Evaluating Perplexity on Language Fashions

A language mannequin is a chance distribution over sequences of tokens. Whenever you prepare a language mannequin, you wish to measure how precisely it predicts human language use. This can be a tough activity, and also you want a metric to judge the mannequin. On this article, you’ll be taught in regards to the perplexity metric. Particularly, you’ll be taught:

• What’s perplexity, and the best way to compute it
• Tips on how to consider the perplexity of a language mannequin with pattern information

Let’s get began.

Evaluating Perplexity on Language Fashions
Picture by Lucas Davis. Some rights reserved.

Overview

This text is split into two elements; they’re:

• What Is Perplexity and Tips on how to Compute It
• Consider the Perplexity of a Language Mannequin with HellaSwag Dataset

What Is Perplexity and Tips on how to Compute It

Perplexity is a measure of how properly a language mannequin predicts a pattern of textual content. It’s outlined because the inverse of the geometric imply of the chances of the tokens within the pattern. Mathematically, perplexity is outlined as:

$$
PPL(x_{1:L}) = prod_{i=1}^L p(x_i)^{-1/L} = expbig(-frac{1}{L} sum_{i=1}^L log p(x_i)large)
$$

Perplexity is a operate of a specific sequence of tokens. In observe, it’s extra handy to compute perplexity because the imply of the log chances, as proven within the method above.

Perplexity is a metric that quantifies how a lot a language mannequin hesitates in regards to the subsequent token on common. If the language mannequin is totally sure, the perplexity is 1. If the language mannequin is totally unsure, then each token within the vocabulary is equally probably; the perplexity is the same as the vocabulary measurement. You shouldn’t anticipate perplexity to transcend this vary.

Consider the Perplexity of a Language Mannequin with HellaSwag Dataset

Perplexity is a dataset-dependent metric. One dataset you need to use is HellaSwag. It’s a dataset with prepare, check, and validation splits. It’s accessible on the Hugging Face hub, and you’ll load it with the next code:

Working this code will print the next:

You may see that the validation break up has 10,042 samples. That is the dataset you’ll use on this article. Every pattern is a dictionary. The important thing "activity_label" describes the exercise class, and the important thing "ctx" gives the context that must be accomplished. The mannequin is predicted to finish the sequence by deciding on one of many 4 endings. The important thing "label", with values 0 to three, signifies which ending is right.

With this, you possibly can write a brief code to judge your individual language mannequin. Let’s use a small mannequin from Hugging Face for example:

This code hundreds the smallest GPT-2 mannequin from the Hugging Face Hub. It’s a 124M-parameter mannequin you can simply run on a low-profile laptop. The mannequin and tokenizer are loaded utilizing the Hugging Face transformers library. You additionally load the HellaSwag validation dataset.

Within the for-loop, you tokenize the exercise label and the context. You additionally tokenize every of the 4 endings. Notice that tokenizer.encode() is the strategy for utilizing the tokenizer from the transformers library. It’s completely different from the tokenizer object you used within the earlier article.

Subsequent, for every ending, you run the concatenated enter and ending to the mannequin. The input_ids tensor is a 2D tensor of integer token IDs with the batch dimension 1. The mannequin returns an object, wherein you extract the output logits tensor. That is completely different from the mannequin you constructed within the earlier article as this can be a mannequin object from the transformers library. You may simply swap it along with your educated mannequin object with minor modifications.

GPT-2 is a decoder-only transformer mannequin. It processes the enter with a causal masks. For an enter tensor of form $(1, L)$, the output logits tensor has form $(1, L, V)$, the place $V$ is the vocabulary measurement. The output at place $p$ corresponds to the mannequin’s estimate of the token at place $p+1$, relying on the enter at positions 1 to $p$. Subsequently, you extract the logits beginning at offset $n-1$, the place $n$ is the size of the mixed exercise label and context. You then convert the logits to log chances and compute the typical over the size of every ending.

The worth token_probs[j, token] is the log chance at place j for the token with ID token. The imply log-probability of every token within the ending is used to compute the perplexity. A very good mannequin is predicted to establish the right ending with the bottom perplexity. You may consider a mannequin by counting the variety of right predictions over the whole HellaSwag validation dataset. Whenever you run this code, you will notice the next:

The code prints the perplexity of every ending and marks the right reply with (O) or (!) and the mannequin’s unsuitable prediction with (X). You may see that GPT-2 has a perplexity of 10 to twenty, even for an accurate reply. Superior LLMs can obtain perplexity under 10, even with a a lot bigger vocabulary measurement than GPT-2. Extra necessary is whether or not the mannequin can establish the right ending: the one which naturally completes the sentence. It needs to be the one with the bottom perplexity; in any other case, the mannequin can’t generate the right ending. GPT-2 achieves solely 30% accuracy on this dataset.

You may as well repeat the code with a special mannequin. Listed below are the outcomes:

• mannequin openai-community/gpt2: That is the smallest GPT-2 mannequin with 124M parameters, used within the code above. The accuracy is 3041/10042 or 30.28%
• mannequin openai-community/gpt2-medium: That is the bigger GPT-2 mannequin with 355M parameters. The accuracy is 3901/10042 or 38.85%
• mannequin meta-llama/Llama-3.2-1B: That is the smallest mannequin within the Llama household with 1B parameters. The accuracy is 5731/10042 or 57.07%

Subsequently, it’s pure to see increased accuracy with bigger fashions.

Notice that you shouldn’t evaluate perplexities throughout fashions with vastly completely different architectures. Since perplexity is a metric within the vary of 1 to the vocabulary measurement, it extremely will depend on the tokenizer. You may see the rationale while you evaluate the perplexity within the code above after changing GPT-2 with Llama 3.2 1B: The perplexity is an order of magnitude increased for Llama 3, however the accuracy is certainly higher. It is because GPT-2 has a vocabulary measurement of solely 50,257, whereas Llama 3.2 1B has a vocabulary measurement of 128,256.

Additional Readings

Under are some assets that you could be discover helpful:

Abstract

On this article, you realized in regards to the perplexity metric and the best way to consider the perplexity of a language mannequin with the HellaSwag dataset. Particularly, you realized:

• Perplexity measures how a lot a mannequin hesitates in regards to the subsequent token on common.
• Perplexity is a metric delicate to vocabulary measurement.
• Computing perplexity means computing the geometric imply of the chances of the tokens within the pattern.