MMLU
MMLU (Massive Multitask Language Understanding) is a benchmark for evaluating LLMs through multiple-choice questions. These questions cover 57 subjects such as math, history, law, and ethics. For more information, visit the MMLU GitHub page.
MMLU covers a broad variety and depth of subjects, and is good at detecting areas where a model may lack understanding in a certain topic.
Arguments
There are two optional arguments when using the MMLU benchmark:
- [Optional]
tasks: a list of tasks (MMLUTaskenums), specifying which of the 57 subject areas to evaluate in the language model. By default, this is set to all tasks. Detailed descriptions of theMMLUTaskenum can be found here. - [Optional]
n_shots: the number of "shots" to use for few-shot learning. This is set to 5 by default and cannot exceed this number.
Example
The code below evaluates a custom mistral_7b model (click here to learn how to use ANY custom LLM) and assesses its performance on High School Computer Science and Astronomy using 3-shot learning.
from deepeval.benchmarks import MMLU
from deepeval.benchmarks.tasks import MMLUTask
# Define benchmark with specific tasks and shots
benchmark = MMLU(
tasks=[MMLUTask.HIGH_SCHOOL_COMPUTER_SCIENCE, MMLUTask.ASTRONOMY],
n_shots=3
)
# Replace 'mistral_7b' with your own custom model
benchmark.evaluate(model=mistral_7b)
print(benchmark.overall_score)
The overall_score for this benchmark ranges from 0 to 1, where 1 signifies perfect performance and 0 indicates no correct answers. The model's score, based on exact matching, is calculated by determining the proportion of multiple-choice questions for which the model produces the precise correct letter answer (e.g. 'A') in relation to the total number of questions.
As a result, utilizing more few-shot prompts (n_shots) can greatly improve the model's robustness in generating answers in the exact correct format and boost the overall score.
MMLU Tasks
The MMLUTask enum classifies the diverse range of subject areas covered in the MMLU benchmark.
from deepeval.benchmarks.tasks import MMLUTask
mm_tasks = [MMLUTask.HIGH_SCHOOL_EUROPEAN_HISTORY]
Below is the comprehensive list of all available tasks:
HIGH_SCHOOL_EUROPEAN_HISTORYBUSINESS_ETHICSCLINICAL_KNOWLEDGEMEDICAL_GENETICSHIGH_SCHOOL_US_HISTORYHIGH_SCHOOL_PHYSICSHIGH_SCHOOL_WORLD_HISTORYVIROLOGYHIGH_SCHOOL_MICROECONOMICSECONOMETRICSCOLLEGE_COMPUTER_SCIENCEHIGH_SCHOOL_BIOLOGYABSTRACT_ALGEBRAPROFESSIONAL_ACCOUNTINGPHILOSOPHYPROFESSIONAL_MEDICINENUTRITIONGLOBAL_FACTSMACHINE_LEARNINGSECURITY_STUDIESPUBLIC_RELATIONSPROFESSIONAL_PSYCHOLOGYPREHISTORYANATOMYHUMAN_SEXUALITYCOLLEGE_MEDICINEHIGH_SCHOOL_GOVERNMENT_AND_POLITICSCOLLEGE_CHEMISTRYLOGICAL_FALLACIESHIGH_SCHOOL_GEOGRAPHYELEMENTARY_MATHEMATICSHUMAN_AGINGCOLLEGE_MATHEMATICSHIGH_SCHOOL_PSYCHOLOGYFORMAL_LOGICHIGH_SCHOOL_STATISTICSINTERNATIONAL_LAWHIGH_SCHOOL_MATHEMATICSHIGH_SCHOOL_COMPUTER_SCIENCECONCEPTUAL_PHYSICSMISCELLANEOUSHIGH_SCHOOL_CHEMISTRYMARKETINGPROFESSIONAL_LAWMANAGEMENTCOLLEGE_PHYSICSJURISPRUDENCEWORLD_RELIGIONSSOCIOLOGYUS_FOREIGN_POLICYHIGH_SCHOOL_MACROECONOMICSCOMPUTER_SECURITYMORAL_SCENARIOSMORAL_DISPUTESELECTRICAL_ENGINEERINGASTRONOMYCOLLEGE_BIOLOGY