DROP
DROP (Discrete Reasoning Over Paragraphs) is a benchmark designed to evaluate language models' advanced reasoning capabilities through complex question answering tasks. It encompasses over 9500 intricate challenges that demand numerical manipulations, multi-step reasoning, and the interpretation of text-based data. For more insights and access to the dataset, you can read the original DROP paper here.
info
DROP challenges models to process textual data, perform numerical reasoning tasks such as addition, subtraction, and counting, and also to comprehend and analyze text to extract or infer answers from paragraphs about NFL and history.
Arguments
There are two optional arguments when using the DROP benchmark:
- [Optional]
tasks: a list of tasks (DROPTaskenums), which specifies the subject areas for model evaluation. By default, this is set to all tasks. The list ofDROPTaskenums can be found here. - [Optional]
n_shots: the number of "shots" to use for few-shot learning. This is set to 5 by default and cannot exceed 5.
note
Notice unlike BIGBenchHard, there is no CoT prompting for the DROP benchmark.
Example
The code below assesses a custom mistral_7b model (click here to learn how to use ANY custom LLM) on HISTORY_1002 and NFL_649 in DROP using 3-shot prompting.
from deepeval.benchmarks import DROP
from deepeval.benchmarks.tasks import DROPTask
# Define benchmark with specific tasks and shots
benchmark = DROP(
tasks=[DROPTask.HISTORY_1002, DROPTask.NFL_649],
n_shots=3
)
# Replace 'mistral_7b' with your own custom model
benchmark.evaluate(model=mistral_7b)
print(benchmark.overall_score)
The overall_score for this benchmark ranges from 0 to 1, where 1 signifies perfect performance and 0 indicates no correct answers. The model's score, based on exact matching, is calculated by determining the proportion of questions for which the model produces the precise correct answer (e.g. '3' or ‘John Doe’) in relation to the total number of questions.
As a result, utilizing more few-shot prompts (n_shots) can greatly improve the model's robustness in generating answers in the exact correct format and boost the overall score.
DROP Tasks
The DROPTask enum classifies the diverse range of categories covered in the DROP benchmark.
from deepeval.benchmarks.tasks import DROPTask
drop_tasks = [NFL_649]
Below is the comprehensive list of available tasks:
NFL_649HISTORY_1418HISTORY_75HISTORY_2785NFL_227NFL_2684HISTORY_1720NFL_1333HISTORY_221HISTORY_2090HISTORY_241HISTORY_2951HISTORY_3897HISTORY_1782HISTORY_4078NFL_692NFL_104NFL_899HISTORY_2641HISTORY_3628HISTORY_488NFL_46HISTORY_752HISTORY_1262HISTORY_4118HISTORY_1425HISTORY_460NFL_1962HISTORY_1308NFL_969NFL_317HISTORY_370HISTORY_1837HISTORY_2626NFL_987NFL_87NFL_2996NFL_2082HISTORY_23HISTORY_787HISTORY_405HISTORY_1401HISTORY_835HISTORY_565HISTORY_1998HISTORY_2176HISTORY_1196HISTORY_1237NFL_244HISTORY_3109HISTORY_1414HISTORY_2771HISTORY_3806NFL_1233NFL_802HISTORY_2270NFL_578HISTORY_1313NFL_1216NFL_256HISTORY_3356HISTORY_1859HISTORY_3103HISTORY_2991HISTORY_2060HISTORY_1408HISTORY_3042NFL_1873NFL_1476NFL_524HISTORY_1316HISTORY_1456HISTORY_104HISTORY_1275HISTORY_1069NFL_3270NFL_1222HISTORY_2704HISTORY_733NFL_1981NFL_592HISTORY_920HISTORY_951NFL_1136HISTORY_2642HISTORY_1065HISTORY_2976NFL_669HISTORY_2846NFL_1996HISTORY_2848NFL_3285HISTORY_2789HISTORY_3722HISTORY_514HISTORY_869HISTORY_2857HISTORY_3237NFL_563HISTORY_990HISTORY_2961NFL_3387HISTORY_124HISTORY_2898HISTORY_2925HISTORY_2788HISTORY_632HISTORY_2619HISTORY_3278NFL_749HISTORY_3726NFL_1096NFL_1207HISTORY_3079HISTORY_2939HISTORY_3581NFL_2777HISTORY_3873HISTORY_1731HISTORY_426NFL_1478HISTORY_3106NFL_1498NFL_3133HISTORY_3345NFL_503HISTORY_801NFL_2931NFL_2482HISTORY_1945NFL_2262HISTORY_3735HISTORY_1151NFL_2415HISTORY_607HISTORY_724HISTORY_1284HISTORY_494NFL_3571NFL_1307HISTORY_2847HISTORY_2650NFL_1586NFL_2478HISTORY_1276NFL_540NFL_894NFL_1492HISTORY_3265HISTORY_686HISTORY_2546NFL_2396HISTORY_2001HISTORY_1793HISTORY_2014HISTORY_2732HISTORY_2927NFL_1195HISTORY_1650NFL_2077HISTORY_3036HISTORY_495HISTORY_3048HISTORY_912HISTORY_936NFL_1329HISTORY_1928HISTORY_3303HISTORY_2199HISTORY_1169HISTORY_115HISTORY_2575HISTORY_1340NFL_988HISTORY_423HISTORY_1959NFL_29HISTORY_2867NFL_2191HISTORY_3754NFL_1021NFL_2269HISTORY_4060HISTORY_1773HISTORY_2757HISTORY_468HISTORY_10HISTORY_2151HISTORY_725NFL_858NFL_122HISTORY_591HISTORY_2948HISTORY_2829HISTORY_4034HISTORY_3717HISTORY_187HISTORY_1995NFL_1566HISTORY_685HISTORY_296HISTORY_1876HISTORY_2733HISTORY_325HISTORY_1898HISTORY_1948NFL_1838HISTORY_3993HISTORY_3366HISTORY_79NFL_2584HISTORY_3241HISTORY_1879HISTORY_2004HISTORY_4050NFL_2668HISTORY_3683HISTORY_836HISTORY_783HISTORY_2953HISTORY_1723NFL_378HISTORY_4137HISTORY_200HISTORY_502HISTORY_175HISTORY_3341HISTORY_2196HISTORY_9NFL_2385NFL_1879HISTORY_1298NFL_2272HISTORY_2170HISTORY_4080HISTORY_3669HISTORY_3647HISTORY_586NFL_1454HISTORY_2760HISTORY_1498HISTORY_1415HISTORY_2361NFL_915HISTORY_986HISTORY_1744HISTORY_1802HISTORY_3075HISTORY_2412NFL_832HISTORY_3435HISTORY_1306HISTORY_3089HISTORY_1002HISTORY_3949HISTORY_1445HISTORY_254HISTORY_991HISTORY_2530HISTORY_447HISTORY_2661HISTORY_1746HISTORY_347NFL_3009HISTORY_1814NFL_3126HISTORY_972NFL_2528HISTORY_2417NFL_1184HISTORY_59HISTORY_1811HISTORY_3115HISTORY_71HISTORY_1935HISTORY_2944HISTORY_1019HISTORY_887HISTORY_533NFL_3195HISTORY_3615HISTORY_4007HISTORY_2950NFL_1672HISTORY_2897HISTORY_1887HISTORY_2836NFL_3356HISTORY_1828HISTORY_3714NFL_2054HISTORY_2709NFL_1883NFL_2042HISTORY_2162NFL_2197NFL_2369HISTORY_2765HISTORY_2021NFL_1152HISTORY_2957HISTORY_1863HISTORY_2064HISTORY_4045HISTORY_3058NFL_153HISTORY_1074HISTORY_159HISTORY_455HISTORY_761HISTORY_1552NFL_1769NFL_880NFL_2234NFL_2995NFL_2823HISTORY_2179HISTORY_1891HISTORY_2474HISTORY_3062NFL_490HISTORY_1416HISTORY_415HISTORY_2609NFL_1618HISTORY_3749HISTORY_68HISTORY_4011NFL_2067NFL_610NFL_2568NFL_1689HISTORY_2044HISTORY_1844HISTORY_3992NFL_716NFL_825HISTORY_806NFL_194HISTORY_2970HISTORY_2878NFL_1652HISTORY_3804HISTORY_90NFL_16HISTORY_515HISTORY_1954HISTORY_2011HISTORY_2832HISTORY_228NFL_2907HISTORY_2752HISTORY_1352HISTORY_3244HISTORY_2941HISTORY_1227HISTORY_130HISTORY_3587HISTORY_69HISTORY_2676NFL_1768NFL_995HISTORY_809HISTORY_941HISTORY_3264NFL_1264HISTORY_1012HISTORY_1450HISTORY_1048NFL_719HISTORY_2762HISTORY_2086HISTORY_1259NFL_1240HISTORY_2234HISTORY_2102HISTORY_688NFL_2114HISTORY_1459HISTORY_1043HISTORY_3609NFL_1223HISTORY_417HISTORY_1884HISTORY_2390NFL_2671HISTORY_2298HISTORY_659HISTORY_459HISTORY_1542NFL_1914HISTORY_1258HISTORY_2164HISTORY_2777NFL_1304HISTORY_4049HISTORY_1423NFL_2994HISTORY_2814HISTORY_2187HISTORY_3280HISTORY_794NFL_3342HISTORY_2153HISTORY_1708NFL_1540HISTORY_92HISTORY_1907NFL_290NFL_1167HISTORY_2885HISTORY_2258HISTORY_1940HISTORY_2380NFL_1245HISTORY_3552HISTORY_534NFL_1193NFL_264NFL_275HISTORY_1042NFL_1829NFL_2571NFL_296NFL_199HISTORY_2434NFL_1486HISTORY_107HISTORY_371NFL_1361HISTORY_1212NFL_2036NFL_913HISTORY_2886HISTORY_2737HISTORY_487NFL_1516NFL_2894HISTORY_3692NFL_496HISTORY_2707HISTORY_655NFL_286HISTORY_13HISTORY_556NFL_962HISTORY_1517HISTORY_1130NFL_624NFL_2125NFL_1670HISTORY_512NFL_1515HISTORY_893HISTORY_1233HISTORY_3116HISTORY_544HISTORY_3807HISTORY_2088NFL_2601HISTORY_1952HISTORY_131HISTORY_3662HISTORY_883HISTORY_2949HISTORY_1965NFL_778HISTORY_2047HISTORY_4009HISTORY_520HISTORY_1748HISTORY_154NFL_493NFL_187HISTORY_1578NFL_1344NFL_3489NFL_246NFL_336NFL_3396NFL_816NFL_1390HISTORY_3363HISTORY_4002HISTORY_4141NFL_1378HISTORY_476NFL_477NFL_1471NFL_3420HISTORY_227HISTORY_3859NFL_715HISTORY_283HISTORY_1943HISTORY_1665HISTORY_1860NFL_2387HISTORY_3253HISTORY_2766HISTORY_671HISTORY_720HISTORY_3141HISTORY_1373HISTORY_2453HISTORY_3608HISTORY_343NFL_2918HISTORY_3866HISTORY_2818NFL_2330NFL_2636NFL_1553HISTORY_1082HISTORY_3900NFL_2202HISTORY_3404HISTORY_103NFL_2409NFL_1412HISTORY_2188NFL_3386NFL_1503NFL_1288NFL_2151NFL_1743HISTORY_2815HISTORY_2671HISTORY_1892NFL_613HISTORY_1356HISTORY_2363HISTORY_424HISTORY_3438HISTORY_148NFL_3290NFL_663HISTORY_732HISTORY_3092HISTORY_408NFL_3460HISTORY_2809HISTORY_530HISTORY_3588HISTORY_1853HISTORY_513HISTORY_918HISTORY_908HISTORY_2869HISTORY_1125HISTORY_796HISTORY_1601HISTORY_1250HISTORY_1092HISTORY_351HISTORY_2142NFL_2255HISTORY_3533HISTORY_3400HISTORY_2456HISTORY_3164HISTORY_2339NFL_2297HISTORY_3105NFL_1596NFL_2893HISTORY_539NFL_1332HISTORY_208NFL_350NFL_2645HISTORY_2921HISTORY_1167HISTORY_2892HISTORY_791NFL_3222NFL_1789NFL_180NFL_3594HISTORY_3143NFL_824NFL_2034