Newmind AI LLM Evaluation Leaderboard

Evaluation Categories

1. ⚔️ Arena-Hard-Auto: Competitive Benchmarking at Scale

Arena-Hard-Auto is a cutting-edge automatic evaluation framework tailored for instruction-tuned Large Language Models (LLMs). Leveraging a tournament-style evaluation methodology, it pits models against each other in head-to-head matchups, with performance rankings determined via the Elo rating system—a method proven to align closely with human judgment, as evidenced in Chatbot Arena benchmarks. This evaluation suite is grounded in real-world use cases, benchmarking models across 11 diverse legal tasks, encompassing an extensive Turkish legal question-answer pairs.

Key Evaluation Pillars

• Automated Judging Relies on specialized judge models to assess and determine the winner in each model-versus-model comparison. Includes dynamic system prompt adaptation to ensure context-aware evaluation based on the specific domain of the query.
• Win Probability Estimation Computes the probability of victory for each model in a matchup using a logistic regression model, offering a probabilistic understanding of comparative strength.
• Skill Rating (Elo Score) Utilizes the Elo rating system to provide a robust measurement of each model’s skill level relative to its competitors, ensuring a competitive and evolving leaderboard.
• Win Rate Measures a model’s dominance by calculating the proportion of head-to-head victories, serving as a direct indicator of real-world performance across the benchmark suite. Arena-Hard-Auto offers a fast, scalable, and reliable method to benchmark LLMs, combining quantitative rigor with realistic interaction scenarios—making it an indispensable tool for understanding model capabilities in high-stakes legal and instructional domains.

To reproduce the results, you can use this repository: https://github.com/lmarena/arena-hard-auto

2. 🔄 EvalMix

EvalMix is a comprehensive evaluation pipeline designed to assess the performance of language models across multiple dimensions. This hybrid evaluation tool automatically analyzes model outputs, computes various semantic, LLM-based, and lexical metrics, and visualizes the results. EvalMix offers the following features:

Comprehensive Evaluation Metrics

• LLM-as-a-Judge: Uses large language models—primarily GPT variants—to evaluate the accuracy, coherence, and relevance of generated responses.
• Lexical Metrics: Calculates traditional NLP metrics such as BLEU, ROUGE-1, ROUGE-2, and ROUGE-L, along with modern metrics like BERTScore (precision, recall, F1) and cosine similarity.
• Comparative Analysis: Enables performance comparison between multiple models on the same dataset.
• Cosine Similarity (Turkish): Assesses performance in the Turkish language using Turkish-specific embedding models.
• Cosine Similarity (Multilingual): Measures multilingual performance using language-agnostic embeddings.

Generation Configuration for Evaluation

The following configuration parameters are used for model generation during evaluation:

{
  "num_samples": 1100,
  "random_seed": 42,
  "temperature": 0.0,
  "max_completion_tokens": 1024
}

3. ⚡ Light-Eval

LightEval is a fast and modular framework designed to evaluate Large Language Models (LLMs) across a diverse range of tasks. It provides a comprehensive performance analysis by benchmarking models on academic, logical, scientific, and mathematical reasoning challenges.

Evaluation Tasks and Objectives LightEval assesses model capabilities using the following six core tasks:

• MMLU (5-shot): Evaluates general knowledge and reasoning skills across academic and professional disciplines.(Proffesional-law task only)
• TruthfulQA (0-shot): Measures the model's ability to generate accurate and truthful responses.
• Winogrande (5-shot): Tests commonsense reasoning and logical inference abilities.
• Hellaswag (10-shot): Assesses the coherence and logical consistency of model predictions based on contextual cues.
• GSM8k (5-shot): Evaluates step-by-step mathematical reasoning and problem-solving capabilities.
• ARC (25-shot): Tests scientific reasoning and the ability to solve science-based problems.

Overall Score Calculation The overall performance score of a model is computed using the average of the six evaluation tasks: LightEval Overall Score = (MMLU_proffesional_law + TruthfulQA + Winogrande + Hellaswag + GSM8k + ARC) / 6

To reproduce the results, you can use this repository: https://github.com/huggingface/lighteval

4. 🐍 Snake-Eval

An evaluation framework where models play the classic Snake game, competing to collect apples while avoiding collisions. Starting from random positions, models must guide their snakes using step-by-step reasoning, with performance measured as an Elo rating. This tests problem-solving ability, spatial awareness, and logical thinking in a challenging environment.

Sample Prompt:

You are controlling a snake in a multi-apple Snake game. The board size is 10x10. 
Normal X,Y coordinates are used. Coordinates range from (0,0) at bottom left to (9,9) at top right.

Apples at: (9, 6), (0, 2), (5, 9), (1, 7), (9, 7)

Your snake ID: 1 which is currently positioned at (5, 1)

Enemy snakes positions:
* Snake #2 is at position (7, 1) with body at []

Board state:
9 . . . . . A . . . .
8 . . . . . . . . . .
7 . A . . . . . . . A
6 . . . . . . . . . A
5 . . . . . . . . . .
4 . . . . . . . . . .
3 . . . . . . . . . .
2 A . . . . . . . . .
1 . . . . . 1 . 2 . .
0 . . . . . . . . . .
  0 1 2 3 4 5 6 7 8 9

--Your last move information:--
Direction: LEFT
Rationale: I'm noticing that (0,2) is the closest apple from our head at (6,1). 
Moving LEFT starts guiding us toward this apple while avoiding the enemy snake.
Strategy: Continue left and then maneuver upward to reach the apple at (0,2).
--End of your last move information.--

Rules:<
1) If you move onto an apple, you grow and gain 1 point.
2) If you hit a wall, another snake, or yourself, you die.
3) The goal is to have the most points by the end.

Decreasing x coordinate: LEFT, increasing x coordinate: RIGHT
Decreasing y coordinate: DOWN, increasing y coordinate: UP

Provide your reasoning and end with: UP, DOWN, LEFT, or RIGHT.

To reproduce the results, you can use this repository: https://github.com/gkamradt/SnakeBench

5. 📚 Retrieval

An evaluation system designed to assess Retrieval-Augmented Generation (RAG) capabilities. It measures how well models can:

• Retrieve relevant information from a knowledge base
• Generate accurate and contextually appropriate responses
• Maintain coherence between retrieved information and generated text
• Handle real-world information retrieval scenarios

Retrieval Metrics

• RAG Success Rate: Percentage of successful retrievals
• Maximum Correct References: Upper limit for correct retrievals per query
• Hallucinated References: Number of irrelevant documents retrieved
• Missed References: Number of relevant documents not retrieved

LLM Judge Evaluation Metrics

• Legal Reasoning: Assesses the model's ability to understand and apply legal concepts
• Factual Legal Accuracy: Measures accuracy of legal facts and references
• Clarity & Precision: Evaluates clarity and precision of responses
• Factual Reliability: Checks for biases and factual accuracy
• Fluency: Assesses language fluency and coherence
• Relevance: Measures response relevance to the query
• Content Safety: Evaluates content safety and appropriateness

Judge Model: nvidia/Llama-3_1-Nemotron-Ultra-253B-v1

RAG Score Calculation The RAG Score is a comprehensive metric that combines multiple performance indicators using dynamic normalization across all models. The formula weights different aspects of retrieval performance:

Formula Components:

• RAG Success Rate (0.9 weight): Direct percentage of successful retrievals (higher is better)
• Normalized False Positives (0.9 weight): Hallucinated references, min-max normalized (lower is better)
• Normalized Max Correct References (0.1 weight): Maximum correct retrievals, min-max normalized (higher is better)
• Normalized Missed References (0.1 weight): Relevant documents not retrieved, min-max normalized (lower is better)

Final Score Formula:

RAG Score = (0.9 × RAG_success_rate + 0.9 × norm_false_positives + 
             0.1 × norm_max_correct + 0.1 × norm_missed_refs) ÷ 2.0

6. 👥 Human Arena

Human Arena is a community-driven evaluation platform where language models are compared through human preferences and voting. This evaluation method captures real-world user preferences and provides insights into model performance from a human perspective.

Evaluation Methodology

• Head-to-Head Comparisons: Models are presented with the same prompts and their responses are compared by human evaluators
• ELO Rating System: Similar to chess rankings, models gain or lose rating points based on wins, losses, and ties against other models
• Community Voting: Real users vote on which model responses they prefer, ensuring diverse evaluation perspectives
• Blind Evaluation: Evaluators see responses without knowing which model generated them, reducing bias

Key Metrics

• ELO Rating: Overall skill level based on tournament-style matchups (higher is better)
• Win Rate: Percentage of head-to-head victories against other models
• Wins/Losses/Ties: Direct comparison statistics showing model performance
• Total Games: Number of evaluation rounds completed
• Votes: Community engagement and evaluation volume
• Provider & Technical Details: Infrastructure and model configuration information

Evaluation Criteria Human evaluators consider multiple factors when comparing model responses:

• Response Quality: Accuracy, completeness, and relevance of answers
• Communication Style: Clarity, coherence, and appropriateness of language
• Helpfulness: How well the response addresses the user's needs
• Safety & Ethics: Adherence to safety guidelines and ethical considerations
• Creativity & Originality: For tasks requiring creative or innovative thinking

Human Arena provides a complementary perspective to automated benchmarks, capturing nuanced human preferences that traditional metrics might miss. This evaluation is particularly valuable for understanding how models perform in real-world conversational scenarios.

7. 🧩 Structured Outputs

Structured Outputs evaluation assesses models' ability to generate properly formatted, structured responses with accurate field extraction and semantic understanding. This benchmark tests how well language models can parse, understand, and extract specific information from documents while maintaining semantic coherence.

Evaluation Methodology: Models are evaluated on their ability to extract structured information from Turkish legal documents. The evaluation uses advanced semantic similarity scoring with Turkish-specific embedding models for accurate assessment.

Technical Implementation:

• Embedding Model: Primary evaluation uses newmindai/TurkEmbed4Retrieval for Turkish-specific semantic understanding
• Similarity Threshold: 0.75 cosine similarity threshold for field matching
• Ground Truth Comparison: MongoDB-stored ground truth data with pre-computed embeddings

Evaluation Metrics:

• Overall: Combined overall performance metric that averages Semantic understanding and Response Format success ratio
• Semantic: Measures semantic understanding and coherence of extracted information using cosine similarity (corresponds to overall_score in scoring)
• Response Format: Success ratio showing successful JSON extractions vs total attempts (success_response/sample_count)
• Name: Accuracy in extracting and identifying name fields from legal documents (20% weight)
• Document Date: Accuracy in date field extraction with multiple format support (20% weight)
• Document Note: Performance in extracting document annotation information using semantic similarity (20% weight)
• From: Performance in extracting source/sender information as lists with semantic matching (20% weight)
• To: Accuracy in extracting destination/recipient information as lists with semantic matching (20% weight)

Scoring Algorithm: The evaluation uses a sophisticated multi-level scoring system:

1. String Fields (name, document_note): Turkish embedding similarity with 0.75 threshold using newmindai/TurkEmbed4Retrieval
2. Date Fields (document_date): Exact date matching with multiple format parsing support
3. List Fields (from, to): One-way similarity from ground truth to predictions using semantic matching
4. Overall Score Calculation: Overall = (Semantic + Response Format) / 2
5. Field Weights: Each extraction field (name, document_date, document_note, from, to) contributes equally with 20% weight to the semantic score