Designing Better Evals
Build robust evaluation frameworks, benchmarks, graders, and regression tests that meaningfully track model improvement.
Calibrating the room
000 • Loading
Frontier AI Research Summit
First Annual Summit on Model Training and Evaluation
- Attendees
- 1,000+
- Researchers
- 70%
- Speakers
- 50+
Our speakers have trained and evaluated models at the following labs and companies:
OpenAIAnthropicMicrosoftNVIDIAxAIDeepMindOpenAIAnthropicMicrosoftNVIDIAxAIDeepMind
01 About the summitN 01 / 07
The rapid improvement of frontier models has created a new set of questions around how models are trained, evaluated, and improved after pretraining. These questions increasingly sit at the center of model development, but they remain distributed across research communities, industry teams, and infrastructure providers.
FAIR Summit is a gathering focused on model training and evaluation. It brings together researchers, engineers, and practitioners working across reinforcement learning, human and synthetic data, evaluations, alignment, model behavior, and infrastructure.
Our goal is to create a community for the people responsible for improving models to be more trustworthy, reliable, and capable.
Presented by
AI Circle
A network of researchers, operators and founders advancing the frontier of AI through post-training and model evaluation.
02 Workshops11 workshops · call for papers open
LLM-as-a-Judge: What Works, What Breaks
Explore judge calibration, bias, agreement, failure modes, and when human evaluation is still required.
Human Evaluation at Scale
Design better rubrics, train evaluators, measure agreement, improve annotation quality, and build reliable adjudication systems.
From SFT to RL: Designing a Post-Training Pipeline
Understand how supervised fine-tuning, preference optimization, reinforcement learning, and data selection fit together.
Reward Modeling and Verifiable Rewards
Design reward signals, build reward models, prevent reward hacking, and identify where verifiable outcomes can improve training.
Synthetic Data and Training Data Design
Generate, curate, filter, and combine synthetic and human data to create higher-quality training mixtures.
Evals for Agents and Tool Use
Evaluate agents across long-horizon tasks, tool calling, browsing, coding, research, and real-world environments.
Building RL Environments
Design environments, tasks, and feedback loops that produce useful learning signals for increasingly capable agents.
Model Behavior and Failure Analysis
Identify systematic failures, build error taxonomies, evaluate robustness, and understand capability tradeoffs.
Regression Testing for Models
Determine whether a new model or checkpoint is actually better, and detect when improving one capability causes another to degrade.
Safety and Alignment Evals
Evaluate jailbreaks, refusals, over-refusal, misuse, robustness, and other behaviors critical to trustworthy deployment.
03 SpeakersLineup · 2027
Speakers
S/01CONFIRMED
Pranay Manocha
AI Research Scientist
MetaS/02CONFIRMED
Harshay Shah
Research Scientist
Google DeepMindS/03CONFIRMED
Michelangelo Conserva
Research Scientist
GoogleS/04CONFIRMED
Dennis Aumiller
Senior Member of Technical Staff
CohereS/05CONFIRMED
Mariya Vasileva
Senior Research Scientist
MetaS/06CONFIRMED
Auriel Wright
Staff PM, Superintelligence Labs
MetaS/07CONFIRMED
Eujeong Choi
AI Deployment Strategist
Mistral AIS/08CONFIRMED
Albert Avetisian
Forward Deployed TPM
MicrosoftS/09CONFIRMED
Angelos Perivolaropoulos
Research Engineer, TTS & STT
ElevenLabsS/10CONFIRMED
Vanessa Piacente
Global Head of Adoption
ElevenLabsS/11CONFIRMED
Kirk McKeown
Founder & CEO
Carbon ArcS/12CONFIRMED
Ahmed Rashad
Founder & CEO
PerleS/13CONFIRMED
Joséphine Parquet
Head of API Products
VEEDS/14CONFIRMED
David Guo
Co-Founder
MidcenturyS/15CONFIRMED
Sunny Shih
GTM | Account Management
ElevenLabsS/16SOON
To be announced
Revealed in waves · 2027
S/17SOON
To be announced
Revealed in waves · 2027
S/18SOON
To be announced
Revealed in waves · 2027
S/19SOON
To be announced
Revealed in waves · 2027
S/20SOON
To be announced
Revealed in waves · 2027
Speaker Application
If you're doing frontier work in model research, training, or evaluation, apply to get involved.
Tickets · Super Early Bird
Be in the room
Join the researchers training and evaluating frontier models at Meta, Google DeepMind, Cohere, ElevenLabs and more. One pass covers the full summit.
Get your pass →AI Circle members get 50% off any pass with their code
04 PartnersAnnounced soon
Partner with
FAIR Summit
Support the researchers shaping how frontier models are trained and evaluated.
Summit hosts are announced in 2027. AI Circle’s current partners include Prolific, Invisible and Encord.