Designing Better Evals
Build robust evaluation frameworks, benchmarks, graders, and regression tests that meaningfully track model improvement.
Calibrating the room
000 • Loading
Frontier AI Research Summit
First Annual Summit on Model Training and Evaluation
Our speakers have trained and evaluated models at the following labs and companies:
OpenAIAnthropicMicrosoftNVIDIAxAIDeepMindOpenAIAnthropicMicrosoftNVIDIAxAIDeepMind
01 About the summitN 01 / 07
The rapid improvement of frontier models has created a new set of questions around how models are trained, evaluated, and improved after pretraining. These questions increasingly sit at the center of model development, but they remain distributed across research communities, industry teams, and infrastructure providers.
FAIR Summit is a gathering focused on model training and evaluation. It brings together researchers, engineers, and practitioners working across reinforcement learning, human and synthetic data, evaluations, alignment, model behavior, and infrastructure.
Our goal is to create a community for the people responsible for improving models to be more trustworthy, reliable, and capable.
Presented by
AI Circle
A network of researchers, operators and founders advancing the frontier of AI through post-training and model evaluation.
02 Workshops11 workshops · call for papers open
LLM-as-a-Judge: What Works, What Breaks
Explore judge calibration, bias, agreement, failure modes, and when human evaluation is still required.
Human Evaluation at Scale
Design better rubrics, train evaluators, measure agreement, improve annotation quality, and build reliable adjudication systems.
From SFT to RL: Designing a Post-Training Pipeline
Understand how supervised fine-tuning, preference optimization, reinforcement learning, and data selection fit together.
Reward Modeling and Verifiable Rewards
Design reward signals, build reward models, prevent reward hacking, and identify where verifiable outcomes can improve training.
Synthetic Data and Training Data Design
Generate, curate, filter, and combine synthetic and human data to create higher-quality training mixtures.
Evals for Agents and Tool Use
Evaluate agents across long-horizon tasks, tool calling, browsing, coding, research, and real-world environments.
Building RL Environments
Design environments, tasks, and feedback loops that produce useful learning signals for increasingly capable agents.
Model Behavior and Failure Analysis
Identify systematic failures, build error taxonomies, evaluate robustness, and understand capability tradeoffs.
Regression Testing for Models
Determine whether a new model or checkpoint is actually better, and detect when improving one capability causes another to degrade.
Safety and Alignment Evals
Evaluate jailbreaks, refusals, over-refusal, misuse, robustness, and other behaviors critical to trustworthy deployment.
03 SpeakersLineup · 2027
Speakers
S/01CONFIRMED
Kirk McKeown
Founder & CEO
Carbon ArcS/02CONFIRMED
Ahmed Rashad
Founder & CEO
PerleS/03CONFIRMED
Vanessa Piacente
Global Head of Adoption
ElevenLabsS/04CONFIRMED
David Guo
Co-Founder
MidcenturyS/05CONFIRMED
Sunny Shih
GTM | Account Management
ElevenLabsS/06CONFIRMED
Albert Avetisian
Forward Deployed TPM
MicrosoftS/07CONFIRMED
Joséphine Parquet
Head of API Products
VEEDS/08SOON
To be announced
Revealed in waves · 2027
S/09SOON
To be announced
Revealed in waves · 2027
S/10SOON
To be announced
Revealed in waves · 2027
Speaker Application
If you're doing frontier work in model research, training, or evaluation, apply to get involved.
04 PartnersAnnounced soon
Partner with
FAIR Summit
Support the researchers shaping how frontier models are trained and evaluated.
Summit hosts are announced in 2027. AI Circle’s current partners include Prolific, Invisible and Encord.