Benchlab is FDL’s IniTiative to maintain SOTA in ML FOR science
Why FDL Benchlab?
Benchlab is unlike any other traditional machine learning competition.
No longer limited by evaluation against static historical datasets, benchmarking is reimagined.
Level up and test your solutions against the best in a live operational context.
Benchlab bridges the gap between model development and operational deployment by challenging participants to predict future data using the live Benchlab platform.
While recent advances in AI have enabled new predictive methods, operationalizing models remains a challenge. This tournament format aims to test submitted workflows in increasingly realistic operational scenarios, where the emphasis is on inference with guaranteed zero data leakage.
The primary goal is to improve critical predictive capabilities across the board by using the community to maintain efficacy. Submissions will be tested against real-world, real-time data to benchmark performance across models and against physics-based baselines.
Onboarding in Benchlab is simple. All hardware and software environments are predefined and shared as templates. Participants ensure their inference pipeline is containerized and compatible, and after a short pre-deployment phase, with code updates permitted, submissions are frozen and inference goes live. Progress through the evaluation process is tracked through a live leaderboard, not just in terms of metrics, but also tangible predictions!
Current Competition
Solar Wind Predication AI Tournament
Forecast the solar wind speed 3 days ahead and help protect our critical modern infrastructure, satellites and Artemis astronauts. Evaluated on live real world data, you know where you stand. Find out more about this competition, including participation details, pre-registration and prize pots below.
