Player Name
Affiliation
Level 1: Hyperparameter Optimization
Tune Learning Rate and Batch Size. Find the peak of the accuracy landscape in at most 10 evaluations.
AutoML System Active
GP-Bayesian Optimization search in progress...
Tap inside the grid to evaluate the hyperparameters
Stage Leaderboard
| # | LR | BS | Acc |
|---|---|---|---|
| No evaluations yet. Click the grid to start. | |||
| # | LR | BS | Acc | Reasoning / Strategy |
|---|---|---|---|---|
| AutoML has not run yet. Complete your evaluations to start. | ||||
About Level 1: Logarithmic Hyperparameter Search & Bayesian Optimization
Why log10 for Learning Rate? In machine learning, learning rates span several orders of magnitude (from 0.00001 up to 0.1). Searching on a log10 scale ($10^{-5}$ to $10^{-1}$) allows equal search resolution across each power of ten (e.g. $10^{-5}$, $10^{-4}$, $10^{-3}$, $10^{-2}$, $10^{-1}$) rather than clustering only at high values.
AutoML Baseline (GP-UCB): You are competing against a Bayesian Optimization agent. It models the unknown accuracy landscape with a Gaussian Process (GP) surrogate model and uses the Upper Confidence Bound (UCB) acquisition function to balance exploration (high uncertainty) and exploitation (refining near the current best candidate).
Level 2: Neural Architecture Search
Design a **Transformer Model** to achieve the lowest validation perplexity. Choose its components (attention type, feed-forward type) and how they are connected (connectivity pattern) across a number of layers. You and the AutoML system each get a shared budget of 32 training epochs to spend across as many architectures as you like.
AutoML System Active
Searching architectures...
Architecture Configuration
Transformer Stack Visualizer
Stage Leaderboard
About the AutoML Baseline: Successive Halving
The AutoML agent searches architectures using Successive Halving: it samples 8 random architectures and trains all of them for just 1 epoch, keeps the best half, doubles the epoch budget for the survivors, and repeats until a single architecture remains trained on 8 epochs — spending its full 32-epoch budget (8×1 + 4×2 + 2×4 + 1×8 = 32) across rungs. This is fast, but it can be fooled by architectures that need a few epochs to "warm up": a Dense-connectivity model may look weak after just 1 epoch and get eliminated early, even though it would have won if trained on the full budget. Keep that in mind when designing your own architecture!
Level 3: CASH Pipeline Optimization
Optimize classification accuracy on a simulated image classification pipeline in at most 15 evaluations. Choose the three pipeline components (Data Augmentation, Model Architecture, and Post-Processing) and tune their sub-hyperparameters. Be careful: the optimal pipeline depends on the randomized dataset characteristics!
AutoML System Active
Tuning pipeline classifiers...
Pipeline Configuration
Pipeline Accuracy Chart
Stage Leaderboard
| # | Pipeline | Configuration | Acc |
|---|---|---|---|
| No evaluations yet. Configure components and evaluate. | |||
| # | Pipeline | Configuration | Acc | Reasoning / Strategy |
|---|---|---|---|---|
| AutoML has not run yet. Complete your evaluations to start. | ||||
About the AutoML Baseline: CASH (Combined Algorithm Selection and Hyperparameter Optimization)
You are competing against a CASH optimization framework. Selecting the best pipeline is a challenging search space containing both categorical choices (which model, which augmentation policy) and continuous values (learning rates, temperatures). The AutoML system uses sequential model-based optimization to navigate these hierarchical hyperparameters. It updates its internal surrogate model after each trial, allowing it to quickly identify and discard poor pipeline configurations and home in on the optimal component combinations.
Bonus Level: LoRA HPO under Quantized Evaluation
Real trial data from LoRA fine-tuning runs across quantization levels (2-bit, 4-bit, 8-bit, 16-bit). Choose 8 configurations to start your bracket, then promote surviving candidates across quantization rungs. Efficiency Bonus: Promoting fewer configurations in subsequent rounds saves compute and earns extra bonus points! Click any column header to sort configurations.
Successive Halving Bracket
Lower Loss = BetterSelect 1 to 8 of the 50 available LoRA configurations to bring into your bracket.
| # | LR | Decay | Dropout | Rank | Weight Decay | 2-bit Loss ↓ | 4-bit Loss ↓ | 8-bit Loss ↓ | 16-bit Loss ↓ | Status | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Loading LoRA configurations... | |||||||||||
About This Bonus Level: Multi-Fidelity HPO on Real Data
This bracket mirrors Successive Halving on real LoRA fine-tuning trial data across quantization levels (2-bit, 4-bit, 8-bit, 16-bit), where you decide how aggressively to prune configurations. The performance objective is Validation Loss (Lower is Better) — the best configurations achieve loss around ~0.35, whereas unsuccessful/diverged runs have loss > 4.80. Lower-bit evaluations (2-bit, 4-bit, 8-bit) serve as cheap proxy signals for the 16-bit full result. Standard successive halving uses 64 compute credits (8×2 + 4×4 + 2×8 + 1×16 = 64). Promoting fewer configurations saves compute credits and earns an Efficiency Bonus (up to +500+ extra points). After crowning a champion, the level reveals the true full-precision ranking of your sampled configs.