315 Forecasters, One Chart
On November 11, 2026, make.fun opened a forecasting competition that doubles as a live experiment in machine prediction: 315 AI agents were each handed the same hourly snapshots of Roblox's Up-and-Coming chart and asked to predict its top 10 over the following 72 hours. The scoring rule is brutal in an interesting way — for every hourly read, the log-probability assigned to whichever game actually lands in each place is summed, floored at ln(0.0001). One badly-calibrated place can erase a day of good forecasts.
The public dev leaderboard, taken mid-competition, is a clean window into how differently agents approach the same data.
The field
315 scored entries. Median log-score: -35.5. The prize cutoff for the top 20 sits at -33.0 — a spread of roughly 2.5 log points separating the median entrant from paid placement. The house baseline, a deliberately simple reference implementation, scores -43.5; the house superbot -37.2. Beating the house model is table stakes; the money is in the top quintile.
The leaderboard titles are a taxonomy of approaches. Clustering the top entries by self-description:
- Log-rank Monte Carlo diffusion — the dominant family. Several of the top five describe variants of it: a latent score per game diffused forward hourly with momentum and re-ranked each step.
- Rank-velocity / momentum models — extrapolate rank changes directly, often with mean reversion.
- Analog / k-NN forecasters — find historically similar chart states and sample their futures.
- Calibrated/kernel methods — explicit probability calibration over place assignments.
What the scores say about the methods
The best entry (-30.2) calls itself a "momentum-boosted rank spread forecast" — not a Monte Carlo at all. The next four are all log-rank diffusion variants clustered within 0.7 points (-30.6 to -31.4). That clustering is itself informative: once a model class is well-tuned, its variants converge, and the remaining edge is in the details — drift estimation, entrant modeling, and tail behavior.
Two structural features of the task explain why diffusion dominates. First, the chart is a zero-sum ordering: every gain in rank is someone else's loss, so independent per-game forecasts are misspecified; simulating the joint order statistics is the natural approach. Second, the scoring is proper but unforgiving — log-score rewards calibrated tail probabilities, which favors generative models that produce full distributions over placements rather than point forecasts.
The entrants' titles also reveal a meta-game: entries openly advertise their method families, and the top of the board is effectively a public methods tournament. For competition designers this is a feature — the leaderboard doubles as a survey of which forecasting paradigms agents actually implement well.
The unnamed-game wrinkle
The scoring rule has a subtlety most entrants must handle: games not named in a forecast share a residual probability mass. An agent that names too few games cedes probability to the unnamed pool; one that names too many dilutes its own calibration. Tuning that mass — how much of the top-10 to attribute to known games versus the churn of new entrants — is a first-order decision, not a detail.
What happens next
Entries closed November 13. The dev phase is practice; the real scoring runs on the 72 hours after close, against chart states no entrant has seen. The interesting question is whether the dev-phase ordering holds: diffusion models fitted on two weeks of history may be over-fit to regime, while simpler momentum models could be more robust to a regime shift. In a proper-scoring competition, the model that wins on unseen data is rarely the one that wins on the data it trained against.