Everything We Tested (And What Actually Works)

The Baseline: 53.7%

Our core model is a 25-state Markov chain. Every batter and pitcher gets a 25×25 transition probability matrix (we call them MSIs — Markov Scoring Indices) built from 10 seasons of play-by-play data. To predict a game, we simulate it 10,000 times using the actual lineup's MSIs.

On the full 2023 season (2,430 games), the Markov chain alone picks the winner 53.7% of the time. That's better than a coin flip, but not by much.

The question: can we do better?

Attempt #1: Parameter Tuning 🔧

Result: ❌ No improvement

The Markov model has several parameters: anchor weights (how much to pull players toward league average), pitcher matchup weight, and home advantage. We ran a 59-configuration parameter sweep.

On a 200-game sample, the best configuration hit 58.5%. We were ecstatic — until we validated on the full 2,430 games.

Every single "winning" configuration regressed to the baseline. The model's architecture matters more than its parameters.

Lesson: Small samples lie. Always validate on full seasons. Baseball has too much variance for 200-game tests to mean anything.

Attempt #2: Elo Ratings 📊

Result: ✅ Major improvement

Instead of simulating individual at-bats, Elo assigns each team a dynamic rating that updates after every game. It captures things the Markov chain can't: defense, baserunning, coaching quality, bullpen depth, roster construction, and momentum.

The Elo system alone outperforms Markov by over 2 percentage points. Blending them 50/50 captures the best of both — Elo's team-level signal plus Markov's matchup-specific granularity.

Lesson: Team strength matters more than individual matchups for picking winners. The best hitter in baseball still makes an out 70% of the time.

Attempt #3: Bullpen Modeling 🐂

Result: ❌ No improvement

We built composite bullpen MSIs from 886,290 relief at-bats across all 30 teams. The idea: switch from starter to bullpen MSI in the 6th or 7th inning for more realistic late-game simulation.

Bullpen-adjusted Markov accuracy: 53.0%. Worse than baseline.

Lesson: Composite bullpen MSIs average over too many different relievers and situations. A team's "average reliever" isn't who they actually use in high-leverage spots.

Attempt #4: Park Factors 🏟️

Result: ⚠️ Inconsistent

We calculated park factors from 7,292 games (2023-2025). The physics are real — Coors Field boosts scoring by 27%, T-Mobile Park suppresses it by 13%.

Individual parks showed dramatic swings — Chicago gained 13.6% accuracy but the Mets' park lost 11.1%. The gains in one season vanished or reversed in the next.

Lesson: Park factors affect scoring (useful for over/under predictions) but don't reliably improve win prediction. The home team's park advantage is already captured by Elo's home-field factor.

Attempt #5: Recency Weighting ⏱️

Result: ❌ Made things worse

Intuition says a player's 2024 performance should matter more than their 2017 stats. We implemented exponential decay with configurable half-lives: 1, 2, 3, and 4 seasons.

A 50-game test showed "60% accuracy" with a 2-year half-life. We got excited. Then we ran the full 2,430-game validation:

Recency weighting actively hurts the model. Why? Markov transition matrices need volume. A 25×25 matrix has 625 cells — you need thousands of at-bats to fill it reliably. Downweighting older data shrinks your effective sample size and makes the matrices noisier.

Lesson: More data beats recent data for transition probability estimation. The base-out state structure of baseball is stable enough that 2017 at-bats are still informative in 2023.

The Variance Problem 📉

The deepest issue with the Markov chain isn't accuracy — it's variance compression. The model predicts approximately 9 total runs for every single game. The standard deviation of predicted scores is 0.45 runs. In reality, it's 4.58 runs.

This happens because 10,000 simulation averages converge. Individual simulations have plenty of variance — 4-1 blowouts, 12-9 slugfests, 1-0 pitchers' duels — but the mean of 10,000 games always lands near 4.5 per side.

This is actually correct math — the expected value IS about 4.5 runs per team. But it means the Markov chain's edge comes from the simulation distribution, not the mean. That's why we use median scores, percentile-based over/under calls, and full probability distributions instead of point estimates.

What We Actually Learned

What's Next

We're not done optimizing. Still on the roadmap:

• Weather integration — Temperature, wind, and humidity affect ball flight. Needs game-time data.
• Platoon splits — Separate MSIs for left/right-handed matchups. We built the handedness database; testing is next.
• Starting pitcher Elo — A pitcher-specific Elo layer could capture ace vs. 5th starter differentiation.
• Calibration tuning — Better mapping of simulation probabilities to real-world outcomes.

But we're also realistic: we're probably within 1-2% of our architectural ceiling. The bigger opportunity is in how we present the predictions — score distributions, confidence levels, trend analysis — not just "Team A wins."