From Random to Strategic: Building an Explainable Pokémon TCG Agent for Kaggle
What 300-game experiments taught me about simplicity, statistics, and the things I had to throw away.
I recently built an agent for Kaggle’s “Pokemon TCG AI Battle” competition: a live-ladder challenge where submitted agents pilot a 60-card Pokemon TCG deck against other competitors’ agents on a continuously-updated, Elo-like rating.
This article walks through the approach, the real experiments behind every claim, and, just as importantly, what I tried and rejected along the way.
The Challenge
The competition runs on the cabt engine (via Kaggle’s kaggle-environments package): a two-player, turn-based, imperfect-information card game, scored on a chess-clock-style overage-time budget rather than a fixed per-turn limit.
Submissions are packaged as a main.py plus a deck list and the engine’s own bundled runtime, then thrown onto a live ladder against thousands of other entries.
Two things made this an interesting engineering problem rather than a pure ML one: the ladder engine’s version has to match the local dev environment bit-for-bit (it doesn’t, out of the box, more on that below), and there is no seed API for reproducible matches.
My Approach
I chose a heuristic decision pipeline over a search or learned policy, for two reasons.
First, explainability: every decision the agent makes should be traceable by a human reviewer in minutes, not buried in opaque weights.
Second, evidence: the strongest publicly documented reference agent for this exact competition is itself rule-based.
So I built a pipeline, not a model.
How the Agent Makes Decisions
The pipeline runs in a fixed order at every decision point:
- Legal-option enumeration. Read the raw observation, produce every legal action.
- Tactical rules, checked before any scoring. Claim a provably-lethal attack the moment one exists (weakness-adjusted damage, never a guess); never let a losing option be picked when a legal attack is available; and, critically, always prefer developing the board (playing energy, evolving, benching) before attacking. That last rule alone was the single biggest local win-rate swing I measured in the whole project.
- Prize-centric weighted scoring. A small linear evaluation of the board state with named weights (prize lead dominant, then KO threat, HP advantage, energy efficiency, bench development, resource preservation). In the shipped greedy configuration it is consulted at every non-fast-path decision as the call site for future lookahead comparison; the actual choice among options comes from the tactical rules above.
- An optional lookahead layer. Built, tested, and ultimately not shipped (see What Did Not Work).
Every decision is wrapped in a watchdog with a conservative time budget and falls back to a simple legal choice on any unexpected exception, so the agent never forfeits on a crash.
Experimentation
Every number below is pulled directly from the project’s append-only experiment ledger, gated through a Wilson confidence-interval / two-proportion z-test harness (minimum 300 games per comparison), never eyeballed.
ExperimentConfigurationWins/NWin rate95% CISanity baselinerandom vs random (self-play)158/30052.7%[0.470, 0.582]Strategic slice (develop-first fix applied)early pipeline vs random250/30083.3%[0.787, 0.871]Final formal strength gateshipped champion vs random271/30090.3%[0.865, 0.932]Deck lock A/BCandidate 1 (Water) vs Candidate 2 (Fighting aggro), same pilot300/300100.0%[0.987, 1.000]
The deck-lock result is the one that surprised me most: with the identical tactical pipeline piloting both sides, changing only the 60-card list produced a 300–0 result.
Deck choice mattered more than any single tactical tweak I tested afterward.
What Worked
- Develop-before-attack ordering. A blind attack-first policy regressed to a 6.3% local win rate; it kept ending turns on 0-damage status attacks instead of building the board first. Reordering to develop-before-attack alone brought that to 83.3% with the same pipeline otherwise.
- Never-guess lethal detection. Only claiming a knockout when weakness-adjusted damage is provably lethal (never estimated) avoided a whole class of tactical misplays.
- Deck selection treated as a real experiment, not a preference: locking the deck via a measured A/B gate (300/300) before any weight tuning began.
- Prize-centric scoring. Weighting the actual win condition (prize lead) far above secondary signals kept the agent’s priorities aligned with how the game is actually won.
- A never-crash envelope around every decision: the agent falls back to a simple legal choice on any exception rather than forfeiting.
What Did Not Work
Three things were built, genuinely tested against the project’s own statistical gate, and rejected, recorded honestly rather than parameter-fished into a false “improvement”:
- Depth-1 forward-model lookahead. Hypothesis: looking one ply ahead at ambiguous decision points would beat the greedy develop-first policy. Measured result: 141/300 (47.0%) against the greedy champion, a NO-DECISION, not statistically significant. The extra search overhead bought nothing measurable at this depth, so the shipped agent stayed greedy.
- Weight-variant tuning. Two single-weight changes were each gated against the champion’s own record (260/300 baseline): raising the KO-threat weight (271/300, 90.3%, CI [0.865, 0.932]) and doubling the bench-development weight (268/300, 89.3%, CI [0.853, 0.923]). Both raw win rates were numerically higher than the baseline, but both confidence intervals overlapped it: two honest NO-DECISIONs, not promoted. Neither weight change shipped.
- Local win rate does not predict ladder standing. The shipped agent wins roughly 90% locally against a random-legal baseline, but that baseline is not representative of the ladder’s field: thousands of entered teams that plausibly include much stronger opponents than “random.” When the agent’s ladder rating didn’t mirror the local margin as a straightforward “should be near the top” signal, that was expected, not a defect in the pipeline: local evaluation exists as a regression gate during development, not a ladder-rank predictor.
The Result
The submission validated and reached a genuine terminal COMPLETE state on the Kaggle ladder (submission ref 54816888). Its public score moved 600.0 (nominal, at validation) → 494.0 → 455.4 within about twenty minutes of submission as other competitors’ matches resolved: a useful reminder that this competition’s rating is Elo-like and continuously time-varying, not a fixed number you can quote once and forget. I’m making no final-rank claim here, only what was observed, and when.
The full, executable solution notebook (self-contained, same pipeline components as the actual submission) is published publicly on Kaggle:
Pokemon TCG AI Battle: Explainable Heuristic Agent (Kaggle notebook)
The competition itself: Pokemon TCG AI Battle on Kaggle
What I Learned
Explainability and rigor paid off more than I expected going in.
The biggest wins came from a tactical bug fix (develop-before-attack) and a deck-selection experiment, not from adding complexity like lookahead. And the discipline of gating every change against a statistical baseline (instead of eyeballing raw percentages) is what let me tell the difference between a real improvement and noise.
Reporting the honest NO-DECISIONs alongside the wins felt like the right call, and it’s a habit I’m carrying into future projects.
Closing
If you’re working on this competition, or on rule-based game-playing agents more generally, I’d genuinely welcome a conversation: feel free to reach out or leave a response below.