XGBoost classifier on 6.3M real PaySim transactions. Time-based split, walk-forward validated across 4 folds, calibrated probabilities, cost-sensitive thresholding, and a bug found and fixed mid-build, documented rather than hidden.
This runs the actual shipped model. The trained trees and calibration curve are
re-implemented in pure Python for this always-on demo (see src/export_model_json.py and
api/score.py), validated against the real scikit-learn/XGBoost output to within
floating-point precision on 20,000 real held-out transactions. Not a mock, not a lookup table.
PaySim's fraud rate is 0.13%. A model that predicts "not fraud" every single time scores 99.87% accuracy and catches nothing. Fraud-ops workflows freeze customer funds on a flag, so false positives carry real trust and regulatory cost. This project holds precision at or above 99% and evaluates on a strict time-based holdout, with no future-state leakage. PaySim is a common tutorial dataset; the contribution here isn't the dataset choice, it's the evaluation rigour: time-based split, calibration, cost-sensitive thresholds, and a five-model comparison on identical pipelines.
Can a model tell fraud apart from legitimate transactions at all? (src/train.py)
Were the model's settings ever actually tested against alternatives? (src/tune.py)
Does it hold up on more than one train/test split? (src/validate.py)
How would we know if the model started drifting in production? (src/monitoring.py)
How does someone without Python actually use this? (dashboard/app.py)
Live-testing the dashboard found the model flagging legitimate account closures as 100% fraud, traced to raw balance columns that were then removed from the model's inputs.
Does one feature dominate the model's decisions, and where's the real precision/recall/cost trade-off? (src/explain.py)
These numbers are the real output of src/train.py run end-to-end against the PaySim
CSV. The script previously could not run at all: CalibratedClassifierCV(cv="prefit") was removed
in scikit-learn ≥1.6. Fixed by wrapping the fitted model in sklearn.frozen.FrozenEstimator and
calibrating on a held-out slice.
The cost-optimal decision threshold swings fold to fold in walk-forward validation. The near-perfect PR-AUC hides that no single fixed threshold is clearly "correct" across the whole time horizon. A real deployment would need to revisit it periodically.
The model can't see whether the destination account's balance was updated consistently, because PaySim's
CASH_OUT type doesn't strictly enforce destination-side accounting. A rule-based patch for this was tested
and rejected: it produced 44,996 false alarms on the training split, up from 136. See MODEL_CARD.md.