Keep Your AI Models Accurate as the World Changes

We define three trigger types for each model: schedule-based (retrain every 30/60/90 days regardless of drift), drift-based (retrain when statistical drift score exceeds threshold), and event-based (retrain when a known business event occurs — Ramadan season, regulatory change, new product launch). Most production models use all three triggers with OR logic: whichever fires first triggers a retraining run. Trigger thresholds are calibrated to the model's measured drift rate during the first 90 days of production.

Champion/challenger is a model promotion strategy where the current production model (champion) and a newly trained candidate (challenger) run simultaneously on a split of live traffic — typically 5-10% to the challenger. Both models' predictions and outcomes are logged. After a defined evaluation period (usually 1-4 weeks), if the challenger outperforms the champion on defined metrics, it is promoted to champion. If it underperforms, it is automatically rolled back. This eliminates the risk of a big-bang model swap where a degraded model is immediately deployed to all traffic.

It depends on the model architecture and data volume. For gradient boosting models (XGBoost, LightGBM), retraining from scratch on a rolling window of recent data is typically faster and more reliable than incremental updates. For neural networks and fine-tuned LLMs, incremental fine-tuning on new data is often more practical. For very large models where full retraining is cost-prohibitive, we use continued fine-tuning with catastrophic forgetting mitigation techniques.

Automated gates prevent promotion. Every retraining pipeline includes performance evaluation gates: the new model must meet minimum thresholds on holdout test data before staging promotion, and must match or exceed the champion's performance on a live traffic sample before production promotion. If the new model fails either gate, it is flagged for human review and the champion remains in production. We investigate root causes: data quality regression, distribution shift in new training data, or labelling errors.