Nearly every guide to fraud detection machine learning spends its energy on the wrong 10% of the problem.
They compare algorithms. Random forest versus XGBoost versus a neural net. Precision, recall, the F1 score, a confusion matrix, maybe a diagram of a decision tree. All of it is accurate, and none of it the thing that decides whether your model works.
The thing that decides it is the label.
Fraud labels arrive late, arrive incomplete, and lie in a specific direction. A chargeback might land 30, 60, sometimes 120 days after the transaction. Which means the most recent slice of your data, the part that reflects how fraud looks right now, is the part you have the least truth about. And “no chargeback filed” is not the same as “this was legitimate.” Plenty of fraud never gets disputed. The cardholder didn’t notice, the amount was small, the account was already compromised and abandoned.
So when you train a model on “chargeback = fraud, no chargeback = clean,” you’re teaching it a definition that’s wrong in a way that’s hard to see and easy to feel later. It will look great on your holdout set and underperform in production, and everyone will blame the algorithm.
Get the labels right and a boring gradient boosting model will beat a brilliant architecture trained on mush. We’d rather spend this piece on that than on another algorithm comparison, but we’ll cover the models too, because the choice does matter once the foundation is solid.
Why rules-based fraud detection keeps losing ground
Rules earn their place. They’re auditable, instant, and easy to explain to a compliance officer, which is why they’ll be part of your stack forever. We’ve made the case that rule-based vs machine learning is a false binary and most good systems run both.
Their weakness is a timing mismatch. Every rule on your list is a fossil of something that already cost you money. It stops that exact pattern and nothing adjacent to it. Fraud rings iterate in days. Your rule review happens when someone has a free afternoon.
Rules also can’t hold combinations in mind. A rule checks a threshold. It can flag orders over $500. It can flag emails younger than 72 hours. It cannot easily notice that this particular mix of order value, account age, device fingerprint, hour of day, shipping distance, and item category is unremarkable for your customers on a weekday afternoon and deeply strange at 3am on a holiday weekend.
And rules generate a false positive bill that nobody itemizes. When a rule declines a real customer, that customer doesn’t complain. They leave. There is no report for revenue that never showed up.
Coinmama ran a hand-written SQL script applying static rules to incoming transactions, rebuilt each time an analysis turned up something new. Moving to predictive scoring cut their false positives by about two-thirds, left them with a much shorter manual review queue, and ranked that queue so the riskiest cases got looked at first. That last part is underrated. Ranking is often worth more than blocking.
Supervised and unsupervised, and where each earns its place
Supervised learning trains on labeled history: transactions where you eventually learned the outcome. It’s the workhorse and it should be your primary. When you have enough confirmed fraud examples, supervised models are more accurate than anything else, by a wide margin.
Unsupervised learning and anomaly detection look for things that don’t fit the normal pattern, without needing labels. This gets oversold badly. Anomaly detection is genuinely useful in two situations: cold start, when you have no labeled fraud yet, and as a secondary net for attack patterns your labels have never seen. As a primary system it produces an alert queue full of weird-but-legitimate behavior, and your review team learns to ignore it within a month.
Most mature setups run supervised as the engine, with an unsupervised layer flagging novel patterns for investigation, plus rules on top as hard guardrails for the things that must never happen regardless of what a model thinks.
The label problem, in detail
Three decisions here matter more than your model choice.
Define the outcome deliberately. “Chargeback within 90 days” and “confirmed fraud after investigation” and “blocked by review” are three different targets that produce three different models. The first is easy to get and includes friendly fraud. The second is cleaner and much rarer. The third teaches your model to imitate your existing rules, complete with their mistakes, which is a genuinely popular way to waste six months.
Respect the maturation window. If disputes take up to 90 days to arrive, transactions from the last 90 days are not labeled, they’re pending. Training them as if “no chargeback yet” means “clean” teaches your model that recent fraud is fine. Hold out the immature window, or handle it explicitly.
Guard against leakage like it’s trying to kill you, because it is. Leakage is when a feature contains information that wasn’t available at prediction time. A “manual review flag” field. A refund timestamp. An account status that got updated after the fraud was discovered. Your model finds it instantly, posts a 0.99 AUC, and everyone celebrates a model that has learned to read the answer key. Leakage is the most common reason a fraud model works in testing and collapses in production, and it’s near-invisible unless you’re specifically hunting for it.
There’s also the imbalance. Fraud might be 0.1% of transactions. A model that predicts “legitimate” every time is 99.9% accurate and worth nothing. Use precision-recall curves, not accuracy. Use AUC-PR, not AUC-ROC, when the positive class is this rare. And evaluate at the operating point you’ll actually run at, meaning the top 1% of scores, not across the whole distribution.
The signals that carry the weight
Fraud signals cluster into groups, and the good models read all of them together.
Transaction attributes. Amount, currency, item mix, quantity, time of day, day of week, shipping method. Rush shipping on a first order is a classic for a reason.
Behavioral signals. How the session went. Time to checkout, whether the card details were typed or pasted, number of payment attempts, whether the cart was browsed or assembled straight from a search. Fraudsters move differently than shoppers.
Device and network context. Device fingerprint, browser and OS, IP reputation, whether the IP geography matches the billing address, proxy and VPN indicators.
Velocity features. Attempts per card, per device, per IP, per shipping address, over the last hour, day, week. Velocity catches card testing better than almost anything else, and it’s one of the strongest feature families in most fraud models.
Relationship features. How many accounts share this device? How many cards share this address. Whether this email appeared elsewhere in your data an hour ago. Fraud is rarely one transaction.
Customer history. Tenure, prior orders, prior disputes, prior refunds, prior support contacts.
Velocity and relationship features are usually where the lift lives, and they’re also the ones teams skip, because building them means constructing aggregations across time windows rather than reading columns off a table. That work is where a lot of fraud projects die.
The models worth your time
Straight talk, because the honest answer is narrower than most articles admit.
Gradient boosting (XGBoost, LightGBM, CatBoost) is the right default for machine learning for fraud detection. Fraud data is tabular, mixed-type, and full of interactions. Boosted trees eat that for breakfast, they handle missing values gracefully, they train fast, and they give you feature importances that a risk analyst can read. For most merchants, this is the answer and the search should stop here.
Logistic regression stays useful as a baseline and in environments where you need a model you can defend line by line. It’ll usually lose to boosting on accuracy. It’ll usually win on explainability, and sometimes that’s the constraint that matters.
Random forest works, sits between the two on both accuracy and interpretability, and is a reasonable choice if boosting feels like too much tuning.
Neural networks earn their place on sequence data, meaning modeling a cardholder’s transaction history as an ordered series rather than a set of aggregates, and at a very large scale. For a mid-market merchant with tabular transaction data, a deep model is usually a more expensive way to get a slightly worse result. We’ve written more on ML models and how these tradeoffs play out on business data.
Isolation forest and autoencoders for anomaly detection fraud work, as the secondary layer described above. Not the engine.
Real-time scoring and the false positive tax
A fraud score is worth nothing at 4am in a batch report. It needs to exist before the authorization decision, which puts you in a latency budget measured in tens of milliseconds.
That constraint shapes the design. Your velocity features need to be computable in real time, which means a feature store or streaming aggregation, not a nightly job. Your model needs to be small enough to score fast, which is one more argument for boosted trees over anything exotic.
The output should feed a graduated response, not a binary. Hard decline for the top sliver. Step-up verification for the medium band, so 3D Secure or an identity check, which recovers customers that a decline would have lost permanently. Hold for review for a narrow slice, ranked by risk so your team’s hours land where they matter. Everything else goes straight through.
That graduation is where the false positive tax gets paid down. A rule has two options. A score has as many as you want.
Set your thresholds by expected value, not by gut. For each band, the math is straightforward: expected fraud loss prevented versus expected margin loss on good orders declined, plus the review cost. Run that per segment, because a new account and a four-year customer should not face the same bar.
How we build fraud models without a data science team
Everything above is real work: label definition, maturation windows, leakage prevention, velocity feature construction, imbalance handling, threshold setting, drift monitoring, retraining. It’s the reason ai fraud detection stays on the roadmap at companies that clearly need it.
Our Predictive AI Agent does that work. You point it at your historical transaction and outcome data, ask which transactions are likely to be fraudulent, and the agent handles the data preparation, the feature engineering including time-window aggregations, model selection, training, and statistical validation. Guardrails against data leakage, overfitting, and unbalanced labels are enforced by default, because those three are what quietly ruin fraud models and they shouldn’t be optional. Scores flow into your systems with the drivers behind each one visible, so a risk analyst can explain a hold to a customer, an auditor, or a very unhappy VP.
No code required. And no requirement to clean and restructure your data first, because transaction data is messy everywhere and the agent is built for the real thing.
Where to start
Pull six months of transactions, excluding the last 90 days. Define your label deliberately and write down what it includes. Check for leakage before you check for accuracy. Build velocity features even if it’s annoying. Evaluate at your real operating point. Then compare against your current rules, on the same window, on net loss rather than catch rate.
If your model can’t beat a well-tuned rule set on that comparison, the model isn’t ready. If it beats it and your false positives drop at the same time, you’ve found money that was already yours.
See how our models detect fraud in your transaction data
Dror is the VP of Data and Analytics at Pecan AI, where he leads the analytics strategy that powers both customer success and Pecan’s own growth. He joined Pecan as Director of Analytics after years of data leadership roles across tech and fintech, bringing a firsthand understanding of what it takes to make data actually useful for business teams.
