Results
The result is obtained from training ~6,00,000 claim data from Health Insurance Board.
{
"xgboost": {
"best_params": {
"max_depth": 10,
"min_child_weight": 14,
"gamma": 1.0530297634027552,
"learning_rate": 0.16393771094031248,
"reg_alpha": 8.53139475196546e-06,
"reg_lambda": 4.350963734093896e-07,
"subsample": 0.7266367682142774,
"colsample_bytree": 0.7611372308560409,
"colsample_bylevel": 0.9905255880335302
},
"best_ntree": 1000,
"best_threshold": 0.3999999999999998,
"test_metrics": {
"roc_auc": 0.9075202031675821,
"pr_auc": 0.8910824419514409,
"accuracy": 0.8318517285389269,
"f1": 0.8430257255048229
}
},
"random_forest": {
"best_params": {
"n_estimators": 300,
"max_depth": 28,
"min_samples_split": 16,
"min_samples_leaf": 2,
"max_features": 0.9077589029724598
},
"best_threshold": 0.4299999999999998,
"test_metrics": {
"roc_auc": 0.9091346324871363,
"pr_auc": 0.8917220443212476,
"accuracy": 0.8349814105765496,
"f1": 0.8457748597151777
}
}
}
Model Performance Data
Metric | XGBoost | Random Forest | Better |
ROC-AUC | 0.9075 | 0.9091 | Random Forest (+0.0016) |
PR-AUC | 0.8911 | 0.8917 | Random Forest (+0.0006) |
Accuracy | 0.8319 | 0.8350 | Random Forest (+0.0031) |
F1 Score | 0.8430 | 0.8458 | Random Forest (+0.0028) |
The Four Quadrants (Using Random Forest)
True Positives (Bottom Right: 40,771)
What it means: The claim was actually Fraud, and the model correctly predicted Fraud.
Business Impact: This is a win. The model successfully caught a fraudulent claim, saving the company money.
True Negatives (Top Left: 34,465)
What it means: The claim was actually Legit, and the model correctly predicted Legit.
Business Impact: This is also a win. A normal customer got their claim processed without unnecessary delays or investigations.
False Positives (Top Right: 10,588)
What it means: The claim was actually Legit, but the model incorrectly flagged it as Fraud.
Business Impact: This is the cost of doing business with an aggressive model. These 10,588 legitimate customers will likely face delays, and human investigators will waste time reviewing their perfectly fine claims.
False Negatives (Bottom Left: 4,281)
What it means: The claim was actually Fraud, but the model incorrectly labeled it as Legit.
Business Impact: This is direct financial loss. These are actual fraudsters who slipped past the AI and got their claims paid out.
The XGBoost Scorecard
Caught the Fraud (True Positives: 40,684)
What it means: The model correctly identified 40,684 actual fraudulent claims.
Context: This is slightly fewer than the Random Forest model (40,771).
Cleared Legit Claims (True Negatives: 34,270)
What it means: The model correctly processed 34,270 legitimate claims without an issue.
Context: Again, slightly fewer than Random Forest (34,465).
The False Alarms (False Positives: 10,783)
What it means: The model wrongly flagged 10,783 perfectly normal claims as "Fraud."
Context: This is where XGBoost struggled a bit more. It created almost 200 more false alarms than the Random Forest model, which means more manual review time wasted by your investigators.
The Ones That Got Away (False Negatives: 4,368)
What it means: The model failed to catch 4,368 actual fraudulent claims, passing them off as legitimate.
Context: This represents direct financial loss, and XGBoost missed 87 more of these than the Random Forest model did.
Scores (0.8430 for XGBoost and 0.8458 for Random Forest)
Both Models are "Healthy"
Random Forest is the Mathematical Winner The F1 score for Random Forest is higher by 0.0028.
The Real-World Impact is a Virtual Tie While Random Forest wins mathematically, a difference of 0.0028 is so incredibly small that they are practically tied.
Highest accuracy we need to choose Random forest but if we observe the speed of model if XGboost is faster we can use this one as well due to very less significant factor