AI-Driven Predictive Analytics for Early Detection of Anomalies and Emerging Patterns in Large-Scale Data

Abstract

Organisations now collect records and sensor readings faster than analysts can inspect them, and the events that matter most, such as fraud, intrusions, equipment faults and the first signs of a change in behaviour, are rare and usually unlabelled. We studied how well unsupervised machine learning detects these events, and at what cost, in two settings. For static records we compared seven detectors from five methodological families (Isolation Forest, ECOD, k-nearest neighbours, LOF, one-class SVM, PCA and an autoencoder) and two rank-averaged ensembles on ten public benchmark datasets ranging from 1,831 to 567,498 records, with five repetitions each. For ordered data we built a two-stage detector that combines the residuals of an autoregressive forecast with a cumulative-sum (CUSUM) statistic, and compared it with four streaming baselines on eight labelled series from the Numenta Anomaly Benchmark. Isolation Forest was the most reliable single detector, with a mean ROC-AUC of 0.880, a mean average precision of 0.541 and the best mean rank (1.9); the differences between methods were significant (Friedman χ² = 39.2, p < 0.001). A rank-averaged ensemble of four diverse detectors ranked second (3.6) but did not beat Isolation Forest (Wilcoxon p = 0.014), and removing its weakest member raised its mean AUC from 0.834 to 0.855. Proximity-based methods failed on clustered anomalies and their run time grew roughly quadratically, reaching 31 s for 80,000 records against 0.5 s for Isolation Forest. In the streaming experiments the residual–CUSUM detector found all 21 labelled anomaly windows, earlier than any baseline (mean delay 28% of the window length), at a cost of 1.95 false-alarm events per 1,000 points. The results suggest that a fast tree-based detector for records, paired with forecast residuals and change accumulation for streams, is a sound default for large-scale monitoring, and that ensembles should be built with care because they inherit the failures of their members.