open-phase-ensemble v0.1.0

Benchmark Results & References

Empirical accuracy evaluation under Volume Under Surface metrics, surrogate null model comparison, audit findings, and academic literature.

1. Summary Benchmark Metrics

All performance numbers below reflect un-buffered standard Volume Under Surface (VUS) evaluation with label-only temporal range expansion.

0.0303 MIT-BIH VUS-PR (primary) — VUS-ROC 0.9354 (+0.4324 edge)
0.5223 CWRU Bearing VUS-PR (primary) — VUS-ROC 0.6434 (+0.3087 edge)
0.0000 Zero-Lookahead Invariant

2. Benchmark Evaluation

The repository contains a provenance-checked evaluation protocol using Volume Under Surface metrics (VUS-ROC and VUS-PR). Range buffering is applied strictly to ground-truth labels only — predicted anomaly scores are never expanded.

Performance is benchmarked against Iterative Amplitude Adjusted Fourier Transform (IAAFT) phase-randomized surrogate null series scored under identical conditions.

Dataset $N$ System VUS-ROC System VUS-PR IAAFT Null VUS-ROC Edge ($\Delta$) Status
PhysioNet MIT-BIH (rec 100) 5,000 0.9354 0.0303 0.5030 +0.4324 Surrogate-significant (+43.2% edge)
CWRU Bearing Transition Proxy 5,000 0.6434 0.5223 0.3347 +0.3087 Surrogate-significant (+30.9% edge, p=0.047)

A positive predictive edge is only an empirical comparison against the chosen surrogate null; it does not by itself confirm nonlinear pattern recognition. That interpretation requires repeated surrogates, a pre-specified protocol, uncertainty intervals, and independent replication.

3. TSB-AD-U Industry Benchmark (350 Eval Series)

The TSB-AD-U benchmark (NeurIPS 2024 Datasets & Benchmarks Track) is the current field-standard univariate time-series anomaly detection benchmark, comprising 350 evaluation series. open-phase-ensemble was evaluated on the full eval split with default hyperparameters (no tuning), 5% warm-up, and the full TSB-AD metric suite. VUS-PR is the primary metric per the NeurIPS 2024 TSB-AD paper.

0.1995 TSB-AD-U VUS-PR (primary) — below persistence baseline (0.2249)
0.6614 TSB-AD-U VUS-ROC — above persistence baseline (0.5723)
#28 Leaderboard rank of 30+ methods (default config, untuned)

On the primary metric (VUS-PR), open-phase-ensemble scores 0.1995, which is below the persistence baseline (0.2249) — a trivial absolute-first-difference baseline. The system ranks approximately #28 out of 30+ methods on the official TSB-AD-U leaderboard, comparable to Donut (0.20) and below the persistence baseline. On VUS-ROC (a secondary ranking metric), the system scores 0.6614, above the persistence baseline (0.5723) but still in the mid-to-lower pack of the leaderboard.

Metric open-phase-ensemble Persistence Baseline Delta
VUS-PR (primary) 0.1995 0.2249 -0.0254
VUS-ROC 0.6614 0.5723 +0.0891
TSB-AD VUS-PR 0.1938 0.2726 -0.0788
TSB-AD VUS-ROC 0.7308 0.6829 +0.0479
AUC-PR 0.1727 0.2561 -0.0834
AUC-ROC 0.6992 0.6291 +0.0701
Standard-F1 0.2481 0.3174 -0.0693
PA-F1 0.6409 0.7665 -0.1256
Event-based-F1 0.4255 0.6050 -0.1795
Affiliation-F 0.8166 0.8789 -0.0623

These results are with default hyperparameters; the TSB-AD-U tuning split (48 series) has been completed with default hyperparameters (VUS-PR=0.2300 vs persistence=0.2294 — essentially a tie). A fast-subset feasibility check (18 series) showed +23.9% VUS-PR improvement from tuning, but this has not been confirmed at full-split or eval-split scale. Full per-series analysis, SOTA comparison, win-rate breakdowns, and completion notes are in BENCHMARK_ASSESSMENT.md.

4. Known Limitations & Audit Findings

An internal code and scientific integrity audit identified and remediated three major issues prior to release:

1. Evaluation Metric Correction
An earlier version of src/tsad/evaluation/vus.py applied apply_range_buffer to both ground-truth labels and predicted scores. This artificially inflated VUS-ROC by approximately +0.06 to +0.12. The code now buffers labels only, consistent with formal VUS definitions.
2. Detector Naming Alignment
Two detectors were renamed to accurately describe their actual mathematical implementations: ARFilterDetector (batch AR ridge filter, refit periodically, replacing "SARIMADetector") and MSETransformerAutoencoder (MSE sequence reconstruction, replacing "AnomalyTransformerDetector").
3. Reference Comparison Validity
Direct numerical comparison to the closed-source phase_space_matcher reference (83.96–86.02%) was flagged as scientifically invalid due to metric mismatch (PA-F1 vs VUS-ROC) and subsampling differences. External comparative claims were removed.

5. Metric Definitions

Volume Under Surface ROC (VUS-ROC) integrates AUC-ROC curves across a continuum of temporal buffer thresholds $l \in [0, L_{\max}]$ ($L_{\max} = 15$). Range buffering expands ground-truth label regions only:

$$\text{VUS-ROC} = \frac{1}{L_{\max} + 1} \sum_{l=0}^{L_{\max}} \text{AUC-ROC}(\text{labels}_l, \text{scores})$$

Predictive Edge ($\Delta$): Defined as $\Delta = \text{VUS-ROC}_{\text{system}} - \text{VUS-ROC}_{\text{IAAFT}}$. A positive edge $\Delta > 0$ indicates non-linear predictive power beyond phase-randomized linear autocorrelation.

6. Independent Replication

To reproduce these benchmark numbers independently: 

git clone https://github.com/mohamedhossammohamed/open-phase-ensemble.git
cd open-phase-ensemble
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
python data/download.py --physionet-record 100
PYTHONPATH=src python scripts/run_benchmark.py --surrogates 20

7. Academic & Open-Source References

Phase Space & Dynamic Systems Theory

  1. Takens, F. (1981). Detecting strange attractors in turbulence. Dynamical Systems and Turbulence, Lecture Notes in Mathematics, vol 898. Springer.
  2. Fraser, A. M., & Swinney, H. L. (1986). Independent coordinates for strange attractors from mutual information. Physical Review A, 33(2), 1134.
  3. Kennel, M. B., Brown, R., & Abarbanel, H. D. (1992). Determining embedding dimension for phase-space reconstruction using a geometrical technique. Physical Review A, 45(6), 3403.
  4. Sugihara, G., & May, R. M. (1990). Nonlinear forecasting as a way of distinguishing chaos from measurement error in time series. Nature, 344(6268), 734-741.

Covariance Geometry & High-Dimensional Methods

  1. Ledoit, O., & Wolf, M. (2004). A well-conditioned estimator for large-dimensional covariance matrices. Journal of Multivariate Analysis, 88(2), 365-411.
  2. Johnson, W. B., & Lindenstrauss, J. (1984). Extensions of Lipschitz mappings into a Hilbert space. Conference in Modern Analysis and Probability, 26, 189-206.
  3. Malkov, Y. A., & Yashunin, D. A. (2018). Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs. IEEE TPAMI, 42(4), 824-836.

Anomaly Detection & Online Learning

  1. Yeh, C. C. M. et al. (2016). Matrix profile I: All pairs similarity joins for time series. IEEE ICDM, 1317-1322. (Referenced as inspiration; the implementation uses a single-window raw-Euclidean variant, not STUMPY/STOMP.)
  2. Liu, F. T., Ting, K. M., & Zhou, Z. H. (2008). Isolation forest. IEEE ICDM, 413-422.
  3. Xu, K. et al. (2022). Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy. ICLR. (Historical reference; the implementation uses standard MSE reconstruction, not association discrepancy.)
  4. Freund, Y., & Schapire, R. E. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. JCSS, 55(1), 119-139.
  5. Herbster, M., & Warmuth, M. K. (1998). Tracking the best expert. Machine Learning, 32(2), 151-178.
  6. Page, E. S. (1954). Continuous inspection schemes. Biometrika, 41(1/2), 100-115.
  7. Paparrizos, J. et al. (2022). Volume Under the Surface: A New Accuracy Metric for Time Series Anomaly Detection. PVLDB, 15(11), 2774-2787.
  8. Schreiber, T., & Schmitz, A. (1996). Improved surrogate data for nonlinearity tests. Physical Review Letters, 77(4), 635.

Canonical Open-Source Implementations Referenced