WinDrift: Early detection of concept drift using corresponding and hierarchical time windows

In today’s interconnected society, large volumes of time-series data are usually collected from real-time applications. This data is generally used for data-driven decision-making. With time, changes may emerge in the statistical characteristics of this data - this is also known as concept drift. A concept drift can be detected using a concept drift detector. An ideal detector should detect drift accurately and efficiently. However, these properties may not be easy to achieve. To address this gap, a novel drift detection method WinDrift (WD) is presented in this research. The foundation of WD is the early detection of concept drift using corresponding and hierarchical time windows. To assess drift, the proposed method uses two sample hypothesis tests with Kolmogorov-Smirnov (KS) statistical distance. These tests are carried out on sliding windows configured on multiple hierarchical levels that assess drift by comparing statistical distance between two windows of corresponding time period on each level. To evaluate the efficacy of WD, 4 real datasets and 10 reproducible synthetic datasets are used. A comparison with 5 existing state-of-the-art drift detection methods demonstrates that WinDrift detects drift efficiently with minimal false alarms and has efficient computational resource usage. The synthetic datasets and the WD code designed for this work have been made publicly available at https://github.com/naureenaqvi/windrift.