In today’s interconnected society, large volumes of time-series data are usually collected from real-time applications. This data is generally used for data-driven decision-making. With time, changes may emerge in the statistical characteristics of this data - this is also known as concept drift. A concept drift can be detected using a concept drift detector. An ideal detector should detect drift accurately and efficiently. However, these properties may not be easy to achieve. To address this gap, a novel drift detection method WinDrift (WD) is presented in this research. The foundation of WD is the early detection of concept drift using corresponding and hierarchical time windows. To assess drift, the proposed method uses two sample hypothesis tests with Kolmogorov-Smirnov (KS) statistical distance. These tests are carried out on sliding windows configured on multiple hierarchical levels that assess drift by comparing statistical distance between two windows of corresponding time period on each level. To evaluate the efficacy of WD, 4 real datasets and 10 reproducible synthetic datasets are used. A comparison with 5 existing state-of-the-art drift detection methods demonstrates that WinDrift detects drift efficiently with minimal false alarms and has efficient computational resource usage. The synthetic datasets and the WD code designed for this work have been made publicly available at https://github.com/naureenaqvi/windrift.
|Title of host publication||Data Mining|
|Subtitle of host publication||20th Australasian Conference, AusDM 2022, Proceedings|
|Editors||Laurence A.F. Park, Simeon Simoff, Heitor Murilo Gomes, Maryam Doborjeh, Yee Ling Boo, Yun Sing Koh, Yanchang Zhao, Graham Williams|
|Number of pages||17|
|Publication status||Published - 06 Dec 2022|
|Event||20th Australasian Data Mining Conference, AusDM 2022: AUSDM’22 - Western Sydney University Parramatta campus, Western Sydney, Australia|
Duration: 12 Dec 2022 → 15 Dec 2022
https://ausdm22.ausdm.org/ (Conference website)
|Name||Communications in Computer and Information Science|
|Conference||20th Australasian Data Mining Conference, AusDM 2022|
|Period||12/12/22 → 15/12/22|
|Other||The Australasian Data Mining Conference has established itself as the premier Australasian meeting for both practitioners and researchers in data mining. It is devoted to the art and science of intelligent analysis of (usually big) data sets for meaningful (and previously unknown) insights. This conference will enable the sharing and learning of research and progress in the local context and new breakthroughs in data mining algorithms and their applications across all industries.|
Since AusDM’02 the conference has showcased research in data mining, providing a forum for presenting and discussing the latest research and developments. Built on this tradition, AusDM’22 will facilitate the cross-disciplinary exchange of ideas, experience and potential research directions. Speciﬁcally, the conference seeks to showcase: Research Prototypes; Industry Case Studies; Practical Analytics Technology; and Research Student Projects. AusDM’22 will be a meeting place for pushing forward the frontiers of data mining in academia and industry. This year’s conference is the 20th anniversary of AusDM and we have an exciting line-up to celebrate this milestone.