kDMI: A Novel Method for Missing Values Imputation Using Two Levels of Horizontal Partitioning in a Data set

Md Geaur Rahman; Md Zahidul Islam

doi:10.1007/978-3-642-53917-6

kDMI: A Novel Method for Missing Values Imputation Using Two Levels of Horizontal Partitioning in a Data set

Md Geaur Rahman, Md Zahidul Islam

Research output: Book chapter/Published conference paper › Conference paper › peer-review

19 Citations (Scopus)

39 Downloads (Pure)

Abstract

Imputation of missing values is an important data mining task for improving the quality of data mining results. The imputation based on similar records is generally more accurate than the imputation based on all records of adata set. Therefore, in this paper we present a novel algorithm called kDMI that employs two levels of horizontal partitioning (based on a decision tree and k-NNalgorithm) of a data set, in order to find the records that are very similar to the one with missing value/s. Additionally, it uses a novel approach to automatically find the value of k for each record. We evaluate the performance of kDMI over three high quality existing methods on two real data sets in terms of four evaluation criteria. Our initial experimental results, including 95% confidence interval analysis and statistical t-test analysis, indicate the superiority of kDMI over the existing methods.

Original language	English
Title of host publication	Lecture Notes in Artiﬁcial Intelligence
Subtitle of host publication	Subseries of Lecture Notes in Computer Science
Place of Publication	Berlin
Publisher	Springer-Verlag London Ltd.
Pages	250-263
Number of pages	14
Volume	8347
ISBN (Electronic)	9783642539176
ISBN (Print)	9783642539169
DOIs	https://doi.org/10.1007/978-3-642-53917-6
Publication status	Published - 2013
Event	The 9th International Conference on Advanced Data Mining and Applications (ADMA 2013) - Zhejiang University, Hangzhou, China Duration: 14 Dec 2013 → 16 Dec 2013 https://web.archive.org/web/20130623022301/http://www.adma2013.org:80/

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer Verlag
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	The 9th International Conference on Advanced Data Mining and Applications (ADMA 2013)
Country/Territory	China
City	Hangzhou
Period	14/12/13 → 16/12/13
Other	The conference aims at bringing together the experts on data mining from around the world, and providing a leading international forum for the dissemination of original research findings in data mining, spanning applications, algorithms, software and systems, as well as different applied disciplines with potential in data mining. ADMA 2013 will promote the same close interaction among practitioners and researchers. Published papers will go through a full peer review process.
Internet address	https://web.archive.org/web/20130623022301/http://www.adma2013.org:80/

Access to Document

10.1007/978-3-642-53917-6

999015772 ManuscriptAccepted author manuscript, 1.1 MB

Cite this

Rahman, M. G., & Islam, M. Z. (2013). kDMI: A Novel Method for Missing Values Imputation Using Two Levels of Horizontal Partitioning in a Data set. In Lecture Notes in Artiﬁcial Intelligence : Subseries of Lecture Notes in Computer Science (Vol. 8347, pp. 250-263). (Lecture Notes in Computer Science). Springer-Verlag London Ltd.. https://doi.org/10.1007/978-3-642-53917-6

@inproceedings{ede54972126b40689fed354ebec1169e,

title = "kDMI: A Novel Method for Missing Values Imputation Using Two Levels of Horizontal Partitioning in a Data set",

abstract = "Imputation of missing values is an important data mining task for improving the quality of data mining results. The imputation based on similar records is generally more accurate than the imputation based on all records of adata set. Therefore, in this paper we present a novel algorithm called kDMI that employs two levels of horizontal partitioning (based on a decision tree and k-NNalgorithm) of a data set, in order to find the records that are very similar to the one with missing value/s. Additionally, it uses a novel approach to automatically find the value of k for each record. We evaluate the performance of kDMI over three high quality existing methods on two real data sets in terms of four evaluation criteria. Our initial experimental results, including 95% confidence interval analysis and statistical t-test analysis, indicate the superiority of kDMI over the existing methods.",

keywords = "Open access version available, Data cleansing, Data pre-processing, Decision Trees, EM algorithm, Missing value imputation",

author = "Rahman, {Md Geaur} and Islam, {Md Zahidul}",

note = "Imported on 12 Apr 2017 - DigiTool details were: Journal title (773t) = Lecture Notes in Computer Science. ISSNs: 0302-9743; ; The 9th International Conference on Advanced Data Mining and Applications (ADMA 2013) ; Conference date: 14-12-2013 Through 16-12-2013",

year = "2013",

doi = "10.1007/978-3-642-53917-6",

language = "English",

isbn = "9783642539169",

volume = "8347",

series = "Lecture Notes in Computer Science",

publisher = "Springer-Verlag London Ltd.",

pages = "250--263",

booktitle = "Lecture Notes in Artiﬁcial Intelligence",

address = "Germany",

url = "https://web.archive.org/web/20130623022301/http://www.adma2013.org:80/",

}

Rahman, MG & Islam, MZ 2013, kDMI: A Novel Method for Missing Values Imputation Using Two Levels of Horizontal Partitioning in a Data set. in Lecture Notes in Artiﬁcial Intelligence : Subseries of Lecture Notes in Computer Science. vol. 8347, Lecture Notes in Computer Science, Springer-Verlag London Ltd., Berlin, pp. 250-263, The 9th International Conference on Advanced Data Mining and Applications (ADMA 2013), Hangzhou, China, 14/12/13. https://doi.org/10.1007/978-3-642-53917-6

kDMI : A Novel Method for Missing Values Imputation Using Two Levels of Horizontal Partitioning in a Data set. / Rahman, Md Geaur ; Islam, Md Zahidul.

Lecture Notes in Artiﬁcial Intelligence : Subseries of Lecture Notes in Computer Science. Vol. 8347 Berlin : Springer-Verlag London Ltd., 2013. p. 250-263 (Lecture Notes in Computer Science).

Research output: Book chapter/Published conference paper › Conference paper › peer-review

TY - GEN

T1 - kDMI

T2 - The 9th International Conference on Advanced Data Mining and Applications (ADMA 2013)

AU - Rahman, Md Geaur

AU - Islam, Md Zahidul

N1 - Imported on 12 Apr 2017 - DigiTool details were: Journal title (773t) = Lecture Notes in Computer Science. ISSNs: 0302-9743;

PY - 2013

Y1 - 2013

N2 - Imputation of missing values is an important data mining task for improving the quality of data mining results. The imputation based on similar records is generally more accurate than the imputation based on all records of adata set. Therefore, in this paper we present a novel algorithm called kDMI that employs two levels of horizontal partitioning (based on a decision tree and k-NNalgorithm) of a data set, in order to find the records that are very similar to the one with missing value/s. Additionally, it uses a novel approach to automatically find the value of k for each record. We evaluate the performance of kDMI over three high quality existing methods on two real data sets in terms of four evaluation criteria. Our initial experimental results, including 95% confidence interval analysis and statistical t-test analysis, indicate the superiority of kDMI over the existing methods.

AB - Imputation of missing values is an important data mining task for improving the quality of data mining results. The imputation based on similar records is generally more accurate than the imputation based on all records of adata set. Therefore, in this paper we present a novel algorithm called kDMI that employs two levels of horizontal partitioning (based on a decision tree and k-NNalgorithm) of a data set, in order to find the records that are very similar to the one with missing value/s. Additionally, it uses a novel approach to automatically find the value of k for each record. We evaluate the performance of kDMI over three high quality existing methods on two real data sets in terms of four evaluation criteria. Our initial experimental results, including 95% confidence interval analysis and statistical t-test analysis, indicate the superiority of kDMI over the existing methods.

KW - Open access version available

KW - Data cleansing

KW - Data pre-processing

KW - Decision Trees

KW - EM algorithm

KW - Missing value imputation

U2 - 10.1007/978-3-642-53917-6

DO - 10.1007/978-3-642-53917-6

M3 - Conference paper

SN - 9783642539169

VL - 8347

T3 - Lecture Notes in Computer Science

SP - 250

EP - 263

BT - Lecture Notes in Artiﬁcial Intelligence

PB - Springer-Verlag London Ltd.

CY - Berlin

Y2 - 14 December 2013 through 16 December 2013

ER -

kDMI: A Novel Method for Missing Values Imputation Using Two Levels of Horizontal Partitioning in a Data set

Abstract

Publication series

Conference

Access to Document

Fingerprint

Cite this