iDMI: A novel technique for missing value imputation using a decision tree and expectation-maximization algorithm

Md Geaur Rahman; Md Zahidul Islam

doi:10.1109/ICCITechn.2014.6997351

iDMI: A novel technique for missing value imputation using a decision tree and expectation-maximization algorithm

Md Geaur Rahman, Md Zahidul Islam

Research output: Book chapter/Published conference paper › Conference paper › peer-review

10 Citations (Scopus)

Abstract

In this paper we present a novel technique called iDMI that imputes missing values of a data set by combining a decision tree algorithm (DT) and an expectation-maximization(EMI) algorithm. We first divide a data set into horizontal segmentsthrough applying a DT algorithm such as C4.5, and thenapply an EMI algorithm on each segment in order to impute themissing values belong to the segment. If all numerical attribute values of a record are missing then we impute them by the meanvalues of the attributes of the records belong to a segment wherethe record falls in, and thereby reduce the computational time complexity of iDMI compare to an existing technique calledDMI which calculate the mean value of an attribute by using all records of a data set. We evaluate the performance of iDMIover three high quality existing techniques on two real data sets in terms of four evaluation criteria. Our initial experimental results, including several statistical significance analysis, indicatethe superiority of iDMI over the existing techniques.

Original language	English
Title of host publication	Proceedings of the 16th International Conference on Computer and Information Technology
Subtitle of host publication	ICCIT 2013
Editors	Rameswar Debnath
Place of Publication	United States
Publisher	IEEE, Institute of Electrical and Electronics Engineers
Pages	496-501
Number of pages	6
ISBN (Electronic)	9781479934980
DOIs	https://doi.org/10.1109/ICCITechn.2014.6997351
Publication status	Published - 2014
Event	16th International Conference on Computer and Information Technology: ICCIT 2013 - Khulna University, Khulna, Bangladesh Duration: 08 Mar 2014 → 10 Mar 2014 https://web.archive.org/web/20130409043745/http://www.iccit.org.bd/2013/

Conference

Conference	16th International Conference on Computer and Information Technology
Country/Territory	Bangladesh
City	Khulna
Period	08/03/14 → 10/03/14
Internet address	https://web.archive.org/web/20130409043745/http://www.iccit.org.bd/2013/

Access to Document

10.1109/ICCITechn.2014.6997351

http://ieeexplore.ieee.org/document/6997351/

Cite this

Rahman, M. G., & Islam, M. Z. (2014). iDMI: A novel technique for missing value imputation using a decision tree and expectation-maximization algorithm. In R. Debnath (Ed.), Proceedings of the 16th International Conference on Computer and Information Technology: ICCIT 2013 (pp. 496-501). IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICCITechn.2014.6997351

Rahman, Md Geaur ; Islam, Md Zahidul. / iDMI : A novel technique for missing value imputation using a decision tree and expectation-maximization algorithm. Proceedings of the 16th International Conference on Computer and Information Technology: ICCIT 2013. editor / Rameswar Debnath. United States : IEEE, Institute of Electrical and Electronics Engineers, 2014. pp. 496-501

@inproceedings{0a01243a34fb44e6bdd1ca78c88218c8,

title = "iDMI: A novel technique for missing value imputation using a decision tree and expectation-maximization algorithm",

abstract = "In this paper we present a novel technique called iDMI that imputes missing values of a data set by combining a decision tree algorithm (DT) and an expectation-maximization(EMI) algorithm. We first divide a data set into horizontal segmentsthrough applying a DT algorithm such as C4.5, and thenapply an EMI algorithm on each segment in order to impute themissing values belong to the segment. If all numerical attribute values of a record are missing then we impute them by the meanvalues of the attributes of the records belong to a segment wherethe record falls in, and thereby reduce the computational time complexity of iDMI compare to an existing technique calledDMI which calculate the mean value of an attribute by using all records of a data set. We evaluate the performance of iDMIover three high quality existing techniques on two real data sets in terms of four evaluation criteria. Our initial experimental results, including several statistical significance analysis, indicatethe superiority of iDMI over the existing techniques.",

keywords = "Data cleansing, Data pre-processing, Missing value",

author = "Rahman, {Md Geaur} and Islam, {Md Zahidul}",

note = "Imported on 03 May 2017 - DigiTool details were: publisher = United States: IEEE, 2014. Event dates (773o) = 21-12-2013-23-12-2013; Parent title (773t) = IEEE International Conference on Computer and Information Technology (ICCIT).; 16th International Conference on Computer and Information Technology : ICCIT 2013 ; Conference date: 08-03-2014 Through 10-03-2014",

year = "2014",

doi = "10.1109/ICCITechn.2014.6997351",

language = "English",

pages = "496--501",

editor = "Rameswar Debnath",

booktitle = "Proceedings of the 16th International Conference on Computer and Information Technology",

publisher = "IEEE, Institute of Electrical and Electronics Engineers",

address = "United States",

url = "https://web.archive.org/web/20130409043745/http://www.iccit.org.bd/2013/",

}

Rahman, MG & Islam, MZ 2014, iDMI: A novel technique for missing value imputation using a decision tree and expectation-maximization algorithm. in R Debnath (ed.), Proceedings of the 16th International Conference on Computer and Information Technology: ICCIT 2013. IEEE, Institute of Electrical and Electronics Engineers, United States, pp. 496-501, 16th International Conference on Computer and Information Technology, Khulna, Bangladesh, 08/03/14. https://doi.org/10.1109/ICCITechn.2014.6997351

iDMI : A novel technique for missing value imputation using a decision tree and expectation-maximization algorithm. / Rahman, Md Geaur ; Islam, Md Zahidul.

Proceedings of the 16th International Conference on Computer and Information Technology: ICCIT 2013. ed. / Rameswar Debnath. United States : IEEE, Institute of Electrical and Electronics Engineers, 2014. p. 496-501.

Research output: Book chapter/Published conference paper › Conference paper › peer-review

TY - GEN

T1 - iDMI

T2 - 16th International Conference on Computer and Information Technology

AU - Rahman, Md Geaur

AU - Islam, Md Zahidul

N1 - Imported on 03 May 2017 - DigiTool details were: publisher = United States: IEEE, 2014. Event dates (773o) = 21-12-2013-23-12-2013; Parent title (773t) = IEEE International Conference on Computer and Information Technology (ICCIT).

PY - 2014

Y1 - 2014

N2 - In this paper we present a novel technique called iDMI that imputes missing values of a data set by combining a decision tree algorithm (DT) and an expectation-maximization(EMI) algorithm. We first divide a data set into horizontal segmentsthrough applying a DT algorithm such as C4.5, and thenapply an EMI algorithm on each segment in order to impute themissing values belong to the segment. If all numerical attribute values of a record are missing then we impute them by the meanvalues of the attributes of the records belong to a segment wherethe record falls in, and thereby reduce the computational time complexity of iDMI compare to an existing technique calledDMI which calculate the mean value of an attribute by using all records of a data set. We evaluate the performance of iDMIover three high quality existing techniques on two real data sets in terms of four evaluation criteria. Our initial experimental results, including several statistical significance analysis, indicatethe superiority of iDMI over the existing techniques.

AB - In this paper we present a novel technique called iDMI that imputes missing values of a data set by combining a decision tree algorithm (DT) and an expectation-maximization(EMI) algorithm. We first divide a data set into horizontal segmentsthrough applying a DT algorithm such as C4.5, and thenapply an EMI algorithm on each segment in order to impute themissing values belong to the segment. If all numerical attribute values of a record are missing then we impute them by the meanvalues of the attributes of the records belong to a segment wherethe record falls in, and thereby reduce the computational time complexity of iDMI compare to an existing technique calledDMI which calculate the mean value of an attribute by using all records of a data set. We evaluate the performance of iDMIover three high quality existing techniques on two real data sets in terms of four evaluation criteria. Our initial experimental results, including several statistical significance analysis, indicatethe superiority of iDMI over the existing techniques.

KW - Data cleansing

KW - Data pre-processing

KW - Missing value

UR - http://toc.proceedings.com/24631webtoc.pdf

U2 - 10.1109/ICCITechn.2014.6997351

DO - 10.1109/ICCITechn.2014.6997351

M3 - Conference paper

SP - 496

EP - 501

BT - Proceedings of the 16th International Conference on Computer and Information Technology

A2 - Debnath, Rameswar

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - United States

Y2 - 8 March 2014 through 10 March 2014

ER -

Rahman MG , Islam MZ. iDMI: A novel technique for missing value imputation using a decision tree and expectation-maximization algorithm. In Debnath R, editor, Proceedings of the 16th International Conference on Computer and Information Technology: ICCIT 2013. United States: IEEE, Institute of Electrical and Electronics Engineers. 2014. p. 496-501 doi: 10.1109/ICCITechn.2014.6997351

iDMI: A novel technique for missing value imputation using a decision tree and expectation-maximization algorithm

Abstract

Conference

Access to Document

Other files and links

Fingerprint

Cite this