Data pre-processing plays a vital role in data mining for ensuring good quality of data. In general data pre-processing tasks include imputation of missing values, identification of outliers, smoothening out of noisy data and correction of inconsistent data. In this paper, we present an efficient missing value imputation technique called DMI, which makes use of a decision tree and expectation maximization (EM) algorithm. We argue that the correlations among attributes within a horizontal partition of a data set can be higher than the correlations over the whole data set. For some existing algorithms such as EM based imputation (EMI) accuracy of imputation is expected to be better for a data set having higher correlations than a data set having lower correlations. Therefore, our technique (DMI) applies EMI on various horizontal segments (of a data set) where correlations among attributes are high. We evaluate DMI on two publicly available natural data sets by comparing its performance with the performance of EMI. We use various patterns of missing values each having different missing ratios up to 10%. Several evaluation criteria such as coefficient of determination (R 2), Index of agreement (d_2) and root mean squared error (RMSE) are used. Our initial experimental results indicate that DMI performs significantly better than EMI.
|Title of host publication||9th Australasian Data Mining Conference|
|Subtitle of host publication||AusDM 2011|
|Editors||V Estivill-Castro, S Simoff|
|Place of Publication||Sydney, Australia|
|Publisher||Australian Computer Society Inc|
|Number of pages||10|
|Publication status||Published - 2011|
|Event||The 9th Australasian Data Mining Conference: AusDM 2011 - University of Ballarat, Ballarat, Australia|
Duration: 01 Dec 2011 → 02 Dec 2011
|Name||Conferences in Research and Practice in Information Technology Series|
|Publisher||Australian Computer Society|
|Conference||The 9th Australasian Data Mining Conference|
|Period||01/12/11 → 02/12/11|
Rahman, M. G., & Islam, M. Z. (2011). A Decision Tree-based Missing Value Imputation Technique for Data Pre-processing. In V. Estivill-Castro, & S. Simoff (Eds.), 9th Australasian Data Mining Conference: AusDM 2011 (Vol. 121, pp. 41-50). (Conferences in Research and Practice in Information Technology Series; Vol. 121). Australian Computer Society Inc.