Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

REMIAN

REMIAN Missing value (MV) imputation is a critical preprocessing means for data mining. Nevertheless, existing MV imputation methods are mostly designed for batch processing, and thus are not applicable to streaming data, especially those with poor quality. In this article, we propose a framework, called Real-time and Error-tolerant Missing vAlue ImputatioN (REMAIN), to impute MVs in poor-quality streaming data. Instead of imputing MVs based on all the observed data, REMAIN first initializes the MV imputation model based on a-RANSAC which is capable of detecting and rejecting anomalies in an efficient manner, and then incrementally updates the model parameters upon the arrival of new data to support real-time MV imputation. As the correlations among attributes of the data may change over time in unforseenable ways, we devise a deterioration detection mechanism to capture the deterioration of the imputation model to further improve the imputation accuracy. Finally, we conduct an extensive evaluation on the proposed algorithms using real-world and synthetic datasets. Experimental results demonstrate that REMAIN achieves significantly higher imputation accuracy over existing solutions. Meanwhile, REMAIN improves up to one order of magnitude in time cost compared with existing approaches. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png ACM Transactions on Knowledge Discovery from Data (TKDD) Association for Computing Machinery

REMIAN

Download PDF
38 pages

Loading next page...
 
/lp/association-for-computing-machinery/remian-ypOdLkhrlH

References

References for this paper are not available at this time. We will be adding them shortly, thank you for your patience.

Publisher
Association for Computing Machinery
Copyright
Copyright © 2020 ACM
ISSN
1556-4681
eISSN
1556-472X
DOI
10.1145/3412364
Publisher site
See Article on Publisher Site

Abstract

Missing value (MV) imputation is a critical preprocessing means for data mining. Nevertheless, existing MV imputation methods are mostly designed for batch processing, and thus are not applicable to streaming data, especially those with poor quality. In this article, we propose a framework, called Real-time and Error-tolerant Missing vAlue ImputatioN (REMAIN), to impute MVs in poor-quality streaming data. Instead of imputing MVs based on all the observed data, REMAIN first initializes the MV imputation model based on a-RANSAC which is capable of detecting and rejecting anomalies in an efficient manner, and then incrementally updates the model parameters upon the arrival of new data to support real-time MV imputation. As the correlations among attributes of the data may change over time in unforseenable ways, we devise a deterioration detection mechanism to capture the deterioration of the imputation model to further improve the imputation accuracy. Finally, we conduct an extensive evaluation on the proposed algorithms using real-world and synthetic datasets. Experimental results demonstrate that REMAIN achieves significantly higher imputation accuracy over existing solutions. Meanwhile, REMAIN improves up to one order of magnitude in time cost compared with existing approaches.

Journal

ACM Transactions on Knowledge Discovery from Data (TKDD) – Association for Computing Machinery

Published: Sep 28, 2020

Keywords: Missing value

References

  • Dealing with missing values in large-scale studies: Microarray data imputation and beyond
    Tero, A.
  • GOOWE: Geometrically optimum and online-weighted ensemble classifier for evolving data streams
    Hamed, R. B.; Fazli, C.
  • Time series anomaly detection using multiple statistical models
    Jyoti, B.
  • Densified winner take all (WTA) hashing for sparse datasets
    Beidi, C.; Anshumali, S.
  • Bursty event detection in twitter streams
    Carmela, C.; Agostino, F.; Clara, P.
  • Learning in the model space for cognitive fault diagnosis
    Huanhuan, C.; Peter, T.; Ali, R.; Xin, Y.
  • Model-based kernel for efficient time series analysis
    Huanhuan, C.; Peter, T.; Ali, R.; Xin, Y.
  • Voila: Visual anomaly detection and monitoring with streaming spatiotemporal data
    Nan, C.; Chaoguang, L.; Qiuhan, Z.; Yu-Ru, L.; Xian, Tand; Xidao, W.
  • Anomaly detection: A survey
    Varun, C.; Arindam, B.; Vipin, K.
  • Overview of the RANSAC algorithm
    Derpanis, K. G.
  • Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography
    Martin, A. F.; Robert, C. B.
  • The yule walker equations for the AR coefficients
    Eshel, G.
  • A survey on concept drift adaptation
    João, G.; Indre, Z.; Albert, B.; Mykola, P.; Abdelhamid, B.
  • A survey of parallel sequential pattern mining
    Wensheng, G.; Jerry, C. W. L.; Philippe, F. V.; Han-Chieh, C.; Philip, S. Y.
  • Deep learning for user interest and response prediction in online display advertising
    Zhabiz, G.; Xingquan, Z.; Arthur, H.; Michael, C.
  • Anomaly detection in streaming environmental sensor data: A data-driven modeling approach
    David, J. H.; Barbara, S. M.
  • Quantitative data cleaning for large databases
    Joseph, M. H.
  • Probability and Statistical Inference
    Hogg, R. V.; Tanis, E. A.; Zimmerman, D. L.
  • Min max normalization based data perturbation method for privacy protection
    Kumar, Y. J.; Santosh, K. B.
  • Generic and scalable framework for automated time-series anomaly detection
    Nikolay, L.; Saeed, A.; Lan, F.
  • ERACER: A database approach for statistical inference and data cleaning
    Chris, M.; Jennifer, N.; Sunil, P.
  • Order-sensitive imputation for clustered missing values
    Qian, M.; Yu, G.; Wang-Chien, L.; Ge, Y.
  • An outlier analysis framework for impedance-based structural health monitoring
    Gyuhae, P.; Amanda, C. R.; Hoon, S.; Charles, R. F.
  • A mathematica version of zeilberger’s algorithm for proving binomial coefficient identities
    Peter, P.; Markus, S.
  • Streaming pattern discovery in multiple time-series
    Spiros, P.; Jimeng, S.; Christos, F.
  • Forecasts of market shares from VAR and BVAR models: A comparison of their accuracy
    Ramos, F. F. Ribeiro
  • Time series anomaly detection: Detection of anomalous drops with limited features and sparse examples in noisy highly periodic data
    Dominique, T. S.; Jason, M. G.; Paolo, M. P.; Stephen, T. E.
  • SCREEN: Stream data cleaning under speed constraints
    Shaoxu, S.; Aoqian, Z.; Jianmin, W.; Philip, S. Y.
  • Using classifier-based nominal imputation to improve machine learning
    Xiaoyuan, S.; Russell, G.; Taghi, M. K.; Amri, N.
  • Generalization of the matrix inversion lemma
    Daniel, J. T.; Guy, R. L. S.
  • Graph matching based on spectral embedding with missing value
    Jin, T.; Bo, J.; Aihua, Z.; Bin, L.
  • On field calibration of an electronic nose for benzene estimation in an urban pollution monitoring scenario
    Saverio, D. V.; Ettore, M.; Marco, P.; Luca, M.; Girolamo, D. F.
  • Concept drift detection for streaming data
    Heng, W.; Zubin, A.
  • Continuous imputation of missing values in streams of pattern-determining time series
    Kevin, W.; Michael, H. B.; Anton, D.; Johann, G.; Hannes, M.
  • Online data mining for co-evolving time sequences
    Byoung-Kee, Y.; Nikolaos, S. D.; Theodore, J.; Jagadish, H. V.; Christos, F.; Alexandros, B.
  • Deep learning: A generic approach for extreme condition traffic forecasting
    Rose, Y.; Yaguang, L.; Cyrus, S.; Ugur, D.; Yan, L.
  • Multiple imputation for missing data: Concepts and new development (Version 9
    Yang, C. Y.
  • Learning individual models for imputation
    Aoqian, Z.; Shaoxu, S.; Yu, S.; Jianmin, W.
  • Time series data cleaning: From anomaly detection to anomaly repairing
    Aoqian, Z.; Shaoxu, S.; Jianmin, W.; Philip, S. Y.
  • GBKII: An imputation method for missing values
    Chengqi, Z.; Xiaofeng, Z.; Jilian, Z.; Yongsong, Q.; Shichao, Z.
  • Optimizing regression models for data streams with missing values
    Indre, Z.; Jaakko, H.
  • Shell-neighbor method and its application in missing data imputation
    Shichao, Z.
  • Missing data imputation by utilizing information within incomplete instances
    Shichao, Z.; Zhi, J.; Xiaofeng, Z.
  • “Missing is useful”: Missing values in cost-sensitive decision trees
    Shichao, Z.; Zhenxing, Q.; Charles, X. L.; Shengli, S.
  • Missing value imputation based on data clustering
    Shichao, Z.; Jilian, Z.; Xiaofeng, Z.; Yongsong, Q.; Chengqi, Z.
  • Efficient utilization of missing data in cost-sensitive learning
    Xiaofeng, Z.; Jianye, Y.; Chengyuan, Z.; Shichao, Z.
  • Missing value estimation for mixed-attribute data sets
    Xiaofeng, Z.; Shichao, Z.; Zhi, J.; Zili, Z.; Zhuoming, X.
  • Missing value estimation for mixed-attribute data sets
    Xiaofeng, Z.; Shichao, Z.; Zhi, J.; Zili, Z.; Zhuoming, X.