Statistical Optimization of Training Data for Semi-Supervised Text Document Clustering

No Thumbnail Available
Newbold, Cody Renae
Journal Title
Journal ISSN
Volume Title
Middle Tennessee State University
Unsupervised machine learning algorithms suffer from uncertainty that results are accurate or useful. In particular, text document clustering algorithms such as Latent Dirichlet Allocation (LDA) and Latent Semantic Analysis (LSA) give no guarantee that documents are clustered in a manner similar to human readers. Using a semi-supervised approach on text document clustering, we show that the selection of training data can be statistically optimized using LDA and LSA. Using this method, a human reader categorizes a percentage of the data as an analysis step, then feeds the partially-labeled data into bootstrap training and testing steps. Using mutual information to discover which documents were better for training, the algorithm does a post-processing step using the optimized training set. The results show that mutual information values are higher when the statistically optimized training set is used and indicate that human-like performance is better achieved with optimized training data.
Afghan war diary, Latent dirichlet allocation, Latent semantic analysis, Topic modeling