This package contains scripts and python tools for running an online implementation of LDA.

Directory Structure and Files

• input: the input directory; contains example input files for testing the program.
• lda.py: main program that runs online LDA.
• run_lda.sh: script that drives the online lda system.
• stopwords.txt: a list of common stopwords to remove when generating the vocabulary.
• vocabulary.py: complementary program that manages the update of vocabulary in documents.

Running the System

• Generate input files according to the input format in the input directory.
• Execute run_lda.sh.
• System output are generated in output-time_slice directories.

Input Format

• time_slice.text: text of the documents, one line per document;
• time_slice.time: time information of the documents, each line maps to the document that has the same line number in time_slice.text.

Parameter Settings

Most parameter values (number of cores to use, minimum frequency threshold of vocabulary, etc) are set in lda.py. The number of topics, T, is specified in run_lda.sh. The size of the sliding window is fixed at 2 time slices. Modification of the code is required to change this parameter.

Credits & Licensing

• The original author of the LDA implementation is Shuyo NAKATANI.
• License: MIT license - http://opensource.org/licenses/MIT.

Publications

• Jey Han Lau, Nigel Collier and Timothy Baldwin (2012). On-line Trend Analysis with Topic Models: #twitter trends detection topic model online. In Proceedings of the 24th International Conference on Computational Linguistics (COLING 2012), Mumbai, India.