LDA

This library implements a form of text clustering and topic modeling called Latent Dirichlet Allocation.

In order to use it, you have to have a seq of documents, each one being itself a seq of strings. These documents can then be indexed through the use of a vocabulary, as follows:

import sequtils, strutils
import lda

let
  rawDocs = @[
      "eat turkey on turkey day holiday",
      "i like to eat cake on holiday",
      "turkey trot race on thanksgiving holiday",
      "snail race the turtle",
      "time travel space race",
      "movie on thanksgiving",
      "movie at air and space museum is cool movie",
      "aspiring movie star"
    ]
  docWords = rawDocs.mapIt(it.split(' '))
  vocab = makeVocab(docWords)
  docs = makeDocs(docWords, vocab)

Once you have the vocabulary vocab , which is just the seq of all word appearing through all documents, and the preoprocessed documents, which are a nested sequence of integer indices, you can traing the model through Collapsed Gibbs Sampling using

let ldaResult = lda(docs, vocabLen = vocab.len, K = 3, iterations = 1000)

Here K denotes the number of desired topics and iterations the number of rounds in the training phase. The result contains a document/topic matrix and a word/topic matrix. These can be used to find the most descriptive words for a topic:

for t in 0 ..< 3:
  echo "TOPIC ", t
  echo bestWords(ldaResult, vocab, t)

or to find the most relevant topics for a document:

for d in 0 ..< docs.len:
  echo "> ", rawDocs[d]
  echo "topic: ", ldaResult.bestTopic(d)

or even to generate text with the same topic distribution as a given document:

echo sample(ldaResult, vocab, doc = 6)

TODO

• parallel training
• variational Bayes sampling
• modified model to account for stop words