Gensim: Documents Are Just Numbers
Updated: May 2
Text is messy and understanding relationships between documents is even messier. In this post we are looking at the popular Gensim package, often used for classic NLP work.
This is not meant to be a comprehensive NLP tutorial. It is about what Gensim does well, what it does awkwardly, and when you should reach for something else entirely.
The Basic Setup
Gensim has one core principle: documents are vectors in high-dimensional space.
Everything else really follows from that and basic geometry. Namely, if you can turn text into numbers, you can measure similarity. If you can measure similarity, you can find patterns. If you can find patterns, you can extract topics, build recommendations, or collapse 10,000 research papers into a handful of context clusters.
The library gives you three main things:
Vectorization — Turn “The cat sat on the mat” into [0.2, 0.0, 0.5, ...]
Similarity — Ask “how close are these two documents?”
Topic extraction — Find the hidden themes across a corpus
The important detail is that these vectors are usually sparse. A document may live in a vocabulary of 100,000 possible words, but each individual document only uses a tiny fraction of them. So Gensim usually does not store document vectors as long lists of mostly-zero values. It stores them as compact lists like this:
[(4, 2), (17, 1), (203, 5)]That means: token ID 4 appears twice, token ID 17 appears once, and token ID 203 appears five times. Everything else is assumed to be zero.
This compact representation is one reason Gensim still matters. Modern embedding models usually give you dense vectors — for example, 384 or 768 floating-point numbers for every document. Gensim’s classic models often begin with sparse count-based representations, then transform them into TF-IDF (Term Frequency–Inverse Document Frequency), LSI (Latent Semantic Indexing), LDA (Latent Dirichlet Allocation), Word2Vec, Doc2Vec, or another vector space.
Worth noting here is that Gensim was built for unsupervised learning on large text corpora. It is not a chatbot framework. It is not BERT (Bidirectional Encoder Representations from Transformers). It is a statistical toolkit that predates the transformer era and remains surprisingly relevant.
A good rule of thumb:
If you want to organise, cluster, compare, or summarise patterns across a large collection of documents, Gensim is worth considering.
If you want a model to understand instructions, generate fluent text, or reason conversationally, you are in LLM territory instead.
The Minimal Example
We start with the smallest possible demonstration. Three sentences about cats and dogs.
This is the fundamental pipeline.
Raw text
→ tokenized words
→ dictionary
→ bag-of-words
→ weighted vectorsThe Dictionary object is central, it functions as your vocabulary. Once built, you can reuse it to transform new documents into the same vector space.
There are two practical details hidden inside this small example.
First, the tokenization is deliberately crude. doc.lower().split() is fine for a toy example, but it treats punctuation and morphology badly. "cat", "cats", and "cat." may become different tokens. For a slightly better default, Gensim provides simple_preprocess:
simple_preprocess() lowercases text, removes punctuation, and drops extremely short or long tokens. It is not full linguistic preprocessing, but it is usually better than plain .split() for blog examples, search prototypes, and quick experiments.
Second, doc2bow() ignores words it does not know. That is usually what you want when transforming new documents after training a dictionary, but it can surprise you:
If you want to inspect what was ignored, use return_missing=True:
If you are still building the dictionary and want new words added, use allow_update=True:
updated_bow = dictionary.doc2bow(query_tokens, allow_update=True)This option should be used with care, since it changes the vocabulary. That is useful during corpus construction, but dangerous after you have trained models, because the meaning of the vector space should stay fixed once downstream models depend on it.
For real corpora, you also usually want to trim the vocabulary. This makes little sense for our tiny example above, so we look at it simply as a hypothetical step:
This matters because common words like 'the', 'and', and 'of' often dominate count vectors without adding much meaning. Extremely rare words can also add noise, especially if they are typos, IDs, or one-off names.
TF-IDF then reweights the bag-of-words counts. The intuition is simple:
A word that appears often in one document is important to that document.
A word that appears in almost every document is less useful for distinguishing documents.
You can control some of this behaviour:
This is not necessarily needed at the start, the default TfidfModel(corpus) is a good starting point. But it helps to know that TF-IDF is not some magical algorithm; it is a weighting scheme. This means that the choices made will have an impact on the outcome as in most models.
Similarity: The Actual Use Case
Vector representations are useful because they let you ask: “which documents are similar?”
In this case 'Document 2' ("Cats and dogs are animals") would score highest, while 'Document 1' ("The dog sat on the log") would score zero, as there is no overlapping words after TF-IDF weighting.
This is the core pattern, you vectorize your corpus once, then query it repeatedly. Gensim caches the index in memory, making lookups fast.
In practice, you rarely want raw scores in the original order. You usually want the top matches:
A score near 1.0 means the vectors are very close. A score near 0.0 means little or no overlap in the chosen vector space. Negative values can appear in some transformed spaces, such as LSI, where dimensions are not simple word counts anymore.
For a small corpus, MatrixSimilarity is convenient. It stores the full index in memory. For larger corpora, prefer Similarity, which can shard the index on disk:
The important parameter is num_features. It tells the index how many dimensions the vectors have. For a plain dictionary-based bag-of-words or TF-IDF model, that is usually len(dictionary), potentially after some filtering and cutting of tokens.
You can also query in batches:
Batch querying is the pattern you want when building search results for many incoming documents, comparing thousands of abstracts, or precomputing nearest neighbours.
When this breaks down: TF-IDF only captures word overlap. It does not understand synonyms. As we already mentioned, "cat" and "feline" are treated as completely unrelated. While "car" and "automobile" are unrelated unless both words appear in similar documents and you use a model that can learn that relationship.
For that, you need embeddings (Word2Vec, FastText) or semantic models (LSI, LDA). Even then, classic Gensim models are still not the same as modern transformer embeddings. They are lighter, cheaper, and often more interpretable, but less semantically powerful.
A useful way to think about these different interpretations is:
TF-IDF similarity asks: do these documents use the same important words?
LSI similarity asks: do these documents point in similar latent directions?
Word2Vec/Doc2Vec similarity asks: were these words or documents learned in similar contexts?
Transformer embedding similarity asks: does a large neural model think these texts mean similar things?
Topic Modeling: Latent Dirichlet Allocation
LDA is Gensim’s signature feature. It finds hidden topics in a collection of documents.
The fundamental idea here is that each document is a mixture of topics, and each topic is a mixture of words.
The output is probabilistic in the sense that the highest contributions in the output in topic are the collection that make up the 'topic' and that all contributions are positive. In the above, Topic 0 is about positioning (sat, on). Topic 1 is about classification (animals, cats, dogs).
This toy example is too small for serious topic modeling, but it shows the shape of the result: a topic is not a label, instead it is a weighted list of words. You would still have to name them.
For example:
This gives you a cleaner Python object:
[("animals", 0.154), ("cats", 0.077), ("dogs", 0.077), ...]That is easier to use if you want to build a report, export topic keywords to a table, or generate labels later.
You can assign documents to topics:
For downstream use, you often want the dominant topic:
The minimum_probability=0.0 option forces Gensim to return all topics, even if their probability is tiny. Without it, very small probabilities may be omitted. This might be more convenient if you need a fixed-length feature vector for every document.
For a slightly more realistic model, you might write:
The key parameters are:
num_topics: how many topics you want the model to find.
passes: how many full passes the model makes over the corpus.
iterations: how hard the model works when estimating topic distributions.
chunksize: how many documents are processed in each training chunk.
random_state: makes results more reproducible.
alpha: controls how many topics each document tends to contain. "auto" lets Gensim learn it.
eta: controls how concentrated topics are around particular words. "auto" lets Gensim learn it.
eval_every: controls perplexity evaluation during training. Setting it to None can make training faster when you do not need that diagnostic.
The trade-off: LDA is powerful but requires tuning. The number of topics is not discovered, you have to decide it upfront. If chosen poorly, you will most likely get nonsense, but if chosen well, you get interpretable clusters.
Opinionated take: Start with num_topics equal to roughly sqrt(num_documents) and iterate. In this regard, LDA is more art than science.
That heuristic is intentionally rough. For 10,000 documents, sqrt(10_000) = 100, which may be far too many if your corpus is narrow and far too few if it covers many domains. A more practical workflow is:
Train several models, such as 10, 20, 40, and 80 topics.
Inspect the top words for each topic.
Check whether topics are coherent or duplicated.
Ask whether the topics support the business or research question.
Pick the smallest model that still separates the important themes.
You can also compute coherence scores with gensim.models.CoherenceModel, but do not outsource judgment entirely to the metric. A topic model is useful only if the resulting topics are interpretable for your use case. This is a big difference between generated topics models and other machine learning models, topics are something that has an immediate meaning for a human.
Word2Vec: When Context Matters
TF-IDF treats every word as independent. Word2Vec learns that "king" and "queen" are related because they appear in similar contexts.
This example is intentionally tiny, so do not take the exact output seriously. Word2Vec needs many repeated contexts to learn meaningful geometry. On four sentences, the result is mostly a demonstration of the API.
The training call has several important parameters:
The choices matter:
vector_size: larger vectors can capture more nuance, but need more data.
window: small windows capture syntactic relationships; larger windows capture broader topical relationships.
min_count: filters rare words. In real data, this removes typos and one-off tokens.
workers: speeds up training on multi-core machines.
sg: 1 uses Skip-gram, often better for rare words; 0 uses CBOW, often faster.
negative: controls negative sampling, one of the standard Word2Vec training tricks.
epochs: more epochs mean more training passes, but also more risk of overfitting small corpora.
seed: helps reproducibility, although multi-threading can still introduce small differences.
Once trained, most interaction happens through model.wv, the keyed-vector store:
This is where Gensim shines, Word2Vec training is efficient and memory-conscious. The library was designed to handle millions of sentences without loading everything into RAM.
The architecture: Word2Vec uses a shallow neural network. It predicts surrounding words from a word (Skip-gram) or predicts a word from its surrounding context (CBOW). The learned weights become your word vectors.
When NOT to use: If you need pre-trained embeddings for general English, just download a pre-trained model such as GloVe, FastText, or a Sentence Transformers model. Training Word2Vec only makes sense if you have domain-specific text: medical records, legal documents, support tickets, engineering logs, niche forums, internal documentation, or research abstracts in a specialised field.
For example, in a hospital corpus, "discharge" will mean something different from its meaning in electrical engineering. In a legal corpus, "consideration" is not just ordinary thoughtfulness. Domain-specific Word2Vec can capture those local meanings better than a generic model.
You can save the trained vectors separately from the full model:
Saving only model.wv is useful when you only need lookup and similarity. Saving the full model is useful if you want to continue training later.
Doc2Vec: The Document-Level Extension
While Word2Vec gives you vectors for words, Doc2Vec gives you vectors for entire documents.
The architecture is similar to Word2Vec but adds a document ID as a special token during training. The resulting vector captures the document’s semantic fingerprint.
The tags field matters. It is how you retrieve document vectors later. Tags can be strings, integers, filenames, database IDs, or any stable identifier:
That makes Doc2Vec useful for tasks such as:
finding similar customer support tickets,
grouping research abstracts,
comparing product descriptions,
deduplicating near-identical documents,
recommending related articles.
You can also infer a vector for a new, unseen document:
This is different from retrieving a vector for a document seen during training. For a new document, Doc2Vec has to estimate a vector while keeping the learned model fixed. The number of inference epochs can affect stability.
A more explicit training setup might look like this:
The two-step build_vocab() + train() style is useful when your training process is more complex. It also makes it clearer that Doc2Vec first learns the vocabulary and then trains the model.
Practical note: Doc2Vec is finicky. It requires careful tuning of vector_size, window, epochs, min_count, and dm. If the results look random, you probably need more training data, more iterations, or cleaner preprocessing.
Also remember that document length matters. Very short texts may not provide enough signal. Very long documents may need chunking, because one vector for an entire book or long report can become too blunt.
Streaming: The Underrated Feature
Most NLP libraries load everything into memory, Gensim has the option of using iterators to avoid this. So, if you have 10GB of text, you can process it line-by-line without ever holding the full dataset in RAM.
This is the design philosophy that is very much in keeping with Python: everything is an iterator. You can chain transformations without materializing intermediate results.
A subtle point here is that MyCorpus() is reusable because it creates a fresh file handle every time iter() is called. A plain generator expression is usually consumed after one pass. That is why the example recreates corpus = MyCorpus() before building the TF-IDF model.
A slightly more realistic streaming corpus might include preprocessing and encoding handling:
Notice that the corpus is created again when needed. This is common in Gensim. A corpus is not necessarily a list; it is often a recipe for producing documents one at a time.
You can also persist a streamed corpus to disk in Matrix Market format:
That gives you a reusable corpus file without converting the whole dataset into a giant Python list.
Why this matters: Most real-world NLP involves large corpora. Gensim was built for researchers dealing with millions of Wikipedia articles, not toy datasets.
Streaming is also helpful for smaller projects because it forces a clean separation between:
how raw text is read,
how it is tokenized,
how the dictionary is built,
how vectors are generated,
how models consume those vectors.
That separation makes the pipeline easier to debug.
LSI: The Dimensionality Reducer
Latent Semantic Indexing (LSI) is like PCA for text. It compresses your document vectors into a lower-dimensional space while preserving similarity.
LSI finds the “concepts” that explain the most variance across documents. Unlike LDA, it is deterministic and faster to train.
In practice, LSI is often used after TF-IDF:
The idea is:
Start with sparse word counts.
Reweight them using TF-IDF.
Compress them into fewer latent dimensions.
Use those dimensions for similarity search or clustering.
You can inspect the discovered dimensions:
The output looks topic-like, but it is not probabilistic in the same way as LDA. LSI dimensions can have positive and negative word weights. For example going back to our simplest example, the following would be a possible part of the outcome
-0.500*"dog" + -0.500*"log" + 0.500*"mat" + 0.500*"cat" + -0.000*"the" + 0.000*"and" + 0.000*"on" + 0.000*"are"That might mean one direction separates animal and object related words from filler words. Useful mathematically, but less intuitive in most cases.
A similarity pipeline with LSI looks like this, continuing with the setup we had just above:
Here num_features=lsi.num_topics, because the vectors now live in the compressed LSI vector space, not the original vocabulary vector space.
The real difference: LDA gives you interpretable topics as mixtures of words. LSI gives you abstract dimensions. If you need to explain your model and especially if you need to explain the different vectors, use LDA. If you just need similarity search, LSI is often good enough.
LSI is especially attractive when:
the corpus is medium-to-large,
exact interpretability is not the main goal,
you want a fast dimensionality reduction step,
TF-IDF word overlap is too brittle,
transformer embeddings feel too heavy for the problem.
When NOT to Use Gensim
Gensim is not the right tool if you need:
1. Pre-trained transformers — Use Hugging Face Transformers for BERT, GPT, or T5. Gensim does not have these.
2. Classification — Gensim is unsupervised. For supervised tasks such as spam detection, sentiment analysis, intent classification, or churn prediction from text features, use scikit-learn, spaCy, or a transformer-based classifier.
3. Production APIs — Gensim models can be memory-intensive. Serving them at scale requires careful architecture. Consider exporting vectors to a dedicated search engine such as Elasticsearch, OpenSearch, FAISS, Qdrant, Weaviate, or another vector database.
4. Real-time chat — Gensim is batch-oriented. If you need conversational AI, look elsewhere.
5. Multilingual embeddings — Gensim supports multilingual workflows if you provide the data, but libraries like Sentence Transformers handle cross-lingual semantic tasks more elegantly.
There are also softer reasons not to use it.
If your dataset is tiny, topic modeling may produce impressive-looking nonsense that has no content. If your audience expects natural-language explanations, LDA topics may not be enough. If your search use case depends on meaning rather than word overlap, a modern embedding model may give better results with less tuning.
The question is not “Is Gensim old?” It is. The better question is: “Is this problem still well served by classic vector-space NLP?” In many cases, the answer may still be yes.
For example:
clustering a backlog of support tickets,
finding duplicate knowledge-base articles,
exploring themes in survey responses,
discovering topics in research abstracts,
building a lightweight recommendation layer,
preparing candidate documents before an LLM reads them.
For those tasks, Gensim can be refreshingly direct and it does not need any LLM tokens to be used.
The Composability Advantage
Gensim’s API is designed to chain transformations.
Each model is a function. You pipe data through them. This makes experimentation fast — swap TF-IDF for Word2Vec, swap LSI for LDA, without rewriting your pipeline.
The one-line query is compact, even if during development you might prefer the more verbose version of changing the quey step by step. It is certanly easier to debug and you can immediately look at intermediate questions:
Did tokenization behave as expected?
Were important words missing from the dictionary?
Did TF-IDF remove or downweight common words?
Did the LSI vector contain useful dimensions?
That is often where the real problem lives. The model may be fine; the preprocessing may not be.
Worth noting: This composability extends to serialization. Save any model with .save() and load it later. The entire pipeline can be persisted to disk.
If you saved a corpus separately, you can load that too:
Then rebuild or load the similarity index:
A full small pipeline might look like this:
That is the way Gensim is designed to be used. You define small reusable transformations, then compose them into a pipeline.
Gensim vs Modern Tools
Gensim occupies a specific niche, namely statistical NLP for large corpora. It will not give you GPT-level language understanding or build chatbots. But if you need to cluster 50,000 customer reviews, find similar research papers, or extract themes from a decade of forum posts, Gensim is a reasonable first choice.
The library’s greatest strength is that it forces you to think about text as vectors. Which, as a concept, transfers directly to modern embeddings and neural search systems.
That is why using Gensim is still useful even if everything tries to be LLM-based. Retrieval-augmented generation, semantic search, reranking, clustering, topic discovery, and recommendation systems all rely on the same basic idea: turn messy text into a space where distance means something.
It can also be combined with LLM-based systems. In those settings Gensim can be used for retrieval before sending documents to an LLM. The combination is surprisingly powerful and saves on token use for the LLM.
A practical hybrid workflow might be:
Use Gensim to cluster or retrieve candidate documents.
Use an LLM to summarise, explain, or compare the selected documents.
Keep Gensim for the cheap, repeatable vector-space work.
Reserve the LLM for the expensive language-heavy step.
That division of labour is often better than asking an LLM to read everything.
Comments