Gensim

Gensim is a free Python library for building documents and extracted features from the text, which are used to conduct natural language processing and text mining. It is developed and maintained by Radim Řehůřek and his team at RaRe Technologies. Gensim is designed to provide a reliable, efficient, and extensible framework for the implementation of topics of various sizes. Its emphasis lies on scalability, performance, and easy deployment.

Gensim is based on the idea of ‘distance’ or ‘similarity’ between documents, which is built around a corpus of text, or a collection of documents. It uses vector representations to represent documents in order to facilitate the measurement of distances between documents. Gensim offers two main approaches to compare documents: topic models (LDA, LSI) and word embeddings (word2vec, glove).

Topic models are built on the assumption that a document can be represented as a ‘bag of words’—or a list of word tokens, without any structure. Each word corresponds to a topic, and the topic model assigns a weight to each topic in the document. These weights then allow us to compare documents and determine which words and topics are more important.

Word embeddings are more sophisticated representations that take into account context and semantic relationships between words rather than just a bag of words. Word embeddings provide a much more accurate interpretation of language, which is why they are popular in natural language processing and text mining. Word embeddings are used to generate feature vectors from words, enabling the comparison of similarity between words and documents.

Gensim also provides a plethora of helpful utilities, like streaming algorithms, and API access. In addition, its ease-of-use makes it popular with developers and data scientists who want to quickly and painlessly build powerful text mining and natural language processing models.