# Word Embeddings¶

Word embedding is a mapping of a word to a d-dimensional vector space. This real valued vector representation captures semantic and syntactic features. Polyglot offers a simple interface to load several formats of word embeddings.

from polyglot.mapping import Embedding


## Formats¶

The Embedding class can read word embeddings from different sources:

• Gensim word2vec objects: (from_gensim method)
• Word2vec binary/text models: (from_word2vec method)
• polyglot pickle files: (load method)
embeddings = Embedding.load("/home/rmyeid/polyglot_data/embeddings2/en/embeddings_pkl.tar.bz2")


## Nearest Neighbors¶

A common way to investigate the space capture by the embeddings is to query for the nearest neightbors of any word.

neighbors = embeddings.nearest_neighbors("green")
neighbors

[u'blue',
u'white',
u'red',
u'yellow',
u'black',
u'grey',
u'purple',
u'pink',
u'light',
u'gray']


to calculate the distance between a word and the nieghbors, we can call the distances method

embeddings.distances("green", neighbors)

array([ 1.34894466,  1.37864077,  1.39504588,  1.39524949,  1.43183875,
1.68007386,  1.75897062,  1.88401115,  1.89186132,  1.902614  ], dtype=float32)


The word embeddings are not unit vectors, actually the more frequent the word is the larger the norm of its own vector.

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

norms = np.linalg.norm(embeddings.vectors, axis=1)
window = 300
smooth_line = np.convolve(norms, np.ones(window)/float(window), mode='valid')
plt.plot(smooth_line)
plt.xlabel("Word Rank"); _ = plt.ylabel("$L_2$ norm")


This could be problematic for some applications and training algorithms. We can normalize them by norms to get unit vectors to reduce effects of word frequency, as the following

embeddings = embeddings.normalize_words()

neighbors = embeddings.nearest_neighbors("green")
for w,d in zip(neighbors, embeddings.distances("green", neighbors)):
print("{:<8}{:.4f}".format(w,d))

white   0.4261
blue    0.4451
black   0.4591
red     0.4786
yellow  0.4947
grey    0.6072
purple  0.6392
light   0.6483
pink    0.6574
colour  0.6824


## Vocabulary Expansion¶

from polyglot.mapping import CaseExpander, DigitExpander


Not all the words are available in the dictionary defined by the word embeddings. Sometimes it would be useful to map new words to similar ones that we have embeddings for.

### Case Expansion¶

For example, the word GREEN is not available in the embeddings,

"GREEN" in embeddings

False


we would like to return the vector that represents the word Green, to do that we apply a case expansion:

embeddings.apply_expansion(CaseExpander)

"GREEN" in embeddings

True

embeddings.nearest_neighbors("GREEN")

[u'White',
u'Black',
u'Brown',
u'Blue',
u'Diamond',
u'Wood',
u'Young',
u'Hudson',
u'Cook',
u'Gold']


### Digit Expansion¶

We reduce the size of the vocabulary while training the embeddings by grouping special classes of words. Once common case of such grouping is digits. Every digit in the training corpus get replaced by the symbol #. For example, a number like 123.54 becomes ###.##. Therefore, querying the embedding for a new number like 434 will result in a failure

"434" in embeddings

False


To fix that, we apply another type of vocabulary expansion DigitExpander. It will map any number to a sequence of #s.

embeddings.apply_expansion(DigitExpander)

"434" in embeddings

True


As expected, the neighbors of the new number 434 will be other numbers:

embeddings.nearest_neighbors("434")

[u'##',
u'#',
u'3',
u'#####',
u'#,###',
u'##,###',
u'##EN##',
u'####',
u'###EN###',
u'n']


## Demo¶

Demo is available here.

### Citation¶

This work is a direct implementation of the research being described in the Polyglot: Distributed Word Representations for Multilingual NLP paper. The author of this library strongly encourage you to cite the following paper if you are using this software.

@InProceedings{polyglot:2013:ACL-CoNLL,
author    = {Al-Rfou, Rami  and  Perozzi, Bryan  and  Skiena, Steven},
title     = {Polyglot: Distributed Word Representations for Multilingual NLP},
booktitle = {Proceedings of the Seventeenth Conference on Computational Natural Language Learning},
month     = {August},
year      = {2013},