Fig. 1

Schematic representation of the neural embedding model for sentences (supervised and unsupervised) consisting of two embedding layers and a final softmax layer over k classes (for the supervised case). In the unsupervised case \(k=|\mathcal {V}|\) and the softmax outputs the probability of the target word (over all vocabulary, as in C-BOW model) given its context: fixed-length context for fastText and entire sentence context for sent2vec. Independently of the training mode (e.g., supervised vs unsupervised) word embeddings are stored as columns in the weight matrix V of the first embedding layer. Note that in the unsupervised case the rows of the weight matrix U of the second embedding layer represent the embeddings for the “negative” words; these embeddings however are not used for the downstream machine learning tasks. In all instances the averaging of embeddings of constituent tokens (\(\mathcal {\hat {\iota }}_{S}\)) is performed by fastText (sent2vec implementation is based on fastText)