6

Neural network language model - prediction for the word at the center or the right of context words?

On Bengio's paper, the model predicts probability by n words for the next word, like predicting probabilities of "book", "car", etc., by n words before it, like "this", "is", "a", "good". However, in tagging problems in NLP, like those in Collobert's papers, a common setup, the window model, is tag prediction for the center word by surrounding words.

Are there some studies on neural network language models for prediction of word probabilities at the center by surrounding words, like predicting probabilities of word at the center like "a", "the" by context words "this", "is" (at the left) and "good", "car" (at the right)?

Tom
  • 788
  • 8
  • 16

2 Answers2

3

The task of finding missing words in a text sometimes referred to as text imputation, or sentence completion.

One paper exploring it with ANN: Solving Text Imputation Using Recurrent Neural Networks. Arathi Mani. CS224D report. 2016. http://cs224d.stanford.edu/reports/ManiArathi.pdf

In this paper, we have shown that the bidirectional RNN yields the best Levenshetein and perplexity scores out of the three models tested for our missing data problem where we try to impute a single word into a sentence that is missing exactly one word from an unknown location.

One paper comparing several approaches including RNN: Zweig, Geoffrey, John C. Platt, Christopher Meek, Christopher JC Burges, Ainur Yessenalina, and Qiang Liu. "Computational approaches to sentence completion." In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pp. 601-610. Association for Computational Linguistics, 2012. https://scholar.google.com/scholar?cluster=4615153328130310080&hl=en&as_sdt=0,22 ; http://www.aclweb.org/anthology/P/P12/P12-1063.pdf

This paper studies the problem of sentencelevel semantic coherence by answering SAT-style sentence completion questions

Franck Dernoncourt
  • 42,093
  • 30
  • 155
  • 271
0

What you are describing is Tomas Mikolov's Word2vec model Word2vec. His implementation has 2 parts the Skip-gram model and the CBOW model. Paper here

CBOW, which is what you need, is trained to predict the target word t from the contextual words that surround it, c, i.e. the goal is to maximise P(t | c) over the training set Quora nice explanation on his paper

Swan87
  • 111
  • 4
  • 1
    Hi, I have read about CBOW and skip-gram before. Skip-gram is obvious not a language model, and CBOW seems not used as language model after some Internet searches. Look for any studies on CBOW-like language models with perplexity evaluations. – Tom Sep 26 '16 at 08:02