Cs224u lec 1 2 3 4 custom word vectors, appendix what is nlu

18 April 2020 - 4 mins read time

Distributed Word Representations

There are a lot of ways to make really good word presentations using different corpora, different reweighting paradigms…

• You can look at particular word windows, and also scale w.r.t distance from target word (i.e. the score is 1/n where n is the distance from target word).

• Larger flatter windows will capture more semantic windows. Smaller, more scaled windows capture more syntactic information.

• You often want to reweight your raw co-occurence counts; Often we use PMI (pointwise mutual information).

** **

Distance Measures

• Cosine distance

• Matching based measures (Appendix 1)

Generalizations w.r.t distiance measures. (For other generalizations, see

Appendix 5)

• Euclidean, Jaccard and Dice with raw counts will favor raw frequency over the distributional patterns.

• Euclidean with L2 norms is equivalent to cosine.

• Jaccard and Dice are equivalent w.r.t ranking.

• Both L2 norms and probabiltiy distrbutions can obscure differences in the amount/strength of evidence, which can have an effect on the realiability of coming/normed-euclidean, and KL Divergence. These shortcomings might be addressed through weighting scemes.

• Skew is KL but with a prelinary step that gives special credence to the refernece distribution

Reweighting

Amplify the thigns that are important and trustworthy.

L2 norming and probability distribution (enforce sum=1) are two ways to do this. Another idea is to amplify counts smaller than expected counts and diminish counts (Appendix 3)

* *

Pointwise Mutual Information (PMI) - observed / expected in log space. The most important reweighting factor.

**Positive PMI (the default for generating word embeddings/vector space models) - **

What’s actually used usually is Positive PMI = ppm(X,ij)= max(0,pmi(X,i,j)

PMI is actually undefined when X_ij = 0. Instead of setting log(0)=0, we cut off all values at 0.

Problem with PMI - PMI tends to really amplify really small values that you cannot trust.

GloVE is basically regularized reduced-dimensional PMI.

TfIdf -

Idf amplifies Tf values that occur in very few documents

Tf-Idf distributions tend to look like PPMI. (Appendix 4)

Dimensionality Reduction

Latent Semantic Analysis (LSA) - 1990, aka Truncated SVD.

Essentially SVD.

Autoencoders

** **

GloVE

Roughly speaking, the obectvei is ot learn vectors to works such that their dot product is proportitional to their probability of coocurrence.

Retrofitting: Distributional representations are powerful and easy to obtain, but they tend to reflect only similarity. Can we get word vectors which also reflect some sort of knowledge about the world (encapsulated in a knowledge graph?)

Extensions to Retrofitting

Appendices

Appendix 1:

Matching based distance measures.

• Matching coefficient (sum smallest of two values along all dimensions)

• Jaccard

• Dice (Intersection over union)

• Overlap

Appendix 2:

You can convert the embedding vectors into a probability distribution, and then use KL distribution between columns. ONLY applications

KL Divergence

Note: KL is not symmetric and may blow up if $q_i ~ 0$

Appendix 3:

Appendix 4:

Cell-value distributions in word embedding matrix for different reweighing schemes

Appendix 5:

Appendix 6: Fasttext - naive way to get subword modeling in.

Appendix 7:

The point of all this is that you get better word vectors~

Appendix 8: How does GloVE work; toy example

Glove Distributions are really nicely spaced out!

Appendix 9:

What is NLU?

Bill McCartney - Works AI /ML Org at Apple, Previously at Google, led NLU for Google Assistant

Chris Potts - Stanford Linguistics Prof

Goals of NLU

Develop artificial assistants. A harder task than NLP.

Brief History of NLU - 1960: Pattern Matching with small rule-sets (SLDLU)

1990s- Statistical NLP.

Recently in academia, a resurgence of NLU

Systems are impressive, but far from solved- big breakthroughs lie in future.

SIRI (Appendix 1) - Still doesn’t understand everything.

Banks and Automated Trading

NLU: Traditional Organization

Big themes of NLU: Semantic Parsing, Learning

Requires - Coreference resolution, domain knowledge, discource knowledge, world knowledge.

Lexical Semantics: Meanings of Words

Compositional Semantics: Meanings of Sentenecs

Language in Context: Meanings of dialogies and discources.

Semantic representaions

• Sentiment analysis

• Vector space models (vectors/topic distributions)

• Relation extraction (Larry Page, founded, Google)

• Semantic Parsing

Course goals - to make you the best NLU resourcher and practitioner. To support you in complerleting a porject that is worthy of a presentation at a top NLP conference.

Assignments: Exemplify best practices for NLU projects.

Appendix 1:

How does SIRI work?