Cs224n lecture 19 20 future of nlp and deep learning
Lecture 20 - Future of NLP and Deep Learning
Kevin Clarke- Areas recently taking off
** **
Harnessing Unlabeled Data - **Backtranslation and unsupervised machine translation**
Monolingual text is easier to acquire! Use a large corpus of unlabeled data and use it to help our billingual translation model.
Pre-training: 1. Separately train encoder and decoder as langauge models 2. Then train jointly on billingual data. Super effective.
Problem with pre-retraining: no “interaction” ebtween the two sentences,
Self-Training: label unlabeled data to get noisy training examples. (Not used very common, its a bad idea, but it inspires back-translation).
Back-Translation
Back-translation helps a lot.
What if there is no billingual data?
Word to word unsupervised translation
Unsupervised Word Translation using cross-lingual word embeddings - shard embeddings space for both languages. Keep word vectors close to their translations from monolingual corpora.
FIrst run word2vec on monolingual corpora, getting word embeddings X and Y.
Learn an orthogonal (rotation) metric such that WX ~ Y.
Learn W with adversarial training:
Dsicriminator: Predict if an embedding is from Y or it is a transformed embedding Wx originally from X. Train W so the discriminator gets “confused”.
Sentence to sentence unsupervised translation
Model: same encoder-decoder model (without attention) used for both languages
- Initialize with cross-lingual word embedding, such that the input and ourput can be English or French. Feed in a special token <Fr> that tells the model to generate in French (or english, or whatever)
Now, this model can be used both as a translation model as well as an autoencoder, because the input can be English or French.
** **
Train it on two objectives:
Training Objective 1: De-noising auto encoder
-
Scramble up the words a little bit and ask the model to reproduce the original sentence before it was scrambled.
-
This ensures that the model can contain all the informaiton about the sentence in the final embedding from the vector produced by the encoder.
Training Objective 2: Back translation
First translate fr -> en
Then use as a “supervised example” to train “en -> fr”
- Why does this work? Cross lingual embeddings and shared encoder gives model a starting point.
Results
** **
**Notes: Attribute transfer: **
You can use the same appraoch above to do attribute transfer, where the attributes now are not English/French but for example, relaxed, annoyed. Data from Twitter.
**Notes: **Doesn’t work so well in languages with very different grammars
** **
Notes: Cross-Lingual BERT
Google concatenated a whole bunch of corpora in differnet langauges and then trained one BERT language model using BERT’s maskLM method.
Facebook porposed to combine this MaskedLM objective with translation
Leads to very very substantial improvements in unsupervised machine translation.
They took cross-lingual BERT and take it as input to an unsupervised machine translation system, and got the
current SOTA in unsupervised machine translation.
Huge Models: GPT-2, **Scaled up pre-training**
StyleGAN is in-between the size of EMLO and BERT
Data Parallelism - Each GPU has a copy of the model, split the minibatch across these models. So if you have 16 GPUs, each sees a batch size of 32, you can aggregate the gradeints of these 16 GPUs and you effectively get a batch size of 512 - allows you to train models much faster.
Model Parallelism - Models get so big they can’t fit on a single GPU. In this case, you ahve to split up the model across multiple compute units, that’s what’s done for models the size of GPT-2. There are new frameworks such as Mesh-Tensorflow which are basically designed to make this model parallelism easier.
GPT2
Just a really big Transformer ML, trained on 40GB of text (10x larger than what previous langauge models has been trained on), quite a lot of effort going into making sure the dataset is good quality; take webpages form reddit links with high karma.
X-axis is log(model size), Y-axis is accuracy. Dotted lines are existing works on these tasks. (How could it be doing translation?- Appendix 1)
GPT2 is much worse at Question answering - it only gets 4% accuracy. However, look at GPT2 QA results, it seems like GPT2 has learned some world knowledge about the world.
Maybe 1 Trillion parameter models will be possible in ~10 years.
Social Ramifications of GPT2 (Appendix 2).
What’s next for NLP?
A lot of the current research “trends” are reactions to BERT. What did BERT “solve” and what do we work on next?
Architecture engineering isn’t providing gains for time improvement!
We need harder NLU tasks. BERT has almost solved SQuAD
New Datasets:
**
**
New Areas of NLP
**Multi-Task Learning - **Another fronteir of NLP is getting one model to perform many tasks. GLUE and DecaNLP are recent examples
Multi-task learning yields improvements on top of BERT.
BERT is a very large model, a better way to use that model is to get it to do many things at once.
Low-Resource Settings
Models that don’t require lots of compute power (can’t use BERT) - especially for mobile devices.
Low-resource langauges
Low-data settings (few-shot learning).
Interpreting/Understanding Models
Can we get explainations for model predictions?
Can we understand what models like BERT know and why they work so well?
Rapidly growing area in NLP, very important for some applications (e.g. healthcare).
Usually tested using Diagnostic/Probing Classifiers - take a model like BERT and run it through several tasks, each of which test for one particular thing, like “does this model know about syntax, or POS”?
This is done by training another classifier on top of BERT to do, say, POS tagging, but we only backprop into the diagnostic classifier itself (we are treating the output of BERT as a fixed input).
The different layers of BERT seem to be representing pretty well the differnet layers of lingustic structure. For example, dependency parsing is a medium-level tasks, and it seems like the medium layers of BERT help most with dependency parsing.
NLP in Industry
NLP is rapidly growing in industry:
-
Dialogue (Chatbots, customer service)
-
Health Care: Understanding health records, understanding biomedical literature.
** **
Lecture 19: Bias in AI Bias in AI - Margaret Mitchell GoogleAI
Prototype - there is “exemplar’ features of objects, i.e. “yellow” bananas.
Human reporting bias - huamns report things not in acordance with real world frequencies
Appendices
Appendix 1:
How can GPT-2 do EN-Fr Translation when it was trained on a giant corpus of English data?
Because in some parts of the English corpus there were bits of French, and quite amazingly it picked up bits and pieces of how to translate into french even though that wasn’t the inded object of the training.
Appendix 2:
Debate about should GPT-2 Release their model
Appendix 3: NLP in high stakes decision making
- Judicial decisions, hiring, grading tests.