Day 13: Introduction to Transformers

Agenda

• 3:45-4:15pm: Course feedback activity
• 4:15-4:45pm: Discussion of Reading on Bias in Word Embeddings
• 4:45-5:25pm: Introduction to Transformers

Course Feedback

We are at the halfway point. Please fill out this survey to let me know how it’s going for you in the course.

Discussion of Reading on Bias in Word Embeddings

Write the main points or themes of the paper on the board. Here are some guiding questions (use them to the extent they are helpful for good conversation):

• How did the paper attempt to remove bias?
• How well did it work?
• What other strategies could you imagine trying?
• Do you think it’s easier to debias a person or a large language model? Why?
• What questions are you wrestling with about removing bias? What’s the world you want these models to reflect?

Further Reading

• Fair is Better than Sensational: Man is to Doctor as Woman is to Doctor
• Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them

Introduction to Transformers

We’ll start on the homework together by watching one of the 3B1B videos and pausing to talk through each of the steps.

To close, let’s check out Figure 5 in the paper What does BERT look at? An Analysis of BERT’s Attention

ML as Optimization Quiz (for your reference)

If you are interested in taking a look at the quiz from last time on ML as optimization, you can find it using the preceding link.