Depth

Transformers quiz

All 11 questions. After you submit, each question shows the right answer and why. Back to the quick check

ExplorerQuestion 1 of 11

The Transformer architecture from 2017 works by passing information through attention rather than by reading words strictly one at a time.

True

False

InvestigatorQuestion 2 of 11

In attention, what is the output for a token?

The token with the longest spelling

A random choice of one other token

A weighted sum of value vectors, with weights from comparing its query to every key

The average of all tokens, always with equal weights

InvestigatorQuestion 3 of 11

By 2020, researchers described the Transformer as the standard architecture for natural-language processing tasks.

True

False

InvestigatorQuestion 4 of 11

Why does a transformer need positional encodings?

To translate between languages

To make the model run faster on GPUs

Because without recurrence or convolution it has no other way to know the order of the tokens

To store the dictionary

ScientistQuestion 5 of 11

In scaled dot-product attention, what are the query–key dot products divided by before the softmax, and why?

The sentence length, so long sentences count less

√dₖ, so large dot products don't push the softmax into regions with extremely small gradients

Nothing — they are used unchanged

The number of layers, to make the model deeper

ScientistQuestion 6 of 11

Two keys give scaled attention scores of 1 and 0. What softmax weight goes to the first key? (3 decimal places)

Your answer

AdvancedQuestion 7 of 11

In the Transformer decoder, the prediction for a position may attend to later positions in the output so it can see what comes next.

True

False

AdvancedQuestion 8 of 11

How does the per-layer computational cost of self-attention grow with sequence length n?

Quadratically, O(n²·d)

Exponentially

Linearly, O(n·d²), like a recurrent layer

It does not depend on n

AdvancedQuestion 9 of 11

What distinguishes BERT's pre-training from a left-to-right language model?

It is trained only on labelled data

It reads text right-to-left only

It uses no attention at all

It conditions on both left and right context in all layers

ExpertQuestion 10 of 11

In the original Transformer, the model dimension was 512 and there were h = 8 heads. What was the per-head key dimension dₖ?

Your answer

ExpertQuestion 11 of 11

What bottleneck did Bahdanau et al. (2014) identify in encoder–decoder translation models, and how did their attention mechanism address it?

Your answer