1 · Language models

Subtitles

1. Tokens, embeddings, and positions
2. Self-attention and causal masks
3. Next-token training and generation
4. Prompting and instruction tuning
5. RAG versus fine-tuning
6. LoRA, quantization, and QLoRA
7. Biomedical evaluation

Milestones

• Trace a token from embedding to next-token prediction.
• Diagnose positional and future-information leakage.
• Choose prompting, RAG, LoRA, or QLoRA for a use case.
• Check LoRA shapes and parameter efficiency.
• Design a factual evaluation beyond average token loss.

Possible exam questions

OPEN · Q1 — How can model A have lower average next-token loss but more factual errors than B? What does the loss establish, can greedy decoding fix it, and how would clinicians compare the letters?

OPEN · Q2 — Can a Transformer locate “left/right opacity” if position vectors are added before the Transformer? What if they are added only afterwards?

OPEN · Q3 — If only the second attention layer has a causal mask, can the representation at “is” already contain information from “absent”?

OPEN · Q4 — How should a system use frequently changing guidelines that exceed the context window and require source/version citations? Why not only fine-tune, and how could it still fail?

OPEN · Q5 — Should three departments pool LoRA adapters when aggregate correctness rises but two departments fall? How can separate adapters share one frozen base model?

RECALL · Q6 — In autoregressive pretraining, where do the prediction targets come from?

RECALL · Q7 — In \(h_i^{(0)} = e(x_i) + p_i\), what information does \(p_i\) supply?

RECALL · Q8 — What distinguishes self-attention from cross-attention?

RECALL · Q9 — What distinguishes zero-shot from few-shot prompting?

RECALL · Q10 — For \(W \in \mathbb{R}^{d \times k}\) and \(\Delta W = BA\) of rank at most \(r\), which shapes must \(A\) and \(B\) have?