Loss Masking
A technique used during LLM fine-tuning where the loss is ignored for the instruction prompt and only calculated for the model's generated response.
Think of It Like This
Like a teacher grading a student's test answer but completely ignoring any spelling mistakes in the printed question itself.
By applying a mask (zeros) over the tokens representing the prompt, backpropagation focuses entirely on optimizing the model to generate the correct completion. This prevents the model from wasting capacity learning how to predict user instructions. It significantly improves the quality of instruction-tuned models.