ALiBi (Attention with Linear Biases)

A positional bias method enabling better extrapolation to longer contexts by adding linear biases to attention scores.