Note

One simple test for LLM bias

in

One simple example of bias in large language models is to ask a question like “Who was President in 2003?”

Leading models will provide an answer that’s only relevant to the USA, without seeking a clarification nor caveating the answer. The answers give no mention to the possibility that I may be asking about the President of Ireland or Nigeria, for example. The model is operating on a biased presumption.

This example isn’t particularly surprising, especially considering how pervasive US-centrism is likely to have been in training data and feedback processes. However it demonstrates the need for guard rails, testing and evaluation when integrating with these models.


You might also like