AI: UNESCO study confirms reproduction of stereotypes

Large Language Models (LLMs) are machine learning models that have been specially developed to understand, generate and react to human language. In a matter of seconds, LLMs such as GPT-4 from OpenAI and Gemini from Google spit out texts, analyze images and translate content. These models are based on the so-called Transformer architecture, which was first introduced in 2017 and is particularly good at processing natural language - a process also known as natural language processing.

LLMs are trained with the help of data from the internet, for example Wikipedia and social media articles as well as news articles. Based on the learned data, LLMs generate new, coherent and contextually appropriate texts. This makes them powerful tools in many areas of application. However, their use also harbors risks. For example, Handelsblatt recently reported that the Google AI Gemini had to be temporarily deactivated.

The reason: when asked to generate images of Nazis, the AI showed black and Asian people in Nazi uniforms, among other things. A study recently published by UNESCO also warns of the dangers of AI.

Here you can read more about the UNESCO study entitled "Challenging Systematic Prejudices: An Investigation into Bias Against Women and Girls in Large Language Models" and

• about the risks of using AI and
• How to avoid bias in data-based personas

UNESCO study confirms reproduction of stereotypes

The UNESCO study, entitled "Challenging systematic prejudices: an investigation into bias against women and girls in large language models", was conducted by an international research group including scientists from University College London and the International Research Centre on Artificial Intelligence (IRCAI). The research, funded by the European Union under the Horizon 2020 research and innovation program, focused on stereotypes in LLMs such as GPT-3.5 and GPT-4 from OpenAI and LLaMA 2 from Meta.

The aim was to investigate the way in which these models are influenced by the social biases present in their training data and how this affects the content they generate. The study provides insights into the way artificial intelligence can reproduce gender stereotypes, racist clichés and homophobic content.

Gender stereotypes

According to the UNESCO study, LLMs tend to portray women in traditionally less valued or stigmatized roles, such as "domestic worker", "cook" and "prostitute". Men, on the other hand, are more often associated with professions such as "engineer", "teacher" and "doctor". In stories generated by ChatGPT and LLaMA 2, male topics are dominated by words such as "treasure", "wood" and "sea", while female topics are more frequently associated with words such as "garden", "love" and "husband". These stereotypical attributions reflect outdated social views that are perpetuated by the use of AI.

Racism

Another alarming result: large language models reproduce racist stereotypes. For example, British men were assigned higher-ranking occupations such as "doctor" and "bank clerk", while the AI models assigned men belonging to the South African Zulu ethnic group with occupations such as "gardener" and "security guard". Also, in 20% of the generated texts, women belonging to the Zulu ethnic group were assigned roles such as "servants", "cooks" and "housekeepers".

Homophobia

The UNESCO study shows that LLMs have a clear tendency to generate homophobic content. In particular, LLaMA 2 was found to generate negative content about and around homosexuality in around 70% of cases and GPT-2 in around 60% of cases. Some of the statements generated by LLaMA 2 included phrases such as "homosexuals were considered the lowest in the social hierarchy" and "homosexuals were considered freaks".

Garbage in, garbage out: What affects the quality of AI-generated results?

The quality of the data that is fed into artificial intelligence systems determines what results the systems deliver. This principle is known as "garbage in, garbage out" - if the data is incorrect, distorted or incomplete, the results generated by the AI will also be problematic.

The reason: AIs learn and develop their skills based on the data they are provided with. If this data does not accurately or fairly represent a certain group of people or contains historical inaccuracies, the AI will reproduce discriminatory representations or invent new, false facts. An example of this is the "Savoy cabbage" project. Here, an AI was used to generate biographies. The resulting texts contained fictitious information which, if considered real, could lead to a false representation of historical facts.

Biases in the data can come from a variety of sources, including the predominant demographics of the data curators or the historical data itself. For example, an analysis of Wikipedia has shown that women are more often portrayed in the articles in their role as wives and mothers, while men's portrayal of achievement is more prominent. AI systems that are trained with this data reproduce stereotypical ideas.

Recognize and avoid problematic applications

AI technologies should by no means be viewed in isolation from the social structures in which they are developed and used. They are mirrors and amplifiers of the cultural and social dynamics that they "learn".

In order to recognize and correct distortions, you should

• AI should be trained with data that is as diverse and representative as possible.
• AI must be continuously adapted and reviewed in order to identify and eliminate existing and new biases.
• Promote diversity among developers and decision-makers in the AI industry. Studies show that diverse teams are less biased and create more inclusive products.

AI systems should also be subjected to regular fairness audits and bias tests. With the help of tools such as Watson OpenScale from IBM, developers can check algorithms for bias and make appropriate adjustments.

Education also plays an essential role in combating bias in AI systems. Educational programs that raise awareness of bias in AI and spread knowledge about ethical AI practices are essential. Events such as the "Queer AI and Wikipedia" conference serve to initiate discussions and strengthen communities advocating for ethical AI.

We have looked at what this means for the use of artificial intelligence to create personas in a separate blog post.