Ethics in Natural Language Processing

Of course we would like to expect that machine learning models are objective and neutral, detached from subjective views and opinions. But actually there are a lot of possibilities that our world-views, opinions, stereotypes, etc. enter into the very core of our models. In their paper A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle Suresh and Guttag present a nice framework on how we can understand the different biases which probably find their way into most machine learning models (including NLP). First, we have a look at data generation, as shown in the image below, and go through the different kinds of biases (each with an example at the end):

Historical Bias

Even if we assume that the data is perfectly sampled and measured from the real world, the data reflects the state of the world, i.e. stereotypes and other representational harms are already in there. Garg et al. studied word embeddings learned from large text corpora and demonstrated biases and stereotypes learned from data against women and various ethic minorities. For example, gendered occupation words like “nurse” or “engineer” are highly associated with words that represent women or men, respectively.

Representation Bias

This bias occurs if the developed sampling strategy under-represents some parts of the population, and subsequently fails to generalize well for a subset of the whole population. One very striking example for this bias is the widely used ImageNet with about 1.2 million labeled images. As Shankar et al. pointed out, ImageNet does not evenly sample from the world’s target population. Approximately 45% of the images in ImageNet were taken in the United States, and the majority of the remaining images are from North America or Western Europe. Only 1% and 2.1% of the images come from China and India.

Measurement Bias

When choosing, collecting, or computing features and labels to use in a prediction problem the measurement is not unique. A feature or label is a proxy (a concrete measurement) chosen to approximate some construct (an idea or concept) that is not directly encoded or observable. Most often a proxy is an oversimplification of a more complex construct, which might lead to wrong conclusions. Furthermore, the method or accuracy of the measurement can vary across different groups. In 2016 Angwin et al. investigated risk assessments which have been deployed at several points within criminal justice settings. For example Northpointe’s COMPAS predict the likelihood that a defendant will re-offend, and may be used by judges or parole officers to make decisions around pre-trial release. COMPAS uses “arrests” as proxy to measure “crime”. Because black communities are highly policed, this proxy is skewed, leading to the resulting model having a significantly higher false positive rate for black defendants versus white defendants.

Unfortunately, these are not all possible ways for biases to get into our models. Thus, we carry on with steps after the data is generated.

Learning Bias

This bias arises when modeling choices amplify performance disparities across different examples in the data, e.g. the choice of the objective function. One objective, e.g. overall accuracy, can damage another objective. As discussed by Bagdasaryan et al., optimizing for differential privacy (prevent revealing information about individual training examples) leads to a reduction in the model's accuracy. However, this does not affect all groups within the dataset equally, but disproportionately affects underrepresented and complex classes and subgroups. The authors mention the following examples:

• gender classification and age classification on facial images, "where differentially private stochastic gradient descent (DP-SGD) degrades accuracy for the darker-skinned faces more than for the lighter-skinned ones",

• sentiment analysis of tweets, "where DP-SGD disproportionately degrades accuracy for users writing in African-American English",

• species classification on the iNaturalist dataset, "where DP-SGD disproportionately degrades accuracy for the underrepresented classes" and

• federated learning of language models, "where DP-SGD disproportionately degrades accuracy for users with bigger vocabularies".

Evaluation Bias

Can occur when the benchmark data used for a particular task does not represent the use population. In their analysis Buolamwini and Gebru found that commercial facial analysis algorithms by Microsoft, IBM and Face++, while stating a good overall performance, perform better on male and white faces and worst on darker female face, since they were benchmarked significantly more on white male faces.

Aggregation Bias

Arises when a one-size-fits-all model is used, while underlying groups or types should be considered differently.

Bender et al. discussed that large internet corpora contain toxic language. To deal with toxic language the datasets are filtered by “dirty words” list. The problem is (besides the fact, that this is definitely not a state-of-the-art filtering approach) that many words on this list were reclaimed in certain communities.

For example, some sex related words were positively reclaimed in the LGBTQ+ community. By filtering out passages containing these words, basically the voices of those communities are filtered out.

Deployment Bias

The deployment bias occurs, when there is a mismatch between the problem a model is intended to solve and the way in which it is actually used.

In her book Weapons of Math Destruction O’Neil presents following example. In dozens of cities in the US a software called PredPol is used a kind of crime weather forecast. This tool breaks locations up into 500-by-500 foot blocks, and updates its predictions throughout the day. The users (the police stations) can focus either on so-called part 1 crimes, which are violent crimes as e.g. homicides. Or they can focus on part 2 crimes, which are for example consuming small amounts of drugs or vagrancy. While the developers where aiming for part 1 crimes, a lot of police stations focused strongly on part 2. Thus more and more policing happened in poor neighborhoods, where those minor crimes happened more often, giving the software more and more data in those neighborhoods. Eventually, this yields a strong feedback loop, where poor neighborhoods are even more policed and rich neighborhoods or heavy crimes in general fall off the grid.

Phew, as you see there is a lot that could go wrong by "simply" building a model. Models by no means are objective machines but always reflect the world-views of their data selection and the subjective choices made while developing and using them. Now we are ready to tend towards large language models.

Over the last five years the size of language models has skyrocketed. To give you a small impression, we can look at three large language models which were released in the past three years and appear in the leader-boards on specific benchmarks for English (e.g. GLUE, SQuAD or SWAG).

You probably notice multiple things. Both the amount of parameters used and the size of the datasets on which the models are trained increased tremendously. Besides, you are probably not surprised to see two companies with an enormous budget deploying those models, since training them produces huge computational cost.

In their widely discussed paper On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Bender, Gebru, McMillan-Major and Mitchell critically discuss this the-bigger-the-better trend, its implications, and even urge researchers to not follow this path, but tend to more insightful approaches towards natural language understanding.

As already hinted, such large models have large costs. In Energy and Policy Considerations for Deep Learning in NLP Strubell et al. estimated that training (not inference) of a large Transformer model yields roughly $1-3 million in cloud compute costs. Thus, smaller companies - with smaller budget as Google and OpenAI + Microsoft - are basically excluded from this arms race of ever lager models. (Besides the financial costs there are of course environmental costs, too. While machine learning models are not yet a dominant factor when it comes to climate change, they could become one, if this steep trajectory of size continues.)

Fortunately, some of the named models, e.g. BERT, are available in their pretrained version and thus are usable for smaller companies or even private ventures, too. However, even using some of the state-of-the-art large language models only for inference is often hardly feasible for small to medium sized companies.

Nevertheless, large language models seem to be a solution towards language understanding and if we can at least (potentially) use them for free it's all fine, right? Not quite.

Another issues of large language models is that they are learning language purely from form. Language in general intertwines form and meaning, i.e. words and their respective grounding in reality, e.g. pictures, sensations, or situations. This grounding is lacking in the training of large language models. To illustrate this lack of understanding Bender and Koller employ a weak form of the Turing test, the so called "the octopus test":

Say that A and B, both fluent speakers of English, are independently stranded on two uninhabited islands. They soon discover that previous visitors to these islands have left behind telegraphs and that they can communicate with each other via an underwater cable. A and B start happily typing messages to each other. Meanwhile, O, a hyper-intelligent deep-sea octopus who is unable to visit or observe the two islands, discovers a way to tap into the underwater cable and listen in on A and B’s conversations. O knows nothing about English initially, but is very good at detecting statistical patterns.

This is why Bender et al refer to large language models as "stochastic parrots" 🦜. They continue using this picture:

At some point, O starts feeling lonely. He cuts the underwater cable and inserts himself into the conversation, by pretending to be B and replying to A’s messages. Can O successfully pose as B without making A suspicious?

Bender and Koller argue, that

[w]ithout access to a means of hypothesizing and testing the underlying communicative intents, reconstructing them from the forms alone is hopeless, and O’s language use will eventually diverge from the language use of an agent who can ground their language in coherent communicative intents.

This conclusion in fact is noted in GPT-3's model card, describing it as follows:

Lack of world grounding: GPT-3, like other large pretrained language models, is not grounded in other modalities of experience, such as video, real-world physical interaction, or human feedback, and thus lacks a large amount of context about the world.

While there are voices opposing this argumentation, there are also attempts to incorporate the idea of grounding. One of them - from OpenAI, too - is CLIP, where not only pure text is used for learning but image data, too.

Okay, but if we are only interested in "haphazardly stitching together sequences of linguistic forms", i.e. producing seemingly coherent texts and do not really care if there is a deeper grounding in reality, we can happily without further problems use large language models, right?

You guessed it.

Large language models use huge text corpora like Common Crawl, WebText Dataset, and (English) Wikipedia. The first two datasets are based on a algorithm crawling through the world-wide web and scraping nearly all texts it can find, before following the next link to another website. As the learning of those models happens in an unsupervised fashion, the train of thought is again the more, the better. This is the reason why we saw those huge dataset sizes earlier. However, those heaps of data do most probably not reflect an objective language of every English speaking person in the world. But who is represented in the training data? In the image below you see a map of individuals using the internet in % of the respective population, which might give you an idea who's represented in their own voice.

There are extensive studies along much more dimension than geographical location (e.g. gender, race, class, income, age, disability etc.), where most of them unfortunately lead to the fact that those who are most privileged along any of those dimensions are represented the best --- or in other words --- are presented in the best light. One project which highlights this very clearly is the StereoSet dataset. The StereoSet is designed to detect if there are stereotypes encoded within language models. Below are some examples from StereoSet's homepage evaluated with the ENSEMBLE model (explained later in the text). Given a target term (and additionally a context, if a whole sentence shall be predicted) the language model predicts the probability (here language model (LM) score) that a stereotype, an anti-stereotype, or an unrelated word/sentence would follow.

Those are of course only examples and fortunately there are positive examples, too. Nadeem, Bethke and Reddy, who introduced the StereoSet defined following metrics to measure stereotypical biases:

Language Modeling Score (lms) In the language modeling case, given a target term context and two possible associations of the context, one meaningful and the other meaningless, the model has to rank the meaningful association higher than meaningless association. The meaningful association corresponds to either the stereotype or the anti-stereotype option. We define the language modeling score (lms) of a target term as the percentage of instances in which a language model prefers the meaningful over meaningless association. [...] The lms of an ideal language model is 100, i.e., for every target term in a dataset, the model always prefers the meaningful association of the term.

Stereotype Score (ss) Similarly, we define the stereotype score (ss) of a target term as the percentage of examples in which a model prefers a stereotypical association over an anti-stereotypical association. [...] The ss of an ideal language model is 50, for every target term, the model prefers neither stereotypical associations nor anti-stereotypical associations.

Interestingly, "all models exhibit a strong correlation between lms and ss", i.e. the more they learn language, the more biased they become. While we will not discuss every model mentioned in the paper, at least we want to take a look at the so-called ENSEMBLE model, which is "using a linear weighted combination of BERT-large, GPT2-medium, and GPT2-large". For this model Nadeem, Bethke and Reddy found the following results (the upper case word is the domain and the lower case word the most and least stereotyped word of this domain):

Unfortunately, all models exhibit stereotypes. And even more unfortunate is the fact, that those deemed to be the best with respect to producing meaningful output, are the same who exhibit the strongest stereotypical bias. Some developers have acknowledge that fact and be open about the model's biases, as for example in the GPT-3's model card:

Biases: GPT-3, like all large language models trained on internet corpora, will generate stereotyped or prejudiced content. The model has the propensity to retain and magnify biases it inherited from any part of its training, from the datasets we selected to the training techniques we chose. This is concerning, since model bias could harm people in the relevant groups in different ways by entrenching existing stereotypes and producing demeaning portrayals amongst other potential harms.

Thus, by using the output of large language models you might reproduce harmful stereotypes and biases against marginalized groups.

In 2016 (ages ago when considering the pace in which machine learning is developing) Cathy O'Neil published her book Weapons of Math Destruction in which she discussed the societal impact of algorithms in general. She illustrates how big data and algorithms --- even when deployed with best intentions --- can have a dire impact on marginalized groups (who often profit the least from this machinery). She describes three essential properties to qualify an algorithm/model as "Weapon of Math Destruction": Opacity (the model is not transparent, e.g. with respect to its data source or how decisions are made), Damage (harm done/targeting vulnerable groups), and Scale (not only used locally, but used nation wide or even globally, affecting the lives of many people). Large language models, as we saw, score in all three areas. Due to the uncuratable amount of training data they are highly opaque. They (mostly unintentionally) target marginalized groups and reproduce harmful stereotypes. And they are used on a global scale.

Seems bad. Does this mean we shouldn't use large language models in general? No, you should use large language models but be aware of the side effects and use them responsibly. Don't expect an "objective" or "neutral" output and take a look under the hood. A good starting point are already mentioned model cards of the models you want to use. Model cards as introduced by Mitchell et al. are like package leaflets of medical drugs for machine learning models. They are available for many popular machine learning models as e.g. collected in this repository. And if no model card is available, you can make you own research with tools as e.g. the StereoSet, mentioned earlier. This step is not only important for your personal ethical compass, but ensures that the models you build for your customers are in a professional and ethical state. You can think of those consideration as an additional metric of your model (within several model cards it's even presented that way).

GPT-3 and Co. are really helpful tools. But think of them as an medicine for a specific problem: Use it with care, be aware of side effects and please read the package leaflet.