Cognitive Bias
Imagine navigating the complex world of decisions we face every day—choosing what task to do, interpreting new information, or deciding who to trust. Our minds often rely on mental shortcuts, known as heuristics, to simplify these decisions. While these strategies are efficient and practical, they can sometimes lead us astray, resulting in cognitive biases. These biases are systematic and predictable errors in judgment that stem from an over-reliance on these shortcuts.
Cognitive biases arise from cognitive limitations, motivational factors, or adaptations to natural environments. These biases often result from applying heuristics in unsuitable contexts, even though heuristics generally work well in typical situations (Fahse et al., 2021; Harris, 2020). Broadly defined, cognitive biases encompass errors related to thinking, judgment, and memory, as well as thinking strategies and heuristics that can lead to such biases, even if they are not inherently biased (Kliegr et al., 2021).
Example of Cognitive Bias in Machine Learning
Here are some well-known examples of cognitive biases from Cardie (1999):
- Subject Accessibility
- The first item mentioned in a sentence is perceived as particularly important. This reflects a broader Focus on Attention bias, where the position of information influences its perceived significance.
- Recency Bias
- People tend to give more weight to recent information when interpreting a sentence, favouring it over older information.
- Restricted Memory
- Short-term memory limitations impact various language comprehension skills, making processing or retaining all relevant details difficult.
- Semantic and Syntactic Priming
- Individuals respond faster to words semantically related to concepts currently focused on during interpretation. Similarly, language processing prefers structures that are syntactically consistent with nearby elements in a sentence.
These biases illustrate how cognitive tendencies can shape our interpretation and understanding of language.
For example, social media content often reflects the cognitive biases of the individuals who create those posts, such as personal opinions or cultural norms (Mavrogiorgos et al., 2014). When Large Language Models are trained on such data, these biases can be embedded into the model’s responses. This type of bias differs from biases introduced by model developers or evaluators during the design or testing phases. While both types of biases originate from humans, the first is tied to the data itself, and the second stems from machine learning models’ implementation and evaluation processes.
Design Approaches for Mitigating Cognitive Bias in Machine Learning
Tackling cognitive bias requires a systematic, proactive approach.
You can get started with these resources:
Free Resources for Cognitive Bias Mitigation
Best Practices and design considerations for mitigating Cognitive Bias from problem definition to model deployment (click Free Downloads).
Bias Design Cards – £399
Empower your team to drive Responsible AI by fostering alignment with interactive design card workshops for design, development and monitoring bias.
Collaborate and take actionable steps with +75 design cards Practical, easy-to-use cards from problem definition to model monitoring Checklists for every phase in the AI/ ML pipelineAI Bias Mitigation Package – £999
Mitigate and resolve 15 Types of Bias specific to your project with detailed guidance from problem definition to model monitoring. Packed with practical methods, research-based strategies, and critical questions to guide your team. Comprehensive checklists with +75 design cards for every phase in the AI/ ML pipelineCustomised AI Bias Mitigation Package – £2499
Mitigate and resolve 15 Types of Bias specific to your project with detailed guidance from problem definition to model monitoring. Packed with practical methods, research-based strategies, and critical questions specific to your use case. Customised checklists and +75 design cards for every phase in the AI/ ML pipeline
Conclusion
Cognitive biases, such as subject accessibility, recency bias, restricted memory, and semantic and syntactic priming, significantly influence how humans process information and make decisions. These biases stem from heuristic shortcuts that simplify complex tasks but often lead to systematic errors in judgment. Organisations can mitigate cognitive biases and foster ethical, human-centred decision-making systems by aligning fairness solutions with specific applications and stakeholder needs.
Sources
Cardie, C., 1999. Integrating case-based learning and cognitive biases for machine learning of natural language. J. Exp. Theor. Artif. Intell., 11(3), pp.297-337.
Dhukaram, A.V. and Baber, C., 2015. Modelling elderly cardiac patients decision making using Cognitive Work Analysis: identifying requirements for patient decision aids. International Journal of Medical Informatics, 84(6), pp.430-443.
Fahse, T., Huber, V. and van Giffen, B., 2021. Managing bias in machine learning projects. In Innovation Through Information Systems: Volume II: A Collection of Latest Research on Technology Issues (pp. 94-109). Springer International Publishing.
Harris, C., 2020, April. Mitigating cognitive biases in machine learning algorithms for decision making. In Companion Proceedings of the Web Conference 2020 (pp. 775-781).
Kliegr, T., Bahník, Š. and Fürnkranz, J., 2021. A review of possible effects of cognitive biases on interpretation of rule-based machine learning models. Artificial Intelligence, 295, p.103458.
Mavrogiorgos, K., Kiourtis, A., Mavrogiorgou, A., Menychtas, A. and Kyriazis, D., 2024. Bias in Machine Learning: A Literature Review. Applied Sciences, 14(19), p.8860.