Using AI Critically
Last updated:
|
|
To promote healthy, informed scepticism of generative AI and develop students’ critical thinking, it is important for students to develop an awareness of the drawbacks and limitations of generative AI tools and learn about how to critically engage with AI generated content.
Technical Limitations of AI
Generative AI possesses several technical limitations, which could lead to unwanted consequences when students use these tools in their learning without proper awareness.
AI could generate incorrect, inaccurate, misleading or fabricated information.
Common generative AI models such as ChatGPT are predominantly “unsupervised”, which is why they are prone to generating incorrect or misleading information. They often generate fabricated data, making up quotes and citations from non-existent sources, sometimes providing full bibliographic details, including fake titles, authors, dates etc. with a fictional URL (Alexander et al., 2023; Baidoo-Anu & Ansah, 2023).
Example of ChatGPT-4.1 mini hallucinating by providing fabricated academic reference
In this example, the user asked AI about the significance of the Lion Rock and requested it to support its answer with academic sources. The three supposedly scholarly publications ChatGPT provided:
Chan, W. T., & Lee, S. M. (2008). The Lion Rock Spirit: Identity and Resilience in Hong Kong. Journal of Asian Studies, 67(3), 789-812.
Smart, A. (2006). Lion Rock and the Making of Hong Kong’s Working Class. Urban History Review, 34(2), 25-40.
Chu, Y. W. (2017). Symbolism and Social Memory: The Case of Lion Rock in Hong Kong. Asian Cultural Studies, 43(1), 101-120.
Are all fictional and fabricated entirely by AI.
Smart, A. (2006). Lion Rock and the Making of Hong Kong’s Working Class. Urban History Review, 34(2), 25-40.
Chu, Y. W. (2017). Symbolism and Social Memory: The Case of Lion Rock in Hong Kong. Asian Cultural Studies, 43(1), 101-120.
Are all fictional and fabricated entirely by AI.
This phenomenon is known as “hallucination”, which describes AI's tendency to generate texts, especially academic references, that are false or simply imaginary to appear convincing (Gimpel et al., 2023)
Example of ChatGPT-4.1 mini hallucinating by making up non-existent bands
In this other example, the user asked ChatGPT to list ten heavy metal bands in Zimbabwe, but only three items on the list are real and verifiable, while the rest are results of AI hallucination.
AI models are more likely to hallucinate when there is not enough data on the subject and when they are asked to create a list. In this case, it is likely that there is very little information on heavy metal bands from Zimbabwe within the AI's training data. AI also prioritizes completing the task (listing ten items) over making sure the information is correct, which often leads to hallucinations.
Q: Why do AI models hallucinate?
A: Due to the “black box” nature of generative AI models, scholars have yet to determine the exact mechanism behind AI hallucination. Some argue that AI lacks metacognition – meaning that it does not think about how it thinks – and relies entirely on calculating the probability of a statement, making it extremely prone to generating false information (Kortemeyer, 2023).
AI-powered research tools
- Scite – AI for Research | Scite
- Scopus AI – Scopus AI - Scopus LibGuide - LibGuides at Elsevier
Let’s try using Scopus AI to search for academic sources. On Scopus's “Start exploring” page, you can find the Scopus AI button. Enter your prompt as usual. In the case shown here, the student used the prompt “Generate a list of articles on the topic *the effect of social media on young people’s mental health*”.
Scopus AI listed a summary of focus and key points of each article related to the topic. The complete references and links to the article can be retrieved on the right side of the result. While Scopus AI and Scite are more reliable, they still possess the shortcomings of other GenAI tools. Thus, caution must always be exercised when using them for your assignment.
Other known issues
AI can give confusing or inaccurate grammatical explanations
In the example below, the student requested ChatGPT to correct the sentence "The research findings are showed that there is a significant correlation between the variables of income levels and mental health outcomes." However, while ChatGPT correctly revised the sentence, the explanation given was inaccurate and confusing. The issue with the original sentence does not stem from the verb form itself, but rather from a misunderstanding of when to use the active and passive voice:
Further prompting did not prove very helpful either. In ChatGPT's second attempt to explain its revision, it skipped over the rationale for choosing "showed" over "are showed" with "findings” and instead elaborated on the differences between the present and past passive voice.
AI could reproduce inherent algorithmic biases and stereotypes
Another prominent limitation of generative AI is its potential to reproduce biases, racial and gender stereotypes and other discriminatory content (Aithal & Aithal, 2023). The example below showed different types of stereotypes being reproduced by a text-to-image AI. For example, AI could associate certain professions with a specific gender (e.g. teachers as women, doctors as men), certain negative words with a specific ethnicity, or certain groups with a specific stereotype (e.g. showed a religious person carrying a lethal weapon).
A: Data is a human construct and is therefore always plagued by human biases. Generative AI’s results rely entirely on its training data, which could contain biased and inaccurate data, discriminatory language as well as racial and gender stereotypes (Chan & Hu, 2023; Mao et al., 2024). Such algorithmic biases are easily reproduced or even amplified since AI cannot assess its answers, resulting in stochastic parroting – like a parrot that performs random guesses (Crawford et al., 2023)
Students could be easily misguided by AI's biased responses and include potentially discriminatory or harmful content in their writing. Therefore, students need to validate AI's response and critically identify any potential biases.
AI lacks Contextual Understanding and Human Nuances
Generative AI models are pre-trained, meaning that they cannot generally adapt to the context of a conversation topic or situation. As a result, they might misinterpret the conversation and provide inaccurate answers (Aithal & Aithal, 2023).
Screenshot of Grok 3 ‘s response to the prompt “write a sentence with these words *add oil*”.
In the example, AI misinterpreted “add oil” by its literal meaning instead of what it means in the Hong Kong context – “go for it” for expressing encouragement.
In terms of EAP learning, voice recognition tools powered by AI may struggle with accents and dialects. In addition, students might be misguided by the narrow and limited representation of language and culture if they over-rely on AI for learning (Wang et al., 2023). AI is also limited in terms of creativity, critical thinking skills, and emotional intelligence (Perera & Lankathilaka, 2023). Thus, students must remain vigilant when using AI to avoid confusion.
AI often produces overused, predictable and unnatural language
As generative AI relies on common phrases and expressions from its training data, it often resorts to clichés or overused terminology, such as “delve” and “underscore” (Juzek & Ward, 2024). Filler phrases like “in the ever-evolving landscape of X…” are also commonly found in texts generated by GenAI. Additionally, AI tends to use rare or complex words excessively in an attempt to sound sophisticated, which can lead to awkward and unnatural phrasing (Opara, 2024). These tendencies contribute to a lack of authenticity in the text.
Overall, generative AI…
- lacks human nuance (e.g. jokes and humour)
- lacks cultural awareness (Baskara & Mukarto, 2023)
- lacks personal perspectives
- lacks a deep understanding of the meaning of words, particularly for tasks that require a nuanced understanding of specific domain knowledge (Perera & Lankathilaka, 2023)
- Uses predictable and superficial language
Privacy and Data Security
Last but not least, generative AI poses a significant risk to privacy and data security if handled improperly. Students need to be aware that AI tool providers might collect personal data and information when users engage with the tools. You must not use AI to process data that contain any sensitive information, including interview transcripts and questionnaire results without consent and ethical approval.
Critically spotting inaccuracy and inconsistency in AI output
As part of AI digital literacy, students will need to master the skill of critically examining AI output for spotting inaccuracies and inconsistencies. Here are a few strategies that you can try:
Ask AI to validate or verify its claim by providing detailed justifications or external links for cross reference.
Screenshot of follow-up question asking ChatGPT-4.1 mini to check the references it provided.
In this case, AI recognized its answers are fabricated and apologized.
Q. Can I compare between two AI models’ answers to check validity?
A. While comparing output between 2 or more AI models might sometimes help you gain a better understanding of a topic, two AI tools agreeing with each other does not mean they are both correct.
In this case, when asked between 9.11 and 9.9 which is bigger, both ChatGPT and Llama 3.1 answered that 9.11 is bigger than 9.9, which is clearly false. Therefore, you should also validate AI answers using reliable sources instead of simply asking another AI model.
Ask the question (again) by clearly stating your context.
When specifying the context as Hong Kong, AI is able to adjust accordingly and use the word in an appropriate sense.
Other methods:
- Use common sense and look out for obvious inconsistencies. E.g. Pay attention to years, names, places, etc.
- Fact-check all AI generation information by finding a reliable supporting source (e.g. academic journal, news articles) before using the idea in your assignments.
