October 23, 2024 By Jennifer Gregory 4 min read

In early 2023, Google’s Bard made headlines for a pretty big mistake, which we now call an AI hallucination. During a demo, the chatbot was asked, “What new discoveries from the James Webb Space Telescope can I tell my 9-year-old about?” Bard answered that JWST, which launched in December 2021, took the “very first pictures” of an exoplanet outside our solar system. However, the European Southern Observatory’s Very Large Telescope took the first picture of an exoplanet in 2004.

What is an AI hallucination?

Simply put, an AI hallucination is when a large language model (LLM), such as a generative AI tool, provides an answer that is incorrect. Sometimes, this means that the answer is totally fabricated, such as making up a research paper that doesn’t exist. Other times, it’s the wrong answer, such as with the Bard debacle.

Reasons for hallucination are varied, but the biggest one is that the data the model uses for training is incorrect — AI is only as accurate as the information it ingests. Input bias is also a top cause. If the data used for training contains biases, then the LLM will find patterns that are actually not there, which leads to incorrect results.

With businesses and consumers increasingly turning to AI for automation and decision-making, especially in key areas like healthcare and finance, the potential for errors poses a big risk. According to Gartner, AI hallucination compromises both decision-making and brand reputation. Additionally, AI hallucinations lead to the spreading of misinformation. Even more so, each AI hallucination leads to people not trusting AI results, which has widespread consequences, and businesses are increasingly turning to this technology.

While it’s tempting to have blind trust in AI, it’s important to use a balanced approach when using AI. By taking precautions to reduce AI hallucinations, organizations can weigh the benefits of AI with the potential complications, which include AI hallucinations.

Explore AI cybersecurity solutions

Organizations increasingly using generative AI for cybersecurity

While the discussion about generative AI often focuses on software development, the issue increasingly affects cybersecurity. The reason is that organizations are starting to use generative AI for cybersecurity purposes.

Many cybersecurity professionals turn to generative AI for threat hunting. While AI-powered security information and event management (SIEM) improves response management, generative AI can use natural language searches for faster threat hunting. Analysts can use natural language chatbots to spot threats. Once a threat is detected, cybersecurity professionals can turn to generative AI to create a playbook based on the specific threat. Because generative AI uses training data to create the output, analysts have access to the latest information to respond to a specific threat with the best action.

Training is another common use for generative AI in cybersecurity. By using generative AI, cybersecurity professionals can use real-time data and current threats to create realistic scenarios. Through the simulation, cybersecurity teams get real-world experience and practice that was previously challenging to find. Because they can practice on similar threats to those they may encounter that day or week, professionals can train on current threats, not ones in the past.

How AI hallucinations affect cybersecurity

One of the biggest issues with AI hallucinations in cybersecurity is that the error can cause an organization to overlook a potential threat. For example, the AI tool may miss a potential threat that ends up causing a cyberattack. Often, this is due to bias in the model that happens through biased training data, which causes the tool to overlook a pattern that ends up affecting the results.

On the flip side, an AI hallucination may create a false alarm. If the generative AI tool fabricates a threat or falsely identifies a vulnerability, then employees will begin to trust the tool less in the future. Additionally, the organization focuses its resources on addressing the false threat, which means that a real attack may be overlooked. Each time that the AI tool produces inaccurate results, employee’s confidence in the tool becomes lower, making it less likely that they will turn to AI or trust the results in the future.

Similarly, a hallucination can provide inaccurate recommendations that prolong detection or recovery. For example, a generative AI tool may accurately spot suspicious activity but provide inaccurate information on the next step or system recommendations. Because the IT team takes the wrong steps, the cyberattack is not stopped and the threat actors gain access.

Reducing the impact of AI hallucinations on cybersecurity

By understanding and anticipating AI hallucinations, organizations can take proactive steps to both reduce the occurrence and the impact.

Here are three tips:

  1. Train employees on prompt engineering. With generative AI, the quality of the results depends greatly on the specific prompts used for the requests. However, many employees create the prompts without formal training or knowledge on how to provide the right information to the model. Organizations that train their IT team on using specific and clear prompts can improve the results and possibly reduce AI hallucinations.
  2. Focus on data cleanliness. AI hallucinations often happen when using poisoned data, meaning there are errors or inaccuracies in the training data. For example, a model that is trained on data that includes cybersecurity threats that were later found to be false reports may identify a threat that is not accurate. By ensuring, as much as possible, that the model uses clean data then your organization can eliminate some AI hallucinations.
  3. Incorporate fact-checking into your process. With today’s current maturity level of generative AI tools, AI hallucinations are likely part of the process. Organizations should assume that errors or inaccurate information may be returned at this stage. By designing a fact-checking process to make sure that all information returned is accurate before employees take action, organizations can reduce the impact of the hallucinations on the business.

Leveling the cyber playing field

Many ransomware gangs and cyber criminals are using generative AI to find vulnerabilities and create attacks. Organizations that use these same tools to fight cyber crime can put themselves on a more level playing field. By also taking proactive measures to prevent and reduce the impact of AI hallucinations, businesses can more successfully use generative AI to help their cybersecurity team better protect data and infrastructure.

More from Artificial Intelligence

Security roundup: Top AI stories in 2024

3 min read - 2024 has been a banner year for artificial intelligence (AI). As enterprises ramp up adoption, however, malicious actors have been exploring new ways to compromise systems with intelligent attacks.With the AI landscape rapidly evolving, it's worth looking back before moving forward. Here are our top five AI security stories for 2024.Can you hear me now? Hackers hijack audio with AIAttackers can fake entire conversations using large language models (LLMs), voice cloning and speech-to-text software. This method is relatively easy to…

Autonomous security for cloud in AWS: Harnessing the power of AI for a secure future

3 min read - As the digital world evolves, businesses increasingly rely on cloud solutions to store data, run operations and manage applications. However, with this growth comes the challenge of ensuring that cloud environments remain secure and compliant with ever-changing regulations. This is where the idea of autonomous security for cloud (ASC) comes into play.Security and compliance aren't just technical buzzwords; they are crucial for businesses of all sizes. With data breaches and cyber threats on the rise, having systems that ensure your…

Cybersecurity Awareness Month: 5 new AI skills cyber pros need

4 min read - The rapid integration of artificial intelligence (AI) across industries, including cybersecurity, has sparked a sense of urgency among professionals. As organizations increasingly adopt AI tools to bolster security defenses, cyber professionals now face a pivotal question: What new skills do I need to stay relevant?October is Cybersecurity Awareness Month, which makes it the perfect time to address this pressing issue. With AI transforming threat detection, prevention and response, what better moment to explore the essential skills professionals might require?Whether you're…

Topic updates

Get email updates and stay ahead of the latest threats to the security landscape, thought leadership and research.
Subscribe today