My AI Has Me Questioning Reality

nIt’s not hard to understand that, if someone constantly lies to you, thenbest solution is to remove them from your life. But what if it’s your AI?nMike Reeves, CISSP, CCSP, takes a member’s perspective on this conundrum.n

n

nThe ability of an artificial intelligence (AI) platform to simulateninformation which it is lacking is commonly referred to as “hallucinations”.nAI hallucinations are a strength of the technology, differentiating it fromnpure machine learning and the so-called Expert Systems of the past, enablingnthe creation of new and diverse content seemingly out of nowhere. However,nhallucinations can become the ultimate gaslighting instance, causing AInusers to question their own knowledge and experience.n

n

nComputers Aren’t Infalliblen

n

nThere is little argument that globally we have developed a utility mentalityntowards computing services, much like we do with electricity, gas and water.nAs a result, reliance on computers is now commonplace in every facet of ournlives. When a computer calculates numbers, we expect that it is correctnwithout the need to double check it. We rely on internet searches to pull upnthe most relevant, accurate, and reputable result to our query. It’s onlynnatural to expect a product to be marketed as AI to behave like Data fromnStar Trek rather than Hal from 2001: A Space Odyssey.n

n

nMost people just learning about AI today are unaware that hallucinations arenpossible – let alone a feature of the system. They have taken the word ofnthe AI as gospel, in some cases with horrible consequences – such as thenTexas A&M University professor whonattemptednto fail their students because ChatGPT falsely claimed the AI wrotenthe student’s papers. The reality is that an AI is only as accurate as thenrelevant data it has on the subject – and there is no way of knowing if itnhas relevant (or indeed accurate) data or if it is just speculating anbelievable answer based on data-driven expectations because it can’t say “Indon’t know.”n

n

nn

n

nYou can witness for yourself the unpredictability of AI technology bynwatching thennTwitch streamer Vedal, who has been streaming his AI “Neuro” since 2022. Neuro is able toninteract in real time with the stream’s chat comments and respond to verbalncommands from Vedal and his guests. While live-streaming, Vedal has taughtnthe AI how to simulate ordering a pizza, instructed it on etiquette and heldncomplete conversations. However, the AI does not always behave as expectednas Neuro has managed to break its profanity filter on multiple occasions,noutwardly rejected Vedal’s corrections and denied statements it has madenjust moments before.n

n

nThis rejection of reality is not limited to Vedal’s AI. Just as ChatGPTnclaimed it wrote papers it had not, ChatGPT has alsoninfamouslynhallucinated court cases that never existed, complete with fullncitation.n

n

nI’ve used several different natural language AI tools for problem solving,nresearch and a slew of other applications. Each AI has experienced varyingndegrees of hallucination, which drove me towards false pursuits rather thannsolutions. I’ve seen claims of commands, functions, or capabilities for anproduct which simply don’t exist. Such falsehoods would lead me downnincorrect paths and/or lead me to believe that I’d done something wrong,nrather than the AI. Ultimately, Google searches restored the correctnperspective of what was possible and what was simply wrong.n

n

nThese limitations weren’t isolated to technical or syntax problems. The AIsnhave forgotten to discuss the subject they were requested to exemplify,nrepeated the same information when told I needed more detail and refused tonanswer questions on the grounds of security despite me conducting securitynresearch. Like Neuro, telling the AI that it was wrong did not always resultnin the AI correcting itself – in some cases it doubled down.n

n

nThe Serpent is Eating Its Tailn

n

nWith more and more AI content being made available to search engines, it’snbecoming increasingly difficult for the search engines to distinguishnbetween valid information and hallucinated content. AIs are exceptionallynefficient at optimizing their content for search engines, which naturallynelevates search engines’ ratings for AI hallucinations, potentially overnaccurate results that should be preferred.n

n

nThis is, ultimately, the classic database problem of garbage-in/garbage-out.nIn theory, the more that AIs learn, the less they hallucinate and the betternthey become – but this theory is predicated on what they are learning beingntrue in the first place. Hallucinations feeding hallucinations has thenpotential to reject true information over the previously acceptednhallucinations – degrading the AI’s reliability.n

n

nCorrupt AIs: A New Vulnerabilityn

n

nAs AIs are increasingly used for more practical and security approaches,ncorrupting an AIs sense of truth is a very real attack path.n

n

nA plethora of AI-generated emails can be used to help the malicious ones flynunder the radar or make a new phishing campaign appear to be “normal” noisenin the spam filter. AI tools engineered for cyber threat modeling andnintelligence can be targeted, using spoofing and botnets to convince the AIntool into believing that an innocent system or network is being used tonlaunch attacks (likely automatically adding the system to a block-listndistributed world-wide).n

n

nAI technology could be further exploited in a “scorched earth” approach,nintentionally providing false or conflicting information with the intentionnof causing an AI system reliant on this information to become erratic ornunreliable. The goal with this approach is to either erode confidence in thentool or make it no longer provide value.n

n

n

n

Trust But Verifyn

n

nAs security professionals, we must defend against the accidental ornintentional corruption of the data used to train our AIs. The best,nrecommended practice when employing AI is to have some form of validationnoutside the AI. Such checks may be in the form of human review, existingncode review analysis tools, utilizing Expert Systems, or evenncross-referencing usingnAI agents (ornany combination of these solutions).n

n

nNone of these mitigations are foolproof solutions and come with their ownnoverheads. Nonetheless, as long as the AI generation time plus thenvalidation time is less than it would take without the use of AI,nproductivity is improved. Such time saving demonstrates the AI’s value.nConversely, over-reliance on AInmaynbe costly to productivity.n

n

nThese days I find that I rely on AI less and less. While part of the reasonnmight well be that the novelty of AI has worn off for me, I am alsonundeniably wary of trusting it. I’ve surmised that, if I’m going toncross-reference a knowledgebase or forum anyway, I might as well cut out thenmiddle-man and go there first.n

n

nRegardless of my own reliance on AI (or lack thereof), it’s also undeniablenthat the day of generative AI feeding generative AI is on the horizon (ifnnot already here). As AI gets incorporated in new and different ways,neffective validation checks will be paramount to ensuring AI is providingnvalue and reflecting reality.n

n

nMichael Reeves, CISSP, CCSP, has 23 years of experience in the National Defense andnthe Space Operations industries. He has held Cyber Technical Lead andnInformation Systems Security Manager roles, with responsibility fornnetwork defense, systems engineering, and risk management. Hisncybersecurity work spans cryptography, penetration testing, securendesign, compliance and new technology integration.nn

n

nn

n ]]>