I hate to break it to you: your shiny new ‘AI-powered’ chatbot isn’t really magic, it’s not even that intelligent – and that matters for Cybersecurity.

Let’s start with a little reality check: when the sales deck says: “powered by AI”, what it most often means is “based on a large-language model (LLM) that predicts words”.  Your shiny new app didn’t get a brain, just an insanely large spreadsheet. In a neural model such as a Transformer (the “T” in ChatGPT) each parameter is akin to a cell in your spreadsheet, but this spreadsheet has hundreds of billions of cells. An LLM like GPT-2 started out at ~1.5 billion parameters but has grown exponentially with every iteration since, jumping 100x to version 3, then another 10x to GPT-4 which is estimated at 1.8 trillion parameters.

For many organisations, especially those thinking about cybersecurity it’s tempting to assume these tools are fully intelligent, autonomous, able to problem-solve like humans, and thus safe. Unfortunately that assumption only creates risk. Because when you confuse pattern-matching for reasoning, you set yourself up for governance gaps, data-leakage hazards, and trust failures.

In this article I’ll attempt to demystify three related-but-distinct concepts: AI (in the broad sense), Machine Learning (ML) and GPT-style models. We’ll go over how they differ, what they can and can’t do, why your organisation should care, and how you might build a secure, pragmatic strategy around them.

The risks of Shadow-AI

So what’s “Shadow AI” I hear you say? It’s simply v2 of the “Shadow IT” problem.  Employees, business units, supply-chain partners, or contractors using Generative AI, aka Gen-AI tools (chatbots, assistants, image-generators, workflow automation) outside the governance, logs, or oversight of the IT/security organisation – it’s a CISO’s nightmare!

According to recent research, organisations are waking up to how pervasive this is. One article describes it thus: “Employees are racing ahead with AI tools, but without oversight, that innovation can quietly turn into risk before anyone notices”.  Another finds that almost half of employees admit to uploading sensitive company information to unauthorised cloud-based Gen-AI platforms.

Why this matters for cybersecurity

  1. Data leakage: Internal IP, sensitive customer data, business strategy, and incident data may be copied into prompts that external tools ingest and may log or reuse as “training”.
  2. Compliance: If regulated data (e.g., financial services, healthcare) is fed into a public-cloud model without data-protection safeguards, you expose yourself to GDPR, and a multitude of regulatory breaches.
  3. Visibility: Because generative tools often reside in browser extensions or SaaS apps, outside the traditional network perimeter, your SOC may not see them.
  4. Trust: If decisions (or advice) come from an unsanctioned generative model, you lack traceability, which is unacceptable for formal incident-response, breach reporting or forensic audit.

So what should we do?

We’ll come back to governance and action later. But let’s just say this: treat Gen-AI tools as you would any other new technology; identify risks, control, monitor, govern, and educate your users.

What is a GPT ?

Limitations of GPTs

Here comes the hard truth. GPT-style models are exceptionally good at what they do: pattern-matching, summarising, generating fluent text. But they are by no measure reasoning machines in the classical sense.  Imagine asking Hercule Poirot to investigate a murder, but he just quotes every murder mystery he’s ever read and picked the most common culprit, Colonel Mustard!

Research backs this up: for example, one study found that GPT performance drops drastically when faced with multi‐step reasoning tasks (e.g., multi-hop inference, domain-shift tasks), and another review states: “While the LLMs excel in certain tasks, they still face challenges in understanding context, generating diverse responses, and handling rare or out-of-domain inputs.” 

What “reasoning” means (and why it matters)

When your cybersecurity analyst asks “what’s the threat vector if Vendor X supplies firmware version Y, given network architecture Z, and the attacker is persistent”, this requires domain reasoning. You’re asking: “Here’s a scenario, what logically comes next given our assets, our controls, our industry profile?”

A GPT is much better at: “Here’s a prompt about Vendor X and firmware version Y, what plausible threats might exist?” It can generate a very convincing narrative. But it’s not actually applying your specific segmentation, your telemetry, your threat intel, and any logical deduction step-by-step.

In effect: you’re asking it to reason, but it only knows how to predict subsequent words. It does a good job of simulating reasoning (fluent text, plausible chain of logic) but it cannot guarantee correctness.

In cybersecurity this means you can’t simply hand over a critical decision to a GPT and assume it will catch novel logic, hidden dependencies, or emergent threat patterns. You must always keep a human in the loop, for now at least.

But isn't GPT's "Thinking Mode" Reasoning?

Hallucinations & why you can’t always trust the output

A key danger in LLMs is hallucination: when the model generates text that it is convinced is factual but is incorrect, fabricated, or misleading.  One review even states: “generative AI can help spread misinformation”, which leads to inevitable ethical debates.

What causes hallucinations?

  • Because GPTs predict the statistically likely next word(s), they are not verifying truth. They don’t “know” in the human sense, they create fluent plausible statements.
  • If you ask: “What are the current CVE’s for Vendor X?” and the model lacks recent data or context, it may invent one based on patterns of previous exploits.
  • Models often operate on a snapshot of data. Unless you feed them fresh telemetry or embed new data, they are blind to events after their training date. 
  • Prompt-sensitivity: small changes in prompt can yield widely different answers, sometimes correct, sometimes implausible.

Why this is a cybersecurity risk

  • If a SOC analyst uses a GPT to draft a threat-report without verification, they may inadvertently use incorrect data or missed logic.
  • If you rely on a GPT for decision-support (e.g., “should we escalate incident X?”) and it hallucinates, you might make the wrong decision, with potentially far reaching consequences.
  • Trust in the tool can erode over time if users discover it’s prone to errors, and yet still gives plausible answers.
  • There’s also a real risk of fictional scenario generation. For example, a GPT might invent an attacker persona or threat that sounds real but has no actual basis in your environment – yet you get distracted spending time and resource considering it.
What to do?

Image-Creation, Deepfakes, & Generative Visual Models

Now the confusing bit! Many organisations now also face the deceptively impressive world of generative images, videos, and voice synthesisation. “Our GPT can also create artwork” only adds to the confusion: how can a text-based LLM generate visuals? And if it’s that good, why can’t it invent a completely new art style? Let’s unpack why it’s not truly creative.

How it works

Text-to-image generative models (e.g., DALL·E 2, Stable Diffusion, Midjourney) are trained on vast datasets of image-text pairs. When you type a prompt, the model maps your text into a virtual space and then generates pixel distributions that match what it has seen as the most statistically probably, given similar prompts.

In English: you’re asking, “given all the images and captions you’ve ever seen, what would most likely correspond to this prompt?” The model stitches together patterns in colour, composition, style and semantics.
So you get (most of the time) very high-quality, plausible, and stylistically coherent visuals, which is why people are astounded at how “artistic” the results appear.

Why they struggle when you ask for something original

Try asking your preferred GPT: “Create a completely brand-new art style that has never existed in history” or perhaps “Visualise a malicious threat actor from the year 2050 that operates beyond all current known adversary models”!  You are asking for novelty outside the scope of the training distribution. The model tries its best but essentially falls back to the nearest things it’s previously experienced.

In other words, visual Gen-AI models remix, recombine, interpolate from known styles, patterns and visuals.  They don’t truly invent from a blank canvas.  This limitation matters where you require true innovation, new visual identities, or unique deep-fake detection.

Gen-AI isn’t just a tech issue; it carries legal and IP risk too.  Marketers need to tread carefully when using GPTs to enhance their creative craft.  There have been several court cases where a models output has mirrored an author’s style (tone, plot, characters) close enough to be actioned, and many law firms advise that there is a real risk of copyright infringement if the output used is substantially similar to protected work.  This underscores the previous points about governance; you must make sure you know where your tool’s training data originated (provenance), whether licenses were secured (compliance), and ensure human verification for potential legal exposure (review).

Other risks specific to image‐models

  • Bias & representation issues: Generated images may reflect stereotypical or skewed distributions of subjects, style, gender, ethnicity. If you rely on them for training staff or threat modelling you might embed bias. 
  • Sensitive-data prompts: Someone might input internal imagery (e.g., network architecture diagram) into an external image generation service which poses a data leak risk.
  • Social engineering: Visual generative tools significantly lower the bar for producing believable fake images or altered imagery, a relatively new surface for cyber-risk.  Malicious business compromise campaigns are becoming more sophisticated yet barely require any skill; it’s no longer just a dodgy phone call pretending to be the IT department – it could be a slick, AI generated video of someone indistinguishable from your boss with an “urgent request”, and we’re rooted in trust when it’s something we physically see and hear – and we know humans are wired to respond to authority and urgency. This topic of “vishing” takes CEO fraud and supercharges it.

What this means for your customers

  • Treat visual-generation tools just like text-generation tools: govern input-data, control prompts, monitor outputs.
  • Use them wisely: great for internal prototyping, concept visuals, creative slides. But if you rely on them for  mission-critical tasks, and ensure you have human/audit oversight.
  • In your cybersecurity context: remind users that vishing isn’t some futuristic concept, it’s already happening, so remember that you can’t trust everything you see – including your own CEO!
No actors required!

How to defend against deepfake vishing

Adopt the mindset that a video is no longer proof of identity. Practical controls include:

  • Second-factor verification: Always validate payment requests, account changes or unusual instructions using a secondary channel.
  • Strict financial approvals: No exceptions because “the CEO said so in a video”.
  • Enhanced Security Awareness Training: Teach staff what deepfakes look like and how attackers use urgency to manipulate them.
  • Monitoring for compromised public footage: Know what voice/video samples of leadership exist online.
  • Communications culture: Normalise asking for proof; it’s perfectly acceptable to ask your CEO for a something only they would know – like where they hide the good biscuits in the office!  Adopt the Zero-Trust mantra “never trust, always verify” (with an embarrassing story from last year’s office party).

Assisted vs Unassisted Machine Learning vs GPTs

If you’re still reading, well done!  I said at the start I’d place all this in context, by contrasting three paradigms, which is especially important given the focus on cybersecurity.  While “AI” gets used as a catch-all moniker, Assisted-ML, Unassisted-ML, and GPT-based systems behave very differently and carry distinct operational risks.

Assisted Machine Learning is what most mature cybersecurity teams rely on today: the model flags suspicious behaviour based on labelled historical data, and a human analyst verifies, interprets and acts. It’s powerful because the final decision remains anchored in human judgement, institutional context and domain expertise. In other words, the machine augments your visibility and speed, but people still ensure accuracy and relevance.

Unassisted Machine Learning by contrast, is fully automated decision-making. Once trained and deployed, the model operates autonomously; it filters emails, scores threats, blocks attacks, and triggers alerts without a human in the loop. This is extremely efficient, but it comes well known risks: model drift, reduced transparency, and brittle edge-case behaviour. If something falls outside the patterns the model has seen before, it can misclassify in surprisingly harmful ways and that’s where you end up with false-positives and false-negatives. In cybersecurity, where novel attacks appear constantly, that risk must be taken seriously.

Generative models like GPT-style systems sit in a completely different category; they don’t classify or detect in the traditional sense.  As you’ve learnt through this article, they generate text, ideas, or explanations based on statistical patterns in language. They’re superb for summarisation, drafting and exploration, but they’re not built for deterministic decision-making. A GPT doesn’t “know” your threat landscape or business context; it predicts the next most likely sentence. Treating generative AI as if it were an analyst or automated control leads to false confidence and governance gaps. Used well, it’s an assistant. Used poorly, it just amplifies your risk.

Assisted Learning vs. Unassisted Learning vs. GPTs

How to decide what goes where

In cybersecurity operations:

  • For detection/classification tasks (e.g., phishing, anomaly detection) you might choose assisted-ML or unassisted-ML depending on risk tolerance.
  • For content/knowledge work (e.g., drafting reports, summarising investigations) you might use GPT-style tools, but even then with human-in-the-loop.
  • For autonomous decision-making (e.g., “is this incident high-severity? should I auto-remediate?”), you should avoid relying purely on generative AI today.

Conclusion

In summary: tools like GPTs and generative-image models are powerful, exciting, and hold material value for businesses, particularly in the cyber threat landscape of SME’s.  But, to repeat how I started, they are not magic. They are not fully-reasoning AI, they don’t inherently understand context or domain-risk, and they carry real governance, data-leakage, and trust risks.

For many organisations the smart approach is this: adopt generative-AI as an assistant, not as a substitute for human expertise. Embed it into your internal risk model: controlled, monitored, human-in-the-loop. Use it to amplify capability, not replace oversight. And treat it as you would any other new attack surface in your cybersecurity posture.

When you apply that mindset, you unlock the positive potential while managing the downside.

The Future of AI – Where do we go from here?

Looking ahead, GPT systems will continue to evolve: larger context windows, better memory, more reliable “thinking modes,” and tighter integration with 3rd party tools that let them call APIs, analyse data, or execute tasks. But even as these models grow more capable, they are still fundamentally just pattern recognition engines, not conscious reasoners. We will surely experience more impressive leaps in accuracy and usefulness, but we should expect the same foundational limitations: no real understanding, no true autonomy, and no guarantee of correctness without supervision. General Artificial Intelligence – the kind that can reason, reflect, and apply judgement across domains, in my opinion remains an ongoing scientific challenge rather than an applied imminent reality. It will arrive one day, but until then we’ll have to settle for an incredibly confident, occasionally wrong autocomplete.

Impact on Cybersecurity: More Power, More Risk

For cybersecurity, the future is a double-edged sword. On one side, we’ll gain extraordinary tools: AI systems that can parse millions of signals, summarise incidents in seconds, assist in threat hunting, and amplify analyst productivity. On the other side, attackers will have access to the same advancements—AI-generated malware, automated reconnaissance, deepfake-driven social engineering, and highly targeted phishing at industrial scale. As generative models become more capable, the speed, sophistication and accessibility of cybercrime will increase. The organisations that thrive will be the ones that pair AI-augmented defences with strong governance, user training, and a culture of healthy scepticism.

The Bottom Line

AI will reshape cybersecurity—but not by replacing humans. Instead, it will force us to evolve: to build smarter controls, to train staff continuously, and to adopt a zero-trust mindset not just for systems, but for information itself. GPTs may become astonishingly helpful co-pilots, but they will remain tools, not oracles. The winners in this next era will be the organisations that harness AI thoughtfully, keep humans firmly in the loop, and prepare for a world where both defenders and attackers wield unprecedented machine-assisted power.

The future of AI is basically us playing schoolteacher to the class overachiever: always handing in homework before the deadline, always answering boldly…but always needing to be marked.