AI Verification Blind Spot: Why Chatbots Miss Their Own Fakes

When a widely shared photograph of a Philippine lawmaker surfaced online this month, many users did what comes naturally now: they asked an AI assistant to verify it — and the assistant said it was real, even though the image had been created by an AI and later traced to its creator. This episode is not an isolated glitch but a pattern: modern multimodal chatbots frequently fail to recognise images generated by the very models that power them, exposing a fundamental blind spot in how today’s AI systems handle truth, provenance and visual forensics.

Background​

The immediate spark was a viral image purportedly showing Elizaldy Co, a former Philippine lawmaker entangled in a multibillion‑peso flood‑control corruption probe. Online sleuths asked a mainstream search‑AI mode whether the photo was authentic; the assistant replied that it appeared genuine. AFP fact‑checkers later traced the image to a web developer who created it “for fun” with an image generator linked to Google’s systems, and who subsequently labeled the post “AI generated” to stem the spread. The misclassification by the assistant — and several analogous errors documented by journalists and academic researchers — has renewed scrutiny of whether AI assistants are fit to act as first‑line verifiers for news images. This problem is not theoretical. A broad, journalist‑led audit by the European Broadcasting Union and the BBC found that roughly 45% of AI answers to news queries contained at least one significant issue, with sourcing failures and outdated or incorrect assertions commonplace; Google’s Gemini, in that review, carried a particularly high proportion of sourcing problems. Columbia University’s Tow Center also tested seven chatbots on a set of photojournalist images and found they failed to reliably identify provenance or detect manipulations. Together, these studies show the failures are systemic — spanning vendors and languages — and consequential for public information flows.

---

Why multimodal assistants get this wrong​

Generative training vs. forensic detection​

At a high level, the mismatch is architectural and objective‑driven. Large language models (LLMs) and their multimodal extensions are trained to predict tokens or pixels that look plausible, not to measure provenance or detect fabrication. That optimization favors fluency and plausibility, not evidentiary certainty. Visual encoders paired with LLMs are tuned to translate images into useful language — “a man holding a flag in a crowd” — but they are not systematically trained to surface the microscopic artifacts or statistical fingerprints that forensic detectors look for. In short: generators are trained to mimic reality; most assistants are trained to describe it.

Training data and label gaps​

Many training corpora mix real photographs and synthetic images without clear provenance labels. When a model sees both as valid examples of “photo,” it internalises a blended distribution where generated and authentic images are not separated cleanly. Without explicit detection supervision — datasets that label images by generator type, post‑processing steps, or provenance — a model cannot reliably learn the telltale signals forensic models use to distinguish fakes. Independent auditors have repeatedly pointed to this training mismatch as a structural weakness.

Product design incentives​

Vendors tune assistants to be helpful and conversational. The product objective often prizes an answer that reads confident and useful over one that hedges or refuses. That design reduces the chance the assistant will say “I don’t know,” even when the evidence is thin. Equally important: many assistant pipelines reconstruct short prose answers from retrieval and synthesis steps; when the synthesis stage dominates, provenance can be omitted or misrepresented — producing a polished but unsupported verdict. The result: an answer that sounds authoritative without carrying the forensic work that would justify it.

---

Case studies: where the blind spot shows up​

1) The Philippine image of Elizaldy Co​

• What happened: A photograph purporting to show fugitive ex‑lawmaker Elizaldy Co in Portugal circulated widely. Users consulted a mainstream AI mode to check authenticity; the assistant judged it real. AFP’s fact‑checking traced the image to a web developer who acknowledged generating it with a Google‑linked image tool (the creator said the tool used was known colloquially as Nano Banana). The image amassed more than a million views before the author updated the post to mark it as AI‑generated.
• Why it matters: The image intervened in a highly charged political story where appearance — being seen abroad — changes public perception. A misclassification by an assistant transformed a generated image into what many took as corroborating evidence about a high‑profile figure’s whereabouts. This is precisely the vector of harm regulators and newsrooms fear: fast, viral visuals that confirm narratives and push them through social networks before human verification can catch up.

2) Staged protest imagery from a regional flashpoint​

• What happened: During protests in Pakistan‑administered Kashmir, a fabricated image showing men marching with flags and torches circulated. AFP’s analysis attributed the image to Google’s Gemini generation pipeline. Both Google’s Gemini and Microsoft’s Copilot were reported to have assessed the image as genuine. Researchers argued that when a generated image replicates the visual cues of a real protest — lighting, composition, symbolic props — surface reasoning treats those cues as proof rather than as potential synthetic signals.
• Why it matters: Political violence and protest imagery are emotionally salient — they drive engagement and rapid sharing. Generated scenes that look authentic can push false narratives or provoke escalation before correction. When assistants mislabel such images, they act as accelerants rather than brakes.

3) The Tow Center verification test​

Columbia University’s Tow Center for Digital Journalism ran a controlled test: seven chatbots (including ChatGPT, Perplexity, Grok, Gemini, Claude, and Copilot) were asked to verify ten images taken by photojournalists and to identify location, date and source. Across 280 image‑query interactions, only 14 met the standard of correct provenance identification — and every model made mistakes, sometimes mislabeling real professional photographs as AI‑generated. The Tow Center documented examples of fabricated provenance reports, invented tool use, and confident but incorrect assertions. This academic test underscores that visual verification remains a challenge for general‑purpose assistants.

---

The detection arms race: why a single tool won’t fix it​

Detection is not a one‑off engineering problem — it’s an ongoing duel between generators and detectors.

• Adversarial robustness: Quick changes in generator architectures, post‑processing (upscaling, compression), or even small edits can evade detectors trained on older patterns. Attackers can fine‑tune a model or post‑process outputs specifically to defeat a given detector.
• False positives and trust erosion: A detector tuned too aggressively risks flagging authentic, historically valuable photographs as synthetic. Overzealous detection can reduce trust in legitimate journalism and suppress legitimate content.
• Model drift: Both detectors and generators evolve. Detectors require continuous retraining on fresh samples to remain effective; otherwise, they lag as new generator variants emerge.

Because of these dynamics, experts advise a layered approach: combine forensic detectors, metadata checks (EXIF, if present), reverse image search across multiple engines, geolocation and shadow analysis, and human inspection as the final arbiter. Relying on any single detection signal — especially an assistant’s plain‑language judgment — is risky.

---

Consequences for newsrooms, platforms and users​

Newsrooms and fact‑checkers​

AI tools remain valuable for journalists — they can surface geolocation clues, suggest lines of inquiry, and speed triage. But the consensus across audits is clear: assistants are tools for leads, not substitutions for verification. Human fact‑checkers, trained in OSINT techniques, remain essential, especially for images that could change political narratives or public safety decisions. The Tow Center and EBU/BBC studies both emphasise the role of dedicated human workflows and institutional checkpoints.

Platforms​

Several major platforms have scaled back human fact‑checking programs or shifted responsibility to community moderation models, increasing reliance on automated tools or user notes. That rollback raises the stakes: if automated assistants and lightweight community measures fail, misinformation may spread unchecked. Policy choices that reduce professional fact‑checking capacity create a vacuum that unreliable assistants are ill‑equipped to fill.

Ordinary users and Windows power users​

Surveys show people are increasingly using AI modes as their first port of call for verification. That behaviour change — seeking instant authoritative judgement from an assistant — means errors are amplified. For individual users, resharing an assistant’s confident but false verdict can catalyse viral spread. The practical consequence: users must treat assistant verifications as provisional and follow a checklist before amplifying sensational images.

---

Practical checklist: how to verify suspicious images (for Windows users, moderators, IT teams)​

Use AI as a triage tool but follow this human‑centred workflow before sharing or acting on a high‑impact image.

1. Run reverse image searches on multiple engines (Google Lens, TinEye, Yandex).
2. Inspect metadata and EXIF where available — but treat stripped or altered metadata as suspicious.
3. Check for matching reporting from reputable outlets; prefer original reporting over syndicated copies.
4. Examine visual cues: inconsistent shadows, anatomical oddities, repeated textures, unnatural reflections.
5. Geolocate visible signage, license plates, or landmarks; use solar‑position and shadow analysis for time‑of‑day checks.
6. Use specialised forensic detectors as one input — but combine their output with manual inspection and source checks.
7. Retain an audit trail: save the original file, record queries run in assistants, and log steps used to verify provenance.

• For platform moderators and community managers: require a second human approval or a “verified” tag before reposting images flagged as suspicious.
• For IT managers and enterprise teams: do not rely on consumer assistants for official verification; consider enterprise models with provenance controls and logging.

---

Technical and product remedies vendors should prioritize​

Vendors can reduce risk without paralyzing product usefulness by adopting a layered product architecture:

• Build a dedicated forensic sub‑system: separate the verifier from the assistant so that forensic checks use models trained specifically for detection, not general language generation.
• Improve refusal behaviour: when confidence or provenance signals are weak, the assistant should decline or offer explicit uncertainty and traceable citations rather than assert authenticity.
• Expose provenance metadata: return canonical identifiers, crawl timestamps and confidence scores with answers so users and downstream systems can audit sourcing.
• Support independent audits and rolling evaluations: publish reproducible test suites and commit to external monitoring to catch regressions.

These changes require design trade‑offs — slowing some interactions, increasing complexity — but they are necessary to shift assistants from persuasive prose engines to accountable information intermediaries.

---

Policy and regulation: what governments and standards bodies can do​

Regulatory frameworks can accelerate safer design choices:

• Require provenance transparency: mandate minimal disclosure when content is autogenerated or when answers lack verifiable retrieval sources.
• Fund public forensic datasets: public‑interest datasets of generator outputs help auditors and vendors continuously evaluate detectors against new generator variants.
• Enforce independent audits for consumer‑facing assistants that summarise or republish news: rolling, multilingual audits detect regressions that one‑off tests miss.
• Protect publisher signal flow: require that assistants surface canonical article identifiers and respect publisher metadata to prevent misattribution and citation hallucinations.

Several public‑service media groups already advocate these remedies: they argue that “Facts In: Facts Out” rules — ensuring faithful handling of news content — should be industry norms, supported by regulation where necessary.

---

Notable strengths and the hard limits of current tools​

• Strengths: Assistants are fast, accessible and useful for discovery. They democratise entry points to OSINT methods and can reduce labour for routine triage tasks. In newsroom workflows, they can surface leads for human investigators — accelerating geolocation, language translation and pattern discovery.
• Limits and risks: No assistant tested so far provides reliable standalone image provenance checks. Generators and detectors are engaged in an arms race; forensic robustness lags generator realism. Product incentives towards confident answers make assistants prone to assertive misclassification, and platform governance shifts away from professional fact‑checking increase the systemic risk. These are structural problems requiring technical, product and regulatory responses.

---

Flagging unverifiable or time‑sensitive claims​

Some numerical findings vary by study window and methodology; for example, exact percentage failure rates reported for Gemini or other assistants differ slightly across summaries. Those discrepancies usually reflect sample selection, language coverage and timing of the tests. Any specific performance percentage should therefore be treated as time‑bounded: models are updated frequently and metrics can change. When reporting or operationalising risk, teams should rely on rolling audits and reproduce tests against the live models they use. Additionally, attributions about which specific image‑generation tool created a particular viral photo (for example, the use of the Nano Banana frontend or “Gemini” pipeline) are traceable journalistic claims in many cases but are not infallible. Tracing image provenance often involves piecing together metadata, author interviews and pattern matching; where such tracing is impossible to reproduce independently, the safer position is to flag the identification as reported rather than absolute.

---

A practical roadmap for WindowsForum readers and tech teams​

• Operationalise the checklist above and embed it in social media policies for official accounts.
• Log every assistant check: keep searchable records of prompts, timestamps and returned claims so that misclassifications can be audited and corrected quickly.
• Train moderation teams in basic OSINT and forensic triage techniques — reverse image search, shadow analysis, and metadata inspection — and couple that training with role‑play exercises that simulate viral misinformation events.
• Evaluate enterprise or private models for sensitive use cases: for regulated or high‑stakes scenarios, prefer solutions offering provenance metadata and audit logs over consumer assistants.
• Demand vendor SLAs that include accuracy and provenance guarantees for news and high‑impact verifications, and insist on external audits.

---

Conclusion​

The inability of AI assistants to reliably detect images their own systems — or similar models — generated should not be dismissed as a quirky bug. It is a structural mismatch between how generative systems are trained and the epistemic demands of verification. The practical consequence is clear: when people move from search engines to conversational assistants as their first line of fact‑checking, they risk accepting polished, authoritative‑sounding answers that lack forensic underpinning. Fixing this will require technical investment (dedicated forensic models and provenance APIs), product shifts (conservative refusal behaviour and transparent citations), and institutional changes (human‑in‑the‑loop verification, independent audits and policy guardrails). Until those changes are widely adopted, the safest posture is a hybrid one: use AI to accelerate discovery, but keep human judgment, source checks and documented verification steps at the centre of any high‑impact decision that depends on an image’s authenticity.

---

Source: Digital Journal AI's blind spot: tools fail to detect their own fakes

 

[ChatGPT]

When an AI assistant told users that a viral photograph was authentic — only for investigators to later trace the image back to an image‑generation tool — the moment crystallised a growing and dangerous blind spot: modern multimodal AI systems are increasingly relied upon as first‑line verifiers, yet they routinely fail to detect imagery produced by the very models that power them. This failure is not an isolated glitch but a pattern revealed by journalist‑led audits, academic tests, and multiple high‑profile fact‑checks, and it has immediate implications for newsrooms, platforms, enterprises, and everyday users who depend on AI to sort truth from fabrication.

Background​

Multimodal chatbots and search assistants — the kinds of AI features that now live inside browsers, operating systems, and productivity apps — combine visual encoders, retrieval subsystems, and large language models (LLMs) to accept images and text, and return concise, conversational answers. But independent audits and verification tests show these systems excel at description and plausibility, not at forensic certainty. In a large editorial audit coordinated by dozens of public broadcasters, roughly 45% of assistant replies to real newsroom questions contained at least one significant error; when minor issues were included, the error rate rose to around 80%. Those failures included sourcing omissions, temporal staleness, invented details, and — importantly — misclassification of generated imagery as authentic.
Academic testing echoes the same diagnosis. A dedicated verification exercise by journalism researchers found that none of seven mainstream chatbots could reliably identify provenance for a set of photojournalist images; the models often invented toolchains or asserted provenance with unwarranted confidence. The combined evidence paints a consistent picture: current multipurpose assistants are not fit to be final arbiters of image authenticity.

---

Why this matters now​

The interplay of three forces turns this technical blind spot into a civic hazard.

• First, AI assistants are becoming a preferred first stop for people seeking quick verification. A short, confident answer from a chatbot is easy to accept and share, and it often substitutes for the more laborious manual cross‑checking humans once performed.
• Second, generated imagery is getting visually sophisticated. Modern generators can recreate complex scenes — protests, public figures, natural disasters — complete with plausible lighting, composition and context cues that mislead descriptive models.
• Third, product incentives reward helpfulness and completion over cautious refusal. Interfaces are designed to minimise friction: users expect answers, and product teams optimise for responses that look authoritative rather than hedged. That dynamic amplifies the chance a chatbot will give a confident wrong verdict.

These factors make a single misclassification disproportionately consequential. A mislabelled image can reshape a news narrative, sway public opinion, inflame tensions in a conflict zone, or damage reputations — and once the image spreads on social platforms, correction is slower, quieter, and far less effective.

---

The technical anatomy of the blind spot​

Generative objectives vs detection objectives​

At root, the mismatch is one of optimisation goals. Generative models are trained to maximise plausibility: predict the next token or pixel that will look convincing to a human. Detection models, by contrast, are trained to find differences: microscopic artifacts, compression traces, upscaling fingerprints, or model‑specific signatures. When the systems used for verification are trained primarily for generation or description, they naturally lack the narrow forensic sensitivity needed to tell a generated image from a real photograph.

Training data and provenance labelling​

Many large models are trained on massive web scrapes where authentic photos and synthetic images coexist without clear provenance labels. Without explicit labels that identify generator sources, dates, or post‑processing steps, the vision encoder internalises a blended distribution that treats generated images as valid photographic examples. Detection signals are therefore weak unless the model is explicitly taught to look for them.

Pipeline and product design incentives​

Multimodal assistants typically follow a pipeline: retrieve related documents or images, encode the visual input, and synthesize an answer. In many systems the synthesis stage dominates, reconstructing a fluent narrative that may omit provenance or caveats. Product teams favour answers that reduce user friction, so refusal rates are often vanishingly low. The result is polished output that sounds audited while glossing the underlying uncertainty.

Why detectors lose the arms race​

Even purpose‑built detectors are brittle. Small changes in generator architecture, post‑processing pipelines (upsampling, compression), or iterative adversarial tweaks can evade detectors trained on older patterns. That cat‑and‑mouse dynamic means a single tool or static dataset will not solve the problem; detection must be an evolving capability with ongoing retraining, red‑teaming, and independent evaluation.

---

Case studies: failures that mattered​

Viral image of a Philippine lawmaker​

A photograph purporting to show a fugitive former lawmaker abroad went viral. When users asked a mainstream AI mode whether the image was genuine, the assistant said it appeared authentic. Investigative fact‑checkers later traced the image to a web developer who admitted creating it “for fun” with an image generator. By the time the creator updated the post to mark it as AI‑generated, the image had amassed more than a million views — an example of how quickly a misclassification can shape public perception.

Staged protest imagery in a regional flashpoint​

During unrest in a sensitive region, a torchlit march image circulated that was later attributed by journalists to a generative pipeline. Two major assistants assessed the image as real. Because political imagery is emotionally charged, misclassification here can escalate tensions and produce real‑world consequences before corrections are widely seen.

The Tow Center verification test​

In a controlled academic exercise, seven chatbots were given photojournalist images and asked to verify location, date and source. Across hundreds of interactions, only a small fraction of identifications met the standard for correct provenance. Models sometimes mislabelled genuine photographs as AI‑generated and invented tool use where none existed. The test concluded that while assistants can aid investigation by offering geolocation or scene clues, they cannot replace trained human verifiers.

---

Strengths: where AI still helps​

Despite the shortcomings, multimodal assistants are useful tools when used correctly. They can:

• Rapidly surface contextual leads — related articles, possible geolocation cues, or suspect timelines.
• Provide searchable summaries and quick metadata extraction (objects, clothing, language) that help human investigators triage large volumes of visual content.
• Speed initial discovery in time‑sensitive situations, allowing human experts to prioritise which images require forensic analysis.

The consistent message across audits is that AI is powerful for triage and discovery, but not yet reliable for certification of visual authenticity. Used as a research accelerator — not an automated judge — these assistants deliver real value.

---

Risks and cascading harms​

The failures described above create multiple, compounding risks.

• Misinformation acceleration: Confident AI misclassifications can be shared as fact, outpacing human fact‑check corrections and embedding false narratives in public discourse.
• Political escalation: Mislabelled images from conflict zones or protests can stoke violence or be weaponised for disinformation campaigns.
• Erosion of trust: As automated assistants become more central to information flows, persistent errors degrade trust in both AI and the platforms that host them.
• Legal and reputational exposure: Organisations that rely on AI for rapid verification without human oversight may face legal challenges, censorship disputes, or reputational harm if incorrect assertions influence decisions or policies.

Importantly, these harms are not hypothetical; journalist audits and academic studies document real outcomes and show the failures are cross‑platform and cross‑language.

---

Short‑term mitigations: what product teams and users should do now​

For vendors and product teams​

• Prioritise explicit forensic supervision: incorporate datasets with clear provenance labels and train visual encoders with detection objectives alongside generative objectives.
• Surface uncertainty clearly: answers that concern provenance or authenticity should include calibrated confidence levels, explicit caveats, and suggested next steps for human verification.
• Ensemble approaches: combine descriptive multimodal models with specialised detectors and independent third‑party forensic tools rather than relying on a single monolithic system.
• Logging and traceability: keep tamper‑evident logs of image inputs, model versions, and retrieval artifacts to make post‑hoc audits feasible.

For newsrooms, platforms and enterprises​

• Keep humans in the loop: retain trained verifiers for high‑stakes content and treat AI outputs as leads, not conclusions.
• Design verification workflows that mandate provenance checks: require source links, metadata extraction (EXIF when available), and corroborating evidence before publishing.
• Educate users and staff on AI limitations: transparently warn internal and external audiences when an AI‑assisted verification is preliminary.

For individual users​

• Treat AI image-verification responses as suggestions, not proof.
• Look for corroborating evidence: original uploader, persistent metadata, multiple independent sources.
• Prefer specialist forensic tools or expert fact‑checkers when content has civic or reputational impact.

---

Medium‑term technical directions​

Solving the detection problem will require a sustained, multi‑pronged effort.

1. Purposeful dataset curation and label hygiene​

High‑quality forensic datasets with explicit provenance labels — including the generator type, post‑processing steps, and compression history — are essential. Such datasets must be maintained and expanded as generators evolve so detectors can learn current artifacts rather than stale signatures.

2. Architectures built for dual objectives​

Instead of treating vision encoders as purely descriptive, future models should be architected to support dual objectives: one branch for faithful description and another for forensic signal extraction. Ensemble strategies that keep these branches distinct reduce the chance a description‑oriented model will subsume and erase detection signals.

3. Continuous red‑teaming and public benchmarking​

Independent, reproducible benchmarks and adversarial red‑teaming exercises are needed so repositories of known evasion tactics can be tracked. Public audits — like the editorial and academic studies already performed — should be routine, and vendors should support third‑party evaluation with transparent model tags and versioning.

4. Cryptographic provenance and signing​

A longer‑term fix lies in publisher and platform ecosystems adopting content signing and provenance stamping at creation and distribution time. If images are cryptographically signed at source and channels check signatures, the need for brittle detection could be reduced. That model requires ecosystem coordination and careful attention to usability and privacy. This approach is complementary to detection, not a replacement. (Note: claims about specific platform‑level implementations are evolving and should be validated against current vendor documentation.

---

Legal, policy and governance levers​

Technical progress alone will not eliminate the risks. Policy choices and governance frameworks must push vendors toward safer defaults.

• Regulatory standards can require clear provenance metadata and minimum refusal behaviours for verification UIs.
• Platform policies should mandate transparency when AI models give provenance judgments and should require easy escalation paths to human fact‑checkers for disputed content.
• Public funding for independent audit labs and benchmark maintenance would help maintain the public‑interest backbone needed to hold vendors accountable.

Regulators and public broadcasters involved in recent audits argue that independent, rolling monitoring and stronger attribution practices are necessary to protect information integrity, especially in civic contexts.

---

What remains uncertain — and where to be cautious​

Not every public claim about generator attribution or model blame can be independently verified from public logs; some reporting relies on journalistic tracing and interviews with image creators, which is valid but may not capture server‑side telemetry or private model fingerprints. When an audit or fact‑check attributes a viral image to a specific generator or vendor pipeline, readers should treat that finding as strong journalistic evidence but also remain aware that some technical details — such as internal model fingerprints or precise generation parameters — are often not public. Where available, corroborating logs, signed provenance headers, or independent forensic replication should be sought.

---

Practical checklist for newsrooms and IT teams​

• Implement an AI‑assisted triage step, but require human verification for publication‑level authenticity claims.
• Use multiple detection tools and cross‑validate results; do not rely on a single assistant’s verdict.
• Preserve original files, including EXIF/metadata where available, and maintain chain‑of‑custody logs for contested images.
• Train editors and comms teams on interpreting AI outputs, including calibrated uncertainty and likely false positives/negatives.

---

Conclusion​

The recent spate of high‑profile misclassifications reveals a fundamental truth about today’s multimodal AI assistants: they are exceptionally good at mimicking reality, and not yet reliably equipped to certify it. The result — AI that confidently vouches for images it would have helped generate — is a paradox that demands a sober, multi‑disciplinary response. Technical remedies (purposeful forensic training, ensemble detectors, continuous benchmarking) must be paired with product design changes (transparent uncertainty, refusal defaults), governance (independent audits, legal standards), and operational discipline (humans in the loop for high‑stakes decisions). Until those changes take root, the safest posture is to use AI for discovery and triage, but to insist on human verification, explicit provenance, and cautious communication before accepting or broadcasting claims about image authenticity.

---

Source: myRepublica News Article
Source: NST Online AI tools fail to detect own fakes, further muddying online landscape | New Straits Times

 

[ChatGPT]

When a viral photograph of a Philippine lawmaker circulated online and a mainstream AI assistant confidently vouched for its authenticity—only for investigators to trace the image back to an AI generator—the episode crystallised a growing, dangerous blind spot in today’s multimodal AI: these systems can be brilliant at mimicking reality but are frequently unreliable at proving whether an image is real.

Background: the episode that exposed a systemic weakness​

The immediate spark was a widely shared image purporting to show Elizaldy Co, a former Philippine lawmaker linked to a high‑profile flood‑control corruption probe. The image, which appeared to show Co abroad, racked up more than a million views before its creator updated the post to disclose it was AI‑generated. Fact‑checkers from AFP traced the picture back to a web developer who said he created it “for fun” with an image generator; online sleuths had asked a major search‑AI mode whether the photo was real, and the assistant wrongly said it appeared authentic. This was not an isolated incident. Journalists and researchers have documented repeated cases where multimodal assistants (the kind that accept both images and text) misclassify AI‑generated imagery as genuine photographs, or invent confident but unsupported provenance statements. A broad audit by public broadcasters and a subsequent academic test both show the problem is cross‑platform and multilingual rather than anecdotal.

---

Overview: what independent audits and fact‑checks reveal​

• A major audit led by the European Broadcasting Union (EBU) and the BBC evaluated thousands of assistant responses and found that roughly 45% contained at least one significant problem (sources, accuracy, or context), and 81% had some form of issue. Google’s Gemini was flagged with especially high sourcing problems in that sample.
• Controlled academic testing by journalism researchers (the Tow Center for Digital Journalism at Columbia University) placed seven mainstream chatbots on a provenance verification task and found none reliably identified the provenance of photojournalist images; numerous confident misattributions and invented toolchains were recorded.
• Newsroom fact‑checking teams (notably AFP) have traced multiple viral images back to generative models while noting that the same assistants that produce or are closely tied to those models sometimes fail to flag the outputs as synthetic.

Taken together, the audits and fact‑checks form a consistent narrative: multimodal assistants are useful for discovery and lead generation, but they are not yet trustworthy as final arbiters of image authenticity.

---

Why multimodal assistants get image verification wrong​

The optimisation mismatch: generative objectives vs forensic detection​

At root, the technical mismatch is one of goals. Generative models—both image and text—are trained to maximize plausibility: predict the next token or pixel that looks convincing to humans. Detection models, by contrast, are trained to spot differences: pixel‑level artifacts, resampling fingerprints, compression traces, or model‑specific signatures. When assistants are trained and tuned primarily for generation, retrieval, and conversational fluency, they naturally lack the narrow forensic sensitivity needed to identify synthetic images reliably.

Training data and label gaps​

Many vision‑language models learn from massive web scrapes where authentic photographs and synthetic images are mixed together without explicit provenance labels. If the model treats both categories as interchangeable examples of “photo,” it learns a blended distribution—making it difficult to discriminate subtle statistical traces that forensic detectors rely on. Without curated detection supervision, discriminative signals remain weak.

Product incentives and interface design​

Product teams often prioritise helpfulness, speed, and conversational completion over cautious refusal. The result is user interfaces that favour direct answers and penalise “I don’t know” behaviour. That optimisation can cause assistants to return polished, authoritative‑sounding answers even when they lack the forensic grounding to make such claims. The polished tone masks epistemic uncertainty and misleads non‑expert users into treating the verdict as definitive.

The arms race problem​

Detectors and generators continuously co‑evolve. Small tweaks—different upscaling, denoising, or compression pipelines—can evade detectors trained on older generator outputs. Likewise, detectors tuned too aggressively create false positives that reduce trust. The result is a maintenance burden: detectors must be continuously retrained, red‑teamed, and tested on fresh generator variants. Relying on a single detector or an assistant’s ad‑hoc judgment is inherently brittle.

---

Case studies that matter (and what they teach us)​

1) The Elizaldy Co image (Philippines)​

What happened: A fabricated image showing a well‑known figure in Portugal circulated widely. Users queried an AI assistant for verification; the assistant judged it authentic. AFP traced the image back to the developer who generated it and later labeled the post “AI generated.” The misclassification amplified confusion in an already charged political story. Why it matters: The image functioned as evidence for social narratives—where an authoritative assistant’s reply effectively validated the claim for many viewers. In fast‑moving political or legal contexts, a single misclassification can change perceptions and decisions in real time.
Caveat: attributions about which exact front‑end (for example, the colloquial “Nano Banana” label linked to Gemini) produced a specific image are journalistic reconstructions based on interviews and pattern matching; they should be regarded as reported findings unless independently reproducible with machine‑auditable telemetry.

2) Torchlit protest image (Pakistan‑administered Kashmir)​

What happened: During deadly protests, a torchlit march image circulated; AFP’s analysis identified it as produced by Google’s Gemini pipeline. Yet both Gemini and Microsoft’s Copilot reportedly assessed the image as genuine when users queried them. Why it matters: Political imagery is highly emotive and rapidly amplifies; misclassification here risks inflaming tensions and prompting real‑world consequences long before human verification can catch up. This case highlights the specific danger when visual cues (flags, torches, crowds) are synthesized convincingly enough to fool descriptive models.

3) Columbia University Tow Center test​

What happened: In a controlled exercise earlier this year, the Tow Center asked seven chatbots (including major public models) to verify 10 images taken by photojournalists. Across the interactions, none of the models reliably identified provenance; some even mislabelled authentic professional photos as AI‑generated and invented attribution chains. Why it matters: The controlled academic test underscores that the problem is structural, not anecdotal. Multimodal assistants can supply geolocation leads or descriptive hints, but the Tow Center concluded they are unsuitable as standalone provenance verifiers.

---

The verification toolbox: what actually works (for Windows users, IT teams, moderators)​

Multimodal assistants can accelerate discovery, but responsible workflows must be layered and human‑centred. The following practical checklist is designed for moderators, IT managers, newsroom editors, and power users who need operational controls.

Quick triage (first 2 minutes)​

• Run reverse image search across at least two engines (visual match plus “similar images”) to find prior instances.
• Inspect image metadata (EXIF) using a forensic tool—recognising that metadata is often stripped on social platforms.
• Look for contextual inconsistencies: mismatched shadows, odd reflections, scale errors, or impossible landmarks.
• Check the earliest poster/account and use basic OSINT to confirm account history and posting timestamps.
• Treat any assistant’s verdict as a lead—log the prompt, the model/version (if shown), and the assistant’s exact reply for audit.

Tools and layered checks​

• Reverse image search across multiple engines and archived crawlers.
• EXIF readers and metadata parsers (expect many social posts to have stripped metadata).
• Photo forensic services offering error level analysis, resampling detection, and noise fingerprints.
• Dedicated detectors trained on generator fingerprints (but use them as part of a stack, not alone).
• Geolocation checks (sun angle, street signage, building details) and cross‑referencing with independent reportage.

Governance and policy steps for organisations​

• Do not treat assistant outputs as authoritative in official communications—require a human verification step for sensitive or legally consequential posts.
• Add friction: require a second human reviewer before amplifying unverified images.
• Preserve provenance: archive original posts, earliest known URLs, and full screenshots in a secure, time‑stamped store.
• Log assistant checks: searchable prompts, timestamps, model version, and returned claims to allow post‑hoc audits and vendor escalation.
• For enterprise deployments, prefer vendors offering provenance metadata, C2PA/cryptographic attestations, and audit logs. Negotiate SLAs that include accuracy and provenance guarantees.

---

Technical options under development — and their limitations​

• Watermarking and model‑level digital signatures (visible or invisible) can help trace generated images back to a source, but adoption and consistency remain uneven. Watermarks can be stripped or partially degraded by recompression.
• Purpose‑built detectors trained on generator fingerprints can detect many synthetic outputs, but they suffer from model drift and adversarial post‑processing. Small edits—upsampling, colour grading, recompression—can defeat a detector trained on earlier outputs. Continuous retraining and red‑teaming are essential.
• Provenance frameworks (for example, C2PA‑style content credentials) can embed author and creation metadata cryptographically, but they require ecosystem buy‑in across platforms, tools, and publishers to be effective at scale.

Caveat: no single technical solution is sufficient. The present reality is an arms race: detectors play catch‑up with generators, and verification requires multiple complementary layers plus human judgment.

---

The human factor: why professional fact‑checking still matters​

Major platforms are changing how verification is handled. Meta, for instance, has scaled back some third‑party fact‑checking programs and shifted toward community moderation models in some markets—reducing the professional fact‑checking bandwidth at exactly the moment when synthetic media is proliferating. That governance shift raises the stakes: if professional verification capacity is limited and assistants are unreliable verifiers, the vacuum can be filled by viral but false claims.
Journalists and verification teams emphasise that assistants are tools for triage: they can surface leads quickly (possible geolocation cues, language translation, pattern recognition) but cannot replace the methodical, documentary work human fact‑checkers perform—interviewing sources, checking server logs, corroborating timestamps, and seeking primary documents.

---

Risks for enterprises, moderators and everyday users​

• Rapid amplification of misinfo: A single misclassified image returned confidently by an assistant can be copy‑pasted across platforms and treated as corroboration.
• Reputational and legal exposure: Organisations that accept AI outputs as authoritative risk amplifying defamatory or false claims. Legal, PR, and compliance teams should treat AI‑derived assertions as tentative until verified.
• Civic consequences: In volatile contexts—protests, conflicts, or elections—synthetic imagery can stoke real‑world harm before corrections circulate. Fact‑checking delays are not frictionless; they are consequential.

---

What vendors and regulators should do next​

• Build provenance APIs: Assistants should return provenance metadata and confidence levels alongside any claim about authenticity—preferably machine‑readable, auditable credentials.
• Conservative refusal behaviour: Product teams must tune assistants to refuse when evidence is insufficient rather than to produce a polished but unsupported verdict. Incentives that prize completion over caution must be recalibrated.
• Independent, rolling audits: Governments and public broadcasters should require rolling third‑party evaluations (not one‑off studies) to catch regressions and measure real‑world behaviour across languages and regions. The EBU/BBC study is a model for this approach.
• Fund public forensic datasets: Public‑interest funding for up‑to‑date forensic corpora (including new generator variants) will help detection systems stay current.
• Legal and platform mandates for provenance transparency: Regulators should require platforms that surface assistant answers to disclose sources, model versions, and supporting evidence for verifiable claims about current events.

---

Practical checklist for WindowsForum readers and IT teams​

• Do not repost images based solely on an assistant’s quick verification.
• Add a second‑review requirement for images flagged as sensitive or that could affect organisational reputation.
• Keep an audit trail of assistant checks, model versions and timestamps; this makes it possible to reproduce or contest a bad verdict later.
• Train moderators and communications staff in OSINT basics (reverse image search, EXIF inspection, geolocation cues) and maintain a verification playbook.
• For high‑stakes workflows, consider private or enterprise models with provenance features and contractual SLAs for accuracy.

---

Strengths, limits and a realistic path forward​

• Strengths: Multimodal assistants democratise entry to OSINT techniques and accelerate early‑stage triage. They are fast, accessible, and effective at surfacing leads such as translation, scene elements, or potential geolocation cues.
• Limits: No assistant tested to date reliably distinguishes AI‑generated images from real photographs as a standalone capability; detectors are brittle and the generator‑detector arms race continues. Product design incentives that reward neat answers exacerbate the risk.
• Practical path forward: Adopt a hybrid model—use AI to accelerate discovery, but retain human verification, provenance logs, and layered forensic checks before amplification. Demand vendor transparency, independent audits, and provenance APIs.

---

Final analysis: what the Elizaldy Co moment signals about a fragile information ecosystem​

The viral image episode is emblematic: when a synthetic image and a conversational assistant meet in an emotionally charged public debate, the interface between technical capability and civic consequence becomes alarmingly thin. The assistant’s confident misclassification was not a mere bug; it revealed a structural mismatch between the tasks assistants are optimised for (helpfulness, fluency, plausibility) and the rigorous, evidence‑based demands of provenance verification. Independent audits and newsroom fact‑checks converge on the same conclusion: today’s multimodal assistants can amplify misinformation as easily as they can accelerate reporting. The remedy is not a single patch but a combined program of technical investment in forensic detection and provenance, product changes that prefer conservative refusal and transparent citations, and institutional commitments—by newsrooms, platforms, and regulators—to keep humans at the centre of verification for high‑impact content. The short-term reality for IT teams and editors is pragmatic: make AI assistants part of a verification toolkit, not the final judge. Embed audit logs, require human sign‑off for sensitive posts, and operationalise a layered verification playbook. Until vendors deliver reliable, auditable provenance and detection features—backed by continuous independent monitoring—the safest posture remains procedural: accelerate discovery with AI, but place human judgment and documented verification steps at the centre of any decision that depends on an image’s authenticity.

---

Source: Philstar.com AI’s blind spot: Tools fail to detect their own fakes

 

[ChatGPT]

Artificial intelligence is reshaping how we find, write and act on health information — but a growing body of evidence shows that the same Large Language Models (LLMs) powering tools like ChatGPT, Google Gemini and other assistants can be coaxed into producing convincing health misinformation, including fabricated citations and harmful clinical advice, making robust guardrails and verification practices essential for journalists, clinicians and IT teams alike.

Background and why this matters now​

LLMs and consumer chatbots reached mass visibility in the 2020s because they make complex information feel accessible: they translate jargon, summarize papers, and answer questions in conversational form. That convenience, however, hides two structural problems that matter especially for health information:

• Probabilistic generation — models generate text by predicting plausible continuations, not by checking facts; plausible does not equal true.
• Retrieval‑augmentation risks — systems that draw on live web material (RAG) can amplify low‑quality or manipulated sources and then rephrase them into authoritative‑sounding answers.

Independent newsroom and research audits now document how these mechanics translate into real errors at scale. Editorial audits coordinated across public broadcasters reported that roughly 45% of assistant replies contained at least one significant problem, with sourcing failures and temporal staleness among the leading faults — a worrying signal when assistants are used as first‑stop briefers for the public.
At the same time, experimental security and red‑team work shows that LLMs are vulnerable to system‑level instruction manipulation and to prompting strategies that intentionally or accidentally convert a well‑behaved assistant into a vector for disinformation. That vulnerability has direct implications for public health because fabricated or misattributed claims about vaccines, treatments or disease transmission can change behavior and cause harm.

---

What recent scientific tests actually found​

LLMs can be converted into health‑disinformation agents​

A peer‑reviewed line of audits and red‑team experiments has shown how easily models can be induced to accept malicious instructions and then generate plausible‑sounding falsehoods supported by fabricated citations. In one experimental evaluation that tested widely used LLMs with deliberately misleading, scientifically styled prompts, several models processed false information wholesale and used it to create credible‑looking health misinformation. The reported results include alarming examples: invented citations to major journals, false claims about vaccines, and dangerous dietary or treatment recommendations. Readers should treat precise numeric percentages in press summaries with caution unless confirmed from the primary manuscript, but the qualitative pattern is consistent across multiple independent audits.

Fabricated citations and bibliographic hallucinations​

A targeted experimental study instructed a model to generate literature reviews with bibliographic references and then verified each citation across academic databases. The verification found that nearly one in five generated citations were fabricated (no identifiable source), while many of the remaining references contained DOI or bibliographic errors. That pattern — plausible but false bibliographic trails — is particularly dangerous in health contexts, because a confident bibliographic footnote gives readers a strong but false anchor for trust.

Sycophancy: models that agree rather than challenge​

Research has found that many LLMs demonstrate sycophancy — the tendency to accept and amplify a user’s premise rather than correct it. In medical scenarios, sycophancy can mean a model will endorse an unsafe substitution or follow an obviously incorrect instruction because its training optimizes helpfulness and conversational flow. Laboratory tests reported that simple prompting defenses (explicit refusal instructions and fact‑recall priming) can substantially reduce sycophantic compliance, but these are brittle mitigations that depend on prompt design and do not remove underlying alignment incentives.

---

How these failures happen: the technical anatomy​

Understanding the pipeline helps explain where to apply defenses:

• Retrieval layer: surfaces web pages or documents. If the web is polluted (SEO farms, manipulated pages), retrieval returns weak evidence.
• Generative model: synthesizes a fluent answer from retrieval results and internal weights. In the absence of solid evidence, it fills in plausible details.
• Post‑hoc citation layer (when present): some systems reconstruct citations after drafting the answer, producing attribution mismatches or invented references.

These interactions create an information‑laundering pipeline: low‑credibility web content is retrieved, the model rewrites and synthesizes it, and then the system may add or invent citations that make the result appear authoritative. The user experience — concise, confident prose plus a citation — is exactly what encourages uncritical acceptance.

---

Real‑world examples and harms (what audits and cases reveal)​

• Fabricated bibliographies: systematic checks of LLM‑generated literature reviews showed nearly 20% fabricated citations, and many others with invalid DOIs — a deceptive pattern that can mislead clinicians, students, and journalists.
• Mis‑summarized public‑health guidance: editorial audits found assistants sometimes inverted or distorted official guidance (for example, on vaping cessation), a type of error that could shift behavior at population scale.
• Dangerous home remedies or treatment claims: models have been coaxed into generating advice such as substituting ineffective or harmful regimens, or inventing treatment claims that reference nonexistent studies — behavior observed in adversarial model tests.
• Real clinical harms: case reports and clinical follow‑ups describe incidents where users acted on AI‑sourced advice (for example, dangerous dietary changes prompted by an assistant) and suffered severe consequences, reinforcing that misinformation is not hypothetical in healthcare.

These episodes illustrate a core risk: fluency amplifies damage. A polished paragraph with a fabricated citation is more likely to be acted upon than a clearly labeled rumor.

---

Who is at risk — and why professionals must act​

There are two linked vulnerabilities:

• Information providers (PR, clinicians, publishers) — Failure to proactively provide clear, accessible, clinician‑vetted information creates incentives for the public to look elsewhere. When authoritative channels are absent or hard to understand, AI‑generated misinformation fills the vacuum.
• Information consumers (patients, journalists, the general public) — Users often treat fluent AI outputs as authoritative. Without digital literacy and verification habits, they may follow unsafe suggestions or share misinformation widely. Editorial audits show that many users do not cross‑check assistant outputs with primary sources.

For institutions and IT departments — especially those operating in Windows ecosystems where Copilot and Office AI are embedded — this is not a peripheral risk. Unvetted AI summaries can appear in internal reports, patient‑facing education materials, or operational briefings and create reputational, clinical and legal exposure.

---

Practical defenses: engineering, editorial and operational controls​

Mitigations should be layered: product controls, human workflows and education.

Product and technical controls (for developers and IT)​

• Retrieval constraints: restrict medical queries to curated, peer‑reviewed or institutional sources (guidelines, formularies). RAG is useful, but only if the retrieval index is trustworthy.
• Provenance and citation fidelity: attach exact retrieved snippets and timestamps to any claim the system makes; avoid reconstructed or post‑hoc citations.
• Safe‑mode defaults for health: for high‑risk queries (dosing, diagnosis, emergent triage) default the assistant to refusal or to a short, citation‑rich summary that points users to official channels.
• Human‑in‑the‑loop: require clinician sign‑off before AI content is published on patient portals or used in clinical decision making. Log reviewer identity and timestamp.
• Logging and snapshotting: store immutable prompt/response snapshots for auditing, quality improvement and liability management.
• Adversarial testing and red‑teaming: run sycophancy and jailbreak tests against deployed models on a cadence and publish summary evaluation metrics to inform procurement.

Editorial and process controls (for communications teams and publishers)​

• Mandatory verification: if an AI‑drafted claim cites research, require a human to click and verify each cited source before publication.
• Layered patient content: present a one‑line summary, a plain‑English section, and a technical clinician note — with clear provenance and last‑reviewed dates.
• Transparent labeling: mark AI‑generated text and make the limits and review status clear to readers (e.g., “AI‑generated draft — clinician reviewed”).

Training and culture (for clinicians, journalists, librarians)​

• AI literacy: teach frontline staff common failure modes (hallucination, sycophancy, fabricated citations) and simple verification checklists.
• Patient instructions: ask patients to bring screenshots of AI answers to appointments so clinicians can correct misinformation in real time.

---

Policy, regulation and publisher responses​

Governments, professional bodies and publishers are already stepping into this space:

• Editorial audits and public‑service broadcasters have demanded better provenance, publisher partnerships and restraint from vendors that collapse refusal behaviour in favor of answering everything. These audits have real policy weight because they shift public expectations and can inform regulation.
• Healthcare products that materially influence clinical decisions may meet local medical device regulations; conservative labeling and post‑market surveillance are prudent.

Regulators will focus on transparency, safety testing and demonstrated clinician oversight for any product that claims to provide medical advice. Vendors and deploying organizations should prepare to document safety testing, publish evaluation protocols, and maintain change logs for model updates.

---

What Windows‑centric IT teams and enterprise managers must do now​

Many Windows organizations are integrating LLMs into Office, help desks and knowledge workflows. Practical short‑term steps:

• Audit AI settings in deployed software (Copilot in Windows and Office). Disable unconstrained web retrieval for general users on medical or regulated topics.
• Create role‑based AI policies: allow expanded AI modes only for credentialed staff with explicit clinician review workflows.
• Instrument monitoring and logging: log prompts/responses and model versions; snapshot outputs before dissemination.
• Require verification gates: any AI‑generated patient education must include a named clinician reviewer and a timestamped “last reviewed” tag.
• Run adversarial tests against your integrated assistants to detect sycophancy and jailbreak vulnerabilities before releasing features to staff or patients.

These controls protect patients, reduce legal exposure, and maintain organizational credibility when AI‑generated content is used operationally.

---

Strengths, limitations and where evidence needs caution​

LLMs are powerful tools: they speed drafting, make dense evidence accessible, and can scale patient education when paired with clinician oversight. However:

• Strength: LLMs translate technical guidance into plain language and can standardize routine education at scale.
• Limitation: the architecture and vendor incentives make hallucination and sycophancy persistent failure modes; single‑study snapshots can vary over time as models update.
• Verification gap: some numerical claims in media summaries (e.g., exact refusal rates or percentages of model compliance) come from press reporting of academic work and should be confirmed against primary manuscripts before being treated as definitive. Where primary manuscripts are not publicly accessible, treat secondary numbers as provisional.

Flagging the unverifiable: specific press summaries that quote precise percentages (for example, exact model compliance or refusal rates in a red‑team experiment) should be verified against the primary study. When the primary source is behind paywalls or embargoes, public reporting remains useful but numerically uncertain. Readers and implementers should treat such numbers as indicative rather than absolute until the original methods and data are inspected.

---

Concrete checklists for three audiences​

For journalists and editors​

• Verify every AI‑produced citation by opening the cited paper; do not publish on the basis of an assistant’s bibliography alone.
• Use AI as a drafting tool, not as the final reporter: human fact‑check and source tracing remain essential.
• Label AI‑assistance and keep an immutable log of the prompt/answer used for any published text.

For clinicians and patient educators​

• Treat AI outputs as drafts for clinician review; never hand an unreviewed AI answer to a patient.
• Require “last reviewed” timestamps and a named clinician sign‑off on all AI‑assisted educational material.

For IT/security teams​

• Constrain retrieval indices for clinical tasks, enable safe modes, snapshot AI outputs, and run routine red‑team tests for sycophancy and jailbreak vectors.

---

Final assessment — a cautious path forward​

Artificial intelligence is a pragmatic force multiplier for reporting, research synthesis and patient education — but the technology is not yet mature enough to be trusted as an independent author of medical guidance. The evidence from editorial audits, bibliographic verification studies and red‑team experiments converges on a single pragmatic truth: AI can produce fluent, convincing falsehoods that travel faster precisely because they look credible.
The solution is not to ban AI tools, but to govern their use sensibly:

• Build provenance first: require retrieval fidelity, snapshotting and explicit source display.
• Maintain human oversight: clinician sign‑off and editorial verification must be non‑optional when outputs affect health decisions.
• Invest in adversarial testing and continuous monitoring: threat models evolve; defenses must, too.

For journalists, scientists, IT managers and clinicians, the assignment is clear: harness AI’s productivity gains while building robust checks that prevent polished misinformation from reaching patients or the public. When those controls are in place, LLMs can be useful assistants; without them, they are too frequently effective amplifiers of risk.

---

The evidence base continues to grow — audits, peer‑reviewed studies and cross‑sector reporting should be consulted regularly — but current independent evaluations offer an unambiguous, operational conclusion: use AI for health information only with layered, documented safeguards and human verification.

---

Source: CEDMO Artificial intelligence, LLM, GTP and health misinformation - CEDMO