AI synthetic media claims: a verification checklist for tools and viral media

What to take away

Common synthetic media vendor claims and what to verify

A local connection, such as an office, a newsroom demo, or a local customer, does not show that a synthetic-media detector works. Treat a vendor's performance pitch as a claim to test, not as evidence.

Ask whether the evidence covers the media you need to check. A result on clean, familiar clips does not establish performance on a phone video shot at night, a livestream, or another difficult condition.

Public benchmarks show how large the gap can be. In Meta's Deepfake Detection Challenge, reported in 2020, the best model scored about 82.6 percent average precision on the public dataset but about 65.2 percent on a black-box set of 10,000 videos the entrants had never seen. Scores on familiar data overstate performance on new material.

Some vendors sell both generation and detection. That is a conflict worth naming. A company that builds synthetic voices and also sells a detector can tune the detector to its own generator, which inflates scores on its own outputs and says little about rivals.

Others claim real-time detection on live video. In practice, live detection trades accuracy for speed. Ask what frame rate the claim assumes, what hardware it needs, and what happens when the stream drops.

When a vendor says its model is "state of the art," ask which benchmark, which version, and who ran it. A claim that cannot be tied to a dated, public leaderboard is a claim you cannot check.

Reading a model card and its stated limits

A model card is the vendor's own description of what a model does, what data trained it, and where it fails. Published model cards are the minimum disclosure you should expect. If a company will not publish one, that silence is itself a finding.

Read the intended use section first. A detector built for social media video may perform poorly on courtroom exhibits, broadcast footage, or audio-only clips. Mismatch between intended use and your use case is the most common reason a tool disappoints.

Then read the out-of-scope section. Good cards name the manipulations the model was not trained on: face swaps, lip sync, voice cloning, full-body puppetry, or scene reenactment. If the card lists none, the vendor either tested narrowly or is hiding gaps.

Check the training data description. Was it scraped from the web, licensed, or generated in-house? Web-scraped data skews toward certain platforms, languages, and lighting conditions. That skew shows up as higher error rates on underrepresented groups.

Look for performance breakdowns by subgroup. A single aggregate accuracy number hides variation. A detector that scores well overall can fail badly on darker skin tones, women, or non-English speakers. If the card reports only one number, ask for the rest. Finally, check the version and date, since models are updated and an old card may describe a model no longer shipped.

How to check NIST evaluation claims

NIST evaluations are task-specific. For example, NIST's face recognition evaluations (the Face Recognition Vendor Test, now run as FRTE) test face-recognition algorithms; their results do not establish how a synthetic-media detector performs. A NIST logo or general reference is not a detector test.

Three NIST efforts are relevant to synthetic media. The NIST GenAI evaluation program pits generators against discriminators, which are detectors, with tracks for text and image. The Open Media Forensics Challenge, successor to the Media Forensics Challenge that supported DARPA's MediFor program, tests detection and localization of manipulations in images and video. NIST AI 100-4, published in November 2024, is a technical report on provenance, labeling and detection, not a test.

If a vendor cites a NIST result, use a repeatable process to check whether it supports the claim.

NIST evaluations are not a pass or fail grade. They are a common yardstick. A model that ranks mid-pack may still suit your workflow if its errors are the kind you can catch elsewhere.

Be careful with benchmark drift. A vendor may cite a result from an older submission while shipping a newer, untested model. Ask for the submission identifier and the date, then verify that it matches the product you are evaluating.

The NIST AI Risk Management Framework as a buyer's checklist

The NIST AI Risk Management Framework is a voluntary guide for managing risks across an AI system's life cycle. It organizes work into govern, map, measure, and manage functions. Use it as a buyer's checklist when a synthetic-media vendor claims its system is trustworthy.

Govern asks who is accountable. Does the vendor have a named owner for model risk? Is there a policy for handling customer reports of errors? Without governance, accuracy claims have no one behind them.

Map asks what the system is for and who it affects. A detector used in journalism affects subjects of coverage, who may be falsely flagged. Ask the vendor to map those impacts and say how they are mitigated.

Measure asks for evidence. This is where NIST evaluations, internal test sets, and third-party audits belong. Ask what metrics the vendor tracks in production, not just at launch.

Manage asks how risks are handled over time. What happens when the model fails? Is there a rollback plan, a human review step, or a disclosure requirement? A vendor with no answer here is selling a demo, not a product.

Provenance signals versus detection scores

Provenance signals are records that show where a piece of media came from: camera signatures, capture metadata, signed manifests, or edit history. The main open standard is C2PA, from the Coalition for Content Provenance and Authenticity, and its consumer-facing label is Content Credentials. They are claims about origin, and they can be verified.

To check Content Credentials, upload the original file to the Content Credentials verify tool and read the issuer, the tools used and the recorded edits. A valid credential confirms origin and integrity, not truth, and a re-saved copy can break the signature; see how political ad deepfakes meet C2PA checks for a worked case. For sensitive files, use a local tool such as the open-source c2patool instead of uploading.

Detection scores are probabilistic. A model outputs a number that suggests likelihood, not proof. A score of 0.92 is not a finding. It is a prompt to look closer, and it can be wrong in both directions.

When both are available, provenance usually carries more weight. A signed capture from a known device, with an unbroken edit history, is stronger evidence than a detector's opinion. Detection helps when provenance is missing, which is common on social platforms.

Provenance has limits too. Metadata can be stripped, forged, or absent by default. Many cameras do not sign images. So provenance is a signal, not a guarantee, and its absence proves nothing. Check provenance first; if it is missing or broken, run detection and weigh the detector's known failure modes. Vendors sometimes sell a "provenance" product that is really a classifier. If there is no signature or manifest, it is detection wearing a provenance label. The guide to checking an unfamiliar account and viral post covers the source side of that order.

Checking a viral piece of media itself

When the question is whether a specific clip or image is synthetic, work through the media before any detector.

See also how to verify suspected AI images, video, and audio.

Questions to ask a synthetic-media vendor before citing its tool

A vendor's answers, or refusals, tell you how much weight to give its tool in your reporting. Put the questions in writing and ask for documents, not assurances.

Ask for the company's SEC filings if it is public or has filed. Registration statements, annual reports, and risk disclosures on SEC.gov can contradict a sales narrative. A vendor that claims enterprise scale while its filings describe a pilot customer is a vendor whose claims need a caveat.

Finally, ask what the tool does not do. Vendors who can name their limits are easier to trust than those who claim full coverage. A clear limit is usable. A vague promise is not.

A worked example: checking one vendor claim

The following case is illustrative, not a real vendor. A startup says its detector is "99 percent accurate and validated by NIST." Apply the table. Accuracy: ask for the test set and conditions; the vendor says only that it was an internal set. Standards: ask for the NIST evaluation name and submission identifier; the vendor can name none.

Provenance and accountability: there is no manifest format and no named owner for errors. Run the detector on 50 clips you know the origin of, and record the misses. The verdict, in print, would be: the claim is unverified, the NIST statement could not be confirmed, and the tool's performance on newsroom material is unknown.

Common synthetic media reporting problems with vendor claims

Deadline pressure can make a vendor's number easy to publish and hard to check, and a local startup or demo in the story is not verification.

A second problem is treating a detector score as proof. A score is one input. Stories that lead with a percentage and skip provenance, context, and human review mislead readers about what is known.

A third problem is ignoring the base rate. If a detector has a one percent false positive rate and you run it on a million clips, you get 10,000 false flags. Reporters rarely mention this arithmetic, so readers assume a flag means a fake.

A fourth is failing to disclose the tool's limits. If the vendor's model card says it was not trained on audio, a story about a cloned voice should say so. A fifth is single-source reliance: one vendor, one expert, one screenshot.

A sixth is not checking the business story. Funding claims, customer counts, and partnerships can be verified through filings and public records. Before you repeat any vendor figure, apply the discipline you would use on a human source, which is the core lesson of common source evaluation mistakes. Name the tool and version, state the score and its known error rate, say what provenance showed, and tell readers what remains unknown.

Common questions