Can you trust a San Francisco AI startup's synthetic media claims?

What to take away

• San Francisco AI synthetic media claims are marketing until you match them against published model cards, NIST AI benchmarks, and independent tests.
• A model card should state training data, intended use, and known failure modes; if it does not, treat the detector as unproven.
• NIST AI benchmarks and the NIST AI Risk Management Framework give you a structured way to compare detection tools and their risk controls.
• Provenance signals from cameras and editing software often carry more weight than a single detection score, especially for court or news use.
• Before citing a Bay Area detection tool, ask for its false positive rate, demographic breakdowns, update policy, and audit trail.
• Vendor accuracy claims that cannot be reproduced on your own sample are not evidence; they are a press release.

What a Bay Area synthetic media vendor actually claims

San Francisco's detection vendors cluster in SoMa and around South Park, near the trust and safety teams at X, Reddit, OpenAI, and Anthropic. Their buyers are platform policy staff and Bay Area newsrooms, not courts. That proximity is why a pitch leads with an accuracy figure, and why the figure lands in coverage.

Vendors lean on the region's reputation. Sand Hill Road money sits 30 miles south, Stanford and UC Berkeley feed the engineering ranks, and pilot deals sit with platforms a short drive from Market Street. That context is real. It is not evidence that the model works on a phone video shot at night.

Some vendors sell both generation and detection. That is a conflict worth naming. A company that builds synthetic voices and also sells a detector can tune the detector to its own generator, which inflates scores on its own outputs and says little about rivals.

Others claim real-time detection on live video. In practice, live detection trades accuracy for speed. Ask what frame rate the claim assumes, what hardware it needs, and what happens when the stream drops. A tool that works on a downloaded clip may fail on a livestream.

When a vendor says its model is "state of the art," ask which benchmark, which version, and who ran it. Self-reported rankings change monthly. A claim that cannot be tied to a dated, public leaderboard is a claim you cannot check.

Reading a model card and its stated limits

A model card is the vendor's own description of what a model does, what data trained it, and where it fails. Published model cards are the minimum disclosure you should expect. If a company will not publish one, that silence is itself a finding.

Read the intended use section first. A detector built for social media video may perform poorly on courtroom exhibits, broadcast footage, or audio-only clips. Mismatch between intended use and your use case is the most common reason a tool disappoints.

Then read the out-of-scope section. Good cards name the manipulations the model was not trained on: face swaps, lip sync, voice cloning, full-body puppetry, or scene reenactment. If the card lists none, the vendor either tested narrowly or is hiding gaps.

Check the training data description. Was it scraped from the web, licensed, or generated in-house? Web-scraped data skews toward certain platforms, languages, and lighting conditions. That skew shows up as higher error rates on underrepresented groups.

Look for performance breakdowns by subgroup. A single aggregate accuracy number hides variation. A detector that scores well overall can fail badly on darker skin tones, women, or non-English speakers. If the card reports only one number, ask for the rest.

Finally, check the version and date. Models are updated. A card from two years ago may describe a model the company no longer ships. Ask for the card that matches the version you are testing.

• Does the card state intended use and out-of-scope uses?
• Does it name training data sources and collection dates?
• Does it report accuracy by demographic subgroup?
• Does it list known failure modes and manipulation types not covered?
• Does it give a version number and a date?
• Does it explain how updates are communicated to customers?
• Does it include contact information for reporting errors?

Testing detectors against NIST AI benchmarks

The National Institute of Standards and Technology runs a continuing evaluation of face analysis software, including synthetic media detection. NIST AI benchmarks give you a neutral scorecard that vendors cannot edit. Start there rather than with a sales deck.

NIST's work on artificial intelligence covers standards, test methods, and research relevant to synthetic media and deepfake detection. The agency publishes reports and maintains evaluations that compare algorithms under controlled conditions. Use those results to see how a vendor's model performs against others on the same tasks.

To test a detector against NIST AI benchmarks, follow a repeatable process.

1. Identify the NIST evaluation that matches your use case, such as face recognition or presentation attack detection.
2. Download the vendor's submitted algorithm entry and its reported error rates.
3. Compare those rates with the evaluation's overall distribution, not just the top performer.
4. Run your own sample through the vendor's API and compare your observed error rate with the NIST figure.
5. Document the gap. A large gap means the vendor's conditions differ from yours.

NIST evaluations are not a pass or fail grade. They are a common yardstick. A model that ranks mid-pack on NIST may still be the right choice for your workflow if its errors are the kind you can catch elsewhere.

Be careful with benchmark drift. A vendor may cite a NIST result from an older submission while shipping a newer, untested model. Ask for the submission identifier and the date. Then verify that the identifier matches the product you are evaluating.

For background on how deepfakes are made and detected, the Wikipedia article on deepfake is a useful starting point, though it is not a primary source for vendor claims.

The NIST AI Risk Management Framework as a buyer's checklist

The NIST AI Risk Management Framework is a voluntary guide for managing risks across an AI system's life cycle. It organizes work into govern, map, measure, and manage functions. You can use it as a buyer's checklist when a San Francisco AI synthetic media vendor claims to be trustworthy.

Govern asks who is accountable. Does the vendor have a named owner for model risk? Is there a policy for handling customer reports of errors? Without governance, accuracy claims have no one behind them.

Map asks what the system is for and who it affects. A detector used in journalism affects subjects of coverage, who may be falsely flagged. Ask the vendor to map those impacts and say how they are mitigated.

Measure asks for evidence. This is where NIST AI benchmarks, internal test sets, and third-party audits belong. Ask what metrics the vendor tracks in production, not just at launch.

Manage asks how risks are handled over time. What happens when the model fails? Is there a rollback plan, a human review step, or a disclosure requirement? A vendor with no answer here is selling a demo, not a product.

Use the framework to structure your questions, then keep the answers in writing. A vendor that engages with the NIST AI Risk Management Framework seriously will have documents to share. One that waves at it will not.

Provenance signals versus detection scores

Provenance signals are records that show where a piece of media came from: camera signatures, capture metadata, signed manifests, or edit history. They are not guesses about whether something is fake. They are claims about origin, and they can be verified.

Detection scores are probabilistic. A model outputs a number that suggests likelihood, not proof. A score of 0.92 is not a finding. It is a prompt to look closer, and it can be wrong in both directions.

When both are available, provenance usually carries more weight. A signed capture from a known device, with an unbroken edit history, is stronger evidence than a detector's opinion. Detection helps when provenance is missing, which is common on social platforms.

Provenance has limits too. Metadata can be stripped, forged, or absent by default. Many cameras do not sign images. Many editors rewrite history. So provenance is a signal, not a guarantee, and its absence proves nothing.

The practical approach is to combine them. Check provenance first. If it is missing or broken, run detection. Then weigh the detector's known failure modes against what you can see yourself. The guide to verify a social media account walks through that order.

Vendors sometimes blur the two. A company may sell a "provenance" product that is really a classifier with a confidence score. Ask which it is. If there is no signature or manifest, it is detection wearing a provenance label.

Questions to ask a San Francisco AI startup before citing its tool

A vendor's answers, or refusals, tell you how much weight to give its tool in your reporting. Put the questions in writing and ask for documents, not assurances.

• What is the false positive rate on material like mine, not on your benchmark?
• Which NIST AI benchmarks has this exact model version been submitted to?
• Will you publish a model card for the version I am testing?
• How do you handle demographic variation in error rates?
• What provenance signals does your tool read, and what does it do when they are absent?
• Who is accountable when the tool is wrong, and what is the correction process?
• Can I reproduce your headline accuracy claim on my own sample?

Ask for the company's SEC filings if it is public or has filed. The company filing database at SEC.gov hosts registration statements, annual reports, and risk disclosures that can contradict a sales narrative. A vendor that claims enterprise scale while its filings describe a pilot customer is a vendor whose claims need a caveat.

Check whether the vendor or its leaders have been cited by fact-checking organizations that follow International Fact-Checking Network standards. The IFCN sets principles for verification work, including transparency about methods. A tool used by signatory organizations has at least been exposed to those standards.

Finally, ask what the tool does not do. Vendors who can name their limits are easier to trust than those who claim full coverage. A clear limit is usable. A vague promise is not.

Claim type	What to ask for	Strong evidence	Weak evidence
Accuracy	Test set composition, subgroup breakdowns	Reproducible results on your sample	A single percentage on a slide
Standards	NIST submission identifier and date	Matching public evaluation entry	A logo or a mention
Provenance	Signature or manifest format	Verifiable capture and edit chain	A confidence score relabeled
Accountability	Named owner, error process	Written policy and contact	Verbal assurance

Common synthetic media reporting problems with vendor claims

San Francisco has more AI coverage per square mile than any other American city. The Chronicle, Mission Local, The San Francisco Standard, and KQED compete on the same beat, and vendors schedule demo days in SoMa to feed it. Deadline pressure makes a vendor's number easy to publish and hard to check.

A second problem is treating a detector score as proof. A score is one input. Stories that lead with a percentage and skip provenance, context, and human review mislead readers about what is known.

A third problem is ignoring the base rate. If a detector has a one percent false positive rate and you run it on a million clips, you get thousands of false flags. Reporters rarely mention this arithmetic, so readers assume a flag means a fake.

A fourth problem is failing to disclose the tool's limits. If the vendor's model card says it was not trained on audio, a story about a cloned voice should say so. Omitting that context overstates the evidence.

A fifth problem is single-source reliance. One vendor's tool, one expert, one screenshot. Verification improves when provenance, detection, and human review are compared. The guide to the visual geolocation workflow shows what each method can and cannot prove.

A sixth problem is not checking the business story. Funding claims, customer counts, and partnerships can be verified through filings and public records. Reporters who skip that step repeat numbers that later collapse.

These failures overlap with broader social account identity signals that show up whenever a vendor's numbers are repeated without testing.

Avoiding these problems is mostly discipline. Name the tool and version. State the score and its known error rate. Say what provenance showed. Then tell readers what remains unknown. For a step-by-step process on handling suspected fakes, see how to apply social post verification.

One more habit helps. Before you repeat any vendor figure, run the same discipline you would apply to a human source, which is the core lesson of common source evaluation mistakes.

Common questions

Can I trust a detector's accuracy percentage? Only if you know the test set, the version, and the conditions. A percentage without those details is a marketing figure, not a measurement.

What if a vendor has no published model card? Treat the tool as unproven and say so in your reporting. The absence of a card is a material fact about the vendor's transparency.

Are NIST AI benchmarks a certification? No. They are comparative evaluations. A good result shows how a model performed under specific conditions, not that it is correct in every case.

Do provenance signals prove a video is real? No. They show origin and edit history, which can be forged or stripped. They are stronger than a detection score but still not absolute.

How do I check a startup's business claims? Look for SEC filings if the company is public or has filed. Compare the risk disclosures and customer descriptions with the sales narrative.

Which fact-checking standards apply to verification tools? Organizations that follow International Fact-Checking Network principles commit to transparent methods, which gives you a benchmark for judging a vendor's disclosure.