Gary Schildhorn received a call that no parent wants to receive. When Schildhorn picked up the phone, the voice of his panicked son told him that he had been in a car accident and was in jail. A second call, moments later, purportedly from a lawyer, gave Schildhorn instructions on how to pay the $9,000 bond. Schildhorn was preparing payment when he received a call from his real son, who was not, in fact, in jail. Schildhorn nearly fell victim to the growing trend of artificial intelligence (AI)-powered voice scams. AI-generated voices are a problem not only for fraud but also for the legal system. Indeed, accusations of AI-generated voice clones have now made their way into the courts, and the way the courts deal with audio recording evidence needs to catch up.
Under the current Federal Rules of Evidence, someone trying to introduce an audio recording of a voice can satisfy the authentication standard for admissibility merely by putting a witness on the stand who says they are familiar with the person’s voice and the recording sounds like them. Specifically, Rule 901 states that the following evidence “satisfies the requirement [for admissibility]: … An opinion identifying a person’s voice—whether heard firsthand or through mechanical or electronic transmission or recording—based on hearing the voice at any time under circumstances that connect it with the alleged speaker.” The rule presumes that this evidence will be “sufficient to support a finding that the item is what the proponent claims it is.”
In the age of artificial intelligence, this presumption is no longer tenable. The Evidence Rulemaking Committee should amend the rules to make the enumerated examples in Rule 901(b) permissive, not mandatory. The examples should illustrate circumstances that may satisfy the authentication requirement while still leaving judges discretion to exclude an item of evidence if there is other proof that it is a fake.
Realism of AI-Powered Voice Clones
Over the past few years, AI-powered voice synthesis and cloning has improved at an impressive clip, culminating this past year in dramatic breakthroughs. Perhaps most striking is the ability to convincingly clone a person’s voice from as
16 Comments
fhd2
Sounds like reasonable changes.
Generally speaking, I think evidence tampering is not a new problem, and even though it's easy in some cases, I don't think it's _that_ widespread. Just like it's possible to lie on the stand, but people usually think twice before they do it, because _if_ they are found to have lied, they're in trouble.
My main concern is rather that legit evidence can now easily be called into question. That seems to me like a much higher risk than fake evidence, considering the overall dynamics.
But ultimately: Humanity has coped without photo, audio or video evidence for most of its existence. I suppose it will cope again.
dghlsakjg
How is AI voice faking any different than any other type of faking? How is it different than a manipulated recording, or a recording where someone is imitating another?
It is just as easy to fake many paper documents, and we have accepted documents as evidence for centuries.
Photos can be faked, video can be edited or faked, witnesses lie or misremember.
Is this just about telling lawyers that unvetted audio recordings can be unreliable? Because that shouldn't be news.
Edit: this is a good faith question. I'm legitimately just curious. Splicing and editing have been around since recording was invented, I was legitimately curious why voice recordings would have been given extra evidential weight when manipulating recordings is a known possibility.
carra
Wait some more time and photos or even video recordings will be deemed just as dangerous. And then what? Even if there is real evidence, it will have to be discarded unless it can't be sufficiently validated. It will get very hard to prove anything.
breakingrules3
[dead]
nottorp
So… those talking head "influencers" who leave multiple hours of voice and video samples on social networking for anyone to download and clone are the most at risk for an attack like this?
treetalker
I question the wisdom of setting the judge up as a superjury / gatekeeper for this kind of situation. This seems like a reliability / weight of the evidence scenario, not a reliability / qualification of the witness scenario (as with an expert witness).
Why would the judge be better qualified to determine whether the voice was authentic, as opposed to the witness? And why should the judge effectively determine the witness's credibility or ability to discern, when that's what juries are for?
All that said, emulated voices do pose big problems for litigation.
exe34
On a related note, why oh why does Lloyds Bank insist on grabbing my voice for login every time I call them? I have to keep saying "no, fcuk off!" a dozen times until it gives up.
Lammy
In the future this stuff will get so good that the public will beg to be surveilled at all times because it will be the only way to prove what you didn't do. You will learn to love Total Information Awareness. Consent status: manufactured :)
tgv
I'll say it again, even though it is rather unpopular here: there has never been a need to develop these tools, nor one to make them easy to deploy, nor one to make them easy to use. Yet all this has happened, and now it may occur that someone is acquitted because AI generated media is so good, the evidence might be artificial. If that happens, and the suspect commits another crime, it's on the conscience of the people that contributed to this. You cannot create something and pretend its use has nothing to do with you.
The tools aren't perfect yet, so it's not too late to stop. Stop the ridiculous image and audio generation tools before it's too late. Nothing of value is lost when these models are made private again, and research is simply halted.
recursive
Here's my plan.
1. Cryptographically hash each piece of media when it's recorded.
2. Submit the hash to a "trusted" authority.
3. It will add a timestamp and sign the result.
4. Now, as long as you keep the original, without re-compressing, and you trust the authority, you have some evidence that the media existed at a timestamp. On or before.
This doesn't prove authenticity, but in many cases, establishing a timestamp would be enough. Forgeries probably wouldn't be created until later, after the shit hit the fan.
Or maybe this doesn't work at all.
tiahura
Why can't objecting party ask to voir dire the witness and find out if they can distinguish between AI and real recordings?
gortok
What shocks (and irritates!) me is that Charles Schwab keeps wanting me to set up voice ID. Why would I want to set up a voice ID for something that is now trivially spoofed?
belter
To avoid the scenarios like the one described in the court case, setup a family password or keyword. The bot will not know it.
TeeMassive
I think we're going to see C2PA (https://c2pa.org/) being mandatory for cellphones. At least when there's at least one implementation and that all digital cameras has an integrated TPM.
TriangleEdge
AI threatens all digital perceptions, not just voice. Images, videos, recordings, … I think soon enough proving things in court beyond a reasonable doubt when the evidence is digital media will be difficult/impossible.
PolieBotics
I've developed a novel approach to creating tamper-evident video via cryptographic feedback loops between projectors and cameras. The process works as follows:
1. A projector displays a challenge pattern (Perlin noise derived from of a hash)
2. A camera captures this projection
3. The system hashes the captured image concatenated with the previous hash and uses it to derive the next projection
4. This chain demonstrates true temporal sequentiality that's difficult to forge
By incorporating random noise derived from Byzantine Fault Tolerant networks and using these networks as timestamping servers, the proofs inherit the network's decentralization properties. ML then confirms that the feature distributions in projection-photograph pairs match expected patterns from the training dataset.
Demo video and GitHub repo available here:
https://www.reddit.com/r/PoliePals/comments/1j8qm2j/truth_be…