Voice-cloning companies hit for lack of safeguards against scammers

Companies creating generative AI (genAI) tools that can clone voices are not doing enough to prevent scammers from using them to defraud customers, according to a study from Consumer Reports.

The study found that four of six companies providing the services didn’t construct enough barriers to prevent individuals’ voices from being fraudulently cloned.

Consumer Reports tested voice cloning tools and found that four companies — ElevenLabs, Speechify, PlayHT, and Lovo — didn’t provide any way to ensure the user had a speaker’s consent to clone their voice. The software also placed no technical guardrails to prevent voice cloning.

The publication was able to easily create voice clones using publicly available audio with tools from the four companies. It only required marking a check box that the user had the legal right to create a voice clone.

Two companies, Descript and Resemble AI, had more safeguards in place by limiting uploads of pre-recorded audio. 

Descript required a user to read the consent statement, which was used as a basis to create the voice clone. Resemble AI required a person to do a voice recording in real time to create a high-quality clone; attempts to create a clone based on pre-recorded audio were of poor quality.

Still, the testers were able to bypass those safeguards, prompting the magazine to call for tighter standards. “We argue that the nascent voice-cloning industry should adopt norms and standards to mitigate the risk of fraud,” Consumer Reports said.

GenAI tools have made voice-cloning a possibility, and it is increasingly being used for fraud.

Scammers use the tools as a form of social engineering to cheat victims out of money or to spread misinformation. In many cases, they create realistic audio of a close relative or friend in trouble and deceive callers to fraudulently send money or to divulge sensitive information.

The technology is also being used to access bank accounts by bypassing voice identity verification. The FBI issued an alert in December about video- and voice-cloning schemes being used for financial fraud.

One UK-based bank, Starling, is recommending customers use phrases to verify the person on the line isn’t a clone. The bank also recommends that users protect their voice by limiting social media access.

Productivity tools from Synthesia and D-ID that create digital avatars from voice and video cloning are being used in enterprises for marketing and presentations. The tools can create realistic videos that combine the video and voice clones of individuals, which saves on the cost of hiring production teams.

Synthesia has an ethics and AI policy to prevent abuse. It prevents non-consensual cloning by putting biometric-based checks in place to ensure the person initiating the cloning request is the same as the person in the footage, said Alexandru Voica, a Synthesia spokesman.

The company also applies robust content moderation at the point of creation, before a video is generated, based on publicly available policies to prevent the creation of harmful content. 

“We work with independent red teaming organizations to ensure our approach is as robust as possible,” Voica said.

Source:: Computer World