This is an old revision of this page, as edited by BrocadeRiverPoems (talk | contribs) at 00:21, 15 November 2024 (Reverting to Talk Page Consensus. This subversion of consensus is clear WP:OWNERSHIPBEHAVIOR. This reversion made re-implements several unreliable sources, misrepresented sources, and subverts consensus based edits for no policy reason.). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.
Revision as of 00:21, 15 November 2024 by BrocadeRiverPoems (talk | contribs) (Reverting to Talk Page Consensus. This subversion of consensus is clear WP:OWNERSHIPBEHAVIOR. This reversion made re-implements several unreliable sources, misrepresented sources, and subverts consensus based edits for no policy reason.)(diff) ← Previous revision | Latest revision (diff) | Newer revision → (diff) Real-time text-to-speech tool using artificial intelligence
This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these messages)
|
Type of site | Artificial intelligence, speech synthesis, machine learning, deep learning |
---|---|
Available in | English |
Founder(s) | 15 |
URL | 15 |
Commercial | No |
Registration | None |
Launched | Initial release: March 12, 2020; 4 years ago (2020-03-12) Stable release: v24.2.1 |
Current status | Under maintenance |
Part of a series on |
Artificial intelligence |
---|
Major goals |
Approaches |
Applications |
Philosophy |
History |
Glossary |
15.ai was a freeware artificial intelligence web application, launched in 2020, that generated text-to-speech voices from fictional characters from various media sources. Created by a pseudonymous developer under the alias 15, the project used a combination of audio synthesis algorithms, speech synthesis deep neural networks, and sentiment analysis models to generate emotive character voices faster than real-time.
In early 2020, 15.ai appeared online as a proof of concept of the democratization of voice acting and dubbing. Its gratis nature, ease of use without user accounts, and improvements over existing text-to-speech implementations made it popular. Some critics and voice actors questioned the legality and ethicality of making such technology so readily accessible.
The site was credited as the impetus behind the popularization of AI voice cloning (also known as audio deepfakes) in content creation. It was embraced by Internet fandoms such as My Little Pony, Team Fortress 2, and SpongeBob SquarePants.
Several commercial alternatives appeared in the following years. In January 2022, the company Voiceverse NFT plagiarized 15.ai's work as part of their platform.
In September 2022, a year after its last stable release, 15.ai was taken offline. As of November 2024, the website was still offline, with the creator's most recent post being dated February 2023.
Features
Available characters included GLaDOS and Wheatley from Portal, characters from Team Fortress 2, Twilight Sparkle and other characters from My Little Pony: Friendship Is Magic, SpongeBob, Daria Morgendorffer and Jane Lane from Daria, the Tenth Doctor Who, HAL 9000 from 2001: A Space Odyssey, the Narrator from The Stanley Parable, Carl Brutananadilewski from Aqua Teen Hunger Force, Steven Universe, Dan from Dan Vs., and Sans from Undertale.
The deep learning model used by the application was nondeterministic: each time speech was generated from the same string of text, the intonation changed slightly. The application supported manually altering the emotion of a generated line using emotional contextualizers (a term coined by this project), a sentence or phrase conveying the emotion of the take that serves as a guide for the model during inference. Emotional contextualizers were representations of the emotional content of a sentence deduced via transfer learned emoji embeddings using DeepMoji, a deep neural network sentiment analysis algorithm developed by the MIT Media Lab in 2017. DeepMoji was trained on 1.2 billion emoji occurrences in Twitter data from 2013 to 2017, and outperformed human subjects in correctly identifying sarcasm in Tweets and other online modes of communication.
15.ai used a multi-speaker model—hundreds of voices were trained concurrently rather than sequentially, decreasing the required training time and enabling the model to learn and generalize shared emotional context, even for voices with no exposure to that context. Consequently, the characters in the application were powered by a single trained model, as opposed to multiple single-speaker models. The lexicon used by 15.ai was scraped from a variety of Internet sources, including Oxford Dictionaries, Wiktionary, the CMU Pronouncing Dictionary, 4chan, Reddit, and Twitter. Pronunciations of unfamiliar words were automatically deduced using phonological rules learned by the deep learning model.
The application supported a simplified phonetic transcription known as ARPABET, to correct mispronunciations and account for heteronyms—words that are spelled the same but are pronounced differently (such as the word read, which can be pronounced as either /ˈrɛd/ or /ˈriːd/ depending on its tense). It followed the CMU Pronouncing Dictionary's ARPABET conventions.
Background
Speech synthesis
Main article: Deep learning speech synthesis See also: Audio deepfakeIn 2016, with the proposal of DeepMind's WaveNet, deep-learning-based models for speech synthesis began to gain popularity as a method of modeling waveforms and generating high-fidelity human-like speech. Tacotron2, a neural network architecture for speech synthesis developed by Google AI, was published in 2018 and required tens of hours of audio data to produce intelligible speech; when trained on 2 hours of speech, the model was able to produce intelligible speech with mediocre quality, and when trained on 36 minutes of speech, the model was unable to produce intelligible speech.
For years, reducing the amount of data required to train a realistic high-quality text-to-speech model has been a primary goal of scientific researchers in the field of deep learning speech synthesis. The developer of 15.ai claims that as little as 15 seconds of data is sufficient to clone a voice up to human standards, a significant reduction in the amount of data required.
Copyrighted material in deep learning
Main article: Authors Guild, Inc. v. Google, Inc.A landmark case between Google and the Authors Guild in 2013 ruled that Google Books—a service that searches the full text of printed copyrighted books—was transformative, thus meeting all requirements for fair use. This case set an important legal precedent for the field of deep learning and artificial intelligence: using copyrighted material to train a discriminative model or a non-commercial generative model was deemed legal. The legality of commercial generative models trained using copyrighted material is still under debate; due to the black-box nature of machine learning models, any allegations of copyright infringement via direct competition would be difficult to prove.
Development
15.ai was designed and created by an anonymous research scientist known by the alias 15.
The algorithm used by the project was dubbed DeepThroat. The developer said the project and algorithm were conceived as part of MIT's Undergraduate Research Opportunities Program, and had been in development for years before the first release of the application.
The developer also worked closely with the Pony Preservation Project from /mlp/, the My Little Pony board of 4chan. This project was a "collaborative effort by /mlp/ to build and curate pony datasets" with the aim of creating applications in artificial intelligence. The Friendship Is Magic voices on 15.ai were trained on a large dataset crowdsourced by the project: audio and dialogue from the show and related media—including all nine seasons of Friendship Is Magic, the 2017 movie, spinoffs, leaks, and various other content voiced by the same voice actors—were parsed, hand-transcribed, and processed to remove background noise.
Reception
15.ai met with a largely positive reception. Liana Ruppert of Game Informer described it "simplistically brilliant" and José Villalobos of LaPS4 wrote that it "works as easy as it looks." Users praised the ability to easily create audio of popular characters that sound believable to those unaware they had been synthesized. Zack Zwiezen of Kotaku reported that " girlfriend was convinced it was a new voice line from GLaDOS' voice actor, Ellen McLain".
Impact
Fandom content creation
15.ai was frequently used for content creation in various fandoms, including the My Little Pony: Friendship Is Magic fandom, the Team Fortress 2 fandom, the Portal fandom, and the SpongeBob SquarePants fandom, with numerous videos and projects containing speech from 15.ai having gone viral.
The My Little Pony: Friendship Is Magic fandom saw a resurgence in video and musical content creation as a result, inspiring a new genre of fan-created content assisted by artificial intelligence. Some fanfictions weren adapted into fully voiced "episodes": The Tax Breaks is a 17-minute long animated video rendition of a fan-written story published in 2014 that uses voices generated from 15.ai with sound effects and audio editing, emulating the episodic style of the early seasons of Friendship Is Magic.
Viral videos from the Team Fortress 2 fandom featuring voices from 15.ai include Spy is a Furry (which gained over 3 million views on YouTube across multiple videos) and The RED Bread Bank, both of which inspired Source Filmmaker animated video renditions. Other fandoms used voices from 15.ai to produce viral videos. As of July 2022, the viral video Among Us Struggles (with voices from Friendship Is Magic) had over 5.5 million views on YouTube; YouTubers, TikTokers, and Twitch streamers also used 15.ai for their videos, such as FitMC's video on the history of 2b2t—one of the oldest running Minecraft servers—and datpon3's TikTok video featuring the main characters of Friendship Is Magic, which have 1.4 million and 510 thousand views, respectively.
Some users created AI virtual assistants using 15.ai and external voice control software. One user on Twitter created a personal desktop assistant inspired by GLaDOS using 15.ai-generated dialogue in tandem with voice control system VoiceAttack.
Troy Baker / Voiceverse NFT plagiarism scandal
See also: Non-fungible token § Plagiarism and fraud
Troy Baker @TroyBakerVA I’m partnering with @VoiceverseNFT to explore ways where together we might bring new tools to new creators to make new things, and allow everyone a chance to own & invest in the IP’s they create. We all have a story to tell. You can hate. Or you can create. What'll it be?
January 14, 2022
In December 2021, the developer of 15.ai posted on Twitter that they had no interest in incorporating non-fungible tokens (NFTs) into their work.
On January 14, 2022, it was discovered that Voiceverse NFT, a company that video game and anime dub voice actor Troy Baker announced his partnership with, had plagiarized voice lines generated from 15.ai as part of their marketing campaign. Log files showed that Voiceverse had generated audio of Twilight Sparkle and Rainbow Dash from the show My Little Pony: Friendship Is Magic using 15.ai, pitched them up to make them sound unrecognizable from the original voices, and appropriated them without credit, to falsely market their own platform—in violation of 15.ai's terms of service.
15 @fifteenai I've been informed that the aforementioned NFT vocal synthesis is actively attempting to appropriate my work for their own benefit. After digging through the log files, I have evidence that some of the voices that they are taking credit for were indeed generated from my own site.
January 14, 2022
Voiceverse Origins @VoiceverseNFT Hey @fifteenai we are extremely sorry about this. The voice was indeed taken from your platform, which our marketing team used without giving proper credit. Chubbiverse team has no knowledge of this. We will make sure this never happens again.
January 14, 2022
15 @fifteenai Go fuck yourself.
January 14, 2022
A week prior to the announcement of the partnership with Baker, Voiceverse made a (now-deleted) Twitter post directly responding to a (now-deleted) video posted by Chubbiverse—an NFT platform with which Voiceverse had partnered—showcasing an AI-generated voice and claimed that it was generated using Voiceverse's platform, remarking "I wonder who created the voice for this? ;)" A few hours after news of the partnership broke, the developer of 15.ai—having been alerted by another Twitter user asking for his opinion on the partnership, to which he speculated that it "sounds like a scam"—posted screenshots of log files that proved that a user of the website (with their IP address redacted) had submitted inputs of the exact words spoken by the AI voice in the video posted by Chubbiverse, and subsequently responded to Voiceverse's claim directly, tweeting "Certainly not you :)".
Following the tweet, Voiceverse admitted to plagiarizing voices from 15.ai as their own platform, claiming that their marketing team had used the project without giving proper credit and that the "Chubbiverse team no knowledge of this." In response to the admission, 15 tweeted "Go fuck yourself." The final tweet went viral, accruing over 75,000 total likes and 13,000 total retweets across multiple reposts.
The initial partnership between Baker and Voiceverse was met with severe backlash and universally negative reception. Critics highlighted the environmental impact of and potential for exit scams associated with NFT sales. Commentators also pointed out the irony in Baker's initial Tweet announcing the partnership, which ended with "You can hate. Or you can create. What'll it be?", hours before the public revelation that the company in question had resorted to theft instead of creating their own product. Baker responded that he appreciated people sharing their thoughts and their responses were "giving a lot to think about," and asked fans on social media to forgive him. Two weeks later, Baker discontinued his partnership with Voiceverse.
Reactions from voice actors
Some voice actors have publicly decried the use of voice cloning technology. Cited reasons include concerns about impersonation and fraud, unauthorized use of an actor's voice in pornography, and the potential of AI being used to make voice actors obsolete.
See also
- Audio deepfake
- Character.ai
- DALL-E
- Deepfake
- Midjourney
- NovelAI
- Stable Diffusion
- Synthetic media
- WaveNet
Notes
- Translated from original quote written in Spanish: "La dirección es 15.AI y funciona tan fácil como parece."
References
- Notes
- ^ Zwiezen, Zack (January 18, 2021). "Website Lets You Make GLaDOS Say Whatever You Want". Kotaku. Archived from the original on January 17, 2021. Retrieved January 18, 2021.
- ^ Ruppert, Liana (January 18, 2021). "Make Portal's GLaDOS And Other Beloved Characters Say The Weirdest Things With This App". Game Informer. Archived from the original on January 18, 2021. Retrieved January 18, 2021.
- ^ Clayton, Natalie (January 19, 2021). "Make the cast of TF2 recite old memes with this AI text-to-speech tool". PC Gamer. Archived from the original on January 19, 2021. Retrieved January 19, 2021.
- Morton, Lauren (January 18, 2021). "Put words in game characters' mouths with this fascinating text to speech tool". Rock, Paper, Shotgun. Archived from the original on January 18, 2021. Retrieved January 18, 2021.
- Ng, Andrew (April 1, 2020). "Voice Cloning for the Masses". The Batch. Archived from the original on August 7, 2020. Retrieved April 5, 2020.
- ^ Lopez, Ule (January 16, 2022). "Troy Baker-backed NFT firm admits using voice lines taken from another service without permission". Wccftech. Archived from the original on January 16, 2022. Retrieved June 7, 2022.
- ^ Kurosawa, Yuki (January 19, 2021). "ゲームキャラ音声読み上げソフト「15.ai」公開中。『Undertale』や『Portal』のキャラに好きなセリフを言ってもらえる". AUTOMATON. Archived from the original on January 19, 2021. Retrieved January 19, 2021.
- ^ Yoshiyuki, Furushima (January 18, 2021). "『Portal』のGLaDOSや『UNDERTALE』のサンズがテキストを読み上げてくれる。文章に込められた感情まで再現することを目指すサービス「15.ai」が話題に". Denfaminicogamer. Archived from the original on January 18, 2021. Retrieved January 18, 2021.
- ^ Williams, Demi (January 18, 2022). "Voiceverse NFT admits to taking voice lines from non-commercial service". NME. Archived from the original on January 18, 2022. Retrieved January 18, 2022.
- ^ Wright, Steve (January 17, 2022). "Troy Baker-backed NFT company admits to using content without permission". Stevivor. Archived from the original on January 17, 2022. Retrieved January 17, 2022.
- ^ Henry, Joseph (January 18, 2022). "Troy Baker's Partner NFT Company Voiceverse Reportedly Steals Voice Lines From 15.ai". Tech Times. Archived from the original on January 26, 2022. Retrieved February 14, 2022.
- @fifteenai (February 23, 2023). "If all goes well, the next update should be the culmination of a year and a half of nonstop work put into a huge number of fixes and major improvements to the algorithm. Just give me a bit more time – it should be worth it" (Tweet) – via Twitter.
- ^ Villalobos, José (January 18, 2021). "Descubre 15.AI, un sitio web en el que podrás hacer que GlaDOS diga lo que quieras". LaPS4. Archived from the original on January 18, 2021. Retrieved January 18, 2021.
- Moto, Eugenio (January 20, 2021). "15.ai, el sitio que te permite usar voces de personajes populares para que digan lo que quieras". Yahoo! Finance. Archived from the original on March 8, 2022. Retrieved January 20, 2021.
- Felbo, Bjarke (2017). "Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm". Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pp. 1615–1625. arXiv:1708.00524. doi:10.18653/v1/D17-1169. S2CID 2493033.
- Corfield, Gareth (August 7, 2017). "A sarcasm detector bot? That sounds absolutely brilliant. Definitely". The Register. Archived from the original on June 2, 2022. Retrieved June 2, 2022.
- "An Algorithm Trained on Emoji Knows When You're Being Sarcastic on Twitter". MIT Technology Review. August 3, 2017. Archived from the original on June 2, 2022. Retrieved June 2, 2022.
- "Emojis help software spot emotion and sarcasm". BBC. August 7, 2017. Archived from the original on June 2, 2022. Retrieved June 2, 2022.
- Lowe, Josh (August 7, 2017). "Emoji-Filled Mean Tweets Help Scientists Create Sarcasm-Detecting Bot That Could Uncover Hate Speech". Newsweek. Archived from the original on June 2, 2022. Retrieved June 2, 2022.
- Valle, Rafael (2020). "Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens". arXiv:1910.11997 .
- Cooper, Erica (2020). "Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings". arXiv:1910.10838 .
- ^ van den Oord, Aäron; Li, Yazhe; Babuschkin, Igor (November 12, 2017). "High-fidelity speech synthesis with WaveNet". DeepMind. Archived from the original on June 18, 2022. Retrieved June 5, 2022.
- Hsu, Wei-Ning (2018). "Hierarchical Generative Modeling for Controllable Speech Synthesis". arXiv:1810.07217 .
- Habib, Raza (2019). "Semi-Supervised Generative Modeling for Controllable Speech Synthesis". arXiv:1910.01709 .
- "Audio samples from "Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis"". August 30, 2018. Archived from the original on November 11, 2020. Retrieved June 5, 2022.
- Shen, Jonathan; Pang, Ruoming; Weiss, Ron J.; Schuster, Mike; Jaitly, Navdeep; Yang, Zongheng; Chen, Zhifeng; Zhang, Yu; Wang, Yuxuan; Skerry-Ryan, RJ; Saurous, Rif A.; Agiomyrgiannakis, Yannis; Wu, Yonghui (2018). "Natural TTS Synthesis by Conditioning WaveNet on Mel-Spectrogram Predictions". arXiv:1712.05884 .
- Chung, Yu-An (2018). "Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis". arXiv:1808.10128 .
- Ren, Yi (2019). "Almost Unsupervised Text to Speech and Automatic Speech Recognition". arXiv:1905.06791 .
- ^ Phillips, Tom (January 17, 2022). "Troy Baker-backed NFT firm admits using voice lines taken from another service without permission". Eurogamer. Archived from the original on January 17, 2022. Retrieved January 17, 2022.
- - F.2d – (2d Cir, 2015). (temporary cites: 2015 U.S. App. LEXIS 17988; Slip opinion (October 16, 2015))
- "15.ai – About". 15.ai. February 20, 2022. Archived from the original on October 6, 2021. Retrieved February 20, 2022.
- Branwen, Gwern (March 6, 2020). ""15.ai", 15, Pony Preservation Project". Gwern.net. Gwern. Archived from the original on March 18, 2022. Retrieved June 17, 2022.
- Scotellaro, Shaun (March 14, 2020). "Neat "Pony Preservation Project" Using Neural Networks to Create Pony Voices". Equestria Daily. Archived from the original on June 23, 2021. Retrieved June 11, 2022.
- "Pony Preservation Project (Thread 108)". 4chan. Desuarchive. February 20, 2022. Retrieved February 20, 2022.
- Scotellaro, Shaun (May 15, 2022). "Full Simple Animated Episode – The Tax Breaks (Twilight)". Equestria Daily. Archived from the original on May 21, 2022. Retrieved May 28, 2022.
- The Terribly Taxing Tribulations of Twilight Sparkle. April 27, 2014. Archived from the original on June 30, 2022. Retrieved April 28, 2022.
{{cite book}}
:|website=
ignored (help) - Phillips, Tom (January 14, 2022). "Video game voice actor Troy Baker is now promoting NFTs". Eurogamer. Archived from the original on January 14, 2022. Retrieved January 14, 2022.
- McWhertor, Michael (January 14, 2022). "The Last of Us voice actor wants to sell 'voice NFTs,' drawing ire". Polygon. Archived from the original on January 14, 2022. Retrieved January 14, 2022.
- "Last Of Us Voice Actor Pisses Everyone Off With NFT Push". Kotaku. January 14, 2022. Archived from the original on January 14, 2022. Retrieved January 14, 2022.
- Purslow, Matt (January 14, 2022). "Troy Baker Is Working With NFTs, but Fans Are Unimpressed". IGN. Archived from the original on January 14, 2022. Retrieved January 14, 2022.
- Strickland, Derek (January 31, 2022). "Last of Us actor Troy Baker heeds fans, abandons NFT plans". Tweaktown. Archived from the original on January 31, 2022. Retrieved January 31, 2022.
- Peters, Jay (January 31, 2022). "The voice of Joel from The Last of Us steps away from NFT project after outcry". The Verge. Archived from the original on February 4, 2022. Retrieved February 4, 2022.
- Tweets
- @TroyBakerVA (January 14, 2022). "I'm partnering with @VoiceverseNFT to explore ways where together we might bring new tools to new creators to make new things, and allow everyone a chance to own & invest in the IP's they create. We all have a story to tell. You can hate. Or you can create. What'll it be?" (Tweet) – via Twitter.
- @fifteenai (December 12, 2021). "I have no interest in incorporating NFTs into any aspect of my work. Please stop asking" (Tweet) – via Twitter.
- @fifteenai (January 14, 2022). "I've been informed that the aforementioned NFT vocal synthesis is actively attempting to appropriate my work for their own benefit. After digging through the log files, I have evidence that some of the voices that they are taking credit for were indeed generated from my own site" (Tweet) – via Twitter.
- @VoiceverseNFT (January 14, 2022). "Hey @fifteenai we are extremely sorry about this. The voice was indeed taken from your platform, which our marketing team used without giving proper credit. Chubbiverse team has no knowledge of this. We will make sure this never happens again" (Tweet) – via Twitter.
- @fifteenai (January 14, 2022). "Go fuck yourself" (Tweet) – via Twitter.
- @VoiceverseNFT (January 7, 2022). "I wonder who created the voice for this? ;)" (Tweet). Archived from the original on January 7, 2022 – via Twitter.
- @fifteenai (January 14, 2022). "Sounds like a scam" (Tweet) – via Twitter.
- @fifteenai (January 14, 2022). "Give proper credit or remove this post" (Tweet) – via Twitter.
- @fifteenai (January 14, 2022). "Certainly not you :)" (Tweet) – via Twitter.
- @fifteenai (January 14, 2022). "Go fuck yourself" (Tweet) – via Twitter.
- @yongyea (January 14, 2022). "The NFT scheme that Troy Baker is promoting is already finding itself in trouble after stealing and profiting off of somebody else's work. Who could've seen this coming" (Tweet) – via Twitter.
- @BronyStruggle (January 15, 2022). "actual" (Tweet) – via Twitter.
- YouTube (referenced for view counts and usage of 15.ai only)
- "SPY IS A FURRY". YouTube. January 17, 2021. Archived from the original on June 13, 2022. Retrieved June 14, 2022.
- "Spy is a Furry Animated". YouTube. Archived from the original on June 14, 2022. Retrieved June 14, 2022.
- "[SFM] – Spy's Confession – [TF2 15.ai]". YouTube. January 15, 2021. Archived from the original on June 30, 2022. Retrieved June 14, 2022.
- "Among Us Struggles". YouTube. September 21, 2020. Retrieved July 15, 2022.
- "The UPDATED 2b2t Timeline (2010–2020)". YouTube. March 14, 2020. Archived from the original on June 1, 2022. Retrieved June 14, 2022.
- TikTok
- "She said " 👹 "". TikTok. Retrieved July 15, 2022.
External links
Differentiable computing | |
---|---|
General | |
Hardware | |
Software libraries | |
Speech synthesis | |||||
---|---|---|---|---|---|
Free software |
| ||||
Proprietary software |
| ||||
Machine | |||||
Applications | |||||
Protocols | |||||
Developers/ Researchers | |||||
Process |
- Speech synthesis
- Deep learning software applications
- Deepfakes
- Internet-related controversies
- Massachusetts Institute of Technology alumni
- My Little Pony: Friendship Is Magic
- My Little Pony fandom
- Internet properties established in 2020
- Web applications
- 2020 in Internet culture
- 2020s fads and trends
- Works involved in plagiarism controversies