Misplaced Pages

DALL-E

Article snapshot taken from Wikipedia with creative commons attribution-sharealike license. Give it a read and then ask your questions in the chat. We can research this topic together.
(Redirected from DALL·E 3) Image-generating deep-learning model

DALL-E
Watermark present on DALL-E images
An image generated by DALL-E 2, from the prompt Teddy bears working on new AI research underwater with 1990s technology
Developer(s)OpenAI
Initial release5 January 2021; 3 years ago (2021-01-05)
Stable releaseDALL-E 3 / 10 August 2023; 16 months ago (2023-08-10)
TypeText-to-image model
Websitelabs.openai.com Edit this on Wikidata
Part of a series on
Artificial intelligence
Major goals
Approaches
Applications
Philosophy
History
Glossary

DALL-E, DALL-E 2, and DALL-E 3 (stylised DALL·E, and pronounced DOLL-E) are text-to-image models developed by OpenAI using deep learning methodologies to generate digital images from natural language descriptions known as prompts.

The first version of DALL-E was announced in January 2021. In the following year, its successor DALL-E 2 was released. DALL-E 3 was released natively into ChatGPT for ChatGPT Plus and ChatGPT Enterprise customers in October 2023, with availability via OpenAI's API and "Labs" platform provided in early November. Microsoft implemented the model in Bing's Image Creator tool and plans to implement it into their Designer app.

History and background

DALL-E was revealed by OpenAI in a blog post on 5 January 2021, and uses a version of GPT-3 modified to generate images.

On 6 April 2022, OpenAI announced DALL-E 2, a successor designed to generate more realistic images at higher resolutions that "can combine concepts, attributes, and styles". On 20 July 2022, DALL-E 2 entered into a beta phase with invitations sent to 1 million waitlisted individuals; users could generate a certain number of images for free every month and may purchase more. Access had previously been restricted to pre-selected users for a research preview due to concerns about ethics and safety. On 28 September 2022, DALL-E 2 was opened to everyone and the waitlist requirement was removed. In September 2023, OpenAI announced their latest image model, DALL-E 3, capable of understanding "significantly more nuance and detail" than previous iterations. In early November 2022, OpenAI released DALL-E 2 as an API, allowing developers to integrate the model into their own applications. Microsoft unveiled their implementation of DALL-E 2 in their Designer app and Image Creator tool included in Bing and Microsoft Edge. The API operates on a cost-per-image basis, with prices varying depending on image resolution. Volume discounts are available to companies working with OpenAI's enterprise team.

The software's name is a portmanteau of the names of animated robot Pixar character WALL-E and the Catalan surrealist artist Salvador Dalí.

In February 2024, OpenAI began adding watermarks to DALL-E generated images, containing metadata in the C2PA (Coalition for Content Provenance and Authenticity) standard promoted by the Content Authenticity Initiative.

Technology

The first generative pre-trained transformer (GPT) model was initially developed by OpenAI in 2018, using a Transformer architecture. The first iteration, GPT-1, was scaled up to produce GPT-2 in 2019; in 2020, it was scaled up again to produce GPT-3, with 175 billion parameters.

DALL-E

DALL-E has three components: a discrete VAE, an autoregressive decoder-only Transformer (12 billion parameters) similar to GPT-3, and a CLIP pair of image encoder and text encoder.

The discrete VAE can convert an image to a sequence of tokens, and conversely, convert a sequence of tokens back to an image. This is necessary as the Transformer does not directly process image data.

The input to the Transformer model is a sequence of tokenized image caption followed by tokenized image patches. The image caption is in English, tokenized by byte pair encoding (vocabulary size 16384), and can be up to 256 tokens long. Each image is a 256×256 RGB image, divided into 32×32 patches of 4×4 each. Each patch is then converted by a discrete variational autoencoder to a token (vocabulary size 8192).

DALL-E was developed and announced to the public in conjunction with CLIP (Contrastive Language-Image Pre-training). CLIP is a separate model based on contrastive learning that was trained on 400 million pairs of images with text captions scraped from the Internet. Its role is to "understand and rank" DALL-E's output by predicting which caption from a list of 32,768 captions randomly selected from the dataset (of which one was the correct answer) is most appropriate for an image.

A trained CLIP pair is used to filter a larger initial list of images generated by DALL-E to select the image that is closest to the text prompt.

DALL-E 2

DALL-E 2 uses 3.5 billion parameters, a smaller number than its predecessor. Instead of an autoregressive Transformer, DALL-E 2 uses a diffusion model conditioned on CLIP image embeddings, which, during inference, are generated from CLIP text embeddings by a prior model. This is the same architecture as that of Stable Diffusion, released a few months later.

Capabilities

DALL-E can generate imagery in multiple styles, including photorealistic imagery, paintings, and emoji. It can "manipulate and rearrange" objects in its images, and can correctly place design elements in novel compositions without explicit instruction. Thom Dunn writing for BoingBoing remarked that "For example, when asked to draw a daikon radish blowing its nose, sipping a latte, or riding a unicycle, DALL-E often draws the handkerchief, hands, and feet in plausible locations." DALL-E showed the ability to "fill in the blanks" to infer appropriate details without specific prompts, such as adding Christmas imagery to prompts commonly associated with the celebration, and appropriately placed shadows to images that did not mention them. Furthermore, DALL-E exhibits a broad understanding of visual and design trends.

DALL-E can produce images for a wide variety of arbitrary descriptions from various viewpoints with only rare failures. Mark Riedl, an associate professor at the Georgia Tech School of Interactive Computing, found that DALL-E could blend concepts (described as a key element of human creativity).

Its visual reasoning ability is sufficient to solve Raven's Matrices (visual tests often administered to humans to measure intelligence).

An image of accurate text generated by DALL-E 3 based on the text prompt "An illustration of an avocado sitting in a therapist's chair, saying 'I just feel so empty inside' with a pit-sized hole in its center. The therapist, a spoon, scribbles notes"

DALL-E 3 follows complex prompts with more accuracy and detail than its predecessors, and is able to generate more coherent and accurate text. DALL-E 3 is integrated into ChatGPT Plus.

Image modification

Two "variations" of Girl With a Pearl Earring generated with DALL-E 2

Given an existing image, DALL-E 2 can produce "variations" of the image as individual outputs based on the original, as well as edit the image to modify or expand upon it. DALL-E 2's "inpainting" and "outpainting" use context from an image to fill in missing areas using a medium consistent with the original, following a given prompt.

For example, this can be used to insert a new subject into an image, or expand an image beyond its original borders. According to OpenAI, "Outpainting takes into account the image’s existing visual elements — including shadows, reflections, and textures — to maintain the context of the original image."

Technical limitations

DALL-E 2's language understanding has limits. It is sometimes unable to distinguish "A yellow book and a red vase" from "A red book and a yellow vase" or "A panda making latte art" from "Latte art of a panda". It generates images of "an astronaut riding a horse" when presented with the prompt "a horse riding an astronaut". It also fails to generate the correct images in a variety of circumstances. Requesting more than three objects, negation, numbers, and connected sentences may result in mistakes, and object features may appear on the wrong object. Additional limitations include handling text — which, even with legible lettering, almost invariably results in dream-like gibberish — and its limited capacity to address scientific information, such as astronomy or medical imagery.

An attempt to generate Japanese text using the prompt "a person pointing at a tanuki, with a speech bubble that says 'これは狸です！'", which results in the text being rendered with nonsensical kanji and kana

Ethical concerns

DALL-E 2's reliance on public datasets influences its results and leads to algorithmic bias in some cases, such as generating higher numbers of men than women for requests that do not mention gender. DALL-E 2's training data was filtered to remove violent and sexual imagery, but this was found to increase bias in some cases such as reducing the frequency of women being generated. OpenAI hypothesize that this may be because women were more likely to be sexualized in training data which caused the filter to influence results. In September 2022, OpenAI confirmed to The Verge that DALL-E invisibly inserts phrases into user prompts to address bias in results; for instance, "black man" and "Asian woman" are inserted into prompts that do not specify gender or race.

A concern about DALL-E 2 and similar image generation models is that they could be used to propagate deepfakes and other forms of misinformation. As an attempt to mitigate this, the software rejects prompts involving public figures and uploads containing human faces. Prompts containing potentially objectionable content are blocked, and uploaded images are analyzed to detect offensive material. A disadvantage of prompt-based filtering is that it is easy to bypass using alternative phrases that result in a similar output. For example, the word "blood" is filtered, but "ketchup" and "red liquid" are not.

Another concern about DALL-E 2 and similar models is that they could cause technological unemployment for artists, photographers, and graphic designers due to their accuracy and popularity. DALL-E 3 is designed to block users from generating art in the style of currently-living artists.

In 2023 Microsoft pitched the United States Department of Defense to use DALL-E models to train battlefield management system. In January 2024 OpenAI removed its blanket ban on military and warfare use from its usage policies.

Reception

Images generated by DALL-E upon the prompt: "an illustration of a baby daikon radish in a tutu walking a dog"

Most coverage of DALL-E focuses on a small subset of "surreal" or "quirky" outputs. DALL-E's output for "an illustration of a baby daikon radish in a tutu walking a dog" was mentioned in pieces from Input, NBC, Nature, and other publications. Its output for "an armchair in the shape of an avocado" was also widely covered.

ExtremeTech stated "you can ask DALL-E for a picture of a phone or vacuum cleaner from a specified period of time, and it understands how those objects have changed". Engadget also noted its unusual capacity for "understanding how telephones and other objects change over time".

According to MIT Technology Review, one of OpenAI's objectives was to "give language models a better grasp of the everyday concepts that humans use to make sense of things".

Wall Street investors have had a positive reception of DALL-E 2, with some firms thinking it could represent a turning point for a future multi-trillion dollar industry. By mid-2019, OpenAI had already received over $1 billion in funding from Microsoft and Khosla Ventures, and in January 2023, following the launch of DALL-E 2 and ChatGPT, received an additional $10 billion in funding from Microsoft.

Japan's anime community has had a negative reaction to DALL-E 2 and similar models. Two arguments are typically presented by artists against the software. The first is that AI art is not art because it is not created by a human with intent. "The juxtaposition of AI-generated images with their own work is degrading and undermines the time and skill that goes into their art. AI-driven image generation tools have been heavily criticized by artists because they are trained on human-made art scraped from the web." The second is the trouble with copyright law and data text-to-image models are trained on. OpenAI has not released information about what dataset(s) were used to train DALL-E 2, inciting concern from some that the work of artists has been used for training without permission. Copyright laws surrounding these topics are inconclusive at the moment.

After integrating DALL-E 3 into Bing Chat and ChatGPT, Microsoft and OpenAI faced criticism for excessive content filtering, with critics saying DALL-E had been "lobotomized." The flagging of images generated by prompts such as "man breaks server rack with sledgehammer" was cited as evidence. Over the first days of its launch, filtering was reportedly increased to the point where images generated by some of Bing's own suggested prompts were being blocked. TechRadar argued that leaning too heavily on the side of caution could limit DALL-E's value as a creative tool.

Open-source implementations

Since OpenAI has not released source code for any of the three models, there have been several attempts to create open-source models offering similar capabilities. Released in 2022 on Hugging Face's Spaces platform, Craiyon (formerly DALL-E Mini until a name change was requested by OpenAI in June 2022) is an AI model based on the original DALL-E that was trained on unfiltered data from the Internet. It attracted substantial media attention in mid-2022, after its release due to its capacity for producing humorous imagery.

See also

References

  1. David, Emilia (20 September 2023). "OpenAI releases third version of DALL-E". The Verge. Archived from the original on 20 September 2023. Retrieved 21 September 2023.
  2. "OpenAI Platform". platform.openai.com. Archived from the original on 20 March 2023. Retrieved 10 November 2023.
  3. Niles, Raymond (10 November 2023) . "DALL-E 3 API". OpenAI help Center. Archived from the original on 10 November 2023. Retrieved 10 November 2023.
  4. Mehdi, Yusuf (21 September 2023). "Announcing Microsoft Copilot, your everyday AI companion". The Official Microsoft Blog. Archived from the original on 21 September 2023. Retrieved 21 September 2023.
  5. ^ Johnson, Khari (5 January 2021). "OpenAI debuts DALL-E for generating images from text". VentureBeat. Archived from the original on 5 January 2021. Retrieved 5 January 2021.
  6. "DALL·E 2". OpenAI. Archived from the original on 6 April 2022. Retrieved 6 July 2022.
  7. ^ "DALL·E Now Available in Beta". OpenAI. 20 July 2022. Archived from the original on 20 July 2022. Retrieved 20 July 2022.
  8. ^ Allyn, Bobby (20 July 2022). "Surreal or too real? Breathtaking AI tool DALL-E takes its images to a bigger stage". NPR. Archived from the original on 20 July 2022. Retrieved 20 July 2022.
  9. "DALL·E Waitlist". labs.openai.com. Archived from the original on 4 July 2022. Retrieved 6 July 2022.
  10. "From Trump Nevermind babies to deep fakes: DALL-E and the ethics of AI art". the Guardian. 18 June 2022. Archived from the original on 6 July 2022. Retrieved 6 July 2022.
  11. "DALL·E Now Available Without Waitlist". OpenAI. 28 September 2022. Archived from the original on 4 October 2022. Retrieved 5 October 2022.
  12. ^ "DALL·E 3". OpenAI. Archived from the original on 20 September 2023. Retrieved 21 September 2023.
  13. "DALL·E API Now Available in Public Beta". OpenAI. 3 November 2022. Archived from the original on 19 November 2022. Retrieved 19 November 2022.
  14. Wiggers, Kyle (3 November 2022). "Now anyone can build apps that use DALL-E 2 to generate images". TechCrunch. Archived from the original on 19 November 2022. Retrieved 19 November 2022.
  15. ^ Coldewey, Devin (5 January 2021). "OpenAI's DALL-E creates plausible images of literally anything you ask it to". Archived from the original on 6 January 2021. Retrieved 5 January 2021.
  16. Growcoot, Matt (8 February 2024). "AI Images Generated on DALL-E Now Contain the Content Authenticity Tag". PetaPixel. Retrieved 4 April 2024.
  17. Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (11 June 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). OpenAI. p. 12. Archived (PDF) from the original on 26 January 2021. Retrieved 23 January 2021.
  18. "GPT-1 to GPT-4: Each of OpenAI's GPT Models Explained and Compared". 11 April 2023. Archived from the original on 15 April 2023. Retrieved 29 April 2023.
  19. Radford, Alec; Wu, Jeffrey; Child, Rewon; et al. (14 February 2019). "Language models are unsupervised multitask learners" (PDF). cdn.openai.com. 1 (8). Archived (PDF) from the original on 6 February 2021. Retrieved 19 December 2020.
  20. Brown, Tom B.; Mann, Benjamin; Ryder, Nick; et al. (22 July 2020). "Language Models are Few-Shot Learners". arXiv:2005.14165 .
  21. Ramesh, Aditya; Pavlov, Mikhail; Goh, Gabriel; et al. (24 February 2021). "Zero-Shot Text-to-Image Generation". arXiv:2102.12092 .
  22. ^ Ramesh, Aditya; Dhariwal, Prafulla; Nichol, Alex; Chu, Casey; Chen, Mark (12 April 2022). "Hierarchical Text-Conditional Image Generation with CLIP Latents". arXiv:2204.06125 .
  23. ^ Heaven, Will Douglas (5 January 2021). "This avocado armchair could be the future of AI". MIT Technology Review. Archived from the original on 5 January 2021. Retrieved 5 January 2021.
  24. Radford, Alec; Kim, Jong Wook; Hallacy, Chris; et al. (1 July 2021). Learning Transferable Visual Models From Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning. PMLR. pp. 8748–8763.
  25. Dunn, Thom (10 February 2021). "This AI neural network transforms text captions into art, like a jellyfish Pikachu". BoingBoing. Archived from the original on 22 February 2021. Retrieved 2 March 2021.
  26. ^ Whitwam, Ryan (6 January 2021). "OpenAI's 'DALL-E' Generates Images From Text Descriptions". ExtremeTech. Archived from the original on 28 January 2021. Retrieved 2 March 2021.
  27. ^ Dent, Steve (6 January 2021). "OpenAI's DALL-E app generates images from just a description". Engadget. Archived from the original on 27 January 2021. Retrieved 2 March 2021.
  28. ^ Marcus, Gary; Davis, Ernest; Aaronson, Scott (2 May 2022). "A very preliminary analysis of DALL-E 2". arXiv:2204.13807 .
  29. ^ Shead, Sam (8 January 2021). "Why everyone is talking about an image generator released by an Elon Musk-backed A.I. lab". CNBC. Archived from the original on 16 July 2022. Retrieved 2 March 2021.
  30. ^ Wakefield, Jane (6 January 2021). "AI draws dog-walking baby radish in a tutu". British Broadcasting Corporation. Archived from the original on 2 March 2021. Retrieved 3 March 2021.
  31. Markowitz, Dale (10 January 2021). "Here's how OpenAI's magical DALL-E image generator works". TheNextWeb. Archived from the original on 23 February 2021. Retrieved 2 March 2021.
  32. "DALL·E: Creating Images from Text". OpenAI. 5 January 2021. Archived from the original on 27 March 2021. Retrieved 13 August 2022.
  33. Edwards, Benj (20 September 2023). "OpenAI's new AI image generator pushes the limits in detail and prompt fidelity". Ars Technica. Archived from the original on 21 September 2023. Retrieved 21 September 2023.
  34. Coldewey, Devin (6 April 2022). "New OpenAI tool draws anything, bigger and better than ever". TechCrunch. Archived from the original on 6 May 2023. Retrieved 26 November 2022.
  35. "DALL·E: Introducing Outpainting". OpenAI. 31 August 2022. Archived from the original on 26 November 2022. Retrieved 26 November 2022.
  36. Saharia, Chitwan; Chan, William; Saxena, Saurabh; et al. (23 May 2022). "Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding". arXiv:2205.11487 .
  37. Marcus, Gary (28 May 2022). "Horse rides astronaut". The Road to AI We Can Trust. Archived from the original on 19 June 2022. Retrieved 18 June 2022.
  38. ^ Strickland, Eliza (14 July 2022). "DALL-E 2's Failures Are the Most Interesting Thing About It". IEEE Spectrum. Archived from the original on 15 July 2022. Retrieved 16 August 2022.
  39. ^ "DALL·E 2 Pre-Training Mitigations". OpenAI. 28 June 2022. Archived from the original on 19 July 2022. Retrieved 18 July 2022.
  40. James Vincent (29 September 2022). "OpenAI's image generator DALL-E is available for anyone to use immediately". The Verge. Archived from the original on 29 September 2022. Retrieved 29 September 2022.
  41. Taylor, Josh (18 June 2022). "From Trump Nevermind babies to deep fakes: DALL-E and the ethics of AI art". The Guardian. Archived from the original on 6 July 2022. Retrieved 2 August 2022.
  42. Knight, Will (13 July 2022). "When AI Makes Art, Humans Supply the Creative Spark". Wired. Archived from the original on 2 August 2022. Retrieved 2 August 2022.
  43. Rose, Janus (24 June 2022). "DALL-E Is Now Generating Realistic Faces of Fake People". Vice. Archived from the original on 30 July 2022. Retrieved 2 August 2022.
  44. ^ OpenAI (19 June 2022). "DALL·E 2 Preview – Risks and Limitations". GitHub. Archived from the original on 2 August 2022. Retrieved 2 August 2022.
  45. Lane, Laura (1 July 2022). "DALL-E, Make Me Another Picasso, Please". The New Yorker. Archived from the original on 2 August 2022. Retrieved 2 August 2022.
  46. Goldman, Sharon (26 July 2022). "OpenAI: Will DALL-E 2 kill creative careers?". Archived from the original on 15 August 2022. Retrieved 16 August 2022.
  47. Blain, Loz (29 July 2022). "DALL-E 2: A dream tool and an existential threat to visual artists". Archived from the original on 17 August 2022. Retrieved 16 August 2022.
  48. Biddle, Sam (10 April 2024). "Microsoft Pitched OpenAI's DALL-E as Battlefield Tool for U.S. Military". The Intercept.
  49. Biddle, Sam (12 January 2024). "OpenAI Quietly Deletes Ban on Using ChatGPT for "Military and Warfare"". The Intercept.
  50. Kasana, Mehreen (7 January 2021). "This AI turns text into surreal, suggestion-driven art". Input. Archived from the original on 29 January 2021. Retrieved 2 March 2021.
  51. Ehrenkranz, Melanie (27 January 2021). "Here's DALL-E: An algorithm learned to draw anything you tell it". NBC News. Archived from the original on 20 February 2021. Retrieved 2 March 2021.
  52. Stove, Emma (5 February 2021). "Tardigrade circus and a tree of life — January's best science images". Nature. Archived from the original on 8 March 2021. Retrieved 2 March 2021.
  53. Knight, Will (26 January 2021). "This AI Could Go From 'Art' to Steering a Self-Driving Car". Wired. Archived from the original on 21 February 2021. Retrieved 2 March 2021.
  54. Metz, Rachel (2 February 2021). "A radish in a tutu walking a dog? This AI can draw it really well". CNN. Archived from the original on 16 July 2022. Retrieved 2 March 2021.
  55. Leswing, Kif (8 October 2022). "Why Silicon Valley is so excited about awkward drawings done by artificial intelligence". CNBC. Archived from the original on 29 July 2023. Retrieved 1 December 2022.
  56. Etherington, Darrell (22 July 2019). "Microsoft invests $1 billion in OpenAI in new multiyear partnership". TechCrunch. Archived from the original on 22 July 2019. Retrieved 21 September 2023.
  57. "OpenAI's first VC backer weighs in on generative A.I." Fortune. Archived from the original on 23 October 2023. Retrieved 21 September 2023.
  58. Metz, Cade; Weise, Karen (23 January 2023). "Microsoft to Invest $10 Billion in OpenAI, the Creator of ChatGPT". The New York Times. ISSN 0362-4331. Archived from the original on 21 September 2023. Retrieved 21 September 2023.
  59. "AI-generated art sparks furious backlash from Japan's anime community". Rest of World. 27 October 2022. Archived from the original on 31 December 2022. Retrieved 3 January 2023.
  60. Roose, Kevin (2 September 2022). "An A.I.-Generated Picture Won an Art Prize. Artists Aren't Happy". The New York Times. ISSN 0362-4331. Archived from the original on 31 May 2023. Retrieved 3 January 2023.
  61. Daws, Ryan (15 December 2022). "ArtStation backlash increases following AI art protest response". AI News. Archived from the original on 3 January 2023. Retrieved 3 January 2023.
  62. ^ Corden, Jez (8 October 2023). "Bing Dall-E 3 image creation was great for a few days, but now Microsoft has predictably lobotomized it". Windows Central. Archived from the original on 10 October 2023. Retrieved 11 October 2023.
  63. ^ Allan, Darren (9 October 2023). "Microsoft reins in Bing AI's Image Creator – and the results don't make much sense". TechRadar. Archived from the original on 10 October 2023. Retrieved 11 October 2023.
  64. Sahar Mor, Stripe (16 April 2022). "How DALL-E 2 could solve major computer vision challenges". VentureBeat. Archived from the original on 24 May 2022. Retrieved 15 June 2022.
  65. "jina-ai/dalle-flow". Jina AI. 17 June 2022. Archived from the original on 17 June 2022. Retrieved 17 June 2022.
  66. Carson, Erin (14 June 2022). "Everything to Know About Dall-E Mini, the Mind-Bending AI Art Creator". CNET. Archived from the original on 15 June 2022. Retrieved 15 June 2022.
  67. Schroeder, Audra (9 June 2022). "AI program DALL-E mini prompts some truly cursed images". Daily Dot. Archived from the original on 10 June 2022. Retrieved 15 June 2022.
  68. Diaz, Ana (15 June 2022). "People are using DALL-E mini to make meme abominations — like pug Pikachu". Polygon. Archived from the original on 15 June 2022. Retrieved 15 June 2022.

External links

OpenAI
Products
Foundation models
People
CEOs
Board of directors
Current
Former
Related
Generative AI
Concepts
Models
Text
Image
Video
Music
Companies
Category
Artificial intelligence
Concepts
Applications
Implementations
Audio–visual
Text
Decisional
People
Architectures
Categories: