This weekend stability.ai started closed beta testing of their Stable Diffusion text-to-image diffusion model. Like Dall-E 2, Craiyon (Dalle Mini), MidJourney, Latent Diffusion, and Disco Diffusion, Stable Diffusion takes text as a prompt and generates images.
After tinkering around with Stable Diffusion for a bit, I recalled seeing a couple prompts of The Great Wave Off Kanagawa by Vincent van Gogh from Imagen and MidJourney and wondered how Stable Diffusion would do at generating famous paintings by alternate artists. So I decided to give it a try and post some of the best results.
# An Aside on Stable Diffusion
From my limited testing over the past day, Stable Diffusion on the default settings is very fastA single 512x512 pixel image is generated in five seconds using the default settings. I am not sure what GPU stability.ai is serving Stable Diffusion from. and usually quite coherent for objects and facesIt might be too good at generating faces. at the default image sizeDoubling the image size in any dimension often results in repeated objects across that dimension., but requires a bit more handholding for an artistic output then MidJourney.
As I understand it, the plan is to gradually increase access to Stable Diffusion, eventually finishing with an open-source release with full code and model weights. stability.ai will also write a paper fully explaining the model architecture and training process.
Some other interesting Stable Diffusion items of note. The twitter tag #stablediffusion has a lot of examples from alpha and beta testers. There are also explorations of how Stable Diffusion interprets inputs, such as artist studies, sampler studies, and examples of changing cfg_scale here and here.
# Reimagining Famous Paintings
This post is divided into three sections.
- Simple Prompts: One painting with one artist
- Multi-Artist Prompts: One painting with multiple artists
- Modified Prompts: A few attempts at fixing some generations with modified prompts
The figure description is the full prompt needed to regenerate the output, including the seed.
Results in each section are presented in alphabetical order.
# Simple Prompts
The following Stable Diffusion generations were all generated using the default settings using the following prompt format:
<painting title> by <different artist> -n 9 -g
The figure description is the full prompt needed to regenerate the output, including the seed.
# A Solitary Temple Amid Clearing Peaks by Artemisia Gentileschi
# A Sunday Afternoon on the Island of La Grande Jatte by Margaret Macdonald Mackintosh
# Christ Pantocrator by Alma Thomas
# Eugene Manet on the Isle of Wight by Frederic Edwin Church
# Girl with a Pearl Earring by Kawanabe Kyōsai
# Inventions of the Monsters by Amrita Sher-Gil
# Nighthawks by Leonardo da Vinci
# Oath of the Horatii by Raja Ravi Varma
# Shoki Riding a Tiger and Attacking a Group of Demons by Rembrandt
# The Arnolfini Wedding by Frida Kahlo
# The Creation of Adam by Banksy
# The Garden of Earthly Delights by Max Ernst
# The Great Wave off Kanagawa by Vincent van Gogh
# The Hay Wain by Jacob Lawrence
# The Persistence of Memory by Sandro Botticelli
# The Raft of the Medusa by Pablo Picasso
# The Scream by Jean-Paul Riopelle
# The Storm on the Sea of Galilee by John Constable
# The Two Fridas by Joshua Johnson
# Multi-Artist Prompts
The following Stable Diffusion generations were all generated using the default settings using the following prompt format:
<painting title> by <artist one> and <artist two> -n 9 -g
The figure description is the full prompt needed to regenerate the output, including the seed.
# Clara the Rhinoceros by Fra Angelico and Elisabeth Sonrel
# Harvesters Resting by Umberto Boccioni and Norman Rockwell
# Judith and her Maidservant by Lady Frieda Harris and Pierre-Auguste Renoir
# Landscape with Ascanius Shooting the Stag of Sylvia by Salvador Dali and Jean-Francois Millet
# Madame de Pompadour by Umberto Boccioni and Hans Hofmann
# Nighthawks by Frida Kahlo and Artemisia Gentileschi
# Saint Jerome Writing by Stanley Pinker and Nicholas of Verdun
# San Giorgio Maggiore at Dusk by Ben Nicholson and George Washington Lambert
# The Death of Julius Caesar by Albrecht Durer and Kees van Dongen
# The Sonnet by Hans Hofmann and Edward Hopper
# The Starry Night by Max Ernst and Amrita Sher-Gil
# The Temptation of St. Anthony by Ravi Varma and Frederic Edwin Church
# Modified Prompts
# Nighthawks by Leonardo da Vinci
# Nighthawks by Leonardo da Vinci
# The Creation of Adam by Banksy
# The Starry Night by Sheikh Hamdullah and Mir Ali Tabrizi
# Bonus: Overpowered Prompts
I also have a folder of prompts where the artist or painting title was overpowered by the other, leading to a result that was mostly the overpowering part of the prompt. Those can be seen here.