Remixed Art History with Stable Diffusion

Famous Paintings by Different Artists

This weekend stability.ai started closed beta testing of their Stable Diffusion text-to-image diffusion model. Like Dall-E 2, Craiyon (Dalle Mini), MidJourney, Latent Diffusion, and Disco Diffusion, Stable Diffusion takes text as a prompt and generates images.

After tinkering around with Stable Diffusion for a bit, I recalled seeing a couple prompts of The Great Wave Off Kanagawa by Vincent van Gogh from Imagen and MidJourney and wondered how Stable Diffusion would do at generating famous paintings by alternate artists. So I decided to give it a try and post some of the best results.

# An Aside on Stable Diffusion

From my limited testing over the past day, Stable Diffusion on the default settings is very fastA single 512x512 pixel image is generated in five seconds using the default settings. I am not sure what GPU stability.ai is serving Stable Diffusion from. and usually quite coherent for objects and facesIt might be too good at generating faces. at the default image sizeDoubling the image size in any dimension often results in repeated objects across that dimension., but requires a bit more handholding for an artistic output then MidJourney.

As I understand it, the plan is to gradually increase access to Stable Diffusion, eventually finishing with an open-source release with full code and model weights. stability.ai will also write a paper fully explaining the model architecture and training process.

Some other interesting Stable Diffusion items of note. The twitter tag #stablediffusion has a lot of examples from alpha and beta testers. There are also explorations of how Stable Diffusion interprets inputs, such as artist studies, sampler studies, and examples of changing cfg_scale here and here.

# Reimagining Famous Paintings

This post is divided into three sections.

  1. Simple Prompts: One painting with one artist
  2. Multi-Artist Prompts: One painting with multiple artists
  3. Modified Prompts: A few attempts at fixing some generations with modified prompts

The figure description is the full prompt needed to regenerate the output, including the seed.

Results in each section are presented in alphabetical order.

# Simple Prompts

The following Stable Diffusion generations were all generated using the default settings using the following prompt format:

<painting title> by <different artist> -n 9 -g

The figure description is the full prompt needed to regenerate the output, including the seed.

# A Solitary Temple Amid Clearing Peaks by Artemisia Gentileschi

A Solitary Temple Amid Clearing Peaks by Artemisia Gentileschi -n 9 -g -S 1579483559

# A Sunday Afternoon on the Island of La Grande Jatte by Margaret Macdonald Mackintosh

A Sunday Afternoon on the Island of La Grande Jatte by Margaret Macdonald Mackintosh -n 9 -g -S 67983670

# Christ Pantocrator by Alma Thomas

Christ Pantocrator by Alma Thomas -n 9 -g -S 1055913541

# Eugene Manet on the Isle of Wight by Frederic Edwin Church

Eugene Manet on the Isle of Wight by Frederic Edwin Church -n 9 -g -S 4090516380

# Girl with a Pearl Earring by Kawanabe Kyōsai

Girl with a Pearl Earring by Kawanabe Kyosai -n 9 -g -S 1300926020

# Inventions of the Monsters by Amrita Sher-Gil

Inventions of the Monsters by Amrita Sher-Gil -n 9 -g -S 646912778

# Nighthawks by Leonardo da Vinci

Nighthawks by Leonardo da Vinci -n 9 -g -S 3849571118

# Oath of the Horatii by Raja Ravi Varma

Oath of the Horatii by Ravi Varma -n 9 -g -S 3953184590

# Shoki Riding a Tiger and Attacking a Group of Demons by Rembrandt

Shoki Riding a Tiger and Attacking a Group of Demons by Rembrandt -n 9 -g -S 676576614

# The Arnolfini Wedding by Frida Kahlo

The Arnolfini Marriage by Frida Kahlo -n 9 -g -S 2400544522

# The Creation of Adam by Banksy

The Creation of Adam by Banksy -n 9 -g -S 2597244766

# The Garden of Earthly Delights by Max Ernst

The Garden of Earthly Delights by Max Ernst -n 9 -g -S 3379957809

# The Great Wave off Kanagawa by Vincent van Gogh

The Great Wave off Kanagawa by Vincent van Gogh -n 9 -g -S 1271764727

# The Hay Wain by Jacob Lawrence

The Hay Wain by Jacob Lawrence -n 9 -g -S 1640004666

# The Persistence of Memory by Sandro Botticelli

The Persistence of Memory by Sandro Botticelli -n 9 -g -S 1611208953

# The Raft of the Medusa by Pablo Picasso

The Raft of the Medusa by Pablo Picasso -n 9 -g -S 75079074

# The Scream by Jean-Paul Riopelle

The Scream by Jean-Paul Riopelle -n 9 -g -S 871238693

# The Storm on the Sea of Galilee by John Constable

The Storm on the Sea of Galilee by John Constable -n 9 -g -S 2957723171

# The Two Fridas by Joshua Johnson

The Two Fridas by Joshua Johnson -n 9 -g -S 4206791884

# Multi-Artist Prompts

The following Stable Diffusion generations were all generated using the default settings using the following prompt format:

<painting title> by <artist one> and <artist two> -n 9 -g

The figure description is the full prompt needed to regenerate the output, including the seed.

# Clara the Rhinoceros by Fra Angelico and Elisabeth Sonrel

Clara the Rhinoceros by Fra Angelico and Elisabeth Sonrel -n 9 -g -S 1276148862

# Harvesters Resting by Umberto Boccioni and Norman Rockwell

Harvesters Resting by Umberto Boccioni and Norman Rockwell -n 9 -g -S 2314378473

# Judith and her Maidservant by Lady Frieda Harris and Pierre-Auguste Renoir

Judith and her Maidservant by Lady Frieda Harris and Pierre-Auguste Renoir -n 9 -g -S 4218746359

# Landscape with Ascanius Shooting the Stag of Sylvia by Salvador Dali and Jean-Francois Millet

Landscape with Ascanius Shooting the Stag of Sylvia by Salvador Dali and Jean-Francois Millet -n 9 -g -S 3600390664

# Madame de Pompadour by Umberto Boccioni and Hans Hofmann

Madame de Pompadour by Umberto Boccioni and Hans Hofmann -n 9 -g -S 3213840066

# Nighthawks by Frida Kahlo and Artemisia Gentileschi

Nighthawks by Frida Kahlo and Artemisia Gentileschi -n 9 -g -S 2493270844

# Saint Jerome Writing by Stanley Pinker and Nicholas of Verdun

Saint Jerome Writing by Stanley Pinker and Nicholas of Verdun -n 9 -g -S 3430314387

# San Giorgio Maggiore at Dusk by Ben Nicholson and George Washington Lambert

San Giorgio Maggiore at Dusk by Ben Nicholson and George Washington Lambert -n 9 -g -S 866517065

# The Death of Julius Caesar by Albrecht Durer and Kees van Dongen

The Death of Julius Caesar by by Albrecht Durer and Kees van Dongen -n 9 -g -S 313344296

# The Sonnet by Hans Hofmann and Edward Hopper

The Sonnet by Hans Hofmann and Edward Hopper -n 9 -g -S 2494379188

# The Starry Night by Max Ernst and Amrita Sher-Gil

The Starry Night by Max Ernst and Amrita Sher-Gil -n 9 -g -S 3892967122

# The Temptation of St. Anthony by Ravi Varma and Frederic Edwin Church

The Temptation of St. Anthony by Ravi Varma and Frederic Edwin Church -n 9 -g -S 3231619068

# Modified Prompts

# Nighthawks by Leonardo da Vinci

Nighthawks by Leonardo da Vinci, tempura on canvas -H 256 -n 9 -g -S 3849571118

# Nighthawks by Leonardo da Vinci

Nighthawks by Leonardo da Vinci, tempura on canvas, chiaroscuro, sfumato -n 9 -g -S 2574299687

# The Creation of Adam by Banksy

The Creation of Adam by Banksy -H 256 -n 9 -g -S 2597244766

# The Starry Night by Sheikh Hamdullah and Mir Ali Tabrizi

The Starry Night by Sheikh Hamdullah and Mir Ali Tabrizi, Calligraphy, Suleyman the Magnificent Period -n 9 -g -S 2157399242

# Bonus: Overpowered Prompts

I also have a folder of prompts where the artist or painting title was overpowered by the other, leading to a result that was mostly the overpowering part of the prompt. Those can be seen here.

Previous

In this post, I explain what Attention Pooling is and how it works. I experiment with Touvron et al’s Learned Aggregation on several small datasets...

Next

While working through Unit 3 of the Hugging Face Reinforcement Learning course, I was feeling impatient by how long it...