We all have seen how artificial intelligence (AI) has been used with photography to do things like fix photographs in Adobe Photoshop, apply fun filters to photographs on Instagram, or transform a photograph to embody the style of a famous painting. The last example, style transfer, uses an AI model called a Generative Adversarial Network (GAN) and can also be used to generate entirely new images based on training it with a collection of images. This is what blew my mind this week.
I am part of a global organization at Microsoft, Commercial Software Engineering (CSE), where we help partners and customers with their most complex software engineering challenges by coding with them. The team is made up of software engineers, data scientists, and program managers like me who all love to hack and build stuff. We just completed a week-long hackathon combining virtual and in-person teams where individuals proposed ideas that interested them to work on and explore – in order to build a team to build something in less than four days.
Meeting Paul
Paul Butler, a software engineer, recent Computer Science graduate, and CSE team member, proposed an idea to use a Style GAN 2-ADA to generate artwork. I had been learning about GANs and my colleague Kevin Ashley just published a book on creating art with AI – and that really peaked my interest. I had never met Paul before, but we quickly found a common thread and interest – as we both went to college in Arizona: he’s an ASU Sun Devil and I’m a UA Wildcat – and he’s super smart. Paul knew how to run the GAN using Python but he didn’t have a training image set to work from – and that’s where I was able to help.
30 Years of Artwork
Ever since I started college studying architecture, I have been keeping visual journals of my artwork, taking a journal and compact supplies with me to document my journey when I travel for work or leisure. Having done this consistently for thirty years, I have filled more than 20 journals with drawings, watercolors, collage, pop-up craft, stickers, circuitry, and photographs. When the pandemic started and I had some time where I couldn’t travel, I scanned and categorized the pages using Adobe Photoshop Lightroom, adding dates, locations, and tags to the images identifying media and medium. Ending up with a collection of over 2,000 journal pages, I had tagged data set of imagery. I proposed to Paul that we use a subset of the pages, 315 of my watercolors, to train the GAN to make an AI that could create watercolors.
Watercoloring
I had started watercoloring when I was 18 on a summer trip with Chuck Albanese, one of my architecture professors to sketch and watercolor in Italy and Greece. Watercoloring starts out hard as you learn to understand how to control the water and color, but like every skill you can get better at it with practice.
I had just finished my first year of architecture school where I learned from Kirby Lockard how to accurately draw freehand perspective, which is crucial in watercolor as a painting often start with a base drawing in pencil or pen. I loved watercoloring because it was portable and quick so I could take my creativity with me wherever I went and create something in as little as 15 minutes.
Journals as Memento Collections
Ever since that trip, for more than thirty years, I’ve been sketching and watercoloring and with the Grail Diary from Indiana Jones and the Last Crusade as inspiration, I started using journals to record my journey, thoughts, and ideas and to collect the ephemera that I gather along the way. What I found is that while I am sitting somewhere creating artwork to go on a page of my journal, my mind is recording everything around me, the people I meet, the conversations I have, what else I observe, and my feelings. The artwork turns into a mnemonic for the moment, a memento. My journal is a serial collection of personal mementos that I can openly share with others, knowing that I don’t have to write a personal narrative to record it. When I look at a page in my journal years later, I instantly recall the experience from years earlier.
Training the GAN
Paul started working on building the Python code to generate images and I started refining my watercolor collection, isolating the images on pages that had other content like line drawings, collage, and making a consistent collection of watercolors. Over the course of more than thirty years my style has evolved so there is a huge amount of variability in the collection, with the only common thread being that they were all watercolors and all created by me. Paul wrote the code so that a single numeric value from 0-999, a random number seed, would be the only input variable to generate each image. He ran the AI model for a few hours and we started seeing results come out of it and I was blown away!
Mementos in the Images
The AI found patterns and my techniques in the watercolor collection and started creating new unique abstract images that instantly triggered memories in me. I not only recognized elements in the images but those images triggered memories of the experiences that I had when creating them, combining these memories in a way that only happened for me in dreams. These images were mementos that spanned time and space.
The image above, #0780 reminds me of the time that I took my son Sam, then 12, to a nude figure drawing session at the local art store five years ago, where we both spent the evening sketching and painting. As you might imagine, it was a memorable evening, and seeing this image, I was brought right back there. As many of my watercolors over the years have been of monumental architecture, there is a heavy influence on the AI model for that kind of subject. In this generated artwork, I also see a hint of the architecture of Antonio Gaudi, one of my favorite architects whose work is the subject of a number of the paintings in the training model.
My Style in the Images
In the more than 1,000 generated images, I saw my style, composition, coloration, linework, and brushwork, but in very abstract forms, much more abstract than I’m comfortable with consciously doing in my work which today would be best characterized as Urban Sketching. Paul tells me that we can do much more to increase the variability, improve the quality, and refine the model. You will likely notice a recurring theme in many of the images of a monumental building on the right of the image; this is either an artifact of the AI model, or a hidden propensity for me to paint buildings on the right side of a watercolor – definitely an interesting direction for investigation. Paul is anxious to build a model against my whole corpus of artwork, which sounds like a very cool idea.
Putting the Artwork in a Gallery
I took more than 200 of the generated images and put them in a virtual art gallery using my Galeryst site. Generated artwork in a generated gallery seems very appropriate. I originally created Galeryst to share my journals with others as bound journals don’t typically exhibit well in physical galleries.
Sharing the Entire Collection
I’ve shared the entire collection of generated watercolor artwork using Adobe Photoshop Lightroom because I want your feedback on the images. You can also see a slideshow of the entire collection on the Lightroom site as well if you click on the … in the upper right of the page. Here is the feedback I would love from you:
- If you like an image, click on the ❤️ heart button in the lower left corner.
- If an image reminds you of something or somewhere, please click on the 💬 comment button in the lower left and leave a comment.
On Adobe Photoshop Lightroom Mobile, there’s a cool feature to “Choose Best Photos”. I ran the analyzer on the collection with a quality threshold of 16, and these are the top 16 that it picked:
Pretty cool, right?