No. 131: 🤖 Stable Diffusion AI
🎨 Is it a threat to digital artists?
Hello,
Greetings from New York 🗽
Well, depending on when you open this, I could well be sitting at Gatwick airport.
But I’ll be mooching around New York all week, teaching, speaking at events, meeting clients, the usual pre-pandemic beat. Let me know if you’re about!
In this hi, tech., we’ve got a lot to cover:
An Apple launch event
Intagram gets fined
Amazon’s health travails
How Github Copilot increases productivity
And a hands-on review of Stable Diffusion
Apple launches some slightly upgraded products
There are a few things in this life that will never excite me, not even one bit:
Lord of the Rings
Inter-brand social media “banter”
Martial arts
An Apple launch event
But Apple did launch some new doodads this week and I’m duty-bound to report on them. There was an iPhone, some AirPods, a new Watch, and more - you can see it all here.
Instagram fined €405M in EU over children’s privacy
“The Instagram complaint focused on the platform’s processing of children’s data for business accounts and on a user registration system it operated which the DPC found could lead to the accounts of child users being set to “public” by default, unless the user changed the account settings to set it to “private.””
Amazon may launch its healthcare business in Japan next
There is a school of thought - a business school of thought, in fact - that the tech giants can conquer any industry at will. They already have customers, they have huge cash reserves, and they of course have lots of valuable data. Moreover, they can subsidise entry to a new market and offer products or services at a loss, until the competition caves.
This theory does not always survive contact with reality.
In Amazon’s case, they are shutting down their Care product, which provided online pharmacy access. Amazon’s acquisition of One Medical is also under investigation by the FTC.
Given the size of the prize on offer, Amazon will keep trying as it keeps trying with grocery shopping. We’d have a better chance of assessing these businesses if instead of lumping them together as “tech giants” and instead pinned down their individual USPs. Amazon is ruthlessly efficient when it comes to logistics. That is rather handy when it comes to delivering prescriptions, but it doesn’t mean Amazon knows anything about the healthcare industry. This explains the company’s latter strategy of simply buying a company, One Medical, that can already handle this aspect well.
It ends up being much more expensive than anyone predicted, nonetheless. That could be a price worth paying in health, albeit not in all other areas.
Because also this week:
Book On Google For Google Flights To Be Discontinued Outside US
We discussed this over 3 years ago in hi, tech. edition 7 (which survives on LinkedIn here), when I looked at how Google Travel would curtail innovation:
The "monopoly spurs innovation" line no longer applies. Why try and compete when there is no chance of winning?
Google Flights will soon shut down, perhaps because it wasn’t worth the hassle for Google (it was managing payments and refunds) or perhaps due to the looming threat of regulatory intervention. It was an open-and-shut case of abusing a monopolistic position, according to this not-exactly-qualified judge.
Apple is gaining on Facebook and Google in online ads after iOS privacy change, report shows
This report focuses on app developer spend on ads within app stores. “Google remains at the top, with 34%. Facebook is second at 28%, followed by Apple at 15%.”
However, Apple plans to extend its advertising reach across the full iOS ecosystem, where its rivals note that Apple is not so heavily restricted by its own App Tracking Transparency (ATT) initiative. How convenient.
It’s not unethical or illegal on Apple’s part - I’d say it is a sensible long-term strategy.
Apple has some benefits when it comes to ads:
It doesn’t depend on advertising revenues. Google and Meta can only grow if they keep bringing in more advertising revenues, but Apple can slowly chip away at their position without the same need for instant, explosive growth.
The aforementioned “privacy push” has positioned Apple as the consumer-friendly option. Regulators don’t quite see it that way, but Apple does certainly have less baggage than Google/Meta.
Apple has whole hardware infrastructure and software ecosystem. They’re gathering all that data, across all those devices, and they have just overtaken Android as the most popular OS in the US.
Their customers are known to have a little more dough than us poor Android users. We know this because Apple customers have, by very definition, spent a fortune on their device. That makes them prime targets for conspicuous consumption: If you bought that phone, you’ll love this Peloton, and so on.
But it also some drawbacks:
Meta’s apps are still superior at matching users with new brands. Apple is dependent on the user searching for something - they only offer direct response ads, in other words. They are building a programmatic infrastructure, but that’s not their area of expertise. As we saw with Amazon Health earlier in this edition, that matters more in some industries than others.
Apple’s search engine is nowhere near the popularity of Google Search.
It lacks the cold, hard purchase data that Amazon has. Apple Pay can be a useful proxy here, however.
Advertising simply isn’t a core focus for Apple. It may, like Google with its Travel initiative, find that it’s not worth the extra hassle.
But my bet would be that it will find quite the contrary; and advertisers should be brushing up on Apple’s plans already.
The definitive guide to what’s in and out in ad tech in 2022
This is a snappy, visual summary of the changing trends in ad tech.
Boston’s ClearMotion funded by Nio Capital to switch on in-car metaverse in China
“ClearMotion, a Boston-based company that aims to make car rides less bumpy and more fun, just raised $39 million led by Nio Capital, a mobility-focused venture capital fund established by William Li, the founder of Chinese electric vehicle upstart Nio.”
But still, who needs an in-car metaverse?
Quantifying Github Copilot’s Impact on Developer Productivity and Happiness
This comes with the usual caveat (Github ran the experiment and of course, they have a vested interest in confirming their own hypotheses), but it’s still fascinating to see the impact of this technology.
Engineers say that they can get more done, more efficiently, and they have time to work on the “fun” parts of programming instead. The experiment results seem to substantiate these opinions. This Twitter thread also shows some of the handy ways Copilot can help identify bugs in code.
Wondering what Github Copilot is? There’s a hi, tech. for that:
That’s a wonderful transitional link to this week’s big story: The impact of Stable Diffusion on digital art.
Apparently, actors are worried that AI will take their jobs. The actors I know can’t get any jobs anyway, so I don’t know what the fuss is about.
But still, let’s entertain the notion.
And now, artists are concerned that image generation technology will replace them. You’ll recall that we have discussed DALL-E 2 on many occasions here, but Stable Diffusion is the new talk of the town:
On August 22, Stability AI released its open source image generation model that arguably matches DALL-E 2 in quality.
It also launched its own commercial website, called DreamStudio, that sells access to compute time for generating images with Stable Diffusion. Unlike DALL-E 2, anyone can use it, and since the Stable Diffusion code is open source, projects can build off it with few restrictions.
The fact that Stable Diffusion is open source has a number of implications on the innovation-accountability spectrum:
It can scale very quickly.
That means Stable Diffusion has little control over what developers might choose to do next. And it could get wacky at best, dangerous at worst. Stable Diffusion has few of the restrictions that apply on DALL-E 2, for example on the use of people’s faces.
However, Stable Diffusion may also have some legal responsiblity for developers’ creations.
TechCrunch reports this week that, “Under the EU’s draft AI Act, open source developers would have to adhere to guidelines for risk management, data governance, technical documentation and transparency, as well as standards of accuracy and cybersecurity.”
Whatever developers learn to do with Stable Diffusion can be applied elsewhere. We’d be naïve to believe that all this technology is intended with the sole purpose of creating funky new images. The underlying techniques could be applied to solve other, more scientific problems in future.
How to use Stable Diffusion
Stable Diffusion doesn’t have an intuitive interface yet, so you do need a bit of coding know-how and access to a decent GPU to run it. (Details on requirements here.)
If you have an M1 or M2 Mac, it’s a little easier - this post contains the code you’ll need to get started.
Or you can use a forked version of the program that wil allow you to create new images, for example here.
There are search engines (such as Lexica) for Stable Diffusion images, too.
I’d expect a user-friendly version to arrive pretty soon.
How does Stable Diffusion work?
Ok, I’ll stick to the bits you need to know here and I’ll link out to the more detailed docs, should you desire them:
It uses latent diffusion models, which can identify shapes in images and bring them into focus based on their predicted match with the words in a prompt. (This video has some nice visualisations of the process.)
The model requires lots of images and lots of metadata. Stability AI uses LAION-5B, a set of 5bn+ images scraped from the Web.
At the training stage, the model assesses the relationships between words and images using Contrastive Language–Image Pre-training (CLIP).
Based on what the model learns, it can create new combinations of images based on text prompts.
The Results
The /r/deepdream/ Sub-Reddit collates a lot of wonderful creations, from Spiderman eating a doughnut to a cat in a teacup. It’s worth visiting just to see what people can conjure up with a few words.
Others have figured out ways to extend Stable Diffusion’s usefulness beyond one-off image generation. This example uses the tech to create a music video, with impressive results.
This guy uses it to convert Monet paintings into realistic images, “de-impressionising” them. Another turns Monet’s images into animations.
The technology is improving at a frightening pace - some developers are already working on systems to create 3D objects and even 3D worlds from text prompts. You can imagine this working as a handy assistant to architects or designers, for example, at the early stages of a project.
The Future Impact
Well, that’s always somewhat challenging to foresee. We’ll do our chances of success some good if we break this down into the different “stakeholders”:
Artists
Positive:
It acts as a visual playground, a stimulus for grander ideas.
You need to know how to use the tools to get the maximum effect, and artists could work with this technology to create entirely new media.
Negative:
It will commoditise some areas of digital art, just like the camera took the lead on photorealism.
Businesses
An obvious use case for these images is as a replacement for those horrendous stock photos companies love using.
Stable Diffusion says it has established a “working relationship” with Adobe already.
We could get to the stage where businesses can submit our audience analysis to Stable Diffusion and it would suggest ad creatives to use across different platforms. It wouldn’t get it 100% right, but it could open up ideas to explore.
Everyday searchers:
We adapt our behaviours to maximise the benefits we receive. Existing image search engines are awful, so we don’t waste time searching for anything too specific.
DALL-E tells users to ask for “unusual or implausible images”, presumably both because it excels in this area and users are typically rather unimaginative. Our senses have been dulled by Google Images.
This has a knock-on effect in other areas of the online experience. For instance, we will come to expect the level of precision we see in DALL-E and Stable Diffusion from results elsewhere. That could play into the hands of more visual platforms (think TikTok, Instagram) if they learn to integrate this much variety into their stable business models.
For example: Imagine that, instead of clicking around and searching for ages for a product you can picture in your mind, you could use AI to create an image of it and then automatically find a match?
It’s still easy to see why artists are concerned about these tools and, not being an artist myself, I need to be wary that I don’t blithely dismiss their unease.
So let’s take stock of what those breezy dismissals might be:
It is true that technology has simply displaced labour in other areas, rather than replacing it altogether. Calculators and accountants have found a way to co-exist, for better and for worse.
It is also true that AI text generation tools have not replaced good writers. They are fine for prosaic tasks that us real writers refuse, but ill-equipped to dish out sarcasm. The robots also, let’s be honest, wouldn’t write a newsletter for free for three years. 😉
And it is additionally true that artists have always faced the threat of cheap knock-offs. Salvador Dalí (he’d get a kick out of the DALL-E name) had one way of dealing with this: he used to sign forgeries (allegedly) of his own paintings. For a fee, naturally. (Fellow surrealist André Breton noted that ‘Avida Dollars’ or ‘money crazy’ was an anagram of Dalí’s name.)
Now, these maxims could lead us to a hearty shrug, served à la française, and the suggestion that artists just deal with the new reality.
But to respond to these points in turn:
Any division of labour between man and machine is poorly defined when it comes to art. The artist either does or does not create the work, and it’s tricky to sub-divide that creation into micro-tasks that could be shared. For instance, Github Copilot has a clear role when it comes to programming and it only takes on some elements of the human programmer’s role.
These new tools do create fantastic images. They have lots of flaws and aren’t ready to take on the world yet, but artists are entitled (even advised) to ask the big questions at this stage. Unlike other tasks (standard calculations in accountancy or statistic-based reporting in journalism, for example), the AI would take over the highest-value aspect of this practice.
In our Dalí forgery example, he could still keep some control over how his image was used. We have no idea how to manage these rights when an open source AI tool is endlessly creating and distributing an artist’s work. (Wait, whatever happened to NFTs? Weren’t they meant to help solve this?)
None of this means we should try and stick a lid back on the AI image box.
Yet, where DALL-E 2 posed intriguing questions, Stable Diffusion has made them urgent.
People will still value - and perhaps even put a premium on - art that is created by another person. It would be a very cold world indeed if we only cared about the material output of the artistic process. Were that the case, we would never go to a gallery or pay for an original work, since a skilled forger could replicate the effect exactly. We’d have a ‘Starry Night’ in every room of our homes.
Furthermore, the net result of these tools could be very beneficial at a broader societal level. Where the work can be completed for a lower cost by technology and to a higher quality, we should of course accept as progress. Plus, rather than mimicking existing artists, what these tools do so well is to create images we would otherwise struggle to develop. It depends on existing styles for training, then combines them in novel ways. That is to be welcomed as a communicative development, as well as a purely expressive one.
We would still do well to remember: Powerful technologies will not arrive at beneficial destinations alone; we must inspect them, correct them - and then direct them there ourselves.
Let’s see what Stable Diffusion makes of that idea:
Perfect.