Welcome to the latest edition of the Responsive Images Guide!
In part 1, I laid out the big idea: a responsive image is a variable image – which adjusts itself to fit variable contexts.
In part 2, we looked at the most common way that an image can do exactly that: scaling itself up and down to fit viewports of different sizes and screens with different densities.
Now, we turn our attention to a different axis of variability: variable image encoding.
What do I mean when I say, “variable encoding”? Two things:
- Delivering an image in different formats (depending on the end-user’s browser).
- Striking different quality/compression tradeoffs (depending on the end user’s screen, connection speed, data plan, and/or preferences)
Let’s tackle both of these, in turn.
Variable Formats
There are many (many) different image formats, and new ones are being developed all of the time.
This varied, evolving landscape has vexed the <img> tag from the very start. To wit—
I'd like to propose a new, optional HTML tag:
IMG
Required argument is SRC="url".
[…]
Browsers should be afforded flexibility as to which image formats they support. Xbm and Xpm are good ones to support, for example. If a browser cannot interpret a given format, it can do whatever it wants instead (X Mosaic will pop up a default bitmap as a placeholder).
[…] I know this is hazy wrt image format, but I don't see an alternative than to just say ``let the browser do what it can'' and wait for the perfect solution to come along (MIME, someday, maybe).
Let me know what you think.........
—Marc Andreessen (February 1993)
“Just let the browser do what it can”
Let’s tug on this historical thread a bit and see where it goes, shall we?
Mosaic 0.10, the browser that Marc and his team released just three weeks (!) after proposing IMG to the world, supported two formats: Gif and XBM.
Support for anything else wasn’t suggested until almost a year later.
Hello,
Is there a way to include Inlined image not in gif format?
—Pini Albilia (January 1994)
This innocent, 13-word question ignited a firestorm of responses. Some of them clung tightly to the idea of hyper*text*:
The goal of WWW is to build a distributed, global, hypertext system; not to implement every "feature du jour" that comes along. The correct, and current, solution to this problem is to simply create a hypertext link to the object in question.
—Tony Sanders (January 1994)
One response proved remarkably prescient:
As a (still learng) ) HTML programmer, could I just request one possible extension for a future version of Mosaic, and that's JPEG handling?
For a specific (but very useful) category of images, they're very useful, basically because they can be far smaller as long as you don't mind a few artefacts. [...] Since bandwidth will always lag behind storage, I think this is quite significant for the 'Web.
—Andy Holyer (January 1994)
But the most important reply, for our purposes, came from a programmer working on Mosaic:
The advantage to using Gifs, however, is that the document will be viewable in all browsers that support inlined images. If you add a new format, the document will only be viewble by a very limited set of people... At least in terms of Mosaic, we have currently only agreed to support gifs and xbms on all three platforms...
Sound familiar?
You can’t use—
—on the web, because if you do, a majority of your users won’t be able to see anything at all. And browsers have strong incentives to be extremely conservative about rocking this boat of stable, universal support for a small set of formats.
Tale as old as time – or at least, <img>.
And so, after two decades, there are only four image formats that enjoy usably-universal support: GIF, JPEG, PNG, and SVG.
“(MIME, someday, maybe)”
Must we limit ourselves to serving up these lowest-common-denominators?
No!
In 2017, we can serve different formats to different users, depending on what their browser can handle. We can, for instance, send a WebP to Chrome, a JPEG-2000 to Safari, and a JPEG-XR to Edge – and still fall back to a trusty ol’ JPEG in Firefox.
How? In one of two ways.
On the client, with <picture> and <source type>
The first (and newest!) way to serve up adaptable-format images is via the <picture> and <source> elements, using the type attribute. Jason Grigsby has a nice write up of the details; in short, we can give the browser a bunch of different URLs and mark them up with those MIME types that Mark referenced way back in 1993. The browser loads the first URL whose type it supports.
On the server, with Accept
There’s also a much older way to accomplish more or less the same thing, called Content negotiation.
If you’re writing markup, <picture>/<source type> pattern is a lot more work than plain old <img src>. You have to generate a bunch of different resources, decide an order of preference, and write out a bunch of extra HTML. Content negotiation says: let’s do all of that work on the server. The client can tell the server which formats it supports up front, via a list of MIME types in the Accept HTTP header. Then the server can do all of the hard work of generating and managing alternate resources, and deciding which ones to send to which clients.
Unless you’re as handy on the back-end as you are on the front-end, this can be difficult to set up yourself. But if you use an image hosting/delivery service like Cloudinary, it’s as easy as adding a few characters to your URL. For instance, this:
<img src="https://res.cloudinary.com/demo/f_auto/sample.jpg" alt="A bee on a flower." />
Delivers a WebP to Chrome, a JPEG-XR to Edge, and a JPEG to Safari and Firefox – all from the same src.
Whether you use multiple URLs in <picture>/<source type> blocks, or a single URL pointing to a smart host, the ability to serve up multiple, alternate formats is not only great for users – it’s also great for the web. The ability to fall back safely to older formats gives developers the ability to safely leap forward and use cutting edge formats before they’re supported in every browser. The next twenty years should be a lot more exciting than the last twenty, when it comes to image formats on the web.
Variable quality/compression
So that’s variable, responsive, image formats, sorted.
What else about the way that images are encoded might we be able to adapt on the fly, in order to give each user the best-possible experience?
Many things! Image compression is complicated and encoders provide a multitude of options, toggles, and switches which let you determine the nitty-gritty details of how your image will be compressed. Most encoders also wrap those myriad settings up into one big, friendly, user-facing setting, called “quality.” A high–“quality” file will be nearly indistinguishable from the original, but will also be quite large, file-size-wise. A low-“quality” file may contain distracting artifacts, but it will weigh much less than a high-quality file would. A well-engineered website strikes a judicious balance, serving up images that look fairly good, and load fairly fast.
In the bad old days when I used to “Save for Web” all of my images by hand out of Photoshop, I had a lucky JPEG quality number: 72. At some point, I’d done some quick-and-dirty trial-and-error and determined that, for most images, “unacceptable” compression artifacts started to creep in when Photoshop’s quality was below 60, but “Save for Web” qualities over 80 or so provided diminishing returns. So for years, I exported everything at 72 and never really thought much more about it.
How low can you go?
Imagine my surprise when a hot new responsive image technique started making the rounds, advocating for a new go-to quality level: zero.
Turns out, JPEG’s compression artifacts aren’t as noticeable when they’re rendered at Hi-DPI, and when browsers scale 2x, highly-compressed images down to fit 1x screens, most of the artifacts get scaled/blurred away, too.
Before we had good, standard, ways to serve up images in multiple, alternate resolutions, compressive images provided a promising path forward. Unfortunately, even though they saved bytes over the wire, compressive images did nothing to save browsers from all of the work that they have to do after they’ve downloaded an over-sized image, namely: decoding, storing, and resizing it. Double turns out, sending high-resolution images to everyone, even if they’re highly-compressed and reasonably light, can lead to slow paint times, janky scrolls, thrashed memory and trashed battery-life on low-powered devices.
So, compressive images were not the one true responsive images solution. But they did teach us something: we can and should send lower-quality/higher-compression images to hi-DPI screens, where compression artifacts are less visible. “Compressive images” provided the first, concrete use case for variable-compression images, and it wasn’t long until lights started going off in people’s heads about a much, much bigger one.
Your next billion users
Ilya Grigorick summarizes it nicely in this article about the Save-Data Client Hint.
In short, billions (billions!) of internet-connected people pay dearly – with their time and money – for what we in the wealthy west consider small amounts of data. Most of the world still connects to the internet via 2G, and the income-adjusted cost of loading a few hundred kilobytes to a user in, say, Afghanistan is staggering. Thus, almost 9% of all page loads and 41% of those from India come from a browser that most westerners have never heard of: UC Browser, which re-compresses and optimizes sites on proxy servers, before sending them to users. And the vast majority of gains that UC Browser and other proxy browsers like it provide come from – you guessed it – re-compressing images.
So: why not cut out the middleman? Why not do that extra compression for users with constrained connections ourselves, on our own servers? What if we could dynamically dial our image quality/compression up and down, per-user, in response to connection speed and user preferences?
This is the promise of the Save-Data Client Hint. The hint is currently only emitted by Blink-based browsers, but that shouldn’t stop you from using it to implement responsively-compressed images today. If you don’t have the ability to implement support for it on the back end, you can craft your own proxy-of-a-sort on the front-end using Service Workers – or just point to an image hosting/delivery service like Cloudinary, which will smartly handle variable compression for you, automatically.
Variable encodings for variable contexts
So, to review:
- A responsive image is a variable image that adapts to meet variable contexts.
- The most important axis of variability, when it comes to responsive images, is resolution.
- In this article, we looked at another: variable image encoding. Which is really an umbrella term for two (related) things: format-switching, based on which formats a given browser supports, and quality/compression-switching, based on a user’s screen resolution, connection, and/or preferences.
So, so far, we’ve looked at dynamically adapting images’ height, width, format, and quality. What else might we want to change about our images in order to respond to users’ particular contexts? Join us next time, when we’ll reach the final frontier of responsive image variability: variable image content.
- Allow me to close the loop on our JPEG support story. Marc left the Mosaic project in ’93 and founded Netscape in April of ’94. When Netscape Navigator was announced to the public in the fall of ’94, JPEG support was one of its three big headline features. Mosaic quickly followed suit, shipping JPEG support six months later. So when Microsoft licensed Mosaic’s code and used it as the foundation of Internet Explorer 1.0, released in August of 1995, it supported JPEGs, too. Thus, supported by both major browsers, JPEG became the standard format for photographic images on the web. Twenty-two years later, it still is. ↩
