Responsive Images: How they Almost Worked and What We Need
Issue № 343

Responsive Images: How they Almost Worked and What We Need

by
Published in Interaction Design, Layout & Grids, Responsive Design
A note from the editors: Since this article ran in Issue No. 343, author Mat Marquis has established and is chairing a Responsive Images Community Group for the W3C. All are invited to participate.

It’s our job, as designers and developers, to pick apart even the seemingly most simple tasks to find ways to improve them. When Ethan Marcotte coined “responsive web design,” he said that a responsive website is made up of three things: a flexible grid, flexible images and media, and media queries. In doing so, he opened up a world of new and exciting things to obsess over. I chose flexible images.

Article Continues Below
Article Continues Below

It’s easy enough to style images so that they scale to fit within a parent container by adding img { max-width: 100%; } to one’s stylesheet. To use this effectively, though, the image must be large enough to scale up to whatever size we can reasonably expect on the largest possible display. This can mean a great deal of overhead. At best it’s just wasteful. At worst, the mobile browser stuffs its hands in its pockets and goes to sulk in the corner, leaving the page partially rendered. A handful of full-bleed images designed for a 13” display could bring a mobile device on an Edge connection to its knees.

Unfortunately, we can’t test bandwidth in any reliable way—not yet, at least. Testing would likely mean introducing a significant download to measure against, which is a lot like setting something on fire to see exactly how flammable it is. What we can determine with some reliability is the size of a device’s screen—and while we can’t necessarily use screen size to make assumptions about bandwidth, it does directly correlate to what we’re trying to accomplish: while a browser’s window size may change, we’ll never need an image larger than the user’s screen.

While we were working on the new Boston Globe website, we devised a technique to mitigate the size of requests for users that may have limited bandwidth. Before I describe it here, I should really warn you up front: it broke. But we planned for that.

How responsive images worked#section2

Scott Jehl brilliantly masterminded Filament Group’s “responsive images” technique. We began with the core philosophy that the technique should err on the side of mobile. With a mobile-first approach, if any part of the process should break down the user should still receive a representative image—even if it’s a bit smaller—and avoid an unnecessarily large request on a device that may have limited bandwidth. Progressive enhancement writ large.

There are three key components to our responsive images script: the markup, the JavaScript, and a server-side redirect rule.

We started with, perhaps unsurprisingly, an image tag:

<img src="mobile-size.gif">

With that as our basis, we’re ensuring that we default to a mobile-sized image. We store the path to the larger image in a data attribute for easy access via JS:

<img src="images/mobile-size.jpg" data-fullsrc="images/desktop-size.jpg">

Now that we have both sources in our markup, we need a way to conditionally load the appropriate source. To do that we need to know the size of the user’s screen. Fortunately, there’s a relatively simple way to determine a device’s screen size through JavaScript by way of a property in the browsers’ window object: window.screen.width, though even that isn’t entirely reliable.

Here’s where we run up against a major challenge: we need to communicate this size to the server in time to defer the request for the image’s original src, if necessary. The server needs to know the client’s screen size before our images are displayed.

We eventually settled on setting the screen size in a cookie. A cookie set in the head of a document would be ready in time for the parsing of the document’s body, and included along with image requests.

I assume that you’re cringing at the idea of cookie-dependent functionality, and I understand—I do. Some of our first iterations involved <noscript> tags and document.write. I still have nightmares about dynamically-injected base tags. These were desperate times, and we left no stone unturned.

Our JavaScript’s second task was far more simple: if the screen was above the size we specified, we swapped the img tag’s original src for the path contained in the data-fullsrc attribute and displayed the larger image in place of the smaller one.

Since the screen size was now available to the server, we toyed with a server-side solution that would automatically resize the original image to suit the screen. We decided against this for several reasons:

  • We wanted to keep server-side dependency to an absolute minimum, and only implement something that could be easily recreated in various server environments.
  • Rather than simply resizing an existing image, we felt it was more important to have the flexibility to crop and zoom the larger image in a way that fully optimizes it for display on a smaller screen.
  • Any of the back-end solutions we experimented with involved scaling a large image down to suit the screen size, which creeped us right out. If the screen’s width was reported incorrectly or if the front-end scripting should break down, we would run the risk of subjecting users to a massive and unnecessary download.

The src-swapping part of the JavaScript handles the lion’s share of the work, but larger-screened devices still make redundant requests—first for the mobile image, then the full-size image. This results in a pretty jarring visual effect as well: the smaller image may be visible before the larger one snaps into place. Since the original src is fetched as the browser parses the img tag, we can’t really dodge that request from the client side. What we can do is mitigate those requests on the server side.

We wrote some simple Apache rewrite rules to intercept requests for an image and check for the cookie we set earlier. If the breakpoint conditions we specified were met, we redirected the request for the mobile-sized image to a 1—1 spacer gif. This kept the size of the redundant request low—especially once cached by the browser—and prevented the mobile-sized image from displaying before we swapped it for the full-size image. Since we didn’t want to apply this logic to every image site-wide, we later introduced a second rule that allowed us to flag images as responsive: the logic above only kicks in if the image’s filename contains “.r.”

<img src="images/mobile-size.r.jpg" data-fullsrc="images/desktop-size.jpg">

Thanks to our mobile-first approach, we had ourselves a pretty scrappy little technique. If any part of the equation should fail, no users would be penalized for their context. A failure on the client side—if cookies or JavaScript were unavailable, for example—would result in a smaller, but perfectly representative image. A failure on the server side would mean a request for the smaller image prior to the full-size image, but in a context where we could at least assume greater bandwidth. No one would be left without images regardless of their device, browser, and features.

This is fortunate really, since within a month or so of launching BostonGlobe.com our responsive image approach broke.

What went wrong#section3

Several newer browsers have implemented an “image prefetching” feature that allows images to be fetched before parsing the document’s body. While it’s hard to argue with a quicker overall loading scheme, it goes against the parsing behavior we’ve come to understand. For our purposes, this feature also invalidates all our methods for communicating the screen size to the server before the images are loaded and breaks our server-side redirect. You can see this on BostonGlobe.com right now: without that redirect you’ll briefly see the mobile-size image before the full-size image is loaded, but it may take sharp eyes and a few page refreshes. Fortunately, this additional overhead is only incurred on desktop browsers, where bandwidth is generally less of a concern.

So, now what?#section4

Long after the Boston Globe site launched, we continued to iterate on our approach. Jason Grigsby has done an incredible job documenting the details of those trials and tribulations in a series of blog posts.

This brings us to the present day, with some of the brightest minds on the web looking for something—anything—that will get the job done. Some think that it isn’t a solvable problem right now, and are placing their bets on user agent detection as a temporary solution. While this is a perfectly viable answer in the short term, I maintain that it’s untenable going forward: with the ever-expanding range of mobile phones and tablets in circulation, we could never hope to maintain a reasonable list of browsers and devices for long.

I believe that the ultimate solution shouldn’t hinge on scripting or CSS—and certainly nothing like UA detection, cookies, custom scripting on the front end, or any server-side shenanigans. Our aim is to represent and serve content appropriately, and for that reason I believe that this should be solved in markup.

The img tag isn’t going to cut it for this, though. It’s effective at conveying the hilarious antics of house cats, but it isn’t well suited to complex logic. It does one thing, and it does it well: it takes a single image source, and it puts it on your screen. If we were to modify this behavior at the browser level, we would never be able to guarantee our changes wouldn’t introduce issues in older browsers. We also know from experience that img doesn’t leave us much (if any) room to polyfill this new behavior.

What we need is a new markup pattern—one that allows us to specify multiple source files, but still specify universally-recognized markup as “fallback content” for browsers that don’t recognize the new tag. This should sound familiar, as this pattern already exists: the video and audio tags.

We know that a video tag can contain references to multiple sources, and that we can specify fallback content within the tag that’s only visible to browsers that don’t support video natively—usually a Flash-based video. What you may not know is that there’s already a way to use media queries to determine which video source to use, though browser support is a little spotty.

<video>
        <source src="high-res.webm" media="min-width:800px" />
        <source src="low-res.webm" />
        <img src="poster.jpg" />
</video>

From there, it doesn’t take much imagination to see how we could use a pattern like this.

<picture>
        <source src="high-res.jpg" media="min-width: 800px" />
        <source src="mobile.jpg" />
        <!-- Fallback content: -->
        <img src="mobile.jpg" />
</picture>

We could have a limitless number of options by using source media queries—higher resolution images for high-res displays over a certain size, for example. If we could reliably detect connection speed, one day we may be able to add media=“connection-speed: edge” or media=“min-speed: 200kbps” to our source elements. If these source elements are implemented per the HTML5 spec, a request will only be sent for the ones that match our media query. What we get is a single, highly-tailored request, with conditional flexibility limited only by a constantly growing roster of media queries.

Once we’ve established that markup as our foundation, we may be able to polyfill the expected behavior for browsers that don’t yet support it. While it’s likely that the polyfills would still involve more than one request, starting with a tried-and-true fallback pattern would allow us to apply polyfills at our discretion.

While we’re at it, I’d also like a pony#section5

As things stand now, a number of developers—myself included—are talking with WHATWG and various browser teams about the details of this new element. A frustrated group of developers pitching a need for a new element is certainly nothing new; we’re not the first, and I’m certain we won’t be the last. In fact, we’re not even the first to reach the exact same conclusion on image delivery: after brainstorming, we learned that a solution much like our own was posted to the W3C’s public mailing list in July of 2007—similar right down to the semantics. This subject has come up multiple times on the WHATWG and W3C discussion lists and quietly died out each time, but never during such a radical shift in browsing context as we’ve experienced over the past year or so, and never in such an exciting context as responsive web design.

While we can’t guarantee that a picture element—or something similar, semantics aside—will ever see the light of day, we’ve recognized that there is a need for such a markup pattern at present, and tremendous potential to such an approach in the future. I’d love to hear your thoughts.

77 Reader Comments

  1. @my ad: As far as I understood you’re talking about a solution to solve the responsive image problem inside CSS, too. This is something you can read in my proposal here: “see blog article”:http://novolo.de/blog/2012/01/19/what-happened-to-responsive-assets/.
    I like the idea of being responsive in either way — informational elements in HTML, non-informational/styling-elements in CSS.

    Is that what you’ve been thinking of? If not, please clarify so we can sort this out.

  2. @anselmh Yes, I would like to see the solution come through CSS as I think it is the best way to approach it for the long term. Adding in javascript workarounds seems messy to me as we look into the future and think it only serves as a crutch to the bigger issues.

    Thinking this through over the morning I’ve found various issues with my original thought and what could be solutions or could lead to complications. Below is a breakdown of how it could work changing around how a few things are currently structured.

    Through the img tag you would nolonger specify an full path to the asset as it is currently structured. ie, img src=”folder/logo.jpg”

    This would be handled through the CSS file using ‘path’ or some other specifier. So the document would now look as follows.

    HTML:

    img src=”logo.jpg”

    CSS:

    img { path: desktop; }

    This would lead the logo.jpg to be pulled from an image folder named ‘desktop’ and retrieve the full size asset.

    This can be changed using media queries to pull the different size assets from a different source folder as follows.

    @media screen and (max-width: 480px) {
    img { path: mobile; }
    }

    If a person doesn’t like to have all images coming out of one directory you could pull images out of various folders through a class element.

    img.company { path: company; }
    img.products { path: products; }

    resulting in – img class”company” src=”logo.jpg”

    and so on.

    You could specify where the assets are coming from depending on the different scenarios that are being designed for. As we look ahead screen sizes and resolutions are going to keep multiplying and we’ll very soon be taking into account televisions adding more complexities.

    Using this approach front end devs that are not concerned with responsive design can still use the traditional path structure without specifying path: ; in the CSS – img src=”folder/logo.jpg” and it would work as expected.

    The part that I’m not fond about is having to create various sizes of the same image however this also opens up a number of design opportunities as you can customize images depending on which display it is being viewed on.

    I see this providing designers a lot of potential to customize how a page would look and deliver images to that specific device instead of needing to use the same image for every instance and simply scaling it.

    This approach keeps the current responsive principles and allows for additional customization but my favorite part of this solution is it only adds one piece of code and leaves the rest of the styled elements via CSS as is.

  3. I should clarify when I say ‘I would like to see the solution come through CSS’. Ideally the best solution would be native HTML enhanced by CSS. The HTML would be a fall back to an enhanced CSS solution.

  4. I still think that that there should be an all emcompassing tag that would allow the browser to gracefully degrade from <video> to to any other element you could use for that area (and dependent on a standard media attribute). It seems like the video tag is only scratching the surface of the power of graceful degradation via HTML, why stop there? What about using a shorter headline or paragraph text if the viewport is smaller? Why not give us the option of giving a html alternative for an image versus relying on basic text in an alt tag?

    Too many tags, not enough scalability.

  5. As our mobile-large screen responsive evolution presses on at warp speed… those on the crest of the wave are held back by seemingly simple technological hurdles. We have to remember the large screen in the equation as brand channels will begin to immerge with one feed -> mobile-large screen.

    Our frustrations are but a small slice in time. What may seem like a solution today, may be an obstacle in only a matter of months.

    I so love your suggested solution. It absolutely makes sense – today… and is evolution enabled.

    Well said and good luck in persuading the powers that be to enable these markup improvements.

    Responsive image nirvana.

  7. This is excellent article in the way it expresses what developers go through in battling how things change and evolve on the web and whereby most designers don’t ever really get the intricacies that developers have to deal with.

    I think it’s not enough for designers to just study design, they need to study structure and code as well. The reverse is equally important but I think there’s plenty of momentum lately for developers to understand the importance of great design. Full disclosure, as a former developer, I may be biased in who I think needs to work harder. 🙂 Nevertheless, nice article!

  11. All these approaches put to much of a burden on developers! They’re clunky, prone to breakage and force the use of more markup. But maybe worst of all they force you to define fixed dimensions where you switch the image and to maintain multiple versions of the image!
    I’m really quite astonished that no one so far has mentioned that JPEG 2000 together with the JPIP protocol handles this use case already… Store the image on the server in the maximum resolution you want to offer, but just download a size appropriate for the user’s display. Open source JPIP servers are available… (Djatoka for instance can serve ‘regular’ formats from a jp2 file, providing a fallback.) Better native support for JPEG 2000 in browsers would be nice of course.

    Some experiments need to be done on how best to use inline JPEG 2000 for responsive images. I suppose one approach could be sending a scaling factor (or max size) with a cookie and do some URI rewriting on the server side to get the desired image size. CSS can still be used to downscale the image for a perfect fit. It’s not optimal, but the main point is to just work with one image and let the server handle the scaling.

    Yes, JPEG 2000 requires encoders/decoders that are complex and more computationally ‘demanding’. But there has to be a tradeoff somewhere.

    And let’s not forget that JPEG 2000 offers many other useful features.

  12. I wrote the comment above in a hurry and forgot to clarify that of course no browser currently implements the JPIP protocol. However I still think using a JPEG 2000 server over HTTP could be a very interesting method towards solving the problem of responsive images. And it would certainly help push native implementations of the JPEG 2000 standard.

  13. What about an approach of using the CSS media type to control the image source? No JavaScript involved, but it does assume that mobile devices come up with a media type of handheld. You probably can’t use the img tag, but maybe a div tag. Then again it changes the function of the div tag and is not semantically correct. A change to the markup to support images like videos would still be the best option.

  14. I’ve been experimenting with this, . I watched the resources as I expanded the window and the different image for the widest display isn’t requested until that CSS is triggered by the window reaching each specified width. (I cused different images so I could see the change easily).

    No Javascript at all.

    .rightBox{

    display:none;
    width:40%;
    background-size:contain;
    background-repeat:no-repeat;
    float:right;
    overflow:auto
    }

    @media screen and (orientation:landscape) and (min-width:480px){

    .rightBox{
    display:block;
    height:40%;
    background-image:url(‘../images/mobile/sevenstep.jpg’);
    }
    }

    @media screen and (orientation:landscape) and (min-width:1024px){

    .rightBox{
    display:block;
    height:40%;
    background-image:url(‘../images/full/planning.jpg’);
    }
    }

  15. I agree, a picture tag was perfect to solve this problem, we have a solution for this, but this solution is not easy to implement, and, at this time, we have to care about many device sizes more and more. Hey W3C, look this post! 😛

  16. @Marc Diethelm: This is entirely the best solution. I’ve written about it as the “perfect solution” in my blog post. The problem is, that JPEG2000 isn’t that good of a format for graphics. So we would need ideally WebP/WebM to be able to progressively download and serve responsive images. Until that is implemented in all browsers (haha!) you will still need another approach, so that’s why we proposed the HTML5 solution for interim.

    @JCOELHO, @yeosteve: A CSS only solution wouldn’t work as it’s not semantic and doesn’t work valid. It still should be possible to do via CSS for non-semantic (styling/layout) purposes, too. Mailing list of W3C didn’t care about this very much 😉

    @Anserson Schmitt: We (Mat and I) are currently discussing on WHATWG lists and if we have a solution over there, it will move to the W3C list. So be sure we do our best W3C cares about this problem 🙂

    @ NatCk: adaptive-images.com is only a polyfill solution and uses some technique not very good for a standard-solution.

  17. Since we’re talking about changing the browser’s behavior anyway, why not actually make it offer up the information to properly solve this problem server-side? Let’s have the browser send the information that is available to the “css media queries”:http://www.w3.org/TR/css3-mediaqueries/ in the HTTP GET request headers caused by the one-and-only tag or CSS url(“whatever.png”).

    To wit:
    X-Media: “screen color:24 resolution:120dpi orientation:portrait device:640×480 viewport:500×300”

    (obviously, the aspect-ratio and device-aspect-ratio can be computed, color-index seems meh)

    The nice thing about this is that it is really up to the user-agent’s current needs to determine what should be displayed, thus what should be requested. If someone zooms in, the user-agent can ask for a bigger version. If they switch to print view, we can ask for the monochrome version in higher DPI and all is warm and fuzzy. If the server doesn’t care, it serves up the one-true-image. If the server cares, it can serve specific versions. Cache logic can be driven by the standard content-negotiation logic (and left non-cached until it is).

    The best part of ALL of this is that the CSS can just reference an image, the html can just include a img tag, etc… and the browser’s request can be driven by what it knows already about the size/placement/device of the image.

  18. Use JavaScript stylesheet manipulation. All responsive images should be rendered in an with a token classname, such as ‘responsive’, and should have fallback ‘src’ value. In <head>, JS will add a stylesheet rule to ‘display:none’ all such elements (to avoid requesting the wrong image resource), then will determine the appropriate screen size category. Script block at the bottom of the page will alter ‘src’ values of all responsive s according to screen size category, per some naming convention, and then update stylesheet rule to ‘display:inline’ all such elements.

    This may cause flicker, more noticeable when HTML is generated/delivered slowly. This can be mitigated by encapsulating the responsive in some server-side component that also emits a script block right after each to do the ‘src’ attribute manipulation and displaying.

    The ‘resize’ event will be handled to perform this logic in case different images need to be re-rendered (can be short-circuited if new image would have lower quality).

    If UA doesn’t support JS or doesn’t support stylesheet manipulation, there will be flicker or simply the fallback content is rendered.

  19. Is there a reason why you can’t just use images as the source in the video tag with the image MIME type, as in the following example??

    <video>
    </video>

    Thanks.

  22. The “Responsive Content”:http://stephanfowler.github.com/responsive-content/ jQuery plugin is an approach which works in a “coarse grained” way, ajax-loading an entire HTML fragment into the page, given current window width (and pixel density). It can be used to tell the server to supply appropriately sized image src URLs etc.

    It’s a fork of Github’s Pjax loader. It does not rely on User Agent or cookies – just window width.

  25. How do would you appreciate these 2 solutions to download only the small images when needed (Smartphone, iPhone, Android)
    1) Use $_SERVER(‘HTTP_USER_AGENT’) in the server script to detect that it is not a PC
    2) Have a main page that detects the screen size and pass it in the URI for subsequent pages.

  26. A great solution to <video>-like functionality. Wondering if a similar functionality can make element accept multiple sources like video does.

Got something to say?

We have turned off comments, but you can see what folks had to say before we did so.

More from ALA

I am a creative.

by
A List Apart founder and web design OG Zeldman ponders the moments of inspiration, the hours of plodding, and the ultimate mystery at the heart of a creative career.
Career