These days, many people advocate for the use “image sets”. Meanwhile, image sets is just a way to put more burden on the developer by asking him to specify two image urls in his markup if he want to support high-resolution screens (phones, tablets, and high-end notebooks).

To me, this is a bad solution to a very important problem. Indeed, you’re sending too much information to the client, the client has to choose which version of the image to download, and if the conditions in which the client view the page changes, it may well need to start over the download to fetch an image with a more important resolution (i.e. a PC with more than one screen, or phone switching from 3G to Wi-Fi). Additionally, a browser which choose for a “high resolution” version will have to wait more before to display a first render of the image.

Using the data we already have

It would be way better if the browser could simply download a “normal-res” version of the image and the server would automatically advertise a “patch” that, when downloaded, allow the browser to ‘upgrade’ the normal-res image to a high-res version.

Indeed, the high-res image is very likely to look a lot like the normal-res image. Most of the data is already known by the client which downloaded the normal-res version, why redownload everything while all we need is to download the high-res details.

I developed a prototype which use a ‘normal-res’ file (1024x683 JPEG, 506Ko) and ‘diff’ file(2048x1365 JPEG 1.12Mo), both at a 100% Quality Level, to reconstruct the original high-res photo of a spider web containing very fine details (2048x1365 JPEG 1.79Mo).

The results are stunning, as I can’t even notice the difference between the original and the reconstructed file, even when the blur of the original 1024x683 file is clearly visible (however, if you make a diff between the two, you’re likely to find they are differences, but those are really very small, normally inferior to 1/255 per color component).

Also, while the 1.12Mo patch is being downloaded, the browser can use the 506Ko normal-res image even on high-res displays immediately, making its first render much faster.

Since the algorithm I use doesn’t depend of the actual image format used to store the image, it also works with PNG or JPEG-XR. It’s visible that the results are somewhat better when you use JPEG-XR instead of JPEG, at equal file size.

Proposed protocol

I propose to add a small HTTP header pointing to the diff file:

HTTP/1.0 200 OK
Image-Resolution-Patch: ./my-image-2x-patch.xrj

The server could be configured to send that header automatically if a “-2x-patch” file exist in the same folder as the requested one, or generate them on the fly from a FlashPix file.

There’s no burden on the developer markup (you just need a traditional <img src> to your 1x file). Since it’s an image format problem, there’s no reason it should have one: developers should not care about image format.

Also, those image patch can apply sequentially, meaning you can make a 1x image file, a (1x to 1.5x) patch and a (1.5x to 2x) patch. Browsers would start by downloading the 1x file, notice they want the 1.5x patch and download it. Then, as the user starts to zoom the page in, the browser could ask the server for the 2x patch and apply it on the 1.5x resulting image.

Proposed algorithm

To create a patch from a low-res version of the image and its high-res version :

1. Use bilinear interpolate to scale the low-res image to the high-res dimension.
   
2. For each color component of the high-res image’s pixels
   
• Compute the difference between the high-res image and the scaled low-res image.
• Add 255 to get a positive number.
• Divide that number by two (int division). Here you loose 1 significative bit
• Use that as the corresponding color component of the patch image
  
3. Write the patch image in a file, using the file format and compression you prefer.

To reconstruct the high-res version of the image from the patch, the algorithm is simpler

1. Use bilinear interpolate to scale the low-res image to the high-res dimension.
2. Add two times the values of the diff image patch to each color component
3. Remove 255 to the value to each color component

Some results

You can download the results here (JPEG-XR format used, 99% quality). For your convenience, I will also add one of the experiments here (note that the images are downscaled):

Original 2x file (1.68Mo JPEG-XR)

Resized 1x file (494Ko JPEG-XR)

Patch file (987Ko JPEG-XR)

Reconstructed file

Difference file (absolute value)

Call to action

If you know somebody who would be happy to create a C++/GPU program to support my algorithm and make experiments with it, send him this link! I’ve made a quick prototype implementation in Visual Basic .NET but I would like to see how fast a native implementation can be.

Otherwise, if you know image format developers, I would like to hear from them about a way to store even more efficiently the diff patch (JPEG-XR is doing fine but maybe it doesn’t take advantage of some properties of the patch which could make the data even smaller).

You can also comment or share this post if you found it interesting :-)

Tweet