Introducing WP Image Processing Queue – On‑the‑Fly Image Processing Done Right

I’ve been bugging core contributors and plugin authors about background processing for a year and half now. To the point where Krogsgard even made fun of me for it at the WordCamp EU after party. “There goes Brad, talking about background processing again.” He was joking, but it’s true. Clearly I think it’s an important subject, but I was surprised to realize that I haven’t talked about it at all on this blog.

Ash wrote about background processing just over a year ago. He presented an awesome library that he coded for WP Offload S3 and released it as a standalone library on GitHub for others to use. I wasn’t the only one who was impressed. WooCommerce and Ninja Forms are using it as well.

The library is great, but I realized there was a problem. If each theme/plugin implements their own background processing queue instead of sharing one queue, we could end up running a bunch of jobs at the same time and impacting server performance. As more and more themes/plugins start to do their own background processing (whether they use this library or do their own thing), the bigger this problem is going to get.

I think the best solution is to get background processing into WordPress core so that all themes/plugins can share a single queue and ensure we don’t impact server performance. And so started my crusade.

At PressNomics, I had a great chat with Mike Schroder. He presented a very good path to core: find a feature that WordPress core needs and that needs background processing. In other words, piggyback! This is exactly how the image optimization stuff made it into core last year: by piggybacking off of responsive images. For background processing, he proposed coming up with an alternative to on-the-fly image processing (OTFIP). Whoa, turns out OTFIP is a problem we regularly deal with for WP Offload S3 as well. This could be a “two birds – one stone” kind of thing. Stars were aligning.

On-the-Fly Image Processing (OTFIP)

We wrote about OTFIP just a few months ago and explained that it tends to cause more problems than it solves. To recap, the main problem OTFIP solves is to cut down on generating (and storing) images that will never be used. If each theme/plugin uses add_image_size() to generate the image sizes they need, the number of image sizes quickly balloons.

Don’t believe me? We have a little competition going on between team members to spot the WordPress install with the most registered image sizes. So far Ash is winning with 49!

the team compares registered image size scores on Slack

To be clear, WordPress generates 49 images of various sizes when an image is uploaded to that WordPress install. That’s easily a minute of waiting for image generation to complete. I’m going to go out on a limb and guess that a solid 29 of those generated images will never be used and yet, they’re taking up hard drive space. This is an extreme example, but we commonly see WordPress installs with 20 or so registered image sizes.

OTFIP FTW

To solve this problem, some developers include OTFIP libraries in their themes/plugins and generate images of the exact size they need when they need it. For example, if you used the popular Aqua Resizer library you might call the following function in your theme/plugin:

<?php $resized_image_url = aq_resize( $image_url, $width, $height, $crop ); ?>

If the resized image doesn’t exist yet, it would be generated and saved alongside the original in the filesystem. This sounds great, but the page load time suffers greatly because it has to wait for the image to be generated before loading the page. And if a page has a bunch of these calls (e.g. one for every blog post on an archive page) it can be a ridiculously long wait and possibly even time out. Plus, image resizing is a CPU and memory intensive operation. So what happens to the server when several of these pages are loaded at the same time? Oops.

Finally, guess what happens when the original image is removed from the Media Library. Yep, the resized image is left behind because WordPress core doesn’t know about it. Plus WordPress core can’t add the resized images to the srcset so no responsive images.

What we need is a solution that doesn’t affect page load time, is mindful of server resources, and works seamlessly with WordPress core as it is today.

Image Processing Queue is Born

At WordCamp US Contributor Day this past December, I decided to try hack together an image processing library that would have all of the benefits of OTFIP but with none of the negatives. And voilà, after a few hours Image Processing Queue was born.

Using it is pretty straightforward. You simply include a file to load Image Processing Queue and its dependencies:

require_once TEMPLATEPATH . '/image-processing-queue/image-processing-queue.php';

Then you can call the ipq_get_theme_image() function in your theme/plugin templates to output an image:

echo ipq_get_theme_image( $post_id, array(
        array( 600, 600, false ),
        array( 1280, 1280, false ),
        array( 1600, 1600, false ),
    ),
    array(
        'class' => 'header-banner'
    )
);

If all the images have already been generated, the output would look something like this:

<img class="alignnone size-full wp-image-22495 header-banner"
width="4096" height="3072" src="https://domain.com/wp-content/uploads/2017/03/image.jpg" 
srcset="https://domain.com/wp-content/uploads/2017/03/image-600x450.jpg 600w,
https://domain.com/wp-content/uploads/2017/03/image-1280x960.jpg 1280w,
https://domain.com/wp-content/uploads/2017/03/image-1600x1200.jpg 1600w,
https://domain.com/wp-content/uploads/2017/03/image-300x225.jpg 300w,
https://domain.com/wp-content/uploads/2017/03/image-1024x768.jpg 1024w"
sizes="(max-width: 720px) 100vw, 720px">

If an image hasn’t been generated yet, it is left out and queued to be generated in the background. So if it’s the first time running the above function call, you might end up with this output:

<img class="alignnone size-full wp-image-22495 header-banner"
width="4096" height="3072" src="https://domain.com/wp-content/uploads/2017/03/image.jpg" 
srcset="https://domain.com/wp-content/uploads/2017/03/image-300x225.jpg 300w,
https://domain.com/wp-content/uploads/2017/03/image-1024x768.jpg 1024w"
sizes="(max-width: 720px) 100vw, 720px">

Regardless of whether the images exist or not, we load the page right away and relegate image generation to the background. No holding up the page load for generating images.

Also, when a new image size is generated, we add metadata to the attachment so that WordPress core understands that the resized image exists. This allows WordPress core to automatically add the generated image sizes to the srcset when generating the image tag. It also allows WordPress core to delete the generated images when the original image is removed from the Media Library. The meta data is in the exact same format as the meta data added for image sizes registered with add_image_size(). Any themes/plugins that use the image size metadata can also use it and therefore use the generated image sizes.

Next Stop, WordPress Core

I would love to see this rolled into WordPress core along with the background processing library. It would make images much easier to manage for WordPress theme/plugin developers and would hopefully result in the abandonment of OTFIP libraries altogether. I’m not sure how something like this would fit into WordPress core’s new release cycles, but I am sure I will continue bugging people about it.

If you’re a theme/plugin author, please consider using Image Processing Queue instead of an OTFIP library next time. You get all the benefits without any of the negatives. And the more this library is used, the easier the case can be made for rolling it into WordPress core.

Of course there is still lots of room for improvement here. Ash is currently working on the background processing library, adding the ability to disable the HTTP worker and spawn multiple queue workers via the CLI. This is great for people with full control of their servers who want more dependable workers. He’s also rebuilding the queue backend, saving jobs to the database by default, but allowing you to swap in any queue backend. For example, you could use Redis to store your jobs.

If you find a bug in either Image Processing Queue or WP Background Processing, please open an issue (or a pull request) on the appropriate GitHub repo.

Have you used OTFIP libraries? Would you give Image Processing Queue a shot instead? What about background processing? What would you like to see happen there? Let us know in the comments.

Update 2017-03-08: I’ve added this library as a plugin on WordPress.org.

About the Author

Brad Touesnard Founder & CEO

As founder of Delicious Brains Inc, Brad has worn many hats. He now spends most of his time managing the product teams and growing the business. Before starting this company, Brad was a freelance web developer, specializing in front-end development.