How to do Background Processing in WordPress Plugins and Themes

wp-background-processing

As WordPress sites become increasingly more complex, so do the tasks that they need to perform which can often be resource intensive or time consuming. I’m talking specifically about tasks which do not need to be actioned instantly for the user’s request to complete, for example sending emails, API calls or database queries. These tasks should never impede your users, as the last thing you want is for them to be staring at a loading screen. Therefore we need to defer such tasks and process them in the background.

WordPress Cron

For most, the WordPress implementation of cron instantly comes to mind and it’s used in core to overcome such issues. If we go back to WordPress 4.3, Boone explained the need for splitting shared taxonomy terms and the difficulties involved with performing the upgrade procedure. The solution was to utilise wp_schedule_single_event and process small batches of 10 taxonomies per run. This process would repeat every 2 minutes until all taxonomies were processed. Nice!

There is, however, one downside to this approach and that’s efficiency. If we have 1000 items to process and we’re only handling 10 items every 2 minutes, it will take over 3 hours to complete, which may or may not be acceptable depending on the task. This is also assuming that the site receives regular traffic or a true cron job has been configured. Simply upping the batch limit isn’t an option as this would likely timeout on shared hosting environments. So what is the solution?

WP Background Processing

WP Background Processing is a library I have created to overcome the issues with using WordPress cron and introduces a queueing system, similar to those found in PHP frameworks, such as Laravel. WP Background Processing has no dependencies or requirements and will run on PHP 5.2, which makes it ideal for use in commercial plugins. Infact, it’s based on the system we use for WP Offload S3 Pro, which allows us to perform find and replacements on the database without ever affecting the user’s browsing experience.

So how do we create a queue system that’s efficient, reliable and compatible with WordPress’ measly system requirements?

First, we remove the need for batch limits. Each batch will continue to process items while PHP memory remains and the total execution time does not exceed 20 seconds. A large majority of hosts (especially shared hosts) have a maximum execution time of 30 seconds, so the 20 seconds limit allows batches to complete before timing out.

We don’t use WordPress cron to handle the queue workers. Instead we use non-blocking asynchronous requests, which were inspired by the TechCrunch team. When a batch reaches either the PHP memory or timeout limit it saves the queue state and instantly dispatches the next batch. This completely negates the need to wait for cron to tick, which means 9 batches can complete, opposed to just 1 batch using the WordPress cron approach.

Queues are synchronous, meaning only one process will work on the queue at any given time. If you attempt to start another process it will gracefully fail and allow the already running process to complete. This ensures that background processing doesn’t eat up your server resources by spawning multiple queue workers.

To ensure a queue worker is available when queue items exists we automatically schedule health checks using WordPress cron. If for whatever reason the queue worker dies, the health check will restart the queue and it will continue where it left off. These checks are performed every 5 minutes (assuming cron is properly configured or the site receives regular traffic).

Let’s take a look at an example.

WP Background Processing Example

I’ve put together a very simple example plugin, which demonstrates how background processing works. The plugin logs random names followed by Lorem Ipsum, which is generated by making a call to the loripsum.net API. To imitate a long running process it sleeps for 5 seconds before processing the queue item.

Clicking the ‘All Users’ link will push 20 random names to the queue and begin processing immediately.

background-processing-example

Entries will begin to populate the debug.log file, roughly every 5 seconds and will continue to do so until the queue is empty.

debug-log

I’m not going to go over the code in this article, but the README.md file details how simple it is to register new queues.

Further Improvements

WP Background Processing is designed to work with WordPress’ minimum requirements, but there are improvements to be made if you have control over the hosting environment. The current implementation saves queue items to the database, which isn’t the most performant method of storing queue data. Instead you could store the data in memory using the likes of Redis or Memcached. Alternatively, you could push your queue data to a third party service, such as Amazon SQS or Iron.io. All are viable options and could be covered in a future post.

Had a similar processing problem in the past? Let us know how you overcame the issue.

About the Author

Ashley Rich

Ashley is a PHP and JavaScript developer with a fondness for solving complex problems with simple, elegant solutions. He also has a love affair with WordPress and learning new technologies.

  • mat_voce

    Yesterday I actually just wrote up code that takes advantage of TechCruch’s https://github.com/techcrunch/wp-async-task/ to kick off expensive tasks in the background.

    I’ll look into this!

  • Wozn2

    Aaaah, this looks awesome, just the thing I’ve been looking for, for a long running import job.

  • Mte90Net

    A little comparison with the TechCrunc library will be useful to evalute that library

    • The TechCrunch library does’t perform background processing, it simply allows you to fire off non-blocking async requests. WP Background Processing builds upon this by allowing to to queue jobs which will continue to process until the job has completed, which alleviates issues with server memory an time limit constraints.

  • Bjorn Holine

    I noticed the article says it will run on PHP 5.2 but then the example repo on GitHub says it requires PHP 5.4. Is that requirement just specific to other code PHP code in the example plugin?

    • WP Background Processing requires 5.2. Only the example repo requires 5.4.

  • Jonathan

    Great post! I have a woocommerce store that receive price updates everyday in a 50 000+ items file. This is exactly what I need to update the prices in the store.

    Cheers!

  • Paul

    Hi, does it require user interaction to start or can it be launched from a WP Cron?

  • Hi Ashley, is there an easy way to stop a background processing job?

  • Thanks for this post, very useful 🙂

  • solepixel

    Thanks for sharing this Ashley, my background process seems to be working OK, but in your example you’re using an admin menu link, which is clicked by the user/admin whenever it needs to be fired off. I’m struggling with trying to get this to fire off automatically. What’s happening is I’m queueing up about 50,000 tasks and as it’s working through them, somehow another queue is generated of another 50,000 tasks. Since all these options have autoload set to “yes”, it’s killing the memory and throwing a WSOD. I tried stopping it from processing whenever a specific transient is set, and I’m setting that transient every time the batch is created, so it shouldn’t be creating new batches while the transient is there, however sometimes it still does. Not sure why. I posted the issue here: https://github.com/A5hleyRich/wp-background-processing/issues/4 and I’d be happy to send you any code I’m using if it helps figure this out. Thanks!

  • iweczek

    Thanks for this example! Is there a good reason to sleep 5 seconds? If one of the items in queue fails, will it repeat or will the next one go? Finally, if PHP memory and execution time is not a concern (can be adjusted), would this work for running larger tasks that might take 5-10 minutes?

    • As mentioned in the article, the 5 second sleep is to demonstrate a long running process. In the real world you would never use `sleep`.

      It all depends on where the job fails. If it fails before it’s removed from the queue then it will repeat on the next batch.

      Yes this will work for longer running tasks.

  • coffee_stewart

    Use this to fix a broken import! – Lovely job.

  • Hey Ashley,

    Brad mentioned this library in the Post Status chat as WooCommerce 2.6 is now using this apparently, but nice work! We’re using a similar mechanism in MailChimp for WordPress but using WP Cron over HTTP requests (which in turn uses HTTP requests but oh well) and inspired by Beanstalk jobs.

    https://github.com/ibericode/mailchimp-for-wordpress/blob/master/includes/class-queue.php
    https://github.com/ibericode/mailchimp-for-wordpress/blob/master/includes/class-queue-job.php

    Anyway, thanks for sharing!

  • DeryckOE

    Hey Ashley,

    I´m trying to use your plugin to write a Backup Plugin. My issue is I read the entirely file tree (52,743 files) and in wp-background-process.php line 103 when update_site_option try to save that huge array the code trows error because of memory limit. Saving that kind of array in database takes a lot of memory and fail. I would like to know if there is a way to avoid this.

    Thanks for your awesome plugin.

    • Yes, the Array gets too big and is stored as a serialized string in the options database table. Reading that serialized option back which is very large can result in memory issues. Best would be to put a counter in your script and create a batch for each X records. So for example very 1000 records, you save() the process so that it can then generate a new batch. Multiple batches of serialized data options instead of just one. I’m yet to test this out but that should do it.

  • Walter Rice

    Awesome! Works like a charm!

  • Peter Rigby

    Hi, thanks for this it works really well! I’m using it to synchronise data between WooCommerce and a retail catalogue system.