As WordPress sites become increasingly more complex, so do the tasks that they need to perform which can often be resource intensive or time consuming. I’m talking specifically about tasks which do not need to be actioned instantly for the user’s request to complete, for example sending emails, API calls or database queries. These tasks should never impede your users, as the last thing you want is for them to be staring at a loading screen. Therefore we need to defer such tasks and process them in the background.

WordPress Cron

For most, the WordPress implementation of cron instantly comes to mind and it’s used in core to overcome such issues. If we go back to WordPress 4.3, Boone explained the need for splitting shared taxonomy terms and the difficulties involved with performing the upgrade procedure. The solution was to utilise wp_schedule_single_event and process small batches of 10 taxonomies per run. This process would repeat every 2 minutes until all taxonomies were processed. Nice!

There is, however, one downside to this approach and that’s efficiency. If we have 1000 items to process and we’re only handling 10 items every 2 minutes, it will take over 3 hours to complete, which may or may not be acceptable depending on the task. This is also assuming that the site receives regular traffic or a true cron job has been configured. Simply upping the batch limit isn’t an option as this would likely timeout on shared hosting environments. So what is the solution?

WP Background Processing

WP Background Processing is a library I have created to overcome the issues with using WordPress cron and introduces a queueing system, similar to those found in PHP frameworks, such as Laravel. WP Background Processing has no dependencies or requirements and will run on PHP 5.2, which makes it ideal for use in commercial plugins. Infact, it’s based on the system we use for WP Offload S3 Pro, which allows us to perform find and replacements on the database without ever affecting the user’s browsing experience.

So how do we create a queue system that’s efficient, reliable and compatible with WordPress’ measly system requirements?

First, we remove the need for batch limits. Each batch will continue to process items while PHP memory remains and the total execution time does not exceed 20 seconds. A large majority of hosts (especially shared hosts) have a maximum execution time of 30 seconds, so the 20 seconds limit allows batches to complete before timing out.

We don’t use WordPress cron to handle the queue workers. Instead we use non-blocking asynchronous requests, which were inspired by the TechCrunch team. When a batch reaches either the PHP memory or timeout limit it saves the queue state and instantly dispatches the next batch. This completely negates the need to wait for cron to tick, which means 9 batches can complete, opposed to just 1 batch using the WordPress cron approach.

Queues are synchronous, meaning only one process will work on the queue at any given time. If you attempt to start another process it will gracefully fail and allow the already running process to complete. This ensures that background processing doesn’t eat up your server resources by spawning multiple queue workers.

To ensure a queue worker is available when queue items exists we automatically schedule health checks using WordPress cron. If for whatever reason the queue worker dies, the health check will restart the queue and it will continue where it left off. These checks are performed every 5 minutes (assuming cron is properly configured or the site receives regular traffic).

Let’s take a look at an example.

WP Background Processing Example

I’ve put together a very simple example plugin, which demonstrates how background processing works. The plugin logs random names followed by Lorem Ipsum, which is generated by making a call to the loripsum.net API. To imitate a long running process it sleeps for 5 seconds before processing the queue item.

Clicking the ‘All Users’ link will push 20 random names to the queue and begin processing immediately.

Entries will begin to populate the debug.log file, roughly every 5 seconds and will continue to do so until the queue is empty.

I’m not going to go over the code in this article, but the README.md file details how simple it is to register new queues.

Further Improvements

WP Background Processing is designed to work with WordPress’ minimum requirements, but there are improvements to be made if you have control over the hosting environment. The current implementation saves queue items to the database, which isn’t the most performant method of storing queue data. Instead you could store the data in memory using the likes of Redis or Memcached. Alternatively, you could push your queue data to a third party service, such as Amazon SQS or Iron.io. All are viable options and could be covered in a future post.

Had a similar processing problem in the past? Let us know how you overcame the issue.