How to do Background Processing in WordPress Plugins and Themes

As WordPress sites become increasingly more complex, so do the tasks that they need to perform which can often be resource intensive or time consuming. I’m talking specifically about tasks which do not need to be actioned instantly for the user’s request to complete, for example sending emails, API calls or database queries. These tasks should never impede your users, as the last thing you want is for them to be staring at a loading screen. Therefore we need to defer such tasks and process them in the background.

WordPress Cron

For most, the WordPress implementation of cron instantly comes to mind and it’s used in core to overcome such issues. If we go back to WordPress 4.3, Boone explained the need for splitting shared taxonomy terms and the difficulties involved with performing the upgrade procedure. The solution was to utilise wp_schedule_single_event and process small batches of 10 taxonomies per run. This process would repeat every 2 minutes until all taxonomies were processed. Nice!

There is, however, one downside to this approach and that’s efficiency. If we have 1000 items to process and we’re only handling 10 items every 2 minutes, it will take over 3 hours to complete, which may or may not be acceptable depending on the task. This is also assuming that the site receives regular traffic or a true cron job has been configured. Simply upping the batch limit isn’t an option as this would likely timeout on shared hosting environments. So what is the solution?

WP Background Processing

WP Background Processing is a library I have created to overcome the issues with using WordPress cron and introduces a queueing system, similar to those found in PHP frameworks, such as Laravel. WP Background Processing has no dependencies or requirements and will run on PHP 5.2, which makes it ideal for use in commercial plugins. Infact, it’s based on the system we use for WP Offload S3 Pro, which allows us to perform find and replacements on the database without ever affecting the user’s browsing experience.

So how do we create a queue system that’s efficient, reliable and compatible with WordPress’ measly system requirements?

First, we remove the need for batch limits. Each batch will continue to process items while PHP memory remains and the total execution time does not exceed 20 seconds. A large majority of hosts (especially shared hosts) have a maximum execution time of 30 seconds, so the 20 seconds limit allows batches to complete before timing out.

We don’t use WordPress cron to handle the queue workers. Instead we use non-blocking asynchronous requests, which were inspired by the TechCrunch team. When a batch reaches either the PHP memory or timeout limit it saves the queue state and instantly dispatches the next batch. This completely negates the need to wait for cron to tick, which means 9 batches can complete, opposed to just 1 batch using the WordPress cron approach.

Queues are synchronous, meaning only one process will work on the queue at any given time. If you attempt to start another process it will gracefully fail and allow the already running process to complete. This ensures that background processing doesn’t eat up your server resources by spawning multiple queue workers.

To ensure a queue worker is available when queue items exists we automatically schedule health checks using WordPress cron. If for whatever reason the queue worker dies, the health check will restart the queue and it will continue where it left off. These checks are performed every 5 minutes (assuming cron is properly configured or the site receives regular traffic).

Let’s take a look at an example.

WP Background Processing Example

I’ve put together a very simple example plugin, which demonstrates how background processing works. The plugin logs random names followed by Lorem Ipsum, which is generated by making a call to the loripsum.net API. To imitate a long running process it sleeps for 5 seconds before processing the queue item.

Clicking the ‘All Users’ link will push 20 random names to the queue and begin processing immediately.

background-processing-example

Entries will begin to populate the debug.log file, roughly every 5 seconds and will continue to do so until the queue is empty.

debug-log

I’m not going to go over the code in this article, but the README.md file details how simple it is to register new queues.

Further Improvements

WP Background Processing is designed to work with WordPress’ minimum requirements, but there are improvements to be made if you have control over the hosting environment. The current implementation saves queue items to the database, which isn’t the most performant method of storing queue data. Instead you could store the data in memory using the likes of Redis or Memcached. Alternatively, you could push your queue data to a third party service, such as Amazon SQS or Iron.io. All are viable options and could be covered in a future post.

Had a similar processing problem in the past? Let us know how you overcame the issue.

About the Author

Ashley Rich

Ashley is a PHP and JavaScript developer with a fondness for solving complex problems with simple, elegant solutions. He also has a love affair with WordPress and learning new technologies.

  • mat_voce

    Yesterday I actually just wrote up code that takes advantage of TechCruch’s https://github.com/techcrunch/wp-async-task/ to kick off expensive tasks in the background.

    I’ll look into this!

  • Wozn2

    Aaaah, this looks awesome, just the thing I’ve been looking for, for a long running import job.

  • Mte90Net

    A little comparison with the TechCrunc library will be useful to evalute that library

    • The TechCrunch library does’t perform background processing, it simply allows you to fire off non-blocking async requests. WP Background Processing builds upon this by allowing to to queue jobs which will continue to process until the job has completed, which alleviates issues with server memory an time limit constraints.

  • Bjorn Holine

    I noticed the article says it will run on PHP 5.2 but then the example repo on GitHub says it requires PHP 5.4. Is that requirement just specific to other code PHP code in the example plugin?

    • WP Background Processing requires 5.2. Only the example repo requires 5.4.

  • Jonathan

    Great post! I have a woocommerce store that receive price updates everyday in a 50 000+ items file. This is exactly what I need to update the prices in the store.

    Cheers!

  • Paul

    Hi, does it require user interaction to start or can it be launched from a WP Cron?

  • Hi Ashley, is there an easy way to stop a background processing job?

  • Thanks for this post, very useful 🙂

  • solepixel

    Thanks for sharing this Ashley, my background process seems to be working OK, but in your example you’re using an admin menu link, which is clicked by the user/admin whenever it needs to be fired off. I’m struggling with trying to get this to fire off automatically. What’s happening is I’m queueing up about 50,000 tasks and as it’s working through them, somehow another queue is generated of another 50,000 tasks. Since all these options have autoload set to “yes”, it’s killing the memory and throwing a WSOD. I tried stopping it from processing whenever a specific transient is set, and I’m setting that transient every time the batch is created, so it shouldn’t be creating new batches while the transient is there, however sometimes it still does. Not sure why. I posted the issue here: https://github.com/A5hleyRich/wp-background-processing/issues/4 and I’d be happy to send you any code I’m using if it helps figure this out. Thanks!

  • iweczek

    Thanks for this example! Is there a good reason to sleep 5 seconds? If one of the items in queue fails, will it repeat or will the next one go? Finally, if PHP memory and execution time is not a concern (can be adjusted), would this work for running larger tasks that might take 5-10 minutes?

    • As mentioned in the article, the 5 second sleep is to demonstrate a long running process. In the real world you would never use `sleep`.

      It all depends on where the job fails. If it fails before it’s removed from the queue then it will repeat on the next batch.

      Yes this will work for longer running tasks.

  • coffee_stewart

    Use this to fix a broken import! – Lovely job.

  • Hey Ashley,

    Brad mentioned this library in the Post Status chat as WooCommerce 2.6 is now using this apparently, but nice work! We’re using a similar mechanism in MailChimp for WordPress but using WP Cron over HTTP requests (which in turn uses HTTP requests but oh well) and inspired by Beanstalk jobs.

    https://github.com/ibericode/mailchimp-for-wordpress/blob/master/includes/class-queue.php
    https://github.com/ibericode/mailchimp-for-wordpress/blob/master/includes/class-queue-job.php

    Anyway, thanks for sharing!

  • DeryckOE

    Hey Ashley,

    I´m trying to use your plugin to write a Backup Plugin. My issue is I read the entirely file tree (52,743 files) and in wp-background-process.php line 103 when update_site_option try to save that huge array the code trows error because of memory limit. Saving that kind of array in database takes a lot of memory and fail. I would like to know if there is a way to avoid this.

    Thanks for your awesome plugin.

    • Yes, the Array gets too big and is stored as a serialized string in the options database table. Reading that serialized option back which is very large can result in memory issues. Best would be to put a counter in your script and create a batch for each X records. So for example very 1000 records, you save() the process so that it can then generate a new batch. Multiple batches of serialized data options instead of just one. I’m yet to test this out but that should do it.

  • Walter Rice

    Awesome! Works like a charm!

  • Peter Rigby

    Hi, thanks for this it works really well! I’m using it to synchronise data between WooCommerce and a retail catalogue system.

  • Donatas Jonas Stirbys

    Ashley,

    I am looking for a solution to keep something like image uploads to S3 in the background, but I guess your approach does not best fits me?

  • Hey Ashley,

    Thank you for your contribution. That save my day. 1+

  • Leonard Cremer

    Hey Ashley

    I am trying to get this to work on my WordPress installation but it does not seem to work.

    Apart from installing the two plugins do I need to do anything else?

    I can activate both options in the Process menu and it reloads the page. I do not see anything in the log file though?

    Is it maybe a configuration issue?

    Thanks

    • WP Offload S3 uses wp_remote_post() to make requests to your site at http://example.com/wp-admin/admin-ajax.php. The server hosting the site needs to be able to resolve the hostname of your site. If you can’t ping your site’s hostname from the server, WP Offload S3 will not be able to dispatch background processes.

  • oluomotoso

    Thanks for a good article and a solution to a serious problem. I’m actually trying to use the plugin to manage social media update when a post is first published. But I can’t get it to work somehow. checked the database and noticed the job is inside the sitemeta table but the job is not getting done. Is there anyway I can debug this or sth. Regards

    • Eddy Rguez

      Hi, Im facing the same issue…did you figure it out?

      • oluomotoso

        Yes, I used a custom developed plugin. I actually had to include the file in my plugin. And it was getting done.

  • Eddy Rguez

    Hi Ashley,

    Im trying yo use your lib to sync data from WP to external system but I could not make it work…im only trying to write to logs…

    require_once plugin_dir_path( __FILE__ ) . ‘classes/wp-async-request.php’;
    require_once plugin_dir_path( __FILE__ ) . ‘classes/wp-background-process.php’;

    class EE_sfSDK extends WP_Background_Process {
    protected $action = ‘attendees_background_process’;
    protected function task( $registration ) {
    //$this->really_long_running_task();
    error_log(“This is the task RUNNING of the WP_Background_Process”);
    return false;
    }
    protected function complete() {
    parent::complete();
    // Show notice to user or perform some other arbitrary task…
    error_log(“This is the task COMPLETED WP_Background_Process”);
    }

    the elements are push correctly inside the queue:

    foreach ($registrations as $registration) {
    if ($registration instanceof EE_Registration) {
    error_log(“Event: ext_sync_to_salesforce push_to_queue!!”);
    $sfSDK->push_to_queue($registration->ID());
    }
    }
    $sfSDK->save()->dispatch();

    the WP table options have a new row:

    wp_attendees_background_process_batch_1f8a9447c3ecec7570a4de37af

    with the values…

    a:7:{i:0;i:2664;i:1;i:2692;i:2;i:2693;i:3;i:2714;i:4;i:2729;i:5;i:2769;i:6;i:2770;}

    but looks like the server is never dispatching the BG process….

    there is any configuration missing to make it work?

    thanks

  • mfacer

    Hi – I have the code *nearly* working.. I can see the request in wp_options, but nothing seems to be being triggered.

    (I include the background processing files prior to this obviously)


    $example_process = new my_background_task();
    $data = array('var1' => 'matt');
    $example_process->push_to_queue( $data );
    $example_process->save()->dispatch();

    my_background_task looks like this:


    class my_background_task extends WP_Background_Process {

    //protected $action = 'example_process';

    protected function task( $data ) {
    error_log("test");
    return false;
    }

    protected function complete() {
    parent::complete();
    // Show notice to user or perform some other arbitrary task...
    }

    }

    I can see in wp_options, the following:

    option name = wp_background_process_batch_a869e7889dfa7af3f601442869170f54
    option value = a:1:{i:0;a:1:{s:4:”var1″;s:4:”matt”;}}

    I would expect to see “hello” in the log should it have triggered. I checked that the cron is enabled… but not sure what else to do. It just doesn’t seem to be triggering? Thanks

    • Eddy Rguez
      • Are you certain the site is able to reach itself at http://domain.com/wp-admin/admin-ajax.php? We’ve seen this in the past where the site is unable to resolve it’s own hostname.

        • Eddy Rguez

          yes, the site it is able to resolve the hostname…but response is coming empty…here are my logs:

          [14-Jul-2017 14:31:04 UTC] insise dispatch() URL =
          [14-Jul-2017 14:31:04 UTC] http://f35.7dd.myftpupload.com/wp-admin/admin-ajax.php?action=wp_attendees_background_process&nonce=ede86c53a0
          [14-Jul-2017 14:31:04 UTC] insise dispatch() ARGS =
          [14-Jul-2017 14:31:04 UTC] Array
          (
          [timeout] => 0.01
          [blocking] =>
          [body] => Array
          (
          [0] => 1561
          [1] => 1574
          [2] => 1599
          )

          [cookies] => Array
          (
          [wordpress_4ce30adadb1eb1d7a95f8acd51da4d2b] => erdguez|1500045528|zJArXvCzKYfUnsITPcdz2WmT21qGn6mUQkm2ye5zD14|4ba96bca3b19648cdd057fbbea3596ff0844a287d5a7b2a6159ae2ef8d6ce6bb
          [PHPSESSID] => as9d3e4l194bu31626ro85kju3
          [wp-settings-time-33] => 1496678727
          [wp-settings-29] => libraryContent=browse&editor=tinymce
          [wp-settings-time-29] => 1497629410
          [wordpress_test_cookie] => WP Cookie check
          [bp_ut_session] => {-q-pageviews-q-:1-c–q-referrer-q-:-q–q–c–q-landingPage-q-:-q-http://f35.7dd.myftpupload.com/login/?redirect_to=http%3A%2F%2Ff35.7dd.myftpupload.com%2Fwp-admin%2F&reauth=1-q–c–q-started-q-:1499872720288}
          [_ga] => GA1.2.1288330373.1496408869
          [wordpress_logged_in_4ce30adadb1eb1d7a95f8acd51da4d2b] => erdguez|1500045528|zJArXvCzKYfUnsITPcdz2WmT21qGn6mUQkm2ye5zD14|fb9cd212ed476140aabaa3b5858e859fb7457b363b3821e130e740a6cce1b65c
          )

          [sslverify] =>
          )

          [14-Jul-2017 14:31:05 UTC] insise dispatch() RESPONSE =
          [14-Jul-2017 14:31:05 UTC] Array
          (
          [headers] => Array
          (
          )

          [body] =>
          [response] => Array
          (
          [code] =>
          [message] =>
          )

          [cookies] => Array
          (
          )

          [http_response] =>
          )

        • Eddy Rguez

          The constructor $sfSDKBackgroundProccess = new EE_sfSDK(); can be done anywhere in the code?

          Im reading it will only work if is constructed at the right time:

          https://github.com/A5hleyRich/wp-background-processing/issues/35

          this global var is not working for me…

          any suggestion to make it work without the use of a global var?

  • AlanP2

    When I activate the example plugin I get ‘Fatal error: Class ‘WP_Async_Request’ not found’.