After nearly 6 months of development I’m pleased to announce the release of WP Offload S3 1.2. For now you’ll need to download it from My Account. Updates through the WordPress dashboard will be enabled soon, hopefully next week.
In this iteration of the plugin we have made some significant under-the-hood improvements, which fundamentally changes how S3 URLs are handled. These changes should make WP Offload S3 more reliable and less of a burden on server resources. Before I go onto discuss the changes, let me explain how WP Offload S3 used to handle S3 URLs and the problems with that implementation.
The Old Implementation
It’s relatively simple. When you inserted an attachment into post content, the S3 URL was added to the editor and saved to the database. However, storing S3 URLs in the database presented a few problems:
- What happens if you want to change the URL format? For example, if you decided to serve your Media Library from CloudFront as opposed to S3.
- If you uninstalled WP Offload S3 and deleted your S3 bucket what happens to existing content?
- What about private attachments? If signed S3 URLs expire after 15 minutes and they were embedded directly into post content they would cease to work once the expiry was reached.
Find and Replace
We could solve problems 1 and 2 easily enough by performing a find and replace on your content. You may have seen the find and replace prompt when changing your URL settings or deactivating the plugin. On small to medium size sites this process is relatively quick and light on server resources. But, things start to get hairy when sites have thousands of posts and/or attachments. That’s before you take into consideration intermediate image sizes and post revisions. Consider the following example:
A site has 100,000 posts, each with 20 revisions and 50,000 attachments. Each attachment has 6 intermediate image sizes. If you do the math that’s 350,000 REPLACE queries that needed to be executed over a data set of 2,000,000 posts and revisions. That many queries is going to bring most servers to halt, or, cause a significant CPU spike.
We could alleviate some of the performance issues by using background processing, but batch processing that many queries takes time. Having to wait 8 hours for changes to fully take effect after you update your URL settings isn’t exactly a great experience.
Handling private attachments is a bit more complicated. Since they expire, you can no longer just store the signed S3 URLs in the database. Our initial idea was to introduce a shortcode, something similar to:
That would allow a new signed URL to be generated on each page request, thus fixing the issue. But, having to use a mix of URLs and shortcodes to output attachments seemed dirty. It would also be a nightmare if you decided to remove WP Offload S3, because you couldn’t do a simple search and replace on those shortcodes.
The New Implementation
In version 1.2 of WP Offload S3 we no longer save the S3 URL to the database. When you insert an attachment into your post content, the S3 URL is added to the editor as before, but when the post is saved we filter the content and ensure only the server URL is saved to the database. We do the opposite when the post is requested by swapping out the server URL for the S3 URL.
This completely removes the need for performing a find and replace on the database. If you change your URL settings, the changes will take full effect instantly across all posts. The same goes for when WP Offload S3 is removed. Because we only store the server URL in the database, deactivating or removing WP Offload S3 will be fine so long as your files haven’t been removed from your server. Private URLs are no longer an issue either, because a new signed URL is generated on every page request.
You may be wondering what impact this has on your site load times, after all the processing is no longer done in the background, but on each individual page request. Let’s take a look at a before and after using Query Monitor.
WP Offload S3 deactivated, URLs pointing to the server:
WP Offload S3 activated, URLs rewritten to S3:
Just 1 extra query and a negligible increase in load time, which we’re pretty pleased with! We managed to keep the content filtering process relatively light due to piggybacking onto responsive images, which was introduced in WordPress 4.4.
These tests were performed using a fresh install of WordPress 4.6.1 on PHP 7. The results were taken from the blog index, which consisted of 10 posts, each with 2 image attachments.
When you upgrade to WP Offload S3 1.2 a new upgrade procedure will kick off, which will trawl through your post content and replace S3 URLs with their server counterpart. Wait, but what about…
That many queries is going to bring most servers to halt, or, at best cause a significant CPU spike.
Don’t worry. We have greatly improved how the find and replace works and it now consumes very little CPU. The upgrade routine will run silently in the background via WordPress cron and you can check on its progress from the WP Offload S3 settings screen.
This release also adds support for custom logos and contains a good amount of bug fixes. Check out the changelog for more details.
We a have lot lined up for WP Offload S3, so stay tuned!