Today we released WP Offload Media 2.3! 🎉

This release has some very important changes in order to support WordPress 5.3, introduces a custom table for the plugin’s data, as well as a plethora of enhancements and bug fixes.

WordPress 5.3 Support

After a relatively quiet period for WordPress core’s media handling, intrepid souls such as Andrew Ozz and Mike Schroder have taken up the gauntlet and spearheaded renewed development efforts.

These changes are long overdue and very welcome, but do mean plugins that work closely with the WordPress Media Library may need development in order to stay compatible.

WordPress 5.3 has made some critical changes to how it handles the uploading of new Media Library items that severely impacts WP Offload Media and other plugins that handle images.

In particular, if you are using WP Offload Media’s “Remove Files From Server” option, then it is very important that you update to WP Offload Media 2.3 before adding new media to WordPress 5.3 in order to avoid failed uploads.

Also, if you intend to upload any images to WordPress 5.3 that are either larger than 2560 pixels in width for example, or are from a phone or other device that does not natively rotate its captured photos to portrait, then again, you must upgrade to WP Offload Media 2.3.

New Upload Process in WordPress 5.3

While on the surface nothing has changed in how you upload media to the WordPress Media Library, under the hood a lot has changed. WordPress 5.3 tries to reduce the likelihood of having to remove and re-upload a new Media Library item due to failures during the generation of sub-sizes (thumbnails).

As part of this change the wp_update_attachment_metadata function is now called many times to save progress before and during the generation of the sub-sizes, rather than once on completion as it was before.

This wp_update_attachment_metadata function fires a filter that WP Offload Media has to use to recognize that a Media Library item has been changed, so this change in usage means WP Offload Media 2.2.1 and earlier may attempt to offload the new files many times, and if “Remove Files From Server” is turned on may stop WordPress 5.3 from being able to generate its sub-sizes.

WP Offload Media 2.3 fixes the problem by waiting until wp_update_attachment_metadata includes semi-complete data it can work with, and by keeping track of what has already been offloaded during the PHP request so as to not duplicate effort.

“-scaled” & “-rotated” Images

There is improved support for large images and images that need rotating in WordPress 5.3. Unfortunately, the scaled and rotated images are renamed but their associated sub-size files have different base names. This breaks how WP Offload Media recognizes the relationship between them under certain circumstances.

It’s an unfortunate fact that many plugins that create custom content do not use the standard WordPress functions for generating the HTML for including images from the Media Library. This means WP Offload Media often has to reverse engineer the local URLs included in content to rewrite them to the required bucket or CDN URL.

Before WordPress5.3, if a URL was for an image sub-size, say “2019/11/cat-1024×768.png”, WP Offload Media could reliably deduce that the original file was named “2019/11/cat.png” and search the database for a matching Media Library item to get its offload data.

With WordPress 5.3 this relationship between sub-size file names and their original can no longer be relied upon due to the way large or rotated image names are generated.

For example, if that “cat.png” file happens to be wider than the default width threshold of 2560px, WordPress 5.3 will generate a new full size file that is scaled down to 2560px wide and renamed “cat-scaled.png”. To keep the quality as high as possible, the original “cat.png” is used to generate sub-sizes and then it is squirreled away for safekeeping. This unfortunately means we get a sub-size such as “2019/11/cat-1024×768.png” who’s original in the database is now called “2019/11/cat-scaled.png”.

This makes it very difficult for plugins like WP Offload Media to reverse engineer “2019/11/cat-1024×768.png” back to “2019/11/cat-scaled.png” as they no longer share a common base file name when the size information is stripped from the sub-size. This is made doubly difficult as WordPress 5.3 does not currently fully protect from having two Media Library items sharing the “-scaled” or “-rotated” full file name.

It took many long days and sleepless nights, but WP Offload Media 2.3 handles the broken relationship between sub-sizes and their full size file name, and also improves the renaming of both original upload and their “-rotated” or “-scaled” files if it detects an offloaded file that could clash, whether removed from the server or not.

Custom Table

Before the changes required for WordPress 5.3 became apparent, the “Big New Thing” for WP Offload Media 2.3 was its new custom table for storing information about offloaded media.

This new custom table makes lookups for offload data related to a Media Library item much faster, especially when reverse engineering a local URL as discussed above.

It was fortuitous that we were already pretty much done with the work for the custom table before we learned of the problems introduced with WordPress 5.3. Having the custom table seriously helped us with implementing WordPress 5.3 support.

Along with the custom table comes some significant refactoring and the introduction of a couple of new PHP classes in the codebase that are passed into a couple of filters that originally took the old array of “amazonS3_info” data. So if you’re a plugin author with WP Offload Media integration, or have custom code that uses some of our filters, please check the WP Offload Media Tweaks plugin for some important changes.

The two most important changed filters that we often see other plugins use are as3cf_pre_update_attachment_metadata and as3cf_get_attachment_url. They both now take an instance of DeliciousBrains\WP_Offload_Media\Items\Media_Library_Item, where previously they took an associative array.

Upgrading

When you first upgrade to WP Offload Media 2.3 it will create the new custom table and then run a background upgrade routine that moves the old “amazonS3_info” postmeta records across in small batches.

Because some WordPress sites are running on severely resource restrained servers, by default the upgrade routine only processes 500 Media Library items every two minutes (assuming WordPress cron is being tickled as it should every minute).

If you have a lot of offloaded Media Library items, and an appropriately provisioned server, you may want to speed up the migration. If so, please check out the as3cf_update_as3cf_items_table_interval, as3cf_update_as3cf_items_table_batch_size and as3cf_update_as3cf_items_table_time_limit filters in the WP Offload Media Tweaks plugin.

For example, I tested migrating 155,000+ offloaded Media Library items in a multisite install with 1,234 subsites on a fairly resource restrained server with the interval set to 1 minute and batch size of 1,000 items. Predictably, it finished in approx 2 hours, I probably could have got away with a batch size of 2,000 to halve that time.

During the migration the bulk offload, download and remove from bucket/server tools are locked out. However, you’ll still be able to add new items to the Media Library and they’ll be offloaded as usual with data stored in the new table.

Media URLs in content will continue to be properly rewritten to their bucket or CDN URLs during the brief upgrade period as there is specific code that falls back to the legacy data if it has yet to be migrated. This includes WooCommerce and Easy Digital Downloads product files.

For media URLs that require reverse engineering as discussed above, any existing post or page will simply use WP Offload Media’s existing URL cache mechanism that maps previously rewritten local URLs to their bucket or CDN version. Not coincidentally, WP Offload Media’s built-in URL cache is faster and more reliable in WP Offload Media 2.3. There should be no broken images on your site while this upgrade is running.

There is no way of downgrading from WP Offload Media 2.3 back to 2.2.1 or earlier. So it probably goes without saying, but I’m going to anyway, please please please backup your site’s database and files before upgrading AND do a dry-run in a development or staging environment before upgrading production!

In fact, you can ensure there is minimal chance of disruption on a very large site by upgrading WP Offload Media on a fresh clone of the production site first. We’ve built the upgrade routine so that if you upgrade the clone first, and then migrate the resultant as3cf_items table back to the production database before upgrading the plugin there, WP Offload Media will migrate any newly offloaded media to the new table first and then simply clean up any legacy metadata it finds is already migrated to the new table. I heartily recommend using a plugin such as WP Migrate DB Pro for the database cloning and table migration! 😉

Other Changes

As always, there are other new features, improvements and bug fixes to be had in a new release of WP Offload Media, you can see them all in the changelog.

Next Up

The custom table was a direct response to some of the feedback we’ve had about how our customers use WP Offload Media. The next release of WP Offload Media will be addressing some frequently requested changes we’ve been receiving from customers, especially in the set up of buckets and the CDN that fronts them.