Add Metadata Tool: For When Your Media is Already in a Bucket

If you already have a cloud storage bucket with your site’s media in it, you probably don’t want to have to upload it all again via WP Offload Media so that it works with your site. There are a few reasons why you might be in this situation:

• The site has been migrated from a hosting service (e.g. WP Engine) that synchronized media files to a storage provider bucket
• Switching to WP Offload Media from another plugin that offloaded the Media Library items
• A command line tool has been used to push uploaded media files from the server to the bucket

With the release of WP Offload Media 2.5 came the ability to tell WP Offload Media that Media Library items have already been offloaded to the bucket by other means.

When run, the tool creates WP Offload Media metadata in the WordPress database for all Media Library items that have yet to be offloaded. The metadata is created using the current Storage Provider settings in WP Offload Media. It uses the provider, bucket and public path, but not the Object Versioning setting.

Once new offload metadata has been added to a Media Library item, it will instantly be treated as if it had been offloaded by WP Offload Media and its local URLs in your content will be rewritten according to your Delivery Provider settings in WP Offload Media.

WARNING: When Not to Use this Tool

There are a number of situations where this tool is not helpful and should not be used:

• It should not be used to add metadata to Media Library items that have an Object Versioning string in their bucket path in cloud storage
• It should not be used to add metadata to Media Library items when the bucket objects are expected to be private and require signed URLs, only public object paths are constructed
• It can not be used to import new media as this tool does not create Media Library item records and does not scan the bucket for new objects

Enabling the Add Metadata Tool

As this tool should only be used in very specific situations, it is not readily available in WP Offload Media’s Tools tab like most other tools. It must be explicitly enabled.

You can enable the Add Metadata tool with the following define in your wp-config.php.

define( 'AS3CF_SHOW_ADD_METADATA_TOOL', true );

Once enabled the Add Metadata tool will be shown in the Tools tab.

Running the Add Metadata Tool

Before running the Add Metadata tool please ensure that the settings seen in the Storage Settings panel for Storage Provider, Bucket, Add Prefix to Bucket Path and Add Year & Month to Bucket Path match the location of your bucket and the path of your existing media files in the bucket.

If the site has been set up to use Object Versioning for normal offloads then that may be kept turned on, but the Add Metadata tool will ignore that setting.

When the Add Metadata tool is run, any Media Library item that has not been offloaded will be analyzed and a new as3cf_items database table record will be created for it with provider and bucket fields set from the current settings. The path will be derived from the item’s upload date and file name in unison with the Add Prefix to Bucket Path and Add Year & Month to Bucket Path settings.

IMPORTANT: As soon as a metadata record has been added to a Media Library item, its URLs will start being rewritten using the settings seen in the Delivery Settings panel. If the media item is missing from the bucket, any rewritten URLs for it in your content will be broken, so we highly recommend running the Verify New Metadata tool after the Add Metadata tool has finished running.

Clicking the Add Metadata button will start the background process of looking for Media Library items that have yet to be offloaded and adding metadata to them so they become offloaded.

Because the operation happens completely on the server, the whole process is very quick and usually finishes within a few minutes, even with thousands of Media Library items.

After the Add Metadata tool has been run, two new links will appear in the tool’s panel:

Verify The New Metadata

The Add Metadata tool adds new metadata records according to your settings without checking if the files already exist in the bucket. There are a few reasons why files might be missing in the bucket:

• WP Offload Media’s Storage Settings didn’t match for the Storage Provider, Bucket or Add Prefix to Bucket Path and Add Year & Month to Bucket Path combination (Key) for the associated objects
• Some objects have been removed from the bucket
• Some newer Media Library item’s files have simply never been uploaded to the bucket
• The original image file was added to the bucket but not the scaled or rotated “full size” version generated by recent versions of WordPress

For this reason WP Offload Media adds a “Verified: No” entry to the “Offload” metabox shown for Media Library items that have new offload metadata created with this tool:

The “Verified: No” entry is an indicator that while WP Offload Media is treating the Media Library item as being offloaded, it has not checked that the expected objects exist in the bucket.

Click the Find items with files missing in bucket and remove metadata, mark others verified link to run the Verify New Metadata tool to check every unverified metadata record created with the Add Metadata tool.

This kicks off a background process that checks for the existence of the full size media file and original image in the bucket.

If there is a large number of unverified metadata records to check, the background process could take a long time to run as it needs to make API requests to the storage provider to verify the existence of every single file that is expected to be in the bucket.

If either the full size file or original image exists at their expected path in the bucket, then the metadata record is deemed good enough to be used and is set as “Verified”, which hides the “Verified: No” line from the item’s “Offload” metabox.

If the Media Library item is a scaled or rotated image and therefore WordPress expects there to be both a full size image file with “-scaled” or “-rotated” in its name, and an original image file, and one of them is missing, then a notice about the Media Library item needing its thumbnails regenerating will be logged.

If the Media Library item is an image and its full size media file and its original image exists in the bucket, then the tool will also check whether all its expected thumbnail sizes exist in the bucket, and again, will log an error notice if any are missing.

If the full size media file and original image are not found in the bucket then the unverified metadata record is deemed unusable and is silently removed, returning the Media Library item to the same state as before the Add Metadata tool was run.

Error Messages

After running the Verify New Metadata tool you may be presented with a notice similar to the following:

You can click the linked item ID to visit the affected Media Library item’s details page and fix the problem.

The most common errors will be thumbnail image files missing from the bucket. If there are a lot of error messages (the tool stops logging messages after the 100th error), then you’ll likely want to run a bulk Regenerate Thumbnails.

Removing The New Metadata

The Remove metadata added by the Add Metadata tool link can be used to remove all new metadata records that have been created by the Add Metadata tool.

When this link is clicked all records in the as3cf_items database table created by the Add Metadata tool will be deleted regardless of whether they have been verified or not.

It may be useful to run this tool if your Storage Settings were not correctly configured to match up with the bucket and path of your existing objects at your cloud storage provider. Once the metadata is removed, you can adjust your settings and run the Add Metadata tool again.

The Remove metadata added by the Add Metadata tool link also works as a background process. It will only remove records from the as3cf_items database table that were created by the Add Metadata tool. It will not remove records created by any other means including the “Copy Files to Bucket” tool or other standard WP Offload Media features.