WP Offload Media has a lot of filters that can be used to alter its behavior, but not a lot of people know about them, or what you could accomplish with them.
WP Offload Media used to work with just Amazon S3, but we recently added support for DigitalOcean Spaces, and then Google Cloud Storage. While I was developing the support for those additional storage providers I tried to make things easier for using WP Offload Media with other storage providers too.
We often get asked whether WP Offload Media can work with this, that, or the other S3 API compatible storage provider. Our usual answer is, “probably, if you use some filters, but we haven’t tried it”.
I thought it high time that I start exploring some of these S3 API compatible storage providers to see how easy (or not) it is to get WP Offload Media working with them. There’s a good few to choose from, such as DreamHost DreamObjects, IBM Cloud Object Storage, Oracle Cloud Storage, Wasabi (see our Wasabi Cloud Storage Quick Start Guide) and Zenko.
I started with MinIO.
What’s MinIO?
MinIO describes itself as …
The 100% Open Source, Enterprise-Grade, Amazon S3 Compatible Object Storage
Hmm, that doesn’t really help much. Maybe it’ll help if I explain why I picked it as my first S3 compatible storage provider to try.
MinIO is an S3 service you can run on your own servers, or even your own desktop machine. This means I didn’t have to sign up for anything, I can use it on my development machine for pure unadulterated speed, and there’s no bandwidth charges while I test, retest, and test yet again WP Offload Media functionality.
Setting Up MinIO
We’re not trying to write an exhaustive guide to using MinIO here, so I’ll just point you to MinIO’s Quick Start Guide. It explains how to quickly install and run MinIO via Docker or with a native binary on macOS, Linux, Windows or FreeBSD.
For me, on my macOS development machine, it was super easy to install via Homebrew.
brew install minio/stable/minio
mkdir -p ~/minio/data
minio server --address :54321 ~/minio/data
This got me up and running with a MinIO server running on port 54321 and storing its data within a subdirectory of my home directory. I chose port 54321 instead of the default of 9000 because I already use that one for PHP and/or Xdebug. Also, 54321 is in the range of “private” ports that are free to be used as you see fit, and I’ll easily remember it!
Because I didn’t use any of the other options for assigning things like Access Keys (you can check the server’s config options with minio server --help), MinIO generated some keys for me.
When I visited http://127.0.0.1:54321 I got a login prompt where I could enter the Access Keys.
This got me to the MinIO Browser, their equivalent of the AWS Console.
That MinIO Browser looks rather empty though, time to fix that.
Configuring WP Offload Media
Now the fun begins as we need to tell WP Offload Media that it is using an S3 service, but that it needs to use different URLs than normal for accessing the bucket.
First up, I took those Access Keys MinIO generated for me and added them to the AS3CF_SETTINGS define in my site’s wp-config.php file.
define( 'AS3CF_SETTINGS', serialize( array(
'provider' => 'aws',
'access-key-id' => 'WA0XXXP74NSR8SGMEWYF',
'secret-access-key' => 'gCkWMwnSl+Z6druGcaItxYuMdeStdfHkQnbQd3bP',
) ) );
That’s the easy bit, WP Offload Media will now use the S3 API with the defined credentials.
Now we need to point WP Offload Media to a different “endpoint” URL for reading and writing buckets and objects. For that, we turn to the WP Offload Media Tweaks plugin which is where we “document” 😉 how to use some of WP Offload Media’s most useful filters.
I downloaded, installed and activated the Tweaks plugin, and then added the following function to its Amazon_S3_and_CloudFront_Tweaks class.
function minio_s3_client_args( $args ) {
// Example changes endpoint to connect to a local MinIO server configured to use port 54321 (the default MinIO port is 9000).
$args['endpoint'] = 'http://127.0.0.1:54321';
// Example forces SDK to use endpoint URLs with bucket name in path rather than domain name as required by MinIO.
$args['use_path_style_endpoint'] = true;
return $args;
}
I also added the following add_filter() call to the __construct() so that the new function will be used.
add_filter( 'as3cf_aws_s3_client_args', array( $this, 'minio_s3_client_args' ) );
Now when WP Offload Media tries to talk to S3 it’ll use my local MinIO server’s URL, and use the correct “bucket in path” URL style that MinIO requires.
While I was there I added the following two bits of code to make sure there’s just the one region as it doesn’t make sense to use more than one with a local MinIO server.
// Added to __construct()
add_filter( 'as3cf_aws_get_regions', array( $this, 'minio_get_regions' ) );
// Added to Amazon_S3_and_CloudFront_Tweaks
function minio_get_regions( $regions ) {
$regions = array(
'us-east-1' => 'Default',
);
return $regions;
}
With that, when WP Offload Media was installed on my site and I visited Settings -> Offload Media I was straight into the “Select Bucket” screen. A click of the “Browse existing buckets” link confirmed to me that WP Offload Media could access the MinIO service without any error, albeit with not a lot to show.
You’ll notice it says “Amazon S3” as the storage provider, but I assure you it was connected to my MinIO server! We’re just using the S3 API internally and at the moment don’t have any filters to change the name. Maybe one day we’ll add that if customers want to be able to change non-critical things like that.
It was a simple affair to use the “Create new bucket” link, enter my new bucket name of “ianmjones-wpom-minio” (yes, yes, I know, terrible name, but I use a similar format for all my test buckets) and clicked “Create New Bucket”.
This worked, and now I was at WP Offload Media’s settings page with all looking as it should.
Well, almost all as it should, but we’ll come to that in a minute!
With everything set up, it was time to offload the Media Library items I had in the site using the “Offload Now” button you saw in the previous screenshot.
Yay, a super fast offload of media with no errors!
When we check the MinIO Browser there’s a lot more to see now.
However, when I viewed one of the Media Library items that had been offloaded, all was not well.
Doh! 🤦♂️ I’d forgotten to change the URL format that WP Offload Media should use to serve offloaded media with. So it had a URL like this…
http://s3.amazonaws.com/ianmjones-wpom-minio/wp-content/uploads/2019/04/17132127/0001_tAKXap853rY.jpeg
In hindsight, when you look at the settings screenshot from just after I’d created the bucket in WP Offload Media, you can see that the “Preview URL” is still using an AWS S3 format.
That’s clearly not going to work, so back to the Tweaks plugin we go!
// Added to __construct()
add_filter( 'as3cf_aws_s3_url_domain', array( $this, 'minio_s3_url_domain' ), 10, 6 );
// Added to Amazon_S3_and_CloudFront_Tweaks
function minio_s3_url_domain( $domain, $bucket, $region, $expires, $args, $preview ) {
// MinIO doesn't need a region prefix, and always puts the bucket in the path.
return '127.0.0.1:54321/' . $bucket;
}
That as3cf_aws_s3_url_domain filter allows you to change the first part of the URL used for serving media, everything up to the point where the object’s path within the bucket starts.
With great anticipation I switched back to the Media Library item’s page and refreshed to see … same problem, different domain. The URL looked right, but still gave me 403 access denied errors.
So I hopped over the MinIO Browser, found the original image and used its little “…” button to get the image’s URL. When I pasted it into a fresh browser tab it worked fine, but the URL was signed!
It started OK, something like this…
http://127.0.0.1:54321/ianmjones-wpom-minio/wp-content/uploads/2019/04/17132127/0001_tAKXap853rY.jpeg
… but then had all the usual params you’d expect for a private object.
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=WA0XXXP74NSR8SGMEWYF%2F20190417%2F%2Fs3%2Faws4_request&X-Amz-Date=20190417T134240Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=305d4e358844b165c7bef04c0c02041fab0f3dfcc4ca43cf9b8ab4e981b40c47
Much investigation ensued. Turns out MinIO doesn’t support object ACLs at all, just bucket policies. By default, the bucket is private. Oh.
Never mind, while WP Offload Media might not be able to change the ACL of objects it offloads to MinIO, it could at least pretend to be writing private files and therefore sign any URLs it generates for them in content. Maybe that’d work?
// Added to __construct()
add_filter( 'as3cf_upload_acl', array( $this, 'minio_upload_acl' ), 10, 1 );
add_filter( 'as3cf_upload_acl_sizes', array( $this, 'minio_upload_acl' ), 10, 1 );
// Added to Amazon_S3_and_CloudFront_Tweaks
function minio_upload_acl( $acl ) {
return 'private';
}
We have two filters for changing the ACL that objects are given in the bucket during an offload, one for the Media Library item’s original file, and one for any thumbnails (sizes) it may have. When these are used to change the ACL to private WP Offload Media remembers this and makes sure to sign any URLs it generates for the Media Library item.
Success! After a quick “Remove all files from bucket” and “Offload Now” dance, we have private media being served from MinIO.
However, I didn’t really want signed URLs for all my media, so looked at how to add a policy to the bucket to allow public read-only access. While the doc on how to use the AWS PHP SDK with MinIO shows the required JSON style policy, you can actually just update the policy via MinIO Browser.
When you hover over the bucket name to the left you can use the “Edit policy” menu option after clicking its vertical “…” button. You then add a prefix, I added “/wp-content”, pick a policy, “Read Only” in this case, and then click the “Add” button.
I therefore commented out the two as3cf_upload_acl* filters in the Tweaks plugin, did the remove-from-bucket-and-re-offload dance again to make sure WP Offload Media thought the objects had public access, and Bob’s your Uncle, we have public media being served from MinIO.
That screenshot may not look very different from the previous one, but look closely at the “Offload” metabox to the right and you’ll see “Access: Public”. Woohoo! 🎉
There was now just one little itch to scratch, the link next to the bucket name in WP Offload Media’s settings page didn’t work. It was trying to go to the AWS Console.
https://console.aws.amazon.com/s3/home?bucket=ianmjones-wpom-minio&prefix=wp-content/uploads/
What I needed was something like…
http://127.0.0.1:54321/minio/ianmjones-wpom-minio/wp-content/uploads/
Luckily, we have filters for that too.
add_filter( 'as3cf_aws_s3_console_url', array( $this, 'minio_s3_console_url' ) );
add_filter( 'as3cf_aws_s3_console_url_prefix_param', array( $this, 'minio_s3_console_url_prefix_param' ) );
// Added to Amazon_S3_and_CloudFront_Tweaks
function minio_s3_console_url( $url ) {
return 'http://127.0.0.1:54321/minio/';
}
function minio_s3_console_url_prefix_param( $param ) {
return '/';
}
The as3cf_aws_s3_console_url filter allows you to change the base URL used to take you to the provider’s console from WP Offload Media’s settings.
The as3cf_aws_s3_console_url_prefix_param filter denotes what should be in the console URL before the path prefix value. For example, the default for AWS/S3 is “?prefix=”, but MinIO just appends the path prefix directly after the bucket name.
And would you Adam and Eve it, it only bloomin’ works! 👀
Wrap Up
By the end of my exploration I had just 5 simple filters to implement to get WP Offload Media working with MinIO. I was pretty chuffed.
You can find all the filters I used to get WP Offload Media working with MinIO in our WP Offload Media Tweaks plugin.
If you’re thinking of using WP Offload Media with an S3 compatible storage service, hopefully this little exploration helps you get an idea of how relatively easy it is.