How to Copy Media Between Buckets

With the release of WP Offload Media 1.5 came the ability to copy all of your existing offloaded media to a new bucket, without the need to download and re-upload locally.

There are a number of reasons why one might want to do this:

• Creating a new staging or development environment to safely mirror your bucket used in production
• Moving to a new bucket
• Consolidating media offloaded across multiple buckets to a single bucket
• Safely updating the bucket used for your media rather than performing a search/replace in the database

Important Note: You can not copy media between buckets in different storage providers, you must first remove your media from the original provider’s bucket using the “Remove all files from bucket” tool before changing provider and then setting a different bucket.

Changing Buckets

When you use the “Change” link next to your site’s current bucket name to change it and there is any media already offloaded, WP Offload Media will prompt you to copy (not move) all of your existing offloaded media to the new bucket.

If you choose “No”, nothing happens and the plugin continues to work the same as it did before: existing offloaded media stays where it is, and new media is uploaded to the newly chosen bucket.

If you choose “Yes”, WP Offload Media will kick off a background process to copy all of your media files which have already been offloaded to the new bucket and update them to reference the new bucket instead. This only affects media which has already been offloaded. It does not have any effect on media which has yet to be uploaded, or that has been offloaded to a different storage provider. A sidebar tool will appear to show the progress and a notice will appear in wp-admin once the operation is complete.

Because the operation happens completely on the storage provider, the whole process is very quick and usually finishes within a few minutes, even with thousands of files.

Using Bucket Constants

If you use bucket or settings constants to define your chosen bucket in your wp-config.php file, then the above prompt will not be available because the ability to choose another bucket through the UI is not possible.

You can still use the Copy Existing Files to New Bucket tool by enabling it in the sidebar with the following define in your wp-config.php.

define( 'AS3CF_SHOW_COPY_BUCKETS_TOOL', true );

Clicking “Begin Copy” will start the copy process, using your currently defined bucket as the target bucket to copy to.

Once the process has been started you may remove the AS3CF_SHOW_COPY_BUCKETS_TOOL define, the tool will continue to work in the background and show its progress in the sidebar, removing itself from the sidebar once complete.

Required Permissions (AWS)

NOTE: If you’re using the same AWS account and Access Keys for the old and new bucket, then you likely don’t need to bother with the rest of this article!

Regardless if you are copying between buckets in the same AWS account or if you would like to copy between buckets in different AWS accounts, both are possible with the proper permissions.

The following permissions are needed to copy between buckets:

• GetBucketLocation – on all source and destination buckets (bucket-root level permission bucket-name/)
• ListBucket – on all source and destination buckets (bucket-root level permission bucket-name/)
• GetObject – on all source buckets (path-level bucket-name/*)
• PutObject – on destination bucket (path-level bucket-name/*)

You need these permissions for the normal operation of WP Offload Media but you will likely experience problems in the process if your credentials do not have the necessary permissions on all buckets involved.

Copying Between Buckets in Different Accounts

Here we will use the example of copying media between buckets in different AWS accounts.

For this example, we’ll assume there is a bunch of media which has already been offloaded to a bucket called old-bucket. We’ll also assume that WP Offload Media has been configured to use a new set of access keys, which give the necessary access to the new bucket.

Copying between buckets in different accounts requires the following steps:

1. Granting access to the old bucket(s) by adding your new account to the bucket’s Access Control List
2. Granting the necessary permissions to your IAM user of the new account

Granting access to the old bucket(s) by adding your new account to the bucket’s Access Control List

For simplicity, we will reference the “new” AWS account which holds the new destination bucket as New Account, and the account which holds the old-bucket bucket as Old Account

1. Log in to the AWS console with New Account
2. Click your name in the top right side of the navigation bar to expand a drop down menu
3. Click My Security Credentials
4. If prompted, click Continue to Security Credentials
5. Expand the section called “Account Identifiers”
6. Copy your Canonical User ID to the clipboard (we’ll need this in just a minute)

We will come back to this account in a minute, so you may want to do the next steps with a different browser or “incognito” window.

1. Log in to the S3 AWS console with Old Account
2. Navigate to the old-bucket bucket
3. Switch to the Permissions tab
4. Click the Access Control List button if it is not already highlighted
5. Find where it says Access for other AWS accounts
6. Click + Add Account
7. Paste in the Canonical User ID copied from before
8. Tick the “Yes” boxes for List objects and Read bucket permissions
9. Click Save
10. Repeat this for each old bucket if there is more than one

You’ve now granted the new account read-only access to old-bucket. Now we just need to grant that access to the user within the new account who’s access keys will be used.

Granting the necessary permissions to your IAM user of the new account

1. Log in to the AWS console with New Account
2. Navigate to the IAM Management Console
3. Click Users
4. Click the username of the user for the new credentials WP Offload Media is using
5. From the Permissions tab, click Add inline policy at the bottom right side of the list of policies attached to the user
6. Select the JSON tab
7. Paste the JSON below into the policy editor
8. Update the Resource section to include the same two lines for each old bucket
9. Replace old-bucket with the name of your “old” source bucket(s)
10. Click Review policy
11. Give the policy a name such as “GrantReadAccessToOldBucket”
12. Click Create policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::old-bucket",
                "arn:aws:s3:::old-bucket/*"
            ]
        }
    ]
}

Now, you are ready to initiate the copy process from within WP Offload Media which should have the credentials configured for New Account.