If you follow us on Twitter you may have a clue as to what we’ve been working on for the past few months, and you might be a little excited about it…
https://twitter.com/JordonRupp/status/1020417656749355008
This is kind of a bonkers, but I'm excited about it.
— Mizner (@miznerism) March 2, 2018
— Jason Adams (@jasontheadams) March 1, 2018
At long last, after 10 months, 10 beta releases, 760+ commits and 93 closed issues, we’re finally ready to announce the release of the Theme & Plugin Files Addon 1.0 🎉🎉🎉! This release is a culmination of many new features and updates, most importantly the ability to migrate your WordPress theme and plugin files along with your database and media files.
Theme & Plugin Files Addon
The ability to migrate theme and plugin files is definitely one of the most requested features we have seen over the years. In fact, it’s a feature we first thought of in October 2015!
The Theme & Plugin Files Addon allows you to migrate themes and/or plugins when you migrate your database and media files. With this addon enabled, the WP Migrate DB Pro interface now includes selectors for themes and plugins when you initiate a push or a pull migration.
We’ve also included an ‘Exclude Files’ text box that allows you to use .gitignore syntax to have fine-grained control over which files are excluded from transfer.
The major benefit of using the Theme & Plugin Files Addon v.s. other methods of transferring files is that it simplifies deployment and setting up a development environment.
For example, say you’re a freelancer and your client only gives you access to their WordPress dashboard — no FTP credentials. Anyone been there before?
✋
Now you can use the Theme & Plugin Files Addon to download the site’s theme and plugin files and get started on the project right away.
The Theme & Plugin Files Addon can also help standardize your workflow across different environments. While some hosts do have SSH access and/or git deployment enabled, quite a few don’t. In these cases this addon makes transferring theme and plugin files a lot easier. No need to search for FTP credentials – you only have to install WP Migrate DB Pro + addons and you’re off to the races.
Of course we’re not advocating ditching SCP or rsync if you’re comfortable using them and SSH access is available. We simply believe the Theme and Plugin Files Addon provides another valid option for getting your files to and from any WordPress host. If you’re using WP Migrate DB Pro anyway to pull the database and media, why not also grab the theme and plugin files at the same time?
How We Did It
For this addon we also decided to revive the beta channel for WP Migrate DB Pro and released 10 beta versions over the last 8 months or so. You can see the progression in the changelogs for each beta release. Using the beta system allowed us to get people using the addon so that we could get feedback on what was working and what wasn’t.
The reason we wanted feedback early is that transferring files is fundamentally different than transferring databases. Database content is essentially text and is in a standard format (SQL). Whereas files can be any sort of content type – from plaintext to binary. Not to mention file encoding (UTF-8 anyone?). For this reason transferring files via PHP over HTTP is much more complex and required some deep thought and trial and error.
How It Works
We started planning how everything would work at our 2017 company retreat in Nova Scotia and got some rudimentary concepts decided upon.
https://twitter.com/dliciousbrains/status/902964783565217793
With the general structure of the addon decided, work began. In our initial discussions we talked about simply zipping up the theme and plugin folders, transferring them, and then unzipping them on the other side. This approach would require much less work than the final approach we landed on, but we ultimately felt that it wouldn’t provide the best user experience. It would be slow waiting for all the files to be zipped up before the transfer could even get started.
The Prototype
The first iteration of the addon migrated files one by one: one file = one network request. This allowed us to show individual file progress in the UI but was extremely slow. If you used one of the earlier betas, I’m sorry. However, this proved that our approach was technically possible and allowed us to iterate on it.
Batching
Once most of the core features were added, we started to look at ways to speed things up. The first thing we implemented was batching files for each request. Using the magic of PHP’s native streams, we were able to batch file content together into payloads, transfer them, and tease them apart on the other side. To be honest, it still feels a bit like magic because it works so well! This dramatically increased the speed of file migrations and got us closer to a final version of the addon.
Example of how payloads are streamed to the client
$handle = $this->payload->create_payload( $batch, $state_data, $state_data['bottleneck'] );
rewind( $handle );
// Read payload line by line and send each line to the output buffer
while ( ! feof( $handle ) ) {
$buffer = fread( $handle, 10 * 10 * 10000 );
echo $buffer;
@ob_flush();
flush();
}
return fclose( $handle );
Chunking
We also implemented file ‘chunking’ functionality for pushes. The reason being that many hosts have a lower post_max_size php.ini setting so trying to fire over a 10MB image file would fail. To get around this, we send large files in smaller ‘chunks’ and re-construct them on the other side. Again, PHP’s streams to the rescue here.
On the topic of pushes, we also noticed that many web hosts don’t like it when a PHP script writes PHP files to their filesystem – go figure? (Seriously though, this is a good security measure and we applaud those hosts. 👏) We’ve mostly solved this issue now, with the one exception still being WP Engine. This is due to the security measures in place and while you can still use this addon for pulls from WP Engine, we don’t have a workaround just yet for pushes to WP Engine.
Verification Measures
Since we’re transferring PHP files and other files essential to your site’s operation it’s important that we don’t white screen your site after a migration. One errant <?php tag and BOOM!
For this reason there are several safety measures in place, from verifying file MD5 hashes to checking file sizes of local and remote files. We also stage files in a tmp folder during a migration and only move them to the final location if the entire migration completes successfully.
These measures were put to the test recently. Brad was getting a file size verification error when pulling certain plugins. After digging into it, we found that the New Relic PHP agent installed on his live site was injecting a JavaScript tracking code into any files that had the string <body>.
Luckily, the file size check reported the size difference and aborted the migration. We plan to investigate the New Relic PHP agent issue further and make sure that any injections like this don’t occur in the future.
Extras
This release also includes a few other odds and ends like the ability to opt-in to our beta channel and usage tracking.
Beta Channel
The beta channel for WP Migrate DB Pro is here to stay. You’ll notice a new option under the “Settings” tab of the plugin to opt-in or out. If you opt-in to the beta channel you’ll get notified when new betas are available. It’s the best way to test out new features (and let us know what you think) before they’re released!
Optional Opt-in Usage Tracking
Another item included with this release is opt-in usage tracking. Usage tracking is disabled by default and is completely optional. Opting-in only sends us information about the types of migrations you run and which settings you’re using. This greatly helps us improve WP Migrate DB Pro as the aim is to figure out which features are used more heavily than others.
Fin
As work on this release was winding down, we had some illuminating meetings about the future of WP Migrate DB Pro and have charted an exciting new course for what’s next. Brad will be discussing this and sharing wireframes in the a full post in the next few weeks. Stay tuned!
Thanks to all our beta testers who helped us get the Theme & Plugin Files Addon ready for public release. If you’d like to know more about how it works or if you were a beta tester be sure to let us know in the comments!
Want to give the new Theme & Plugin Files Addon a go? The addon is available with all Developer licenses (and higher) of WP Migrate DB Pro: https://deliciousbrains.com/wp-migrate-db-pro/pricing/
