Behind-the-Scenes: Planning The Future of WP Migrate DB Pro

We often get questions about what’s coming next to WP Migrate DB Pro or if we ever plan to include X feature or support Y sites. After shipping our latest major release, the new Theme & Plugin Files addon, a few weeks ago, we started to ask that question ourselves.

For anyone curious about where WP Migrate DB Pro is going, this article will give you a peek inside our working roadmap for the next several versions.

Where We’ve Been

Around this time last year, I wrote about why the WP Migrate DB Pro release had taken twice as long as it should have. And here we are again, nearly another year gone by and only one major release to speak of: the new Theme & Plugin Files addon we released a few weeks ago.

The good news is we didn’t repeat the same mistakes this past year. The main reason it took so long is that the minimum version of the Theme & Plugin Files addon turned out to be a lot of work. Transferring files over HTTP in a performant way (as good or better than FTP) and ensuring data integrity (checksum ftw!) was a challenging feat as Peter described in the release post.

The bad news is the technical debt in WP Migrate DB Pro that I wrote about last year slowed us down a lot. And so we’ve decided to focus on servicing technical debt for the next release.

Next Release

Peter has been performing a major reorganization of the code over the past 6 weeks or so, which should speed up development going forward. Believe it or not we’re on track to release that along with an all-new Backups tab later this month. Yup, another major release after just a couple of months!

The Backups tab will simply list the files in your backups folder and have download and delete functionality to start:

Please keep in mind that these are wireframes and that the finished product may look significantly different.

This release will also require PHP version 5.4+, so if you still need to use WP Migrate DB Pro on a server running PHP 5.2 or 5.3 you’ll either need to upgrade or use an older version of WP Migrate DB Pro.

If you’ve turned on beta update notices, you should see one very soon. To turn on beta update notices, simply visit the Settings tab in WP Migrate DB Pro.

The Release After Next: 2.0

The release after next is going to be a biggie. The plugin’s Javascript (JS) is in terrible condition. It was written in jQuery six years ago and is a tangled mess of nested callback hell. The team absolutely dreads working on it and it’s painfully slow going to add or update anything in there.

We’re going to throw it all out and start over with React. Why React over Vue.js or another JS framework? I’ll let Peter start the holy war here:

React is the obvious choice for a JavaScript rewrite as it’s been chosen by the Gutenberg team and will be available in WordPress core going forward. I also personally have more experience with React so it’s a no-brainer for me. If you’re here for the holy war, check out the comments of the Vue vs React post I published a couple of months ago. 🔥

Because starting over with React involves heavy updates to the UI, we’re also going to refresh the UI a bit, better grouping settings together and other small details that are long overdue.

We’ll also be adding some more significant features, like the option to back up the database without running a migration or to automatically save a recent migration so it can easily be reused or saved as a migration profile.

A new solid UI foundation with fresh, well-written JS is going to do wonders for us going forward, allowing us to work more quickly and painlessly. Which should mean we can deliver on more features and other product improvements faster.

Beyond 2.0

What should WP Migrate DB Pro look like in a couple of years time? That’s the question we asked ourselves. So we opened Balsamiq and started sketching what our customers have told us they’d like to see and what we’d like to see ourselves.

Although we haven’t prioritized yet, we definitely want to tackle the following items in the future.

Dry Run

It would be pretty sweet if in addition to pausing a migration before it completes (like you can already), you could view the search and replace changes and decide whether you want to proceed with replacing the database or cancel and make some adjustments before running it again.

Friendlier Backups

Backup file names are currently not descriptive at all. It would be nice to know why a backup exists in addition to when it occurred:

It would also be nice to restore a backup right there on the Backups tab.

Safer Backups

Something that has always irked me about WP Migrate DB Pro’s backups is their location. They’re in a publicly accessible folder! 😬

Sure we suffix each filename with a salt string so it’s hard to guess, but yikes, that’s still not very safe. It would be infinitely better if you could choose an offline folder for them to be saved to instead:

Independent Connections

Currently the connection info for remote sites is saved to each profile. But what if you have a few different profiles for the same site and you need to update the connection info? Yup, you need to update each profile. Annoying. Making the connection info independent of profiles will allow you to update it once.

Push/Pull One Post

Once the connection info is independent of profiles, we can do other things with it, like push an individual post to a remote site or pull in a post from a remote site.

Migrate Files without the Database

Although we have an easy little hack for migrating files without the database, it would be great if there was a more intuitive way to do it. We plan to have exactly that:

Preserving Options

Currently you can preserve the active_plugins option from the wp_options table when performing a migration:

But what about other options, like which theme is active and whether or not the site can be indexed by search engines? We’d like to enable you to easily preserve any option in the wp_options table:

Currently you can achieve this is with the wpmdb_preserved_options filter, but it will be much more convenient to be able to do it via the UI.

Regular Expressions and Case Sensitivity

Currently WP Migrate DB Pro’s search & replace is case-sensitive and does not allow case-insensitive searches, nor does it allow regular expressions (regex). We’d like you to be able to toggle these on and off.

CLI Command Builder

Migration commands can be pretty long and ugly. It’s not usually pleasant to type them out, so we thought it would be handy if you could set up a migration via the UI and it would spit out a CLI command for you to turn into an alias or set up a cron to run it on a schedule.

Import/Export Migration Profiles

Many of our customers have told us that they’d like to import and export migration profiles. We plan to implement this the same way many plugins do: JSON.

We considered allowing you to push/pull profiles but JSON has the advantage of being easily editable. Don’t want to import all the profiles? Just delete that part of the JSON. Need to change the name of a few profiles? Open it in your favorite editor and run a search and replace.

Subsite-to-Subsite for the Multisite Tools Addon

Currently there is one major feature missing from the Multisite Tools addon: the ability to push/pull a subsite (database and media) from one multisite install to another multisite install. We need to tackle this at some point.

CLI Support for the Theme & Plugin Files Addon

We tried to keep the first release of the Theme & Plugin Files addon as minimal as possible to get it released sooner than later. This meant sacrificing features like CLI support. There is currently no CLI support for the Theme & Plugin Files addon but it’s definitely something we want to add in the future.

Speeding Up Migrations

When we built the new high-performance file transfer system for the Theme & Plugin Files addon, we made sure that it could transfer any data, not just files. We plan to swap out the old transfer systems for database and media file migrations which should make a massive speed improvement when migrating data and media files.

Resume Migration on Error

There are few things more frustrating than running a large migration and a network blip causes the whole thing to come crashing to a halt. Currently the only thing you can do is start over and hold your breath again. We plan to have better recovery from errors like this in the future.

Handle Different Table Prefixes

Currently, if you try running a push or pull migration between two sites that have different database table prefixes, you get a warning message like this:

I think it’s high time we ditch this warning and gracefully handle this for you. It was a complicated problem when we last looked at it years ago but it’s at least worth revisiting.

Conclusion

WP Migrate DB Pro turned five years old this year and people still love it…

This will take a while. Better get some ☕️
(thank you @dliciousbrains for this awesome plugin) pic.twitter.com/x8UMTrWWq3
— Nicolas Bazinet (@CaptnBaz) October 23, 2018

I already love Migrate DB Pro from @dliciousbrains, and recently we’ve been using extensively to import/export between multisite and single install and I have to say, it’s been flawless. 💯
— Mike Hemberger (@JiveDig) October 16, 2018

So the Theme and Plugin files add-on for WP Migrate DB Pro from @dliciousbrains is 😙👌 Paired with the Media add-on and I rarely have to use FTP at all now!
— Jack McConnell (@jackmcconnell) October 8, 2018

https://twitter.com/allisonplus/status/1040300604508258304

There's only one plugin that has truly shocked me with its reliability and value: WP Migrate DB Pro from @dliciousbrains
— Matt (@matthew_zak) August 14, 2018

But as the length of this post demonstrates there is plenty of room for improvement and we’re far from done.

What are the top 3 things you would like to see in WP Migrate DB Pro? Something from this list or something else? Let us know in the comments.

This entry was posted in WP Migrate DB Pro and tagged WP Migrate DB Pro, New Release, Project Management, Behind the Scenes, Future, Planning, Wireframes, Release Planning.

.replybox-fallback, .replybox-fallback ul { list-style: none; }

Alex Wright says:

November 6, 2018 at 6:55 am

Thanks for your dedication! To answer the question of top things we’d like to see in WP Migrate DB moving forward – backups is definitely not one. We use WP Engine who takes care of all backups, and another option to backup a database isn’t helpful. But keep up all the good work!

rick gregory says:

November 6, 2018 at 8:02 am

I’ll echo this. There are a lot of good backup solutions out there (updraft etc) and services like ManageWP do this also. One thing I’d guard against is making the plugin (which I LOVE) a catchall for things that are vaguely related to migration. I like and use it because it makes migrating back and forth very easy and stupidly reliable. Please don’t clutter the code or the UI with things that are not related to migration or related only loosely.

Reply
Brad says:

November 7, 2018 at 4:49 am

I 100% agree you shouldn’t be relying on WP Migrate DB Pro for site backups. It’s certainly not designed for that use case. But if you’re pushing a database to another site it doesn’t hurt to backup the database before overwriting it. That’s the main purpose of WP Migrate DB Pro’s backup feature. And having visibility into those backups and the ability to manage them is important. Being able to run a quick backup of the database before you do something else (something other than a database push) can be handy too, hence the dedicated Backup option. In my opinion, WordPress plugins are not a good solution for scheduled site backups. A server-side backup is not going to tax your server as much and will be more reliable (e.g. some plugin you have installed causes a problem for the backup plugin and backups don’t run, are corrupted, etc). A couple of years ago I shared a short bash script that can be used to run daily backups and send them to Amazon Glacier. If you host WordPress yourself, this is an option. Otherwise, your host should be doing this for you.

Reply

JorgeLuisBorges says:

November 6, 2018 at 6:59 am

+1 to restore backups. Or some kind of control of the settings/compatibility of the backups – we get issues restoring from old sites on older servers. Keep up the good work people.

Athlone Harris-Compton says:

November 6, 2018 at 7:00 am

I use this plugin on multiple corporate sites that have multiple environments. I love it. I can’t imagine working without it, but one thing I really, really want to see is the ability to preserve options outside of the wp_options table. We currently run an MU plugin to preserve specific options in that table, but there are times when I would have really liked to have been able to preserve options outside of it too.

Brad says:

November 7, 2018 at 4:32 am

Would the wpmdb_preserved_options filter work for you or do you need a UI for some reason?

Reply
- Athlone Harris-Compton says:
  
  November 7, 2018 at 5:09 am
  
  Yes, that does work. That’s what we already use in the MU plugin I mentioned, but it only works for the wp_options table. I’ve had a couple times where I wanted to exclude database items from migration that were outside that table, and there currently isn’t a way to do that as far as I know.
  
  Reply

Dave Schmidt says:

November 6, 2018 at 7:00 am

Awesome work, guys! WP Migrate DB Pro is one of the indispensable tools I use to run my business, so I love that you’re continuously improving it. For a while now I’ve said it would be great to see a list of changes before running a Search and Replace. I’ve gone back to the ‘ole reliable script by Interconnect/it because it shows a detailed list of database changes. Thanks again for welcoming feedback.

Jon says:

November 6, 2018 at 7:03 am

Gutenberg support! (j/king). Seriously, though… There are so many great things in there I can’t list them all, but I am thrilled to see my "CLI command builder" idea is in the mix. I will add one more… allow MDBP to work when there is a minor version mismatch. It’s frustrating that so often I would go to pull a DB to local only to be told my versions are out of sync, so I have to login to one or both sites and update them, then return to pulling. 50% of the time I’d instead just go to phpmyadmin. MDBP saves a ton of time, except when it doesn’t, and version mismatches blocking migrations is on of those times it doesn’t.

Brad says:

November 7, 2018 at 4:29 am

Hey bud, I’ve had a different solution in mind for this problem since the very beginning… https://uploads.disquscdn.com/images/2d1c8a57941f73a6d077377c35f00b479b0955043b748510758217a9cb9fdabd.png Just haven’t gotten around to it!

Reply
- Jon says:
  
  November 7, 2018 at 9:57 am
  
  Update remotely sounds cool! Just something short of the hoops I regularly jump through now 😀
  
  Reply

Konrad Bąk says:

November 6, 2018 at 7:08 am

Thanks for sharing your roadmap! Do you plan to reboot a database merging/synch feature, either in WP Migrate DB or in a dedicated plugin à la MergeBot? Thank you, Guido

Brad says:

November 7, 2018 at 4:25 am

So far the only plan along those lines is to be able to push/pull one post at a time.

Reply

Ulrich says:

November 6, 2018 at 7:11 am

The two features I would love to see would "Push/Pull One Post" and "Resume Migration on Error". I work on big multisites where the migration fails when syncing using the UI. The alternative that we use is using the CLI to export and scp to move the ZIP file and import again with the CLI. The push and pull posts would useful as we have done flying switches between the old site and the new redesigned site. We have normally migrated the last posts that have been added to the old site to the redesign site. MergeBot would have been to complex for us. A simpler solution would be cool. It would be quicker to update a specific post that I want to test with the latest content locally then get the whole site again. Regarding the CLI Command Builder you already have "aliases" with the profile subcommand. https://deliciousbrains.com/wp-migrate-db-pro/doc/cli-profile-subcommand/

Joe Fletcher says:

November 12, 2018 at 8:06 am

I believe they mean a bash alias to save long migrate commands. What I’d like to see is a renamable alias for the profile IDs. Currently, you can do wp migratedb profile 1 – but profile id 1 can mean different things on different dev sites. On one machine, wp migratedb profile 1 could be a push to production, but on another wp migratedb profile 1 could be a pull. I’d like to have the ability to do "wp migrate profile push-to-production" that I set up to always push to production or "push-to-production-db-media" that always does db & media. That gives me the ability to standardize my CLI commands across dev machines and across different sites. This would be possible by adding an editable "alias" or nickname in addition to the existing profile id.

Reply

Luke Cavanagh says:

November 6, 2018 at 7:24 am

WP Migrate DB Pro is one killer product. The new features look very solid.

Brian Bourn says:

November 6, 2018 at 8:26 am

I use WP DB Migrate Pro every week & it’s the best!. Within the media files add-on it would be great to include an option to "use media from production". For small sites I pull down all the media files locally, but for larger sites we work on, the media library can be quite large and take hours to download and often times the process times out.

Sam Carlton says:

November 6, 2018 at 11:57 am

Love that you guys are moving to React, although I do like Vue more right now you’re reasoning totally makes sense. +1 for CLI Support for the Theme & Plugin Files Addon The Theme & Plugins addon has been a serious life saver for deploys to barebones hosting platforms and I’d love to see it get even better.

mikelking says:

November 6, 2018 at 12:17 pm

While all of those new features are prett nice, the most important feature you can add is: Subsite-to-Subsite for the Multisite Tools A tool/feature not on your radar that would be ever so helpful is a DB diet/shrinker for converting GIANT production site DBs into local dev DBs. Nothing like have a huge DB dump from prod that doesn’t fit on your local dev… :/

Yu-Cheng Wang says:

November 6, 2018 at 6:12 pm

it’s always good to read all the details about the product. Using WP Migrate DB Pro everyday.

Adam Norwood says:

November 8, 2018 at 11:20 am

Big things like the Multisite subsite-to-subsite tools would be great, definitely, but here’s one presumably smaller tweak that would make me very happy in the meantime. For the Tables section on the Migrate tab, it’d be awesome if there were a way to filter the list of tables quickly (e.g. an input where I could enter _30_ to narrow down to just the tables for the site with ID 30). Even more useful would be an additional radio button option on Multisite installs: "Migrate only the tables for site: [dropdown / autocomplete list of sites by name]", which would save me the extra step of looking up the ID for a site. (maybe the proposed subsite-to-subsite tool would basically make this request pointless, but this is how I’m currently bouncing data for single sites from a production instance down to my local testing environment…if there’s already an easier trick, I’m all ears) Anyhow thanks for the always great work, and I look forward to seeing features from the roadmap come down the pike!

Matt McDowell says:

November 12, 2018 at 9:31 am

I’d like to see CLI support for the Theme & Plugin Files add-on. Sounds like this is being planned, but no timeline as yet? Thanks!

Dac Chartrand says:

November 13, 2018 at 3:58 pm

Thanks for posting. We’re doing similar research at Pressbooks and have been looking at posts like these for inspiration. > is a tangled mess of nested callback hell I wanted to ask if you’ve looked into Event Streams? Here’s where we are at with our research so far: + https://pressbooks.org/blog/2018/11/12/refactoring-slow-forms-using-php-generators-and-event-streams/ We can now update the UI, in real time with a single ajax request. The main ideas are that 1) It’s not necessary to wait until the request finishes, PHP can emit event-stream responses (SSE) back to the web browser while it is working on something and 2) PHP Generators are a relatively simple refactoring hack to get those responses back to the browser. (available since PHP 5.5.0) If the browser doesn’t support EventSource, this polyfill is reliable: https://github.com/Yaffle/EventSource If you haven’t used EventSource, check it out? If you have, any drawbacks? Regards,

seabee says:

November 14, 2018 at 2:55 am

Believe it’s theoretically possible now, if I know what tables/queries to make – but would like to be able to database merge with WooCommerce in particular (and also forms, such as Gravity Forms). Problem is customer requests work to a website, so I use pull it to local machine to work on. Then push it to a staging machine that the customer can see. This might take days or weeks before sign-off. Then I need to put live – and guess what, there’s been loads of WooCommerce orders and form submissions that I can’t lose. Currently I’m having to make notes of changes and apply manually – a real time waster and risk-prone.

Jason Pomerleau says:

December 27, 2018 at 10:47 am

Excited for the speed improvements. Have some sites with gigabytes of media files. I often just use rsync to move those over, but that’s not always an option. Having assets move over faster would be a welcome improvement. 🙂

TrevorAGreen says:

January 15, 2019 at 7:59 am

Maybe a two stage process where you can compare local to remote during a push/pull to detect any changes and help you decide which tables to migrate. Say you are pushing a site from some local changes where you have gravity forms. You really need to not overwrite any new form entries, but if you made changes you might still want to push those tables. So it would be nice to have a preflight where it tells you if you have differences to help you decide what to do. I have also had issues with some security plugin tables just failing the migration. If I just remove them and stick with the core wordpress stuff there isn’t really an issue, but +1 for better failure detection. If there was a preflight mechanism you could maybe handle this with a way to publish the tables that are working and dump those that have errored out. It might also be nice to have a selector when setting up a profile to auto select only the core wordpress tables, rather than having to manually select them.