Dependency Management and WordPress: A Proposal

#

‘Dependency hell’ is a problem faced by all software, and it has been rearing its ugly head in the WordPress space over the last few years with more and more plugins using third-party libraries of code.

We come across this issue every couple of months with our Amazon S3 plugin WP Offload S3. It’s a very real problem for Delicious Brains and can serve as a good concrete example of the issue.

The Problem

Our plugin transfers media to Amazon S3 when you upload to the WordPress media library, and to do that we make use of the AWS SDK (bundled as the Amazon Web Services plugin). The SDK went through some major changes between versions 2 and 3, resulting in the SDK bumping its minimum required version of PHP from 5.3 to 5.5. With 42.7% of all WordPress installs running on servers with PHP 5.2, 5.3 or 5.4, stipulating a minimum of PHP 5.5 just wouldn’t work for our plugin.

When customers use Offload S3 along with another plugin that relies on and bundles the AWS SDK, unless that plugin is using v2, we encounter an issue of incompatible libraries. WordPress loads plugins in the order they were activated, and PHP loads files as they are required and classes as they are called (when using autoloaders), so it is a real lottery which plugin calls the SDK code first. The other plugin is the one that suffers.

The most frustrating thing about this issue is that it’s caused by having the best of intentions! Developers use third-party code to be efficient and avoid reinventing the wheel. The code has been written by others and used and battled tested by many.

Why not just include the library in your plugin with the namespace or classes prefixed? That would work, no clashes would occur. The Ibericode guys did this to include the Pimple package in their plugin. This works fine for them as Pimple is a small two file package. But this just doesn’t scale. Doing this manually for something as large as the AWS SDK would be too much work, and would require doing it over and over every time the package is updated. Which is why people turn to Composer in general, as it makes it easy to manage and integrate third-party libraries into a codebase. But unfortunately Composer is often viewed as the culprit of this very problem.

This is NOT a Composer Problem!

The WordPress community has a hard enough time already trying to get onboard with Composer (unlike the rest of the PHP world), without it getting tarred with the wrong brush!

The problem is already there, without Composer, you just don’t notice it as prominently. Packaging any third-party library with your public plugins, whether you use Composer or not, will cause issues with WordPress.

Alain SchlesserPost Status Slack.

Peter Suhm wrote about the dangers of using Composer with WordPress because of this very issue, but luckily Coen Jacobs jumped in to Composer’s defence to point out his error. This sparked some further discussion between Peter and Coen which helped promote a potential solution from Coen, ironically using Composer. It leverages the Composer Merge Plugin, built for Wikimedia, to merge multiple composer.json files at runtime.

This makes perfect sense, as Composer is the de facto PHP package and dependency manager, but it needs some bending to make it work across a whole WordPress site which might have multiple plugins using their own composer.json file to manage their dependencies. This is only a proof of concept from Coen but I love the direction here, and I will come back to solving the problem with Composer later.

The Ideal Approach

Ryan McCue wrote a great piece about how he would solve the plugin dependencies problem within WordPress, that didn’t focus so much on the technical implementations but more on what was required from an end user experience. To summarize, the solution needs to:

  • Catch potential conflicts with dependencies up front before a plugin is installed
  • Installation should automatically take care of installing a plugin’s dependencies
  • Updates to the plugin should be dependency aware, and stop and warn if updates will create conflicts

WordPress as a project is very much focused on the user and not the developer. ‘Decisions not options’ is a mantra that has lead many of the project’s features in recent years, automatic background updates being an important one. Dependency management needs to be handled in the same fashion, where the user doesn’t have to think too much and it just works for them.

This type of solution is a big ask, covering a lot of ground and facing some large challenges. Critically, Ryan makes the point about where the solution needs to sit – WordPress core:

The end goal here is core integration. If the solution doesn’t end up in core at the end, the project has failed, as it’s not ubiquitous. If this happens, throw out what you need and try again, but it must be in core to be a viable solution for many users.

A Proposal

I’m not the greatest WordPress developer. I can use Composer but I am no expert. I’m certainly not a PHP internals guy and I’m just about keeping up with this problem as I write it, but… I think the problem can be solved and here is my high-level proposal for a solution and possible core feature plugin.

As I mentioned before, I feel Coen was definitely on the right track with his Composer based solution and some of this is directly inspired by his project, but I think we need to take it further.

1. Third-Party Code via Composer

Any plugin or theme using third-party code needs to define their dependencies using a composer.json file, and should not distribute their vendor directory.

2. Composer in Core

WordPress would have to come bundled with Composer removing the need for end users to have to install it manually. This would be like any other library WordPress requires, such as Requests or Simple Pie.

3. Custom Composer Behavior

We would then need to heavily customize Composer’s core behavior to integrate with WordPress. Specifically, this would integrate with the plugin/theme installation process to read the plugin’s composer.json file, merge it into all the others and install the dependencies.

It would need to catch the conflicts and reply back to the user (via the admin UI or CLI) to inform them of issues. It would also need to handle this when a plugin is being updated.

Drupal has had the similar Composer Manager since 2013:

Composer Manager allows each contributed module to ship with its own composer.json file, listing the module-specific requirements. It then merges the requirements of all found modules into the consolidated composer.json file. This results in a single vendor/ directory shared across all modules which prevents code duplication and version mismatches.

4. Package Sandboxing

The last part of my proposal is a bit more blue sky thinking, but some recent work from Coen has given me hope. The ideal scenario for WordPress and users is to not have to deal with conflicts at all. Every plugin should be able to run their dependencies safely without the threat of interfering with others. Loading multiple versions of the same class just isn’t possible in PHP, but prefixing namespaces or classes so they can be used in isolation is something that might work, and has been discussed previously from a Composer perspective.

Mozart is a new script from Coen that automatically prefixes PSR-0 and PSR-4 compatible packages with a custom namespace, so they can be safely used by a plugin. Building this kind of functionality into the Composer part of WordPress would mean all dependencies used by plugins would be sandboxed and no conflicts would arise. Of course, more work would need to be done to Mozart to support other types of packages, such as those with classmap autoloaders or without namespaces, and this approach will struggle with more complex packages that make use of type hinting and dependency injection, but the potential is there.

Considerations

There are some obvious issues with this proposal, number one is that Composer has a minimum requirement of PHP 5.3. I am optimistic that if by the time this proposal goes any further than just my words in this post, that support for PHP 5.2 in WordPress is a thing of the past. And if it isn’t and this project gains traction, then I hope it serves as added weight to the argument of bumping the minimum version.

The next biggest issue is adoption. This will be a seismic shift in how people write plugins, in effectively what has been the Wild West of coding practices up until now. Any third-party code included will need to be done via Composer. The WordPress.org plugin review team will need to be onboard, and some automated reviewing of existing plugins would be an option. Also, as WP-CLI is now a WordPress.org project, it would make sense to add a more defined standard to scaffolding a plugin to include a composer.json file.

If packages were not sandboxed and conflicts were still an issue, then using Composer like this introduces an issue around versioning. A lot of packages have unnecessarily strict SemVer requirements. This isn’t a big deal if you can just make a change or submit a PR. For end users that just cannot install a plugin because its requirements are too strict, it will be hugely problematic. WordPress would need to be smart with its decision making here and override restrictions within sensible limits, so end user experience doesn’t suffer.

Performance is another concern. The servers WordPress typically run on are often optimized for serving web requests, which does not necessarily consume a lot of memory. Composer dependency resolution is a memory intensive process, at times creating a tree of more than 100k+ rule nodes that need to be solved. This can easily require 256MB or even 512MB of memory to run reliably. Luckily this is being improved, but a WordPress installation with plugins requiring multiple dependencies will undoubtedly increase the install time and consume more server resources than simply downloading and extracting a zip file.

Lastly, an issue for me is implementation. I can see what needs to be done but not exactly how to do it. This kind of project needs multiple people to work on it, and I wouldn’t know where to begin with getting this turned into a feature project.

Thank you to Alain Schlesser for helping with this proposal, reviewing my thinking, and answering my (incessant) questions.

Going Forward

The proposal needs lots more planning and thinking through but I believe it is possible. So who’s with me? I know Alain is keen, although he is quite busy at the moment!

If you have any suggestions or comments on the proposal, or this is something you would be interested in helping with then let me know in the comments.