Dependency Management and WordPress: A Proposal

‘Dependency hell’ is a problem faced by all software, and it has been rearing its ugly head in the WordPress space over the last few years with more and more plugins using third-party libraries of code.

We come across this issue every couple of months with our Amazon S3 plugin WP Offload S3. It’s a very real problem for Delicious Brains and can serve as a good concrete example of the issue.

The Problem

Our plugin transfers media to Amazon S3 when you upload to the WordPress media library, and to do that we make use of the AWS SDK (bundled as the Amazon Web Services plugin). The SDK went through some major changes between versions 2 and 3, resulting in the SDK bumping its minimum required version of PHP from 5.3 to 5.5. With 42.7% of all WordPress installs running on servers with PHP 5.2, 5.3 or 5.4, stipulating a minimum of PHP 5.5 just wouldn’t work for our plugin.

When customers use Offload S3 along with another plugin that relies on and bundles the AWS SDK, unless that plugin is using v2, we encounter an issue of incompatible libraries. WordPress loads plugins in the order they were activated, and PHP loads files as they are required and classes as they are called (when using autoloaders), so it is a real lottery which plugin calls the SDK code first. The other plugin is the one that suffers.

The most frustrating thing about this issue is that it’s caused by having the best of intentions! Developers use third-party code to be efficient and avoid reinventing the wheel. The code has been written by others and used and battled tested by many.

Why not just include the library in your plugin with the namespace or classes prefixed? That would work, no clashes would occur. The Ibericode guys did this to include the Pimple package in their plugin. This works fine for them as Pimple is a small two file package. But this just doesn’t scale. Doing this manually for something as large as the AWS SDK would be too much work, and would require doing it over and over every time the package is updated. Which is why people turn to Composer in general, as it makes it easy to manage and integrate third-party libraries into a codebase. But unfortunately Composer is often viewed as the culprit of this very problem.

This is NOT a Composer Problem!

The WordPress community has a hard enough time already trying to get onboard with Composer (unlike the rest of the PHP world), without it getting tarred with the wrong brush!

The problem is already there, without Composer, you just don’t notice it as prominently. Packaging any third-party library with your public plugins, whether you use Composer or not, will cause issues with WordPress.

Alain SchlesserPost Status Slack.

Peter Suhm wrote about the dangers of using Composer with WordPress because of this very issue, but luckily Coen Jacobs jumped in to Composer’s defence to point out his error. This sparked some further discussion between Peter and Coen which helped promote a potential solution from Coen, ironically using Composer. It leverages the Composer Merge Plugin, built for Wikimedia, to merge multiple composer.json files at runtime.

This makes perfect sense, as Composer is the de facto PHP package and dependency manager, but it needs some bending to make it work across a whole WordPress site which might have multiple plugins using their own composer.json file to manage their dependencies. This is only a proof of concept from Coen but I love the direction here, and I will come back to solving the problem with Composer later.

The Ideal Approach

Ryan McCue wrote a great piece about how he would solve the plugin dependencies problem within WordPress, that didn’t focus so much on the technical implementations but more on what was required from an end user experience. To summarize, the solution needs to:

  • Catch potential conflicts with dependencies up front before a plugin is installed
  • Installation should automatically take care of installing a plugin’s dependencies
  • Updates to the plugin should be dependency aware, and stop and warn if updates will create conflicts

WordPress as a project is very much focused on the user and not the developer. ‘Decisions not options’ is a mantra that has lead many of the project’s features in recent years, automatic background updates being an important one. Dependency management needs to be handled in the same fashion, where the user doesn’t have to think too much and it just works for them.

This type of solution is a big ask, covering a lot of ground and facing some large challenges. Critically, Ryan makes the point about where the solution needs to sit – WordPress core:

The end goal here is core integration. If the solution doesn’t end up in core at the end, the project has failed, as it’s not ubiquitous. If this happens, throw out what you need and try again, but it must be in core to be a viable solution for many users.

A Proposal

I’m not the greatest WordPress developer. I can use Composer but I am no expert. I’m certainly not a PHP internals guy and I’m just about keeping up with this problem as I write it, but… I think the problem can be solved and here is my high-level proposal for a solution and possible core feature plugin.

As I mentioned before, I feel Coen was definitely on the right track with his Composer based solution and some of this is directly inspired by his project, but I think we need to take it further.

1. Third-Party Code via Composer

Any plugin or theme using third-party code needs to define their dependencies using a composer.json file, and should not distribute their vendor directory.

2. Composer in Core

WordPress would have to come bundled with Composer removing the need for end users to have to install it manually. This would be like any other library WordPress requires, such as Requests or Simple Pie.

3. Custom Composer Behavior

We would then need to heavily customize Composer’s core behavior to integrate with WordPress. Specifically, this would integrate with the plugin/theme installation process to read the plugin’s composer.json file, merge it into all the others and install the dependencies.

It would need to catch the conflicts and reply back to the user (via the admin UI or CLI) to inform them of issues. It would also need to handle this when a plugin is being updated.

Drupal has had the similar Composer Manager since 2013:

Composer Manager allows each contributed module to ship with its own composer.json file, listing the module-specific requirements. It then merges the requirements of all found modules into the consolidated composer.json file. This results in a single vendor/ directory shared across all modules which prevents code duplication and version mismatches.

4. Package Sandboxing

The last part of my proposal is a bit more blue sky thinking, but some recent work from Coen has given me hope. The ideal scenario for WordPress and users is to not have to deal with conflicts at all. Every plugin should be able to run their dependencies safely without the threat of interfering with others. Loading multiple versions of the same class just isn’t possible in PHP, but prefixing namespaces or classes so they can be used in isolation is something that might work, and has been discussed previously from a Composer perspective.

Mozart is a new script from Coen that automatically prefixes PSR-0 and PSR-4 compatible packages with a custom namespace, so they can be safely used by a plugin. Building this kind of functionality into the Composer part of WordPress would mean all dependencies used by plugins would be sandboxed and no conflicts would arise. Of course, more work would need to be done to Mozart to support other types of packages, such as those with classmap autoloaders or without namespaces, and this approach will struggle with more complex packages that make use of type hinting and dependency injection, but the potential is there.

Considerations

There are some obvious issues with this proposal, number one is that Composer has a minimum requirement of PHP 5.3. I am optimistic that if by the time this proposal goes any further than just my words in this post, that support for PHP 5.2 in WordPress is a thing of the past. And if it isn’t and this project gains traction, then I hope it serves as added weight to the argument of bumping the minimum version.

The next biggest issue is adoption. This will be a seismic shift in how people write plugins, in effectively what has been the Wild West of coding practices up until now. Any third-party code included will need to be done via Composer. The WordPress.org plugin review team will need to be onboard, and some automated reviewing of existing plugins would be an option. Also, as WP-CLI is now a WordPress.org project, it would make sense to add a more defined standard to scaffolding a plugin to include a composer.json file.

If packages were not sandboxed and conflicts were still an issue, then using Composer like this introduces an issue around versioning. A lot of packages have unnecessarily strict SemVer requirements. This isn’t a big deal if you can just make a change or submit a PR. For end users that just cannot install a plugin because its requirements are too strict, it will be hugely problematic. WordPress would need to be smart with its decision making here and override restrictions within sensible limits, so end user experience doesn’t suffer.

Performance is another concern. The servers WordPress typically run on are often optimized for serving web requests, which does not necessarily consume a lot of memory. Composer dependency resolution is a memory intensive process, at times creating a tree of more than 100k+ rule nodes that need to be solved. This can easily require 256MB or even 512MB of memory to run reliably. Luckily this is being improved, but a WordPress installation with plugins requiring multiple dependencies will undoubtedly increase the install time and consume more server resources than simply downloading and extracting a zip file.

Lastly, an issue for me is implementation. I can see what needs to be done but not exactly how to do it. This kind of project needs multiple people to work on it, and I wouldn’t know where to begin with getting this turned into a feature project.

Thank you to Alain Schlesser for helping with this proposal, reviewing my thinking, and answering my (incessant) questions.

Going Forward

The proposal needs lots more planning and thinking through but I believe it is possible. So who’s with me? I know Alain is keen, although he is quite busy at the moment!

If you have any suggestions or comments on the proposal, or this is something you would be interested in helping with then let me know in the comments.

About the Author

Iain Poulson

Iain is a WordPress and PHP developer from England. He builds free and premium plugins, as well as occasionally blogging about WordPress. Moonlights as a PhpStorm evangelist.

  • If this were a feature project, how would you take a design-first approach with it?

    Also, given this original problem statement:

    When customers use Offload S3 along with another plugin that relies on and bundles the AWS SDK, unless that plugin is using v2, we encounter an issue of incompatible libraries. WordPress loads plugins in the order they were activated, and PHP loads files as they are required and classes as they are called (when using autoloaders), so it is a real lottery which plugin calls the SDK code first. The other plugin is the one that suffers.

    What’s the ideal outcome / user experience for the end user? In what ways would dependency management improve the user experience?

    • i think the user experience of things mysteriously (mysterious to a non-dev regular plugin user) is pretty poor. Dependency management should allow things to simply work without conflict and that does improve the UX significanlty

    • The ideal outcome is not having the conflict in the first place (see section 4). The next best thing is having WordPress tell the user there is a conflict between plugins upfront. This is more ideal than the current situation where a plugin will just break, error, white screen, not work, or silently fail and the user is completely in the dark about what is wrong. Resulting in a trip to the support channel of one or both plugins, if the user identifies the cause themselves.

    • I think the UX would be simply raising a message on the install page that these two plugins are incompatible with each other. The user doesn’t need to know the exact library and version conflict. The alternative is them realizing there is a problem when they experience a fatal error, or if they are lucky one of the two plugins blaming the other one for a conflict.

      • Coen Jacobs

        This has been discussed a number of times in Trac tickets already. Not being able to install a plugin because it has some invisible conflict with another plugin, is not acceptable for the core team (anyone got a recent comment link for this?).

        The end user is never going to be held accountable for any incompatibilities between plugins, within the WordPress ecosystem. That’s just never going to change. So, I set on a quest to find the way to solve this, by making it a problem for the developers to fix. Sandboxing/prefixing turned out to be the least bad problem and so Mozart was born.

        • Yeah, I’ve seen those comments multiple times, I just disagree with them. Plugin conflicts are so prevalent even without including issues due to version conflicting libraries. They just tend to be hidden a little better than fatal errors.

          Saying that we should minimize the conflicts is an admirable goal, and prefixing is probably the only good way to do it, but that shouldn’t stop the simple and straightforward solution being implemented first since it will be needed anyways as not all packages will be able to be prefixed. Why can’t this be done iteratively?

        • mzalewskinz

          But doesn’t that already happens? If PluginB conflicts with PluginA and generates a fatal error, the plugin activation will fail. (Ignoring the silent/delayed conflict issues for now)

          Surely dependency conflict message would be a lot more user friendly than “A fatal error occurred”?

          • The library conflict won’t always result in a fatal on activation. Plugin B might simply not work as intended as it is using the wrong library.

  • Caleb Smith

    This has been a big issue for me in my development. There are a couple of libraries that are key for API connections that would take so much effort to recreate for a simple plugin that I want to provide to the community. I don’t want to cause issues for other plugins, but I also haven’t found a good alternative for dealing with dependencies. I’ll keep an eye on your posts and look forward to any resolutions you come across!

  • Ed

    I know this is only a small part of the deep integration you’re talking about, but we’ve had great results using composer with johnpbloch’s wordpress fork and wpackagist. They are at the core of the Roots.io wordpress stack, and go a long way towards alleviating the problem (at least for developers).

    Ironically, the only plugins we can’t manage with composer are premium ones like WP S3 Offload and DB Migrate Pro 😀

    • Using Composer to manage your whole site (WP, plugins and themes), like Roots/Bedrock does is great but it doesn’t solve the issue here. If, with Composer, you require plugin A which bundles library X v1.0, and you also require plugin B which bundles library X v2.0, depending on how those plugins work with the different library versions can result in the conflict described in the post.

      Also our premium plugins are Composer friendly 🙂 https://deliciousbrains.com/wp-migrate-db-pro/doc/installing-via-composer/

      • Ed

        Well, this just made my day – I’m off to update my composer files 🙂

        Thanks!

    • Tobias Geisler

      Also recently switched to Bedrock. The approach of the boilerplate is amazing but it does not solve the issue of premium plugins, which is a whole other kind of issue, as Iain just pointed out.

      To solve the problem, you mentioned, Ed, we basically deployed a Satis (https://github.com/composer/satis) instance where we add the Deliciousbrains (and WPML) download links as suggested in the post Iain just pointed out. So, in our project we just have to add our Satis repo to our repositories. This way, premium plugins are managed centrally in our Satis instance. To make things even simpler, we wrote a script to parse a easy-to-maintain yaml file into satis.json (JSON is nice, but tedious to write). It also scans a dist directory on the server and inserts previously uploaded zip packages automatically into the satis.json. We use this for premium plugins which cannot be added by URL (such as WooCommerce Add-ons at the time of writing).

      Would love to see premium plugin providers like Deliciousbrains host their own Composer Repo (accessible maybe by access token) to host premium plugins, so integrating them into composer managed WP installs would be a breeze – and older versions could also be depended on. At least until a real solution comes along to manage premium plugins through composer.

      A big thanks to Deliciousbrains for for offering the end point, though, WooCommerce doesn’t even have it anywhere in the roadmap as their support assured me. Linking their download links directly in composer.json leads to broken archives which do not unzip.

      • Ed

        I’ve been thinking along similar lines for a while now. I’ll give Satis a go – thanks for the suggestion 🙂

      • We certainly have it on our radar to improve our Composer support, specifically to allow specific version downloads, but it’s not high up the priority list 🙂

        • Tobias Geisler

          Awesome! Looking forward to hearing about it sometime in the future.

      • mikelking

        Yes Premium plugins can be a RPITA when it comes to composer updates. What we have done at my company we have a private git repository for each premium plugin. We are using composer to require those into the appropriate site, but it is a lot of DevOps management on the initial as we have to make sure we properly tag each version correctly and it’s more to maintain.

        Take everyone’s favorite SEO plugin. Every time there’s an update we must request the lasted download link. Then unpack it, commit it to our repo and tag it. It’s a few hours of turn around but since we have an unlimited license all of our sites receive the update from our private repo and we can push out the update relatively quickly form there.

        The are alternatives, but I think all get messy. As each premium provider has their own way of doing things. We’ve just ensured it’s standardize at our edge.

        Cheers,
        m

      • mikelking

        Your Satis link is borked. It lands in GitHub 404 because the paren is being appended.

        https://github.com/composer/satis

    • Another solution for managing premium plugins is https://github.com/blazersix/satispress .

      It has the advantage that you can make use of the standard WordPress update mechnism to load new versions.

      • Good shout, Alain – I use that myself 🙂

      • Tobias Geisler

        Great hint, how did I miss this?!
        I assume one sets up a WordPress instance just to expose the packages, right? Only drawback would be single-install licences I guess.

  • Paul Benbow

    Doesn’t roots.io already deal with this?

    • Roots allows you to manage your whole site with Composer, but the problem can still occur. See http://disq.us/p/1h7b5gg 🙂

    • Tobias Geisler

      With Bedrock? Iain is referring to dependencies of plugins, not of entire WP projects (which bedrock kinda solves).

  • i would be extremely happy to see something composer based in the core

    we were able to solve the similar jQuery issue years ago and now it’s time to solve it with generic dependencies

  • Great article. And it is only one weak part of the problem: think about the numerous JS libs called, sometimes twice with different versions and, as a user, I am horrified that I can not master what my website is calling outside its domain.

  • Ugh. I am one of the vocal opponents of including Composer in WordPress. I won’t repeat my objections here because they can easily be found in Trac but I will leave this ditty:

    “A WordPress developer had a problem managing dependencies and thought ‘Wow, I can use Composer to solve that problem!’ Now the developer has two problems.”

    • BTW, I addressed the dependency problem in Imperative back in 2012/13. But I really needed some changes in core to make it work flawlessly and I could get no interest from the powers that be to see those changes in core. So I gave up on the issue and focused my business away from building plugins for clients for distribution.

      Imperative addresses the first point from Ryan McCue: “Catch potential conflicts with dependencies up front before a plugin is installed.”

      That said, with changes something Imperative-like would be a much easier for people in the WordPress world and would not require all the hackery that is required to get Composer to accept and allow the WordPress defaults, nor would it require all WordPress plugins to go back and implement a composer.json file (yet another flawed aspect of Composer, IMO.)

      • Wouldn’t plugins need to retrofit to the Imperative method?

    • Coen Jacobs

      Mike, I know you are an opponent of Composer in WordPress, but let me repeat the most important quote from this article, in this whole ‘why use Composer’-debate:

      > The problem is already there, without Composer, you just don’t notice it as prominently. Packaging any third-party library with your public plugins, whether you use Composer or not, will cause issues with WordPress.

      It’s not a problem with Composer.

  • chrismccoy

    rarst has a good page regarding composer in wordpress

    http://composer.rarst.net/

  • mzalewskinz

    See https://github.com/mzalewski/wppm

    It bundles Composer and uses Composer’s dependency solver to resolve plugins conflicts (or notify the user that PluginA cannot be activated).

    My question is: Do we want to allow users to automatically resolve conflicts by downloading different dependency versions? Composer is a dev tool, not a user tool.
    While I definitely think WordPress needs to add some kind of dependency management to handle conflicts, I don’t think it should be allowed to modify the WP installation and potentially introduce some site-breaking changes that could be a nightmare to debug.

    • Thanks for the link, that looks like an awesome piece of work – I wish I had seen it sooner!

      Different dependency versions can’t be loaded together without some form of prefixing, and if that were possible I don’t think the user needs to know that is what is happening, WordPress just needs to do it without worrying the user.

  • Nick Barrett

    This is a great well written article. Like many of the commenters below, we have been using Bedrock from roots.io for our base install.
    I recently had the need to integrate a Laravel app with a WordPress backend. Using the s3 drivers from League’s Flysystem for Laravel 5 requires AWS SDK v3, and the delicious brains plugins use v2 – basically an unsolvable dependency conflict! It’s time WordPress catered more for developers by integrating with the tried and tested industry standard solutions out there like Composer.

  • mikelking

    Iain,

    I feel that this is a big problem for WordPress in that all of the other serious PHP based projects are composer friendly and we’ve all just kind of done our own thing. I started down this road (see: https://github.com/mikelking/dummy). In any event I got dummy working to the point of building a working site out of several different layers of composer abstraction. Ultimately getting hung up on some of composer’s ‘There can be only one’ obliterate the destination directory magick and making certain it works in ALL dev environments.

    Keep in mind that .org has abandoned PHP5 altogether (as an end of life prospect). See https://wordpress.org/about/requirements/ which stats the following:

    * PHP version 7 or greater
    * MySQL version 5.6 or greater OR MariaDB version 10.0 or greater
    * HTTPS support

    I don’t think that you need to get .org onboard w/ composer out of the gate. My vision for the dummy project was to make it easy for a developer to fork dummy and then define their dependencies in the composer manifest and build their plugin or theme without having to worry about the other details.

    The secondary goal was to make it possible for anyone to to fork dummy to easily define a new site in a consistent manner using the composer manifest and then use tools like jenkins or deploybot to ship the code to their hosting provider.

    In both cases the composer magick happens before the code hits the web heads.

    I think there are some synergies to what you want to do and what I’ve already done, thus if you want to discuss hit me on twitter:

    @mikelking:disqus

    • Hey Mike, thanks – I will take a look at your project 🙂

      Unfortunately, WordPress haven’t abandoned PHP5 altogether, they still support backwards compatibility to 5.2 but ‘advise’ PHP 7 should be used.

  • Xedin Unknown

    Good article, and I would love to see Composer in WP, as well as contribute to the development of the associated software, etc. If someone takes this on, feel free to get in touch with me. There are however a few points I would like to make:

    1. Mozart is not a good solution. The namespace is part of a class’ FQN, the whole point of which is to be able to identify a class uniquely. PSR-4 makes this possible by requiring a vendor/package prefix. Of course, it is possible for 2 vendors to have the same name, and for them to have 2 projects with the same name also. However, this is quite unlikely, and if this happens it can easily be fixed. Therefore, I don’t see any problem here. And no need to screw with other people’s code. Leave the namespaces as is, and just use PSR-4. Clever people have already created all the standards and tools we need to make this possible. It’s just that WordPress developers are so attached to usability, that they tend to forget about practicality. In this case, trying to make any combination of plugins possible is something that is unnecessary. There is such a thing as incompatibility, and it’s alright. It’s there for a reason.
    2. No need to “heavily customize” anything. Of course, there is need for some customization. The `composer/installers` package works great for these things. `oomphinc/composer-installers-extender` can be used to avoid having to write an installer, if needs be, even if temporarily. `wikimedia/composer-merge-plugin` can allow us to merge configs from multiple locations, which looks like a good way to solve some of the problems. And we’ll probably end up writing a Composer plugin, yes. But this is not necessarily huge. Many problems can be solved via convention. On the other hand, I understand that this is probably a case of just abusing the word “heavily”.
    3. Plugins can also be required. There’s no problem with requiring other plugins, if they provide something useful. A plugin is just a package. However, it would be much better if plugin dependencies were libraries instead. And for this, developers need to write better code. So what? Many people have been writing good code for like ever. If some people can’t, that’s OK. Maybe, it’s even impossible to provide a way for consistent management of plugins via Composer. Maybe, that’s OK too. As long as it’s possible to manage plugin dependencies through WordPress with Composer.
    4. Develop the functionality standalone. As Carl Alexander says, you’ve got to have really thick skin to be able to stand against the human factor when contributing to WP, and I am not willing to put myself through that torture. Instead, just develop a standalone tool, and wrap it in a WordPress plugin. If WP Core wants to integrate it, they can. If not, well, what’s the problem? The tool would be available with an open-source GPL-compatible license such as MIT, and those who want to use it will use it. Which I’m guessing will be a great many.

    Anyhow, like I said, I’m available to help out here, whether it’s hands-on dev work, or planning, project management, whatever. You can check out my work at RebelCode and Dhii to learn about the way I do things.

    • Hey Xedin,

      Regarding your first point, I think you misunderstood what Mozart tries to achieve. This is not about making two separate packages work that have conflicting namespaces. This is about dealing with the fact that WordPress does not have one central location to use Composer. So, if you use Composer at the plugin-level, it is used several times in isolation, and cannot achieve its main mission of only loading 1 version of a given dependency used across all plugins, because it doesn’t know about the other plugins. This causes libraries to be included several times, once for each plugin, and potentially in different, conflicting versions. As the autoloader still only loads only one single version, the actual version you get is very random and will very probably lead to bugs.

      Mozart tries to solve this by prefixing each package, and thus loading all of them, one for every plugin that uses it, so that every plugin gets the version it needs.

      • Xedin Unknown

        Hi Alain,

        This is what I meant, since it is effectively the same things. Two packages that have colliding namespaces will work together, because the namespaces are prefixed by Mozart. I believe my point still applies.

        And yes, if we use Composer for each plugin separately, this is a problem somewhat. But we are talking about installation-wide management, aren’t we? This problem should go away on its own.

        • Yes, installation-wide management would be the ideal outcome, indeed.

          Mozart is more of a plan B if all efforts to introduce a centralized Composer fail. Mozart tries to make Composer at the plugin-level work across multiple plugins.

  • This is a great topic and I would love to help see this happen. I’ll do what I can to support it as a project, but if my efforts with contributing to WP-CLI are anything to go by, I’ll mostly be logging issues as I find them in using the solution.

    I’m currently developing a plugin for a client that bundles the AWS SDK, I’d be interested to hear how you have worked around the problem in its current incarnation. Or if you get a support request do you just ask the conflicting plugin developer to update their SDK version?

  • Addition of composer in WP great..!!

  • I would love to see composer in WP