PHP and cURL: How WordPress makes HTTP requests

cURL is the workhorse of the modern internet. As its tagline says, cURL is a utility piece of software used to ‘transfer data with urls‘. According to the cURL website, the library is used by billions of people daily in everything from cars and television sets, to mobile phones. It’s the networking backbone of thousands of applications and services. Unsurprisingly, it’s also a core utility used by WordPress’ own Requests API as well as our own WP Migrate DB Pro.

If you’re curious about the power of the cURL library, how it works with WordPress and what to watch out for (especially on macOS), then you’re in the right place.

What is cURL?

Let’s start by going over what cURL is. cURL is really two pieces, libcurl which is the C library that does all the magic, and the cURL CLI program. Programming languages like PHP include the libcurl library as a module, allowing them to provide the cURL functionality natively.

The libcurl library is an open source URL transfer library and supports a wide variety of protocols. Not just HTTP, but HTTPS, SCP, SFTP, HTTP/2 and even Gopher. Pretty much every protocol you can imagine – cURL supports. cURL actually turns 19 years old in 2017; however, it’s still quite powerful and modern. While it does have its quirkiness and issues, it’s useful for a developer to know how it works and what it does.

I can hear you saying “OK Peter, great, you like this cURL thing, why should I learn about it?”. There are a couple reasons actually!

Reason one is that cURL is neat, and I mean really neat. It’s kind of like an internet swiss army knife. Essentially, if you have a piece of software that needs to make a network request – be it an HTTP POST request to a remote URL or an SFTP file download – cURL is often the simplest choice.

For example, to send a HTTP POST request with a file upload, using the cURL CLI, run:

curl --form name=Peter --form age=34 --form upload=@/Users/petertasker/photos/image-1.jpg http://httpbin.org/post

How about download a large file?

curl -O http://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.14.tar.gz

Or getting the HTTP headers from a server?

curl -I https://deliciousbrains.com

The second reason that cURL is worth learning about is that it is available on pretty much every platform and can be installed quickly and easily. If you’ve got a web server, chances are you’ve got cURL.

For example, if you don’t have cURL already installed on macOS you can easily get it with Homebrew by running brew install curl.

So far we’ve been looking at cURL, the CLI tool, but cURL bindings are also available for most languages, including PHP. If you’re using PHP software that makes network requests, you’ve likely been using cURL!

cURL and PHP

In PHP land, cURL support is like any other module that you may rely on, like mysqli or the GD library.

Most versions of PHP are compiled with cURL by default, but cURL integration is technically an extension, just like mysqli and everything else listed in the extensions section of your phpinfo() output:

cURL phpinfo() output

PHP’s cURL implementation however, leaves a little to be desired. Whereas the cURL CLI tool is relatively straightforward, PHP’s implementation is a bit more complicated.

When working with PHP’s implementation of cURL you’re required to use the curl_setopt() function. This function lets you set cURL options. For example, setting up a HTTP POST request looks like this:

$curl = curl_init( 'https://httpbin.org/post' );
curl_setopt( $curl, CURLOPT_POST, true );
curl_setopt( $curl, CURLOPT_POSTFIELDS, array( 'field1' => 'some data', 'field2' => 'some more data' ) );
curl_setopt( $curl, CURLOPT_RETURNTRANSFER, true );
$response = curl_exec( $curl );
curl_close( $curl );

While not awful, making requests this way can get a little out of hand with larger requests and more complicated CURLOPT_ parameters.

Fortunately, the wonderful PHP community has created libraries that abstract a lot of the complexity away. Two of the more popular networking libraries are Guzzle and Requests. Because Requests supports older versions of PHP and WordPress still supports PHP 5.2 (😩), the Requests library is used in WordPress core.

Requests and WordPress

Internally, WordPress uses the WP_Http class for network requests, which in turn relies on the Requests library. This means that all of the HTTP utility methods like wp_remote_get() and wp_remote_post() use Requests. At a high level, WordPress updates, plugin downloads, plugin updates, and pretty much any upload/download functionality in WordPress core is using the Requests abstraction of the cURL bindings and options.

Let’s take a quick peek at how the Requests library makes HTTP requests. If you pop open wp-includes/class-http.php you’ll be able to check out a lot of the internal plumbing that drives HTTP requests within WordPress. As of WordPress 4.6, the WP_Http::request(); method uses the Requests::request() method.

In WordPress 4.7.3 you can find this call on line 366 of the WP_Http class referenced above.

$requests_response = Requests::request( $url, $headers, $data, $type, $options );

Pretty simple right? Now compare the HTTP POST request below to the raw CURLOPTS method above:

$data = array( 'key1' => 'value1', 'key2' => 'value2' );
$response = Requests::post( 'http://httpbin.org/post', array(), $data );

Much simpler. And if you’re working in the context of a WordPress plugin or theme, you can use the wp_remote_post() function for even more abstraction:

$data = array( 'key1' => 'value1', 'key2' => 'value2' );
$response = wp_remote_post( 'http://httpbin.org/post', array( 'data' => $data ) );

Now we’re talking! wp_remote_post() simply calls WP_Http::request() with POST as the method parameter.

Now as far as Requests works internally, let’s take a look at wp-includes/class-requests.php around line 357.

In the Requests::request() method, you can see that the code first looks for a $transport method. In the case of WordPress’ implementation of Requests there are only 2 default options, cURL and fsockopen, in that order. fsockopen uses PHP streams, and is a fallback for when the cURL extension isn’t installed.

...
if (!empty($options['transport'])) {
    $transport = $options['transport'];

    if (is_string($options['transport'])) {
        $transport = new $transport();
    }
}
else {
    $need_ssl = (0 === stripos($url, 'https://'));
    $capabilities = array('ssl' => $need_ssl);
    $transport = self::get_transport($capabilities);
}
$response = $transport->request($url, $headers, $data, $options);
...

Once the transport is determined, the request is passed into the chosen $transport class. Since we’re covering cURL in this article we’ll quickly take a look at how Requests uses cURL.

Way down in wp-includes/Requests/Transport/cURL.php around line 130 we’ll see how Requests really works. This class highlights how complex working with cURL in PHP can be. Most of the logic within the class is about verifying and handling request and response headers, as well as setting the correct CURLOPTS based on the parameters passed to the method.

A lot of parameter setting is also handled in the Requests_Transport_cURL::setup_handler() method on line 309, which switches over the the options passed in and correctly sets the right CURLOPT:

...
switch ($options['type']) {
        case Requests::POST:
            curl_setopt($this->handle, CURLOPT_POST, true);
            curl_setopt($this->handle, CURLOPT_POSTFIELDS, $data);
            break;
        case Requests::HEAD:
            curl_setopt($this->handle, CURLOPT_CUSTOMREQUEST, $options['type']);
            curl_setopt($this->handle, CURLOPT_NOBODY, true);
            break;
        case Requests::TRACE:
            curl_setopt($this->handle, CURLOPT_CUSTOMREQUEST, $options['type']);
            break;
        case Requests::PATCH:
        case Requests::PUT:
        case Requests::DELETE:
        case Requests::OPTIONS:
        default:
            curl_setopt($this->handle, CURLOPT_CUSTOMREQUEST, $options['type']);
            if (!empty($data)) {
                curl_setopt($this->handle, CURLOPT_POSTFIELDS, $data);
            }
    }
...

Ultimately this all boils down to a curl_exec() call once all of the options have been set. If it looks complicated, that’s because it is! Different servers and hosts have different requirements for HTTP headers and SSL handling. Requests does a good job of trying to accommodate a wide variety of setups.

Additionally, there are some hooks within WordPress’ networking functions that allow cURL options to be overridden if needed. For example, I have the following commented out in an mu-plugin in my local development environment:

 add_action('http_api_curl', function( $handle ){
    //Don't verify SSL certs
    curl_setopt($handle, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($handle, CURLOPT_SSL_VERIFYHOST, false);

    //Use Charles HTTP Proxy
    curl_setopt($handle, CURLOPT_PROXY, "127.0.0.1");
    curl_setopt($handle, CURLOPT_PROXYPORT, 8888);
 }, 10);

In the above example I’m using the http_api_curl hook to first disable SSL certificate verification. This is helpful for when you’re working with a development site that has a self-signed certificate that doesn’t need to be validated.

The second block allows me to proxy through Charles to inspect PHP’s network requests as they go over the wire. Charles is an awesome tool for debugging network requests that lets you see the nitty-gritty details of each request in your local environment.

Common Issues

One of the issues we have seen come up from time to time in support requests for WP Migrate DB Pro revolve around how cURL and SSL work together.

OpenSSL is an industry standard SSL/TLS toolkit for handling encrypted communications. Like cURL, it’s another software library. cURL is compiled with an SSL/TLS toolkit so that it can make connections over the TLS protocol. In the case of WP Migrate DB Pro, this would be when you try to push or pull from a site that has an SSL certificate installed (HTTPS sites). An issue can arise on macOS when cURL isn’t compiled with OpenSSL.

SSL handling with cURL is a huge topic (we have written support documentation on the topic) but with macOS environments the issue is normally that a different SSL/TLS library called SecureTransport is used. This can cause a known issue, and if your local system is using a version of cURL compiled with SecureTransport while your remote server is using OpenSSL, PHP’s cURL bindings will likely throw a SSLRead() error or similar.

On the CLI a little more information is presented, indicating the SSL Handshake failed. You can fake this response by using the cURL CLI command with the insecure -sslv3 version.

curl -sslv3 https://deliciousbrains.com

A good test to see which version of OpenSSL PHP is using is by using grep to search the phpinfo output for the ‘SSL Version’ string.

php -i | grep "SSL Version"

In the output you should see if you’re using OpenSSL or SecureTransport.

SSL Version => OpenSSL/1.0.2j

There can also be issues if both servers are using different versions of OpenSSL and it’s generally best to have matching versions. This can lead to the same errors you would see when using SecureTransport.

Closing Thoughts

So now we know what the cURL library is and how to use it. We levelled up by learning the basics of how PHP and WordPress core use the libcurl bindings and covered the details of a common macOS cURL issue.

What are your thoughts about networking and cURL? Do you have any tips or tricks that you use in your workflow? Let us know in the comments section below!

About the Author

Peter Tasker

Peter is a PHP and JavaScript developer from Ottawa, Ontario, Canada. In a previous life he worked for marketing and public relations agencies. Loves WordPress, dislikes FTP.

  • Peter, great article on cURL and WordPress, how I could have used this a short while ago 😉

    I would add that in my experience some WP Managed hosts have certain limitations on cURL requests from within a WordPress site. So best to test on the major ones if possible.

    • Peter Tasker

      This is true. I think that whole topic of hosts and cURL/PHP support could be a whole other article!

  • Been working a lot with cURL these days. Building an API based on two third party API’s with it. I am not sure about it being the standard or not, but at least it standardized things at the end of the SaaS app I am building.

    Glad to see you write about it.

    P.S. Try the pjson package for coloured output of JSON response.

    • Forgot to add the link for pjson, I guess this tweet serves the purpose.

      https://twitter.com/MrAhmadAwais/status/847091316513230848

    • Peter Tasker

      Nice! I have a Chrome addon that makes JSON pretty, but this looks neat for the CLI. Cheers!

      • Yup, for Chrome, JSON formatter is the best one. It even makes the JSON available in console as a JS Object, I have a video that I made about it, will be sharing soon on my blog.

  • Nocare

    Doing some work recently I discovered this: http://php.net/manual/en/function.curl-setopt-array.php
    Next time I need to with wordpress i’ll definitely be checking out their requests class.

    • Peter Tasker

      Requests is definitely worth a look if you’re working in WordPress land. It simplifies a lot of the cURL ugliness. Good tip on the `curl_setopt_array()` function!

  • guy

    Thanks Peter, interesting article. It might help me understand why WP Migrate Pro is failing to connect to the Delicious Brains API from my staging environment if I use SSL combined PHP7.2

    • Peter Tasker

      Yeah, debugging cURL issues can be tricky. Your best bet may be to write a simple cURL PHP script and log out the errors.

  • Josh McKee

    Great article! I wish to consume your delicious brains.

  • I have a local server environment set up without MAMP and was getting errors thrown using Migrate Pro DB and used this SO guide to solve the conflict by using homebrew to install a version of cURL which uses OpenSSL instead of SecureTransport: http://stackoverflow.com/questions/26461966/osx-10-10-curl-post-to-https-url-gives-sslread-error

    • Peter Tasker

      Hi Alex, that post is indeed a good one. We reference it in our documentation as well.

      • Awesome. I edited the SO article to update it to use
        brew install –with-homebrew-curl –with-httpd24 php55

        because without the –with-httpd24, the /usr/local/opt/php55/libexec/apache2/libphp5.so file didn’t get written

        That article and all your documentation and these blog posts are super helpful! Keep up the great work 🙂

  • A little tip on creating cURL requests where you don’t want to wait for the response before continuing your code execution, set CURLOPT_TIMEOUT to 1. That way it’ll trigger the request and timeout after one second. Useful if you want to run a series of requests without needing to wait for their response.

    • Bob Chip

      I like that. How about a CURLOPT_TIMEOUT even less?

      • I’m sure you can go less, there is a CURLOPT_TIMEOUT_MS setting that allows you to set timeout in milliseconds.

  • I created a ticket (40142) to hopefully have more HTTP utility methods (DELETE, PUT, etc) in WordPress Core.

    Ticket Link: https://core.trac.wordpress.org/ticket/40142

    • Peter Tasker

      Nice Brandon! That would be super helpful. I did notice when looking through the Requests implementation that there were some HTTP methods missing.