Testing protocols and gathering information with cURL

In the Unix/Linux world, a huge number of people use the cURL command. It’s very useful, but do you really know how powerful it is?

What does cURL stand for?

First of all, curl is actually written cURL, short for “client URL request library”.

The tool first appeared in 1997 under the name urlget.
cURL is mainly used to test a protocol and check that content is available, but it can also collect information (by downloading it, for example).

The program relies on the libcurl library, written in C.

Using cURL

Most of the time, we use cURL in a very basic way, to check that a site is up and online.

For example, to check that https://safecode.fr is available, I simply run:

$ curl https://safecode.fr

The command returns the page’s source code (shortened here for the example).

<!DOCTYPE html>
<html lang="fr-FR">
<head>
<title>Safecode - Plus de sécurité dans un monde connecté</title>
etc...

Note that the http:// prefix in http://safecode.fr sets the protocol to use, but it’s the default anyway, so we can also run:

$ curl safecode.fr

<!DOCTYPE html>
<html lang="fr-FR">
<head>
<title>Safecode - Plus de sécurité dans un monde connecté</title>
etc...

This article doesn’t aim to show every possible option (there are loads), but here are two more practical uses.

With cURL, you can retrieve the response headers, for example, with:

$ curl --head https://safecode.fr

HTTP/2 301
date: Wed, 18 Mar 2020 13:49:00 GMT
content-type: text/html; charset=UTF-8
set-cookie: __cfduid=d8f73dac546c5e7133b80de13a299b7ea1584539340; expires=Fri, 17-Apr-20 13:49:00 GMT; path=/; domain=.safecode.fr; HttpOnly; SameSite=Lax; Secure
x-redirect-by: WordPress
set-cookie: xRqKjnJzEhr=AUgY%2AozumXR; expires=Thu, 19-Mar-2020 13:49:00 GMT; Max-Age=86400; path=/
set-cookie: tesuYAwgFhEaoG_=u.2NWG3nSsg4JQx; expires=Thu, 19-Mar-2020 13:49:00 GMT; Max-Age=86400; path=/
set-cookie: lzadmWBXZKgc=XWNCr%5B7lh09m; expires=Thu, 19-Mar-2020 13:49:00 GMT; Max-Age=86400; path=/
location: https://safecode.fr/
cf-cache-status: DYNAMIC
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 575f6a1e1ee9ce73-LHR
alt-svc: h3-27=":443"; ma=86400, h3-25=":443"; ma=86400, h3-24=":443"; ma=86400, h3-23=":443"; ma=86400

This command can reveal a lot of valuable information about a website. For instance, you can see that the site runs on WordPress and uses Cloudflare as its CDN. That’s exactly the kind of reconnaissance I used in the E-corp OSINT challenge.

The returned data can easily be saved. To store the output in test.txt, just run:

$ curl -o test.txt https://safecode.fr
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 16123    0 16123    0     0  35174      0 --:--:-- --:--:-- --:--:-- 35126

The -o option, for output, sets the output path, i.e. where we want to store the response.

Specifying a port

As with many other tools, you can specify a port. The default port for HTTP is 80, and the default for HTTPS is 443.

$ curl https://safecode.fr
$ curl https://safecode.fr:443

The two commands above therefore return the same result. If HTTPS isn’t served on the default port, use url:<port>.

If the port is 444 instead of 443, the command becomes:

$ curl https://safecode.fr:444

---

Many supported protocols

DICT, FTP, FTPS, Gopher, HTTP/1[10] (with HTTP/2 support), HTTP POST, HTTP PUT, HTTP proxy tunneling, HTTPS, IMAP, Kerberos, LDAP, POP3, RTSP, SCP, SMTP, SMB, FTP, SFTP, TFTP, Telnet

If this command has caught your interest, man is your best friend. And to go further with HTTP testing, see my guide to load testing with HTTPerf.

$ man curl

---

Further reading on cURL

• Wikipedia
• Usage examples (in French)