netcat to 👀 peep HTTP: gotchas

netcat is an old tool sometime qualified as the swiss army knife of network utilities.
- My memory not so long ago

That was what I remembered as I wondered how I could capture HTTP traffic between my reverse proxy and one of my service on my snowflake. After a quick internet research, I found several source citing command similar to this one:

mkfifo answer
touch in out
nc -vv -l -k -p 8001 <answer | tee -a in /dev/fd/2 | nc service.com 80 | tee -a out answer

Command explanation

It’s quite a lengthy command. Let’s decompose this:

Everything from the first nc is chained through unix pipes | which redirects content of left hand side stdout to right hand side stdin.
If you have difficulties wrapping your head around it, try playing with individual commands like nc, tee, read about unix pipes and named pipes.

Possible issues

Which flavor of netcat?!

Alright, netcat has different implementations, namely openbsd and gnu. Examples in this article assumes you have openbsd netcat.

Knowing which implementation of netcat you use is important; several options are behaving differently. Use `nc -h` to know which version you use and what does the options do.

On Ubuntu, both versions exists with packages netcat-openbsd and netcat-traditional. Don’t forget to sudo update-alternatives --config nc to use appropriate implementation.

Additionnaly, you have a similar tool named `ncat` which comes from `nmap` project. It has interesting options such as `-o` which dump a netcat session to a file, `-k` which makes the listening socket accept several connections and `-c` which execute command on incomming connection. Actually, `ncat` is more suited to our task since we can achieve our little peeking proxy with this command:
ncat -vv -l -k -p 8001 -c "ncat service.com 80" -o dump

Server closes the connection

Our approach works… but it’s quite fragile.

In fact, the program which connects to distant server to communicate quits when connection ends, it does not reconnect automatically.
If connection to server.com ends, our chain is broken and incomming requests to 8001 will just hang.

In order to observe this behavior, you can use header Connection: close on your HTTP request. Full example:

$ nc -vv -l -k -p 8001 <answer | tee -a in /dev/fd/2 | nc google.com 80 | tee -a out answer & # Google will do fine for testing
$ http localhost:8001 # using httpie
# Request is OK
$ http localhost:8001
# Still OK
$ http localhost:8001 "Connection: close"
# Still OK, although server just closed the connection, any other request will hang
$ http localhost:8001
# NOK, Request hangs

You can use ss command to observe opened socket throughout the testing

watch ss  -ap 'sport == :8001 or dport == :8001 or sport == :80 or dport == :80' # poor man monitoring of socket on 8001 and 80

It doesn’t work on fish shell

If you are using fish shell you may have noticed that nc -vv -l -k -p 8001 <answer | tee -a in /dev/fd/2 | nc google.com 80 | tee -a out answer doesn’t even start a listening socket (connection refused on localhost:8001).

This is because, contrary to bash, fish tries to open redirected files in the parent process, before forking different part of the pipeline, resulting in a deadlock.

One can rewrite the command to avoid redirected files in order to be compliant with fish:

cat answer | nc -vv -l -k -p 8001 | tee -a in /dev/fd/2 | nc google.com 80 | tee -a out answer

I will let you explore this stackoverflow thread for more details.


  1. an article explaining named pipes ↩︎

  2. openbsd netcat man ↩︎