netcat to 👀 peep HTTP: gotchas
netcat is an old tool sometime qualified as theswiss army knifeof network utilities.
- My memory not so long ago
That was what I remembered as I wondered how I could capture HTTP traffic between my reverse proxy and one of my service on my snowflake. After a quick internet research, I found several source citing command similar to this one:
mkfifo answer touch in out nc -vv -l -k -p 8001 <answer | tee -a in /dev/fd/2 | nc service.com 80 | tee -a out answer
It’s quite a lengthy command. Let’s decompose this:
mkfifo answer: Make a named pipe1. In essence the same thing than the regular unix pipe
|, except it exists on the filesystem thus you reference it by its name.
touch in out: Create files named
out, respectively for persisting the streams of incoming request and outgoing responses.
nc -vv -l -k -p 8001 <answer:
netcat2 in listen mode will print incomming request content to
stdinback to client.
It listens to tcp port 8001, then reads value of named pipe
answerto send it back to the client.
-kenable netcat to continue listening after connection close (netcat can accept several successive connections)
tee -a in /dev/fd/2:
stdinto write back to
stdoutand files in parameters.
stdinhere being content of HTTP requests, it will write it to
infile and to
stderr, to see request content in console). Implicitly
teewill also print to
nc service.com 80:
service.comon port 80 and write
stdincontent on the socket. Server response is then written on
stdinhere being HTTP request content which is chained up until here.
tee -a out answer: You know what
teedoes now. Note that we write back server response to the
Everything from the first
nc is chained through unix pipes
| which redirects content of left hand side
stdout to right hand side
If you have difficulties wrapping your head around it, try playing with individual commands like
tee, read about unix pipes and named pipes.
Which flavor of netcat?!
Alright, netcat has different implementations, namely openbsd and gnu. Examples in this article assumes you have openbsd netcat.
Knowing which implementation of netcat you use is important; several options are behaving differently. Use `nc -h` to know which version you use and what does the options do.
On Ubuntu, both versions exists with packages
netcat-traditional. Don’t forget to
sudo update-alternatives --config nc to use appropriate implementation.
Additionnaly, you have a similar tool named `ncat` which comes from `nmap` project. It has interesting options such as `-o` which dump a netcat session to a file, `-k` which makes the listening socket accept several connections and `-c` which execute command on incomming connection. Actually, `ncat` is more suited to our task since we can achieve our little peeking proxy with this command:
ncat -vv -l -k -p 8001 -c "ncat service.com 80" -o dump
Server closes the connection
Our approach works… but it’s quite fragile.
In fact, the program which connects to distant server to communicate quits when connection ends, it does not reconnect automatically.
If connection to
server.com ends, our chain is broken and incomming requests to
8001 will just hang.
In order to observe this behavior, you can use header
Connection: close on your HTTP request.
$ nc -vv -l -k -p 8001 <answer | tee -a in /dev/fd/2 | nc google.com 80 | tee -a out answer & # Google will do fine for testing $ http localhost:8001 # using httpie # Request is OK $ http localhost:8001 # Still OK $ http localhost:8001 "Connection: close" # Still OK, although server just closed the connection, any other request will hang $ http localhost:8001 # NOK, Request hangs
You can use
ss command to observe opened socket throughout the testing
watch ss -ap 'sport == :8001 or dport == :8001 or sport == :80 or dport == :80' # poor man monitoring of socket on 8001 and 80
It doesn’t work on fish shell
If you are using fish shell you may have noticed that
nc -vv -l -k -p 8001 <answer | tee -a in /dev/fd/2 | nc google.com 80 | tee -a out answer doesn’t even start a listening socket (connection refused on localhost:8001).
This is because, contrary to bash, fish tries to open redirected files in the parent process, before forking different part of the pipeline, resulting in a deadlock.
One can rewrite the command to avoid redirected files in order to be compliant with fish:
cat answer | nc -vv -l -k -p 8001 | tee -a in /dev/fd/2 | nc google.com 80 | tee -a out answer
I will let you explore this stackoverflow thread for more details.