Saturday, May 25, 2013

Redirecting Standard Input and Output and Why This Makes Netcat Really Useful

Check out this really basic C program:

#include <stdio.h>
int main()
    int c;
    while ((c = fgetc(stdin)) != EOF)
        fputc(c, stdout);

It takes whatever you type in and prints it back to you. So, if you compile it to a binary called "exe" ($ gcc [source file].c -o exe), you can see it repeat everything you type, for example:

$ ./exe
something else
something else

You can hit CTRL+D to send the EOF character, terminating the program. The "cat" utility ($ cat) has the same behavior (and probably a few other programs, too).

On its own, this isn't very exciting. On Unix-like systems, however, programs are meant to work together, so that several simple programs can be used together to do something more useful (google for "The Unix Way"). You can redirect the output and input of a program to a file, so "exe" becomes a (pretty horrible) text editor

$ ./exe > file
This will be written to a file called "file". 

[hit CTRL+D]

and a text reader

$ ./exe < file
This will be written to a file called "file".

and a file copier

$ ./exe < file > copyOfFile

which is pretty good for less than ten lines of code. You can also redirect the output of one program to be the input of another (called "piping"). For example, let's get a directory listing:

$ ls -1

Adding | ./exe to the end of that command will give the same output since "exe" just copies its input to its output. For example:

$ ls -1 | ./exe | cat | ./exe

There are utilities actually designed to be useful in this case, however, like "less", which will let you scroll up and down through the directory listing:

$ ls -1 | less
[hit 'q' to exit]

or "grep" which will filter out lines that match a pattern:

$ ls -1 | grep "\.c"

This is all basic Unix stuff (and there's a lot more to it which I'm not getting into), but here's where it gets interesting. There's a utility called "netcat" or "nc" (both of which are typically sym-linked to the same executable), which is similar to "cat" (and our toy example "exe"), but works across a network. This means its input and output connect to a socket.

Let's say that you're sitting at a computer with an IP of and you want to talk to one with IP address (the LAN IPs of two of my computers at the moment). On .3, you start netcat as a server listening on some port

$ nc -l 1234
what's up?

and on the other machine, you start netcat as a client

$ nc 1234
what's up?

and you've just improvised yourself a two-way chat program. Hitting CTRL+D on either end of the connection will close both programs.

Want to proxy that connection through a third machine, let's say at The following will set up a one-way connection from .3 to .31:

On  $ nc -l 1234
On  $ nc 1234 | nc -l 1234
On $ nc 1234

The only reason this isn't two-way is that the pipe on .5 only runs from the first netcat instance to the second. BSD allows bidirectional pipes, but I don't think Linux does, so let's add our own for the reverse connection. Everything stays the same except on .5:

$ mkfifo mypipe
$ nc 1234 < mypipe | nc -l 1234 > mypipe
$ rm mypipe

Similarly, you can have both netcat servers on .5, so .3 and .31 both connect as clients, and so on. Instead, let's forget about the proxy and transfer a file from .3 to .31:

On  $ nc -l 1234 < song.mp3
On $ nc 1234 > song.mp3

Let's add a somewhat unnecessary step to this:

On  $ cat < song.mp3 | nc -l 1234
On $ nc 1234 | cat > song.mp3

If you trace this through, cat reads the MP3 file then passes the data to netcat. On the receiving end, netcat gets the data then passes it to cat, which writes the received mp3. The end result is the same as in the first case. What if, however, instead of "cat", we use some data compression software ("gzip"/"gunzip")?

On  $ gzip < song.mp3 | nc -l 1234
On $ nc 1234 | gunzip > song.mp3

The end result is the same yet again, but this time less data is sent over the network because it is first compressed, then decompressed on the receiving end. Also, be careful with this because while "cat < file" and "cat file" do the same thing, "gzip file" and "gzip < file" do not. 

Let's encrypt the transfer, too:

On  $ gzip < song.mp3 | gpg -c -o - | nc -l 1234
On $ nc 1234 | gpg | gunzip > song.mp3

GnuPG (gpg) should prompt for a passphrase. If you want that to work without password prompts:

On  $ echo THEPASSWD > passphrase; gzip < song.mp3 | gpg --batch --passphrase-fd 3 -c -o - 3< passphrase | nc -l 1234
On $ echo THEPASSWD > passphrase; nc 1234 | gpg --batch --passphrase-fd 3 3< passphrase | gunzip > song.mp3

A "--passphrase" argument and a "--passphrase-file" argument do exist, but I've only had sporadic success with them, so unless that gets fixed at some point, I would avoid them. The "3<" simply means to open the file "passphrase" with a file descriptor of 3, which gpg then looks for.

Finally, instead of saving song.mp3 to a file, let's just stream it. There's no change on the sender side, and only the very end of the receiver side changes.

On  $ echo THEPASSWD > passphrase; gzip < song.mp3 | gpg --batch --passphrase-fd 3 -c -o - 3< passphrase | nc -l 1234
On $ echo THEPASSWD > passphrase; nc 1234 | gpg --batch --passphrase-fd 3 3< passphrase | gunzip | mplayer -

What this means is that using only programs commonly found on a Unix system, we can compress, encrypt and stream data over a network, which is pretty awesome. Note that trying to compress an MP3 as I have is probably not worth it. If you don't care about encryption, either, then all you need is:

On  $ nc -l 1234 < song.mp3
On $ nc 1234 | mplayer -

You can go really nuts with this, though. Streaming video is the same as audio. It's not that hard to stream audio currently being recorded, either. Do that in two directions, and you get a voice chat. You can easily pipe in some sort of audio compression. Multiple files can be compressed and sent by using the "tar" utility. 

Also, netcat can use UDP connections instead of TCP ones with the "-u" argument. I won't go into details, but this has performance benefits if you don't mind your data getting a little corrupted from time to time (e.g. for audio streaming).

There is some advocacy for using netcat as a port scanner (with the "-z" argument). I would probably use nmap.

Finally, one thing that I really wish I'd known about when I first studied socket programming, but I hadn't heard of netcat back then: you can use netcat as a generic client/server, so you don't have to write your client and your server code at once. It can make debugging a lot less annoying.

No comments:

Post a Comment