Daemonlogger for packet capture and redirection

Daemonlogger has two features of note. First, it has a user-friendly ring buffer mode. In other words, it can log packets within a specified number of files and file sizes. Why is this useful? It's the simplest worry-free network packet capture implementation you could ask for. Being able to specify a defined amount of space to capture Pcap data means you can start a tool like Daemonlogger in ring buffer mode and never touch it again. You'll always have the last 300 GB, or 100 GB, or whatever amount of hard drive space you allocate, of network traffic available for analysis.

More Traffic Talk Network security monitoring: Know your network Using Wireshark and Tshark display filters for troubleshooting DNS troubleshooting and analysis

# tcpdump -n -i le1 -s 1515 -C 1 -W 5 -w /tmp/test1.lpc -U

I produced the following directory listing while collecting packets with Tcpdump.

-rw-r--r-- 1 root wheel 373K Dec 9 20:26 test1.lpc0
-rw-r--r-- 1 root wheel 977K Dec 9 20:26 test1.lpc1
-rw-r--r-- 1 root wheel 977K Dec 9 20:26 test1.lpc2
-rw-r--r-- 1 root wheel 977K Dec 9 20:26 test1.lpc3
-rw-r--r-- 1 root wheel 978K Dec 9 20:26 test1.lpc4

Notice the first file is smaller than the others. That's because Tcpdump wrote the five files, then started overwriting the first file. When I grabbed that output it had written 373 KB in place of the original test1.lpc0 file. I don't like the way Tcpdump manages its file names in that respect. I also don't like seeing file names like NAME0, NAME1, etc.

Let's see how to implement a similar capture with Daemonlogger. Notice the different switches required, especially the -r to active ring buffer mode and the separation between the logging directory (-l /tmp) and the name of the files in the directory (-n dltest1.lpc). Also note the specification of the file size using bytes instead of MB as was the case with Tcpdump.

daemonlogger -i le1 -S 1515 -s 1000000 -m 5 -r -l /tmp/ -n dltest1.lpc

[-] Interface set to le1
[-] Logpath set to /tmp/
[-] Max files to write set to 5
[-] Log filename set to "dltest1.lpc"
[-] Pidfile configured to "daemonlogger.pid"
[-] Pidpath configured to "/var/run"
[-] Ringbuffer active
[-] Rollover size set to 1000000 bytes
[-] Snaplen set to 1515
[-] Rollover time configured for 0 seconds
[-] Pruning behavior set to oldest IN DIRECTORY

-*> DaemonLogger <*-
Version 1.2.1
By Martin Roesch
(C) Copyright 2006-2007 Sourcefire Inc., All rights reserved

Checking partition stats for log directory "/tmp//."
sniffing on interface le1
Logging packets to /tmp//dltest1.lpc.1228873098

I like that Daemonlogger tells you what it's doing. You could avoid seeing this output if you use the -d switch to daemonize at startup. Based on this output I noticed I neglected to enable packet-buffered output, which I could've activated using the -F switch to produce this notice:

[-] Packet-buffered output activated

As Daemonlogger works it reports that it's rolling over logfiles.

Size limit reached (1000506 - 1000000 = 506), rolling over!
Rolling over logfile...
Logging packets to /tmp//dltest1.lpc.1228873102
Size limit reached (1000576 - 1000000 = 576), rolling over!
Rolling over logfile...
Logging packets to /tmp//dltest1.lpc.1228873109
Size limit reached (1000386 - 1000000 = 386), rolling over!
Rolling over logfile...
Logging packets to /tmp//dltest1.lpc.1228873117
Size limit reached (1000716 - 1000000 = 716), rolling over!
Rolling over logfile...
Logging packets to /tmp//dltest1.lpc.1228873122
Size limit reached (1000574 - 1000000 = 574), rolling over!
Rolling over logfile...
[!] Ringbuffer: deleting /tmp//dltest1.lpc.1228873098
Logging packets to /tmp//dltest1.lpc.1228873127
Size limit reached (1000716 - 1000000 = 716), rolling over!
Rolling over logfile...
[!] Ringbuffer: deleting /tmp//dltest1.lpc.1228873102
Logging packets to /tmp//dltest1.lpc.1228873133
Size limit reached (1000480 - 1000000 = 480), rolling over!
Rolling over logfile...
[!] Ringbuffer: deleting /tmp//dltest1.lpc.1228873109
Logging packets to /tmp//dltest1.lpc.1228873138

If I take an arbitrary look at the directory where Daemonlogger stores its Pcap data, I'll see the following:

-rw-r--r-- 1 root wheel 977K Dec 9 20:38 dltest1.lpc.1228873117
-rw-r--r-- 1 root wheel 977K Dec 9 20:38 dltest1.lpc.1228873122
-rw-r--r-- 1 root wheel 977K Dec 9 20:38 dltest1.lpc.1228873127
-rw-r--r-- 1 root wheel 977K Dec 9 20:38 dltest1.lpc.1228873133
-rw-r--r-- 1 root wheel 884K Dec 9 20:39 dltest1.lpc.1228873138

There are only five files, as I would expect. However, I can always be sure that the newest file has the newest data. You can verify this by checking the timestamp at the end of the files. Here I check the timestamp on the first and last files. They're in Unix time.

freebsd70:/tmp# date -r 1228873117
Tue Dec 9 20:38:37 EST 2008
freebsd70:/tmp# date -r 1228873138
Tue Dec 9 20:38:58 EST 2008

You can dispense with determining the number of individual files you want to log and just use a percentage of available partition space. In the following I tell Daemonlogger to start rolling over its ring buffer when it approaches 65% utilization (via the -M 65 switch).

/daemonlogger -i le1 -S 1515 -s 1000000 -M 65 -r -l /tmp/ -n dltest1.lpc -F

[-] Packet-buffered output activated
[-] Interface set to le1
[-] Logpath set to /tmp/
[-] Log filename set to "dltest1.lpc"
[-] Pidfile configured to "daemonlogger.pid"
[-] Pidpath configured to "/var/run"
[-] Ringbuffer active
[-] Rollover size set to 1000000 bytes
[-] Snaplen set to 1515
[-] Rollover time configured for 0 seconds
[-] Pruning behavior set to oldest IN DIRECTORY

-*> DaemonLogger <*-
Version 1.2.1
By Martin Roesch
(C) Copyright 2006-2007 Sourcefire Inc., All rights reserved

Checking partition stats for log directory "/tmp//."
65% max disk utilization = 666358 blocks free (out of 1903879)
Blocksize = 2048
Rollsize = 488 blocks

sniffing on interface le1
Logging packets to /tmp//dltest1.lpc.1228874034
Size limit reached (1000722 - 1000000 = 722), rolling over!
Rolling over logfile...
Logging packets to /tmp//dltest1.lpc.1228874043
Size limit reached (1000798 - 1000000 = 798), rolling over!
Rolling over logfile...
Logging packets to /tmp//dltest1.lpc.1228874047
Size limit reached (1000215 - 1000000 = 215), rolling over!
Rolling over logfile...

Once the 65% threshold is reached Daemonlogger lets us know.

Logging packets to /tmp//dltest1.lpc.1228874206
Size limit reached (1000503 - 1000000 = 503), rolling over!
Rolling over logfile...
Disk max utilization reached, rolling over and pruning
[!] Ringbuffer: deleting /tmp//dltest1.lpc.1228874034
Logging packets to /tmp//dltest1.lpc.1228874215
Size limit reached (1001167 - 1000000 = 1167), rolling over!
Rolling over logfile...
Disk max utilization reached, rolling over and pruning
[!] Ringbuffer: deleting /tmp//dltest1.lpc.1228874043
Logging packets to /tmp//dltest1.lpc.1228874221

You can also tell Daemonlogger to drop privileges using the -u and -g switches, e.g. -u nobody -g nobody:

[-] Setting user ID to nobody

Checking ps output confirms this.

$ ps -auxww | grep daemonlogger
nobody 3332 0.0 0.3 3312 1360 p0 S+ 9:01PM 0:00.07 ./daemonlogger -i le1 -S 1515 -s 1000000 -M 65 -r -l /tmp/ -n dltest1.lpc -F -u nobody -g nobody

Dropping privileges helps limit what an intruder could do if he were able to exploit a vulnerability in Daemonlogger.

The final aspect of Daemonlogger that I'd like to demonstrate is its "soft tap" functionality. With Daemonlogger you can listen to traffic on one interface and retransmit on another. This could be useful if you wish to create a network traffic director that performs I/O intensive tasks (like filtering or splitting traffic) in tandem with other systems that take actions, like logging traffic to disk. I called this "Distributed Traffic Collection" in a 2005 blog post (http://taosecurity.blogspot.com/2005/07/distributed-traffic-collection-with-pf.html) where I showed how to achieve this with the Pf packet filter on FreeBSD.

To simulate taking traffic from one interface and sending it elsewhere, I'll create a fake interface on FreeBSD. This edsc(4) interface imitates an Ethernet device, and is intended to take the place of a real interface connected to another computer.

First I'll create the edsc0 interface.

freebsd70:/root# ifconfig edsc0 create
freebsd70:/root# ifconfig edsc0 up
freebsd70:/root# ifconfig edsc0
edsc0: flags=8843 metric 0 mtu 1500
ether 00:00:00:00:00:00

With it created, I'll capture some traffic with Daemonlogger and send only ICMP packets over to the edsc0 interface.

daemonlogger -F -i le1 -o edsc0 icmp
[-] Packet-buffered output activated
[-] Interface set to le1
[-] Log filename set to "daemonlogger.pcap"
[-] Tap output interface set to edsc0[-] Pidfile configured to "daemonlogger.pid"
[-] Pidpath configured to "/var/run"
[-] Rollover size set to 2147483648 bytes
[-] Rollover time configured for 0 seconds
[-] Pruning behavior set to oldest IN DIRECTORY

-*> DaemonLogger <*-
Version 1.2.1
By Martin Roesch
(C) Copyright 2006-2007 Sourcefire Inc., All rights reserved

sniffing on interface le1

If I sniff on the edsc0 interface I'll only see ICMP traffic, e.g.

# tcpdump -n -i edsc0
tcpdump: WARNING: BIOCPROMISC: Invalid argument
tcpdump: WARNING: edsc0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on edsc0, link-type EN10MB (Ethernet), capture size 96 bytes
21:13:59.694057 IP 172.16.99.131 > 64.233.169.104: ICMP echo request, id 27405, seq 0, length 64
21:13:59.694058 IP 64.233.169.104 > 172.16.99.131: ICMP echo reply, id 27405, seq 0, length 64

On a single system you might wonder why you shouldn't just have your sniffing program watch edsc0 for ICMP traffic. Remember, here edsc0 is simulating a connection to a separate computer. Imagine a directly connected second system plugged into the interface represented by edsc0. The second computer would only see ICMP traffic.

Daemonlogger only supports one invocation of the -o switch, so you can't specify multiple outputs for one input. If you need that functionality, Pf can do it.. It's possible to run more than one instance of Daemonlogger, but you could encounter performance issues. Still, this soft tap functionality probably has other uses that I haven't imagined.

About the author

Richard Bejtlich is Director of Incident Response for General Electric. Richard wrote several books on network security monitoring and forensics, and also operates the TaoSecurity blog (http://taosecurity.blogspot.com).