I reach for tcpdump far sooner than most people do when troubleshooting issues. The insight gained from a targeted packet capture session usually cuts through to the root of most problems that trip up distributed systems.
For some people, packet capturing is an esoteric art, a black voodoo of network wizardry that is to be marvelled at but not fully understood. Like iptables configuration, strace or low-level object code debugging, packet capturing seems to confound people.
Let's fix that shall we?
Invoking tcpdump ----------------
Until you get comfortable with tcpdump, here's the only two sets of command-line options you ever need to remember:
$ sudo tcpdump -Xnv -i eth0 ${filter}
$ sudo tcpdump -vs0 -w foo.pcap -i eth0 ${filter}
The first incantation will sniff all traffic transiting eth0, matching the given ${filter}, print the full segment, minus the link-level headers, in both hex and ASCII (-X), without converting numbers into names (-n), and print additional packet metadata like TTL, window sizes, etc. (-v).
The second incantation also dumps all traffic across eth0 that matches the ${filter}, but it will write the data to a file (-w foo.pcap), and won't cut off packet payloads at 64k (-s0). The -v on this one serves to print out a running tally of number of packets processed by the kernel, and the number of packets caught by the filter. That way, you can Ctrl-C the process once you've captured enough packets. The out.pcap file can then be pulled off of the box and analyzed with tools like Wireshark.
Do you need a filter?
Yes. You need a filter. If you're running tcpdump locally you will invariably see noisy background traffic from mDNS daemons, SSH sessions, web browsers doing things via AJAX, etc. If you're on a remote server, tcpdump will happily dump your SSH packets to the screen, which will cause more SSH packets to be sent, which will then be caught and printed, leading to ... well ... chaos and confusion.
Always use a filter.
Here's a few to get you started:
port 5432
Show me all traffic to or from TCP/5432 (PostgreSQL default server port).
port 80 and host 10.0.0.42
Dump HTTP conversations between me and the remote 10.0.0.42 host.
udp port 53 and host 8.8.8.8
Watch the DNS queries being made out to Google's public DNS service.
Deciphering The Output ----------------------
So tcpdump is off doing its thing, pulling packets off the wire and dumping them to your terminal. Here's what you get (only the first packet is shown)
12:25:18.777311 IP (tos 0x0, ttl 64, id 65444, offset 0, flags [DF], proto TCP (6), length 60) 10.0.0.42.55523 > 10.240.99.5432: Flags [S], cksum 0xfe30 (incorrect -> 0x2866), seq 2270010050, win 43690, options [mss 65495,sackOK,TS val 121136486 ecr 0,nop,wscale 6], length 00x0000: 4500 003c ffa4 4000 4006 3d15 0a00 002a E..<..@.@.=..... 0x0010: 0af0 0063 d8e3 1538 874d 9ac2 0000 0000 ...c...8.M...... 0x0020: a002 aaaa fe30 0000 0204 ffd7 0402 080a .....0.......... 0x0030: 0738 6566 0000 0000 0103 0306 .8ef........
(I've added some gratuitious line breaks to help readability)
Before we dive into that scary looking hex dump there, and all that gibberish text on the right-hand side, let's look at all that lovely metadata in the first couple of lines.
12:25:18.777311 IP (tos 0x0, ttl 64, id 65444, offset 0, flags [DF], proto TCP (6), length 60)So this is a TCP connection, and this packet is 60 octets long, including the IP and TCP headers.
10.0.0.42.55523 > 10.240.0.99.5432: Flags [S], cksum 0xfe30 (incorrect -> 0x2866), seq 2270010050, win 43690, options [mss 65495,sackOK,TS val 121136486 ecr 0,nop,wscale 6], length 0Turns out this is a SYN packet, from 10.0.0.42 (me) bound for 10.240.0.99. The local side of the connection has bound port 55523 as the source port for the connection, and is attempting to contact the remote host on the standard PostgreSQL port 5432.
Neat!
Let's dive into that hex dump now.
The IPv4 Header
The IP header is the first thing we see. It's small-ish, only 20 octets:
0x0000: 4500 003c ffa4 4000 4006 3d15 0a00 002a E..<..@.@.=.....0x0010: 0af0 0063 d8e3 1538 874d 9ac2 0000 0000 .......8.M......0x0020: a002 aaaa fe30 0000 0204 ffd7 0402 080a .....0..........0x0030: 0738 6566 0000 0000 0103 0306 .8ef........The first octet contains the 4-bit version field and the 4-bit header length:
0x0000: 4500 003c ffa4 4000 4006 3d15 0a00 002a E..<..@.@.=.....0x0010: 0af0 0063 d8e3 1538 874d 9ac2 0000 0000 ...c...8.M......0x0020: a002 aaaa fe30 0000 0204 ffd7 0402 080a .....0..........0x0030: 0738 6566 0000 0000 0103 0306 .8ef........Get used to seeing that "E", which is 45 in hex. That 4 means that this is an IPv4 packet, and then 5 tells you that the IP header is 5 (32-bit) words long, which is 20 octets.
The remainder of the IPv4 header is nicely summarized by its defining RFCs, primarily RFC791, but if you really want to nerd out, see RFC2474 and RFC3168.
The TCP Header
The TCP header is up next, at offset 0x12:
0x0000: 4500 003c ffa4 4000 4006 3d15 0a00 002a E..<..@.@.=.....0x0010: 0af0 0063 d8e3 1538 874d 9ac2 0000 0000 .......8.M......0x0020: a002 aaaa fe30 0000 0204 ffd7 0402 080a .....0..........0x0030: 0738 6566 0000 0000 0103 0306 .8ef........As we saw with the metadata, the local host is sourcing from port 55523 (hex d8e3), and connecting to 5432 (hex 1538).
0x0000: 4500 003c ffa4 4000 4006 3d15 0a00 002a E..<..@.@.=.....0x0010: 0af0 0063 d8e3 1538 874d 9ac2 0000 0000 ...c...8.M......0x0020: a002 aaaa fe30 0000 0204 ffd7 0402 080a .....0..........0x0030: 0738 6566 0000 0000 0103 0306 .8ef........The next 8 octets hold the sequence and acknowledgment numbers:
0x0000: 4500 003c ffa4 4000 4006 3d15 0a00 002a E..<..@.@.=.....0x0010: 0af0 0063 d8e3 1538 874d 9ac2 0000 0000 ...c...8.M......0x0020: a002 aaaa fe30 0000 0204 ffd7 0402 080a .....0..........0x0030: 0738 6566 0000 0000 0103 0306 .8ef........The following four bits store the data offset, which specifies both the size of the TCP header in units of 32-bit (4-octet) words, and also the offset at which the payload data of the segment starts (hence the name):
0x0000: 4500 003c ffa4 4000 4006 3d15 0a00 002a E..<..@.@.=.....0x0010: 0af0 0063 d8e3 1538 874d 9ac2 0000 0000 ...c...8.M......0x0020: a002 aaaa fe30 0000 0204 ffd7 0402 080a .....0..........0x0030: 0738 6566 0000 0000 0103 0306 .8ef........The data offset is important because TCP headers vary in size, based on what options are present. Without this value, you won't be able to tell where the header ends and your application payload starts.
This packet has a data offset of a, which is hex for 10. That means the TCP header will be 10 (32-bit) words long, or 40 octets. In the hex dump, each octet is represented by two hexadecimal digits, and octets are paired into space separated groups.
Finding the start of the payload then is straightforward, if not really easy:
- Start at the beginning of the TCP header
- Convert the data offset hex value into decimal (
a→ 10) - Count down on every other grouping of four hex digits
- Stop when you hit zero
0x0000: 4500 003c ffa4 4000 4006 3d15 0a00 002a E..<..@.@.=.....0x0010: 0af0 0063 d8e3 1538 874d 9ac2 0000 0000 ...c...8.M......0x0020: a002 aaaa fe30 0000 0204 ffd7 0402 080a .....0..........0x0030: 0738 6566 0000 0000 0103 0306 DATA... .8ef........dataJust count down on the highlighted octet pairs, until you reach the data...
A More Complete Picture -----------------------
That was just one packet, and it was a SYN packet, which don't generally hold any data. Let's look at the full conversation, where I was attempting to verify that a PostgreSQL authentication session was working properly.
(This is gonna get intense)
Three-Way Handshake
10.0.0.42.55523 > 10.240.0.99.5432: Flags [S], ...0x0000: 4500 003c ffa4 4000 4006 3d15 0a00 002a E..<..@.@.=....\*0x0010: 0af0 0063 d8e3 1538 874d 9ac2 0000 0000 ...c...8.M......0x0020: a002 aaaa fe30 0000 0204 ffd7 0402 080a .....0..........0x0030: 0738 6566 0000 0000 0103 0306 .8ef........We've already analyzed this packet to death. There's no data, because it's a regular old SYN packet, but make a note of the sequence number (in purplse). Also, look at the acknowledged sequence number (yellow) — it's all zeroes, because this is the only packet you'll see that doesn't come after another packet.
Feel free to wow your friends and coworkers with that little bit of trivia next time you're discussing corner cases in standard networking protocols.
10.0.0.42.5432 > 10.240.0.99.55523: Flags [S.], ...gth 00x0000: 4500 003c 0000 4000 4006 3cba 0a00 002a E..<..@.@.<....\*0x0010: 0af0 0063 1538 d8e3 6853 2b31 874d 9ac3 ...c.8..hS+1.M..0x0020: a012 aaaa fe30 0000 0204 ffd7 0402 080a .....0..........0x0030: 0738 6566 0738 6566 0103 0306 .8ef.8ef....This is our SYN/ACK response, from the remote end back to us. Note that the acknowledged sequence number (highlighted) matches the sequence number of the previous packet. This is how TCP provides reliability on an unreliable protocol like IP.
10.0.0.42.55523 > 10.240.0.99.5432: Flags [.], ...0x0000: 4500 0034 ffa5 4000 4006 3d1c 0a00 002a E..4..@.@.=....\*0x0010: 0af0 0063 d8e3 1538 874d 9ac3 6853 2b32 ...c...8.M..hS+20x0020: 8010 02ab fe28 0000 0101 080a 0738 6566 .....(.......8ef0x0030: 0738 6566 .8efThe last part of our TCP handshake arrives as a lonely ACK packet (with no data). Note the chain of sequence number → acknowledged sequence number.
Let The Data Flow
10.0.0.42.55523 > 10.240.0.99.5432: Flags [P.], ...0x0000: 4500 003c ffa6 4000 4006 3d13 0a00 002a E..<..@.@.=....\*0x0010: 0af0 0063 d8e3 1538 874d 9ac3 6853 2b32 ...c...8.M..hS+20x0020: 8018 02ab fe30 0000 0101 080a 0738 6566 .....0.......8ef0x0030: 0738 6566 0000 0008 04d2 162f .8ef......./Now that our connection is nailed up, the client starts pushing protocol data packets. This is the first one, an SSLRequest of 8 octets (yellow) followed by the magic number 04d2 162f, which converts to decimal as 1234 5679. This all makes sense to PostgreSQL, per its protocol documentation.
You may have noticed that this is a PSH/ACK message ([P.], highlighted in blue). PSH messages signal the receiving end to attempt an immediate push the receive buffer up to the application stack. If the server software is waiting in a select(2) loop, receipt of a PSH message may result in a return from the select() with a readable socket descriptor.
In the wild, you'll often see PSH messages for interactive protocols (like SSH / telnet) as well as protocols with very small messages (like PostgreSQL).
10.0.0.42.5432 > 10.240.0.99.55523: Flags [.], ...0x0000: 4500 0034 4e54 4000 4006 ee6d 0a00 002a E..4NT@.@..m...\*0x0010: 0af0 0063 1538 d8e3 6853 2b32 874d 9acb ...c.8..hS+2.M..0x0020: 8010 02ab fe28 0000 0101 080a 0738 6566 .....(.......8ef0x0030: 0738 6566 .8efWe get an immediate ACK to our pushed data...
10.0.0.42.5432 > 10.240.0.99.55523: Flags [P.], ...0x0000: 4500 0035 4e55 4000 4006 ee6b 0a00 002a E..5NU@.@..k...\*0x0010: 0af0 0063 1538 d8e3 6853 2b32 874d 9acb ...c.8..hS+2.M..0x0020: 8018 02ab fe29 0000 0101 080a 0738 6566 .....).......8ef0x0030: 0738 6566 4e .8efN... followed by another data packet, this time containing the server response, the single character N (the PostgreSQL protocol says that means "nope, not gonna do SSL").
10.0.0.42.55523 > 10.240.0.99.5432: Flags [.], ...0x0000: 4500 0034 ffa7 4000 4006 3d1a 0a00 002a E..4..@.@.=....\*0x0010: 0af0 0063 d8e3 1538 874d 9acb 6853 2b33 ...c...8.M..hS+30x0020: 8010 02ab fe28 0000 0101 080a 0738 6566 .....(.......8ef0x0030: 0738 6566 .8efand then an ACK from the client. Again, no data.
10.0.0.42.55523 > 10.240.0.99.5432: Flags [P.], ...0x0000: 4500 0076 ffa8 4000 4006 3cd7 0a00 002a E..v..@.@.<....\*0x0010: 0af0 0063 d8e3 1538 874d 9acb 6853 2b33 ...c...8.M..hS+30x0020: 8018 02ab fe6a 0000 0101 080a 0738 6566 .....j.......8ef0x0030: 0738 6566 0000 0042 0003 0000 7573 6572 .8ef...B....user0x0040: 0068 6564 6765 686f 6700 6461 7461 6261 .hedgehog.databa0x0050: 7365 0068 6564 6765 686f 6700 6170 706c se.hedgehog.appl0x0060: 6963 6174 696f 6e5f 6e61 6d65 0070 6762 ication_name.pgb0x0070: 656e 6368 0000 ench..With SSL negotiation out of the way, the client gets down to business and sends the Postgres StartupMessage, in which the initial 4 octets (yellow) indicate the length (0x42, decimal 66). The next 4 octets (purple) identify that the client speaks v3.0 (0003.0000) of the PostgreSQL protocol. Connection information (username, database, app name, etc.) follows.
If you look closely, you'll see that this TCP segment acks 6853 2b33 which we just acked in the previous segment. This is normal, and just shows that this segment must follow that packet.
10.0.0.42.5432 > 10.240.0.99.55523: Flags [P.], ...0x0000: 4500 0041 4e56 4000 4006 ee5e 0a00 002a E..ANV@.@..^...\*0x0010: 0af0 0063 1538 d8e3 6853 2b33 874d 9b0d ...c.8..hS+3.M..0x0020: 8018 02ab fe35 0000 0101 080a 0738 6566 .....5.......8ef0x0030: 0738 6566 5200 0000 0c00 0000 0500 0000 .8efR...........0x0040: 00 .The server replies by sending an AuthenticationMD5Password message (type 'R', code 5).
10.0.0.42.55523 > 10.240.0.99.5432: Flags [P.], ...0x0000: 4500 005d ffa9 4000 4006 3cef 0a00 002a E..]..@.@.<....\*0x0010: 0af0 0063 d8e3 1538 874d 9b0d 6853 2b40 ...c...8.M..hS+@0x0020: 8018 02ab fe51 0000 0101 080a 0738 6566 .....Q.......8ef0x0030: 0738 6566 7000 0000 286d 6435 3563 3033 .8efp...(md55c030x0040: 3639 3864 3738 3336 6664 3462 3637 3138 698d7836fd4b67180x0050: 3366 3237 3536 3766 6638 3136 00 3f27567ff816.The client dutifully obeys, and returns a PasswordMessage (type 'p') with a secret MD5 one-way hash of 5c03698d7836fd4b67183f27567ff816.
10.0.0.42.5432 > 10.240.0.99.55523: Flags [P.], ...0x0000: 4500 003d 4e57 4000 4006 ee61 0a00 002a E..=NW@.@..a...\*0x0010: 0af0 0063 1538 d8e3 6853 2b40 874d 9b36 ..c..8..hS+@.M.60x0020: 8018 02ab fe31 0000 0101 080a 0738 6567 .....1.......8eg0x0030: 0738 6566 5200 0000 0800 0000 00 .8efR........The server, having verified the password by way of the one-way MD5 hash, responds with an AuthenticationOk message (type 'R', code 0). Success!
Now What? ---------
I hope this exercise has shown that tcpdump can be useful in day-to-day diagnostic and troubleshooting work, and that the hex dumps it provides are a veritable treasure trove of useful information about everything happening on the wire.
Next time you have a failing service or experience slow response times, maybe you'll warm up a packet capture and dive right in?
Happy Hacking!
James (@iamjameshunt) works on the Internet, spends his weekends developing new and interesting bits of software and his nights trying to make sense of research papers.
Currently exploring Kubernetes, as both a floor wax and a dessert topping.