This morning, I was trying to spin up two Dell R710s in our lab environment, so that I could install Openstack. Thankfully, these things have iDRACs (remote out-of-band baseboard management controllers over IPMI), so I don't have to crouch down in the supply closet to run the installation.
I have to use a VirtualBox IE6 VM with the --ahem-- security turned way down low, but it beats a sharp stick in the eye.
I'm trying to set up two machines,
openstack1, over their iDRACs. I've got PXE and tFTP all set, complete with all the operating system disks and the requisite configuration to do the installs unattended, but the PXE menu configuration refuses to select a boot target automatically. This is by design.
I've got my IE6 VM up and connected to both iDRACs, logged in, on the virtual consoles, and I've got both of the installations in-flight, when all of the sudden, the console stops responding on exactly one of the iDRACs.
Great. Maybe the management card is flaking. I check the networking in the closet to make sure someone didn't jostle something or need a network drop and scavenge one. Nope. All good there. Most of my co-workers are watching something on youtube in the conference room.
Time to start basic networking troubleshooting.
Can I ping both machines from my Macbook? Check.
Can I ping them from the IE6 VM? Check.
Let's try a port-scan. The iDRAC that won't connect is first on the block:
→ nmap 10.200.0.120 Starting Nmap 7.31 ( https://nmap.org ) at 2017-03-20 10:42 EDT Nmap scan report for 10.200.0.120 Host is up (0.0086s latency). Not shown: 994 closed ports PORT STATE SERVICE 7676/tcp open imqbrokerd 8000/tcp open http-alt 8001/tcp open vcom-tunnel 8002/tcp open teradataordbms 8080/tcp open http-proxy 9999/tcp open abyss Nmap done: 1 IP address (1 host up) scanned in 0.22 seconds
Well, it's got ports open and listening, but I don't see HTTP (80) or HTTPS (443), which is strange. What about the other (functioning) iDRAC?
→ nmap 10.200.0.121 Starting Nmap 7.31 ( https://nmap.org ) at 2017-03-20 10:42 EDT Nmap scan report for 10.200.0.121 Host is up (0.012s latency). Not shown: 995 closed ports PORT STATE SERVICE 22/tcp open ssh 80/tcp open http 111/tcp open rpcbind 443/tcp open https 5900/tcp open vnc Nmap done: 1 IP address (1 host up) scanned in 1.48 seconds
Also up, but ... with ... different ... ports.
These IPs (
.121) are both statically assigned in the iDRAC configuration (the one bit of in-front-of-the-hardware setup I suffered through).
On a lark, I checked the ARP table on my macbook (which is also on the
→ ping -c1 10.200.0.120 PING 10.200.0.120 (10.200.0.120): 56 data bytes 64 bytes from 10.200.0.120: icmp_seq=0 ttl=64 time=2.287 ms --- 10.200.0.120 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 2.287/2.287/2.287/0.000 ms → ping -c1 10.200.0.121 PING 10.200.0.121 (10.200.0.121): 56 data bytes 64 bytes from 10.200.0.121: icmp_seq=0 ttl=64 time=1.504 ms --- 10.200.0.121 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 1.504/1.504/1.504/0.000 ms → arp -a | grep 10.200.0.12 ? (10.200.0.120) at fc:f1:36:11:8a:1c on en0 ifscope [ethernet] ? (10.200.0.121) at a4:ba:db:15:2b:95 on en0 ifscope [ethernet]
ping runs help ensure that the ARP entries haven't aged out of the ARP table.)
Now, I was expecting both MAC addresses to share the same OUI, but they are radically different. Running both of them through Wireshark's online OUI lookup tool reveals the problem:
A4:BA:DB Dell Inc. FC:F1:36 Samsung Electronics Co.,Ltd
Remember how I told you that no one was mucking with the wiring in the closet because they were watching a video on the Internet in the big conference room? The reason they were in that particular room is because that room has a 75" Samsung flat panel TV.
Oh, a Samsung? Interesting...
Once they were done, I got on the Smasung and took a look at its networking configuration. It's configured for DHCP and sure enough, the DHCP server had handed out my iDRAC IP address.
tl;dr: informing your DHCP server of pool addresses you are going to statically assign is a Best Practice ™