This morning, I was trying to spin up two Dell R710s in our lab environment, so that I could install Openstack. Thankfully, these things have iDRACs (remote out-of-band baseboard management controllers over IPMI), so I don't have to crouch down in the supply closet to run the installation.
I have to use a VirtualBox IE6 VM with the --ahem-- security turned way down low, but it beats a sharp stick in the eye.
I'm trying to set up two machines, openstack0 and openstack1, over their iDRACs. I've got PXE and tFTP all set, complete with all the operating system disks and the requisite configuration to do the installs unattended, but the PXE menu configuration refuses to select a boot target automatically. This is by design.
I've got my IE6 VM up and connected to both iDRACs, logged in, on the virtual consoles, and I've got both of the installations in-flight, when all of the sudden, the console stops responding on exactly one of the iDRACs.
Great. Maybe the management card is flaking. I check the networking in the closet to make sure someone didn't jostle something or need a network drop and scavenge one. Nope. All good there. Most of my co-workers are watching something on youtube in the conference room.
Time to start basic networking troubleshooting.
Can I ping both machines from my Macbook? Check.
Can I ping them from the IE6 VM? Check.
Let's try a port-scan. The iDRAC that won't connect is first on the block:
→ nmap 10.200.0.120
Starting Nmap 7.31 ( https://nmap.org ) at 2017-03-20 10:42 EDT
Nmap scan report for 10.200.0.120
Host is up (0.0086s latency).
Not shown: 994 closed ports
PORT STATE SERVICE
7676/tcp open imqbrokerd
8000/tcp open http-alt
8001/tcp open vcom-tunnel
8002/tcp open teradataordbms
8080/tcp open http-proxy
9999/tcp open abyss
Nmap done: 1 IP address (1 host up) scanned in 0.22 seconds
Well, it's got ports open and listening, but I don't see HTTP (80) or HTTPS (443), which is strange. What about the other (functioning) iDRAC?
→ nmap 10.200.0.121
Starting Nmap 7.31 ( https://nmap.org ) at 2017-03-20 10:42 EDT
Nmap scan report for 10.200.0.121
Host is up (0.012s latency).
Not shown: 995 closed ports
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
111/tcp open rpcbind
443/tcp open https
5900/tcp open vnc
Nmap done: 1 IP address (1 host up) scanned in 1.48 seconds
Also up, but ... with ... different ... ports.
These IPs (10.200.0.120 and .121) are both statically assigned in the iDRAC configuration (the one bit of in-front-of-the-hardware setup I suffered through).
On a lark, I checked the ARP table on my macbook (which is also on the 10.200.x.x segment):
→ ping -c1 10.200.0.120
PING 10.200.0.120 (10.200.0.120): 56 data bytes
64 bytes from 10.200.0.120: icmp_seq=0 ttl=64 time=2.287 ms
--- 10.200.0.120 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 2.287/2.287/2.287/0.000 ms
→ ping -c1 10.200.0.121
PING 10.200.0.121 (10.200.0.121): 56 data bytes
64 bytes from 10.200.0.121: icmp_seq=0 ttl=64 time=1.504 ms
--- 10.200.0.121 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.504/1.504/1.504/0.000 ms
→ arp -a | grep 10.200.0.12
? (10.200.0.120) at fc:f1:36:11:8a:1c on en0 ifscope [ethernet]
? (10.200.0.121) at a4:ba:db:15:2b:95 on en0 ifscope [ethernet]
(The two ping runs help ensure that the ARP entries haven't aged out of the ARP table.)
Now, I was expecting both MAC addresses to share the same OUI, but they are radically different. Running both of them through Wireshark's online OUI lookup tool reveals the problem:
A4:BA:DB Dell Inc.
FC:F1:36 Samsung Electronics Co.,Ltd
Remember how I told you that no one was mucking with the wiring in the closet because they were watching a video on the Internet in the big conference room? The reason they were in that particular room is because that room has a 75" Samsung flat panel TV.
Oh, a Samsung? Interesting...
Once they were done, I got on the Smasung and took a look at its networking configuration. It's configured for DHCP and sure enough, the DHCP server had handed out my iDRAC IP address.
tl;dr: informing your DHCP server of pool addresses you are going to statically assign is a Best Practice ™
Happy Hacking!
James (@iamjameshunt) works on the Internet, spends his weekends developing new and interesting bits of software and his nights trying to make sense of research papers.
Currently exploring Kubernetes, as both a floor wax and a dessert topping.