NIC Discovery on CentOS

When adding a NIC to CentOS, one normally expects to see the device appear in /etc/udev/rules.d/70-persistent-net.rules. But things don’t always go your way.

In A Perfect World…

The 70-persistent-net.rules file is a view into the kernel’s udev networking subsystem which serves to locate interfaces during startup, assign drivers, and define persistent names for their access.

• If the unique device key — the MAC address in the case of network interfaces — does not already exist in the file, default sequential numbering is applied and an entry is added.
• However, if the matching key for that interface already exists, then the kernel will use that NAME to create the device.

# PCI device 0x8086:0x1075 (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:14:5e:41:42:62", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x1076 (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:14:5e:41:42:63", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

I have talked about persistent-net in the past, editing the file to restore expected interface order when a VM is cloned and the new VMNICs are detected. It can also be useful, for example, if you didn’t like the default assignment of ‘em1‘ as the first NIC on a Dell motherboard: you could edit persistent-net and change it to ‘eth0‘. (And then either reboot, or run ‘udevadm trigger’.)

Of course, once the Ethernet device names are known, you would then create the appropriate ifcfg-<Name> configuration files in /etc/sysconfig/network-scripts/:

Self-Discovery

But what if persistent-net does not exist? This is the situation I found myself in when adding a quad-port NIC on several CentOS 6.4 and 6.5 physical hosts. I knew the cards were recognized because I could see them in the output of ‘lspci‘. But what did the kernel create the interfaces as? How was I to reference the ports by name? How should I define my ‘ifcfg-XXX‘ configuration files?

There are 2 ways to determine this. The first involves installing the ‘lshw‘ utility to view various hardware classes. Note that the interface name appears as the “logical name”:

# sudo yum -y install lshw
# lshw -class network
  *-network
       description: Ethernet interface
       product: 82547GI Gigabit Ethernet Controller
       vendor: Intel Corporation
       physical id: 1
       bus info: pci@0000:02:01.0
       logical name: eth0
       version: 00
       serial: 00:14:5e:41:42:62
       width: 32 bits
       clock: 66MHz
       capabilities: bus_master cap_list rom ethernet physical
       configuration: broadcast=yes driver=e1000 driverversion=7.3.21-k6-1-NAPI firmware=N/A ip=192.168.2.254 latency=0 mingnt=255 multicast=yes
       resources: irq:18 memory:d0120000-d013ffff memory:d0100000-d011ffff ioport:2000(size=32) memory:80000000-8001ffff(prefetchable)

The second method takes advantage of the sysfs to walk the hardware tree. (I’m betting this is just what ‘lshw‘ is doing.) Attributes for each device are kept in text files under the appropriate named-device directory.

# ls /sys/class/net
eth0  eth1  lo  virbr0  virbr0-nic
# cat /sys/class/net/eth0/address
00:14:5e:41:42:62

So once I knew the discovered interface names — p1p1 thru p1p4 in the case of the quad — I created the appropriate ifcfg-p1pX files and connected my SAN.

# ifup p1p1
# ping -I p1p1 10.10.10.100
PING 10.10.10.100 (10.10.10.100) 56(84) bytes of data.
64 bytes from 10.10.10.100: icmp_seq=1 ttl=64 time=0.237 ms
64 bytes from 10.10.10.100: icmp_seq=3 ttl=64 time=0.225 ms
64 bytes from 10.10.10.100: icmp_seq=4 ttl=64 time=0.283 ms
^C

All done, right? Not quite.

I could ping my SAN, but iSCSI discovery & login were failing with connect timeout:

# iscsiadm -m discovery -t sendtargets -p 10.10.10.100
iscsiadm: connect to 10.10.10.100 timed out
iscsiadm: connect to 10.10.10.100 timed out
iscsiadm: connect to 10.10.10.100 timed out
iscsiadm: connect to 10.10.10.100 timed out
iscsiadm: connect to 10.10.10.100 timed out
iscsiadm: connect to 10.10.10.100 timed out
iscsiadm: connection login retries (reopen_max) 5 exceeded
iscsiadm: No portals found

After much hair pulling, I finally realized that iSCSI utilizes its own set of interface names, and these too were not auto-discovered with NIC addition. The pre-existing interfaces ‘iface0‘ and ‘iface1‘ lived under /var/lib/iscsi/ifaces, but nothing had been created for the recently-added quad.

Now it might be tempting to just add iface2, iface3, etc, but that would not be very efficient. iSCSI discovery tries each iface in sequence (with 5 retries for each), so if you put iSCSI on p1p4 and no other, you’d have to weather 6×5 timeouts before the ‘sendtargets‘ ultimately succeeded. (If you didn’t want that default search behavior, you could specify the exact interface with the ‘-I <iface-name>‘ option.)

The better method is to keep only the ifaces that are used for iSCSI. Either the MAC address (iface.hwaddress) or interface name (iface.net_ifacename) are the glue that joins iSCSI iface files to kernel device identity. So in my case, I just updated the existing iface0 to use the MAC data for p1p1. Alternatively, I could have created a new interface using the p1p1 name. Both approaches can be done by editing at the /var/lib/iscsi/ifaces file level, or through iscsiadm commands as follows:

# iscsiadm -m iface -I iface0 -o update -n iface.hwaddress -v 90:E2:BA:7F:39:26
#
# iscsiadm -m iface -I iscsi-p1p1 -o new
# iscsiadm -m iface -I iscsi-p1p1 -o update -n iface.net_ifacename -v p1p1

At that point, discovery and login worked as we had originally expected when all this began:

# iscsiadm -mode discovery -t sendtargets -I iscsi-p1p1 -p 10.10.10.100
10.10.10.100:3260,1 iqn.1984-08.com.dell:powervault.md3200i.d64ae5200075d08d000000004f6b3002
# iscsiadm -m node -I iscsi-p1p1 -l
Logging in to [iface: iscsi-p1p1, target: iqn.1984-08.com.dell:powervault.md3200i.d64ae5200075d08d000000004f6b3002, portal: 10.10.10.100,3260] (multiple)

Understanding the ways in which subsystems such as udev and iscsi work by learning as much as you can about the config files is fantastic foundational knowledge. But to be able to pair that up with the commands needed to effect a change is the icing on the cake, because it opens the door to scripting and automation. Later on, I’ll post a script that does all the work of udev‘s write_net_rules for cases when persistent-net can’t be relied upon…

Share/Bookmark