Cisco MSE pains

I (and many others) have bumped in to problems when installing a Cisco MSE 3300 in a customer environment. Being a Linux zealot in my free time I’ve tried to make things easier on myself by tweaking some of the setup process.

Cisco’s setup script (in 7.0 at least) does some things to the network init scripts that are not nice and are not what I’d consider best practices. One problem people have run in to is after reboot the NICs will swap assignment (e.g. eth0 is now eth1). From my understanding the main reason for the instability is that the initial setup script rewrites the network config file and the new version doesn’t contain the MAC anymore. If I understand the udev process correctly, Linux dynamically allocates hardware assignment which is useful for things like plugging in a flash drive. The network hardware also goes through this process so if the network script doesn’t try to remember which NIC is eth0 and which is eth1 it’s basically a race condition for that assignment. I suspect that a lot of people are actually in this state but don’t realize it because they only configure one of the NICs via Cisco’s script. If you leave eth1 alone, it retains the original network script (which still contains the MAC) and basically eth1 always gets the MAC it wants and eth0 gets “the other one”. Things going poorly without anyone knowing….or at least this is what I suspect.

To fix this neat little issue you’ll want to add HWADDR=11:22:33:44:55:66 (subbing out your appropriate MAC) somewhere in the network config file located at /etc/sysconfig/network-scripts/ifcfg-eth0. The same obviously goes for eth1 and it’s corresponding MAC if you’ve set that up as well. I use a bit of CLI parsing to get that in there without using much effort. It’s probably not as pretty as it could be but it was quick and dirty and works.

[root@MSE /]# echo HWADDR=`ifconfig eth0 | grep -o -E '([[:xdigit:]]{1,2}:){5}[[:xdigit:]]{1,2}'` >> /etc/sysconfig/network-scripts/ifcfg-eth0
[root@MSE /]#
[root@MSE /]# echo HWADDR=`ifconfig eth1 | grep -o -E '([[:xdigit:]]{1,2}:){5}[[:xdigit:]]{1,2}'` >> /etc/sysconfig/network-scripts/ifcfg-eth1
[root@MSE /]#

It’s probably worth checking this just to be sure. Even when copying and pasting things you never know when something went wrong.

[root@MSE /]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
BROADCAST=172.16.30.255
IPADDR=172.16.30.5
NETMASK=255.255.255.0
NETWORK=172.16.30.0
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
HWADDR=00:42:17:D1:B6:14
[root@MSE /]#

Another neat little thing the setup script does is clobber the gateway parameter. Obviously this will be dynamic based on your environment so sub out your appropriate IP for my example.

[root@MSE /]# echo GATEWAY=172.16.30.1 >;>; /etc/sysconfig/network-scripts/ifcfg-eth0
[root@MSE /]#
[root@MSE /]# echo GATEWAY=172.16.50.1 >;>; /etc/sysconfig/network-scripts/ifcfg-eth1
[root@MSE /]#

Let’s check again to make sure things look correct.

[root@CDW-MSE /]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
BROADCAST=172.16.30.255
IPADDR=172.16.30.5
NETMASK=255.255.255.0
NETWORK=172.16.30.0
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
HWADDR=00:42:17:D1:B6:14
GATEWAY=172.16.30.1
[root@CDW-MSE /]#

See this link for a description of this behavior:

Another little interesting gotcha for this problem is that the markings on the server casing don’t actually read eth0 and eth1 but could be listed as GE1 or BMC1. I’ve seen a small paper tag listing the MACs affixed to the back of the MSE which makes it much easier to figure out which is which. Finding the current binding in Linux is as easy as the following.

[root@MSE /]# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:42:17:D1:B6:14
          inet addr:172.16.30.5  Bcast:172.16.30.255  Mask:255.255.255.0
          inet6 addr: fe80::215:17ff:fed0:b444/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:100 errors:0 dropped:0 overruns:0 frame:0
          TX packets:33 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:12386 (12.0 KiB)  TX bytes:2410 (2.3 KiB)
          Base address:0x2000 Memory:e8180000-e81a0000
[root@MSE /]#

After all that is sorted out, I like to restart the network services on the box rather than doing a reboot. This lets me check to see if things are working without possibly putting my NICs back in an unknown state if I messed something up.

[root@MSE /]# network service restart
[root@MSE /]#

If the MSE is unreachable from WCS/NCS make sure the service is running using one or all of these:

[root@MSE /]# /opt/mse/setup/msed status
[root@MSE /]# /opt/mse/setup/msed start
[root@MSE /]# /opt/mse/setup/msed restart

After that you should be good to go.

Some other notes
A lot of the following is found in the Cisco guide, make sure you read that thoroughly. These are just some of the things I see brought up repeatedly and felt worth mentioning.

  • According to Cisco the MSE services are not supported on both interfaces at once. The second interface can be used for heartbeat services but I’m not aware of any other further official use. SSHD should bind to both interfaces so you could use it for remote access.

  • When asked for the hostname during the setup script put in only the hostname and not the FQDN.

  • If you’re starting from scratch and have to download the software from CCO, keep an eye on how you transfer the file over to your MSE. If using an ftp client this should be done in binary mode so avoid ASCII and possibly any sort of auto mode. Binary will copy a stream (bit for bit) so you end up with what you started with. ASCII mode might strip out some characters in the stream for control purposes which will break your file. Description found right here. Alternately consider using something like scp which is typically native in Linux and OSX. I like Winscp for Windows. To decompress and make the file executable:

    gzip -d filename.bin.gz *or* gunzip filename.bin.gz
    chmod 755 filename.bin *or* chmod +x filename.bin

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>