It took me another two weeks, but I think the HSRA network is finally stable. As I mentioned in the previous post, this was perhaps the most frustrating and hardest-to-troubleshoot technology issue I’ve dealt with in all my years of doing this type of work.
Lots of red herrings and false starts during this nightmare – old switches in the server room, stray switches everywhere, loop-backs, cabling, copier network card, server network card, firewall, power surges, mice, squirrels, strange liquids dripping from the ceiling, people bringing in foreign network devices. I tried isolation techniques to pinpoint the problem, but it always seemed to come back after a few hours. It also didn’t help that everything would work at night, only to fail once students and staff came in during the day.
After one all-nighter and four days in a row of late-night re-wiring sessions, I finally decided last week to switch over as much gear to the wireless network as I could. It seemed to be more stable through all of this and I was getting desperate. I just added four new Airport Extreme base stations this year, so I knew it should be able to handle the load from 50 new iMacs. I removed all of the older eMacs from the floor, as they do not have wireless cards. Amazingly, the network stayed up all day.
I added an isolated switch to the firewall and started connecting essential wired devices: printers, network security cameras and a few primary workstations (that lacked wireless cards). The key measure to see if the network would stay up was the system log on the server – link errors would appear there when the network was about to lose it’s mind:
Didn’t realize we still had AppleTalk turned on – it was apparently being used for printer setup. Turned it off on the server (which made the server log errors go away) and started reconfiguring all of the clients as well. Bonjour-only printer setup from now on…
With a functioning wireless network (and limited wired components), we decided that the issue had to lie with the physical wiring on the main student floor. This area had been re-constructed over the summer, with walls being torn down and wires pulled out of the old connector boxes. I already had pulled all of the stray Linksys switches off the floor, so we spent one morning last week with a Fluke wire tester and checked all 35 active ports – all checked out fine. I also checked all of the patch panel cables in the server room – also fine.
With the cables cleared of guilt, we started adding back student iMacs one advisory at a time using the wired Ethernet ports. Links errors reappeared within two minutes. At first we though it was one computer out of the first five, but after further testing, they all would give a link error.
I remembered reading a forum post that talked about manual Ethernet settings and IPv6. I couldn’t find the exact post again, so I just started experimenting with the settings. Out of the box, a new iMac is set to have IPv6 turned on and Ethernet set to automatic. After turning off IPv6 and setting Ethernet to manually, 100baseTX, full-duplex, flow-control and 1500 MTU, we experienced no more link errors.
I tend to think that the IPv6 change wasn’t necessary, but I’m keeping it off at this point. I added two more new 24-port Linksys switches to the server room and converted all of the machines back to the wired network successfully. We will hopefully be moving to HP ProCurve gig switches soon, but at least we are solid now.
Students were also experiencing two other issues that I considered unrelated: network login rejection when the server was at high CPU-utilization and kernel panic crashes on logout. These were both known issues that I found mentioned on the Apple discussion forums. The former was a server bug that has since been addressed in this weekend’s security update and the latter is caused by duplicate fonts in a student’s network directory. I deleted all fonts from the student home directories on Friday, so we’ll see if that helps this week.
Today I finally got to install the new iLife ’09 and iWork ’09 updates school-wide using the awesome K-12 site license we purchased ($250 for each suite). Bravo to Apple for the great recession-friendly school pricing.
We’ll see how this week goes, but it seems good so far…
Originally published by DK on February 16, 2009 at 9:36 pm