This week I had a troubling issue at work I’ve never seen in all my years of working with technology. Making matters even more frustrating, it just cleared up by itself today with no action on my part.

HSRA has a number of security cameras set up to record digitally to two servers. Those servers record motion-sensitive movies and take a still shot of each camera every 30 seconds. The application uses a standard ftp file transfer to move those still images to a web server. On Tuesday night, I noticed that the cameras showed students in the building, but upon closer inspection, saw that the time stamp on the photos was from earlier that morning.

I fired up VPN Tracker to connect to the school’s network. Both security camera servers were up and running, so I used Apple Remote Desktop to take over the screen of each one. I noticed that all the cameras were recording video, but the ftp file transfers were stuck on “logging in.” I checked the web server and ftp appeared to be running fine with nothing unusual in the logs. Thinking it was maybe a password issue, I changed the password and tried the ftp transfer again. Nothing.

Next, I tried to connect to the web server using AFP. The login window appeared right away and the file window popped up with no delay at all. For kicks, I decided to restart all three servers. Still no change, so I decided I would wait until the morning to power cycle the Sonicwall device we use for firewall/NAT/filtering, which has cleared up wonky network issues in the past (even though it really shouldn’t be a cause for internal LAN service items).

When I arrived at school, I went to the server room and immediately restarted the Sonicwall. I then restarted the security servers and looked at the ftp status window. Still stuck at login. I opened a terminal window and tried to do a manual ftp connection to the web server. After about 30 seconds, the attempt timed out and failed. I tried Transmit (a GUI-based ftp client) on my laptop to attempt an ftp connection to the web server and it also failed, but got a little bit further in the login process. I tried it several times and was able to make a successful connection about once out of every five tries.

Some additional experiments yielded interesting results:

• SSH also seemed to be affected and would always time out
• FTP and SSH to servers outside the LAN would connect right away
• Firewall settings on all boxes were either permissive for ftp or off entirely
• Other active protocols (AFP, ARD, web, mail) did not seem affected
• FTP and SSH between any other internal clients I tried all timed out

This really had me stumped. After power cycling the Sonicwall and the servers again, I remembered that our gigabit switches had caused weird things before and decided to power cycle them too. All that caused was a few additional problems with printers, file server logins and internet access that were easily fixed by client restarts.

I decided to really study the Sonicwall via the admin interface and the log settings. Everything seemed to be functioning normally. I cruised through their support forums and ended up writing my own post there. I received a reply from the moderator that said local services on the subnet should not be impacted by the Sonicwall, just as I had earlier suspected.

Final step that day was to ping my network guru friend Chuck Goolsbee of Digital Forest. He said the same thing as the Sonicwall moderator and that I should try to isolate the issue before replacing network components (I have a spare Cisco 1712 router and considered swapping that with the Sonicwall). Dejected and tired, I left for the day.

When I arrived the next morning, things were still the same. I started reading about paid support options with Sonicwall, the feature-set of the Cisco unit, alternate filtering and firewall options like Dan’s Guardian and IPcop and anything else I could do to either fix this or get a working replacement. Out of the corner of my eye, I noticed the ftp status windows on the security servers started flashing activity! Without me touching anything, ftp just started working (and has continued to do so since then).

The network gnomes have left the building, at least for now…DK