"We have met the enemy and he is us." - Walt Kelly (1972)

Submitted by tarvid on Wed, 07/29/2015 - 00:04

As you are painfully aware, LS.Net has not been the model of stability for several weeks. I thought things were under control when I returned to Fairfax after spending the first half of the month at the homestead in Grayson County. My daughter had bought tickets to the Eagles concert in Baltimore on Monday the 20th. The network went down while we on the way to the concert and I managed to get the TWR side working by "bouncing" power to the router serving fiber and wireless customers when I got back. No amount of long distance thrashing would bring the web server back although our monitoring service in Prague kept getting "pings" from the web server in Galax. Backup engineers were contacted. Both were out of town.

I finally accepted the situation but realized I was physically in need of sleep before heading to Galax. I grabbed a nap and then visited by favorite surplus computer hardware dealer in nearby Chantilly. I had not unpacked my clothes, medicines, etc. so all I had to do was gather electronic resources. I went to bed early and awoke at the break of dawn. I dumped some stuff off at my daughters garage and headed to Galax.

​It was a beautiful day for a drive up the mountain and down the Shenandoah. I listened to a collection of essays selected by Christopher Hitchens titled "Arguably". It brought back 50 year old memories of reading Emerson, Paine, Adams, Lowell, Thoreau, Whitman (and many others). Great stuff to rattle the mind (better than the twitter over Donald Trump). I arrived in Galax with a well exercised mind and the problem looked the same in person as it did over the 'net. Helen, the web server could talk but not listen. I could examine the server from the console (one thing I could not do over the 'net). The only "odd" things were ARP messages in the system logs.

I thought "ARP poisoning". Just as I was about to bounce the switch which contained the ARP table I suspected of being corrupt I heard the generator start up. It chugged a bit and dumped the whole network. When things settled down, Helen was still "deaf" but not "dumb". Switches are fundamental. They simply take data from one port and route it to another. Dumb switches do this task blindly. But I had installed a "smart" managed switch and deluded myself by concluding this was simply brought on by power instability. I ordered a UPS from Walmart to be delivered overnight (this was Thursday). Fedex put it on a truck to Indianapolis instead of a plane to Winston. VDOT removed the bridge that connects me to Beaver Dam Road. Fedex showed up late Monday.

I hurriedly installed the battery, turned it on, got two beeps every two seconds, the sound of a bad battery. Vanella from Bangalore quietly led me through some diagnostics and offered to send a new replacement but suggested plugging it on with the power turned off and let the battery get a decent charge overnight. Om mani padme hum.

Turned it on this morning, hooked up a couple of computers, a monitor and desk lamp. Passed all the diagnostics but a few hours of testing seemed appropriate. After noon, I packed up the UPS and two servers and a small table and took the detour to get to US 21 and US 58 and then Galax. I agonized for an hour and finally took the web server down, installed the UPS and brought everything back up. Again the wireless and fiber equipment came up but the web server did not. Finally I decided, "It's the switch, stupid."

The gateway router has 4 ports and I had 6 devices but only three were critical to restoring service. I ripped out all the cables and connected the three devices directly to the gateway, Now I had fiber, wireless and a web server. The backup server was still being difficult but I can connect to it remotely from the homestead. I need something to do tomorrow anyway.

Sometimes you have to sweep away the delusions and look at things as they really are. If you see problems email me, tarvid@ls.net, or call my voicemail, 703-594-7297 and leave a message.