Laughing. And powering.

comments 3

Everyone who has ever done tech support has Stories and these Stories get traded, enjoyed and, of course, embellished. Some eventually become the stuff of legend.

Well, very small legends in very small circles.

A couple of my favourites are The Case of the 500 Mile Email and all the stories from people providing help configuring modems over the phone. (Inevitably, those stories end with the user/supplicant/whatever trying to get their modem to dial the phone line that they’re, um, talking on. It never works.) As you might expect, those stories are less common these days than once they were.

Why are those stories amusing? I’m not sure. But consider:

This (tech support, not modems) is a running gag — or several running gags — on the British sitcom ‘The IT Crowd’ which I’ve mentioned before. In particular, the titular IT department (a dynamic go-getter, a genius, and a man from Ireland) has a habit of answering the phone and saying something like

“Hello, IT. Have you tried turning it off and on again?”

Some years ago, we had a single machine providing basic services (email, shell access, file storage, usenet, personal web pages, and so on) to students. Early in its lifetime, this worked very well. We upgraded the hardware and software from time to time and things continued to work, but there were more and more times when the machine struggled under the load. Much of the problem was I/O; after all, with X thousand people checking their mail constantly, the poor server spent all it’s time logging people on, checking the status of the mail disk, and logging them off:

Wait for I/O. “Hello, John Smith.” Wait for I/O. “There is no new mail for you — just like the last hundred times you’ve asked.”

So we upgraded the server. And the storage — the vendor had just announced a new disk array. I won’t go into many details, but disk arrays can speed up I/O for any number of reasons and they can increase performance under load (they get slower slower, if that makes sense).

These things typically also add gobs of cache, multiple fiber optic paths, redundant power supplies and fault-tolerant disk configurations.

We installed the new hardware and all was well again. For a while. Then it failed. One day it just… vanished. Not literally, of course — it was still there, sitting on its shelf with its status lights all glowing a nice, friendly, green.

The problem was that the server couldn’t ‘see’ it anymore, which meant that, as far as it was concerned, a huge fraction of its disks just… went away.

Servers hate that. They don’t react… well. Plus, of course, everyone with data on those disks — data like, I don’t know, email — couldn’t get to it. It was bad. The phone started ringing and didn’t stop. I rebooted the server — no change — and phoned The Vendor. If you’ve read ‘Patching‘ or ‘Enveloping‘ or ‘Franking‘, you know that this doesn’t always work as well as you might hope.

I spoke to a voice. I explained the problem. The voice transferred me to the operating system (OS) group. “Oh, you have one of those. I’ve never seen one of those. How do you like it?” He then transferred me to the storage group. They didn’t know what to do either and transferred me back to the OS group. (The rest of the day was a see-saw between the OS and storage groups.) Try this. Try that. Try the other thing.  (One of the things was a reset sequence for the disk array.) Here’s a command that looks like someone banged his head on the keyboard — what’s the output from this command? Here’s another one. What’s the output from it? Nothing worked. Nothing helped.

Transfer, transfer, transfer.

The day stretched on. My call was Escalated. An angry crowd of undergraduates with pitchforks and torches gathered and chanted. Well, not really, but the phone continued to ring and various folks regularly stuck their heads around the corner to ask “Anything yet?” And my evening curling game grew closer and closer.

I’ve  talked  a  bit  about  spares, about  how  finding  a  spare  is Good. It’s  also time consuming and  ties up the phone.  I didn’t have much time and I didn’t want to  tie up the phone because at any minute there might  be a call  from the brain  trust with some  new suggestion that probably  wouldn’t work. I didn’t  get a spare.

During one of the many phone calls with the brain trust, I asked if there would be value in power-cycling (turning it off and on again) the disk array. “Nah. That reset we had you do (over and over again, I might add) does the same thing. It wouldn’t help.”  More time passed. More suggestions — each one more desperate than the last.  Unplug this and plug it in again. Pull out that and put it back. Turn around three times and put your head in a bucket. Nothing helped. More requests for the output from cryptic commands. No inspiration was forthcoming. Curling time arrived. (They won without me. A sign? Perhaps.) Finally, sometime late that evening, without being told to, I turned the array off, unplugged it, plugged it back in and turned it back on.

And Lo! The server saw it and within minutes we were up and running.

The phone gradually stopped ringing.

I told the brain trust. Their reaction? Well, they were happy — they were tired of me, after all — but they were unanimous about one thing: that shouldn’t have worked. It shouldn’t have done anything.

So sometimes the punchline to a joke can be funny and not funny.

“Hello, IT. Have you tried turning it off and on again?”


The Author

Rose Glace is the pseudonym of nobody important.


  1. Pingback: Existing | Rose Glace's Blog

  2. Pingback: Sparing (5, if you’re keeping score). Being an ill-mannered lout. | Rose Glace's Blog

  3. Pingback: Majoring (in disaster) – Rose Glace's Blog

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s