h2g2 The Hitchhiker's Guide to the Galaxy: Earth Edition

I can cope with most IT major incidents at work.

I first talk to the people on the help desk, to see who's ringing in with the problems, which locations they're at and what is/isn't working. I also talk to the technical support teams, who triage the incident and work out the best plan of attack for getting users working again. Sometimes they've seen it happen before, so they know how to fix it. Sometimes they can switch users over to some sort of backup system while they investigate. Sometimes they see that it's a problem with a product, so they raise a call with the vendor.

I usually open a conference call so everyone can discuss what they're doing. I send out communications via SMS or e-mail to the users affected, as well as to the management. I get recorded messages put on the help desk phone system, too. Eventually everything gets fixed, then we investigate what caused it to happen in the first place and work out how we can prevent it in future. I'll then write a report about it all.

Typically it takes half a day to a day to get all this done (assuming we fix it in that time). I have to stop everything else I'm doing to manage a major incident, so it creates a backlog of the day-to-day tasks, which I then have to catch up with in due course.

I can cope with a few major incidents, maybe one a week, without it seriously affecting my schedule. For the last few months, that was about the rate they came in - until October. October was meltdown. We had around 17 major incidents, including one that took nearly a week to fix (I was working days and nights on it.)

The worst thing is when they happen simultaneously. Not only does the help desk get overwhelmed, but the triage gets confused too. It's not immediately apparent whether or not the different incidents are caused by the same fault. Do you assume they are and get, say, the networks team to look at it? Or do you assume they're different, and get the server team to look at one and the database team to look at the other one? Communications get confused too. Do you send out separate e-mails to the people who reported the incidents, or do you send out a wider e-mail to everyone who might be affected if the cause is wider?

Fortunately, getting simultaneous major incidents is rare.

Or, it used to be. This morning I had 3 simultaneous ones

That must be why they pay you the big bucks!

I don't know; if you didn't have incidents you would be bored. I think incidents are created to keep you interested. Just imagine the perfect program that never went wrong.

I love programs that never go wrong.

I think I would take boredom!

That's pretty hard work, Icy. Do you get tune if in lieu, or is it all seen as part of the job?

Yes, Vip, they recently gave us the option of taking either overtime or free music downloads.

()

I'm not a fan of LOL, but that really did!

Deb

I like the sound of tune in lieu.

[Amy P]