h2g2 The Hitchhiker's Guide to the Galaxy: Earth Edition

Here's TW's line for the reason:

>>Dear all,

On New Year’s Eve there was severe service disruption to a number of our
services including the loss of TV services nationally.

We would like to apologise for the inconvenience this caused our
customers, and thank them for the patience and understanding they gave
our staff whilst we worked to restore services.

A full investigation is taking place. In the meantime, I wanted to give
further details of how the issue was managed.

At approx. 22:00 on New Year’s Eve, severe weather caused flooding in
our Digital Control Centre in Knowsley, this in turn triggered automatic
alarms to the fire brigade.

The fire brigade arrived on site by 22:10, and ordered a complete
evacuation of the site and the immediate shutdown of all electrical
power and water supplies. This was done to ensure the safety of staff
and fire personnel dealing with the issue.

The removal of power at the site disrupted digital TV and some analogue
TV services nationwide. Additionally, Internet services (broadband and
dialup) and telephony in the Liverpool region were also disrupted, as
was access to Internet services dependent on servers in the Centre; such
as e-mail, PWP, and the blueyonder and Telewest websites, including the
status page.

Due to the loss of access, we regret that our technical support team
were unable to update the status page, mailing lists or the announce
newsgroups. Given the scale of disruption the number of calls coming in
to our technical support teams caused congestion problems, which
prevented some customers getting through.

By 22:30, after a full assessment of the situation by senior management,
a decision was made to invoke our contingency plan. This involved
routing as many services as possible to a designated recovery centre at
Hayes.

All on call engineers nationwide were en route by 23:00, despite being
hampered by traffic congestion due to New Year’s celebrations.

By 23:30 the fire service had contained the flood, and some power was
allowed to be restored from back-up systems. By 01:00 on New Year’s Day,
we restored e-mail and PWP services, but were still unable to restore
full power due to health & safety and power equipment integrity
considerations.

Re-routing of traffic from Hayes was completed at 04:00, resulting in
the restoration of the majority of TV services nationally, although in
some areas customers we experiencing some residual freezing pictures issues.

At 06:00 we were allowed to restore all power in Knowsley by the fire
brigade, and we subsequently re-routed all traffic from Hayes back to
Knowsley. This restored all services except in the Liverpool area. By
08:00, all services were fully restored nationally.

Again, please accept our sincerest apologies for any inconvenience this
incident has caused you and please be assured all involved in Telewest
concentrated efforts to minimise the disruption caused by this serious
problem.

Regards,


Colin Whitbread
Group Operations Director
Networks, Technology & IT
Telewest Broadband<<

21:54 Picture freeze on ITV2 The 'Real' Beckhams.

Keyed 101, black screen.

Power-on reset. (Have a SA box) the LED channel dispaly showed:

tunE

with a dot moving around the display. ie.

t.unE >> tu.nE >> tu'nE >> tun'E

Telephone 151 'faults' unable to connect, as was 150.

Not being far from Knowsley and knowing people that are not that far from the knowsley site I can confirm the "severe weather" as nonsense.

A mild sleet had fallen about 9pm, very light drizzel followed. Yet cold enough for slush to remain even on gritted roads until about 11pm the grids dealing with any flow created by the steady flow of traffic. Mainly Taxis. Winds moderate and steady.

Knowsley is *still* above sea level. Not known for flooding.

However if Knowsley is now to be identified as the center for signal distribution...

Why do Telewest staff tell people its from London, or Scotland? Or to quote staff "its whats sent to us" - from where? Surly not "beamed to Billinge then to your arial" as told to a customer. Billinge is a hill, that happens to have comms equipment on it (an ntl: site), overlooks the Merseyside area, high enough to see Mt Snowdon in Wales, about 60 miles distant as the crow flies. Gives an exelent outlook over Liverpool, along the River Mersey into Cheshire, the Widnes Runcorn bridge crossing being a landmark. Infact high enough to look round in a 360 motion to se Manchester, the pennines, round further to see Bolton. So is an exelent view point to anybody travling towards Knowsley to get to the A580. - Nope no "severe weather" - sorry Telewest, not going for that one!

Analogue TV comes from 'Winter Hill' further north and dwafs 'Billing Hill'

This "severe weather" did nothing to affect analouge signal. Despite transmition sources being the place for 'weather' to happen.

Why would any company have a national broadcast center without independent power backup?

Do people really need to sit next to the 'head end' for it to work?

Who says any exposed part need be on the ground floor anyway?

Flooding "cause fire alarms to go off" - What! So every bit of cable Telewest have, is left of the floor, ready to short, even fire alarm systems? - Can then understand why they had been told to leave the building.

Have these people never heard of running a hot swap backup system?

Even a old office PC Novell Netware 4 system is more robust than this, one server 'dies' at one site a second takes over near instantly at a second site.

What a fantastic display of Microsoft Exchange then? - The software Telewest have *claimed* to use ie. "be the reason previous problems with e-mail."

If the equipment is so sophisticated they can fit it all in one building to service the whole country, why is management not sophisticated enough to identify a hazard, get the staff out and not wait to "be told" to leave?

The picture qualty had been deteriorating for days before, so no customer phoned in about this? Why always wait for a catastrophic failure?

Are we all going to get a "good will gesture" of a credit? Or does a "national fault" nullify this? - Its broken so bad it affects everybody so tuff! - Great customer care as ever from Telewest?

Next new year who will it be at the unoffical office party to get so pissed, trip and break something f**k-off expensive during an attempt to fondle "Mandy from accounts" in the equipment room? - A more plausable explination?

The Telewest Services in Essex went off around 8:30pm, the tv services were up and ready when I got up in the morning but it was not until the afternoon before I could access the email services.


Kat

it went about 930 here, n the interactive service was down all the following day. Telewest are just full of sh*t!