h2g2 The Hitchhiker's Guide to the Galaxy: Earth Edition

To all those inconvenienced by the DNA servers taking unofficial strike action in the last hour, I have to apologise profusely.

I had asked our server people to run some SQL monitoring on the database server, in order to get a live snapshot of what queries were happening when, and which ones were running slowly (so that I can try and improve those areas which are tending to run slowly based on real data, and not just guesses). Unfortunately, it would appear that this monitoring was taking up so much server resource itself that it caused the database server to grind to a halt.

Once this was corrected, and all the web servers were restarted, everything sprang back to normal.

So it was all my fault, and I promise not to do it again.

And it looks like I'm back to guessing...

Naughty Jim! At least you did it with the best of intentions though. I guess any data collected during this time is of no value then.

*slaps hand*

Don't do it again, and please write...

....*I must not upset hootooers again*.....

100 times please!

have ya tried giving the atomic vector plotter a nice fresh hot cup of tea

without you there wouldn't be so much of h2g2 to monitor in the first place ...


for all the coding!

"I guess any data collected during this time is of no value then."

I'm not hopeful, but there's over 200M of data in the trace file. We'll see.

can you not run a small H2G2 server and databse on a development server and moniter that way?

Wellt hats how I do it at work anyway...

-- DoctorMO --

It was a good idea, Jim. Pity it didn't work out... it's always the way, isn't it?

Surely a separate server would mean there wouldn't be any queries to monitor? Unless you expect jim to sit at a computer loading random pages for a few weeks?

Tango

"can you not run a small H2G2 server and databse on a development server and moniter that way?"

We do that all the time, but when you want to know why your live site is going slowly sometimes, testing a dev site won't necessarily give you the same information. What I was hoping to see is a pattern of particular queries taking a long time, which might point me to areas where we can improve concurrency in the database.

Even testing with stress testers doesn't necessarily give an accurate indication.

I've found out why that monitoring hammered the system so much - it appears they turned on a *lot* of events that I wasn't expecting - so that hundreds (possibly thousands) of events were being logged every second. It wasn't long before the server started running low on resources...

It took just 10 minutes for the profiler to generate 200M of logs. And there's practically nothing of interest there. Oh well.

I still say it is nearly always something to with threads. At least they seem to be affected sooner and worse than pages. It doesn't matter how long the thread is in that short threads may be affected but it may matter on which page it is with other threads if that invokes some degree of recursion to generate the next/previous links in the same way as long threads do. The worst threads would then be the long ones on long forum lists (possibly with long lists of subscriptions to each or even from a user with a long subscription list).

It is definitely nothing to do with the NARTHURs. Fortunately that became obvious before you started the new rotation loading system.

We can try to keep track of which times and which threads from outside but only you can monitor the internal states of things or adjust them. I would be very tempted to put lower limits on all the recursion (and link generation) stuff just to see if it made a difference.

Shame the monitoring took such a chunk out of the servers... the data might have been very useful.

Well, at least it happened with good intentions....

.....but my punishment still stands....

I want 100 lines on my desk by the end of the day

Careful Vicky, you know Jim has an edit button and the power to replace your personal space with

"_________________________________________________" x 100

perhaps I should ask him to send it to you then!

Oh - I forgot about the editing powers of the italics!

The BBCi Service Not Available messages seem mostly to have gone away now and I'm back to having great difficulty getting into threads at all and even more difficulty replying (getting AddThread page). In addition I'm getting the DNS messages which seemed last time to be the result of my ISP breaking the POST function by setting a ludicrously low size limit (I've just sent them an email again). Some of those are on page accesses though which shouldn't be a POST issue. The new(?) weirdness is that I've been getting partially loaded theads, eg just a couple of complete posts with no bottom part to the page or 4 or 5 of them with the tree structure cut off halfway through.