Any Linux server experts about?

On 5 Aug 2004 10:49:37 GMT Charles Sweeney <me RemoveThis @charlessweeney.com>
broke off from drinking a cup of tea at No thank you to write:

 >Matt Probert wrote:
 >
  >> Any Linux server experts about?
  >>
  >> We have a problem with our server being a little overloaded, I
  >> eventually managed to get the following out of "top":
  >>
  >> 5:17am up 118 days, 15:02, 1 user, load average: 141.56, 144.92,
  >> 144.34
 >
 >As you may know those numbers are VERY high!
 >

VERY being an understatement! Couldn't even telnet in - okay I
eventually got in and got the top line of top!

 >Looks like you may have a runaway process.

You might have thought so, so we did a reboot. That turfed some of the
concurrent connections off <g> sorry lads! Anyway, it was just over
use. Too damned popular! Either that or someone was running multiple
searches in a loop as a DoS, it will be a while before I can check
logs to ascertain this, but it seems unlikely.

Matt<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Any Linux server experts about? 

Matt Probert wrote:

 > Any Linux server experts about?
 >
 > We have a problem with our server being a little overloaded, I
 > eventually managed to get the following out of "top":
 >
 > 5:17am up 118 days, 15:02, 1 user, load average: 141.56, 144.92,
 > 144.34
 >
 > But that's all, I can't even connect its so busy!
 >
 > Any (sensible) suggestions on what could help reduce the problem?

I'm amazed it still managed to chug along with a load like that!

One of the servers I manage normally has a load around the 0.00 to 0.50 mark
(gets around 1500 visitors / 30000 pageviews per day, so not heaps). About
50% of the pageviews generate a dynamic image and there's one in particular
that we changed so it made several of these images. It was creating about
50000 of these images per day.

All of a sudden we were having problems with the load spiking up to about 24
and it was causing the site to appear unresponsive. I put it down to the
extra images being generated so we turned those off.

Within a few days we again noticed the load going up and making the site
slow. I logged in and restarted Apache (fully stopped and started again) -
it had spawned about 100 sub processes (normally there's between about 5
and 20) and didn't seem to be cleaning up after itself.

I've been logging the load and number of running processes for the last
couple of weeks where it's logged into a text file once a minute, and need
to do some analysis of where it first peaks and what's being requested at
the time. I'm guessing the script generating the image is sometimes not
stopping running or something along those lines, so the sub process never
stops running and Apache doesn't clean up after itself.

I know this doesn't answer any questions for you, but perhaps there's
something similar going on? What sort of things are you running on the
server to serve the pages (ie scripting etc)? Could any of those scripts be
the cause of issues with Apache? Are the pages database driven? If so, are
the queries taking a long time to run and your tables need some indexing?

If you want to log the processes etc like I've been doing, you can do a one
line script that's invoked by cron once per minute like so:

cat /proc/loadavg | echo -e "`date "+%F %H:%M:%S"`\t`awk -F " |/" {'print
$1"\t"$5'}`" >> /path/to/proc.`date +%F`.txt

It writes out the date and time, a tab, the current load, a tab, the current
number of processes. The data can then be imported into a database and
analysed further if required. The file it writes out to is in the format
proc.YYYY-MM-DD.txt where YYYY, MM and DD are the year month and day, so
there's one file for each day.

--
Chris Hope - The Electric Toolbox - <a style='text-decoration: underline;' href="http://www.electrictoolbox.com/" target="_blank">http://www.electrictoolbox.com/</a><!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Any Linux server experts about? 

On Thu, 05 Aug 2004 22:42:04 +1200 Chris Hope
<blackhole.TakeThisOut@electrictoolbox.com> broke off from drinking a cup of tea
at to write:

 >Matt Probert wrote:
 >
  >> Any Linux server experts about?
  >>
  >> We have a problem with our server being a little overloaded, I
  >> eventually managed to get the following out of "top":
  >>
  >> 5:17am up 118 days, 15:02, 1 user, load average: 141.56, 144.92,
  >> 144.34
  >>
  >> But that's all, I can't even connect its so busy!
  >>
  >> Any (sensible) suggestions on what could help reduce the problem?
 >
 >I'm amazed it still managed to chug along with a load like that!
 >
 >One of the servers I manage normally has a load around the 0.00 to 0.50 mark
 >(gets around 1500 visitors / 30000 pageviews per day, so not heaps). About
 >50% of the pageviews generate a dynamic image and there's one in particular
 >that we changed so it made several of these images. It was creating about
 >50000 of these images per day.
 >
 >All of a sudden we were having problems with the load spiking up to about 24
 >and it was causing the site to appear unresponsive. I put it down to the
 >extra images being generated so we turned those off.
 >
 >Within a few days we again noticed the load going up and making the site
 >slow. I logged in and restarted Apache (fully stopped and started again) -
 >it had spawned about 100 sub processes (normally there's between about 5
 >and 20) and didn't seem to be cleaning up after itself.
 >
 >I've been logging the load and number of running processes for the last
 >couple of weeks where it's logged into a text file once a minute, and need
 >to do some analysis of where it first peaks and what's being requested at
 >the time. I'm guessing the script generating the image is sometimes not
 >stopping running or something along those lines, so the sub process never
 >stops running and Apache doesn't clean up after itself.
 >
 >I know this doesn't answer any questions for you, but perhaps there's
 >something similar going on? What sort of things are you running on the
 >server to serve the pages (ie scripting etc)? Could any of those scripts be
 >the cause of issues with Apache? Are the pages database driven? If so, are
 >the queries taking a long time to run and your tables need some indexing?
 >
 >If you want to log the processes etc like I've been doing, you can do a one
 >line script that's invoked by cron once per minute like so:
 >
 >cat /proc/loadavg | echo -e "`date "+%F %H:%M:%S"`\t`awk -F " |/" {'print
 >$1"\t"$5'}`" >> /path/to/proc.`date +%F`.txt
 >
 >It writes out the date and time, a tab, the current load, a tab, the current
 >number of processes. The data can then be imported into a database and
 >analysed further if required. The file it writes out to is in the format
 >proc.YYYY-MM-DD.txt where YYYY, MM and DD are the year month and day, so
 >there's one file for each day.
 >

Thanks for the suggestions, Chris.

Well, with the help of Travis at tera-byte (our hosting company) we
have found the problem to be one of excessive popularity. At the time
of the severe problems - a period of some ninety minutes today - there
were 60 concurrent connections, almost all running searches. The
search script is as optimised as it can be, you'll just have to take
my word for that <g>

No, in short, as I thought we need a load balancing solution whereby
we can operate two or more mirrored servers through the one
connection, sharing incoming requests through them in turn. Not a
cheap solution as far as I can tell, and I think one which given our
current income may prove economically unviable. Unless any hosts wants
to quote us for three dedicated servers and a load balancing router,
all hosted remotely?

Matt<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Any Linux server experts about? 

Chris Hope wrote:
 > Matt Probert wrote:
 >
  >> 5:17am up 118 days, 15:02, 1 user, load average: 141.56, 144.92,
  >> 144.34
 >
 > I'm amazed it still managed to chug along with a load like that!

(This song rocks!)

An old server at work (just before it died a horrible smoking[1] death)
got much higher than that: in the 200s.

Matt, do you cache your common searches? That could take some of the load
off the server.

Otherwise, try putting a check in at the beginning of your search script
to check CPU load. If CPU load is high then switch to a less CPU-intensive
algorithm. For example:

if ($cpuload < 50) {
  $query = 'SELECT term,entry FROM encycopaedia '.
   "WHERE term='$search' OR entry ILIKE '%$search%'";
} else {
  $query = 'SELECT term,entry FROM encycopaedia '.
   "WHERE term='$search'";
}

Then when CPU load is high, a simpler search is tried that gives less good
results, but runs faster.

____
[1] Yes, it literally started smoking. Massive hardware failure. Poor
diddums was only a P166 though.

--
Toby A Inkster BSc (Hons) ARCS
Contact Me ~ <a style='text-decoration: underline;' href="http://tobyinkster.co.uk/contact" target="_blank">http://tobyinkster.co.uk/contact</a>
Now Playing ~ ./william_shatner_and_joe_jackson_with_ben_folds_-_common_people.ogg<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Any Linux server experts about? 

Matt Probert wrote:

 > On Thu, 05 Aug 2004 22:42:04 +1200 Chris Hope
 > <blackhole DeleteThis @electrictoolbox.com> broke off from drinking a cup of tea
 > at to write:
 >
  >>Matt Probert wrote:
  >>
   >>> Any Linux server experts about?
   >>>
   >>> We have a problem with our server being a little overloaded, I
   >>> eventually managed to get the following out of "top":
   >>>
   >>> 5:17am up 118 days, 15:02, 1 user, load average: 141.56, 144.92,
   >>> 144.34
   >>>
   >>> But that's all, I can't even connect its so busy!
   >>>
   >>> Any (sensible) suggestions on what could help reduce the problem?
  >>
  >>I'm amazed it still managed to chug along with a load like that!
  >>
  >>One of the servers I manage normally has a load around the 0.00 to
  >>0.50 mark (gets around 1500 visitors / 30000 pageviews per day, so not
  >>heaps). About 50% of the pageviews generate a dynamic image and
  >>there's one in particular that we changed so it made several of these
  >>images. It was creating about 50000 of these images per day.
  >>
  >>All of a sudden we were having problems with the load spiking up to
  >>about 24 and it was causing the site to appear unresponsive. I put it
  >>down to the extra images being generated so we turned those off.
  >>
  >>Within a few days we again noticed the load going up and making the
  >>site slow. I logged in and restarted Apache (fully stopped and started
  >>again) - it had spawned about 100 sub processes (normally there's
  >>between about 5 and 20) and didn't seem to be cleaning up after
  >>itself.
  >>
  >>I've been logging the load and number of running processes for the
  >>last couple of weeks where it's logged into a text file once a minute,
  >>and need to do some analysis of where it first peaks and what's being
  >>requested at the time. I'm guessing the script generating the image is
  >>sometimes not stopping running or something along those lines, so the
  >>sub process never stops running and Apache doesn't clean up after
  >>itself.
  >>
  >>I know this doesn't answer any questions for you, but perhaps there's
  >>something similar going on? What sort of things are you running on the
  >>server to serve the pages (ie scripting etc)? Could any of those
  >>scripts be the cause of issues with Apache? Are the pages database
  >>driven? If so, are the queries taking a long time to run and your
  >>tables need some indexing?
  >>
  >>If you want to log the processes etc like I've been doing, you can do
  >>a one line script that's invoked by cron once per minute like so:
  >>
  >>cat /proc/loadavg | echo -e "`date "+%F %H:%M:%S"`\t`awk -F " |/"
  >>{'print $1"\t"$5'}`" >> /path/to/proc.`date +%F`.txt
  >>
  >>It writes out the date and time, a tab, the current load, a tab, the
  >>current number of processes. The data can then be imported into a
  >>database and analysed further if required. The file it writes out to
  >>is in the format proc.YYYY-MM-DD.txt where YYYY, MM and DD are the
  >>year month and day, so there's one file for each day.
  >>
 >
 > Thanks for the suggestions, Chris.
 >
 > Well, with the help of Travis at tera-byte (our hosting company) we
 > have found the problem to be one of excessive popularity. At the time
 > of the severe problems - a period of some ninety minutes today - there
 > were 60 concurrent connections, almost all running searches. The
 > search script is as optimised as it can be, you'll just have to take
 > my word for that <g>
 >
 > No, in short, as I thought we need a load balancing solution whereby
 > we can operate two or more mirrored servers through the one
 > connection, sharing incoming requests through them in turn. Not a
 > cheap solution as far as I can tell, and I think one which given our
 > current income may prove economically unviable. Unless any hosts wants
 > to quote us for three dedicated servers and a load balancing router,
 > all hosted remotely?

Can't you upgrade the current server's memory and CPU?

--
Charles Sweeney
<a style='text-decoration: underline;' href="http://CharlesSweeney.com" target="_blank">http://CharlesSweeney.com</a><!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Any Linux server experts about? 

comments.RemoveThis@probertencyclopaedia.com (Matt Probert) wrote in
news:411225b2.19531054@news.ntlworld.com:

 > On Thu, 05 Aug 2004 22:42:04 +1200 Chris Hope
 > <blackhole.RemoveThis@electrictoolbox.com> broke off from drinking a cup of tea
 > at to write:
 >
  >>Matt Probert wrote:
  >>
   >>> Any Linux server experts about?
   >>>
   >>> We have a problem with our server being a little overloaded, I
   >>> eventually managed to get the following out of "top":
   >>>
   >>> 5:17am up 118 days, 15:02, 1 user, load average: 141.56, 144.92,
   >>> 144.34
   >>>
   >>> But that's all, I can't even connect its so busy!
   >>>
   >>> Any (sensible) suggestions on what could help reduce the problem?
  >>
  >>I'm amazed it still managed to chug along with a load like that!
  >>
  >>One of the servers I manage normally has a load around the 0.00 to
  >>0.50 mark (gets around 1500 visitors / 30000 pageviews per day, so not
  >>heaps). About 50% of the pageviews generate a dynamic image and
  >>there's one in particular that we changed so it made several of these
  >>images. It was creating about 50000 of these images per day.
  >>
  >>All of a sudden we were having problems with the load spiking up to
  >>about 24 and it was causing the site to appear unresponsive. I put it
  >>down to the extra images being generated so we turned those off.
  >>
  >>Within a few days we again noticed the load going up and making the
  >>site slow. I logged in and restarted Apache (fully stopped and started
  >>again) - it had spawned about 100 sub processes (normally there's
  >>between about 5 and 20) and didn't seem to be cleaning up after
  >>itself.
  >>
  >>I've been logging the load and number of running processes for the
  >>last couple of weeks where it's logged into a text file once a minute,
  >>and need to do some analysis of where it first peaks and what's being
  >>requested at the time. I'm guessing the script generating the image is
  >>sometimes not stopping running or something along those lines, so the
  >>sub process never stops running and Apache doesn't clean up after
  >>itself.
  >>
  >>I know this doesn't answer any questions for you, but perhaps there's
  >>something similar going on? What sort of things are you running on the
  >>server to serve the pages (ie scripting etc)? Could any of those
  >>scripts be the cause of issues with Apache? Are the pages database
  >>driven? If so, are the queries taking a long time to run and your
  >>tables need some indexing?
  >>
  >>If you want to log the processes etc like I've been doing, you can do
  >>a one line script that's invoked by cron once per minute like so:
  >>
  >>cat /proc/loadavg | echo -e "`date "+%F %H:%M:%S"`\t`awk -F " |/"
  >>{'print $1"\t"$5'}`" >> /path/to/proc.`date +%F`.txt
  >>
  >>It writes out the date and time, a tab, the current load, a tab, the
  >>current number of processes. The data can then be imported into a
  >>database and analysed further if required. The file it writes out to
  >>is in the format proc.YYYY-MM-DD.txt where YYYY, MM and DD are the
  >>year month and day, so there's one file for each day.
  >>
 >
 > Thanks for the suggestions, Chris.
 >
 > Well, with the help of Travis at tera-byte (our hosting company) we
 > have found the problem to be one of excessive popularity. At the time
 > of the severe problems - a period of some ninety minutes today - there
 > were 60 concurrent connections, almost all running searches. The
 > search script is as optimised as it can be, you'll just have to take
 > my word for that <g>
 >
 > No, in short, as I thought we need a load balancing solution whereby
 > we can operate two or more mirrored servers through the one
 > connection, sharing incoming requests through them in turn. Not a
 > cheap solution as far as I can tell, and I think one which given our
 > current income may prove economically unviable. Unless any hosts wants
 > to quote us for three dedicated servers and a load balancing router,
 > all hosted remotely?
 >
 > Matt

60 concurrent connections just doesn't sound like a lot for one server
as long as it has the right specs.

What about adding more memory, mirror harddrives so it can perform two
simultaneous reads, add a second processor if the motherboard supports
it. Maybe the simples is to upgrade to a dual xeon server with plenty
of ram, 15rpm mirrored scsi harddrives. Might set you back $350/month
from a dedi provider. Definitely much cheaper than 3 servers with a
load balancing router.

--
Edward Alfert
<a style='text-decoration: underline;' href="http://www.rootmode.com/" target="_blank">http://www.rootmode.com/</a>
Multiple Domain Hosting and Reseller Hosting Plans
Promotional Code (Recurring $5/month Discount): newsgroup<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Any Linux server experts about? 

On Thu, 05 Aug 2004 17:07:11 +0100 Toby Inkster
<usenet200408 RemoveThis @tobyinkster.co.uk> broke off from drinking a cup of tea
at to write:

 >Chris Hope wrote:
  >> Matt Probert wrote:
  >>
   >>> 5:17am up 118 days, 15:02, 1 user, load average: 141.56, 144.92,
   >>> 144.34
  >>
  >> I'm amazed it still managed to chug along with a load like that!
 >
 >(This song rocks!)
 >
 >An old server at work (just before it died a horrible smoking[1] death)
 >got much higher than that: in the 200s.
 >
 >Matt, do you cache your common searches? That could take some of the load
 >off the server.
 >
 >Otherwise, try putting a check in at the beginning of your search script
 >to check CPU load. If CPU load is high then switch to a less CPU-intensive
 >algorithm. For example:
 >
 >if ($cpuload < 50) {
  > $query = 'SELECT term,entry FROM encycopaedia '.
   > "WHERE term='$search' OR entry ILIKE '%$search%'";
 >} else {
  > $query = 'SELECT term,entry FROM encycopaedia '.
   > "WHERE term='$search'";
 >}
 >
 >Then when CPU load is high, a simpler search is tried that gives less good
 >results, but runs faster.
 >

The principle sounds good, basically test cpuload (can I do that with
an environment variable? A rhetorical question, I'll try and see what
happens) and behave accordingly. Not a solution, but may help the
system from jamming up while a permanent solution is arranged.

Matt<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Any Linux server experts about? 

Toby Inkster wrote:

 > else {
  > $query = 'SELECT term,entry FROM encycopaedia '.
   > "WHERE term='$search'";
 > }

Even better -- if CPU load is high -- forward the query on to Google

--
Toby A Inkster BSc (Hons) ARCS
Contact Me ~ <a style='text-decoration: underline;' href="http://tobyinkster.co.uk/contact" target="_blank">http://tobyinkster.co.uk/contact</a><!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Any Linux server experts about? 

Matt Probert wrote:

[putolin]

 > Well, with the help of Travis at tera-byte (our hosting company) we
 > have found the problem to be one of excessive popularity. At the time
 > of the severe problems - a period of some ninety minutes today - there
 > were 60 concurrent connections, almost all running searches. The
 > search script is as optimised as it can be, you'll just have to take
 > my word for that <g>
 >
 > No, in short, as I thought we need a load balancing solution whereby
 > we can operate two or more mirrored servers through the one
 > connection, sharing incoming requests through them in turn. Not a
 > cheap solution as far as I can tell, and I think one which given our
 > current income may prove economically unviable. Unless any hosts wants
 > to quote us for three dedicated servers and a load balancing router,
 > all hosted remotely?
 >
 > Matt

You can do load balancing (crudely but it works) with nothing more than the
3 machines and a properly setup DNS. Setup the DNS with the A records
having the same domain but the three differrent IP's. I have the doc's
someplace, I think you may be able to fetch the DNS Docs/HOWTOS from
<a style='text-decoration: underline;' href="http://www.tldp.org/docs.html." target="_blank">http://www.tldp.org/docs.html.</a> The DNS will then dish out connections,
first one to server 1, second one to server 2 and third one to server 3,
then repeats.

--
There are 10 types of people in this world
Those that understand binary and those that don't<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Any Linux server experts about? 

On 5 Aug 2004 12:39:05 GMT Charles Sweeney <me.DeleteThis@charlessweeney.com>
broke off from drinking a cup of tea at No thank you to write:

 >
 >Can't you upgrade the current server's memory and CPU?
 >

As the server isn't swapping, it doesn't need any additional RAM. But
no, the CPU can not be upgraded, though I could migrate to a faster
CPU-based machine, but it seems unlikely it would have that much
effect.

Matt<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Any Linux server experts about? 

On 5 Aug 2004 12:55:44 GMT Edward Alfert <ealfert DeleteThis @rootmode.com> broke
off from drinking a cup of tea at RootMode LLC to write:

 >
 >60 concurrent connections just doesn't sound like a lot for one server
 >as long as it has the right specs.

The 60 concurrent connections was a snap shot. Further investigation
reveals we regularly have in excess of 100 concurrent connections, but
less than 200.

The problem is when so many simultaneous searches occur.

 >
 >What about adding more memory, mirror harddrives so it can perform two
 >simultaneous reads, add a second processor if the motherboard supports
 >it. Maybe the simples is to upgrade to a dual xeon server with plenty
 >of ram, 15rpm mirrored scsi harddrives. Might set you back $350/month
 >from a dedi provider. Definitely much cheaper than 3 servers with a
 >load balancing router.

Cheaper, yes. But not applicable/possible in this circumstance.

Matt<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Any Linux server experts about? 

Matt Probert wrote:
 > On Thu, 05 Aug 2004 17:07:11 +0100 Toby Inkster
 > <usenet200408.DeleteThis@tobyinkster.co.uk> broke off from drinking a cup of tea
 > at to write:
  >> ...
  >> Otherwise, try putting a check in at the beginning of your search
  >> script to check CPU load. If CPU load is high then switch to a less
  >> CPU-intensive algorithm.
  >>
 >
 > The principle sounds good, basically test cpuload (can I do that with
 > an environment variable? A rhetorical question, I'll try and see what
 > happens) and behave accordingly. Not a solution, but may help the
 > system from jamming up while a permanent solution is arranged.

Alternatively you could build in a delay when the server is busy. Spamcop
uses a client side implementation - I expect there are server-side solutions
that don't add much to cpu load.

--
William Tasso
Recommended reading ...
<a style='text-decoration: underline;' href="http://www.catb.org/~esr/faqs/smart-questions.html" target="_blank">http://www.catb.org/~esr/faqs/smart-questions.html</a>
<a style='text-decoration: underline;' href="http://www.aww-faq.org/" target="_blank">http://www.aww-faq.org/</a><!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Any Linux server experts about? 

Matt Probert wrote:

 > The principle sounds good, basically test cpuload (can I do that with
 > an environment variable?

Certainly not an environment variable on a Unix box. It ought to be
possible to write a 10 line C program to get the CPU load (using the
getloadavg system call), then you can grab the load from your PHP/Perl/
whatever script using:

  $cpuload = `/path/to/cpuloadprogram`;

Also another idea: "man nice".

--
Toby A Inkster BSc (Hons) ARCS
Contact Me ~ <a style='text-decoration: underline;' href="http://tobyinkster.co.uk/contact" target="_blank">http://tobyinkster.co.uk/contact</a>
Now Playing ~ ./dodgy_-_good_enough.ogg<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Any Linux server experts about?