The root of the problem is not enough contiguous cache to allow adequate
# of child daemons to spawn and stay ready to meet demand.
What happens is , on a typical linux server is this :
Apache, MySQL (and usually a mailer) all fight running at the same
priority to spawn / cache x # of child daemons.
One, or all gets a surge in usage or abused .. and now there's not
enough contiguous blocks of cache for new children to live.
So, mysql, apache, php (if using fastcgi) must now
1 - fork for every request
2 - swap (skid to disk)
This is the cause of your I/O bottlenecks.
Simply putting Apache and MySQL on different VM's will eliminate most of
the problem, because you are also eliminating the competition for
contiguous space to cache.
You can also try :
Using something like spri, available at rfxnetworks.com to re-prioritize
your process tree based on demands. This helps quite a bit.
Build PHP sensibly. Typically its bloated at about 15 MB, you can
normally get it down to as low as 3 - 5 MB in size with all
functionality you need.
Use FastCGI! Stop making everything fork each time (ahem, php hehe)
Increase the # of idle HTTP and sql servers.
... etc .. etc.
Breaking up your over-malloc()'ing daemons however is step 1, if you
hope to get it under control.
Then, you can look at what's really causing I/O loads in your
applications, without having to weed through what's just dirty paging.
Log your dirty* in slabinfo for a week, xen the machine, break up your
services a bit then log it again. You'll see a major improvement.
HTH
Tim
On Thu, 2006-08-31 at 20:01 -0700, Tom Brown wrote:
> On Thu, 31 Aug 2006, Jason wrote:
>
> > CPU wont be your bottleneck, drive IO will be. Any time one of my DomU's
> > goes IO crazy, the rest of my DomU's might as well be hammered. You can of
> > course minimize that impact by using SCSI controllers or other IO devices
> > that
> > offload that work from your main CPU.
>
> CPU is unlikely to be your problem... which of course is what you opened
> with, but then suggest using hardware to offload the CPU ... huh?
>
> My suggestion is to use more (possibly smaller) drives and spread the load
> across the spindles... if you put each domU on a separate drive I doubt
> you will see much affect on the other domUs if one goes I/O bound...
>
> (of course if you're using iSCSI, AOE, NFS, or other network storage, then
> you may hit network or CPU bottlenecks....)
>
> If you do, that's a scheduling issue which should be solved shortly with
> the new scheduler.
>
> I/O bottle necks IMHO are one of the most common system problems.
>
> -Tom
>
>
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
|
Home