Not enough space: couldn't spawn child process:

[bnooch]

Something weird is going on with the machine my web server is on and I'm afraid disaster is about to strike.

I am the webmaster for the school of business at the U of U. I have been assigned the role of sys admin backup for when our sys admin is on vacation. Currently he is in Africa for 3 weeks :|
I do know some sys admin stuff, but I am not experienced by any means.

I've searched the archives and it looks like people have had a similar "couldn't spawn child process" issue, but those were because the system couldn't find a path...not because there's not enough space (just so you know I did my homework).

We have a Sun x4200 running Solaris 10 containers.
Running apache 2.0.59
I don't think we are running mod_perl, I think we do something else for perl but I'm not sure what.

Here is the situation.

The backend for my Content Management System is written in perl and is intermittently giving me 500 internal server errors. I looked at the httpd logs and they said that there is not enough space to spawn a child process:

[Tue Jun 19 09:45:58 2007] [error] [client 128.110.240.100] (12)Not enough space: couldn't spawn child process: /vhost/[URL unfurl="true"]www-prod/apps/cgi/bcom/bcom.cgi,[/URL] referer: [URL unfurl="true"]http://helpdesk.business.utah.edu/go/articles/1[/URL]

My first guess was that it was a problem with swap space. So, I googled the error and that's what other people have seemed to conclude as well. It also seemed that this only happens with cgi, not php (which is good because our public website is in php and thank god it is still functioning).

I did a 'swap -s' and the output was:

total: 483812k bytes allocated + 6530088k reserved = 7013900k used, 45880k available

I'm guessing that 45 MBs is not a whole helluva lotta space for swap. But really, I have no idea.

Anyway, I was going to restart the server, but then hesitated because the public website is working fine and I am nervous about restarting the server in the middle of the day unless absolutely necessary.

When it started spontaneously working again, I did another swap -s.

total: 412844k bytes allocated + 5496400k reserved = 5909244k used, 1163836k available

I'm really worried this is going to take out our public website. I'm a little nervous messing with swap space as I have never really dealt with swap space before.

I don't know if this helps, but here is a print out of df -k

I could give a print out of what my df -k is giving me, but I don't know if that would be a bad idea in terms of security (...?)

Has anyone seen this problem before and have a quick fix? (wishful thinking i know)

Any insight would be GREATLY appreciated.