Check out 'ulimit'. You can set limits on per-user basis. Don't remember the exact name, but there's something like /etc/login.defs that has a lot of 'ulimit' type of stuff defined (very distro-dependant mind you). Like DanielHozac said though, if the app needs it, it will probably die if you don't let it get the memory it needs. 'ulimit' is good for keeping runaway processes from taking down the box though (like when you use PHP and try experimenting with recursive MySQL queries and it goes up in smoke ;-) )
----
JBR