Can you dump the file in binary? I know old progress database s have a 2GB dump size limit ( even in JFS with large files enabled ), but if you dump the file in binary it will span to multiple dump files.
Any of the above mentioned posts could be correct as well.
You could start by verify you can create a file larger than 2 GB in the FileSystem you are trying to write the dump too. If you can create a 3 GB file, then it's the dump utility, not FileSystem or the ulimit.