Our Solaris 10 server is a SunFire x4550, running Solaris/Generic_141445-09.
We use NFS heavily in our environment. The options we use are simple: "tcp,hard,intr,bg,rw" (vers=3)
We have one directory where files are written to and deleted which sometimes (not always) leaves behind a ton of .nfsXXX locks that prevent directories form being removed and I'm puzzled as to why. It only seems to be happening in one directory, yet that directory is not mounted any differently than others.
The other night, there were 1088 .nfsXXXXXXX files that all had the same timestamp. Eventually they did disappear, however one of our developers complained loudly that this was interfering with production.
I've attempted to debug this, but I don't know whether we're hitting a bug or if this is a symptom of the slowness of NFS and lock operations, where a process writes a directory structure then deletes it right after -- is it too fast?
The resources on the system are more than adequate. I've tuned /etc/default/nfs with the following:
NFSD_LISTEN_BACKLOG=32
NFSD_PROTOCOL=ALL
NFSD_SERVERS=512
LOCKD_LISTEN_BACKLOG=256
LOCKD_SERVERS=128
LOCKD_RETRANSMIT_TIMEOUT=5
GRACE_PERIOD=90
NFS_SERVER_VERSMAX=3
NFS_CLIENT_VERSMAX=3
We recently changed our NFS mounts from UDP (which they were using long before I joined) to TCP, due to retrans (mostly caused by congestion on the back-end, non-routed switches that are connected by 1GigE). That seems to have resolved that problem for the most part.
Any insight or pointers would be greatly appreciated.
Thanks!