Deadlock On Same Mutex

[Nosferatu]

Hello...

I am using a mutex synch mechanism in an multi-threaded environmnet (apache).

When I got several of the threads working - note that Apache choses the threads to be runned from a thread pool - the system gets into a deadlock and all the threads are waiting on the same mutex, without even timing out properly (10 seconds timeout, takes several minutes to get a timeout error).

I've checked and double checked, triple checked, quad checked the program and there is nothing wrong in the logic. Any thread releases the mutex once done with it and there is no way a thread enters an infinite loop or anything like that.
The only thing I can think of is a thread being terminated or killed and the mutex reference does not get signaled, which I don't think it is possible, given the mutex implementation on Windows.

The mutexes are implemented in some sys-dependent function wrappers, inside a DLL, and are spanned across several DLLs. How can this affect the mutexes? Or can it, at all??

I am totally lost on this. Two days of head-banging proved no result whatsoever.

The deadlock occurs in various places. If I removed the deadlocking mutex, I got another deadlock, in another part of the application, on another mutex.

There are no crossed mutex references - no other mutexes are asked for while a thread is holding a particular mutex, I have been careful on circular dependencies on this. And still this happens.


Any guides are greatly appreciated.

~~Razvan


[red]Nosferatu[/red]
We are what we eat...
There's no such thing as free meal...
once stated: methane@personal.ro

 

[xwb]

I don't know very much about the particular problem you are talking about. However, you could have a look at the implementation and see if they are using CreateMutex for the creator and OpenMutex for the user. It should be using WaitForSingleObject to wait for the mutex. Maybe it is doing something 'clever' or has its own implementation of mutexes.

If all else fails, try changing the mutex calls to semaphores. These are the same thing at an inter-process level instead of an inter-thread level.

 

[teriviret]

Just some ideas:

Did you look for cases where a thread might grab a mutex twice but only release it once? Some mutex implementations allow a thread to grab a mutex more than once, and they keep a semaphore-like count where the mutex is only truly released once the thread releases it as many times as it grabbed it.

Are there any places where a thread may be grabbing a mutex but then throwing an exception soon after, skipping the code that would normally release the mutex?

Another thing you could try is insert some additional code wherever a mutex is grabbed or released, to print out to a log file a line saying which mutex was grabbed/released by which thread. This might be a lot of tedious work though.

I like what xwb said -- turn the mutexes into semaphores, and watch to make sure the semaphore counters actually go back to the right values whenever they are released. You could add some code right before every release call, to check to make sure the semaphore counters are what you expect them to be.

You could also add a variable for each mutex, where a thread stores its own ID number just after it grabs a mutex. Then you can inspect it at any time to see which thread actually owns the mutex.