C++/VB - Gracefully shutting down a thread in a DLL

XP/Server 2003.

We have a DLL, very old, and being used by hundreds of applications.  I'm in
the process of adding a maintenance thread (the first) to that DLL.  It has
to be done in a way that won't require me changing hundreds of other
programs.  Many of these are customer programs outside of my control.

I wrote the thread and am starting it when the DllMain is called with
DLL_PROCESS_ATTACH.  So far so good.

The thread starts, runs, and everything works, until it's time to shut down.

I can't get the thread to gracefully exit.  When the host exe exits, Windows
calls DllMain with DLL_PROCESS_DETACH.  In that I set a terminate flag for
the thread to see, but thread in the dll doesn't run long enough to see it,
so it never completes, and never cleans up.

In researching this on the web, it turns out that DLL_PROCESS_DETACH isn't
called until after all process threads have already been halted,
ungracefully if necessary.  That makes it sound impossible to signal the
thread at that time to have it exit.

So is there any good way to get the thread in a dll to gracefully exit
without modifying the exe's that use the dll?

Thanks for any help.

Bruce.

There is really no good solution to your problem, and, you should try
to have your APIs/functions to express an Initialize/Deinitialize paradigm.

First of all, creating a thread in DllMain is unsafe, because if your
DllMain returns FALSE, then, you may end-up unloading the module
while the thread is still executing.

Likewise, even if you succeed, your thread must acquire a reference
on the module (by either forcefully calling LoadLibrary,
or GetModuleHandleEx), to guarantee that nobody can make the thread
run on an unloaded module, by artificially calling FreeLibrary.

As a secondary item, synchronizing Dll-Termination and a thread shutdown
is almost always guarantee to get you into a loader lock deadlock,
because you are holding the loader lock in DllMain(PROCESS_DETACH)
and, you need the LoaderLock to call ExitThread.

The only paradigm that has a chance to work is an explicit
Initialize/Deinitialize
pair of functions, for each thread.
That is the way in which, for example, ws2_2/wsock32
does cleanup of worker threads, by using WSAStartup/WSACleanup
to count the number of outstanding users, and, separating the DLL-Lifetime
by the users lifetime. Other examples are CoInitialize/CoUninitialize,
that may optionally perform cleanup of extra worker thread created
to perform decoupling of inter-apartment calls.

--

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

m in
has

If you possibly can, don't do this :-)

What is the thread for?  Could the same
functionality be provided using, for example,
a windows timer?

If you do have to do this then terminating
the thread is hard.
Ivan's suggestion is the best/safest, but I guess may
not be an option given the large installed client base.

Another choice is to make use of a helper DLL, and
the FreeLibraryAndExitThread API.

Your existing DLL would load a new helper DLL and
call a method to start a thread in that DLL.  This thread
bumps the use count of the new DLL.
[Note: don't start the thread when in DllMain, try to hook
an existing method call that makes the thread necessary]

The DLL unload for your existing DLL is changed to call
a method in the new DLL that flags the thread to complete
and then it unloads the DLL.  The use count drops from
2 to 1 and your existing DLL exits cleanly.  The new DLL is
still in memory (as the use count is 1) and the thread is still
running.

Eventually the the thread gets scheduled, and it calls
the FreeLibraryAndExitThread API to drop the use count
to 0 and unload the new DLL.

Jeffrey Richter wrote about this in MSJ -- start at
http://www.microsoft.com/msj/archive/S202B.aspx
in the answer to the question


Regards,
Roger.

I'm not sure.  How would a windows timer help clean up resources at dll
unload time?

The thread will be maintaining and aging a cache of objects that are created
on the fly as the dll functions.  I can't depend on when the dll will be
called by the host exe's, so I need a worker thread to do the aging.  It's
working now, other than for the exit cleanup problem.

The dlll doesn't have a message handler, if that's what you mean by windows
timer.

Thanks for the help.

Bruce,

Well, if I ever get a chance to design a new interface, I will do that next
time.  The one I'm dealing with is at least 15 years old and already woven
in to hundreds of our and customer programs, so that's not an option.

Thanks anyway.

Bruce.

Bad idea. You did a lot of work on 99% that will never be able to work
because of the rest of 1% ;-|

Seriously: there's no way. The cleanup thread must be terminated
before PROCESS_DETACH is called by the loader. Perhaps if you hook the
quit?

Christian

s

Yup.  If you have not got one then that idea is no use :-)

Well, if it is guaranteed to run in a UI thread then you can let the main
app's message loop dispatch your messages to you.

As many have pointed out, there is no always-correct
solution to your problem.
Managing a thread lifetime from DllMain() is too error prone.

Maybe if you can reformulate your requirements, there are other
viable solutions, such as using the Kernel32/NtDll thread pool

--

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

Unfortunately, about 95% of the exe's using this dll do not have a message
queue either.

Thanks,
Bruce.

That sounds like a possibility.  I have never hooked an API so I will do some
research on that unless someone knows why that might not work.

Thanks for the idea!

Bruce.

Then I suggest you carefully study Roger's first message about using a
helper DLL.

I'm struggling to understand your suggestion.

From my reading of your suggestion, you proposes the thread is not in my old
dll, but a new dll.

What good does that do me?  My thread needs access to the various resouces
(globals, STL lists, etc) of the old dll, so my thread must be located in
the old dll.  Right?  What good does a new dll and a thread located there
buy me?

Bruce.

You'll be able to shutdown your thread in your primary DLL DllMain
DLL_PROCESS_DETACH handler, by signalling it through an event or a message.
Actual shutdown will happen at the thread's leisure time. When the thread
exits, it will unload the secondary DLL by calling ExitThreadFreeLibrary.

In subsequent postings, this part of Mr. Orr's message seems to be ignored
by everyone including Mr. Orr:


That's a pretty good article, on par with his books.



If you possibly can, don't do this :-)

What is the thread for?  Could the same
functionality be provided using, for example,
a windows timer?

If you do have to do this then terminating
the thread is hard.
Ivan's suggestion is the best/safest, but I guess may
not be an option given the large installed client base.

Another choice is to make use of a helper DLL, and
the FreeLibraryAndExitThread API.

Your existing DLL would load a new helper DLL and
call a method to start a thread in that DLL.  This thread
bumps the use count of the new DLL.
[Note: don't start the thread when in DllMain, try to hook
an existing method call that makes the thread necessary]

The DLL unload for your existing DLL is changed to call
a method in the new DLL that flags the thread to complete
and then it unloads the DLL.  The use count drops from
2 to 1 and your existing DLL exits cleanly.  The new DLL is
still in memory (as the use count is 1) and the thread is still
running.

Eventually the the thread gets scheduled, and it calls
the FreeLibraryAndExitThread API to drop the use count
to 0 and unload the new DLL.

Jeffrey Richter wrote about this in MSJ -- start at
http://www.microsoft.com/msj/archive/S202B.aspx
in the answer to the question


Regards,
Roger.

ld

The thread function is in the new DLL, and the thread is responsible
for unloading this DLL.
So the original DLL can be unloaded leaving the thread still exiting.

Read Jeffrey's article which explains it better than I am.

Regards,
Roger.

I'm lost in the details and trying to see the bigger purpose, and I'm
feeling really dense here.  Let me see if I can ask some dumb questions and
hopefully the light will trigger at some point.


How?  By explicit linkage or by LoadLibrary?  So in order, my exe would load
my old dll, which would load the new dll?

call a method to start a thread in that DLL.

From where?  In the DLL_PROCESS_ATTACH for the old dll?  Is the new library
even loaded at this point?

bumps the use count of the new DLL.

Just calling _beginthread does that?


By that I assume you mean the DllMain of the old DLL?  So I should find some
other event that happens after the old DLL is loaded to call the thread
starter in the new DLL?

a method in the new DLL that flags the thread to complete
and then it unloads the DLL.

You lost me there.  What is "The DLL unload for your existing DLL is changed
...."?  Do you mean modify the old DLL's DLL_PROCESS_DETACH code?

2 to 1 and your existing DLL exits cleanly.

Why?  What even dropped the usage code?   Usage count for the new dll or for
the old dll?  Why is the usage count important?  How did it get to be 2?


Ok, I don't understand how we got here, but I do understand that as the
goal.  To somehow trigger my old DLL theads to complete before
DLL_PROCESS_DETACH in the old dll is called.

Is part of the trick here to get the system to call the new dll
DLL_PROCESS_DETACH *before* it calls the DLL_PROCESS_DETACH in the old dll?


I guess I don't understand usage counts.  If the exe has already shutdown,
and if the old dll has already shut down, wouldn't the usage count of the
new dll already be zero?  Weren't those the only 2 users of the new dll?

Bruce.

Let's try to address the bigger purpose ...

Windows ensures that calls to Dll entry points are always serialised,
using the internal "LoaderLock".

This means that, while handling a DLL_PROCESS_ATTACH or
DLL_PROCESS_DETACH, any
threads that attempt to stop or start will block until the
DLL_PROCESS_XXX completes.
Once the loader lock is released these other threads can then do
DLL_THREAD_ATTACH/DETACH
processing.

So you can't wait for a thread to complete in a dll unload.
But if you *don't* wait for a thread to complete it might be executing
in the DLL itself
and will fail nastily if the DLL code gets unload 'under its feet'

The best solution is not to get there -- ensure the DLL is not
unloaded until all
the threads are completed.  This seems not an option for you given the
large existing codebase.

The proposed solution is to create a new DLL, that provides an entry
point
for your thread function and that is only unloaded when the thread
completes.

So you don't need to wait for the thread to complete inside
DLL_PROCESS_DETACH
just request it to safely 'commit suicide' later on using
FreeLibraryAndExitThread.

Note: if the process as a whole exits then you have additional
problems tidying up -
the threads will be unceremoniously aborted during process exit.

Regards,
Roger,

OK, I've re-read your very first posting more carefully and this seems
to be your situation,
rather than that of safely unloading the DLL in a running process.
Apologies
for failing to notice this.

What tidying up needs doing - can you do it in the main thread?

Regards,
Roger.

The only option for dealing with this, is a helper process waiting on the
handle of the main process (which is signaled at exit).

I just read that. Dear God, that is now in my top ten list of the
cleverest hacks I have ever read about.

Bob Moore
http://bobmoore.mvps.org/

Correct.   The dll is never dynamically loaded.  It is loaded when the exe's
or other dll's load, and stays resident nutil the host exe or dll exits.  I
just need to get the thread in the dll thread to end gracefully when the
host exe exits.


There is no main thread my dll.  Only an API interface that gets called by
hundreds of exe's and other dll's, many written by customers.  The only
threads that exist are the exe's which I can't change.

There are a variey of resouces in the DLL that are allocated as needed, some
in shared memory, some gobals in the DLL, a cache, STL lists, etc.  The DLL
needs to unwind what it does when it unloads or those reasources (such as
shared memory and shared globals) will be left in an undefined state.

The maintenance thread in the DLL is the one I need to add.  It will start
when the DLL loads and end when the DLL unloads, but as I'm finding out
here, that's not as simple as it sounds.

The maintenance thread in the dll may be in the middle of modifying those
resources when the exe unloads, so I need to be sure the dll thread exits
gracefully in that situation.

Bruce.

I'm sorry.  I just don't see what a a thread in a new dll is going to buy
me.  That's not what I need.  I need the thread to be in the old dll so to
have access to it's resources.


And that's exactly the problem I have.  I need to get the thread in the old
dll to gracefully exit before everything else is shut down.  I thought
DLL_PROCESS_DETACH would to that, but my thread has already been stopped at
that point, never again to run.  So I guess I need an even earlier
notification of an impending unload that happens even before
DLL_PROCESS_DETACH.  Is there such a thing?

Bruce.

I am just not getting this extra dll and thread thing.  So while I think
about that more, I am pursuing hooking ExitProcess to see if that will do
what I need.

Bruce.

If a client process plays by the rules, he'll call FreeLibrary before
calling ExitProcess. Then your thread will shutdown more or less gracefully,
and the secondary DLL will be unloaded by ExitThreadAndFreeLibrary.

If a client loaded your "primary" DLL by implicit load (referenced by other
DLL or main EXE imports), then you will get non-NULL lpReserved argument in
DLL_PROCESS_DETACH. In this case your best bet is to simply allow the thread
to run. During process exit, the libraries are not unloaded, they are simply
discarded with the rest of the process memory. It's safe to let the thread
to run until it gets terminated.

Again, you don't need to hook ExitProxess, that would be a terrible hack.

Since no one calls LoadLibrary, no one calls FreeLibrary.


That's the problem, the thread never runs to completion.  It is frozen once
DLL_PROCESS_DETACH. is called and never runs to completion.


I'm desperate.

Bruce.

Roger, I've been reading the web page about using a stub dll and reference
counts about 50 times so far and I'm very slowly begining to understand what
it's talking about and how the scheme works.

However, in re-reading the above, it occurs to be that scheme just can't
work because that's exactly what's happening here.  ExitProcess is being
called by the exe exiting, and that shuts down all threads, whereever they
might be, in and out of the dll.  At that point reference counts and stub
dll become irrelavant because everything is stopped.

So if I'm undersstanding this better now, I have nothing to gain by pursuing
a stub dll.  Is that your understanding too?

And that my only salvation is the hooking of ExitProcess.

Bruce.

STL lists, cache, globals all will cease to exist when the process exits.

Unless another process has the shared memory open, Windows will clean it
automatically at process exit.

The only thing you need to be concerned about is consistency of persistent
storage.  If your thread was in the middle of writing a file when the
process exited, you could have issues.

This last sentence is not normally correct.  The thread needs to be specially
designed to be safe to terminate at any time...

Normally, if you have ExitProcess while your secondary thread is busy doing
stuff, the design is already broken.

If a process is shutdown orderly, any secondary threads would be quiet. If a
process is shut down abruptly, it doesn't make sense to care about those
threads.

As I've mentioned elsewhere, we sell an API that lives in a DLL.  The
programs that use our DLL are written by those customers and totally out of
our control.  So the DLL has to do whatever it can to compensate for bad
application designs.

The DLL has shared memory open that has to be cleaned up at process
termination time.  For a simple example, imagine a object counter sitting in
shared memory, monitored by multiple exes using the same DLL, all using a
variable number of objects.  When one process exits, someone must decrement
the object counter by those that were in use at ExitProcess time.  Without
that cleanup, the counter would be rendered useless for all the other
processes still running.

Bruce.

You need another process running, managing the objects, and cleaning up
after each process when it exits (this is essentially what happens with all
kernel resources, the driver is running independently and reclaims the
resources when the process using them exits).  Have each DLL register the
process handle with the central cleanup process when it loads, then have the
cleanup process call WaitForMultipleObjects to detect when each client
process exits.

Or switch over completely to a client-server architecture, so none of your
resources are actually owned by the third-party process.

This is a classic case of broken by design ;)



The fact that your shared memory structure requires this type of metadata to
be updated cooperatively means that it cannot ever work well in your
environment.  As others have mentioned though, there are solutions that won't
require any API changes.



The simplest is to create a resource arbiter / garbage collector.  This
would take the form of either a UM service of a KM driver and would ensure
that abandoned resources were returned to the free list etc.



Another solution, perhaps more complex to code but maybe better suiting your
situation, would be to redesign the shared memory structure to not require
cleanup.  This would be the case if the acquire / use algorithms were able
to detect abandoned resources and free or reuse them as appropriate.



The second solution is a more robust solution but has the performance
penalty that each allocation / access would need to verify consistency.
This may or may not be a problem for your situation.  The first solution has
the benefit of being easier to code and performing better in the general
case, but being more fragile at execution time because things can go subtly
wrong and it could be hard to detect (ie termination of the service process,
change of ACL on the driver, etc.)

Hi Bruce,



Did you get the solution for your problem finally? Im also in a similar situation like yours.