C++/VB - RPC_E_CANTCALLOUT_ININPUTSYNCCALL

Asked By domehead100

05-Nov-09 07:40 PM

I am getting this error (RPC_E_CANTCALLOUT_ININPUTSYNCCALL) when trying
to use an IDispatch pointer from another process.

Here is the situation:
1) I have an app ("my app") and a DLL injected into a target process.
As the user interacts with the target process, the DLL will
periodically put some data into shared memory and use
SendMessageTimeout to notify the my app that it needs to take a look
at the data.  Part of that data potentially includes an IDispatch
pointer to an object in the target app that the my app might need to
manipulate.
2)  My app inspects the data, and under certain conditions, it will
unmarshal an IDispatch pointer from shared memory and attempt to make
some Invoke calls on the IDispatch.  Before any of this occurs, the
main app has already done a ReplyMessage to the message sent from the
DLL.

This is when I get the RPC_E_CANTCALLOUT_ININPUTSYNCCALL error.  My
understanding is that this is because COM thinks my app is still
processing the SendMessage call from the DLL and will not allow it to
make any cross-thread/cross-process COM calls while this is so to
prevent any possible deadlocks.

Just before my first Invoke, where the error occurs, ::InSendMessageEx
returns ISMEX_SEND | ISMEX_REPLIED, indicating that my app HAS replied
to the message.  The MSDN documentation indicates that in this case
the calling thread (that did the SendMessage) is not blocked.  So, COM
should realize this and not return the error; but, perhaps COM is just
using ::InSendMessage, which returns TRUE?

I do not think I can use the technique of doing a PostMessage the way
things are coded, because the processing needs to be completely
synchronous between the DLL and the my app; e.g. the
SendMessageTimeout is used to suspend the target app briefly while my
app has a chance to do some quick processing.

Currently the code that processes the message from the DLL is in the
GUI thread in my app (though the GUI is usually hidden when the user
is working in the target app).

I thought that perhaps I could put the code that handles the message
from the DLL into a separate thread, and then use events rather than
SendMessage to fire off processing in that thread in my app.  Instead
of doing a SendMessageTimeout call, the DLL would do a
WaitForMultipleObjects and my app would set one of two events
indicating that it does or does not need to process the data.

Any ideas if this would work or if there is an easier way?  it will be a
fair amount of effort to move the processing code into another thread.

Thanks much for any help!

~Mike

03-Nov-09 11:53 AM

You are trying to make an out-of-apartment COM call inside a message =
handler invoked in response to a sent message (as opposed to a posted =
one; see SendMessage, PostMessage). You cannot do that. Post yourself a =
message, then make the call from that message's handler.


You have already violated this principle when you called ReplyMessage. =
The moment you did, SendMessageTimeout returned on the DLL's side, and =
the DLL proceeded on its merry way. The two processes are now running in =
parallel, not synchronized in any way.


But you finished your quick processing when you called ReplyMessage, =
right? So there ishould be no harm in PostMessage after that.
--=20
With best wishes,
Igor Tandetnik

With sufficient thrust, pigs fly just fine. However, this is not =
necessarily a good idea. It is hard to be sure where they are going to =
land, and it could be dangerous sitting under them as they fly overhead. =
-- RFC 1925

03-Nov-09 01:48 PM

That error occurs when you call a COM method from inside a window message
handler that is triggered by one of the SendMessage...() functions.  COM
uses window messages internally, and thus would deadlock if the call were
allowed to be performed while SendMessage...() is busy.  Thus, you cannot
call COM methods while handling SendMessage...() requests.


You will have to re-design your code a little bit.  If you do not want to
change the DLL code, then the app's message handler will have to make a copy
of the data and cache it somewhere, and then PostMessage() a second message
back to itself, or otherwise signal itself in any other way of your
choosing, so the app knows that data is still pending.  That will allow the
SendMessage() to fully complete, unblocking the DLL and the app's message
queue.  The second message/signal can then safely make the IDispatch call,
using the copied data as needed.  Or, perform the COM call in a different
thread than the one that is receiving the message.



Even though you have replied to the DLL, thus unblocking the DLL so it can
continue on its way, your app's message queue is still blocked while your
message handler is still running.  You have to exit your mssage handler to
unblock the message queue.  Thus the need to perform the COM call
asynchronously.


Yes.


The ISMEX_SEND flag indicates that the calling thread is still processing a
received message.


True, but the receiving thread is still blocked.


That was true in old OS versions, but in Windows 2000 COM was changed to use
InSendMessageEx() instead.  Even then, that still does not solve the root
problem.  The calling thread is still blocked and cannot make an outbound
COM call yet.


You can use PostMessage(), but on the app side privately, not in the DLL.
The DLL can continue using SendMessageTimeout().  Since you are calling
ReplyMessage() on the app side before making the COM call, that means the
DLL does not have to wait for the app to fully process the data before
continuing on its way, so a private PostMessage() on the app side to delay
the COM call a few extra milliseconds should work just fine.


Since you are using SendMessage...() to send the message, it does not matter
which thread receives the message, if you perform the COM call in the same
thread.  As long as the original message handler is still running, that
thread is blocked from making outbound COM calls.  So either exit the
message handler and do the COM call from a secondary posted message, or else
have the original message handler delegate the COM call to another thread so
it is not blocked by the receiving thread's message handler.

05-Nov-09 07:40 PM

Thanks for the help Igor (again :o)).  But, I am still a bit stuck...

My main question is:  Do you think that if I used events instead of
SendMessageTimeout/ReplyMessage, that would appease the COM subsystem?

You're right, once I do the ReplyMessage, that part is not truly
synchronous.  The reason for SendMessageTimeout/ReplyMessage is to be
PostMessage, but I think it would be unreliable since then I am waiting
for a system-defined context switch rather than a direct process that
I have some control of.  After the ReplyMessage, there is further
processing and manipulation of the target app that is truly
synchronous, as you have helped me see more clearly, that is not my
problem.

My quandry with the ReplyMessage is that, as you point out, "The
moment you did [ReplyMessage], SendMessageTimeout returned on the
DLL's side, and the DLL proceeded on its merry way. The two processes
are now running in parallel, not synchronized in any way."  BUT, COM
is apparently not smart enough to figure this out (apparently it uses
InSendMessage instead of InSendMessageEx).

If my app has nothing to do, it simply returns early to the
SendMessageTimeout.  It only does the ReplyMessage if it has some
manipulation/automation of the target app to perform.  That's the
reason for the ReplyMessage, to prevent a deadlock in this case.

~Mike

05-Nov-09 07:40 PM

....

Remy, thanks very, very much.  What you said makes a lot of sense.  I
interpreted the RPC_E_CANNOTCALLOUT error to mean only that COM would
not let me call out because it though the thread that sent the message
was blocking waiting on me, which would cause a deadlock.  I never
considered COM's use of the message loop in my app.  (vaguely
wondering if a MTA help in any way here, but do not even want to go
there even if it was)

I am going to experiment with some combination of PostMessage and
events and see what happens.

Thanks again,

~Mike

M
ot
y
that
et
o
opy
ge
the
ll,
nt
n
to
a

03-Nov-09 03:28 PM

Yes. Though I do not quite see what you expect to gain from the added =
complexity, over simply posting yourself a message.


I do not understand. If you believe your code would have race conditions =
with PostMessage, then it already has race conditions without =
PostMessage. You seem to be arguing that without PostMessage, your =
application will crash and burn less often. I am not sure this is a =
strong argument to make.

What control do you believe you have over a context switch? Recall that =
Windows features preemptive multitasking, and there are systems with =
multiple CPUs that can run two processes truly simultaneously.


So help it, by posting a message to yourself.


So instead of ReplyMessage, call PostMessage and prevent a deadlock this =
way.
--=20
With best wishes,
Igor Tandetnik

With sufficient thrust, pigs fly just fine. However, this is not =
necessarily a good idea. It is hard to be sure where they are going to =
land, and it could be dangerous sitting under them as they fly overhead. =
-- RFC 1925

05-Nov-09 07:40 PM

ith PostMessage, then it already has race
essage, your application will crash and

I think there is an inevitable race condition.  I am trying to get in
and manipulate the target app while the user is potentially typing
keystrokes.  When they type a key that requires my app to take action,
which may involve inserting keystrokes into the input queue, it needs
to begin manipulating the target app before the user presses the next
key or at least before the target app processes that next keystroke
message.  I am trying to always win the race.

indows features preemptive multitasking,
simultaneously.

I have no control over a context switch; however, as soon as the DLL
calls SendMessageTimeout, the target app is blocked until I return or
reply to it (or the timeout occurs).  So, the target will not get a
time slice or if it does it will not do anything because it is
blocking.  With PostMessage, as soon as the DLL calls PostMessage, the
target app is free to do whatever immediately.  The difference is that
with SendMessage, I can take a few milliseconds to:  a) determine if I
need to take action, and b) do some additional processing if needed,
before returning or replying to the message.  By the time ReplyMessage
is called, my app has already determined what it needs to do (often
just SendInput), has queued up whatever data it needs, and is ready
virtually in the next line of code after ReplyMessage to take action
(e.g., SendInput).  The race is between the time I ReplyMessage and
the time I take action.  With PostMessage, the race begins as soon as
PostMessage is called.  I am trying to get a head start to make sure I
always win the race, if that makes sense.  This has all worked
superbly, only now I am trying to add functionality of scripting or
manipulating the target app through automation, and now I have run into
this COM safety feature.  I am not complaining about it, it makes
sense, I was just confused by the documentation.

Thanks again for taking the time to help me with this.  I realize that
perhaps I am being stupid or stubborn or whatever; hope I have not
been irritating though :o).

I will try PostMessage and see how that works, and probably some
combination of events as well (I do not think events will be all that
complex if I do not have to add another thread).

~Mike

03-Nov-09 09:35 PM

I think you misunderstand. I do not suggest that the DLL use PostMessage =
instead of SendMessageTimeout. I suggest that your controlling =
application call PostMessage on _itself_ in the same place where now it =
calls ReplyMessage, then return immediately. Any processing that you =
currently perform after ReplyMessage should be moved to the posted =
message's handler. Like this:

// Before
OnSentMessage() {
DoQuickProcessing();
ReplyMessage();
DoSlowProcessing();
}

// After

OnSentMessage() {
DoQuickProcessing();
PostMessage(UserDefinedMessage);
}

OnPostedMessage() {
DoSlowProcessing();
}

--=20
With best wishes,
Igor Tandetnik

With sufficient thrust, pigs fly just fine. However, this is not =
necessarily a good idea. It is hard to be sure where they are going to =
land, and it could be dangerous sitting under them as they fly overhead. =
-- RFC 1925