Delegates, function pointers, and evil unmanaged code that should die

I'm trying to write a plugin for a popular program in C# instead of C/C++. I decided to make an API so people could easily write these plugins in C#. The basic requirements are as follows.

Make a dll that exports the following two functions:

extern "C" int __declspec(dllexport) plugin_version( );
extern "C" int __declspec(dllexport) plugin_main( char* eventname, void* ptr );

In the first call to from the main program to plugin_main in the plugin, ptr points to a structure with an element that is a pointer to a function in the main program with the following signature. I also listed the delegate I made for it on the C# code.

int plugin_send(char *guid, char *event, void *data);

internal delegate int plugin_function_send( [MarshalAs( UnmanagedType.LPStr )]string guid, [MarshalAs( UnmanagedType.LPStr )]string eventname, [In, Out, MarshalAs( UnmanagedType.AsAny )]object data );

All communication FROM the plugin TO the main program goes through this callback.

Problem #1: While debugging, my calls from the managed code to this callback (using the delegate formed from using Marshal.PtrToStructure where the field in the structure is specified as this delegate type) result in an error from the managed debugging assistant that says:

Managed Debugging Assistant 'PInvokeStackImbalance' has detected a problem in '<mainexecutable.exe>'.

Additional Information: A call to PInvoke function 'InteropTest!InteropTest.plugin_function_send::Invoke' has unbalanced the stack. This is likely because the managed PInvoke signature does not match the unmanaged target signature. Check that the calling convention and parameters of the PInvoke signature match the target unmanaged signature.

If, instead of directly calling through the delegate, I create a function in my wrapper dll and DllImport it with the Cdecl calling convention explicitly specified and give it the function pointer for the delegate and the parameters that should be passed to it (wrap the callback back into the main program), it works without the error. It seems the delegate needs to be called with the Cdecl calling convention but the managed code isn't doing that Unfortunately the callback in the main program is not exported as a symbol and can only be reached through the function pointer that it gives the plugin, so I can't DllImport it with the Cdecl calling convention like I can the functions in my wrapper.

In this callback to the main program, I pass an instance of a class (pointer to struct in C) in the data parameter, one element of which is a delegate that is to be used as a callback back into the plugin code. (My head hurts already). Aside from the two exported functions above, ALL calls from the unmanaged code will be to some function with the following signature, again with the delegate I have made for it:

typedef int (*ttkCallback)(int windowID, char *subwindow, char *eventname, void *data, void *userData);

internal delegate int ttkCallback( int windowID, [MarshalAs( UnmanagedType.LPStr )]string subwindow, [MarshalAs( UnmanagedType.LPStr )]string eventname, [In, Out]IntPtr data, IntPtr userData );

data in most cases is either a pointer to a structure (there are many MANY different ones), a pointer to a string, or NULL. You know which one it is by reading the string passed in eventname and looking up the matching data type in the API, and casting void* data to that type (if it was C code). In my current test code, I am testing just one of these callbacks. I use Marshal.PtrToStructure to read data into the type of structure I know it is. This works, regardless of whether or not I wrap the call to plugin_send. The fields of the resulting structure the main program sends the callback it was given look just as they should. One field in the particular structure sent in my test case is char* text. On return from the callback, that field holds the text that should be printed to the window in the main program. According to the SDK documentation, returning a NULL pointer in this field (AKA leaving it alone since it starts as NULL) means no text should be printed.

Problem #2: Even if I just return 0; (callback must return 0) and don't touch the pointer in the void* data parameter or the memory it points to, I get an error. The error is a buffer overrun detected. Just before that, the plugin callback I talk about in the paragraph just above this one is called AGAIN by the main program. This time data contains "the same pointer to a text string that was returned by the plugin callback on the previous call." In this case, it should be NULL, but the debugger proves that it points to some location in memory that I haven't identified, but by looking at the memory before and after that location I can tell it's somewhere in the program's data memory.

I tried again, this time using Marshal.StringToCoTaskMemAnsi and Marshal.WriteIntPtr to set the pointer text in data on the first call to "Test string," just in case returning a NULL pointer was not in fact acceptable. The buffer overrun detected error still occurred, and the pointer given to the callback the second time in the data parameter pointed to the same location as before, and not to the string previously allocated by Marshal.StringToCoTaskMemAnsi.

Answer this question