SDK - Question About Objects

[JTaylor]

Not sure how to ask this and will probably end up sounding silly but...

Is there a way to pass an object(?) to WinBatch and then Pass it back?   I will use a database record set as an example.  If I have a pointer(?) to query results can I pass that to WinBatch and then pass it back to the Extender to do things like stepping through the rows?   If so, what type do I need to use for Returnresult()?

Jim

[td]

Assuming you are referring to COM Automation objects then yes, it is possible.  COM Automation objects are represented using the Win32 variant data structure. More specifically they are usually variants of type VT_DISPATCH.  Many years ago I wrote an example extender for testing the WIL extender interface that also manipulated variant data structure of various types, including passing them to and from the extender to WinBatch and back. 

That said, I don't know if the code still exists. I also vaguely recall finding a few limitations that were never addressed because no users had any interest in doing that kind of code-intensive programming.

[JTaylor]

Also, if willing...would you show me how to pass back a two dimensional array?  If I had something like the following, what types should I use for the array that Returnresult will like it?  Thanks.

Jim

Code (c) Select Expand



datatype?  arr[2][2];

arr[0,0] = "Hello";
arr[0,1] = "World";
arr[1,0] = "April";
arr[1,1] = "2020";

return Returnresult(arr);



[JTaylor]

Okay.  I will see if I can sort that out.   Have hacked away at it some but thought I should ask before spending more time.   If you get bored, some examples of passing various types, like this, arrays, etc., back from the Extender would be greatly appreciated  :)

Jim

[td]

Addons.h documents the DllArrayServices and DllArrHandler DLL callbacks for array creation and handling.   For the most part, returning a value of a specific WIL type is simply a matter of filling in the appropriate LPVIPERVAR structure members and then making a call to the LPVIPERCALLBACK function.

[JTaylor]

I always forget about addons.h.   Was thinking there was more info but wasn't finding it.  Thanks.

Jim

[td]

A couple of more notes on WIL extenders and COM Automation objects. 

The internal WIL representation of COM Automation objects and objects in general is a somewhat complex set of template interface wrappers that are instantiated as needed to express the functionality provided by the object.  However, the WIL extender interface presents COM Automation objects to extenders as a variant of type VT_DISPATCH.  This conversion happens automagically when a variable containing an object reference is passed to an extender function that expects the variant parameter type.

The old C++ extender example in the WIL SDK contains the IsMember extender function which is a simple example of an extender function that accepts a COM Automation object.

[JTaylor]

Okay.  Thanks.  Thought it would be interesting to create an SQLite Extender and got the opening/creation and command execution working but stuck on SELECT operations as that requires more complicated stuff.   Have had to replace the wall beside my desk three times already from banging my head against it and other than knowing a lot of ways that don't work haven't got much further  :)  Not sure how many more hours I can afford and while I am continually learning from doing such things the array stuff still doesn't make much sense in regards to what is needed.   I read through Kirby's array extender source and that helped a little.  Will take a look at IsMember and see if that turns on any light bulbs.   Would love an example extender (in the latest format) that demonstrates all the various options.  If I keep plugging away maybe some of this will make sense, without examples, before I die.

In any event, decided to try a .NET DLL and see how that worked.  Turns out it works okay.  Not as nice as an extender but only requires the one DLL and no registrations so no different regarding distribution.

Thanks again.

Jim

[td]

COM and DCOM are fundamental parts of the Windows OS.  The technology forms the foundation of everything from the Windows shell to .Net and Windows Store apps. To that end, MSFT's native COM developer documentation is the best place to learn the fundamentals of COM programming.   The extender specific programming part is limited to passing data to and from the extender to the WIL interpreter and that is fairly straight forward.

[td]

Haven't looked at it or used it but SQLite does have a straight C/C++ API.  The API might make for a fast single DLL (assuming you can statically link to the SQLite dll) solution.  Just a thought.

[JTaylor]

Sounds good in theory but in practice, for people as ignorant as me, not so much ;)   I was impressed I got the parts working that I did.  I may tackle it again though.   Would like to figure out how to do the more complicated data types and need a practical and useful project to learn on.   That is where I was getting stuck.  Wasn't trying to implement a COM interface....at least I don't think so :)   The .NET option works but not as smooth as I would like.   Not sure it is all that big of a deal.   Mostly trying to learn and this is something I think others would find useful.   Thanks.

Jim

The extender specific programming part is limited to passing data to and from the extender to the WIL interpreter and that is fairly straight forward.

[td]

There is a lot more to the Windows COM API than implementing interfaces.  There are native functions for all manor of variant conversions and safearray manipulation.  There are also many rules of the road for the relationships between interfaces and the handling of data types. Knowledge of COM data structures and APIs is or at least was fundamental to Windows programming.  Once MSFT embarked on the effort to dumb down Windows programming, the knowledge became less essential until you bumped up against the limits of MSFT's "simpler" programming model.

I guess the reason I stayed in the business was that I like challenges and like learning new things.  The reward is sufficient to do the hard work of digging into the details.  Sometimes that digging involves reading many pages of MSFT's dry API documentation.  Other times it involves reverse engineering a subsystem to understand how to use its components.  Mostly, it is just disciplined hard work.

[kdmoyers]

Thought it would be interesting to create an SQLite Extender

That sounds fantastic.  There are plenty of use cases outside my office lan, and sqlite seems like the perfectly sized solution. Let me know privately if this effort could benefit from underwriting.

[td]

We have looked into integrating SQLite directly into the WIL interpreter but decided that resources could be better spent elsewhere.  It is a little hard to justify when there are already multiple ways to access SQLite in WinBatch scripts.

I guess I would be interested in hearing justification for an extender when both COM and dotNet access is already readily available.  The two advantages I thought of off the top of the addled cranium where speed and the conveniences of a no install single file (the extender dll.)

[JTaylor]

Speed is always good but issues related to distribution is a BIG one for me.   Plus native stuff is, generally, cleaner to use.

I am happy to do the work as it is how I generally learn new stuff.   Especially if you could throw me a tid-bit of help here and there with the interface :)

Hope to get back to hacking away at it again soon either way.  Trying to get this HtmlAgilityPack thing sorted at the moment. 

Jim

[td]

Sounds good in theory but in practice, for people as ignorant as me, not so much ;)   I was impressed I got the parts working that I did.  I may tackle it again though.   Would like to figure out how to do the more complicated data types and need a practical and useful project to learn on.   That is where I was getting stuck.  Wasn't trying to implement a COM interface....at least I don't think so :)   The .NET option works but not as smooth as I would like.   Not sure it is all that big of a deal.   Mostly trying to learn and this is something I think others would find useful.   Thanks.

Jim

One last note:
Passing various data types to and from the extender to the interpreter is really straight forward in practice as well as theory.  Look at both the old and new C++ examples in the SDK.  There are method overloads for returning most of the supported WIL datatypes including variants. For an array, it is simply a matter of using the overload that has an LPVIPERVAR as its parameter.  That's the LPVIPERVAR you obtained using the DllArrayServices function.  As for passing data types into the extender, it is simply a matter of specifying the correct parameter type in a function's table entry.

Based on tech support questions and SDK downloads there is very little interest in creating WIL extenders. The level of C/C++ language skills necessary to create extenders is no longer that common among WinBatch users. The mindset of users has changed as well.   The drive to simplify programming has changed how code writers perceive the task.  There is more expectation of gluing together API calls and less interest in taking the time to discover how things work. Not saying that the new approach to coding is necessarily always a bad thing.  Creating WIL extenders is a matter of gluing together API calls but also requires a bit of work to discover how the extender interface works.  I know the first time I created an extender there was less information about it than there is now. As I started working with the API my understanding of how to accomplish what I wanted to do slowly expanded.  Extender writing does have something of a small barrier to entry and that is probably a good thing for multiple reasons. 

Buried deep in the WinBatch todo list is a task to create a C# SDK for WIL extenders.  It's a very low priority item but you never know...

[JTaylor]

Making a bit of progress.  Able to create an Array and pass it back properly so that WinBatch recognizes it but stuck now and not seeing how to make other steps function. Mostly due to data type incompatibility.  It says to use a "vipervar" for a couple of options but not seeing a way to cast that to LPBYTE.    LPVIPERVAR will Cast but then nothing ever returns.   Same thing with assigning values to array.  It just closes with no response.   A little [direct] guidance would be appreciated :-)  Thanks.

I know the code is a bit messy with all the comments and various attempts.   Just trying to work through the options to make sure I know how and that they work before I actually use them for something useful.   Hopefully it make enough sense to respond.

Jim

Code (c) Select Expand




LPVIPERVAR Arr;
LONG nRequest = 2;
DWORD lpBuf[5] = { 40, 40, 0, 0, 0 };

DWORD dwSub1 = 1;
DWORD dwSub2 = 1;
DWORD dwSub3 = 0;
DWORD dwSub4 = 0;
DWORD dwSub5 = 0;

LPVIPERVAR init_arr;
DLLARRAYSERVICES1 arr_info;
long J;

vipervar vv;   //OR DO I USE LPVIPERVAR?


// THIS ONE WORKS!!!!!!!!
J = DllArrayServices(Arr, (LPBYTE) lpBuf, sizeof(lpBuf), 2);

//HOW DO I CAST 2nd Parameter to LPBYTE?  It says there is no way.
J = DllArrayServices(Arr, (LPBYTE)arr_info, sizeof(arr_info), 1);

//Using LPVIPERVAR allows me to CAST to LPBYTE but NOTHING Ever Returns.  No errors or anything.   It says this should be vipervar but how do I cast to LPBYTE?
init_arr->lpstr = "";
J = DllArrayServices(Arr, (LPBYTE)init_arr, sizeof(init_arr), 3);

//Using LPVIPERVAR allows me to CAST to LPBYTE but NOTHING Ever Returns.  No errors or anything.  "vipervar" doesn't allow a CAST.
//Also, What is the proper way to do the subscripts?  Do I just use zero for those not being addressed?  Am I defining them properly?

vv.lpstr = "HELLO";
J = DllArrHandler(Arr, dwSub1, dwSub2, dwSub3, dwSub4, dwSub5, vv, 2);

return ReturnResult(Arr);
int row_cnt = 0;




Jim

[td]

Your issues have more to do with the fundamentals of the C and C++ languages than the specifics of the extender WIL interface.

The documentation in addons.h for the DllArrayServices function option one states,

1  =  get array info    (lpBuf = pointer to DLLARRAYSERVICES1 structure)

Note the usage of the term "pointer". You can't cast a structure to a pointer (a structure is not an integral type) but you can cast a pointer to a structure to a different type of pointer. So you would end up with

        J = DllArrayServices(Arr, (LPBYTE)&arr_info, sizeof(arr_info), 1);

or in proper C++ (from an old example)

   // Get an array descriptor.
   DLLARRAYSERVICES1 ArrayInfo; // WIL array desctiptor.
   if ( DllArrayServices(vvParam1, reinterpret_cast <LPBYTE>(&ArrayInfo), sizeof(DLLARRAYSERVICES1), 1) ) {
      return ReturnError(RESULT_MINOR_ERROR, IDS_ARRAYDESCFAIL); // 201: Unable to read array information
   }

The same concept applies to a call to the DllArrHandler API. The vipervar must be a pointer but the pointer must point to something. You also need to indicate the WIL variable type of the vipervar in the inittype member of the structure. From addons.h:
  int     inittype;                    // bitmask of VARTYPE_* bits

[JTaylor]

I am sure I am missing many fundamentals but did note the pointer requirement and thought I had tried what you posted.  Have tried so many things it is hard to remember all the combinations.   That was just my latest attempt.   What you posted doesn't work either but assuming I am missing something else.   It did respond with the error you put in, once I added it to the Error Table anyway.   None of the ARRAY error messages from previous SDKs are in the latest one, as far as I can tell, if that is something that should be there.  Here is my latest.  I stripped out all the error message stuff to keep the code small for here.   

Whatever call I make, apart from creating the array, the script just goes away with no message at all.  I do have a Message() in the winbatch script right after the function call so it never returns to the script.   I have experimented commenting out all but one, apart from creation, and they all act the same.   I am obviously missing something but no idea what since what you posted acts like everything I have done.

Sorry if I am being a pain but I have put in a lot of hours of reading and trying things but end up at the same place no matter what I do.  Appreciate any more pointers on what I am doing wrong here.  Thanks again.

Jim

Code (c) Select Expand




LPVIPERVAR Arr;
DWORD lpBuf[5] = { 40, 40, 0, 0, 0 };
DWORD dwSub1 = 0;
DWORD dwSub2 = 0;
DWORD dwSub3 = 0;
DWORD dwSub4 = 0;
DWORD dwSub5 = 0;
vipervar init_arr;
int rcode = 0;
vipervar vv;   


// THIS ONE WORKS!!!!!!!!
rcode = DllArrayServices(Arr, (LPBYTE)lpBuf, sizeof(lpBuf), 2);

vv.lpstr = "HELLO" ;
  rcode = DllArrHandler(Arr, dwSub1, dwSub2, dwSub3, dwSub4, dwSub5, (LPVIPERVAR)vv, 2);  //Type Cast Issue using vipervar.  Can remove error using LPVIPERVAR
return ReturnResult(rcode);

       init_arr.x = 1;
       rcode = DllArrayServices(Arr, (LPBYTE)init_arr, sizeof(init_arr), 3);   //Type Cast Issue using vipervar.  Can remove error using LPVIPERVAR

     DLLARRAYSERVICES1 ArrayInfo; // WIL array desctiptor.
     rcode = DllArrayServices(Arr, reinterpret_cast <LPBYTE>(&ArrayInfo), sizeof(DLLARRAYSERVICES1), 1);
   

    return ReturnResult(Arr);
int row_cnt = 0;


[td]

The code I posted comes from an old same extender that still works just fine in the context of that extender.  It is not something you can just cut & paste (hoping we are beyond that) as I did not include parameter declarations or other context.  It is something that you should read to understand the point I made about pointers. 

However, you seem to have completely ignored the comments from my previous post about pointers which makes the code below wrong.
        vv.lpstr = "HELLO" ;
        rcode = DllArrHandler(Arr, dwSub1, dwSub2, dwSub3, dwSub4, dwSub5, (LPVIPERVAR)vv, 2); 
        return ReturnResult(rcode);

You are not using a pointer to a vipervar. You are using the structure itself.  You are also not setting the WIL Variable type. You are potentially causing all kinds of stack memory corruption.

Also, the array related extender callbacks have not changed in at least 10 to 12 years.

[JTaylor]

I did not ignore what you said.  I have also done a LOT of reading.   I have freely admitted my ignorance, which is what I am trying to correct here.  Option 1 and 2 of the DLLArrayServices say they need a pointer and is what you referenced in a previous post regarding Service Request #1.   None of the other options mention needing a pointer other than #2 and why I hadn't included that in those.  If they need pointers it might be good to update documentation.   Also, the documentation could use some sample code and save you all this hassle, like in the WinBatch documentation.

That aside I have made a bit of progress as I am now getting error codes as a response for the Handler and Request 1 of the Service, rather than it just quietly closing. 

Also, I don't understand about setting the WIL variable type.  It sounds like you mean as part of the setup for making the DLLArrayHandler/Service call.  If so, I need an example of what that looks like because I don't see how I would know what is needed from the documentation.  I know what you are talking about regarding the inittype option within the structure from ADDONS.h, assuming that is what you mean.   If you mean something else please clarify.

Did I define the subscripts for the Handler call properly?   If I had a 2 dimensional, 40x40, array and wanted to put "HELLO" in 0,0 would I just make them all DWORDs = 0?  Would like to at least scratch that off my list as a potential problem as something about the Handler call is corrupting my Array that I have created, and WinBatch likes, before I call the Handler.


Jim

[td]

I did not ignore what you said.  I have also done a LOT of reading.   I have freely admitted my ignorance, which is what I am trying to correct here.  Option 1 and 2 of the DLLArrayServices say they need a pointer and is what you referenced in a previous post regarding Service Request #1.   None of the other options mention needing a pointer other than #2 and why I hadn't included that in those.  If they need pointers it might be good to update documentation.   Also, the documentation could use some sample code and save you all this hassle, like in the WinBatch documentation.

First, DllArrayHandler always needs a pointer or a NULL pointer to the LPVIPERVAR parameters. That is why the prototype has an LPVIPERVAR as the variable type instead of vipervar. If it didn't need a pointer it wouldn't be defined as a pointer in the function prototype.  (Note that LPVIPERVAR is declared as a pointer in addons.h)  This is the type of stuff that is basic C/C++.

That aside I have made a bit of progress as I am now getting error codes as a response for the Handler and Request 1 of the Service, rather than it just quietly closing. 

Also, I don't understand about setting the WIL variable type.  It sounds like you mean as part of the setup for making the DLLArrayHandler/Service call.  If so, I need an example of what that looks like because I don't see how I would know what is needed from the documentation.  I know what you are talking about regarding the inittype option within the structure from ADDONS.h, assuming that is what you mean.   If you mean something else please clarify.

There are almost a dozen examples in the SDK of setting the WIL variable type before returning a value to the DLL.  Just look at any of the C++ overloads of the ReturnResult member. As previously mentioned you need to set the inittype member of the vipervar structure with a VARTYPE_* bit as also defined in addons.h.
  int     inittype;                    // bitmask of VARTYPE_* bits

Did I define the subscripts for the Handler call properly?   If I had a 2 dimensional, 40x40, array and wanted to put "HELLO" in 0,0 would I just make them all DWORDs = 0?  Would like to at least scratch that off my list as a potential problem as something about the Handler call is corrupting my Array that I have created, and WinBatch likes, before I call the Handler.

It is very simple. The DWORD parameters are just array dimensions.  Since WIL arrays can have up to 5 dimensions there are 5 DWORD parameters. You access array elements just like you would in WIL syntax [0,0,0,0,0].  The difference is that you specify all five dimensions even when the array has fewer than five and you are passing parameters instead of using square bracket syntax. Just set the unused dimensions to zero (although it doesn't matter much because the DLL will only examine the dimensioned that have been defined for the array.)

My approach to learning and teaching is lacking and not a good match for some.  I do not want to neglect the needs of other users so I really can't spend any more time on this.  It is all doable if you take the time to think through it.

[ChuckC]

It's been ages since I last looked at the source code I wrote when I developed the Windows Terminal Server extender and the EHLLAPI extender, but both of them have functions in them that return WIL arrays from extender functions.

If there are no objections on Tony's part, I could share them out-of-band of JTaylor.  They are functional examples of complete C++ extender code that exercise quite a number of WIL extender interface features.

[JTaylor]

Thank you for your time and sorry I took up any of it.   In all fairness, all I wanted, and asked for, from the beginning was some [working] examples which I could use to understand how the calls are made.   That probably would have ended the discussion there as I could work backwards through the call requirements and would have saved both of use a great deal of time and frustration.

Chuck, if allowed, that would be GREATLY appreciated.

Thanks again.

Jim

[JTaylor]

I did just find a 2004 version of the SDK which has some examples.   Hasn't born fruit yet but at least something to study.   Thanks again.

Jim

[td]

Thank you for your time and sorry I took up any of it.   In all fairness, all I wanted, and asked for, from the beginning was some [working] examples which I could use to understand how the calls are made.   That probably would have ended the discussion there as I could work backwards through the call requirements and would have saved both of use a great deal of time and frustration.

It has been my experience that encouraging cut & paste programming eventually leads to more time spent and frustration instead of less.  The approach is to learn the concepts instead of imitating them.

[JTaylor]

Goal wasn't to cut & paste.  It was a way to understand the calls.  Figured out the problem though after looking through the example I found.  It was with the initial array definition.   Since it worked I thought I was good on that front.   Once I corrected that problem everything else now works.    In any event, as with all problems created by a simple oversight and misdirection I learned some things so not all wasted time, on my part anyway.   Now to see how it works in real usage.

Thanks again.

Jim