DOMData Extender (Alpha?)

[JTaylor]

This will probably be my last Extender.   At least I can't think of anything else I need/want.   Plus I have a lot of work I want to do on the SQLite Extender and, of course, finishing this one.   I plan for this to be a JSON, XML and HTML Extender.   JSON is the only thing currently available.    Thought XML would be first but having wonky problems on that front so switched to JSON.   

Mainly looking for suggestions and feedback on what I have done.  I normally convert JSON to XML in my scripts as there hasn't been a good JSON option for WinBatch so other than some Javascript usage I haven't done a lot with JSON.   I just incorporated this into a production version of an app I distribute and it worked a treat.   Much simpler than using the SQLite Extender and its JSON capabilities.  I am probably missing some needed functionality but having trouble thinking of what else to add before I switch back to the XML portion so thought I would ask any interested parties.    The Help is a bit rough but hopefully clear enough to give it a go.  Don't forget you may need to "Unblock" the help file.

You can only work with one Parsed document at a time.   I probably need to make it so more than one can be active but curious if that is needed.  Do you ever have more than one document you are actively working with at the same time?

Also, my focus on this Extender is the extraction of data and not a tool for creating/modifying DOM elements/documents.  Open to requests, though they will be lower on the list of priorities and no promises.

         http://www.jtdata.com/anonymous/DomData.zip

Jim

[stanl]

I have great test. https://api.weather.gov/alerts/active?area=NC - this was in a script I posted a couple of months ago where Kirby suggested a RegEx string to get at the Json return data. I attached the script if interested.


This looks like its going to be a great addition ;) ;) ;)

[jmburton2001]

Also, my focus on this Extender is the extraction of data...

I've been tinkering with XML data extraction on and off for the past few years with no real success. I'd be very interested in the XML functions when (if) you add them.

[JTaylor]

Thanks.

I have great test. https://api.weather.gov/alerts/active?area=NC - this was in a script I posted a couple of months ago where Kirby suggested a RegEx string to get at the Json return data. I attached the script if interested.


This looks like its going to be a great addition ;) ;) ;)

[stanl]

This is nice. Script below opens attached Json file based on response data weather url. This was a difficult Json to parse, even Powershell had a hard time when I tried, but your dmjGe tTreePath() was flawless. I had to save the path to a file then inspect to manually obtain specific paths to concatenate for /headlines I wished to display. Does it make sense to return the path as Tab-Delimited so specific elements could be searched/extracted with WB functions or a ForEach loop.

Code (WINBATCH) Select Expand



AddExtender("wbdomdata.dll")
cJson = Dirscript():"nc.json"
cFile = Dirscript():"path.txt"


dmParse(FileGet(cJson),@dmJSON)
path = dmjGetTreePath()


;create path list to test distinct values
;FilePut(cFile,path)




value = dmjGetValue("/features/0/properties/headline"):@LF:dmjGetValue("/features/1/properties/headline")
Message(cJson,value)


Exit
               


[stanl]

Attached is clumsy way to iterate and get values for 'headlines' in the Json. I initially get a Visual Studio Assertation which I can click ignore on. It also seems to get more than the value if explicitly looked for.

Code (WINBATCH) Select Expand



AddExtender("wbdomdata.dll")
cJson = Dirscript():"nc.json"
cFile = Dirscript():"path.txt"


dmParse(FileGet(cJson),@dmJSON)
path = StrReplace(dmjGetTreePath(),@LF,@TAB)


n= ItemLocateWild("*headline*",path,@TAB, 1)
value = ""
If n>0
   cVal = ItemExtract(n,path,@TAB)
   value = value:dmjGetValue('"':cVal:'"'):@LF
   Message(cVal,value)
   While 1
      n1 = ItemLocateWild("*headline*",path,@TAB, n+1)
      If n1>0
         cVal = ItemExtract(n1,path,@TAB)
         value = value:dmjGetValue('"':cVal:'"'):@LF
         Message(cVal,value)
         
         n=n1
      Else
         Break
      Endif
         
   EndWhile
Else
   Message(cJson,"No headline values found")
Endif
;create path list to test distinct values
;FilePut(cFile,path)
;value = dmjGetValue("/features/0/properties/headline"):@LF:dmjGetValue("/features/1/properties/headline")
Message(cJson,value)
Exit
               


[JTaylor]

Yes.   I realized there was a big gap after you posted the script yesterday and I went to convert it.  I thought I had things covered but realized I had missed something obvious.   Stay-tuned...

I will probably go with returning a Map compatible string unless you can make a case why TAB delimited would be better.   I keep trying to make a case for TAB delimited to myself but can't see why so happy to hear other thoughts.

Jim

[JTaylor]

Thanks.  This kind of stuff is helpful as it also points out my oversight.   

Jim

Attached is clumsy way to iterate and get values for 'headlines' in the Json. I initially get a Visual Studio Assertation which I can click ignore on. It also seems to get more than the value if explicitly looked for.

[JTaylor]

After reading this I realized it could be misconstrued...I am serious...Thank you.

Thanks.  This kind of stuff is helpful as it also points out my oversight.   

Jim

Attached is clumsy way to iterate and get values for 'headlines' in the Json. I initially get a Visual Studio Assertation which I can click ignore on. It also seems to get more than the value if explicitly looked for.

[JTaylor]

See if this solves the problem.

   http://www.jtdata.com/anonymous/DomData.zip

Code (winbatch) Select Expand


AddExtender(DirScript():"wbdomdata.dll")
cJson = Dirscript():"nc.json"
cFile = Dirscript():"path.txt"

dmParse(FileGet(cJson),@dmJSON)
;path = StrReplace(dmjGetTreePath(),@LF,@TAB)
value = ""

emap  = dmjGetElementMap("/features")
jmap  = MapCreate(emap,@TAB,@CR)

ForEach key in jmap
   value = value:dmjGetValue(jmap[key]:"/properties/headline"):@LF
Next

If value == "" Then
   Message(cJson,"No headline values found")
EndIf

;create path list to test distinct values
;FilePut(cFile,path)
;value = dmjGetValue("/features/0/properties/headline"):@LF:dmjGetValue("/features/1/properties/headline")
Message(cJson,value)
Exit
               


[JTaylor]

Also, just a general note/warning.  I am not settled on function names yet and also open to suggestions.

Jim

[stanl]

I will probably go with returning a Map compatible string unless you can make a case why TAB delimited would be better.

Not trying to make a case for anything. Your last post worked perfectly. What I was having problems with as although your dmjGetValue() function can accept jmap[key] as a variable even with a path I can hard-code and it works I either get blank returns or an error [attached]. Include snippet below in either your or my script, test for all three returns

Code (WINBATCH) Select Expand



dmParse(FileGet(cJson),@dmJSON)
path = StrReplace(dmjGetTreePath(),@LF,@TAB)


n= ItemLocateWild("*headline*",path,@TAB, 1)
cVal = ItemExtract(n,path,@TAB)
Message("",cVal)
;Message("",dmjGetValue("%cVal%")) ;returns blank
;Message("",dmjGetValue(cVal)) ;returns blank


var='"':cVal:'"'
Message("",var)
Message("",dmjGetValue(var)) ;assertation error


[td]

Looks like the DLL was compiled with MSFT's debug defined.

Anyway, I debated whether or not to mentions this but ILC has been specing out a JSON extender as well.  The high-level design parameters include robust, high-performance, and user-friendly with a small footprint.  Not sure that it will happen because it is not clear how needed it is yet but thought it should be mentioned.

This forum topic will be very useful for developing detailed specs if we should decide to go ahead with the project. 

[JTaylor]

Good to know and won't bother me.  I just need something practical to make myself learn and not having good JSON capabilities has been a real hassle over the years.   Finally learned enough I could do something about it.   

Having that Map option in the Extender would be really handy right now though :)

Jim

Looks like the DLL was compiled with MSFT's debug defined.

Anyway, I debated whether or not to mentions this but ILC has been specing out a JSON extender as well.  The high-level design parameters include robust, high-performance, and user-friendly with a small footprint.  Not sure that it will happen because it is not clear how needed it is yet but thought it should be mentioned.

This forum topic will be very useful for developing detailed specs if we should decide to go ahead with the project.

[JTaylor]

This was really more of a question.  That is,  Is there any reason to return a TAB delimited list over, or in addition to, the Map option?

I am not seeing the Assertion error.   Not compiling with DEBUG as far as I know.  The library I am using has Assertions everywhere and I have to trap everything.

Will try this script.

Jim

[/quote]


Not trying to make a case for anything. Your last post worked perfectly. What I was having problems with as although your dmjGetValue() function can accept jmap[key] as a variable even with a path I can hard-code and it works I either get blank returns or an error [attached]. Include snippet below in either your or my script, test for all three returns


[/quote]

[JTaylor]

Two issues.   

First, do not enclose the path in quotes,  unless you are submitting it directly of course (i.e. dmjGetValue("/my/0/path") ).  Simply use cVal  as it is Extracted.

Second, The TreePath has @CRLF on the end so there was a @CR left after ItemExtract().   Didn't think about someone using that as a delimited list but rather a reference.  I will change it to a @LF as that will keep the display but also provide a single delimiter.  I have noted this in the documentation.  I think if you fix these two things and account for the @CRLF until the next release the script should work.

Again, thank you.  This is very helpful.

Jim

I will probably go with returning a Map compatible string unless you can make a case why TAB delimited would be better.

Not trying to make a case for anything. Your last post worked perfectly. What I was having problems with as although your dmjGetValue() function can accept jmap[key] as a variable even with a path I can hard-code and it works I either get blank returns or an error [attached]. Include snippet below in either your or my script, test for all three returns

Code (WINBATCH) Select Expand



dmParse(FileGet(cJson),@dmJSON)
path = StrReplace(dmjGetTreePath(),@LF,@TAB)


n= ItemLocateWild("*headline*",path,@TAB, 1)
cVal = ItemExtract(n,path,@TAB)
Message("",cVal)
;Message("",dmjGetValue("%cVal%")) ;returns blank
;Message("",dmjGetValue(cVal)) ;returns blank


var='"':cVal:'"'
Message("",var)
Message("",dmjGetValue(var)) ;assertation error


[td]

I am not seeing the Assertion error.   Not compiling with DEBUG as far as I know.  The library I am using has Assertions everywhere and I have to trap everything.

Generally, assertions are the result of the code containing one or more either runtime library assert macros or a developer implementing their own version of the same.  Runtime assertions are defined in the <assert.h> header and are normally turned off for production source compiles by include a #define NDEBUG preprocessor statement or defining it on the compiler's command line in the scope of any code modules containing the runtime assert macro.

[JTaylor]

Understood.  I have had NDEBUG defined in the properties and compile under the "Release" profile but still get those.

Jim

[td]

It could be because some library you are linking into still has assertions enabled or your 3rd party source code is undefining the NDEBUG or has its own assertions that don't respect NDEFINE.  There are cases of MSFT leaving assertions active in production libraries. Sometimes this appears intentional and other times not so much.

[JTaylor]

Understood.   Probably something along those lines.  Upside, especially at this point, is it is catching things I would miss.   I think with this library it is intentionally set somewhere so one must trap everything appropriately, which is probably a good idea.

Jim

[stanl]

Guess I should make a goal of mine clear. If I already knew a lot about a Json tree I could hard-code the path in a function. I'm more interested in a generic parsing mechanism that can iterate the tree into elements or objects or lists [whichever is convenient] then select values from those and then write out correct parsing code. It is something related to work, which I call pre-pre-validation. Not easily explained, but take the weather alert URL - the ask is how to break it down into reasonable elements where accessing the values of the elements is the goal... code to write code... like going back to working with Clipper.

[JTaylor]

This includes the beginning of the XML stuff.  A lot to do yet but hoping on feedback so as to guide the rest of the work.   Be honest.  You are not going to hurt my feelings.  Telling me it is good when it is not, doesn't help any of us.   Does it work is, of course, important but it being intuitive and smooth is of equal importance.   Also, if there are double-quotes in the text you will encounter problems on Map functions.  That is, when trying to use the result in a MapCreate().   Just learned that is an issue.

Just to repeat, do not get married to function names.  Those are a work in progress.
   
   http://www.jtdata.com/anonymous/DomData.zip

Jim

[td]

Guess I should make a goal of mine clear. If I already knew a lot about a Json tree I could hard-code the path in a function. I'm more interested in a generic parsing mechanism that can iterate the tree into elements or objects or lists [whichever is convenient] then select values from those and then write out correct parsing code. It is something related to work, which I call pre-pre-validation. Not easily explained, but take the weather alert URL - the ask is how to break it down into reasonable elements where accessing the values of the elements is the goal... code to write code... like going back to working with Clipper.

By values are you referring to the left-hand name side of the pair? Or do you want to find a specific right-hand js typed side and get its "path"?

[td]

Understood.   Probably something along those lines.  Upside, especially at this point, is it is catching things I would miss.   I think with this library it is intentionally set somewhere so one must trap everything appropriately, which is probably a good idea.

Jim

Traditionally assertions should only be used during development to document assumptions made by the code's author and should never appear in production code. They are handy during development and testing when using another author's source or library. I am also of that view.

[JTaylor]

Agreed.

jim

[JTaylor]

After thinking about this a bit more, I think you can [sort of] do what you want already.   Again, you may want types and such added.

If you submit the following you will get a list of all the root nodes.  You could then sumbit with the desired node and get the next level for each of those nodes.  Am I headed in the right direction?

tree  = dmjGetElementMap("")


Jim

Guess I should make a goal of mine clear. If I already knew a lot about a Json tree I could hard-code the path in a function. I'm more interested in a generic parsing mechanism that can iterate the tree into elements or objects or lists [whichever is convenient] then select values from those and then write out correct parsing code. It is something related to work, which I call pre-pre-validation. Not easily explained, but take the weather alert URL - the ask is how to break it down into reasonable elements where accessing the values of the elements is the goal... code to write code... like going back to working with Clipper.

[JTaylor]

Thought I posted this earlier....

Keeping in mind I don't have a way to pass you "Objects" and using the TreePath as a model.  How would you like that to look and in what format?   Do you want element Types included?   

Jim

Guess I should make a goal of mine clear. If I already knew a lot about a Json tree I could hard-code the path in a function. I'm more interested in a generic parsing mechanism that can iterate the tree into elements or objects or lists [whichever is convenient] then select values from those and then write out correct parsing code. It is something related to work, which I call pre-pre-validation. Not easily explained, but take the weather alert URL - the ask is how to break it down into reasonable elements where accessing the values of the elements is the goal... code to write code... like going back to working with Clipper.

[stanl]

The weather alert Json I provided has multiple possibilities to test.

   
• a Json Polygon type for coordinates
• pairs -

"status": "Actual",
"messageType": "Update",
"category": "Met",
"severity": "Severe",
"certainty": "Observed",
"urgency": "Immediate",
"event": "Flood Warning",

   
• hyperlinks

Below is code to create the json output for any state you want to test:

Code (WINBATCH) Select Expand



;Winbatch CLR: save json output to file
IntControl(73,1,0,0,0)
ar = "NC" ;choose a different state to test outut
request = "https://api.weather.gov/alerts/active?area=%ar%"
cFile = Dirscript():ar:".json"
If FileExist(cFile) Then FileDelete(cFile)
ObjectClrOption("useany", "System")
ObjectClrOption("useany", "System.Net.Http")               
oClient = ObjectClrNew('System.Net.Http.HttpClient')
oTask = ObjectClrNew('System.Threading.Tasks.Task')
oResponse = ObjectClrNew('System.Net.Http.HttpResponseMessage')
oHdr = ObjectClrNew('System.Net.Http.Headers.MediaTypeWithQualityHeaderValue')
oClient.DefaultRequestHeaders.Add("User-Agent","Other")
oClient.DefaultRequestHeaders.Add("Accept","application/geo+json")
Response = oClient.GetAsync(request).Result
n=0
While n<=10
   If Response.IsSuccessStatusCode Then Break
   TimeDelay(1)
   n=n+1
   If n>9
      Pause("Giving Up","Unable To Create Request")
      goto theend
   Endif
EndWhile
TaskStr = Response.Content.ReadAsStringAsync()
html=TaskStr.Result
FilePut(cFile,html)


:theend
oTask = 0
oClient = 0
Exit


:WBERRORHANDLER
oTask = 0
oClient = 0
wberroradditionalinfo = wberrorarray[6]
lasterr = wberrorarray[0]
handlerline = wberrorarray[1]
textstring = wberrorarray[5]
linenumber = wberrorarray[8]
errmsg = "Error: ":lasterr:@LF:textstring:@LF:"Line (":linenumber:")":@LF:wberroradditionalinfo
Terminate(@TRUE,"Error Encountered",errmsg)




[JTaylor]

If this was a source of JSON to use for testing, thanks.  It is helpful.   

If it was an answer to my questions about what you were wanting to accomplish I am not understanding.

Jim

[stanl]

If this was a source of JSON to use for testing, thanks.  It is helpful.   
If it was an answer to my questions about what you were wanting to accomplish I am not understanding.
Jim

C'mon... you should know me better than that, ;D .

   
• I adjusted my script to replace @CRLF with @TAB instead of @LF - that fixed it.. sorta of... [something you can test]
• Right now there are no answers, maybe just bad questions.

Let me put things this way:  with the weather alert json I could manually examine it, parse it and comply with a request from a user to get the data [in Excel, text... whatever].


but the user says... I want a script that figures out what can be done manually so I can choose a result.


I personally think most Json implementations suck, but also preferred the Kinks over the Beatles :(


[EDIT]


here is a simple one to play with: https://jsonplaceholder.typicode.com/users

[JTaylor]

So what isn't available now that would be needed to accomplish what you want?   You can easily walk the tree path level by level or pull the entire tree.   

If you talk Tony into adding a TreeView control to the dialog suite it would probably be a great boon to what you want, assuming I understand.  I added one in my Extender but without Select event capability it didn't seem very useful.

Jim

[td]

I don't think you need a tree view control to view a JSON tree.  All you need is a browser and a URL to a Website. A "write my code for me" or "no code" option is another issue but I am not sure that is what is being requested either.

[JTaylor]

I know...but still think a TreeView would be nice but guessing you have stopped dialog work until you decide where you are headed on that front :) 

Yeah...I am still not sure either.   Happy to make it easier if I can but need more direction.   At the very least it makes me think about what I am doing in different ways which is always helpful.

Jim

I don't think you need a tree view control to view a JSON tree.  All you need is a browser and a URL to a Website. A "write my code for me" or "no code" option is another issue but I am not sure that is what is being requested either.

[JTaylor]

...although, regarding the tree it wasn't the viewing so much as the user being able to interact with the tree.

Jim

[td]

Obviously, but the interaction part could be accomplished without a pretty treeview control also.