This will probably be my last Extender. At least I can't think of anything else I need/want. Plus I have a lot of work I want to do on the SQLite Extender and, of course, finishing this one. I plan for this to be a JSON, XML and HTML Extender. JSON is the only thing currently available. Thought XML would be first but having wonky problems on that front so switched to JSON.
Mainly looking for suggestions and feedback on what I have done. I normally convert JSON to XML in my scripts as there hasn't been a good JSON option for WinBatch so other than some Javascript usage I haven't done a lot with JSON. I just incorporated this into a production version of an app I distribute and it worked a treat. Much simpler than using the SQLite Extender and its JSON capabilities. I am probably missing some needed functionality but having trouble thinking of what else to add before I switch back to the XML portion so thought I would ask any interested parties. The Help is a bit rough but hopefully clear enough to give it a go. Don't forget you may need to "Unblock" the help file.
You can only work with one Parsed document at a time. I probably need to make it so more than one can be active but curious if that is needed. Do you ever have more than one document you are actively working with at the same time?
Also, my focus on this Extender is the extraction of data and not a tool for creating/modifying DOM elements/documents. Open to requests, though they will be lower on the list of priorities and no promises.
http://www.jtdata.com/anonymous/DomData.zip
Jim
I have great test. https://api.weather.gov/alerts/active?area=NC - this was in a script I posted a couple of months ago where Kirby suggested a RegEx string to get at the Json return data. I attached the script if interested.
This looks like its going to be a great addition ;) ;) ;)
Quote from: JTaylor on January 18, 2021, 06:16:03 PM
Also, my focus on this Extender is the extraction of data...
I've been tinkering with XML data extraction on and off for the past few years with no real success. I'd be very interested in the XML functions when (if) you add them.
Thanks.
Quote from: stanl on January 19, 2021, 03:18:22 AM
I have great test. https://api.weather.gov/alerts/active?area=NC - this was in a script I posted a couple of months ago where Kirby suggested a RegEx string to get at the Json return data. I attached the script if interested.
This looks like its going to be a great addition ;) ;) ;)
This is nice. Script below opens attached Json file based on response data weather url. This was a difficult Json to parse, even Powershell had a hard time when I tried, but your dmjGe tTreePath() was flawless. I had to save the path to a file then inspect to manually obtain specific paths to concatenate for /headlines I wished to display. Does it make sense to return the path as Tab-Delimited so specific elements could be searched/extracted with WB functions or a ForEach loop.
AddExtender("wbdomdata.dll")
cJson = Dirscript():"nc.json"
cFile = Dirscript():"path.txt"
dmParse(FileGet(cJson),@dmJSON)
path = dmjGetTreePath()
;create path list to test distinct values
;FilePut(cFile,path)
value = dmjGetValue("/features/0/properties/headline"):@LF:dmjGetValue("/features/1/properties/headline")
Message(cJson,value)
Exit
Attached is clumsy way to iterate and get values for 'headlines' in the Json. I initially get a Visual Studio Assertation which I can click ignore on. It also seems to get more than the value if explicitly looked for.
AddExtender("wbdomdata.dll")
cJson = Dirscript():"nc.json"
cFile = Dirscript():"path.txt"
dmParse(FileGet(cJson),@dmJSON)
path = StrReplace(dmjGetTreePath(),@LF,@TAB)
n= ItemLocateWild("*headline*",path,@TAB, 1)
value = ""
If n>0
cVal = ItemExtract(n,path,@TAB)
value = value:dmjGetValue('"':cVal:'"'):@LF
Message(cVal,value)
While 1
n1 = ItemLocateWild("*headline*",path,@TAB, n+1)
If n1>0
cVal = ItemExtract(n1,path,@TAB)
value = value:dmjGetValue('"':cVal:'"'):@LF
Message(cVal,value)
n=n1
Else
Break
Endif
EndWhile
Else
Message(cJson,"No headline values found")
Endif
;create path list to test distinct values
;FilePut(cFile,path)
;value = dmjGetValue("/features/0/properties/headline"):@LF:dmjGetValue("/features/1/properties/headline")
Message(cJson,value)
Exit
Yes. I realized there was a big gap after you posted the script yesterday and I went to convert it. I thought I had things covered but realized I had missed something obvious. Stay-tuned...
I will probably go with returning a Map compatible string unless you can make a case why TAB delimited would be better. I keep trying to make a case for TAB delimited to myself but can't see why so happy to hear other thoughts.
Jim
Thanks. This kind of stuff is helpful as it also points out my oversight.
Jim
Quote from: stanl on January 20, 2021, 05:05:40 AM
Attached is clumsy way to iterate and get values for 'headlines' in the Json. I initially get a Visual Studio Assertation which I can click ignore on. It also seems to get more than the value if explicitly looked for.
After reading this I realized it could be misconstrued...I am serious...Thank you.
Quote from: JTaylor on January 20, 2021, 06:43:53 AM
Thanks. This kind of stuff is helpful as it also points out my oversight.
Jim
Quote from: stanl on January 20, 2021, 05:05:40 AM
Attached is clumsy way to iterate and get values for 'headlines' in the Json. I initially get a Visual Studio Assertation which I can click ignore on. It also seems to get more than the value if explicitly looked for.
See if this solves the problem.
http://www.jtdata.com/anonymous/DomData.zip
AddExtender(DirScript():"wbdomdata.dll")
cJson = Dirscript():"nc.json"
cFile = Dirscript():"path.txt"
dmParse(FileGet(cJson),@dmJSON)
;path = StrReplace(dmjGetTreePath(),@LF,@TAB)
value = ""
emap = dmjGetElementMap("/features")
jmap = MapCreate(emap,@TAB,@CR)
ForEach key in jmap
value = value:dmjGetValue(jmap[key]:"/properties/headline"):@LF
Next
If value == "" Then
Message(cJson,"No headline values found")
EndIf
;create path list to test distinct values
;FilePut(cFile,path)
;value = dmjGetValue("/features/0/properties/headline"):@LF:dmjGetValue("/features/1/properties/headline")
Message(cJson,value)
Exit
Also, just a general note/warning. I am not settled on function names yet and also open to suggestions.
Jim
Quote from: JTaylor on January 20, 2021, 06:42:19 AM
I will probably go with returning a Map compatible string unless you can make a case why TAB delimited would be better.
Not trying to make a case for anything. Your last post worked perfectly. What I was having problems with as although your dmjGetValue() function can accept jmap[key] as a variable even with a path I can hard-code and it works I either get blank returns or an error [attached]. Include snippet below in either your or my script, test for all three returns
dmParse(FileGet(cJson),@dmJSON)
path = StrReplace(dmjGetTreePath(),@LF,@TAB)
n= ItemLocateWild("*headline*",path,@TAB, 1)
cVal = ItemExtract(n,path,@TAB)
Message("",cVal)
;Message("",dmjGetValue("%cVal%")) ;returns blank
;Message("",dmjGetValue(cVal)) ;returns blank
var='"':cVal:'"'
Message("",var)
Message("",dmjGetValue(var)) ;assertation error
Looks like the DLL was compiled with MSFT's debug defined.
Anyway, I debated whether or not to mentions this but ILC has been specing out a JSON extender as well. The high-level design parameters include robust, high-performance, and user-friendly with a small footprint. Not sure that it will happen because it is not clear how needed it is yet but thought it should be mentioned.
This forum topic will be very useful for developing detailed specs if we should decide to go ahead with the project.
Good to know and won't bother me. I just need something practical to make myself learn and not having good JSON capabilities has been a real hassle over the years. Finally learned enough I could do something about it.
Having that Map option in the Extender would be really handy right now though :)
Jim
Quote from: td on January 20, 2021, 10:20:39 AM
Looks like the DLL was compiled with MSFT's debug defined.
Anyway, I debated whether or not to mentions this but ILC has been specing out a JSON extender as well. The high-level design parameters include robust, high-performance, and user-friendly with a small footprint. Not sure that it will happen because it is not clear how needed it is yet but thought it should be mentioned.
This forum topic will be very useful for developing detailed specs if we should decide to go ahead with the project.
This was really more of a question. That is, Is there any reason to return a TAB delimited list over, or in addition to, the Map option?
I am not seeing the Assertion error. Not compiling with DEBUG as far as I know. The library I am using has Assertions everywhere and I have to trap everything.
Will try this script.
Jim
[/quote]
Not trying to make a case for anything. Your last post worked perfectly. What I was having problems with as although your dmjGetValue() function can accept jmap[key] as a variable even with a path I can hard-code and it works I either get blank returns or an error [attached]. Include snippet below in either your or my script, test for all three returns
[/quote]
Two issues.
First, do not enclose the path in quotes, unless you are submitting it directly of course (i.e. dmjGetValue("/my/0/path") ). Simply use cVal as it is Extracted.
Second, The TreePath has @CRLF on the end so there was a @CR left after ItemExtract(). Didn't think about someone using that as a delimited list but rather a reference. I will change it to a @LF as that will keep the display but also provide a single delimiter. I have noted this in the documentation. I think if you fix these two things and account for the @CRLF until the next release the script should work.
Again, thank you. This is very helpful.
Jim
Quote from: stanl on January 20, 2021, 09:50:58 AM
Quote from: JTaylor on January 20, 2021, 06:42:19 AM
I will probably go with returning a Map compatible string unless you can make a case why TAB delimited would be better.
Not trying to make a case for anything. Your last post worked perfectly. What I was having problems with as although your dmjGetValue() function can accept jmap[key] as a variable even with a path I can hard-code and it works I either get blank returns or an error [attached]. Include snippet below in either your or my script, test for all three returns
dmParse(FileGet(cJson),@dmJSON)
path = StrReplace(dmjGetTreePath(),@LF,@TAB)
n= ItemLocateWild("*headline*",path,@TAB, 1)
cVal = ItemExtract(n,path,@TAB)
Message("",cVal)
;Message("",dmjGetValue("%cVal%")) ;returns blank
;Message("",dmjGetValue(cVal)) ;returns blank
var='"':cVal:'"'
Message("",var)
Message("",dmjGetValue(var)) ;assertation error
Quote from: JTaylor on January 20, 2021, 11:19:45 AM
I am not seeing the Assertion error. Not compiling with DEBUG as far as I know. The library I am using has Assertions everywhere and I have to trap everything.
Generally, assertions are the result of the code containing one or more either runtime library assert macros or a developer implementing their own version of the same. Runtime assertions are defined in the <assert.h> header and are normally turned off for production source compiles by include a #define NDEBUG preprocessor statement or defining it on the compiler's command line in the scope of any code modules containing the runtime assert macro.
Understood. I have had NDEBUG defined in the properties and compile under the "Release" profile but still get those.
Jim
It could be because some library you are linking into still has assertions enabled or your 3rd party source code is undefining the NDEBUG or has its own assertions that don't respect NDEFINE. There are cases of MSFT leaving assertions active in production libraries. Sometimes this appears intentional and other times not so much.
Understood. Probably something along those lines. Upside, especially at this point, is it is catching things I would miss. I think with this library it is intentionally set somewhere so one must trap everything appropriately, which is probably a good idea.
Jim
Guess I should make a goal of mine clear. If I already knew a lot about a Json tree I could hard-code the path in a function. I'm more interested in a generic parsing mechanism that can iterate the tree into elements or objects or lists [whichever is convenient] then select values from those and then write out correct parsing code. It is something related to work, which I call pre-pre-validation. Not easily explained, but take the weather alert URL - the ask is how to break it down into reasonable elements where accessing the values of the elements is the goal... code to write code... like going back to working with Clipper.
This includes the beginning of the XML stuff. A lot to do yet but hoping on feedback so as to guide the rest of the work. Be honest. You are not going to hurt my feelings. Telling me it is good when it is not, doesn't help any of us. Does it work is, of course, important but it being intuitive and smooth is of equal importance. Also, if there are double-quotes in the text you will encounter problems on Map functions. That is, when trying to use the result in a MapCreate(). Just learned that is an issue.
Just to repeat, do not get married to function names. Those are a work in progress.
http://www.jtdata.com/anonymous/DomData.zip
Jim
Quote from: stanl on January 20, 2021, 03:01:46 PM
Guess I should make a goal of mine clear. If I already knew a lot about a Json tree I could hard-code the path in a function. I'm more interested in a generic parsing mechanism that can iterate the tree into elements or objects or lists [whichever is convenient] then select values from those and then write out correct parsing code. It is something related to work, which I call pre-pre-validation. Not easily explained, but take the weather alert URL - the ask is how to break it down into reasonable elements where accessing the values of the elements is the goal... code to write code... like going back to working with Clipper.
By values are you referring to the left-hand name side of the pair? Or do you want to find a specific right-hand js typed side and get its "path"?
Quote from: JTaylor on January 20, 2021, 02:32:04 PM
Understood. Probably something along those lines. Upside, especially at this point, is it is catching things I would miss. I think with this library it is intentionally set somewhere so one must trap everything appropriately, which is probably a good idea.
Jim
Traditionally assertions should only be used during development to document assumptions made by the code's author and should never appear in production code. They are handy during development and testing when using another author's source or library. I am also of that view.
Agreed.
jim
After thinking about this a bit more, I think you can [sort of] do what you want already. Again, you may want types and such added.
If you submit the following you will get a list of all the root nodes. You could then sumbit with the desired node and get the next level for each of those nodes. Am I headed in the right direction?
tree = dmjGetElementMap("")
Jim
Quote from: stanl on January 20, 2021, 03:01:46 PM
Guess I should make a goal of mine clear. If I already knew a lot about a Json tree I could hard-code the path in a function. I'm more interested in a generic parsing mechanism that can iterate the tree into elements or objects or lists [whichever is convenient] then select values from those and then write out correct parsing code. It is something related to work, which I call pre-pre-validation. Not easily explained, but take the weather alert URL - the ask is how to break it down into reasonable elements where accessing the values of the elements is the goal... code to write code... like going back to working with Clipper.
Thought I posted this earlier....
Keeping in mind I don't have a way to pass you "Objects" and using the TreePath as a model. How would you like that to look and in what format? Do you want element Types included?
Jim
Quote from: stanl on January 20, 2021, 03:01:46 PM
Guess I should make a goal of mine clear. If I already knew a lot about a Json tree I could hard-code the path in a function. I'm more interested in a generic parsing mechanism that can iterate the tree into elements or objects or lists [whichever is convenient] then select values from those and then write out correct parsing code. It is something related to work, which I call pre-pre-validation. Not easily explained, but take the weather alert URL - the ask is how to break it down into reasonable elements where accessing the values of the elements is the goal... code to write code... like going back to working with Clipper.
The weather alert Json I provided has multiple possibilities to test.
- a Json Polygon type for coordinates
- pairs -
"status": "Actual",
"messageType": "Update",
"category": "Met",
"severity": "Severe",
"certainty": "Observed",
"urgency": "Immediate",
"event": "Flood Warning",
Below is code to create the json output for any state you want to test:
;Winbatch CLR: save json output to file
IntControl(73,1,0,0,0)
ar = "NC" ;choose a different state to test outut
request = "https://api.weather.gov/alerts/active?area=%ar%"
cFile = Dirscript():ar:".json"
If FileExist(cFile) Then FileDelete(cFile)
ObjectClrOption("useany", "System")
ObjectClrOption("useany", "System.Net.Http")
oClient = ObjectClrNew('System.Net.Http.HttpClient')
oTask = ObjectClrNew('System.Threading.Tasks.Task')
oResponse = ObjectClrNew('System.Net.Http.HttpResponseMessage')
oHdr = ObjectClrNew('System.Net.Http.Headers.MediaTypeWithQualityHeaderValue')
oClient.DefaultRequestHeaders.Add("User-Agent","Other")
oClient.DefaultRequestHeaders.Add("Accept","application/geo+json")
Response = oClient.GetAsync(request).Result
n=0
While n<=10
If Response.IsSuccessStatusCode Then Break
TimeDelay(1)
n=n+1
If n>9
Pause("Giving Up","Unable To Create Request")
goto theend
Endif
EndWhile
TaskStr = Response.Content.ReadAsStringAsync()
html=TaskStr.Result
FilePut(cFile,html)
:theend
oTask = 0
oClient = 0
Exit
:WBERRORHANDLER
oTask = 0
oClient = 0
wberroradditionalinfo = wberrorarray[6]
lasterr = wberrorarray[0]
handlerline = wberrorarray[1]
textstring = wberrorarray[5]
linenumber = wberrorarray[8]
errmsg = "Error: ":lasterr:@LF:textstring:@LF:"Line (":linenumber:")":@LF:wberroradditionalinfo
Terminate(@TRUE,"Error Encountered",errmsg)
If this was a source of JSON to use for testing, thanks. It is helpful.
If it was an answer to my questions about what you were wanting to accomplish I am not understanding.
Jim
Quote from: JTaylor on January 21, 2021, 05:55:19 AM
If this was a source of JSON to use for testing, thanks. It is helpful.
If it was an answer to my questions about what you were wanting to accomplish I am not understanding.
Jim
C'mon... you should know me better than that, ;D .
- I adjusted my script to replace @CRLF with @TAB instead of @LF - that fixed it.. sorta of... [something you can test]
- Right now there are no answers, maybe just bad questions.
Let me put things this way: with the weather alert json I could manually examine it, parse it and comply with a request from a user to get the data [in Excel, text... whatever].
but the user says... I want a script that figures out what can be done manually so I can choose a result.
I personally think most Json implementations suck, but also preferred the Kinks over the Beatles :(
[EDIT]
here is a simple one to play with: https://jsonplaceholder.typicode.com/users
So what isn't available now that would be needed to accomplish what you want? You can easily walk the tree path level by level or pull the entire tree.
If you talk Tony into adding a TreeView control to the dialog suite it would probably be a great boon to what you want, assuming I understand. I added one in my Extender but without Select event capability it didn't seem very useful.
Jim
I don't think you need a tree view control to view a JSON tree. All you need is a browser and a URL to a Website. A "write my code for me" or "no code" option is another issue but I am not sure that is what is being requested either.
I know...but still think a TreeView would be nice but guessing you have stopped dialog work until you decide where you are headed on that front :)
Yeah...I am still not sure either. Happy to make it easier if I can but need more direction. At the very least it makes me think about what I am doing in different ways which is always helpful.
Jim
Quote from: td on January 21, 2021, 07:57:32 AM
I don't think you need a tree view control to view a JSON tree. All you need is a browser and a URL to a Website. A "write my code for me" or "no code" option is another issue but I am not sure that is what is being requested either.
...although, regarding the tree it wasn't the viewing so much as the user being able to interact with the tree.
Jim
Obviously, but the interaction part could be accomplished without a pretty treeview control also.
Not disagreeing. Always more than one way to skin a cat.
Jim
Hope my cat didn't read this...
Might make him think twice before misbehaving.
Quote from: td on January 21, 2021, 09:25:49 AM
Hope my cat didn't read this...
Here is the latest. Added more to XML stuff as well as fix/tweak here and there. Can now extract attributes as well as an Array and CSV output from 2 levels of XML. Now off to use it in a project and see what I have missed.
http://www.jtdata.com/anonymous/DomData.zip
Jim
After attempting to use the XML stuff I have renamed most all the XML functions. Also, I don't think I like the way this library does things so need to do some better "wrapping" as I don't really like much of what I have done so far. I think I stuck too closely to the library way of doing it. Wish the previous one had worked but could never figure out why it would work one time and then immediately get gibberish the next run.
http://www.jtdata.com/anonymous/DomData.zip
Jim
Things are moving along well here. As for a treeview object I kinda agree with Tony that it is not necessary. Unless I missed it, some sort of count() function would be helpful. Using the Json file:
tree = dmjGetElementMap("/features/0") creates a tab-delimted variable that can become a WB map. But I'd like a way to know there are 4 elements. But, it could be a dumb question.
Thought about adding that, and intended to put something in the Help, but since you can do:
cnt = ArrInfo(treemap,1)
I decided not to bother. I have added that to the help file so thanks for the reminder.
Jim
Glad to hear it. Let me know if there is something I can do to make what you mentioned easier. Still not sure what more you need so just let me know, with pictures :)
Jim
Quote from: stanl on January 22, 2021, 03:23:01 AM
Things are moving along well here. As for a treeview object I kinda agree with Tony that it is not necessary. Unless I missed it, some sort of count() function would be helpful. Using the Json file:
tree = dmjGetElementMap("/features/0") creates a tab-delimted variable that can become a WB map. But I'd like a way to know there are 4 elements. But, it could be a dumb question.
Maybe I missed a version...but
AddExtender("wbdomdata.dll")
cJson = Dirscript():"nc.json"
cFile = Dirscript():"tree.txt"
dmParse(FileGet(cJson),@dmJSON)
tree = dmjGetElementMap("/features/0")
cnt = ArrInfo(tree,1)
will error that tree is not an array. Or I would have figured it out.
You have to create the Map. I am told that there are plans to make Maps available as part of the Extender SDK but not sure of the timeline on that option. Once that occurs I can send back a Map and the extra step won't be necessary.
jim
tree = MapCreate(dmjGetElementMap("/features/0"),@TAB,@CR)
Quote from: JTaylor on January 22, 2021, 03:16:38 PM
You have to create the Map.
Yep. I had my RFM moment. Stepping through the tree is just a matter of developing a recursion structure as well as a structure for output. Your Extender provides all that is needed - the iteration is a matter of style or a teachable moment ::)
more clumsy code below, just going to continue plugging things in to see what shakes out: Eventually see multiple threads for Json, XML, HTML....
AddExtender("wbdomdata.dll")
cJson = Dirscript():"nc.json"
dmParse(FileGet(cJson),@dmJSON)
;choose element to test
element = "/features/0" ;will return sub-tree
;stepping through sub-tree will get to single element/item pair
;element = "/features/0/id" ;will return selected element as item
tree = MapCreate(dmjGetElementMap(element),@TAB,@CR)
cnt = ArrInfo(tree,1)
If cnt==0
fld = ItemExtract(ItemCount(element,"/"),element,"/")
Message(element,fld:@TAB:dmjGetValue(element))
Else
For i = 0 to cnt-1
Message("Item:":i+1,tree[i])
Next
Endif
Exit
I am sure you know this but on the outside chance you don't and for the sake of thoroughness you can also do the following and avoid the counters.
Foreach leaf in tree
Message("Item: ", tree[leaf])
Next
Quote from: JTaylor on January 23, 2021, 06:01:06 AM
I am sure you know this but on the outside chance you don't and for the sake of thoroughness you can also do the following and avoid the counters.
Foreach leaf in tree
Message("Item: ", tree[leaf])
Next
Yes, I know that. Having a count is more important, at the moment...