**************** New JSON Extender (Experimental) * Part Deux ***********

[td]

Maybe this should be the embellished version?

Version 44003  Sep 17, 2021 (Experimental release.)

      Modified the jsKeyPaths function to return
      a single dimension array instead of an item
      list.
     
      Improved performance of the jsValueGet and
      jsValueType functions.

The performance improvements are significant but this version also breaks any script using jsKeyPath because the function now returns an array instead of an item list. It is experimental.

The documentation has been gussy up some as well.

It is available at the usual place:

https://files.winbatch.com/downloads/wb/ilcjs44i.zip

[stanl]

Maybe this should be the embellished version?

A lot faster.... I adjusted my script for array rather than list; eliminated the map lookup; modified the Boxtext to not display each node;only wrote out the node and value... tested on first 10,000 nodes. There is one test line where I thought the ObjectType for the jsKeyPaths would return ARRAY, but might be using the function wrong.


[EDIT]: my bad, use VarType() not ObjectTypeGet()... the use of "Object" in the Extender docs confused me.






Anyway, this is an excellent Extender and folks.... Json ain't going away :D

Code (WINBATCH) Select Expand



;Winbatch 2021C - Census Data - tests large json return
;uses  WinHttp.WinHttpRequest.5.1 for JSON return
;Stan Littlefield, September 14, 2021
;Updated: September 18, 2021 - based on new Extender using array rather than delimited list
;======================================================================================================
IntControl(73,1,0,0,0)
Gosub udfs
AddExtender('ilcjs44i.dll', 0, 'ilcjs64i.dll')
cUrl = "https://api.census.gov/data/2019/acs/acs1/variables.json"
BoxOpen("Parsing:":cUrl,"Please Wait...")
cFile = Dirscript():"CensusAPI.txt"
cJson = Dirscript():"CensusAPI.json"
If FileExist(cFile) Then FileDelete(cFile)
If FileExist(cJson) Then FileDelete(cJson)       
request = Createobject("WinHttp.WinHttpRequest.5.1")
request.Open("GET", cUrl, @False )
request.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
request.SetRequestHeader("Accept", "application/json")
request.Send()
jdata = request.ResponseText
request = 0


;persist .json file
;FilePut(cJson,jdata) ;uncomment if needed
;=================================
If jsValid(jdata)
   ;jstypes = crtypes()  ;no need for map, not persisting node types
   Object = jsParse(jdata)
   BoxTitle("Creating text Output ":cFile )
   crTree()
   jsObjClose()
   BoxShut()
   Message("Json Tree Created",cFile)
Else 
   jsObjClose()
   BoxShut()
   Message(cJson,"Not Valid Json")
Endif


Exit


:WBERRORHANDLER
geterror()
Terminate(@TRUE,"Error Encountered",errmsg)
Exit


:udfs
#DefineSubRoutine geterror()
   wberroradditionalinfo = wberrorarray[6]
   lasterr = wberrorarray[0]
   handlerline = wberrorarray[1]
   textstring = wberrorarray[5]
   linenumber = wberrorarray[8]
   errmsg = "Error: ":lasterr:@LF:textstring:@LF:"Line (":linenumber:")":@LF:wberroradditionalinfo
   Return(errmsg)
#EndSubRoutine


#DefineSubRoutine crTree()
   Fx = FileOpen(cFile,"Write")
   output = "node":@tab:"value"
   FileWrite(Fx,output)
   tree =jsKeyPaths(Object,"")


   ;just for test
   Message("Object Type for tree",ObjectTypeGet(tree)) ;would not expect blank return




   Cnt = Arrinfo(tree,1)
   i=1
   ForEach item in tree
      iVal = jsValueGet(Object,item)
      If jsValueType(Object,item) <> @JsonObj
         output = item:@tab:iVal
         If (i mod 100 == 0) Then BoxText("Processing ":i:" of ":Cnt)
         FileWrite(Fx,output)
         i +=1
         If i>10000 Then Break
      Endif
   Next
   FileClose(Fx)
   Drop(Fx)
   Return(1)
#EndSubRoutine


#DefineSubRoutine crtypes()
maps = $"1,NULL
2,BOOL
4,NUMBER
8,STRING
16,ARRAY
32,OBJECT
$"
jstypes = MapCreate(maps,',',@lf)
Return(jstypes)
#EndSubRoutine


Return


[td]

Thanks for taking the time to check the latest version. If you place the  "iVal = jsValueGet(Object,item)" inside the "if" block under the line "If jsValueType(Object,item) <> @JsonObj", your script will execute even faster and use less memory. You don't use the value for JSON objects so no reason to waste CPU cycles obtaining them. JSON objects are the most expensive JSON type the jsValueGet function processes.

For the curious, the term "object" is a general computer science concept and as such is used in many different contexts. In this case, the term is part of the formal description of JSON.

https://datatracker.ietf.org/doc/html/rfc7159

[bottomleypotts]

Well I'm having trouble (see attached). Would appreciate any help people can offer. Would also be nice to be able to see the paths variable when debugging.

[bottomleypotts]

Thanks for the reply Jim. The JSON is valid. I can use javascript to parse it.

[stanl]

Well I'm having trouble (see attached). Would appreciate any help people can offer. Would also be nice to be able to see the paths variable when debugging.

First, you have a variable error in your code jsValueGet(Object, path) should be jsValueGet(obj, path)



I tried paths = jsKeyPaths(obj,"") instead of paths = jsKeyPaths(obj,"id")... didn't help much


Bur, see what happens with prefacing the array and using obj,""

Code Select Expand



{ "items":
[
   {
      "id": "F2EVUSJVLDUCI",
      "generated": "2021-09-19T14:45:46.000Z"
   },
   {
      "id": "PFEPH53XKWDHQ",
      "generated": "2021-09-19T14:45:46.000Z"
   },
   {
      "id": "YLCDX6BKM5FB6",
      "generated": "2021-09-19T14:45:45.000Z"
   },
   {
      "id": "6OH7YI5GVDNI4",
      "generated": "2021-09-19T14:45:45.000Z"
   },
   {
      "id": "SOEQCKWWQ2UNQ",
      "generated": "2021-09-19T14:45:45.000Z"
   },
   {
      "id": "GUIUJV2HXCT6W",
      "generated": "2021-09-19T14:45:45.000Z"
   },
   {
      "id": "DKYFE4GOFF73A",
      "generated": "2021-09-19T14:45:44.000Z"
   },
   {
      "id": "CGG6B2GY5PH2M",
      "generated": "2021-09-19T14:45:44.000Z"
   },
   {
      "id": "5IVIANXJBG7FA",
      "generated": "2021-09-19T14:45:44.000Z"
   },
   {
      "id": "YKO537IUD2QKE",
      "generated": "2021-09-19T14:45:44.000Z"
   },
   {
      "id": "7MG2VEF5TNSDI",
      "generated": "2021-09-19T14:45:44.000Z"
   },
   {
      "id": "NUTFET4Y63D4I",
      "generated": "2021-09-19T14:45:44.000Z"
   },
   {
      "id": "PYGUPZV3KSQG6",
      "generated": "2021-09-19T14:45:44.000Z"
   },
   {
      "id": "K7YOX3V7PCL62",
      "generated": "2021-09-18T23:46:22.000Z"
   },
   {
      "id": "ST2G5JLV3TLRO",
      "generated": "2021-09-18T23:46:22.000Z"
   }
]}



Then try again with jsKeyPaths(obj,"items").... the extender will kick out @jsonobj references and the numbers change each iterations. Main reason why I put  If jsValueType(Object,item) <> @JsonObj in my code. Initially I thought if I referenced any of those I would get a path, i.e. a sub-array.... 


[EDIT] .... Oh, and use displaythis:=path:@TAB:jsValueGet(obj, path):@CR for a better view of data


Good to see more users playing with Json.

[td]

There is definitely a problem with the jsParse function and how it handles nameless array containers so it will require modification.

As I have mentioned before, constructive user feedback is invaluable.

[stanl]

Odd....I didn't have the "items" preceding the data.  Must have been why nothing I tried worked.

Jim

No, I added it to make a point.

   
• the Op had valid Json
• jskeypaths() seemed to fail
• they would work with added "items" and jsKeyPaths(obj,"") but not with jsKeyPaths(obj,"items")
• The Extender @jsonObj needs more explanation...

and, of course, this leads to more semanatic arguments over the term "Object".  But, let's assume the OP might ultimately want the Json parsed as


item,id,generated => basic .csv output


so something like

Code Select Expand


output=""
ForEach i in sKeyPaths(obj,"items")
   output:=i.id:",":i.generated:@CR
Next



would, in my opinion, be adequate Json parsing.

[JTaylor]

Glad I diagnosed that one correctly.   :)     Removed my other posts before I saw this one as I thought I messed up and didn't want to confuse things.

Jim

There is definitely a problem with the jsParse function and how it handles nameless array containers so it will require modification.

As I have mentioned before, constructive user feedback is invaluable.

[stanl]

There is definitely a problem with the jsParse function and how it handles nameless array containers so it will require modification.

As I have mentioned before, constructive user feedback is invaluable.

sorry, Tony... I was typing while you were typing. :-\ :-\ :-\

[stanl]

Glad I diagnosed that one correctly. 

Jim;


although I didn't diagnose anything I did bring up that jsparse() returns a vartype() array, not an ObjectTypeGet() array... which is where we digress to semantic quibbling.  Of course any nuances can be handled internally but any additional code would be specific to certain Json formats.

[JTaylor]

...and it just dawned on me that this issue is why all my attempts at using the Extender keep failing in my current project.  Been beating my head against a wall thinking I was doing something wrong  :)

Jim

[JTaylor]

Yeah...JSON seems to be a topic where I see many variations in the semantics.   I spent quite a bit of time trying to decide what terms to use in my Extender.   Even googling for the "correct" terminology brought up differences.

Jim

Glad I diagnosed that one correctly. 

Jim;


although I didn't diagnose anything I did bring up that jsparse() returns a vartype() array, not an ObjectTypeGet() array... which is where we digress to semantic quibbling.  Of course any nuances can be handled internally but any additional code would be specific to certain Json formats.

[stanl]

Here is another interesting API to play with: https://www.colourlovers.com/api/colors/top?format=json&numResults=100