Anyone want to help with JSON

[stanl]

https://api.weather.gov/alerts/active?area=NC    [substitute any state]


returns some interesting JSON. While I will probably make a deeper dive with PS, would be interested in any advanced WB parsing hints.  Mostly interest in sections like:


"headline": "Flood Advisory issued September 25 at 10:46AM EDT until September 25 at 11:45AM EDT by NWS Raleigh NC", "description": "The National Weather Service in Raleigh has issued a\n\n* Flood Advisory for Poor Drainage Areas for...\nAnson County in central North Carolina...\nNorthwestern Cumberland County in central North Carolina...\nHarnett County in central North Carolina...\nHoke County in central North Carolina...\nLee County in central North Carolina...\nSoutheastern Montgomery County in central North Carolina...\nMoore County in central North Carolina...\nRichmond County in central North Carolina...\nScotland County in central North Carolina...\n\n* Until 1145 AM EDT.\n\n* At 1046 AM EDT, Doppler radar indicated heavy rain. Overflowing\npoor drainage areas will cause minor flooding in the advisory area.\n\nSome locations that will experience flooding include...\nFayetteville, Sanford, Laurinburg, Southern Pines, Rockingham,\nWadesboro, Raeford, Lillington, Carthage, Fort Bragg, Pinehurst,\nDunn, Hamlet, Aberdeen, Angier, Hoffman, Antioch, Pope AFB, Spring\nLake and Erwin.",

[ChuckC]

The Newtonsoft JSON NuGet package is what I use in my c# projects for heavy duty JSON parsing.

With .NET Standard & .NET Core, there is a new built-in JSON parser but is is more limited.

PowerShell has some nice support for serializing & de-serializing JSON text, either between objects and strings or between objects and string content in a file.

Obviously, all of the methods mentioned above are accessible within WIL via the CLR support.

[JTaylor]

I am usually lazy and use C# to convert to XML. :)    If I get this current project I am on done and this thread is still active I might jump in.

Jim

[kdmoyers]

The squeamish may want to avert their eyes: ugly hacks ahead!

Code (winbatch) Select Expand

  curl = "https://api.weather.gov/alerts/active?area=NC"
  text = fileget(shortcutdir("desktop"):"\weatherdata.json")

  ; array of matches
  arr = arrdimension(99,3)

  re = '"headline": "([^"]+)",\r\n *"description": "(.+?)"'

  ; fill 2d array with match results
  objRE = ObjectOpen("VBScript.RegExp")
  objRE.Pattern    = re
  objRE.IgnoreCase = @true ; canse sensitivity
  objRE.MultiLine  = @true ; multiline mode
  objRE.Global     = @true ; match more than once
  matches = objRE.execute(text)
  arr[0,0] = 0 + matches.count
  if matches.count
    for i = 0 to matches.count-1
      arr[i+1,0] = matches.item(i).value                  ; full match...
      for j = 0 to matches.item(i).submatches.count-1
        arr[i+1,j+1] = matches.item(i).submatches.item(j) ; ...and each submatch
      next j
    next i
  endif
  objRE = 0

  ; display contents of array
  if arr[0,0]
    for ii = 1 to arr[0,0]
      message(arr[ii,1], strreplace(arr[ii,2],"\n",@lf)  )
    next ii
  else
    message("no match",re)
  endif

  EXIT

[stanl]

I LOVE the Regex approach. Think I'll try a CLR : ObjectClrNew('System.Net.WebRequest') along with ObjectClrNew('System.Text.RegularExpressions.Regex',cPattern).... just for another ugly hack.


What is interesting is I ran the URL this morning and NC had flash flood alerts, but running it now the JSON has no headlines. However, try CA as the state. Again, probably why the Regex is cool.

[kdmoyers]

Here's what I get for a CLR based RegEx solution.  I like it, thanks for the pointer!

Code (winbatch) Select Expand

  curl = "https://api.weather.gov/alerts/active?area=NC"
  text = fileget(shortcutdir("desktop"):"\weatherdata.json")
  regexpr = '"headline": "([^"]+)",\r\n *"description": "(.+?)"'

  REX = "System.Text.RegularExpressions"
  ObjectClrOption('useany', REX)
  R = ObjectClrNew(REX:'.RegexOptions')
  R = R.IgnoreCase | R.Multiline | R.Singleline
  ReOx = ObjectClrType(REX:'.RegexOptions',R)
  RE = ObjectClrNew(REX:'.Regex', regexpr, ReOx)

  matches = RE.Matches(text)
  message('count',matches.Count)

  foreach match in matches
      groups = match.Groups

      t1 = groups.Item(1).Captures.Item(0).Value
      t2 = groups.Item(2).Captures.Item(0).Value
       
      message(t1,t2)     

  next

  EXIT

[stanl]

Nice Kirby;


Now my 'je ne sais quoi'. You can open the link in a browser and it feeds back Json quickly. Tried the CLR route with the code below as well as with System.Net.Http.HttpClient and in both cases the response is access is forbidden.

Code (WINBATCH) Select Expand



strUrl = "https://api.weather.gov/alerts/active?area=NC"
ObjectClrOption('useany', 'System')
objWebUtil = ObjectClrNew('System.Net.WebUtility')


objUri = ObjectClrNew('System.Uri', strUrl)
objSvcManager = ObjectClrNew('System.Net.ServicePointManager')
protocols = ObjectClrType("System.Net.SecurityProtocolType",3072|768) 
objSvcManager.SecurityProtocol = protocols
objSvcPoint = objSvcManager.FindServicePoint(objUri)


objWebRequest = ObjectClrNew('System.Net.WebRequest')
objWebRequest = objWebRequest.Create(objUri)
objWebRequest.Timeout = objWebRequest.Timeout * 6
objResponse = objWebRequest.GetResponse()
Message("",objResponse) ; 403 forbidden error
exit


[stanl]

I think adding headers to the request will avoid the forbidden error. Little confused on headers for WebRequest class - assume it would need "application/json" or something like that.


but it is so easy with Powershell

Code Select Expand



[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$j = Invoke-WebRequest 'https://api.weather.gov/alerts/active?area=NC'  | ConvertFrom-Json | ConvertTo-Json
$j | Out-File "c:\temp\NC.json"


[stanl]

And below is my shot with HTTPClient.  According to the MSFT docs I read the System.Net.Http.HttpClientHandler class should be able to set SSL Protocols, but the class wasn't even recognized. The script will either give a timeout message after 10 seconds or show a forbidden response.  Otherwise, there is a lot of interesting Json and the Powershell code only gets part of it.

Code (WINBATCH) Select Expand



IntControl(73,1,0,0,0)
request = "https://api.weather.gov/alerts/active?area=NC"


ObjectClrOption("useany", "System")
ObjectClrOption("useany", "System.Net.Http")               


oClient = ObjectClrNew('System.Net.Http.HttpClient')
oTask = ObjectClrNew('System.Threading.Tasks.Task')
oResponse = ObjectClrNew('System.Net.Http.HttpResponseMessage')


;trying to set security protocol but fails
;oHandler = ObjectClrNew('System.Net.Http.HttpClientHandler')
;oHandler.SslProtocols = 3072


Response = oClient.GetAsync(request).Result
n=0
While n<=10
   If Response.IsSuccessStatusCode Then Break
   TimeDelay(1)
   n=n+1
   If n>9
      Pause("Giving Up","Unable To Create Request")
      goto theend
   Endif
EndWhile
TaskStr = Response.Content.ReadAsStringAsync()
html=TaskStr.Result
Pause("response",html)


:theend
oTask = 0
oClient = 0
Exit


:WBERRORHANDLER
oTask = 0
oClient = 0
wberroradditionalinfo = wberrorarray[6]
lasterr = wberrorarray[0]
handlerline = wberrorarray[1]
textstring = wberrorarray[5]
linenumber = wberrorarray[8]
errmsg = "Error: ":lasterr:@LF:textstring:@LF:"Line (":linenumber:")":@LF:wberroradditionalinfo
Terminate(@TRUE,"Error Encountered",errmsg)




[stanl]

I think I got it. No messing around with protocols using HTTP Client.  I simply added:

Code (WINBATCH) Select Expand


oClient.DefaultRequestHeaders.Add("User-Agent","Other")



and Json was returned. Let me add an error-handler and combine with Kirby's code and should be able to post a workable script. ;)

[Nerrio]

I might be interested to help, when are do you want to start? I will be on vacation all next week at my friend's property in Sicily here but after I am free and can do some work.

[stanl]

I might be interested to help, when are do you want to start? I will be on vacation all next week at my friend's property in Sicily but after I am free and can do some work.

Appreciate it. In 2 other threads related to this I feel comfortable with using CLR to obtain the Json. What remains is parsing it.

[ChuckC]

PowerShell does an excellent job of parsing JSON and returning a PSCustomObject that functions much like a nested hash table with key-value pairs where some values can, in turn, be hash tables.

[stanl]

PowerShell does an excellent job of parsing JSON and returning a PSCustomObject that functions much like a nested hash table with key-value pairs where some values can, in turn, be hash tables.

Agreed. I did notice new namespaces for Json introduced to Core 3.0/3.1 an {I may be worng} also .net 4.8 -
but neither work with CLR

Code (WINBATCH) Select Expand



ObjectClrOption("useany","System.Text.Json")
ObjectClrOption("System.Text.Json.Serialization")



The API used for this and other threads provides a pretty complex Json example, therefore an excellent learning experience. Certainly WB can call a PS script for Json processing, and Kirby's Regex has merit.

[td]

The namespace "System.Text.Json" is part of ".Net Core" but not part of the ".Net Framework" to use MSFT's terminology. That is why WinBatch CLR hosting can't find it.

[kdmoyers]

I feel I should point out that using RegEx for this kind of parsing is like using a chain saw for wood:

• Nothing churns through work with such savage speed
• Less-than-cautious handling results in tragedy
• Severe limitations outside of it's main task

In this case, it's not obvious how you would prevent the regex from finding matches inside a substructure that you did not want, if there was one. 

In other words, RegEx is structure-blind.  There are ways to fight this, but, it gets messy.

Better would be some kind of json query language... but that is outside my experience.

-Kirby

[ChuckC]

The new JSON serialization support in .NET Core v3.x is significantly streamlined and higher performing compared to Newtonsoft JSON, but it is also significantly more limited, especially as it pertains to opt in/out for serialization of fields & properties, custom serialization extensions, etc....  I find that the limitations are so great with it that as long as Newtonsoft JSON remains usable with both .NET Core and .NET Framework, I will continue to use Newtonsoft JSON in all of my current c# projects and most likely in most new c# projects, too.

[stanl]

I feel I should point out that using RegEx for this kind of parsing is like using a chain saw for wood:
-Kirby

Lucky you. Never been in a hurricane.


Regex was valuable for a good section of the url Json. As for Tony's comments about system.text.json - yeah, just another part of the CLR learning curve.


What can or can't be done in WB - thought this was a good exercise. No, the web url had nothing to do with my work but I am using the url as an example for Json as a volunteer to hold Saturday classes for S.T.E.M students.  { they are more interested in Python} :o

[td]

Python is the programming language du jour but with staying power. It is cross-platform but practically baked into many Linux distros like Raspberry Pi OS.