https://api.weather.gov/alerts/active?area=NC (https://api.weather.gov/alerts/active?area=NC) [substitute any state]
returns some interesting JSON. While I will probably make a deeper dive with PS, would be interested in any advanced WB parsing hints. Mostly interest in sections like:
"headline": "Flood Advisory issued September 25 at 10:46AM EDT until September 25 at 11:45AM EDT by NWS Raleigh NC", "description": "The National Weather Service in Raleigh has issued a\n\n* Flood Advisory for Poor Drainage Areas for...\nAnson County in central North Carolina...\nNorthwestern Cumberland County in central North Carolina...\nHarnett County in central North Carolina...\nHoke County in central North Carolina...\nLee County in central North Carolina...\nSoutheastern Montgomery County in central North Carolina...\nMoore County in central North Carolina...\nRichmond County in central North Carolina...\nScotland County in central North Carolina...\n\n* Until 1145 AM EDT.\n\n* At 1046 AM EDT, Doppler radar indicated heavy rain. Overflowing\npoor drainage areas will cause minor flooding in the advisory area.\n\nSome locations that will experience flooding include...\nFayetteville, Sanford, Laurinburg, Southern Pines, Rockingham,\nWadesboro, Raeford, Lillington, Carthage, Fort Bragg, Pinehurst,\nDunn, Hamlet, Aberdeen, Angier, Hoffman, Antioch, Pope AFB, Spring\nLake and Erwin.",
The Newtonsoft JSON NuGet package is what I use in my c# projects for heavy duty JSON parsing.
With .NET Standard & .NET Core, there is a new built-in JSON parser but is is more limited.
PowerShell has some nice support for serializing & de-serializing JSON text, either between objects and strings or between objects and string content in a file.
Obviously, all of the methods mentioned above are accessible within WIL via the CLR support.
I am usually lazy and use C# to convert to XML. :) If I get this current project I am on done and this thread is still active I might jump in.
Jim
The squeamish may want to avert their eyes: ugly hacks ahead!
curl = "https://api.weather.gov/alerts/active?area=NC"
text = fileget(shortcutdir("desktop"):"\weatherdata.json")
; array of matches
arr = arrdimension(99,3)
re = '"headline": "([^"]+)",\r\n *"description": "(.+?)"'
; fill 2d array with match results
objRE = ObjectOpen("VBScript.RegExp")
objRE.Pattern = re
objRE.IgnoreCase = @true ; canse sensitivity
objRE.MultiLine = @true ; multiline mode
objRE.Global = @true ; match more than once
matches = objRE.execute(text)
arr[0,0] = 0 + matches.count
if matches.count
for i = 0 to matches.count-1
arr[i+1,0] = matches.item(i).value ; full match...
for j = 0 to matches.item(i).submatches.count-1
arr[i+1,j+1] = matches.item(i).submatches.item(j) ; ...and each submatch
next j
next i
endif
objRE = 0
; display contents of array
if arr[0,0]
for ii = 1 to arr[0,0]
message(arr[ii,1], strreplace(arr[ii,2],"\n",@lf) )
next ii
else
message("no match",re)
endif
EXIT
I LOVE the Regex approach. Think I'll try a CLR : ObjectClrNew('System.Net.WebRequest') along with ObjectClrNew('System.Text.RegularExpressions.Regex',cPattern).... just for another ugly hack.
What is interesting is I ran the URL this morning and NC had flash flood alerts, but running it now the JSON has no headlines. However, try CA as the state. Again, probably why the Regex is cool.
Here's what I get for a CLR based RegEx solution. I like it, thanks for the pointer!
curl = "https://api.weather.gov/alerts/active?area=NC"
text = fileget(shortcutdir("desktop"):"\weatherdata.json")
regexpr = '"headline": "([^"]+)",\r\n *"description": "(.+?)"'
REX = "System.Text.RegularExpressions"
ObjectClrOption('useany', REX)
R = ObjectClrNew(REX:'.RegexOptions')
R = R.IgnoreCase | R.Multiline | R.Singleline
ReOx = ObjectClrType(REX:'.RegexOptions',R)
RE = ObjectClrNew(REX:'.Regex', regexpr, ReOx)
matches = RE.Matches(text)
message('count',matches.Count)
foreach match in matches
groups = match.Groups
t1 = groups.Item(1).Captures.Item(0).Value
t2 = groups.Item(2).Captures.Item(0).Value
message(t1,t2)
next
EXIT
Nice Kirby;
Now my 'je ne sais quoi'. You can open the link in a browser and it feeds back Json quickly. Tried the CLR route with the code below as well as with System.Net.Http.HttpClient and in both cases the response is access is forbidden.
strUrl = "https://api.weather.gov/alerts/active?area=NC"
ObjectClrOption('useany', 'System')
objWebUtil = ObjectClrNew('System.Net.WebUtility')
objUri = ObjectClrNew('System.Uri', strUrl)
objSvcManager = ObjectClrNew('System.Net.ServicePointManager')
protocols = ObjectClrType("System.Net.SecurityProtocolType",3072|768)
objSvcManager.SecurityProtocol = protocols
objSvcPoint = objSvcManager.FindServicePoint(objUri)
objWebRequest = ObjectClrNew('System.Net.WebRequest')
objWebRequest = objWebRequest.Create(objUri)
objWebRequest.Timeout = objWebRequest.Timeout * 6
objResponse = objWebRequest.GetResponse()
Message("",objResponse) ; 403 forbidden error
exit
I think adding headers to the request will avoid the forbidden error. Little confused on headers for WebRequest class - assume it would need "application/json" or something like that.
but it is so easy with Powershell
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$j = Invoke-WebRequest 'https://api.weather.gov/alerts/active?area=NC' | ConvertFrom-Json | ConvertTo-Json
$j | Out-File "c:\temp\NC.json"
And below is my shot with HTTPClient. According to the MSFT docs I read the System.Net.Http.HttpClientHandler class should be able to set SSL Protocols, but the class wasn't even recognized. The script will either give a timeout message after 10 seconds or show a forbidden response. Otherwise, there is a lot of interesting Json and the Powershell code only gets part of it.
IntControl(73,1,0,0,0)
request = "https://api.weather.gov/alerts/active?area=NC"
ObjectClrOption("useany", "System")
ObjectClrOption("useany", "System.Net.Http")
oClient = ObjectClrNew('System.Net.Http.HttpClient')
oTask = ObjectClrNew('System.Threading.Tasks.Task')
oResponse = ObjectClrNew('System.Net.Http.HttpResponseMessage')
;trying to set security protocol but fails
;oHandler = ObjectClrNew('System.Net.Http.HttpClientHandler')
;oHandler.SslProtocols = 3072
Response = oClient.GetAsync(request).Result
n=0
While n<=10
If Response.IsSuccessStatusCode Then Break
TimeDelay(1)
n=n+1
If n>9
Pause("Giving Up","Unable To Create Request")
goto theend
Endif
EndWhile
TaskStr = Response.Content.ReadAsStringAsync()
html=TaskStr.Result
Pause("response",html)
:theend
oTask = 0
oClient = 0
Exit
:WBERRORHANDLER
oTask = 0
oClient = 0
wberroradditionalinfo = wberrorarray[6]
lasterr = wberrorarray[0]
handlerline = wberrorarray[1]
textstring = wberrorarray[5]
linenumber = wberrorarray[8]
errmsg = "Error: ":lasterr:@LF:textstring:@LF:"Line (":linenumber:")":@LF:wberroradditionalinfo
Terminate(@TRUE,"Error Encountered",errmsg)
I think I got it. No messing around with protocols using HTTP Client. I simply added:
oClient.DefaultRequestHeaders.Add("User-Agent","Other")
and Json was returned. Let me add an error-handler and combine with Kirby's code and should be able to post a workable script. ;)
I might be interested to help, when are do you want to start? I will be on vacation all next week at my friend's property in Sicily here (https://tranio.com/italy/sicily/) but after I am free and can do some work.
Quote from: Nerrio on September 27, 2020, 08:13:38 AM
I might be interested to help, when are do you want to start? I will be on vacation all next week at my friend's property in Sicily but after I am free and can do some work.
Appreciate it. In 2 other threads related to this I feel comfortable with using CLR to obtain the Json. What remains is parsing it.
PowerShell does an excellent job of parsing JSON and returning a PSCustomObject that functions much like a nested hash table with key-value pairs where some values can, in turn, be hash tables.
Quote from: ChuckC on September 28, 2020, 04:23:41 AM
PowerShell does an excellent job of parsing JSON and returning a PSCustomObject that functions much like a nested hash table with key-value pairs where some values can, in turn, be hash tables.
Agreed. I did notice new namespaces for Json introduced to Core 3.0/3.1 an {I may be worng} also .net 4.8 -
but neither work with CLR
ObjectClrOption("useany","System.Text.Json")
ObjectClrOption("System.Text.Json.Serialization")
The API used for this and other threads provides a pretty complex Json example, therefore an excellent learning experience. Certainly WB can call a PS script for Json processing, and Kirby's Regex has merit.
The namespace "System.Text.Json" is part of ".Net Core" but not part of the ".Net Framework" to use MSFT's terminology. That is why WinBatch CLR hosting can't find it.
I feel I should point out that using RegEx for this kind of parsing is like using a chain saw for wood:
- Nothing churns through work with such savage speed
- Less-than-cautious handling results in tragedy
- Severe limitations outside of it's main task
In this case, it's not obvious how you would prevent the regex from finding matches inside a substructure that you did not want, if there was one.
In other words, RegEx is structure-blind. There are ways to fight this, but, it gets messy.
Better would be some kind of json query language... but that is outside my experience.
-Kirby
The new JSON serialization support in .NET Core v3.x is significantly streamlined and higher performing compared to Newtonsoft JSON, but it is also significantly more limited, especially as it pertains to opt in/out for serialization of fields & properties, custom serialization extensions, etc.... I find that the limitations are so great with it that as long as Newtonsoft JSON remains usable with both .NET Core and .NET Framework, I will continue to use Newtonsoft JSON in all of my current c# projects and most likely in most new c# projects, too.
Quote from: kdmoyers on September 28, 2020, 11:51:50 AM
I feel I should point out that using RegEx for this kind of parsing is like using a chain saw for wood:
-Kirby
Lucky you. Never been in a hurricane.
Regex was valuable for a good section of the url Json. As for Tony's comments about system.text.json - yeah, just another part of the CLR learning curve.
What can or can't be done in WB - thought this was a good exercise. No, the web url had nothing to do with my work but I am using the url as an example for Json as a volunteer to hold Saturday classes for S.T.E.M students. { they are more interested in Python} :o
Python is the programming language du jour but with staying power. It is cross-platform but practically baked into many Linux distros like Raspberry Pi OS.