viewpoint-particle

Author Topic: CLR: WebRequest / headers only  (Read 316 times)

stanl

  • Pundit
  • *****
  • Posts: 1573
CLR: WebRequest / headers only
« on: September 28, 2021, 04:27:26 pm »
Just wondering is there is a way to get around the attached error the script produces on line headers = oWebRequest.GetResponse().Headers


Code: Winbatch

;CLR - attempt Web Request to get headers only
Url = 'https://community.idera.com/'
ObjectClrOption('useany', 'System')
oWebUtil = ObjectClrNew('System.Net.WebUtility')


oUri = ObjectClrNew('System.Uri', Url)
oSvcManager = ObjectClrNew('System.Net.ServicePointManager')
protocols = ObjectClrType("System.Net.SecurityProtocolType",3072|768)  
oSvcManager.SecurityProtocol = protocols
oSvcPoint = oSvcManager.FindServicePoint(oUri)


oWebRequest = ObjectClrNew('System.Net.WebRequest')
oRequest = oWebRequest.Create(oUri)
oRequest.Timeout = oRequest.Timeout * 6
headers = oRequest.GetResponse().Headers
;gives error here


keys = headers.AllKeys
output="key":@tab:"Value":@lf


ForEach k in keys
    output:= k:@tab:headers.GetValues(k):@lf
Next


oResponse.Close()


Message(Url,output)


Exit

 

stanl

  • Pundit
  • *****
  • Posts: 1573
Re: CLR: WebRequest / headers only
« Reply #1 on: September 29, 2021, 10:47:45 am »
Oh, and before the put-downs or other arguments start: yes, this does work in powershell
Code: [Select]

$url = 'https://community.idera.com/'
$request = [System.Net.WebRequest]::Create( $Url )
$headers = $request.GetResponse().Headers
$headers.AllKeys |
     Select-Object @{ Name = "Key"; Expression = { $_ }},
     @{ Name = "Value"; Expression = { $headers.GetValues( $_ ) } }


with 2 points:
  • doing as a favor for old client who wanted to know urls that contained cookies in headers
  • GetResponse() is defined as an 'abstract class' so would be interested in if / how they may work with WB's CDL

no harm / no foul

td

  • Tech Support
  • *****
  • Posts: 3952
    • WinBatch
Re: CLR: WebRequest / headers only
« Reply #2 on: September 29, 2021, 10:57:48 am »
That is a bizarre error message. I have no idea why the CLR is regurgitating that exception because the WIL DLL is not attempting to use an IDispatch interface to access anything in your script. The response class is abstract so the behavior of the object returned by GetResponse can vary based on the concreate class returned but the SupportsHeaders property returns true. For example,

Code: Winbatch
objResponse = oRequest.GetResponse()
if objResponse.SupportsHeaders then strText = 'Headers supported'
else strText = 'Headers not supported'
Message('Are Headers supported', strText)
 


Also, the WebHeaderCollection class returned by the Headers property is marked as ComVisibleAttribute so it shouldn't make any difference anyway.  Here's a link to the class:

https://docs.microsoft.com/en-us/dotnet/api/system.net.webheadercollection?view=netframework-4.8

I will have to do a little bit more investigation in an attempt to at least come up with a better explanation of what is going on.

You might be able to use the GetResponseHeader method as an alternative.

https://docs.microsoft.com/en-us/dotnet/api/system.net.httpwebresponse.getresponseheader?view=netframework-4.8
"No one who sees a peregrine falcon fly can ever forget the beauty and thrill of that flight."
  - Dr. Tom Cade

td

  • Tech Support
  • *****
  • Posts: 3952
    • WinBatch
Re: CLR: WebRequest / headers only
« Reply #3 on: September 29, 2021, 11:01:47 am »
Oh, and before the put-downs or other arguments start

I was under the impression that we have gotten past that point...
"No one who sees a peregrine falcon fly can ever forget the beauty and thrill of that flight."
  - Dr. Tom Cade

stanl

  • Pundit
  • *****
  • Posts: 1573
Re: CLR: WebRequest / headers only
« Reply #4 on: September 29, 2021, 12:07:08 pm »
Oh, and before the put-downs or other arguments start

I was under the impression that we have gotten past that point...


like I wrote... no harm / no foul..


but, if you execute the PS code against 'https://forum.winbatch.com/' you will get 403 forbidden error, which is useful info [I'm doing this as a favor for an old client, not for money]...


Just turned out an interesting error.... I think there is a javascript getresponseheaders() function... dunno

td

  • Tech Support
  • *****
  • Posts: 3952
    • WinBatch
Re: CLR: WebRequest / headers only
« Reply #5 on: September 29, 2021, 12:37:04 pm »
The GetResponseHeader method appears to work. Using your script with the method call GetResponseHeader('Content-Type') I received the header content "text/html: charset=utf-8" which is what you would expect.
"No one who sees a peregrine falcon fly can ever forget the beauty and thrill of that flight."
  - Dr. Tom Cade

stanl

  • Pundit
  • *****
  • Posts: 1573
Re: CLR: WebRequest / headers only
« Reply #6 on: September 29, 2021, 03:01:09 pm »
The GetResponseHeader method appears to work. Using your script with the method call GetResponseHeader('Content-Type') I received the header content "text/html: charset=utf-8" which is what you would expect.


Understood. But the object was to iterate all header keys not just a single one as web request will return different or different amount of keys. And, in testing I missed your use of the function



oResponse = oRequest.GetResponse()
test = oResponse.GetResponseHeader('Content-Type')
Message("",test)          FAILS

td

  • Tech Support
  • *****
  • Posts: 3952
    • WinBatch
Re: CLR: WebRequest / headers only
« Reply #7 on: September 29, 2021, 04:58:02 pm »
There is a defined list of headers that are supported by the response object so those would be the only ones you need to test for in a straightforward loop. Or just use a list of common header names.  I thought that would be self-evident.  I can't remember where the list is on MSFT's website or maybe I just imaged it. Anyway, all I can tell you is that I tried the method on the site in your script and it worked repeatedly.
"No one who sees a peregrine falcon fly can ever forget the beauty and thrill of that flight."
  - Dr. Tom Cade

td

  • Tech Support
  • *****
  • Posts: 3952
    • WinBatch
Re: CLR: WebRequest / headers only
« Reply #8 on: September 29, 2021, 05:06:12 pm »
This hack of your script works in repeated tests.

Code: Winbatch
Url = 'https://community.idera.com/'
ObjectClrOption('useany', 'System')
oWebUtil = ObjectClrNew('System.Net.WebUtility')


oUri = ObjectClrNew('System.Uri', Url)
oSvcManager = ObjectClrNew('System.Net.ServicePointManager')
protocols = ObjectClrType("System.Net.SecurityProtocolType",3072|768)  
oSvcManager.SecurityProtocol = protocols
oSvcPoint = oSvcManager.FindServicePoint(oUri)


oWebRequest = ObjectClrNew('System.Net.WebRequest')
oRequest = oWebRequest.Create(oUri)
oRequest.Timeout = oRequest.Timeout * 6
;;headers = oRequest.GetResponse().Headers
;gives error here
objResponse = oRequest.GetResponse()
if objResponse.SupportsHeaders then strText = 'Headers supported'
else strText = 'Headers not supported'
Message('Are Headers supported', strText)

header = objResponse.GetResponseHeader('Content-Type')
message('Content-Type Header', header)
 
"No one who sees a peregrine falcon fly can ever forget the beauty and thrill of that flight."
  - Dr. Tom Cade

stanl

  • Pundit
  • *****
  • Posts: 1573
Re: CLR: WebRequest / headers only
« Reply #9 on: September 30, 2021, 03:00:11 am »
Your hack worked. I probably had a typo error when I tried GetResponseHeader(). I attached the header list for the Idera url. I checked with the person I will be compiling this for and he is only interested in the set-cookie header. That was easy to test with both the Idera url and WB tech db url.  I can adapt the code to a UDF chkheader(url,header). Didn't mean to come off so contentious in my replies. My hack to your hack can test for Set-Cookie
Code: Winbatch

;CLR - attempt Web Request to get headers only
hdr = 'Set-Cookie'
;comment/uncomment url to test
Url = 'https://community.idera.com/'   ;header should exist
;Url = 'https://techsupt.winbatch.com/'  ;header should not exist
;==============================================================
ObjectClrOption('useany', 'System')
oWebUtil = ObjectClrNew('System.Net.WebUtility')
oUri = ObjectClrNew('System.Uri', Url)
oSvcManager = ObjectClrNew('System.Net.ServicePointManager')
protocols = ObjectClrType("System.Net.SecurityProtocolType",3072|768)  
oSvcManager.SecurityProtocol = protocols
oSvcPoint = oSvcManager.FindServicePoint(oUri)


oWebRequest = ObjectClrNew('System.Net.WebRequest')
oRequest = oWebRequest.Create(Url)
oRequest.Timeout = oRequest.Timeout * 6
objResponse = oRequest.GetResponse()
if objResponse.SupportsHeaders
   header = objResponse.GetResponseHeader(hdr)
   if Strlen(header)==0
      message('Set-Cookie Header', "Does Not Exist")
   else
      message(hdr:' Header', header)
   endif
else
   Message('Are Headers supported','Headers not supported')
endif
objResponse.Close()
Exit
 

td

  • Tech Support
  • *****
  • Posts: 3952
    • WinBatch
Re: CLR: WebRequest / headers only
« Reply #10 on: September 30, 2021, 09:03:41 am »
For verification purposes, I had gotten a dump of all headers the old-fashioned way.

Code: Winbatch
Url = 'https://community.idera.com/'
ObjHttp = ObjectCreate("WinHttp.WinHttpRequest.5.1")

TimeoutVal = 59000
ObjHttp.SetTimeouts(TimeoutVal, TimeoutVal, TimeoutVal, TimeoutVal)  
ObjHttp.Open("GET", Url)
ObjHttp.Send()

strHeaders = ObjHttp.GetAllResponseHeaders()

;; More stuff here.
 

And yes, it still works on Windows 11.
"No one who sees a peregrine falcon fly can ever forget the beauty and thrill of that flight."
  - Dr. Tom Cade

stanl

  • Pundit
  • *****
  • Posts: 1573
Re: CLR: WebRequest / headers only
« Reply #11 on: September 30, 2021, 03:01:30 pm »
touché

td

  • Tech Support
  • *****
  • Posts: 3952
    • WinBatch
Re: CLR: WebRequest / headers only
« Reply #12 on: October 01, 2021, 01:33:05 pm »
I confess to an ulterior but benign motive. Our discussions seem to attract a few eyeballs so I thought I would drop a nugget from the past to maybe remind other users that there is more than one way to perform the task.
"No one who sees a peregrine falcon fly can ever forget the beauty and thrill of that flight."
  - Dr. Tom Cade

stanl

  • Pundit
  • *****
  • Posts: 1573
Re: CLR: WebRequest / headers only
« Reply #13 on: October 02, 2021, 02:41:24 am »
I confess to an ulterior but benign motive. Our discussions seem to attract a few eyeballs so I thought I would drop a nugget from the past to maybe remind other users that there is more than one way to perform the task.


I alluded to 'another' method in my early posts, but WinHttp had slipped my mind although it has served as a swiss army knife for many scripts I have written.  I did end up sending a compiled exe to my friend via the CLR route and I'm sure I will have other CLR questions coming up.