viewpoint-particle

Author Topic: HtmlAgilityPack  (Read 447 times)

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
HtmlAgilityPack
« on: April 07, 2020, 08:34:08 pm »
For anyone looking for an HTML parser...

I often have a need to parse HTML outside of IE and have been looking for a replacement for the DOM Extender since it will not handle Unicode.   Have tried many things with little success but ran across HtmlAgilityPack.  Thought I would see if I could make it COM accessible but turns out it already is set up for that.   If you download the project and compile the NET45 project and then register the DLL using regasm it will work fairly well.  Might be useful to someone so thought I would mention it.   Here is an example script.  It uses XPath syntax.

Code: Winbatch

html = FileGet("test.html")

Message("HTML",html)

hDoc      = ObjectCreate("HtmlAgilityPack.HtmlDocument")

             hDoc.LoadHtml(html)

hBody = hDoc.DocumentNode.SelectSingleNode("//body")
Message("Single Node - Body Node",hBody)

htm = hBody.InnerHtml
Message("Body Node - InnerHtml",htm)

divNodes = hDoc.DocumentNode.SelectNodes("//div")
For x = 0 to divNodes.count-1
  div_txt = divNodes.Item(x).InnerHtml
  Message("Multiple Nodes - Node# %x%",div_txt)
Next



divNodes = hDoc.DocumentNode.SelectNodes("//div[@data-asin]")   ;DIVs with data-asin attribute
For x = 0 to divNodes.count-1
  div_txt = divNodes.Item(x).GetAttributeValue("data-asin", 1)
  Message("Attributes - Node# %x%",div_txt)
Next



ObjectClose(hDoc)


 


Jim

stanl

  • Pundit
  • *****
  • Posts: 1249
Re: HtmlAgilityPack
« Reply #1 on: April 08, 2020, 02:53:51 am »
I used one of the first releases in 2012. You can reference the .dll directly from powershell, but wrap the PS code in WB
Code: [Select]

add-type -Path 'C:\Agility\HtmlAgilityPack.dll'
$HTMLDocument = New-Object HtmlAgilityPack.HtmlDocument
$wc = New-Object System.Net.WebClient
$result = $HTMLDocument.LoadHTML($wc.DownloadString("http://www.simple-talk.com/blogbits/philf/quotations.html"))
$HTMLDocument.DocumentNode.SelectSingleNode("(//table)[2]").InnerText

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #2 on: April 09, 2020, 03:33:32 pm »
Okay.   Discovered many of the Methods were not COM accessible because they used Generic types so spent the day making most things work with WinBatch.   There are MANY more Methods than what the online documentation shows and more than what you will find in my test script.   Can't guarantee they will all work but I altered most that i thought would be useful and, if in my test script, should work, obviously.   I did comment out the Messages so you will need to uncomment any for which you wish to see the results.   

I am attaching the altered HtmlNode.cs and htmlNodeCollections.cs files.  Just rename the original one and replace it with mine if you wish to live dangerously.   I am also including the test script and test.html file.   You will need to change DOMEx to HtmlAgilityPack in the test script, unless you change your namespace like I did.

Don't laugh too much at my code.   I am learning but very much a noob when it comes to C#.   No guarantees as I have only "tested" this and not used it in production so may encounter issues under real use.   Let me know of anything you find and I will fix it.  Hopefully these are the only ones I changed that you need for WinBatch.

Jim

stanl

  • Pundit
  • *****
  • Posts: 1249
Re: HtmlAgilityPack
« Reply #3 on: April 11, 2020, 04:53:33 am »
Good stuff.  I can't find regasm and without all of the warnings not about to download. The initial reason I went the powershell route was to be able to access the dll without a lot of hoops. To my chagrin, I confused PS add-type with WB's CLR 'appbase' - so this fails
Code: Winbatch

ObjectClrOption('appbase','C:\Scripts\Powershell\HtmlAgilityPack.dll')
oHTML = ObjectClrNew("HtmlAgilityPack.HtmlDocument")
 


Got plenty of time, so maybe I'll try gacutil.exe and if that works maybe can help with DetectEncoding (at the bottom of your test.wbt file).


EDIT:  Not bad, ran gacutil.exe -i HtmlAgilityPack.dll and this worked
Code: Winbatch

ObjectClrOption('useany','HtmlAgilityPack')
oHTML = ObjectClrNew("HtmlAgilityPack.HtmlDocument")
Message("HtmlAgilityPack",oHTML)
 


but to move forward probably have to include System.Net.WebClient as container.

stanl

  • Pundit
  • *****
  • Posts: 1249
Re: HtmlAgilityPack
« Reply #4 on: April 11, 2020, 07:03:07 am »
Got this to work using WB CLR (after using gacutil to insert HtmlAgilityPack.dll into GAC). If Tony looks at this thread he can probably school me on creating a secure channel so that oWeb.DownloadData("http://www.simple-talk.com/blogbits/philf/quotations.html") doesn't error.


Not sure if using CLR can contribute any more to your use of dll through COM.
Code: Winbatch

IntControl(73,1,0,0,0)
gosub udfs
html=""
ObjectClrOption("useany", "System.Data")
oWeb = ObjectClrNew('System.Net.WebClient') ;to obtain html string from URL
ObjectClrOption('useany','HtmlAgilityPack')
oHTML = ObjectClrNew("HtmlAgilityPack.HtmlDocument")


Message("HtmlAgilityPack",oHTML) ;test that object is created


;this will error could not create SSL/TSL secure channel
;html = oWeb.DownloadData("http://www.simple-talk.com/blogbits/philf/quotations.html")
;Message("Test",html)


test() ;sample html string
oHTML.LoadHtml(html)
htmlBody = oHTML.DocumentNode.SelectSingleNode("//body")


Message("Test",htmlBody.OuterHtml)
oWeb=0
oHTML=0
Exit


:WBERRORHANDLER
oDiag=0
geterror()
Message("Error Encountered",errmsg)
Exit


:udfs
#DefineSubRoutine geterror()
   wberroradditionalinfo = wberrorarray[6]
   lasterr = wberrorarray[0]
   handlerline = wberrorarray[1]
   textstring = wberrorarray[5]
   linenumber = wberrorarray[8]
   errmsg = "Error: ":lasterr:@LF:textstring:@LF:"Line (":linenumber:")":@LF:wberroradditionalinfo
   Return(errmsg)
#EndSubRoutine


#DefineSubRoutine test()
;simple test for HTML string
html = $"<!DOCTYPE html>
<html>
<body>
   <h1>This is bold heading</h1>
   <p>This is underlined paragraph</p>
   <h2>This is italic heading</h2>
</body>
</html> "
$"

Return(html)
#EndSubRoutine
Return

 


EDIT: Jim, this worked for DetectEncoding [partial code]


Code: Winbatch

oHTML.LoadHtml(html)
oHTML.Save("C:\temp\test.html")
encoding = oHTML.DetectEncoding("C:\temp\test.html")
Message("Encoding",encoding.ToString())
 

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #5 on: April 11, 2020, 08:45:28 am »
Thanks.   Should have thought about the GAC...probably wouldn't have had to rewrite stuff but I did learn a lot as well as discovering that the documentation doesn't cover very much. 

Also, some of what they did didn't appear to make much sense, especially in regards to optional parameters.   Maybe I missed the point but redid most of those by removing what I considered unnecessary duplication of methods.   They created two methods.  One without a parameter whose only purpose seemed to be to call the other function with the desired default parameter.

Regasm.exe can be found under the C:\Winodws\Microsoft.NET\Framework(64)\ path.   Depends on versions you have installed.  One for every version but don't know that it matters which you use.  More recent the better I would assume.   Again, may not even need it with you using GAC.

Had actually forgotten about the note on DetectEncoding.   Thanks.   I will have to test that again.

Jim

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #6 on: April 11, 2020, 08:58:33 am »
I still get the "specified ole variant is invalid" error on Detect encoding through COM.   I don't know that I would use that Method but was trying to make "everything" work. 

Jim

stanl

  • Pundit
  • *****
  • Posts: 1249
Re: HtmlAgilityPack
« Reply #7 on: April 11, 2020, 10:16:08 am »
As I implied: COM vs. CLR is six of one....


Re-read some original posts, when I first brought up https://api.pwnedpasswords.com/range/ and needed TLS security. Tony provided code to resolve
Code: Winbatch

objUri = ObjectClrNew('System.Uri', strUrl)
objSvcManager = ObjectClrNew('System.Net.ServicePointManager')
protocols = ObjectClrType("System.Net.SecurityProtocolType",3072|768)  
objSvcManager.SecurityProtocol = protocols
objSvcPoint = objSvcManager.FindServicePoint(objUri)
 


so think I can work this in to use HtmlAgility against URL's. 


In reference to regasm.exe - I did a complete search up through .NET 4.8 and it doesn't exist anywhere, and I have version 1909 on my Win10 machine.  In any event, my biggest question is why 'appbase' fails if the .dll can easily be placed in the GAC and then used by CLR?

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #8 on: April 11, 2020, 10:30:27 am »
Interesting.   I did notice that it doesn't show in anything past 4.0 so maybe that has gone away.   I am running that version of Win10 as well.  I can send you a copy if you want to try it out.   If so, just email directly so I have your current email.

Maybe Tony has anthropomorphized his computer and now has to stay 6 feet away?   ;)

In any event, appreciate your input.   I am starting some real world use of it to see how it goes.   I have about 70 scripts where I use the DOM Extender so will change over a couple and see if it is stable in production and then work through the others if all is good.   This will be a big help with the foreign language stuff I have to parse.

Jim

stanl

  • Pundit
  • *****
  • Posts: 1249
Re: HtmlAgilityPack
« Reply #9 on: April 11, 2020, 10:39:16 am »
Figured out the URL stuff... this worked via WB CLR with .dll in GAC
Code: Winbatch

IntControl(73,1,0,0,0)
gosub udfs
html=""
ObjectClrOption("useany", "System.Data")
oWeb = ObjectClrNew('System.Net.WebClient') ;to obtain html string from URL
ObjectClrOption('useany','HtmlAgilityPack')
oHTML = ObjectClrNew("HtmlAgilityPack.HtmlDocument")
;oNode = ObjectClrNew("HtmlAgilityPack.HtmlDocument.HtmlNode")


Message("HtmlAgilityPack",oHTML) ;test that object is created
;avoid secure channel errors
strUrl="http://www.simple-talk.com/blogbits/philf/quotations.html"
objUri = ObjectClrNew('System.Uri', strUrl)
objSvcManager = ObjectClrNew('System.Net.ServicePointManager')
protocols = ObjectClrType("System.Net.SecurityProtocolType",3072|768|192|48|0)  
objSvcManager.SecurityProtocol = protocols
objSvcPoint = objSvcManager.FindServicePoint(objUri)


html = oWeb.DownloadString(strUrl)


oHTML.LoadHtml(html)
htmlBody = oHTML.DocumentNode.SelectSingleNode("(//table)[2]").InnerText
Message("Insults",htmlBody)
oHTML.Save("C:\temp\test.html")
encoding = oHTML.DetectEncoding("C:\temp\test.html")
Message("Encoding",encoding.ToString())


oWeb=0
oHTML=0
Exit


:WBERRORHANDLER
oDiag=0
geterror()
Message("Error Encountered",errmsg)
Exit


:udfs
#DefineSubRoutine geterror()
   wberroradditionalinfo = wberrorarray[6]
   lasterr = wberrorarray[0]
   handlerline = wberrorarray[1]
   textstring = wberrorarray[5]
   linenumber = wberrorarray[8]
   errmsg = "Error: ":lasterr:@LF:textstring:@LF:"Line (":linenumber:")":@LF:wberroradditionalinfo
   Return(errmsg)
#EndSubRoutine


Return


 

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #10 on: April 11, 2020, 11:38:42 am »
Excellent.  Thanks again.

I haven't gotten around to testing that via COM yet.   Will post my script as well if it works.

Jim

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #11 on: April 11, 2020, 12:11:55 pm »
Here is COM version for loading from URL.

Jim

Code: Winbatch

url  = "http://www.yourserver.com/test.html";

hDoc      = ObjectCreate("DomEx.HtmlDocument")
htmlNode  = ObjectCreate("DomEx.HtmlNode")
hWeb      = ObjectCreate("DomEx.HtmlWeb")

hDoc = hWeb.Load(url);

hBody = hDoc.DocumentNode.SelectSingleNode("//body")
htm = hBody.OuterHtml
Message("Body Node - InnerHtml",htm)


divNodes = hDoc.DocumentNode.SelectNodes("//div[@data-asin]")
For x = 0 to divNodes.count-1

  div_txt = divNodes.Item(x).OuterHTML
  Message("Get Data ASIN Attribute",div_txt)

Next

ObjectClose(hDoc)
ObjectClose(hWeb)
ObjectClose(htmlNode)


 

stanl

  • Pundit
  • *****
  • Posts: 1249
Re: HtmlAgilityPack
« Reply #12 on: April 11, 2020, 12:58:35 pm »
Nice. Still waiting for Tony's comment on why 'appbase' fails even though CLR works with .dll if placed in GAC.


EDIT: no need, just make first line: ObjectClrOption("Appbase", 'C:\scripts\powershell\') ;or wherever you place .dll.  So now can use dll w/out either regasm or gactutil. 

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #13 on: April 11, 2020, 01:41:25 pm »
Most Excellent.  I thought there was a way to do that but found what I was thinking about and it was something different so gave up.  Thank you.

Jim

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #14 on: April 11, 2020, 02:02:21 pm »
Does this work for you?  I get an error on "hDoc.LoadHtml(html) ".   I have not loaded this into GAC.

Jim

Code: Winbatch

html = FileGet("hap_test.html")

ObjectClrOption("Appbase", DirScript())
ObjectClrOption('useany','HtmlAgilityPack')
hDoc  = ObjectClrNew("HtmlAgilityPack.HtmlDocument")

hDoc.LoadHtml(html)

 

stanl

  • Pundit
  • *****
  • Posts: 1249
Re: HtmlAgilityPack
« Reply #15 on: April 11, 2020, 03:34:19 pm »
try "use" not "useany"

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #16 on: April 11, 2020, 03:40:41 pm »
Same error.   So it works for you?   GAC may have affected that I guess???

Jim

stanl

  • Pundit
  • *****
  • Posts: 1249
Re: HtmlAgilityPack
« Reply #17 on: April 12, 2020, 03:58:29 am »
Hmmm.  I ran gacutil.exe /u HTMLAgilityPack which uninstalled from GAC. I then ran my script with Appbase and it worked. I did notice a slight time difference in execution but could be the Internet.
Code: Winbatch

IntControl(73,1,0,0,0)
gosub udfs
html=""
ObjectClrOption("Appbase", 'C:\scripts\powershell\')
ObjectClrOption("useany", "System.Data")
oWeb = ObjectClrNew('System.Net.WebClient') ;to obtain html string from URL
ObjectClrOption('use','HtmlAgilityPack')
oHTML = ObjectClrNew("HtmlAgilityPack.HtmlDocument")
;avoid secure channel errors
strUrl="http://www.simple-talk.com/blogbits/philf/quotations.html"
objUri = ObjectClrNew('System.Uri', strUrl)
objSvcManager = ObjectClrNew('System.Net.ServicePointManager')
protocols = ObjectClrType("System.Net.SecurityProtocolType",3072|768|192|48|0)  
objSvcManager.SecurityProtocol = protocols
objSvcPoint = objSvcManager.FindServicePoint(objUri)


html = oWeb.DownloadString(strUrl)


oHTML.LoadHtml(html)
htmlBody = oHTML.DocumentNode.SelectSingleNode("(//table)[2]").InnerText
Message("Insults",htmlBody)
;save html and display encoding
oHTML.Save("C:\temp\test.html")
encoding = oHTML.DetectEncoding("C:\temp\test.html")
Message("Encoding",encoding.ToString())


oWeb=0
oHTML=0
Exit


:WBERRORHANDLER
oWeb=0
oHTML=0
geterror()
Message("Error Encountered",errmsg)
Exit


:udfs
#DefineSubRoutine geterror()
   wberroradditionalinfo = wberrorarray[6]
   lasterr = wberrorarray[0]
   handlerline = wberrorarray[1]
   textstring = wberrorarray[5]
   linenumber = wberrorarray[8]
   errmsg = "Error: ":lasterr:@LF:textstring:@LF:"Line (":linenumber:")":@LF:wberroradditionalinfo
   Return(errmsg)
#EndSubRoutine


Return
 
[size=78%] [/size]

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #18 on: April 12, 2020, 08:22:07 am »
Happy Easter!!!

Here is the error I get.   Maybe I need to go back and Build the original project and see what happens???

Jim

Error: 1261
Ole: Exception
Line (21)
COM/CLR Exception:

    HtmlAgilityPack
    Object reference not set to an instance of an object.

stanl

  • Pundit
  • *****
  • Posts: 1249
Re: HtmlAgilityPack
« Reply #19 on: April 12, 2020, 10:04:47 am »
I didn't really build anything. Just downloaded the latest html-agility-pack-master.zip, unzipped, and fleshed out the .dll to my scripts\powershell folder. Oh, and I had re-booted after uninstalling from the GAC and Appbase still works. On a tangent right now trying to integrate a filestream object into the mix as it seems to offer a way to save in specified encoding. Tried accessing a URL as a webstream and seeing if Agility can decode - but error was "URI encoding not supported".... but this has been fun

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #20 on: April 12, 2020, 02:27:11 pm »
Apparently it is something with the changes I made because if I Build the solution right out of the box it works but I am back to the "Generic Types cannot be Marshaled to the COM Interface Pointers" message for some of the Methods even though I am using the CLR stuff via WinBatch.

Jim

stanl

  • Pundit
  • *****
  • Posts: 1249
Re: HtmlAgilityPack
« Reply #21 on: April 13, 2020, 09:02:23 am »
So, bottom line: were you able to get CLR to initiate at all?

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #22 on: April 13, 2020, 09:49:41 am »
Yes, with the version right out of the box,  but many of the methods are unusable because it still gives me the same error I was getting via COM before all my changes.   

    "Generic Types cannot be Marshaled to the COM Interface Pointers"

Jim

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #23 on: April 13, 2020, 09:50:58 am »
Is WinBatch able to create a "Collection" natively?   That is, a list that ForEach can be used with?

Jim

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #24 on: April 13, 2020, 05:58:07 pm »
FINALLY!!! Tracked down the problem with using CLR and my version.    Related to the Generic Problem but wasn't nice enough to tell me that.   It is their use of "Dictionary", which is a Generic Collection.   IDictionary is supposedly a non-generic version, which should theoretically work but haven't cracked the code yet.   Only took me a couple of hours and lots of MessageBoxes to track it down.

Jim

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #25 on: April 22, 2020, 07:42:48 pm »
Just to pose this as a question...Is there a way to overcome the  "Generic Types cannot be Marshaled to the COM Interface Pointers" issue with some setting on compile or initialization or does it require a rewrite?   I load it as follows.  The load works fine and some methods are good.   Just those passing back Dictionaries and other such generic types.  Thanks.


Code: Winbatch

html = FileGet("hap_test.html")

ObjectClrOption("Appbase", DirScript())
ObjectClrOption('use','HtmlAgilityPack')
hDoc = ObjectClrNew("HtmlAgilityPack.HtmlDocument")

hDoc.LoadHtml(html)


 

Jim

stanl

  • Pundit
  • *****
  • Posts: 1249
Re: HtmlAgilityPack
« Reply #26 on: April 23, 2020, 03:44:16 am »
Maybe a dumb answer but since I never encountered that error is it possible that your running REGASM is somehow interfering with loading via the CDL?

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #27 on: April 23, 2020, 05:11:38 am »
I unregistered it but is possible I suppose.  When registered it won't load, if I remember correctly.  I have tried so many things it gets fuzzy.  Here is a one that fails.

Code: Winbatch
divNodes = hDoc.DocumentNode.SelectNodes("//div[@data-asin]")
ForEach iNode in divNodes

  div_txt = iNode.GetDataAttribute("asin")        ;Returns Attributes with an attribute of "data-"+parameter
   Message("Get Data ASIN Attribute",div_txt.Value)

  div_txt  = iNode.GetDataAttributes()            ;Returns Attributes that begin with "data-"
  Foreach y in div_txt
    Message("Get Data Attributes",y.Value)
  Next

Next
 

Jim

JTaylor

  • Pundit
  • *****
  • Posts: 1397
    • Data & Stuff Inc.
Re: HtmlAgilityPack
« Reply #28 on: April 23, 2020, 08:48:01 am »
I went through the registry and cleaned out anything related but still fails.   I know everything you posted would have worked.   Appreciate it if you tried that last bit of code and see what happens. 

Jim