WebView2 Extender Update

[JTaylor]

I have posted an update to my WebView2 Extender, in the event anyone uses this or is interested.   I decided I preferred the COM approach to the functionality.  I left a copy of the old approach in the distribution files.  See WebView in the Help file.

For clarification, this gives you a way to embed a current browser into a Dialog.  With the loss of IE this became practically impossible as most sites check the browser type so things like Shell.Explore2 no longer works very well.  This restores the ability to browse and scrape sites again via a DOM model.


http://www.jtdata.com/anonymous/wbOmnibus.zip


Jim

[JTaylor]

Just a bit of direction:

Under Sample code there is a WV2 directory.  Unless I have missed something, you will only need that and the Webview2 files from Microsoft.  I normally use the x86 Fixed Version.  If you create an .\edge subdirectory under the wv2 folder and place the files there, the example I created should work.  You could also try the Installer for the Runtime files and that may work as well.

An added note, if you place a copy of the edge (or whatever you name it) subdirectory in the WinBatch\System\ folder and the WebView2Loader.dll file in the System folder itself.  You can run scripts without having to compile them.

https://developer.microsoft.com/en-us/Microsoft-edge/webview2/?form=MA13LH

Also, I would be very interested to know if you try this and/or are using it.

Jim

[kdmoyers]

If I had time, I'd be all over this.  It sounds cool.

[td]

Does indeed.

[JTaylor]

A note, in case someone gives this a go...

After a couple of days of searching for the problem, I discovered that one needs to include

       WinShow("WebView2") ; Whatever the caption of your Dialog Title bar might be.

in your code for the browser to show.  Working on fixing this in the Extender code, if possible, but wanted to throw this out there until it was either fixed or documented.  Not sure why it is needed because it can be placed before the browser control is even created.  If you don't do this, all you see is the control that hosts the browser.

Jim

[JTaylor]

That problem should be resolved.   Update posted with that issue fixed plus a few other improvements.  The Help is better as well but still needs work.

https://www.data-n-stuff.com/anonymous/wbOmnibus.zip

Jim

[JTaylor]

Added Click() and Check() methods.

https://www.data-n-stuff.com/anonymous/wbOmnibus.zip

Jim

[JTaylor]

If you use the Navigate Method very much I would suggest a WinBatch Helper function like this to save code.  If you use an EDITBOX event option. You might also want to pass that HWND value add a DialogControlSet() function at the beginning to set the value to something like "NavBegin"

Code Select Expand

#DefineFunction wvBrowse(WV_Handle, obj, url)

    DialogProcOptions(WV_Handle, @dpoDisable,2)
    obj.Navigate(url)
    While obj.IsNavComplete() == @FALSE
    EndWhile
    DialogProcOptions(WV_Handle, @dpoDisable,0)

    Return obj.Document()
   
#EndFunction

[kdmoyers]

THANKS. This sounds like excellent advice, I'll try to tuck it away somewhere I'll find it later when I get back to trying this.  Or maybe insert that snippet into the help?

[JTaylor]

Also excellent advice and done.

Also, I should mention that I am in the midst of integrating this into one of my major applications and, as a result, am finding that all my testing was lacking so I am constantly fixing and improving things so, if one goes to give this a try, please grab the project again to be sure you have the latest.

Jim

[spl]

Attached jpeg is the structure/folder on my E: drive to test Jim's latest post. The folder Webview2 was created and files associated with the webview extender copied. A folder \edge was created and the runtimes were extracted there. Two of the test .wbt files were compiled as small exes and I moved the current WBDYC44I.dll to the folder (might have to change this if you do not have the latest version).

The WBOmniWV2.exe ran as expected. Pretty cool.

One nitpick. When run, Webview data files are placed in user appdata\local under wvWebview2\EBView (and if not there created). However, the EBView folder is also part of the runtimes (so would appear under the \edge folder). Delete that folder (assuming redundancy with the appdata folder) and scripts of compiled exe will hang. Line 50 of the WBOmniWVB2.wbt is

obj = wvCreate(cwin,starting_url, ewin, ".\edge", 0,0)

so I'm assuming that subfolder might hold the datafiles for script runs and the appdata folder is redundant. Again, a minor point and I could have done something wrong to begin with. I plan my own test dialog for https://www.scrapingcourse.com/ecommerce/ which I have referenced in earlier posts and serves as a great site to test links, attributes etc..

All in all, Jim did great work and deserves more responses as he is the first to help out for others.

[JTaylor]

Appreciate the plug.

Assuming I understand and to clarify, the EBWebView folder in the RunTime files only contain a couple of DLLs which are necessary for the browser to run.  The cache for WebView2, which defaults to the "EBWebView" name, which I direct to the AppData path are necessary and will always be created somewhere but can be deleted at any time without any adverse affects, apart from the expected when one clears their browser cache.  If one is going to run multiple instances of the browser then you will want to manage the cache locations.  I provide a mechanism for doing that and it is explained in the Help File.  Couldn't think of a good, automatic, solution that might not come back to bite someone.

Again, thanks for the plug.

Jim

[spl]

Asked and answered. Just wanted to clear up EBView in case anyone else questioned it. As we have discussed in private, I have similar PS scripts running in .net 64-bit which also create EBView in appdata, but instead my folder was called Webview2. When I changed my script to replicate your data folder

New-Variable -Name 'data' -Value "$([Environment]::GetEnvironmentVariable('LOCALAPPDATA'))\wvWebview2" -Scope Global -Force

it worked just as well, so decided to keep your location and can run both yours and mine to datafiles in same location. Also, as we discussed, your click events can return links, so plan a dialog similar to attached where if Links button is pressed links will appear and if one is selected webview will re-navigate to that particular section. Have to consider best way to replicate the gridview, as I think reportview would involve excessive code, but askitemlist() would suffice. Maybe have a working script ready to post over weekend for others to test with your Extender.

[JTaylor]

If any interested parties...

I have posted an update.  Things are much improved and, I believe, relatively solid at this point.

https://www.data-n-stuff.com/anonymous/wbOmnibus.zip

As always, feedback is appreciated, if you give it a go.

Jim

[spl]

Have pestered Jim for last couple of weeks to think more about just a basic scraping variation which employs the Extender [no dialog or web browser view]. While there are options with Selenium or Agility, his Extender is equal to the task and stays entirely within WB. The attached script is a very basic use of the Extender in this capacity to return data from a url. I also attached what should be the result of the script execution as a text file. There is an error-handler to indicate issues others might have running the script and it goes without saying Jim would appreciate that feedback.
But keep in Mind:"

• Script should be run in same folder as the Extender files
• Separate sub-folder .\edge should have the X86 runtimes
• Script should be compiled, but to avoid this place a copy of the .\edge folder under Program Files (X86) Winbatch \system folder and the .wbt can be run in Studio
  

Again, this is very basic web-scraping but works well. You will have to be versed in DOM parsing functions used by Extender (detailed in the help file). If you have followed this thread, the previous post by Jim has the latest update to ensure success of the script. If any might confirm script success I can post more advanced scraping scripts.

[bottomleypotts]

Thanks Jim as always.

And thank you for the background Stan -- I agree regarding webscraping -- the extender seems awesome for this. I appreciate the example - keep them coming as it will make learning this so much more easier -- for when I have time to play.

[spl]

Something worth mentioning about the script I posted. There is an additional option parameter in the wvCreate() function which has to do with cached data. It defaults to @False/don't delete, which I assume might be fine for the dialog view, but for basic scraping obj = wvCreate(0,url,0, ".\edge", "",@true)   

might be more efficient, especially if scraping with recursion. Also, question for Tony - is there a way to unload an Extender, or just [extender]=0 prior to Exit, and would this be beneficial for webview2?

[JTaylor]

In case this affects Tony's reponse, I do close my objects in the "Shutdown" section of the Extender.

Also, in case wondering minds want to know, I made the conscious decision to default to @FALSE on the file deletion solely because I didn't want to be deleting files on someone's system without their permission. I always hate adding such things anyway because I am afraid I will screw it up and delete something I shouldn't.  That is the reason one cannot specify a path for the Temp files, apart from it being a subdirectory. Wanted to avoid someone deleting something they didn't want to delete.

Jim

[spl]

In case this affects Tony's reponse, I do close my objects in the "Shutdown" section of the Extender.

Where is that? I surveyed the help file and searched for 'Shutdown'. So, by adding @true as final parameter for obj = wvCreate(0,url,0, ".\edge", "",@true) in the scraping script, when does the deletion occur. Is it temp files or cached memory? The script does not close the Extender, so are you saying the cache deletion occurs on Exit. Adding @true or leaving the default seems to have no ill effects even after running the script 10 times in repetition, the speed is the same, the text file created and displayed and WB closes in Studio.

Hope I haven't inadvertently stirred up a hornets nest. 

[spl]

Just tested and confirmed a folder structure like attached is all that is needed to run web-scraping with Extender. Of course .\edge folder needs to be copied to the WB \system folder in Program Files (X86) otherwise script would need to be compiled.

[JTaylor]

It is part of the Extender Structure.  Things you want to happen when an Extender is launched and when it is closed, which happens as part of the script exiting.  I was thinking there was a way to unload an extender but haven't found it.

Jim

[spl]

For those interested. The attached script performs recursion on https://books.toscrape.com. You might want to open in a browser to see the main page and the multiple categories. Under the 'several ways to skin the cat' Jim decided recursion by page rather than my attempts at recursion by links.href got the job done, and he was correct. Again, I ran this from a modified folder for just web-scraping with the Extender (as I displayed a couple of posts back) and same rules for running or compiling it.

In spite of the sub-folder files needed, the Extender parses incredibly fast. You may notice in the text file output that some of the book titles end in ... as the title is quite long. This may be an encoding issue (I highly doubt) so maybe some testing will figure it out.

Please test and supply feedback

[td]

It is part of the Extender Structure.  Things you want to happen when an Extender is launched and when it is closed, which happens as part of the script exiting.  I was thinking there was a way to unload an extender but haven't found it.

Jim

Other than IntControl 99?

[spl]

Other than IntControl 99?

oops! Added this to end of both scripts I posted. No error, so no harm. Thanks Tony.

Code Select Expand

ObjectClose(doc)
ObjectClose(obj)
IntControl(99,"wbOmniWV2.dll",0,0,0)
Exit

[JTaylor]

Not sure if there is any interest, and I know there is an official json extender in addition to the one I produced, but I have added JSON parsing to the WebView Extender so if you navigate to a URL that returns JSON, it will detect that fact and the Document() Method will return a JSON document object rather than an HTML document object.   See the JSON section under the WebView2 topic.  I will probably be removing the old JSON, XML and HTML Extenders in the near future.  If you happen to use those make sure you have a backup copy.  Plus, I would love to know if they are in use as I don't mind leaving them.  Just trying to eliminate clutter if they are not being used.

     http://www.data-n-stuff.com/anonymous/wbOmnibus.zip

Jim

[bottomleypotts]

Hey Jim,

If you do remove your code for JSON, can you split it off into a standalone extender please.

[JTaylor]

The old JSON parser is standalone like all of the Extenders.  I use to have them all in one Extender but the conflicts were becoming too great so I broke them out into their own. There are 15 now, if I am counting right. 

There would be no reason to use the old one unless you already have code using it as I have no intention of updating it.  Not to mention the official Extender that is now available.  The WebView Extender (wbOmniWV2.dll) will now handle HTML, XML and JSON parsing.  I am quite proud of this one.  It is slicker than snot on a door knob.  It has solved the many problems created with the death of Internet Explorer.

Appreciate the interest.

    https://www.data-n-stuff.com/anonymous/wbOmnibus.zip

Jim