viewpoint-particle

Author Topic: WB still excellent web scraper  (Read 317 times)

stanl

  • Pundit
  • *****
  • Posts: 668
WB still excellent web scraper
« on: May 20, 2017, 10:15:42 am »
Need to put Tony in a better mood >:(

I took on a task to get a data field's (numeric code) description ( multi line text) from a web url, where you must input 1 numeric code and the results pop out as a javascript. Having not done this in 4 years javascript urls get pretty sophisticated...  this one blocked forward and backward caching, and all of the hrefs pointed back to the same menu. I was looking at a Python add-in and script or working with Firefox, or trying to figure out the jquery/AJAX stuff in the site. I then noticed that in spite of all the sophistication, the basic navigation was BaseURL: \Search\ : input number.  This added the secret sauce to the hrefs and by re-navigating to the results from the search I got what I needed.  Basically 2 navigates with basic WB code - in 20 lines of code.

I have been with WB since 1998. It never fails in a pinch. I'm too old to be trying to replace it with Python, Ruby or Visual Studio - I have worked with them but I don't have the flow and would not be competitive. Scripting and WB in particular helped pay off our small house and pay off our daughter's Duke loans.