Login to site and navigating to pages

[hdsouza]

Hi,
I can login to a site by referencing the objects on the page, But once I have logged into I would like to navigate the pages using iHostConnect.
I realize it sounds strange, But I have to navigate several thousand pages on the site, and I want to avoid using msie.navigate and then check if the page is busy or fully loaded as doing is several times is prone to failure.

So I can login to the site:

Code (winbatch) Select Expand


lc_Login = "admin@yahoo.com"
lc_pass = "pass"
url_Note = "https://www.lendingclub.com/account/loanDetail.action?loan_id=3457872"
msie = ObjectOpen("InternetExplorer.Application")
msie.Visible = @True
msie.Navigate(url_Note)
GoSub WaitForMSIE

InputCollection = msie.document.GetElementsByTagName("INPUT")
ForEach Input In InputCollection
   Input.style.border = ".25mm solid red"
   if input.name == "login_email" THEN input.value = lc_Login
   if input.name == "login_password" THEN input.value = lc_Pass
   if input.id == "master_accountSubmit" THEN
      Input.click
      Break
   endif
Next
Exit

:WaitForMSIE
MsieBusy = msie.busy
while msie.busy
    MsieBusy = msie.busy
    ;several additional lines removed to make the script easily readable
endwhile
Return


Now that i am able to login i have several 1000 Loan details to extract and i have to do this one at a time. Navigating with msie.navigate is error prone so I want to try using  iHostConnect to navigate to each site.

Code (winbatch) Select Expand


File_Hostconnect = "c:\temp\ File_Hostconnect.txt"
:HostConnect_Extract
URL_Note1 = strtrim(url_Note)
URL_Temp = strreplace (URL_Note1, "https://", "")
URL_Temp = strreplace (URL_Temp, "http://", "")
Url_Main =  strtrim(Itemextract(1, URL_Temp, "/" ))
Url_Remain = strtrim(strreplace(URL_Temp, "%Url_Main%/", ""))

tophandle=iBegin(0,"","")
connecthandle=iHostConnect(tophandle, Url_Main, @HTTP, "","")
datahandle=iHttpInit(connecthandle, "GET", Url_Remain, "",0)
rslt=iHttpOpen(datahandle, "", 0, 0)
If rslt=="ERROR" || rslt!=200 then
   URL_Error = rslt
   iClose(datahandle)
   iClose(connecthandle)
   iClose(tophandle)
   Page_Error = 1
   Timedelay(1)
EndIf
If Page_Error == 1 then goto HostConnect_Extract
iReadData(datahandle, File_Hostconnect)
iClose(datahandle)
iClose(connecthandle)
iClose(tophandle)


[JTaylor]

Is there a question here?  You have mentioned no problems and asked no questions as far as I can tell.   Does your code not work?  Does it give an error?   Not sure if it will remember the fact that you logged in previously in a different fashion but maybe.   Probably depends on the site.


Also, you might also look at the WinSock extender and the httpGet...() functions.  Might find them useful for the URL parsing.

Jim

[hdsouza]

Hi Jim,
ihostconnect (2 nd script) is not able to access the logged in site. It still believes the site is not logged in.
(Sure I could use the first script and do it all, but as I mentioned earlier doing an MSIE.NAVIGATE over 1000 records is error prone)

[JTaylor]

Assumed that was the case but since you never really said I wasn't sure.    Since I am unfamiliar with the site it is difficult to offer concrete suggestions. 

You can use "click" instead of navigate and see if that works any better.  I do a lot of scraping and am aware of how touchy it can be sometimes.   

There are a number of applications which will download entire sites.  Not sure if any of those would be helpful or not in this context.

If you did a POST request with the credentials you could get to the first page but not sure if you can navigate from there with subsequent calls.  Maybe someone else will chime in with ideas.

Wish I could offer something more definite but to many unknowns.

Jim

[hdsouza]

Thanks Jim.
I guess I may have to go with cookie way.
That being said  i tried setting and then getting the cookie, but it says I am not logged in.
The script  executes correctly till the end but when i open the outfile, i can see its not logged in.

Code (winbatch) Select Expand


AddExtender("WWINT44I.DLL")
outfile = "c:\temp\1.txt"
lc_Login = "admin@yahoo.com"
lc_pass = "password123"
Srvr = 'https://www.lendingclub.com'
iCookieSet(Srvr, lc_Login, lc_pass)

icookieget(Srvr)
hTop  = ibegin(0,"","")
hData = iurlopen(hTop, "https://www.lendingclub.com/account/summary.action")
if hData==0
      Ptr = binaryindexnc(Htm, ptr2, BlkPtr, @fwdscan)
      if Ptr then continue
endif
Success = ireaddata(hData, outfile)
if !Success
   message('Could not read from page!', 'ireaddata(hData, fVid)')
   exit
endif
iclose(hData)
iclose(hTop)
exit


So is it possible to go through all available cookies on the system and find the cookie I am looking for?

[JTaylor]

Was hoping someone else might chime in.  Cookies are files so I assume you could loop through them but have never really done anything with cookies so no certainty.

Have you tried an app that is designed to do what you are asking?  I am guessing you are going to keep hitting a wall going the route you are trying unless the cookie idea works.

Jim

[stanl]

Looking at an earlier part of the thread, the OP said that msie.navigate was prone to error.  There is navigate2() which I posted a script to deal with multiple tabs, and it is also useful when dealing with sites that rely heavily on AJAX and hidden DIV's...  just .02

[hdsouza]

Thanks Stan, Jim.
Here is why I am saying that the  msie.navigate approach is error prone:
I am writing a main winbatch script  (Parent) which will call other winbatch scripts (child scripts). The child scripts will be doing the heavy lifting with IE. When different IEs are opened from different scripts, at times, believe it or not , the window ID is the same (frequently). When this happens child scripts  fight each other over the same IE browser. A fallout from this is Orphaned IE browsers. Then there are also the browser load times. And if I ran 8 such child scrips it makes the error handling process even more complex. I have written similar parent-child scripts in the past and  have ways around these problems, but its never completely clean, even after writing tons of loops for error handling.
Thats why I was hoping to get some flavor of iHostConnect or icookie to work.. or maybe some other approach which does not need opening a browser.

Thanks
Hil

[JTaylor]

Weird...I often call multiple IE instances and don't recall encountering this problem.   Wish I could be more help.

Jim