Script execution speed

Started by hdsouza, January 21, 2018, 06:19:01 AM

Previous topic - Next topic

kdmoyers

Before we leave off, I'd like to request that larger brains than mine consider this performance request:
Code (winbatch) Select

;Presently, the BinaryTag loop works like this:

    struc = BinaryTagInit(bb,@lf,@cr)
    while 1
      struc = BinaryTagFind(struc)
      if struc == "" then break
 
    endwhile

;But it seems like it would be faster if it worked like this:

    struc = BinaryTagInit(bb,@lf,@cr)
    while BinaryTagFind(struc)
 
    endwhile

;I don't know if that's possible.
The mind is everything; What you think, you become.

JTaylor

...or maybe...

    ForEach struc in BinaryTagInit(bb,@lf,@cr,@TRUE)
        .....
    Next

Last Parameter an optional one indicating if it is a Collection or not, for backwards compatibility.

Jim

kdmoyers

<< ForEach struc in ... >>

Oooohh, fancy!
The mind is everything; What you think, you become.

td

Either approach would certainly look prettier but the performance gain would likely be negligible.   Pretty syntax has its place but it also often hides ugly secrets...   
"No one who sees a peregrine falcon fly can ever forget the beauty and thrill of that flight."
  - Dr. Tom Cade

td

One possible compromise would be to allow assignment in while statements.

While struc = BinaryTagFind(struc)
   whatever
Endwhile

But that could be a bit of a challenge to implement.
"No one who sees a peregrine falcon fly can ever forget the beauty and thrill of that flight."
  - Dr. Tom Cade

JTaylor

Figured there wouldn't be any gains on performance but assume having a Collection would, I assume, open up some potentially useful possibilities in addition to being pretty.  In any event, I do hope you find time to implement the prior suggestions on the BinaryTag front.

Jim


Quote from: td on January 26, 2018, 01:49:47 PM
Either approach would certainly look prettier but the performance gain would likely be negligible.   Pretty syntax has its place but it also often hides ugly secrets...   

JTaylor

Guess I should clarify...these changes would be nice but I meant the ones requested sometime back in a different thread.

Jim

hdsouza

There is a reference to .net and winbatch,  http://techsupt.winbatch.com/webcgi/webbatch.exe?techsupt/nftechsupt.web+WinBatch/OLE~COM~ADO~CDO~ADSI~LDAP+WinBatch~and~Dot~NET.txt but its not clear whether it should be working or not.. or maybe I am not reading correctly. I mention .net as the initial PS code was .net code in PS.

Also while the PS code is still faster than wbt, Should the input file  be large and a lot of data is being written to the output file, that will slow the speed down. So should anyone try to use the code, here is why:
As we add more lines to the $NotesList the script takes longer and longer. I believe .NET is duplicating the entire array in memory, adding the new item, and deleting the old copy in memory. It is explained here along with a solution, but for now I cannot get it to work. https://blogs.technet.microsoft.com/ashleymcglone/2017/07/12/slow-code-top-5-ways-to-make-your-powershell-scripts-run-faster/
I did go a different route in splitting the $NotesList into smaller parts and then combine the parts together at the end with a FOR loop... although I am thinking the earlier solution maybe better.

Not trying to pollute the winbatch forum with PS code,  but I noticed interest on the post for a quicker solution. So if someone should use the initial code, this may get better speeds (at least for now :) )

Code (powershell) Select

$File_Path = "C:\Temp\"
$File_NotesDownload = $File_Path  + "SecondaryMarketAllNotes.csv"
$File_NotesEval     = $File_Path  + "SecNotesEval.txt"
if (Test-Path $File_NotesEval)     { remove-item $File_NotesEval }

$Start = Get-Date
$r = [IO.File]::OpenText($File_NotesDownload)
$Count_EvalNotes = 0
$CountOutFile = 0
while ($r.Peek() -ge 0)
    {
     $Note = $r.ReadLine()
    $Note = ($Note -replace '"','').ToLower()
    $Loan_Id = ($Note -split ',')[0].trim()
    $Loan_Id = $Loan_Id -as [int]
    if (($Loan_Id -is [int])  -eq $false) {Continue}

    $NeverLate         = ($Note -split ',')[13].trim()
    if (($Loan_StatusValue -eq "current") -and ($NeverLate -ne "true")) {Continue}

    $Ask_price       = [float]($Note -split ',')[6].trim()
    if ($Ask_price -gt 25) {Continue}

    # several of the filters have been removed ......
    # several of the filters have been removed ......

    $Count_EvalNotes = $Count_EvalNotes + 1
    $ModCount_EvalNotes = $Count_EvalNotes/200
    $index = "$ModCount_EvalNotes".IndexOf(".")

    if (($index -eq -1) -or ($Count_EvalNotes -eq 1))
       {
       $CountOutFile = $CountOutFile + 1
       New-Variable -Name "List_$CountOutFile"
       }

    $NotesList = $Note + "`r`n"
    $val=Get-Variable -Name "List_$CountOutFile" -ValueOnly
    Set-Variable -Name "List_$CountOutFile" -value $val$NotesList
    }
for ($i=1; $i -le $CountOutFile; $i++)
    {
    $val=Get-Variable -Name "List_$i" -ValueOnly
    $val  >> $File_NotesEval
    }

$r.Dispose()
$End = Get-Date
$TotalTime = New-Timespan -Start $Start -End $End
$TotalTimeSecs = $TotalTime.TotalSeconds
$TotalTimeSecs


stanl

Like I posted, way back in the thread - make your choice. The pyrotechnics over arrays and processing speed were something that has always happened on this board and for the betterment of WB and society. I have never been concerned with the speed of getting data, only with validation and error handling...   :o

kdmoyers

Quote from: td on January 26, 2018, 01:53:38 PM
One possible compromise would be to allow assignment in while statements.

While struc = BinaryTagFind(struc)
   whatever
Endwhile

But that could be a bit of a challenge to implement.

Hmmm...

Maybe if BinaryTagFind took a pointer? Maybe it would operate directly on that parameter, modifying it.  Also, it could return a simple true/false. So the code becomes:

Code (winbatch) Select
While BinaryTagFind(&struc)
   whatever
Endwhile


I dunno.  Probably not very important in the grand scheme! (smile)
The mind is everything; What you think, you become.

td

Ideas are always appreciated and so keeping them coming but pointers and references are expensive in WnBatch so it certainly wouldn't be a performance enhancer.  It would be quite the opposite.    There are several other issues that I won't go into.     
"No one who sees a peregrine falcon fly can ever forget the beauty and thrill of that flight."
  - Dr. Tom Cade

kdmoyers

Quote from: td on January 30, 2018, 06:26:33 AM
pointers and references are expensive in WinBatch so it certainly wouldn't be a performance enhancer
Yup, you're right of course. I use them all the time, but I hardly ever do speed-sensitive code, so I had forgotten they are not quick.
The mind is everything; What you think, you become.

kdmoyers

I just realized I can write it myself.  Cool - it works! But you are right -- it's slower, not faster.
Code (winbatch) Select
#definefunction BinaryTagFindEx(pStructure)
  *pStructure = BinaryTagFind(*pStructure)
  return *pStructure != ""
#endfunction
The mind is everything; What you think, you become.

td

You have the UDF overhead too but the reason references take more time than plain variables is that the interpreter has to parse the pointer and make two trips to the variable instead of one.   In the vast majority of scripts, this overhead is insignificant but like everything else, it adds up in a tight loop.
"No one who sees a peregrine falcon fly can ever forget the beauty and thrill of that flight."
  - Dr. Tom Cade