Script execution speed

[kdmoyers]

Before we leave off, I'd like to request that larger brains than mine consider this performance request:

Code (winbatch) Select Expand


;Presently, the BinaryTag loop works like this:

    struc = BinaryTagInit(bb,@lf,@cr)
    while 1
      struc = BinaryTagFind(struc)
      if struc == "" then break
 
    endwhile

;But it seems like it would be faster if it worked like this:

    struc = BinaryTagInit(bb,@lf,@cr)
    while BinaryTagFind(struc)
 
    endwhile

;I don't know if that's possible.


[JTaylor]

...or maybe...

    ForEach struc in BinaryTagInit(bb,@lf,@cr,@TRUE)
        .....
    Next

Last Parameter an optional one indicating if it is a Collection or not, for backwards compatibility.

Jim

[kdmoyers]

<< ForEach struc in ... >>

Oooohh, fancy!

[td]

Either approach would certainly look prettier but the performance gain would likely be negligible.   Pretty syntax has its place but it also often hides ugly secrets...   

[td]

One possible compromise would be to allow assignment in while statements.

While struc = BinaryTagFind(struc)
   whatever
Endwhile

But that could be a bit of a challenge to implement.

[JTaylor]

Figured there wouldn't be any gains on performance but assume having a Collection would, I assume, open up some potentially useful possibilities in addition to being pretty.  In any event, I do hope you find time to implement the prior suggestions on the BinaryTag front.

Jim

Either approach would certainly look prettier but the performance gain would likely be negligible.   Pretty syntax has its place but it also often hides ugly secrets...   

[JTaylor]

Guess I should clarify...these changes would be nice but I meant the ones requested sometime back in a different thread.

Jim

[hdsouza]

There is a reference to .net and winbatch,  http://techsupt.winbatch.com/webcgi/webbatch.exe?techsupt/nftechsupt.web+WinBatch/OLE~COM~ADO~CDO~ADSI~LDAP+WinBatch~and~Dot~NET.txt but its not clear whether it should be working or not.. or maybe I am not reading correctly. I mention .net as the initial PS code was .net code in PS.

Also while the PS code is still faster than wbt, Should the input file  be large and a lot of data is being written to the output file, that will slow the speed down. So should anyone try to use the code, here is why:
As we add more lines to the $NotesList the script takes longer and longer. I believe .NET is duplicating the entire array in memory, adding the new item, and deleting the old copy in memory. It is explained here along with a solution, but for now I cannot get it to work. https://blogs.technet.microsoft.com/ashleymcglone/2017/07/12/slow-code-top-5-ways-to-make-your-powershell-scripts-run-faster/
I did go a different route in splitting the $NotesList into smaller parts and then combine the parts together at the end with a FOR loop... although I am thinking the earlier solution maybe better.

Not trying to pollute the winbatch forum with PS code,  but I noticed interest on the post for a quicker solution. So if someone should use the initial code, this may get better speeds (at least for now :) )

Code (powershell) Select Expand


$File_Path = "C:\Temp\"
$File_NotesDownload = $File_Path  + "SecondaryMarketAllNotes.csv"
$File_NotesEval     = $File_Path  + "SecNotesEval.txt"
if (Test-Path $File_NotesEval)     { remove-item $File_NotesEval }

$Start = Get-Date
$r = [IO.File]::OpenText($File_NotesDownload)
$Count_EvalNotes = 0
$CountOutFile = 0
while ($r.Peek() -ge 0)
    {
     $Note = $r.ReadLine()
    $Note = ($Note -replace '"','').ToLower()
    $Loan_Id = ($Note -split ',')[0].trim()
    $Loan_Id = $Loan_Id -as [int]
    if (($Loan_Id -is [int])  -eq $false) {Continue}

    $NeverLate         = ($Note -split ',')[13].trim()
    if (($Loan_StatusValue -eq "current") -and ($NeverLate -ne "true")) {Continue}

    $Ask_price       = [float]($Note -split ',')[6].trim()
    if ($Ask_price -gt 25) {Continue}

    # several of the filters have been removed ......
    # several of the filters have been removed ......

    $Count_EvalNotes = $Count_EvalNotes + 1
    $ModCount_EvalNotes = $Count_EvalNotes/200
    $index = "$ModCount_EvalNotes".IndexOf(".")

    if (($index -eq -1) -or ($Count_EvalNotes -eq 1))
       {
       $CountOutFile = $CountOutFile + 1
       New-Variable -Name "List_$CountOutFile"
       }

    $NotesList = $Note + "`r`n"
    $val=Get-Variable -Name "List_$CountOutFile" -ValueOnly
    Set-Variable -Name "List_$CountOutFile" -value $val$NotesList
    }
for ($i=1; $i -le $CountOutFile; $i++)
    {
    $val=Get-Variable -Name "List_$i" -ValueOnly
    $val  >> $File_NotesEval
    }

$r.Dispose()
$End = Get-Date
$TotalTime = New-Timespan -Start $Start -End $End
$TotalTimeSecs = $TotalTime.TotalSeconds
$TotalTimeSecs


[stanl]

Like I posted, way back in the thread - make your choice. The pyrotechnics over arrays and processing speed were something that has always happened on this board and for the betterment of WB and society. I have never been concerned with the speed of getting data, only with validation and error handling...   :o

[kdmoyers]

One possible compromise would be to allow assignment in while statements.

While struc = BinaryTagFind(struc)
   whatever
Endwhile

But that could be a bit of a challenge to implement.

Hmmm...

Maybe if BinaryTagFind took a pointer? Maybe it would operate directly on that parameter, modifying it.  Also, it could return a simple true/false. So the code becomes:

Code (winbatch) Select Expand

While BinaryTagFind(&struc)
   whatever
Endwhile

I dunno.  Probably not very important in the grand scheme! (smile)

[td]

Ideas are always appreciated and so keeping them coming but pointers and references are expensive in WnBatch so it certainly wouldn't be a performance enhancer.  It would be quite the opposite.    There are several other issues that I won't go into.     

[kdmoyers]

pointers and references are expensive in WinBatch so it certainly wouldn't be a performance enhancer

Yup, you're right of course. I use them all the time, but I hardly ever do speed-sensitive code, so I had forgotten they are not quick.

[kdmoyers]

I just realized I can write it myself.  Cool - it works! But you are right -- it's slower, not faster.

Code (winbatch) Select Expand

#definefunction BinaryTagFindEx(pStructure)
  *pStructure = BinaryTagFind(*pStructure)
  return *pStructure != ""
#endfunction


[td]

You have the UDF overhead too but the reason references take more time than plain variables is that the interpreter has to parse the pointer and make two trips to the variable instead of one.   In the vast majority of scripts, this overhead is insignificant but like everything else, it adds up in a tight loop.