viewpoint-particle

Author Topic: Script execution speed  (Read 3990 times)

kdmoyers

  • Full Member
  • ***
  • Posts: 227
Re: Script execution speed
« Reply #35 on: January 26, 2018, 07:46:57 am »
Before we leave off, I'd like to request that larger brains than mine consider this performance request:
Code: Winbatch
;Presently, the BinaryTag loop works like this:

    struc = BinaryTagInit(bb,@lf,@cr)
    while 1
      struc = BinaryTagFind(struc)
      if struc == "" then break
 
    endwhile

;But it seems like it would be faster if it worked like this:

    struc = BinaryTagInit(bb,@lf,@cr)
    while BinaryTagFind(struc)
 
    endwhile

;I don't know if that's possible.
 
The mind is everything; What you think, you become.

JTaylor

  • Pundit
  • *****
  • Posts: 922
    • Data & Stuff Inc.
Re: Script execution speed
« Reply #36 on: January 26, 2018, 07:57:48 am »
...or maybe...

    ForEach struc in BinaryTagInit(bb,@lf,@cr,@TRUE)
        .....
    Next

Last Parameter an optional one indicating if it is a Collection or not, for backwards compatibility.

Jim

kdmoyers

  • Full Member
  • ***
  • Posts: 227
Re: Script execution speed
« Reply #37 on: January 26, 2018, 10:57:18 am »
<< ForEach struc in ... >>

Oooohh, fancy!
The mind is everything; What you think, you become.

td

  • Tech Support
  • *****
  • Posts: 2629
    • WinBatch
Re: Script execution speed
« Reply #38 on: January 26, 2018, 01:49:47 pm »
Either approach would certainly look prettier but the performance gain would likely be negligible.   Pretty syntax has its place but it also often hides ugly secrets...   
"Success is a lousy teacher. It seduces smart people into thinking they can't lose."
  - Bill Gates


td

  • Tech Support
  • *****
  • Posts: 2629
    • WinBatch
Re: Script execution speed
« Reply #39 on: January 26, 2018, 01:53:38 pm »
One possible compromise would be to allow assignment in while statements.

While struc = BinaryTagFind(struc)
   whatever
Endwhile

But that could be a bit of a challenge to implement.
"Success is a lousy teacher. It seduces smart people into thinking they can't lose."
  - Bill Gates


JTaylor

  • Pundit
  • *****
  • Posts: 922
    • Data & Stuff Inc.
Re: Script execution speed
« Reply #40 on: January 26, 2018, 07:44:41 pm »
Figured there wouldn't be any gains on performance but assume having a Collection would, I assume, open up some potentially useful possibilities in addition to being pretty.  In any event, I do hope you find time to implement the prior suggestions on the BinaryTag front.

Jim


Either approach would certainly look prettier but the performance gain would likely be negligible.   Pretty syntax has its place but it also often hides ugly secrets...   

JTaylor

  • Pundit
  • *****
  • Posts: 922
    • Data & Stuff Inc.
Re: Script execution speed
« Reply #41 on: January 27, 2018, 10:08:40 am »
Guess I should clarify...these changes would be nice but I meant the ones requested sometime back in a different thread.

Jim

hdsouza

  • Full Member
  • ***
  • Posts: 144
Re: Script execution speed
« Reply #42 on: January 28, 2018, 09:03:23 am »
There is a reference to .net and winbatch,  http://techsupt.winbatch.com/webcgi/webbatch.exe?techsupt/nftechsupt.web+WinBatch/OLE~COM~ADO~CDO~ADSI~LDAP+WinBatch~and~Dot~NET.txt but its not clear whether it should be working or not.. or maybe I am not reading correctly. I mention .net as the initial PS code was .net code in PS.

Also while the PS code is still faster than wbt, Should the input file  be large and a lot of data is being written to the output file, that will slow the speed down. So should anyone try to use the code, here is why:
As we add more lines to the $NotesList the script takes longer and longer. I believe .NET is duplicating the entire array in memory, adding the new item, and deleting the old copy in memory. It is explained here along with a solution, but for now I cannot get it to work. https://blogs.technet.microsoft.com/ashleymcglone/2017/07/12/slow-code-top-5-ways-to-make-your-powershell-scripts-run-faster/
I did go a different route in splitting the $NotesList into smaller parts and then combine the parts together at the end with a FOR loop... although I am thinking the earlier solution maybe better.

Not trying to pollute the winbatch forum with PS code,  but I noticed interest on the post for a quicker solution. So if someone should use the initial code, this may get better speeds (at least for now :) )

Code: PowerShell
$File_Path = "C:\Temp\"
$File_NotesDownload = $File_Path  + "SecondaryMarketAllNotes.csv"
$File_NotesEval     = $File_Path  + "SecNotesEval.txt"
if (Test-Path $File_NotesEval)     { remove-item $File_NotesEval }

$Start = Get-Date
$r = [IO.File]::OpenText($File_NotesDownload)
$Count_EvalNotes = 0
$CountOutFile = 0
while ($r.Peek() -ge 0)
    {
     $Note = $r.ReadLine()
    $Note = ($Note -replace '"','').ToLower()
    $Loan_Id = ($Note -split ',')[0].trim()
    $Loan_Id = $Loan_Id -as [int]
    if (($Loan_Id -is [int])  -eq $false) {Continue}

    $NeverLate         = ($Note -split ',')[13].trim()
    if (($Loan_StatusValue -eq "current") -and ($NeverLate -ne "true")) {Continue}

    $Ask_price       = [float]($Note -split ',')[6].trim()
    if ($Ask_price -gt 25) {Continue}

    # several of the filters have been removed ......
    # several of the filters have been removed ......

    $Count_EvalNotes = $Count_EvalNotes + 1
    $ModCount_EvalNotes = $Count_EvalNotes/200
    $index = "$ModCount_EvalNotes".IndexOf(".")

    if (($index -eq -1) -or ($Count_EvalNotes -eq 1))
       {
       $CountOutFile = $CountOutFile + 1
       New-Variable -Name "List_$CountOutFile"
       }

    $NotesList = $Note + "`r`n"
    $val=Get-Variable -Name "List_$CountOutFile" -ValueOnly
    Set-Variable -Name "List_$CountOutFile" -value $val$NotesList
    }
for ($i=1; $i -le $CountOutFile; $i++)
    {
    $val=Get-Variable -Name "List_$i" -ValueOnly
    $val  >> $File_NotesEval
    }

$r.Dispose()
$End = Get-Date
$TotalTime = New-Timespan -Start $Start -End $End
$TotalTimeSecs = $TotalTime.TotalSeconds
$TotalTimeSecs
 

stanl

  • Pundit
  • *****
  • Posts: 801
Re: Script execution speed
« Reply #43 on: January 28, 2018, 01:27:01 pm »
Like I posted, way back in the thread - make your choice. The pyrotechnics over arrays and processing speed were something that has always happened on this board and for the betterment of WB and society. I have never been concerned with the speed of getting data, only with validation and error handling...   :o

kdmoyers

  • Full Member
  • ***
  • Posts: 227
Re: Script execution speed
« Reply #44 on: January 30, 2018, 05:43:30 am »
One possible compromise would be to allow assignment in while statements.

While struc = BinaryTagFind(struc)
   whatever
Endwhile

But that could be a bit of a challenge to implement.

Hmmm...

Maybe if BinaryTagFind took a pointer? Maybe it would operate directly on that parameter, modifying it.  Also, it could return a simple true/false. So the code becomes:

Code: Winbatch
While BinaryTagFind(&struc)
   whatever
Endwhile

I dunno.  Probably not very important in the grand scheme! (smile)
The mind is everything; What you think, you become.

td

  • Tech Support
  • *****
  • Posts: 2629
    • WinBatch
Re: Script execution speed
« Reply #45 on: January 30, 2018, 06:26:33 am »
Ideas are always appreciated and so keeping them coming but pointers and references are expensive in WnBatch so it certainly wouldn't be a performance enhancer.  It would be quite the opposite.    There are several other issues that I won't go into.     
"Success is a lousy teacher. It seduces smart people into thinking they can't lose."
  - Bill Gates


kdmoyers

  • Full Member
  • ***
  • Posts: 227
Re: Script execution speed
« Reply #46 on: January 30, 2018, 08:56:59 am »
pointers and references are expensive in WinBatch so it certainly wouldn't be a performance enhancer
Yup, you're right of course. I use them all the time, but I hardly ever do speed-sensitive code, so I had forgotten they are not quick.
The mind is everything; What you think, you become.

kdmoyers

  • Full Member
  • ***
  • Posts: 227
Re: Script execution speed
« Reply #47 on: January 30, 2018, 12:44:45 pm »
I just realized I can write it myself.  Cool - it works! But you are right -- it's slower, not faster.
Code: Winbatch
#definefunction BinaryTagFindEx(pStructure)
  *pStructure = BinaryTagFind(*pStructure)
  return *pStructure != ""
#endfunction
 
The mind is everything; What you think, you become.

td

  • Tech Support
  • *****
  • Posts: 2629
    • WinBatch
Re: Script execution speed
« Reply #48 on: January 30, 2018, 01:36:11 pm »
You have the UDF overhead too but the reason references take more time than plain variables is that the interpreter has to parse the pointer and make two trips to the variable instead of one.   In the vast majority of scripts, this overhead is insignificant but like everything else, it adds up in a tight loop.
"Success is a lousy teacher. It seduces smart people into thinking they can't lose."
  - Bill Gates