Streamlining Code

[oradba4u]

All:
Does anyone have a document (or know where there is one) that can help me streamline my code to execute faster?
I mean tips, tricks, and traps (things to avoid) to shorten the code, or make it faster.
Is there a program that will help me capture/analyze the performance of my Winbatch programs and point out where some "traffic jams" are occurring?

Thanks in advance,
a Flash wannabe.

[ChuckC]

Some starting points for you...

Examine looping constructs that repeatedly perform tasks inside the loop that can be done just a single time [outside the loop] before iteration begins.  Any task/step/operation that has a high latency should not be performed repeatedly unless absolutely necessary.  High latency can be associated with network communications, database interactions and file system activity, among other things.

GetTickCount() may be of some assistance for measuring how long it takes a specific portion of a script to execute.  However, it has approximately millisecond precision so that may not be suitable for segments of code that execute quite quickly.  It is, however, reasonably suitable for measuring the overall execution time of an loop from its beginnning until its end in many cases.


I'm sure others will chime in with additional thoughts & suggestions.


If you have more specific questions about particular coding techniques or usage cases for API functions, COM interfaces or .NET, post examples of code that you have written when asking those questions.

[kdmoyers]

Speed things I usually have good luck with:

Anything that does any sort of file I/O takes F O R E V E R.  Try to get it out of your loops.

Anything that does network or screen I/O is slow.  Try to get it out of your loops.

Stare at your code -- what things get pointlessly recomputed each time around the loop?
Move those things outside the loop.

Be careful of loops within loops.  Sometimes this results in a wild number of iterations
that you were not expecting.

Broadly speaking, try to reduce the number of lines of code that execute.  Winbatch
has some overhead per line of code.

Simplify the essential kernel that must run fast.  Try to remove function calls from
it it you can.  If the operation was simple, just do it right there in the main
program and avoid the overhead of function call.

If you are desperate, there is a technique called loop unrolling: if you are only doing a loop
five times, maybe just explicitly do the five things and avoid the overhead of the loop.

Think about your overall algorithm -- is there a faster way?  Sometimes a change of tactics
can result in a stunning performance improvement.

[ChuckC]

"Back in the day", the organization of arrays in memory was a concern when it came to the issue of "row-wise vs. column-wise" processing, and how that carries to higher order performance issues with arrays with a rank greater than 2.

And, upon reflection, despite having used WinBatch since circa 1995-1996, I don't recall ever having asked how WIL arrays are stored in memory what access patterns are affected by accessing them...

Seems like a good time for Tony to check in...

[td]

WIL arrays are O(n). The array's rank does not change this.

One interesting optimization in WIL that is a bit counter-intuitive is using user-defined subroutines (not user-defined functions) in loops with structured "if" statements inside the loop.  The idea is to place the lines of code inside the "if" block in a subroutine and call the subroutine containing the lines in the "if" block as the only statement in the block or call the subroutine from an immediate "if" statement. This works were the following criteria are met:

• The loop is repeated many times.
• The loop contains more than a few lines inside a structured "if" block.
• The structured "if" statement frequently does not evaluate to "@True".

 
The performance gained by using this technique varies widely based on how closely the optimized script reflects the above criteria.

[edit] WIL arrays are O(1) and not O(n).

[kdmoyers]

Fascinating.  Sounds like the sort of optimization I'd flub though.
I once worked for hours on a speedup idea only to measure and
find I'd made my program 10x slower! (laugh)
-Kirby

[stanl]

Fascinating.  Sounds like the sort of optimization I'd flub though.

Agreed fascinating. My biggest mistake was looking at Worksheetfunction transpose() for optimization.

[kdmoyers]

Not that anybody asked, but I do a lot of scanning big text files. 
My fastest algorithm for this is below.
It does 411K lines in 18 seconds on my cheezy PC.

Code: Winbatch

  InFile = "SomeLargeFile.txt"

  AddExtender("wilx44i.dll",0,"wilx64i.dll")

  FileSiz = filesize(InFile)                ; size of file
  BB = binaryalloc(FileSiz)                 ; reserve space
  binaryreadEX(BB,0,InFile,0,FileSiz)       ; gulp the file
  LoopStruct = binarytaginit(BB, "", @lf)   ; set up to scan text lines
  LineNum = 0
  while 1
    LoopStruct = binarytagfind(LoopStruct)  ; advance to next line
    if LoopStruct == "" then break          ; all done?
    LineNum += 1

    TheLine = binarytagextr(LoopStruct,1)   ; the line text
    LinePtr = xhex(strsub(LoopStruct,5,12)) ; optional: binary offset

    ; OK do stuff here

  endwhile
  message('line count', LineNum)

  exit

[td]

Fascinating.  Sounds like the sort of optimization I'd flub though.
I once worked for hours on a speedup idea only to measure and
find I'd made my program 10x slower! (laugh)
-Kirby

The only hard part is determining whether or not you have a section of a script that would be enhanced by the modification.  A script that parses dialog templates that is a part of the WinBatch Studio context menu implementation saw a 20% increase in performance using the subroutine technique.  Basically, places where it helps are not that common but it does produce a significant boost when you find a case with a need for speed.

[stanl]

Moving from theory.... perhaps we should scroll back to the original request from the OP who wanted to streamline his code. Going back to Sep 2019 the Op opened a thread with



Hello:
I want to "scrape" the following table (see attachment) from this website:
https://www.baseball-reference.com/teams/CIN/1975-batting-orders.shtml


which got a lot of hits and responses.  Perhaps the OP will post any code he eventually came up with and y'all can help streamline.

[kdmoyers]

Moving from theory....  and y'all can help streamline.

Oh, sorry. Yes, sure thing.

[stanl]

Moving from theory....  and y'all can help streamline.

Oh, sorry. Yes, sure thing.

Seriously... nothing personal. My personal opinion about streamlining code is optimizing readability [and guilty as charged, having written a lot of crap]. Working to cut a few milliseconds from a process at the expense of making the code more arcane for someone inheriting it not always the best strategy.

[td]

Seriously... nothing personal. My personal opinion about streamlining code is optimizing readability [and guilty as charged, having written a lot of crap]. Working to cut a few milliseconds from a process at the expense of making the code more arcane for someone inheriting it not always the best strategy.

Couldn't agree more. There are times when implementing an optimization is justified by a big payoff. Knowing when and where to use optimization techniques via analysis is a skill learned through experience.