suggestions for speed up

Started by MW4, October 23, 2017, 08:36:27 AM

Previous topic - Next topic

MW4

I have the following code which goes through a folder and based upon the create date, it puts the pdf's into a zip file by quarter.

It works, it's just painfully slow.

Any suggestions?


Code (winbatch) Select
AddExtender( "WWZIP44I.DLL" )

GoSub udfs

intDaysOld = 120
intDateType = 4
strRootDir = "Y:\FFiles\ROS\" ; '.\'

arrFileInfo = udfArrFileOlderThan (strRootDir, intDaysOld, intDateType)

arrLen = ArrInfo( arrFileInfo, 1 )

DirChange("\\fdc01\FArchive\FFiles\Archived ROS")

For xxx = 0 to arrLen -1

zZipFiles( "m" , strcat("ROS Archive ",udfgetqtr(arrFileInfo[xxx,4]),".zip") , arrFileInfo[xxx,0] , "" )

next

message("Zip Process","Finished")

Exit

:udfs
;--------------------------------------------------------------------------------;
;    ;
;                    udfArrFileOlderThan : START OF FUNCTION                     ;
;                                                                                ;
;--------------------------------------------------------------------------------;
; udfArrFileOlderThan : Returns an array of files older than a specified day     ;
;--------------------------------------------------------------------------------;
; strRootDir  : Directory path that contains files                               ;
; intDaysOld  : Number of days old (90 days).                                    ;
; intDateType : LastModified=2 LastAccessed=3 LastCreated=4                      ;
;--------------------------------------------------------------------------------;
; returns     : Array of file information of files older than specified date     ;
;--------------------------------------------------------------------------------;

#DefineFunction udfArrFileOlderThan (strRootDir, intDaysOld, intDateType)

; Check Parameters.
If !DirExist (strRootDir)
   Pause ('Error:udfFileOlderThan', 'Invalid directory specified.')
   Exit
EndIf
If !IsNumber (intDaysOld)
   Pause ('Error:udfFileOlderThan', 'Invalid number of days specified.')
   Exit
EndIf
If intDateType < 2 || intDateType > 4
   Pause ('Error:udfFileOlderThan', 'Invalid DateType Specified')
   Exit
EndIf

; Get all file data from RootDir.
arrFileInfo = FileInfoToArray (strRootDir : '*', 1 | 2)
; Remove first row.
ArrayRemove (arrFileInfo, 0, 1)

; Now loop thru array removing all files newer files.
intI = 0
While intI < ArrInfo (arrFileInfo, 1)
   strDTLastMod = arrFileInfo[intI, intDateType]
   intDiff = TimeDiffDays (TimeYmdHms (), strDTLastMod)
   ;Remove file data from array if newer than specified days.
   If intDiff < intDaysOld
      ArrayRemove (arrFileInfo, intI, 1)
   Else
      intI = intI + 1
   EndIf
EndWhile
Return arrFileInfo
#EndFunction

;--------------------------------------------------------------------------------;
;    ;
;                    udfArrFileOlderThan : END OF FUNCTION                       ;
;                                                                                ;
;--------------------------------------------------------------------------------;


;--------------------------------------------------------------------------------;
;    ;
;                    udfGetQtr : START OF FUNCTION                               ;
;                                                                                ;
;--------------------------------------------------------------------------------;
; udfGetQtr   : Returns a string as 1Q, 2Q, 3Q, 4Q                               ;
;--------------------------------------------------------------------------------;
; date4Qtr    : Date that a quarter is needed                                    ;
;--------------------------------------------------------------------------------;
; returns     : Returns a string as 1Q, 2Q, 3Q, 4Q ;
;--------------------------------------------------------------------------------;
#DefineFunction udfGetQtr( date4Qtr )

DS_MM   = StrSub(date4Qtr, 6, 2)
DS_YY = StrSub(date4Qtr, 1, 4)
If DS_MM < 4 then
QTR = "1Q"
else
if DS_MM < 7 then
QTR = "2Q"
else
if DS_MM < 10 then
QTR = "3Q"
else
QTR = "4Q"
endif
endif
endif

return strcat(DS_YY,"-",QTR)

#EndFunction
;--------------------------------------------------------------------------------;
;    ;
;                    udfGetQtr : END OF FUNCTION                                 ;
;                                                                                ;
;--------------------------------------------------------------------------------;
;--------------------------------------------------------------------------------;
;--------------------------------------------------------------------------------;
Return ;Return from GoSub udfs Line 4

MW4

I was thinking also of restructuring how it runs something along the lines of first moving them to a folder for each year and quarter then zipping all files in that folder to that archive.
Not sure if that would be any faster

td

You might want to consider first determining where the execution time is being spent.  The zZipFiles function is likely the biggest CPU cycles burner but by what percentage?  If the function is consuming say for the sake of argument 90% of the total script time, then taking the function out of a loop may benefit you.  If however processing the 'for-next' loop statements is taking up 20% or more of the script time, then moving the function out of the loop may not help much because you still have to have a loop to move the files by their age.  It is difficult to say whether or not calling zZipFiles once would be all that much faster than calling it for each file.  It may very well be a lot faster but you would have to do a comparative test to find out.

The zZipFiles function got slower after we updated the extender to the latest version of Info-ZIP about six years ago because users wanted support for bigger files and archives, and support for password protected archives. 

The old 34i version of the extender is still around and can be found by ftping to files.winbatch.com and moving to the downloads\wb sub-directory.  The file name is wwzip34i.zip.  The old version of zZipFiles has been reported to be faster but does have limited file size support and lacks multiple enhancements.  Don't know whether or not it would be that big of an improvement so again, you would have to try it to find out.
"No one who sees a peregrine falcon fly can ever forget the beauty and thrill of that flight."
  - Dr. Tom Cade