ForEach by Delimiter

[JTaylor]

First, want to say thanks again for this feature.  It is awesome!!!

My question though is, is there a limit to the size or number of lines?  I get an Out of Memory for Strings error at 2,286 lines and 80,577 characters.  I am iterating through a text file, line by line.  Using @LF as the delimiter.  I did a replace on @CRLF.

I tried just repeating the word hello instead of using long lines of text and I could have 5457 lines and 38,204 characters.  Same error if I added another line.

I do know I could use FileRead but would be good to know if there is a limit or a bug.

Thanks.

Jim

[kdmoyers]

I don't know about this case, but I will say that there is something strange with this feature when used in large programs with big arrays and maps and stuff.  So far, my response has been to replace the Foreach-with-delimiter loop with an old ItemExtract loop, and the problem goes away. 

It has resisted my attempts to create a "smoking gun" sample -- when I try to create a simple case, it always works perfectly.  Which just means my case was too simple.  I haven't given up, I just haven't got it yet.

Thing is, you can't conclude much from this report: until I find a smoking gun, the most likely explanation is simply an unexpected delimiter in the string.

[JTaylor]

That is why I tried the lines of "Hello" as I thought the same thing.  Appreciate the reply.

I don't know about this case, but I will say that there is something strange with this feature when used in large programs with big arrays and maps and stuff.  So far, my response has been to replace the Foreach-with-delimiter loop with an old ItemExtract loop, and the problem goes away. 

It has resisted my attempts to create a "smoking gun" sample -- when I try to create a simple case, it always works perfectly.  Which just means my case was too simple.  I haven't given up, I just haven't got it yet.

Thing is, you can't conclude much from this report: until I find a smoking gun, the most likely explanation is simply an unexpected delimiter in the string.

[kdmoyers]

I agree with you though -- I dearly love this foreach delimiter feature, so I will continue to lean on it whenever time allows.

[td]

I don't know about this case, but I will say that there is something strange with this feature when used in large programs with big arrays and maps and stuff.  So far, my response has been to replace the Foreach-with-delimiter loop with an old ItemExtract loop, and the problem goes away. 

It has resisted my attempts to create a "smoking gun" sample -- when I try to create a simple case, it always works perfectly.  Which just means my case was too simple.  I haven't given up, I just haven't got it yet.

Thing is, you can't conclude much from this report: until I find a smoking gun, the most likely explanation is simply an unexpected delimiter in the string.

Putting in the effort to create an example is much appreciated. The script below does illustrate the problem on my system.

Code Select Expand

; Build a test string
Text = 'Hello world':@lf
strContent = StrFill(Text, 42000)

; This errors at about line  3367.
count = 0
Foreach line in strContent by @lf
   dummy = line
   count += 1
next
exit

[JTaylor]

Sorry about that.  Wish I was as smart as you, as your approach is much easier to figure out the fail point :-)   I spent so much time narrowing down the file looking to see if there was a some character causing the problem and then narrowing down the Hello list it didn't cross my mind.  I was just happy to confirm I wasn't doing anything stupid to cause the problem.   


So, is it a bug or a feature?   I guess it will depend on whether you fix it or document it???

Thanks.


Jim

[spl]

Something I asked Jim about, but also fails in WB 64. Not sure CLR can pull into a .Net Arraylist (I think PS can)

[td]

Or just use the itemlist functions or the Arrayize function on any very large lists until the next release.

[td]

This works as well, but it is a bit slower.

Code Select Expand

objAList = ObjectClrNew( 'System.Collections.ArrayList')
for x = 1 to 4000
   objAList.Add('Hello world')
next
y = 0
ForEach objitem in objAList
   y +=  1
Next

exit

[JTaylor]

Sounds good.  Thanks.

Jim

[kdmoyers]

Or just use the itemlist functions or the Arrayize function on any very large lists until the next release.

Works for me!

[td]

This works if you want to use a ForEach structure.

Code Select Expand

; Build a test string
Text = 'Hello world':@lf
strContent = StrFill(Text, 42000)
aContent = Arrayize(strContent, @lf)

; This does not error.
count = 0
Foreach line in aContent
   dummy = line
   count += 1
next

[kdmoyers]

Got it, nice.

Hmmm... would this work too?

Code Select Expand

Foreach line in Arrayize(strContent, @lf)
   dummy = line
   count += 1
next

[JTaylor]

Nice.

Thanks.

Jim

[td]

Got it, nice.

Hmmm... would this work too?

Code Select Expand

Foreach line in Arrayize(strContent, @lf)
   dummy = line
   count += 1
next

WIL has an optimization that prevents recalling the function on each iteration, as long as the physical line does not change. However, making changes to the strContent variable from within the foreach next loop are not reflected in the contents of each line variable in the loop. This illustrates the point:

Code Select Expand

; This does not error.
count = 0
Foreach line in Arrayize(strContent, @lf)
   dummy = line
   count += 1
   if IsDefined(strContent) then drop(strContent)
next
exit

[edit] Also, using substitution for the parameters to the Arrayize function can cause a physical change to the line and result in reevaluation of the expression containing the Arrayize function call.

[kdmoyers]

Super interesting Tony, thanks!

[spl]

Just curious. But while on the subject of Foreach and Arrays. Could ForEach with ArrayFileGet be preferable to a FileRead() loop - dependent on any maxsize for ArrayFileGet and accounting for the 4096 char line length.

[td]

I haven't tested, but I don't think you would get much of a performance boost. It does save some typing, e.g., not using an integer array index. I tend to use ForEach because I am lazy.