reading UTF-8 files

[kdmoyers]

I'm having trouble understanding how to FileRead a UTF-8 file.
why does this not work?

Code Select Expand

; an experiment in FileRead on a UTF-8 file
;
;

testPath = Environment("TEMP") : "\Utf8BomReadTest.txt"

; Make a no-BOM UTF-8 file: "Aé?B" LF "X" LF
byteList = "65,195,169,228,184,173,66,10,88,10"
byteTotal = ItemCount(byteList, ",")
buf = BinaryAlloc(byteTotal)
For i = 1 To byteTotal
    BinaryPoke(buf, i - 1, ItemExtract(i, byteList, ","))
Next
BinaryWrite(buf, testPath)
BinaryFree(buf)

; now simulate the idea of prepending a BOM mark:
fileBytes = FileSize(testPath)
buf = BinaryAlloc(fileBytes + 3)
BinaryPoke(buf, 0, 239) ;EF
BinaryPoke(buf, 1, 187) ;BB
BinaryPoke(buf, 2, 191) ;BF
BinaryReadEx(buf, 3, testPath, 0, fileBytes)      ; file contents go after the BOM
BinaryWrite(buf, testPath)
BinaryFree(buf)

; Read it back.
hIn = FileOpen(testPath, "READ", @true) ; ask for unicode processing? (fails either way)
line1 = FileRead(hIn)
line2 = FileRead(hIn)
FileClose(hIn)

codes = ""
For i = 1 To StrLen(line1)
    codes = codes : Char2Num(StrSub(line1, i, 1)) : " "
Next
Message("UTF-8 BOM read test", "line 1 codes: " : codes : @CRLF : "line 2: [" : line2 : "]")

;this is not what I expect, line 1 should have only 4 utf-8 chars, not 10

Perhaps I should not be using StrLen and StrSub to parse the lines?
I don't see anything like ChrStrLen and ChrStrSub but I'm probably misunderstanding how this works.

EDIT-- I get it now, it's reading it ok, and displaying it correctly if I just message('',line1).   
The problem is that StrLen and StrSub don't see it as utf-8 chars.  So that's my question:
how do I look at the chars on the line when it's utf-8 characters?

((Interestingly, if I paste those four utf-8 characters into this forum message, the page crashes when I click Save! ))

[kdmoyers]

I got it, here's the corrected code.  It works fine, I messed up the filenames and forgot
that I need to put in chrsetcodepage(65001) to make it work.

Code Select Expand

; an experiment in FileRead on a UTF-8 file
;
;

testPath = Environment("TEMP") : "\Utf8BomReadTest.txt"
testPath2 = Environment("TEMP") : "\Utf8BomReadTest2.txt"

; Make a no-BOM UTF-8 file: "Aé?B" LF "X" LF
byteList = "65,195,169,228,184,173,66,10,88,10"
byteTotal = ItemCount(byteList, ",")
buf = BinaryAlloc(byteTotal)
For i = 1 To byteTotal
    BinaryPoke(buf, i - 1, ItemExtract(i, byteList, ","))
Next
BinaryWrite(buf, testPath)
BinaryFree(buf)


if 1==1
    ; now simulate the idea of prepending a BOM mark:
    fileBytes = FileSize(testPath)
    buf = BinaryAlloc(fileBytes + 3)
    BinaryPoke(buf, 0, 239) ;EF
    BinaryPoke(buf, 1, 187) ;BB
    BinaryPoke(buf, 2, 191) ;BF
    BinaryReadEx(buf, 3, testPath, 0, fileBytes)      ; file contents go after the BOM
    BinaryWrite(buf, testPath2)
    BinaryFree(buf)
endif

chrsetcodepage(65001)

; Read it back.
hIn = FileOpen(testPath, "READ", @true) ; ask for unicode processing? (fails either way)
line1 = FileRead(hIn)
line2 = FileRead(hIn)
FileClose(hIn)

codes = ""
For i = 1 To StrLen(line1)
    codes = codes : Char2Num(StrSub(line1, i, 1)) : " "
Next
Message("UTF-8 BOM read test", "line 1 codes: " : codes : @CRLF : "line 1: [" : line1 : "]")
clipput(line1)

[kdmoyers]

Sorry for using the forum as my rubber duck!

[spl]

Sorry for using the forum as my rubber duck!

But question. How is it determined file is UTF-8 in the first place? Would make you solution a udf.. i.e. processesUDF8(). That would be cool.