I'm having trouble understanding how to FileRead a UTF-8 file.
why does this not work?
; an experiment in FileRead on a UTF-8 file
;
;
testPath = Environment("TEMP") : "\Utf8BomReadTest.txt"
; Make a no-BOM UTF-8 file: "Aé?B" LF "X" LF
byteList = "65,195,169,228,184,173,66,10,88,10"
byteTotal = ItemCount(byteList, ",")
buf = BinaryAlloc(byteTotal)
For i = 1 To byteTotal
BinaryPoke(buf, i - 1, ItemExtract(i, byteList, ","))
Next
BinaryWrite(buf, testPath)
BinaryFree(buf)
; now simulate the idea of prepending a BOM mark:
fileBytes = FileSize(testPath)
buf = BinaryAlloc(fileBytes + 3)
BinaryPoke(buf, 0, 239) ;EF
BinaryPoke(buf, 1, 187) ;BB
BinaryPoke(buf, 2, 191) ;BF
BinaryReadEx(buf, 3, testPath, 0, fileBytes) ; file contents go after the BOM
BinaryWrite(buf, testPath)
BinaryFree(buf)
; Read it back.
hIn = FileOpen(testPath, "READ", @true) ; ask for unicode processing? (fails either way)
line1 = FileRead(hIn)
line2 = FileRead(hIn)
FileClose(hIn)
codes = ""
For i = 1 To StrLen(line1)
codes = codes : Char2Num(StrSub(line1, i, 1)) : " "
Next
Message("UTF-8 BOM read test", "line 1 codes: " : codes : @CRLF : "line 2: [" : line2 : "]")
;this is not what I expect, line 1 should have only 4 utf-8 chars, not 10
Perhaps I should not be using StrLen and StrSub to parse the lines?
I don't see anything like ChrStrLen and ChrStrSub but I'm probably misunderstanding how this works.
EDIT-- I get it now, it's reading it ok, and displaying it correctly if I just message('',line1).
The problem is that StrLen and StrSub don't see it as utf-8 chars. So that's my question:
how do I look at the chars on the line when it's utf-8 characters?