Random ("strange") issues when processing OneDrive files

[bottomleypotts]

Does anyone have any suggestions why I would be having random issues with OneDrive? Just doing random stuff such as using FileDigest. Error 1800.

Decided to switch to using CertUtil and that has the same random issues. Unable to find file.

[ChuckC]

Are you able to open the files with something like notepad.exe or the "TYPE" command in a CMD.exe console window?

Microsoft implements support for presenting content from OneDrive as folders & files on a local drive via the use of mini-filter drivers and reparse points.  Depending on how it is configured, there is the possibility that you may be seeing only a "stub" for the file in question and getting an error when trying to open it if OneDrive isn't able to immediately retrieve the body of the file from cloud storage and replicate it locally.

[bottomleypotts]

Sorry. Should have made it more clear. Am iterating through 100,000 files in 2,000 folders and subfolders. All files have been retrieved and stored locally. Just looked at the log and seems like I'm having an error rate of 5%. Just will not open the file. Of course, then I use notepad, excel, word or whatever ... no problem.

Just rapidly hitting the files seems to be causing errors. And running through for a second time will maybe result in a 1% change. But it will usually be the same files.

I find it odd.

Also, using 32bit compiled WB scripts.

[bottomleypotts]

Just a follow up because I have no idea what the issue is.

I decided to trap the filedigest command. On error, I can fileget the file without issue. I can also filecopy the file. So why filedigest fails is beyond me. So my workaroud is to filecopy the file and use that value.

Stupid!

[td]

FileDigest and FileGet use exactly the same Windows APIs to open a file and the problem is most likely a Window bug (there are several relating to the NTFS in Windows 10 and newer system.) FileDigest does provide the system error number as additional error information. That information would be useful to know.

[bottomleypotts]

Thanks Tony.

The error is:

1800: Error opening file

Clicking on "More Error Info"

CreateFile error: 3

[td]

I think it is very likely a Windows bug. Error 3 is "The system cannot find the path specified." I have seen cases on Windows 10/11 where error 3 is generated when a file is copied (but not moved) to two different locations. It is as if the file system locks the source file even though the first copy is complete. That is not exactly the problem you are having but it does suggest a broader issue with the Windows file system.

[bottomleypotts]

Thanks for the replies. Does not surprise me in the least that this is a Microsoft issue.

[ChuckC]

I decided to trap the filedigest command. On error, I can fileget the file without issue. I can also filecopy the file. So why filedigest fails is beyond me. So my workaroud is to filecopy the file and use that value.

Just out of curiosity, have you tried using an altered form of your script where FileGet() is used in place of FileDigest() in the main loop you have for processing the files?

I'm curious as to whether FileGet() fails in the same percentage of the time as FileDigest() when it is the function used for the initial access for each file.

I am still quite suspicious of OneDrive's mini-filter driver not retrieving the files as fast as you're trying to access them, and for any file for which it cannot retrieve the content from the backend cloud storage, it results in system error code 3 being returned.

[bottomleypotts]

Just out of curiosity, have you tried using an altered form of your script where FileGet() is used in place of FileDigest() in the main loop you have for processing the files?

I'm curious as to whether FileGet() fails in the same percentage of the time as FileDigest() when it is the function used for the initial access for each file.

I am still quite suspicious of OneDrive's mini-filter driver not retrieving the files as fast as you're trying to access them, and for any file for which it cannot retrieve the content from the backend cloud storage, it results in system error code 3 being returned.

No. But will run a test for you! It is frustrating having random errors crop up like this. Just seems to be the era of IT we live in.

[bottomleypotts]

Just iterated through 250,000 files. Had no issues with fileget.

[td]

As mentioned previously, both functions use exactly the same Win32 APIs to open and read a file. The only difference is that FileDiget does more processing of the contents of the file once they are in memory. It can, therefore, take a bit more time between files depending on the hash algorithm selected.