Detecting the ACL problem

[snowsnowsnow]

So, there's this long-standing Windows bug having to do with files moved into shared directories.

Specifically, if you create a file somewhere on your system - outside of a shared area - and then move (not copy) the file into the shared area, then the "ACL"s on the file will be such that the file cannot be read by the remote user (i.e., the user accessing your "shared" area).

The standard workarounds/fixes for this problem are:

1) Do a copy, not a move.  This is ugly and inefficient (if the file is large), but it does work.

2) Use a tool to "fix" the ACLs on the file, once it has been moved into the shared area.  As it happens, I asked about this some years ago, and a helpful forum user provided me with the answer, which is the following script:

Code (WinBatch) Select Expand


; vim:fo-=t fo+=ro1 comments+=b\:;
IntControl(50,0,0,0,0) ; Turn off Web Page Support
AddExtender("WWWNT34I.DLL")

#DefineSubroutine udsTry(expr,errstr,cancelstr)
IntControl(73,1,1,0,0)
Return %expr%
:wberrorhandler
Return errstr
:cancel
Return cancelstr
#EndSubroutine

IF !IsDefined(Prog) THEN Prog = IntControl(1004,0,0,0,0)
ProgDate = StrCat(Prog," - ",FileTimeGet(Prog))

Param1 = FileFullName(Param1)
Terminate(!udsTry(`wntAccessMod("",Param1,300,2,0)`,0,0),ProgDate,"File/directory not found: %Param1%")
wntAccessMod("",Param1,300,3,1)
IF DirExist(Param1) THEN it = "Directory"
   ELSE it = "File"
Pause(ProgDate,"%it%: %Param1% has been processed successfully...")


Now, my question is this.  Suppose I want to detect files that haven't been fixed, but I don't want to just blindly "fix" them all.  I.e., the form of this post/thread is not "How do I fix it?", but "How do I detect which files have the disease?"

As you many have guessed, the underlying issue is that I have a very large directory (lots of files) and I think that only a few files have the disease, and it would be time-consuming to just blindly fix all of the files in the directory.

P.S.  Since it has been quite a while since I've done it, what is the "proper" way to post WB code here so that it gets correctly highlighted?  It looks to me like the above does not invoke the highlighting.  I believe this functionality was fixed/improved when we moved from the old WebBoard to this one.

P.P.S.  Also, is there a way to insert raw HTML code into your post, and have it get displayed correctly (i.e., as formatted pretty display, not just raw HTML text) ?

[td]

The obvious answer is to use the network extender to check for the missing ACL combination(s) and act accordingly.   Presumably,  there is some reason why you have not pursued this?   

The forum does not process most raw HTML by design.  You can use BB codes in place of some HTML but for WIL script syntax coloring you don't need to.  Just select WinBatch from the Code drop list when you are in the topic editor and  paste your script between the tags. You can also display syntax colored raw HTML by selecting HTML.

[ChuckC]

The cost of retrieving every ACL on every folder and file and manually computing what the inherited ACEs should look like vs. what they actually are is more expensive than simply allowing the extender to invoke the underlying Win32 API function that propagates inheritable ACEs down thru a folder hierarchy on disk.

[snowsnowsnow]

The obvious answer is to use the network extender to check for the missing ACL combination(s) and act accordingly.   Presumably,  there is some reason why you have not pursued this?   

The forum does not process most raw HTML by design.  You can use BB codes in place of some HTML but for WIL script syntax coloring you don't need to.  Just select WinBatch from the Code drop list when you are in the topic editor and  paste your script between the tags. You can also display syntax colored raw HTML by selecting HTML.

First of all, to answer the formatting questions:
1) I figured out how to make it highlight as WinBatch and have edited the original post accordingly.
2) My reason for asking about raw HTML is that VIM has the ability to "Convert to HTML" - which means it will take your source code and convert it to HTML which, when displayed or printed shows as syntax highlighting.  For various personal reasons, I actually prefer (or did prefer at one time) VIM's version of syntax highlighting to the various options provided herein for doing the highlighting.  In the old version of the forum software, it was possible to use VIM to generate the HTML code and then paste that code in here and it all worked.  It seems (from what you've said) that this is no longer possible.  So, my question is, is there any other way to get that effect (that is, to take HTML generated by VIM and paste it in here) ?

Now, regarding your comment "Presumably,  there is some reason why you have not pursued this?", the short answer is that I really don't understand any of it.  I was just hoping someone could spoon-feed me the answer, like they (and by "they", I think I mean "ChopC") did before.

Finally, regarding this comment:

The cost of retrieving every ACL on every folder and file and manually computing what the inherited ACEs should look like vs. what they actually are is more expensive than simply allowing the extender to invoke the underlying Win32 API function that propagates inheritable ACEs down thru a folder hierarchy on disk.

I kinda suspected that might be true.  But, as it turns out, I'm not really concerned all that much by "cost" or "expense".  I'm really only going to do this once - it is not an on-going process.  And, as I tried to make clear in the original post, I'm really more interested in actually knowing (that is, in the "detection" mode) which files aren't like the others - than in just blindly fixing them all (and, in the process, destroying the evidence that I am trying to decipher).  So, if it can be done, I'd be grateful for some code that does it.

[td]

The Forum blocks processing of most raw HTML in posts for security reasons and that isn't likely to change.  You could perhaps write a script to convert your HTML colorized output to BBcode.  I converted Detlev's WinBatch Studio HTML syntax colorizer script to use BBcode instead of HTML tags and it produced more or less the same results when the output was posted to this forum.   Of course that effort became superfluous once we added syntax coloring...

Performing your disease detection could be relatively straight forward or incredibly convoluted.  I suppose it depends on how many possible variables you actually need to consider.   While is it only a small part of what you would need to do and my not be of any use at all, here is a link to a Tech Database article that demonstrates collecting all the users/groups permissions on a share:

http://techsupt.winbatch.com/webcgi/webbatch.exe?techsupt/nftechsupt.web+WinBatch/Networks~-~Servers/Microsoft~Client/wNT+Obtain~Share~Permissions.txt 

I will poke around for any other examples that may be of some use.

[snowsnowsnow]

When I started this thread, I had hopes that there existed a simple, server-side, solution - hopefully just a simple variation on the code that I posted.  Based on the responses I've gotten, it seems that there isn't such a solution.

So, I decided to try a client-side solution - that is, iterating over all the files and doing a simple:

fh = FileOpen(file,"READ")
FileClose(fh)

to see which files, if any, caused an abort.  However, this didn't reveal anything interesting, so the search continues.

In any case, thanks for all the help so far.

[ChuckC]

If your main interest is in finding the files that have incorrect inherited ACEs on them due to having been moved with the underlying MoveFile() Win32 API function, rather than just blindly correcting their security settings to be properly inherited, then the methodology to use should be similar to the following:

1)  Write some code that iterates through the file system and discovers the structure of the folder hierarchy on disk.  Achieve this however you like.

2)  For each folder that was enumerated, create within it a single file with a unique non-conflicting name, and also create a child folder with a unique non-conflicting name.

3)  Read the DACL [a.k.a "permissions"] for the file and directory that you just created, and retain them in memory for reference purposes.

4)  Enumerate all immediate subordinate files and folders under the current directory.  Do not do a recursive enumeration of the entire subtree under the current directory.

5)  For each child file/folder that is enumerated, read the DACL and compare it to the appropriate file or folder DACL for the ones that you explicitly created.  If there are differences, log or store them in some way for later reference.

6)  Delete the file & folder that were created in step #2.

[td]

It would be great to be proven wrong about this and I can't spend as much time as I would like on this at the moment.  But based on a very brief perusal of MSFT documentation and a few simple tests, the probability of success using the network extender in this endeavor does not approximate certainty.  And as the targeted operation systems becomes more resent, the probability diminishes to the point of approaching a very long shot when you arrive at Windows 2012 R2. (Assuming this type of file security inheritance problem is still an issue on Windows 2012 R2.) 

.

[ChuckC]

The problem still exists as of Win2K12R2.

The root cause of the problem is that the Win32 API function MoveFile() actually invokes DeviceIoControl() and causes it to execute a very low-level file system operation to move an existing directory entry for a folder, file or reparse point to another directory on the same volume.  Due to the fact that the propagation of inheritable ACEs and computation of new ACLs occurs at a higher level within the Win32 API library itself [e.g. SetSecurity()/SetNamedSecurity()], none of the inheritance calculations get performed on the file system object that is being moved, nor on any of its subordinates if it is a directory that is being moved.

I'll grant that there are some newer ACE types for Dynamic Access Control that are supported on Win2K12R2 that the extender may not be aware of and cannot properly handle, but if none of those ACEs are involved, then the extender should be able to retrieve all of the ACEs in the DACL for purposes of comparing them against a reference DACL to determine if there are any discrepancies between the two different DACLs.

I will, however, reiterate that it's easier to simply allow the underlying SetSecurity()/SetNamedSecurity() Win32 API functions do the bulk inheritance calculation and simply compute correct DACLs for all folders and files on the volume and leave it at that.  The amount of time involved in writing and debugging a script to do that analysis and report on the file system objects with incorrect ACEs will take longer than allowing the API function to do the work.

Given that this "non-computation of new security settings on move" issue still exists and has existed since Win2K came into being, the best thing to do is to keep track of moves that are performed and to be aware that the problem will most definitely occur if a move is performed on the same volume.

[td]

I am fully aware of the underlying cause of the problem.  However, based on past experience with MSFT I like to note when making an assumption that is not verified.  And simple testing on Win2k3, Win2k8, win2k12 R2 throws the assertion that the extender can extract all  the necessary security information into doubt.  But as previously mentioned, It would be great to be shown otherwise.

[td]

Out of an abundance of curiosity, created a local file on the Win2012 R2 server with only User and Administrators privileges.  Moved the file to a shared directory on the same volume using the WIL FileMove function which uses the MoveFile  (not MoveFileEx) Win32 API.  The file inherited the Everyone privileges from the share but lost the User privileges according to wntAccessGet.  Tried the same thing from a remote host and got the same result except that wntAccessGet will not list any Everyone privileges remotely.   Checked using wntAccessGet locally instead.     

[ChuckC]

Permissions on shares themselves don't ever inherit down into the file system.

Effective access is computed using the intersection of the share permissions and the NTFS permissions.

[td]

Permissions on shares themselves don't ever inherit down into the file system.

Let me restate to make it clearer.  The file inherited the Everyone privileges from the directory that is shared but lost the User privileges.

Effective access is computed using the intersection of the share permissions and the NTFS permissions.

Yes, the more restrictive permissions are applied according to MSFT.

[ChuckC]

Just wanted to run thru a couple of other ways of achieving a "move" operation and what the end result was regarding the inherited ACEs.

I did a quick test on a Win2KR2 server as follows:

1)  Via the Windows Explorer, I created 2 directories at the root of the C: drive, named "Folder01" and "Folder02".

2)  Via the Windows Explorer, I viewed the security settings of each folder I created and disabled ACL inheritance, with the option to convert inherited ACEs to explicitly assigned ACEs.

3)  On "Folder01", I added a new explicitly assigned access-allowed ACE for "TestGroup1", and gave it full control permissions.

4)  On "Folder02", I added a new explicitly assigned access-allowed ACE for "TestGroup2", and gave it full control permissions.

5)  I created 2 text files in "Folder01", and they were named "FileA.txt" and "FileB.txt".

6)  I moved "FileA.txt" from "Folder01" to "Folder02" via the Windows Explorer drag & drop.

7)  I moved "FileB.txt" from "Folder01" to "Folder02" from a CMD.EXE window with the MOVE command.

8)  Using both DumpACL and DumpSD, I dumped out the security settings on both "FileA.txt" and "FileB".txt, and noted that although "FileA.txt" had correct permissions assigned to it, with the inherited ACE for "TestGroup1" having been removed and a new inherited ACE for "TestGroup2" having been added.  However, "FileB.txt" still had an inherited ACE for "TestGroup1" on it, and no inherited ACE for "TestGroup2".

9)  Using the Windows Explorer to view the security settings on both files, I observed the same inconsistencies in the inherited ACEs as when viewing them via command line tools.


So, even when doing this all locally on a fully patched & up to date Win2K12R2 system, the problem can be made to manifest itself when a low-level Win32 API function is used to perform the move and the higher level shell-related functions used by the Windows Explorer or its own built-in "move" logic aren't involved.

This link is helpful in shedding some light on the problem and how/when usage of the Windows Explorer affects how the security gets manipulated during a move on the same volume:

https://blogs.msdn.microsoft.com/oldnewthing/20130924-00/?p=3143

That link also refers to this older article that goes into more depth discussing the underlying issue:

https://blogs.msdn.microsoft.com/oldnewthing/20060824-16/?p=29973/

[td]

If you scroll back up a few post, you will notice that you mentioned the MoveFile as an offending API.  My purpose was to test that function in a Window 2012 R2 environment.  Clearly existing ACLs are stripped and inheritance is enforced contrary to your comments about, my expectations of, and MSFTs documentation of the function.

That the Windows shell and and the command shell Move command behave differently and the Move command does not apply inherited privileges is not surprising on Windows 2012 R2.  What is somewhat surprising and the point I was trying to make is that the MoveFile API does not behave as expected.

I should have but did not realize that the Windows shell behavior changed in this regard between XP/2003 and newer systems.  Something to remember for future reference.   

[ChuckC]

Yes, I noted that you had used WinBatch's FileMove() function to do your test.

However, I just tested some basic C/C++ code that is calling MoveFileW(), MoveFileExW() and MoveFileWithProgressW() [the Unicode wide character versions of the functions], and in all cases, the file is being moved w/o inherited ACEs being recalculated.  I'm doing this on the same Win2K12R2 server that I used when I did my test with the Windows Explorer and CMD.EXE's internal "MOVE" command.

So... this begs the question... Is WinBatch's FileMove() function really using MoveFile()?  If so, is it using MoveFileA() or MoveFileW()?

[td]

FileMove is using either MoveFileA or MoveFileW.  My tests involved the wide version and there is no question about that the results.   The WinBatch FileMove function does some parameter parsing and checking because it accepts multiple files but after that it's a straight call to MoveFileA or MoveFileW for each file. 

[ChuckC]

OK.

I repeated my test with the code modified to explicitly call MoveFileA(), and I'm still seeing the problem occur as it always has, with the inherited ACEs not being recomputed after the file was moved to a different folder with a different DACL on it that should have resulted in the file having a different set of inherited ACEs if the file's DACL had been recomputed after the move.

Does WinBatch's FileMove() have any additional code in it that might take action to force the inherited ACEs to be recalculated?

Have you verified that the source and target folders are, in fact, on the same NTFS volume, and that neither of them is a reparse point such as a mount point, a junction or a directory symbolic link?

Also, for the user account that you're using when calling FileMove() in a script, does the process have any elevated LSA privileges, or otherwise have administrator rights, and is it running under a restricted token or an elevated token if UAC is enabled?

[td]

I built the server... The test was on a single volume and did not involve any reparse points of any kind nor does FileMove touch any DACLs via any APIs other than MoveFile. 

I do not have the luxury of burning any more time on this topic and must move on.   

[ChuckC]

As a follow up to my earlier, testing, I just put WinBatch 2016A on my test Win2K12R2 system and performed the same test using "Interactive WIL" on PopMenu to execute FileMove() using a text file and the same 2 directories I did my other testing with.  I'm still seeing the inherited ACEs failing to be recalculated after the move in my test environment.

At this point, I'll still stand by my earlier assessment that this issue persists to this day in current versions of Windows, and could result in files in real world environments not having their inherited ACEs recalculated and their DACLs updated after having been moved by tools or utilities that make use of MoveFile(), MoveFileEx() and/or MoveFileWithProgress() [both ANSI and Unicode variants], and that the Win32 Network Extender is still viable when it comes to detecting and correcting these incorrect ACLs.

[td]

I disagree with regard to MoveFile.  But I do agree that the Win32 Network Extender is viable as long as you use it locally and are using it against SMB+NTFS. If not then your results may vary.