NtQueryDirectoryFile fails on NFS paths longer than about 260 chars

NtQueryDirectoryFile fails on NFS paths longer than about 260 chars

Post by Corinna Vi » Sat, 20 Dec 2008 03:01:52


Hi,

This problem occurs with the NFS client from SFU 3.5 on XP SP3, as well
as with the NFS client from the on-board SFU on Windows Server 2008.

Consider a directory tree on a NFS share on some remote NFS server box
which, just for instance, is 1K chars deep, and every subdir contains
just one more subdir:

\\??\\UNC\\server\\share\dir\\dir\dir\\dir\\dir\\.....

Now you start a simple testcase which does basically this (vaguely
C-shaped pseudo code):

allocate PFILE_NAMES_INFORMATION buffer of 60K
native_nt_dirname = "\\??\\UNC\\server\\share\dir"
while length (native_nt_dirname) < 1024
{
status = NtOpenFile (&handle, dir)
status = NtQueryDirectoryFile (handle, [...], buffer, 60K,
FileNamesInformation, FALSE, NULL, TRUE)
[skip . and .. entries]
if (NT_SUCCESS (status) && is_a_dir (buffer.entry))
native_nt_dirname = native_nt_dirname + "\\" + buffer.entry
NtClose (handle)
}

I don't know if my pseudo code is clear enough, but the real code works fine
for long path names of up to 32K on NTFS, but it fails on NFS as soon as
the native_nt_dirname is longer than about 260 WCHARs. Which is *very*
strange

What fails is *not* the NtOpenFile on the dir, but rather the subsequent
call to NtQueryDirectoryFile. It also fails with a rather weird status
code, STATUS_BUFFER_TOO_SMALL. I tried various code changes like
reading only one directory entry at a time instead of what fits into the
buffer, or using the FileDirectoryInformation class instead of
FileNamesInformation, but to no avail.

The bad joke is, this only happens with NtQueryDirectoryFile. Calls
to NtQueryInformationFile on files on an NFS share with any length
work fine!

This occurs in our Cygwin lib. The effect is that an `ls' shows no
files in a subdir > 260 chars, but a 'stat dir" within the same
directory works fine. Then ls again doesn't work, but a cd works, and
so on. Calling stat (which basically translates to NtQueryInformationFile)
at any point in the directory tree shows that the subdir exists and every
one has another inode number.

For the sake of sanity, I tried this with a SFU shell from SFU 3.5 on
XP and with a SFU shell on Windows 2008. And guess what? It suffers
the same problem. It doesn't recognize any directories below a path
of ~260 chars because NtQueryDirectoryFile fails. But it's no problem
to cd into the next subdirectory if you happen to know its name.

Additionally it should be noted that the 260 chars border doesn't make
any sense on the native NT level, which Cygwin as well as SFU utilize.

Is this perhaps a known bug in the NFS client?


Corinna

--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat
 
 
 

NtQueryDirectoryFile fails on NFS paths longer than about 260 chars

Post by Alan Carr » Sat, 20 Dec 2008 03:48:36

Corinna Vinschen" < XXXX@XXXXX.COM > wrote in message
news:gie36g$4k0$ XXXX@XXXXX.COM ...


You have to use the Unicode version of the function (whatever it's called,
maybe add a W at the end). I am assuming this is the same limitation as you
see for functions such as CreateFile or FindFirstFileEx and so on. There's
always this little blurb in the help:

FindFirstFileEx/CreateFile etc...
--------------------------------
In the ANSI version of this function, the name is limited to MAX_PATH
characters. To extend this limit to approximately 32,000 wide characters,
call the Unicode version of the function and prepend "\\?\" to the path.

Also note the following from windef.h:
--------------------

// windef.h
#define MAX_PATH 260

--------------------

So no surprises there.

- Alan Carre



 
 
 

NtQueryDirectoryFile fails on NFS paths longer than about 260 chars

Post by Corinna Vi » Sat, 20 Dec 2008 05:47:59

Alan,





If you look closely, I'm talking about using native NT functions.
These are always using UNICODE, so this is entirely unrelated to
the restrictions of the Win32 ANSI API. MAX_PATH has no meaning
in the NT API.


Corinna

--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat
 
 
 

NtQueryDirectoryFile fails on NFS paths longer than about 260 chars

Post by Alan Carr » Sat, 20 Dec 2008 09:05:06

ee below...

"Corinna Vinschen" < XXXX@XXXXX.COM > wrote in message
news:giectv$7p7$ XXXX@XXXXX.COM ...

===============

Well ok, I guess you know better than me since I've never used these
functions. However you *are* passing in Ansi strings in those examples, so
that's why I suspect it may be related... I mean, it seems just a little too
coincidental to me that you're running up against that 260 byte limit just
as the WinAPI functions do. Perhaps (I mean not perhaps, certainly) the
WinAPI functions must eventually pass on something to NTDLL.DLL no?

Anyway, it's not documented in my local MSDN thingy here, so I tried dumpbin
on it and this is what I get:

dumpbin /exports e:\WINNT\system32\ntdll.dll | find/i "querydi"

234 E2 0000DF5E NtQueryDirectoryFile
235 E3 0000DF73 NtQueryDirectoryObject

1043 412 0000DF5E ZwQueryDirectoryFile
1044 413 0000DF73 ZwQueryDirectoryObject

So it appears like there's 2 versions of everything. Some start with "Nt"
and others with Zw. Now call me crazy, but Zw sounds like what one might use
for shorthand for something like "Zero-terminated-wide-string" ...

There's a ZwQueryXXX for every NtQueryXXX it seems. Here's the list: (see 2
links following list):

228 DC 0000DEE0 NtQueryAttributesFile
229 DD 0000DEF5 NtQueryBootEntryOrder
230 DE 0000DF0A NtQueryBootOptions
231 DF 0000DF1F NtQueryDebugFilterState
232 E0 0000DF34 NtQueryDefaultLocale
233 E1 0000DF49 NtQueryDefaultUILanguage
234 E2 0000DF5E NtQueryDirectoryFile
235 E3 0000DF73 NtQueryDirectoryObject
236 E4 0000DF88 NtQueryEaFile
237 E5 0000DF9D NtQueryEvent
238 E6 0000DFB2 NtQueryFullAttributesFile
239 E7 0000DFC7 NtQueryInformationAtom
240 E8 0000DFDC NtQueryInformationFile
241 E9 0000DFF1 NtQueryInformationJobObject
242 EA 0000E006 NtQueryInformationPort
243 EB 0000E01B NtQueryInformationProcess
244 EC 0000E030 NtQueryInformationThread
245 ED 0000E045 NtQueryInformationToken
246 EE 0000E05A NtQueryInstallUILanguage
247 EF 0000E06F NtQueryIntervalProfile
248 F0 0000E084 NtQueryIoCompletion
249 F1 0000E099 NtQueryKey
250 F2 0000E0AE NtQueryMultipleValueKey
251 F3 0000E0C3 NtQueryMutant
252 F4 0000E0D8 NtQueryObject
253 F5 0000E0ED NtQueryOpenSubKeys
254 F6 0000E102 NtQueryPerformanceCounter
255 F7 0000EAB0 NtQueryPortInformationProcess
256 F8 0000E117 NtQueryQuotaInformationFile
257 F9 0000E12C NtQuerySection
258 FA 0000E141 NtQuerySecurityObject
259 FB 0000E156 NtQuerySemaphore
260 FC 0000E16B NtQuerySymbolicLinkObject
261 FD 0000E180 NtQuerySystemEnvironmentValue
262 FE 0000E195 NtQuerySystemEnvironmentValueEx
263 FF 0000E1AA NtQuerySystemInformation
264 100 0000E1BF NtQuerySystemTime
265 101 0000E1D4 NtQueryTimer
266 102 0000E1E9 NtQueryTimerResolution
267 103 0000E1FE NtQueryValueKey
268 104 0000E213 NtQueryVirtualMemory
269 105 0000E228 NtQueryVolumeInformationFile

Zw versions:
=========

1037 40C 0000DEE0 ZwQueryAttributesFile
1038 40D 0000DEF5 ZwQueryBootEntryOrder
1039 40E 0000DF0A ZwQueryBootOptions
1040 40F 0000DF1F ZwQueryDebugFilterState
1041 410 0000DF34 ZwQueryDefaultLocale
1042 411 0000DF49 ZwQueryDefaultUILanguage
1043 412 0000DF5E ZwQueryDirectoryFile
1044 413 0000DF73 Zw
 
 
 

NtQueryDirectoryFile fails on NFS paths longer than about 260 chars

Post by m » Sat, 20 Dec 2008 09:06:25

MAX_PATH is variously used; and you have just found another case of this
bug!
 
 
 

NtQueryDirectoryFile fails on NFS paths longer than about 260 chars

Post by Alan Carr » Sat, 20 Dec 2008 09:09:56


BTW. MAX_PATH has a very very obvious meaning (nothing to do with WinAPI or
anything):

C:\<256>NULL == 260

That is, 260 is what you get when a filename length is restricted to a
single byte of storage. It's just a universal constant.

You can probably guess why the limit is about 32K when you jump to Unicode.
I mean... 2 bytes perhaps, signed?

Nah! ;)

- Alan Carre
 
 
 

NtQueryDirectoryFile fails on NFS paths longer than about 260 chars

Post by David Crai » Sat, 20 Dec 2008 10:06:00

ook at: http://www.osronline.com/article.cfm?id=257

This explains the differences between Zw and Nt. And the primary answer is
not wide characters, but security when drivers do it. Drivers should use Zw
instead of Nt when possible. With Zw, the buffers for a read can be located
in the kernel address space, while Nt versions do not permit this.

The problem Corinna is having may deal with the interface to the network
interfaces with the remote file system, but I don't know. Start at page 316
in the "Native API Reference" for documentation on these calls and the type
of file objects that can be queried with the various types of queries. Also
check the WDK for more up to date docs if it is documented.


"Alan Carre" < XXXX@XXXXX.COM > wrote in message
news:%23bo% XXXX@XXXXX.COM ...


 
 
 

NtQueryDirectoryFile fails on NFS paths longer than about 260 chars

Post by Alan Carr » Sat, 20 Dec 2008 10:50:56


So what's the explanation for that guy who passed a Unicode string into
NtQueryDirectoryFile and it always returns failure? The docs say to send in
Unicode... sure seems something is wrong somewhere in err Denmark? I mean,
I'm not saying it must be because of my guess (I honestly know nothing!),
but it's interesting to try and guess why this very specific "coincidence"
[I mean it's not a coincidence clearly] would come up.

Here's another guess:
Guess 2: Perhaps the function interprets the incoming string as either Ansi
or Unicode right away, and then behaves accordingly (ie. 256 length for the
Ansi, 32k for the Unicode). Notice that it really does need to decide
right-off how to store the length part of the string. Why? because that will
be the first "element" in the array; in true "PASCAL" form. So it needs to
know where to begin (or put another way, how much to *skip over*) before
writing down the string. The last step is to fill in the first element (the
counter element) because that's the last thing it actually learns.

And there's no zero terminator in NTDLL obviously, it must be PASCAL
strings. Clearly 'Z' in "Zw" can't mean "zero"... that was a bad guess! It
has to be pascal strings internally because you get the full 256 characters
out of your lowly single-byte counter. [C:\] + 256 + [NULL] == 260 , so
clearly all 4 bytes are assumed; 3 represent the device, and the last is our
C null terminator.

Maybe all she needs to do is put an L in front of that string and it'll
work? Or no, scratch that... could be misinterpreted if you hit the wrong
unicode char. Perhaps 0xFFFE as the first 2 bytes and that tells it the data
to follow is in fact Unicode?

I think it's got to be this Ansi/Unicode mixup. Somehow, or somewhere along
the line it's deciding on Ansi and skipping over 1 byte then writing. Has to
be... the question is why?

- Alan Carre
 
 
 

NtQueryDirectoryFile fails on NFS paths longer than about 260 chars

Post by Alexander » Sat, 20 Dec 2008 11:46:58

ince an UNICODE_STRING can never be longer than 32K characters, a full
filename can't be, either.

"David Craig" < XXXX@XXXXX.COM > wrote in message
news: XXXX@XXXXX.COM ...


 
 
 

NtQueryDirectoryFile fails on NFS paths longer than about 260 chars

Post by Alan Carr » Sat, 20 Dec 2008 12:30:56


Not according to the header file (Ntsecapi.h):


typedef struct _LSA_UNICODE_STRING {
USHORT Length;
USHORT MaximumLength;
#ifdef MIDL_PASS
[size_is(MaximumLength/2), length_is(Length/2)]
#endif // MIDL_PASS
PWSTR Buffer;
} LSA_UNICODE_STRING, *PLSA_UNICODE_STRING;

typedef LSA_UNICODE_STRING UNICODE_STRING, *PUNICODE_STRING;


Should be able to have up to 64k length UNICODE_STRING's

- Alan Carre

P.S. Something of interest... regardless of whether or not the OS can even
store the "full path" of an object, that actually says nothing
*theoretically* about how long it's path can be. In fact you can have a file
with an infinitely long path (I actually made such a file once under FAT by
pointing one of it's root directory's subdirs back to a grandparent
directory of that same directory - if you ran dir command it just ran along
till it crashed at MAX_PATH). But even if it's not an infinite loop, all you
really need to know to go up and down levels with zero ambiguity, is the
current node. You can then just go to a "next" or "prev", directory in the
chain. Edit a file and save it. That is, even if the number of steps to get
to where you are was a billion "CD SomeDir"s, meaning the actual path from
the root of the device is *billions* of CHARs/WCHARs long, there still is a
definite and completely unabiguous path. If you remembered how you did it
the 1st time you could get back there again always. But you would never
store this path as a string file system "as a file name". There may be an
algorithm say (that would be fun)... to get to your famous file. Say the
Fibonacci sequence to the 10000th element (or whatever you think is real
cool), but that's about it. Hmm... now I must learn NTFS so I can make a
file who's "path" is 10,000 fibonacci numbers as each parent dir ;).
Definite path, can't store it ANYWHERE!
 
 
 

NtQueryDirectoryFile fails on NFS paths longer than about 260 chars

Post by Alexander » Sat, 20 Dec 2008 12:55:27

These are in bytes.

And your dreams about unlimited length are just that - dreams.
 
 
 

NtQueryDirectoryFile fails on NFS paths longer than about 260 chars

Post by Corinna Vi » Sat, 20 Dec 2008 18:59:56


The NT API has nothing to with MAX_PATH in the first place. MAX_PATH is
an artificial restriction of the Win32 API ANSI functions, something to
do with internal static buffers (like for the CWD in the PEB), backward
compatibility, moon phase, etc. Using MAX_PATH on the NT level or below,
in drivers, isn't sane.

To reiterate.

The problem I'm reporting is a problem which occurs when calling native
NT functions on an NFS share. It does not occur on NTFS, the code works
just fine there. So I assume it's a bug in the SFU NFS client driver.

It restricts directory searches to absolute paths with ~260 chars (yes,
that *is* MAX_PATH, and maybe not even coincidentally). This, together
with the weird status code STATUS_BUFFER_TOO_SMALL points vaguely to a
too small static internal buffer in the NFS client driver. That's just
a wild guess, of course, talking about closed source software, but here
we are.

What's especially weird is that the NtOpenFile function, the only one in
this context which actually *gets* the path name, works fine and returns
a valid handle, which can be used in any other OS function. As I wrote
in my OP, NtQueryInformationFile works fine on the handle returned by
NtOpenFile. Only the NtQueryDirectoryFile function using the handle
fails if the path given to NtOpenFile is longer than ~260 characters.

It does not only occur in Cygwin, it also occurs in an Interix/SFU shell
when accessing an NFS FS. So this is neither a problem with my code,
nor a problem restricted to the Win32 subsystem. Rather, it's a generic
problem.

Given that PATH_MAX (no, not MAX_PATH) is 512 in Interix, and given that
PATH_MAX does not even specify the maximum absolute path length as
MAX_PATH does for the Win32 ANSI API, it looks like a bug which affects
a Microsoft product as well.

I'd really like to go back to my original report instead of discussing
how UNICODE_STRING works.

If possible, I'd like to ask for a reaction from the Microsoft guys
scanning this newsgroup.


Corinna

--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat
 
 
 

NtQueryDirectoryFile fails on NFS paths longer than about 260 chars

Post by Stefan Kuh » Sat, 20 Dec 2008 23:41:05

Hi Corinna,



Either that or it is there on purpose because otherwise Explorer.exe
will have hard time navigating into such a directory.

Just a thought,

--
Stefan Kuhr
 
 
 

NtQueryDirectoryFile fails on NFS paths longer than about 260 chars

Post by Alexander » Sat, 20 Dec 2008 23:52:33

Corina,

Check if that happen in plain cmd.exe/dir with plain SMB shares, as well as
with NFS. You can feed those long UNICODE names to cmd.exe.
 
 
 

NtQueryDirectoryFile fails on NFS paths longer than about 260 chars

Post by Corinna Vi » Sun, 21 Dec 2008 01:36:06


If so, that would be very silly, IMHO. Just because Explorer is still
not able to cope with long pathnames (after all these years...) doesn't
mean the system should cripple all applications, including even the
POSIX subsystem.


Corinna

--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat