"Attempt to execute non-executable address" on Server 2008 when ??Terminal Services installed

"Attempt to execute non-executable address" on Server 2008 when ??Terminal Services installed

Post by Corinna Vi » Fri, 31 Oct 2008 02:20:35


XXXX@XXXXX.COM wrote:

No worries.


Thanks, that was very helpful!

I just did that and the single page which contains the function is not
executable when the crash happens. This looks like a OS problem,
but read further.

For my private test application (an augmented bash built in debug mode),
the .text segment is at 0x401000 up to 46b000. `objdump -h' prints

bash-g.exe: file format pei-i386

Sections:
Idx Name Size VMA LMA File off Algn
0 .text 00069510 00401000 00401000 00000400 2**4
CONTENTS, ALLOC, LOAD, CODE
1 .data 00002380 0046b000 0046b000 00069600 2**5
CONTENTS, ALLOC, LOAD, DATA
[...]

The start address of the crashing function is 0x419d97. The printout of
!vadump for the pages in the .text segment looks like this when the crash
occurs:

BaseAddress: 00401000
RegionSize: 0000c000
State: 00001000 MEM_COMMIT
Protect: 00000080 PAGE_EXECUTE_WRITECOPY
Type: 01000000 MEM_IMAGE

BaseAddress: 0040d000
RegionSize: 00001000
State: 00001000 MEM_COMMIT
Protect: 00000040 PAGE_EXECUTE_READWRITE
Type: 01000000 MEM_IMAGE

[...]

BaseAddress: 00419000
RegionSize: 00001000
State: 00001000 MEM_COMMIT
Protect: 00000008 PAGE_WRITECOPY !!!!!
Type: 01000000 MEM_IMAGE

BaseAddress: 0041a000
RegionSize: 00002000
State: 00001000 MEM_COMMIT
Protect: 00000080 PAGE_EXECUTE_WRITECOPY
Type: 01000000 MEM_IMAGE

[...]

!vprot 0x419000 prints additionally

AllocationBase 00400000
AllocationProtect 00000080 PAGE_EXECUTE_WRITECOPY

So *something* has actually changed the protection. Not only on this
page but also on some other arbitrary pages in the .text segment.

When debugging the same on a non-TS Server 2008 machine, all pages in
the .text segment are still either PAGE_EXECUTE_WRITECOPY or
PAGE_EXECUTE_READWRITE when setting a breakpoint to the instruction
which crashes on the TS machine.

I debugged this further with WinDbg and the change of protection already
occurs before any Cygwin code ran. I set a breakpoint at the start of
the DLL entry routine and at the time it's called from the Windows
loader for the DLL_PROCESS_ATTACH rat race, the protection is already
PAGE_WRITECOPY. The test application loads three other Cygwin specific
DLLs, but as far as I can see, all these libs are loaded *after* the
Cygwin DLL. So, if I didn't miss anything, the protection of this page
has actually changed before any application code has been called.

What now? I'm not overly fluent in WinDbg since I'm using GDB all the
time. Is there a way in WinDbg to break on a page protection change?
The `ba' command can't be used before the application has been started
and I don't know if it would be triggered by a protection change anyway.
I also tried to set a breakpoint to VirtualProtect, but it's not hit
at load time.

Does that description qualify for a re-evaluation of the support case at
Microsoft Professional Support, maybe?


Corinna

--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat
 
 
 

"Attempt to execute non-executable address" on Server 2008 when ??Terminal Services installed

Post by Alexander » Fri, 31 Oct 2008 12:51:54

s the function being called in a DLL or in the EXE? Does the call's target
address contain a fixup jump or actual code? Are the Cygwin DLLs loaded on
the preferred address under TS?

"Corinna Vinschen" < XXXX@XXXXX.COM > wrote in message
news:gea613$3a1$ XXXX@XXXXX.COM ...



 
 
 

"Attempt to execute non-executable address" on Server 2008 when ??Terminal Services installed

Post by Corinna Vi » Fri, 31 Oct 2008 17:48:49


In the EXE, as I described in my original posting already.


Actual code.


Yes.


Corinna

--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat
 
 
 

"Attempt to execute non-executable address" on Server 2008 when ??Terminal Services installed

Post by Corinna Vi » Fri, 31 Oct 2008 20:03:26


I debugged this further with WinDbg. I set a breakpoint to
ntdll!NtProtectVirtualMemory and followed through all calls from
starting the application until the page protection of the page in
question has changed to PAGE_WRITECOPY.

It seems there's a Terminal Service component called tsappcmd.dll, which
changes the page protection of arbitrary pages in the .text segment for
some dubious reason.

Usually you see two calls to NtProtectVirtualMemory in a row, called
from tsappcmp!TermsrvUpdateAllUserMenu+0x8xx. The first one sets the
protection to READWRITE, the next one sets it back to EXECUTE_READWRITE
or EXECUTE_WRITECOPY, whatever the original state of the page was.
However, sometimes there goes something wrong, apparently. For some
pages where the original protection was EXECUTE_WRITECOPY, the first
call sets the protection to READWRITE, as usual, but the second call
sets it to just WRITECOPY instead of EXECUTE_WRITECOPY.

So the page is left in a non-executable state and a subsequent try to
execute a function located in this page crashes with a SEGV.

This all happens already at application load time. I set a breakpoint
to the Cygwin DLLs entry function. It's never hit before the protection
has already been changed by tsappcmd.dll. So all the above already
happens before any application code ran.


Corinna

--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat
 
 
 

"Attempt to execute non-executable address" on Server 2008 when ??Terminal Services installed

Post by roger.or » Fri, 31 Oct 2008 20:31:20


Hmm, seems a bit odd that the 'READONLY' flag isn't shown.
....


Wonder whether there's an issue with the OS loader copying pages and
changing the protection?

....


I don't know of one -- but what happens if you break on
NtProtectVirtualMemory in Windbg?


It starts to look like it...

Regards,
Roger.
 
 
 

"Attempt to execute non-executable address" on Server 2008 when ??Terminal Services installed

Post by Corinna Vi » Fri, 31 Oct 2008 20:42:27


^^^^^^^^
tsappcmp, sorry.

Corinna

--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat
 
 
 

"Attempt to execute non-executable address" on Server 2008 when ??Terminal Services installed

Post by Pavel A » Sat, 01 Nov 2008 01:29:21

How this tsappcmp.dll is loaded in your process -
does TS add it to AppInitDLLs?

The "appcmp" reminds "application compatibility", so maybe there
is some appcompat rule that matches your exe - by name
or by some exe file attributes.

--PA|
 
 
 

"Attempt to execute non-executable address" on Server 2008 when ??Terminal Services installed

Post by Corinna Vi » Sat, 01 Nov 2008 02:12:16


I don't know. I never saw this DLL before I started debugging this
problem and I never had a reason before. The AppInitDLLs value is
empty, though. I assume it's a dependency of some other DLL after
installing TS, or the Windows loader does it by itself if TS is
installed.


Maybe, but how am I supposed to know the rule?


Corinna

--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat
 
 
 

"Attempt to execute non-executable address" on Server 2008 when ??Terminal Services installed

Post by Pavel A » Sat, 01 Nov 2008 05:42:33


Hmm. Seems that it doesn't help to know the rules:
they change them all the time :(

--PA
 
 
 

"Attempt to execute non-executable address" on Server 2008 when ??Terminal Services installed

Post by Pavel A » Sun, 02 Nov 2008 05:23:21

If tsappcmp.dll indeed belongs to the application compatibility
scheme of Win2008 or TS, maybe this gives more chances with the MS support.
The appcompat stuff is designed to make 3rd party apps work, not to break
them.
So maybe you can directly contact the appcompat team.

This program on MS Connect site specially addresses TS compatibilty :

https://connect.microsoft.com/tsappcompat

( to see this page you need to log on to the Connect site )

Regards,
--PA
 
 
 

"Attempt to execute non-executable address" on Server 2008 when ??Terminal Services installed

Post by Corinna Vi » Sun, 02 Nov 2008 18:38:27


Thanks for this URL! I'll try and explain the problem to them.


Corinna

--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat
 
 
 

"Attempt to execute non-executable address" on Server 2008 when ??Terminal Services installed

Post by GP » Tue, 04 Nov 2008 17:48:26

Hello Corinna!


Maybe you can report to the group if you've successfully solved the issue
with MS.


GP
 
 
 

"Attempt to execute non-executable address" on Server 2008 when ??Terminal Services installed

Post by Corinna Vi » Wed, 05 Nov 2008 18:16:57


Sure. Right now the MS support is still trying to find a way to avoid
support rather than trying to debug the problem. Stay tuned.

I have another open case which I originally discussed on this newsgroup
(quadratic timing behaviour accessing long paths in NT 5.x) which, after
5 months, still nobody at MS even tried to debug. I got a lecture about
correct testing instead. Well, hope never fades...


Corinna

--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat
 
 
 

"Attempt to execute non-executable address" on Server 2008 when ??Terminal Services installed

Post by Alexander » Wed, 05 Nov 2008 22:57:40

That's my experience with MS, too. Unless the OS crashes and burns, they
won't do anything. Even if the checked kernel ASSERTs.