x64 calling convention .c to .asm

x64 calling convention .c to .asm

Post by bmFpcm » Thu, 04 Aug 2005 09:21:02

Hey everybody,

First off, I hope I've selected the correct forum to post this in. I'm
looking for help on the calling convention for an x64 driver written in C
that calls into an Assembly file. The majority of my code is in a .c file
with a few small funtions written in assembly (no I cannot use the available
instrinsics due to 2 lines of code).

For testing purposes, I stole the ioclt sample code from the 3790 DDK (it
is very similar to what I need), and added a call to my assembly. I can
compile both files to .obj, and then link them into a .sys, but when I deploy
the driver, OSR Driver Loader (version 2.2) tells me the driver is invalid
(if I comment out just the call to the assembly function, the driver works
like I expect). I'm pretty sure my problem is either in the way I declare the
extern funtion or in the format of my assembly file.

My compiling commands are:
For my .c file: cl /c test.c --- test.obj
For my .asm: ml64 /c test64.asm --- test64.obj

My linking command is:
C:\WINDDK\3790\lib\wnet\amd64\ntoskrnl.lib test64.obj test.obj /OUT:test.sys

Does anybody here see anything wrong with my compile/link commands?

Does somebody here have information on where I can find a properly formatted
assembly file (preferably one that is called from a .c file and not .cpp),
that will compile with the DDK compiler (ml64) ?


x64 calling convention .c to .asm

Post by Ivan Brugi » Thu, 04 Aug 2005 11:44:21

There ain't msvcrt.dll in kernel mode, as far as I can remember.
If you have the DDK, try first to build the driver with
the ASM file using the standard rules for `*.asm` files
via the SOURCES file.
When the driver correctly builds and loads, you can then analyze
and attempt to replicate the result using other unsupported build mechanism.

This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at


x64 calling convention .c to .asm

Post by UGF2ZWwgQS » Thu, 04 Aug 2005 20:26:04

have you tried /subsystem:native ?

x64 calling convention .c to .asm

Post by bmFpcm » Fri, 05 Aug 2005 05:59:07

Thanks Ivan and Pavel.

My problem was solved by implementing both your suggestions (I had to remove
msvcrt.lib and use /SUBSYSTEM:NATIVE).

Thanks again,