Nirvana v2: Electric Boogaloo

Return-provenance detection for raw and gadgeted direct syscalls.

That is the shortest description of Nirvana v2: Electric Boogaloo. The name is a nod to the older Nirvana-style process instrumentation callback technique; this project is my native x64 version that makes the callback path useful for direct-syscall return provenance. It can also be read as a tripwire primitive: once a process knows where a syscall returned from, it can decide whether to keep watching, report, alter policy, or leave.

Most direct-syscall discussions start at the ntdll stub.

That is a reasonable starting point. On native x64 Windows, user mode normally enters the kernel through small syscall stubs exported by ntdll.dll, and GUI-related syscalls commonly route through win32u.dll. The classic stub shape is familiar:

mov     r10, rcx
mov     eax, <syscall_number>
syscall
ret

The offensive version of the story is also familiar. If an endpoint product hooks ntdll!NtAllocateVirtualMemory, an implant can skip the export body, load the syscall number directly, execute syscall, and avoid that user-mode hook.

The defensive version is more interesting:

skipping the exported function entry is not the same thing as returning from the expected syscall instruction.

That distinction is the core of this project.

Instead of asking only, “did this thread call ntdll!NtAllocateVirtualMemory?”, the detector asks a more precise question:

The kernel just serviced syscall X.
When control returned to user mode, did the thread return to the post-syscall
address that belongs to syscall X?

If the syscall number says NtProtectVirtualMemory, but the post-syscall return address belongs to NtQuerySystemTime, that is not a normal exported syscall. That is a syscall-gadget mismatch.

The implementation repository is here:

https://github.com/CameronBabcock/DirectSyscallDetector

Research notes: the raw WinDbg, IDA, kernel-symbol, and harness notes behind this post are published with the project as ResearchNotes.md. This article is the polished narrative; the notes preserve the evidence trail.

The Short Version

The detector does four things:

1. Builds a live syscall catalog from the currently loaded ntdll.dll and win32u.dll.
2. Installs a process instrumentation callback with NtSetInformationProcess(ProcessInstrumentationCallback).
3. Captures the previous user PC from callback-entry R10, plus stack arguments 5 and 6 from the saved user stack.
4. Uses a monitor thread to query NtQueryInformationThread(ThreadLastSystemCall) for the syscall number and first argument, then classifies the event.

The important invariant is:

syscall number ownership must match return-address ownership

That produces these verdicts:

Recovered syscall ID	Previous PC after syscall	Verdict
Known ID	Expected return for that same ID	Clean
Known ID	Not in any cataloged stub	Raw direct syscall
Known ID	In a cataloged stub for a different syscall ID	Syscall-return mismatch
Unknown ID	Any return address	Unknown syscall

The Visual Studio demo proves all three useful cases:

Probe	What happens	Expected classification
Baseline ntdll!NtAllocateVirtualMemory	Calls the real exported native API	Clean
Raw MASM NtAllocateVirtualMemory	Executes syscall from project-owned assembly	Raw direct syscall
Raw MASM NtProtectVirtualMemory	Executes another syscall from project-owned assembly	Raw direct syscall
Gadgeted NtProtectVirtualMemory	Loads the NtProtectVirtualMemory syscall number, then jumps to a syscall instruction inside ntdll!NtQuerySystemTime	Syscall-return mismatch

The final output is intentionally direct:

[baseline ntdll!NtAllocateVirtualMemory] clean
  syscall: 0x0018 (ntdll.dll!NtAllocateVirtualMemory)
  previous pc owner: ntdll.dll!NtAllocateVirtualMemory

[raw MASM NtAllocateVirtualMemory] raw direct syscall
  syscall: 0x0018 (ntdll.dll!NtAllocateVirtualMemory)
  previous pc: 0x00007FF7A6D9970B

[ntdll syscall gadget for NtProtectVirtualMemory] syscall-return mismatch
  syscall: 0x0050 (ntdll.dll!NtProtectVirtualMemory)
  previous pc owner: ntdll.dll!NtQuerySystemTime

That last line is the whole detection idea in one sentence:

This was NtProtectVirtualMemory, but it returned through NtQuerySystemTime.

Why Return Address Beats Entry-Point Thinking

A simple user-mode API monitor often starts from function entry:

Did execution enter ntdll!NtAllocateVirtualMemory?

Direct syscalls are designed to make that question less useful. They can skip the export entry entirely.

A slightly better question is:

Did the syscall instruction live somewhere inside ntdll?

That catches raw syscalls from arbitrary executable memory, but it is still too broad. An indirect syscall can borrow a syscall instruction from a clean ntdll stub while loading a different syscall number.

This project asks a narrower question:

The kernel recorded syscall ID N.
Which export owns syscall ID N?
Which export owns the address the thread returned to?
Do those owners match?

That is why the gadget probe is detected. The return PC is inside ntdll.dll, but it is inside the wrong stub.

The Native x64 Syscall Stub

For a normal native call, the ntdll stub is short. The exact bytes vary by build and patch state, but the logical shape is:

mov     r10, rcx
mov     eax, ServiceNumber
syscall
ret

There are two details worth making explicit.

First, the syscall number is loaded into EAX. Those numbers are not stable across Windows builds. A serious detector should not ship a hard-coded table and pretend it describes every machine.

Second, the expected return address for a normal stub is the instruction immediately after syscall. Since syscall is two bytes, the detector stores:

ExpectedReturnAddress = SyscallInstructionAddress + 2

That syscall + 2 address is the user-mode return anchor.

Building The Runtime Catalog

The detector builds its catalog from the loaded modules in the current process:

ntdll.dll
win32u.dll

It walks each PE export directory, skips forwarded exports, scans candidate functions for mov eax, imm32 followed by syscall, and records:

syscall number
module name
export/function name
function address
syscall instruction address
expected return address
generated PHNT-backed signature metadata

The catalog has two important indexes:

syscall number          -> one or more exported syscall stubs
expected return address -> owning exported syscall stub

This split is not cosmetic. In a mismatch case, the syscall number and return address point to different functions. The detector must use the syscall number to decide what actually ran, and the return address to decide whether the path was suspicious.

For example:

SystemCallNumber:
  0x0050 -> ntdll.dll!NtProtectVirtualMemory

PreviousProgramCounter:
  ntdll.dll!NtQuerySystemTime + 0x14

Verdict:
  syscall-return mismatch

The signature printer also follows the recovered syscall ID, not the previous PC. If a gadget returns through NtQuerySystemTime, but the kernel says the syscall was NtProtectVirtualMemory, the right arguments to print are NtProtectVirtualMemory arguments.

Generated PHNT Signatures

The project vendors PHNT and generates a compile-checked signature registry from PHNT’s native API declarations.

The generator emits factories like this:

using TFunction = std::remove_pointer_t<decltype(&::NtAllocateVirtualMemory)>;
return MakeSyscallSignature<TFunction>(
    "NtAllocateVirtualMemory",
    std::array<std::string_view, 6>{
        "ProcessHandle",
        "BaseAddress",
        "ZeroBits",
        "RegionSize",
        "AllocationType",
        "PageProtection",
    },
    std::array<std::string_view, 6>{
        "HANDLE",
        "PVOID*",
        "ULONG_PTR",
        "PSIZE_T",
        "ULONG",
        "ULONG",
    });

The decltype(&::NtAllocateVirtualMemory) part matters. It makes the generated file compile against PHNT. If the generated registry drifts away from the available declaration, the build fails instead of printing stale metadata.

At runtime, the flow is:

ThreadLastSystemCall says syscall ID 0x18
  -> catalog resolves 0x18 to NtAllocateVirtualMemory
  -> generated registry provides NtAllocateVirtualMemory metadata
  -> printer emits captured argument names, types, and values

The current demo prints only values it actually captures:

NtAllocateVirtualMemory:
  arg1 ProcessHandle
  arg5 AllocationType
  arg6 PageProtection

NtProtectVirtualMemory:
  arg1 ProcessHandle
  arg5 OldProtection

Arguments 2 through 4 are not fabricated.

What The Kernel Actually Does

This was the part I wanted to understand before trusting the technique.

The user-mode callback is installed with:

NtSetInformationProcess(
    NtCurrentProcess(),
    ProcessInstrumentationCallback,
    ...);

On the tested Windows 26100 x64 target, ProcessInstrumentationCallback is process information class 0x28 / decimal 40.

The kernel stores the callback in the process object:

_KPROCESS+0x168 InstrumentationCallback

The researched path is:

NtSetInformationProcess
  -> case ProcessInstrumentationCallback / 0x28
  -> validate callback target with MmValidateUserCallTarget
  -> lock the process
  -> write _KPROCESS.InstrumentationCallback

The validation detail matters for implementation. Use an image-backed callback thunk unless you are deliberately handling CFG-valid dynamic code.

The Return Rewrite: KiSetupForInstrumentationReturn

The most important kernel function in the notes is:

nt!KiSetupForInstrumentationReturn

The observed x64 logic is roughly:

mov rax, gs:188h
mov rdx, [rax+0B8h]
mov r8,  [rdx+168h]       ; _KPROCESS.InstrumentationCallback
test r8, r8
jnz  callback_present
ret

callback_present:
cmp word ptr [rcx+170h], 33h
jnz ret
mov rax, [rcx+168h]       ; old trap-frame Rip
mov [rcx+58h], rax        ; trap-frame R10 = old Rip
mov [rcx+168h], r8        ; trap-frame Rip = callback
ret

RCX is the trap-frame pointer.

The key offsets are:

_KTRAP_FRAME+0x058 R10
_KTRAP_FRAME+0x168 Rip
_KTRAP_FRAME+0x170 SegCs
_KTRAP_FRAME+0x180 Rsp

The kernel checks SegCs == 0x33, which is the user x64 code segment, then rewrites the trap frame:

old Rip -> trap-frame R10
callback -> trap-frame Rip

So callback entry looks like:

RIP = callback
R10 = previous user PC
RSP = original user RSP
RAX = return value / existing trap-frame RAX

That is the subtle but crucial point:

on this x64 path, callback-entry R10 is the previous PC. RAX is not.

That discovery shaped the final MASM thunk. The early research harness crashed when it treated RAX as the previous PC. Switching to R10 made the probe complete cleanly.

The research notes also checked TEB instrumentation fields:

TEB+0x2d0 InstrumentationCallbackSp
TEB+0x2d8 InstrumentationCallbackPreviousPc
TEB+0x2e0 InstrumentationCallbackPreviousSp

Those fields exist, but in the tested path they were not the source of the previous PC. The authoritative handoff was trap-frame R10.

Why The Callback Must Start In Assembly

Because the previous PC arrives in R10, the callback cannot safely begin as normal C++.

The compiler owns volatile registers. If you enter a C++ function first, the prologue and call setup may destroy the evidence before you inspect it.

The callback must be a save-first assembly thunk:

pushfq
push rax
push rcx
push rdx
push rbx
push rbp
push rsi
push rdi
push r8
push r9
push r10
push r11
push r12
push r13
push r14
push r15

; publish evidence to a fixed shared record

pop r15
pop r14
pop r13
pop r12
pop r11
pop r10
pop r9
pop r8
pop rdi
pop rsi
pop rbp
pop rbx
pop rdx
pop rcx
pop rax
popfq

mov r11, r10
jmp r11

The final project’s callback follows that idea. It saves state, filters to the armed worker thread, publishes the evidence into a fixed callback slot, waits for the monitor to acknowledge, restores state, and jumps back to the previous PC.

It does not call native APIs. It does not allocate. It does not format strings. It does not take C++ locks.

The rule is:

the callback captures; the monitor interprets

Where ThreadLastSystemCall Fits

The other kernel chain is syscall metadata.

On x64, kernel syscall entry starts by reconstructing the first argument register state:

nt!KiSystemCall64:
swapgs
...
mov rcx, r10

Later, the user syscall dispatcher stores metadata in the current thread:

nt!KiSystemServiceUser+0xc6:
mov [rbx+88h], rcx     ; _KTHREAD.FirstArgument
mov [rbx+80h], eax     ; _KTHREAD.SystemCallNumber
mov [rbx+90h], rsp     ; _KTHREAD.TrapFrame

The relevant researched offsets were:

_KTHREAD+0x080 SystemCallNumber
_KTHREAD+0x088 FirstArgument
_KTHREAD+0x090 TrapFrame
_KTHREAD+0x184 State
_KTHREAD+0x232 PreviousMode

NtQueryInformationThread(ThreadLastSystemCall) reaches that data through:

NtQueryInformationThread
  -> ThreadLastSystemCall case 21
  -> PspQueryLastCallThread
  -> _KTHREAD.SystemCallNumber
  -> _KTHREAD.FirstArgument

But this API has restrictions. The researched PspQueryLastCallThread path:

rejects the current thread
requires KTHREAD.State == Waiting
requires PreviousMode == UserMode
checks that ContextSwitches remained stable
returns FirstArgument and SystemCallNumber

That means ThreadLastSystemCall is not a same-thread, in-callback escape hatch.

If the instrumentation callback calls NtQueryInformationThread on itself, two bad things happen:

1. PspQueryLastCallThread rejects the current thread.
2. The query syscall would update the caller’s own latest syscall metadata anyway.

The final demo therefore uses a monitor thread. The callback pauses the target in user mode long enough for the monitor to query the worker thread from the outside.

Why Arguments 2 Through 4 Are Missing

This is one of the places where the detector deliberately chooses honesty over pretty output.

The research allocation harness tried to recover a direct NtAllocateVirtualMemory call:

NtStatus status = DirectNtAllocateVirtualMemory(
    (HANDLE)-1,
    &base,
    0,
    &region_size,
    MEM_RESERVE | MEM_COMMIT,
    PAGE_READWRITE);

The callback could recover:

previous PC from callback-entry R10
syscall number from bytes near the direct stub
return status from RAX
stack arg 5 from saved user RSP
stack arg 6 from saved user RSP

It could not reliably recover:

arg1 from callback-entry RCX
arg2 from callback-entry RDX
arg3 from callback-entry R8
arg4 from callback-entry R9

The observed callback-entry state for the first four volatile argument registers did not match the original user-mode function arguments. The caller home-space slots were not useful either.

That is why the final detector prints:

ThreadLastSystemCall:
  first argument

callback stack snapshot:
  fifth argument
  sixth argument

For NtAllocateVirtualMemory, that is enough to show:

ProcessHandle(HANDLE)=0xFFFFFFFFFFFFFFFF,
AllocationType(ULONG)=0x00003000,
PageProtection(ULONG)=0x00000004

0x3000 is MEM_RESERVE | MEM_COMMIT. 0x4 is PAGE_READWRITE.

That is useful telemetry, but it is not pretending to be a full argument trace.

The Callback Ring

The final project uses a small global callback ring:

constexpr std::size_t CallbackSlotCount{ 16 };

struct alignas(64) CallbackSlot
{
    volatile LONG State{};
    ULONG Reserved0{};
    volatile std::uint64_t ThreadId{};
    volatile std::uint64_t PreviousProgramCounter{};
    volatile std::uint64_t StackPointer{};
    volatile std::uint64_t StackArgument5{};
    volatile std::uint64_t StackArgument6{};
    volatile std::uint64_t Reserved1{};
    volatile std::uint64_t Reserved2{};
};

The state machine is intentionally tiny:

Free -> Writing -> Ready -> Free

The MASM callback:

1. claims a free slot with an interlocked compare-exchange;
2. writes thread ID, previous PC, stack pointer, stack argument 5, and stack argument 6;
3. marks the slot ready;
4. waits until the monitor clears it.

The monitor:

1. waits for a ready slot;
2. suspends or opens the worker thread as needed;
3. calls NtQueryInformationThread(ThreadLastSystemCall);
4. copies the callback record;
5. clears the slot;
6. classifies the event.

This is still a demo, not a production event transport. A production sensor would need better answers for lost events, callback ownership, timeout behavior, cross-process deployment, telemetry export, protected process boundaries, and overhead. The ring is there to keep the callback path minimal and to make the concurrency boundary explicit.

Classification Logic

Once the monitor has both data sources, the classifier is simple.

observedEntry = catalog.FindPrimaryBySyscallNumber(SystemCallNumber)
returnOwner   = catalog.FindByReturnAddress(PreviousProgramCounter)

if query failed:
    QueryFailed
else if observedEntry is null:
    UnknownSyscall
else if returnOwner is null:
    RawDirectSyscall
else if returnOwner.Number != observedEntry.Number:
    SyscallGadgetMismatch
else:
    Clean

That logic catches two different bypass shapes.

Raw Direct Syscall

The demo’s raw MASM helper does this:

mov r10, rcx
mov eax, dword ptr [g_DsdNtAllocateVirtualMemorySyscall]
syscall
ret

The syscall number is valid because it came from the live catalog. The previous PC is suspicious because it points back into the demo executable’s assembly helper.

observedEntry: NtAllocateVirtualMemory
returnOwner:   none
verdict:       raw direct syscall

Syscall Gadget Mismatch

The gadget helper does this:

mov r10, rcx
mov eax, dword ptr [g_DsdNtProtectVirtualMemorySyscall]
mov r11, qword ptr [g_DsdGadgetSyscallInstruction]
jmp r11

g_DsdGadgetSyscallInstruction points to the syscall instruction inside a different ntdll stub, such as NtQuerySystemTime.

The syscall ID is NtProtectVirtualMemory. The return PC owner is NtQuerySystemTime.

observedEntry: NtProtectVirtualMemory
returnOwner:   NtQuerySystemTime
verdict:       syscall-return mismatch

This is the key difference from “return address is inside ntdll” checks. The detector is not allergic to ntdll. It is checking whether the right ntdll stub owns the return.

Tripwire Capabilities

The demo classifies events and prints them, but the same primitive can support a more active tripwire model.

At callback time, the process has three useful facts:

what syscall just completed
where user mode is about to resume
what status value is currently in RAX

That is enough to separate passive telemetry from policy. The monitor can decide that a known-good return path is just evidence, an unknown return path is worth logging, and a raw or gadgeted path is a tripwire event.

This is especially interesting because the signal is not tied to a single API hook. A module can avoid ntdll!NtProtectVirtualMemory and still leave a post-syscall return address. A gadget can borrow a clean syscall instruction and still return through the wrong stub. The tripwire watches the return provenance after the kernel has already done the work.

Scenario	Tripwire signal	Possible policy response	Why it matters
Anti-cheat or anti-tamper module in a game process	Unfamiliar module-owned return paths around memory query, protection, or handle syscalls	Record, raise local confidence, or move the protected component into a safer mode	Detects inspection behavior without relying only on hooked API entry points
Offensive malware, anti-EDR, or anti-AV research in an authorized lab	Raw or gadgeted syscall activity near sensitive allocation, protection, section, thread, or handle operations	Self-unload, reduce capability, delay, or emit operator telemetry	Gives an agent a way to notice that its assumptions about the process changed
EDR canary process or protected sensor component	Unexpected direct-syscall provenance inside a process that should have boring syscall paths	Generate tamper telemetry, snapshot context, or correlate with image-load and memory events	Converts a low-level primitive into a higher-confidence tamper signal
Productized endpoint sensor	Known syscall ID with mismatched return owner, plus suspicious module provenance	Enrich the event with module identity, loaded-image history, and memory provenance	Helps distinguish noisy native API usage from deliberate syscall-path manipulation

The return-value angle is separate and should be treated carefully. On the researched x64 path, callback-entry RAX contains the syscall return status. The current project preserves it and returns to the previous PC; it is observe-only. A policy-aware variant could choose to preserve, normalize, or fail a result before resuming user mode, but that belongs in controlled research or productized sensor logic, not in an indiscriminate demo.

That distinction matters for public work. The useful idea is not “here is a trick to interfere with a named product.” The useful idea is:

post-syscall return provenance can become a process-local tripwire

For defensive engineering, that can mean agent self-protection, canary processes, or tamper-aware telemetry. For CNO and pentest agent work, it can mean authorized self-awareness: unload if the environment starts behaving in a way the agent has never seen before, or report that an anti-cheat, anti-EDR, anti-AV, or other inspection component appears to be present.

Evidence Trail

Here is the compact claim-to-evidence map from the notes and final project:

Claim	Evidence
ProcessInstrumentationCallback is set through NtSetInformationProcess, class 0x28 / decimal 40	IDA switch case and live NtSetInformationProcess breakpoint
The callback is stored in _KPROCESS.InstrumentationCallback	WinDbg dt nt!_KPROCESS Instrumentation* and live store at NtSetInformationProcess+0x2330
x64 previous PC is delivered in callback-entry R10	KiSetupForInstrumentationReturn disassembly, trap-frame before/after, callback-entry registers
RAX is not previous PC	Trap-frame and callback-entry values; allocation probe showed RAX as syscall return status
TEB instrumentation previous-PC fields were not the source in this path	TEB field dump at callback entry
ThreadLastSystemCall reaches PspQueryLastCallThread	IDA NtQueryInformationThread case 21 and WinDbg disassembly
PspQueryLastCallThread reads _KTHREAD.FirstArgument and _KTHREAD.SystemCallNumber	WinDbg disassembly and _KTHREAD offsets
Cross-thread blocked syscall metadata can be queried	Direct NtWaitForSingleObject harness matched event handle and syscall number
Direct allocation callback can recover previous PC, return status, and stack args 5/6	Direct NtAllocateVirtualMemory harness output
First four arguments are not reliable at callback entry	Allocation harness register and home-space results
Final detector catches clean, raw, and gadget cases	Visual Studio demo verdict checks

The full raw evidence is in ResearchNotes.md.

Reproducible Project Shape

The solution has two projects:

DirectSyscallDetectorLib
  static library
  C++23 / latest MSVC mode
  MASM enabled
  PHNT vendored under third_party/phnt

DirectSyscallDetectorDemo
  console app
  links the library
  runs the four probes

The intended verification commands are:

& 'C:\Program Files\Microsoft Visual Studio\18\Insiders\MSBuild\Current\Bin\amd64\MSBuild.exe' `
  .\DirectSyscallDetector.sln /m /p:Configuration=Debug /p:Platform=x64

& 'C:\Program Files\Microsoft Visual Studio\18\Insiders\MSBuild\Current\Bin\amd64\MSBuild.exe' `
  .\DirectSyscallDetector.sln /m /p:Configuration=Release /p:Platform=x64

.\x64\Release\DirectSyscallDetectorDemo.exe

The demo returns nonzero if an expected verdict fails:

expect baseline ntdll allocate: pass
expect raw allocate: pass
expect raw protect: pass
expect ntdll gadget protect: pass

Productization Notes

This is an in-process research demo, not an injectable EDR sensor.

That distinction matters. The signal is useful, but a production design has more work to do:

• decide how to install or broker callbacks across many target processes;
• handle processes where another component already owns the instrumentation callback;
• avoid unacceptable overhead in syscall-heavy workloads;
• design backpressure and lost-event behavior;
• export telemetry without calling complex APIs inside the callback;
• account for PPL and protected-process boundaries;
• correlate with image loads, memory provenance, ETW, call stacks, and suspicious allocation/protection behavior;
• track Windows-build differences in private callback and ThreadLastSystemCall behavior.

There are also research questions worth keeping open:

• What happens when an attacker borrows a same-ID alias stub?
• How should a sensor treat patched or remapped syscall stubs?
• How often do legitimate components use syscall gadgets or custom syscall thunks?
• Which syscall families produce the best detection value relative to callback overhead?
• How stable is this path across Windows versions and processor modes?

Those are not reasons to ignore the signal. They are the engineering questions that turn a sharp primitive into a deployable detection.

What To Remember

The mental model is:

1. Catalog every live syscall stub in ntdll.dll and win32u.dll.
2. Store syscall number -> syscall metadata.
3. Store syscall + 2 return address -> stub owner.
4. Install a process instrumentation callback.
5. Capture callback-entry R10 as previous PC.
6. Capture stack args 5/6 from saved user RSP.
7. Query ThreadLastSystemCall from a monitor thread for syscall ID and arg1.
8. Compare syscall-ID ownership to return-address ownership.

The return PC tells you whether the path was suspicious. The recovered syscall ID tells you what actually ran.

That is why the detector can say:

This was NtProtectVirtualMemory,
but it returned through NtQuerySystemTime's syscall stub.

That is a much better sentence than “the return address was in ntdll.”

Closing

I built this because I wanted a direct-syscall detector that was grounded in the kernel handoff, not just another hard-coded syscall-number trampoline.

Runtime cataloging, process instrumentation callbacks, ThreadLastSystemCall, and PHNT-backed generated signatures fit together into a clean model:

after the kernel did the work, where did the thread come back?

For direct and gadgeted syscalls, that question carries a lot of signal.

If you work on EDR, CNO, pentest agent work, endpoint security, Windows internals, or detection engineering and this kind of research is relevant to your team, I would be happy to talk.

• Email: [email protected]
• LinkedIn: https://www.linkedin.com/in/cameronbabcock/