TL;DR
Me trying to find ways to make a CTF challenge that is hard for LLMs to solve in memory forensics. The focus is on finding ways to bypass some plugins in Volatility.
Background
Random thoughts
As I was working at my company, I realized that I have come far in my cybersecurity journey. I have been participating in CTFs for a while, and now instead of only solving them, I am creating my own challenges for others to solve. But LLMs these days are really good at solving CTF challenges crazy fast. CTFs are becoming less fun to solve because of that, everyone just “slopping” now. So it came to my mind: how can I make a CTF challenge that is not solvable by LLMs, or at least make it harder for them to solve?
LLMs are very strong at text-based and code-based challenges. If a challenge can be copied into a prompt, reduced to a few files, or turned into a clear mechanical task, an LLM-assisted solver has a major advantage. It can summarize code, detect common patterns, write scripts, and suggest possible attack paths much faster than most humans. At first, I thought the answer was to simply make the challenge bigger: more files, larger logs, huge binaries, and heavier artifacts. But size alone is no longer enough. Even with large files, solvers can use tools like strings, grep, jq, CLI utilities, or MCP integrations with tools like IDA to quickly extract useful artifacts. The LLM does not need to read every byte. It only needs the human to feed it the right evidence. Or you just have tons of tokens to spend and let the LLM do the work, that’s one way to go lol.
It took me a while to realize that the answer is not to make the challenge bigger, but to make it more complex. The challenge should be designed in a way that requires human intuition, creativity, and problem-solving skills that LLMs currently lack. This could involve creating challenges that require understanding of context, making connections between seemingly unrelated pieces of information, or requiring knowledge of specific domains that are not easily accessible to LLMs.
Intentions
Now what I have in mind is a forensics challenge that needs human intuition to solve, or at least includes a “human in the loop” with LLMs. That gave me a few options to consider, but I decided to go with memory forensics.
My mentor (@teebow1e) said this:
Mememmememe, the two most popular memory forensics tools are Volatility and MemProcFS. These tools are commonly used and still getting updates. AND THEY ARE OPEN SOURCE!!!
Because these tools are open source, I can analyze the source code and understand how they work. That helps me create a challenge that these tools may have trouble parsing, thus giving LLMs a harder time solving it.
Volatility analysis
Cloning the source code took no time zzzz. First, let’s check out the plugins that are available in Volatility. My focus was on the windows plugins since I wanted to create a challenge based on Windows memory forensics.
The docs for these plugins can be found here. I was looking for plugins that are common in malware analysis. Some plugins can be skipped for this research because they are normal first-pass plugins to try when analyzing a memory dump, like pslist, pstree, cmdline, filescan, …
Let’s dive in the plugins that I found interesting and can be used to create a challenge that is hard for LLMs to solve.
malfind plugin
Basis
Located in volatility3/volatility3/framework/plugins/windows/malware/malfind.py
Its own description says:
class Malfind(interfaces.plugins.PluginInterface):
"""Lists process memory ranges that potentially contain injected code."""In simple words, malfind tries to find memory regions that look like injected code.
It does not prove that something is malware. It only says:
This memory region looks suspicious.
The plugin mainly checks VADs.
Explanation - VAD
VAD, or Virtual Address Descriptor, is a data structure used by the Windows operating system to manage the virtual memory of a process. Each process has its own virtual address space, which is divided into pages. The VAD keeps track of which pages are allocated, which are free, and what type of memory they represent (e.g., code, data, stack, heap).
Example:
0x10000000 - 0x1000ffff : read/write memory
0x20000000 - 0x2002ffff : executable image mapping
0x30000000 - 0x30001fff : private shellcode-like memoryHow it works
The plugin’s file structure is as follows:
| File | Role |
|---|---|
plugins/windows/malware/malfind.py | main malfind logic |
plugins/windows/pslist.py | gives malfind the process list |
plugins/windows/vadinfo.py | helps decode VAD protection and dump VADs |
symbols/windows/extensions/__init__.py | has VAD helpers like traverse(), get_tag(), get_start(), get_end() |
The flow of the plugin is as follows:
So the short version is:
process -> VAD -> protection -> dirty pages -> outputFirst part: Process selection
malfind first gets processes through pslist:
So the plugin starts from the normal Windows process list.
This means malfind can be used against all processes or only selected PIDs:
python vol.py -f memory.raw windows.malware.malfind
python vol.py -f memory.raw windows.malware.malfind --pid 556So malfind could be bypassed if the process is not in the process list. But that would be kind of guessy if the player does not know what process to look for. So this is not what I want to do.
Second part: VAD scanning
After choosing a process, malfind walks the process VAD tree:
proc.get_vad_root() is a helper function from Volatility’s Windows _EPROCESS extension. It finds the root of that process’s VAD tree while traverse() walks the VAD tree and returns each VAD one by one.
So malfind could be bypassed if Volatility’s VAD walk is broken.
In symbols/windows/extensions/__init__.py the following line could help with that:
While walking the tree, Volatility also checks the VAD pool tag to decide whether the node is really a VAD. That means if the pool tag before the target VAD is corrupted, traverse() may never return that VAD to malfind, thus weakening the plugin. But messing with the pool tag is not easy. It requires kernel-level access to the system. I will come back to this later.
Third part: Protection check
The most important check is memory protection.
malfind cares about executable memory, especially memory that is also writable.
Examples that look suspicious:
| Protection | Symbol | Why |
|---|---|---|
PAGE_EXECUTE_READWRITE | RWX | code can run and also be changed |
PAGE_EXECUTE_WRITECOPY | RX + COW | code can run, and writes create a private changed copy |
Normal-looking examples:
| Protection | Symbol | Why |
|---|---|---|
PAGE_READONLY | R | readable only, cannot execute |
PAGE_READWRITE | RW | writable, but cannot execute |
PAGE_EXECUTE_READ | RX | executable, but not directly writable |
protect_values() reads MmProtectToValue from the memory dump.
This can be bypassed by avoiding obvious suspicious protection values like RWX or RX + COW. A common trick is to write the payload while the memory is RW, then change it to RX. At the time of analysis, the memory no longer looks writable, so it looks more normal.
But that is not enough. malfind has a second check for dirty pages.
Fourth part: Dirty page check
Explanation - Dirty pages
A dirty page is a page that has been modified after it was loaded or mapped into memory. So even if the page is now only RX, the dirty bit can still show that the page was changed before.
As mentioned above, an injector can avoid leaving memory as RWX by using RW -> RX: write the payload first, then change the page protection to executable-read. However, malfind also checks whether executable pages are dirty.
In the source, malfind does this only when the VAD is executable but not writable:
So the logic is: if the memory is executable, but it is not writable anymore, check whether any page was modified before.
This makes the simple RW -> RX trick weaker. The memory may look clean from its current protection, but the dirty page check can still reveal that something changed inside it.
So to bypass this part, the payload should avoid looking like modified executable memory. For example, module stomping is more interesting here because the VAD can still look like a normal image mapping, such as PAGE_EXECUTE_WRITECOPY, instead of a simple private RW -> RX shellcode region.
But this also depends on how the memory is mapped and how the dirty bit appears in the dump, so this is not a perfect rule. It is better to think of malfind this way: it catches many common injection patterns, but not all of them.
Flow summary
dlllist plugin
Basis
Located in volatility3/volatility3/framework/plugins/windows/dlllist.py
Its description says:
class DllList(interfaces.plugins.PluginInterface, timeliner.TimeLinerInterface):
"""Lists the loaded DLLs in a particular windows memory image."""In simple words:
dlllistshows the modules loaded by a process.
This is useful because malware often uses DLL injection to hide inside a normal process. DLL injection has a few variants, from classic DLL injection to more advanced ones like reflective DLL injection. So this plugin is very useful for malware analysis.
How it works
The plugin’s file structure is as follows:
| File | Role |
|---|---|
plugins/windows/dlllist.py | main plugin |
plugins/windows/pslist.py | gives dlllist the process list |
plugins/windows/psscan.py | used when checking a process by physical offset |
plugins/windows/pedump.py | dumps DLLs when --dump is used |
symbols/windows/extensions/__init__.py | has PEB and loader-list helpers |
symbols/windows/extensions/pe.py | reconstructs PE files when dumping |
The flow of the plugin is as follows:
The short flow is:
process -> PEB -> loader list -> DLL entry -> outputFirst part: Process selection
dlllist first gets processes through pslist:
So the plugin starts from the normal Windows process list, same idea as malfind.
This means dlllist can be used against all processes or only selected PIDs:
python vol.py -f memory.raw windows.dlllist
python vol.py -f memory.raw windows.dlllist --pid 556It also supports checking a process by physical offset:
python vol.py -f memory.raw windows.dlllist --offset 0x12345678When --offset is used, it uses psscan to find that process object instead of the normal pslist route.
So dlllist could also be bypassed if the target process is not in the normal process list. But same as malfind, this is kind of guessy and not very fun for the player. If the player does not know what process to look for, hiding the whole process just makes the challenge annoying.
Second part: PEB loader list
After choosing a process, dlllist walks the loaded module list of that process:
load_order_modules() is a helper from Volatility’s Windows _EPROCESS extension. It walks the loader list from the process PEB.
Explanation - PEB
PEB or Process Environment Block is a user-mode structure that stores information about a process. One important part of the PEB is Ldr, which keeps track of loaded modules like DLLs.
So when Windows loads a DLL normally, the DLL usually appears in the PEB loader lists.
Each list entry is a _LDR_DATA_TABLE_ENTRY. This structure contains information like:
| Field | Meaning |
|---|---|
DllBase | where the DLL is mapped in process memory |
SizeOfImage | size of the DLL image |
BaseDllName | short DLL name |
FullDllName | full DLL path |
LoadTime | when the DLL was loaded, if available |
This is the main strength of the plugin. It gives clean module metadata very quickly.
But this is also the first big weakness. If the malware unlinks its DLL from the PEB loader list, dlllist may not show it anymore. This technique is usually called PEB unlinking.
In simple words: the DLL is still mapped in memory, but the loader list no longer points to it, so dlllist does not see it.
For CTF challenges, this is useful because the process can still be visible, but the suspicious DLL metadata can be hidden.
Third part: DLL metadata output
For each DLL entry, dlllist prints these columns:
PID
Process
Base
Size
Name
Path
LoadCount
LoadTime
File outputThe most important fields for analysis are:
| Column | Why it matters |
|---|---|
PID / Process | tells which process loaded the DLL |
Base | tells where the DLL starts in memory |
Size | tells how large the image is |
Name | short DLL name |
Path | full DLL path |
File output | tells whether dumping worked |
Example output:
This kind of output is very useful for CTF players. If a process loads a weird DLL, that is already a strong clue.
But for a challenge author, this is also dangerous. If the plugin shows the stomped module name directly, then an LLM can reason from the metadata without needing to dump anything.
So hiding or misleading this metadata becomes important.
Fourth part: Dumping DLLs
dlllist can also dump DLLs with --dump:
python vol.py -f memory.raw windows.dlllist --pid 556 --dumpWhen --dump is used, dlllist calls pedump:
So dlllist --dump reads the DLL from process memory using the DllBase from the loader entry.
Explanation - DllBase
DllBase is the base address where the DLL is mapped inside the process.
If a DLL is loaded at 0x7ffb100000, then pedump expects to find a valid PE file starting at that address.
This dump feature is very useful when the DLL is normal. But it has one very clear weakness: the PE header must still be valid.
When dumping, the PE helper checks things like:
| Field | Expected value |
|---|---|
| DOS magic | MZ |
| NT header | PE |
SizeOfImage | reasonable size |
| section table | valid section information |
So if the PE header is stomped, dlllist --dump can fail.
This is why PE header stomping can bypass the dump part of dlllist.
Flow summary
ldrmodules plugin
Now why ldrmodules? After I found out how to bypass malfind and dlllist, I created a challenge that implemented those bypasses. I gave the challenge to a few people to “slop” and it took about 1 hour to 2 hours to solve it. The LLMs did not get a good result from malfind and dlllist, but they still managed to find the necessary information to solve the challenge with ldrmodules. So I needed to find a way to bypass ldrmodules as well.
Basis
Located in: volatility3/volatility3/framework/plugins/windows/ldrmodules.py
Its description says:
class LdrModules(
interfaces.plugins.PluginInterface,
deprecation.PluginRenameClass,
replacement_class=ldrmodules.LdrModules,
removal_date="2026-06-07",
):
"""Lists the loaded modules in a particular windows memory image."""In simple words:
ldrmodulescompares loaded DLL metadata with mapped PE files in memory.
This plugin is interesting because it is usually used after dlllist.
dlllist trusts the PEB loader list. If malware unlinks a DLL from the PEB, dlllist may miss it. But ldrmodules also checks VADs, so it can catch that kind of hiding.
So this plugin is basically asking:
Is this mapped PE file also present in the loader lists?How it works
The plugin’s file structure is as follows:
| File | Role |
|---|---|
plugins/windows/malware/ldrmodules.py | main ldrmodules logic |
plugins/windows/ldrmodules.py | wrapper for the plugin |
plugins/windows/pslist.py | gives ldrmodules the process list |
plugins/windows/vadinfo.py | walks VADs for the process |
symbols/windows/extensions/__init__.py | has PEB loader-list helpers and VAD helpers |
symbols/windows/extensions/pe.py | gives PE header structures like _IMAGE_DOS_HEADER |
The flow of the plugin is as follows:
First part: Process selection
ldrmodules first gets processes through pslist:
So same as malfind and normal dlllist, it starts from the normal Windows process list.
This means it can be used against all processes or only selected PIDs:
python vol.py -f memory.raw windows.ldrmodules
python vol.py -f memory.raw windows.ldrmodules --pid 556So yes, hiding the whole process from pslist could affect this plugin. But same as before, this is not the most interesting bypass for a CTF because it makes the player guess what process to hunt for.
Second part: PEB loader lists
For each process, ldrmodules walks three loader lists:
These are the three common PEB loader lists:
| List | Output column | Meaning |
|---|---|---|
InLoadOrderModuleList | InLoad | DLL appears in load order |
InInitializationOrderModuleList | InInit | DLL appears in init order |
InMemoryOrderModuleList | InMem | DLL appears in memory order |
This is the part that catches bad PEB unlinking.
If a DLL is removed from one list but not the others, ldrmodules can show something weird like:
InLoad InInit InMem
False True TrueIf it is removed from all three lists, but still mapped in memory, it can show:
InLoad InInit InMem
False False FalseThus, the module looks suspicious.
Third part: VAD scanning
After reading the PEB loader lists, ldrmodules walks the process VADs:
Inside vadinfo, this eventually uses:
This is the main reason ldrmodules is stronger than dlllist. Even if the DLL is removed from the PEB loader list, the memory mapping may still be visible in the VAD tree. So to bypass this, I came back to the idea that I mentioned earlier about corrupting the pool tag before the target VAD.
Fourth part: PE header check
For each VAD, ldrmodules checks whether the VAD starts with an MZ header:
This is a very important bypass point.
If the VAD exists, but the first bytes no longer look like a PE file, ldrmodules skips it.
So PE header stomping that I found earlier that bypasses dlllist --dump can also bypass ldrmodules.
Flow summary
Designing a working bypass
Further analysis of the Volatility plugins that LLMs used to solve the first version of my challenge showed that many of those paths depend on the VAD tree. That made VAD pool tag corruption a good target.
The idea is simple: if Volatility cannot reliably walk the relevant VAD nodes, plugins such as malfind, vadinfo, vadtree, vaddump, and parts of dumpfiles lose the memory region that contains the planted payload.
The hard part is that VAD metadata lives in kernel memory. Modifying it from user mode is not realistic on modern Windows. Older Windows versions may be easier because public kernel bugs exist, but I wanted the challenge to run on a more realistic Windows 10 setup.
So I used a simpler and more practical route: BYOVD, or Bring Your Own Vulnerable Driver. The program drops and loads a known vulnerable signed driver, then uses the driver’s read/write primitive to modify kernel memory.
The final bypass chain became:
memory-only private PE staging
-> loader metadata scrubbing
-> BYOVD kernel write
-> VAD pool tag corruptionVisualizing the VAD
Every Windows process has an EPROCESS object in kernel memory. Inside that object is a pointer to the process VAD tree.
As I said earlier, VAD means Virtual Address Descriptor. A VAD node describes a range of virtual memory inside the process.
This is the layout that I think is correct. Lul
The left side is the process. The middle is the tree used for lookup. The right side is the content of one VAD node.
The two VAD forms are not two separate concepts. They are two structure sizes:
_MMVAD_SHORT = enough fields for a basic memory range
_MMVAD = _MMVAD_SHORT-style range data plus extra mapped-file informationVPN means Virtual Page Number. Windows describes VAD ranges by page number. On x64 Windows, a normal page is 0x1000 bytes:
VPN = virtual_address >> 12
virtual_address = VPN << 12
So a VAD is not saying this allocation starts at byte 0x001a0000 directly. It stores the page range, and the address is reconstructed by shifting the VPN back left by 12 bits.
P/S: Sometimes I don’t even know what I am writing when I review it later. So I hope this is correct and you can understand it :P
I want the bytes to remain recoverable for the player. I only want to weaken the automatic metadata path for the tools.
Why Pool Tags Matter
VAD objects live in kernel pool memory. Kernel pool allocations have a small header before the actual object. One field in that header is the pool tag.
Pool tags are short four-byte labels used for debugging, accounting, and object identification.
For VAD allocations, the tag usually looks like a VAD-related value:
VadS
VadF
Vadl
Vad The memory layout kinda looks like this:
Volatility uses this kind of information as a sanity check. When it walks VADs, it does not only follow pointers. It also tries to decide whether the object it reached looks like the expected VAD type.
Remember this snippet of code:
If a non-root VAD node does not have a VAD-looking pool tag, Volatility can drop that node and its subtree.
So if the pool tag changes from:
VadSto something that does not look like a VAD tag:
GIBBERISHthen Windows may still have a usable VAD object, but Volatility may reject it during offline parsing. That creates the mismatch I wanted.
Before and after corruption
Before the VAD pool tag corruption, Volatility can do this:
The memory layout:
After the VAD pool tag corruption, the payload bytes are still there, but the parser-facing metadata is damaged.
The memory layout:
The payload is still mapped in memory, but Volatility’s VAD parsing path is no longer reliable, so the obvious Volatility route is weakened.
What does this break?
This mainly affects plugins that begin with VAD traversal:
The practical result is:
- malfind does not print the useful private executable region
- vadinfo/vadtree do not give a clean address range
- vaddump cannot directly dump the region
- dumpfiles loses VAD-backed file discovery
- ldrmodules loses part of its cross-checking evidence
This does not make every plugin useless. Basic triage can still work:
- pslist
- pstree
- cmdline
- filescan
- handles
- netscan
- registry plugins
Other possibilities?
There are many things that could be corrupted in theory:
VAD tree links
StartingVpn / EndingVpn
Subsection
ControlArea
Segment
prototype PTE chains
process handle tableBut those structures may still be used by the live Windows kernel. If I break one of them too aggressively, the process can crash or the whole VM can bugcheck before the memory dump is captured. But that will be an interesting topic for future research or challenges. Am I right? :P
Making the design work!
As I said before, to use BYOVD to modify kernel memory, I need a vulnerable driver that is signed and can be loaded on Windows 10. I also need a driver that is old enough to be public and documented, so I can understand how to use it.
So I used RTCore64.sys, the MSI Afterburner driver affected by CVE-2019-16098. It is old, public, and well documented, which makes it practical for a CTF challenge while still demonstrating the real technique.
Credits and References
Credit for the vulnerability and public PoC:
- NVD entry: CVE-2019-16098
- Public PoC/research I used as reference: Barakat/CVE-2019-16098
One thing to note is that Barakat’s PoC was made for a specific Windows version. If you want to reuse it on a different Windows build, you cannot just copy and run it. You have to edit the PoC and fix the Windows-specific offsets first.
What the vulnerable driver gives
The important bug is not “execute shellcode in kernel”. The useful part is that user mode can ask the driver to read or write kernel virtual memory.
So the vulnerable driver becomes a small kernel memory read/write helper.
In the implementation, the useful IOCTLs are:
0x80002048 = read kernel memory
0x8000204c = write kernel memoryExplanation - IOCTL
IOCTL means I/O control code. It is basically a command number sent from a user-mode program to a driver.
The two hex values above are not random. They are command IDs that RTCore64.sys understands. One command reaches the driver’s memory-read path, and the other reaches the memory-write path. These values come from the public PoC and from looking at the driver’s device-control handler.
The user-mode program calls DeviceIoControl() and gives Windows:
driver handle
IOCTL command number
input buffer
output buffer
The program sends a small 48-byte buffer to the driver. The buffer contains the kernel address, the access size, and the value for writes.
Runtime flow
So I made the BYOVD layer run after the payload is planted in the target process. The program loads the vulnerable driver, then uses the driver’s read/write primitive to modify the target process VAD pool tag.
The implementation drops the driver to temp, creates a kernel service with sc.exe, starts it, then opens the driver device. This layer needs admin permission because loading a driver needs admin. If the driver does not load, the program just skips this layer and keeps the user-mode tricks.
One annoying part of kernel memory work is offsets.
EPROCESS and MMVAD fields are not a stable public API. If the offset is wrong, the code may read a random value and feed it back to the driver as a kernel pointer. That can crash the VM, which occurred to me a few times and caused a bunch of BSODs and snapshot restores.
Finding the target process in kernel memory
From user mode, I have the target PID. But to modify the VAD, I need the kernel EPROCESS address for that PID.
The implementation for this is simple:
This is basically the kernel version of walking the process list.
It also needs to check that every pointer looks like a real x64 kernel pointer before using it again. This matters because the vulnerable driver will dereference whatever address it is given. Bad pointer means possible bugcheck.
From EPROCESS to VAD root
Once the target EPROCESS is found, the code reads the process VAD root.
For the tested Windows 10 builds, the important offset is:
EPROCESS + 0x7D8 = VadRoot
VadRoot + 0x00 = Root pointerThis matched the Volatility side too. Volatility’s get_vad_root() has a Windows 8.1/Windows 10 path that treats VadRoot as an _RTL_AVL_TREE and dereferences its Root.
This part had to be correct. Reading the old-style field layout would give the wrong pointer and the whole VAD corruption would silently do nothing.
My intention for the planted PE is that it will be in private process memory. The implementation knows the base and size because it allocated and wrote the bytes itself.
For each VAD node, I read the modern Windows 10 range fields:
StartingVpn at VAD + 0x18
EndingVpn at VAD + 0x1C
StartingVpnHigh at VAD + 0x20
EndingVpnHigh at VAD + 0x21Then I reconstruct the virtual addresses:
startVa = StartingVpn << 12
endVa = (EndingVpn + 1) << 12If the VAD range overlaps the planted PE range, that is the VAD I want to damage.
Full implementation flow
Here’s a graph of what I have in mind for my BYOVD implementation. Pretty messy, but I hope it is understandable.
I have already created a challenge that implements this design. I will write about the challenge in another post since it is for a competition and I do not want to spoil it for the players. But I can say that the challenge is solvable with the right tools and techniques. LLMs do have a hard time solving it, but it is not impossible. This does not guarantee that the LLMs can’t solve it, a frontier model with sufficient tokens or if a human can steer the LLM to the right path, it can still solve it pretty quickly. But isn’t that the point of LLMs, to help humans solve problems faster? :P