</th>	iOS</th>	OS X</th></tr></thead>
Minimum tested version</td>	7.1</td>	10.11.1</td></tr>
Fixed version</td>	9.3.2</td>	10.11.5</td></tr> </tbody></table> #</a > Other findings</h1> In exploring this bug I thought it was useful to test out various applications to see how they would handle this type of invalid image. Since the chunks all contain a CRC32 of the data, you cannot modify the image outright and then upload it to almost any service. Twitter and imgur were two hosts I tried and they both would not accept the image because of the invalid CRC32. I wrote a simple Go utility</a> that iterates the chunks and fixes any invalid CRCs.</p> After fixing the chunk both hosts accepted the image just fine. imgur (and likely most other hosts) do not bother to strip unknown chunks so uploading the image to imgur puts those users at risk. Twitter and Facebook on the other hand will re-encode any image as a JPEG which will obviously remove the malicious chunk.</p> I think that this is interesting and really important for privacy-concerned people. Before investigating this bug I had not considered additional chunks not</em> being stripped when uploading an image to services. It makes sense, but it seems like this would be an easy way for vendors to hide additional info about the device which took the image outside of the EXIF-related chunks and have them survive re-encoding.</p> #</a > Timeline</h1> Dec 16, 2015: Reported vulnerability to vendor</li> Dec 17, 2015: Vendor acknowledged vulnerability</li> Dec 27, 2015: Posted pic to Twitter to see what would happen</li> Dec 27, 2015: Vendor said the bug was undergoing triage</li> Mar 21, 2016 (91 days since disclosure): iOS 9.3 released and bug still not fixed, status update requested</li> Mar 22, 2016: Vendor notified me that a fix is "in progress"</li> April 22, 2016: Public disclosure</li> May 16, 2016: iOS 9.3.2 and macOS 10.11.5 were released, giving this vulnerability CVE-2016-1811</li> </ul> If you'd like to have a sample image, you can find it here</a>. NOTE</strong> if you are using Safari, your browser's renderer process will crash. If you aren't, you will see a screenshot of a Kanye tweet.</p> tl;dr</strong> a custom PNG chunk with a 0-length data field will trigger null pointer dereference causing the application to crash similar to the CoreText</a> crash from 2013.</p> shoutouts to my boys ed snowden and james comey</p> Finding a CSRF vulnerability in phpBB 2016-01-25T22:05:08-08:00 The phpBB team released phpBB version 3.1.7-PL1</a> on Jan 11, 2016 which fixed a CSRF issue I found in the admin control panel BBCode creation form. Since BBCode is basically whitelisted HTML created by admins this CSRF vulnerability could allow an attacker to inject arbitrary HTML or JavaScript into forum posts.</p> This was my first time looking at phpBB and I was very happy with actually being able to find something with significant impact within a few hours.</p> #</a > Finding a target</h2> A good starting point for understanding the features and flow of phpBB controllers would be to look at something users would have access to with complex operations going on. `./phpbb/phpBB/posting.php</code> is a decent starting point since this is the page users hit when creating topics/posts. This controller should have permission checks, record creation, form submission, and maybe some HTML escaping. At the top of the file we can see something interesting:</p>` // Grab only parameters needed here</span></span> $</span>post_id</span> =</span> request_var</span>('</span>p</span>',</span> 0</span>);</span></span> $</span>topic_id</span> =</span> request_var</span>('</span>t</span>',</span> 0</span>);</span></span> $</span>forum_id</span> =</span> request_var</span>('</span>f</span>',</span> 0</span>);</span></span> $</span>draft_id</span> =</span> request_var</span>('</span>d</span>',</span> 0</span>);</span></span> $</span>lastclick</span> =</span> request_var</span>('</span>lastclick</span>',</span> 0</span>);</span></span> </span> $</span>preview</span> = (</span>isset</span>($</span>_POST</span>['</span>preview</span>'])) ? true : false;</span></span> $</span>save</span> = (</span>isset</span>($</span>_POST</span>['</span>save</span>'])) ? true : false;</span></span> $</span>load</span> = (</span>isset</span>($</span>_POST</span>['</span>load</span>'])) ? true : false;</span></span> $</span>confirm</span> = $</span>request</span>-></span>is_set_post</span>('</span>confirm</span>');</span></span> $</span>cancel</span> = (</span>isset</span>($</span>_POST</span>['</span>cancel</span>']) && !</span>isset</span>($</span>_POST</span>['</span>save</span>'])) ? true : false;</span></span></code></pre> Mostly all of the request variables used in this controller are defined a the top of the file, some of which are taken from this weird request_var()</code> function which IntelliJ tells me is deprecated. This function is basically a wrapper for \phpbb\request\request_interface::variable()</code>. Looking at \phpbb\request\request::variable()</code> it can be seen that this method returns the requested var from some associative array. The array is the concatenation of the $_POST</code> and $_GET</code> global variables. For those of you not familiar with PHP these are globals which contain the POST request and query vars (respectively). All of these variables are also trim()</code>'d and type casted to match the type of whatever the default value is.</p> This is a key bit of information: if we can find some place in a form where a POST request is made we should also be able to make the request with a GET. Maybe we should start looking for CSRF bugs?</p> #</a > How CSRF tokens work in phpBB</h2> Doing a simple find in file for "form" in IDEA I came across this line:</p> if</span> ($</span>submit</span> &&</span> check_form_key</span>('</span>posting</span>'))</span></span></code></pre> Looking into the check_form_key()</code> function it's clear that this is the function to check the CSRF token (using ===</code> mind you)... but it's done manually. And further down</em> the file:</p> add_form_key</span>('</span>posting</span>');</span></span></code></pre> So adding and checking the CSRF token is done manually. This smells like something that can lead to errors!</p> #</a > Finding CSRF bugs!</h2> Let's search the project for add_form_key()</code> and check_form_key()</code>. What we're looking for is files that show up in the result for add_form_key()</code> but not for check_form_key()</code>.</p> The following image is the result of the search for add_form_key()</code> and check_form_key()</code> respectively, with the admin control panel includes folder expanded:</p> </p> The greater number of check_form_key()</code> calls to add_form_key()</code> calls isn't really concerning since you can check the form key as many times as you'd like. What we're looking for is places where a form key is added but not checked. We can see there are two places where calls to check_form_key()</code> are definitely missing:</p> acp_bbcode.php</code></li> acp_extensions.php</code></li> </ul> acp_extensions.php</code> isn't too interesting since that just lets admins toggle showing unstable versions when checking extensions for updates, so the worse a CSRF vuln here does is allow an attacker to make an admin think their extensions are outdated.</p> The check in acp_bbcode.php</code> is of interest though since, although the form is submitted via POST, the request_var()</code> method is used to retrieve all form variables. We should be able to create BBCode over GET with a CSRF token omitted!</p> Here's a demonstration of this vulnerability:</p> {{< youtube 7NsUoE32cyQ >}}</p> #</a > Notes</h2> Although this vulnerability can lead to XSS, it's really not practical. By default phpBB enforces re-authentication when admins go to the ACP and gives the admin a different admin CP session ID. The SID also needs to be present in both the cookie and query string by default.</p> In theory a timing attack is possible since session IDs are checked with the equals operator ($this->session_id !== $session_id</code>) but this is also not very practical since sessions are tied to IP, browser, and some other information but most importantly doing it over the network isn't exactly easy.</p> The only way I see this being practical is if there's also an XSS vulnerability on some admin page that would allow you to inject a script to get the admin SID from document.location</code> then perform the exploit.</p> #</a > Timeline</h1> July 11, 2015: Reported vulnerability</li> August 4, 2015: Received response from two different project members requesting more information</li> December 23, 2015: Followed-up with project members (never received notification of response and I sort of forgot about this bug)</li> December 23, 2015: Vendor confirmed bug</li> January 11, 2016: Fix released</li> </ul> CVE-2016-1902: Symfony SecureRandom 2016-01-24T15:48:01-08:00 #</a > Overview</h2> Recently the Symfony project published a security advisory</a> to the SecureRandom</a> class in their Security component that affects Symfony versions 2.3.0-2.3.36, 2.6.0-2.6.12, 2.7.0-2.7.8. On most sane systems there is no problem, but in the event that something goes wrong the SecureRandom::nextBytes()</code> falls back to a custom random number generator which creates insecure random numbers.</p> #</a > Details</h2> The nextBytes()</code> function has three methods of RNG:</p> If the PHP 7 random_bytes()</code></a> function exists, that is used.</li> If OpenSSL is installed on the system then openssl_random_pseudo_bytes()</code></a> is used and the result is checked to ensure RNG succeeded.</li> If all of the above fail, a custom scheme is used. This is where the problem lies.</li> </ol> The custom scheme can take in a seed file which doesn't necessarily need to exist. If the file does exist then the inital RNG seed is read from that file, otherwise the seed is initialized with uniqid(mt_rand(), true)</code>.</p> Here is where the random number is generated</a>:</p> $</span>bytes</span> = '';</span></span> while</span> (</span>strlen</span>($</span>bytes</span>) < $</span>nbBytes</span>) {</span></span> static</span> $</span>incr</span> =</span> 1</span>;</span></span> $</span>bytes</span> .=</span> hash</span>('</span>sha512</span>', $</span>incr</span>++.$this-></span>seed</span>.</span>uniqid</span>(</span>mt_rand</span>(), true).$</span>nbBytes</span>, true);</span></span> $this-></span>seed</span> =</span> base64_encode</span>(</span>hash</span>('</span>sha512</span>', $this-></span>seed</span>.$</span>bytes</span>.$</span>nbBytes</span>, true));</span></span> $this-></span>updateSeed</span>();</span></span> }</span></span> </span> </span> return</span> substr</span>($</span>bytes</span>,</span> 0</span>, $</span>nbBytes</span>);</span></span></code></pre> Upon inspection we can see the bytes that are eventually returned to the user are generated from a SHA-512 hash. This is probably done to make the output look uniform. A red flag is raised that maybe whatever is being fed to hash()</code> isn't</em> uniformly random. What's actually being fed to the core of this algorithm is the concatenation of some counter, seed, and a uniqid()</code>. Let's take a look at each of these individually:</p> The counter starts out at 1 and is declared static. Since PHP is stateless, the counter is predictable if you know how many times the function has been called for your request.</p> </li> mt_rand()</code> is used as the $prefix</code> to uniqid()</code>. The PHP documentation for both of these functions explicitly state that they should not be used for cryptographic purposes. uniqid()</code> inparticular isn't meant to even be random -- just unique. At its core it's just a concatenation of prefix, seconds, microseconds, and php_combined_lgc()</code></a> (the latter explained here</a>).</p> </li> The seed can</em> be something we don't know if the file already exists</em>. There are two cases where we can predict the seed though:</p> 1. If the user provides a path that doesn't exist</span></span> 2. If the user provides a path to a file that the application does not</span></span></code></pre> have permission to read or write.</p> </li> </ul> (Sidenote: Hugo's markdown generator doesn't support ordered vs unordered lists... what?)</em></p> Point #2 is kind of interesting here because both failure to read or</em> write the file fail silently and because PHP isn't exactly a type-safe language, nothing explodes.</p> readSeed()</code></a>:</p> private function</span> readSeed</span>()</span></span> {</span></span> return</span> json_decode</span>(</span>file_get_contents</span>($this-></span>seedFile</span>));</span></span> }</span></span></code></pre> If file_get_contents()</code> fails it returns FALSE</code> and is passed to json_decode()</code>. If json_decode()</code> fails, it simply returns NULL and you have to manually check json_last_error()</code> to see why it failed. So if the file can't be read, your seed is null</code>.</p> updateSeed()</code></a> fails very similarly. The result of file_put_contents()</code> is never checked and will fail silently.</p> #</a > tl;dr</h2> SecureRandom::nextBytes()</code> will generate insecure random numbers if you're not using PHP 7, don't have random_compat, and OpenSSL fails for some reason.</p> Even if this isn't a very practical or exploitable bug, it shows how a weakness at the core of a framework can go indiscovered for so long (this one almost 3 years!). I also mainly did this writeup to note that this is my first of hopefully many more CVEs.</p> #</a > Timeline:</h2> Dec 14, 2015: Vulnerability identified and reported</li> Dec 30, 2015: No response, requested follow-up</li> Jan 14, 2016: Patch released for 2.3, 2.6, 2.7</li> Jan 18, 2016: Security advisory published</li> </ul>

Reverse Engineering Binaries With AI

2026-03-08T00:00:00+00:00

#</a > Getting Into Security</h2>
In middle school the gateway drug known as Halo 3 and unapproved mods shared on the game's File Share system got me interested in programming and security.
Through these mods and YouTube tutorials I discovered the (long defunct) Xbox-Tampers forum, followed tutorials on how to edit map files in a hex editor and make very basic MSN Messenger nudge bomb applications, and met some great people through that site who I'm still friends with.
I remember the struggles in the late 2000s and early 2010s of using cracked</del> expensive copies of IDA Pro to reverse engineer bits of the Xbox 360 to better write modding tools, and begging my friends who were much better than me (like @carrot_c4k3</a>, Xenomega</a>, and @Grimdoomer</a>) to help me reverse some Xbox 360 OS functions that I could not understand.
I've invested a lot of time and energy to try to get better at these things over the past 18 or so years. It's not what I live and breathe anymore and I'm not insanely cracked at RE, but I feel I've gotten pretty decent at being able to throw a binary into a static analysis tool and with disassembly/pseudocode can reasonably understand what's going on in an an application.
I've worked professional as a security engineer for the past 10 years and of course, AI "happened" and it's shaken up a lot of things, but I've still felt fairly comfortable about my skills not being terribly threatened. I've seen people try traditional static analysis tools, data flow analysis, etc., and all tooling falls short somehow, so surely this won't be too different in the near-term -- especially for such a niche skill -- right?
I think I was wrong.

#</a > Programming w/ AI</h2>
For non-ethical reasons I've felt pretty anti-LLM for coding. Mostly because it made me feel disconnected, dumb, and lose context. When Copilot first came out I gave it a shot but I felt like a monkey just smashing tab to have it complete some code. My subscription was soon cancelled as I felt like I was losing my edge when I didn't have AI assistance readily available.
At my job, however, AI adoption is being driven fairly hard and I've taken to using Claude Code for some of my side projects to understand AI usage better.
A couple years ago I created a program called WoWs Toolkit</a> which for about the last year went without any major updates because I just lost the steam to context switch all the time.
It's a fairly straightforward application designed to datamine the naval warfare videogame World of Warships, but has some slightly complicated internals which are used to read game files, understand the match replay format, etc:

I've had people in the community ask for various features over time which were mostly filed away in the "Nice To Have Eventually" folder as some require significant time investment.
One requested feature was the ability to read the game's proprietary 3D model format to understand the armor layout of ships in the game better. I started to reverse engineer the format about a year ago, lost steam, and just kind of dropped it.
There are various moving parts for that, some of which I'd already done:

Reading the game's virtual filesystem (VFS) without intermediate tools.</li>
Understanding the game's "GameParams" database which holds meta information about ships, upgrades, each component's 3D model path, etc.</li> </ol>
And parts I'd not done:

In the main VFS is a file called "assets.bin" which is a format very similar to the VFS it's contained within, but different enough to break my parser and it also had a second string table that I hadn't yet understood.</li>
Any research on the `.geometry</code>, .visual</code>, or .model</code> files -- all of which are required to paint a complete picture of the mesh, model positions, world transform, etc.</li>`
`Any understanding of how ship armor was represented and how it works in relation to the mesh.</li> </ol> I was making some changes to the app and thought, "Screw it, why not see if Claude can figure it out?"`#</a > Claude Code + Binary Ninja MCP</h2> I installed one of the MCP plugins for Binary Ninja, set it up in Claude Code, and gave it the following prompt: We have the Binary Ninja MCP running with WorldOfWarships64.exe running. There's some code I'd like you to reverse engineer in order to figure out a file format. Please document findings in G:\dev\wowsunpack\MODELS.md We're going to be reverse engineering the WoWs .geometry file format. An example file can be found here: G:\wows_dump\res\content\gameplay\uk\ship\aircarrier\BSA013_Colossus_1945\BSA013_Colossus_1945.geometry It's a 3D model format that I believe is proprietary to Wargaming's BigWorld engine. There are some magic constants in the file like 45 4E 43 44</code> that I've used to find a possible candidate function to start at: sub_140a5a940 This has some interesting strings such as: bufferHeader->m_magicVal == EncodedBufferHeader::EncodedMagicVal</code> There are other functions with this magic value too that I have not looked at. Please remember if you search the binary in binary ninja for this magic value, it will need to be little-endian encoded. If you want to write code to test, feel free to add a new command to the wowsunpack CLI and start adding code to a new models module. We should use winnow for our binary parsing. Do not commit any changes. </blockquote> Then came some early results: Note: I'm using Claude Desktop here for no good reason. Normally I'd use the CLI or ACP interfaces. And after processing for a bit: After about an hour or two (part of that time was waiting on me to accept some action prompt) I had some 3D models extracting into a format I could load into Blender piece-by-piece: And after more prompting, it was able to export the entire ship with its paint textures: And as a side note, it did label things/create structs! As far as code is concerned, it tried to do some very dumb things along the way. For example, in one of these files there's some XML data at some point. It generated code where after parsing a section header it would scan forward looking for the XML blob's opening tag and do the same from that position to find the closing tag. I had to curse and yell telling it, "You're screwing this up, stop taking shortcuts. Do this the the right way. No heuristics." Now obviously this binary has checked assertions (which I hope remain in the binary after this blog post :) ) and those make analysis overall easier. So maybe this is an extremely positive case. But honestly, I'm impressed with what Claude was able to do. From this very dumb prompt I was able to go from a bit of info from myself + a very surface-level parser for a dependency of the model loading pipeline (not even the 3D model itself!) to code which is able to export 3D models of ships with their textures and armor models, at any LOD. And it built a custom 3D model viewer into WoWs Toolkit. It even did some weird things I never thought of, like cracking hashes</a> from a set of known strings in the binary to figure out items in the tree structure. I barely wrote a line of code for this and it cost maybe ~$150 in tokens: If you would like to read its full analysis of the 3D model format, here is its braindump</a>. And here is the code</a>. #</a > Learnings</h2> While I really don't have much to share about the specific thing being reverse engineered, I did learn quite a bit about interacting with Claude. I'm sure people who live and breathe LLMs have their own advice, but what I found personally valuable: Developing custom CLIs for rapid testing/iteration. I already had a CLI for interacting with game files which Claude was able to easily extend to test its analysis of the binary.</li> The existing tooling beyond just that single CLI was critical. Other CLI tools for examining related file formats or deobfuscating data were crucial. This wasn't just a "point it at Binja and profit" sort of deal to achieve the polished result.</li> Watching the code as it streams in, or ensuring a mostly comprehensive review for reasons I've already mentioned like the AI trying to take shortcuts, is very important. I interrupted Claude many times when I noticed it was taking a subpar approach.</li> My project is in Rust, and Rust has a strong type system. Having the AI use strong newtypes for things like identifiers helped quite a bit with ensuring bug-free code.</li> My code was originally split into a couple different repos (one for the CLI tool and libraries, one for the UI). Swapping editor windows before all of this was kind of annoying, and it was even more annoying when trying to have the agents do cross-repo changes.</li> If you want to sniff out AI-assisted or generated code changes, look for unicode symbols like the fancy Unicode equivalent of <-></code> or -></code>, comments in Rust starting with //!</code>, or sudden introduction of section dividers like the following:</li> </ul> // --------------------------------------------------------------------------- // Helper: parse an array at a given offset, wrapping winnow errors // ---------------------------------------------------------------------------</code></pre> And most importantly: there were still bugs along the way. Trying to debug some of these things and prompt for fixes without knowing specifics of how the pieces were interconnected was really annoying (e.g. ship turrets were sometimes not facing to the correct default orientation). For things I really care about being done right, I think it's still worthwhile to go through the pains myself first and have AI assistance later if I can if for not other reason than to save time in correcting its mistakes. #</a > Mixed Feelings</h2> I would consider myself a low-level person who thinks about allocations in my application, the performance cost of various approaches, and I like knowing the details of how something works. I might do the wrong thing at times for certain tradeoffs, but I like to be aware of that tradeoff existing and which decision was made. For this task though? I have no idea how it works. Before starting this project I had zero knowledge of 3D rendering, model formats, and while I did manually reverse engineer the game's general virtual filesystem format for its packed files, I got about halfway through reversing the similar virtual filesystem used for 3D model assets. I have no idea how it works beyond what I started. I read the code as Claude went along, pointing out things that I thought were obvious anti-patterns or incorrect... but I couldn't tell you how the different ship mesh sections are read, then paired to their transformation matrix (which is located in another file). That kind of sucks. Don't get me wrong -- I am happy that I'm giving people something they want and it otherwise may not have ever gotten done without Claude helping accelerate things. I'm also happy that the barrier to these types of tasks are lowered for a modest fee. There's just not "my code" and the iteration over failure to be proud of, and I want people to feel confident that the person behind the software fully understands what it's doing. Or supposed to be doing. Even back in the days of asking my friends for help, I was still building a relationship with them in some capacity and they could help guide me. In some cases they would even just reverse engineer things, hand me the completed package, and explain to me the bullshit they had to understand, and I'd happily give them a shoutout for helping with the project. Now it's a one-way trip with Claude and asking it to do a brain dump. I suppose that just like I told the AI not to take shortcuts though, I find it hard to accept <Insert This Month's Best Model> doing these things at the cost of building my own context, expanding my skillset to new areas, and keeping my existing skills sharp.

A File Format Uncracked for 20 Years

2025-11-06T00:00:00+00:00

Splinter Cell (2002)</a> was one of the first games I had on the original Xbox and still remains one of my favorite games of all time. The game was developed by Ubisoft using Unreal Engine 2 -- licensed from a small indie dev called Epic Games who continues to use and license its game engine technology for contemporary small-budget indie games such as Fortnite and Halo: Campaign Evolved.

I got into programming/hacking through video games and I still enjoy data mining/exploring cut content from the few games I play nowadays. I recently randomly decided to look online for cut content from Splinter Cell and I was kind of surprised by the lack of datamined info. There isn't really much information on the topic aside from an OG Xbox review copy</a> of the game which contained two levels cut from the retail Xbox version and some other minor differences.

Naturally, I decided to legally backup my personal disc copy of the game and got to digging into the files.

My initial objective was to examine the format of the game data and sniff out if there's any indicators of cut content such as textures, models, interesting strings -- whatever. Some nice finds would be debug menus, voice lines, weapon concepts, or levels that are unreachable through normal game progression.

The game's (truncated) file tree looks like this:

.</span></span>
├── contentimage.xbx</span></span>
├── dashupdate.xbe</span></span>
├── default.xbe</span></span>
├── downloader.xbe</span></span>
├── dynamicxbox.umd</span></span>
├── LMaps</span></span>
│   ├── 000_menu</span></span>
│   │   ├── common.lin</span></span>
│   │   └── menu.lin</span></span>
│   ├── 001_Training</span></span>
│   │   ├── 0_0_2_Training.bik</span></span>
│   │   ├── 0_0_2_Training.lin</span></span>
│   │   ├── 0_0_2_Training_progress.tga</span></span>
│   │   ├── 0_0_2_Training_start.tga</span></span>
│   │   ├── 0_0_3_Training.lin</span></span>
│   │   ├── 0_0_3_Training_complete.tga</span></span>
│   │   ├── 0_0_3_Training_progress.tga</span></span>
│   │   ├── common.lin</span></span>
│   │   └── French</span></span>
│   │       ├── 0_0_2_Training_progress.tga</span></span>
│   │       ├── 0_0_2_Training_start.tga</span></span>
│   │       ├── 0_0_3_Training_complete.tga</span></span>
│   │       └── 0_0_3_Training_progress.tga</span></span></code></pre>
.xbe</code> files are Xbox Executables, .bik</code> are Bink Video</a> files, and .tga</code> are images... but .lin</code> is new to me.</p>
In Splinter Cell the maps are divided into separate parts. So in the training mission 001_Training</code>, you likely have 0_0_2_Training.lin</code> for the first part and 0_0_3_Training.lin</code> for the second which gets loaded via an in-game loading sequence after advancing to some zone in the map.</p>
I instantly thought that common.lin</code> might contain data common to both of these parts as a way to reduce file size. The Halo games for instance have a shared.map</code> containing assets which are shared across most maps, and load data at a fixed address so that the file can be trivially transmuted from a binary blob to its in-memory data structures.</p>
Examining the common.lin</code> file in a hex editor, a few things become immediately apparent:</p>
┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐</span></span>
│00000000│ 04 00 00 00 0c 00 00 00 ┊ 78 9c 7b d7 97 c2 00 00 │........┊x.{.....│</span></span>
│00000010│ 06 2e 01 e1 04 00 00 00 ┊ 0c 00 00 00 78 9c 63 60 │........┊....x.c`│</span></span>
│00000020│ 90 66 00 00 00 3a 00 1c ┊ 04 00 00 00 0c 00 00 00 │.f...:..┊........│</span></span>
│00000030│ 78 9c 73 48 67 60 00 00 ┊ 02 39 00 a8 04 00 00 00 │x.sHg`..┊.9......│</span></span>
│00000040│ 0c 00 00 00 78 9c b3 e0 ┊ 65 60 00 00 01 0b 00 46 │....x...┊e`.....F│</span></span>
└────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘</span></span></code></pre>

Data between 0x0..0x4</code> and 0x4..0x8</code> are little-endian 32-bit integers: 0x00000004</code> and 0x0000000C</code>.</li>
At offset 0x8</code> is what appears to be a zlib-compressed chunk of data -- denoted by the 'x' in the ASCII view and 0x78 0x9c</code> in the hex view.</li>
There's another sequence of this at offset 0x14</code>, which happens to be 0xC</code> bytes past the offset of the zlib data (0x8</code>), and another at 0x28</code>.</li>
</ul>
Presumably the format here is {decompressed_data_len, compressed_data_len, zlib_block[compressed_data_len]}</code> repeated.</p>
So far so good.</p>
I wrote a quick tool to decompress the archive and without a hitch ended up with a 64k file containing 4 u32</code>s prefixing it. Since these 4 are in their own dedicated zlib-compressed chunks I consider to be separate from the main data. I later reverse engineered and identified how they are used:</p>
uncompressed_data_size: 0x648EEE</span></span>
texture_cache_size? - later used when calling D3DDevice_CreateTexture2: 0x1B0000</span></span>
vertex_buffer_size? - ditto, D3DDevice_CreateVertexBuffer2: 0x6740</span></span>
index_buffer_size? - ditto, XGSetIndexBufferHeader: 0xD38</span></span></code></pre>
And this is what the main data section's first 0x100 bytes look like:</p>
┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐</span></span>
│00000000│ 5c 58 9e 13 00 a3 c5 e3 ┊ 9f b4 92 9b 13 5c 58 9e │\X......┊.....\X.│</span></span>
│00000010│ 13 01 00 00 00 04 2a d6 ┊ fe 7e 37 13 4d 61 70 73 │......*.┊.~7.Maps│</span></span>
│00000020│ 5c 6d 65 6e 75 5c 6d 65 ┊ 6e 75 2e 75 6e 72 00 00 │\menu\me┊nu.unr..│</span></span>
│00000030│ 00 00 00 ee de 00 00 00 ┊ 00 00 00 16 4d 61 70 73 │........┊....Maps│</span></span>
│00000040│ 5c 31 5f 31 5f 30 54 62 ┊ 69 6c 69 73 69 2e 75 6e │\1_1_0Tb┊ilisi.un│</span></span>
│00000050│ 72 00 f0 de 00 00 6d c9 ┊ 17 00 00 00 00 00 16 4d │r.....m.┊.......M│</span></span>
│00000060│ 61 70 73 5c 31 5f 31 5f ┊ 31 54 62 69 6c 69 73 69 │aps\1_1_┊1Tbilisi│</span></span>
│00000070│ 2e 75 6e 72 00 60 a8 18 ┊ 00 98 34 21 00 00 00 00 │.unr.`..┊..4!....│</span></span>
│00000080│ 00 16 4d 61 70 73 5c 31 ┊ 5f 31 5f 32 54 62 69 6c │..Maps\1┊_1_2Tbil│</span></span>
│00000090│ 69 73 69 2e 75 6e 72 00 ┊ 00 dd 39 00 89 63 19 00 │isi.unr.┊..9..c..│</span></span>
│000000a0│ 00 00 00 00 18 4d 61 70 ┊ 73 5c 30 5f 30 5f 32 5f │.....Map┊s\0_0_2_│</span></span>
│000000b0│ 54 72 61 69 6e 69 6e 67 ┊ 2e 75 6e 72 00 90 40 53 │Training┊.unr..@S│</span></span>
│000000c0│ 00 0f 9f 0c 00 00 00 00 ┊ 00 18 4d 61 70 73 5c 30 │........┊..Maps\0│</span></span>
│000000d0│ 5f 30 5f 33 5f 54 72 61 ┊ 69 6e 69 6e 67 2e 75 6e │_0_3_Tra┊ining.un│</span></span>
│000000e0│ 72 00 a0 df 5f 00 48 86 ┊ 11 00 00 00 00 00 1e 4d │r..._.H.┊.......M│</span></span>
│000000f0│ 61 70 73 5c 31 5f 32 5f ┊ 31 44 65 66 65 6e 73 65 │aps\1_2_┊1Defense│</span></span>
└────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘</span></span></code></pre>
And at what appears to be the end of the file table:</p>
┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐</span></span>
│0002c580│ 79 6e 63 68 5c 69 6e 74 ┊ 5c 55 73 61 53 6f 6c 64 │ynch\int┊\UsaSold│</span></span>
│0002c590│ 69 65 72 5c 55 53 4f 55 ┊ 4e 43 5f 33 2e 62 69 6e │ier\USOU┊NC_3.bin│</span></span>
│0002c5a0│ 00 40 8d 9b 13 74 05 00 ┊ 00 00 00 00 00 c1 83 2a │.@...t..┊.......*│</span></span>
│0002c5b0│ 9e 64 00 11 00 01 00 00 ┊ 00 10 0e 00 00 88 00 00 │.d......┊........│</span></span>
│0002c5c0│ 00 fa 0f 00 00 f3 7a 11 ┊ 00 4e 00 00 00 3e 78 11 │......z.┊.N...>x.│</span></span>
│0002c5d0│ 00 de ad f0 0f 42 01 9c ┊ 90 92 8f 96 93 9e 8b 96 │.....B..┊........│</span></span>
│0002c5e0│ 90 91 9a 9c 97 9a 93 90 ┊ 91 df af bc ba bc b7 ba │........┊........│</span></span>
│0002c5f0│ b3 b0 b1 df a6 c5 a3 ba ┊ bc b7 ba b3 b0 b1 a3 ac │........┊........│</span></span>
│0002c600│ a6 ac ab ba b2 a3 df ce ┊ cf d0 cd c9 d0 cf cd df │........┊........│</span></span>
│0002c610│ cd ce c5 cf cd c5 ce cb ┊ ff 00 00 00 00 00 00 00 │........┊........│</span></span>
│0002c620│ 00 00 00 00 00 00 00 00 ┊ 00 01 00 00 00 fa 0f 00 │........┊........│</span></span>
│0002c630│ 00 10 0e 00 00 05 4e 6f ┊ 6e 65 00 10 04 07 04 06 │......No┊ne......│</span></span>
│0002c640│ 43 6f 6c 6f 72 00 10 04 ┊ 07 04 0d 49 6e 74 65 72 │Color...┊...Inter│</span></span>
│0002c650│ 6e 61 6c 54 69 6d 65 00 ┊ 10 00 07 00 07 45 6e 67 │nalTime.┊.....Eng│</span></span>
│0002c660│ 69 6e 65 00 10 00 07 04 ┊ 05 43 6f 72 65 00 10 00 │ine.....┊.Core...│</span></span>
│0002c670│ 07 04 07 53 79 73 74 65 ┊ 6d 00 10 00 07 04 06 55 │...Syste┊m......U│</span></span>
└────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘</span></span></code></pre>
Just to save some blog space on my trial and error process here, I'm going to drop some of the resources I found which discuss this format:</p>

https://oldunreal.com/phpBB3/viewtopic.php?t=4885</a></li>
https://zenhax.com/viewtopic.php@t=1049.html</a></li>
https://reshax.com/topic/1421-ubisoft-unreal-engine-2-open-season-2006-video-game-umd-also-lin-xbox-xbox-360-pc-and-liv-latter-being-exclusive-to-xbox-360/</a></li>
https://www.unrealarchive.org/wikis/unreal-wiki/Legacy:UMOD/File_Format.html</a></li>
</ul>
The last two posts in particular had structure info that was helpful in figuring out the packed int format (think UTF-8 and its variable-length encoding) and a couple unknown vars.</p>
What I gathered from all of these posts was that over time, nobody's really been able to figure out this format's quirks sufficiently to unpack the data. Everyone seems to think that some kind of VFS is created and the data gets mapped at a specific address and then read. Which may be true for some titles or consoles, but is not really the case for this one.</p>
My objective has now changed:</strong> I now want to reverse engineer this file format and be able to dump individual files from this filesystem. Then I can achieve my core goal of looking for cut content. Then I can maybe play the game.</p>
#</a
>
tl;dr of the general .lin</code> structure</h2>
common.lin</code> has a different layout from the other .lin</code> files that looks roughly like:</p>
/* ==== Standard Data ==== */</span></span>
</span>
// These three, from research + reverse engineering, should not be considered</span></span>
// as part of the "whole" file</span></span>
u32</span>             maybe_load_address</span>;</span> // 5C 58 9E 13 (0x139e585c) in common.lin</span></span>
compressed_int  name_length</span>;</span>        // 0 in common.lin</span></span>
char</span>            name</span>[</span>name_length</span>];</span></span>
</span>
/* ==== common.lin-specific file header ==== */</span></span>
</span>
u32</span>             magic</span>;</span>              // 0x9fe3c5a3 in little endian, i.e. A3 C5 E3 9F</span></span>
</span>
u32</span>             unk_address</span>;</span>        // B4 92 9B 13, (0x139b92b4) suspiciously similar to maybe_load_address.</span></span>
                                    // unk_address - load_address gives you the start of the file</span></span>
                                    // table, relative to the magic?</span></span>
u32</span>             load_address2</span>;</span>      // 5C 58 9E 13 same as maybe_load_address</span></span>
</span>
u8</span>              unknown</span>[</span>8</span>];</span>         // 01 00 00 00 04 2A D6 FE</span></span>
compressed_int  file_entry_count</span>;</span></span>
</span>
FileEntry</span>       file_entries</span>[</span>file_entry_count</span>];</span></span>
</span>
struct</span> FileEntry</span> {</span></span>
    compressed_int  name_len</span>;</span></span>
    char</span>            name</span>[</span>name_len</span>];</span></span>
    u32</span>             offset</span>;</span></span>
    u32</span>             len</span>;</span></span>
    u32</span>             unk</span>;</span></span>
}</span></span></code></pre>
Then immediately following the FileEntry</code> table are 54 Unreal Engine Package files in sequence (identified via their 0x9E2A83C1</code> magic -- these are also referred to as Linker</strong> files) that presumably map to the files in the file table.</p>
The map-specific files like menu.lin</code> and 0_0_2_Training.lin</code> do not have the file table, but they do have the first 3 fields (and a non-null string like "menu\x0" for the name field) then a sequence of Linker files.</p>
But the difficulty with parsing this data starts with the file table.</p>
#</a
>
Problems</h2>
#</a
>
File Table</h3>
The file table is a very simple format that I'm able to parse with my program:</p>
FileEntry</span> {</span></span>
    name</span>:</span> Maps</span>\\menu\\menu</span>.</span>unr</span>,</span></span>
    offset</span>:</span> 0x0</span>,</span></span>
    len</span>:</span> 0xDEEE</span>,</span></span>
    unk</span>:</span> 0x0</span>,</span></span>
},</span></span>
FileEntry</span> {</span></span>
    name</span>:</span> Maps</span>\\1_1_0Tbilisi</span>.</span>unr</span>,</span></span>
    offset</span>:</span> 0xDEF0</span>,</span></span>
    len</span>:</span> 0x17C96D</span>,</span></span>
    unk</span>:</span> 0x0</span>,</span></span>
},</span></span>
FileEntry</span> {</span></span>
    name</span>:</span> Maps</span>\\1_1_1Tbilisi</span>.</span>unr</span>,</span></span>
    offset</span>:</span> 0x18A860</span>,</span></span>
    len</span>:</span> 0x213498</span>,</span></span>
    unk</span>:</span> 0x0</span>,</span></span>
},</span></span>
FileEntry</span> {</span></span>
    name</span>:</span> Maps</span>\\1_1_2Tbilisi</span>.</span>unr</span>,</span></span>
    offset</span>:</span> 0x39DD00</span>,</span></span>
    len</span>:</span> 0x196389</span>,</span></span>
    unk</span>:</span> 0x0</span>,</span></span>
},</span></span>
FileEntry</span> {</span></span>
    name</span>:</span> Maps</span>\\0_0_2_Training</span>.</span>unr</span>,</span></span>
    offset</span>:</span> 0x534090</span>,</span></span>
    len</span>:</span> 0xC9F0F</span>,</span></span>
    unk</span>:</span> 0x0</span>,</span></span>
},</span></span>
FileEntry</span> {</span></span>
    name</span>:</span> Maps</span>\\0_0_3_Training</span>.</span>unr</span>,</span></span>
    offset</span>:</span> 0x5FDFA0</span>,</span></span>
    len</span>:</span> 0x118648</span>,</span></span>
    unk</span>:</span> 0x0</span>,</span></span>
},</span></span>
FileEntry</span> {</span></span>
    name</span>:</span> Maps</span>\\1_2_1DefenseMinistry</span>.</span>unr</span>,</span></span>
    offset</span>:</span> 0x7165F0</span>,</span></span>
    len</span>:</span> 0x249AF6</span>,</span></span>
    unk</span>:</span> 0x0</span>,</span></span>
},</span></span>
FileEntry</span> {</span></span>
    name</span>:</span> Maps</span>\\1_2_2DefenseMinistry</span>.</span>unr</span>,</span></span>
    offset</span>:</span> 0x9600F0</span>,</span></span>
    len</span>:</span> 0x20F662</span>,</span></span>
    unk</span>:</span> 0x0</span>,</span></span>
},</span></span>
<</span>snip</span>></span></span></code></pre>
At first glance the files seem to be laid out sequentially, aligned to a pointer-width boundary. Except, notice that last file's offset... 0x9600F0</code>. This is way outside of the range of my 0x648EEE</code>-length file, and this file list contains 3,582 files! Not 54 as expected from the count of Unreal Package magics!</p>
The mismatch file count could be explained by not every file in this container being an Unreal Package, but the offsets so far are extremely</em> wrong.</p>
#</a
>
File Reading</h3>
After debugging the game in the Original Xbox emulator xemu</a>, I was able to find the routine which opens the file, as well as the function which reads and decompresses data.</p>

    Function Identification Methodology</summary>
    
        If anyone's curious on the methodology: I identified NtCreateFile</code>, set a breakpoint, recorded the HANDLE</code> returned for the file path I cared about, then set a breakpoint at NtReadFile</code> and broke when the input HANDLE</code> matched the expected value. The call stack/stepping from here helped identify interesting callers. Alternatively, the string "unknown compression method</code>" is useful in finding the decompression routine inflateInit2</code>.</p>
This is not super relevant to the blog post which is why it's in this little collapse section. I hate reading posts like this that skip over a detail I'm interested in like it's just common knowledge how something is done, so I'm trying to avoid doing that :)
    </div>
</details>
Note: Click images to see in higher res</em>.</p>
</a></p>
This function basically checks the requested read size against how much data it has precached in its decompressed data buffer. It will then copy as much data as it can from its precached buffer to the output buffer, then read the next block of compressed zlib data into its precache buffer if the previous one was exhausted. Repeat this process until the request is satisfied.</p>
Identifying this function was pretty important for my reverse engineering process. I could now set breakpoints on the code which copies data to the output buffer and see who's calling this function when data is read from offsets I care about.</p>
I stepped through this code, set Memory Read breakpoints on data I didn't yet understand, and noted something interesting early on!</p>
Those "addresses" from the header (0x139e585c</code>)? Those are actually passed to what I can only guess is a Seek</code> routine which updates the position</code> property of the file reader, which then makes an indirect call to another function that literally does nothing</strong>.</p>
The entire content of the function is:</p>
retn    4</span></span></code></pre>
That's it.</p>
Then the reads just... continue from their last position? Since the function is an indirect call, I can only assume that I was looking at some composed C++ object where the outer class object updates its own position</code> in Seek()</code> and then calls its underlying file reader's Seek()</code>... which is a no-op?</p>
After setting Memory Read breakpoints on the object's position</code> field, I noticed it's only ever used in their file reader equivalent of FTell()</code>. It doesn't affect where data is actually being read from at all.</p>
The reason for the Seek()</code> being a no-op is likely because the underlying file reader is reading directly from the compressed buffer, which reads in 0x4000</code>-byte chunks. Since you cannot reasonably map an uncompressed data offset to a compressed offset the format must be designed to ignore seeks and just read data linearly.</p>
...the .lin</code> extension makes a lot more sense.</p>
💡 In order to read these files, you have to assume that you cannot seek forward/backward. Easy enough.</p>
#</a
>
Load Order Matters</h3>
We still have a problem that has not been addressed: why does the file table have a large count of files with bad offsets?</p>
I continued to use breakpoints inside of the file read function to trace where interesting bits of data were read and forced a break when the data immediately following the file table was read. Eventually I traced the file read operation back far enough to find this function, StaticLoadObject</code>:</p>
</a></p>
This function calls ResolveName</code> which I was able to log the arguments to via a debugger breakpoint script, which told me the InName</code> was ini:Engine.Engine.GameEngine</code>:</p>
</a></p>
This ini:Engine.Engine.GameEngine</code> name gets parsed as:</p>

ini:</code> <- resolve the name from the game's INI files</li>
Engine.Engine</code> <- the INI table to read from</li>
GameEngine</code> <- the key from the table to read</li>
</ul>
If I look in UW.ini</code> included with the game, this table is defined as:</p>
[Engine.Engine]</span></span>
RenderDevice</span>=</span>D3DDrv.D3DRenderDevice</span></span>
GameRenderDevice</span>=</span>D3DDrv.D3DRenderDevice</span></span>
AudioDevice</span>=</span>XboxAudio.XboxAudioSubsystem</span></span>
Console</span>=</span>Engine.Console</span></span>
DefaultPlayerMenu</span>=</span>UPreview.UPreviewRootWindow</span></span>
Language</span>=</span>int</span></span>
GameEngine</span>=</span>Engine.GameEngine</span></span>
EditorEngine</span>=</span>Editor.EditorEngine</span></span>
WindowedRenderDevice</span>=</span>D3DDrv.D3DRenderDevice</span></span>
DefaultGame</span>=</span>Echelon.EchelonGameInfo</span></span>
DefaultServerGame</span>=</span>WarfareGame.WarfareTeamGame</span></span>
ViewportManager</span>=</span>XboxDrv.XboxClient</span></span>
Render</span>=</span>Render.Render</span></span>
Input</span>=</span>Engine.Input</span></span>
Canvas</span>=</span>Echelon.ECanvas</span></span>
Editor3DRenderDevice</span>=</span>D3DDrv.D3DRenderDevice</span></span></code></pre>
So the resulting value returned from this function is Engine.GameEngine</code>, which matches what this function resolves.</p>
This is then used to resolve the package</strong> Engine</code> and its exported object</strong> GameEngine</code>. The game binary looks for the file Engine</code> in its available sources (partial matching strategy), which includes searching against the LIN file table, and then resolves that name as System\Engine.u</code>. My tool that reads the file table confirms that this is declared in the LIN file:</p>
FileEntry</span> {</span></span>
    name</span>:</span> System</span>\\</span>Engine</span>.</span>u</span>,</span></span>
    offset</span>:</span> 0x13482120</span>,</span></span>
    len</span>:</span> 0x127DA1</span>,</span></span>
    unk</span>:</span> 0x0</span>,</span></span>
},</span></span></code></pre>
Except the file start offset + len don't make sense. If I assume the Engine.u</code> file is the first file immediately following the file table, advancing forward by this length appears to land right in the middle of some string?</p>
┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐</span></span>
│00154330│ 09 45 4d 65 73 68 53 46 ┊ 58 00 10 00 07 00 1b 43 │.EMeshSF┊X......C│</span></span>
│00154340│ 68 61 6e 64 65 72 6c 65 ┊ 72 43 72 79 73 74 61 6c │handerle┊rCrystal│</span></span>
│00154350│ 50 61 72 74 69 63 75 6c ┊ 65 00 10 00 07 00 12 46 │Particul┊e......F│</span></span>
└────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘</span></span></code></pre>
I'll save some time and just say that I did not identify the wrong file. The lengths just don't matter, and for all intents and purposes are wrong. The reader in the game engine must just read the data in-order using its self-description in its own header?</p>
The Unreal Engine Package/Linker file format has been well documented</a> and does include some sizes in its header. The packages contain about what you'd expect of some object-oriented programming (OOP) script/data format.</p>
It has exported</em> objects which are named instances of some OOP type and has properties and data. Or the object can be a class/struct definition. The exports may rely on types exported from other packages which are declared as imports</em>. Both of these have names or string data associated with them which are defined in the name</em> table.</p>
I mapped the existing documentation to the following Rust struct:</p>
pub</span> struct</span> PackageHeader</span><'</span>i</span>> {</span></span>
    pub</span> version</span>:</span> u32</span>,</span></span>
    pub</span> flags</span>:</span> u32</span>,</span></span>
    pub</span> name_count</span>:</span> u32</span>,</span></span>
    pub</span> name_offset</span>:</span> u32</span>,</span></span>
    pub</span> export_count</span>:</span> u32</span>,</span></span>
    pub</span> export_offset</span>:</span> u32</span>,</span></span>
    pub</span> import_count</span>:</span> u32</span>,</span></span>
    pub</span> import_offset</span>:</span> u32</span>,</span></span>
</span>
    // Note: this is not in the above documented description</span></span>
    pub</span> unk</span>:</span> u32</span>,</span></span>
    // Ditto.</span></span>
    // Not shown: compressed int for length of this data at this position</span></span>
    pub</span> unknown_data</span>: &'</span>i</span> [</span>u8</span>],</span></span>
</span>
    pub</span> guid_a</span>:</span> u32</span>,</span></span>
    pub</span> guid_b</span>:</span> u32</span>,</span></span>
    pub</span> guid_c</span>:</span> u32</span>,</span></span>
    pub</span> guid_d</span>:</span> u32</span>,</span></span>
    // Not shown: compressed int for length of this data at this position.</span></span>
    pub</span> generations</span>:</span> Vec</span><</span>GenerationInfo</span>>,</span></span>
}</span></span></code></pre>
And of course, the offsets in this format are also unusable (e.g. the name_offset</code> lands you after</em> the start of the name table). But the counts look good:</p>
PackageHeader</span> {</span></span>
    version</span>:</span> 0x110064</span>,</span></span>
    flags</span>:</span> 0x1</span>,</span></span>
    name_count</span>:</span> 0xE10</span>,</span></span>
    name_offset</span>:</span> 0x88</span>,</span></span>
    export_count</span>:</span> 0xFFA</span>,</span></span>
    export_offset</span>:</span> 0x117AF3</span>,</span></span>
    import_count</span>:</span> 0x4E</span>,</span></span>
    import_offset</span>:</span> 0x11783E</span>,</span></span>
    unk</span>:</span> 0xFF0ADDE</span>,</span></span>
    unknown_data</span>: [</span></span>
      ...</span></span>
    ]</span></span>
    guid_a</span>:</span> 0x0</span>,</span></span>
    guid_b</span>:</span> 0x0</span>,</span></span>
    guid_c</span>:</span> 0x0</span>,</span></span>
    guid_d</span>:</span> 0x0</span>,</span></span>
    generations</span>: [</span></span>
        GenerationInfo</span> {</span></span>
            export_count</span>:</span> 0xFFA</span>,</span></span>
            name_count</span>:</span> 0xE10</span>,</span></span>
        },</span></span>
    ],</span></span>
}</span></span></code></pre>
Now with my tool updated to read these tables -- parsing by assuming that they immediately follow this header and each other -- I have imports that look like:</p>
Package Core</span>.</span>Core</span></span>
Import</span> {</span> class_package</span>:</span> 4</span>,</span> class_name</span>:</span> B64</span>,</span> package_index</span>:</span> 0</span>,</span> object_name</span>:</span> 4</span>,</span> object</span>:</span> None</span> }</span></span>
</span>
Class Core</span>.</span>Object</span></span>
Import</span> {</span> class_package</span>:</span> 4</span>,</span> class_name</span>:</span> B62</span>,</span> package_index</span>:</span> FFFFFFFF</span>,</span> object_name</span>:</span> 13</span>,</span> object</span>:</span> None</span> }</span></span>
</span>
Class Core</span>.</span>Function</span></span>
Import</span> {</span> class_package</span>:</span> 4</span>,</span> class_name</span>:</span> B62</span>,</span> package_index</span>:</span> FFFFFFFF</span>,</span> object_name</span>:</span> BBD</span>,</span> object</span>:</span> None</span> }</span></span></code></pre>
And exports:</p>
Class Actor</span></span>
(</span>0x0</span>)</span> ObjectExport</span> {</span></span>
    class_index</span>:</span> 0x0</span>,</span></span>
    super_index</span>:</span> 0xFFFFFFFE</span>,</span></span>
    package_index</span>:</span> 0x0</span>,</span></span>
    object_name</span>:</span> 0x206</span>,</span></span>
    object_flags</span>:</span> 0x40F0004</span>,</span></span>
    serial_size</span>:</span> 0x3A8</span>,</span></span>
    serial_offset</span>:</span> 0xF719</span>,</span></span>
}</span></span>
</span>
Class Pawn</span></span>
(</span>0x1</span>)</span> ObjectExport</span> {</span></span>
    class_index</span>:</span> 0x0</span>,</span></span>
    super_index</span>:</span> 0x1</span>,</span></span>
    package_index</span>:</span> 0x0</span>,</span></span>
    object_name</span>:</span> 0x1A</span>,</span></span>
    object_flags</span>:</span> 0x40F0004</span>,</span></span>
    serial_size</span>:</span> 0x281</span>,</span></span>
    serial_offset</span>:</span> 0xFAC1</span>,</span></span>
}</span></span>
</span>
...</span></span>
</span>
Class GameEngine</span></span>
(</span>0xEFB</span>)</span> ObjectExport</span> {</span></span>
    class_index</span>:</span> 0x0</span>,</span></span>
    super_index</span>:</span> 0x1C8</span>,</span></span>
    package_index</span>:</span> 0x0</span>,</span></span>
    object_name</span>:</span> 0x1D8</span>,</span></span>
    object_flags</span>:</span> 0x40F0004</span>,</span></span>
    serial_size</span>:</span> 0x5B</span>,</span></span>
    serial_offset</span>:</span> 0xC50DB</span>,</span></span>
}</span></span></code></pre>
So the GameEngine</code> object has export index 0xEFB</code> and its data is supposedly located at offset 0xC50DB</code> relative to the package start. You guessed it though, its offset is wrong!</p>
#</a
>
Export Data</h3>
Up to this point we know:</p>

You cannot seek in the file reader.</li>
The offsets do not map cleanly to the on-disk representation and aren't really used other than for position tracking.</li>
The sizes (at least in the file table, and I soon realized in the export data) are incorrect.</li>
We know GameEngine</code> is the first object requested by the C++ side of the game and is export index 0xEFB</code> in the Engine</code> package. It may not be the first object actually parsed</em>, but it's the first object requested.</li>
</ol>
Now, to achieve my goal of dumping these files I attempted to simply sum the size of these exports to figure out the end offset of the file... but trying a combination of that calculated size + any of the {end_of_export_table, start_of_file}</code> offsets landed me in weird places with other Linker files in-between.</p>
By referencing Unreal-Library</a> to help fill in some of the blanks, I observed the following high-level parsing logic in the game engine:</p>

An exported object is requested by the game. If it isn't loaded already, the export is lazy loaded.</li>
Lazy loading requires resolving the super</code> type's object. For some things this is the Class</code> or Struct</code> base types, for other things this is a different parent class which will eventually have Class</code> as its parent type.</li>
Exports have properties which can be of varying size. As you read an export, you deserialize its data as described by its serial_size</code> and serial_offset</code> fields, and however the types exported from the C++ side defines the deserialization routine.</li>
</ol>
Which visually results in something like the following flow when resolving imports/exports:</p>
</p>
To give a concrete example, imagine that GameEngine</code> has the following class hierarchy:</p>
GameEngine -> Engine -> Subsystem -> Class</code></p>
Also imagine that GameEngine</code> is the very first object ever parsed -- nothing else has been loaded yet. Requesting to load GameEngine</code> from the Engine.u</code> package will trigger the following sequence of events:</p>

Engine.u</code> header read/parse (since no package has been created yet)</li>
Lookup Engine</code>'s GameEngine</code> export. It's not yet parsed, so we need to construct this object by constructing/deserializing it.</li>
GameEngine</code>'s parent class is Engine.Engine</code>. It has not yet been parsed, so we need to deserialize it before</strong> GameEngine</code>.</li>
Core.Subsystem</code> is Engine.Engine</code>'s parent class. Same thing.</li>
Core.u</code> header read/parse (since Core</code> hasn't been loaded yet)</li>
Core.Class</code> is Core.Subsystem</code>'s parent class (and the base class). Construct this object.</li>
Core.Class</code> property deserialization. We can now continue with Core.Subsystem</code> creation.</li>
Core.Subsystem</code> property deserialization...</li>
Engine.Engine</code> property deserialization..</li>
Engine.GameEngine</code> property deserialization...</li>
We can now return the fully constructed Engine.GameEngine</code>.</li>
</ol>
I believe this can result in export data that is interleaved, unfortunately. For the above scenario the data may be on disk like the following diagram. Note:</strong> for space/simplicity I've omitted Core.Class</code>, as well as the potential for the properties themselves</em> to trigger deserializing of other exports.</p>

And now if you imagine that there's a second</em> object which also extends from Engine</code> loaded after GameEngine</code>, then their common the super class Engine</code> has already been parsed and its information is already in-memory. i.e. if you serialize two objects of the same exact type, the first object might have all the data for its parent classes interleaved with its own</em> export data and the second object only contains its own property data.</p>
Unfortunately, this means that to read these files statically (even for just static recompilation) you need to have full knowledge of how each C++-implemented type is parsed in order to parse all exports and their properties. Additionally, reading one export may trigger resolving of imports in your own Linker object, which in turn trigger deserialization of exports in another Linker object.</p>
This results in the export data's size not necessarily being wrong</em> per se, but not super usable without actually doing full parsing. If other exports are deserialized in the middle of deserializing an export, they will seek around and restore the original position. When the export is done deserializing it subtracts the post-deserialization position</code> of the file reader from the saved pre-deserialization position</code> and asserts that it equals the export's expected length. It's misleading though as you cannot simply read SerialSize</code> bytes from its offset.</p>
Note: I'm not 100% confident in the data being interleaved vs just sequential. Through observing seek/read operations for various exports, I do see seeks going to a wildly different offset in the middle of deserializing an export, then another export deserializing, then seeking back to the original export and continuing to deserialize it again This is a PITA to debug though</em>.</p>
#</a
>
Why??????</h3>
I imagine there's a very good reason for packaging data this way. It's best to consider the constraints of the time:</p>

The game is being shipped on a physical disc.</li>
The Xbox has 64MB of RAM shared between the CPU and GPU, with some portion of that being dedicated to the OS.</li>
The CPU wasn't terribly slow for the time, but wasting cycles would have been noticed.</li>
</ol>
The .lin</code> format mitigates these issues with:</p>

Compressing data means you save space on the disc... If you conveniently ignore the fact that common.lin</code> is duplicated in each map's directory and is the same for every map I tested, which kinda negates part of this.</li>
Streaming data in from the file instead of decompressing the whole thing at once saves on overall memory pressure during the data loading phase.</li>
Laying out the file in a byte-for-byte exact read order increases I/O speeds by not having to seek around the physical media, and ensures that you don't need magic to translate an uncompressed offset to a compressed one in a performant manner.</li>
</ol>
#</a
>
Logging Load Order for Static Recompilation</h2>
I really, really wanted to avoid doing any runtime dumping that requires playing the game in an emulator or physical console. It doesn't scale well to other games and is generally less flexible. But doing runtime observations are extremely useful in making sense of the format, so I went ahead and added some logging to get an idea of the file read order from the compressed archive when booting the game:</p>
..\System\Engine.u</span></span>
..\System\Core.u</span></span>
..\System\Echelon.u</span></span>
..\Textures\HUD.utx</span></span>
..\Sounds\FisherFoley.uax</span></span>
..\Sounds\CommonMusic.uax</span></span>
..\System\EchelonEffect.u</span></span>
..\Textures\ETexSFX.utx</span></span>
..\Textures\2-1_CIA_tex.utx</span></span>
..\Textures\generic_shaders.utx</span></span>
..\Textures\LightGenTex.utx</span></span>
..\Textures\5_1_PresidentialPalace_tex.utx</span></span>
..\Textures\1_2_Def_Ministry_tex.utx</span></span>
..\Textures\EGO_Tex.utx</span></span>
..\Textures\ETexIngredient.utx</span></span>
..\Textures\1-1_TBilisi_tex.utx</span></span>
..\Textures\1_3_CaspianOilRefinery_TEX.utx</span></span>
..\StaticMeshes\EMeshSFX.usx</span></span>
..\StaticMeshes\EGO_OBJ.usx</span></span>
..\Textures\ETexCharacter.utx</span></span>
..\Textures\4_3_Chinese_Embassy_tex.utx</span></span>
..\Textures\4_3_0_Chinese_Embassy_tex.utx</span></span>
..\Textures\4_3_2_Chinese_Embassy_tex.utx</span></span>
..\Sounds\water.uax</span></span>
..\Sounds\DestroyableObjet.uax</span></span>
..\Sounds\FisherVoice.uax</span></span>
..\Sounds\FisherEquipement.uax</span></span>
..\Sounds\GunCommon.uax</span></span>
..\Sounds\Interface.uax</span></span>
..\Sounds\Electronic.uax</span></span>
..\Sounds\Dog.uax</span></span>
..\Sounds\Lambert.uax</span></span>
..\StaticMeshes\EMeshIngredient.usx</span></span>
..\StaticMeshes\EMeshCharacter.usx</span></span>
..\Textures\2_2_1_Kalinatek_tex.utx</span></span>
..\StaticMeshes\LightGenOBJ.usx</span></span>
..\Textures\ETexRenderer.utx</span></span>
..\Sounds\Door.uax</span></span>
..\Sounds\GenericLife.uax</span></span>
..\Sounds\Special.uax</span></span>
..\Sounds\ThrowObject.uax</span></span>
..\StaticMeshes\Generic_Mesh.usx</span></span>
..\StaticMeshes\prog\generic_obj.usx</span></span>
..\Textures\0_0_Training_tex.utx</span></span>
..\Textures\3_4_Severo_tex.utx</span></span>
..\System\EchelonIngredient.u</span></span>
..\Sounds\Gun.uax</span></span>
..\System\EchelonGameObject.u</span></span>
..\Animations\ESkelIngredients.ukx</span></span>
..\Sounds\Metal.uax</span></span>
..\Animations\ETrk.ukx</span></span>
..\StaticMeshes\2-1_cia_obj.usx</span></span>
..\System\EchelonHUD.u</span></span>
..\Animations\ESam.ukx</span></span>
..\Maps\menu\menu.unr             // <--- # 55</span></span>
..\Textures\2_2_Kalinatek_tex.utx</span></span>
..\StaticMeshes\2_2_Kalinatek_OBJ.usx</span></span>
..\System\EchelonPattern.u</span></span>
..\Sounds\S3_4_2Voice.uax</span></span>
..\Sounds\S3_4_3Voice.uax</span></span>
..\Sounds\S2_2_2Voice.uax</span></span>
..\Sounds\S2_1_2Voice.uax</span></span>
..\Sounds\S5_1_2Voice.uax</span></span>
..\Sounds\S3_2_2Voice.uax</span></span>
..\Sounds\S4_2_2Voice.uax</span></span>
..\Sounds\S4_1_1Voice.uax</span></span>
..\Sounds\S1_2_1Voice.uax</span></span>
..\Sounds\S1_1_2Voice.uax</span></span>
..\Sounds\S0_0_3Voice.uax</span></span>
..\Sounds\S3_2_1Voice.uax</span></span>
..\Sounds\S4_2_1Voice.uax</span></span>
..\Sounds\S1_3_3Voice.uax</span></span>
..\Sounds\S0_0_2Voice.uax</span></span>
..\Sounds\S4_3_2Voice.uax</span></span>
..\Sounds\S1_1_1Voice.uax</span></span>
..\Sounds\S2_2_1Voice.uax</span></span>
..\Sounds\S4_3_1Voice.uax</span></span>
..\Sounds\S5_1_1Voice.uax</span></span>
..\Sounds\S4_1_2Voice.uax</span></span>
..\Sounds\S2_1_1Voice.uax</span></span>
..\Sounds\S1_1_0Voice.uax</span></span>
..\Sounds\S2_2_3Voice.uax</span></span>
..\Sounds\S2_1_0Voice.uax</span></span>
..\Sounds\S1_2_2Voice.uax</span></span>
..\Sounds\Vehicules.uax</span></span>
..\Sounds\S1_1_Voice.uax</span></span>
..\Sounds\S2_1_Voice.uax</span></span>
..\Sounds\S4_3_0Voice.uax</span></span>
..\Sounds\S1_3_2Voice.uax</span></span>
..\Sounds\Machine.uax</span></span>
..\Sounds\FireSound.uax</span></span>
..\Sounds\SoundEvent.uax</span></span>
..\Sounds\S0_0_Voice.uax</span></span>
..\Sounds\S4_3_Voice.uax</span></span>
..\Sounds\S4_2_Voice.uax</span></span>
..\Sounds\S5_1_Voice.uax</span></span>
..\Sounds\XboxLive.uax</span></span>
..\System\EchelonCharacter.u</span></span>
..\Sounds\GearCommon.uax</span></span>
..\Animations\ENPC.ukx</span></span>
..\Sounds\Exspetsnaz.uax</span></span>
..\Sounds\GeorgianSoldier.uax</span></span>
..\Sounds\RussianMafioso.uax</span></span>
..\Sounds\GeorgianCop.uax</span></span>
..\Sounds\EliteForce.uax</span></span>
..\Sounds\CiaSecurity.uax</span></span>
..\Sounds\CiaAgentMale.uax</span></span>
..\Sounds\ChineseSoldier.uax</span></span>
..\Animations\EFemale.ukx</span></span>
..\Animations\EDog.ukx</span></span>
..\Sounds\GeorgianPalaceGuard.uax</span></span></code></pre>
    File Dumping Script</summary>
    
        I set a breakpoint in the prologue of a function with the string "LinkerExists</code>" that I later determined to be the constructor for an object called ULinkerLoad</code>. One of the arguments is the file name for this object.</p>
When triggered, the breakpoint executes the following IDA Python script which reads the filename pointer, then the filename, outputs it to the IDA console, and continues execution:</p>
import</span> ida_idd</span>,</span> ida_kernwin</span>,</span> ctypes</span></span>
p</span>=</span>ida_dbg</span>.</span>get_reg_val</span>("</span>ebx</span>")</span></span>
s</span>=</span>b</span>""</span></span>
while</span> True:</span></span>
    c</span> =</span> ida_idd</span>.</span>dbg_read_memory</span>(</span>p</span>,</span>2</span>)</span></span>
    if</span> not</span> c</span> or</span> c</span> ==</span> b</span>"</span>\x00\x00</span>":</span> break</span></span>
    s</span> +=</span> c; p</span>+=</span>2</span></span>
</span>
ida_kernwin</span>.</span>msg</span>("</span>ULinkerLoad: </span>" +</span> s</span>.</span>decode</span>('</span>utf-16-le</span>')+"</span>\n</span>")</span></span></code></pre>
    </div>
</details>
In the above file load order I annotated file #55 which is ..\Maps\menu\menu.unr</code>. The common.lin</code> file has 54 Linker files and #55 in the above listing happens to be the map which is loading and has its own dedicated .lin</code> file. This is a strong indicator that the common.lin</code> archive genuinely contains only 54 files and anything else is read from level-specific archives.</p>
I also set a breakpoint in the function which deserializes exports (called Preload</code>) and did some logging of which export is read and when a stream seek occurred:</p>
ULinkerLoad: ..\System\Engine.u</span></span>
ULinkerLoad: ..\System\Core.u</span></span>
Export offset: 0x0,0x0,0x0,0x97,0x40f0004,0x4d,0x1b05</span></span>
Seeking to/from: 0x1b05,0x10883</span></span>
Export offset: 0xfffffffe,0x0,0x3,0x13d,0x70004,0x1c,0x6531</span></span>
Seeking to/from: 0x6531,0x1b18</span></span>
Read complete: 0xfffffffe,0x0,0x3,0x13d,0x70004,0x1c,0x6531</span></span>
Seeking to/from: 0x1b18,0x654d</span></span>
Export offset: 0xfffffffe,0x0,0x3,0x13c,0x70004,0x1c,0x6515</span></span>
Seeking to/from: 0x6515,0x1b18</span></span>
Read complete: 0xfffffffe,0x0,0x3,0x13c,0x70004,0x1c,0x6515</span></span>
Seeking to/from: 0x1b18,0x6531</span></span>
Export offset: 0xfffffffe,0x0,0x3,0x119d,0x70004,0x2c,0x6432</span></span>
Seeking to/from: 0x6432,0x1b18</span></span>
Seeking to/from: 0x6451,0x6452</span></span>
Seeking to/from: 0x6453,0x6454</span></span>
Seeking to/from: 0x6454,0x6455</span></span>
Seeking to/from: 0x6455,0x6456</span></span>
Export offset: 0xfffffffd,0x0,0x2d7,0x477,0x70004,0xb,0x1c35</span></span>
Seeking to/from: 0x1c35,0x6457</span></span>
Read complete: 0xfffffffd,0x0,0x2d7,0x477,0x70004,0xb,0x1c35</span></span>
Seeking to/from: 0x6457,0x1c40</span></span>
Export offset: 0xfffffffd,0x0,0x2d7,0x46d,0x70004,0xb,0x2736</span></span></code></pre>
    Export Preload Script</summary>
    
        IDA Python breakpoint script at Preload</code> entry, identifiable by the string "SerialSize</code>" and</strong> after the deserialization routine is called:</p>
import</span> ida_dbg</span>,</span> ida_idd</span>,</span> ida_kernwin</span>,</span> ctypes</span>,</span> time</span></span>
</span>
export_addr</span>=</span>ida_dbg</span>.</span>get_reg_val</span>("</span>ebp</span>")</span></span>
</span>
class_index</span> =</span> int</span>.</span>from_bytes</span>(</span>ida_idd</span>.</span>dbg_read_memory</span>(</span>export_addr</span>,</span> 4</span>), "</span>little</span>")</span></span>
super_index</span> =</span> int</span>.</span>from_bytes</span>(</span>ida_idd</span>.</span>dbg_read_memory</span>(</span>export_addr</span> +</span> 4</span>,</span> 4</span>), "</span>little</span>")</span></span>
package_index</span> =</span> int</span>.</span>from_bytes</span>(</span>ida_idd</span>.</span>dbg_read_memory</span>(</span>export_addr</span> +</span> 8</span>,</span> 4</span>), "</span>little</span>")</span></span>
object_name</span> =</span> int</span>.</span>from_bytes</span>(</span>ida_idd</span>.</span>dbg_read_memory</span>(</span>export_addr</span> +</span> 12</span>,</span> 4</span>), "</span>little</span>")</span></span>
object_flags</span> =</span> int</span>.</span>from_bytes</span>(</span>ida_idd</span>.</span>dbg_read_memory</span>(</span>export_addr</span> +</span> 16</span>,</span> 4</span>), "</span>little</span>")</span></span>
serial_size</span> =</span> int</span>.</span>from_bytes</span>(</span>ida_idd</span>.</span>dbg_read_memory</span>(</span>export_addr</span> +</span> 20</span>,</span> 4</span>), "</span>little</span>")</span></span>
serial_offset</span> =</span> int</span>.</span>from_bytes</span>(</span>ida_idd</span>.</span>dbg_read_memory</span>(</span>export_addr</span> +</span> 24</span>,</span> 4</span>), "</span>little</span>")</span></span>
</span>
edx</span>=</span>ida_dbg</span>.</span>get_reg_val</span>("</span>edx</span>")</span></span>
</span>
properties</span> = [</span>class_index</span>,</span> super_index</span>,</span> package_index</span>,</span> object_name</span>,</span> object_flags</span>,</span> serial_size</span>,</span> serial_offset</span>]</span></span>
</span>
ida_kernwin</span>.</span>msg</span>("</span>Export data: </span>" + "</span>,</span>".</span>join</span>(</span>hex</span>(</span>n</span>)</span> for</span> n</span> in</span> properties</span>) +"</span>\n</span>")</span></span></code></pre>
    </div>
</details>
There is really no discernable pattern to the loads at all. The file/export load order seems to be just satisfying the dependency graph (exports required for parents/properties of yet-to-be-parsed types) for requested objects from the C++ side of the house.</p>
I think an acceptable compromise to doing this statically would be requiring dumping the file/export load order from the game... but more work is needed to prove the viability of this approach.</p>
I adjusted my program to read my logged lines into a queue of exports to be parsed, using the completed</strong> reads (lines starting with Read complete</code> rather than Export offset</code>). It then attempted to find the matching export in the export table across any package, and read its size. Repeat until the next Linker object is encountered, parse that, add it to the list, and repeat.</p>
This quickly proved to be non-viable with my very barebones program. I would hit a point where I failed to find a matching export for the line logged, presumably because I was not reading the correct amount of data required to reach the next Unreal Package where that export was declared.</p>
This was either a bug, or maybe some of the types attempt to seek+read without triggering a Preload()</code>. At any rate, I had now invested a week or longer on the static approach with no data successfully dumped yet.</p>
#</a
>
Dumping at Runtime</h2>
At some point during the above research, I discovered the EnhancedSC</a> project -- a community patch for Splinter Cell 1 on PC which fixes bugs, adds gameplay improvements, and has folks who certainly know the game engine better than me. I joined their Discord and asked if anyone knew about this format and they said that it's been a dead end for anyone who's bothered.</p>
They were quite interested though in any progress achieved as they want to port some content from the Xbox versions of the games to PC. Through this community I got some great help with various theories, ideas, and introduced to tooling like UE-Explorer</a>.</p>
After spending about a week on static recompilation I didn't want to spend even more time investing in getting things dumped only to hit a hard wall. For example discovering that the files were wildly different than expected, wouldn't work on PC, or wouldn't work with UE Explorer. I needed to dump something</em>.</p>
The game can obviously read the data fine. The thought came to me that perhaps I could just dump the data into some crappy format after it's read that makes piecing it back together easy.</p>
While doing static analysis I came across a function that was very peculiar to me. I identified the ULinkerLoad</code> function mentioned earlier by searching for the Unreal Package file magic (highlighted below), and found the following function:</p>
</a></p>
As expected, the file magic is checked against what's read from disk. But there's another result for the magic in a different function that is setting</strong> some structure's field to the magic:</p>
</a></p>
And what is the purpose of this code? As it turns out, user game saves are just Unreal Objects serialized in the same format -- sans compression and other oddities that go along with it!</p>
</a></p>
#</a
>
Patching OG Xbox Binaries</h3>
In order to do interesting things, we need to run our own code alongside the game. Debugger scripts are simply too slow and unreliable, so we need something running in the emulator or on a physical device. It'd also be cool if I could write a QEMU plugin for the emulator... but that's another rabbit hole.</p>
Injecting code into a game on Windows or Unix is easy. You can CreateRemoteThread()</code> or DLL hijack on Windows, and on Unix use LD_PRELOAD</code>. On Xbox 360 you can "inject" persistent DLLs. On original Xbox, you have one process with (as far as I know), no DLLs.</p>
This could probably be a blog post on its own since modern information is pretty scarce (RIP XboxHacker.org), but there are at least two tools I know of that can be used to manipulate original Xbox executables.</p>

The Python library pyxbe</a></li>
The CLI tool XboxImageExploder</a></li>
</ol>
Both of these tools allow you to add a new section to an executable and basically create a code cave that you can use for placing additional code or data. When the system loads the image, it maps that newly added section with the appropriate permissions. You then need to patch some place in the original executable so that your code runs.</p>
Using XboxImageExploder and XePatcher I was able to write a patch</a> which calls the serialization routine on an object after it gets loaded into memory.</p>
tl;dr</strong> of the patch:</p>

Define a hook point at the end of theLoadMap()</code> function. This definition will cause XePatcher to write these instructions that jump execution to Hack_LoadMap</code> at the declared file offset.</li>
Hack_LoadMap</code> calls Hack_DumpAllLinkers</code> and does the standard epilogue cleanup for LoadMap()</code> which won't be executed since we hijacked execution</li>
Hack_DumpAllLinkers</code> iterates a global list of Linker</code> objects and calls Hack_DumpFile</code> with that linker as an argument.</li>
Hack_DumpFile</code> ensures that the output directory for the given Linker</code> file is created, then calls the game-provided function which serializes the Linker</code> to that path. For example, the ..\System\Engine.u</code> linker file from the common.lin</code> file will be written to z:\System\Engine.u</code>.</li>
</ol>
;---------------------------------------------------------</span></span>
; At the very end of the LoadMap() routine</span></span>
;---------------------------------------------------------</span></span>
; file offset, not a VA</span></span>
dd</span>      73698h</span></span>
dd</span>      (_load_map_return_end - _load_map_return_start)</span></span>
_load_map_return_start</span>:</span></span>
</span>
    ; Jump to our detour function</span></span>
    push    esi</span></span>
    mov     eax</span>, Hack_LoadMap</span></span>
    jmp     eax</span></span>
</span>
_load_map_return_end</span>:</span></span>
</span>
</span>
_Hack_LoadMapCalled</span>:</span></span>
    dd</span>      0</span></span>
</span>
_Hack_LoadMap</span>:</span></span>
    mov     eax</span>, Hack_DumpAllLinkers</span></span>
    call    eax</span></span>
</span>
    mov     eax</span>, Hack_LoadMapCalled</span></span>
    mov</span>     dword</span> [</span>eax</span>], </span>1</span></span>
</span>
    _load_map_restore_registers</span>:</span></span>
    ; return value that we clobbered in the</span></span>
    ; hook</span></span>
    pop     eax</span></span>
</span>
    ; Since we patched in the prologue, we will just</span></span>
    ; do the register restore ourselves</span></span>
    pop     edi</span></span>
    pop     esi</span></span>
    pop     ebx</span></span>
    mov     esp</span>, </span>ebp</span></span>
    pop     ebp</span></span>
    retn</span>    8</span></span>
</span>
_Hack_DumpAllLinkers</span>:</span></span>
    push    ebx</span></span>
    push    esi</span></span>
</span>
    %</span>define</span> g_ObjectLinkers 0033c42ch</span></span>
</span>
    ; Load the linker count</span></span>
    mov     ebx</span>, [g_ObjectLinkers + </span>4</span>]</span></span>
    test    ebx</span>, </span>ebx</span></span>
    jz</span>      _dump_all_linkers_restore_registers</span></span>
</span>
    ; esi will be our index</span></span>
    mov     esi</span>, </span>0</span></span>
</span>
    _dump_all_linkers_linker_loop_start</span>:</span></span>
</span>
    cmp     esi</span>, </span>ebx</span></span>
    jz</span>      _dump_all_linkers_linker_loop_finish</span></span>
</span>
    ; Iterate the linkers</span></span>
    mov     eax</span>, [g_ObjectLinkers]</span></span>
    mov     ecx</span>, </span>esi</span></span>
    imul    ecx</span>, </span>4</span></span>
</span>
    add     eax</span>, </span>ecx</span></span>
    mov     eax</span>, [</span>eax</span>]</span></span>
    push    eax</span></span>
    mov     ecx</span>, Hack_DumpFile</span></span>
    call    ecx</span></span>
    add     esp</span>, (</span>4</span> * </span>1</span>)</span></span>
</span>
    _dump_all_linkers_linker_loop_end</span>:</span></span>
    inc     esi</span></span>
    jmp</span>     _dump_all_linkers_linker_loop_start</span></span>
</span>
    _dump_all_linkers_linker_loop_finish</span>:</span></span>
    _dump_all_linkers_restore_registers</span>:</span></span>
    pop     esi</span></span>
    pop     ebx</span></span>
    ret</span></span>
</span>
_Hack_DumpFile</span>:</span></span>
    ; Load the argument representing the</span></span>
    ; object that's being saved</span></span>
    mov     eax</span>, [</span>esp</span> + </span>4</span>]</span></span>
</span>
    ; Save registers</span></span>
    push    edi</span></span>
    push    esi</span></span>
    push    ebx</span></span>
</span>
    mov     edi</span>, </span>eax</span></span>
</span>
    _dump_file_do_dump</span>:</span></span>
</span>
    ; Iterate the object's exports and save their flags</span></span>
</span>
    ; ==== NOT USED</span></span>
    ; Grab the export data pointer</span></span>
    ;mov     ecx, [edi + 0x88]</span></span>
    ; Grab the number of exports</span></span>
    ;mov     ebx, [edi + 0x8C]</span></span>
    ; ==== NOT USED</span></span>
</span>
    ; Allocate space for the file path</span></span>
    sub     esp</span>, </span>0x200</span></span>
</span>
</span>
    ; Grab the linker's filename</span></span>
    mov     eax</span>, [</span>edi</span> + </span>0x98</span>]</span></span>
</span>
    ; Put the input filename in esi</span></span>
    mov     esi</span>, </span>eax</span></span>
</span>
    ; If the input filename is empty, jump to the cleanup routine</span></span>
    ; since this is not a file that's in the packed .lin</span></span>
    cmp</span>    word</span> [</span>eax</span>], </span>0</span></span>
    jz</span>     _Hack_DumpFile_Done</span></span>
</span>
    ;===== DIRECTORY CREATION</span></span>
    ; The file path is located at the beginning of the stack</span></span>
    mov     ebx</span>, </span>esp</span></span>
</span>
    ; Set the filename on the stack to `z:`</span></span>
    ; This has to be a char*, not a wchar_t*</span></span>
    mov</span>     byte</span> [</span>esp</span>], 'z'</span></span>
    mov</span>     byte</span> [</span>esp</span> + </span>1</span>], ':'</span></span>
</span>
    ; This will hold our position in the path we're building</span></span>
    mov     ebx</span>, </span>0</span></span>
</span>
    _Hack_DumpFile_File_Directory</span>:</span></span>
</span>
    ; We are looking for a backslash</span></span>
    ; this is wchar_t `\`</span></span>
    push</span>    0x005c</span></span>
    ; Grab the position of the last backslash for the</span></span>
    ; input file</span></span>
    push    esi</span></span>
    mov     eax</span>, appStrchr</span></span>
    call    eax</span></span>
    add     esp</span>, (</span>4</span> * </span>2</span>)</span></span>
</span>
    ; Not found</span></span>
    test    eax</span>, </span>eax</span></span>
    jz</span>      _Hack_DumpFile_Directory_Finish</span></span>
</span>
    ; We found a slash -- check if we've discarded the first</span></span>
    ; bit of data before the slash (it's expected to start</span></span>
    ; with "..\" )</span></span>
    test    ebx</span>, </span>ebx</span></span>
    jnz</span>     _Hack_DumpFile_File_Directory_Create_Directory</span></span>
</span>
    ; Update ebx to point to the first slash so we can use it</span></span>
    ; for later copying.</span></span>
    mov     ebx</span>, </span>eax</span></span>
    jmp</span>     _hack_dumpfile_directory_end</span></span>
</span>
    _Hack_DumpFile_File_Directory_Create_Directory</span>:</span></span>
</span>
    ; Skip the Z: part for the dest file path</span></span>
    lea     ecx</span>, [</span>esp</span> + </span>2</span>]</span></span>
    push    edx</span></span>
    push    esi</span></span>
    ; Start of the linker's file path</span></span>
    mov     esi</span>, </span>ebx</span></span>
</span>
    ; Copy from ebx to eax</span></span>
    _hack_dump_file_copy_directory_loop</span>:</span></span>
    cmp     esi</span>, </span>eax</span></span>
    je</span>      _hack_dump_file_copy_directory_loop_finish</span></span>
</span>
    mov     dl</span>,   [</span>esi</span>]</span></span>
    mov</span>     [</span>ecx</span>], </span>dl</span></span>
    inc     ecx</span></span>
    ; we're doing some janky wchar_t to char</span></span>
    ; conversion tricks</span></span>
    add     esi</span>, </span>2</span></span>
</span>
    jmp</span>     _hack_dump_file_copy_directory_loop</span></span>
</span>
    _hack_dump_file_copy_directory_loop_finish</span>:</span></span>
    ; Add null terminator</span></span>
    mov</span>     byte</span> [</span>ecx</span>], </span>0</span></span>
</span>
    pop     esi</span></span>
    pop     edx</span></span>
</span>
    mov     ecx</span>, </span>esp</span></span>
    ; Make sure we don't clobber eax</span></span>
    push    eax</span></span>
</span>
    ; Attributes</span></span>
    push</span>    0x0</span></span>
    ; Create this directory</span></span>
    push    ecx</span></span>
    mov     ecx</span>, CreateDirectory</span></span>
    call    ecx</span></span>
    ; cdecl function, it cleans up</span></span>
</span>
    pop     eax</span></span>
</span>
    _hack_dumpfile_directory_end</span>:</span></span>
    ; Save the position</span></span>
    lea     esi</span>, [</span>eax</span> + </span>2</span>]</span></span>
    jmp</span>     _Hack_DumpFile_File_Directory</span></span>
</span>
    _Hack_DumpFile_Directory_Finish</span>:</span></span>
</span>
    ; Set the file path we want to copy</span></span>
    mov     esi</span>, </span>ebx</span></span>
</span>
    ;===== FILE CREATION</span></span>
</span>
    ; The file path is located at the beginning of the stack</span></span>
    mov     ebx</span>, </span>esp</span></span>
</span>
    ; Set the start of VeryLongString to `Z:`</span></span>
    push</span>    ZDrive</span></span>
    push    ebx</span></span>
    mov     eax</span>, wstrcpy</span></span>
    call    eax</span></span>
    add     esp</span>, (</span>4</span> * </span>2</span>)</span></span>
</span>
    ; Set the copy target to the bytes immediatley</span></span>
    ; following `z:`, so the result should be</span></span>
    ; `z:\filename`</span></span>
    lea     eax</span>, [</span>ebx</span> + </span>4</span>]</span></span>
</span>
    ; Copy the filename to the path buffer</span></span>
    push    esi</span></span>
</span>
    ; Set ESI to the full file path for later use</span></span>
    mov     esi</span>, </span>ebx</span></span>
</span>
    push    eax</span></span>
    mov     eax</span>, wstrcpy</span></span>
    call    eax</span></span>
    add     esp</span>, (</span>4</span> * </span>2</span>)</span></span>
</span>
    ; Error</span></span>
    mov     edx</span>, </span>dword</span> [GlobalError]</span></span>
    ; InOuter</span></span>
    mov     eax</span>, [</span>edi</span> + </span>2Ch</span>]</span></span>
</span>
    ; Pad size?</span></span>
    push</span>    0xFFFFFFFF</span></span>
    ; Conform</span></span>
    push</span>    0x0</span></span>
    ; Error</span></span>
    push    edx</span></span>
    ; Filename</span></span>
    push    esi</span></span>
    ; TopLeveLFlags</span></span>
    push</span>    -</span>1</span></span>
    ; Base</span></span>
    push    edi</span></span>
    ; InOuter</span></span>
    push    eax</span></span>
</span>
    ; ( UObject* InOuter,</span></span>
    ;   UObject* Base,</span></span>
    ;   DWORD TopLevelFlags,</span></span>
    ;   const TCHAR* Filename,</span></span>
    ;   FOutputDevice* Error=GError,</span></span>
    ;   ULinkerLoad* Conform=NULL );</span></span>
    mov     eax</span>, UObject_SavePackage</span></span>
    call    eax</span></span>
    add     esp</span>, (</span>7</span> * </span>4</span>)</span></span>
</span>
</span>
    _Hack_DumpFile_Done</span>:</span></span>
</span>
    ; Restore the stack to clean up the file</span></span>
    ; path</span></span>
    add     esp</span>, </span>0x200</span></span>
</span>
    ; Restore the export flags</span></span>
</span>
    _dump_file_restore_registers</span>:</span></span>
</span>
    ; Restore saved registers</span></span>
    pop     ebx</span></span>
    pop     esi</span></span>
    pop     edi</span></span>
</span>
    ret</span></span>
</span></code></pre>
Why assembly?</em> Sunk cost and not wanting to figure out tooling required to compile C to shell code targeting this platform. The functions could be written in C. The detour hooks must remain assembly.</p>
#</a
>
Results</h2>
We can now read the output files in UE Explorer, and even run the Xbox main menu and the in-engine cinematic from the first level on PC... albeit with some bugged lighting and textures. Anything past that first level cinematic, including the interactive bit of the level itself, has failed to load.</p>
The above patch, dumping at LoadMap()</code> end, resulted in the most reliable file dumping out of my many experiments. At the end of this function it seems nearly all data is read and ready to go, but there are definitely a couple of objects deserialized after this point. Dumping after all object reads are complete though actually seems to make things worse -- maybe because some object properties have changed in-memory from their default values?</p>
Loading an Xbox file in UE Explorer:</p>
</a></p>
First level cinematic which should have a dimly-lit hallway with shadows:</p>
</a></p>
Swapping a texture had some problems:</p>
</a></p>
In fact, the textures were just straight up incomplete data. Since literally every texture object had a small size I figured that between the time the object was deserialized and the point I dumped it the original texture data must have been transformed to the target format and the original free'd.</p>
To confirm my assumption, I set a breakpoint in the function which kicks off object deserialization so that it would break immediately before:</p>

    Targeted Export Breakpoint Code</summary>
    
        I set a breakpoint in the Preload()</code> routine immediately before object deserialization with the following script.</p>
import</span> ida_dbg</span>,</span> ida_idd</span>,</span> ida_kernwin</span>,</span> ctypes</span>,</span> time</span></span>
</span>
export_addr</span>=</span>ida_dbg</span>.</span>get_reg_val</span>("</span>ebp</span>")</span></span>
</span>
serial_offset</span> =</span> int</span>.</span>from_bytes</span>(</span>ida_idd</span>.</span>dbg_read_memory</span>(</span>export_addr</span> +</span> 24</span>,</span> 4</span>), "</span>little</span>")</span></span>
</span>
class_index</span> =</span> int</span>.</span>from_bytes</span>(</span>ida_idd</span>.</span>dbg_read_memory</span>(</span>export_addr</span>,</span> 4</span>), "</span>little</span>")</span></span>
super_index</span> =</span> int</span>.</span>from_bytes</span>(</span>ida_idd</span>.</span>dbg_read_memory</span>(</span>export_addr</span> +</span> 4</span>,</span> 4</span>), "</span>little</span>")</span></span>
package_index</span> =</span> int</span>.</span>from_bytes</span>(</span>ida_idd</span>.</span>dbg_read_memory</span>(</span>export_addr</span> +</span> 8</span>,</span> 4</span>), "</span>little</span>")</span></span>
object_name</span> =</span> int</span>.</span>from_bytes</span>(</span>ida_idd</span>.</span>dbg_read_memory</span>(</span>export_addr</span> +</span> 12</span>,</span> 4</span>), "</span>little</span>")</span></span>
object_flags</span> =</span> int</span>.</span>from_bytes</span>(</span>ida_idd</span>.</span>dbg_read_memory</span>(</span>export_addr</span> +</span> 16</span>,</span> 4</span>), "</span>little</span>")</span></span>
serial_size</span> =</span> int</span>.</span>from_bytes</span>(</span>ida_idd</span>.</span>dbg_read_memory</span>(</span>export_addr</span> +</span> 20</span>,</span> 4</span>), "</span>little</span>")</span></span>
serial_offset</span> =</span> int</span>.</span>from_bytes</span>(</span>ida_idd</span>.</span>dbg_read_memory</span>(</span>export_addr</span> +</span> 24</span>,</span> 4</span>), "</span>little</span>")</span></span>
</span>
edx</span>=</span>ida_dbg</span>.</span>get_reg_val</span>("</span>edx</span>")</span></span>
</span>
properties</span> = [</span>class_index</span>,</span> super_index</span>,</span> package_index</span>,</span> object_name</span>,</span> object_flags</span>,</span> serial_size</span>,</span> serial_offset</span>]</span></span>
</span>
ida_kernwin</span>.</span>msg</span>("</span>Export offset: </span>" + "</span>,</span>".</span>join</span>(</span>hex</span>(</span>n</span>)</span> for</span> n</span> in</span> properties</span>) +"</span>\n</span>")</span></span>
</span>
# Break only when the object offset matches my target</span></span>
return</span> serial_offset</span> ==</span> 0x</span>11f65f</span></span></code></pre>
    </div>
</details>
Then I set a breakpoint in the data read function and examined where the texture data from disk was being copied to. I had to do some stepping through to figure out where this destination pointer originated, then located immediately next to that</em> pointer is the dynamic array's length and capacity. Set a Memory Write breakpoint on the length field and wait for it to go to zero. Wherever it happened, that was the code and the realloc</code> immediately following had to simply get nop'd out to prevent the texture data from being evicted.</p>
After adjusting my patch and dumping data again I noted that the texture file's size changed dramatically and I now had some nice textures to look at:</p>
</a></p>
</a></p>
#</a
>
General Issues With Dumping</h2>
Since exports are lazy loaded, you can only dump what's used in a level. The main menu uses some functionality from Engine</code> and Core</code>, but not all of it. So if I load the main menu map and dump all the linkers when it's finished loading, I will only have a partial representation of Engine</code> and Core</code>.</p>
In the same vein, anything unreferenced or unused which might by happenstance be in the archive cannot be easily recovered without intelligent brute forcing since you don't know where its data starts. e.g. the main menu has some brushes which are in the export table but appear to be unused, so nothing ever triggers their appropriate load.</p>
#</a
>
Next Steps</h2>
While we've had some wins with dumping data and I feel I've accomplished a lot, I'm not going to be satisfied until I can cleanly dump anything I want from the game. A major milestone would be to get the training mission on Xbox completely working on PC. I'm going to try to make some strides here and if I hit a wall, I'll at least try to dump some data from the review copy of the game.</p>
This format can certainly be read statically with</em> load-order knowledge from the game engine. I'm hoping for now that someone from the community can use the work presented in this blog post to get it working in Unreal-Library.</p>
Static recompilation would be a much more general approach which I hope only requires two debugger breakpoint scripts to dump package filenames and export loads on a per-game basis rather than a binary patch. We already have this for SC1. The difficulty would be in contributing to UELib (or some other project) so that it can match the game's exact I/O behavior and deserialize multiple packages simultaneously using the load-order data. If this interests you, check out the issue I filed in the project repo</a>.</p>
If you have questions, feel free to reach out to me on Twitter: @landaire</a>.</p>
#</a
>
Thanks</h2>

Grimdoomer for getting me up to speed with writing OG Xbox patches and for listening to my rants about this format.</li>
To the EnhancedSC developer community for helping inspect my dumped files and for investing in what success we've had so far.</li>
EliotVU for developing the great UE Explorer and UELib.</li>
The folks who documented their own findings on this format before me. Every little bit of information helps.</li>
</ul>

Saving My Commit With `jj evolog`

2025-10-03T00:00:00+00:00

jj</code> (jujutsu)</a> is a newish git-compatible version control system that has some fresh ideas and a pretty great CLI UX (compared to git</code>). I had a moment recently where I hadn't yet committed my changes and while attempting to format the code I inadvertently made my diff way larger than it should have been.

Here's what happened:

I made some changes that I was quite happy with and did a quick jj diff</code> / jj status</code> to make sure there was nothing I was accidentally including in the change.</li> </ol>
❯ : jj status Working copy changes: M Cargo.toml M src/de.rs M src/lib.rs M src/ser.rs M src/value.rs M src/value_impls.rs Working copy (@) : zxsrvopz 43a4bc7d master | de: very rough PoC of refcounted data Parent commit (@-): xlzooroo 9b8d55a1 core: update quickcheck + rand and tests (tests failing)</code></pre>

I noticed that the changes were not well-formatted, so I ran cargo fmt</code> </li>
I ran jj status</code> and had an oh shit moment when I realized way more files were changed from the cargo fmt</code> than I thought would be. </li> </ol>
❯ : jj status Working copy changes: M Cargo.toml M src/consts.rs M src/de.rs M src/error.rs M src/lib.rs M src/ser.rs M src/value.rs M src/value_impls.rs M test/arby.rs M test/mod.rs Working copy (@) : zxsrvopz 69322ff0 master | de: very rough PoC of refcounted data Parent commit (@-): xlzooroo 9b8d55a1 core: update quickcheck + rand and tests (tests failing)</code></pre> So now I have a problem: how do I undo the cargo fmt</code> step?
Note: A lobste.rs reader helpfully pointed out</a> that jj undo</code> would actually suffice at this point! Let's pretend like this is more involved though :p
#</a > Enter jj evolog</code></h2> One of jj</code>'s commands is evolog</code></a>, which "shows how a change has evolved over time". jj</code> takes a snapshot of the working copy every time a command is run, so when I ran jj diff</code> to check my work, it recorded a snapshot of my working copy from the last time I ran a jj</code> command. Running jj evolog</code> you'll see something like: ❯ : jj evolog ○ mmxmynwt hidden example@example.com 2025-10-03 18:08:05 7aa68914 │ (no description set) │ -- operation f606802e1b09 (2025-10-03 18:08:05) snapshot working copy ○ mmxmynwt hidden example@example.com 2025-10-03 18:07:58 4970219f (empty) (no description set) -- operation bb8323da564f (2025-10-03 18:07:58) new empty commit</code></pre> Pretending that there's more than the empty commit and snapshot working copy operations, this on its own is not super helpful in identifying what happened in each operation. Running jj evolog -p</code> shows more details: ❯ : jj evolog -p @ zxsrvopz example@example.com 2025-10-03 18:07:03 master 69322ff0 │ de: very rough PoC of refcounted data │ -- operation 0df245508eb8 (2025-10-03 18:07:03) snapshot working copy │ src/consts.rs --- Rust │ No syntactic changes. │ │ src/error.rs --- 1/3 --- Rust │ 6 6 │ 7 //! Error objects and codes 7 //! Error objects and codes │ 8 8 │ 9 use std::fmt; 9 use serde::{de, ser}; │ 10 use std::io; 10 use std::error; │ 11 use std::error; 11 use std::fmt; ...skipping... ○ zxsrvopz hidden example@example.com 2025-10-02 19:31:08 43a4bc7d de: very rough PoC of refcounted data Cargo.toml --- TOML 10 edition = "2024" 11 12 [dependencies] 13 serde = { version = "1.0.104", features = ["rc"] } 14 byteorder = "1.3.2" 15 num-bigint = "0.4.0" 16 num-traits = "0.2.10" src/lib.rs --- Rust 68 68 //! 69 69 //! The minimum supported version of the toolchain is 1.41.1. 70 .. 71 .. #![cfg_attr(feature = "unstable", feature(test))] 72 70 73 71 pub use self::ser::{ 74 72 Serializer, /* The rest */</code></pre> There are two important lines here: @ zxsrvopz example@example.com 2025-10-03 18:07:03 master 69322ff0</code></pre> And ○ zxsrvopz hidden example@example.com 2025-10-02 19:31:08 43a4bc7d</code></pre> zxsrvopz</code> is the change ID (which is the same for both lines) while 43a4bc7d</code> and 69322ff0</code> are the unique commit IDs. In the above output I just searched for zx</code> to jump from the first commit to the second and saw that the second is what I wanted to restore to. So now I can jj edit 43a4bc7d</code> to jump back to that specific operation and I see the following output: ❯ : jj edit 43a4bc7d Working copy (@) now at: zxsrvopz?? 43a4bc7d de: very rough PoC of refcounted data Parent commit (@-) : xlzooroo 9b8d55a1 core: update quickcheck + rand and tests (tests failing) Added 0 files, modified 8 files, removed 0 files</code></pre> See the zxsrvopz??</code> change ID? If the above was appropriately highlighted you'd see it in red, but jj</code> now is telling me that I have a conflicted change by placing that text in red and showing the ??</code> after the change ID. If I run jj log</code> I see my history has branched and I have two changes with the same ID: ❯ : jj log @ zxsrvopz?? example@example.com 2025-10-02 19:31:08 43a4bc7d │ de: very rough PoC of refcounted data │ ○ zxsrvopz?? example@example.com 2025-10-03 18:07:03 master 69322ff0 ├─╯ de: very rough PoC of refcounted data ○ xlzooroo example@example.com 2025-10-02 17:00:07 git_head() 9b8d55a1 │ core: update quickcheck + rand and tests (tests failing)</code></pre> The change starting with @</code> is the revision I'm currently on while the other with a ○</code> is the forked revision. I can now resolve this conflict by simply abandoning the other change: > : jj abandon 69322ff0 Abandoned 1 commits: zxsrvopz?? 69322ff0 master | de: very rough PoC of refcounted data Deleted bookmarks: master ❯ : jj log @ zxsrvopz example@example.com 2025-10-02 19:31:08 43a4bc7d │ de: very rough PoC of refcounted data ○ xlzooroo example@example.com 2025-10-02 17:00:07 git_head() 9b8d55a1 │ core: update quickcheck + rand and tests (tests failing)</code></pre> And we're fixed! I've reproduced what happened here in a short shell session which will have better highlighting and more info (note: clicking will take you to asciinema.org</code>): </a>

Writing a PE Loader for the Xbox in 2024

2024-08-13T00:00:00+00:00

Emma (@carrot_c4k3</a>) is a good friend of mine. We met in 2007 from the Xbox 360 scene and have remained friends ever since. She recently participated in pwn2own in the Windows LPE category and ended up using a great bug for LPE.

The bug far exceeded the category though: this vulnerability was also a sandbox escape, i.e. it's in an NT syscall which is reachable from the UWP sandbox. A couple months ago she got a wild idea: why not try to port the exploit over to the Xbox One? (Modern Xboxes, not to be confused with the OG Xbox)

#</a > Brief Primer on the Xbox One's Security</h2>
Since I'll be talking about this in the context of the Xbox, it's worthwhile to spend a moment discussing the Xbox One's security model. There's a very great and in-depth overview of the Xbox One's security model on YouTube</a> presented by Tony Chen who is one of the folks who designed it. I highly recommend watching it if you're interested, but I'll do my best at giving a crash course:
┌────────────────────────────┐ ┌────────────────────────────┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ERA (GameOS) │ │ SystemOS │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └────────────────────────────┘ └─────┼─────────────┼┬───────┤ │ │ ││ VMBus │ │ │ │└───────┘ ┌──────────────┼─────────────────────────┼─────────────┼────────┐ │ │ ▼ ▼ │ │ │ HostOS ┌──────────┐ ┌───────────────┤ │ │ │ Synthetic│ │ VSPs/Normal │ │ └──────────────────▶│ Devices │ │ Hyper-V Stuff │ └──────────────────────────────────┴──────────┴─┴───────────────┘ ┌───────────────────────────────────────────────────────────────┐ │ │ │ Hypervisor │ │ │ │ │ └───────────────────────────────────────────────────────────────┘</code></pre>
This is a very, very simplified drawing of what you'd find on Microsoft's Hyper-V Architecture page</a>. The main thing I'm trying to highlight here is that there are 3 VMs with 3 different purposes:

HostOS, which acts very similar to your standard Hyper-V host.</li>
ERA OS (aka GameOS) which is where games run.</li>
SystemOS which is where applications run. Here you'll find the system shell and UWP applications from the Windows Store.</li> </ol>
Each VM is running a very slimmed down version of Windows based on Windows Core OS (WCOS). The Hyper-V architecture is mostly what you'd encounter on a normal PC but with some additional Xbox-specific VSPs/functionality.
Missing from the above diagram is the security processor (SP). The Xbox One's security processor should be the only thing on the Xbox which can reveal a title's plaintext on Xbox One. (Random fact: Microsoft's Pluton Processor</a> is based on learnings from the Xbox One's security processor)
The core idea behind all of this is to make piracy extremely difficult, if not impossible without breaking the SP. If you do hack the Xbox One, you can't do it online trivially because the SP will attest that the console's state is something unexpected.
#</a > OK, How Does This Relate to the PE loader?</h2>
Unrelated to her pwn2own entry, Emma found a vulnerability/feature in an application on the Xbox One marketplace called GameScript, which is an ImGui UI for messing with the Ape programming language</a>.
Through this vulnerability</a> Emma was able to read/write arbitrary memory and run shellcode. So we have arbitrary code execution in SystemOS, but now the problem: writing shellcode is a pain, so how can we run arbitrary executables easily?
We have the ability to read/write arbitrary memory and change page permissions which is enough to write a portable executable (PE/.exe) loader. Emma asked if I would write one since it would simplify the exploit development pipeline while she worked on porting her LPE exploit over and it'll be useful for homebrew later on too. Easy enough right?
Wrong.
#</a > Reinventing the Wheel</h2>
The specific technique of PE loading outlined here is referred to as "Reflective PE Loading". To me this sounds like some #redteam term I'd never heard before embarking on this project, and is not very descriptive in my opinion... but a "reflective PE loader" is simply some user-mode code that can load and execute a PE without going through the normal LoadLibrary()</code> / CreateProcess()</code> routines .
Avoiding LoadLibrary()</code> and CreateProcess()</code> is very important for us since those will check for code integrity and any code we write will not be properly signed.
I took a look at the work involved and decided I wanted to write my own loader for multiple reasons:
I despise dealing with C/C++ build systems. </li> Since I'm targeting Xbox Windows and not desktop Windows, I might encounter some problems and I know how to debug my own code better than someone else's. </li> On the Xbox we're required to use a PE loader for running unsigned executables until we eventually break code integrity. So we better know how it works and be able to load complex applications. </li> I don't give a shit about EDR evasion or any #redteam stuff like that. </li> We originally had some very, very strict size constraints that we found a workaround for, but we want to be able to control the loader size as much as possible. </li>It seemed simple enough at the time to just rewrite it in Rust, so I did. </li> </ol> For my project's base I combined two open-source Rust projects: b1tg/rust-windows-shellcode</a> which provided a great template for writing and building Windows shellcode in Rust.</li> Thoxy67/rspe</a> which provides a basic reflective loader.</li> </ul> rspe already got me most of the way there, but with a few caveats: It needed some cleanup (e.g. lots of unnecessary copies)</li> It did not support loading imports by ordinal</li> It did not support thread-local storage at all</li> It did not support command line arguments</li> It did not support environments with W^X mitigations</li> It did not work with shellcode-based programming in mind.</li> </ul> What is shellcode-based programming?</summary> On that last point above you might be wondering, "What is shellcode-based programming?" Well why don't I just give an example. Here's how a VirtualAlloc()</code> call in rspe</code> worked before: #[link(name = "kernel32")] extern "system" { pub fn VirtualAlloc( lpaddress: *const c_void, dwsize: usize, flallocationtype: VIRTUAL_ALLOCATION_TYPE, flprotect: PAGE_PROTECTION_FLAGS, ) -> *mut c_void; } // Allocate memory for the image let baseptr = VirtualAlloc( core::ptr::null_mut(), // lpAddress: A pointer to the starting address of the region to allocate. imagesize, // dwSize: The size of the region, in bytes. MEM_COMMIT, // flAllocationType: The type of memory allocation. PAGE_EXECUTE_READWRITE, // flProtect: The memory protection for the region of pages to be allocated. );</code></pre> And here's how this would look with shellcode-based programming: pub type VirtualAllocFn = unsafe extern "system" fn( lpAddress: *const c_void, dwSize: usize, flAllocationType: u32, flProtect: u32, ) -> PVOID; pub fn fetch_virtual_alloc(kernelbase_ptr: PVOID) -> VirtualAllocFn { // this is some macro that, using `kernelbase_ptr`, parses kernelbase's export table to find `VirtualAlloc` // and return its address. i.e. kind of a self-made version of `GetProcAddress` resolve_func!(kernelbase_ptr, "VirtualAlloc") } let VirtualAlloc = fetch_virtual_alloc(kernelbase_ptr); let baseptr = (VirtualAlloc)( preferred_load_addr, // lpAddress: A pointer to the starting address of the region to allocate. imagesize, // dwSize: The size of the region, in bytes. MEM_COMMIT, // flAllocationType: The type of memory allocation. PAGE_READWRITE, // flProtect: The memory protection for the region of pages to be allocated. );</code></pre> As you might have noticed, we're not linking against any libraries and calling those imports directly. Instead we're using indirect calls to functions whose addresses we manually resolved at runtime. All you need for shellcode development that isn't painful is to find kernelbase.dll</code> which can be done using the gs</code> register to grab the PEB: pub fn get_module_by_name(module_name: *const u16) -> Option<PVOID> { let peb: *mut PEB; unsafe { asm!( "mov {}, gs:[0x60]", out(reg) peb, ); let ldr = (*peb).Ldr; let module_list = &((*ldr).InLoadOrderModuleList); // The first entry of LDR_DATA_TABLE_ENTRY is a LIST_ENTRY, so transmuting this address // from LIST_ENTRY to LDR_DATA_TABLE_ENTRY is legal. let mut cur_module: *const LDR_DATA_TABLE_ENTRY = core::mem::transmute(module_list); // The list is doubly-linked, so eventually we will wrap back around to the head. let module_list_head = cur_module; loop { let cur_name = (*cur_module).BaseDllName.Buffer; if !cur_name.is_null() && icmp_raw_str_u16(module_name, cur_name) { return Some((*cur_module).BaseAddress); } let flink = (*cur_module).InLoadOrderModuleList.Flink; cur_module = flink as *const LDR_DATA_TABLE_ENTRY; if cur_module == module_list_head { // We wrapped the whole list and didn't find a result. return None; } } } }</code></pre> Then parse the PE's export table</a> to find GetModuleHandleA()</code>, and GetProcAddress()</code>. </div> </details> #</a > The Easy Parts</h2> Although it's been talked about before</a>, I'll give a brief overview of how a basic loader works: Parse the PE headers and VirtualAlloc()</code> some memory for the "cloned" PE with all the fixups applied. You'll try to VirtualAlloc()</code> at the PE's preferred load address, but if you don't get it fall back to a random address. This is your load address. From here you calculate the delta between the preferred and actual load address and this will be used for fixing relocations. (Note: copying the PE just to change its fields isn't strictly necessary but simplifies some things) </li> Iterate each PE section and copy it over to the newly VirtualAlloc</code>'d region</a>. The virtual addresses here are relative virtual addresses, so you just take each section's VirtualAddress, add it to the load address, and copy the section from its old location to the new address. </li> Fix section permissions</a>. For each section, look at its Characteristics</code> field and determine the correct permissions. VirtualProtect()</code> the section according to the permissions. </li> Fix imports</a>. For each import in the import table (IMAGE_DIRECTORY_ENTRY_IMPORT</code>), ensure the imported DLL is loaded. Then use the loaded DLL's handle with GetProcAddress()</code> to get the address of the function being imported. For each import in the table, write the real address in the import's thunk. Instead of GetProcAddress()</code> could also parse the module's exports and match things up, but I took the lazy way. </li> Fix relocations</a>. This basically involves walking the IMAGE_DIRECTORY_ENTRY_BASERELOC</code> directory and fixing each IMAGE_BASE_RELOCATION</code> such that you add the delta calculated in step 1 to the relocation's VirtualAddress</code> field. There's some nuance here where you need to only modify certain bits, etc. etc. but this is the basic idea. </li> Call the module's entrypoints</a>. </li> </ol> I learned through this experience that PEs can have multiple thread-local storage callbacks called before the actual module entrypoint. Calling these is fairly straightforward: let tls_directory = &ntheader_ref.OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_TLS as usize]; // Grab the TLS data from the PE we're loading let tls_data_addr = baseptr.offset(tls_directory.VirtualAddress as isize) as *mut IMAGE_TLS_DIRECTORY64; let tls_data: &mut IMAGE_TLS_DIRECTORY64 = unsafe { core::mem::transmute(tls_data_addr) }; let mut callbacks_addr = tls_data.AddressOfCallBacks as *const *const c_void; if !callbacks_addr.is_null() { let mut callback = unsafe { *callbacks_addr }; while !callback.is_null() { execute_tls_callback(baseptr, callback); callbacks_addr = callbacks_addr.add(1); callback = unsafe { *callbacks_addr }; } } unsafe fn execute_tls_callback(baseptr: *const c_void, entrypoint: *const c_void) { let func: ImageTlsCallbackFn = core::mem::transmute(entrypoint); func(baseptr, DLL_THREAD_ATTACH, ptr::null_mut()); }</code></pre> Executing the image entrypoint is pretty similar: let entrypoint = (baseptr as usize + (*(ntheader as *const IMAGE_NT_HEADERS64)) .OptionalHeader .AddressOfEntryPoint as usize) as *const c_void; // Create a new thread to execute the image execute_image(baseptr, entrypoint, context.fns.create_thread_fn); unsafe fn execute_image( dll_base: *const c_void, entrypoint: *const c_void, create_thread_fn: CreateThreadFn, ) { let func: extern "system" fn(*const c_void, u32, *const c_void) -> u32 = core::mem::transmute(entrypoint); func(dll_base, DLL_PROCESS_ATTACH, ptr::null()); }</code></pre>#</a > The Hard Parts</h2> There were some parts that really kicked my ass in figuring out, but in my opinion were very important for what I wanted in the PE loader. The exploit / PE loader must not cause the hijacked application to become unreliable. I don't want to be debugging crashes in some of the existing threads that broke simply because we're hijacking the address space. </li> We must be able to run complex applications. Since we're using this technique to bypass code integrity, this will be our main method of running arbitrary applications. </li> The application shouldn't know it's been reflectively loaded, or care. </li> </ol> #</a > Thread-Local Storage</h3> Related to #2, the absolute biggest challenge I faced was with applications that use thread-local storage (TLS). Having done all of my development in Rust, my test program that I was loading was also written in Rust. I kept crashing on int 29</code> instructions (RtlFailFast(code)</code>) shortly after executing the module's entrypoint. This was extremely painful to debug but eventually I figured out that I was failing after fetching data from TLS </a> </a> I was kind of confused because I didn't expect my application to use TLS, but apparently even the most basic "hello world" Rust program uses TLS: </a> It turns out that this is related to Rust's thread initialization code that sets some thread-locals for the current thread and thread ID: https://github.com/rust-lang/rust/blob/2e630267b2bce50af3258ce4817e377fa09c145b/library/std/src/thread/mod.rs#L694</a> So I came to realize that my original idea for how I was handling TLS data was completely flawed. Originally I was allocating new memory for my module's TLS, but didn't even realize it had some default state associated with it that I had to copy over. Simple fix right? diff --git a/crates/loader/src/lib.rs b/crates/loader/src/lib.rs index 97311d0..d66773d 100755 --- a/crates/loader/src/lib.rs +++ b/crates/loader/src/lib.rs @@ -180,34 +185,53 @@ unsafe fn reflective_loader_impl(context: LoaderContext) { .OptionalHeader .AddressOfEntryPoint as usize) as *const c_void; - let tls_directory = &ntheader_ref.OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_TLS]; + let tls_directory = + &ntheader_ref.OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_TLS as usize]; + + // Grab the TLS data from the PE we're loading + let tls_data_addr = + baseptr.offset(tls_directory.VirtualAddress as isize) as *mut IMAGE_TLS_DIRECTORY64; + + // TODO: Patch the module list + let tls_index = patch_module_list( + context.image_name, + baseptr, + imagesize, + context.fns.get_module_handle_fn, + tls_data_addr, + context.fns.virtual_protect, + entrypoint, + ); + if tls_directory.Size > 0 { // Grab the TLS data from the PE we're loading let tls_data_addr = baseptr.offset(tls_directory.VirtualAddress as isize) as *mut IMAGE_TLS_DIRECTORY64; - let tls_data: &IMAGE_TLS_DIRECTORY64 = unsafe { core::mem::transmute(tls_data_addr) }; + let tls_data: &mut IMAGE_TLS_DIRECTORY64 = unsafe { core::mem::transmute(tls_data_addr) }; // Grab the TLS start from the TEB let tls_start: *mut *mut c_void; unsafe { core::arch::asm!("mov {}, gs:[0x58]", out(reg) tls_start) } - let tls_index = unsafe { *(tls_data.AddressOfIndex as *const u32) }; - let tls_slot = tls_start.offset(tls_index as isize); let raw_data_size = tls_data.EndAddressOfRawData - tls_data.StartAddressOfRawData; - *tls_slot = (context.fns.virtual_alloc)( + let tls_data_addr = (context.fns.virtual_alloc)( ptr::null(), - raw_data_size as usize, + raw_data_size as usize, // + tls_data.SizeOfZeroFill as usize, MEM_COMMIT, PAGE_READWRITE, ); - // if !tls_start.is_null() { - // // Zero out this memory - // let tls_slots: &mut [u64] = unsafe { core::slice::from_raw_parts_mut(tls_start, 64) }; - // tls_slots.iter_mut().for_each(|slot| *slot = 0); - // } + core::ptr::copy_nonoverlapping( + tls_data.StartAddressOfRawData as *const _, + tls_data_addr, + raw_data_size as usize, + ); + + // Update the TLS index + core::ptr::write(tls_data.AddressOfIndex as *mut u32, tls_index); + *tls_slot = tls_data_addr; let mut callbacks_addr = tls_data.AddressOfCallBacks as *const *const c_void; if !callbacks_addr.is_null() {</code></pre> This code worked, but not for long. I obviously had no idea how TLS worked, and soon discovered that in a multi-threaded application I was again getting similar crashes because the TLS data was bad. Through much pain and debugging I ended up learning: Changing the TLS for your current thread is obviously not enough. New threads that spawn won't have the modifications I did above, so they'll have "default" TLS without my module included since the changes I did above are only reflected for the current thread. Duh. </li> TLS is allocated in slots for the current thread and each slot is a pointer to the TLS data. </li> Windows keeps a cache of TLS directories for each loaded module, which means you can't just pave over the hijacked module's TLS data with your new TLS data and things will "just work". You'll have to update the cache. </li> </ul> #</a > Fixing TLS Data</h3> In the above section I mentioned that Windows keeps a cache of TLS directories for each loaded module, and I think this is a critical reason why the reflective PE loaders I sampled didn't bother with TLS data (only one loader sampled seemed to support TLS data</a>). I really only discovered this by painfully debugging and figuring out the application only crashed when spawning new threads, that the crashes were relating to data in TLS, and figuring that something must be wrong with the TLS data. It finally clicked when I noticed that the ThreadLocalStoragePointer</code> for the crashing thread's TEB didn't match the spawning thread's... </a> </a> This is super obvious in hindsight! Each thread's TLS has to be unique, but I don't know... I thought the ThreadLocalStoragePointer</code> was a pointer to the default state TLS and the per-thread slots were in the TEB's TlsSlots</code> field? Anyways, I set a breakpoint at the thread initialization routine, LdrpInitializeThread</code>, and debugged it to see if there was anything that stood out for TLS initialization. Like magic, I eventually stepped into LdrpAllocateTls</code>: </a> The ReactOS source code</a> was of huge help here in figuring out what was going on, but essentially what happens when spawning a new thread is: If any of the currently loaded modules has TLS, allocate a ThreadLocalStoragePointer</code>.</li> The size of this memory block is sizeof(void*) * NUM_MODULES_WITH_TLS_DATA</code>.</li> Iterate some TlsLinks</code> list. This is a list of LDRP_TLS_DATA</code>:</li> </ol> typedef struct _LDRP_TLS_DATA { LIST_ENTRY TlsLinks; IMAGE_TLS_DIRECTORY TlsDirectory; } LDRP_TLS_DATA, *PLDRP_TLS_DATA;</code></pre> Calculate the size of the TLS data based on the TlsDirectory</code>, and copy its contents.</li> Put the pointer to the memory allocated in step 4 in the appropriate slot, recorded as TlsData->TlsDirectory.Characteristics</code>.</li> </ol> Now that I know the TLS data is cached, can't I just overwrite the TlsDirectory</code> data in this list from the host module with the data from the new module? Well yes... and no. The LDRP_TLS_DATA</code> is heap-allocated, so I'd have to scan the heap which would be pretty bug-prone. #</a > Janky Approach to Fixing TLS Data</h4> tl;dr: use a private ntdll</code> function that returns the cached TLS_ENTRY</code> from a LDR_DATA_TABLE_ENTRY*</code> to find the hijacked module's TLS data. Once found, overwrite the cached IMAGE_TLS_DIRECTORY</code> with the new module's. This has a big problem: if the program you're loading requires TLS, you must inject into a program with TLS. Otherwise you'll be replacing a random DLL's TLS data if you aren't careful. List Patching Details</summary> I popped ntdll.dll</code> into IDA to see what functions were using this LdrpTlsList</code> to see if maybe there was some other way I could grab the list's address. </a> I found that in Windows (but not ReactOS) is a function, "LdrpFindTlsList</code>", which will return a PTLS_ENTRY</code> (the actual name of the Windows data structure for ReactOS's LDRP_TLS_DATA</code>) given a PLDR_DATA_TABLE_ENTRY</code>. Ken Johnson even conviently provided the source code on his blog</a>. The PLDR_DATA_TABLE_ENTRY</code> can be found in the PEB which you can explore using the !peb</code> command in WinDbg. The complete code: /// Returns the Thread Environment Block (TEB) pub fn teb() -> *mut TEB { let mut teb: *mut TEB; unsafe { core::arch::asm!("mov {}, gs:[0x30]", out(reg) teb) } teb } pub unsafe fn patch_module_list( image_name: Option<&[u16]>, new_base_address: *mut c_void, module_size: usize, get_module_handle_fn: GetModuleHandleAFn, this_tls_data: *const IMAGE_TLS_DIRECTORY64, virtual_protect: VirtualProtectFn, entrypoint: *const c_void, ) -> u32 { let current_module = get_module_handle_fn(core::ptr::null()); let teb = teb(); let peb = (*teb).ProcessEnvironmentBlock; let ldr_data = (*peb).Ldr; let module_list_head = &mut (*ldr_data).InMemoryOrderModuleList as *mut LIST_ENTRY; let mut next = (*module_list_head).Flink; while next != module_list_head { // -1 because this is the second field in the LDR_DATA_TABLE_ENTRY struct. // the first one is also a LIST_ENTRY let module_info = (next.offset(-1)) as *mut LDR_DATA_TABLE_ENTRY; if (*module_info).DllBase == current_module { (*module_info).DllBase = new_base_address; // EntryPoint (*module_info).Reserved3[0] = entrypoint as *mut c_void; // SizeOfImage (*module_info).Reserved3[1] = module_size as *mut c_void; if !this_tls_data.is_null() { let ntdll_addr = get_module_handle_fn("ntdll.dll\0".as_ptr() as *const _); if let Some(ntdll_text) = get_module_section(ntdll_addr as *mut _, b".text") { for window in ntdll_text.windows(LDRP_FIND_TLS_ENTRY_SIGNATURE_BYTES.len()) { if window == LDRP_FIND_TLS_ENTRY_SIGNATURE_BYTES { // Get this window's pointer and move backwards to find the start of the fn let mut ptr = window.as_ptr(); loop { let behind = ptr.offset(-1); if *behind == 0xcc { break; } ptr = ptr.offset(-1); } let LdrpFindTlsEntry: LdrpFindTlSEntryFn = core::mem::transmute(ptr); let list_entry = LdrpFindTlsEntry(module_info); (*list_entry).TlsDirectory = *this_tls_data; } } } } break; } next = (*next).Flink; } // This stuff here is mostly unnecessary, but I did it anyways as a "just in case". // The idea is to overwrite the `IMAGE_TLS_DIRECTORY` of the hijacked module // to point at the new module's. // And to get the hijacked module's TLS index since that's the slot we'll be // hijacking for our new module. if !this_tls_data.is_null() { let dosheader = get_dos_header(current_module); let ntheader = get_nt_header(current_module, dosheader); #[cfg(target_arch = "x86_64")] let ntheader_ref: &mut IMAGE_NT_HEADERS64 = unsafe { core::mem::transmute(ntheader) }; #[cfg(target_arch = "x86")] let ntheader_ref: &mut IMAGE_NT_HEADERS32 = unsafe { core::mem::transmute(ntheader) }; let real_module_tls_entry = &mut ntheader_ref.OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_TLS as usize]; let real_module_tls_dir = current_module .offset(real_module_tls_entry.VirtualAddress as isize) as *mut IMAGE_TLS_DIRECTORY64; let mut old_perms = 0; virtual_protect( real_module_tls_dir as *mut _ as *const _, core::mem::size_of::<IMAGE_TLS_DIRECTORY64>(), PAGE_READWRITE, &mut old_perms, ); let idx = *((*real_module_tls_dir).AddressOfIndex as *const u32); *real_module_tls_dir = *this_tls_data; idx } else { 0 } }</code></pre> </div> </details> #</a > The Good Method</h4> Remember how I said only one loader sampled seemed to support TLS data</a>? This happens to be the same approach they took. Building on the above list patching method, I saw a different function called LdrpAllocateTlsEntry</code> referenced the LdrpTlsList</code> cache and is called by LdrpHandleTlsData</code>. The latter function is called when a new module is loaded and is responsible for setting up almost all of the state relating to a module's TLS. It has no sanity checks on whether or not the module's TLS data has already been handled. Which is awesome, and actually makes sense! Why sanity check if this function is only ever called once during real loader scenarios? We can abuse this by performing the following operations: Update the hijacked module's LDR_DATA_TABLE_ENTRY</code> (found via the PEB) to point to our new module's base address.</li> Release the hijacked module's TLS data (LdrpReleaseTlsEntry</code>)</li> Call LdrpHandleTlsData</code> with the hijacked module to force the new TLS data to be loaded.</li> </ol> This also solves all of the problems we had with both prior methods! We can inject into any process and not just processes that have TLS data</li> According to the Ken Johnson code</a> this function updates the TLS info in the PEB (or maybe some kernel data?)</li> And according to the Ken Johnson code updates other threads</li> Is less code than both other solutions</li> Doesn't require me to manually update the new module's TLS index</li> </ul> const LDRP_RELEASE_TLS_ENTRY_SIGNATURE_BYTES: [u8; 7] = [0x83, 0xE1, 0x07, 0x48, 0xC1, 0xEA, 0x03]; const LDRP_HANDLE_TLS_DATA_SIGNATURE_BYTES: [u8; 9] = [0xBA, 0x23, 0x00, 0x00, 0x00, 0x48, 0x83, 0xC9, 0xFF]; // Function signature/type alias for LdrpReleaseTlsEntry type LdrpReleaseTlsEntryFn = unsafe extern "system" fn(entry: *mut LDR_DATA_TABLE_ENTRY, unk: *mut c_void) -> NTSTATUS; // Function signature/type alias for LdrpHandleTlsData type LdrpHandleTlsDataFn = unsafe extern "system" fn(entry: *mut LDR_DATA_TABLE_ENTRY); /// Returns the Thread Environment Block (TEB) pub fn teb() -> *mut TEB { let mut teb: *mut TEB; unsafe { core::arch::asm!("mov {}, gs:[0x30]", out(reg) teb) } teb } /// Patches the module list to change the hijacked module's DLL base and entrypoint. /// /// TODO: Patch image name. /// /// This is useful to ensure that a program that depends on `GetModuleHandle*` /// doesn't fail simply because its module is not found pub unsafe fn patch_ldr_data( new_base_address: *mut c_void, module_size: usize, get_module_handle_fn: GetModuleHandleAFn, this_tls_data: *const IMAGE_TLS_DIRECTORY64, entrypoint: *const c_void, ) { let current_module = get_module_handle_fn(core::ptr::null()); let teb = teb(); let peb = (*teb).ProcessEnvironmentBlock; let ldr_data = (*peb).Ldr; let module_list_head = &mut (*ldr_data).InMemoryOrderModuleList as *mut LIST_ENTRY; let mut next = (*module_list_head).Flink; while next != module_list_head { // -1 because this is the second field in the LDR_DATA_TABLE_ENTRY struct. // the first one is also a LIST_ENTRY let module_info = (next.offset(-1)) as *mut LDR_DATA_TABLE_ENTRY; if (*module_info).DllBase != current_module { next = (*next).Flink; continue; } (*module_info).DllBase = new_base_address; // EntryPoint (*module_info).Reserved3[0] = entrypoint as *mut c_void; // SizeOfImage (*module_info).Reserved3[1] = module_size as *mut c_void; if this_tls_data.is_null() { break; } let ntdll_addr = get_module_handle_fn("ntdll.dll\0".as_ptr() as *const _); let ntdll_text = get_module_section(ntdll_addr as *mut _, b".text"); if ntdll_text.is_none() { break; } let ntdll_text = ntdll_text.unwrap(); // Get the TLS entry for the current module and remove it from the list if let Some(window) = ntdll_text .windows(LDRP_RELEASE_TLS_ENTRY_SIGNATURE_BYTES.len()) .find(|&window| window == LDRP_RELEASE_TLS_ENTRY_SIGNATURE_BYTES) { // Get this window's pointer. It will land us in the middle of this function though let mut ptr = window.as_ptr(); // Walk backwards until we find the prologue. Pray this function retains padding loop { if *ptr.offset(-1) == 0xcc && *ptr.offset(-2) == 0xcc { break; } ptr = ptr.offset(-1); } #[allow(non_snake_case)] let LdrpReleaseTlsEntry: LdrpReleaseTlsEntryFn = core::mem::transmute(ptr); LdrpReleaseTlsEntry(module_info, core::ptr::null_mut()); } if let Some(window) = ntdll_text .windows(LDRP_HANDLE_TLS_DATA_SIGNATURE_BYTES.len()) .find(|&window| window == LDRP_HANDLE_TLS_DATA_SIGNATURE_BYTES) { // Get this window's pointer. It will land us in the middle of this function though let mut ptr = window.as_ptr(); // Walk backwards until we find the prologue. Pray this function retains padding loop { if *ptr.offset(-1) == 0xcc && *ptr.offset(-2) == 0xcc { break; } ptr = ptr.offset(-1); } #[allow(non_snake_case)] let LdrpHandleTlsData: LdrpHandleTlsDataFn = core::mem::transmute(ptr); LdrpHandleTlsData(module_info); } break; } }</code></pre>#</a > Patching Command-Line Args</h3> This has been done by other PE loaders, but I wanted to call this out as well: while the PEB contains the image name and process arugments, so does kernelbase.dll</code>! Why? For GetCommandLineW</code> and GetCommandLineA</code> of course. This one wasn't too bad to patch so long as you want to rely on the fact that the UNICODE_STRING</code> structure for the PEB and in kernelbase.dll</code> share the same backing buffer (i.e. the latter is a shallow copy of the former). That also doesn't account for the ANSI_STRING</code> variant... but 🤷‍♂️ tl;dr of the following code: we scan the global memory of kernelbase.dll</code> looking for the previously mentioned UNICODE_STRING</code> buffer pointer we obtained from the PEB then, once found, update its pointer and length to match our new pointer and length. pub unsafe fn patch_cli_args(args: Option<&[u16]>, kernelbase_ptr: *mut u8) { if let Some(args) = args { let peb = (*teb()).ProcessEnvironmentBlock; // This buffer pointer should match the cached UNICODE_STRING in kernelbase let buffer = (*(*peb).ProcessParameters).CommandLine.Buffer; // Search this pointer in kernel32's .data section if let Some(kernelbase_data) = get_module_section(kernelbase_ptr, b".data") { let ptr = kernelbase_data.as_mut_ptr(); let len = kernelbase_data.len() / 2; // Do not have two mutable references to the same memory range let data_as_wordsize = core::slice::from_raw_parts(ptr as *const usize, len); if let Some(found) = data_as_wordsize .iter() .position(|ptr| *ptr == buffer as usize) { // We originally found this while scanning usize-sized data, so we have to translate // this to a byte index let found_buffer_byte_pos = found * core::mem::size_of::<usize>(); // Get the start of the unicode string let unicode_str_start = found_buffer_byte_pos - core::mem::offset_of!(UNICODE_STRING, Buffer); let unicode_str = core::mem::transmute::<_, &mut UNICODE_STRING>( ptr.offset(unicode_str_start as isize), ); let args_byte_len = args.len() * core::mem::size_of::<u16>(); unicode_str.Buffer = args.as_ptr() as *mut _; unicode_str.Length = args_byte_len as u16; unicode_str.MaximumLength = args_byte_len as u16; } } } }</code></pre>#</a > Preventing Hijacked Application Crashes</h3> I thought a great idea to prevent the hijacked application from crashing by suspending all of its threads. I was surprised to learn that not only was this fairly easy to do on Windows, it was even easier to accidentally do this from a non-admin session for all other Medium-IL processes! </a> Yeah, don't call CreateToolhelp32Snapshot()</code> incorrectly. The Windows examples were fairly straightforward but on Xbox the code crashed. And that's because the kernel32_ptr</code> here actually needs to be a pointer to kernel32legacy.dll</code> since on Xbox kernel32.dll</code> doesn't exist. That took me a while to figure out and hunt down and double-check where the functions got relocated to. Here is the code I eventually came up with: pub unsafe fn suspend_threads(kernel32_ptr: PVOID, kernelbase_ptr: PVOID) { // kernel32legacy.dll on xbox let CreateToolhelp32Snapshot = fetch_create_tool_help32(kernel32_ptr); let Thread32Next = fetch_thread_32_next(kernel32_ptr); let Thread32First = fetch_thread_32_first(kernel32_ptr); // kernelbase.dll on xbox let GetCurrentThreadId = fetch_get_current_thread_id(kernelbase_ptr); let GetCurrentProcessId = fetch_get_current_process_id(kernelbase_ptr); let OpenThread = fetch_open_thread(kernelbase_ptr); let SuspendThread = fetch_suspend_thread(kernelbase_ptr); let CloseHandle = fetch_close_handle(kernelbase_ptr); let pid = GetCurrentProcessId(); // Suspend all other threads except this one let h = CreateToolhelp32Snapshot(TH32CS_SNAPTHREAD, pid); let current_thread = GetCurrentThreadId(); let mut te: THREADENTRY32 = core::mem::zeroed(); te.dwSize = core::mem::size_of_val(&te) as u32; if Thread32First(h, &mut te as *mut _) != 0 { loop { if te.dwSize as usize >= offset_of!(THREADENTRY32, th32OwnerProcessID) + core::mem::size_of_val(&te.th32OwnerProcessID) && te.th32OwnerProcessID == pid && current_thread != te.th32ThreadID { let thread_handle = OpenThread(THREAD_SUSPEND_RESUME, false, te.th32ThreadID); SuspendThread(thread_handle); } if Thread32Next(h, &mut te as *mut _) == 0 { break; } te.dwSize = core::mem::size_of_val(&te) as u32; } } CloseHandle(h); }</code></pre>#</a > TODO</h2> There are still some remaining items for the loader: Ensure that GetModuleHandle(NULL)</code> (handle to self) works correctly.</li> Maybe load .NET binaries? We already have a technique for launching .NET code, but having an all-in-one solution might be nice.</li> Maybe make this a generic crate?</li> </ul> #</a > fin</h2> This was a fun exercise that taught me a lot about how Windows binaries are loaded. I'd like to thank carrot_c4k3, tuxuser, and 0e9ca321209eca529d6988c276e4e4ed for their help/support. With this work, we're now able to do cool things on Xbox! For example, using the PE loader we can launch the main GameScript exploit, launch Emma's Windows exploit binary to elevate privileges, spawn a new process as suspended, inject our shellcode/PE loader, and execute a custom SSH/SFTP daemon which uses tokio for async. I think I accomplished my goal of loading complex applications :) </video> </div> The top terminal session is the payload server running on my PC, while the bottom netcat session is the output from the exploit and SSH daemon running on my Xbox. tuxuser even managed to get toasts working! </a>

On Dependency Usage in Rust

2024-06-03T00:00:00+00:00

#</a > Context</h2>
A couple months back I read "C isn't a Hangover; Rust isn't a Hangover Cure"</a> (original Medium link</a>) by John Viega. I responded to the post</a> already on Twitter (sometimes known as X) and in hindsight should have just written a blog post to begin with since the platform is so terrible for longform comment.
What follows is hopefully a more organized, digestable, and better response to John's post than what I wrote on Twitter. If you haven't read his post, I recommend giving it a read for full context instead of reading just what I've decided to directly respond to.
John's post goes into some concerns about using Rust and if Rust is really the right choice over something GC'd, and covers a few angles including whether memory safety really matters for you, what language best fits your team, and something that came up multiple times is dependency usage.
I disagree with some of the arguments John made surrounding dependencies and I frequently hear similar sentiments said by crowds who are anti-Rust. The idea that a program is less desirable or less secure because it has more dependencies. I find these arguments to be an easy jab lacking substance, and wanted to take an opportunity to challenge them.
My big complaint with John's points is that he spells out negatives but ignores most positives, instead telling the reader to figure those out for themselves.
It's important to understand who I am for a frame of reference: my background is in security and I first learned to program in C# making tools for Xbox 360 modding. That involved some reverse engineering and learning C++ for writing trainers/tools/cheats or whatever stuff would have to run on the console. Following my Xbox hacking in my teens, I professionally did web dev for some years (PHP), then web security, then hypervisor security at Microsoft, and now native code security focused on mobile applications. With the exception of my ~3 years in professional web dev, my title has always had "Security Engineer" in it -- not "Software Engineer". I sure as shit write a lot of code though.
I was entirely self-taught in programming at age 13 mostly through following online tutorials, and got a bachelor's degree in computer science.
My language of choice for about 8 years now has been Rust. This was settled upon after trying Go, D, Python, and some other languages inbetween. I am nobody in C/C++ circles and I'm a nobody in Rust circles, but I do pay close attention to the Rust community because I love the language.

#</a > Why are dependencies seen as insecure?</h2>
John talks about this at good length and it's worth reading his thoughts. If you're too lazy to do so, I think it can be sufficiently summarized as:

"Code review is a lot harder to do well than writing code"</li>
Dependencies can come from anyone and can generally be contributed to by anyone. Therefore the more dependencies you have, the larger your implicit circle of trust, and any break in that circle breaks your security. They become a single point of failure.</li>
You trust the code you write, and you know the code you write.</li> </ol>
On #1, I don't agree -- at least not broadly. It's probably true for small bits of code that you have the technical know-how to write yourself, but in memory-safe langauges what's the worst thing you can miss in a code review of something that's not technically complicated? Probably minor bugs that would cause a DoS. So you bring in a dependency that you didn't audit super closely and now you have a DoS in your application. Depends on your threat model how important this is to you, and whether that impacts your mental quality rating of the dependency.
On #2 I mostly agree. You are opening up your circle of trust, but done right you can protect yourself. The npm left-pad incident</a> is a prime example of what can go wrong from even a non-malicious dependency failure.
In the left-pad incident a package named "left-pad" was removed from the npm registry causing widespread build failures for almost every node.js application. The broad usage of this dependency shocked people since it was less than 50 lines of code and could be written by anyone.
If you're pulling in a dependency that's already compromised then you're a bit late, but for avoiding future compromise you can:
Use a package service that does not delete yanked dependencies. This should only be possible in extremely rare scenarios where e.g. someone's private information was exposed. crates.io, Rust's default package source, does not permit deletion</a>.</li>
Commit lockfiles to ensure that builds are reproducible, the same dependencies are pulled every time, and a future compromise of a dependency doesn't impact you unless you explicitly update. This is the default behavior for Cargo and npm. The lockfile will also ensure that the dependency's location is preserved, preventing dependency substitution attacks and should ensure that with the first point above that even a yanked dependency can still be resolved.</li>
Vendor dependencies so that you have a true complete snapshot of things without relying on 3rd parties. This weighs a lot more and is harder to manage over time but is an immediate solution to both of the above points.</li> </ul>
On #3, sure. This is reasonable, but there are costs to writing that code that I'll cover later on.
#</a > Just because C/C++ users suffer doesn't mean everyone else has to</h2>
I'm going to quote a few of John's paragraphs for full context and then dissect some of them one-by-one:

Rust makes it easy to pull in outside dependencies, and much like in the JavaScript ecosystem, it seems to have encouraged lots of tiny dependencies. That makes it a lot harder to monitor and manage the problem.
But Rust’s situation is even worse than in most languages, in that core Rust libraries (major libraries officially maintained by the Rust project) make heavy use of third party dependencies. The project needs to take ownership and provide oversight for their libraries.
To me, this has long been one of the biggest risks in software. I can write C code that is reasonably defensive, but I have a hard time trusting any single dependency I use, never mind scaling that out.
Properly securing your dependency supply chain is a much harder problem than writing safe C code. Personally, I only pull in dependencies beyond standard libraries if the work I’d have to do in order to credibly replace the functionality is so great that, if I didn’t bring in a dependency, I would choose not to do the work.
C is a lot better than Rust in this regard, but it’s not particularly great. Partially, that’s because the C standard libraries (which I am always willing to use; the core language implementation and runtime is a given) are not at all extensive. People who write a lot of C end up building things themselves once and keeping them around and adapting them for decades, including basic data structures like hash tables. </blockquote>
First of all, I strongly disagree with the sentiment that securing your dependency supply chain is harder than writing safe C/C++ code.
You have to be at least a moderately advanced user in C++/core memory safety ideas to come to the realization that modifying a container while iterating it with iterators is a bad idea, or that there are subtly different ways to zero-initialize a structure that result in subtly different ways of it being zero-initialized (which may or may not include its padding), or that some types of pointer arithmetic/comparisons are undefined behavior</a>.
You don't need to be an advanced programmer to do a short sniff test to see if a dependency you're bringing in to your application looks fairly widely used and trusted by a community. Sure, the XZ backdoor is an extreme example of even experts who were members of the project missed something snuck in over time, but this is not what we're talking about here.

Rust makes it easy to pull in outside dependencies </blockquote>
Honestly? Thank god.
While C/C++ applications generally require fewer dependencies, most of the time you're relying on the project maintainer to provide you with a list of those dependencies and how to install them with your platform's preferred package manager.
Something that I always find myself saying when try to build a C/C++ application from source is "shit I'm missing a header" and general complaints about the build tools themselves. And of course the build/configuration tool tells you what you lack but doesn't tell you what you need to install because that's not its responsibility. The build tool may support building on your favorite OS, but it doesn't know how to install packages on that OS or even what the package is.
So you're left with a terrible error message that can leave you wondering "Do I need to install lib-dev-whatever2 or just lib-whatever2? Is this even available via my OS's package manager?"
Don't forget all of the dependencies you need to install just to install the dependency too: pkgconf, autoconf, autotools, ninja, cmake, whatever, and any other libraries this single dependency may rely on. Kicking the problem into a Dockerfile is also not a good substitution for a quality build tool.
The developer experience surrounding dependencies in C/C++ is so awful that you just default to not using any at all. Or you bring in a "header-only library" that makes integration easy because bringing in multiple external source/header files makes people want to turn off their computer and consider another career.

[Easy dependency usage] seems to have encouraged lots of tiny dependencies. That makes it a lot harder to monitor and manage the problem. </blockquote>
I disagree that the dependency story becomes harder to manage. There are multiple tools to monitor and manage your dependency usage in Rust:

cargo-tree</a> can tell you your dependency tree</li>
cargo-geiger</a> can tell you if any of your dependencies in the graph use `unsafe{}</code></li>`
`cargo-acl</a> can tell you which crates use unsafe{}</code>, run build scripts to see if any use network/filesystem, and provides you with API usage information to see if a crate is doing things unexpected. It can even sandbox the build tools.</li>`
cargo-audit</a> can tell you if any of your crates are affected by a known security vulnerability and fix the used package version automatically.</li> </ul> These are all possible because Rust's tooling ecosystem is so good. These are not things that are run by default so the argument is a bit weaker, but the fact that they exist means you have the option of using them if you want to. For example, I've seen many CI pipelines that use cargo-audit</code> to ensure vulnerable crates aren't being used. People who write a lot of C end up building things themselves once and keeping them around and adapting them for decades, including basic data structures like hash tables. </blockquote> I see this as a bad thing. You're probably going to write bugs and it's going to be hard to fix affected applications. No proper version tracking or update mechanism means that depending on how you use and manage this ad-hoc dependency, tracking where it's used and patching affected programs might be difficult. A hash table is also not necessarily a "basic" data structure but I would definitely consider it a common data structure. Common data structures and algorithms can still have bugs, and there are many examples of this (and if it's so common why isn't it in the stdlib?). No intent here to shame these folks, but just some examples: smallvec</a>, libwebp's Huffman tree decoding</a>, and glibc's qsort()</code></a>. (I'm aware that glibc and libwebp would typically be installed using your distro's package manager but that's besides the point.) So why are we shooting ourselves in the foot by making it difficult to track and manage our dependencies for C/C++, including even our own first-party dependencies? @Lucretiel summarized this same sentiment fairly well on Twitter</a>: Quick reminder that I C doesn't have a culture of minimal dependencies because of some kind of ingrained strong principles in its community, C has a culture of minimal dependencies because adding a dependency in C is a pain in the fucking ass. Rust and Node.js have smaller projects and deeper dependency trees than C++ or Python for literally no other reason than the fact that the former languages make it very easy to create, publish, distribute, and declare dependencies. This is systemic incentives 101. </blockquote> #</a > Rust isn't as "batteries included" as other languages</h2> One point John makes is: Languages like Go and Python that have extensive standard libraries that the language maintainers take responsibility for are actually the best case scenario in my opinion. Yes, more people touch the code, but the DIY economics are often the wrong choice, and having organizations willing to both be accountable, and provide an environment where people can focus on minimizing dependencies if they feel its important, is a good thing. ... Generally, I think Rust (and pretty much any programming language) would be served well to take ownership of their standard libraries. Pull in all the dependencies, and be willing to take ownership. </blockquote> I agree with John that having batteries included simplifies things for both new and established users, but I don't think we should be so quick to add more batteries to the collection without sufficient testing. My understanding is Rust has learned from the mistakes of other languages and explicitly tries not to include things in the standard library that the Rust core team believes don't quite fit, including for reasons of figuring out the API. You simply have more freedom with packages: they're semver-versioned and you can break compat with an appropriate version bump. You don't necessarily know the warts of an API until you start to really use it widely. Take for example the rand</code> crate</a>. There is no random number generator in the Rust standard library, and rand</code> is the de facto standard crate for this task. That's a bit odd, no? Random number generation is fairly common and one would think it's in the standard library. There's even a tracking issue for adding one: #27703</a> (and #36999</a>). While I agree in principle, putting something into the standard library mostly means that the APIs for it are immutable. You know what's changed in fairly minor but meaningful ways since those issues were closed? The rand</code> crate's APIs. If these had been brought into the standard library as-is we'd be mostly forever stuck with certain warts like Rng::gen_range()</code></a> accepting 2 args (low, high) instead of the more-natural Range</code> (using low..=high</code> syntax). Rust itself is still a growing and changing language as well, and it may not make sense to land on an API that would be better once language improvements land. Good luck changing a stabilized API without breaking compat. #</a > Package management in other languages also suck</h2> Yes, Python has become so popular, that plenty of people use outside dependencies, and there are several popular package managers. However, it’s still in a vastly better place from a supply chain perspective than JavaScript, which has become famous among developers for hidden dependencies on trivially small packages. </blockquote> There's no way to really sugarcoat this, but Python and Go package management really fucking sucked (Python still sucks, but some semi-recent tools are making it suck less). So Python has monolithic dependencies... but why? Because the tooling and uploading dependencies</a> is high-friction^{^{(I've never uploaded -- maybe it's easier than I think). And we're supposed to praise this? In what world is Python + pip "in a vastly better place from a supply chain perspective than JavaScript" because of this fact either?}} pip dependencies are by default global which causes conflicts with other Python applications, forcing you to use virtual environments. Note: /u/encyclopedist on Reddit</a> pointed out that this has recently changed with PEP 668</a>.</li> If pip hits a version conflict within your own project's package graph you're in for a headache</li> Packages with native dependencies are a mystery to basically everyone except the package author. Or is this just me?</li> There's no strong lockfile containing metadata sufficient for guaranteeing the bits someone installing a project's dependencies for the first time match the bits when the lockfile was generated (i.e. package hashes).</li> </ul> pip in my experience has been so frustrating to use for dependency management that it inspires me to just simply not use dependencies to begin with. Yes there are tools that make this easier, but they are not defaults or even agreed upon by the community. And before Go had its package management renaissance</a> does anyone remember what it looked like to use dependencies in Go? You imported a library like this in your code: import ( "github.com/codegangsta/cli" )</code></pre> You use go get</code> to download the dependencies to your local machine, and built the application. There were no lock files, no versions, nothing. The latest version of the source code was grabbed and used until you updated it which may have had breaking changes. The community had to resort to package proxies to version packages. Today it's pretty insane to think about letting a 3rd party man-in-the-middle your packages and deliver it to you with no integrity checks just to work around warts in the tools. And what about other languages like C#? NuGet, .NET's package manager, was also terrible. I don't know how it is today, but around ~2017 while working at Microsoft I discovered that NuGet had a "feature" where the client would reach out to all of your package feeds in parallel to fetch a package and whichever responded first won. I can't find the issue for it on GitHub, but someone had reported this behavior and it was considered "by-design". Even still when I presented the problem to the NuGet team internally, they did not see it as a vulnerability. The obvious problem here was that we were leaking our internal package names to external package feeds and a name collision could result in the wrong package being used (this was before dependency substitution/confusion attacks were widely known). NuGet also had no lock files, no integrity checks, and conveniently provides install/build scripts and usually what you're receiving is prebuilt binaries. Code integrity isn't verified and the only thing that would have prevented you from using a completely different binary was Strong naming</a> which is not a security boundary. In fact, I've seen a lot of projects publish their strong name key. Rust was fortunately blessed from the beginning (pre-1.0, 2014!</a>) with people who knew how to build a package manager. Cargo is not perfect, but it works pretty damn well for the majority of Rust users. It is because the package management story in Rust is so good compared to other languages that the standard library doesn't need to be as feature-complete. Shipping with a fantastic package manager in 1.0 allowed the community package ecosystem to explode without having to pause and shift towards better or different solutions (NuGet's change to JSON-based projects</a>, Go's shift away from go get</code> using git-based imports to Go modules, and many different Python package managers like poetry, rye, pipenv). Rust developers are not bleeding from using the tools they depend on and it's absurd to me that this is considered a weakness. #</a > What about dependency explosion?</h2> Here is an example of an application I'm working on that reads files in a custom filesystem specific to the Xbox 360 (known as XContent / STFS). There's crypto involved for signing the header and verifying file data and conceptually this single file contains many others similar to a tarball or zip file. It's a CLI application with the following dependencies in its Cargo.toml</code> file: [dependencies] # For mmaping the input file memmap2 = "0.9" # Parsing arguments clap = { version = "4.5.4", features = ["derive"] } # Easy error handling anyhow = "1.0" # Data serialization serde = { version = "1.0" } # Reading the input file's filesystem stfs = { version = "0.1", path = "../stfs" } # Also for reading the input file's filesystem xcontent = { path = "../xcontent" } # Date/time operations chrono = "0.4.38" # Converting data to/from hexadecimal hex = "0.4.3" # Converting file sizes to something human-readable humansize = "2.1.3" # Serializing data to JSON serde_json = "1.0"</code></pre> According to cargo deps-list</code></a> this results in 102 dependencies from the 10 direct dependencies I specified. cargo depgraph</code></a> produced this graph: </a> Click for a larger image #</a > The sniff test</h3> I've mentioned my "quick checks" or "sniff test" a couple times in this blog post, so it's worth calling out what it is. I 100% did not audit all 102 of these dependencies in the above graph, but for each of the 10 I directly brought in to my application I looked at the author to see if I knew of them, looked at their project setup, and decided their goals align with mine which led me to using the crate. I've passed on crates that to me looked like someone not necessarily intending for others to use their work, or simply did not pass my vibe check. Here is what my personal flow looks like: Do a search for the topic I'm interested in</li> </ol> </a> Check the crates.io page</li> </ol> </a> ✅ The crate has good examples and information. I don't recognize the author, but that's not terribly uncommon. Check the crate's stats to get an idea of how widely used it is</li> </ol> </a> ✅ Tons of usage. These numbers can be gamed, but probably not to this level. Check the versions.</li> </ol> </a> ✅ 9 versions since its original release 8 years ago. The last release was 3 years ago. The author isn't changing stuff all the time which is good as I don't expect a hex crate to have heavy code churn. Check who is using this crate to see if I recognize any of them</li> </ol> </a> ✅ The hex crate has over 4,000 other crates depending on it and I recognize all of the top 5 biggest users. Check the repo</li> </ol> </a> ✅ This is not a great example but the hex crate has some stars, the active development about matches the crates.io page (keep in mind the repository doesn't have to match what's uploaded to crates.io!), and the project looks decently put together. There's also no build.rs script that I need to check out. The crate passes all of my standard checks! I feel comfortable pulling the crate into my repository #</a > Economic factors</h3> Something John mentions multiple times is weighing "economic factors" when considering what language or dependencies to use. Avoid unnecessary dependencies. I will leave ‘unnecessary’ vaguely defined here; you need to be educated and judge all the economic factors. But note that, there are often other benefits to fewer dependencies, from shorter build times to less surface to test, to less risk from API changes or bugs from downstream dependencies. </blockquote> Are all of these 10 crates I used above strictly necessarily? No. I could get away with writing my own hex converter, human-readable size converter, command-line argument parser, drop mmap support, drop support for chrono date/time, and rewrite to use standard Result<T, E></code> instead of using anyhow</code>. This is what such a Cargo.toml</code> would look like: [dependencies] # Data serialization serde = { version = "1.0" } # Reading the input file's filesystem stfs = { version = "0.1", path = "../stfs" } # Also for reading the input file's filesystem xcontent = { path = "../xcontent" } # Serializing data to JSON serde_json = "1.0"</code></pre> But you know what I get from splurging on 6 extra deps? I can just write my fucking code. That's the biggest economic factor I care about. I don't have to worry about making my argument parser print out help and keeping its flags and info up-to-date and manually pretty. I don't have to leave the user with a shitty DateTime</code> field because I can't write a good one for them since my app doesn't revolve around dates. I don't have to write boilerplate for bubbling up errors. The overall quality of the application and my dev experience is improved. hex</code> and humansize</code> are arguably my application's left-pad</code></a>. Converting to/from hex is not terribly complex and converting a number of bytes to the best unit of KB/MB/GB is extremely easy. In fact, I'm pretty sure I originally wrote it for this project and then removed it. These dependencies do one task that's simple enough for me to write but I didn't. Why? Because each handles some edge cases that may matter for me, and I'm not wanting to spend 30m of my time writing something that's not core to my application when someone already wrote the code and did it better than I would in those 30m. Instead I took 3 minutes to search around to find the crate, ensure it fit my needs and to sanity check it looked kinda legit, and then used it in my application. I got. Shit. Done. I will say that there have been times where I've compiled something and thought, "Holy shit 500+ dependencies?" But to me this isn't a signal of it's security but rather bloat and complexity. I have to think about everything the application does, its complexity, and consider if it's just bloated for no reason or if there is good reason for having so many dependencies. This can impact my judgement on the application's quality and how likely I am to use really use the tool. @Lucretiel said something else recently on Twitter</a> said something that loosely fits into this topic: It’s a good thing we’re keeping our dependency count low, I think to myself, as I read about how my UI framework also provides threads, networking utilities, data structures, floating point math, D-Bus, cryptographic utilities, geographic utilities, and a Bluetooth implementation </blockquote> Bloat is everywhere. You just need to know how to look for it. #</a > Circle of trust</h3> Anyway, the more dependencies you have, the larger your circle of implicit trust is, the larger your attack surface is, and the more supply chain risk you’re taking. </blockquote> I'd rather assume an author of a crate that looks like it provides what I need has non-malicious intent than the other way around. Maybe that perspective will change if I ever get burned and my laptop gets ransomwared because I missed a build.rs</code> file or a proc macro that does sketchy things. But consider for a second: do you consider your OS as part of your circle of trust? It's unlikely you'll ever get backdoored by your OS, but bugs are certainly present and depending on your threat model a vulnerable OS means a problem for you. Do you know how much attack surface there is with say image parsing on iOS/macOS? You can choose to not bring in libjpeg/libpng/libwebp and just use ImageIO (which is used by UIKit/CoreGraphics). Easy! Except you now have at least 30 different image formats on your attack surface that you didn't know about. And there's no way to turn them off. And now you're stuck ensuring that the image you're parsing is a trusted image format. You might be screaming, "But Apple is trusted! And Apple publishes updates!" Ok? Did you validate how many of those updates are backported to major iOS versions used by your users? Did you audit Apple's closed-source lib and discover this attack surface and then weigh the economic costs of not using it? I mean, ImageIO is mostly just a wrapper around libjpeg and libpng, so why not just use them directly? Likely answer: Because it's convenient and you might not care about the problems I am describing because you aren't some security nerd. #</a > Closing thoughts</h2> I would like to thank John for sharing his thoughts and perspective. I do outright agree with some his points: Avoiding unnecessary dependencies may be better long-term for better compile times and less potential problems down the road. This is a tradeoff worth considering, but to me is a minor point.</li> The bigger your dependency graph, the bigger your single-points of failure. I see this as a tradeoff for rapid development.</li> Understand what makes sense for your team and your own threat model.</li> </ul> But I disagree with some of the foundational arguments like: C’s advantage in terms of lack of dependencies (which can come with a lower attack surface in general) is large, but still doesn’t make it the right economic choice in the first place. It might still be wiser to choose Rust when all economic factors are considered, but the security argument is just not one I find compelling enough. </blockquote> The security risk of dependency use is simply not one I find compelling enough to select C over Rust, and certainly is not scarier than a buffer overflow. C lacking first-class support for dependencies should be considered a strong disadvantage since you can't even get good support for 1st-party dependencies. The benefits Rust provides as a language are already enough for a lot of people to select it over C -- myself included. A stellar default package manager and build tool makes it all the better to use. In my opinion "dependency usage" should be a minor footnote (and John explicitly says to weigh these kinds of factors yourself). Additionally, I'd argue that a critical mem safety issue is statistically way more likely to happen and can have critical impact even with modern mitigations. Some of the memory safety bugs that we're finding are old enough to drink in the US, showing that they can be very difficult to find. The xz</code> backdoor required around 3 years worth of effort to attempt to sneak into the application and was discovered in less than a week after it went live. Don't live in fear of dependencies. Do what provides the least friction for you to accomplish the engineering you enjoy doing within your personal or team parameters.

Deobfuscating World of Warships' Python Scripts

2023-11-08T00:00:00+00:00

#</a > Background</h2>
This blog post is something I'm writing 3 years after my initial research/development, and about 2 years after I stopped actively working on the tool. Some of the details in this blog may not be fully accurate from time slippage and a lot of the initial research notes I made were lost or scattered in Discord conversations.
I am only now writing this as the game is somewhat dead and the development team chooses to continually release content that makes gameplay worse. Submarines, hybrid battleships, aircraft carriers, and HE-spamming battleships with cruiser concealment that overmatch everything have led to a less enjoyable gameplay experience.
The tool and techniques have been kept to a very tight circle so that data mining could continue without a potential cat-and-mouse game with the developer. I apologize to that community if that does occur.
Two years ago I open-sourced a tool I called `unfuck</code></a> which I described as a "Python 2.7 bytecode deobfuscator</del> unfucker". That tool was the result of the work I had done for World of Warships, but isn't capable of completely deobfuscating World of Warships files out-of-the-box. unfuck</code>` `made the rounds on HN</a> where I got supportive responses such as:`
Look, I get that edgy names are fun, but I'm happy that I will never have to use this tool for work, and I pity the fool who has to explain why "unfuck" was needed to solve a real problem. </blockquote> And: Ooof, really bad name. Makes me think the project or maintainer are immature... </blockquote> And someone respecting my licensing choices: Interesting dual licensing </blockquote> This project is dual-licensed under MIT and the ABSE ("Anyone But Stefan Esser") license. Note that an additional exception to the license is added, forbidding use/redistribution of said content to his trainees as well, but only when in a 5 mile radius from "Stefan Esser" or while holding any sort of (video)conference/chat with him. </blockquote> Note that this license will only be used as long as what would capstone decode / that one other arm64 ida plugin thing by i0n1c ("Stefan Esser") are not under the MIT license. afterwards, all exceptions are cleared and basically MIT license applies </blockquote> </blockquote> #</a > What is World of Warships?</h2> World of Warships (WoWs) is a free-to-play naval warfare multiplayer game released in 2015 by Wargaming and their Lesta Studio. It supports multiple forms of modifications</a> via Adobe Flash for UI mods, XML/audio files for audio mods, custom ship textures, and Python for basically everything else. The Python APIs are somewhat limited, but can observe some in-game events that allow mod developers to surface information that the game doesn't show you by default. Unlike a lot of multiplayer games, WoWs uses an authoritative server model where each client is only receiving events that the server believes they should receive. For example, enemy ship locations and related information is only pushed down to clients when they are intended to be painted to your screen. There are no wall hacks and things like auto-aim are somewhat negated by player skill. That doesn't mean there aren't illegal mods/cheats</a> that show you where to aim, where incoming artillery shells will land, etc. With all of that said, there is very little incentive to cheat in this game. A top-tier player in World of Warships will have the game sense and skill that comes within a close margin to someone playing with cheats. Therefore deobfuscation of scripts doesn't necessarily help with cheat development apart from illegal mods, but may help modders create better mods in general. #</a > Game Scripts</h2> Although the World of Warships developers are mostly transparent about how game mechanics work, there have still been some mysteries. For example, the algorithm for how the dispersion ellipse of your shells is calculated</a> and the dispersion of the shells themselves</a>. The format of match replay files is also mostly documented</a> through reverse engineering work of a few researchers and from reviewing publicly available engine source code, but there are still unknown elements to the serialized format. For these reasons it's highly desirable to review the implementation code to unmask these mysteries. There has been at least one other individual who managed to deobfuscate and decompile the game scripts as far back as 2016. A World of Warships EU forum member by the name of "TehRick" / "ThiSpawn" made multiple posts</a> showing they had reverse engineered some of the C++ and Python logic. In fact, they explicitly called out the "Lesta anti-noob protection" obfuscated Python module name in one of their forum posts: In a Reddit post</a> by the same username they also linked to a decompiled Python source file: https://pastebin.com/y3Yk43Nd</a>. Although TehRick seems to have disappeared soon after these posts, their code was still being referenced 4 years later! If you are TehRick / ThiSpawn feel free to reach out to me -- I would love to talk about your deobfuscator! #</a > Python VM Primer</h2> When Python source code is executed, you may notice that a .pyc</code> file is created. The Python interpreter doesn't interpret raw source code -- it first compiles that source code to an intermediate representation that can be fed to the VM as instructions. The VM is stack-based with some "registers" which are used for runtime storage of variables in pre-defined variable (or unnamed) slots. Almost every VM operation with the exception of loads/stores will directly modify values on the stack in some way. Variable registers cannot be modified in-place, and must first be put directly on the stack. Each object on the stack or in a variable slot is a deserialized Python object representing one of the following types: None</li> StopIteration</li> Ellipsis</li> Bool</li> Long</li> Float</li> Complex</li> Bytes</li> String</li> Tuple</li> List</li> Dict</li> Set</li> FrozenSet</li> Code</li> </ul> Basically the raw primitive types you're probably used to when writing Python. All module definitions, functions, etc. are defined as Code</code> objects that live in the co_consts</code> section of their parent code object. Ditto with any lists, tuples, etc. that are hard coded in source code. #</a > Instructions</h3> World of Warships uses Python 2.7 which has helpful documentation covering all instructions here: https://docs.python.org/2/library/dis.html</a>. All instructions are at least 1 byte for the opcode and up to 3 bytes for instructions which have a 16-bit argument. For example, a JUMP_ABSOLUTE 300</code> may be encoded as 0x71_012c</code> and a POP_TOP</code> may be encoded as 0x01</code>. The WoWs developers did not do any opcode remapping, which is a fairly common obfuscation trick when an application has the flexibility of embedding the Python VM. #</a > Prior Work on Deobfuscation</h2> #</a > lpcvoid's Findings</h3> World of Warships' core game logic is contained within a scripts.zip</code> file that is handled by a special loader. The loader reads compiled Python (files ending in .pyc</code>) out of this zip archive and even uses a special technique of handling the serialized Python code object. This logic and deobfuscating the first stage of the matryoshka doll has already been described by lpcvoid on his blog</a> and he even has a part 2</a> going into some of the junk instructions. I highly recommend reading his blog posts before continuing, as I will not rehash his hard work. To summarize his findings: the module loader deserializes the bytecode object and uses the bytecode as an encryption key for some ciphertext stored in const data. After decrypting the ciphertext, the plaintext is decompressed (zlib) and is executed as a code object. #</a > Rapid Analysis of Bytecode Using pyasm</h3> I wanted to take a quick moment to call out that my friend @gabe_k wrote a tool for a CTF challenge he put together many years ago called pyasm</code> that is specifically designed for this type of scenario. It can gracefully handle disassembling of bad instructions, and even supports recompiling a .pyasm</code> file back to a serialized code object. I've forked the project and made some quality of life improvements/bug fixes here: https://github.com/landaire/pyasm</a>. Here is an example of what one of these .pyc</code> files look like when converted to a .pyasm</code> file (redacted since it's very long): code stack_size 3 flags 66 consts 4 none string "Wargaming.net | Lesta Studio" string "an error occurred while loading module" string "\x9a\xb6\x85A.^lPGO0Z/\xeeY3o\x11$,FCPCi&:U\x04\t\x02IS\x155EJ\x1a3\x96\x1f=\xe2?\xa95\xa5\tf\x13hl\x92,\x12\x14T\x06\xc8o:\x08\x16\xd4\xfd\xd0\x8c\xc1T\xa0\x9b\xe5b\xc3%\x0eD\x8e\x85\xfb%\x83\x9b\xffL\rH\ruk\x14\xf2=\xd7\x86\t\x13\x0bk\x83HL\tH\xbf\x076IM\xa1\x0bTP\xdc\xc7<\n\x08\x8c\xdd\x05\'\x19\xf0\xa2FL\t\xbb\xd7\x15=\xd8\xc3\xba3\x19\x0fk\xd9\xe4\xf0\xbf\x9c?\nTJP\xc1\xcc\x17\xce\x04\x18\xfe98\x0e\x1d\x86\xdc\xab\x19\xe6)M,\x0e\xd4\xd8\xd4\x02\xb1\x0b(&;(o_\x1c\x1b\x16\xd8\xe1!-\xadw\xda%\xd0/\xa2\x1a\x087\xc7\x9d\x107\x0f\xe4\xcc\x1c\x0c)... end names 1 string "locals" end instructions <255> 65532 UNARY_INVERT SETUP_EXCEPT 15 LOAD_CONST 1 # Wargaming.net | Lesta Studio LOAD_NAME 0 # locals CALL_FUNCTION 0 DUP_TOP EXEC_STMT POP_BLOCK JUMP_FORWARD 12 3 POP_TOP LOAD_CONST 2 # an error occurred while loading module PRINT_ITEM PRINT_NEWLINE JUMP_FORWARD 1 END_FINALLY LOAD_CONST 0 # None RETURN_VALUE DELETE_NAME 23519 UNARY_NOT DELETE_NAME 23438 <216> 64603 INPLACE_OR UNARY_NOT DELETE_NAME 23308 ... end</code></pre> Throughout my analysis of the obfuscation tricks I frequently leaned on pyasm for manually reordering/deleting instructions in order to figure out which sections were causing hiccups for the decompiler. #</a > WoWs - Generic Obfuscation Tricks</h2> #</a > 1: Bogus Instructions</h3> After decrypting and decompressing the 2nd stage code object, you'll find that tools like uncompyle6</a> fail to decompile the bytecode to source with an error. In his 2nd blog post lpcvoid covered one of Lesta's tricks of inserting bogus instructions that contain completely invalid opcodes which confuse these types of tools. One thing not mentioned is that these instructions can also be valid but really mess up the VM's stack and cause static analysis to enter a bad state. Unfortunately, tools like uncompyle are mostly intended to run on Python bytecode generated cleanly by a Python VM. Invalid opcodes are definitely not supported, and neither are invalid instruction operands. This trick is the most straightforward for causing a decompiler to choke. #</a > 2: Stack Reordering</h3> Different Python code patterns -- even if semantically the same -- have very slight nuance in the emitted instructions. Tools like uncompyle in some cases rely on fragile instruction patterns for mapping to source code and will easily encounter errors when a wonky pattern is encountered. The following is a completely made-up example, but consider the following instruction sequence: 224 LOAD_CONST 0 227 MAKE_FUNCTION_0 0 None L. 351 230 STORE_FAST 6 'f333'</code></pre> This loads some code object from the const section, makes a function, and stores that function in co_varnames[6]</code>. This would typically result in something like the following Python code: def f333(): pass</code></pre> Now imagine that the instructions were rewritten to push and immediately pop a value to the stack randomly in the middle of creating the function: 224 LOAD_CONST 0 227 MAKE_FUNCTION_0 0 None ... LOAD_CONST None ... POP_TOP L. 351 230 STORE_FAST 6 'f333'</code></pre> The LOAD_CONST</code>/MAKE_FUNCTION</code>/STORE_FAST</code> pattern is broken by instructions that are effectively a no-op, and the signatures used by the decompilers are now broken. #</a > 3: Const Predicates</h3> Related to trick #2, instruction patterns may be broken by inserting conditions that evaluate to a constant value. One side of the branch may bring you to some garbage instructions (trick #1) and on the other side of the branch will be the next set of valid code to be executed. Consider the following pseudocode: 224 LOAD_CONST 0 227 MAKE_FUNCTION_0 0 None if len({1, 2, 3} & {2}) > 0: L. 351 230 STORE_FAST 6 'f333' else: ... POP_TOP ... POP_TOP ... POP_TOP ... RETURN_VALUE </code></pre> In the middle of defining a function we've inserted a check to see if two sets overlap, and if so we store the function in variable slot #6 (STORE_FAST 6</code>). If the sets do not overlap, we go down the bogus code path that screws up stack state. #</a > 4: Variable Renaming</h3> This one really doesn't impact tools but definitely impacts any end user who manages to partially decompile anything: local variables and function names (in the serialized function object) are renamed to things that are illegal in Python source code such as keywords, operators, and a combination of these things with spaces. Function objects are typically renamed to be very large unique numbers. Here is a real example from when I first set on this project: def f333(impf, f222): continue r ; = 66 h >> ] = 87 assert } break = 538 l else try 6 = 199 continue r ; += 66 h >> ] -= 538 assert } break = 199 if not continue r ; + h >> ] >= assert } break - h >> ] + l else try 6: k try 4 = 113 k try 4 = None 2 c = 22 v as lambda n = 433 6 p = 147 k try 4 += 113 2 c -= 433 v as lambda n = 147 if not k try 4 + 2 c >= v as lambda n - 2 c + 6 p: pass (j in, % /= p, 6 { +, y [, += import r) = ()</code></pre>#</a > 5: Implicit Returns</h3> The RETURN_VALUE</code> instruction in Python returns the value located at the top of the VM's stack. Python will, as far as I'm aware, always emit a RETURN_VALUE</code> with the immediately preceding instruction setting up the value to be returned. For example: LOAD_FAST 0 RETURN_VALUE</code></pre> This loads the value stored at co_varnames[0]</code> to the top of stack and returns it. Splitting up these instructions will break the pattern decompilers use to transform this into return varname</code>. Imagine if the in the following code tos</code> literally represented the top of the stack (and not a variable slot): if condition: tos = get_return_value() else: tos = other_return_value() return</code></pre> This would create the following control flow: In this scenario the instruction immediately preceding the RETURN_VALUE</code> isn't the instruction setting up the value -- it's likely some type of JUMP</code> instruction or quite possibly anything else! It's not clear to me if this is more of a code optimization trick, or an obfuscation trick, but really what's the difference? #</a > 6: Weird Jumps</h3> Something else I noticed was that some jumps were just... weird? There were randomish-looking JUMP_FORWARD N</code> instructions that didn't make sense (but usually were just jumping over garbage instructions), and were hard to disambiguate from those legitimately generated by the Python compiler. For example, the following Python code may insert a JUMP_FORWARD 0</code> (jump to the next instruction): if !foo: # POP_JUMP_IF_FALSE # EMIT THIS CODE FIRST if foo: print "target" else: print "else" # AFTER THE ABOVE BLOCK IS EMITTED, INSERT JUMP_FORWARD 0 else: # EMIT BLOCK print "main target"</code></pre> This probably doesn't make much sense, so let me show a real example of the control flow at an instruction level: </a> Do you notice the JUMP_FORWARD 0</code> in the left-center node? It's completely unnecessary! The layout of these instructions when serialized is: POP_TOP</code>, POP_TOP</code>, POP_TOP</code>, JUMP_FORWARD 0</code>, LOAD_FAST 2</code>. The execution sequence of these instructions is exactly the same as well. You could remove the JUMP_FORWARD 0</code> and nothing of value would be lost. So why is it there? It's just a side effect of how the Python compiler does codegen: def visitIf(self, node): end = self.newBlock() numtests = len(node.tests) for i in range(numtests): test, suite = node.tests[i] if is_constant_false(test): # XXX will need to check generator stuff here continue self.set_lineno(test) self.visit(test) nextTest = self.newBlock() self.emit('POP_JUMP_IF_FALSE', nextTest) self.nextBlock() self.visit(suite) self.emit('JUMP_FORWARD', end) # <--- UNCONDITIONALLY ADD `JUMP_FORWARD` self.startBlock(nextTest) if node.else_: self.visit(node.else_) self.nextBlock(end)</code></pre> Sometimes that JUMP_FORWARD</code> isn't jumping 0 bytes and may be some value that jumps over garbage instructions (and was inserted by the obfuscator). Other times this JUMP_FORWARD</code> comes immediately after some other unconditional control flow instruction and will never be executed by a Python VM (or picked up by my instruction decoder). So why can't it be removed if it's usually unnecessary? Unfortunately uncompyle relies on the presence of the JUMP_FORWARD</code> to determine what type of condition has occurred. #</a > 7: Generally Weird Control Flow</h3> This one is hard to express without sounding psychotic, but let me just say: fuck loops, fuck exception handlers. If you combine the weird jumps and false predicates in loops, you may be able to generate some code that looks like a loop but is only ever executed for one iteration before just jumping to some other part of the code because of a const predicate. And what about exception handlers that intentionally raise an exception that's supposed to be caught to trigger the "good" code path? And what about nesting exceptions inside of loops with const predicates? After encountering such a scenario I was starting to feel like this: #</a > Deobfuscation</h2> Deobfuscating everything together requires a solid framework that can achieve: Parsing all instructions in a manner that doesn't break on bogus opcodes</li> Evaluating conditions to determine used/unused code</li> Restoring instruction ordering</li> Restoring variable names</li> Deoptimizing code (implicit returns)</li> Normalize weird control flow</li> </ol> ...so yeah the natural path I took here was to rebuild the Python VM in Rust in 2 months and write somewhat spaghetti code that revisiting two years later has me thinking "wtf was I smoking?". #</a > Avoiding Bad Instructions</h3> Parsing instructions is fairly straightforward: an instruction with no arguments is 1 byte (opcode), and an instruction with an argument is 3 bytes (opcode + uint16). You start parsing instructions by reading from offset 0 in the co_code</code> section of the code object and just continue this in a loop. Decompilers and disassemblers tend to read instructions linearly -- i.e. if the first instruction is JUMP_ABSOLUTE 200</code> it's going to disassemble JUMP_ABSOLUTE 200</code> from offset 0, then disassemble the next instruction from offset 3. This isn't great because you will run into a bunch of bogus instructions that can be avoided by simply creating a decoder that understands control flow. To mitigate this in my deobfuscator I instead add instruction offsets to a queue. A JUMP_ABSOLUTE 200</code> will add offset 200 as next in the queue, and a JUMP_IF_{TRUE,FALSE} <TARGET></code> will add the offset for the target and the next instruction to the queue. Along the way I also compile an instruction graph where each node represents a basic block with edges to other basic blocks: </a> #</a > Removing const predicates</h3> Remember how I said I rebuilt the Python VM in Rust? I was serious, and it was just to solve this problem.</a> I do what I've called "partial execution". Certain builtin functions are handled, and an individual code object's opcodes are executed to obtain a snapshot of the VM's stack and perform taint tracking. The main function signature is: /// Executes an instruction, altering the input state and returning an error /// when the instruction cannot be correctly emulated. For example, some complex /// instructions are not currently supported at this time. pub fn execute_instruction<O: Opcode<Mnemonic = py27::Mnemonic>, F, T>( instr: &Instruction<O>, code: Arc<Code>, stack: &mut VmStack<T>, vars: &mut VmVars<T>, names: &mut VmNames<T>, globals: &mut VmNames<T>, names_loaded: LoadedNames, mut function_callback: F, access_tracking: T, ) -> Result<(), Error<O>>;</code></pre> Brief rundown of all inputs: instr</code>: the instruction to be executed</li> code</code>: the deserialized Python code object for which the instruction belongs to</li> stack</code>: current snapshot of the VM stack</li> vars</code>: all variable slots and their current values</li> names</code>: map to a vars</code> slot</li> globals</code>: similar to vars</code> but on a global level</li> names_loaded</code>: modules imported</li> function_callback</code>: callback for CALL_FUNCTION</code> instruction and can be used for handling builtins or calling other code objects if desired</li> access_tracking</code>: data to associate with a VM stack value</li> </ul> When an instruction is executed and the stack is modified, the modified stack value will represent a tuple of (Option<value>, [access_tracking])</code>. The first value in the tuple is the value that resulted from executing an instruction if it could be determined, otherwise None</code>. The second value will represent some metadata for looking up instructions which contributed to generating/modifying that stack value (a basic block index + instruction index). The execution loop for a basic block looks something like this: for instruction in basic_block: if instruction.is_conditional_jump(): if tos[0].is_some(): # Determine the truthiness of top-of-stack. # # Based off of the opcode (JUMP_IF_{TRUE,FALSE}), determine which # branch is never taken and remove the edge from this BB to the BB # we will never branch to. # # We also iterate tos[1] and remove all of the instructions which # contributed to tos[0]. else: execute_instruction(instruction, ...)</code></pre> This will effectively remove instructions related to const conditions and (usually) the basic block not taken from the graph. These inserted, fake basic blocks are implicitly removed since they become orphaned from the main code graph. There are some issues with this approach: I need to correctly implement almost all VM instructions</li> Large traces will balloon in computation complexity and memory</li> Complex arithmetic or unsupported instructions will immediately stop the VM, which leaves code only partially deobfuscated</li> </ol> #</a > Restoring Variable Names</h3> Variable names are unfortunately always lost. Instead of "restoring", I simply iterate and replace odd var names with the following script: def fix_varnames(varnames): global unknowns newvars = [] for var in varnames: var = var.strip() unallowed_chars = '=!@#$%^&()"\'/, ' banned_char = False banned_words = ['assert', 'in', 'continue', 'break', 'for', 'def', 'as', 'elif', 'else', 'for', 'from', 'global', 'if', 'import', 'is', 'lambda', 'not', 'or', 'pass', 'print', 'return', 'while', 'with'] for c in unallowed_chars: if c in var: banned_char = True if not banned_char: if var in banned_words: banned_char = True if banned_char: newvars.append('unknown_{0}'.format(unknowns)) unknowns += 1 else: newvars.append(var) return tuple(newvars)</code></pre> Pretty simple replacement of illegal-looking variable name with unknown_N</code>. #</a > Restoring Function Names</h3> Unlike var names, function names generally can be restored due to an oversight in the obfuscator. When a function is defined in a module, the bytecode for setting it up looks like the following: LOAD_CONST <CONST_INDEX> # Load the code object MAKE_FUNCTION # Take the loaded code object and tell the interpreter to turn it into a function STORE_NAME <NAME_INDEX> # Store the created function at the specified named index</code></pre> NAME_INDEX</code> corresponds to a name</code> string value located in the co_names</code> array in the code object. The code object for the function also contains its name, which is generally what's used by decompilers to label a function. I leveraged this in my instruction handler loop by checking: for instruction in basic_block: if instruction.is_store_name(): accessed_instructions = tos[1] if accessed_instructions[-1].is_make_function(): # change the function name on the code object to match what this # scope sees as the function name fix_function_name(tos[0], co.co_names[instruction.argument]) execute_instruction(instruction, ...)</code></pre> Strictly speaking these names aren't even necessary for the VM to run the code since they're really only present for debugging purposes. I suspect that this is either an oversight by Lesta or they deliberately left these in for debugging crashes on clients (although I'm not certain if they send back Python crash reports). Maybe there's something I don't know though. #</a > Deoptimizing Code</h3> I think that RETURN_VALUE</code> is really the only case I had to correct, and it was a fairly simple fix. Look for basic blocks that contain only a RETURN_VALUE</code> instruction</li> For each incoming edge, replace the final instruction in the basic block to be RETURN_VALUE</code></li> Remove the basic block from step 1</li> </ol> The example control flow for return values then provided in the section about implicit returns</a> will now look like this: #</a > Fixing Bad Instructions</h3> There are some scenarios where code paths containing bad instructions can't be outright removed. Usually it's because a condition couldn't be proven to be a const predicate, or there was some other factor involved that led to it not being removed. Even though I know they're bad, I'm hesitant to outright outright remove the nodes and conditions as gaps in the VM and data mixing may lead to incorrect instruction removal. To correct these basic blocks I calculate the depth of the stack at the location of the bad instruction, insert enough POP_TOP</code> instructions to clear out the stack, and finally put a LOAD_CONST None</code> and RETURN_VALUE</code> at the end of the basic block to force a return None</code>. #</a > The Rest of The Matryoshka Doll</h2> There are four distinct "stages" to loading the Python module, two of which we've already loosely discussed from lpcvoid's blog (encrypted code, and the compressed code). #</a > Stage 2 - Decompressed Code</h3> The following is an example stage 3 payload: import sys, marshal, copy_reg if id(marshal.loads) != copy_reg.mmId: return code = sys._getframe().f_back.f_code.co_code impf = (isinstance(builtins, dict) or builtins).import if 1 else builtins['import'] if not hasattr(impf, 'func_code') or hash(impf.func_code.co_code) != 1236377808: if hasattr(impf, 'func_code') and type(impf.func_globals['common']) != type(marshal.loads): return def f123--- This code section failed: --- 0 LOAD_GLOBAL 0 'common' 3 LOAD_FAST 0 'arg' 6 LOAD_FAST 1 'kw' 9 CALL_FUNCTION_VAR_KW_0 0 None 12 STORE_FAST 2 'res' 15 LOAD_GLOBAL 1 'type' 18 LOAD_FAST 2 'res' 21 CALL_FUNCTION_1 1 None 24 LOAD_ATTR 2 'name' 27 LOAD_CONST 'module' 30 COMPARE_OP 2 == 33 POP_JUMP_IF_FALSE 155 'to 155' 36 LOAD_GLOBAL 3 'hasattr' 39 LOAD_FAST 2 'res' 42 LOAD_CONST 'file' 45 CALL_FUNCTION_2 2 None 48 POP_JUMP_IF_FALSE 155 'to 155' 51 LOAD_GLOBAL 3 'hasattr' 54 LOAD_FAST 2 'res' 57 LOAD_CONST 'gCPLBx86' 60 CALL_FUNCTION_2 2 None 63 UNARY_NOT 64 POP_JUMP_IF_TRUE 82 'to 82' 67 LOAD_FAST 2 'res' 70 LOAD_ATTR 4 'gCPLBx86' 73 LOAD_CONST '1663084375' 76 COMPARE_OP 3 != 79_0 COME_FROM 64 '64' 79 POP_JUMP_IF_FALSE 155 'to 155' 82 LOAD_FAST 0 'arg' 85 LOAD_CONST 0 88 BINARY_SUBSCR 89 LOAD_CONST ('collections', 'utf8_test', 'copy_reg') 92 COMPARE_OP 7 not-in 95_0 COME_FROM 79 '79' 95_1 COME_FROM 48 '48' 95_2 COME_FROM 33 '33' 95 POP_JUMP_IF_FALSE 155 'to 155' 98 SETUP_EXCEPT 48 'to 149' 101 LOAD_GLOBAL 5 'loaded' 104 LOAD_ATTR 6 'add' 107 LOAD_GLOBAL 7 'id' 110 LOAD_FAST 2 'res' 113 CALL_FUNCTION_1 1 None 116 LOAD_FAST 0 'arg' 119 LOAD_CONST 0 122 BINARY_SUBSCR 123 BUILD_TUPLE_2 2 126 CALL_FUNCTION_1 1 None 129 POP_TOP 130 LOAD_GLOBAL 8 'sys' 133 DUP_TOP 134 LOAD_ATTR 9 'errCnt' 137 LOAD_CONST 1 140 INPLACE_ADD 141 ROT_TWO 142 STORE_ATTR 9 'errCnt' 145 POP_BLOCK 146 JUMP_FORWARD 6 'to 155' 149_0 COME_FROM 98 '98' 149 POP_TOP 150 POP_TOP 151 POP_TOP 152 JUMP_FORWARD 0 'to 155' 155_0 COME_FROM 152 '152' 155_1 COME_FROM 146 '146' 155 LOAD_FAST 2 'res' 158 RETURN_VALUE -1 RETURN_LAST Parse error at or near `None' instruction at offset -1 def f123(impf, f222): import sys f222.func_globals['loaded'] = set() if not isinstance(builtins, dict): f222.func_globals['common'] = builtins.import if 1 else builtins['import'] f222.func_globals['sys'] = sys sys.errCnt = 0 builtins.import = isinstance(builtins, dict) or f222 else: builtins['import'] = f222 sys.settrace(None) sys.settrace = sys.getrefcount sys.setprofile(None) sys.setprofile = sys.getrefcount sys.gettrace = sys.exit sys.getprofile = sys.exit return f333(impf, f222) impf = f222 def f123(marshaled): swapMap = {0: 151, 1: 235, 2: 9, 3: 249, 4: 100, 5: 10, 6: 188, 7: 106, 8: 128, 9: 122, 10: 220, 11: 189, 12: 242, 13: 253, 14: 210, 15: 243, 16: 5, 17: 27, 18: 222, 19: 90, 20: 139, 21: 18, 22: 79, 23: 255, 24: 230, 25: 83, 26: 20, 27: 74, 28: 89, 29: 141, 30: 219, 31: 123, 32: 203, 33: 51, 34: 98, 35: 53, 36: 103, 37: 204, 38: 190, 39: 118, 40: 62, 41: 161, 42: 41, 43: 241, 44: 247, 45: 101, 46: 196, 47: 153, 48: 181, 49: 40, 50: 152, 51: 174, 52: 140, 53: 171, 54: 44, 55: 134, 56: 158, 57: 88, 58: 70, 59: 132, 60: 173, 61: 2, 62: 129, 63: 8, 64: 86, 65: 21, 66: 148, 67: 145, 68: 211, 69: 127, 70: 224, 71: 167, 72: 185, 73: 237, 74: 147, 75: 233, 76: 58, 77: 175, 78: 14, 79: 252, 80: 209, 81: 155, 82: 37, 83: 162, 84: 42, 85: 227, 86: 78, 87: 136, 88: 12, 89: 246, 90: 81, 91: 126, 92: 186, 93: 6, 94: 87, 95: 150, 96: 96, 97: 39, 98: 193, 99: 28, 100: 55, 101: 59, 102: 200, 103: 30, 104: 225, 105: 197, 106: 212, 107: 213, 108: 245, 109: 179, 110: 105, 111: 111, 112: 112, 113: 32, 114: 156, 115: 91, 116: 68, 117: 50, 118: 13, 119: 66, 120: 84, 121: 159, 122: 182, 123: 102, 124: 221, 125: 154, 126: 57, 127: 254, 128: 130, 129: 17, 130: 82, 131: 77, 132: 104, 133: 95, 134: 146, 135: 48, 136: 169, 137: 164, 138: 121, 139: 223, 140: 11, 141: 232, 142: 244, 143: 218, 144: 85, 145: 113, 146: 177, 147: 166, 148: 52, 149: 24, 150: 170, 151: 4, 152: 73, 153: 144, 154: 236, 155: 34, 156: 205, 157: 115, 158: 114, 159: 226, 160: 45, 161: 234, 162: 19, 163: 133, 164: 168, 165: 135, 166: 194, 167: 99, 168: 138, 169: 251, 170: 46, 171: 72, 172: 60, 173: 94, 174: 31, 175: 75, 176: 3, 177: 178, 178: 116, 179: 238, 180: 7, 181: 143, 182: 92, 183: 142, 184: 176, 185: 25, 186: 108, 187: 250, 188: 16, 189: 160, 190: 107, 191: 240, 192: 208, 193: 0, 194: 187, 195: 49, 196: 15, 197: 184, 198: 199, 199: 43, 200: 165, 201: 38, 202: 125, 203: 76, 204: 110, 205: 71, 206: 33, 207: 217, 208: 1, 209: 229, 210: 120, 211: 131, 212: 195, 213: 69, 214: 231, 215: 97, 216: 248, 217: 201, 218: 206, 219: 22, 220: 23, 221: 35, 222: 207, 223: 124, 224: 137, 225: 65, 226: 157, 227: 93, 228: 180, 229: 56, 230: 117, 231: 63, 232: 191, 233: 109, 234: 239, 235: 36, 236: 202, 237: 163, 238: 119, 239: 214, 240: 183, 241: 54, 242: 172, 243: 29, 244: 47, 245: 228, 246: 198, 247: 61, 248: 26, 249: 149, 250: 67, 251: 216, 252: 192, 253: 80, 254: 64, 255: 215} marshaled = ('').join(map(chr, [ swapMap[ord(n)] for n in marshaled ])) return marshaled co_code = [ chr(((byte ^ 38) & 126 | (byte ^ 38) >> 7 & 1 | ((byte ^ 38) & 1) << 7) ^ 89) for byte in [ ord(byte) for byte in f123(code) ] ] locDict = {} locDict['globs'] = sys._getframe().f_back.f_globals locDict['code'] = marshal.loads(('').join(co_code[::-1])) locDict['marshal'] = marshal exec locDict['code'] in locDict def f111(): pass f111() del f111</code></pre> There are some checks to ensure that certain state is set up, but in general this will: Load the co_code</code> from the original stage 1 file</li> Apply a substitution cipher over each byte</li> Do some bit arithmetic on each byte of the result from step 2</li> Execute the code that was just decoded</li> </ol> In my deobfuscator I was able to leverage the custom Python VM to apply the swapmap for me by creating a state machine. Essentially I scan for certain instructions that look like they're applying the swapmap, then execute that function with some fake VM stack set up. That code can be found here: https://github.com/landaire/wowsdeob/blob/ffeeedaea9390c1d1e9ba785360e75aaa1aa10d0/src/smallvm.rs</a> #</a > Stage 3</h3> This stage is pretty boring all things considered. The Stage 3 code object is just another compressed code object that's been base64 encoded and had the result reversed. No big tricks here. The deobfuscator logic can be found here: https://github.com/landaire/wowsdeob/blob/ffeeedaea9390c1d1e9ba785360e75aaa1aa10d0/src/main.rs#L290-L297</a>. Worth noting that this is the stage which references the Lestas "Anti noobs protection"! #</a > Stage 4</h3> We have the final module! This module now needs to have all generic deobfuscation tricks applied to get decompilation. #</a > End Result</h2> The end result of this effort is going from a file that fails to decompile: ❯ uncompyle6 ./output/AirplaneUtils_stage4.pyc # uncompyle6 version 3.8.0 # Python bytecode 2.7 (62211) # Decompiled from: Python 2.7.18 (default, Sep 28 2022, 20:52:16) # [GCC Apple LLVM 14.0.0 (clang-1400.0.29.102)] # Warning: this version of Python has problems handling the Python 3 byte type in constants properly. # Embedded file name: 26977129990194521 # Compiled at: 2020-12-14 08:10:48 Traceback (most recent call last): File "/Users/lander/.pyenv/versions/2.7.18/bin/uncompyle6", line 10, in <module> sys.exit(main_bin()) File "/Users/lander/.pyenv/versions/2.7.18/lib/python2.7/site-packages/uncompyle6/bin/uncompile.py", line 194, in main_bin **options) File "/Users/lander/.pyenv/versions/2.7.18/lib/python2.7/site-packages/uncompyle6/main.py", line 328, in main do_fragments, File "/Users/lander/.pyenv/versions/2.7.18/lib/python2.7/site-packages/uncompyle6/main.py", line 230, in decompile_file do_fragments=do_fragments, File "/Users/lander/.pyenv/versions/2.7.18/lib/python2.7/site-packages/uncompyle6/main.py", line 149, in decompile co, out, bytecode_version, debug_opts=debug_opts, is_pypy=is_pypy File "/Users/lander/.pyenv/versions/2.7.18/lib/python2.7/site-packages/uncompyle6/semantics/pysource.py", line 2578, in code_deparse co, code_objects=code_objects, show_asm=debug_opts["asm"] File "/Users/lander/.pyenv/versions/2.7.18/lib/python2.7/site-packages/uncompyle6/scanners/scanner2.py", line 350, in ingest pattr = names[oparg] IndexError: tuple index out of range zsh: exit 1 uncompyle6 ./output/AirplaneUtils_stage4.pyc</code></pre> To: # uncompyle6 version 3.8.0 # Python bytecode 2.7 (62211) # Decompiled from: Python 2.7.18 (default, Sep 28 2022, 20:52:16) # [GCC Apple LLVM 14.0.0 (clang-1400.0.29.102)] # Warning: this version of Python has problems handling the Python 3 byte type in constants properly. # Embedded file name: 123823449462 # Compiled at: 2020-12-14 08:10:48 import math, random from ConstantsUtils import idGenerator import GameParams, Junk, Lesta from Math import Vector3 from AirPlanes.AirplaneConstants import DeathReason, SquadronStateEnum, Throttle, TurnDirection, PlaneTypeNames from AirplaneConstants import SQUADRON_DEPARTURE_BIT, SQUADRON_PURPOSE_BIT, SQUADRON_INDEX_BIT, PLANETYPE_2_PARAMSNAME, PLANE_TORPEDO_CONE_HALF_WIDTH, PlaneTypes, PLANE_PROJECTILE_GRAVITY from mc0f1198d import devMode from md0ce06f9 import LOG_ERROR from m79622f13 import normaliseAngle, getDirectionFromYaw, lerp, lerpAngles, getDirectionFromYawPitch, EPSILON from PlanesDEFConverter import PlanesDictConverter from PyMagic import pTuple from mc062022a import ShipTypes from shared_constants.m22c5a818 import PLANE_AMMO_TYPES class WayPoint: enum = idGenerator(0) GENERATED = next(enum) RESET = next(enum) LAUNCHING_START_NODE = next(enum) LAUNCHING_END_NODE = next(enum) LANDING_START_NODE = next(enum) LANDING_END_NODE = next(enum) del enum def init(self, pos, yaw, pitch, time, waypointType=GENERATED): self.pos = pos self.yaw = yaw self.pitch = pitch self.time = time self.sent = False self.type = waypointType def repr(self): return ('<< Waypoint pos:{0}, time:{1}, type:{2}, yaw:{3}, pitch: {4}>>').format(self.pos, self.time, self.type, self.yaw, self.pitch) def toDict(self): return {'position': Vector3(self.pos), 'yaw': self.yaw, 'pitch': int(normaliseAngle(self.pitch, False) 127 / math.pi), 'time': int(self.time 1000), 'type': self.type} @staticmethod def fromDict(dict): return WayPoint(Vector3(dict['position']), dict['yaw'], dict['pitch'] math.pi / 127.0, dict['time'] / 1000.0, dict['type']) @staticmethod def _splineReference(point1, point2, t): unknown_0 = point1.pos.flatDistTo(point2.pos) if unknown_0 > 0: unknown_1 = 1.0 - t unknown_2 = unknown_0 getDirectionFromYaw(point1.yaw) 1 = unknown_0 getDirectionFromYaw(point2.yaw) unknown_3 = unknown_1 unknown_1 * (1.0 + 2.0 t) point1.pos + t t (1.0 + 2.0 unknown_1) point2.pos + unknown_1 unknown_1 t unknown_2 - unknown_1 t t 1 unknown_4 = point1.pos.flatDistTo(unknown_3) / unknown_0 unknown_3.y = lerp(point1.pos.y, point2.pos.y, unknown_4) unknown_5 = lerpAngles(point1.yaw, point2.yaw, t) unknown_6 = 0.0 else: unknown_3 = lerp(point1.pos, point2.pos, t) unknown_5 = lerpAngles(point1.yaw, point2.yaw, t) unknown_6 = 0.0 return (unknown_3, unknown_5, unknown_6) def spline(point1, point2, t): return Lesta.splineWayPoints(point1.pos, point1.yaw, point1.pitch, point2.pos, point2.yaw, point2.pitch, t) spline = _splineOptimised def generateSquadronId(shipId, index, purpose, departureId): """ Generates id of the squadron based on the given arguments. :param shipId: id of the owner :type shipId: int :param index: index of the squadron within the owner :type index: int :param purpose: the function that squadron is performing :type purpose: int (AirplaneConstants.SquadronPurpose) :param departureId: unique departure id of the squadron incremented with every subsequent id generation :type departureId: int :return id of the squadron :rtype: int """ unknown_7 = departureId << SQUADRON_DEPARTURE_BIT | purpose << SQUADRON_PURPOSE_BIT | index + 1 << SQUADRON_INDEX_BIT | shipId assert retrieveOwnerID(unknown_7) == shipId assert retrieveSquadronIndex(unknown_7) == index assert retrieveSquadronPurpose(unknown_7) == purpose return unknown_7 def retrieveSquadronIndex(squadronId): unknown_8 = 7 return (squadronId >> SQUADRON_INDEX_BIT & unknown_8) - 1 def retrieveSquadronPurpose(squadronId): unknown_9 = 7 return squadronId >> SQUADRON_PURPOSE_BIT & unknown_9 def retrieveSquadronDeparture(squadronId): unknown_10 = 7 return squadronId >> SQUADRON_DEPARTURE_BIT & unknown_10 def parseSquadronId(squadronId): return (retrieveSquadronIndex(squadronId), retrieveSquadronPurpose(squadronId), retrieveSquadronDeparture(squadronId)) def retrieveOwnerID(id): return id & 4294967295L def getPlaneName(params, planeType): unknown_11 = PLANETYPE_2_PARAMSNAME.get(planeType) if planeType: return params.dict[unknown_11].planeType return planeType def getTorpedoingArea--- This code section failed: --- 0 LOAD_GLOBAL 0 'getDirectionFromYaw' 3 LOAD_FAST 1 'attackDir' 6 LOAD_ATTR 1 'yaw' 9 LOAD_GLOBAL 2 'math' 12 LOAD_ATTR 3 'pi' 15 LOAD_CONST 2 18 BINARY_DIVIDE 19 BINARY_ADD 20 CALL_FUNCTION_1 1 None 23 STORE_FAST 6 'unknown_14' 26 LOAD_GLOBAL 4 'min' 29 LOAD_CODE <code_object 369464740902> 32 MAKE_FUNCTION_0 0 None 35 LOAD_FAST 4 'formation' 38 LOAD_ATTR 5 'positions' 41 GET_ITER 42 CALL_FUNCTION_1 1 None 45 CALL_FUNCTION_1 1 None 48 STORE_FAST 7 'unknown_15' 51 LOAD_GLOBAL 6 'max' 54 LOAD_CODE <code_object 369537115186> 57 MAKE_FUNCTION_0 0 None 60 LOAD_FAST 4 'formation' 63 LOAD_ATTR 5 'positions' 66 GET_ITER 67 CALL_FUNCTION_1 1 None 70 CALL_FUNCTION_1 1 None 73 STORE_FAST 8 'unknown_16' 76 LOAD_FAST 8 'unknown_16' 79 LOAD_FAST 7 'unknown_15' 82 BINARY_SUBTRACT 83 STORE_FAST 9 'unknown_17' 86 LOAD_FAST 9 'unknown_17' 89 LOAD_FAST 5 'currentPlaneCount' 92 BINARY_MULTIPLY 93 LOAD_FAST 4 'formation' 96 LOAD_ATTR 7 'npositions' 99 BINARY_DIVIDE 100 STORE_FAST 10 'unknown_18' 103 LOAD_FAST 0 'attackPoint' 106 LOAD_FAST 10 'unknown_18' 109 LOAD_CONST 2 112 BINARY_DIVIDE 113 LOAD_GLOBAL 8 'PLANE_TORPEDO_CONE_HALF_WIDTH' 116 BINARY_SUBTRACT 117 LOAD_FAST 6 'unknown_14' 120 BINARY_MULTIPLY 121 BINARY_SUBTRACT 122 STORE_FAST 11 'unknown_19' 125 LOAD_FAST 0 'attackPoint' 128 LOAD_FAST 10 'unknown_18' 131 LOAD_CONST 2 134 BINARY_DIVIDE 135 LOAD_GLOBAL 8 'PLANE_TORPEDO_CONE_HALF_WIDTH' 138 BINARY_ADD 139 LOAD_FAST 6 'unknown_14' 142 BINARY_MULTIPLY 143 BINARY_ADD 144 STORE_FAST 12 'unknown_20' 147 LOAD_FAST 11 'unknown_19' 150 LOAD_FAST 12 'unknown_20' 153 COMPARE_OP 2 == 156 POP_JUMP_IF_FALSE 176 'to 176' 159 LOAD_FAST 12 'unknown_20' 162 LOAD_FAST 6 'unknown_14' 165 LOAD_CONST 0.001 168 BINARY_MULTIPLY 169 INPLACE_ADD 170 STORE_FAST 12 'unknown_20' 173 JUMP_FORWARD 0 'to 176' 176_0 COME_FROM 173 '173' 176 LOAD_FAST 0 'attackPoint' 179 LOAD_FAST 1 'attackDir' 182 LOAD_FAST 2 'planeParams' 185 LOAD_ATTR 9 'torpedoAimDist' 188 BINARY_MULTIPLY 189 BINARY_ADD 190 STORE_FAST 13 'unknown_21' 193 LOAD_FAST 13 'unknown_21' 196 LOAD_FAST 6 'unknown_14' 199 LOAD_FAST 3 'spreading' 202 BINARY_MULTIPLY 203 LOAD_CONST 0.5 206 BINARY_MULTIPLY 207 BINARY_SUBTRACT 208 STORE_FAST 14 'unknown_22' 211 LOAD_FAST 13 'unknown_21' 214 LOAD_FAST 6 'unknown_14' 217 LOAD_FAST 3 'spreading' 220 BINARY_MULTIPLY 221 LOAD_CONST 0.5 224 BINARY_MULTIPLY 225 BINARY_ADD 226 STORE_FAST 15 'unknown_23' 229 LOAD_FAST 11 'unknown_19' 232 LOAD_FAST 12 'unknown_20' 235 LOAD_FAST 14 'unknown_22' 238 LOAD_FAST 15 'unknown_23' 241 BUILD_TUPLE_4 4 244 RETURN_VALUE -1 RETURN_LAST Parse error at or near `None' instruction at offset -1 def getBombingZone(planeParams, modifierParams, aimAccuracy, attackerStrength=1.0): unknown_24 = planeParams.maxSpread unknown_25 = planeParams.minSpread unknown_26 = lerp(unknown_24[0], unknown_25[0], aimAccuracy) modifierParams.planeSpreadMultiplier unknown_27 = lerp(unknown_24[1], unknown_25[1], aimAccuracy) modifierParams.planeSpreadMultiplier 0 = planeParams.outerSalvoSize[0] unknown_26 attackerStrength unknown_28 = planeParams.outerSalvoSize[1] unknown_27 unknown_29 = planeParams.innerSalvoSize[0] unknown_26 attackerStrength unknown_30 = planeParams.innerSalvoSize[1] unknown_27 return (0, unknown_28, unknown_29, unknown_30) # snip</code></pre> Clearly some functions still fail to decompile, but it may be enough to just read the instructions at this point to understand the intent. And code objects that go from this: </a> To this: </a> #</a > Closing Thoughts</h2> There's one common theme in this post that I hope some readers picked up on: we are constantly battling the decompiler's ability to unravel code based off of heuristics instead of battling the obfuscator injecting garbage. The const predicates for example aren't even that big of a deal -- they just insert false control flow that at a source level is fairly straightforward to see is garbage. However, this false control flow is enough to throw off the decompiler's ability to figure out a single source code pattern that results in the entire code object failing to decompile. This isn't a jab at uncompyle</code> either -- it's a great tool that works fairly well considering there's zero competition in this space. However, I think that if I were to solve this problem from scratch in 2023 I'd solve it very differently by working on a better decompiler that erodes away mapping of 1:1 source code to bytecode, and instead focuses on rebuilding source with the same functional semantics. As reverse engineers we don't really care about how the code was originally written, we just want to understand its intent at a higher level. #</a > Thanks</h2> Thank you, reader, for making it this far. I'd like to extend thanks to the following people for their support in this research/providing deobfuscator feedback: lpcvoid (without his initial blog post I wouldn't have been nearly as motivated to go down this endeavour)</li> Track</li> TTaro</li> 901234</li> notyourfather</li> gabe_k</li> Scout1Treia</li> EdibleBug</li> </ul>

One Weird Trick to Improve Bug Finding With ASAN

2023-02-10T00:00:00+00:00

#</a > ASAN Primer</h2>

If you're already an ASAN expert, feel free to skip to the next section.

AddressSanitizer</a> (ASAN) is an extremely useful tool in software testing, debugging, and security testing for finding memory safety issues in native applications. It's extremely straightforward to use on most platforms -- all you need to do is pass -fsanitize=address</code> to clang/gcc and run the application.

As your application runs it builds metadata about its memory state into what's called a shadow memory. The shadow memory is essentiallly a compressed representation of the application's address space and is used to look up memory ranges that are considered addressable. Memory ranges that are not addressable will be referred to as "poisoned memory".

When ASAN detects a memory safety issue it will print a report to the console and stop the application. The first bit of the report is as follows:

=================================================================</span></span>
==1==ERROR: AddressSanitizer: container-overflow on address 0x602000000010 at pc 0x560696424930 bp 0x7ffce1e0f150 sp 0x7ffce1e0f148</span></span>
WRITE of size 4 at 0x602000000010 thread T0</span></span>
    #0 0x56069642492f in main /app/example.cpp:14:15</span></span>
    #1 0x7fecaaf9a082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)</span></span>
    #2 0x56069636335d in _start (/app/output.s+0x2135d)</span></span>
</span>
0x602000000010 is located 0 bytes inside of 12-byte region [0x602000000010,0x60200000001c)</span></span>
allocated by thread T0 here:</span></span>
    #0 0x56069642211d in operator new(unsigned long) /root/llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:95:3</span></span>
    #1 0x560696427824 in void* std::__1::__libcpp_operator_new[abi:v15000]<unsigned long>(unsigned long) /opt/compiler-explorer/clang-15.0.0/bin/../include/c++/v1/new:246:10</span></span>
    #2 0x560696427808 in std::__1::__libcpp_allocate[abi:v15000](unsigned long, unsigned long) /opt/compiler-explorer/clang-15.0.0/bin/../include/c++/v1/new:272:10</span></span>
    #3 0x5606964277a9 in std::__1::allocator<Foo>::allocate[abi:v15000](unsigned long) /opt/compiler-explorer/clang-15.0.0/bin/../include/c++/v1/__memory/allocator.h:112:38</span></span>
    #4 0x5606964275e0 in std::__1::__allocation_result<std::__1::allocator_traits<std::__1::allocator<Foo>>::pointer> std::__1::__allocate_at_least[abi:v15000]<std::__1::allocator<Foo>>(std::__1::allocator<Foo>&, unsigned long) /opt/compiler-explorer/clang-15.0.0/bin/../include/c++/v1/__memory/allocate_at_least.h:54:19</span></span>
    #5 0x560696426479 in std::__1::__split_buffer<Foo, std::__1::allocator<Foo>&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<Foo>&) /opt/compiler-explorer/clang-15.0.0/bin/../include/c++/v1/__split_buffer:316:29</span></span>
    #6 0x560696425927 in void std::__1::vector<Foo, std::__1::allocator<Foo>>::__push_back_slow_path<Foo>(Foo&&) /opt/compiler-explorer/clang-15.0.0/bin/../include/c++/v1/vector:1535:49</span></span>
    #7 0x560696424d3b in std::__1::vector<Foo, std::__1::allocator<Foo>>::push_back[abi:v15000](Foo&&) /opt/compiler-explorer/clang-15.0.0/bin/../include/c++/v1/vector:1567:9</span></span>
    #8 0x5606964248d1 in main /app/example.cpp:11:10</span></span>
    #9 0x7fecaaf9a082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)</span></span></code></pre>
It tells us there's a container-overflow</em>, what address the container overflow occurred at, the call stack of where the overflow occurred, and finally where the memory we're faulting on was originally allocated.</p>
The next bit of the report is the shadow memory that was mentioned above:</p>



    
</a>
The big arrow here is pointing into the shadow memory at [06]</code>, which according to the legend at the bottom of the screenshot tells us there are six addressable bytes followed by a global redzone</em> (represented by the red 0xf9 in the shadow bytes).</p>
#</a
>
Runtime Instrumentation</h3>
ASAN builds its shadow memory at runtime with the help of its runtime library, libclang_rt.asan_{target_platform}_dynamic.dylib</code>. The runtime library provides some of the following:</p>

Memory management hooks for malloc()</code>, free()</code>, operator new()</code>, etc. Whenever a memory allocation/free occurs ASAN will update its shadow memory</li>
Functions for checking if memory is addressable or poisoned.</li>
Hooks for some common memory manipulation functions (strncpy</code>, strcpy</code>, memcpy</code>, memcmp</code>, etc.).</li>
</ul>
For checking if memory is addressable, ASAN's runtime provides some simple APIs that are used by its compiler instrumentation such as:</p>

__asan_load1</code></li>
__asan_store1</code></li>
__asan_load2</code></li>
__asan_store2</code></li>
__asan_load8</code></li>
__asan_store8</code></li>
...</li>
__asan_loadN</code></li>
</ul>
Note: This is certainly not a definitive list of APIs, but are relatively common</em>.</p>
These all essentially do the same thing under the hood:</p>
extern</span> "</span>C</span>"</span> NOINLINE INTERFACE_ATTRIBUTE </span>void</span> __asan_exp_loadN</span>(</span>uptr </span>addr</span>,</span> uptr </span>size</span>,</span></span>
u32 </span>exp</span>) {</span></span>
</span>
    if</span> (</span>__asan_region_is_poisoned</span>(</span>addr</span>,</span> size</span>)) {</span></span>
        GET_CALLER_PC_BP_SP</span>;</span></span>
        ReportGenericError</span>(</span>pc</span>,</span> bp</span>,</span> sp</span>,</span> addr</span>, false,</span> size</span>,</span> exp</span>, true, false);</span></span>
    }</span></span>
</span>
}</span></span></code></pre>
They take an address and size, check if memory in that range is poisoned, and reports a generic error if it is.</p>
#</a
>
Compiler Instrumentation</h3>
The compiler instrumentation is primarily used for poisoning stack memory and inserting calls into the runtime library for "interesting" loads/stores. Of course, not every</em> load/store will be instrumented by ASAN as that'd be a bit too heavy weight and a lot of things can be determined to be "safe" statically in the compiler.</p>
I'm not a compiler expert and truthfully don't care to dive into the source code at this time to figure out how ASAN determines what an "interesting" load/store is. With that said, when one is encountered ASAN's compiler pass will insert calls to the __asan_{load,store}{size}</code> runtime functions to check the operation.</p>
#</a
>
You're Probably Missing Out-of-Bounds Accesses</h2>
With the crash course on ASAN out of the way, we can dive in to the main point of this blog post: you're probably missing OOBR/W in your applications if you're using C++/Rust/whatever language containers.</p>
#</a
>
The Problem With Vectors</h3>
Here is some example code that should raise an out-of-bounds access violation:</p>
#include</span> <</span>vector</span>></span></span>
#include</span> <</span>stdio.h</span>></span></span>
#include</span> <</span>string.h</span>></span></span>
</span>
int</span> main</span>() {</span></span>
    // Allocate a vector to store some data generated by our fuzzer</span></span>
    std</span>::</span>vector</span><</span>char</span>></span> fuzzed</span>;</span></span>
    // Fuzzer pushes 5 bytes to the vector</span></span>
    fuzzed</span>.</span>push_back</span>(</span>0x41</span>);</span></span>
    fuzzed</span>.</span>push_back</span>(</span>0x42</span>);</span></span>
    fuzzed</span>.</span>push_back</span>(</span>0x43</span>);</span></span>
    fuzzed</span>.</span>push_back</span>(</span>0x44</span>);</span></span>
    fuzzed</span>.</span>push_back</span>(</span>0x45</span>);</span></span>
</span>
    // Copy 8 bytes from the vector to a test buffer</span></span>
    char</span> test</span>[</span>8</span>] = {</span>0</span>};</span></span>
    memcpy</span>(&</span>test</span>,</span> fuzzed</span>.</span>data</span>(), sizeof(</span>test</span>));</span></span>
    for</span> (</span>size_t</span> i </span>=</span> 0</span>;</span> i </span>< sizeof(</span>test</span>);</span> i</span>++) {</span></span>
        printf</span>("</span>%02X</span>",</span> test</span>[</span>i</span>]);</span></span>
    }</span></span>
</span>
    printf</span>("</span>\n</span>size(</span>%zu</span>), capacity(</span>%zu</span>)</span>\n</span>",</span> fuzzed</span>.</span>size</span>(),</span> fuzzed</span>.</span>capacity</span>());</span></span>
</span>
    return</span> 0</span>;</span></span>
}</span></span></code></pre>
We have a vector with 5 bytes that we then try to copy 8 bytes from. Pretty standard out-of-bounds read. When we run this with ASAN however...</p>
Program returned: 0</span></span>
Program stdout</span></span>
</span>
4142434445FFFFFFBEFFFFFFBEFFFFFFBE</span></span>
size(5), capacity(8)</span></span></code></pre>
https://godbolt.org/z/cecf6Pjz8</a></em></p>
No crash! You might notice something interesting in the last line of the output though: the size of the vector is 5, but its capacity is 8</em>.</p>
Some readers probably know that when you push_back()</code> or insert data into a vector</code> that's at its capacity, it reallocates the buffer to be double</em> its current size, copies the data to the new buffer, and frees the old one (or just does a realloc()</code>). As a vector starts to grow from 0 elements up to N, its growth looks like the following:</p>
</p>
Source</a></em></p>
This is very problematic for us. We're not catching an out-of-bounds access because of some implementation detail. All ASAN knows is that the application requested a buffer with 8 bytes -- it doesn't know that in our case 3 of those bytes are unused memory that aren't safe for us to use yet.</p>
In the general case, any memory accesses in the range from [vector.data() + vector.size(), vector.data() + vector.capacity()]</code> won't be detected as an out-of-bounds access!</p>
#</a
>
The Problem With Strings</h3>
Here's an example that's basically the same as the vector example above -- except, we're now constructing an std::string</code> with a static C string.</p>
#include</span> <</span>stdio.h</span>></span></span>
#include</span> <</span>string</span>></span></span>
#include</span> <</span>string.h</span>></span></span>
</span>
int</span> main</span>() {</span></span>
    std</span>::</span>string</span> test</span>("</span>four</span>");</span></span>
    char</span> temp</span>[</span>10</span>] = {</span>0</span>};</span></span>
    memcpy</span>(&</span>temp</span>,</span> test</span>.</span>data</span>(), sizeof(</span>temp</span>));</span></span>
</span>
    for</span> (</span>size_t</span> i </span>=</span> 0</span>;</span> i </span>< sizeof(</span>temp</span>);</span> i</span>++) {</span></span>
        printf</span>("</span>%02X</span>",</span> temp</span>[</span>i</span>]);</span></span>
    }</span></span>
</span>
    printf</span>("</span>\n</span>size(</span>%lu</span>), capacity(</span>%lu</span>)</span>\n</span>",</span> test</span>.</span>size</span>(),</span> test</span>.</span>capacity</span>());</span></span>
</span>
    return</span> 0</span>;</span></span>
}</span></span>
</span></code></pre>
Again, this doesn't trigger a crash:</p>
Program returned: 0</span></span>
Program stdout</span></span>
</span>
666F7572000000000000</span></span>
size(4), capacity(15)</span></span></code></pre>
https://godbolt.org/z/hdjK1WoKo</a></em></p>
So the four-character string actually has a total capacity of 15, i.e. the std::string</code> has over-allocated memory. If you tried initializing an std::vector</code> with an explicit initializer list it would allocate only the exact number of elements needed... why are strings different?</p>
Let's take a look at LLVM's libc++ string</code> code (simplified version will follow):</p>
#ifdef</span> _LIBCPP_BIG_ENDIAN</span></span>
    static const</span> size_type __short_mask </span>=</span> 0x01</span>;</span></span>
    static const</span> size_type __long_mask  </span>=</span> 0x1ul</span>;</span></span>
#else</span>  // _LIBCPP_BIG_ENDIAN</span></span>
    static const</span> size_type __short_mask </span>=</span> 0x80</span>;</span></span>
    static const</span> size_type __long_mask  </span>= ~(</span>size_type</span>(~</span>0</span>) >></span> 1</span>);</span></span>
#endif</span> // _LIBCPP_BIG_ENDIAN</span></span>
</span>
    enum</span> {</span>__min_cap</span> = (sizeof(</span>__long</span>) -</span> 1</span>)/sizeof(</span>value_type</span>) ></span> 2</span> ?</span></span>
                      (sizeof(__long) - 1)/sizeof(value_type) : 2</span>};</span></span>
</span>
    struct</span> __short</span></span>
    {</span></span>
        value_type </span>__data_</span>[</span>__min_cap</span>];</span></span>
        struct</span></span>
            :</span> __padding</span><</span>value_type</span>></span></span>
        {</span></span>
            unsigned char</span> __size_</span>;</span></span>
        };</span></span>
    };</span></span>
</span>
#else</span></span>
</span>
    struct</span> __long</span></span>
    {</span></span>
        size_type __cap_</span>;</span></span>
        size_type __size_</span>;</span></span>
        pointer   __data_</span>;</span></span>
    };</span></span>
</span>
#ifdef</span> _LIBCPP_BIG_ENDIAN</span></span>
    static const</span> size_type __short_mask </span>=</span> 0x80</span>;</span></span>
    static const</span> size_type __long_mask  </span>= ~(</span>size_type</span>(~</span>0</span>) >></span> 1</span>);</span></span>
#else</span>  // _LIBCPP_BIG_ENDIAN</span></span>
    static const</span> size_type __short_mask </span>=</span> 0x01</span>;</span></span>
    static const</span> size_type __long_mask  </span>=</span> 0x1ul</span>;</span></span>
#endif</span> // _LIBCPP_BIG_ENDIAN</span></span>
</span>
    enum</span> {</span>__min_cap</span> = (sizeof(</span>__long</span>) -</span> 1</span>)/sizeof(</span>value_type</span>) ></span> 2</span> ?</span></span>
                      (sizeof(__long) - 1)/sizeof(value_type) : 2</span>};</span></span>
</span>
    struct</span> __short</span></span>
    {</span></span>
        union</span></span>
        {</span></span>
            unsigned char</span> __size_</span>;</span></span>
            value_type __lx</span>;</span></span>
        };</span></span>
        value_type </span>__data_</span>[</span>__min_cap</span>];</span></span>
    };</span></span>
</span>
#endif</span> // _LIBCPP_ABI_ALTERNATE_STRING_LAYOUT</span></span>
</span>
    union</span> __ulx</span>{</span>__long __lx</span>;</span> __short __lxx</span>;};</span></span>
</span>
    enum</span> {</span>__n_words</span> = sizeof(</span>__ulx</span>) / sizeof(</span>size_type</span>)};</span></span>
</span>
    struct</span> __raw</span></span>
    {</span></span>
        size_type </span>__words</span>[</span>__n_words</span>];</span></span>
    };</span></span>
</span>
    struct</span> __rep</span></span>
    {</span></span>
        union</span></span>
        {</span></span>
            __long  __l</span>;</span></span>
            __short __s</span>;</span></span>
            __raw   __r</span>;</span></span>
        };</span></span>
    };</span></span>
</span>
    __compressed_pair</span><</span>__rep</span>,</span> allocator_type</span>></span> __r_</span>;</span></span></code></pre>
GitHub link.</a></p>
Yuck</em>. This is not simple to understand, but we can see that there's some interesting inline buffer stuff going on with the __short</code> struct at least. I've rewritten this code to be definitely not</em> the same layout as an std::string</code> but shows what's going on easier to understand:</p>
class</span> string</span> {</span></span>
	char</span> short_optimization</span>[</span>15</span>];</span></span>
	size_t</span> len</span>;</span></span>
	size_t</span> capacity</span>;</span></span>
	char</span> *</span>heap_longer_string</span>;</span></span>
}</span></span></code></pre>
std::string</code> has an optimization for short strings that allows it to avoid a heap allocation. Unfortunately, this means that for small strings we won't detect small out-of-bounds reads (OOBR) similar to the std::vector</code> problem. And similar to the std::vector</code> problem, heap-allocated strings grow in a way that over-allocates memory to reduce the number of allocations every time you push more data to it.</p>
#</a
>
Fixes</h2>
#</a
>
The "One Weird Trick"</h3>
This isn't really documented anywhere, but std::vector</code> actually does have ASAN enlightenment to detect this exact problem we're talking about:</p>
    // The following functions are no-ops outside of AddressSanitizer mode.</span></span>
    // We call annotatations only for the default Allocator because other allocators</span></span>
    // may not meet the AddressSanitizer alignment constraints.</span></span>
    // See the documentation for __sanitizer_annotate_contiguous_container for more details.</span></span>
#ifndef</span> _LIBCPP_HAS_NO_ASAN</span></span>
    _LIBCPP_CONSTEXPR_SINCE_CXX20</span></span>
    void</span> __annotate_contiguous_container</span>(</span>const void *</span>__beg</span>,</span> const void *</span>__end</span>,</span></span>
                                         const void *</span>__old_mid</span>,</span></span>
                                         const void *</span>__new_mid</span>)</span> const</span></span>
    {</span></span>
</span>
      if</span> (!</span>__libcpp_is_constant_evaluated</span>() &&</span> __beg </span>&&</span> is_same</span><</span>allocator_type</span>,</span> __default_allocator_type</span>>::</span>value</span>)</span></span>
        __sanitizer_annotate_contiguous_container</span>(</span>__beg</span>,</span> __end</span>,</span> __old_mid</span>,</span> __new_mid</span>);</span></span>
    }</span></span>
#else</span></span>
    _LIBCPP_CONSTEXPR_SINCE_CXX20 _LIBCPP_HIDE_FROM_ABI</span></span>
    void</span> __annotate_contiguous_container</span>(</span>const void*</span>,</span> const void*</span>,</span> const void*</span>,</span></span>
                                         const void*</span>)</span> const</span> _NOEXCEPT </span>{}</span></span>
#endif</span></span></code></pre>
GitHub Link</a>.</p>
When the _LIBCPP_HAS_NO_ASAN</code> preprocessor macro is not defined it has some logic for informing ASAN about the contiguous region of a vector as well as the contiguous region that's allocated but yet-unused. The preprocessor macro is only defined when:</p>
#    if !__has_feature(address_sanitizer)</span></span>
#      define _LIBCPP_HAS_NO_ASAN</span></span>
#    endif</span></span></code></pre>
GitHub Link</a>.</p>
So why the hell aren't we getting this enlightenment? We never defined it ourselves.</p>
I don't even remember why I tried this, but it seems you need to explicitly pass -stdlib=libc++</code> and just like magic, it works. Our example for an std::vector</code> will now detect the small OOBR with this flag:</p>
=================================================================</span></span>
==1==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000075 at pc 0x5640917cd227 bp 0x7ffe3ad2ee30 sp 0x7ffe3ad2e600</span></span>
READ of size 8 at 0x602000000075 thread T0</span></span>
    #0 0x5640917cd226 in __asan_memcpy /root/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:22:3</span></span>
    #1 0x56409180aa25 in main /app/example.cpp:17:5</span></span>
    #2 0x7f35ef2f1082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)</span></span>
    #3 0x56409174935d in _start (/app/output.s+0x2135d)</span></span></code></pre>
https://godbolt.org/z/ao64GcT7f</a>.</em></p>
There are some downsides to this:</p>

std::vector</code> is the only container with this enlightenment. But it does automatically update the poisoned region whenever we insert, remove, or clear the elements which is very nice.</li>
Our std::string</code> example still doesn't detect the OOBR with this compiler flag: https://godbolt.org/z/3bj6nnGxG</a>.</li>
You may not want to enable this if you have modules you cannot compile with this flag that may share an std::vector</code>. The module that's not enlightened would not poison memory correctly, leading to false-positives. There may be ABI compatability issues as well.</li>
</ul>
#</a
>
Code-Level Fix</h3>
Google's ASAN wiki</a> provides documentation for how to manually poison memory yourself using ASAN_POISON_MEMORY_REGION(addr, size)</code> and ASAN_UNPOISON_MEMORY_REGION(addr, size)</code>. We can use this as follows:</p>
#if</span> __has_feature</span>(</span>address_sanitizer</span>) </span>||</span> defined</span>(</span>__SANITIZE_ADDRESS__</span>)</span></span>
#include</span> <</span>sanitizer/asan_interface.h</span>></span></span>
#endif</span></span>
</span>
const uint8_t</span> *</span>extra_start </span>=</span> fuzzed</span>.</span>data</span>() +</span> fuzzed</span>.</span>size</span>();</span></span>
size_t</span> extra_len </span>=</span> fuzzed</span>.</span>capacity</span>() -</span> fuzzed</span>.</span>size</span>();</span></span>
</span>
</span>
#if</span> __has_feature</span>(</span>address_sanitizer</span>) </span>||</span> defined</span>(</span>__SANITIZE_ADDRESS__</span>)</span></span>
ASAN_POISON_MEMORY_REGION</span>(</span>extra_start</span>,</span> extra_len</span>);</span></span>
#endif</span></span></code></pre>
Or if for some reason you don't want to pull in the ASAN interface you could just copy data to a vector with the appropriate pre-allocated size:</p>
std</span>::</span>vector</span><</span>uint8_t</span>></span> copied</span>;</span></span>
copied</span>.</span>reserve</span>(</span>fuzzed</span>.</span>size</span>());</span></span>
std</span>::</span>copy</span>(</span></span>
    fuzzed</span>.</span>begin</span>(),</span></span>
    fuzzed</span>.</span>end</span>(),</span></span>
    std</span>::</span>back_inserter</span>(</span>copied</span>)</span></span>
);</span></span>
assert_eq</span>(</span>copied</span>.</span>capacity</span>(),</span> copied</span>.</span>size</span>())</span></span></code></pre>
Copying data sucks, but do what works for you. Note:</em> avoid using std::vector::shrink_to_fit()</code>. Per cppreference</a>, "It depends on the implementation whether the request is fulfilled."</p>
#</a
>
Other Tricks</h2>
While I have your attention I wanted to call out some other things you can do to improve your ability to find bugs.</p>
#</a
>
Failfast</h3>
If you have an abstraction that's intended to safely handle memory, why wait for your test or fuzzing harness to find the bug? For example, in my opinion a span</code> implementation should never be given an invalid memory range. We can enforce this at its constructor by checking if the provided memory region is poisoned and trigger a controlled crash:</p>
#include</span> <</span>sanitizer/asan_interface.h</span>></span></span>
</span>
template</span><</span>typename</span> T</span>></span></span>
class</span> span</span><</span>T</span>> {</span></span>
    public</span>:</span></span>
    span</span>(</span>T</span> *</span>data</span>,</span> size_t</span> count</span>) {</span></span>
</span>
        if</span> (</span>__asan_region_is_poisoned</span>(static_cast<</span>void</span>*>(</span>data</span>),</span> count </span>* sizeof(</span>T</span>))) {</span></span>
            assert</span>(false);</span></span>
        }</span></span>
</span>
    }</span></span>
}</span></span></code></pre>#</a
>
Sanitizer Recovery</h3>
Whenever you repro a bug with ASAN, try to remember to compile with -fsanitize-recover=address</code>. This will essentially allow the application to recover and continue running when ASAN triggers a violation.</p>
It may seem like a strange choice, but let's say you have a small out-of-bounds read that looks relatively boring. That bug may be hiding something much juicier that's triggered only</em> when the OOBR occurs! -fsanitize-recover=address</code> will allow the application to run until either a hard fault occurs or the application exits, but will still print any ASAN violation that occurs along the way.</p>
#</a
>
Closing Thoughts</h2>
ASAN is a very powerful tool, but has limitations on what it can provide you by default. When using abstractions that allocate memory for you, keep in mind that they may reduce ASAN's effectiveness. The examples shown here were exclusively C++ examples, but can be easily applied to other languages as well.</p>
Rust, for example, has zero ASAN englightenment at the time of this blog post. That means unsafe { }</code> code manually reading from a Vec<T></code>'s data pointer or passing the pointer across an FFI boundary may run into similar false-negatives. Ditto for the String</code> type, OSString</code>, etc.</p>

Reverse Engineering Yaesu FT-70D Firmware Encryption

2021-12-27T00:00:00+00:00

#</a > Background</h2>
Ham radios are a fun way of learning how the radio spectrum works, and more importantly: they're embedded devices that may run weird chips/firmware! I got curious how easy it'd be to hack my Yaesu FT-70D, so I started doing some research. The only existing resource I could find for Yaesu radios was someone who posted about custom firmware for their Yaesu FT1DR</a>.
The Reddit poster mentioned that if you go through the firmware update process via USB, the radio exposes its Renesas H8SX microcontroller and can have its flash modified using the Renesas SDK. This was a great start and looked promising, but the SDK wasn't trivial to configure and I wasn't sure if it could even dump the firmware... so I didn't use it for very long.

#</a > Other Avenues</h2>

Yaesu provides a Windows application on their website that can be used to update a radio's firmware over USB: </a>

The zip contains the following files:

1.2 MB  Wed Nov  8 14:34:38 2017  FT-70D_ver111(USA).exe</span></span>
682 KB  Tue Nov 14 00:00:00 2017  FT-70DR_DE_Firmware_Update_Information_ENG_1711-B.pdf</span></span>
8 MB  Mon Apr 23 00:00:00 2018  FT-70DR_DE_MAIN_Firmware_Ver_Up_Manual_ENG_1804-B.pdf</span></span>
3.2 MB  Fri Jan  6 17:54:44 2012  HMSEUSBDRIVER.exe</span></span>
160 KB  Sat Sep 17 15:14:16 2011  RComms.dll</span></span>
61 KB  Tue Oct 23 17:02:08 2012  RFP_USB_VB.dll</span></span>
1.7 MB  Fri Mar 29 11:54:02 2013  vcredist_x86.exe</span></span></code></pre>
I'm going to assume that the file specific to the FT-70D, "FT-70D_ver111(USA).exe", will likely contain our firmware image. A PE file (.exe) can contain binary resources in the .rsrc</code> section -- let's see what this file contains using XPEViewer</a>:</p>



    
</a>
Resources fit into one of many different resource types</a>, but a firmware image would likely be put into a custom type. What's this last entry, "23"? Expanding that node we have a couple of interesting items:</p>



    
</a>
RES_START_DIALOG</code> is a custom string the updater shows when preparing an update, so we're in the right area!</p>



    
</a>
RES_UPDATE_INFO</code> looks like just binary data -- perhaps this is our firmware image? Unfortunately looking at the "Strings" tab in XPEViewer or running the strings</code> utility over this data doesn't yield anything legible. The firmware image is likely encrypted.</p>
#</a
>
Reverse Engineering the Binary</h2>
Let's load the update utility into our disassembler of choice to figure out how the data is encrypted. I'll be using IDA Pro, but Ghidra (free!), radare2 (free!), or Binary Ninja are all great alternatives. Where possible in this article I'll try to show my rewritten code in C since it'll be a closer match to the decompiler and machine code output.</p>
A good starting point is the the string we saw above, RES_UPDATE_INFO</code>. Windows applications load resources by calling one of the FindResource*</code> APIs</a>. FindResourceA</code> has the following parameters:</p>

HMODULE</code>, a handle to the module to look for the resource in.</li>
lpName</code>, the resource name.</li>
lpType</code>, the resource type.</li>
</ol>
In our disassembler we can find references to the RES_UPDATE_INFO</code> string and look for calls to FindResourceA</code> with this string as an argument in the lpName</code> position.</p>



    
</a>
We find a match in a function which happens to find/load all</em> of these custom resources under type 23</code>.</p>



    
</a>
We know where the data is loaded by the application, so now we need to see how it's used. Doing static analysis from this point may be more work than it's worth if the data isn't operated on immediately. To speed things up I'm going to use a debugger's assistance. I used WinDbg's Time Travel Debugging</a> to record an execution trace of the updater while it updates my radio. TTD is an invaluable tool and I'd highly recommend using it when possible. rr</a> is an alternative for non-Windows platforms.</p>
The decompiler output shows this function copies the RES_UPDATE_INFO</code> resource to a dynamically allocated buffer. The qmemcpy()</code> is inlined and represented by a rep movsd</code> instruction in the disassembly, so we need to break at this instruction and examine the edi</code> register's (destination address) value. I set a breakpoint by typing bp 0x406968</code> in the command window, allow the application to continue running, and when it breaks we can see the edi</code> register value is 0x2be5020</code>. We can now set a memory access breakpoint at this address using ba r4 0x2be5020</code> to break whenever this data is read.</p>
Our breakpoint is hit at 0x4047DC</code> -- back to the disassembler. In IDA you can press G</code> and enter this address to jump to it. We're finally at what looks like the data processing function:</p>



    
</a>
We broke when dereferencing v2</code> and IDA has automatically named the variable it's being assigned to as Time</code>. The Time</code> variable is passed to another function which formats it as a string with %Y%m%d%H%M%S</code>. Let's clean up the variables to reflect what we know:</p>
bool</span> __thiscall </span>sub_4047B0</span>(</span>char</span> *</span>this</span>)</span></span>
{</span></span>
  char</span> *</span>encrypted_data</span>;</span> // esi</span></span>
  BOOL v3</span>;</span> // ebx</span></span>
  char</span> *</span>v4</span>;</span> // eax</span></span>
  char</span> *</span>time_string</span>;</span> // [esp+Ch] [ebp-320h] BYREF</span></span>
  int</span> v7</span>;</span> // [esp+10h] [ebp-31Ch] BYREF</span></span>
  __time64_t</span> Time</span>;</span> // [esp+14h] [ebp-318h] BYREF</span></span>
  int</span> (</span>__thiscall </span>**</span>v9</span>)(</span>void</span> *,</span> char</span>);</span> // [esp+1Ch] [ebp-310h]</span></span>
  int</span> v10</span>;</span> // [esp+328h] [ebp-4h]</span></span>
</span>
  // rename v2 to encrypted_data</span></span>
  encrypted_data </span>= *(</span>char</span> **)(*((</span>_DWORD </span>*)</span>AfxGetModuleState</span>() +</span> 1</span>) +</span> 160</span>);</span></span>
  Time </span>= *(</span>int</span> *)</span>encrypted_data</span>;</span></span>
  // rename this function and its 2nd parameter</span></span>
  format_timestamp</span>(&</span>Time</span>, (</span>int</span>)&</span>time_string</span>, "</span>%Y%m</span>%d</span>%H%M</span>%S</span>");</span></span>
  v10 </span>=</span> 1</span>;</span></span>
  v7 </span>=</span> 0</span>;</span></span>
  v9 </span>=</span> off_4244A0</span>;</span></span>
  sub_4082C0</span>(</span>time_string</span>);</span></span>
  v3 </span>=</span> sub_408350</span>(</span>encrypted_data </span>+</span> 4</span>,</span> 0x100000</span>,</span> this </span>+</span> 92</span>,</span> 0x100000</span>, &</span>v7</span>) ==</span> 0</span>;</span></span>
  v4 </span>=</span> time_string </span>-</span> 16</span>;</span></span>
  v9 </span>=</span> off_4244A0</span>;</span></span>
  v10 </span>= -</span>1</span>;</span></span>
  if</span> (</span> _InterlockedDecrement</span>((</span>volatile signed</span> __int32 </span>*)</span>time_string </span>-</span> 1</span>) <=</span> 0</span> )</span></span>
    (*(</span>void</span> (</span>__stdcall </span>**)(</span>char</span> *))(**(</span>_DWORD </span>**)</span>v4 </span>+</span> 4</span>))(</span>v4</span>);</span></span>
  return</span> v3</span>;</span></span>
}</span></span></code></pre>
The timestamp string is passed to sub_4082c0</code> on line 20 and the remainder of the update image is passed to sub_408350</code> on line 21. I'm going to focus on sub_408350</code> since I only care about the firmware data right now and based on how this function is called I'd wager its signature is something like:</p>
status_t</span> sub_408350</span>(</span>uint8_t</span> *</span>input</span>,</span> size_t</span> input_len</span>,</span> uint8_t</span> *</span>output</span>,</span> output_len</span>,</span> size_t</span> *</span>out_data_processed</span>);</span></span></code></pre>
Let's see what it does:</p>
int</span> __stdcall </span>sub_408350</span>(</span>char</span> *</span>a1</span>,</span> int</span> a2</span>,</span> int</span> a3</span>,</span> int</span> a4</span>,</span> _DWORD </span>*</span>a5</span>)</span></span>
{</span></span>
  int</span> v5</span>;</span> // edx</span></span>
  int</span> v7</span>;</span> // ebp</span></span>
  int</span> v8</span>;</span> // esi</span></span>
  unsigned int</span> i</span>;</span> // ecx</span></span>
  char</span> v10</span>;</span> // al</span></span>
  char</span> *</span>v11</span>;</span> // eax</span></span>
  int</span> v13</span>;</span> // [esp+10h] [ebp-54h]</span></span>
  char</span> v14</span>[</span>64</span>];</span> // [esp+20h] [ebp-44h] BYREF</span></span>
</span>
  v5 </span>=</span> a2</span>;</span></span>
  v7 </span>=</span> 0</span>;</span></span>
  memset</span>(</span>v14</span>,</span> 0</span>, sizeof(</span>v14</span>));</span></span>
  if</span> (</span> a2 </span><=</span> 0</span> )</span></span>
  {</span></span>
LABEL_13:</span></span>
    *</span>a5 </span>=</span> v7</span>;</span></span>
    return</span> 0</span>;</span></span>
  }</span></span>
  else</span></span>
  {</span></span>
    while</span> (</span> 1</span> )</span></span>
    {</span></span>
      v8 </span>=</span> v5</span>;</span></span>
      if</span> (</span> v5 </span>>=</span> 8</span> )</span></span>
        v8 </span>=</span> 8</span>;</span></span>
      v13 </span>=</span> v5 </span>-</span> v8</span>;</span></span>
      for</span> (</span> i </span>=</span> 0</span>;</span> i </span><</span> 0x40</span>;</span> i </span>+=</span> 8</span> )</span></span>
      {</span></span>
        v10 </span>= *</span>a1</span>;</span></span>
        v14</span>[</span>i</span>] = (</span>unsigned</span> __int8</span>)*</span>a1 </span>>></span> 7</span>;</span></span>
        v14</span>[</span>i </span>+</span> 1</span>] = (</span>v10 </span>&</span> 0x40</span>) !=</span> 0</span>;</span></span>
        v14</span>[</span>i </span>+</span> 2</span>] = (</span>v10 </span>&</span> 0x20</span>) !=</span> 0</span>;</span></span>
        v14</span>[</span>i </span>+</span> 3</span>] = (</span>v10 </span>&</span> 0x10</span>) !=</span> 0</span>;</span></span>
        v14</span>[</span>i </span>+</span> 4</span>] = (</span>v10 </span>&</span> 8</span>) !=</span> 0</span>;</span></span>
        v14</span>[</span>i </span>+</span> 5</span>] = (</span>v10 </span>&</span> 4</span>) !=</span> 0</span>;</span></span>
        v14</span>[</span>i </span>+</span> 6</span>] = (</span>v10 </span>&</span> 2</span>) !=</span> 0</span>;</span></span>
        v14</span>[</span>i </span>+</span> 7</span>] =</span> v10 </span>&</span> 1</span>;</span></span>
        ++</span>a1</span>;</span></span>
      }</span></span>
      sub_407980</span>(</span>v14</span>,</span> 0</span>);</span></span>
      if</span> (</span> v8 </span>)</span></span>
        break</span>;</span></span>
LABEL_12:</span></span>
      if</span> (</span> v13 </span><=</span> 0</span> )</span></span>
        goto</span> LABEL_13</span>;</span></span>
      v5 </span>=</span> v13</span>;</span></span>
    }</span></span>
    v11 </span>= &</span>v14</span>[</span>1</span>];</span></span>
    while</span> (</span> 1</span> )</span></span>
    {</span></span>
      --</span>v8</span>;</span></span>
      if</span> (</span> v7 </span>>=</span> a4 </span>)</span></span>
        return</span> -</span>101</span>;</span></span>
      *(</span>_BYTE </span>*)(</span>a3 </span>+</span> v7</span>++) =</span> v11</span>[</span>6</span>] | (</span>2</span></span>
                                      * (</span>v11</span>[</span>5</span>] | (</span>2</span></span>
                                                 * (</span>v11</span>[</span>4</span>] | (</span>2</span></span>
                                                            * (</span>v11</span>[</span>3</span>] | (</span>2</span></span>
                                                                       * (</span>v11</span>[</span>2</span>] | (</span>2</span></span>
                                                                                  * (</span>v11</span>[</span>1</span>] | (</span>2</span></span>
                                                                                             * (*</span>v11 </span>| (</span>2</span> * *(</span>v11 </span>-</span> 1</span>))))))))))))));</span></span>
      v11 </span>+=</span> 8</span>;</span></span>
      if</span> ( !</span>v8 </span>)</span></span>
        goto</span> LABEL_12</span>;</span></span>
    }</span></span>
  }</span></span>
}</span></span></code></pre>
I think we've found our function that starts decrypting the firmware! To confirm, we want to see what the output</code> parameter's data looks like before and after this function is called. I set a breakpoint in the debugger at the address where it's called (bp 0x404842</code>) and put the value of the edi</code> register (0x2d7507c</code>) in WinDbg's memory window.</p>
Here's the data before:</p>



    
</a>
After stepping over the function call:</p>



    
</a>
We can dump this data to a file using the following command:</p>
.writemem C:\users\lander\documents\maybe_deobfuscated.bin 0x2d7507c L100000</span></span></code></pre>
010 Editor has a built-in strings utility (Search > Find Strings...) and if we scroll down a bit in the results, we have real strings that appear in my radio!</p>



    
</a>
At this point if we were just interested in getting the plaintext firmware we could stop messing with the binary and load the firmware into IDA Pro</a>... but I want to know how this encryption works.</p>
#</a
>
Encryption Details</h2>
Just to recap from the last section:</p>

We've identified our data processing routine (let's call this function decrypt_update_info</code>).</li>
We know that the first 4 bytes of the update data are a Unix timestamp that's formatted as a string and used for an unknown purpose.</li>
We know which function begins decrypting our firmware image.</li>
</ul>
#</a
>
Data Decryption</h3>
Let's look at the firmware image decryption routine with some renamed variables:</p>
int</span> __thiscall </span>decrypt_data</span>(</span></span>
        void</span> *</span>this</span>,</span></span>
        char</span> *</span>encrypted_data</span>,</span></span>
        int</span> encrypted_data_len</span>,</span></span>
        char</span> *</span>output_data</span>,</span></span>
        int</span> output_data_len</span>,</span></span>
        _DWORD </span>*</span>bytes_written</span>)</span></span>
{</span></span>
  int</span> data_len</span>;</span> // edx</span></span>
  int</span> output_index</span>;</span> // ebp</span></span>
  int</span> block_size</span>;</span> // esi</span></span>
  unsigned int</span> i</span>;</span> // ecx</span></span>
  char</span> encrypted_byte</span>;</span> // al</span></span>
  char</span> *</span>idata</span>;</span> // eax</span></span>
  int</span> remaining_data</span>;</span> // [esp+10h] [ebp-54h]</span></span>
  char</span> inflated_data</span>[</span>64</span>];</span> // [esp+20h] [ebp-44h] BYREF</span></span>
</span>
  data_len </span>=</span> encrypted_data_len</span>;</span></span>
  output_index </span>=</span> 0</span>;</span></span>
  memset</span>(</span>inflated_data</span>,</span> 0</span>, sizeof(</span>inflated_data</span>));</span></span>
  if</span> (</span> encrypted_data_len </span><=</span> 0</span> )</span></span>
  {</span></span>
LABEL_13:</span></span>
    *</span>bytes_written </span>=</span> output_index</span>;</span></span>
    return</span> 0</span>;</span></span>
  }</span></span>
  else</span></span>
  {</span></span>
    while</span> (</span> 1</span> )</span></span>
    {</span></span>
      block_size </span>=</span> data_len</span>;</span></span>
      if</span> (</span> data_len </span>>=</span> 8</span> )</span></span>
        block_size </span>=</span> 8</span>;</span></span>
      remaining_data </span>=</span> data_len </span>-</span> block_size</span>;</span></span>
</span>
      // inflate 1 byte of input data to 8 bytes of its bit representation</span></span>
      for</span> (</span> i </span>=</span> 0</span>;</span> i </span><</span> 0x40</span>;</span> i </span>+=</span> 8</span> )</span></span>
      {</span></span>
        encrypted_byte </span>= *</span>encrypted_data</span>;</span></span>
        inflated_data</span>[</span>i</span>] = (</span>unsigned</span> __int8</span>)*</span>encrypted_data </span>>></span> 7</span>;</span></span>
        inflated_data</span>[</span>i </span>+</span> 1</span>] = (</span>encrypted_byte </span>&</span> 0x40</span>) !=</span> 0</span>;</span></span>
        inflated_data</span>[</span>i </span>+</span> 2</span>] = (</span>encrypted_byte </span>&</span> 0x20</span>) !=</span> 0</span>;</span></span>
        inflated_data</span>[</span>i </span>+</span> 3</span>] = (</span>encrypted_byte </span>&</span> 0x10</span>) !=</span> 0</span>;</span></span>
        inflated_data</span>[</span>i </span>+</span> 4</span>] = (</span>encrypted_byte </span>&</span> 8</span>) !=</span> 0</span>;</span></span>
        inflated_data</span>[</span>i </span>+</span> 5</span>] = (</span>encrypted_byte </span>&</span> 4</span>) !=</span> 0</span>;</span></span>
        inflated_data</span>[</span>i </span>+</span> 6</span>] = (</span>encrypted_byte </span>&</span> 2</span>) !=</span> 0</span>;</span></span>
        inflated_data</span>[</span>i </span>+</span> 7</span>] =</span> encrypted_byte </span>&</span> 1</span>;</span></span>
        ++</span>encrypted_data</span>;</span></span>
      }</span></span>
      // do something with the inflated data</span></span>
      sub_407980</span>(</span>this</span>,</span> inflated_data</span>,</span> 0</span>);</span></span>
      if</span> (</span> block_size </span>)</span></span>
        break</span>;</span></span>
LABEL_12:</span></span>
      if</span> (</span> remaining_data </span><=</span> 0</span> )</span></span>
        goto</span> LABEL_13</span>;</span></span>
      data_len </span>=</span> remaining_data</span>;</span></span>
    }</span></span>
    // deflate the data back to bytes</span></span>
    idata </span>= &</span>inflated_data</span>[</span>1</span>];</span></span>
    while</span> (</span> 1</span> )</span></span>
    {</span></span>
      --</span>block_size</span>;</span></span>
      if</span> (</span> output_index </span>>=</span> output_data_len </span>)</span></span>
        return</span> -</span>101</span>;</span></span>
      output_data</span>[</span>output_index</span>++] =</span> idata</span>[</span>6</span>] | (</span>2</span></span>
                                              * (</span>idata</span>[</span>5</span>] | (</span>2</span></span>
                                                           * (</span>idata</span>[</span>4</span>] | (</span>2</span></span>
                                                                        * (</span>idata</span>[</span>3</span>] | (</span>2</span></span>
                                                                                     * (</span>idata</span>[</span>2</span>] | (</span>2</span></span>
                                                                                                  * (</span>idata</span>[</span>1</span>] | (</span>2</span> * (*</span>idata </span>| (</span>2</span> * *(</span>idata </span>-</span> 1</span>))))))))))))));</span></span>
      idata </span>+=</span> 8</span>;</span></span>
      if</span> ( !</span>block_size </span>)</span></span>
        goto</span> LABEL_12</span>;</span></span>
    }</span></span>
  }</span></span>
}</span></span></code></pre>
At a high level this routine:</p>

Allocates a 64-byte scratch buffer</li>
Checks if there's any data to process. If not, set the output variable out_data_processed</code> to the number of bytes processed and return 0x0 (STATUS_SUCCESS</code>)</li>
Loop over the input data in 8-byte chunks and inflate each byte to its bit representation.</li>
After the 8-byte chunk is inflated, call sub_407980</code> with the scratch buffer and 0</code> as arguments.</li>
Loop over the scratch buffer and reassemble 8 sequential bits as 1 byte, then set the byte at the appropriate index in the output buffer.</li>
</ol>
Lots going on here, but let's take a look at step #3. If we take the bytes 0xAA</code> and 0x77</code> which have bit representations of 0b1010_1010</code> and 0b0111_1111</code> respectively and inflate them to a 16-byte array using the algorithm above, we end up with:</p>
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |    | 8 | 9 | A | B | C | D | E | F |</span></span>
|---|---|---|---|---|---|---|---|----|---|---|---|---|---|---|---|---|</span></span>
| 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 |    | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 |</span></span></code></pre>
This routine does this process over 8 bytes at a time and completely fills the 64-byte scratch buffer with 1s and 0s just like the table above.</p>
Now let's look at step #4 and see what's going on in sub_407980</code>:</p>
_BYTE </span>*</span>__thiscall </span>sub_407980</span>(</span>void</span> *</span>this</span>,</span> _BYTE </span>*</span>a2</span>,</span> int</span> a3</span>)</span></span>
{</span></span>
  // long list of stack vars removed for clarity</span></span>
</span>
  v3 </span>= (</span>int</span>)</span>this</span>;</span></span>
  v4 </span>=</span> 15</span>;</span></span>
  v5 </span>=</span> a3</span>;</span></span>
  v32</span>[</span>0</span>] = (</span>int</span>)</span>this</span>;</span></span>
  v28 </span>=</span> 0</span>;</span></span>
  v31 </span>=</span> 15</span>;</span></span>
  do</span></span>
  {</span></span>
    for</span> (</span> i </span>=</span> 0</span>;</span> i </span><</span> 48</span>; *((</span>_BYTE </span>*)&</span>v33 </span>+</span> i </span>+</span> 3</span>) =</span> v18 </span>)</span></span>
    {</span></span>
      v7 </span>=</span> v28</span>;</span></span>
      if</span> ( !</span>v5 </span>)</span></span>
        v7 </span>=</span> v4</span>;</span></span>
      v8 </span>= *(</span>_BYTE </span>*)(</span>i </span>+</span> 48</span> *</span> v7 </span>+</span> v3 </span>+</span> 4</span>) ^</span> a2</span>[(</span>unsigned</span> __int8</span>)</span>byte_424E50</span>[</span>i</span>] +</span> 31</span>];</span></span>
      v9 </span>=</span> v28</span>;</span></span>
      *(&</span>v34 </span>+</span> i</span>) =</span> v8</span>;</span></span>
      if</span> ( !</span>v5 </span>)</span></span>
        v9 </span>=</span> v4</span>;</span></span>
      v10 </span>= *(</span>_BYTE </span>*)(</span>i </span>+</span> 48</span> *</span> v9 </span>+</span> v3 </span>+</span> 5</span>) ^</span> a2</span>[(</span>unsigned</span> __int8</span>)</span>byte_424E51</span>[</span>i</span>] +</span> 31</span>];</span></span>
      v11 </span>=</span> v28</span>;</span></span>
      *(&</span>v35 </span>+</span> i</span>) =</span> v10</span>;</span></span>
      if</span> ( !</span>v5 </span>)</span></span>
        v11 </span>=</span> v4</span>;</span></span>
      v12 </span>= *(</span>_BYTE </span>*)(</span>i </span>+</span> 48</span> *</span> v11 </span>+</span> v3 </span>+</span> 6</span>) ^</span> a2</span>[(</span>unsigned</span> __int8</span>)</span>byte_424E52</span>[</span>i</span>] +</span> 31</span>];</span></span>
      v13 </span>=</span> v28</span>;</span></span>
      *(&</span>v36 </span>+</span> i</span>) =</span> v12</span>;</span></span>
      if</span> ( !</span>v5 </span>)</span></span>
        v13 </span>=</span> v4</span>;</span></span>
      v14 </span>= *(</span>_BYTE </span>*)(</span>i </span>+</span> 48</span> *</span> v13 </span>+</span> v3 </span>+</span> 7</span>) ^</span> a2</span>[(</span>unsigned</span> __int8</span>)</span>byte_424E53</span>[</span>i</span>] +</span> 31</span>];</span></span>
      v15 </span>=</span> v28</span>;</span></span>
      v38</span>[</span>i </span>-</span> 1</span>] =</span> v14</span>;</span></span>
      if</span> ( !</span>v5 </span>)</span></span>
        v15 </span>=</span> v4</span>;</span></span>
      v16 </span>= *(</span>_BYTE </span>*)(</span>i </span>+</span> 48</span> *</span> v15 </span>+</span> v3 </span>+</span> 8</span>) ^</span> a2</span>[(</span>unsigned</span> __int8</span>)</span>byte_424E54</span>[</span>i</span>] +</span> 31</span>];</span></span>
      v17 </span>=</span> v28</span>;</span></span>
      v38</span>[</span>i</span>] =</span> v16</span>;</span></span>
      if</span> ( !</span>v5 </span>)</span></span>
        v17 </span>=</span> v4</span>;</span></span>
      v18 </span>= *(</span>_BYTE </span>*)(</span>i </span>+</span> 48</span> *</span> v17 </span>+</span> v3 </span>+</span> 9</span>) ^</span> a2</span>[(</span>unsigned</span> __int8</span>)</span>byte_424E55</span>[</span>i</span>] +</span> 31</span>];</span></span>
      i </span>+=</span> 6</span>;</span></span>
    }</span></span>
    v32</span>[</span>1</span>] = *(</span>int</span> *)((</span>char</span> *)&</span>dword_424E80</span></span>
                    + (((</span>unsigned</span> __int8</span>)</span>v38</span>[</span>0</span>] +</span> 2</span>) | (</span>32</span> *</span> v34 </span>+</span> 2</span>) | (</span>16</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>1</span>] +</span> 2</span>) | (</span>8</span> *</span> v35 </span>+</span> 2</span>) | (</span>4</span> *</span> v36 </span>+</span> 2</span>) | (</span>2</span> *</span> v37 </span>+</span> 2</span>)));</span></span>
    v32</span>[</span>2</span>] = *(</span>int</span> *)((</span>char</span> *)&</span>dword_424F80</span></span>
                    + (((</span>unsigned</span> __int8</span>)</span>v38</span>[</span>6</span>] +</span> 2</span>) | (</span>32</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>2</span>] +</span> 2</span>) | (</span>16</span></span>
                                                                                           * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>7</span>]</span></span>
                                                                                           +</span> 2</span>) | (</span>8</span></span>
                                                                                                 * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>3</span>]</span></span>
                                                                                                 +</span> 2</span>) | (</span>4</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>4</span>] +</span> 2</span>) | (</span>2</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>5</span>] +</span> 2</span>)));</span></span>
    v32</span>[</span>3</span>] = *(</span>int</span> *)((</span>char</span> *)&</span>dword_425080</span></span>
                    + (((</span>unsigned</span> __int8</span>)</span>v38</span>[</span>12</span>] +</span> 2</span>) | (</span>32</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>8</span>] +</span> 2</span>) | (</span>16</span></span>
                                                                                            * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>13</span>]</span></span>
                                                                                            +</span> 2</span>) | (</span>8</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>9</span>]</span></span>
                                                                                                  +</span> 2</span>) | (</span>4</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>10</span>] +</span> 2</span>) | (</span>2</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>11</span>] +</span> 2</span>)));</span></span>
    v32</span>[</span>4</span>] = *(</span>int</span> *)((</span>char</span> *)&</span>dword_425180</span></span>
                    + (((</span>unsigned</span> __int8</span>)</span>v38</span>[</span>18</span>] +</span> 2</span>) | (</span>32</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>14</span>] +</span> 2</span>) | (</span>16</span></span>
                                                                                             * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>19</span>]</span></span>
                                                                                             +</span> 2</span>) | (</span>8</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>15</span>] +</span> 2</span>) | (</span>4</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>16</span>] +</span> 2</span>) | (</span>2</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>17</span>] +</span> 2</span>)));</span></span>
    v32</span>[</span>5</span>] = *(</span>int</span> *)((</span>char</span> *)&</span>dword_425280</span></span>
                    + (((</span>unsigned</span> __int8</span>)</span>v38</span>[</span>24</span>] +</span> 2</span>) | (</span>32</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>20</span>] +</span> 2</span>) | (</span>16</span></span>
                                                                                             * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>25</span>]</span></span>
                                                                                             +</span> 2</span>) | (</span>8</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>21</span>] +</span> 2</span>) | (</span>4</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>22</span>] +</span> 2</span>) | (</span>2</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>23</span>] +</span> 2</span>)));</span></span>
    v32</span>[</span>6</span>] = *(</span>int</span> *)((</span>char</span> *)&</span>dword_425380</span></span>
                    + (((</span>unsigned</span> __int8</span>)</span>v38</span>[</span>30</span>] +</span> 2</span>) | (</span>32</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>26</span>] +</span> 2</span>) | (</span>16</span></span>
                                                                                             * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>31</span>]</span></span>
                                                                                             +</span> 2</span>) | (</span>8</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>27</span>] +</span> 2</span>) | (</span>4</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>28</span>] +</span> 2</span>) | (</span>2</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>29</span>] +</span> 2</span>)));</span></span>
    v32</span>[</span>7</span>] = *(</span>int</span> *)((</span>char</span> *)&</span>dword_425480</span></span>
                    + (((</span>unsigned</span> __int8</span>)</span>v38</span>[</span>36</span>] +</span> 2</span>) | (</span>32</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>32</span>] +</span> 2</span>) | (</span>16</span></span>
                                                                                             * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>37</span>]</span></span>
                                                                                             +</span> 2</span>) | (</span>8</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>33</span>] +</span> 2</span>) | (</span>4</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>34</span>] +</span> 2</span>) | (</span>2</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>35</span>] +</span> 2</span>)));</span></span>
    v19 </span>= (</span>char</span> *)(&</span>unk_425681 </span>- (</span>_UNKNOWN </span>*)</span>a2</span>);</span></span>
    v20 </span>= &</span>unk_425680 </span>- (</span>_UNKNOWN </span>*)</span>a2</span>;</span></span>
    v33 </span>= *(</span>int</span> *)((</span>char</span> *)&</span>dword_425580</span></span>
                 + (((</span>unsigned</span> __int8</span>)</span>v38</span>[</span>42</span>] +</span> 2</span>) | (</span>32</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>38</span>] +</span> 2</span>) | (</span>16</span></span>
                                                                                          * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>43</span>]</span></span>
                                                                                          +</span> 2</span>) | (</span>8</span></span>
                                                                                                * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>39</span>]</span></span>
                                                                                                +</span> 2</span>) | (</span>4</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>40</span>] +</span> 2</span>) | (</span>2</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>41</span>] +</span> 2</span>)));</span></span>
    result </span>=</span> a2</span>;</span></span>
    if</span> (</span> v4 </span><=</span> 0</span> )</span></span>
    {</span></span>
      v30 </span>=</span> 8</span>;</span></span>
      do</span></span>
      {</span></span>
        *</span>result </span>^= *((</span>_BYTE </span>*)</span>v32 </span>+ (</span>unsigned</span> __int8</span>)</span>result</span>[</span>v20</span>] +</span> 3</span>);</span></span>
        result</span>[</span>1</span>] ^= *((</span>_BYTE </span>*)</span>v32 </span>+ (</span>unsigned</span> __int8</span>)</span>v19</span>[(</span>_DWORD</span>)</span>result</span>] +</span> 3</span>);</span></span>
        result</span>[</span>2</span>] ^= *((</span>_BYTE </span>*)</span>v32 </span>+ (</span>unsigned</span> __int8</span>)</span>result</span>[&</span>unk_425682 </span>- (</span>_UNKNOWN </span>*)</span>a2</span>] +</span> 3</span>);</span></span>
        result</span>[</span>3</span>] ^= *((</span>_BYTE </span>*)</span>v32 </span>+ (</span>unsigned</span> __int8</span>)</span>result</span>[</span>byte_425683 </span>-</span> a2</span>] +</span> 3</span>);</span></span>
        result </span>+=</span> 4</span>;</span></span>
        --</span>v30</span>;</span></span>
      }</span></span>
      while</span> (</span> v30 </span>);</span></span>
    }</span></span>
    else</span></span>
    {</span></span>
      v29 </span>=</span> 8</span>;</span></span>
      do</span></span>
      {</span></span>
        v24 </span>=</span> result</span>[</span>32</span>];</span></span>
        v22 </span>= *</span>result </span>^ *((</span>_BYTE </span>*)</span>v32 </span>+ (</span>unsigned</span> __int8</span>)</span>result</span>[</span>v20</span>] +</span> 3</span>);</span></span>
        result </span>+=</span> 4</span>;</span></span>
        result</span>[</span>28</span>] =</span> v22</span>;</span></span>
        *(</span>result </span>-</span> 4</span>) =</span> v24</span>;</span></span>
        v25 </span>=</span> result</span>[</span>29</span>];</span></span>
        result</span>[</span>29</span>] = *(</span>result </span>-</span> 3</span>) ^ *((</span>_BYTE </span>*)</span>v32 </span>+ (</span>unsigned</span> __int8</span>)</span>result</span>[(</span>_DWORD</span>)</span>v19 </span>-</span> 4</span>] +</span> 3</span>);</span></span>
        *(</span>result </span>-</span> 3</span>) =</span> v25</span>;</span></span>
        v26 </span>=</span> result</span>[</span>30</span>];</span></span>
        result</span>[</span>30</span>] = *(</span>result </span>-</span> 2</span>) ^ *((</span>_BYTE </span>*)</span>v32 </span>+ (</span>unsigned</span> __int8</span>)</span>result</span>[&</span>unk_425682 </span>- (</span>_UNKNOWN </span>*)</span>a2 </span>-</span> 4</span>] +</span> 3</span>);</span></span>
        *(</span>result </span>-</span> 2</span>) =</span> v26</span>;</span></span>
        v27 </span>=</span> result</span>[</span>31</span>];</span></span>
        result</span>[</span>31</span>] = *(</span>result </span>-</span> 1</span>) ^ *((</span>_BYTE </span>*)</span>v32 </span>+ (</span>unsigned</span> __int8</span>)</span>result</span>[</span>byte_425683 </span>-</span> a2 </span>-</span> 4</span>] +</span> 3</span>);</span></span>
        *(</span>result </span>-</span> 1</span>) =</span> v27</span>;</span></span>
        --</span>v29</span>;</span></span>
      }</span></span>
      while</span> (</span> v29 </span>);</span></span>
    }</span></span>
    v5 </span>=</span> a3</span>;</span></span>
    v3 </span>=</span> v32</span>[</span>0</span>];</span></span>
    v4 </span>=</span> v31 </span>-</span> 1</span>;</span></span>
    v23 </span>=</span> v31 </span>-</span> 1</span> <= -</span>1</span>;</span></span>
    ++</span>v28</span>;</span></span>
    --</span>v31</span>;</span></span>
  }</span></span>
  while</span> ( !</span>v23 </span>);</span></span>
  return</span> result</span>;</span></span>
}</span></span></code></pre>
Oof. This is substantially more complicated but looks like the meat of the decryption algorithm. We'll refer to this function, sub_407980</code>, as decrypt_data</code> from here on out. We can see what may be an immediate roadblock: this function takes in a C++ this</code> pointer (line 5) and performs bitwise operations on one of its members (line 18, 23, etc.). For now let's call this class member key</code> and come back to it later.</p>
This function is the perfect example of decompilers emitting less than ideal code as a result of compiler optimizations/code reordering. For me, TTD was essential for following how data flows through this function. It took a few hours of banging my head against IDA and WinDbg to understand, but this function can be broken up into 3 high-level phases:</p>

Building a 48-byte buffer containing our key material XOR'd with data from a static table.</li>
</ol>
 1</span>  int</span> v33</span>;</span></span>
 2</span>  unsigned</span> __int8 v34</span>;</span> // [esp+44h] [ebp-34h]</span></span>
 3</span>  unsigned</span> __int8 v35</span>;</span> // [esp+45h] [ebp-33h]</span></span>
 4</span>  unsigned</span> __int8 v36</span>;</span> // [esp+46h] [ebp-32h]</span></span>
 5</span>  unsigned</span> __int8 v37</span>;</span> // [esp+47h] [ebp-31h]</span></span>
 6</span>  char</span> v38</span>[</span>44</span>];</span> // [esp+48h] [ebp-30h]</span></span>
 7</span></span>
 8</span>  v3 </span>= (</span>int</span>)</span>this</span>;</span></span>
 9</span>  v4 </span>=</span> 15</span>;</span></span>
10</span>  v5 </span>=</span> a3</span>;</span></span>
11</span>  v32</span>[</span>0</span>] = (</span>int</span>)</span>this</span>;</span></span>
12</span>  v28 </span>=</span> 0</span>;</span></span>
13</span>  v31 </span>=</span> 15</span>;</span></span>
14</span>  do</span></span>
15</span>  {</span></span>
16</span>    // The end statement of this loop is strange -- it's writing a byte somewhere? come back</span></span>
17</span>    // to this later</span></span>
18</span>    for</span> (</span> i </span>=</span> 0</span>;</span> i </span><</span> 48</span>; *((</span>_BYTE </span>*)&</span>v33 </span>+</span> i </span>+</span> 3</span>) =</span> v18 </span>)</span></span>
19</span>    {</span></span>
20</span>    // v28 Starts at 0 but is incremented by 1 during each iteration of the outer `while` loop</span></span>
21</span>      v7 </span>=</span> v28</span>;</span></span>
22</span>      // v5 is our last argument which was 0</span></span>
23</span>      if</span> ( !</span>v5 </span>)</span></span>
24</span>        // overwrite v7 with v4, which begins at 15 but is decremented by 1 during each iteration</span></span>
25</span>        // of the outer `while` loop</span></span>
26</span>        v7 </span>=</span> v4</span>;</span></span>
27</span>      // left-hand side of the xor, *(_BYTE *)(i + 48 * v7 + v3 + 4)</span></span>
28</span>      //     v3 in this context is our `this` pointer + 4, giving us *(_BYTE *)(i + (48 * v7) + this->maybe_key)</span></span>
29</span>      //     so the left-hand side of the xor is likely indexing into our key material:</span></span>
30</span>      //     this->maybe_key[i + 48 * loop_multiplier]</span></span>
31</span>      //</span></span>
32</span>      // right-hand side of the xor, a2[(unsigned __int8)byte_424E50[i] + 31]</span></span>
33</span>      //     a2 is our input encrypted data, and byte_424E50 is some static data</span></span>
34</span>      //</span></span>
35</span>      // this full statement can be rewritten as:</span></span>
36</span>      //     v8 = this->maybe_key[i + 48 * loop_multiplier] ^ encrypted_data[byte_424E50[i] + 31]</span></span>
37</span>      v8 </span>= *(</span>_BYTE </span>*)(</span>i </span>+</span> 48</span> *</span> v7 </span>+</span> v3 </span>+</span> 4</span>) ^</span> a2</span>[(</span>unsigned</span> __int8</span>)</span>byte_424E50</span>[</span>i</span>] +</span> 31</span>];</span></span>
38</span></span>
39</span>      v9 </span>=</span> v28</span>;</span></span>
40</span></span>
41</span>      // write the result of `key_data ^ input_data` to a scratch buffer (v34)</span></span>
42</span>      // v34 looks to be declared as the wrong type. v33 is actually a 52-byte buffer</span></span>
43</span>      *(&</span>v34 </span>+</span> i</span>) =</span> v8</span>;</span></span>
44</span></span>
45</span>      // repeat the above 5 more times</span></span>
46</span>      if</span> ( !</span>v5 </span>)</span></span>
47</span>        v9 </span>=</span> v4</span>;</span></span>
48</span>      v10 </span>= *(</span>_BYTE </span>*)(</span>i </span>+</span> 48</span> *</span> v9 </span>+</span> v3 </span>+</span> 5</span>) ^</span> a2</span>[(</span>unsigned</span> __int8</span>)</span>byte_424E51</span>[</span>i</span>] +</span> 31</span>];</span></span>
49</span>      v11 </span>=</span> v28</span>;</span></span>
50</span>      *(&</span>v35 </span>+</span> i</span>) =</span> v10</span>;</span></span>
51</span></span>
52</span>      // snip</span></span>
53</span></span>
54</span>      // v18 gets written to the scratch buffer at the end of the loop...</span></span>
55</span>      v18 </span>= *(</span>_BYTE </span>*)(</span>i </span>+</span> 48</span> *</span> v17 </span>+</span> v3 </span>+</span> 9</span>) ^</span> a2</span>[(</span>unsigned</span> __int8</span>)</span>byte_424E55</span>[</span>i</span>] +</span> 31</span>];</span></span>
56</span></span>
57</span>      // this was probably the *real* last statement of the for-loop</span></span>
58</span>      // i.e. for (int i = 0; i < 48; i += 6)</span></span>
59</span>      i </span>+=</span> 6</span>;</span></span>
60</span>    }</span></span></code></pre>

Build a 32-byte buffer containing data from an 0x800-byte static table, with indexes into this table originating from indices built from the buffer in step #1. Combine this 32-byte buffer with the 48-byte buffer in step #1.</li>
</ol>
    // dword_424E80 -- some static data</span></span>
    // (unsigned __int8)v38[0] + 2) -- the original decompiler output has this wrong.</span></span>
    //     v33 should be a 52-byte buffer which consumes v38, so v38 is actually data set up in</span></span>
    //     the loop above.</span></span>
    // (32 * v34 + 2) -- v34 should be some data from the above loop as well. This looks like</span></span>
    //     a binary shift optimization</span></span>
    // repeat with different multipliers...</span></span>
    //</span></span>
    // This can be simplified as:</span></span>
    //     size_t index  = ((v34 << 5) + 2)</span></span>
    //                     | ((v37[1] << 4) + 2)</span></span>
    //                     | ((v35 << 3) + 2)</span></span>
    //                     | ((v36 << 2) + 2)</span></span>
    //                     | ((v37 << 1) + 2)</span></span>
    //                     | v38[0]</span></span>
    //     v32[1] = *(int*)(((char*)&dword_424e80)[index])</span></span>
    v32</span>[</span>1</span>] = *(</span>int</span> *)((</span>char</span> *)&</span>dword_424E80</span></span>
                    + (((</span>unsigned</span> __int8</span>)</span>v38</span>[</span>0</span>] +</span> 2</span>) | (</span>32</span> *</span> v34 </span>+</span> 2</span>) | (</span>16</span> * (</span>unsigned</span> __int8</span>)</span>v38</span>[</span>1</span>] +</span> 2</span>) | (</span>8</span> *</span> v35 </span>+</span> 2</span>) | (</span>4</span> *</span> v36 </span>+</span> 2</span>) | (</span>2</span> *</span> v37 </span>+</span> 2</span>)));</span></span>
    // repeat 7 times. each time the reference to dword_424e80 is shifted forward by 0x100.</span></span>
    // note: if you do the math, the next line uses dword_424e80[64]. We shift by 0x100 instead of</span></span>
    // 64 because is misleading because dword_424e80 is declared as an int array -- not a char array.</span></span></code></pre>

Iterate over the next 8 bytes of the output buffer. For each byte index of the output buffer, index into yet another</em> static 32-byte buffer and use that as the index into the table from step #2. XOR this value with the value at the current index of the output buffer.</li>
</ol>
// Not really sure why this calculation works like this. It ends up just being `unk_425681`'s address</span></span>
// when it's used.</span></span>
    v19 </span>= (</span>char</span> *)(&</span>unk_425681 </span>- (</span>_UNKNOWN </span>*)</span>a2</span>);</span></span>
    v20 </span>= &</span>unk_425680 </span>- (</span>_UNKNOWN </span>*)</span>a2</span>;</span></span>
</span>
// v4 is a number that's decremented on every iteration -- possibly bytes remaining?</span></span>
    if</span> (</span> v4 </span><=</span> 0</span> )</span></span>
    {</span></span>
        // Loop over 8 bytes</span></span>
      v30 </span>=</span> 8</span>;</span></span>
      do</span></span>
      {</span></span>
        // Start XORing the output bytes with some of the data generated in step 2.</span></span>
        //</span></span>
        // Cheating here and doing the "draw the rest of the owl", but if you observe that</span></span>
        // we use `unk_425680` (v20), `unk_425681` (v19), `unk_425682`, and byte_425683, the</span></span>
        // the decompiler generated suboptimal code. We can simplify to be relative to just</span></span>
        // `unk_425680`</span></span>
        //</span></span>
        // *result ^= step2_bytes[unk_425680[output_index] - 1]</span></span>
        *</span>result </span>^= *((</span>_BYTE </span>*)</span>v32 </span>+ (</span>unsigned</span> __int8</span>)</span>result</span>[</span>v20</span>] +</span> 3</span>);</span></span>
</span>
        // result[1] ^= step2_bytes[unk_425680[output_index] + 1]</span></span>
        result</span>[</span>1</span>] ^= *((</span>_BYTE </span>*)</span>v32 </span>+ (</span>unsigned</span> __int8</span>)</span>v19</span>[(</span>_DWORD</span>)</span>result</span>] +</span> 3</span>);</span></span>
</span>
        // result[2] ^= step2_bytes[unk_425680[output_index] + 2]</span></span>
        result</span>[</span>2</span>] ^= *((</span>_BYTE </span>*)</span>v32 </span>+ (</span>unsigned</span> __int8</span>)</span>result</span>[&</span>unk_425682 </span>- (</span>_UNKNOWN </span>*)</span>a2</span>] +</span> 3</span>);</span></span>
</span>
        // result[3] ^= step2_bytes[unk_425680[output_index] + 3]</span></span>
        result</span>[</span>3</span>] ^= *((</span>_BYTE </span>*)</span>v32 </span>+ (</span>unsigned</span> __int8</span>)</span>result</span>[</span>byte_425683 </span>-</span> a2</span>] +</span> 3</span>);</span></span>
        // Move our our pointer to the output buffer forward by 4 bytes</span></span>
        result </span>+=</span> 4</span>;</span></span>
        --</span>v30</span>;</span></span>
      }</span></span>
      while</span> (</span> v30 </span>);</span></span>
    }</span></span>
    else</span></span>
    {</span></span>
        // loop over 8 bytes</span></span>
      v29 </span>=</span> 8</span>;</span></span>
      do</span></span>
      {</span></span>
        // grab the byte at 0x20, we're swapping this later</span></span>
        v24 </span>=</span> result</span>[</span>32</span>];</span></span>
</span>
        // v22 = *result ^ step2_bytes[unk_425680[output_index] - 1]</span></span>
        v22 </span>= *</span>result </span>^ *((</span>_BYTE </span>*)</span>v32 </span>+ (</span>unsigned</span> __int8</span>)</span>result</span>[</span>v20</span>] +</span> 3</span>);</span></span>
</span>
        // I'm not sure why the output buffer pointer is incremented here, but</span></span>
        // this really makes the code ugly</span></span>
        result </span>+=</span> 4</span>;</span></span>
</span>
        // Write the byte generated above to offset 0x1c</span></span>
        result</span>[</span>28</span>] =</span> v22</span>;</span></span>
        // Write the byte at 0x20 to offset 0</span></span>
        *(</span>result </span>-</span> 4</span>) =</span> v24</span>;</span></span>
</span>
        // rinse, repeat with slightly different offsets each time...</span></span>
        v25 </span>=</span> result</span>[</span>29</span>];</span></span>
        result</span>[</span>29</span>] = *(</span>result </span>-</span> 3</span>) ^ *((</span>_BYTE </span>*)</span>v32 </span>+ (</span>unsigned</span> __int8</span>)</span>result</span>[(</span>_DWORD</span>)</span>v19 </span>-</span> 4</span>] +</span> 3</span>);</span></span>
        *(</span>result </span>-</span> 3</span>) =</span> v25</span>;</span></span>
        v26 </span>=</span> result</span>[</span>30</span>];</span></span>
        result</span>[</span>30</span>] = *(</span>result </span>-</span> 2</span>) ^ *((</span>_BYTE </span>*)</span>v32 </span>+ (</span>unsigned</span> __int8</span>)</span>result</span>[&</span>unk_425682 </span>- (</span>_UNKNOWN </span>*)</span>a2 </span>-</span> 4</span>] +</span> 3</span>);</span></span>
        *(</span>result </span>-</span> 2</span>) =</span> v26</span>;</span></span>
        v27 </span>=</span> result</span>[</span>31</span>];</span></span>
        result</span>[</span>31</span>] = *(</span>result </span>-</span> 1</span>) ^ *((</span>_BYTE </span>*)</span>v32 </span>+ (</span>unsigned</span> __int8</span>)</span>result</span>[</span>byte_425683 </span>-</span> a2 </span>-</span> 4</span>] +</span> 3</span>);</span></span>
        *(</span>result </span>-</span> 1</span>) =</span> v27</span>;</span></span>
        --</span>v29</span>;</span></span>
      }</span></span>
      while</span> (</span> v29 </span>);</span></span>
    }</span></span></code></pre>
The inner loop in the else</code> branch above I think is kind of nasty, so here it is reimplemented in Rust:</p>
for</span> _</span> in 0</span>..</span>8</span> {</span></span>
    // we swap the `first` index with the `second`</span></span>
    for</span> (</span>first</span>,</span> second</span>)</span> in</span> (</span>0x1c</span>..=</span>0x1f</span>).</span>zip</span>(</span>0</span>..</span>4</span>) {</span></span>
        let</span> original_byte_idx</span> =</span> first</span> +</span> output_offset</span> +</span> 4</span>;</span></span>
</span>
        let</span> original_byte</span> =</span> outbuf</span>[</span>original_byte_idx</span>];</span></span>
</span>
        let</span> constant</span> =</span> unk_425680</span>[</span>output_offset</span> +</span> second</span>]</span> as</span> usize</span>;</span></span>
</span>
        let</span> new_byte</span> =</span> outbuf</span>[</span>output_offset</span> +</span> second</span>] ^</span> generated_bytes_from_step2</span>[</span>constant</span> -</span> 1</span>];</span></span>
</span>
        let</span> new_idx</span> =</span> original_byte_idx</span>;</span></span>
        outbuf</span>[</span>new_idx</span>] =</span> new_byte</span>;</span></span>
        outbuf</span>[</span>output_offset</span> +</span> second</span>] =</span> original_byte</span>;</span></span>
    }</span></span>
</span>
    output_offset</span> +=</span> 4</span>;</span></span>
}</span></span></code></pre>#</a
>
Key Setup</h3>
We now need to figure out how our key is set up for usage in the decrypt_data</code> function above. My approach here is to set a breakpoint at the first instruction to use the key data in decrypt_data</code>, which happens to be xor bl, [ecx + esi + 4]</code> at 0x4079d3</code>. I know this is where we should break because in the decompiler output the left-hand side of the XOR operation, the key material, will be the second</em> operand in the xor</code> instruction. As a reminder, the decompiler shows the XOR as:</p>
v8 </span>= *(</span>_BYTE </span>*)(</span>i </span>+</span> 48</span> *</span> v7 </span>+</span> v3 </span>+</span> 4</span>) ^</span> a2</span>[(</span>unsigned</span> __int8</span>)</span>byte_424E50</span>[</span>i</span>] +</span> 31</span>];</span></span></code></pre>
The breakpoint is hit and the address we're loading from is 0x19f5c4</code>. We can now lean on TTD to help us figure out where this data was last written. Set a 1-byte memory write breakpoint at this address using ba w1 0x19f5c4</code> and press the Go Back</code> button. If you've never used TTD before, this operates exactly as Go</code> would except backwards</em> in the program's trace. In this case it will execute backward until either a breakpoint is hit, interrupt is generated, or we reach the start of the program.</p>
Our memory write breakpoint gets triggered at 0x4078fb</code> -- a function we haven't seen before. The callstack shows that it's called not terribly far from the decrypt_update_info</code> routine!</p>

set_key</code> (we are here -- function is originally called sub_407850</code>)</li>
sub_4082c0</code></li>
decrypt_update_info</code></li>
</ul>
What's sub_4082c0</code>?</p>



    
</a>
Not a lot to see here except the same function called 4 times, initially with the timestamp string as an argument in position 0, a 64-byte buffer, and bunch of function calls using the return value of the last as its input. The function our debugger just broke into takes only 1 argument, which is the 64-byte buffer used across all</em> of these function calls. So what's going on in sub_407e80</code>?</p>



    
</a>
The bitwise operations that look supsiciously similar to the byte to bit inflation we saw above with the firmware data. After renaming things and performing some loop unrolling, things look like this:</p>
// sub_407850</span></span>
int</span> inflate_timestamp</span>(</span>void</span> *</span>this</span>,</span> char</span> *</span>timestamp_str</span>,</span> char</span> *</span>output</span>,</span> uint8_t</span> *</span>key</span>) {</span></span>
    for</span> (</span>size_t</span> output_idx </span>=</span> 0</span>;</span> output_idx </span><</span> 8</span>;</span> output_idx</span>++) {</span></span>
        uint8_t</span> ts_byte </span>= *</span>timestamp_str</span>;</span></span>
        if</span> (</span>ts_byte</span>) {</span></span>
            timestamp_str </span>+=</span> 1</span>;</span></span>
        }</span></span>
</span>
        for</span> (</span>int</span> bit_idx </span>=</span> 0</span>;</span> bit_idx </span><</span> 8</span>;</span> bit_idx</span>++) {</span></span>
            uint8_t</span> bit_value </span>= (</span>ts_byte </span>>> (</span>7</span> -</span> bit_idx</span>)) &</span> 1</span>;</span></span>
            output</span>[(</span>output_idx </span>*</span> 8</span>) +</span> bit_idx</span>] ^=</span> bit_value</span>;</span></span>
        }</span></span>
    }</span></span>
</span>
    set_key</span>(</span>this</span>,</span> key</span>);</span></span>
    decrypt_data</span>(</span>this</span>,</span> output</span>,</span> 1</span>);</span></span>
</span>
    return</span> timestamp_str</span>;</span></span>
}</span></span>
</span>
// sub_4082c0</span></span>
int</span> set_key_to_timestamp</span>(</span>void</span> *</span>this</span>,</span> char</span> *</span>timestamp_str</span>) {</span></span>
    uint8_t</span> key_buf</span>[</span>64</span>];</span></span>
    memset</span>(&</span>key_buf</span>,</span> 0</span>, sizeof(</span>key_buf</span>));</span></span>
</span>
    char</span> *</span>str_ptr </span>=</span> inflate_timestamp</span>(</span>this</span>,</span> timestamp_str</span>, &</span>key_buf</span>, &</span>static_key_1</span>);</span></span>
    str_ptr </span>=</span> inflate_timestamp</span>(</span>this</span>,</span> str_ptr</span>, &</span>key_buf</span>, &</span>static_key_2</span>);</span></span>
    str_ptr </span>=</span> inflate_timestamp</span>(</span>this</span>,</span> str_ptr</span>, &</span>key_buf</span>, &</span>static_key_3</span>);</span></span>
    inflate_timestamp</span>(</span>this</span>,</span> str_ptr</span>, &</span>key_buf</span>, &</span>static_key_4</span>);</span></span>
</span>
    set_key</span>(</span>this</span>, &</span>key_buf</span>);</span></span>
}</span></span></code></pre>
The only mystery now is the set_key</code> routine:</p>
int</span> __thiscall </span>set_key</span>(</span>char</span> *</span>this</span>,</span> const void</span> *</span>a2</span>)</span></span>
{</span></span>
  _DWORD </span>*</span>v2</span>;</span> // ebp</span></span>
  char</span> *</span>v3</span>;</span> // edx</span></span>
  char</span> v4</span>;</span> // al</span></span>
  char</span> v5</span>;</span> // al</span></span>
  char</span> v6</span>;</span> // al</span></span>
  char</span> v7</span>;</span> // al</span></span>
  int</span> result</span>;</span> // eax</span></span>
  char</span> v10</span>[</span>56</span>];</span> // [esp+Ch] [ebp-3Ch] BYREF</span></span>
</span>
  qmemcpy</span>(</span>v10</span>,</span> a2</span>, sizeof(</span>v10</span>));</span></span>
  v2 </span>= &</span>unk_424DE0</span>;</span></span>
  v3 </span>=</span> this </span>+</span> 5</span>;</span></span>
  do</span></span>
  {</span></span>
    v4 </span>=</span> v10</span>[</span>0</span>];</span></span>
    qmemcpy</span>(</span>v10</span>, &</span>v10</span>[</span>1</span>],</span> 0x1Bu</span>);</span></span>
    v10</span>[</span>27</span>] =</span> v4</span>;</span></span>
    v5 </span>=</span> v10</span>[</span>28</span>];</span></span>
    qmemcpy</span>(&</span>v10</span>[</span>28</span>], &</span>v10</span>[</span>29</span>],</span> 0x1Bu</span>);</span></span>
    v10</span>[</span>55</span>] =</span> v5</span>;</span></span>
    if</span> ( *</span>v2 </span>==</span> 2</span> )</span></span>
    {</span></span>
      v6 </span>=</span> v10</span>[</span>0</span>];</span></span>
      qmemcpy</span>(</span>v10</span>, &</span>v10</span>[</span>1</span>],</span> 0x1Bu</span>);</span></span>
      v10</span>[</span>27</span>] =</span> v6</span>;</span></span>
      v7 </span>=</span> v10</span>[</span>28</span>];</span></span>
      qmemcpy</span>(&</span>v10</span>[</span>28</span>], &</span>v10</span>[</span>29</span>],</span> 0x1Bu</span>);</span></span>
      v10</span>[</span>55</span>] =</span> v7</span>;</span></span>
    }</span></span>
    for</span> (</span> result </span>=</span> 0</span>;</span> result </span><</span> 48</span>;</span> result </span>+=</span> 6</span> )</span></span>
    {</span></span>
      v3</span>[</span>result </span>-</span> 1</span>] =</span> v10</span>[(</span>unsigned</span> __int8</span>)</span>byte_424E20</span>[</span>result</span>] -</span> 1</span>];</span></span>
      v3</span>[</span>result</span>] =</span> v10</span>[(</span>unsigned</span> __int8</span>)</span>byte_424E21</span>[</span>result</span>] -</span> 1</span>];</span></span>
      v3</span>[</span>result </span>+</span> 1</span>] =</span> v10</span>[(</span>unsigned</span> __int8</span>)</span>byte_424E22</span>[</span>result</span>] -</span> 1</span>];</span></span>
      v3</span>[</span>result </span>+</span> 2</span>] =</span> v10</span>[(</span>unsigned</span> __int8</span>)</span>byte_424E23</span>[</span>result</span>] -</span> 1</span>];</span></span>
      v3</span>[</span>result </span>+</span> 3</span>] =</span> v10</span>[(</span>unsigned</span> __int8</span>)</span>byte_424E24</span>[</span>result</span>] -</span> 1</span>];</span></span>
      v3</span>[</span>result </span>+</span> 4</span>] =</span> v10</span>[(</span>unsigned</span> __int8</span>)</span>byte_424E25</span>[</span>result</span>] -</span> 1</span>];</span></span>
    }</span></span>
    ++</span>v2</span>;</span></span>
    v3 </span>+=</span> 48</span>;</span></span>
  }</span></span>
  while</span> ( (</span>int</span>)</span>v2 </span>< (</span>int</span>)</span>byte_424E20 </span>);</span></span>
  return</span> result</span>;</span></span>
}</span></span></code></pre>
This function is a bit more straightforward to reimplement:</p>
void</span> set_key</span>(</span>void</span> *</span>this</span>,</span> uint8_t</span> *</span>key</span>) {</span></span>
    uint8_t</span> scrambled_key</span>[</span>56</span>];</span></span>
    memcpy</span>(&</span>scrambled_key</span>,</span> key</span>, sizeof(</span>scrambled_key</span>));</span></span>
</span>
    for</span> (</span>size_t</span> i </span>=</span> 0</span>;</span> i </span><</span> 16</span>;</span> i</span>++) {</span></span>
        size_t</span> swap_rounds </span>=</span> 1</span>;</span></span>
        if</span> (((</span>uint32_t</span>*)</span>GLOBAL_KEY_ROUNDS_CONFIG</span>)[</span>i</span>] ==</span> 2</span>) {</span></span>
            swap_rounds </span>=</span> 2</span>;</span></span>
        }</span></span>
</span>
        for</span> (</span>int</span> i </span>=</span> 0</span>;</span> i </span><</span> swap_rounds</span>;</span> i</span>++) {</span></span>
            uint8_t</span> temp </span>=</span> scrambled_key</span>[</span>0</span>];</span></span>
            memcpy</span>(&</span>scrambled_key</span>, &</span>scrambled_key</span>[</span>1</span>],</span> 27</span>);</span></span>
            scrambled_key</span>[</span>27</span>] =</span> temp</span>;</span></span>
</span>
            temp </span>=</span> scrambled_key</span>[</span>28</span>];</span></span>
            memcpy</span>(&</span>scrambled_key</span>[</span>28</span>], &</span>scrambled_key</span>[</span>29</span>],</span> 27</span>);</span></span>
            scrambled_key</span>[</span>55</span>] =</span> temp</span>;</span></span>
        }</span></span>
</span>
        for</span> (</span>size_t</span> swap_idx </span>=</span> 0</span>;</span> swap_idx </span><</span> 48</span>;</span> swap_idx</span>++) {</span></span>
            size_t</span> scrambled_key_idx </span>=</span> GLOBAL_KEY_SWAP_TABLE</span>[</span>swap_idx</span>] -</span> 1</span>;</span></span>
</span>
            size_t</span> persistent_key_idx </span>=</span> swap_idx </span>+ (</span>i </span>*</span> 48</span>);</span></span>
            this</span>-></span>key</span>[</span>persistent_key_idx</span>] =</span> scrambled_key</span>[</span>scrambled_key_idx</span>];</span></span>
        }</span></span>
    }</span></span>
}</span></span></code></pre>#</a
>
Putting Everything Together</h3>

Update data is read from resources</li>
The first 4 bytes of the update data are a Unix timestamp</li>
The timestamp is formatted as a string, has each byte inflated to its bit representation, and decrypted using some static key material as the key. This is repeated 4 times with the output of the previous run used as an input to the next.</li>
The resulting data from step 3 is used as a key for decrypting data.</li>
The remainder of the firmware update image is inflated to its bit representation 8 bytes at a time and uses the dynamic key and 3 other unique static lookup tables to transform the inflated input data.</li>
The result from step 5 is deflated back into its byte</em> representation.</li>
</ol>
My decryption utility which completely reimplements this magic in Rust can be found at https://github.com/landaire/porkchop</a>.</p>
#</a
>
Loading the Firmware in IDA Pro</h2>
IDA thankfully supports disassembling the Hitachi/Rensas H8SX architecture. If we load our firmware into IDA and select the "Hitachi H8SX advanced" processsor type, use the default options for the "Disassembly memory organization" dialog, then finally choose "H8S/2215R" in the "Choose the device name" dialog...:</p>



    
</a>
We don't have shit. I'm not an embedded systems expert, but my friend suggested that the first few DWORDs look like they may belong to a vector table. If we right-click address 0 and select "Double word 0x142A", we can click on the new variable unk_142A</code> to go to its location. Press C</code> at this location to define it as Code, then press P</code> to create a function at this address:</p>



    
</a>
We can now reverse engineer our firmware :)</p>

Finding a CSRF vulnerability in phpBB

CVE-2016-1902: Symfony SecureRandom

`#</a > Timeline:</h2> Dec 14, 2015: Vulnerability identified and reported</li> Dec 30, 2015: No response, requested follow-up</li> Jan 14, 2016: Patch released for 2.3, 2.6, 2.7</li> Jan 18, 2016: Security advisory published</li> </ul>`

lander's posts

Reverse Engineering Binaries With AI

A File Format Uncracked for 20 Years

Saving My Commit With `jj evolog`

Writing a PE Loader for the Xbox in 2024

MITMing the Xbox 360 Dashboard for Fun and RCE

On Dependency Usage in Rust

Deobfuscating World of Warships' Python Scripts

One Weird Trick to Improve Bug Finding With ASAN

Reverse Engineering Yaesu FT-70D Firmware Encryption

Apple ImageIO Denial of Service

lander's posts

Reverse Engineering Binaries With AI

A File Format Uncracked for 20 Years

Saving My Commit With `jj evolog`

Writing a PE Loader for the Xbox in 2024

#</a > TODO</h2> There are still some remaining items for the loader:</p> Ensure that GetModuleHandle(NULL)</code> (handle to self) works correctly.</li> Maybe load .NET binaries? We already have a technique for launching .NET code, but having an all-in-one solution might be nice.</li>

MITMing the Xbox 360 Dashboard for Fun and RCE

On Dependency Usage in Rust

Deobfuscating World of Warships' Python Scripts

One Weird Trick to Improve Bug Finding With ASAN

Reverse Engineering Yaesu FT-70D Firmware Encryption

Apple ImageIO Denial of Service

Finding a CSRF vulnerability in phpBB

CVE-2016-1902: Symfony SecureRandom