So, you want to start reverse engineering MikroTik routers. Where do you start? As opposed to many routers which act more as a collection of independent binaries for each service, MikroTik devices implement a system of interconnected binaries which perform tasks for one another. Unfortunately, there is limited publicly available information about how this system-wide implementation works, and the good, technical information available is now a few years old. In that time, MikroTik released a number of minor version updates and one major revision software upgrade, making some of the technical details obsolete.
Consequently, we are left generally in the dark as to how MikroTik works, and digging into its dense, hand-rolled C++ binaries filled with custom library calls is a daunting task.
This blog post, which overviews our presentation at REcon 2022, outlines key knowledge and introduces tools that we created during our research over the past handful of months.
The goal of that talk, and this post, is to refresh the publicly available MikroTik knowledge and provide a crash course on MikroTik internals that will bring you from potentially zero experience to a point where you are familiar and comfortable with key MikroTik concepts and abstractions.
This knowledge will jump-start your research, tool development, or whatever MikroTik-related tinkering in which you are interested. Let’s get started!
Overview
We approach our overarching goal in four ways: first, we dive into MikroTik’s RouterOS operating system, understanding how it loads firmware and boots processes. Specifically, we focus on how firmware packages are cryptographically signed, and how we can bypass signing to obtain a developer shell in a MikroTik RouterOS virtual machine. Next, we focus on a key concept central to MikroTik: its proprietary messaging protocol used for IPC. Third, we dive into how we can authenticate to different services, specifically reviewing a proprietary, hand-rolled cryptographic protocol used for multiple publicly exposed services. Finally, we introduce a novel post-authentication jailbreak for MikroTik devices running v6 firmware (current long-term release branch) that pops a shell on any virtual or physical device.
Diving Deep into RouterOS Internals
NPK Files and Backdoors
Unlike some IoT devices which frustratingly require intercepting software downloads or extracting firmware directly from hardware, MikroTik hosts its proprietary firmware on its software downloads page. This conveniently allows us to investigate firmware components and understand how RouterOS, MikroTik’s customized operating system, is organized. Opening up the file system, we see the following components:
/flash/rw/{disk, logs, tmp, store...}– writable region/lib– core libraries/nova/bin– system binaries/nova/lib– system libraries/nova/etc– system configuration/pckg/{name}/nova/{bin, lib, etc}– package data
RouterOS software is distributed in .npk files (which we think stands for “nova package”). In recent versions of RouterOS, each NPK file contains a squashfs with the package data along with a cryptographic signature that RouterOS verifies during installation and every reboot.
While RouterOS is locked down – meaning we cannot easily get a developer shell – there is a known backdoor that has existed for a long time. Specifically, if we login as the devel user with the admin password and have the option package installed, RouterOS launches /pckg/option/bin/bash instead of the default restricted shell.
That is exactly what we want for security research! However, there are two problems:
- The
optionpackage does not exist outside of MikroTik offices (of course…) - Packages are signed, which means we cannot easily construct our own
optionpackage
In some previous versions of RouterOS it was possible to leverage known CVEs to “install” the option package post-boot and enable this developer backdoor. See Jacob Baines’s Cleaner Wrasse program for an example of an automated tool that accomplishes this.
However, since version 6.44 (2019), this tool no longer works and we need a different strategy…
Bypassing Signature Validation
Since we are hackers with unfettered access to the RouterOS firmware, let’s go straight to the source and figure out how RouterOS actually validates packages. After a bit of poking around, we discover that package validation occurs in the init binary, a part of the compressed initrd.rgz file located in the disk image’s boot sector.
Luckily for us, the init binary invokes a single function to validate each package. We can find this function by looking for a reference to the %s/flash/var/pdb/%s/disabled string. In pseudo-code, the function works as follows:
int check_signature(...){
// magic
snprintf(buf, 0x80, "%s/flash/var/pdb/%s/disabled");
return is_valid;
}All we need to do to bypass signature validation is find this function and patch it to return 1 every time. However, we run into a problem when we try to recompress this and patch our original initrd.rgz…
It turns out that the kernel is very finicky about what initrd.rgz looks like. Specifically, we need to make sure that we match the expected size (both compressed and decompressed) and also the exact position in the disk image. If we do not match these properties then the kernel fails to decompress initrd.rgz and the router fails to boot.
The “Entropy Trick”
To solve the first two constraints, we make use of an “entropy trick.” Specifically, we create a dummy file, pad, inside our initrd directory and adjust its size to match the decompressed size of the original initrd directory. Then, by adjusting the amount of entropy in the file, we control the compressed size of initrd.rgz:
For example, if pad contains all zeros (low entropy), its compressed size is small. On the other hand, if pad contains all random bytes (high entropy), its compressed size is large. As long as our target size falls between these two extremes, we can perform a binary search on the ratio of zeros to random bytes in order to exactly match the original compressed size.
Ctrl+H
Unfortunately, if we now mount the filesystem in the boot image and copy over our modified initrd.rgz file, the kernel still won’t boot. This is because the kernel expects that initrd.rgz resides at a very specific location in the boot image. When we mount the filesystem and copy the file, it adjusts the position of the actual data. This problem is relatively easy to fix; we can simply do a find-and-replace for every 512 byte sector of the original initrd.rgz and swap it with our modified initrd.rgz. This strategy effectively operates on the raw bytes in the disk image instead of mounting the boot sector as a filesystem.
Unlocking the Backdoor
Now that we have successfully patched out signature validation, we are free to install a fake option package (with an invalid signature) and enable our persistent developer backdoor!
It is also helpful to include busybox in the option package for reverse engineering research, since recent versions of RouterOS do not actually ship with any standard /bin tools.
Once we reboot, we can simply telnet -l devel and provide the admin password to get a familiar bash shell!
RouterOS IPC
Nova Messages
MikroTik designs RouterOS in a very modular fashion. The operating system contains more than 80 processes which communicate with each other through internal messages, and each process is generally responsible for one specific feature. For example, the user binary handles authentication for all other processes.
Upon boot, the init process spawns /nova/bin/loader which is RouterOS’s main control process. loader is responsible for spawning all of the other processes and managing interprocess communication. In some sense, /nova/bin/loader is “the router’s router.”
RouterOS implements the bulk of its IPC in the libumsg.so shared library. This library contains methods for serializing and deserializing messages, constructing abstraction layers to handle requests, and facilitating process-wide communication abstractions. The extensive utilization of this custom framework across RouterOS binaries is one of the things that makes RouterOS a difficult reverse engineering target.
So let’s break it down together.
We’ll start with the core player: “Nova Message” (nv::message internally). A nova message is a typed, key-value mapping. It comes in two flavors: a pseudo-JSON variant (now deprecated), and a serialized binary variant. You can recognize the binary messages because they always start with M2 in ascii:
Dissecting a Message
Reverse engineering this message protocol shows that there are six types of data, which can each exist as a single value or as a list. We include the following cheat sheet to describe the serialization format in depth, and you can also find some open-source libraries which implement this protocol.
Each nova message key is a 24-bit integer and certain keys have a special meaning inside RouterOS. For example, keys of the form 0xFFxxxx correspond to the SYS namespace and are used during message routing:
Particularly of interest are the keys for SYS_TO (destination binary), SYS_FROM (origin binary), and SYS_CMD (what operation to invoke).
Armed with this information, we can
