Introduction

One of the projects at oscamp involved replacing the system calls in musl with function calls into the ArceOS kernel, making it possible to run Linux ELF binaries on the ArceOS unikernel. That was when I first recognized the enormous potential of this libc replacement approach. Recently, I experimented with redirecting glibc calls to run Linux ELF binaries on macOS, and named the project magiclibc.

Functions, Dynamic Linking, and System Calls

For most computer science students, their first program probably looks something like this:

1
2
3
4
5
6

#include <stdio.h>

int main() {
    printf("hello, world\n");
    return 0;
}

Here, we call the printf function to output some text. This function is part of the C standard library. If you compile the program with gcc, the standard library implementation is typically glibc, the most widely used C standard library implementation on Linux. The implementation of printf ultimately invokes a system call to write data to the terminal device. On Linux, that system call is write(2).

If you have studied operating systems, the following diagram should be easy to understand: binaries interact with one another through the platform ABI, while processes interact with the kernel through system calls.

flowchart LR
    A[printf] -- Linux ABI --> B[glibc]
    B -- write(2) --> C[Linux Kernel]

libmagic: A Mach-O glibc-compatible Library

The diagram above suggests a possibility: we could replace glibc with a Mach-O library that implements the Linux ABI (aarch64-unknown-linux-gnu on AArch64) and ultimately uses Darwin system calls, as illustrated below:

flowchart LR
    A[printf] -- Linux ABI --> B[Mach-O glibc-compatible library]
    B -- write(2) --> C[Darwin/XNU]

Implementing the entire library in assembly would indeed make this possible, but it would hardly be efficient to develop. I implemented the library using Rust’s extern "C", but the resulting binary naturally follows the Darwin ABI rather than the Linux ABI. Before entering the actual function, we therefore need an assembly “trampoline” that converts arguments from the Linux ABI to the Darwin ABI and then calls the real implementation. I call this trampoline the bridge. The actual architecture looks like this:

flowchart LR
    A[printf] -- Linux ABI --> B[libmagic bridge]
    B -- Darwin ABI --> C[libmagic implementation]
    C -- write(2) --> D[Darwin/XNU]

For functions with a fixed number of arguments, the two ABIs are still highly similar, but they differ significantly in how they handle variadic functions. For this reason, libmagic currently implements puts rather than printf. The only notable difference is that on aarch64-apple-darwin, the x18 register is reserved as a platform register, whereas it is not reserved on aarch64-unknown-linux-gnu. Although this could be addressed by compiling with -ffixed-x18, I would prefer to avoid requiring recompilation.

To make the explanation easier, let us borrow the concepts of host and guest from virtual machines. The host is the program running on macOS, while the guest is the program built for Linux. Before entering the guest, we save the host’s x18 register. When switching from the guest back to the host, we save the guest’s x18 register and restore the host’s x18; the reverse happens when switching back. This resembles switching stack frames during a function call, except that we initially need caller-saved behavior and callee-saved behavior thereafter.

Only part of the core code is shown here. For the complete implementation, see magiclibc.

The implementation of puts is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

#[unsafe(no_mangle)]
pub unsafe extern "C" fn puts(text: *const c_char) -> c_int {
    if text.is_null() {
        return -1;
    }

    let result = std::panic::catch_unwind(|| {
        // SAFETY: The ABI contract above is enforced by the trusted guest.
        let bytes = unsafe { CStr::from_ptr(text) }.to_bytes();
        let stdout = io::stdout();
        let mut stdout = stdout.lock();
        stdout.write_all(bytes)?;
        stdout.write_all(b"\n")?;
        stdout.flush()
    });

    match result {
        Ok(Ok(())) => 0,
        Ok(Err(_)) | Err(_) => -1,
    }
}

The bridge is implemented as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

.text
.p2align 2

// Locate the active context whose registered stack contains the bridge's
// current SP, then install that thread's Darwin x18. Arguments x0-x8 are
// left untouched for the eventual host function.
_magic_install_host_x18:
  adrp x9, _MAGIC_THREAD_CONTEXTS@PAGE
  add  x9, x9, _MAGIC_THREAD_CONTEXTS@PAGEOFF
  mov  x10, #64
1:
  ldar x11, [x9]
  cmp x11, #2
  b.ne 2f
  ldr x12, [x9, #8]
  cmp sp, x12
  b.lo 2f
  ldr x13, [x9, #16]
  cmp sp, x13
  b.hs 2f
  ldr x18, [x9, #24]
  ret
2:
  add x9, x9, #32
  subs x10, x10, #1
  b.ne 1b
  brk #0x18

.globl _magic_bridge_puts
_magic_bridge_puts:
  stp x18, x30, [sp, #-16]!
  bl _magic_install_host_x18
  bl _puts
  ldp x18, x30, [sp], #16
  ret

ELF Loading and Dynamic Link Redirection

macOS cannot run ELF files directly; it supports only Mach-O. We therefore need an ELF loader to handle this task. I use the elf_loader crate for this. Because it does not support loading ELF files from the filesystem on macOS, I use mmap to map the ELF file into memory and then pass it to elf_loader for loading.

Next comes the central piece: dynamic link redirection. On Linux, dynamic linking is usually implemented through the PLT, while the interpreter specified by PT_INTERP—for example, /lib/ld-linux-aarch64.so.1—populates the GOT to redirect function calls. Here, our runtime takes over the role of PT_INTERP, redirecting function entries for libc.so.6 to the corresponding bridge function entries in libmagic.dylib.

Before actually entering the guest, we also need to construct a fake initial Linux user stack like the one present before control reaches _start. It follows the Linux ABI stack layout and contains information such as argc, argv, envp, and auxv.

Once everything is ready, all that remains is to switch to the fabricated Linux user stack and enter the guest’s _start.

More

By wrapping libSystem, magiclibc also implements functionality such as pthread. The source code is available on GitHub.

Conclusion

magiclibc demonstrates that it is possible to run Linux ELF binaries on macOS without using either a virtual machine or system-call interception. However, because this approach is far less general than intercepting system calls, considerably more work would be required to turn it into a complete Linux compatibility layer.