{'fraction_non_alphanumeric': 0.037621470690890384, 'fraction_numerical': 0.012620092951485203, 'mean_word_length': 4.876295336787565, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Transkernel: An Executor for Commodity Kernels on Peripheral Cores

Modern mobile and embedded platforms see a large number of ephemeral tasks driven by background activities. In order to execute such a task, the OS kernel wakes up the platform beforehand and puts it back to sleep afterwards. In doing so, the kernel operates various IO devices and orchestrates their power state transitions. Such kernel execution phases are lengthy, having high energy cost, and yet difficult to optimize. We advocate for relieving the CPU from these kernel phases by executing them on a low-power, microcontroller-like core, dubbed peripheral core, hence leaving the CPU off. Yet, for a peripheral core to execute phases in a complex commodity kernel (e.g. Linux), existing approaches either incur high engineering effort or high runtime overhead. We take a radical approach with a new executor model called transkernel. Running on a peripheral core, a transkernel executes the binary of the commodity kernel through cross-ISA, dynamic binary translation (DBT). The transkernel translates stateful kernel code while emulating a small set of stateless kernel services; it sets a narrow, stable binary interface for emulated services; it specializes for kernel's beaten paths; it exploits ISA similarities for low DBT cost. With a concrete implementation on a heterogeneous ARM SoC, we demonstrate the feasibility and benefit of transkernel. Our result contributes a new OS structure that combines cross-ISA DBT and emulation for harnessing a heterogeneous SoC. Our result demonstrates that while cross-ISA DBT is typically used under the assumption of efficiency loss, it can be used for efficiency gain, even atop off-the-shelf hardware.


Introduction
Modern embedded platforms 1 often run a large number of ephemeral tasks. These tasks are driven by periodic or background activities. Example tasks include acquiring sensor readings, refreshing smart watch display [40], push notifications [34], and periodic data sync [88]. They drain a substantial fraction of battery life, e.g., 30% for smartphones [14,13] and smart watches [42], and almost the entire battery life of smart things for surveillance [81]. To execute an ephemeral task, a commodity OS kernel drives the whole platform out of deep sleep beforehand and puts it back to deep sleep afterwards. This procedure has been an integral part of the kernel for almost two decades, and now is a heavy burden. Recent work shows this kernel procedure can consume much more energy than the ephemeral task itself [41], up to 10× [34]. Why is the kernel procedure so inefficient? Recent studies [25,89,41] show the bottlenecks as the two kernel phases called device suspend/resume as illustrated in Figure 1. In the phases, the kernel operates a variety of IO devices (or simply devices in this paper): it invokes the respective device drivers, cleans up pending IO tasks, and ensures these devices to reach the expected power states. The device suspend/resume phases are complex, incur numerous CPU idle epochs, and are proven difficult to optimize ( §2) [89,46,43].
To relieve CPU from the inefficiency, we deem that device suspend/resume should be executed by a low-power, microcontroller-like core on the same platform. As exemplified by ARM Cortex-M, such cores have trimmed-down ISAs and simple microarchitectures. They incarnate as peripheral cores on a wide range of modern SoCs used by popular products [48,58,2]. For workloads with heavy IO and low performance demands, a peripheral core delivers much higher efficiency than the CPU [38,39,50,1].
Towards offloading a commodity kernel's phases to a peripheral core, we are challenged by i) the complex, fast-evolving kernel code [63] and ii) the peripheral core's different ISA and wimpy hardware. More specifically, device suspend/resume invokes diverse drivers and multiple kernel layers, which are difficult to transplant to the peripheral core. To maintain a single OS image over the CPU and the peripheral core, existing approaches such as a multikernel [6,56,7] require to craft a wide interface for synchronizing kernel state between the two ISAs. This interface is easily broken due to changes in the kernel's build configurations and updates to its source code.
We seek to enable the offloading with the following goals: i) tractable engineering effort with good reuse of the commodity kernel code; ii) developing and compiling the software for a peripheral core once; running the software with many builds of the kernel -generated from different configurations and source versions; iii) low runtime overhead.
We take a seemingly infeasible approach, shown in Figure 1: we make the peripheral core execute unmodified binary of a commodity kernel through cross-ISA, dynamic binary translation (DBT), a technique previously regarded as expensive [6] and was never tested on microcontroller-like cores to our knowledge.
We hence propose transkernel, a new executor model for a peripheral core. A transkernel runs a DBT engine, which translates the unmodified kernel binary for most code executed in the offloaded kernel phases; underneath the translated code, the transkernel provides a small set of emulated services, as lightweight, drop-in replacements for their counterparts in the commodity kernel. The transkernel follows four principles: i) translating stateful code while emulating stateless kernel services; ii) choosing narrow and stable interfaces for emulation; iii) specializing for beaten paths; iv) exploiting similarities between heterogeneous ISAs for DBT. By applying the model, we have built a transkernel prototype called ARK. Of an ARM-based heterogeneous SoC, ARK runs on a Cortex-M3 peripheral core alongside Linux running on the Cortex-A9 CPU. ARK demonstrates the feasibility of transkernel. ARK transparently reuses unmodified Linux kernel drivers and libraries. It depends on a narrow, stable binary interface (ABI) of only 12 kernel functions and one variable. ARK is able to execute kernel phases that invoke diverse drivers (e.g. USB and WiFi NIC) with rich functionalities (e.g. DMA and firmware loading) and sophisticated kernel services (e.g. scheduling and interrupt handling). In busy execution, ARK only incurs 2.7× overhead as compared to native kernel execution on CPU. The low cost makes ARK beneficial: ARK reduces device suspend/resume energy by 34%, which results in tangible battery life extension under real-world usage.
We make the following contributions on OS and DBT: • We present the transkernel model for empowering a peripheral core to execute phases in a commodity kernel on behalf of CPU. To the design space of OSes for heterogeneous multiprocessors, the transkernel represents a new point that combines DBT for bridging ISA gaps and emulation for catering to core asymmetry.
• Crucial to the practicality of transkernel, we contribute a new paradigm of cross-ISA DBT, in which a microcontrollerlike core dynamically translates unmodified binary built for a full-fledged CPU. We contribute new DBT optimizations that systematically exploit ISA similarities. We demonstrate that while cross-ISA DBT is typically used under the assumption of efficiency loss, it can be used for efficiency gain, even on off-the-shelf hardware.
• We report a transkernel implementation, ARK, atop a het-erogeneous ARM SoC. ARK contributes concrete designs of kernel service emulation. Its DBT engine, to our knowledge, is the first cross-ISA DBT implementation runs atop a microcontroller-like core. ARK meets our goal of tractable engineering efforts and "build once run with many". ARK offers tangible benefits of energy efficiency.

Motivations
We next discuss device suspend/resume, the major kernel bottleneck in ephemeral tasks. We argue to mitigate its inefficiency with a peripheral core. We show difficulties in known approaches, and accordingly motivate our design objectives.

Kernel execution in device suspend/resume
Expecting a long period of system inactivity, an OS kernel puts the whole platform into deep sleep: in brief, the kernel synchronizes file systems with storage, freezes all user tasks, turns off IO devices (i.e. device suspend/resume), and finally powers off the CPU. To wake up from deep sleep, the kernel performs a mirrored procedure. For a detailed description, see Linux documentation [12]. In a typical ephemeral task, the above kernel execution takes hundreds of milliseconds [29] while user work often takes tens of milliseconds [41]; this kernel execution often consumes several times more energy than the user code [34].
Problem: device suspend/resume By profiling recent Linux on multiple embedded platforms, recent work [89] shows the aforementioned kernel execution is bottlenecked by device suspend/resume, the kernel phases right before powering off the CPU and right after powering on the CPU. In the phases, the kernel cleans up pending IO tasks and drives transitions of device power states. We briefly summarize the prior findings below. i) Device suspend/resume is inefficient. It contributes 54% on average and up to 66% to the total kernel energy consumption. The transitions of device power states take long. CPU idles frequently in numerous short epochs, typically in milliseconds. ii) Devices are diverse. On a platform, the kernel often suspends and resumes tens of different devices. Across platforms, different devices incur long kernel executions. iii) Optimization is difficult. Device power state transitions are bound by slow hardware and low-speed buses, as well as physical factors (e.g. voltage ramp-up). Devices have implicit power, voltage, and clock dependencies, requiring certain power transitions to happen sequentially. Modern Linux already overlaps the transitions with best efforts [46,43]. Yet, as shown in the prior work, CPU idle still constitutes up to 68% of the duration of device suspend/resume.
Challenge: Widespread, complex code invoked Device suspend/resume crosses multiple kernel layers [63,30], from the callbacks in individual drivers (e.g. for a MMC controller), to driver libraries (e.g. the generic clock framework), to kernel libraries (e.g. for radix trees), and to kernel services (e.g. scheduler). The execution is control-heavy, with dense branches and callbacks. In recent a Linux source tree (4.4), we find over 1000 device drivers (covering almost all driver classes) implementing suspend/resume callbacks (154K SLoC), which invoke over 43K SLoC in driver libraries, 8K SLoC in kernel libraries, and 43K SLoC in kernel services.
Opportunities We identify the following kernel behaviors in device suspend/resume as opportunities. i) Beaten kernel paths A successful suspend/resume phase follows beaten paths [37], on which the kernel acquires all needed resources and encounters no failures. Off the beaten paths, the kernel handles rare events such as races between IO events, resource shortage, and hardware failures. These branches typically cancel the current suspend/resume attempt, perform diagnostics, and retry later. Compared to the beaten paths, they invoke very different kernel services, e.g. syslog. ii) Simple concurrency exists among the syscall path (which initiates the platform suspend/resume), interrupt handlers, and deferred kernel work. The concurrency is for leveraging hardware asynchrony and kernel modularity rather than exploiting multicore parallelism. iii) Low sensitivity to execution delay On embedded platforms, most ephemeral tasks are driven by background activities [34,49,14] and thus insensitive to the execution time. This contrasts to many servers for interactive user requests [90,49].
Summary: design implications We are compelled to treat device suspend/resume systematically. We face challenges that the invoked kernel code is diverse, complex, and crosslayer; we see opportunities that allow focusing on beaten kernel paths, specializing for simple concurrency, and higher efficiency at the cost of increased execution time.

A peripheral core in a heterogeneous SoC
We deem that the inefficiency of device suspend/resume can be substantially mitigated with a peripheral core on a modern heterogeneous SoC, which has the following characteristics.
Hardware model i) Asymmetric processors: The CPU and the peripheral core offer disparate performance-efficiency trade-offs. The peripheral core has no MMU but memory protection unit (MPU), which can map at most tens memory regions. It cannot run commodity OSes. ii) Heterogeneous, yet similar ISAs: The two processors have different ISAs. Many instructions from the two ISAs have similar semantics, as will be discussed in detail below. iii) Loose coupling: The two processors are located in separate power domains and can be turned on/off independently. iv) Shared platform resources: Both processors share access to system DRAM and IO devices. IO interrupts are physically wired to both processors. Many SoCs, including those in popular products (e.g. iPad Pro and Azure Sphere), fit this hardware model [77,78,59,71].
Promise of high efficiency A peripheral core is known for high efficiency for IO-heavy workloads with low performance demand [38,50,74,1,72]. It benefits the kernel's device suspend/resume in the following ways: i) The peripheral core can operate while leaving the CPU offline. ii) The idle  power of a peripheral core is often one magnitude lower than CPU [39,60], minimizing system power during numerous CPU idles. iii) Its simple microarchitecture suits kernel execution, whose irregular behaviors often see marginal benefits from higher power, advanced microarchitectures [54]. Note that a peripheral core offers much higher efficiency than a LITTLE core in ARM big.LITTLE [23], which mandate a homogeneous ISA and tight core coupling. We will examine big.LITTLE in Section 7.
ISA similarity On an SoC we target, the CPU and the peripheral core often have ISAs from the same family, e.g. ARM.
The two ISAs often implement instructions with similar semantics, despite likely in different encoding.
The common examples are SoCs integrating ARMv7a ISA and ARMv7m ISA [57,58,68,48,59]. Other families also provide their ISAs amenable to same-SoC integration, e.g. NanoMIPS and MIPS32. We believe the ISA similarities are by choice. i) For ISA designers, it is feasible to explore performance-efficiency tradeoffs within one ISA family, since the family choice is merely about instruction syntax rather than semantics [9]. The designers likely start from common instruction semantics and instantiate them differently. ii) For SoC vendors, incorporating same-family ISAs on one chip reduces the software efforts [36] as well as facilitating silicon design and ISA licensing.

Design space exploration
We next explore OS designs that empower a peripheral core to execute a commodity kernel's phases. Our challenges are i) the complex, fast-evolving kernel code and ii) the peripheral core's different ISA and wimpy hardware.
Code transplant creates fragile interfaces Given that a commodity kernel (we use Linux as the example below) cannot span heterogeneous ISAs out-of-box [39,6], one may be tempted to carve out the related source code from the Linux kernel, cross-compile the source, and run it atop a "peripheral kernel" on the peripheral core. The peripheral kernel is vital to the autonomy of the offloaded execution. This approach results in a multikernel OS [7] shown in Figure 2(a). However, the two resultant interfaces (shown as in the figure) are difficult to implement and maintain. 1 The interface between the two kernels, needed for their data dependency. Through the interface, both kernels (in heterogeneous ISAs) synchronize the Linux kernel state, e.g.  devices configurations, pending IO tasks, and locks, before and after the offloading. Whether the interface is based on messages [56,7] or software shared memory [39,6,22], it is essentially built on an agreement on the definitions of shared Linux kernel data types, including their semantics and/or memory layout. The agreement is highly fragile as the kernel data types are altered by ISA choices, kernel configurations, and the Linux kernel evolution. Table 1(a) summarizes numerous changes to the data types referenced in device suspend/resume. The efforts in building this interface is not only tedious (for keeping the data types consistent between heterogeneous ISAs) but repetitive: any data type change would break the interface and require to revise and rebuild the peripheral kernel [21,22]. 2 The interface between the transplant code and the peripheral kernel, needed for resolving the former's functional dependency on the latter. The interface is determined by the choice of transplant boundary. Common transplant boundaries include the bottom of device-specific code [21,22,76], that of driver classes [11,75], and that of driver libraries [39]. All these choices expose at least hundreds of Linux kernel functions on this interface, as summarized in Table 1(b). This is due to diverse drivers on embedded platforms and Linux's sophisticated internals. Implementing such an interface is a daunting task; maintaining the interface as Linux evolves is even more difficult. As shown in Table 1(a), the ABI of these functions significantly changes as Linux evolves [33]. One must revise and rebuild the peripheral kernel frequently to implement the updated ABI, which is a moving target. Current cross-ISA DBT is unaffordable Alternatively, one may propose to run DBT on the peripheral core for translating the Linux kernel for an entire offloaded phase, shown in Figure 2(b). DBT allows a host processor (e.g. the peripheral core) to execute instructions in a foreign guest ISA (e.g. the CPU). DBT is free of the above interface difficulties: the translated code precisely reproduces the kernel behaviors and directly operates the kernel state ( 3 ). The DBT interacts with the Linux kernel at a low, stable interface: the CPU ISA ( 4 ). However, existing cross-ISA DBT incurs high overhead [5]. The overhead is further exacerbated by that our DBT paradigm is inverse to the common one. Whereas existing cross-ISA DBT is engineered for a brawny host (e.g. an x86 desktop) serving a wimpier guest (e.g. an emulated ARM smartphone) [16,85], our DBT host, a peripheral core with simple hardware, serves a full-fledged CPU. A straightforward port of a popular DBT engine exhibits up to 25× slowdown as will be shown in Section 7. The overhead would negate any efficiency promise of hardware and result in overall efficiency loss. Furthermore, cross-ISA DBT for translating the whole Linux kernel is complex [8]. A peripheral core lacks necessary environment, e.g. multiple address spaces and POSIX support, for developing and debugging such complex software.

Design objectives for software on a peripheral core
To overcome the difficulties, we set threefold objectives. G1. Tractable engineering effort. We set to reuse much of the commodity kernel source, in particular the rich, fast-evolving drivers that are impractical to build anew. We target a simple structure for the peripheral core's software. G2. Build once, work with many. One build of the peripheral core's software should work with a commodity kernel's binaries built from a wide range of configurations and source versions. This requires the peripheral core's software to interact with the commodity kernel through a stable ABI. G3. Low overhead. The offloaded kernel phases should yield a tangible efficiency gain.

The Transkernel Model
We propose a new executor model called transkernel. Running on a peripheral core, a transkernel consists of two key components: a DBT engine for translating and executing the unmodified kernel binary; a set of emulated, minimalistic kernel services that underpins the translated kernel code. A concrete transkernel implementation targets a specific commodity kernel, e.g. Linux. To achieve G1-G3 above, the transkernel follows four principles: Translating stateful code; emulating stateless services By stateful code, we refer to the offloaded code that must share state with the kernel execution on CPU. We deem that the stateful code include device drivers, driver libraries, and a small set of kernel services. They cover the most diverse and widespread code invoked in device suspend/resume ( §2). Through translation, the transkernel reuses commodity kernel code without transplant (G1); the translated code operates kernel state without the need for a fragile ABI (G2).
We relax the semantics of the emulated services to be stateless, so that the state of these services only lives within one device suspend/resume phase. Being stateless, the emulated services do not need to synchronize state with the kernel execution on CPU. (G2) Choosing a narrow, stable interface for emulation We determine the translation/emulation boundary to be a small set of kernel functions. We ensure that the ABI of the chosen kernel   Specializing for the beaten path The transkernel only executes the beaten path of device suspend/resume; in the rare events of the execution goes off the beaten path, it transparently falls back to CPU. Rather than precisely reproducing the kernel's behaviors, the emulated services seek functional equivalence; under the same interfaces, they only implement features needed by the beaten path. This is in the spirit of kernel specialization as in library OSes [19,66,47]. It entails tractable implementation (G1).
Exploiting ISA similarities for DBT We depart from generic cross-DBT designs that bridge an arbitrary guest and an arbitrary host, but instead systematically exploit similarities in instructions semantics, register usage, and control flow transfer. This makes cross-ISA DBT affordable to the peripheral core and ultimately makes transkernel practical. (G3) Section 4 below describes how we apply the model to a concrete transkernel, in particular our translation/emulation decisions for major kernel services, and our choices of the emulation interface. Section 5 will describe our DBT design.

A Transkernel Implementation
We implement a transkernel called ARK. ARK targets an ARM SoC fitting our hardware model in Section 2.2. The CPU has the ARMv7A ISA and the peripheral core has ARMv7m which is a popular combination. The CPU runs Linux v4.4.
The offloading workflow ARK is shipped as a standalone executable for the peripheral core, accompanied by a small Linux kernel module for the CPU to transfer control to/from the peripheral core. We refer to such control transfer as handoff. Prior to a device suspend phase, the kernel shuts down all but one CPU cores, passes control to the peripheral core, and shuts down the last CPU core. Then, ARK completes the device phase to suspend the entire platform. Device resume is normally executed by ARK on the peripheral core; in case of urgent wakeup events (e.g. a user unlocking a smartphone screen), the kernel resumes on CPU with native execution.
System structure As shown in Figure 3, ARK runs a DBT  engine, a set of emulated kernel services, and a small library for managing the peripheral core's private hardware, e.g. controllers of cache and interrupt. Upon booting, ARK replicates the linear memory mappings of the Linux kernel, so that it can address the Linux kernel's memory objects, which is similar to prior systems [39,22]. To access I/O regions, ARK allocates MPU entries for them; in case of limited number of entries, ARK time-multiplexes the regions. ARK translates all device-specific code, the libraries invoked by them, and a few kernel services we deem must be stateful (summarized in Table 2 and examined below). The translated code interacts with the emulated services through a narrow, stable ABI: the emulated services serves downcalls from the translated code and makes upcalls into the translated code. The interface is illustrated as in Figure 3 and summarized in Table 2. To support concurrency in the kernel phases, ARK runs multiple DBT contexts. Each context has its own DBT state (e.g., virtual CPU registers and a stack) and independently executes DBT and the emulated services. Switch among DBT contexts is as cheap as updating the pointer to the DBT state.
ARK executes the beaten paths. Upon entering unbeaten branches pre-defined by us, e.g. blacklisted kernel functions, ARK migrates all the DBT contexts of translated code back to the CPU and continues as native execution there ( §6).

A Scheduler of DBT Contexts
ARK emulates a scheduler, which shares no state, e.g., priorities or runtime statistics, with the Linux scheduler on the CPU. Corresponding to the simple concurrency model of suspend/resume ( §2), ARK eschews reproducing Linux's preemptive multithreading but instead maintains and switches among cooperative DBT contexts: one primary context for executing the syscall path of suspend/resume, one for executing IRQ handlers ( §4.2), and multiple for deferred work ( §4.3). Managing no more than tens of contexts, ARK uses a simple, round-robin scheduling strategy. It begins the execution in the syscall context; when the syscall context blocks (e.g., by calling msleep()), ARK switches to the next ready context to execute deferred functions until they finish or block. When a hardware interrupt occurs, ARK switches to the IRQ context to execute the kernel interrupt handler ( §4.2).

Interrupt and Exception Handling
During the offloaded device phase, all interrupts are routed to the peripheral core and handled by ARK.
Kernel interrupt handlers ARK emulates a short, early stage of interrupt handling while translating the kernel code for the remainder. This is because this early stage is ISA-specific (e.g., for manipulating the interrupt stack), on which the CPU (v7a) and the peripheral core (v7m) differ. Hence, the emulated services implement a v7m-specific routine and install it as the hardware interrupt handler. Once an interrupt happens, the routine is invoked to finish the v7m-specific task and make an upcall to the kernel's ISA-neutral interrupt handling routine (listed in Table 2), from where the ARK translates the kernel to finish handling the interrupt.
Hardware interrupt controller ARK emulates the CPU's hardware interrupt controller. This is needed as the two cores have separate, heterogeneous interrupt controllers. The CPU controller's registers are unmapped in the peripheral core; upon accessing them (e.g. for masking interrupt sources) the translated code triggers faults. ARK handles the faults and operates the peripheral core's controller accordingly.
Exception: unsupported We don't expect any exception in the offloaded kernel phases. In case exception happens, ARK uses its fallback mechanism ( §6) to migrate back to CPU.

Deferred Work
In the device phase, device drivers frequently schedules functions to be executed in the future. ARK translates the Linux services that schedule the deferred work; it also translates the actual execution of the deferred work. The translation choice is because such services must be stateful: the peripheral core may need to execute deferred work created on the CPU prior to the offloading, e.g. freeing pending WiFi packets; it may defer new work until after the completion of resume.
ARK maintains dedicated DBT contexts for executing the deferred work (Section 4.1). While the Linux kernel often executes deferred work in kernel threads (daemons), our insight is that deferred work is oblivious to its execution context (e.g., a real Linux thread or a DBT context in ARK). Beyond this, ARK only has to run the deferred work that may sleep with separate DBT contexts so that they do not block other deferred work. From these DBT contexts, ARK translates the main functions of the aforementioned kernel daemons, which retrieve and invoke the deferred work. Threaded IRQ defers heavy-lifting IRQ work (bottom halves) to a kernel thread which executes the work after the hardware IRQ is handled. A threaded IRQ handler may sleep. Therefore, ARK maintains per-IRQ DBT contexts for executing these handlers. Each context makes upcalls into irq_thread() (the main function of threaded irq daemon, listed in Table 2).
ARK creates one dedicated context for executing all nonsleepable tasklets and per-workqeueue contexts for executing workitems so that one workqueue will not block others. These contexts make upcalls to the main functions of the kernel daemons (do_softirq(), worker_thread(), and run_local_timers()), translating them for retrieving and executing deferred work.

Locking
Spinlocks ARK emulates spinlocks, because their implementation is core-specific and that ARK can safely assume all spinlocks are free at handoff points: as described in early Section 4, handoff happens between one CPU core and one peripheral core, which do not hold any spinlock; all other CPU cores are offline and cannot hold spinlocks. Hence, ARK emulates spinlock acquire/release by pausing/resuming interrupt handling. This is because ARK runs on one peripheral core and the only hardware concurrency comes from interrupts.
Sleepable locks ARK translates sleepable locks (e.g., mutex, semaphore) because these locks are stateful: for example, the kernel's clock framework may hold a mutex preventing suspend/resume from concurrently changing clock configuration [52]. Furthermore, mutex's seemingly simple interface (i.e., compare and exchange in fast path) has unstable ABI and therefore unsuitable for emulation: a mutex's reference count type changes from int to long (v4.10), breaking the ABI compatibility. The translated operations on sleepable locks may invoke spinlocks or the scheduler, e.g., when updating reference counts or putting the caller to sleep, for which the translated execution makes downcalls to the emulated services.

Memory Allocation
The device phase frequently requests dynamic memory, often at granularities of tens to hundreds of bytes. By Linux design, such requests are served by the kernel slab allocator backed by a buddy system for page allocation (fast path); when the physical pages runs low, the kernel may trigger swapping or kill user processes (slow path).
ARK provides memory allocation as a stateful service. It translates the kernel code for the fast path, including the slab allocator and the buddy system. In the rare case that the allocation enters the slow path (e.g. due to low physical memory), ARK aborts offloading. With a stateful allocator, the offloaded execution can free dynamic memory allocated by during the kernel execution on CPU, and vice versa. Compare to prior work that instantiates per-kernel allocators with split physical memory [39], ARK reduces memory fragmentation and avoids tracking which processor should free what dynamic memory pieces. Our experience in Section 7 show that ARK is able to handle intensive memory allocation/free requests such as in loading firmware to a WiFi NIC.

Delays & Timekeeping
Delays ARK emulates udelay() and msleep() for busy waiting and sleeping. ARK converts the expected wait time to the hardware timer cycles on the peripheral core. ARK implements msleep() by pausing scheduling the caller context. jiffies The Linux kernel periodically updates jiffies, a global integer, as a low-overhead measure of elapsed time. By consulting the peripheral core's hardware timer, ARK directly updates the jiffies. It is thus the only shared variable on the kernel ABI that ARK depends (all others are functions).

The Cross-ISA DBT Engine
A Cross-ISA DBT Primer DBT, among its other uses [55,45,26], is a known technique allowing a host processor to execute instructions in a foreign guest ISA. In such cross-ISA DBT, the host processor runs a program called DBT engine. At run time, the engine reads in guest instructions, translates them to host instructions based on the engine's builtin translation rules, and executes these host instructions. The engine translates guest instructions in the unit of translation block -a sequence (typically tens) of guest instructions that has one entry and one or more exits. After translating a block, the engine saves the resultant host instructions to its code cache in the host memory, so that future execution of this translated block can be directed to the code cache. Design overview We build ARK atop QEMU [8], a popular, opensource cross-ISA DBT engine. ARK inherits QEMU's infrastructure but departs from its generic design which translates between arbitrary ISAs. ARK targets two well-known DBT optimizations: i) to emit as few host instructions as possible; ii) to exit from the code cache to the DBT engine as rarely as possible. We exploit the following similarities between the CPU's and the peripheral core's ISAs Beyond the similarities, the two ISAs have important discrepancies. Below, we describe our exploitation of the ISA similarities and our treatment for caveats.

Exploiting Similar Instruction Semantics
We devise translation rules with a principled approach by parsing a machine-readable, formal ISA specification recently published by ARM [67]. Our overall guideline is to map each    Table 3 summarizes ARK's translation rules for all 558 v7a instructions. Among them, 80% can be translated with identity rules, for which ARK only needs to convert instruction encoding at run time. 15% of v7a instructions have v7m counterparts but may require amendment instructions, which fortunately fall into a few categories: i) Side effects. After load/store, v7a instructions may additionally update memory content or register values (shown in Table 4, G1). ARK emits amendment instruction to emulate the extra side effect (H3). ii) Constraints on constants. The range of constants that can be encoded in a v7m instruction is often narrower (Table 4, G2). In such cases, the amendment instructions load the constant to a scratch register, operate it, and emulate any side effects (e.g. index update) the guest instruction may have. iii) Richer shift modes. v7a instructions support richer shift modes and larger shift ranges than their v7m counterparts. This is exemplified by Table 4 G1, where a v7m instruction cannot perform LSR (logic shift right) inline as its v7a counterpart. Similar to above, the amendment instructions load the operand to a scratch register and perform shift on the register.
Beyond the above, only 27 v7a instructions have no v7m counterparts, for which we manually devise translation rules.
In summary, through systematic exploitation of similar instruction semantics, ARK emits compact host code at run time.
In the example shown in Table 4, three v7a instructions are translated into seven v7m instructions by ARK, while to 27 instructions by our QEMU baseline.

Passthrough of CPU registers
General purpose registers Both the guest (v7a) and the host (v7m) have the same set (13) of general-purpose registers. In emitting a host instruction, ARK follows register allocation in the guest counterpart with best efforts (e.g., one-to-one mapping in best case, as in Table 4, G1). ARK emits much fewer host instructions than QEMU, which emulates all guest registers in memory and manipulates them with load /store. Caveats fixed The amendment host instructions operate scratch registers as exemplified by t0 in Table 4, H2-H6. However, the wimpy host faces higher register pressure, as it (v7m) has no more registers than the brawny guest (v7a). To spill some registers to memory while still reusing the guest's register allocation, we make the following tradeoff: we designate one host register as the dedicated scratch register, and emulates its guest counterpart register in memory. We pick the least used one in the guest binary as the dedicated scratch register; we experimentally determined it as R10 by analyzing the ARM Linux kernel. We find that most amendment instructions are satisfied by one scratch register; in rare cases when extra scratch registers are needed, ARK follows a common design to allocate dead registers and spill unused ones to memory.
Condition flags Both the guest and the host ISAs involve five condition flags (e.g. zero and carry) with identical semantics. The host instructions emitted by ARK hence directly set and test the host's hardware condition flags. Fortunately, most guest (v7a) instructions and their host (v7m) counterparts have identical behaviors in testing/setting flags, verified by us parsing the ISA specifications [67]. Such flag passthrough hence incurs much lower overhead than QEMU, which emulates guest CPU flags as individual variables in host memory and emits as many as 7 host instructions to manipulate emulated flags, even for each flag-setting guest instruction. This cost is especially high for the control-heavy suspend/resume, which contains extensive conditional branches ( §2). Caveats fixed Amendment host instructions may affect the hardware condition flags unexpectedly. For amendment instructions (notably comparison and testing) that must update the flags as mandated by ISA, ARK emits two host instructions to save/restore the flags in a scratch register around the execution of these amendment instructions.

Control Transfer and Stack Manipulation
Function call/return Both guest (v7a) and host (v7m) use PC (program counter) and LR (link register, storing function return address) to maintain the control flow. QEMU emulates guest PC and LR in host memory. For each guest function calls and returns, the emitted host code manipulates the emulated PC and LR. However, this is expensive since the return address, loaded from stack or the emulated LR, points to a guest address. Each function return hence causes the DBT to step in and look up the corresponding code cache address. This overhead is magnified in the control-heavy device phase.
By contrast, ARK never emits host code to emulate the guest PC or LR. For each guest function calls, the host code pushes the return-to addresses in code cache to stack and to LR; for each guest function returns, the host code loads the hardware PC with the return address (which points to code cache) popped from the stack or from the hardware LR. By doing so, ARK no longer participates in all function returns. Our optimization is inspired by same-ISA DBT [31].
Stack and SP QEMU emulates the guest stack with an array and the guest SP in the host memory. Each guest push/pop translates to multiple host instructions updating the emulated SP and the stack array. This is costly to suspend/resume, which frequently makes function calls and operates stack heavily.
The ARK avoids such expensive stack emulation by emitting host push/pop instructions to directly operate the guest stack in place. This is possible because ARK emulates the Linux kernel's virtual address space ( §4). ARK also ensures the host code generate the same stack frames as the guest would do by making amendment instructions avoid using stack, which would introduce extra stack contents. In addition, this further facilitates the migration in abort ( §6). Caveats fixed i) As the host saves on the guest stack the code cache addresses, which are meaningless to the guest CPU, upon migrating from the peripheral core (host) to the CPU (guest), the DBT rewrites all code cache addresses on stack with their corresponding guest addresses. ii) guest push/pop instruction may involve emulated registers (i.e., scratch register). ARK must emit multiple host instructions to correctly synchronize the emulated registers in memory.

Translated-to-Native Fallback
As described in Section 3, when going off the beaten paths, ARK migrates the kernel phase back to the CPU and continues as native execution, analogous to virtual-to-physical migration of VMs [82]. Migrating one DBT context is natural, as ARK passes through most CPU registers and uses the kernel stack in place ( §5.3). Yet, to migrate all active DBT contexts, ARK address the following unique challenges.
Migrate DBT contexts for deferred work After fallback, all blocked workitems should be allowed to continue their execution on the CPU. The challenge is that their enclosing DBT contexts do not have counterparts in the Linux kernel. To solve this issue, we again exploit the insight that the workitems are oblivious to their execution contexts. Upon migration, the Linux kernel creates temporary kernel threads as "receivers" for blocked workitems to execute in. Once the migrated workitems complete, the receiver threads terminate.
Migrate DBT context for interrupt If fallback happens in the execution of the ISA-neutral interrupt handler (translated), the remainder of the handler should migrate to the CPU. This challenge, again, is that ARK's interrupt context has no coun-terpart on the CPU: the interrupt never occurs to the CPU. ARK addresses this by rethrowing the interrupt as an IPI (interprocessor interrupt) from the peripheral core to the CPU; the Linux kernel uses the IPI context as the receiver for the migrated interrupt handler, and starts executing from the fallback point in the interrupt handler. ARK gives up complete transparency: if Linux checks the CPU's interrupt controller, it will find that the interrupt comes from IPI instead of an IO device.

Evaluation
We seek to answer the following questions: 1. Does ARK incur tractable engineering efforts? ( §7.2) 2. Does ARK incur low execution overhead? ( §7.3) 3. Does ARK yield energy efficiency benefit? What are the major factors impacting the benefit? ( §7.4)

Methodology
Test Platform We evaluate ARK on Pandaboard-ES Rev B1 equipped with TI OMAP4460 SoC [77]. As summarized in Table 6, it has dual ARM Cortex-A9 CPU and dual Cortex-M3. Among a variety of platforms that fit our hardware model ( §2.2), we chose this board due to its good documentation and long-time kernel support (since 2.6.11 in 2011), which allows our study of kernel ABI over a long timespan in Section 2.
As Cortex-M3 on the platform is incapable of DVFS, for fair comparison, we run both cores at their highest clock rates.  (2)(3)(4)(6)(7)(8), slab/buddy allocator (1-4,6-9), softirq (9), DMA (2,6-9), threaded IRQ (1,5,9), and firmware upload (9). We measure device suspend/resume executed by ARK on Cortex-M3 and report the measured results. We compare ARK to native Linux execution on Cortex-A9. We further compare to a baseline ARK version, of which the DBT is a straightforward v7m port of QEMU while lacking the optimizations described in Section 5. We report measurements taken with warm DBT code cache, as this reflects the real-world scenario where device suspend/resume is frequently exercised.

Analysis of engineering efforts
ARK eliminates the tedious Linux kernel transplant ( §2.3). As shown in Table 5, in our test, ARK transparently reuses     [31,20]. The result validates our principle of specializing these emulated services.
Our code analysis shows that ARK meets our goal of "build once, run with many". We verify that the ARK binary works with a variety of kernel configuration variants (including defconfig-omap4 and yes-to-all) of Linux 4.4. We also verify that ARK works with a wide range of Linux versions, from version 3.16 (2014) to 4.17 (most recent at the time of writing). This is because ARK only depends on a narrow ABI shown in Table 2; the ABI has not changed since Linux 3.16.

Measured execution characteristics
Core activity We trace core states during ARK execution. Figure 4 (a) shows the breakdown of execution time. Compared to the native execution on CPU, ARK shows the same amount of accumulated idle time but much longer (16×) busy time. The reasons are Cortex-M3's much lower clockrate (1/6 of the A9's clockrate) and ARK's execution overhead. Despite the extended busy time, ARK still yields energy benefit, as we will show below.
Memory activity We collect DRAM activities by sampling the hardware counters of the SoC's DDR controller. We observed that ARK on Cortex-M3 generates much higher average DRAM utilization (32 MB/s read and 2MB/s write) than the native execution on A9 (only 8MB/s read and 4MB/s write). We attribute this difference to M3's small (32KB) last-level cache (LLC). Throughout the test, the ARK emitted and executed around 230KB host instructions, which far exceeds the LLC capacity and likely causes thrashing. By contrast, Cortex- A9 has a much larger LLC (1MB), which absorbs most of the kernel memory access. The memory activity has a strong energy impact, as will be shown below.
Busy execution overhead Our measurement shows that ARK incurs low overhead in busy kernel execution, which includes both DBT and emulation. We report the overhead as the ratio between ARK's cycle count on Cortex-M3 to the Linux's cycle count on A9. Note that an M3 cycle length is 6× of an A9 cycle due to different clockrates.
Overall, the execution overhead is 2.7× on average (suspend: 2.9×; resume: 2.6×). Figure 5 shows for individual drivers the execution overhead, which ranges from 1.1× to 4.5×. Our DBT design and optimizations ( §5) have strong impact on the execution overhead, as shown in Figure 5. With our baseline design, the average execution overhead is 13.9×, 5.2× higher than ARK. Applying register passthrough ( §5.2) to the baseline reduces the overhead by 2.5×, to 5.5×. The remaining optimizations (e.g. control transfer) collectively reduce the overhead by additional 2×. We notice that our optimizations are less effective on drivers with very dense control transfer (e.g. USB) due to high DBT cost. As we will show below, the low overhead is crucial to overall energy benefit.
Emulated services Our profiling shows that ARK's emulated services incur low overhead. Overall, the emulated services only contribute 1% of total busy execution. i) The early, ISAspecific interrupt handling ( §4.2) takes 3.9K Cortex-M3 cycles, only 1.5-2× more cycles than the native execution. ii) Emulated workqueues ( §4.3) incurs a typical queueing delay of tens of thousands M3 cycles. The delay is longer than the native execution but does not break the deferred execution semantics. iii) For migrating one DBT context to the CPU in fallback, ARK spends around 20 us on rewriting code cache addresses on stack ( §5.3), 17 us to flush Cortex-M3's cache, and 2 us to wake up the CPU through an IPI. CPU Peripheral core Cortex A9 @ 1.2GHz Cortex M3 @ 200MHz L1:64KB + L2:1MB L1:32KB P cpu_busy/idle :630mW/80mW [79] P pc_busy/idle :17mW/1mW [80] DRAM power model: Micron LPDDR2; self-refresh (P mem_sr ): 1.3mW [51] IO average power (P io ): 5mW [87]  Methodology We model the system power based on measured hardware activities. We choose modeling because i) our test platform is a development board with unoptimized board-level power; ii) the board does not provide measurement points for DRAM power [18]. We consider the power of cores, DRAM, and IO. As summarized in Table 6, we model core power as a function of core activities with TI's modeling tool [79,80]; we model DRAM power as a function of DRAM power state and read/write activities, with Micron's modeling tool [51]. These power tools are official and have production quality. Based on prior work [87] we assume 5mW of average IO power over device suspend/resume, which reasonably penalizes ARK for its extended execution time.
• The system energy with native execution is given by: E cpu = T idle · (P cpu_idle + P mem_sr + P io ) + T busy · (P cpu_busy + P mem + P io ) • The system energy of ARK is given by: Here, all T s are elapsed time measured in native execution.
Ps are power consumptions for cores and DRAM in different power states. The DRAM's active power P mem is derived from measured activities. F captures the ratio between the frequencies of CPU and the peripheral core. C is the average measured overhead in busy execution reported earlier, as the ratio between cycle counts of ARK and the native execution. Energy saving ARK consumes 66% energy (a reduction of 34%) of the native execution, despite its longer execution time. The energy breakdown in Figure 4(b) shows that the benefit comes from two portions. i) energy reduction in busy execution: due to its low overhead (on average 2.7×), the ARK's energy efficiency in busy execution is 23% higher than the native execution. ii) ARK reduces the system idle energy to a negligible portion, since the peripheral core's idle power is 1.25% of the CPU's. Figure 4 also shows that our DBT optimizations are crucial to energy benefit. Although the baseline benefits from lower idle power, its high execution overhead ultimately leads to 5.1× energy compared to the native execution. Recommendation to architects: ARK consumes more DRAM energy than the native execution, due to Cortex-M3's tiny LLC (32KB) as describe earlier. The LLC size trades off between the core power and the DRAM power. Our result suggests that the current size is suboptimal for the offloaded kernel execution. We expect that a careful increase in the LLC size (to 64 KB or 128 KB) will significantly reduce DRAM power at the cost of moderate increase in core power.  What-if analysis We further analyze ARK's energy benefit under the changes to two major factors: the DBT overhead (ARK's behavior) and the fraction of CPU busy time in native execution (Linux's behavior). We estimate energy consumption by using the above power model and plugging in different values of the two factors. Our analysis results in in Figure 6 show two findings. i) ARK's energy benefit will be more pronounced when the kernel incurs less CPU busy time (i.e. more CPU idle). This is because ARK's efficiency advantage over CPU is higher during idle periods than during busy execution. ii) ARK's energy benefit critically depends on its DBT overhead. When the overhead drops to below 3.5×, ARK saves energy even for all-busy kernel phases; when the overhead exceeds 5.2× (the break-even point), ARK wastes energy even for 20% busy execution time, the lowest fraction observed on embedded platforms in prior work [89].
Qualitative comparison with big.LITTLE We estimate ARK saves tangible energy compared to a LITTLE core. From recent big.LITTLE characterizations [61,24], we assume that compared to the CPU on our platform, a LITTLE core would offer 1.3× energy efficiency at 70% clockrate [44]; it consumes 40 mW when idling [64]. We make a favorable assumption for LITTLE that its DRAM utilization is as low as the CPU, while in reality the utilization should be higher due to LITTLE's smaller LLC. Even with this favorable assumption for LITTLE and unfavorable hardware for ARK (tiny LLC), our estimation shows that LITTLE consumes 77% energy of the native execution, more than ARK (66%). The reason is LITTLE's idle power is 40× of Cortex-M3; its energy efficiency in busy execution is 5% lower than ARK.
Battery life extension Based on ARK's energy reduction in device suspend/resume, we project the battery life extension for ephemeral tasks reported in prior work [34]. When the ephemeral tasks are executed at 5-second intervals and the native device suspend/resume consumes 90% system energy in a wakeup cycle, ARK extends the battery life by 18% (4.3 hours per day); with 30-second task intervals and a 50% energy consumption percentage, ARK extends the battery life by 7% (1.6 hours per day). This extension is tangible compared to complementary approaches in prior work [34,87].

Related Work
OS for heterogeneous processors A multikernel OS instantiates per-processor kernels [6,39,56,3,56]. It however faces interface difficulty as described in Section 2.3. Unlike it, transkernel translates kernel code across processors using DBT. Prior systems distribute OS functions over CPU and accelerators [53,73]. The accelerators cannot operate autonomously, which is required by offloading kernel phases.
DBT DBT has been used for system emulation [8] and binary instrumentation [31,26,45,20]; DeVuyst et al. [17] uses DBT to speed up process migration. Related to transkernel, prior systems run translated user programs atop an emulated syscall interface [8,28,85]. Unlike them, transkernel translates kernel code and emulates a narrow interface inside the kernel. Prior systems use DBT to run binaries in commodity ISAs (e.g. x86) on specialized VLIW cores and hence gain efficiency [10,32,69,70]. transkernel demonstrates that DBT can gain efficiency even on off-the-shelf cores. Existing DBT engines leverage ISA similarities, e.g. between aarch32 and aarch64 [16,15]. They still fall into the classic DBT paradigm, where the host ISA is brawny and the guest ISA is wimpy (i.e., lower register pressure). With an inverse DBT paradigm, ARK addresses very different challenges. Much work is done on optimizing DBT translation rules, using optimizers [27,4] or machine learning [83]. Compared to them, ARK leverages ISA similarities and therefore reuses code optimization already present in the guest code by the guest compilers.
Kernel and drivers Prior kernel studies show rapid evolution of the Linux kernel and the interfaces between kernel layers are unstable [63,62]. This observation motivates transkernel. Extensive work transplants device drivers to a separate core [22], user space [21], or a separate VM [35]. However, the transplant code cannot operate independent of the kernel, whereas transkernel must execute autonomously.
Suspend/resume Its inefficiency raises attention for cloud servers [49,86] and mobile [34]. Drowsy [34] mitigates the inefficiency by reducing the devices involved in suspend/resume through a user/kernel co-design; Xi et al. propose to reorder devices to resume [86]. Our approach is complementary to them and requires no changes to the userspace. PowerNap [49] takes a hardware approach to speed up suspend/resume for servers. It does not treat kernel execution for operating diverse IO on embedded platforms. A kernel may put idle devices to low power modes at run time [87], which is complementary to suspend/resume that ensures all devices are off.

Conclusions
We present transkernel, a new executor model for a peripheral core to execute a commodity kernel's phases, notably device suspend/resume. The transkernel executes the kernel binary through cross-ISA DBT. It translates stateful code while emulating stateless services; it picks a stable ABI for emulation; it specializes for beaten paths; it exploits ISA similarities for DBT. We experimentally demonstrate that the approach is feasible and beneficial. The transkernel represents a new OS design point for harnessing heterogeneous SoCs.