{'fraction_non_alphanumeric': 0.027723062738359665, 'fraction_numerical': 0.00894765516952349, 'mean_word_length': 5.074719959266803, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Compiler-Driven FPGA Virtualization with SYNERGY

FPGAs are increasingly common in modern applications, and cloud providers now support on-demand FPGA acceleration in data centers. Applications in data centers run on virtual infrastructure, where consolidation, multi-tenancy, and workload migration enable economies of scale that are fundamental to the provider's business. However, a general strategy for virtualizing FPGAs has yet to emerge. While manufacturers struggle with hardware-based approaches, we propose a compiler/runtime-based solution called Synergy. We show a compiler transformation for Verilog programs that produces code able to yield control to software at sub-clock-tick granularity according to the semantics of the original program. Synergy uses this property to efficiently support core virtualization primitives: suspend and resume, program migration, and spatial/temporal multiplexing, on hardware which is available today. We use Synergy to virtualize FPGA workloads across a cluster of Altera SoCs and Xilinx FPGAs on Amazon F1. The workloads require no modification, run within 3-4x of unvirtualized performance, and incur a modest increase in FPGA fabric utilization.


INTRODUCTION
Field-Programmable Gate Arrays (FPGAs) combine the functional efficiency of hardware with the programmability of software. FP-GAs can exceed general-purpose CPU performance by orders of magnitude [13,76] and offer lower cost and time to market than ASICs. FPGAs have become a compelling acceleration alternative for machine learning [16,80,86,100], databases [12,43,61], finance [39,58], graph processing [20,69], communication [10,38,41,76,90], and image processing [68]. In data centers, FPGAs serve diverse hardware needs with a single technology. Amazon now provides F1 instances with large FPGAs attached [23] and Microsoft deploys FPGAs in new data center construction [64].
Virtualization is fundamental to data centers. It decouples software from hardware, enabling economies of scale through consolidation. However, a standard technique for virtualizing FPGAs has yet to emerge. There are no widely agreed upon methods for supporting key primitives such as workload migration (suspending and resuming a hardware program or relocating it between FPGAs mid-execution) or multi-tenancy (multiplexing multiple hardware programs on a single FPGA). Better virtualization support is required for FPGAs to become a mainstream accelerator technology.
Virtualizing FPGAs is difficult because they lack a well-defined interposable application binary interface (ABI) and state capture primitives. On CPUs, hardware registers are restricted to a small, static set and access to data is abstracted through virtual memory, making it trivial to save and restore state. In contrast, the state of an FPGA program is distributed throughout its reprogrammable fabric in a program-and hardware-dependent fashion, making it inaccessible to the OS. Without knowing how programs are compiled for an FPGA, there is no way to share the FPGA with other programs or to relocate programs mid-execution. FPGA vendors are pursuing hardware-based solutions to enable sharing by partitioning the device into smaller, isolated fabrics. However, lacking state capture primitives, this does not solve the fundamental problem and cannot support features like workload migration.
We argue that the right place to support FPGA virtualization is in a combined compiler/runtime environment. Our system, Synergy, combines a just-in-time (JIT) runtime for Verilog, canonical interfaces to OS-managed resources, and an OS-level protection layer to abstract and isolate shared resources. The key insight behind Synergy is that a compiler can transparently re-write Verilog code to compensate for the missing ABI and explicitly expose application state to the OS. The core technique in Synergy is a static analysis to transform the user's code into a distributed-system-like intermediate representation (IR) consisting of monadic sub-programs which can be moved back and forth mid-execution between a software interpreter and native FPGA execution. This is possible because the transformations produce code that can trap to software at arbitrary execution points without violating the semantics of Verilog.
Synergy's first contribution is a set of compiler transformations to produce code that can be interrupted at sub-clock-tick granularity ( §3) according to the semantics of the original program. This allows Synergy to support a large class of unsynthesizable Verilog. Traditional Verilog uses unsynthesizable language constructs for testing and debugging in a simulator: Synergy uses these to expose interfaces to OS-managed resources and to start, stop, and save the state of a program at any point in its execution. This allows Synergy to perform context switch and workload migration without hardware support or modifications to Verilog.
Synergy's second contribution is a new technique for FPGA multi-tenancy ( §4). Synergy introduces a hypervisor layer into the compiler's runtime which can combine the sub-program representations from multiple applications into a single hardware program, which is kept hidden from those instances. This module is responsible for interleaving asynchronous data and control requests between each of those instances and the FPGA. In contrast to hardware-based approaches, manipulating each instance's state is straightforward, as the hypervisor has access to every instance's source and knows how it is mapped onto the device.
Synergy's final contribution is a compiler backend targeting an OS-level protection layer for process isolation, fair scheduling, and cross-platform compatibility ( §5). Recent OS-FPGA proposals harden vendor shells and export interfaces for the application to assist the OS with state capture for context switch [46,63]. A major obstacle to using these systems is the requirement that the developer implement those state capture interfaces. Synergy satisfies the state capture requirement transparently by using compiler analysis to identify the set of variables that comprise a program's state and emitting code to interact with state capture and quiescence interfaces. For applications which natively support such interfaces, Synergy can use these to dramatically reduce overhead for context switch and migration.
Our Synergy prototype extends the Cascade [78] JIT compiler and composes it with the AmorphOS [46] FPGA OS. We measure Synergy in real-world contexts that represent the heterogeneity of the data center. We show the ability to suspend and resume programs running on a cluster of Altera SoCs and Xilinx FPGAs running on Amazon's F1 cloud instances, to transition applications between the two, and to temporally and spatially multiplex both devices efficiently with strong OS-level isolation guarantees. This is done without exposing the architectural differences between the platforms, or requiring extensions to the Verilog language or modifications to the programs. We achieve performance within 3 − 4× of unvirtualized code with a reasonable fabric cost.

BACKGROUND
Verilog [88] is one of two standard HDLs used to program FPGAs. VHDL [6] is essentially isomorphic. Verilog consists of synthesizable and unsynthesizable constructs. Synthesizable Verilog describes computation which can be lowered onto an FPGA. Unsynthesizable Verilog includes tasks such as print statements, which are more expressive and aid in debugging, but must be executed in software.
Verilog programs are declarative and organized hierarchically in units called modules. An example Verilog module is shown in Figure 1. The interface to a module is defined in terms of its input/output ports (clock, res). Its semantics are defined in terms of arbitrary-width wires (x,y) and registers (r), logic gates (e.g. &), primitive arithmetic (e.g. +), and nested sub-modules (sm). The value of a wire is functionally determined by its inputs (lines 5, 21), whereas a register is updated at discrete intervals (lines 6,11,13). For brevity, our discussion ignores Verilog's rules of type inference (reg may be demoted to wire). Synergy does not.
Verilog supports sequential and concurrent semantics. Continuous assignments (lines 5, 21) are scheduled when the value of their right-hand-side changes. Procedural blocks (lines 9-19) are scheduled when their guard is satisfied (e.g. clock changes from 0 to 1). The ordering of these events is undefined, and their evaluation is non-atomic. Any of the statements in a fork/join block may be evaluated in any order. Only a begin/end block is guaranteed to be evaluated sequentially. Procedural blocks can contain two types of assignments to registers: blocking (=) and non-blocking (<=). Blocking assignments are executed immediately, whereas nonblocking assignments must wait until all continuous assignments or control blocks are finished.
When used idiomatically, these semantics map directly onto hardware primtives: wires appear to change value instantly and registers appear to change value with the clock. However, unsynthesizable statements have no analogue. The print statement on line 18 is non-deterministic, it can be interleaved with any assignment in lines 10-14. So too is the first execution of lines 12 and 14, which can be interleaved with the assignment on line 5. While the assignment on line 11 is visible immediately, the assignment on line 13 is only performed after every block and assignment has been scheduled, thus the value 3 only appears the second time line 10 is executed.

Cascade
Cascade is the first JIT compiler for Verilog. Using Cascade, Verilog is parsed one line at a time, added to the user's program, and its side effects made visible immediately. This can include the results of executing unsynthesizable Verilog. While JIT compilation is orthogonal to Synergy, Cascade's runtime techniques are a fundamental building block. Cascade applies transformations to the user's program that produce code which can trap into the Cascade runtime at the end of the logical clock tick. These traps are used to handle unsynthesizable statements in a way that is consistent with Verilog's scheduling semantics, even during hardware execution. Synergy improves upon this to trap into the runtime at sub-clocktick granularity according to the semantics of the original program and to enable context switch ( §3).
Cascade uses the syntax of Verilog to manage programs at the module granularity. Its IR expresses a distributed system of Verilog sub-programs, each corresponding to a single module in the user's program. A sub-program's state is represented by a data structure known as an engine. Sub-programs start as low-performance, software-simulated engines and are replaced over time by highperformance FPGA-resident engines. Cascade retains the flexibility to relocate engines by imposing a constrained ABI on its IR, mediated by messages sent over the runtime's data/control plane. Relevant to our discussion is a subset of that ABI: get/set and evaluate/update messages. The get/set messages read and write an engine's inputs, outputs, and program variables. The evaluate/update messages request that an engine run until no more continuous assigns or procedural blocks can be scheduled, and latch the result of non-blocking assignments, respectively.
Unsynthesizable traps are placed on an ordered interrupt queue and evaluated between clock ticks, when changes to engine state have fixed-pointed and the program is in a consistent state. This limits support for unsynthesizable Verilog to output-only. For example, print statements can occur at any point in a program, but their side effects are only made visible between clock-ticks. There is no way to schedule an interrupt between the statements in a begin/end block, block on the result, and continue execution. Synergy removes these limitations.

AmorphOS
AmorphOS is an FPGA runtime infrastructure which supports cross-program protection and compatibility at very high degrees of multi-tenancy. AmorphOS allows hardware programs to scale dynamically in response to FPGA load and availability and can transparently change mappings between user logic and FPGA fabric to increase utilization by avoiding fragmentation. AmorphOS extends processes with Morphlets, an abstraction for FPGA-based execution. AmorphOS can spatially share an FPGA among Morphlets from different protection domains and falls back to timesharing when space-sharing is infeasible. AmorphOS mediates OSmanaged resources through a shell-like component called a hull, which provides an isolation boundary and a compatibility layer. This enables AmorphOS to co-locate several Morphlets in a single reconfigurable zone to increase utilization without compromising security. AmorphOS leaves the problems of efficient context switch, over-subscription, and support for multiple FPGAs mostly unsolved by relying on a programmer-exposed quiescence interface and a programmer-populated compilation cache.
AmorphOS's quiescence interface forces the programmer to write state-capture code ( §1), which requires explicitly identifying live state. The interface is simple to support for request-response style programs such as DNN inference acceleration [80], but difficult, say, for a RISC core that can execute unbounded sequences of instructions. This can subject an OS-scheduler to arbitrary latency based on a program's implementation and introduces the need for forced revocation mechanisms as a fallback. Transparent state capture mechanisms which insulate the programmer from low-level details of on-fabric state are not supported.

VIRTUALIZATION PRIMITIVES
In this Section, we describe a sound transformation for Verilog that allows a program to yield control at sub-clock-tick granularity. This transformation allows Synergy to support the entire unsynthesizable Verilog standard from hardware, including $save and $restart, the two primitives which are necessary for supporting workload migration. We frame this discussion with a file IO case study. While file IO is not necessary for virtualization, it provides a clear perspective from which to understand the transformation. Moreover, supporting file IO in hardware enables a more expressive programming environment in which applications have access to OS-managed resources through canonical hardware-independent interfaces. We leave a discussion of other applications which can benefit from the ability to yield control at the sub-clock-cycle granularity (say, step-through debuggers) to future work.

Motivating Example: File I/O
Consider the program shown in Figure 2, which uses unsynthesizable IO tasks to sum the values contained in a large file. The program opens the file (line 4) and on every clock tick, attempts to read a 32-bit value (line 9). When the program reaches the endof-file, it prints the running sum and returns control to the host (lines 10-12). Otherwise, it adds the value to the running sum and continues (line 14). While this program is simple, its structure is typical of applications that perform streaming computation over large data-sets.
The key obstacle to supporting this program is that the IO tasks introduce data-dependencies within a single clock-tick. The endof-file check on line 10 depends on the result of the read operation on line 9, as does the assignment on line 14. Because the semantics of these operations involve an interaction with the file system, we always @(posedge clock) begin 9: $fread(fd, r); // TASK 1 10: if ($feof(fd)) // FEOF 1 11: $display(sum); // TASK 2 12: $finish(0); // TASK 3 13: else 14: sum <= sum + r; 15: end 16: endmodule  must not only pause the execution of the program mid-cycle while control is transferred to the host, but also block for an arbitrary amount of time until the host produces a result. Our solution is to transform the program into a state machine which implements a coroutine style semantics. While a programmer could adopt this idiom, the changes would harm both readability and maintainability.

Scheduling Transformations
Synergy uses the transformations sketched in Figure 3 to establish the invariant that all procedural logic appears in a single control statement. Any fork/join block may be replaced by an equivalent begin/end block, as the sequential semantics of the latter are a valid scheduling of the former. Also, any nested set of begin/end blocks may be flattened into a single block as there are no scheduling constraints implied by nested blocks. Next, we combine every procedural control statement in the program into a single statement called the core. The core is guarded by the union of the events that guard each individual statement. This is sound, as Verilog only allows disjunctive guards. Next, we set the body of always @(posedge __clk) always@ ( the core to a new begin/end block containing the conjunction of the bodies of each individual block. This is sound as well, as sequential execution is a valid scheduling of active procedural control statements. Finally, we guard each conjunct with a name-mangled version of its original guard (e.g. __pos_x, details below) as all of the conjuncts would otherwise be executed when the core is triggered. We note that these transformations are sound even for programs with multiple clock domains.

Control Transformations
The transformations in Figure 4 modify the control structure of the core so that it is compatible with the Cascade ABI. Recall that the Cascade ABI requires that all of the inputs to an IR sub-program including clocks will be presented as values contained in set messages which may be separated by many native clock cycles on the target device. Thus we declare state to hold the previous values of variables that appear in the core's guard, and wires that capture their semantics in the original program (e.g. __pos_x is true whenever a set message last changed x from false to true. We also declare new variables (__state and __task) to track the control state of the core, and whether a system task requires the attention of the runtime. Finally, we replace the core's guard by a posedge trigger for the native clock on the target device (__clk).

State Machine Transformations
The body of the core is lowered onto a state machine with the following semantics. States consist of as many synthesizable statements as possible and are terminated either by unsynthesizable tasks or the guard of an if or case statement. A new state is created for each branch of a conditional statement, and an SSA-style phi state is used to rejoin control flow. A compiler has flexibility in how it chooses to lower the resulting state machine onto Verilog text. Figure 5 shows one possible implementation. Each state is materialized as an if statement that performs the logic associated with the state, takes a transition, and sets the __task register if the state ended in an unsynthesizable  statement. Control enters the first state when the variable associated with the original guard (__pos_clock) evaluates to true, and continues via the fall-through semantics of Verilog until a task is triggered. When this happens, a runtime which is compatible with the Cascade ABI can take control, place its results (if any) in the appropriate hardware location, and yields back to the target device by asserting the __cont signal. When control enters the final state, the program asserts the __done signal, indicating that there is no further work to do be done. Collectively, these steps represent the compute portion of the evaluate and update requests required by the ABI.

Workload Migration
With these transformations, support for the $save and $restart system tasks is straightforward. Both can be materialized as traps triggered by the value of __task in a runtime compatible with the Cascade ABI. The former prompts the runtime to save the state of the program through a series of get requests, and the latter HW OTHER SW NET Hypervisor Engine  prompts a sequence of set requests. Either statement can be triggered in the course of normal program execution, or via an eval statement. Once a program's state is read out, it can be suspended, migrated to another physical machine if necessary, and resumed.

HYPERVISOR DESIGN
In this section we describe Synergy's support for the two primary forms of hardware multiplexing: spatial (where two programs are run simultaneously on the same fabric) and temporal (where two programs share resources using time-slice scheduling). Synergy provides an indirection layer that allows multiple runtime instances to share a compiler at the hypervisor layer. Figure 6 shows a sketch of Synergy during an execution in which two applications share a single hardware fabric. In addition to the scheduler and data/control plane introduced in § 2, we have called out the compilers associated with both the runtime instance running those applications, and the Synergy hypervisor. These compilers are responsible for lowering a sub-program onto a targetspecific engine that satisfies Cascade's distributed-system ABI.

Program Coalescing
The compiler in the runtime instance connects to the hypervisor 1 , which runs on a known port. It sends the source code for a sub-program over the connection, where it is passed to the native hardware compiler in the hypervisor, which produces a targetspecific implementation of an engine and places it on the FPGA fabric 2 . The hypervisor responds with a unique identifier representing the engine 3 and the runtime's compiler creates an engine which remains permanently in software and is configured with the unique identifier 4 . The resulting engine interacts with the runtime as usual. However, its implementation of the Cascade ABI is simply to forward requests across the network to the hypervisor 5 and block further execution until a reply is obtained.
The key idea that makes this possible is that the compiler in the hypervisor has access to the source code for every sub-program in every connected instance. This allows the compiler to support multitenancy by combining the source code for each sub-program into a single monolithic program. Whenever the text of any subprogram changes, the combined program is recompiled to support the new logic. Whenever an application finishes executing, all of its sub-programs are flagged for removal on the next recompilation. The implementation of this combined program is straightforward. The text of the sub-programs is placed in modules named after their unique hypervisor identifier. The combined program concatenates these modules together and routes ABI requests to the appropriate module based on their identifier. By isolating both sub-program code and communication, the FPGA fabric can be shared securely.
The overhead of the Synergy hypervisor depends primarily on the application. While regular communication can become a bottleneck, optimizations [78] can reduce the ABI requests between the runtime and an engine to a tolerable level. For batch-style applications, fewer than one ABI request per second is required, and we are able to achieve near-native performance even for programs separated from the hypervisor by a network connection. In contrast, applications that invoke frequent ABI calls (e.g. for file I/O) will have overheads that scale with the frequency of interaction. While our discussion presents a hypervisor which compiles all of its sub-programs to FPGA fabric, this is not fundamental. The virtualization layer nests, and it is both possible and performant for a hypervisor to delegate the compilation of a sub-program to a second hypervisor 6 , say if the device is full.

Scheduling State-Safe Compilation
The Synergy hypervisor schedules ABI requests sequentially to avoid resource contention. The one exception is compilation, which can take a very long time to complete. If compilation were serialized between ABI requests, it could render applications noninteractive. But scheduling compilation asynchronously leads to a key implementation challenge: changing the text of one instance's sub-programs requires that the entire FPGA be reprogrammed, a process which would destroy all connected instances' state. The solution is to schedule these destructive events when all connected instances are between logical clock-ticks and have saved their state. Figure 7 shows the handshake protocol used to establish these invariants. Compilation requests are scheduled asynchronously 1 , and run until they would do something destructive. The hypervisor then sends a request to every connected runtime instance 2 to schedule an interrupt between their logical clock-ticks when they are in a consistent state 3 . The interrupt causes the instances to send get requests to Synergy 4 to save their program state. When they have finished, the instances send a reply indicating it is safe to reprogram the device 5 and block until they receive an acknowledgement. Compilation proceeds after the final reply. The device is reprogrammed and the handshake finishes in the opposite fashion. The hypervisor informs the instances it is finished, they HW OTHER SW NET Hypervisor Engine  send set requests to restore their state on the target device and control proceeds as normal.

Multitenancy
Collectively, these techniques suffice to enable multitenancy. Spatial multiplexing is accomplished by combining the sub-programs from each connected runtime into a single monolithic program on the target device. Temporal multiplexing is accomplished by serializing ABI requests that involve an IO resource (say, a connection to an inmemory dataset) which is in use by another sub-program. Sharing preserves tenant protection boundaries using AmorphOS, which provides support for isolating sub-programs sharing the FPGA fabric ( §2.2).

IMPLEMENTATION
Our implementation of Synergy comprises the hypervisor described in §4, compilation passes which enable sub-clock-tick granularity support for the unsynthesizable primitives described in §3, and both Intel and AmorphOS backends.

Intel Backends
Our implementation of Synergy extends Cascade's support for the DE10 Nano SoC to the full family of Intel devices that feature reprogrammable fabric and an ARM core. This describes a range of targets, including the high-performance Stratix 10. The core feature these targets share is that they support Intel's Avalon interface for memory-mapped IO. This allows us to lower the transformations described in §3 onto a Verilog module that converts reads and writes on the Avalon memory-mapped slave interface into ABI requests. Adding support for a new Intel backend amounts to compiling this module in a hardware context which contains an Avalon memory-mapped master whose control registers are mmap'ed into the same process space as the runtime or hypervisor. Compiling the logic for these interfaces can be expensive, so Synergy augments the Intel family of backends with a compilation cache similar to the one used by AmorphOS. This allows Synergy to transition gracefully from using Cascade's JIT interface for iterative development to using fast database lookup for mature virtualized applications that require rapid transitions to hardware execution. Unlike the AmorphOS backend described below, our DE10 backend does not yet support the AmorphOS protection layer.

AmorphOS Backends
Synergy uses a similar strategy for supporting multiple AmorphOS backends. We lower the transformations described in §3 onto a Verilog module implementing the AmorphOS CntrlReg interface. The module runs as a Morphlet inside the AmorphOS hull, which provides cross-domain protection and thus preserves tenant isolation boundaries. It also enables Synergy to take advantage of the large degree of multitenancy AmorphOS offers. The Synergy hypervisor communicates with the Morphlet via a library from AmorphOS. This makes adding support for a new AmorphOS backend as simple as bringing AmorphOS up on that target.
A key difference between the DE10 and F1 is the size and speed of the reprogrammable fabric they provide. Each F1 FPGA has 10× more LUTs and operates 5× faster than a DE10. This enables Synergy to accelerate larger applications, but also makes achieving timing closure challenging. Synergy adopts two solutions. The first is to pipeline access to program variables which are modified by get/set requests. For writes, Synergy adds buffer registers between the AmorphOS hull and the variables. For reads, Synergy builds a tree with the program's variables at the leaves and the hull at the trunk. By adding buffer registers at certain branches, this logic is removed from the critical timing path. The second solution is to iteratively reduce the target device frequency until the design does meets timing. This is automated by Synergy's build scripts, which can also preserve synthesis, placement, and routing data to help offset the cost of performing multiple compiles.

Quiescence Interface
AmorphOS provides a quiescence interface that notifies applications when they will lose access to the FPGA (e.g. during reconfiguration), allowing them to quiesce and back up their state accordingly. Table 1: Benchmarks were chosen to represent a mix of batch-and streaming-style computation (marked ★).

Name Description
adpcm Pulse-code modulation encoder/decoder bitcoin Bitcoin mining accelerator df Double-precision arithmetic circuits mips32 Bubble-sort on a 32-bit MIPS processor nw ★ DNA sequence alignment regex ★ Streaming regular expression matcher Synergy supports this interface by handling the implementation of execution control and state management for developers. By default, all program variables are considered non_volatile, and will be saved and restored automatically.
For applications that implement quiescence, Synergy introduces an optional, non-standard $yield task, shown in Figure 8. Developers can assert $yield to signal that the program has entered an application-specific consistent state. When present, Synergy will only perform state-safe compilations at the end of a logical clock tick in which $yield was asserted. The use of $yield causes stateful program variables to be considered volatile by default. Volatile variables are ignored by state-safe compilations, making it is the user's responsibility to restore or reset their values at the beginning of each logical clock tick following an invocation of $yield. Users may override this behavior by annotating a variable as non_volatile.

EVALUATION
We evaluated Synergy using a combination of Altera DE10 SoCs and Amazon F1 cloud instances. The DE10s consist of a Cyclone V device [36] with an 800 MHz dual core ARM processor, reprogrammable fabric of 110K LUTs, 50 MHz clock, and 1 GB of shared DDR3 memory. Synergy's DE10 backend was configured to generate bitstreams using Intel's Quartus Lite Compiler and to interact with the DE10s' FPGA fabric via a soft-IP implementation of an Avalon Memory-Mapped master. The F1 cloud instances [23] support multiple Xilinx UltraScale+ VU9Ps running at 250 MHz and four 16 GB DDR4 channels. Synergy's F1 backend was configured to use build tools adapted from the F1 toolchain and to communicate with the instances' FPGA fabric over PCIe. Table 1 summarizes the benchmarks used in our evaluation, a combination of batch and streaming computations. The ability to handle file IO directly from hardware made the latter easy to support, as developing these benchmarks amounted to repurposing testbench code designed for functional debugging. Benchmarks were compiled prior to running experiments to prime Synergy's bitstream caches. This was appropriate as Synergy's goal is to provide virtualization support for applications which have spent sufficient time in the compile-test-debug cycle to converge on a stable design.
We find that Synergy improves upon Cascade's performance. Despite targeting a 5× higher frequency on F1, implementing a more complex program transformation, and accounting for device frequency overheads, it still achieves a virtual clock frequency [78]  within 3 − 4× of native unvirtualized performance and maintains a reasonable fabric cost. While non-negligible, these figures do not represent a lower-bound, and we expect further engineering to reduce them considerably. Figure 9 plots the performance of bitcoin as it is moved back and forth between software and hardware on two different target architectures. This workflow is typical of suspend and resume style virtualization. The application combines a block of data with a nonce, applies several rounds of SHA-256 hashing, and loops until it discovers a nonce that produces a hash under a target value.

Workload Migration
The application begins execution in a new instance of Synergy and, after running briefly in software, transitions to hardware execution on a DE10 ( = 5) where it achieves a peak throughput of 16M nonces evaluated per second. At ( = 15) we emit a signal which causes the instance to evaluate a $save task. Control then transitions temporarily to software as the runtime evacuates the program's state. The application's throughput drops significantly during this window, but quickly returns to steady-state as control returns to hardware ( = 22). Synergy is then terminated ( = 30), and similar process is initiated on an F1 instance ( = 39). In this case, the instance evaluates a restart task to restore the context which was saved on the DE10 ( = 50). Due to the larger, higher performance hardware on F1, the program achieves a higher throughput (83M), but suffers from higher performance degradation during the restart as it takes longer to reconfigure. Figure 10 plots the performance of a single-cycle 32-bit MIPS processor consisting of registers, a datapath, and on-chip memory. The CPU repeatedly randomizes and sorts an in-memory array, with execution transitioning between two FPGAs. The workload is typical of long-running batch computations which are coalesced to improve data center utilization.
The curves show two different execution contexts: one where the program is migrated between nodes in a cluster of DE10s, and one where it is migrated between F1 instances. The timing of key events is synchronized to highlight the differences between the environments. In both cases control begins in software and transitions shortly thereafter to hardware ( = 2, 4) where the targets achieve throughputs of 14M and 41M instructions per second, respectively. At ( = 15) we emit a signal which causes both contexts to evaluate $save/$restart tasks as the program is moved between FPGAs. A short time later ( = 20), performance returns to peak. Compared to the previous example and the same experiment run on some of the other benchmarks, the performance degradation during hardware/software transitions is more pronounced for mips32, with the virtual frequency temporarily lowering to 2K on F1. This is partially due to the large amount of state which must be managed by get/set requests compared to other benchmarks (the state of a MIPS processor consists of its registers, data memory, and instruction memory). Figure 11 plots the performance of two streaming-style computations on a DE10. Both read inputs from data files that are too large to store on-chip. The first (regex) reads in characters and generates statistics on the stream using a regular expression matching algorithm. The second (nw) reads in DNA sequences and evaluates how well they match using a tile-based alignment algorithm.

Multitenancy
The regular expression matcher begins execution in a new instance of Synergy and, at time ( = 10), transitions to hardware where it achieves a peak throughput of 500, 000 reads per second. At ( = 15), the sequence aligner begins execution in a second instance of Synergy. For the next few seconds, the performance of the matcher is unaffected. At ( = 24), the aligner transitions to hardware and the hypervisor is forced to temporally multiplex the execution of both applications, as they now contend on a common IO path between software and hardware. During the period where both applications are active ( = 24 − 60), the matcher's throughput drops to slightly less than 50%. This is due to the hypervisor's use of round-robin scheduling and the fact that the primitive read operations performed by the matcher (characters) require less time to run to completion than the primitive read operations performed by the aligner (strings). At ( = 60), the sequence aligner completes execution, and the throughput for the matcher returns to its peak value shortly thereafter. Compared to previous examples, the time required to transition between performance regimes is slightly more pronounced. This is due to Synergy's use of adaptive refinement [78] to determine the time spent in hardware execution before yielding control back to the REPL. It takes several seconds after the aligner finishes execution for Cascade to adjust back to a schedule which achieves peak throughput while also maintaining interactivity. Figure 12 plots the performance of some batch-style computations on an F1 instance. The first two applications read small inputs sets and transition to long-running computation before returning a result. The former (df) performs double-precision floating-point computations characteristic of numeric simulations, and the latter (bitcoin) is the miner described in §6.1. Compared to the previous examples, the results are unremarkable. Without resource contention, the hypervisor is able to run both in parallel. The applications begin software execution in separate instances of Synergy ( = 0, 22) and after transitioning to hardware ( = 2, 24) achieve a virtual clock rate [78] of 83 MHz. At ( = 42), another batch-style application that encodes and decodes audio data (adpcm) begins execution in a new instance of Synergy. While the hypervisor can run this application in parallel with the first two, lowering its application logic onto the F1 instance causes the resulting design to no longer meet timing at the peak frequency of 250 MHz. To accommodate all three applications, the global clock is set to 125 MHz, reducing their virtual clock frequencies to 41 MHz. The Synergy hypervisor hides the number of applications running simultaneously from the user. As a result, this can lead to unexpected performance regressions in our prototype. Future work can address this by running each application in an appropriate clock domain, with clock-crossing logic added automatically as needed.

Quiescence
Saving and restoring large volumes of state not only degrades reconfiguration performance ( Figure 10) but also requires a large amount of device-side resources to implement ( § 6.4). Synergy's quiescence interface allows developers to signal when a program is quiescent and which variables are stateful at that time. We found that most of our benchmarks had a large number of volatile variables, including 99%, 96%, and 71% of df's, bitcoin's, and mips32's state. For these applications, implementing quiescence resulted in an average LUT and FF savings of 50% and 15%, respectively. In our other benchmarks, 1/8 to 1/4 of the state was volatile. Implementing quiescence for them resulted in an average LUT and FF savings of 2% and 9%, respectively.

Overheads
There are two major sources of overheads in programs constructed by Synergy. The first are discrete, non-fundamental overheads resulting from how programs are virtualized in hardware in the Synergy prototype. Implementing the semantics of the original program with the ability to pause execution in the middle of a virtual clock cycle involves toggling the virtual clock variable, evaluating relevant program logic, and latching variable assignments. When these are done in separate hardware cycles, there is a minimum 3× performance overhead. This is an artifact of our implementation rather than a fundamental requirement and can be improved with future work on target-specific backends.
The second source of overheads comes from the state access and execution control logic added by Synergy, which has a lessobvious impact on designs targeting FPGA hardware. To evaluate these overheads, we compile our benchmarks under a number of different conditions and measure the resource usage and achieved device frequency. We perform these compilations using Vivado on F1, target AmorphOS's maximum frequency (250 MHz), and use the reported delay to determine the frequency achieved.
As a baseline, we compile our benchmarks natively on Amor-phOS, providing an upper bound on resource and frequency overheads. We also simulate a Cascade on AmorphOS baseline by compiling our benchmarks without system tasks, which avoids overheads introduced by our new state machine transformations. Finally, we modified our benchmarks to implement the quiescence protocol, allowing us to estimate the savings of exposing reconfiguration to developers and establishing a lower bound on state access overhead. Due to the complexity of fully rewriting our benchmarks for these cases, we only focus on replicating overheads and not functionality. Figures 13 and 14 show Synergy's FF and LUT usage is generally 2 − 4× and 1 − 6× native, respectively. For adpcm and mips32, the results exceed the height of the graph and have been labeled with the appropriate values. These two benchmarks are exceptional due to their use of large on-chip RAMs, which Vivado implements using FFs under Synergy instead of LUTRAMs. Creating RAMs out of FFs can also require additional LUTs to implement muxing logic. The adpcm* and mips32* results compare against Amor-phOS using FFs for RAMs and show that Synergy's overheads are resonable under these conditions. Future work should enable state access transformations that preserve the memories in user designs, which would eliminate this problem. Overall, we find that Synergy's overheads are similar to Cascade's and that using quiescence annotations can provide savings of up to ∼2×. Figure 15 shows that Synergy does not reduce the design's operating frequency in most cases. However, adpcm is an exception, likely due to its use of system tasks from inside its complex control   logic, which makes execution control much more expensive to implement. We see that Synergy's frequency overhead for mips32 is almost entirely due to forcing the use of FFs to implement RAMs. When normalized against AmorphOS using FFs (AOS FF), Synergy was less than 6% slower despite supporting full state capture. We find that for nw, Synergy and the Cascade baseline achieve higher frequencies than native, likely due to nw's complexity creating a higher-than-normal volatility in compiler outcomes. When combined with the previous 3× overhead, we find that Synergy's overall execution overhead is within 3 − 4× that of native.
While not shown in the data above, we tried compiling adpcm and nw with an anti-congestion strategy to see if it would help with their complex designs. We found that this improved both their frequencies under Synergy by 47%. With quiescence annotations, adpcm still improved 23% and nw, 37%. Applying the same strategy to nw under AOS only gave an improvement of 26%. This indicates that optimizing compiler strategies could be a great avenue for offsetting the costs of code transformations.

LIMITATIONS
Source Code. Synergy guarantees isolation between runtime instances with respect to hardware execution. However information leak between distrustful parties through source code and side channels is still possible. Leaks through sharing source could be mitigated with cryptographically secure channels, but Synergy would still function as a trusted party.
Compilation Cache. Synergy's backends rely on compilation caches to reduce overhead in production environments, as the alternative would be to transition back to software and wait through recompilation on virtualization events. Synergy uses techniques such as deterministic code generation to increase cache hit rates. As more applications use FPGAs, cache hit rates may drop and symmetry-breaking or speculative compilation may be needed to compensate.
FPGA compilers incur significant overhead. Synergy adapts many overhead-reduction techniques from the software domain such as eliminating redundant re-compilation [29,74,89], distributed build caching [2,25,32], and JIT compilation [87]. FPGA compilation can be further improved with a virtualization layer. Overlay-based virtualization [7,42,48,49,52,95] abstracts away target-specific details and enables fast compilation and lower deployment latency. The approach reduces utilization and performance. Synergy and [78] work at this level, while AmorphOS [46] works at the application-OS boundary.
Hardware-Software Partitioning. The FPGA design cycle relies on developing a design in a high-fidelity simulator [35,98] before compiling to hardware. Simulation incurs order of magnitude slowdowns compared to hardware execution, but is necessary for debugging. Many attempts have been made to bridge this performance gap, including systems which enable migration between different speed simulators [11], higher-level languages with partitioned runtime environments [4], and OS-managed communication channels [1]. Synergy builds on the approach taken by Cascade as it does not require explicit interface changes [1,4] or hardware support [11].

CONCLUSION
FPGAs are emerging in data centers so techniques for virtualizing them are urgently needed to enable them as a practical resource for on-demand hardware acceleration. Synergy is a compiler/runtime solution that supports multi-tenancy and workload migration on hardware which is available today.

A ARTIFACT APPENDIX A.1 Abstract
This artifact appendix documents the requirements and instructions for setting up Synergy and how to reproduce the results of the experiments presented in our ASPLOS'21 paper. Code for our various backends can be found at:

A.5 Experiment Workflow
Our primary experiments consist of running several benchmarks on Synergy and demonstrating the ability to save/restore state, migrate execution across FPGAs, and share hardware resources. This can be accomplished from the command line, with directives specified dynamically through the REPL, or through scripting via the C++ library interface. Synergy tracks when these directives are executed as well as the virtual application's frequency and can print this information to the console or log it to a file.

A.6 Evaluation and Expected Results
In order to help simplify the process of running experiments, we provide several C++ programs that use Synergy's library interface to automate experiment execution. These can be found in the artifact branch of the AWS F1 repo, in the experiments folder. The process of building and running these experiments is documented at https://github.com/JoshuaLandgraf/ cascade/blob/artifact/experiments/README.md. We currently provide two versions of the experiments, one for native F1 execution, and one for software simulation. The expected results are displayed in this paper and can be compared with the performance data obtained by running the experiments.

A.7 Experiment Customization
Since Synergy is a runtime, it enables a wide variety of experiments beyond those shown in the paper. It is especially simple to look over the experiments provided and tweak them to run for longer, use different or more benchmarks, or execute more complex sequences of operations. Many of our benchmarks already have different top-level files that tweak the inputs, actions performed, or number of execution iterations. Users can also modify the provided benchmarks or run their own Verilog programs on Synergy.