{'fraction_non_alphanumeric': 0.0284251693109917, 'fraction_numerical': 0.0021302979237544117, 'mean_word_length': 5.185250737463127, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Future of Consumer Edge-AI Computing

Deep Learning has proliferated dramatically across consumer devices in less than a decade, but has been largely powered through the hardware acceleration within isolated devices. Nonetheless, clear signals exist that the next decade of consumer intelligence will require levels of resources, a mixing of modalities and a collaboration of devices that will demand a significant pivot beyond hardware alone. To accomplish this, we believe a new Edge-AI paradigm will be necessary for this transition to be possible in a sustainable manner, without trespassing user-privacy or hurting quality of experience.

Since their very advent, Deep Neural Networks (DNNs) have been getting deeper and wider in their attempt to be more accurate without losing generality. Simultaneously, higher accuracies have also been a result of combining multiple models (ensembles or cascades) or inventing more exotic architectures, manually or automatically, that offer higher capacity, better generalisation or fewer inductive biases [6].
More recently, there have been emerging trends in AI that are changing the computational landscape quite significantly. On the one hand, the training of hyperscale models [3] that act as foundations in latent spaces for solving a multitude of downstream tasks in one or multiple modalities has been dominating computation in cloud AI. On the other hand, as devices have been getting more capable, an increasing number of DNNs have been getting deployed on-device, many times required to run simultaneously. Furthermore, the advent of fields like Federated Learning (FL) and on-device personalisation introduce the requirement of training on-device.
Despite the forward-looking use-cases, such workloads have been pushing the compute and memory requirements to unprecedented scales ( Fig. 1), along with their data ingestion needs, a scale with which individual edge devices have not kept up with. While the consumer edge becomes increasingly populated by smart devices, these continue to operate as standalone entities in isolation from their compute environment. Therefore, there are many missed opportunities for shaping a common context to learn and perform higherlevel or fidelity tasks under a collaborative environment.
As such, there exists a gap between compute requirements and resource availability for deploying intelligence at the consumer edge, unlikely to be bridged only through traditional hardware scaling techniques. In this paper, we propose a new paradigm for organising resources at the edge when executing emerging AI-tasks. Departing from isolated devices and moving into more capable EdgeAI-Hubs, we argue that the fluid sharing of compute and the among-device sharing of context information are key ingredients of an architecture that would deliver on the requirements of modern AI-tasks, with privacy and sustainability as vital components for deploying of state-of-the-art AI at the consumer edge.

Deep Learning trends
DNNs have traditionally grown in size in their strive for higher accuracies in the pursuit of intelligence. In a few years, we moved from the traditional multi-layer perceptron into deep networks of various forms and architectures. While the inductive bias of previous DNNs offered a convenient "shortcut" for learning in different modality inputs and a native mapping to widely available hardware, it turned out that convolutions were not "all that we needed." The advent of the transformers [6] brought a breath of fresh air into deep learning, and enabled the advent of hyperscale models able to organise and query the knowledge over massive datasets. Unprecedented accuracies and novel use-cases were, however, accompanied by unprecedented model footprints, only able to be trained distributedly in large datacenters. Even at inference time, these models remain notoriously difficult to deploy, and much more so with real-time performance.
Besides execution time, the large memory footprint of recent models has a direct energy impact on hardware, which becomes the main scaling bottleneck [13]. Memory accesses dominate energy, with more than 100× higher consumption than computation, while on-chip caches often account for 50% of the processor's energy budget. Indicatively, executing TinyBERT (fullprecision weights: 255MB) on mobile processors, such as the Edge TPU (8MB cache), requires an excessive amount of off-chip memory accesses and intensive use of the on-chip cache. As a result, even if new mobile processors could execute such models fast, the battery would be drained at an unacceptable rate.
At the same time, privacy of user data has become a top priority. Thus, alternative models of decentralised training have emerged, namely FL and on-device personalisation. While gamechanging, since it enables collaborative learning and pushes computation to where data reside, training on-device creates a much more memoryand energy-consuming workload that not all mobile devices can support. Indicatively, training SmallBERT on-device can consume more that 8GB of peak memory, while inference requires 1/16th of that [7]. Simultaneously, such in-thewild deployments need to deal with data and system heterogeneity, along with partial availability and dubious robustness of clients, and compete for resources with other co-habitating workloads.

The evolution of ML compute
The adoption and continuous upscaling of DNNs would not have been possible without the scaling of compute capabilities. Without a doubt, GPUs and their re-purposing from graphical to neural accelerators initially paved the way towards training deep networks, followed by specialised ASIC or FPGA-based accelerators. On-device compute for DNNs. Nevertheless, DNNs were never intended to stay limited to the premises of the datacenter. AI advancements made their way into the consumer world [2], with smart devices coming in different shapes and forms, ranging from smartphones and wearables to IoT devices and even robots. Their omnipresence along with their sensing capabilities enabled the production of unprecedented scales of local data, available for harvesting.
While initially fully offloaded to the cloud at the cost of privacy and latency, DNN computation has since progressively been "onloaded" to devices. What made this possible was advances in mobile hardware, which leverages integrated GPUs and Neural Processing Units (NPUs) on the same System-on-Chip (SoC) [9] or external accelerators, e.g. Edge TPU, Movidius. In addition, EfficientML techniques have enabled performant on-device execution for a multitude of AI-tasks.
Still, in contrast to typical cloud-based in-frastructure, the computational landscape in-thewild is much more heterogeneous [2] and failureprone [8], while the device still needs to remain responsive for tasks of the respective user. This brings new challenges that need to be tackled by device manufacturers and developers for making the mobile devices truly "smart." On-device hardware vs. offloading. DNN compute requirements and the respective hardware have not necessarily scaled at the same pace, as their development cycles are very different. In essence, developing and deploying highly specialised hardware is a high-cost process and needs to be balanced from a utility-cost standpoint as well as from a device lifecycle perspective. Be that as it may, cloud infrastructure still remains much more capable and allows for performant general-purpose compute without dealing with the constraints or heterogeneity of the edge, at the expense of client latency, provider cost and potentially privacy. Thus, the ubiquitous question on whether (and how much) computation to offload or whether to dedicate on-device hardware is posed in the consumer Edge-AI space.

Unlocking new potentials
Currently, there are emerging and forwardlooking applications in the consumer domain, which are still too costly to be deployed to the edge, or lack a common context that could be shared amongst devices in the ecosystem. In this setting, there is a lot of unfulfilled potential that is currently limited by the way intelligence is organised, deployed and distributed.

Devices are islands
While more and more devices penetrate the realm of the smart edge, they generally remain siloed, integrating standalone hardware to support their narrow requirements and do not advertise their capabilities to other local devices. In such an inelastic deployment, not only many compute cycles are wasted, but devices remain constrained in their own settings, and replicate similar sensor integration to perform different tasks. As such, the per-device cost is higher, the utilisation remains low and the hardware investment gets retired at the end of the device's lifecycle.

Shortcomings of previous paradigms
There have been various previous paradigms to create a flexible compute environment between devices [14], [16], with varying levels of success and adoption. These are not necessarily mutually exclusive, but provide different levels of interaction for the involved entities and different allocation of resources. Specifically: On-device ML [2]. A straightforward execution model is to run AI-tasks locally. While simple, it is far from simplistic as embedded and mobile devices come with severe constraints, spanning across computation, energy and thermals. Thus, specialised hardware integration along with techniques from EfficientML become important. Mobile Cloud Computing (MCC) [14]. MCC suggests that storage and compute are outsourced from the mobile device to more powerful cloud resources. Latency and privacy remain key issues of this paradigm. Edge Computing & Mobile Edge Computing (MEC) [14]. Edge computing is motivated by the movement of compute closer to where the data are produced, i.e. the edge. It provides lower latency compared to cloud offloading (i.e. MCC) and can be anywhere in between the cloud and the end-device, including base station servers (i.e. MEC). Privacy issues remain. Dew Computing [16]. Dew computing is defined as the ground level of cloud and edge computing, with a focus on the personal area sensor network and peer-to-peer ad-hoc networking. Based on microservices, it offers lower latency and realtime capability with dynamic network configuration without depending on any central entity. Fog Computing [14]. One of the most promising attempts, the most akin to our proposal, but yet to be fully realised, has been the model of Fog Computing. Motivated by the advent of IoT and building upon the technologies of cyberforaging, cloudlets and network virtualisation, fog computing proposed a decentralised computing model reliant on no single component, unlike cloud computing. Devices could leverage the nearby idle resources to perform low-latency computation while for more complex processing the cloud is still used. While many system principles were carefully outlined, its generality meant that it never found a "killer" use-case driving its adoption, and devices remained isolated with opportunistic collaboration if any. Volunteer Computing [12]. Inversely, there have been initiatives for volunteering spare cycles of edge compute for scientific purposes Facilitated by the explosion of compute at the consumer end, this paradigm proved helpful for harvesting ambient compute in the wild and showcases a prime example of computation "onloading".

Consumer Edge-AI 2.0
Given the accelerated rise of new AI-based use-cases and the inability of all smart devices to scale up their capabilities to that dynamic, we propose a new architecture that aims at ML execution, by means of flexible sharing of compute resources with smarter placement of specialised hardware, so that edge devices' processing power is augmented without extreme duplication. At the same time, their sensing capabilities can also be augmented through sharing of their situational context. We erect our proposed architecture around the following pillars:

Shared compute
Hardware resource allocation. Specifically, we shift from the paradigm of developing and integrating highly specialised NPUs into every available device to a model where higher-end and more general hardware accelerators can be hosted in central-device hubs. These hubs can be standalone devices that only serve this purpose (e.g. a home accelerator) or "piggy-backed" into devices with longer life cycles that are omnipresent at the edge (e.g. a router or a TV). Thus, accelerators can take advantage of larger budget in terms of area, cooling and power to provide acceleration to more operations and users than before. In fact, this hardware can even be based on reconfigurable computing platforms (FPGAs) [19], so that they can be adjusted to the user needs, use-cases and available devices. Last, they can be designed in a way that many hubs can be interconnected to scale out capabilities.
We distinguish two levels of compute sharing: i) static partitioning and ii) dynamic resource sharing. Static partitioning refers to the placement of dedicated compute units to devices. While certain IoT devices can become as thin as a sensor with a network adapter, others still need to maintain some autonomy or mobility. As such, one cannot simply centralise all available compute and strip devices out of any meaningful capabilities. Therefore, a balance need to be struck between the two ends of the spectrum. For example, on-device inference may be a common AI-task, but opportunistic participation 1 in FL has not shown a real need so far. Thus, smartphones may not need to integrate train-capable hardware. Instead, a training-ready NPU could be integrated to a home hub device where training can be offloaded.
At the same time, resource sharing can happen dynamically between device-hub or in a peerto-peer manner, compatibly with Fog Computing. Workload can be offloaded on demand, based on either the static capabilities of the device or their instantaneous dynamic load. How the equilibrium of resource partitioning is met is a function of i) the nature of AI-tasks at hand, ii) their urgency to complete and iii) the intersection of tasks between devices.
Networking & scheduling. Central to this elastic compute is a capable multi-channel networking infrastructure on top of which different requests for AI-tasks can be scheduled and prioritised. Such an infrastructure should offer multi-channel access, as different devices support diverse protocols (e.g. Wi-Fi, BLE, Zigbee, LoRa, etc.), loadbalancing among available channels and bandwidth slicing for better QoE. Additionally, task deadlines with preemption under multi-tenancy are core features for the scheduler to guarantee QoE for all active users. For example, the upscaling of live streaming video for a user would need to be given higher priority than the classification of newly acquired gallery photos on a user's device, which can be done offline. Thus, it is one of the hub's primary objectives not only to provide the resources, but also to co-ordinate them.

Shared context
By having interconnected compute, it is possible to also share a common context between 1 Participation in FL usually happens when device is plugged in and connected to an unmetered connection. sensing devices to fulfill collaborative tasks. As such, a smart speaker may not only serve as a standalone device, but could also be used for recognising a user in a room and personalising their experience by loading their profiles on ambient devices. At the same time, it can serve as a secondary microphone for noise cancelling or even for intrusion detection along with a smart camera. Context sharing can be i) explicit, through sensor-data exchange or ii) implicit, by embedding subsets of available sensors into a common subspace that can be leveraged for different tasks. Last, different tasks can also share common DNN backends, instead of replicating them per device. For example, the obstacle detection subsystem of a robotic vacuum cleaner could share parts of a DNN with a pet surveillance camera. In fact, both can act as sensors for enhancing the classification result through multiview perspectives, with one even offering active vision capabilities.

Privacy
In this era of Edge-AI, consumer data are increasingly important to protect. This is further exacerbated with the advent of distributed ML, be it in the form of collaborative inference, split computing [11] or FL [8] Therefore, privacy becomes an innate design component of nextgeneration consumer Edge-AI.
Privacy-preserving ML. Specifically, we differentiate between privacy and robustness at inference and training time. In the former setting, the input and outputs 2 of the model are sensitive, rather than the model itself. 3 However, when training or personalising a model, along with the data involved, the model parameters and gradients become also privacy-sensitive, as they may leak user data. Overall, privacy is not binary, you either have it or not, but spans in a continuous spectrum, i.e. a privacy budget, and therefore should be treated as such. Trust zones. Simultaneously, different data have different sensitivity towards different entities/adversaries. For example, there are data that one might want accessible from home-owned devices but private to the public, such as holiday trip photos. Conversely, other data one could share privately with a third-party, but not with their household, such as their browsing preferences for ad personalisation. As such, trust zones are formed that shape the data flow of collaboration in Edge-AI. This is depicted in Fig. 2 and shows that different tasks and devices can collaborate or exchange data based on the zones of trust.

Sustainable-AI
Additionally to user privacy, it is also important to respect the environmental ecosystem in which Edge-AI operates.We acknowledge the downsides of onloading more computation to the consumer edge where energy sources and efficiency may be suboptimal compared to datacenters [13] and propose ways that our paradigm can offset them. EfficientML. A particularly important component in any embedded deployment of AI is the optimisation of resources. As aforementioned, embedded and mobile devices integrate lower computational capabilities than state-of-the-art servers which affects their energy requirements. To that direction, techniques from EfficientML reduce the computational, memory or bandwidth requirements of AI-tasks, by means of changing the architecture [1], representation [8] or execution [10] of the DNN in an offline or dynamic manner. These, not only affect the QoE of users, but can also significantly affect the energy consumption of DNN execution. Lower manufacturing costs. Our paradigm also economises on the placement of highly-specific accelerators. By centralising certain components of the edge compute infrastructure, the manufacturing cost of mobile or IoT SoCs can be lower and the effective utilisation rate higher. Device upcycling. The unprecedented pace of computer systems advancement means that devices become quickly deprecated and turn from useful companions to effectively e-waste. However, old devices still integrate various sensors and oftentimes enough compute power to be useful [15]. We embrace the previous-generation devices co-existence and integrate them in a sustainable manner to the smart edge context, through upcycling and repurposing.
However, the landscape of upcycled devices is even more heterogeneous and potentially offers decreased robustness and reliability 4 . Providing support for such an ecosystem is tough and requires an asymmetric effort compared to the associated economic gains. Thus, a communitydriven development model, based on personal utility, or a subscription-based model for useful services through such devices could be a sustainable model of maintenance.

Proposed Architecture
Here, we lay out a reference architecture of the systems implementing our paradigm and how different components interact to accomplish the end goal, i.e. supporting and enabling AI-task execution through a common computational and contextual fabric, while preserving user-privacy and sustainability. Orchestrator. We suggest a client-server design where a central node (the orchestrator) is responsible for the subscription and management of resources in the local edge. The orchestrator should reside in a non-mobile (for availability) device of the edge network (i.e. the EdgeAI-Hub) and should be relatively high-end in terms of compute and networking capabilities for fine-grained scheduling and high-fidelity cross-channel data communication, respectively. Finally, it can be co-located with an AI-capable device or can be simply coordinating these.
A reference design of the orchestrator is depicted in Fig. 3a. Effectively, the orchestrator is the coordinator of the AI-task processes among device resources. To this direction, it comprises a resource manager to keep track of available resources and dynamic load of devices and a scheduler to allocate jobs. Each device maintains its own queue, with preemptible tasks based on their relative priority. To facilitate resource-totask matching, a performance controller assesses an AI-tasks's runtime on a certain device through analytical or historical estimators.
Upon scheduling a training or inference task, the orchestrator tracks its execution; the respective controller monitors the task progress, and handles data and model access between devices or even aggregation of parameters, if applicable. This way, context sharing and collaborative learning are enabled in a robust manner, while access to sensitive data remains controlled. EdgeAI-Hub. Fig. 3b showcases the technology stack of an EdgeAI-Hub. At the bottom, we have the hardware that is responsible for running the AI-tasks. The underlying SoC would integrate general-purpose and specialised hardware, i.e. an NPU, to accelerate common DNN operations, from convolutional and fully-connected layers to
Transformer blocks [6]. Contrary to smartphone SoCs nowadays, EdgeAI-Hub hardware should be optimised for i) a broader set of operations, ii) multi-precision support, iii) large multi-level memory for training with Direct Memory Access (DMA) support for zero-copy distributed ML, iv) virtualisation for fast context switching and task preemption under multi-user tenancy, and v) hardware-based app sandboxing with TEE for secure processing of sensitive information.
Network-wise, we envision the EdgeAI-Hub to support simultaneous communication over multiple interfaces, both for supporting different devices as well as for load-balancing and higher throughput. MIMO over multiple antennas could increase communication capacity, along with mesh networking over proxy repeaters for densely covering the deployment setting. Last, device discovery and handshaking should be made possible over any supported channel, e.g. BLE, NFC, UWB. Extensibility through removable components could also future-proof the EdgeAI-Hub's hardware.
Data are a first-class citizen in AI and, thus, storage becomes important. We propose a hierarchical storage solution with fast caching for iterative operations, like model training or model/context sharing, and traditional drives for long-term persistence. Hardware-level encryption and user-management with Access Control List (ACL) support from the filesystem is important to ensure security and privacy of data. Finally, redundancy could be offered through hardware (e.g. RAID) or distributed replication.
Moving up, the local operating system (OS) is responsible for local resource management and task scheduling. Low-level ML compilers and BLAS libraries would reside on top of the OS, along with the actual high-level ML framework interpreters for DNN execution.
The middleware and application layers form the top of the stack. The former is responsible for the coordination of distributed execution of tasks, including the orchestrator, preemptive scheduler and performance monitor of local and remote resources. The application may then span across different use-cases from the following section.

Enabling upcoming use-cases
There are various existent or upcoming applications and technologies that can be enabled or benefit from the proposed paradigm.

Virtual assistants & enhanced interaction
Home assistants have been progressively penetrating the consumer-edge. However, they remain largely thin, with most computation happening on the cloud. The proliferation of large language models that could be -at least partially -hosted on premises could significantly improve interaction, but also enable new use-cases of general or context-or user-specific question answering. Additionally, these models can be combined with other modalities (e.g. vision for lip reading) or repurposed for downstream tasks, such as person identification for space personalisation by pre-loading user profiles on nearby devices.

Virtual spaces
AR/VR technologies have been undoubtedly getting more capable and the promise of virtual worlds like the Metaverse are becoming increasingly relevant. Furthermore, the importance of online communication has been further enhanced by the COVID-19 pandemic and alternative models of collaboration, such as telepresence, have emerged. As the physical world is entering virtual reality, ML plays a central role and new architectures for scene representation or object mapping (e.g. NERF) are emerging, requiring significant computational power. As such, preparing the consumer ecosystem for such upcoming tasks may be an enabler for the deployment of next-gen technologies, be it for mutliplayer VR gaming, hybrid collaboration or immersive entertainment.

Cyber-physical agents
Inversely, computer agents have started having physical presence. Robotic agents, in the form of vacuums cleaners, assistants for the elderly or multi-articulated arms for cooking have showed up and are pushing the envelope of what is possible. In this realm, agents may co-exist with one another or with other sensors that can enhance their perception and spatial/environmental awareness. Moreover, mobile agents can act as active learning agents for gathering knowledge and reducing model uncertainty.

e-Health
Additionally, e-health is undoubtedly an important component that has stayed focused on the wearable space. However, such signals (e.g. heartrate monitoring) can be enhanced by additional multi-sensory signals on the consumer edge (e.g. vision for human pose estimation, , audio for cough detection) for diagnosing additional situations. In parallel, new use-cases such as cooking instructions or fitness tracking through audiovisual signals can be enabled through the existence of multi-device sensors and ambient compute for food readiness detection or pose estimation during exercise at home.

On-premise learning
As the edge is becoming increasingly capable in performing AI-tasks, on-premise training becomes a core workload. Both from a privacy and utility standpoint, it can enable learning on local data for personalisation or collaboration between devices, tasks and across input modalities. This becomes especially important in the case where training involves sensitive user data, which can never leave the premises of their ring of trust. Such tasks can span across various use-cases, such as classifying or enhancing private photos on a user's gallery, personalising speech recognition to a user's voice through interactions with a smart speaker or even summarising notifications and discussions a user has on their phone.

Challenges
The advent of such new consumer ecosystems brings about several challenges. These remain largely open issues and we anticipate them becoming prevalent topics of future research.

ML-related challenges
Non-IID data & multitude of tasks. Data in the wild are highly heterogeneous from client to client and even potentially temporally changing. As such, techniques from domain adaptation, FL and Continual Learning become increasingly relevant to generalisation. Model co-habitation. Nowadays smart-devices (e.g. smartphones) perform a multitude of tasks concurrently. As such, several DNNs may be cohabitating the device running in parallel. How these are scheduled to be executed efficiently is an open problem [19]. Global/private utility & privacy. FL is becoming a prevalent paradigm for pushing training to the device, the custodian of data. However, there is a trade-off between global and private utility, as optimisations goals can clash. Adding DP noise to the updates adds another degree of freedom trading utility for privacy. Striking a balance between all these factors can be challenging. Lack of annotations. When operating in the wild, generated data available for training may not be "clean" or even annotated with labels. This poses a major challenge for supervised training algorithms and paves the way for alternative methods in the field of semi-supervised, unsupervised and active learning for generalisation. Right to be forgotten. While learning is the most important task of AI, it is becoming increasingly essential for users to control where their data are being used. As such, model unlearning becomes a new field of study on how a model can extract and remove knowledge tied to specific data. Confidence assessment. As AI-tasks progressively make their way into the physical world, through robotic assistants, it is necessary for them to not only make decisions in a binary manner, but also assess the confidence of their decisions before acting, both for safety and interpretability.

System-level challenges
System heterogeneity and availability. Contrary to the datacenter, devices in the wild can be very heterogeneous and may arbitrarily become unavailable. As such, they should be treated accordingly, as this can affect the overall performance, robustness or even fairness of the (eco)system. Fault-tolerance & task preemption. As resources may become unavailable, high-priority tasks should be given some fault-tolerance margins or to be restarted indepotently on available resources in a transparent manner to the user. Interoperability. Another repercussion to extreme heterogeneity is that devices of different manufacturers, generations and tiers will need to co-operate through a common interconnect. As such, backwards compatibility, standardisation of interfaces and open middleware become requirements to break away from siloes and enable device federation.
Incentives. An equally important factor for realisation of such a paradigm is incentivisation of the stakeholders: i) Manufacturers to support federated operation of devices and unlocking the potential of old devices; ii) Clients to accept onpremise execution of AI-tasks that can potentially benefit more users (e.g. FL).

Conclusions
Historically, there has been a pendulum effect between the centralisation and decentralisation of compute. First, hardware breakthroughs appear in the datacenter, in turn enabling solutions to problems previously intractable. Eventually, hardware advances make their way into the consumer devices. However, algorithmic solutions continue to scale up in requirements, resources for which can only be found out-of-premise. AI seems to be no exception and current trends in the field require a consolidation and reorganisation of edge resources for scaling compute to upcoming requirements sustainably.