{'fraction_non_alphanumeric': 0.01984732824427481, 'fraction_numerical': 0.0045190839694656485, 'mean_word_length': 4.842311808776311, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Running Jobs in the Vacuum

We present a model for the operation of computing nodes at a site using Virtual Machines (VMs), in which VMs are created and contextualized for experiments by the site itself. For the experiment, these VMs appear to be produced spontaneously "in the vacuum" rather having to ask the site to create each one. This model takes advantage of the existing pilot job frameworks adopted by many experiments. In the Vacuum model, the contextualization process starts a job agent within the VM and real jobs are fetched from the central task queue as normal. An implementation of the Vacuum scheme, Vac, is presented in which a VM factory runs on each physical worker node to create and contextualize its set of VMs. With this system, each node's VM factory can decide which experiments' VMs to run, based on site-wide target shares and on a peer-to-peer protocol in which the site's VM factories query each other to discover which VM types they are running. A property of this system is that there is no gate keeper service, head node, or batch system accepting and then directing jobs to particular worker nodes, avoiding several central points of failure. Finally, we describe tests of the Vac system using jobs from the central LHCb task queue, using the same contextualization procedure for VMs developed by LHCb for Clouds.


Introduction
There is now a considerable amount of interest within the scientific grid and HEP communities in running jobs in virtualized environments on Cloud services [1,2,3]. Virtualized environments provide several attractive features for sites and experiments, particularly by decoupling operating system versions between the "bare iron" run by the site and the environment seen by the experiment software. This shifts responsibility for supporting the experiments' operating system requirements from the site to the experiments' own operations teams, in return for the experiment having greater control within the VM.
Cloud environments largely depend on virtualization technologies to decouple the specific requirements of customers, and are also the most widely discussed way of providing virtualized environments to experiments that want to use this technology.
In this paper we present the Vacuum model as an alternative way of providing virtualized environments, and locate its place within the evolution of Grid and Cloud models since the turn of this century. Whereas Grid and Cloud models require that experiments push each job or VM request into each site, in the Vacuum model they see their VMs apparently being produced spontaneously by sites "in the vacuum", to use an analogy with the quantum mechanical production of virtual particles in nature. These virtual machines then request work from the experiments' central services, and so the Vacuum model is an inversion of the more usual Cloud model of Infrastructure-as-a-Service, and these sites can be said to operate as Infrastructure-asa-Client (IaaC). Figures 1, 2, and 3 compare the three models and illustrate their shared aspects, and the progressive simplification that has occurred. In all of the models, LHC experiments such as LHCb and some of the larger experiments elsewhere have developed "pilot job" systems [4,5,6,7], which operate as private overlay grids in which tasks are pulled from central task queues. The Grid was originally conceived as a push system, but during the 2000s the experiments added pilot jobs in the payloads they pushed to sites, which in turn pull tasks when they arrive at sites. These overlay grids, such as LHCb's DIRAC, are also readily adaptable for use with Cloud sites, in which case the VM runs the client part of the pilot job and requests tasks from the central task queue as before.

Grid, Cloud, and Vacuum
As with Grids, Cloud interfaces provide a general way for users (or customers) to run arbitrary work at a site. However, we have observed that many HEP sites run jobs for a comparatively small number of experiments or other virtual organisations. For example, at the University of Manchester Tier-2 centre, in excess of 95% of the resources are used by the ATLAS and LHCb experiments. Consequently, the flexbility provided by Cloud and Grid interfaces is largely unneeded for the vast majority of the site's work. This observation prompted an examination of simpler ways of providing virtualized environments to experiments, with the aim of simplifying the local support burden and benefitting from the robustness associated with simplicity. The resulting Vacuum model is a third way of making use of nodes at a site. The Vacuum model can be defined as a scenario in which VMs are created and contextualized for experiments by the site itself. The contexualization procedures are supplied in advance by the experiments and launch clients within the virtual machines to obtain tasks to work on from the experiments' central queue of tasks. The Grid model: originally conceived with central brokers forwarding users' jobs to sites, the Grid has evolved towards direct submission of jobs to sites. Large experiments have created their own overlay infrastructures, such as LHCb's DIRAC, with users' jobs fetched from a central task queue by pilot jobs once they arrive at the site. The Cloud model: an evolution of the demand for server rental in the commercial world, which has led to highly elastic VM hosting services. Some experiments are now looking at using either commercial Cloud providers or sites using software developed for Clouds, and are adding support for VMs to their private infrastructures.
As described below, LHCb has demonstrated using a common contextualization procedure for Cloud and Vacuum sites, using a client or JobAgent adapted from their existing grid pilot jobs. In this way, these two models are complementary and the Vacuum can be seen as a way of operating VMs at a site for a specific set of experiments, but using a considerably simpler software stack than at Cloud or Grid sites. As before, user jobs are fetched from the experiment's central task queues. This leverages the existing work on supporting VMs for Clouds and the experiment's private overlay infrastructures such as DIRAC.

Vac implementation
An implementation of the Vacuum scheme, Vac, has been developed at Manchester. A VM Factory daemon runs on each physical worker node to create and contextualise that node's set of VMs. The libvirt[8] and Linux KVM[9] VM frameworks are used by Vac for lower level VM management functions. Each node's VM Factory decides which experiments' VMs to run, and this choice is based on site-wide target shares and information gathered by a peer-to-peer protocol between the factory nodes in that site or Vac "space". This decision is made whenever a VM slot becomes available. Local copies of common configuration files are updated when necessary by the sites' preferred operating system management tool, such as Puppet [10]. This scheme allows sites to provide virtual environments to multiple experiments and still maintain the fair share allocation of capacity between experiments that is currently achieved with grid interfaces to conventional batch queuing systems. However, this is achieved without a gate keeper service, head node, or batch system accepting and then directing jobs to particular worker nodes. Instead, the Vac factories co-ordinate the site's behaviour amongst themselves using the peer-to-peer protocol and their copies of the configuration files. This avoids several central points of failure, and considerably simplifies the installation and maintenance of these sites.
The Vac software and documentation is made available under the BSD license from the Vac website[11].

UDP protocol
Each factory has a list of all the factories in the same Vac space and queries the other factory nodes when deciding what VM flavours to create. These queries are sent in UDP packets containing a JSON-encoded Python dictionary.
Each factory runs a responder process which listens on UDP port 995 and replies to these queries when they are received. The responses are UDP packets containing a JSON-encoded dictionary for each VM a ssigned to that factory. These replies include the VM's state (shutdown / starting / running), the VM type (roughly corresponding to the experiment), the VM's start time, and the outcomes of the last VM instances run for each VM type. The UDP messages also include randomly chosen cookies to prevent some simpler denial of service attacks.

Backoff procedure
To avoid loading the experiment's central Matcher/TaskQueue service with requests when no work is available to this site, Vac implements a "backoff" procedure. If a VM finishes and returns "no work" / "banned" / "site misconfigured" outcomes to the factory, then it is classed as aborted. If no outcome is returned, then if the VM finishes after less than the parameter fizzle seconds (by default 600 seconds) then it is also classed as an abort. For each VM type, if an abort has happened on any factory in the space within a time range given by the parameter backoff seconds (again, 600 seconds by default), then no more VMs of that type will be started.
After this initial period has passed, it is possible for a VM to be created for the VM type in question to test the current situation. To avoid a flood of unnecessary VMs, if an abort happened within the last backoff seconds + fizzle seconds and any new VMs have run for less than fizzle seconds, then no more VMs of that type will be started. This gives time for one or two trial VMs to run and potentially abort if no work is available. Once backoff seconds + fizzle seconds have passed without any aborts, then VMs can start again as fast as slots become available.
For the worst case scenario of an experiment having no work for the site for an extended period of time, the backoff procedure typically results in one or two VMs being started for the experiment every backoff seconds which run for one to five minutes before failing to find work and aborting. Although this does represent wasted resources, it only amounts to one or two cores per Vac space being used occasionally when no work is available from the experiment. This cost of treating the factories as loosely coupled peers which follow the backoff procedure has to be compared to the saving of not requiring a dedicated multicore head node machine, which would centrally orchestrate the operation of the VMs or jobs across the space in a traditional grid or cloud site.

Target shares
To attempt to achieve the allocation of resources between experiments that the site wants, Vac uses the concept of target shares. These are simply the desired fractions of the resources to be delivered to each experiment. When deciding which type of VM to create, each Vac factory node tries to start types which are currently under represented in the Vac space according to their target shares. The backoff procedure described above prevents the excessive creation of VMs for experiments which currently have no work.
This approach is designed to be highly robust and to allow nodes to carry on working even when their neighbours have problems. To avoid relying on a central head node which decides which VM type to create, the target shares are recorded in a configuration file which exists on each factory node. It is assumed that the site will use their existing node configuration mechanism (such as Puppet) to copy these files to the nodes when required.
The overall design of the Vac system envisages that these target shares may be modified over time to achieve particular shares over the timescale of months, quarters, or years. As Vac makes accounting records available in standard log file formats expected by batch and grid accounting systems, it would be possible for a site to use their local accounting database, such as APEL, to automate the generation of the target shares file to favour experiments which have not received the desired share so far in the accounting period. Again, the advantage of this retrospective rather than dynamic approach is that if the automatic regeneration of target shares files fails,

Graceful termination and shutdown codes
Vac's strategy for graceful termination is based around the proposed HEPiX machinefeatures mechanism[12]. When starting a VM, the Vac factory creates directories containing the machinefeatures files and exports them into the VM using NFS. The VM contextualization then mounts these directories within the VM. The shutdowntime value is used by Vac to notify the VM of the absolute time at which it will be destroyed if still running. In practice, if the contextualization imposes limits on the length of tasks requested from the experiment's central task queue then the VM will normally be able to terminate before the enforced shutdown time is reached. Nevertheless, this mechanism provides protection in cases where the VM fails to terminate itself and would otherwise continue occupying resources indefinitely. In addition, the shutdown command option is provided to the VM by Vac, which is to be run by the VM when it terminates. We have extended this command by allowing it to take command-line arguments, which are recorded in a writeable NFS directory hosted by the Vac factory. The arguments allow the VM to report why it terminated. They are of particular use in the backoff procedure, which otherwise relies on the fizzle seconds parameter to attempt to determine if a VM failed or ran successfully.
The arguments consist of three digit codes followed by human readable messages, in a similar way to status messages in internet protocols such as HTTP [13] and SMTP [14]. The values are listed in Table 1. Table 1. Shutdown codes and messages 100 Shutdown as requested by the VM's host/hypervisor 200 Intended work completed ok 300 No more work available from task queue 400 Site/host/VM is currently banned/disabled from receiving more work 500 Problem detected with environment/VM provided by the site 600 Error related to job agent or application within VM As with HTTP codes, this scheme provides room to insert more numbers for finer grained information in the future. Vac uses this information programmatically, but it may also be useful to site administrators to help identify where responsibility for problems lie, especially given the relatively opaque nature of experiments' VMs from a site's point of view.

Tests with LHCb production jobs
To create a VM for an experiment, the experiment must supply a contextualization procedure to the site which the Vac factory daemon applies. A suitable contextualization was developed by the LHCb collaboration at CERN for use with Clouds [15], and has been extended to work with Vac. During the summer of 2013, we successfully demonstrated running production jobs from the main LHCb central task queues without any modification to the central DIRAC services or operations procedures. During these production runs, Vac was in operation at 3 sites in the UK (Manchester, Lancaster, Imperial College) and successfully ran 3300 normal production jobs from the central task queue. Figure 4 shows sustained job execution during a 30 days period during these tests.  The contextualization procedure causes the LHCb JobAgent previously used within pilot jobs to be started in the VM. In the simplest mode, the JobAgent requests one task from the LHCb central task queue, runs it as a job for up to 24 hours, and then terminates the virtual machine. The JobAgent's existing mechanism for discovering the time left in a batch queue slot is being extended to use the equivalent HEPiX machinefeatures information, and this could be used to run several tasks consecutively within the same VM instance before the hard shutdown time imposed by Vac is reached.

Conclusion
We have presented an alternative model for operating VMs at sites and successfully demonstrated running LHCb production jobs at several such sites in the UK as part of the standard LHCb distributed computing infrastructure based on DIRAC. This is a general approach which could be used by any site and by any experiment which prepares a contextualization procedure to allow its jobs to run inside VMs. As such, we next aim to increase the number of sites evaluating the Vac software and demonstrate creating and contextualisaing VMs for other experiments with Vac.