{'fraction_non_alphanumeric': 0.023692992213570633, 'fraction_numerical': 0.006117908787541713, 'mean_word_length': 4.789439793947199, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Belle II public and private cloud management in VMDIRAC system.

The role of cloud computing technology in the distributed computing for HEP experiments grows rapidly. Some experiments (Atlas, BES-III, LHCb) already exploit private and public cloud resources for the data processing. Future experiments such as Belle II or upgraded LHC experiments will largely rely on the availability of cloud resources and therefore their computing models have to be adjusted to the specific features of cloud environment, in particular to the on-demand computing paradigm. Belle II experiment at SuperKEKB will start physics run in 2017. Belle II computing requirements are challenging. The data size at the level of hundred PB is expected after several years of operation, around 2020. The baseline solution selected for distributed processing is the DIRAC system. DIRAC can handle variety of computing resources including Grids, Clouds and independent clusters. Cloud resources can be connected by VMDIRAC module through public interfaces. In particular the mechanism of dynamic activation of new virtual machines with reserved job slots for new tasks in case of an increasing demand for computing resources is introduced. This work is focused on VMDIRAC interaction with public (Amazon EC2) and private (CC1) cloud. The solution applied by Belle II experiment and the experience from Monte Carlo production campaigns will be presented. Updated computation costs for different use cases will be shown.


Introduction
Belle II experiment [1] is able to compute on a variety kinds of resources, i.e. grids, standalone clusters, academic clouds and commercial clouds. Clouds are becoming increasingly important for the Belle II computing model [2].
For now, the Belle II distributed system can handle communication to EC2, OCCI and Nova APIs. It means that we can run virtual machines on Amazon, CC1, OpenStack and other compatible interfaces. The Belle II software is located on the same CVMFS Stratum-1 mirror servers which support LHC experiments. It gives us more stable and up-to-date access to the experiment's software.
Squid proxy server on each cloud region (Site) is set up for caching the software mounted by the CVMFS [3] to the virtual machine.
Nowadays, Belle II collaborators introduced at least three different ways to handle cloud infrastructure (VMDIRAC, Cloud-Scheduler, Dynamic-Torque). This paper covers VMDIRAC solution only.

Commercial and scientific clouds overview
In our work, we used two types of cloud resources, called Infrastructure-as-a-Service, public commercial and private scientific ones. An example of commercial cloud is Amazon EC2, which is a part of the Amazon Web Services [4]. Representative of private cloud connected to the Belle II computing model is the CC1 system [5] located in Krakow, Poland. Both types of cloud provide a public API called EC2. In general, private clouds are easier to manage. Often an amount of CPUs is provided by particular organization for free, based on MOU (memorandum of understanding). In case of commercial clouds we need to lead computations in the cheapest possible way, so that we are using non-persistent machines, like Amazon Spot Instances which are at about 70% cheaper than standard On-demand machines. Spot Instance is a type of virtual machine for which the price is constantly changing and often it is much cheaper than other type of instance. The price is different not only in each AWS Region but also in each AWS Zone (part of Region). Moreover, Spot Instance can be rapidly destroyed if the market price reach our maximum price, what is the main issue of such model. Fortunately for the Belle II we can apply such non-persistent model thanks to the Job Scheduler, a part of DIRAC WMS [8] which has a great failover mechanism. Also, Amazon EC2 provides On-demand instances, which are persistent and the cost is stable but much higher than Spot Instances. We are using On-demand machines for the Squid cache servers which need to have very high availability.

The DIRAC system and VMDIRAC extension
The DIRAC[6] (Distributed Infrastructure with Remote Agent Control) system is a framework to build a complex distributed environment in a relatively simple way. It is used by large communities like Belle II, LHCb, BES-III, but also smaller groups for example NGI multi-VO portals.
Cloud resources can be connected to the DIRAC by the VMDIRAC [8] module which provides a mechanism of dynamic activation of new virtual machines with reserved job slots for new tasks. All of this activates automatically in case of an increasing demand for computing resources. VMDIRAC is an extension of the DIRAC system. It is used to manage a pool of different kind of clouds by API drivers like EC2, OCCI, rOCCI, Nova. For the VMDIRAC Amazon driver we developed new extensions for a better control of different AWS regions and virtual machines based on Spot Instances.

VMDIRAC cloud resources map
The variety of cloud resources raises the management needs. For such purpose we prepared a DIRAC extension [9] which represents usage of cloud resources managed by VMDIRAC in the interactive Google Map [ fig.3]. Cloud Resources Map is delivered by the BelleDIRAC software.

EC2UserData Agent
The DIRAC Configuration System is a main knowledge-base about distributed infrastructure, where we can check all available resources and its configuration. EC2UserData agent periodically checks the configuration and generates so called user data script for each cloud endpoint. EC2 machines are contextualized by the user data script which starts during the VM boot process. For the private VM images we are using Cloud-init software for this purpose.

EC2SpotPrice Agent
This is the crucial agent to control EC2 Spot Instances. It calculate an optimal MaxBiddingPrice limit for the Spot Instance requests in each Amazon region and zone. Also, it increases limits of VM amount for the cheapest zone and decreases VM limits for expensive zones. Calculated results are stored in the main DIRAC Configuration. To compare prices on particular AWS Zone we calculate weighted moving average (WMA) indicator from the period of last two hours. It gives us smoother chart of changing price. Final parameter 'MaxBiddingPrice' in our case is always lower than 80% of OnDemand instance price and little bit higher than than WMA indicator [ fig.4]. It is our solution to balance between low costs and stability of the system which we can get for such price.

Results and Summary
Belle II jobs (gBasf2) can take many hours of CPU, so any interruption is crucial. With the DIRAC system we achieved great failover mechanism of many services, like job submission [8] and even sudden Spot Instances shutdown can be easily handled and job can be rescheduled, thanks to this mechanism the efficiency of whole Belle II distributed system is close to 100%. Processing on Amazon Spot Instances gives us efficiency at about 98% if there are no strong fluctuations of price. Such situation can be observed in one of the chart below. AWS Sydney region became expensive in Australian working hours, so a lot of computing machines have been destroyed [ fig.5]. For the long job which can take many hours, the risk of losing generated data and than rescheduling jobs from the beginning is high. For this reason it would be good to upload output data periodically or at least to send 'halted' signal in the last minute of the instance life and than resubmit the job. In the current Belle II distributed computing model such halted job need to be resubmit form the scratch, even if the job was processing many hours. For this reason we need to operate Spot Instances carefully and strive for short jobs submission to all temporary resources. In this paper we present working examples of the MC generation on clouds. Commercial clouds comparing to the private, are more challenging for few reasons, one is mutable price for computations and outbound data transfers which need to be handled very carefully. We need to be ready for an unexpected destruction of the virtual machine what implicate a necessity of quick restore of resources so the reaction time is crucial. In some AWS Regions, price for the Spot instances is changing in more determined way, this indicates a possibility to use learning systems to decide where to start next computation machine to have processing more stable.

Acknowledgements
We are grateful for the support and the provision of computing resources by CoEPP in Australia, HEPHY in Austria, McGill HPC in Canada, CESNET in the Czech Republic, ACK Cyfronet AGH is an academic computer centre which is providing lots of computational and storage resources for Belle II jobs processing. Data generated on the private cloud CC1 are directly stored in Cyfronet's storage element.
The project was supported by the Polish National Science Centre (NCN) and the Polish Ministry of Science and Higher Education, grant reference no. 804/N-JAPONIA/2010/0.