{'fraction_non_alphanumeric': 0.021637786964552732, 'fraction_numerical': 0.009235640777552996, 'mean_word_length': 4.780376207422471, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Current Status of the Ceph Based Storage Systems at the RACF

Ceph based storage solutions are becoming increasingly popular within the HEP/NP community over the last few years. With the current status of Ceph project, both object storage and block storage (RBD) layers are production ready on a large scale, and the Ceph file system storage layer (CephFS) is rapidly getting to that state as well. This contribution contains a thorough review of various functionality, performance and stability tests performed with all three (object storage, block storage and file system) levels of Ceph by using the RACF computing resources in 2012-2014 on various hardware platforms and with different networking solutions (10/40 GbE and IPoIB/4X FDR Infiniband based). We also report the status of commissioning a large scale (1 PB of usable capacity, 4k HDDs behind the RAID arrays by design) Ceph based object storage system provided with Amazon S3 complaint RadosGW interfaces deployed in RACF, as well as performance results obtained while testing the RBD and CephFS storage layers of our Ceph clusters.


Introduction
Over a last decade, the RHIC and ATLAS Computing Facility (RACF) at BNL have been using conventional storage system that provides the access to its namespace operation as well as the content of the data. During the same period, the new type of storage, which separates the operations of the data from those of its namespace, has emerged. These new kinds of storage systems operate on a principle that the storage service should provide the stable and scalable service accessing the content of data without performance limitations of manipulating the metadata associated with the namespace. They are commonly called object storage systems, emphasizing the data over its metadata, such as Amazon S3 [1], GlusterFS [2], and Ceph [3].
At RACF, Ceph based storage clusters were chosen to be tested for the following reasons. At first, it is free software for us to deploy on top of our older or even retired hardware without any licensing and other financial obligations. Secondly, based on reliable autonomic distributed data store (RADOS) [4], Ceph provides three distinctive use cases: object storage with S3 interfaces, block devices and a file system. The support of S3 APIs, which were originally developed by Amazon to access their services, reduces the learning curve of using the new type of storage. The RADOS block device layer (RBD) [5] can be used as any part of existing mounted storage system. Finally, the Ceph file system (CephFS) [6] allows us to study the use case of a resilient and scalable distributed file system for possible applications in our HEP specific data storage environment. The networking and storage layout of both RACF Ceph clusters is show in Figure 1. More details on the configuration and performance of these systems can be found in the reports [8] and [9].

Ceph Object Gateway
Ceph Object Gateway (RadosGW) provides two interfaces to Ceph storage clusters: S3 and Swift. The S3 Interface provides the compatible APIs used by Amazon S3 while the Swift interface is compatible with OpenStack Swift APIs [10]. The common interface used in Amazon and Ceph makes the use of S3 APIs are convenient for developers to create new and adapt the exiting application. The object store does not provide block devices and/or file system by itself -it only provides containers, call buckets, for data to be stored, requiring the use of external file catalog to retrieve/modify the content of the containers. ATLAS experiment has already started to implement the use of S3 storage in the computing framework called ATLAS Event Service [11] as a data store for a very large number (~10 9 ) of small data objects (~a few MB each), utilizing the advantage of not having a file system.

Ceph Block Device
Ceph (RADOS) block device (RBD) provides one of the most convenient ways to integrate the Ceph storage clusters into any existing storage services. It provides the regular mountable device to OS. Once such a device is mounted, it can be formatted to be used as a normal space.
One example of using Ceph block device on our existing dCache system is shown in Figure 2. First, multiple RBD objects are created in the Ceph storage cluster. Then, they are provided to multiple hosts where they can be mounted and formatted to be used as storage on those hosts, allowing to be added as dCache [12] storage pools. With dCache as the top layer of the storage service, Ceph storage immediately becomes a part of any production service, providing all necessary, existing access interface without complication of learning how to access a new type of storage. One disadvantage of the using RBDs is that it does not fully utilize the redundancy of Ceph storage system by only allowing single-mount of RBD on a particular host, resulting in the reliance in the stability of RBDmounted host system to serve the data regardless of the availability of the Ceph storage cluster.

Ceph File System
Ceph file system, CephFS, is a POSIX compliant (with minor exceptions summarized in [13]) file system built on top of the Ceph storage cluster mention in the previous sections. Using the Ceph storage technology, the CephFS provides the familiar interface to users without any complication of learning new APIs with all the benefits of Ceph storage cluster. The distributed nature of the CephFS makes the service to be reliable and resilient. In addition, unlike the RBDs mentioned in the last section, the same storage can be accessed on the multiple hosts, making some interesting use case.
Since the same storage can be presented to multiple hosts, when the above layer of a storage service are state-less in serving the data, it can be configured to increase the reliability of the service horizontally by adding any additional hosts. For example, one of the candidates for such a layer of a state-less storage service on top of CephFS is the XRootD data access service [14]. Figure 3 illustrates this configuration. CephFS provides a common file system. This same name space is mounted in multiple XRootD data server hosts. Since any data can be accessed through any of the data servers, CephFS with XRootD can provide a very reliable storage services with horizontal scalability.

Test setup configurations
The throughput tests were conducted using two setups:  Setup S1 consisting of 74 client hosts each provided with 1 GbE network interfaces plugged to a non-blocking networking core of the RACF ATLAS Farm. Each of the client nodes was deployed under Scientific Linux (SL) 6.5 x86_64 OS provided with the custom built Linux kernel 3.19.1 and was mounting CephFS directly from the Main Ceph cluster. The maximum network throughput was limited by 74 Gbps = 9.25 GB/s on the client side. The optimal values for the stripe_unit and object_size parameters were determined experimentally while trying to maximize performance of I/O operations with a single large file in a series of preliminary CephFS performance tests with one client node mounting the CephFS directly from the cluster involved.

Read and write throughput with test setup S1
The largest scalability test for CephFS directly mounted on the client nodes was performed with the group of 74 1 GbE network attached clients, each mounting the CephFS directly from the Main Ceph cluster. The test was conducted in the following 3 consequent stages: 1. Measure the network bandwidth available between the clients and the Ceph cluster head nodes with iperf3 in one and several simultaneous iperf3 TCP session established on every client node. The maximum aggregated network throughput of 8.9 GB/s was reached in these tests with two sessions per client. This corresponds to 96% of the throughput limitation on the client side, showing that there is no obstruction on the network level between the client nodes and the Ceph cluster. bandwidth of up to 8.7 GB/s was observed, which corresponds to 98% of the value measured with iperf in the first stage of the CephFS scalability test. Thus, the CephFS read performance was limited only by the network capabilities of the client nodes, while the sustained write performance observed was about 50% of the network throughput limit on the client side. Further Ceph cluster level optimizations and similar scalability tests are needed to fully understand the ways in which the write performance can be further improved in our setup.

Read and write throughput with test setup S2
To measure write throughput of CephFS, a set of distinct files with relatively large size (approximately 3.6 GB per file) from BNLs production dCache storage system were used as the data source. They were written to CephFS instance in order to be later retrieved via XRootD protocol. Figure 4 (top) shows the network activity during the one hour long test. The network values in the figure include network activity within the Ceph storage cluster associated with internal replications. The actual write throughput values are about 1/3 of those indicated by the green line. Although the tests were conducted repeatedly with varying number of clients, the maximum write throughputs did not change, indicating the limit in the current configuration of the Ceph cluster. Additionally, we have observed that, as more data is written, the slow degradation of performance was seen Figure 4 (bottom). It might be attributed to the performance issues associated with the Ceph Metadata Server (MDS), even though the clustered MDS configuration was used. Even faster degradation was observed with just a nonclustered MDS configuration with just one active Ceph MDS component in the cluster.   In order to measure read throughput of CephFS, source test files written in the previous section (write tests) were used as source data to be read by clients. Again, XRootD protocol was used here. Figure 5 (top) and (bottom) show network activities seen in the Ceph Storage clusters during the read tests using four or eight XRootD data servers. The colors indicate different hosts used for the data server. The read tests show that in the current setup, four data servers were sufficient to reach the saturation of the stable read-throughput of about 2 GB/s. The tests also revealed the existence of the initial large spike, indicating the effect of cache in the CephFS and OSD read cache on the Ceph cluster head nodes.

Conclusion
Two Ceph storage clusters were successfully implemented in RACF at BNL using the existing hardware and reaching the scale of 4k spinning drives and 1 PB of aggregate usable capacity across both clusters while maintaining factor of 3 internal data replication. All three types of Ceph storage layers (object storage/S3, RBD and CephFS) were incorporated as a part of the HEP specific storage services provided by RACF for ATLAS experiment. The experience gained while operating these two clusters indicates that while RBD and CephFS storage layers of Ceph can be used as a part of common storage service, the S3 storage layer provides the opportunity to utilize the data store without the limitations typically associated with file systems. The stability of services provided (including those based on CephFS) is now verified on the timescale of about 10 months, starting from the moment when our first large scale Ceph cluster entered production in 2014.