{'fraction_non_alphanumeric': 0.016887816646562123, 'fraction_numerical': 0.003399495558723544, 'mean_word_length': 4.972495088408644, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A New Design for Live Neutron Event Data Visualisation for ISIS and the ESS

Viewing and reducing event data live during an experiment is an important component of contemporary experiment control systems. At ISIS a simple TCP streaming mechanism for neutron event data allows data to be processed by Mantid during an experiment; however, this existing solution is likely to encounter performance issues with the new generation of acquisition hardware. As part of an ESS in-kind collaboration, ISIS is working on developing a new data streaming system based on Apache Kafka for managing and delivering live experiment data to multiple clients. This work provides an ideal opportunity to replace the existing ISIS streaming mechanism with this new system, which not only provides enhanced functionality for ISIS but also functions as a prototype for developing and testing requirements of the ESS.


Introduction
A range of different data must be aggregated during an experiment on a beamline at a pulsed neutron source. These include measurements from sensors monitoring the sample environment, details of the proton pulse from the accelerator and experiment metadata, such as the names of the current users of the beamline. The majority of data by frequency and volume are typically neutron detection events, which comprise information about the location on a detector that a neutron was detected, its time of flight from source to detector and the wall-clock time of the neutron pulse which the neutron is associated with.
Live visualisation of neutron events and other data is increasingly seen as a critical component of experiment control. Visualisation during an experiment allows the scientist to optimise the experiment configuration via the control system based on this continuous feedback. This feedback loop has the highly valuable result of making more efficient use of beamtime compared to the alternative of loading and visualising the data from a file written at the end of each experiment run.
Transporting data from the detectors and other data sources to where it is processed and visualised during an experiment requires a robust and high-throughput data streaming system. Contemporary streaming systems must handle the increasing data rate requirements of facilities which exhibit a trend towards higher intensity neutron sources [1], detectors with larger detector areas, higher spatial resolutions and increased detection efficiencies [2]. Such systems must also be reliable in the sense that they should neither waste beamtime due to system downtime nor compromise scientific validity of results due to data loss.
By working with the European Spallation Source (ESS) Data Management Group in addition to their BrightnESS [3] partners Paul Scherrer Institute and ELETTRA Sincrotrone Trieste in an in-kind project, ISIS has an excellent opportunity to improve on current systems in use and to prototype the future ESS system. It will be invaluable to the ESS to have proven software with well understood operational requirements ahead of the facility's start of operations in 2019 [4]. The ESS will use the Experimental Physics and Industrial Control System (EPICS) [5] for instrument control and has chosen to use the Mantid Framework [6] for live data processing and visualisation. Both of these technologies are in use at ISIS and therefore sharing knowledge and implemented software across the institutions is mutually beneficial.

Existing system
The Instrument Control Program (ICP) aggregates data on a single machine for each beamline at ISIS. It is connected to the detector electronics and has access to other data sources such as metadata databases and sample environment measurements via a local area network. The ICP is responsible for writing a file in the NeXus format [7] which is then copied to network attached storage from where it can be read for analysis and visualisation using the Mantid Framework. It also serves data to several existing visualisation clients, including Mantid's Live Listener, LabVIEW clients and the IBEX control system [8].
The current data streaming solution has several limitations. Firstly, each new connection places additional strain on the instrument server. Since this server is responsible for aggregating and writing data to file it is crucial that it is not overloaded. Secondly, the clients are only able to capture data from the moment they join the stream and will lose data if they cannot process and display data at the same rate as the stream. Data can also be lost on the wire without the knowledge of the client, which is of critical concern when streaming data for analysis or to a file writer. Finally, there are some data types, such as images from the EPICS's areaDetector module [9], which are not yet supported.
For the ESS, the facility's high beam intensity and cutting edge detectors are expected to result in neutron detection rates one to two orders of magnitude higher than existing institutions [10]. In the past, the volume of data would be reduced early in the pipeline by histogramming events, however modern analysis requires retaining full event data [11]. Therefore, detection rates of up to 10 8 neutrons per second per beamline create throughput requirements of up to 10 Gbps for neutron data from a single beamline. Implementing a system without the aforementioned limitations is essential since having a single instrument server responsible for aggregation, file writing and serving data with this bandwidth is beyond current hardware capabilities. Furthermore, robustness and data delivery guarantees are of particular importance due to the intention to carry out live data reduction at the ESS [12].

Apache Kafka
The key technology on which the new design for ISIS and the ESS is based is Apache Kafka.
Kafka is an open-source, high throughput, message streaming platform with a configurable level of message delivery guarantees [13]. Kafka implements a publish-subscribe architecture, which is depicted in Figure 1. Producer clients publish messages to named data streams, called topics, on a cluster of Kafka brokers. Consumer clients subscribe to one or more topics to receive messages.
Topics are persisted on disk on brokers in one or more partitions. Each newly produced message is appended to the end of a partition, thus partitions are log files of messages. Partitions are distributed across brokers in the cluster to allow throughput for a topic to exceed the capacity of a single broker. Topics can also be configured to replicate their partitions on different brokers, with priority given to brokers in different server racks. Broker machines or entire racks can therefore fail without the system losing data or clients experiencing loss of service. The oldest messages, at the start of the partition, are removed once their age reaches a configurable retention time or to ensure a specified limit on data volume is not exceeded.   Figure 2 shows the initial prototype of the new system at ISIS. In this design the ICP functions as a producer and streams aggregated data into the Kafka cluster. Since the scalable Kafka cluster is then responsible for serving data to any consumers this approach solves the issue of additional clients placing strain on the ICP. As Kafka stores messages in partitions on the cluster, clients do not have to miss data if their processing cannot keep up with data production. An additional benefit is that clients can request historical data, for example data from the start of the current experimental run even if they join the stream part way through.

Initial prototype
Mantid's Live Listener interface was updated to allow it to function as a consumer client for the Kafka-based system. The Live Listener subscribes to a topic containing neutron event data and updates a data structure, a workspace, at a specifiable interval with information that has been streamed to it. At each interval, processing algorithms can be configured to run on the new, or cumulative, data and the instrument view visualisation can be updated. An example of Mantid's Instrument View is shown in Figure 3 and displays accumulated counts of detected neutrons on a three-dimensional view of the detectors of the WISH beamline at ISIS during an experiment run.

Future
In the prototype system the ICP remains responsible for aggregating data and writing the NeXus file. As previously mentioned, the expected throughput requirements of the ESS make this approach impossible for the new facility. Instead, various producer clients are being implemented to publish data from different sources, such as EPICS and databases providing metadata. Similarly a separate, NeXus file writing, consumer client is in development by colleagues at the Paul Scherrer Institute as part of the BrightnESS project [3]. The system at ISIS will be  updated to make use of these clients and mimic the planned architecture for the ESS, as shown in Figure 4. Not only will this benefit ISIS by adding support for datatypes not streamed by the current system, but also make the system a true prototype for the ESS.