{'fraction_non_alphanumeric': 0.022224061581757232, 'fraction_numerical': 0.0031453048048669455, 'mean_word_length': 4.8072578707041576, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CinemAirSim: A Camera-Realistic Robotics Simulator for Cinematographic Purposes

Drones and Unmanned Aerial Vehicles (UAV's) are becoming increasingly popular in the film and entertainment industries in part because of their maneuverability and the dynamic shots and perspectives they enable. While there exists methods for controlling the position and orientation of the drones for visibility, other artistic elements of the filming process, such as focal blur and light control, remain unexplored in the robotics community. The lack of cinemetographic robotics solutions is partly due to the cost associated with the cameras and devices used in the filming industry, but also because state-of-the-art photo-realistic robotics simulators only utilize a full in-focus pinhole camera model which does incorporate these desired artistic attributes. To overcome this, the main contribution of this work is to endow the well-known drone simulator, AirSim, with a cinematic camera as well as extended its API to control all of its parameters in real time, including various filming lenses and common cinematographic properties. In this paper, we detail the implementation of our AirSim modification, CinemAirSim, present examples that illustrate the potential of the new tool, and highlight the new research opportunities that the use of cinematic cameras can bring to research in robotics and control. https://github.com/ppueyor/CinematicAirSim


I. INTRODUCTION
The use of Unmanned Aerial Vehicles (UAVs) is widespread in industry and robotics research [1]. Among the list of potential uses of UAVs, there is growing interest in the film industry for those seeking a more unique cinematic experience. Drones enable the recording of scenes and perspectives that are typically inaccessible by typical filming cranes or other infrastructure.
In the robotics community, there is interest in developing control and planning algorithms for cinematographic purposes. For example, model predictive control is used in [2] to calculate trajectories when filming a desired aerial shot. An interface that allows novice users to generate optimized paths according to their artistic requests is presented in [3]. A high-level API that incorporates camera path planning and autonomous drone control for shooting purposes is exhibited in [4]. A 3D tool that eases the shooting and the design of trajectories of UAVs for non-robotic experts following the  guidelines of knowledgeable cinematographers is developed in [5]. Visibility constraints, in terms of the focal length are discussed in [6]. Finally, autonomous key frame selection during the filming is discussed in [7]. All these works are interested in the design of trajectories, i.e., sequences of desired positions and orientations, of the drones for filming purposes. However, to the best of our knowledge, there is no prior work that explores the most important factors of the cinematographic scope: the aesthetic and artistic issues associated to photographic aspects. This is probably due to the additional complications associated with modeling cinematographic cameras, not to mention the expensive price of these sensors. Including cinematographic cameras in an existing robotics simulator could be a simple and cheap way to avoid these complications (see our solution to this problem in Fig. 1).
The research progress in robotics would be almost impossible without some software to virtually reproduce the robots as close to reality as possible. There are many examples of robotics simulators that have been used and improved for decades. Stage and Gazebo [8] are two of the most popular state-of-the-art simulators, thanks to their native integration with the Robotic Operating System (ROS). Webots [9] is another interesting platform that includes a variety of robotic models and sensors and an advanced physics engine. However, in order to account for the cinematographic aspects of filming, it is also very important that simulators have a powerful graphic engine capable of rendering photo-realistic scenes. Despite the features that make these simulators useful for general robotics -their powerful physics engines, their easy connection with ROS, or the wide community of users that makes development easier -the lack of realism in their graphic engines limits their potential in the context of cinematography.
The last few years have witnessed the appearance of new simulators that make use of state-of-the-art graphic engines, such as Unreal [10] or Unity [11]. On one hand, Flight-Goggles [12] is a photo-realistic simulator able to render realistic environments thanks to the Unity Engine, as well as its capability of gathering data and the integration of lifelike simulated sensors. On the other hand, AirSim [13], developed by Microsoft Research, is built upon Unreal Engine. Both simulators meet all the necessary requirements for the cinematographic purposes, but currently none of them offer the possibility of online and real-time modification of camera parameters associated with standard filming photographic effects, i.e., focal length, focus distance, and aperture.
The main contribution of this paper is a modification of AirSim, CinemAirSim, that empowers the simulator with the artistic features of real filming cameras. The existing AirSim API is extended with new methods that enable users to access and control a camera's focal length, focal distance, or the lens aperture. Our contribution allows different types of lenses to be selected and considered in the control problem by exploiting the potential of the Unreal Engine. All the files required to modify the original simulator can be downloaded for free from the Github page of the project 1 . The paper also includes examples where the different parameters are adjusted during the drone's flight to mimic scenes from movies awarded with the Academy Award for Best Cinematography in different years.
The rest of the paper is organized as follows. Section II describes the two main software elements of the simulator, Airsim and Unreal. Section III gives all the details of the integration of the cinematic camera into AirSim. Section IV shows different examples using the simulator to control both the drone and the camera parameters in order to reproduce real movie scenes. Finally, Section V presents the conclusions of the work and discusses the future lines of research that our software can enable in the robotics community.

II. AIRSIM AND UNREAL
In this section we describe the two main programs that sustain CinemAirSim: the graphics engine Unreal and the robotics simulator AirSim. The main software components of these programs are schematized in Fig. 2.

A. Unreal
The Unreal Engine is a graphics engine that was developed by Epic Games and is widely used in the video game industry. It's powerful graphic capabilities, open-source license, and the ability to create virtual environments make this tool extremely appealing for research.
Since the principal aim of the cinematographic simulator presented in this paper is to enable robotics research oriented to the acquisition of artistic shots, the quality of the visualization is an important aspect in its development. In that sense, Unreal comes with a powerful graphics engine, which has a high number of characteristics that will be important to the aspirations of the simulator. In particular, it is worth-mentioning the powerful rendering capabilities, the visualization during the execution of the play using multiple configurable cameras, and the high fidelity simulation of the light, weather, and reflections of scene objects.
In addition, the environment editor of Unreal offers wide capabilities to create these realistic scenes. There are existing tools that allow users to design environments that are very close to reality in a quick and user-friendly way. There also exists an extensive community of active users that are creating 3D models and maps available to download for free. In fact, some of the models are created using photogrammetry techniques, which makes the environments precisely lifelike. All of this simplifies the task of creating custom scenarios to the robotics researchers.
Finally, Unreal comes with C++ code that allows to move the elements, direct the cameras, or control the light sources of the scene in real time. Moreover, there are some works that provide an easy integration of Unreal with ROS [14] and OpenCV [15].

B. AirSim
AirSim (Aerial Informatics and Robotics Platform) was launched in 2017 [13] by Microsoft Research as a plugin for the Unreal Engine 4. It is very easy to install; in fact it is as simple as copying a folder into your Unreal project and compiling. Like Unreal, AirSim also has an extensive community of users that are actively improving and discussing potentially interesting features of the simulator. Moreover, the possibility of generating binaries of code that integrates all the elements (environments, configurations) in a ready-to-run file enables AirSim to be used with a specific purpose by those users that are not so familiar to the robotics scope, such as cinematographers in the film industry. This is the method that Microsoft followed for the drone competition Game of Drones 2 during NeurIPS 2019.
AirSim provides a realistic simulation of a generic quadrotor and offers the possibility to customize it to accommodate to different commercial models. The design of AirSim takes into account diverse factors in order to achieve the best realism when emulating the behaviour of actual drones. The model of the vehicles and its actuators, the effect of the environment variables such as gravity, magnetic field and air pressure on the robots, or the accelerations when moving, to name a few. Moreover, the simulator uses the physics engine included in Unreal to detect and model collisions.
While several models of sensors are embedded to the robots, given the scope of the paper, we focus on the vision sensors and describe the current properties of these sensors in AirSim. The default camera provided by AirSim is called "PIP camera" due to the possibility to simulate different vision sensors within the same layout, i.e., RGB image, depth perspective, or object segmentation. AirSim allows users to attach to the vehicle several of these cameras with flexibility in their location. The visualization of the images in AirSim can be done in two ways: i) using the main viewport of Unreal and ii) using a SceneCaptureComponent. This component is also used to acquire the images of the cameras for further post processing. More importantly for the cinematic purpose, the main intrinsic configurations are the resolution (the sensor's width and height) and the fieldof-view, which can only be modified offline in the program settings. However, this type of camera follows the classic pinhole model, which limits its use on the cinematographic scope when compared to more complex optic models, like the thin-lens models.
Finally, all these elements and parameters are controlled through the provided API. When the execution of the flight starts, a Remote Procedure Call (RPC) server is launched. This server allows to make RPC calls to control the different drones in real time.

III. CINEMATOGRAPHIC CAMERAS IN AIRSIM
In this section we describe the modifications made in Unreal and AirSim to make the simulation of cinematographic cameras inside the robotics simulator possible. We also describe all of the new functionalities that can be controlled in real-time with examples to visualize their influence in the resulting images.

A. Cinematic Camera Model
Unreal provides a specialized camera component for cinematic photography refered to as the CineCameraComponent [16]. The CineCameraComponent simulates cameras currently used in the film industry in a highly-configurable and realistic way. Specifically, it gives users the ability to specify a camera's aperture and focal depth as well as the lens' f-stop and focal length. This camera can be modeled by thin-lens camera model whose projection equation is given by where f represents the focal length, Z represents the focus distance (distance of the object to the lens) and z represents the focal depth (distance from the lens to the sensor). The thin-lens model differs from the common pinhole camera model in that the aperture is not infinitesimally small which allows multiple rays of light to hit the same pixel on the sensor. This property enables the out-of-focus quality of objects that are not at the focus distance for a given focal length and focal depth.

B. Cinematic Camera Integration
In order to integrate the cinematic camera model in the architecture of Unreal and Airsim, we have remodeled several of their modules. The parts modified for CinemAirSim are highlighted in red in Fig. 2.
Firstly, in the Sensor Simulation module of AirSim, we have replaced the base pinhole cameras, CameraComponent, that are embedded in the drones for cinematic cameras, CineCameraComponent. Since the weight of the real drone and the mounted camera will be different depending on the commercial models, we have left this setting with the default values. Nevertheless, this parameter can be set in the standard configuration files of AirSim. Secondly, in order to adjust the lens characteristics of the CineCameraComponent onthe-fly, we have increased the number of accessors in the class in charge of controlling the camera sensor, PIPCamera, implementing all the getters and setters that permit us to update these parameters. The user can still use the methods from the basic cameras of Unreal, as it is just an extended version of the original cameras.
These changes have required other minor adjustments in intermediate files up to the API layer. The API server and client have been endowed with new methods that give the user the possibility of changing the lens parameters. The settings of the camera can be read and written by making RPC calls as it is done in the base version of AirSim.
Finally, the rendering module of Unreal additionally required several modifications to maintain the protocol for image acquisition of AirSim, i.e., using the SceneCapture-Component described in the previous version. An alternative way to achieve this without modifying Unreal would be by editing PIPCamera to acquire images directly from the main viewport of the simulator. However, this solution has been discarded sd it does not allow to get images from multiple robots simultaneously.

C. Customizable Settings
We describe in more detail all the new parameters that can be adjusted using the API. Their influence in the image is shown with different examples in Figure 4. They can be compared to Fig. 3, that shows the RGB camera output of the base version of AirSim.

1) FilmBack:
This parameter controls the size of the filmback or digital sensor of the camera. CinemAirSim enables the control of this feature either by setting a custom horizontal and vertical size (width and height) in millimetres or to choose among a list of some common preset filmbacks sizes (16:9 DSLR, 35mm Academy, IMAX 70mm, etc.). Changing the filmback will automatically change the aspect ratio of the images acquired with the camera.
2) Lens: In Unreal, changing the lens of the camera determines the minimum and maximum focal lengths and apertures, the number of blades of the diaphragm that influences the shape of the Bokeh effect, and the minimum focus distance, i.e., the distance to the closest object the lens can focus on. The ability to change the lens of the simulator enables the user to test different lenses without the need of buying new cameras. As in the filmback, there is a predefined set of lenses available, each with particular parameters, but it is also possible to manually set custom minimum and maximum values for the focal length and aperture.
3) Focus distance: The focus distance is the space in centimeters between the camera and the subject of the plane where the image is in focus. The user can decide whether to activate the manual focus or not. If it is activated, it will focus the image in terms of the focus distance, stated by the user through the API. Moreover, we have included the option to show the debug focus plane in the visualization. This is a colored plane that is drawn in the image at the distance where the elements are in focus. This helps to control where to adjust the focus distance with precision.

4) Focal length:
The focal length is the distance between the optical center of the lens and the camera's digital sensor, expressed in millimeters. In this way it is possible to get closer or farther shots of a desired object without the need of moving the robot. The focal length also controls the fieldof-view (FOV) and the zoom in the image.
5) Focus aperture: The aperture of the diaphragm controls the amount of light that enters to the image sensor and is expressed in terms of the f-stop number. The diameter of the opening where the light passes decreases as this number gets higher, allowing less light to reach the sensor. The quantity of light that infers to the sensor will determine the depth of field, which is the area between the closest and the farthest points of the image that are in focus. Larger aperture values in terms of f-stop will make the range bigger. Choosing different values of focus distance will determine the position of the area where the objects will be focused. By tweaking the focal length and the focus distance, it is possible to get focused outputs from objects that are apparently far. In this section, we have reproduced some scenes of films that were awarded with the Academy's Oscar for Best Photography to demonstrate the potential of the simulator and to highlight new avenues of research where it can be used. We want to remark that the goal of these experiments is not to propose a new planning and control solution for the camera parameters but to demonstrate that these type of plans can be efficiently tested and validated using the simulator. Therefore, in the experiments we have generated open-loop combined trajectories for both the camera and drone parameters that reproduce realistically scenes filmed with cinematographic cameras and we have compared them with equivalent experiments modifying only the trajectory of the drone and using a conventional camera. Figure 5 shows from a third person view the two scenes with the filming drones. We refer the reader to the supplementary video, where all these scenes and comparisons can be observed with more detail. Fig. 6. Example of focal length change. The first row shows screenshots from the real movie. The second row shows the images acquired using CinemAirSim for a reproduction of the scene. The third row shows the images acquired by the default camera of AirSim in the same locations.

A. Focal length changes
In this experiment we reproduce a scene acquired by a drone where the focal length is controlled. The scene is extracted from the movie "Road to Perdition", which obtained the Academy Award in 2002. Changing this parameter affects to the field of view, generating a zoom-in or zoomout effect without the need of getting closer or farther. We have chosen outdoors long-distance scenes, so that the other focus parameters do not affect the final result. Figure 6 shows two frames of the real scene (top row), as well as the simulated versions using CinemAirSim (middle row) and AirSim with the standard camera (bottom row). In the scene, we have reduced linearly the focal length while moving the drone in a straight line, rotating so that the car in the scene keeps inside the field of view. In the supplementary video there is another experiment of similar characteristics extracted from the movie "The Revenant" (awarded in 2015), Fig. 1.

B. Focus distance and aperture changes
In this experiment we highlight the focus parameters of the cinematographic camera, i.e., the aperture and focus distance. The objective is to provide values for these two parameters in such a way that the objects that are in focus match with the real scene. The scene we have chosen is extracted again from "Road to Perdition". Figure 7 shows the results of this experiment. In the left column, the focus is on the main character, sat at the end of the table, whereas close and far away objects, like the green cups and the chandelier respectively, appear out of focus, thanks to the low f-stop value. In the bottom row we can see that the standard camera cannot reproduce this effect. As the scene evolves the focus shifts towards the back of the room (middle column) defocusing the main character in the process. Finally, at the end of the scene (right column), the focus returns to the character. To achieve all these changes, we have fixed the aperture at a low level, f/1.2, and we have used the focus distance trajectory depicted in Fig. 8.

C. Focus Distance Closed-loop Control
In this subsection we describe a simple control experiment to adjust the focus distance in terms of the real space that exists between the drone to the object that should be in focus. For this case, we have chosen the 3D model of Stan Lee to be in focus (Fig. 9). The aperture is fixed to a low f-stop value to make the focus change more significative. We have used the tools from Unreal and AirSim to get the ground-truth location of the drone and the target. The drone follows a trajectory where its distance to the main character is continuously changing, ranging between far away and close distances. During the execution of the trajectory, the focus distance is adjusted in terms of the real distance in meters that separates the drone and the target. This auto focus control occurs in real time. The top row of Fig. 9 shows some screenshots of the overall result of the experiment. The plane that represents the part of the image that is in focus is depicted in Fig 9, middle row. The evolution of the focus distance, as well as the position and rotation of the drone are represented in the bottom row Fig. 9. The top row shows the frames acquired by the camera at three instants. The middle row shows the same frames with the focus plane, to verify that the target is in focus. The bottom row shows the focus distance trajectory (left) and the drone position (middle) and orientation (right).

V. CONCLUSIONS AND NEW RESEARCH POSSIBILITIES
In this paper we have presented CinemAirSim, an extended version of AirSim that enables the simulation and control of cinematographic cameras in real time. We have described the main tools that form the basic structure of the simulator, namely Unreal and Airsim, and how they have been modified to allow the simulation of a thinlens camera model. With CinemAirSim, all the parameters associated to these realistic cameras can be easily controlled in real-time by means of the extended API in the same way as all the elements present in the original simulator. In addition, to illustrate the potential of CinemAirSim, we have developed a set of experiments where we have provided a single drone open-loop trajectories for pose and camera parameters with the objective of reproducing some scenes of movies recognized for the quality of their photography. The results demonstrate that the simulator can achieve high levels of reality under different conditions and parameter configurations.
CinemAirSim also paves the way for novel research directions in the fields of robotics and control such as the design of new control algorithms that autonomously decide the best photographic parameters together with the drone trajectories. We can also envsion designing new multi-drone coordinated shots, where different robots put the focus on different parts of the scene and exploit other camera parameters to avoid filming each other. Another major research direction includes robotic perception and navigation algorithms that consider the thin-lens model, such as Depth From Focus (DfF) methods instead of the classic pinhole camera model. We believe CinemAirSim will prove to be an important tool in the development of new robotic solutions in the near future.