{'fraction_non_alphanumeric': 0.030927835051546393, 'fraction_numerical': 0.012050400916380297, 'mean_word_length': 4.82946047008547, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Cloth in the Wind: A Case Study of Physical Measurement through Simulation

For many of the physical phenomena around us, we have developed sophisticated models explaining their behavior. Nevertheless, measuring physical properties from visual observations is challenging due to the high number of causally underlying physical parameters -- including material properties and external forces. In this paper, we propose to measure latent physical properties for cloth in the wind without ever having seen a real example before. Our solution is an iterative refinement procedure with simulation at its core. The algorithm gradually updates the physical model parameters by running a simulation of the observed phenomenon and comparing the current simulation to a real-world observation. The correspondence is measured using an embedding function that maps physically similar examples to nearby points. We consider a case study of cloth in the wind, with curling flags as our leading example -- a seemingly simple phenomena but physically highly involved. Based on the physics of cloth and its visual manifestation, we propose an instantiation of the embedding function. For this mapping, modeled as a deep network, we introduce a spectral layer that decomposes a video volume into its temporal spectral power and corresponding frequencies. Our experiments demonstrate that the proposed method compares favorably to prior work on the task of measuring cloth material properties and external wind force from a real-world video.


Introduction
There is substantial evidence [17,10] that humans run mental models to predict physical phenomena. We predict the trajectory of objects in mid-air, estimate a liquid's viscosity and gauge the velocity at which an object slides down a ramp. In analogy, simulation models usually optimize their parameters by performing trial runs and selecting the best. Over the years, physical models of the world have become so visually appealing through simulations and rendering [46,28,7,36] that it is worthwhile to consider them for physical scene understanding. This alleviates the need for meticulous annotation of the pose, illumination, texture and scene dynamics as the model delivers them for free.  Figure 1. We propose to measure real-world physical cloth parameters without ever having seen the phenomena before. From cloth simulations only, we learn a distance metric that encodes both intrinsic and extrinsic physical properties. After learning, we use the embedding function to measure physical parameters from real-world video by comparison to its simulated counterpart.
In this paper, we consider flags and cloth in the wind as a case study. Measurements and visual models of flags and cloth are important for virtual clothing try-on [56], energy harvesting and biological systems [37,19]. The cloth's intrinsic material properties, together with the external wind force, determine its dynamics. Untangling the dynamics of fabric is challenging due to the involved nature of the air-cloth interaction: a flag exerts inertial and elastic forces on the surrounding air, while the air acts on the fabric through pressure and viscosity [19]. As we seek to measure both the cloth's intrinsic material properties and the external wind force, our physical model couples a non-linear cloth model [46] with external wind force [49].
The task is challenging, as physical models of cloth tend to have high numbers of unknown parameters and bear intricate coupling of intrinsic and external forces. Our solution is to compare pairs of real and simulated observations and measure their physical similarity. As there is a fundamental caveat in the use of simulation and rendering for learning: "visually appealing" does not necessarily imply the result is realistic, the main question is how to assess the similarity of the causally underlying physical parameters rather than visual correspondence. It might be the case that the image looks real but never occurs in reality. At the core of our measurement is a cloth simulation engine with unknown parameters θ to be determined. The outcome of a simulation (e.g. 3D meshes, points clouds, flow vectors) is converted to the image space using a render engine. We then compare the simulated visual data with a realworld observation of the particular phenomenon ( Figure 1). Accordingly, we propose to learn a physical similarity metric from simulations only, without ever perceiving a real-world example. In the learned embedding space, observations with similar physical parameters will wind up close, while dissimilar example pairs will be further away. Guided by the physical similarity, the simulation's parameters are refined in each step. As a result, we obtain a complete computational solution for the refined measurements of physical parameters.
Our contributions are as follows: (1) We propose to train a perception-based physical cloth measurement device from simulations only, without ever observing a real-world manifestation of the phenomena. Our measurement device is formulated as a comparison between two visual observations implemented as a Siamese network that we train with contrastive loss. (2) In a case study of cloth, we propose a specific instantiation of the physical embedding function. At its core is a new spectral decomposition layer that measures the spectral power over the cloth's surface. Our solution compares favorably to existing work that recovers intrinsic and extrinsic physical properties from visual observations. (3) To evaluate our method, we record real-world video of flags with the ground-truth wind speed gauged using an anemometer. (4) Finally, we iteratively refine physics simulations from a single real-world observation towards maximizing the physical similarity between the real-world and its simulation.

Related Work
Previous work has measured physical properties by perceiving real-world objects or phenomena -including material properties [11], cloth stiffness and bending parameters [6,55], mechanical features [52,26,27,23], fluid characteristics [51,40,35] and surface properties [25]. The primary focus of the existing literature has been on estimating intrinsic material properties from visual input. However, physical phenomena are often described by the interaction between intrinsic and extrinsic properties. Therefore, we consider the more complex scenario of jointly estimating intrinsic material properties and extrinsic forces from a single real-world video through the iterative refinement of physics simulations.
Our case study focuses on the physics of cloth and flags, both of which belong to the broader category of wind-excited bodies. The visual manifestation of wind has received modest attention in computer vision, e.g. the oscillation of tree branches [54,41], water surfaces [40], and hanging cloth [6,55,48,9]. Our leading example of a flag curling in the wind may appear simple at first, but its motion is highly complex. Its dynamics are an important and wellstudied topic in the field of fluid-body interactions [37,42,43]. Inspired by this work and existing visual cloth representations that characterize wrinkles, folds and silhouette [4,14,50,56], we propose a novel spectral decomposition layer which encodes the frequency distribution over the cloth's surface.
Previous work has considered the task of measuring intrinsic cloth parameters [4,6,55] or external forces [9] from images or video. Notably, Bouman et al. [6] use complex steerable pyramids to describe hanging cloth in a video, while both Yang et al. [55] and Cardona et al. [9] propose a learningbased approach by combining a convolutional network and recurrent network. In our experiments we will compare our cloth frequency-based representations with Cardona et al. [9] on flags while Yang et al. [55] is a reference on the hanging cloth dataset of Bouman et al. [6].
Our approach of measuring physical parameters by iterative refinement of simulations shares similarity to the Monte Carlo-based parameter optimization of [52] and the particle swarm refinement of clothing parameters from static images [56]. In particular, the work of [56] resembles ours as they infer garment properties from images for the purpose of virtual clothing try-on. However, our work is different in an important aspect: we estimate intrinsic and extrinsic physical parameters from video while their work focuses on estimating intrinsic cloth properties from static equilibrium images. Recently, Liang et al. [24] have proposed a differentiable cloth simulator which could potentially be used as an alternative to our approach for cloth parameter estimation.

Method
We consider the scenario in which we make an observation of some phenomena with a physical model explaining its manifestation available to us. Based on the perception of reality, our goal is to measure the D p unknown continuous parameters of the physical model θ ∈ R D p , consisting of intrinsic parameters θ i and extrinsic parameters θ e through an iterative refinement of a computer simulation that implements the physical phenomena at hand. In particular, we consider observations in the form of short video clips  Figure 3. We propose the perception-based measurement of physical scene properties. Given an observation of a real-world physical phenomenon, here represented as video clip x target , our algorithm measures the underlying parameters of the physical scene. Central is a simulation engine implementing the physical model, parametrized by intrinsic material properties θ i and the characterization of external forces θ e . A render engine, with render parameters ζ , maps the simulator's output to the image space producing video clip x sim . Using an embedding function s φ (x), both real and simulated examples are mapped to a manifold on which physically similar examples are assigned to nearby points. To measure the similarity between both clips, we evaluate a distance metric D i j (·, ·) in the embedding space. Its result serves as the objective for an optimization module that refines the physical parameters θ towards the actual observation.
iteration, the simulator runs with current model parameters θ to produce some intermediate representation (e.g. 3D meshes, point clouds or flow vectors), succeeded by a render engine with parameters ζ that yields a simulated video clip x sim ∈ R C×N t ×H×W . Our insight is that the physical similarity between real-world observation and simulation can be measured in some embedding space using pairwise distance: where s φ (x) : R C×N t ×H×W → R D e an embedding function parametrized by φ that maps the data manifold R C×N t ×H×W to some embedding manifold R D e on which physically similar examples should lie close. In each iteration, guided by the pairwise distance (1) between real and simulated instance, the physical model is refined to maximize physical similarity. This procedure ends whenever the physical model parameters have been measured accurately enough or when the evaluation budget is finished. The output comprises the measured physical parameters θ * and corresponding simulation x * sim of the real-world phenomenon. An overview of the proposed method is presented in Figure 3.

Physical Similarity
For the measurement to be successful, it is crucial to measure the similarity between simulation x sim and real-world observation x target . The similarity function must reflect correspondence in physical dynamics between the two instances. The prerequisite is that the physical model must describe the phenomenon's behavior at the scale that coincides with the observational scale. For example, the quantum mechanical understanding of a pendulum will be less meaningful than its formulation in classical mechanics when capturing its appearance using a regular video camera. Given the physical model and its implementation as a simulation engine, we generate a dataset of simulations with its parameters θ randomly sampled from some predefined search space. For each of these simulated representations of the physical phenomenon, we use a 3D render engine to generate multiple video clips x i sim, with different render parameters ζ i . As a result, we obtain a dataset with multiple renders for each simulation instance. Given this dataset we propose the following training strategy to learn a distance metric quantifying the physical similarity between observations.
We employ a contrastive loss [15] and consider positive example pairs to be rendered video clips originating from the same simulation (i.e. sharing physical parameters) while negative example pairs have different physical parameters. Both rendered video clips of an example pair are mapped to the embedding space through s φ (x) in Siamese fashion [8]. In the embedding space, the physical similarity will be evaluated using the squared Euclidean distance: If optimized over a collection of rendered video clips, the contrastive loss asserts that physically similar examples are pulled together, whereas physically dissimilar points will be pushed apart. As a result, by training on simulations only, we can learn to measure the similarity between simulations and the real-world pairs.

Simulation Parameter Optimization
We will arrive at a measurement through gradual refinement of the simulations (Figure 3). To optimize the physical parameters we draw the parallel with the problem of hyperpa-rameter optimization [39,3]. In light of this correspondence, our collection of model parameters is analogous to the hyperparameters involved by training deep neural networks (e.g. learning rate, weight decay, dropout). Formally, we seek to find the global optimum of physical parameters: where the target example is fixed and the simulated example depends on the current set of physical parameters θ. Adjusting the parameters θ at each iteration is challenging as it is hard to make parametric assumptions on (2) as function of θ and accessing the gradient is costly due to the simulations' computational complexity. Our goal is, therefore, to estimate the global minimum with as few evaluations as possible.
Considering this, we adopt Bayesian optimization [39] for updating parameters θ. Its philosophy is to leverage all available information from previous observations of (2) and not only use local gradient information. We treat the optimization as-is and use a modified implementation of Spearmint [39] with the Matérn52 kernel and improved initialization of the acquisition function [29]. Note that the embedding function s φ (x) is fixed throughout this optimization.

Physics, Simulation and Appearance of Cloth
Up until now, we have discussed the proposed method in general terms and made no assumptions on physical phenomena. In this paper, we will consider two cases of cloth exposed to the wind: curling flags and hanging cloth ( Figure 4). To proceed, we need to confine the parameters θ and design an appropriate embedding function s φ (x).

Physical Model
The physical understanding of cloth and its interaction with external forces has been assimilated by the computer graphics community. Most successful methods treat cloth as a mass-spring model: a dense grid of point masses organized in a planar structure, inter-connected with different types of springs which properties determine the fabric's behavior [1,33,46,2,28]. We adopt Wang's et al. [46] non-linear and anisotropic mass-spring model for cloth. This model uses a piecewise linear bending and stretching formulation. The stretching model is a generalization of Hooke's law for continuous media [38]. As our experiments focus on flags in the wind for which the stretching properties are of minimal relevance, our experiments will focus on flags in the wind, typically made of strong weather-resistant material such as polyester and nylon. Therefore, the material's stretching properties are of minimal relevance and we will emphasize on the cloth's bending model [46] and external forces [49].
Bending Model (θ i ). The bending model is based on the linear bending force equation first proposed in [7]. The model   . Left: we consider two cases of cloth exposed in the wind: (a) a flag curling in the wind; and (b) cloth fabric hanging from a rod. In both cases, the fabric fabric is treated as a mass-spring model in which a dense grid of point masses is inter-connected with multiple springs. Right: the bending and stretching springs determine the materials behavior. Flexion springs act over shared edges whereas structural springs connect to direct neighbors.
formulates the elastic bending force F e over triangular meshes sharing an edge ( Figure 4). For two triangles separated by the dihedral angle ϕ, the bending force reads: where k e is the material dependent bending stiffness, N 1 , N 2 are the weighted surface normals of the two triangles, E represents the edge vector and u is the bending mode (see Figure 1 in [7]). The bending stiffness k e is non-linearly related to the dihedral angle ϕ. This is realized by treating k e as piecewise linear function of the reparametrization α = sin(ϕ/2)(N 1 + N 2 ) −1 . After this reparametrization, for a certain fabric, the parameter space is sampled for N b angles yielding a total of 3N b parameters across the three directions. Wang et al. [46] empirically found that 5 measurements are sufficient for most fabrics, producing 15 bending parameters.
External Forces (θ e ). For the dynamics of cloth, we consider two external forces acting upon its planar surface. First, the Earth's gravitational acceleration (F g = ma g ) naturally pushes down the fabric. The total mass is defined by the cloth's area weight ρ A multiplied by surface area. More interestingly, we consider the fabric exposed to a constant wind field. Again, modeling the cloth as a grid of point masses, the drag force on each mass is stipulated by Stokes's equation F d = 6πRηv w in terms of the surface area, the air's dynamic viscosity and the wind velocity v w [49,28]. By all means, this is a simplification of reality. Our model ignores terms associated with the Reynolds number (such as the cloth's drag coefficient), which will also affect a real cloth's dynamics. However, it appears that the model is accurate enough to cover the spectrum of cloth dynamics.

Simulation Engine
We employ the non-differentiable ArcSim simulation engine [28] which efficiently implements the complex physical model described in Section 4.1. On top of the physical model, the simulator incorporates anisotropic remeshing to improve detail in densely wrinkled regions while coarsening flat regions. As input, the simulator expects the cloth's initial mesh, its material properties and the configuration of external forces. At each time step, the engine solves the system for implicit time integration using a sparse Cholesky-based solver. This produces a sequence of 3D cloth meshes based on the physical properties of the scene. As our goal is to learn a physical distance metric in image space between simulation and a real-world observation, we pass the sequence of meshes through a 3D render engine [5]. Given render parameters ζ comprising of camera position, scene geometry, lighting conditions and the cloth's visual texture, the renderer produces a simulated video clip (x sim ) which we can compare directly to the real-world observation (x target ). We emphasize that our focus is neither on inferring render parameters ζ from observations nor on attaining visual realism for our renders.
Parameter Search Space (θ i , θ e ). The ArcSim simulator [28] operates in metric units, enabling convenient comparison with real-world dynamics. As the base material for our flag experiments, we use "Camel Ponte Roma" from [46]. Made of 60% polyester and 40% nylon, this material closely resembles widely used flag fabrics [46]. The fabric's bending coefficients, stretching coefficients, and area weight were accurately measured in a mechanical setup by the authors. We adopt and fix their stretching parameters and use the bending stiffness and area weight as initialization for our cloth material. Specifically, using their respective parameters we confine a search space that is used during our parameter refinement. We determine ρ A ∼ Uniform(0.10, 0.17) kg m −2 after consulting various flag materials at online retailers. And, we restrict the range of the bending stiffness coefficients by multiplying the base material's k e in (3) by 10 −1 and 10 to obtain the most flexible and stiffest material respectively. As the bending coefficients have a complex effect on the cloth's appearance, we independently optimize the 15 bending coefficients instead of only tuning the one-dimensional multiplier. The full parameter search space is listed in Table 1.

Spectral Decomposition Network
The dominant source of variation is in the geometry of the waves in cloth rather than in its texture. Therefore, we seek a perceptual model that can encode the cloth's dynamics such as high-frequent streamwise waves, the number of nodes in the fabric, violent flapping at the trailing edge, rolling motion of the corners and its silhouette [37,42,13]. As our goal is to measure sim-to-sim and sim-to-real similarity, a crucial underpinning is that our embedding function is able to disentangle and extract the relevant signal for domain adaptation [32,20]. Therefore, we propose modeling the spatial distribution of temporal spectral power over the cloth's surface. Together with direction awareness, this effectively characterizes the traveling waves and flapping behavior from visual observations. Spectral Decomposition Layer. The proposed solution is a novel spectral decomposition layer that distills temporal frequencies from a video. Specifically, similar to [34], we treat an input video volume as a collection of signals for each spatial position (i.e. H × W signals) and map the signals into the frequency domain using the Discrete Fourier Transform (DFT) to estimate the videos' spatial distribution of temporal spectral power. The DFT maps a signal f [n] for n ∈ [0, N t −1] into the frequency domain [30] as formalized by: We proceed by mapping the DFT's complex output to a real-valued representation. The periodogram of a signal is a representation of its spectral power and is defined as I(ω) = 1 N t |F( jω)| 2 with F( jω) as defined in (4). This provides the spectral power magnitude at each sampled frequency. To effectively reduce the dimensionality and emphasize on the videos' discriminative frequencies, we select the top-k strongest frequencies and corresponding spectral power from the periodogram. Given a signal of arbitrary length, this produces k pairs containing I(ω max i ) and ω max i for i ∈ [0, k] yielding a total of 2k scalar values.
Considering an input video volume, treated as a collection of H × W signals of length N t , the procedure extracts the discriminative frequency and its corresponding power at each spatial position. In other words, the spectral decomposition layer performs the mapping R C×N t ×H×W → R 2kC×H×W . The videos' temporal dimension is squeezed and the result can be considered a multi-channel feature map -to be further processed by any 2D convolutional layer. We reduce spectral leakage using a Hanning window before applying the DFT. The batched version of the proposed layer is formalized as algorithm in the supplementary material.  Figure 5. Overview of our SDN architecture s φ (x) for learning the physical correspondence between the simulation and real-world observation of dynamic flags. Given a 3D video volume as input, we first apply a 0 th -order temporal Gaussian filter followed by two directional 1 st -order Gaussian derivative filters and then spatially subsample both filtered video volumes by a factor two. The proposed spectral decomposition layer then applies the Fourier transform and selects the maximum power and corresponding frequencies densely for all spatial locations. This produces 2D multi-channel feature maps which we process with 2D ResNet blocks to learn the embedding.
Embedding Function. The specification of s φ (x), with the spectral decomposition layer at its core, is illustrated in Figure 5. First, our model convolves the input video x with a temporal Gaussian filter followed by two spatially oriented first-order derivative filters. Both resulting video volumes are two-times spatially subsampled by means of max-pooling. Successively, the filtered video representations are fed through the spectral decomposition layer to produce spectral power and frequency maps. The outputs are stacked into a multi-channel feature map to be further processed by a number of 2D convolutional filters with trainable weights φ. We use 3 standard ResNet blocks [16] and a final linear layer that maps to the R D e embedding space. We refer to our network as Spectral Decomposition Network (SDN). [31] and is publicly available1. Unless mentioned otherwise, all network inputs are temporally sampled at 25 fps. After that, we use a temporal Gaussian with σ t = 1 and first-order Gaussian derivative filters with σ x,y = 2. For training the embedding function with the contrastive loss, we adopt a margin of 1 and use the BatchAll sampling strategy [18,12]. The spectral decomposition layer selects the single most discriminative frequency (i.e. k = 1). Adding secondary frequency peaks to the feature maps did not yield substantial performance gains. The size of our embeddings is fixed (D e = 512) for the paper. Input video clips of size 224 × 224 are converted to grayscale. We optimize the weights of the trainable ResNet blocks using Adam [22] with mini-batches of 32, learning rate 10 −2 and a weight decay of 2 · 10 −3 .

Real and Simulated Datasets
Real-world Flag Videos. To evaluate our method's ability to infer physical parameters from real-world observations, we have set out to collect video recordings of real-world flags 1https://tomrunia.github.io/projects/cloth/ with ground-truth wind speed. We used two anemometers ( Figure 6) to measure the wind speed at the flag's position. After calibration and verification of the meters, we hoisted one of them in the flagpole to the center height of the flag to ensure accurate and local measurements. A Panasonic HC-V770 camera was used for video recording. In total, we have acquired more than an hour of video over the course of 5 days in varying wind and weather conditions. We divide the dataset in 2.7K train and 1.3K non-overlapping test video clips and use 1-minute average wind speeds as ground-truth. The train and test video clips are recorded on different days with varying weather conditions. Examples are displayed in Figure 6 and the dataset is available through our website.

FlagSim Dataset.
To train the embedding function s φ (x) as discussed in Section 3.1, we introduce the FlagSim dataset consisting of flag simulations and their rendered animations. We simulate flags by random sampling a set of physical parameters θ from Table 1 and feed them to ArcSim. For each flag simulation, represented as sequence of 3D meshes, we use Blender [5] to render multiple flag animations x i sim at different render settings ζ i . We position the camera at a varying distance from the flagpole and assert that the cloth surface is visible by keeping a minimum angle of 15 • between the wind direction and camera axis. From   [9]. We regress the wind speed (v w ∈ θ e ) in the range 0 m s −1 to 15.5 m s −1 and report numbers on the evaluation set.

Results and Discussion
Real-world Extrinsic Wind Speed Measurement (θ e ). We first assess the effectiveness of the proposed spectral decomposition network by measuring the wind speed on the recently proposed real-world flag dataset by Cardona et al. [9]. Their method, consisting of an ImageNet-pretrained ResNet-18 [16] with LSTM, will be the main comparison. We also train ResNet-18 with multiple input frames, followed by temporal average pooling of the final activations [21]. After training all methods, we report the root mean squared error (RMSE) and accuracy within 0.5 m s −1 (Acc@0.5) in Table 2. While our method has significantly fewer parameters (2.6M versus 11.2M and 42.1M), the SDN outperforms the existing work on the task of real-world wind speed regression. This indicates the SDN's effectiveness in modeling the spatial distribution of spectral power over the cloth's surface and its descriptiveness for the task at hand. The supplementary material contains the results on our FlagSim dataset. (θ i , θ e ). We evaluate the physical similarity embeddings after training with contrastive loss. To quantify the ability to separate examples with similar and dissimilar physical parameters, we report the triplet accuracy [45]. We construct 3.5K FlagSim triplets from the  test set as described in Section 3.1. We consider the SDN trained for video clips of a varying number of input frames and report its accuracies in Table 3. The results indicate the effectiveness of the learned distance metric to quantify the physical similarity between different observations. When considering flags, we conclude that 30 input frames are best with a triplet accuracy of 96.3% and therefore use 30 input frames in the remainder of this paper. In Figure 7 we visualize a subset of the embedding space and observe that the flag instances with low wind speeds are clustered in the top-right corner whereas strong wind speeds live in the bottom-left.

Real-world Intrinsic Cloth Parameter Recovery (θ i ).
In this experiment, we assess the effectiveness of our SDN for estimating intrinsic cloth material properties from a realworld video. We compare against Yang et al. [55] on the hanging cloth dataset of Bouman et al. [6] (Figure 2). Each of the 90 videos shows one of 30 cloth types hanging down while being excited by a fan at 3 wind speeds (W1-3). The goal is to infer the cloth's stiffness and area weight. From our SDN trained on FlagSim with contrastive loss, we extract the embedding vectors for the 90 videos and project them into a 50-dimensional space using PCA. Then we train a linear regression model using leave-one-out following [6]. The results are displayed in Figure 9. While not outperforming the specialized method of [55], we find that our flag-based features generalize to intrinsic cloth material recovery. This is noteworthy, as our SDN was trained on flags of lightweight materials exhibiting predominantly horizontal motion. Bottom: development throughout the refinement process for 50 iteration steps. We plot the distance between simulation and target instance in the embedding space and the estimated wind speed (m s −1 ). We annotate the ground-truth wind speed with a dashed line. As the plot indicates, the refinement process converges towards the real wind speed.

Real-world Combined Parameter Refinement (θ i , θ e ).
Putting everything together, our goal is measuring physics parameters based on real-world observations. We demonstrate the full measurement procedure (Figure 3) by optimizing over intrinsic and extrinsic model parameters (θ i , θ e ) from real-world flag videos (and present hanging cloth refinement results in the supplementary material). First, we randomly sample a real-world flag recording as subject of the measurement. The parameter range of the intrinsic (16×) and extrinsic (1×) is normalized to the domain [−1, +1] and are all initialized to 0, i.e. their center values. We fix the render parameters ζ manually as our focus is not on inferring those from real-video. However, these parameters are not carefully determined as the residual blocks in the embedding function can handle such variation (Figure 7). In each step, we simulate the cloth meshes with current parameters θ i , θ e and render its video clip with fixed render parameters ζ . Both the simulation and real-world video clips are then projected onto the embedding space using s φ (x), and we compute their pairwise distance (1). Finally, the Bayesian optimization's acquisition function (Section 3.2) determines where to make the next evaluation θ i , θ e ∈ [−1, +1] to maximize the expected improvement, i.e. improving the measurement. The next iteration starts by denormalizing the parameters and running the simulation. We run the algorithm for 50 refinement steps. In Figure 8, we demonstrate our method's measurements throughout optimization. Most importantly, we observe a gradual decrease in the pairwise distance between simulation and real-world example, indicating a successful measurement of the physical parameters. Importantly, we note that the wind speed converges towards the ground-truth wind speed within a few iterations, as indicated with a dashed line. More examples are given in the supplementary material.

Conclusion
We have presented a method for measuring intrinsic and extrinsic physical parameters for cloth in the wind without perceiving real cloth before. The iterative measurement gradually improves by assessing the similarity between the current cloth simulation and the real-world observation. By leveraging only simulations, we have proposed a method to train a physical similarity function. This enables measuring the physical correspondence between real and simulated data. To encode cloth dynamics, we have introduced a spectral decomposition layer that extracts the relevant features from the signal and generalizes from simulation to real observations. We compare the proposed method to prior work that considers flags in the wind and hanging cloth and obtain favorable results. For future work, given an appropriate physical embedding function, our method could be considered for other physical phenomena such as fire, smoke, fluid or mechanical problems. We report the Pearson correlation coefficients (higher is better) between predicted material type and both ground-truth stiffness/density on the Bouman et al. [6] hanging cloth dataset. The dashed red line indicates human performance as determined by [6].

A. Table of Contents
The supplementary material has the following content: • Appendix B and Algorithm 1: Formalization of the batched version of the spectral decomposition layer.
• Appendix C and Figure 10: Details on the collection of real wind speed measurements and video recordings.
• Appendix D: Details on the video data augmentation and optimization for the training of all networks.
• Appendix E and Figure 4: Additional experiments on refined measurements for the hanging cloth dataset and details on our ClothSim dataset.
• Table 4: Supplement to Table 2 in the main paper with wind speed regression results on our FlagSim dataset.
• Figure 3: Supplement to Figure 8 in the main paper to include more refined measurement examples.
• Listing 1, Listing 2 and Listing 3: Specification of ArcSim scene and material configuration files for flag and cloth simulations. •

B. Spectral Decomposition Layer
To support Section 4.3 in the main paper, we formalize the batched version of the spectral decomposition layer in Algorithm 1. Given a batch of video clips as input, our spectral layer applies the discrete Fourier transform along the temporal dimension to compute the temporal frequency spectrum. From the periodogram, we can select the topk strongest frequency responses and their corresponding spectral power. The resulting frequency maps and power maps all have the same dimensions and can, therefore, be stacked as a multi-channel image. These tensors can be further processed by standard 2D convolution layers to learn frequency-based feature representations. The proposed layer is efficiently implemented in PyTorch [31] to run on the GPU using the torch.irfft operation. The source code is available through the project website.

C. Real-World Flag Dataset Acquisition
We here describe our data acquisition to obtain real-world wind speed measurements serving as ground-truth for our To verify the correctness of both anemometers, we have checked that both wind meters report the same wind speeds before usage. After that, we use the SkyWatch BL-400 anemometer for our measurements as it measures omnidirectional which is more convenient. We hoisted the anemometer in a flag pole such that the wind speeds are measured at the same height as the flag. Wind speed measurements are recorded at 1 second intervals and interfaced to the computer. In Figure 10 we display an example measurement and report the dataset's wind speed distribution. For the experiments (Section 6, main paper, last section), we randomly sample video clips of 30 consecutive frames from our video recordings and consider the ground-truth wind speed to be the average over the last minute. This procedure ensures that small wind speed deviations and measurement errors are averaged out over time.
To capture the videos, we use a Panasonic HC-V770 video camera. The camera records at 1920 × 1080 at 60 frames per second. We perform post-processing of the videos in the following ways. Firstly, we temporally subsample the video frames at 25 fps such that the clips are in accordance with the frame step size in the physics simulator. Moreover, we assert that the video recordings are temporally aligned with the wind speed measurements using their timestamps. Secondly, we manually crop the videos such that the curling flag appears in the approximate center of the frame. After this, the frames are spatially subsampled to 300 × 300, again in agreement with animations obtained from the render engine.

D. Training Details
Data Augmentation. The examples in the FlagSim dataset are stored as a sequence of 60 JPEG frames of size 300 × 300. During training, when using less than 60 input frames (30 is used in all experiments), we randomly sample N t successive frames from each video clip. This is achieved by uniform sampling of a temporal offset within the video. After this, for the sampled sequence of frames, we convert images to grayscale, perform multi-scale random cropping and apply random horizontal flipping [47] to obtain a N t × 1 × 224 × 224 input clip. Finally, we subtract the mean and divide by the standard deviation for each video clip.
Optimization Details. We train all networks using stochastic gradient descent with Adam [22]. We initialize training with a learning rate of 10 −2 and decay the learning rate with a factor 10 after 20 epochs. To prevent overfitting, we utilize weight decay of 2 · 10 −3 for all networks. Training continues until validation loss plateaus -typically around 40 epochs. Total training time for our spectral decomposition network is about 4 hours on a single Nvidia GeForce GTX Titan X. When training the recurrent models [9,55] we also perform gradient clipping (max norm of 10) to improve training stability.

E. Experiments on Hanging Cloth Video
Our real-world flag dataset enables us to evaluate our method's measurement performance of external parameters (v w ∈ θ e ). However, the cloth's internal parameters are unknown and cannot be evaluated beyond visual inspection. Therefore, we also perform experiments on the hanging cloth dataset of Bouman et al. [6]. The authors have carefully determined the internal cloth material properties, which we can leverage for quantitative evaluation of our simulatedrefined measurements. Specifically, we assess our method's ability to measure the cloth's area weight (kg m −2 ). The method is identical to that explained in the main paper with its results presented in the final experiment of Section 6. However, we retrain the embedding function s φ (x) on a dataset of hanging cloth simulations, which we refer to as ClothSim. In this section, we will briefly discuss the characteristics of this dataset and report experimental results.
ClothSim Dataset. Following the same procedure as for the FlagSim dataset, we additionally generate a dataset of simulated hanging cloth excited by a constant wind force. The main difference between the FlagSim dataset is the wider variety of cloth material. Specifically, we use all the materials presented in [46] available in ArcSim. The increased diversity allows us to model the dynamics in real-world hanging cloth recording [6]. Our dataset shares similarity with the simulated hanging cloth dataset of [55]. However, in their work, the dataset is employed to train a classifier for predicting the material class. In Listing 2 and Table 5 we present an exhaustive overview of the simulation and render parameters that were used for generating the dataset.
Real-world Parameter Refinement (θ i , θ e ). Given the embedding function s φ (x) trained on ClothSim using contrastive loss, we run our refined measurement experiment on the hanging cloth dataset of Bouman et al. [6]. Our goal is to measure the cloth's area weight as we have access to its ground-truth measurement. Unlike for our real-world flag dataset, we do not know the true wind speed beyond the setting of the industrial fan that was used for exciting the fabric artificially. In Figure 4 we report the results for 3 randomly sampled real-world videos.  . Results for our hanging cloth refined-measurements for a random target video capturing the hanging cloth as recorded by Bouman et al. [6]. The five images are the center frames of the real-world target video (left) and simulations throughout the refinement process after t = 0, 10, 20, 40 optimization steps. Optimization is performed over all 16 intrinsic cloth parameters θ i and 1 external wind speed θ e . We plot the estimated cloth material area weight (kg m −2 ) and wind speed velocity (m s −1 ), although we only have access to the true material area weight (dashed horizontal line). For this dataset, the wind speed has three settings of increasing wind speed: W1, W2 and W3.   Additional results for our FlagReal refined-measurements for a random target video capturing the flag in the wind (corresponding to Figure 8 in the main paper). The five images are the center frames of the real-world target video (left) and simulations throughout the refinement process after t = 0, 10, 20, 40 optimization steps. Optimization is performed over all 16 intrinsic cloth parameters θ i and 1 external wind speed θ e . We only visualize the simulated wind speed as it is the only parameter for which we have ground-truth (dashed line). Top row: successful optimization example; although the scale between real observation and simulation is different, our method is able to precisely determine the external wind speed. The real-world video has a ground-truth wind speed of 2.46 m s −1 while the refinement procedure finds a wind speed of 2.34 m s −1 in less than 10 optimization steps. Center row: Another successful optimization example. The real-world video has a ground-truth wind speed of 2.96 m s −1 while the refinement procedure finds a wind speed of 2.36 m s −1 after 45 refinement steps. Bottom row: failure case; even though after 45 steps the wind speed is approximately correct, the optimization procedure has not converged. Listing 3. The ArcSim material configuration [28] as JSON file to be consumed by the simulator. As base material, we use "camel ponte roma" with its properties determined in the mechanical setup by [46]. This file specifies the cloth's area weight, bending stiffness coefficients and stretching coefficients. As flags are of strong, weather-resistant material, we optimize over the area weight (1×) and bending parameters (15×). Together these 16 parameters define θ i . For hanging cloth, we also keep the bending parameters fixed to constrain the number of free parameters. The flag/cloth texture Sampled from 12 countries Sampled from [55]