{'fraction_non_alphanumeric': 0.03264529093440976, 'fraction_numerical': 0.011757240309530945, 'mean_word_length': 4.699417319670484, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Descriptive Modeling of Textiles using FE Simulations and Deep Learning

In this work we propose a novel and fully automated method for extracting the yarn geometrical features in woven composites so that a direct parametrization of the textile reinforcement is achieved (e.g., FE mesh). Thus, our aim is not only to perform yarn segmentation from tomographic images but rather to provide a complete descriptive modeling of the fabric. As such, this direct approach improves on previous methods that use voxel-wise masks as intermediate representations followed by re-meshing operations (yarn envelope estimation). The proposed approach employs two deep neural network architectures (U-Net and Mask RCNN). First, we train the U-Net to generate synthetic CT images from the corresponding FE simulations. This allows to generate large quantities of annotated data without requiring costly manual annotations. This data is then used to train the Mask R-CNN, which is focused on predicting contour points around each of the yarns in the image. Experimental results show that our method is accurate and robust for performing yarn instance segmentation on CT images, this is further validated by quantitative and qualitative analyses.


Introduction
Thanks to their elevated specific properties, woven composite materials are widely used in many fields where the final part weight is of upmost importance (e.g., aeronautical, aerospace and submarine industries) [1,2]. These structured composites are obtained by weaving yarns (e.g., carbon fiber bundles) in a prescribed pattern. Given that this weaving pattern is designed so as to achieved certain material properties, a great interest has been given to properly describe the effective manufactured arrangement of yarns. Such is the goal of numerical modeling of textiles: to obtain a geometrical description of the yarns (i.e., yarn paths and cross sections) for the final textile configuration. This can be done using predictive approaches that employ numerical simulations, or descriptive ones that use (X-ray) tomography images to describe real manufactured parts.
The simplest group of predictive approaches considers the well-established Textile Geometrical Preprocessors (TGP), such as WiseTex [3,4] and TexGen [5]. These approaches model the yarns following only geometrical considerations, hence they constitute a very idealized representation. A more advanced group of predictive approaches are those that employ mechanical behavior laws, either at the sub-meso [6,7,8,9,10,11] or mesoscale [12] (scale of the textile).
On the other hand, descriptive approaches leverage multiple image and data processing techniques to extract useful information from the acquired volume images. As such, they can go further than only image segmentation. Moreover, most of these approaches are based on a local image descriptor called the structure tensor [13,14,15,16,17,18]. The local principal image orientations and degree of anisotropy that result from such operation allow to differentiate image voxels belonging to the reinforcement (highly anisotropic) and matrix (highly isotropic). Hence, it can help distinguishing different yarn orientations (e.g., warp and weft) whenever the image contrast and resolution allow it.
In cases in which these derived features cannot be easily separable (e.g., simple threshold on anisotropy level or orientation angle), clustering techniques have had some success. These approaches aim at finding complex rules that partition the domain spawned by these derived features. In such sense, k-means clustering [14] and Gaussian Mixture Models (GMM) [15,16,17] have proven well suited for separating the different yarn orientations (e.g., warp and weft yarns). In the case of 3D woven composites with binder yarns, further work is necessary due to the similarities they share with warp yarns (e.g., the same orientation) [17]. It should be noted that, these methods are only capable of distinguishing between yarn orientations but not of isolating each yarn. For cases in which the yarns are not heavily compacted, elementary morphological operations can be employed [19]. However, in more complex cases, such as two yarns of the same type being in contact, some manual intervention may be necessary [20]. Besides, if an individual identification of the yarns is desired; that is, assign an ID to them, even more steps are required, e.g., semi-automatic identification on the first slice and propagation [16].
At this point, the output of these methods are no more than binary image masks. Hence, unless a voxel-based formalism is employed for the material simulation [21], they do necessitate further processing in order to provide usable textile models such as Finite Element (FE) meshes [13,15,17]. It is important to note that this category of descriptive methods presents some challenges. Given that these approaches do not account for spatial information, they often require some clean-up post-processing steps such as morphological operations or smoothing before and after the construction of the textile models. Indeed, the concatenation of these operations, while useful and effective, can lead to complex pipelines that would seem to demand heavy re-parametrization for every new analyzed sample (e.g., re-identification of the mixture model).
Other novel descriptive approaches based on textile models have recently been proposed [22,23]. In both cases an a priori 3D model of the textile is progressively deformed (optimized) so as to conform to the observed sample. The starting model is obtained by manually extracting the yarn paths and using constant elliptical cross sections. These latter can progressively be adapted using polygonal shapes until the correct yarn shape is obtained [23]. This group of methods (based on a priori ) do present a great advantage over the previous ones since they directly provide the sought textile model without requiring extra operations.
Finally, some recent descriptive approaches based on deep neural networks have been proposed for performing the semantic segmentation task [24,25]. Indeed, given the unprecedented performance of deep learning techniques in other domains (e.g., medical imaging, natural images), it seems only natural to apply it into the analysis of materials. In both cases, "ready-to-use" network architectures were employed: Sinchuk et al. [24] used the U-Net architecture [26] available on the DragonFly software, while Ali et al. [25] implemented DeepLab v3+ [27] in MATLAB using the Deep Learning Toolbox. Similarly, they both employ a 2D strategy for the segmentation (i.e., independently feed slices of the stack to the network), albeit Ali et al. [25] used a less common top-down approach (the slices are taken in the thickness direction). To the best of our knowledge, currently, these are the only deep learning-based descriptive methods in the literature.
Moreover, due to the supervised nature of the training that the networks undergo, these approaches require the constitution of a properly annotated database. This means that for a given tomography slice the corresponding pixel-wise masks are known for each class (e.g., warp yarns, weft yarns, matrix). Sinchuk et al. [24] employ the "polygon selections" function in ImageJ while Ali et al. [25] use the MATLAB Image Labeler Tool. It should be noted that these semantic segmentation approaches are similar to those based on the structure tensor in the sense that they only segment the yarn orientations (i.e., no individualization of yarns) and would require extra post-processing steps to generate usable textile models. Now, the aim of the method proposed here is to address the yarn segmentation problem as an instance segmentation one using deep learning methods. Here, instance segmentation refers to the process of simultaneously solving the problem of object detection as well as that of semantic segmentation. Additionally, we will achieve such results by treating the problem of segmentation as one of keypoint estimation. Hence, no binary masks of the yarns will be obtained, but rather a set of points describing the outlines of all yarn sections (instances). To the best of our knowledge, we are the first to propose a method of this nature. The advantage of this approach is that it provides the yarn cross sections as a collection of control points that can be directly used for constructing the textile model (e.g., FE mesh).
Nonetheless, an inherent challenge of supervised deep learning algorithms is the need of considerable amounts of annotated datasets during training. This crucial information is very expensive to obtain since considerable manual operations are required for properly annotating all yarns in an image (i.e., delimiting their cross sections and paths), even when some semi-automated procedures are employed [25]. Here, we propose to circumvent this problem by using a limited amount of (imprecise) annotations in a weakly supervised manner rather than a fully supervised one. In particular, we will employ a deep neural network for creating synthetic CT images from these weakly annotated images, we refer to these as pseudo-CT images. After training, the network will be used on "annotated" images that come from FE simulations (the yarn information is numerically accessible). As such, we are now capable of generating many pseudo-CT images where the annotations come from simulations rather than by manual operations. Finally, another deep network will take this completely numerical database for training the instance segmentation model, which will later be used on real images.
The available materials and an overview of the method will given in section 2. The generation of pseudo-CT images and the instance segmentation steps will be detailed in sections 3 and 4 respectively. The use of these results for obtaining textile models will be discussed in section 5. Finally, a quantitative evaluation of the obtained results will be proposed in section 6.

Tomographic volume
The 3D woven fabric studied in this work is a ply-to-ply angle-interlock, composed of 75 carbon fiber yarns: 39 warp and 36 weft. All yarns are of the same type and size. The warp yarns are distributed alternately in a sequence of 4 and 3 yarns in two consecutive columns. Similarly, two consecutive weft columns are composed of 5 and 4 yarns respectively. In this sample, there are 11 warp planes and 8 weft columns.
The sample was scanned with a GE Phoenix-Xray tomograph (GE v|tome|x L300) at a resolution of 20µm. The reconstructed image I CT is 1698 × 1814 × 402 voxels in size, and is normalized so that its dynamic range is [0, 1]. In the following, the X, Y and Z axes are aligned with the warp, the weft and the thickness orientations respectively. This information is also given in table 1, which summarizes all relevant information for the employed images.

Simple textile model
The creation of a geometrical textile model from the acquired CT volume I CT was done in a semiautomatic manner. First, the yarn paths are manually identified by locating the coordinates of their crossover points where crimp is maximum ; that is, at intersections of warp and weft directions. These are defined for each column in the orthogonal direction (e.g., 8 cross-over points for the warp and 11 for the weft) and are manually placed at the approximate center of the corresponding cross section. Then, a B-spline interpolation is used with this first set of control points, and if required, each yarn path can be further adjusted by adding more control points. As such, the final number of control points is not necessary identical for each yarn. These local adjustments are rather necessary in zones with high crimp.
No identification of the yarn cross section is performed for this model. Hence a uniform elliptical shape is attributed to all 75 yarns, uniform all along each yarn and equal for all yarns. A first estimation of the ellipse radii is obtained by manually measuring some yarn sections in a collection of carefully selected CT volume images. Since in practice, the intra-yarn fiber volume fraction is not constant all along the yarn path, the radii are then adjusted to better match the theoretical overall fiber volume fraction value of the composite.

Mechanical simulations
Numerical Finite Element (FE) calculations are carried out using the in-house software Multifil [11] devoted to simulating mechanical behaviors of entangled fibrous materials using a quasi-static formulation. It is based on the solution of the mechanical equilibrium of a collection of beams in contact-friction interaction. Here, each yarn is modeled using 61 beams (also called virtual fibers or macro-filaments) twisted together so as to mimic the internal cohesion of fibers. Given that the employed number of virtual fibers is sufficient for properly describing the deformation of the yarn cross section [28], each beam is described by the isotropic Saint-Venant law.
This algorithm proposes a novel strategy for obtaining the "as-woven" state. Instead of starting from a nominal configuration close to the "as-woven" state (as most methods do [9,29]), the starting configuration places all yarns in the same plane [8,11]. Hence, the algorithm progressively separates the virtual fibers using a contact-friction algorithm until reaching the global mechanical equilibrium of the targeted weaving pattern.
The goal of the mechanical simulations presented here is to provide a rich and useful information for the subsequent deep learning models. As such, instead of just using the textile model at the "as-woven" state, we propose to perform further compaction simulations on this model. These latter will provide a much greater variability of textile density.
The weaving pattern used for the textile model is the same as in the acquired CT volume I CT . However, we only considered 64 yarns for the simulation: 28 warp and 36 weft (i.e., 11 warp less than in the original volume). Once the as-woven configuration is reached (mechanical equilibrium), a subsequent sub-mesoscale simulation of transverse compression is performed until a desired final compacted state is achieved. Assuming an initial thickness H init and a final H f inal , the increment compression is (H init − H f inal )/N steps with N steps = 12. These different compaction steps emphasize the variability of the textile morphology during a forming process (e.g., changes in both yarn trajectories and cross sections). Note that the simulation step closest to that of the original sample is the initial one (i.e., the as-woven state).
All the 12 simulation steps are considered so as to obtain their respective textile models. However, given that these simulations (determination of the initial configuration and transverse compression) are performed at the sub-mesoscale, the yarn envelopes need to be reconstructed from the virtual fibers. This considers the determination of the yarn paths and the estimation of the corresponding yarn cross sections.
Here, the yarn cross sections were computed at 38 and 49 locations along the warp and weft yarn paths. The, 100 points were used for estimating the polygonal envelopes for all cross sections, but only 10 equally sampled points are taken for the following. Additionally, the center point (the centroid of all cross section points) is considered for each cross section. Hence, 11 points are used for representing the yarn at each location.

Voxel conversion
The previous simple textile model and the 12 textile models from FE simulations are expressed using a parametric formalism. In order to convert them into "images", the in-house software REVoxel [21] is used for the voxel conversion. This software is capable of handling this type of parametric description since it was developed for predicting the mechanical properties of woven composite materials using voxel-FE models.
The produced volumes are called I GT (as in ground truth) for the simple textile model and I

(k)
F E with k ∈ [1,12] for the numerically obtained models. The assigned intensity levels (i.e., gray values) are the yarn IDs, hence the dynamic range is as large as the number of yarns (75 and 64, respectively). Moreover, these labeled images are calculated at the same resolution (i.e., voxel size) as that of the acquired CT volume I CT . Thus, I GT and I CT have the same size, but I (k) F E is 1401 × 1401 × 321 voxels in size. This smaller thickness (321 voxels) results from the average thickness from all twelve simulation steps and is positioned so that half-plane (orthogonal to the thickness) remains constant. Then, in the generated images, the compaction seems to be symmetric with respect to a center plane.

Extraction of 2D slices
In the following, all 3D volumes will be analyzed through 2D slices taken along the warp and weft orientations. As such, 3512 slices (1814 XZ slices and 1698 YZ slices) can be extracted from both images I CT and I GT . Whereas only 2802 slices (1401 XZ slices and 1401 YZ slices) can be extracted from each of the twelve images I This approach, known as 2.5D in the medical field, is extremely advantageous because it will allow to use existing architectures and frameworks, most of which are developed for dealing with 2D images rather than 3D volumes.

Overview of the method
As it can be seen in the diagram of the proposed method in figure 1, it is composed of two main parts: (i) the generation of a virtual dataset using a simple annotations, and (ii) the use of this dataset for training an instance segmentation model. In both cases, deep neural networks are employed.
The first part, detailed in section 3, consists in performing a sort of "inverse segmentation" that allows converting labeled images into realistic CT images (i.e., pseudo-CT). Here, the "simple labels" obtained from the simple textile model are used as input during the training phase for a U-Net network [26]. Then, the images obtained from mechanical simulations can be used as inputs for the network (during inference) so as to obtain the corresponding pseudo-CT images, which constitute the database for the next part.
The second part, detailed in section 4, employs a Mask R-CNN architecture [30] for performing the instance segmentation task. Here, the "perfectly" annotated images that come from the mechanical simulations and their corresponding pseudo-CT images constitute an entirely numerical source of information for the network. Indeed, the network only encounters real CT images when used during inference, never during training. Then, this network can be used on the acquired CT image for obtaining the description of the yarn cross sections using a set of keypoints.

Pre-processing of labeled images
The labeled images, obtained after voxelization of either the real CT or the FE simulations, contain voxels whose values correspond to the yarn ID. This information is not useful for the creation of the sought pseudo-CT images. Hence, they will be given new labels that represent the yarn type. Two strategies are possible: (i) distinguish between warp and weft orientations, or (ii) distinguish between yarns seen longitudinally and yarns seen transversally.
The first approach presents the advantage of "encoding" the yarn type in the labels. However, given the 2D nature of the analysis, any given yarn (e.g., a warp yarn) will sometimes be seen longitudinally (in XZ slices) and sometimes transversally (in YZ slices). This is problematic for the network since it needs to comprehend this duality. Indeed, our initial experiments using this approach failed to provide satisfactory results since it only seemed to work for one yarn direction but not the other.
On the other hand, the second approach has the advantage of always presenting the transversally seen yarns using the same label, independently of the yarn type (i.e., warp or weft). Also, even if the yarn type is "lost" in the new labels, it can still be deduced from the images. Indeed, as it can be seen in figures 5d and 6d the warp yarns (seen longitudinally) present a much higher wave amplitude than the weft yarns. This type of contextual information should be easily captured by a convolutional neural network. For this reason, the second approach is taken hereinafter.
The voxel-wise transformation function f tri is constructed so that for any 2D input image (e.g.,

Neural network architecture
The goal is to construct a mapping function f pseudo so that with f unet a slightly modified version of the U-Net network [26]. The U-Net architecture consists of a contracting part to capture context and a symmetric expanding part that enables precise localization. The contracting part follows the architecture of a typical convolutional network. It consists of repeated application of convolutions, each followed by a rectified linear unit (ReLU) and a max pooling operation. Next, the expanding part is composed of a sequence of up-convolutions and concatenations with high-resolution features from the contracting part (via skip-connections). This u-shaped architecture combines high-resolution (spatial) information from the contracting part and the condensed feature information that is successively up-sampled.
The modification proposed here is two-fold. First, a long skip-connection is added between the input and output of the last convolution layer. This type of connection has proven to be useful in image generation tasks without affecting computational times [31]. Second, a sigmoid activation function is added to this new output so that the function is bounded to the range [0, 1]. This can be written as for any input image u, with f unet the "classical" U-Net network [26] and Sigm the sigmoid function. Figure 2 schematizes the U-Net architecture. For completeness, the following functions are employed: with a and b denoting the sizes of input and output feature spaces. For further information or more details for each of these, along with the ReLU, Sigm, BatchNorm, MaxPool, Concat, Upsample, Dense and Conv operations, see the respective references [26,32].

Training of the network
The network is trained from scratch on a random 80% of the x GT and x CT datasets (2810 image pairs). The remaining 20% (702 image pairs) is used for validation purposes. No test set is needed here since the network will be used on the virtual images issued from simulation.
The optimization is performed with respect to a custom loss function for any input image u and any target image w, with a classical pixel-wise L 2 metric, and L per a perceptual loss [33] that encourages the output image to be "perceptually similar" to the target defined as The latter, also called feature reconstruction loss, aims to measure high-level perceptual and semantic differences between images [33]. It makes use of a loss network f vgg pre-trained for image classification, meaning that L per is itself a deep convolutional neural network. Here, f vgg is a subset of the 16-layer VGG network (e.g., VGG-16) [32] pre-trained on a subset of the ImageNet dataset [34], a large database with more than one million of manually annotated natural images. Selecting a subset of the complete network allows the evaluation of reconstructed features to be performed at an "inner" layer. Indeed, while using early layers will tend to favor visually indistinguishable images, later layers will favor image content and overall spatial structure over color, texture, and exact shape. Here, the chosen layer is last (the third) of the fourth block (out of five in total). This desired relu4_3 [32,33] output corresponds to the function f vgg , while f vgg corresponds to the output of the entire VGG-16 architecture, see figure 3 for details on the location of these functions.
In this context, the VGG-16 is being used as a "backbone" or feature extracting network. It should be noted that the choice of the network itself or the dataset it was trained on are relatively arbitrary. Yet, the sheer amount of annotated images in the ImageNet dataset and (pre-)training of the VGG-16 network guarantee a very general feature extractor useful for describing any type of images (from ImageNet or otherwise).
Finally, the training seeks to minimize L unet (x GT , x CT ) for the labeled images x GT as inputs and the CT images x CT as output. Note that the weights of the VGG-16 network do not need updating during training.

Implementation details
The experiment was carried out using the library PyTorch Lightning [35] since it provides a highlevel interface to PyTorch [36]. Both open-source Python libraries facilitate the access to deep learning experiments. In fact, the framework provided by PyTorch Lightning organizes PyTorch code so as to dissociate research from engineering. In practice this means that scalable deep learning models can easily be created and run on distributed setups while keeping the models almost intact (i.e., hardware agnostic).
Hence, once the models were tested at a small scale, they were easily deployed on 6 GPU Nvidia Tesla T4 (16Gb RAM) using the "Distributed Data Parallel" accelerator on PyTorch Lightning. The gradient descent algorithm used is Adam [37]. Moreover, the defaults proposed by PyTorch and PyTorch Lightning were used for all other parameters (e.g., cross-validation, event logging) and hyperparameters (e.g., learning rate, batch size).
Yet, special attention was required for the batches. As previously explained, the dataset is composed by the collection of XZ and YZ slices from both volumes. However, these slices have different sizes, 1698 × 402 and 1814×402 respectively. Then, whenever a YZ slice is (randomly) chosen to form a batch, it is horizontally trimmed so that they all have the same size of 1698 × 402 pixels.
The loss evolution for the training and testing datasets is shown in figure 4. Training was stopped after 40 epochs since it became clear that the "total" loss function on the validation dataset L val unet had attained a plateau. This empirical early-stop was performed so as to avoid over-fitting.

Results
Once the network has been trained, the VGG-16 backbone is no longer needed for inference, only the modified U-Net. As such, the model can be used for predicting on the already seen labeled images for the CT volume I GT (see figures 5 and 6), as well as on the twelve simulated volumes I F E (see figures 7 to 10). Moreover, the network can also be used on "hand-drawn" images representing other textile architectures as it can be seen in Appendix A. A visualization of the manner in which the network progressively obtains these results is given in Appendix B.
The performance displayed in figures 5 and 6 is of very high quality. The model is capable of generating extremely realistic pseudo-CT images, comparable to the real one. Moreover, the generated pseudo-CT images do not present any tomographic reconstruction artifacts (e.g., ring and cupping artifacts), unlike the original real ones.
It is important to note that the model learnt to transform the simple elliptical cross sections into more complex ones just by observing the CT volume, which clearly does not present yarns with perfect ellipses. Also, it is capable of better handling the proximal regions between yarns. As it can be seen, the in-house software used for voxel conversion [21] at times creates some undesirable "lips" in these regions (see figures 5a and 6a). Yet, the model has properly learnt to not reproduce this numerical artifact. In conclusion, the model is capable of overcoming both numerical limitations thanks to the learning process.
The results for the simulated textile, shown in figures 7 to 10 demonstrate that creating realistic CT images from completely numerical results is possible. The quality of the pseudo-CT images produced for a median compaction level (6th simulation step), shown in figures 7 and 8 is so convincing that, at a first glance, one could be fooled into taking them as real CT images. This is less true for the highest compaction level (12th simulation step), shown in figures 9 and 10, where the inner yarn texture is less faithful to such a case. However, it is important to remark that real samples displaying such extreme compression have not   It is also interesting to note that the case of the simulated textile presents a relevant challenge. As it was stated, the network has learnt to convert simple elliptical cross sections into realistic ones. This "problem" is non-existent for the images issued from our simulation, since the employed software (MultiFil) produces "correct" yarn cross sections. Interestingly, the model mostly respects the input yarn cross sections, but in some cases it deforms them further. This process is guided by some global spatial clues that the network had to learn for the images in the training dataset.     4. Instance segmentation using simulated images

Neural network architecture
We propose to solve the problem of yarn segmentation as one of keypoint estimation. As such, we employ the state-of-the-art Mask-RCNN [30] architecture that specializes in instance segmentation. This implies that a first task is to detect all yarns present in an image by defining their respective bonding boxes. Then, for each of these detected objects, the second task is to predict a fixed number of keypoints that describe each individual cross section. Currently, the most-performing approach for the first task is a two-step one: first several bounding box proposals are generated, and then each of these are classified as either containing a relevant object or not. A schematic representation of these steps is shown in figure 11.

Convolutional Features
The first step consists in computing some features on the image. This is done using a convolutional neural network (CNN), more precisely a fully convolutional network (FCN). This calculation is very fast since it computes features for several regions of the image while sharing computation between neighboring regions. In addition, by using a pre-trained FCN (i.e., the backbone), one can get a very powerful feature  extractor. We denote this network as f backbone and the features it generates as h = f backbone (u) for any input image u. Here we use a ResNet [38] pre-trained on ImageNet [34] as backbone.

Region Proposal Network
The Region Proposal Network (RPN), as its name suggest, is a CNN whose task is to propose regions. To do this, the network evaluates a fixed set of possible regions for every spatial unit on the convolutional feature map h provided by the backbone network f backbone . These fixed regions are known as anchors and are just bounding boxes with a predefined scale and aspect ratio, as shown in figure 12. Then, the RPN will apply a Conv layer that will output an intermediate feature map (with 256 dimensions at every spatial unit in our case). This feature map is then fed through two independent branches f rpnscores and f rpncoords so as to obtain the scores h scores = f rpnscores (h) and coordinates h coords = f rpncoords (h). The score in h scores represents the probability that one anchor contains an object or background (i.e., 2 parameters since encoded as a one-hot vector). The coordinates in h coords represent the transformation that needs to be applied to a given anchor so that it better contains the object. This transformation typically depicts 2D translation and 2D scaling (i.e., 4 parameters). As every spatial unit in the feature map h is given the same treatment, for a given set of α anchors, 2α scores and 4α coordinates will be produced per unit.

ROI Align
For all regions classified as containing an object by h scores in the previous stage, the information provided in f rpncoords along with the anchor type allow determining the ROI inside the feature map. However, all selected ROI will not have the same size (i.e., different anchors) nor be aligned with the spatial grid (i.e., sub-pixel precision). Then, a technique called ROI align [30] is used to warp them into a fixed size (e.g., 14 × 14) without loosing information about their spatial positioning. This approach essentially consists in performing bilinear interpolation on the impacted region so as obtain an intermediate image that is a multiple of the desired size (e.g., 28 × 28). Then a pooling operation (e.g., 2 × 2 max-pooling) provides the final image with the desired size. We denote this operation as f align and its output h align = f align (h ROI ) for any one region h ROI in h identified as containing an object by h scores . An example of this process is shown in figure 13.

Bounding Box Classification and Refinement
These newly obtained fixed-sized ROI are now employed for object classification and bounding box refinement. The first part consists in assigning one of multiple classes to the ROI. Given that in this case we only concern with one class of object, each ROI will be classified as containing either a "yarn object" or a "background object" (e.g., longitudinally seen yarns). This operation differs to the one in RPN where the classification was used to decide if an object was present or not. This classification task is performed with

Keypoints estimation
One of the major improvement that Mask-RCNN [30] brought was to introduce a parallel branch to the Box Classification and Refinement stage that was dedicated to the instance segmentation task. This parallel branch aims to predict the position of N kp = 11 keypoints for each instance. It consists in some Conv layers followed by an Upsample operation to obtain fixed-sized feature maps S for each keypoint (e.g., 11×56×56). Each feature map will represent the probability distribution of the localization of the corresponding keypoint. Then, the 2D coordinates of each keypoint are obtained by taking the position with maximum score for each of the feature maps.

Training of the network
First, it may be useful to recall the definitions of the Huber loss H δ , commonly used in robust regression thanks to its "soft" L 1 norm being less sensitive to outliers, the binary cross entropy loss H 2 , commonly used in binary classification, and the cross entropy loss H for discrete probability distributions. These are: Also, it should be noted that, the coordinates q coords defining the bounding boxes for each yarn are computed as a pre-processing step. These are easily obtained by extracting the minimum and maximum values of the keypoints coordinates of every yarn.
Training of the networks f backbone , f rpncoords and f rpnscores is achieved by optimizing the weighted sum of a classification loss L class and a regression loss L box .
The classification loss is defined as where i ∈ [1, α] is the index of the considered anchors, p i is the score given by the network for that particular anchor (obtained from h scores ), and the ground truth (GT) label q i is the boolean result (i.e., 1 or 0) of the operation IoU(anchor, q coords ) > τ , with the intersection over union (IoU, also known as Jaccard index) The optimization of this loss will encourage the RPN to correctly distinguish between anchors that contain objects and those that do not. Then, the regression loss is defined as where h coords represents the coordinates (4 parameters) predicted by the network, and q coords represents the GT coordinates for the bounding boxes. The optimization of this loss will tune the Box Classification and Refinement stage so that the anchors correctly include the objects (i.e., yarns). This loss is only computed for anchors that have been labeled as positives.
Next, training of the keypoint estimation branch is performed by optimizing the loss function where S represents the distributions estimated by the network for each keypoint in the form of feature maps (e.g., 11 × 56 × 56), and S coords represents the GT distribution for the known keypoints. This latter is computed for each keypoint as a one-hot vector of dimension 3136 reshaped into a 56 × 56 map. As such, the keypoint estimation task becomes a classification problem where the objective is to determine for each keypoint what position it belongs to (out of the 3136 possible locations). Finally, training is performed on the pseudo-CT images I pCT generated in the previous section and the corresponding keypoints obtained from FE (see figures 7d, 8d, 9d and 10d). All 33624 slices ((1401+1401)·12) are used for training of the networks.

Implementation details
The open source project Detectron2 [40] was used to implement this part of our system. This software system is also powered by the PyTorch [36] framework and implements many state-of-the-art object detection algorithms, such as the Mask R-CNN used here. This implementation works almost out-of-the-box with the default parameters, and only required adapting the learning rate to a smaller value of 2.5 × 10 −5 . However, a more substantial modification is required to modify the default number of 17 keypoints used for human pose estimation into the N kp = 11 keypoints used for describing the yarns (10 for the cross section and 1 for the yarn path). Finally, the model was trained on 1 GPU Nvidia Quadro P4000 (8Gb RAM). The loss evolution for the three losses detailed in the previous section is shown in figure 14. Clearly, L class is the first to attain a lower-valued plateau, followed by L box and L kpt . These latter two however, continue improving (albeit slowly) at the moment training was stopped.

Results
After training on the numerical dataset (from FE), the network can be used for predicting on the acquired CT volume I CT . First, an example of the predicted score maps for some keypoints is shown in figure 15. As it can be seen, the keypoints coordinates can be directly extracted by identifying the position at which the maximum score value occurs. Then, figure 16 displays the results of both stages, bounding box detection and keypoint estimation. The system performs remarkably well in the detection stage, as it shows no false positives (false yarn detection) but some few false negatives (missing yarn detection). An example of this can be seen in figure 16d, where a a yarn is not detected on the bottom left part of the image. This issue will be further explored in the next section.

Completion of missing sections
Since no consistency is enforced between slices, in some cases a yarn cross section may be detected in a given slice but not detected in the subsequent, and vice versa. Hence a simple procedure is proposed for "filling" the missing sections.
Given that the number of warp and weft yarns is known (39 warp and 36 weft), a simple look-up on the number of detected instances per slice will yield the "incomplete" slices (those with less yarn cross sections than expected) as well as the complete ones. Indeed, from the 3512 analyzed slices (1814 XZ slices and 1698 YZ slices), 37 (1%) present one more instance than excepted and only 25% of the total contain the exact number of instances. As table 2 shows, most slices (62%) are only missing one or two instances, with the warp direction being more precise than the weft one. Thus, 95% of all instances (all cross sections for all yarns for all slices) are correctly detected by the neural network.  It also is important to note that the missing yarn cross sections in those incomplete slices are in their majority (84%) from the upper-most and lower-most layers of the textile (i.e., the boundaries). Similarly, most of the cases in which a yarn cross section is missing in a large number of consecutive slices only happen at the lateral extremes of the volume, region in which the tomographic artifact is most prominent (see figure 5d). It should be noted that the Mask R-CNN was not trained on images presenting tomographic artifacts because the U-Net does not generate them. Thus, the network is prone to under-perform in this regions.
After simple examination of the predicted keypoints it became clear that the center keypoint often was not placed in the "center" of the yarn. Let us recall that no particular attention was brought to this center keypoint, rather all 11 were treated equally. Hence, the center keypoint will be recomputed as the centroid from the correctly predicted 10 contour points. These new center keypoints are used hereinafter.
Then, all pairs of subsequent slices that are composed of one complete and one incomplete slice are identified by computing the pair-wise distance matrix between the computed center keypoints present in both. This matrix helps pairing each yarn in the incomplete slice to the correct yarn in the complete slice by observing their distances. Next, all 11 keypoints for the unpaired yarns in the complete slice are simply copied onto the incomplete one. This procedure is repeated for all incomplete slices until none is left. It was observed that this naive "copy" mechanism was effective in sight of the good consistency of the network and the almost non existent complicated cases (e.g., a long portion of a missing yarn).

Mesh of yarn path
As it can be seen in figure 17a, the identified yarn paths are very ragged as a consequence of per-slice (i.e., independent) prediction. A simple median filtering is applied along each yarn path so as to smoothen them out. Next, in order to alleviate the meshes to be generated, a sub-sampling procedure is applied. Then, by retaining only 1 cross section every 25 slices, 66 (out of 1698) for warp yarns and 71 (out of 1814) for weft yarns are kept. The corresponding meshes (using L1 elements) can be seen in figures 17 and 19b.

Mesh of reinforcement
The generation of a conformal FE mesh for the yarns is a trivial process. Since the instances provided by the network contain the same number of keypoints describing they yarn cross sections, these can simply be connected. First, the (local) polar coordinates of the proposed keypoints are computed for obtaning a consistent order. Then, all pairs of keypoints in consecutive cross sections are connected (e.g., keypoint 1 in section 1 with keypoint 1 in section 2). As such, a mesh surface composed of quadrilateral (Q4) elements can be generated for each yarn. These can be seen in figure 19c.
Moreover, these surface elements can be connected to the previous (L1) mesh for the yarn path so as to construct a volume mesh for each yarn using triangular prisms (P6, wedges).
It should be noted these meshes can be taken directly for simulation purposes, or used as input geometries for a more advanced meshing software.

Mesh of composite
Here, the textile description (yarn path and associated cross sections) is fed to the in-house software REVoxel [21] in order to obtain an intermediate voxel-FE mesh. This latter is then fed to the in-house software Mirax (developed by Alain Rassineux at UTC) which outputs a fully conformal mesh using tetrahedral (T4) elements. The resulting mesh is showed in figure 19d, and a detailed view can be seen in figure 18. Figure 19: (a) Acquired CT volume I CT of the studied unit cell, and resulting FE meshes for (b) the yarn path, (c) the reinforcement (surface or volume), and (d) the composite (volume)

Validation of textile model
Usually, the validation of segmentation models is performed qualitatively [24,25]. However some quantitatively validation can be obtained by extracting relevant features (e.g., yarn path, section areas [41]), using similarity metrics [23] (i.e., comparing binary mask, local directions and yarn inter-penetration), or by assessing the mechanical or thermal properties of the textile [19,20].
In our case, the simple textile model acquired using the semi-automatic procedure described in section 2.2 can be used as ground truth for validating the performance of the algorithm. However, it should be noted that only the yarn paths can be considered as "true" since the cross sections are simple ellipses.
Hence, the Haussdorf metric will be used for the quantitative evaluation of the yarn paths, while the intra-yarn fiber volume fraction will be used for validation of the cross sections. Note that the assessment of the instance segmentation is performed on the final meshed model.

Path assessment
In order to provide a quantitative comparison, we propose to use the one-sided Hausdorff distance, a commonly used metric in 3D segmentation tasks defined as: with the sets A and B and δ H (A, B) measuring the distance from A to B.
In the present case, the measurement is computed all yarns independently. Thus, for each yarn i, the set A i contains the predicted center keypoints (after the smoothing process described previously), and the set B i contains the known keypoints (ground truth). The obtained distances δ H (A i , B i ) for all yarns i can be summarized in the histogram shown in figure 20a. Moreover, the distance for each yarn is displayed in figures 20b and 20c following an arrangement that mimics that of the textile (i.e., each cell represents a yarn). It should become clear that the vast majority of yarns have very small distances which validates the performance of the system. Some few yarns do have greater distances, but these are rather outliers with very few occurrences. These are highlighted in figure 20  the weft orientation is the one that poses the most problems. It is important to note however, that the three weft yarns responsible for the greatest Hausdorff distances had many consecutive missing slices (as manifested by the straight lines that result from the "copy" mechanism) and are located on the upper-most and bottom-most layers (as noted in section 5.1).

Section assessment
As said previously the assessment of yarn sections using classical metrics (e.g., IoU, F1-score) cannot be done since the simple textile model only considers elliptical cross sections. Nevertheless, the textile geometrical information, in the form of the intra-yarn fiber volume fraction (FVF), can be used for assessing the quality of the cross section estimation.
More precisely, instead of estimating the average FVF per yarn, we compute the section-wise FVF, or fiber section fraction (FSF), as the ratio between the nominal section and the estimated sections. The nominal section is computed as N · π · R 2 with N the number of fibers and R their radii, whereas the estimated section is computed as the polygonal area that the keypoints enclose.
The histogram for all 5130 estimated FSF values (66 × 39 + 71 × 36) is shown in figure 22a and the spatial distribution is shown in figure 23. It can be seen that for both warp and weft cases, the values are centered around 70% and range overall from 55 to 90%. These results are in good agreement with similar numerical studies without post-processing (50-95%) [17], yet experimentally it is difficult to exceed 70-75% [17,42,43]. This global overestimation of the FSF is in part due to a modest but systematic underestimation of the yarn cross section by the network. However, it is also explained by the method for the calculation of the FSF employed here. Indeed, a rigorous calculation of the section areas should be done orthogonal to the yarn path (even with low crimp), instead of using the ones aligned with the image axes. Similarly, a better estimation of sections area using higher order interpolators would prove useful for accurately computing the FSF value. It should be noted that cases in which the estimated section was smaller than the nominal one, the FSF was set to 100%. This explains the slight increase observed in the histogram for the highest values close to 100%. Moreover, considering that a FSF of 91% is the theoretical limit for a compact hexagonal arrangement in a 2D plane [43], sections that surpass this value could be considered as not physically-admissible. These constitute 2.6% of analyzed sections, they are highlighted in figures 22b and 22c and shown in figure 24. Figure 24: Visualization of sections with FSF higher than the theoretical limit of 91%

Conclusions and perspectives
A new segmentation method based on deep learning has been proposed in this study to predict the yarn envelopes in slices of CT images for 3D woven composites. It proposes a complete system that includes a first step of generation of synthetic images (U-Net), and a second step of instance segmentation (Mask R-CNN). These constitute the two main contributions of this study. Moreover, the fact that a fully annotated database is not available, but rather a weakly annotated one, is the reason why the first step is crucial.
We propose to construct a numerical database using the FE mechanical simulations, thus profiting from this substantial knowledge-base and the possibility of performing mechanically sound data augmentation by extracting different textile configurations (e.g., with variable density). Moreover, the dataset creation problem is treated as one of inverse segmentation from labeled images (issued from FE) into realistic pseudo-CT ones. This allows to circumvent the classical problem of requiring a heavily annotated dataset for training a proper instance segmentation model.
The formulation of instance segmentation task used here allows the individual identification of each yarn (e.g., by ID) as well as a 10-point parametrization useful for constructing a textile model. It should be noted that 10 keypoints were found to be sufficient for mechanic elastic homogenization, but the modifications made to the Detectron2 framework (for Mask R-CNN) allow to freely adjust this number of keypoints if required.
Having the system output the yarn keypoints facilitates the use of the readily available meshing strategies (linear, surface, volume) to obtain a description of only the reinforcement or the complete composite.
Then, we assess the results using the Hausdorff distance for the yarn path and the local intra-yarn fiber volume fraction (i.e., FSF) for the yarn cross sections. This two-step validation is dictated by the fact that only a weakly annotated dataset is available. Additionally, it is a novel strategy that quantifies the results from both, a relative criterion (Hausdorff) as well as an absolute one that represents a material property. These quantitative validation strategies show an overall good performance while also highlighting areas of improvement. For example the overestimation of FSF could be introduced as a penalization factor in the loss functions, thus avoiding unrealistically high values (e.g., higher than 91%). Similarly, the number of keypoints could be adapted according to the yarn or fabric types (i.e., higher or lower crimp fabrics).
A more substantial improvement could be to emphasize the center keypoint (i.e., differentiate it from the others) so that it truly represents the yarn path. Alternatively, this keypoint could be completely disregarded and just computed on the fly from the remaining 10 ones. Another approach would be to use a polar representation (with the origin at the center keypoint) with some constraints on the expected distances between contour keypoints. This implies a sizable departure from the classical Mask R-CNN and may be explored in future works.
Another area for improvement is the spatial coherence through multiple slices, this problem is explored in Appendix C. Indeed, the 2.5D approach taken here does not impose any connection between consecutive slices (other than them belonging to the same manifold). The simplest approach is then to adapt existing 2D and temporal frameworks so that the temporal coherence is interpreted as a spatial one for our case. This would have the great advantage of eliminating the completion step employed here, since the network would not omit nor add yarns sporadically. A more complex strategy would be to conceive a fully 3D system that would deal with 3D sub-volumes instead of slices. However, current works in this area are still developing and have yet to establish themselves.
Finally, the most direct method for improvement is to apply the system to other fabrics, such as other interlock types, injected samples, higher crimp fabrics or reinforcements with different yarn sizes. This would add robustness to the networks and allow them to generalize to increasingly more complex and unseen cases such as full composite parts. We are currently exploring this in order to better gauge the limits of this approach. and A.25i present two interesting characteristics. First, the "texture" for the longitudinal yarns presents clear discontinuities. This is a sign that the network cannot handle these elevated curvatures (nonexistent in the ply-to-ply). Second, the too-perfect positioning of the cross section yarns (e.g., completely vertical columns) presents a curious challenge to the network. It becomes clear that one of the contextual clues the network employs is the relative positioning of yarns. Yet, when these remain constant, the network produces images for these yarns that are almost the same (see lower right of figure A.25i). Figure A.25j show that the network is capable of "accommodating" the longitudinal yarns so as to comply with the variable size of the cross section yarns. Moreover, the two cases of contacting yarns present two different approaches: the first case (third layer) is to slightly separate the yarns, while the second one (fourth layer) calls for a more radical strategy, almost merging the yarns. Clearly this is related to the cases of yarn inter-penetration that were present in the training phase. Finally, a common quirk visible in all images is the addition of the film layer that was present on the sample during the acquisition (see bottom of figure 19a). While this "feature" may seem innocuous at first, it does imply that some effort during training was devoted to this banal aspect of the image. This is a nice demonstration of one of the challenges related to deep learning, the at-times lack of control over what the network learns and what not.

Appendix B. Visualization of learnt feature maps
In recent years, much effort has been dedicated to "explaining" the inner functioning of deep neural networks [44]. However, the deep nature of the model constitutes a challenge since the number of parameters in such a system are considerably many. For example, the U-Net model contains over 9 million parameters that describe the convolutional filters that are applied to any input image. While it is true that one could visualize these learnt filters as images, this would not be very useful in understanding what does each of them do. However, it is possible to visualize the intermediate outputs that follow these filters, these are called feature maps. As such, the transformations that an input image undergoes (as it progresses through the network) will become apparent. This strategy is chosen and the outputs of each block in the U-Net is shown in figure B.26, see figure 2 for reference on the convolutional blocks. The inverse relationship between size of the feature maps and their number becomes apparent. One of the issues of the current 2.5D approach is that no consistency is imposed between consecutive slices. As a consequence, when we seek to "gather" all results into a 3D space, some issues arise for the instance segmentation with Mask R-CNN (see section 5.1). For completeness, we will explore the impact it has on the generation of pseudo-CT images (i.e., the U-Net).
All YZ slices (orthogonal to warp direction) of the acquired CT volume I CT are fed to the U-Net and the outputs are stacked one after the other. This effectively generates a new volume I YZ pCT that can be observed in either the warp or the weft orientations (i.e., YZ or XZ). In the first case, we will observe the natural response of the network. While in the second one, the resulting slices show the consistency of the network through the volume. The same can be done for the weft orientation so as to obtain I XZ pCT with the equivalent orientations of interest XZ and YZ. The central regions of the these volumes (i.e., mid-slices) are shown in figure C.27. A striking difference can be observed between the volumes seen along the generated direction (i.e., figures C.27b and C.27e) versus those seen in the orthogonal one (i.e., figures C.27c and C.27f). However, the fact that the slices in figures C.27c and C.27f do not show many sharp gradients implies that, even if it is not enforced, the network does provide results that are relatively consistent. This highly desirable behavior implies that the network has learnt how to obtain contextual clues from the arrangement of yarns so that it knows how to "convert" the elliptical cross sections into realistic ones in a very consistent manner. If this was not the case, this conversion would be almost random.