{'fraction_non_alphanumeric': 0.02667637519030913, 'fraction_numerical': 0.008472893360693718, 'mean_word_length': 4.883177570093458, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Predicting realistic and precise human body models under clothing based on orthogonal-view photos

Accurate and realistic digital human body models are required by many research applications, for example in the areas of ergonomics, technology, and computer graphics. The already difficult research problem becomes more challenging if the individual subjects to be modelled are dressed in normal or loose-fit clothing. In this study, we present an intelligent two-phase method to customize 3D digital human body models based on two orthogonal-view photos of the customers. It integrates both image-based and example-based modelling techniques to create human body models for individual customers with precise body measurements and realistic appearance. It fills up the research gap of human model customization; without the need of taking body scan, any customers can create their 3D digital body models only based on their orthogonal-view photos in normal or loose-fit clothing. Experimental results have shown that the proposed method can efficiently and accurately customize human models of diverse shapes, meeting the specific needs of the clothing industry.


Introduction
An accurate digital human body model is a necessity in clothing related research or many ergonomic applications. Therefore, 3D digital human body modelling has received much research attention in last decades. In general, there are two classes of method for developing digital human body models, including construction methods and reconstruction methods. The key difference between the two is the involvement of body scanning. The former class of methods, namely, constructive methods, normally uses some projection devices to detect the customer's body shape and generate shape model. In order to obtain accurate body shape, the subjects being scanned must wear tight-fit clothing. Scanning devices are usually bulky and expensive. To overcome such limitations, researchers proposed reconstructive methods to capture, from images or size measurements, customer's body features, which are used to deform a template model using deformation technologies. Similar to the scanning-based construction methods, reconstruction methods also require customers being nude or dressing in tight-fit clothing for taking pictures or being measured. Recently, some methods were introduced to estimate the human body shape under clothing both from scanner or images. However, the results are not accurate enough for ergonomic or clothing applications.
In this study, we propose an intelligent two-phase method to customize 3D digital human body model based on two orthogonal-view photos of the customer. In our method, the customer needs not be in nude or with specialized tight-fit clothing before taking the photos, but can be dressed in normal or loose-fit clothing. To demonstrate the effectiveness of our method, we recruited a total of 15 female and 6 male subjects for experimental verification. Each subject was asked to have his/her body scanned and also have two photos taken with normal clothing in order to customize a human model using our method. The customized models and scans are compared in several aspects, including size measurements, areas and cross-section shape. We also compare our method with the method of [1], which can customize human models in tight-fit clothing. Experimental results have shown that the proposed method can customize human models of diverse shapes efficiently and accurately, meeting the specific needs of the clothing industry.

Related work
Accurate human body models are required in many research areas, such as clothing design, computer vision and ergonomic applications. A large number of research work have been reported in the literature for modelling human subjects in the past two or three decades. Since the 1980s, different types of scanners have been used to obtain accurate models of human body, e.g. head scanners, foot scanners and entire body scanners. Most scanners use either laser or white light to measure the depth information of the body surface for modeling purpose. Although scanning can obtain accurate and detailed 3D models, their applications are restricted by expensive, and often bulky, equipment.
In response to this, many researchers then proposed reconstructive modelling methods, which used information such as partial scans [2], images [4,5] and measurements [2,6] to estimate the 3D shape of the body skin surface by morphing a deformable template model. Most deformable models were developed by the so-called example-based methods [2,3], which learnt statistically the shape models from a large range of scan data. One of most famous methods is SCAPE [2], which combined both shape and pose deformation into one template. Nevertheless, SCAPE [2] can describe general shape features (e.g. slim or fat body type) of human subjects well, but it cannot effectively deform detailed local shape features (e.g. slopy shoulder or the waist level).
Recently, Zhu et al. [1] developed a method to customize body shape of individual subjects from two orthogonalview photos. They described the human body 3D shape by 17 key feature parts. For each part, they learnt a shape prediction function from a large scale of human body scans. They extracted all 17 local features of individual subject from the orthogonal-view photos, from which they assembled 3D shape according to the relative position of the 17 local features defined in the photos. Their method can customize human models with high measurement accuracy. Unfortunately, their method suffers from a drawback, similar to that of body scanning, that subjects must dress in tight-fit clothing in order to ensure the model accuracy.
In recent years, a few research work were reported that modelled human bodies under clothing. Hasler et al. [7] introduced a method for estimating the naked body shape from a dressed people's scan. It deformed an examplebased deformable model to fit the dressed scan iteratively until meeting some defined constraints. In 2010, Guan et al. [8] described a generative model that combines the contour of 2D human body and the deformation of clothing. This model can be used to estimate 2D body shape and underlying poses from images. Zhou et al. [9] presented a method that reshapes human bodies in images. It first detects body profile in the input image, and matches the profile with a morphable 3D model. Next, it morphs the 3D model so as to drive the rendering of images for reshaping the body of the subject in the image. The above two research work focused on estimating or deforming body shape under clothing in images, thus they mainly deal with 2D images. Extending the work to 3D space, Hasler et al. [10] developed a method based on a multi-linear model for 3D human pose and body shape deformation. It can estimate deformation parameters that drive the deformation of the 3D multi-linear deformable model, based on the silhouette captured from images. However, it is important to note all these statistical learnt deformable models can only capture the average shape deformation, but cannot reach the accuracy of body model customization. Therefore, none of these research applications is aim at obtaining an accurate 3D body shape model for a dressed subject in the images.

Methodology
In this paper, we proposed a two-phase method to customize human body models from photos in which subjects are dressed in normal or loose-fit clothing. In the first phase, we predict a 2D feature of the subject's body shape under clothing. Based on the predicted 2D feature, we construct a 3D body shape feature in second phase, and such 3D shape feature can be used to customize a detailed 3D model of the subject.

2D feature prediction
Our method aims to create accurate body models under clothing based on customer's photos. Since photos only contain 2D information, similar to other related studies, we define customer's 2D feature as body profilesfrontview and side-view profiles. Since most part of body profile is being covered by clothing in the photos, the most challenging task is to predict a complete profile based on some cues not being covered by clothes.
To do so, we first establish a database of normalized body profiles. The profiles are extracted from more than 5000 scans of real subjects with different body shapes. The database covers a wide range of body shapes; some example profiles are showed in Fig.1.
Second, we extract body feature cues from photos and use which to predict the complete profiles using the profile database. Due to the complex situations of input images, e.g. different clothing and noisy background, we realize the cue extraction by manually selecting features points on the photos. These feature points are categorized into two types: reference points and boundary points. The reference points are used to define locations of key body features on the images, such as neck and ankle locations. The boundary points are used to define the potential body shape of the subjects. Generally speaking, boundary points should be defined at locations where the body contours are not being covered by clothing. To allow more flexibility, users can manually define, based on their experience, the boundary point positions anywhere on the photos, even at positions where body contours are being covered by clothing (e.g. the red point in Fig.2(a)). It is another reason why manual feature extraction is preferred in our method. We normally define a total of 7-9 boundary points on the front-view and side-view photos, which are used to predict the complete 2D front-and side-view profiles. In our method, the extracted boundary points are normalized, using the defined reference points, to match the format of profiles in the database. The normalized boundary points are used to search profiles with similar features in the profile database by three steps: 1) calculating the difference  between boundary points and relevant feature points at the same level of the profile; 2) searching a number of N profiles with least total differences of all defined boundary points; and 3) synthesizing one single profile by combining the N selected profiles. Fig.2 (b) shows the example of estimated profile based on the points extracted in Fig.2(a).

3D feature reconstruction
Similar to the work of [1], we define a framework to represent 3D body shape feature. In [1], the framework was constructed by defining the locations of 17 key cross-sections of body shape model from the photos. However, this method is not suitable for modelling clothed subjects because of two following reasons: (1) most of the body contour is covered by clothing, which makes it very difficult to define all the 17 cross sections; (2) the location identification of a number of cross-sections is very tedious. In this paper, we define a framework involving 30 crosssections (as shown in Fig.2 (c)). The framework is built by first automatically recognising a small number of key cross-sections from 2D profiles, and then by interpolating extra cross-sections between recognized key crosssections.
The second phase of the proposed method on clothed subject body model customization is to reconstruct the subject's 3D body feature from the estimated 2D profiles. To do so, we first extract local and global body features from predicted profiles, obtained in the phase one. The local features refer to widths and depths of the 30 crosssections defining the 3D shape feature, and such cross-sectional widths and depths can be obtained by locating the relevant levels of the predicted front-view and side-view profiles. The global features refer to the relative positions of the key cross-sections. The cross-sectional widths and depths are then used to predict 3D shape of the particular cross-section. The prediction is based on relationship models learnt between local features and cross-sectional shape from a large scale of real human scanned models. With the predicted cross-sectional 3D shapes, we assemble or reconstruct 3D body shape feature using the global features extracted from the profiles. With the reconstructed 3D shape feature, a template model is then deformed to the shape of the particular subject using combined triangular Free Form Deformation (ct-FFD) algorithm [1]. Fig.2 (d) shows the deformed model of the example subject.

Loose-fit results
An experiment was carried out to evaluate the effectiveness of our model customization method. A total of 21 subjects, including 6 males and 15 females, were recruited. The subjects were found with diverse body builds that can be classified as underweight, normal and overweight. All subjects have their front-view and side-view photos dressing in loose-fit clothes taken for model customization. All subjects also had their body scanned by [TC] 2 NX16 scanning system for comparison purpose. Fig. 3 compares some customized results with corresponding scanned models. Table 1 shows the ranges of discrepancy between the extracted girth measurement of the customized models and that of the scanned models. It can be found that all mean discrepancies are lower than 2.0cm, which is within the size tolerance of the clothing industry. Apart from girth measurements, the key cross-section at chest/bust, waist and hips are compared in Fig. 4.

Tight-fit results
Since the proposed method can customize models using two-view photos, it can customize body models for subjects being dressed in loose-fit clothing in the photos, and it also can customize body models for subjects dressed in tight-fit clothing. We therefore customized all 30 models reported in Zhu et al. [1]. Fig. 5 compares some results of our method, that of [1] and the scans. The size discrepancy between deformed models and scans at six girth measurements are shown in Table 2. For comparison, Table 3 lists the size discrepancy between the customization results of [1] and the scanned models. As shown, we can find that our method has smaller mean size discrepancy and standard deviation. It is probably because 30 cross-sections better describe the 3D shape of human models. Moreover, our method reduces the amount of manual operation than [1], which reduces the human errors.

Conclusion
We have proposed in this paper a rapid method for reconstructing precise 3D body models from customer's photos. Compared with the work of Zhu et al [1], it reduces tedious feature extraction operation on customer's photos. Moreover, it can reconstruct customer's detailed geometric characteristics even when subjects are dressed in loose-fit clothing in the photos. Experimental results have proved that (1) the method can customize customers' body models based on the photos of the customers who wore loose-fit clothing in the photos; (2) the resulting models have realistic appearance and accurate size measurements; (3) the customization process is efficient with minimal interactive operations; and (4) the process meets the requirement for real-time applications. To conclude, the method contributes in accurate human body model customization from photos.