{'fraction_non_alphanumeric': 0.024130450678821264, 'fraction_numerical': 0.009189070612035976, 'mean_word_length': 4.759932612663204, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The effect of information and communication technologies on creativity in collaborative design

This study compares the degree of creativity of forty-two conceptual designs proposed as solutions to two innovative design problems developed in face-to-face and virtual collaborative environments. The solutions obtained were evaluated by three experts applying the Moss metric, which considers the level of usefulness and the level of unusualness. The average values and the trends of the data were obtained, and an analysis of the variance was also performed to determine whether the environment influences the degree of creativity. The results show that it cannot be proved that the level of creativity is influenced by working face-to-face or virtually, that is, whether information and communication technologies are used or not has no effect on the final result.


Introduction
In a world based on interconnection and the continuous exchange of information, product design has evolved to be performed in a decentralized way. The progress in computation and real-time access to any content from anywhere around the world makes smooth collaboration between multidisciplinary teams possible. This results in the elimination of physical barriers and improving synergies between designer and industry. Hence, the development of the new information and communication technologies (ICT) has led to a change in the way of working for society in general and for the design engineer in particular (Rahman et al. 2012). Now, the designers have the new technologies available for design assistance (Adenauer et al. 2013) and design engineer communication within project work (Regli 1997). For the last few decades, the design process could not be understood without the integration of the new technologies in all its phases-from idea conceptualization and formal research (Grabska 2012;Seichter 2003) to its transfer to industry (Groover and Zimmers 1983;Jezernik and Hren 2003;Yu et al. 1993). These points motivate that future engineers need to be trained in the use of ICT right from the academic stage of their preparation (Flowers 2001).

Background
One of the concepts that arouse most interest and concern in the field of virtual design teams is creativity, since it is considered one of the factors determining the success of the design outcomes. The wide range of methodologies oriented towards creativity is evidence of that fact (Gundy 1981;Jones 1980;Thompson and Lordan 1999), as is the huge number of scientific studies that analyse and compare the effect of creative methods, sketching and other stimuli during idea generation (Lopez-Mesa et al. 2011;Thompson and Lordan 1999; Van der Lugt 2002) among others. MacGregor and Torres-Coronas (2007) edited a collection of research articles on the topic of creativity stimulation in the field of virtual teams. This book proposes a set of solutions to avoid conflicts when working in virtual teams. The points where more research is needed are also identified, such as the search for evidence about what kinds of activities are adequate for working in virtual teams.
Studies conducted up to now evidence that the use of virtual design teams does not lessen creativity, but it in fact also seems to be encouraged by factors such as the kind of interaction or communication. Leenders et al. (2003) focused on new product development team's communication pattern as a key factor of team creativity. The study analysed the impact of three factors in the communication exercised during the development of electronic products: the team member proximity, the communication modality and the task structure. The paper concluded that the proximity of the team members in early stages of the work helps to achieve greater creativity, but if the task is highly complex, high levels of virtuality appear unproductive. Similarly, Kratzer et al. (2006) conducted a study with 44 research and design virtual teams, in order to analyse the factors that explain their creative performance. The study concluded that the performance achieved depended on the variability of similar factors: the physical proximity (the creative performance is higher for teams that are highly variable in physical proximity), the use of communication modalities (the creative performance is higher for teams that combines face-to-face and virtual communication) and the manner in which the task is coordinated (teams that carried out variable intra-team interdependencies achieved higher creativity). Thereby, observations in research groups working in electronic product development show that the virtuality of research and development teams is neither generally positive nor negative for their creative performance: the creative outcome is a function of how virtuality is managed in these teams. Moreover, several authors have developed new ICT-based tools that allow the interaction of a group of persons who are distributed during the creative phases of the design and concluded that creativity can be enhanced by means of naturally sharing knowledge (Shigeki and Zafer 2008).
Previous work on asynchronous virtual teams concludes that, despite the fact that there are elements that complicate creativity, the chance to connect a huge number of people has great potential for establishing a collaborative environment that generates a creativity stimulation on its own (Ocker 2005). A more recent study demonstrates that distributed collaborative work produces better results in idea generation when teams work in a synchronous mode than when they do so in an asynchronous mode (Rahman et al. 2012). Regarding the university educational context, the use of virtual teams has been analysed in a practical experiment. Thirty-four students were asked to use Web 2.0 technologies, such as Wikideas and Creativity Connector tool, in a context of project-based learning. The study concluded that a positive influence on creativity is appreciated when these tools are used (Ardaiz-Villanueva et al. 2011).
Moreover, several studies have been conducted to evaluate and compare virtual teams and face-to-face teams. Warkentin et al. (1997) analysed the effectiveness of a World Wide Web-based asynchronous computer conference system compared to traditional face-to-face meetings. Both teams had to perform some collaborative tasks and exchange information during the process. The study found out that virtual groups showed a lower level of cohesion and satisfaction with results than face-to-face groups, so the authors recommend combining virtual work with traditional meetings to strengthen the relationship of the working groups. Andres (2002) carried out a comparative study on virtual and face-to-face work in software programmers, which concluded that face-to-face work is more productive, but there are no differences in levels of satisfaction. De Pillis (De Pillis and Furumo 2007) analysed the behaviour that students showed in a intellective task, divided into small virtual and face-to-face groups. Similarly, the conclusion was that the virtual teams displayed lower average performance than the face-to-face teams because they needed more time to complete the task. Additionally, the virtual teams showed less cohesion and satisfaction, due to a higher level of deadbeat members. More recently, Tang et al. (2011) performed a similar analysis in the field of design, by comparing design teams in traditional and digital environments when generating creative solutions. In his study, the digital environment tries to emulate the traditional one, so the designers worked together in a synchronous mode using audiovisual communication. Functionality and innovation were two of the parameters used to assess the outcomes of the experience. Conclusions suggest that the design process in both pen and paper and digital drawing environments does not show any significant differences.
Although many of the studies conducted in recent years seek to quantify the effectiveness of both ways of working, few have explored the impact that the different working tools have in the creativity degree of designer teams when facing conceptual product design. Moreover, most of the cited works emphasize the use of certain technology, but there are also few of them concerned about trying to compare the effect of the use of technology against not using it when the main point is not the technology by itself. In order to complement previous studies, a new study is proposed in this article with a particular use of technology for conceptual design and with an appreciable number of cases which allow to have a sample size to achieve better analysis conclusions. The present work aims to compare the level of creativity of the solutions to design problems presented by design teams each made up of three members when they work face-to-face or virtually. In the face-toface mode, they work in the same place using paper and pencils, while in the virtual mode they use digital sketching tools and a shared synchronous document. In this last case, the designers work in separate rooms and they communicate through chat. As research results, the aim is to determine whether the conceptual solutions generated by the teams show higher, lower or similar levels of creativity when they have been working face-to-face or virtually. The starting hypothesis is that the creativity level will be similar in both cases because, despite the fact that virtual teams communicate only by chat, the expected team interaction efforts will be more focused on analysing and solving the design problem, so this will not cause a negative effect. Furthermore, the attention level achieved when reading the written information on the chat, which can be reread several times, is expected to be higher than when talking in a face-to-face group.
There are several metrics that can be used to assess the degree of creativity of the conceptual solutions for the innovative design problems, like Sarkar and Chakrabarti's SAPPhIRE (State change-Action-Part-Phenomenon-Input-oRgan-Effect) model (Sarkar and Chakrabarti 2008), Besemer and O'Quin's taxonomy of CPSS (Creative Product Semantic Scale) (O'Quin and Besemer 1989), the rating questionnaires of Kudrowitz and Wallace (2012), the metrics for measuring ideation effectiveness of Shah et al. (2003) or the comparative scale of Moss (Moss 1966). This last one was chosen for the present study because it is specifically designed to assess creativity by comparing a set of solutions provided for the same problem, and it uses a continuous scale for this purpose, which is useful for statistical treatment. This metric assesses creativity as the combination of usefulness and unusualness, which is a common itemization of creativity used in creativity metrics for product design, and which produces similar results compared with previous ones, as is demonstrated in the study conducted by Chulvi et al. (2012a, b, c). The study showed the evaluation of five objects by means of four different creativity assessment methods: Sarkar and Chakrabarti (2008), Moss (1966), EPI (Justel 2006) and CPSS (O'Quin and Besemer 1989). There, CPSS is a questionnaire that comprises several evaluators and provides the creativity value in a continuous scale; the methods of Moss and EPI comprise one single evaluator, and the results are in a continuous scale; the method of Sarkar and Chakrabarti comprises one single evaluator and provides the creativity values as ordinal data. The study concludes that all four evaluation methods provide similar results regarding creativity.
Moreover, the Moss scale has been used in several research works for more than four decades, so it is a highly accepted metric for measuring creativity (Michael 2001;Stricker 2008;Smyth 2010;Chulvi et al. 2013;Sarkar and Chakrabarti 2014).
2 Methodology of the study 2.1 Description of the experiment Groups of three individuals were formed for the experiment. The subjects were students in the last year of their Design Engineering degree, so they all offered a similar profile and academic training. Design Engineering degree combines the learning of technical and creative subjects (Howard et al. 2008). Sixty-six students participated in the experiment, 25 males and 41 females, and were divided up into a total of 22 groups. The experiment was part of a practical class of their degree.
Each group was asked to solve two conceptual design problems during the experiment. In one of them, they had to work in face-to-face groups, and in the other one, they were to work in virtual teams by using ICT. No guidance was given to them regarding the use of any design method, the aim being to avoid interferences, since one method could be more or less effective depending on the working mode. The problems chosen were related to the design of a new product that allows the automation of a function that has been performed manually up to now and that could be commercialized in the domestic sphere. The teams were provided with the next problem statements in the corresponding experiment phase: • Problem 1: design an automated domestic device to paint fingernails. • Problem 2: design an automated domestic device to take the fishbone out of small and medium-sized fish.

Materials and technologies
For face-to-face working, the design groups were taken to a cubicle with abundant natural light and they were provided with drawing material for elaborating sketches and designs. A camera was placed in a corner of the room in order to record the design process, so that the way the team worked could be analysed in future studies. In the case of those working in virtual teams, each member of the team was led to a separate cubicle with a computer connected to the Internet and equipped with flexible tablets with electronic pencils (Flexi Style Pixel, of NGS-www.ngs.eu) in order to make their working conditions and those of the face-to-face teams more similar. The collaborative drawing tool selected for this study is the one provided by Google Drive (Fig. 1). The simplicity of this tool and its intuitive interface make it suitable for a fast learning process. Moreover, the similarity with the Windows ''Paint'' application means that the students may already know several aspects and functions of the tool, since they are used to similar workspaces. This computer application provides them with a workspace comparable to the one used in face-toface work, including the use of similar drawing resources (kind and width of lines, colours, text), and it also provides them with a chat application that allows the team members to communicate efficiently with one another. A screencast application was installed in the computers in order to capture what is happening on the screen, so the working process of virtual teams could also be archived. Table 1 resumes the working conditions, while Fig. 2 shows an overview of the experimental settings.

Development of the experiment
Twelve participants were called simultaneously in at random to conduct the initial training and carry out the experience. The experiment was performed in three phases with only 5 min of rest between them, as represented in Fig. 2.
In this phase, a group of 12 students were put together in the same room. At this stage, they were taught about how the tools they were going to use during the experiment worked, and they were given a short time to become familiar with them. They were asked to draw anything using the drawing tool of Google Drive with the flexible tablets with electronic pencils and later to use these tools with the student sitting at their side. At this point, the participants did not know who their team mates were.
Twelve students were led to the cubicles, where they were divided up randomly into four teams of three students ( Fig. 3) and asked to solve a design problem. In order to avoid any interferences caused by the order of solving a problem or the use of a technology, half of the teams started with problem P1 and the other half with problem P2. Moreover, half of the teams that were beginning with each problem started to work in face-to-face groups and the other half started working in virtual groups (using ICT). So, in this phase, there is a team working face-to-face which is solving problem P1; another team working faceto-face which is solving problem P2; a team working in virtual groups which is solving problem P1; and a last team working in virtual groups which is solving problem P2. The assignment of tools and problems is given in Table 2.
The schedule given for problem solving was as follows: • Time for reading the problem statement and asking moderators about any doubts: 5 0 . • Problem solving: 30 0 .
• Time for documenting the final result: 10 0 .
When phase 2 ends, the students that had been working face-to-face were separated and led to the isolated cubicles to work virtually and those teams that had been working in virtual teams in the previous phase were put together to work face-to-face. They followed the same schedule as in phase 2, and they were asked to solve a different problem, as also shown in Table 2. So, for example, the team who worked face-to-face in order to solve problem P1 in the previous phase is now working virtually to solve problem P2.
The reason for alternating the beginning of each phase with or without ICT and with the problem (P1 and P2) is to eliminate the interferences caused by the order of the problem solving. The possible combinations of problems, phases and groups can be seen in Table 3, together with the codification model for each experiment output.
A code is assigned to each run of the experiment, which is in turn composed of three different codes separated by dashes. The first code corresponds to the number assigned to the group. So, G1 is the first group, and G2 is the second one, etc. The second code is defined by the letters ST and CT, which indicates whether the outcome comes from a face-to-face group (ST for the abbreviation of ''without ICT'' in Spanish) or from virtual teams (CT for the abbreviation of ''with ICT'' in Spanish), which are followed by a number that indicates the order in which the experience was performed by the team. So, for example, the code ST1 will mean that this design corresponds to a face-to-face work, and this experiment was performed first (in phase 2). The third code corresponds to the problem solved by the group.

Analysis tools
The creativity of the outcomes was evaluated using the metric of Moss (Moss 1966). Moss elaborated a method of identification and estimation of the creativity level of a product by combining two different parameters: usefulness and unusualness. According to Moss, the variable usefulness is determined by comparing the level of accomplishment of the product requirements with respect to a standard solution considered as satisfactory, which is called the ''teacher's solution''. This parameter is evaluated within a range of 0-3, where 0 means that the design fails to reach the minimum requirements, and a value of 3 corresponds to a solution that is better than the ''teacher's solution''. If the solution fulfils just the basic requirements but in a vague or imprecise way, its punctuation is 1, while the value of 2  means that the solution has achieved the same functional quality as the ''teacher's solution''. Despite the fact that Moss' metric considers that only one evaluator is needed, in the present research it was preferred to make the assessment by the agreement of three experts in the field of Design Engineering, in order to avoid subjectivity. So, three experts deliberated together about the value of unusualness, usefulness and creativity, and one value for each parameter was provided for statistical analysis. The same three experts determine together at the beginning of the evaluation phase which the ''teacher's solution'' will be. The unusualness variable is determined by the inverse probability that this same concept appears within a homogenous group of solutions. Thus, the product must be compared with the rest of the products elaborated within a group of solutions for the same problem. This is a comparative method, and therefore, the evaluator must be familiar with the possible solutions that could appear and with which frequency they could do so. He or she must also be capable of deducing which level of probability of appearance is considered normal for average products, and of turning the deviations of probability into unusualness ratios. A value between 0 and 3 is also assigned to these ratios. Here, a value of 0 means that this is a common solution, and a value of 3 corresponds to an exceptionally original idea. Lastly, the level of creativity of the final product is obtained by multiplying those two variables, Fig. 3 Students' distribution and phases of the experiment being the final score between a minimum value of 0 and a maximum value of 9, calculated as creativity = usefulness 9 unusualness (Table 4).

Results
Forty-two conceptual designs were obtained as a result of the experiment. Twenty-one of them involved a home device for painting fingernails (Figs. 4, 5), and the other 21 corresponded to a domestic device for extracting the fishbone from small and medium-sized fish (Figs. 6, 7). Handmade designs by the physical working groups were digitized in the same format as those produced using ICT tools to avoid conditioning the evaluators.
Using the metric of Moss (1966), three experts in the field of Design Engineering evaluated together, by consensus, parameters such as unusualness, usefulness and creativity for each of the 42 designs submitted. According to this assessment method, unusualness is evaluated by comparing the conceptual designs provided, but, in order to be able to perform the usefulness evaluation, first the ''teacher's solution'' must be defined. The assessment criteria that the evaluators together concluded by agreement for both problems are, in functional terms, as follows:   Home device for painting fingernails: • The device must paint the nails and only the nails. At the end of the process, fingers must be clean. A mechanical system of blocks to limit the nail is proposed. • The device must have several deposits for different colours, and it must have a selection system to choose which single colour is desired (such as buttons). • Nails will be painted using a brush.
• The device will run automatically, so electrical energy is needed.
Home device to extract the fishbone: • The device must remove the fishbone, the tail and the head of the fish. • This action will be performed using between one and three solid blades. • The fishbone will be removed by one or two hooks.
• The device will run automatically, so electrical energy is needed.
Accordingly, Table 5 shows some examples of evaluation of the problems.
The results of the evaluation of all the outcomes are given in Table 6. It can be seen that many of the solutions have a very low score on creativity, either because the solution does not meet the demands or does so in a very basic way (utility 0 or 1) or because there are at least 6-10 % of similar ideas (Unusualness 0 or 1), or due to both reasons simultaneously. Figure 8 shows the box plots of the degree of usefulness, unusualness and creativity in order to better visualize the similarities between the two ways of working. The graph shows that the degree of unusualness and creativity is distributed in the same way when working with or without technology, in all cases averages with a value of 0 being obtained, which means that half of the results have that value. In the degree of usefulness, however, differences are observed between using or not using technologies, as the average score when using technologies is 1, while its value is 2 when they are not used. Table 7 shows the number of times that the solutions of the same design team had been rated better when working without technologies and the number of times that their solutions had been rated better when they worked with technologies. This value is indicated as a total and separately for each of the two problems (P1 and P2). As can be observed, there is no clear trend showing an influence of the work environment on the degree of creativity of the results, as there are groups that produce better-rated results when working with technologies and groups in which this happens when working without technologies. Of the 21 groups, there are 7 that obtain the same ratings in both cases, whereas in the degree of unusualness 8 groups are better with technologies and 6 are better without technologies, something similar also happening with the other two parameters. This allows us to deduce that the possible influence of the working environment on the ratings of the designed solutions is not consistent. Table 8 shows the values of usefulness, unusualness and creativity of the proposals according to the problem, according to the working mode (physically or virtually) and combining both parameters. This table shows that problem 1 (P1) seems to give better results in terms of usefulness and creativity, while problem 2 (P2) seems to work better in unusualness. Similarly, the averages of the three parameters are higher when working in physical teams. With the aim of verifying whether these differences are significant, the data were analysed statistically.
The average values in Table 8 show that the degrees of usefulness, unusualness and creativity are slightly higher when working face-to-face, but, as seen in Table 7, there are groups that get better ratings when working face-to-face, while in other design groups the opposite occurs. On analysing the degree of utility and unusualness, it can be seen that the ratings are slightly higher for groups without ICTs. Regarding the degree of usefulness, the differences between using and not using technologies are slightly higher than in the unusualness parameter. From the box plot graph, it can be deduced that the data do not follow a normal distribution, which has also been checked with a Shapiro-Wilks test. So, the median of the two samples has to be compared. In this analysis, the data are paired, that is, the design groups of the to solve the problems is the same'') against the alternative hypothesis (''the median of the result of solving any of the problems described is different'' and ''the median of the result of using ICT or not to solve the problems is different'') with an a error = 0.05. In this test, the differences of the subjects in two samples are calculated and ranked to obtain a statistic called ''T'' which is compared against the lower and upper critical values (W) of Wilcoxon signed rank test. In addition, it has been considered that the statistic T follows a normal distribution and the value z is obtained Fig. 6 Conceptual designs for a home device for extracting the fishbone, with no ICTs Fig. 7 Conceptual designs for a home device for extracting the fishbone, with ICTs Table 5 Examples of evaluation of the problems G01-ST1-P1 Usefulness = 1. The device accomplishes the basic function of painting the nails, but it has no deposits to store colours. It also has no system to ensure that the fingers will not get painted Unusualness = 0. The concept is similar to several of the other solutions submitted Creativity = 1 9 0 = 0 G02-ST2-P1 Usefulness = 3. The device accomplishes all the functions proposed in the ''teacher's solution'', and it also presents several improvements like a scanner for nail detection, robotic arm, ergonomic hand rest or USB connection for selecting designs Unusualness = 3. The concept is not similar to the others, and all the evaluators coincide with the originality of the idea Creativity = 3 9 3 = 9 G05-CT2-P2 Usefulness = 0. The opinion of all the evaluators is that dropping the fish onto the blades will not ensure that it is going to be cut into the three parts shown in the drawing. This could only be accomplished if the fish is dropped well on target and does not bend. Moreover, the thickness of the blade would probably not separate the head from the body, but just split it with the rest of the fish Unusualness = 0. The concept of dropping the fish onto blades has been presented by several teams The device performs all the functions proposed in the ''teacher's solution'' (automated fish opening and fishbone extraction), and the evaluators agree that those functions would achieve the same level of quality Unusualness = 2. The concept is not exact but close to another proposed solution Creativity = 2 9 2 = 4  from the experimental value of T and tested against a standard normal distribution (critical value 1.96 for a a = 0.05). Thus: • If T 2 T crit low ; T crit up Â Ã , the null hypothesis is accepted (the problem/use of ICT has no effect).
• If T does not belong to the interval of the lower and upper critical values, the null hypothesis is rejected.
• If Z [ criticalZ, the null hypothesis is rejected (i.e., the problem/use of ICT has an effect). • If Z B criticalZ, the null hypothesis is accepted (i.e., the problem/use of ICT has no effect).
The first statistical analysis was performed to check whether the problem has influenced the results of unusualness, usefulness and creativity. The Wilcoxon analysis results are given in Table 9, where it is possible to see how the problem has not affected the results although the usefulness is in the limit.
On the other hand, we also analysed the influence that the use of ICTs, or not, had on the results of unusualness, usefulness and creativity. Table 10 shows the results of statistical analyses performed for this purpose. As we can see, the data suggest that the fact of working in person or virtually did not influence the levels of unusualness, usefulness or creativity of the results.
However, since the first analysis showed that the usefulness was very near the critical value when the differences against the design problem have been analysed, we decided to check whether in-person or distance work, using ICT tools or not, has any influence on the unusualness, usefulness and creativity values of the solutions, the results being separated according to the problem solved. Since the samples are separated for each problem, now the data of each sample are independent and the design teams solving a specific problem (problem 1 for instance) using ICT are different to the design teams that did not use ICT. So, the nonparametric U de Mann-Whitney test has been applied in this case. The test results displayed in Table 11 show       that in this case the use of ICT tools, or not, has hardly any influence. Also, it has been seen that the problem seems to affect the results of usefulness of the designs, but a new analysis must be done in order to see whether the problem order has any effect on the results. Again, a U de Mann-Whitney test has been done providing the results shown in Table 12. There it can be seen that the order in which the problem is solved has no significative effect in the unusualness, usefulness or creativity of the results.

Discussion
The assessment of creativity in the concepts generated using the Moss scale shows that when the group works face-to-face, the degree of creativity is somewhat higher (1.74) than when working and communicating through technologies (1.14) ( Table 8). These values are modest, since on the Moss scale creativity values range between 0 and 9. If we examine the degree of utility and of unusualness separately, the same thing happens: the results are slightly higher for groups that work without technology, but in both cases the values obtained are modest, their values ranging from 0 to 3, and they are slightly lower, particularly in the degree of unusualness. In the degree of usefulness, the differences between using ICTs, or not, are a little more noticeable. Furthermore, the influence of using technologies or not is not consistent, as 6 of the 12 groups obtained higher scores on the degree of creativity when they worked face-to-face, while the other 6 groups scored better on creativity when they worked using technologies.
Wilcoxon analysis shows that the difference in the degree of creativity is no significant, and therefore, it can be stated that the values of novelty, usefulness and creativity obtained are not affected by the use of ICTs in design teams. This means that these three values did not vary significantly depending on whether the groups worked in person or virtually. These findings suggest that both modes are equally effective, and thus, the implementation of a virtual work mode is generally positive and does not affect the results negatively.
The fact that the degree of creativity is little higher for face-to-face work may be because in the scenario in which the members of each group communicated through the computer, they could not communicate by voice or see each other, so this could somehow affect the ability of the group to generate ideas. In this regard, the fact of not being able to make eye contact eliminates the option of communicating ideas through gestures and nonverbal communication (Knapp and Hall 2009), which can be detrimental for group creativity. Indeed, there are studies that claim that the visual transmission of information acts as cognitive stimulation for creative processes (Rivera et al. 2010).
The results of this study are similar to those obtained in the study by Tang et al. (2011), in which third-year industrial design students working in pairs generated solutions to a creative problem, alternatively in a face-toface situation and in a virtual environment that simulated face-to-face conditions. In Tang's study, the solutions were scored numerically by experts on seven criteria, including functionality and innovation, which would be more akin to the usefulness and unusualness criteria used in this study. Although in the traditional environment the scores were a little bit higher than in the digital one, statistics show that the setting employed to generate creative solutions to a new problem had no influence on the results.
Furthermore, the results obtained in this experiment may have been influenced by the fact that all participants are university students, that is, young designers who are very familiar with the use of these technologies. While it is true that the same experiment in a more adult population could vary in its results (Atman et al. 2005), we can predict that the profile of the designer who will work with virtual design tools in the future will be more like those who participated in this study, so that the results can be considered as valid.
On the other hand, the scores of the solutions vary depending on the design problem, being more creative and useful in the case of the nail polish device; although the unusualness score is somewhat higher in the fishbone extractor device again, Wilcoxon analysis shows that there is no significant difference in the results depending on the design problem.
This result may be due to both the experience and motivation of the designers. In fact, the profile of the participants in the experiment indicates that they were young people and more than 60 % were women. This means that at least 60 % had previous experience in the nail painting process and the problems that resulted from it, while the fact that the participants' profile is defined by young students suggests that most of them probably still live with their parents and have never or almost never faced the problem of removing a fishbone. Informal conversations with students conducted at the end of the experiment confirm that motivation was different for two problems, being lower for the problem related to filleting the fish. In this design problem, it is worth noting that the scores between the groups that solved it using ICTs and those who solved it working in person are more similar (Table 8) than the scores obtained when they worked with the problem related to the nail polish device. However, the study does not supply sufficient data to determine whether the motivation from the problem may be a factor that exerts a significant influence on the creativity of the results.
Following this line, a deeper analysis of participant's opinion was carried out, whose conclusions can be seen in . The results of this study show that the students find that the interface is pleasant and they have enjoyed the experience of using it. They feel satisfied with the amount of information available and the organization of the information in the screen. Design students also think that it is easy to learn to use the IT tool and that it is fine to develop a creative design task in virtual teams. However, students think that the assigned design task is more difficult to do in virtual teams and that it can be solved in a better and faster way in face-to-face teams. A 30 % of the students have felt equally satisfied, but a 46 % have felt more satisfied in the face-toface. As a conclusion of the study, this kind of creative design tasks is more efficient, in the opinion of the students, in faceto-face teams than in virtual teams using chat.
Another possible explanation of the results of the experiment is that by not following any particular design method the designers focused on solving the problem using the means available. That could have been different if they had been required to apply a particular design method, because in that case perhaps the environment would have interfered more significantly. For example, using chat tools versus direct audio communication may slow down the generation of ideas in a brainstorming session. For that reason, we decided not to use any particular method, since some methods may be more focused on direct interaction and others may work better with the support of new technologies. As our intention in this study was to analyse the influence of using ICTs, or not, on design results in terms of novelty, usefulness and creativity, not using any methodology to address the design of the proposals makes it possible to eliminate the variable relating to the method. However, previous studies have shown that without a working methodology the scores obtained in these three parameters are generally lower (Chulvi et al. 2012a, b), so that it becomes more difficult to appreciate significant differences in the scores from the results of in-person teams and those that used ICTs.

Conclusions
The main results of the study suggest that the values of novelty, usefulness and creativity of the proposals created by the teams-without using any particular method-do not appear to be strongly influenced by working face-to-face or virtually. This means that the use of ICTs, or not, during the conceptualization process seems not to affect the qualities of the final result. According to Wilcoxon results, it cannot be demonstrated that the quality of the results is affected by the working mode related to use or not ICTs. As there is no difference in the results in both working modes, this conclusion can give greater validity to the studies on the improvements that future virtual design applications may have on these parameters. So, the advantages provided by the use of technology can be analysed with the certainty that the improvement in the results will be caused due to the technology by itself and not by the reactions of the design teams by using technologies.
Furthermore, as noted above, previous studies have shown that the design methodology affects the levels of novelty, usefulness and creativity of the results (Chulvi et al. 2012a. That is why it will also be useful to know whether certain types of design methods are better for face-to-face working or produce better results when used with virtual teams. Knowing this information would also be useful when developing new virtual work applications for design teams. Therefore, a subject of interest for this study in the short term focuses on comparing the effect of the design methodology to be applied in physical and virtual teams in order to see whether there is any interaction between methodology and working mode (in person or virtual). On the other hand, the work to be carried out in the medium or long term would be to create a virtual tool to assist designers in their work remotely, by using methodologies that have previously been shown to be more effective for this purpose.
Moreover, in the results obtained, it has also been noticed that the motivation to solve the problem might have an influence on the creativity of the results. However, by not considering this possibility in the initial hypothesis, we have not collected enough data in this experiment to analyse whether this factor is really influential. Therefore, it would be interesting to perform an experiment in the future based on this new hypothesis, as well as testing whether there could be a significant interaction between personal motivation and in-person or virtual work modes.