{'fraction_non_alphanumeric': 0.028112733830757807, 'fraction_numerical': 0.008274691467166449, 'mean_word_length': 5.115916955017301, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Linking Living Lab Characteristics and Their Outcomes : Towards a Conceptual Framework

Despite almost a decade of living lab activity all over Europe, there still is a lack of empirical research into the practical implementation and the related outcomes of living labs. Therefore, this article proposes a framework to create a better understanding of the characteristics and outcomes of living labs. We investigate three living labs in Belgium and one in Finland to learn how the different building blocks of living lab environments contribute to the outputs of innovation projects launched within the lab. The findings imply that managers and researchers contemplating innovation in living labs need to consider the intended inputs and outcomes, and reframe their innovation activities accordingly. We formulate practical guidelines on how living labs should be managed on the levels of community interaction, stakeholder engagement, and methodological setup to succeed in implementing living lab projects and to create user-centred innovations. That way, living lab practitioners can work towards a more sustainable way of setting up living labs that can run innovation projects over a longer period of time. Remember the two benefits of failure. First, if you do fail, you learn what does not work. Second, the failure gives you the opportunity to try a new approach.


Introduction
Co-creation links distributed sources of knowledge (Tanev et al., 2011;timreview.ca/article/496) and conceptualizes innovation as the collaborative development between two or more stakeholders. Co-creation is also described as the act of creating value to the mutual benefit of two or more actors, beyond creating actual product or service innovation in a collaborative way (Allen et al., 2009;timreview.ca/article/301). In particular, living labs are regarded as an emerging open innovation approach that involves multiple stakeholders, including users, to co-create value that eventually leads to innovation. Living labs are a new way of structuring research and help companies rapidly commercialize and upscale an innovation through validation and testing in real-life contexts (Leminen and Westerlund, 2012;tinyurl.com/orlnfh5). Living labs offer a more reliable market evaluation than test markets, and they give users power in innovation processes (Salter and White, 2013; tinyurl.com/lknek7b).
To date, there exists no consistent and commonly accepted definition of the living lab. Instead of a general definition, several authors have suggested various key characteristics and principles or have tried to harmonize the different methods and tools (cf. Mulder et al., 2008;tinyurl.com/8su2mal). However, none of these efforts link the characteristics or principles of living labs to tangible outcomes. Therefore, the objective of this study is to: i) investigate the different building blocks of a living lab environment, and ii) examine how they contribute to the outputs of innovation projects launched within the lab, based on an analysis of actual living lab projects and experiences. To reach this objective, we first discuss the definition and basic characteristics of living labs. Second, we establish our framework, based on living lab literature, to detail the triangulation between environment, approach, and outcome in living labs. Then, we describe the research design constructed for a validation of our proposed framework based on data of four living labs, and we report the results and lessons learned from our empirical Despite almost a decade of living lab activity all over Europe, there still is a lack of empirical research into the practical implementation and the related outcomes of living labs. Therefore, this article proposes a framework to create a better understanding of the characteristics and outcomes of living labs. We investigate three living labs in Belgium and one in Finland to learn how the different building blocks of living lab environments contribute to the outputs of innovation projects launched within the lab. The findings imply that managers and researchers contemplating innovation in living labs need to consider the intended inputs and outcomes, and reframe their innovation activities accordingly. We formulate practical guidelines on how living labs should be managed on the levels of community interaction, stakeholder engagement, and methodological setup to succeed in implementing living lab projects and to create user-centred innovations. That way, living lab practitioners can work towards a more sustainable way of setting up living labs that can run innovation projects over a longer period of time.

Multiple Definitions of the Living Lab Concept
The living lab concept appeared in academic discussion in the 1990s, but really took off only in 2006 when the European Commission kicked off projects to advance, coordinate, and promote a common European innovation system based on living labs (Dutilleul et al., 2010;tinyurl.com/lgz3svv). Several international organizations, representing industrial living lab initiatives in information and communication technologies (ICT), were founded in order to stimulate living lab research. The European Network of Living Labs (ENoLL; openliving labs.eu) is the most influential initiative covering living labs from all over the world. Living labs were put forward as an institution to overcome the "European Paradox" (tinyurl.com/kjm8735) or the gap between research leadership and commercial success of innovation. This increasing attention and the accompanying monetary support for living labs has unfortunately led to a wide variety of projects carried out under the "living labs" umbrella, and a proliferation of research papers that use the term "living labs" in a sense that is only loosely relate to the subject.
Despite the booming interest in living labs, they remain an under-researched area due to the lack of common understanding of the concept and its underlying mechanisms (Bergvall-Kåreborn and Ståhlbröst, 2009;tinyurl.com/kfazp4o). They have been discussed from different perspectives, and a wide diversity of thematic approaches, constellations, methodologies, and tools for living labs exist (Almirall et al., 2012; timreview.ca/article/603). The living lab has been conceptualized as an environment (Ballon et al., 2005; tinyurl.com/k2zflmz), a methodology or innovation approach (Bergvall- Kåreborn et al., 2009;tinyurl.com/kn9rzjx), an organization or an innovation intermediary (Schuurman et al., 2012; tinyurl.com/lbsjwod), a network (Leminen and Westerlund, 2012; tinyurl.com/nk2bv2r), and a system (ENoLL, 2007;tinyurl.com/nv4hhdb). This lack of common understanding makes it difficult to advance research focused on living labs.
We follow the definition by Westerlund and Leminen (2011; timreview.ca/article/489) because it stresses the multistakeholder aspect, the real-life context, and the various stages of the development process. They view living labs as "physical regions or virtual realities where stakeholders form public-private-people partnerships (4Ps) of firms, public agencies, universities, institutes and users, all collaborating for creation, prototyping, validating and testing of new technologies, services, products and systems in real-life contexts". In the living lab environment, different stakeholders can cooperate and share their resources, knowledge, and expertise, which is crucial to startups and small firms that have challenges acquiring venture capital (Eriksson et al., 2005; tinyurl.com/8fv3jkp). Living labs can have a demographic or geographical focus, they are either research or industry driven, and they are led by utilizers, enablers, providers, or users (Leminen et al., 2012; timreview.ca/article/602). Although the implementations vary, notions about the role of users and their engagement in the innovation process remain central. Living labs research the whole innovation process from concept to effective usage (Salter and White, 2013; tinyurl.com/lknek7b).

The Living Lab Triangle
Based on a literature review and the authors' earlier research (Veeckman et al., 2012; tinyurl.com/mm2at5q), a comprehensive framework was established to analyse the link between the building blocks of living labs and their effect on the living lab outcomes. The Living Lab Triangle framework ( Figure 1) has three pillars and consists of 11 key characteristics. The foundation of our framework is based on the characterizing purposes of Følstad (2008; tinyurl.com/l7s99ph). Making a distinctive profile of each living lab was initially difficult, because Følstad's characteristics were both insufficient to identify the main building blocks of living labs that act as differentiators and incapable of assessing the impact of the living lab's R&D activities. Every living lab obtained the same score for the setup of their innovative characteristics, whereas in practice they had different outcomes. Therefore, some of the Følstad's characteristics were adjusted and combined with the key principles of good practice by Eriksson et al. (2005; tinyurl.com/8fv3jkp).
The identified characteristics are divided on a generic level (i.e., the living lab environment) and on a project level (i.e., the living lab approach). The set of characteristics on the generic level refers to material, immaterial, and contextual elements of a living lab environment, and the set on the project level defines the methodological aspects. www.timreview.ca

Carina Veeckman, Dimitri Schuurman, Seppo Leminen, and Mika Westerlund
Pillar 1: Building blocks of the living lab environment 1. Technical infrastructure: When assessing or co-creating innovations, a technical component should be available for the test users within the living lab. In ideal circumstances, the testing of the innovation also includes monitoring of the technical performance during usage and non-usage of the innovation.

Ecosystem approach:
Various stakeholders, from industrial partners to users and research organizations, interact to develop and evaluate a certain process, product, or service within the living lab ecosystem. Similarly, ad-hoc business ecosystems are constructed within the living lab projects (Peltoniemi and Vuori, 2004; tinyurl.com/cwtd63x). When creating an ecosystem, it is important to create value to attract and retain members, and to share the value within the ecosystem (Iansiti and Levien, 2004; tinyurl.com/ bqaol6f). In practice, this means that there should be an added value for all partners involved, in order to create long-term engagement and identification with the living lab or at least on a project level (cf. Apollon project, 2012; www.apollon-pilot.eu).

Level of openness:
One of the key principles in living labs is that the innovation process should be as open as possible, because a multitude of perspectives might speed up the development and bring more innovative ideas (Bergvall-Kåreborn et al., 2009;tinyurl.com/9nqmrdy). This study incorporates two levels of openness, namely how intellectual property rights are being handled (i.e., the extent of knowledge sharing) and the degree to which new partners are embraced.

Community:
Users participating in the living lab are part of a community, which can range from a "community of interest" to a "community of practice", whether or not it is geographically bound.   Carina Veeckman,Dimitri Schuurman,Seppo Leminen,and Mika Westerlund 5. Lifespan: This characteristic refers to the duration of the living lab, and not of a single innovation project launched within the lab. For example, a short-term living lab initiative might last less than six months, whereas a long-term initiative might have a two-year duration, and a very long-term initiative might have an indeterminate end date.

Scale:
This characteristic refers to the number of users involved in living lab research activities such as the living lab panel. A small-scale living lab panel may involve fewer than 100 users, whereas a largescale living lab may have more than 500 users. These numbers are defined on the generic level of the living lab, and not on a project level because the type of innovation or user study will define how many users can participate within the project or research activity.

Real-world context:
Users should be studied within a real-life context, which implies a familiar context that reflects the users' natural environment as much as possible. For example, users are studied within their home environment rather than in a laboratory setting.
Pillar 2: Building blocks of the living lab approach 1. Evaluation, context research, and co-creation: Within a living lab setting, test users are involved through different phases of the innovation cycle in which they can test, evaluate, and co-create the innovation. This means that test users must be able to give a positive or negative assessment of the innovation through, for example, surveys or in-depth interviews. Test users should be given the opportunity to shape the innovation in interaction with researchers and developers. Co-creation should be iterative and make use of, for example, participatory methods. Furthermore, the usage context should be taken into account as a critical element that influences usage behaviour through, for example, ethnographic tools (cf. Veeckman and Lievens, 2013; tinyurl.com/ny457sg).

User role:
Leminen, Westerlund, and Nystöm (2014; tinyurl.com/ma9ja59) identified four distinct user roles in living labs on the basis of the degree of user activity and the firm's view of co-creation: i) informant, ii) tester, iii) contributor, and iv) co-creator. We propose that user roles depend on the view that companies pursue for integrating users in living labs and the degree of user activity within these living lab activities.

Pillar 3: The innovation outcome
To evaluate the success of a living lab, the innovation outcome must be considered. Knowledge of the tangible outcomes enables us to assess impact and determine which approaches worked best. Thus, the living lab setup can be improved, which leads to better implementation of future living lab projects. However, the literature is silent about which components affect the outcome in living labs, with the exception of Leminen, Westerlund, and Kortelainen (2012; tinyurl.com/kklefus) who found that it depends on: i) strategic intention; ii) passion; iii) knowledge and skills; iv) other resources; and v) partners in the living lab network (Table 1).

Carina Veeckman, Dimitri Schuurman, Seppo Leminen, and Mika Westerlund Operationalization
We operationalized the previously discussed building blocks on a four-point scale. A low score means that a specific characteristic is not present and a high score means that it is clearly identifiable and contributes to the operation of the living lab. We modified several of the Følstad's characteristics based on the findings by Veeckman and colleagues (2012; tinyurl.com/l7mc5hx). For instance, "discovery" and "familiar context" were covered by other concepts and left out on the level of the living lab environment. The former is a principle maintained during the whole living lab process, and it forms an integral part of the way the methodologies are set up. The latter is a principle of testing in a real-world context and implies a familiar context that reflects the users' natural environment as much as possible.
We also added four new building blocks: i) the ecosystem approach, ii) level of openness, iii) community aspect, and iv) user role. We also added a new pillar: innovation outcome. These adjustments will lead to a better characterization of living labs and are essential to assess the impact of diverse setups of living lab operations. The new pillar will make a more direct link between the building blocks of a living lab setup and the outcomes of innovation projects launched within the lab. Through these additions, the interplay between the living lab environment and its projects will be considered more attentively, as we assume that the environment intentionally and unintentionally shapes the projects. Table 2 details the operationalization of our framework.

Research Design
We conducted a multiple case-study analysis of four distinct living Labs in two European countries: FLEL-LAP, LeYLab, and Mediatuin located in Belgium, and the Laurea Living Labs Network in Finland (Box 1). This research approach was deemed appropriate because we are dealing with new and poorly understood phenomena (cf. Eisenhardt, 1989; tinyurl.com/n666sey).

Box 1. Information about investigated living labs
www.timreview.ca

Carina Veeckman, Dimitri Schuurman, Seppo Leminen, and Mika Westerlund
By analyzing the main characteristics in different living labs, conclusions can be made on how these building blocks should be set up and how they affect the outcome of an innovation project. The results of this study will contribute to the current understanding and knowledge building of the living lab concept, but will also give practical guidelines on how to overcome possible challenges in the living lab setup, or how one can strive for a successful implementation of this innovation instrument.
The data was collected between 2007 and 2013, including expert interviews with the living lab staff and secondary data including various documents of the design and outcomes of the projects. Our analysis relies on the coding technique of an operationalized framework of living lab characteristics, which is grounded on a literature review and earlier testing (Veeckman et al., 2012; tinyurl.com/mm2at5q). Two of the authors in this study independently assessed the characteristics of the living labs, assigning a low score to reflect that a characteristic was not present and assigning a high score to show that the characteristic was clearly identifiable and contributed to the operation of the living lab. Disagreements in coding results were re-examined and resolved together. Table 3 shows a characterizing profile for each living lab and illustrates that the new added building blocks act as a differentiator. However, it should be noted that these results were coded on the generic level of the living lab, and not on a project level. If we applied the framework to each living lab project separately, the results could be very different because methodologies and objectives vary within those cases.

Carina Veeckman, Dimitri Schuurman, Seppo Leminen, and Mika Westerlund
The Laurea Living Labs Network obtains the highest result for most characteristics. Although it has a different approach in recruiting users and setting up the infrastructure, its main merits are the ecosystem approach and the level of openness. The strengths of LeYLab and Laurea are the fixed infrastructure and the passion of some stakeholders to move on, which make FLELLAP the weakest in class. This result is mainly due to the malfunctioning ecosystem and project-based approach, which also caused the ending of the initiative. The other two Flemish living labs, Mediatuin and LeYLab, were able to build a more sustainable model. They are still running and have the opportunity to live on in other open innovation initiatives.
There is a remarkable difference between the Flemish and the Finnish labs in terms of how the living lab approach is applied. In the Flemish labs, there is little initiative towards the evaluation or co-creation of the scope of the living lab. For example, FLELLAP only conducted a quarterly survey amongst their general panel, which related to the three thematic domains. Conversely, Laurea Living Labs has a more thematically focused research track on the generic level with co-creation, development, validation, and testing of innovations. If the Flemish living labs had a more clearly defined research track on the generic level of the living lab environment, and a mixed set of living lab tools, the possibilities of finding new opportunities or innovative ideas would be higher and projects within the lab would be better supported. Next, we present some lessons learned and discuss how a more successful implementation of living lab projects can be achieved.
Lesson 1: Create value and share it with everyone FLELLAP and LeYLab obtained lower scores for their ecosystem approach as compared to Mediatuin and Laurea Living Labs. This result may be due to the missing links in their value chains and the unequal contribution of stakeholders. For example, FLELLAP focused on smart grids even though there was no thematic expert or electricity supplier involved. This gap brought about missed opportunities for building more innovative services in that domain. The malfunctioning ecosystem of FLELLAP resulted in the closure of the initiative in March 2013. Therefore, we recommend that, when setting up a living lab, there should be: i) a clear thematic focus for the strategy and ii) a good variety of stakeholders. A clear thematic focus will lead to complementary, shared motives for collaboration within the living lab, which in turn will benefit the community aspect (e.g., through increased engagement towards a given topic) and creation of new partnerships (e.g., less differentiated domains).
The results from our analysis also show that the type of infrastructure (i.e., an ad-hoc or fixed infrastructure) will determine the thematic focus. When opting for a fixed infrastructure (e.g., the fibre infrastructure in LeYLab), all projects running in the lab can make use of it. On one hand, it will lead to a clearer focus in the type of projects because the stakeholders should test an innovation that fits with the infrastructure. On the other hand, it will also restrain their testing possibilities because they are not able to extend beyond it. When opting for an ad-hoc infrastructure, as did FLEL-LAP, stakeholders feel less restricted in testing out innovations that are linked to the thematic focus of the living lab. The disadvantage is that every time a new project starts, users are equipped with new infrastructure or devices. It requires the panel manager to put in extra effort to guide each project and subpanel.
All these aspects underpin the strategic intention of the living lab and should be thoroughly discussed at the start of the initiative. It must be ensured that everyone will collaborate when diverse stakeholders are brought together, even if they have different interests, resources, and ways of operating. As illustrated by FLEL-LAP, which failed in building a mutual vision or a common purpose, it is of vital importance that value can be created and shared amongst every stakeholder when joining the living lab initiative. After all, living labs break down traditional and hierarchical approaches to innovation and frame them in a more experimental and collaborative manner (Hellström Reimer et al., 2012;tinyurl.com/ob925t4).

Lesson 2: When there is no value, there is no openness
A low score on the ecosystem approach may result in an even lower score for the level of openness. When there is no added value for the involved stakeholders in the ecosystem, industrial partners are less eager to share the results. Consider FLELLAP and Mediatuin, where stakeholders were reluctant to present their results to other partners or to give updates on scheduled technical improvements. Stakeholders feared competition and wanted to keep their agendas confidential.
Mainly due to the lack of common purpose within these Flemish living labs, there was little to no interaction and information sharing among these stakeholders. In better circumstances, the involved stakeholders would have been able to draw on each other's knowledge, capacities, and resources.
www.timreview.ca Carina Veeckman, Dimitri Schuurman, Seppo Leminen, and Mika Westerlund In FLELLAP, the pooled resources were exclusively used by the key stakeholders, which caused information blockages and inefficiencies in the innovation process. It was tremendously difficult to build up a good ecosystem and find new interested stakeholders. Furthermore, the lack of openness restrained small-and medium-sized enterprises and startups from accessing the critical assets that were afforded by the involved large businesses. For example, two smart media projects (Fietsnet and MUFO-LIVE) in FLELLAP were not able to make use of the wireless Internet infrastructure of one of the large companies due to the lack of shared value creation. Therefore, we stress the importance of creating shared motives for collaboration, so that the living lab resources can be made available to each stakeholder.

Lesson 3: Community engagement is crucial
The differences in community engagement between the studied living labs are a remarkable finding. High performers on this scale included LeYLab, which is a geographical community and a community of practice through its installed fibre connection, and Laurea Living Labs, which consists of students and staff members. This engagement resulted in an active participation in panels and projects. Low performers are Mediatuin and FLEL-LAP with arranged panels based on a mutual interest for media and ICT. FLELLAP evidenced that a frequent communication (e.g., mailing bi-monthly newsletters, sharing results and pictures of the projects) helps to create a community from scratch. Moreover, a survey on the motivations for collaboration showed that intrinsic motivations were highest among the panel members, meaning that panel members had a personal interest in making a valuable contribution to the innovation.
Based on these results, the management of the panel and its communication could be set up more efficiently. The efforts of this approach eventually paid off in the studied labs, as evidenced by higher participation rates of FLELLAP over time relative to Mediatuin. For the panel managers of the living lab, this participation level meant a strong decrease in time and effort required in the recruitment of new people. Therefore, we recommend that, when setting up a living lab, one must have an access to a specific set of users and establish a strong communication link with them. Otherwise, there will be a need to recruit new people each time a new project starts, which means more effort and a loss of accumulated knowledge. In addition, community support will keep users motivated to participate in a living lab.

Conclusion
This article studied how the main characteristics, or building blocks, of living lab environments can impact the daily living lab operations and the outcomes of the projects. The Living Lab Triangle framework makes it possible to study the interplay between the setup of the living lab environment and the outputs of the projects within the lab. It triangulates the characteristics of the living lab environment, the living lab approach, and the innovation outcome. The study demonstrates that the living lab environment shapes the undertaken projects and that innovation practitioners should consider the intended inputs and outcomes and reframe their innovation activities accordingly.
Based on the findings from the studied living labs, we make five recommendations. For more successful implementation of projects, a living lab should establish: 1. A clear strategic intention 2. A minimum of shared value creation and sharing among all stakeholders 3. A minimum level of openness 4. A minimum set of users and establish a strong communication 5. A mixed set of living lab tools to discover new opportunities Our framework is more comprehensive than previous conceptualizations on living labs. In addition, this study updates the current knowledge about living labs with some new real-life empirical data. However, future research should further explore the main building blocks and operationalization of the framework. Given that this study involved a small number of living labs cases, the framework should also be further validated on a larger scale. This validation should take place through a large number of living labs focusing on different domains. It would also be interesting to code the framework on the level of each living lab project, instead of the generic level, and assess to what extent the living lab environment contributes to the implementation of the projects.