{'fraction_non_alphanumeric': 0.03122326775021386, 'fraction_numerical': 0.002619760479041916, 'mean_word_length': 4.9437559580552906, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Putting the Education into Educational Simulations: Pedagogical Structures, Guidance and Feedback

Learn-by-doing simulations can provide effective learning, allowing learners to practice skills in a realistic environment in a safe way, free of real-life consequences after mistakes. This method is particularly well-suited for corporate training situations in which the goal is for learners to perform their jobs optimally. The success—in terms of improved skill development and job performance—of an educational simulation requires an emphasis on the educational components, not just the simulation aspect. I suggest here that the goals of a learn-by-doing simulation must be not only to provide a practice environment, but to provide a specific learning environment (with guidance and feedback for the learner) and carefully-created situations in order for a learner to become proficient in the required skills.


Introduction
Learn-by-doing simulations can provide tremendously effective learning, allowing learners to practice skills in a realistic environment in a safe way, free of real-life consequences after mistakes. This method is particularly well-suited for corporate training situations in which the goal is for learners to perform their jobs as well as possible. The success-in terms of improved skill development and job performance-of an educational simulation requires an emphasis on the educational components rather than just the simulation aspect. All too often, corporate simulation-based training programs focus solely on the simulated environment and do not give enough thought to the educational aspects of the training, and thus do not have an optimal effect on job performance. I suggest here that the goals of a learn-by-doing simulation must be not only to provide a practice environment, but to provide a specific learning environment (with some type of guidance and feedback for the learner) and carefully-created situations in order for the learner to become proficient in the required skills.
In this paper I look at the design and development of learn-by-doing simulations from a pedagogical perspective and describe examples from a variety of learn-by-doing simulations. By following an educational approach to simulation design, designers can ensure that learners not only have a practice environment, but have the pedagogical support so that learners acquire skills efficiently and are able to transfer what they learn in the simulation to their job.

Background and Goals
Online simulation is such an appealing method because it allows people to learn from experience in a safe, simulated environment-"safe" in that learners can experiment in ways that they cannot in real life, and in that mistakes made in the simulation will not have real-life consequences, such as a crashed plane or an angry customer, as they would in real life. While in some ways there is no substitute for actual experience, a well-constructed simulation can provide three key factors that real-life experience cannot match: • A simulation can be built around scenarios and situations that have been carefully designed to bring out key learning points. This approach can accelerate the learning process by ensuring that learners face certain situations. • A simulation can provide appropriate guidance and feedback to the learner, while reallife experiences often can pass by without the learner taking away the key points. • Learners in a simulation can gain access to relevant expert stories (again, if the simulation and stories are well-designed) at various points in their simulated experience, in a way that they cannot in real life.

Coaching Guidance and Feedback
Different, successful learn-by-doing simulations (Guralnick, 1996;Guralnick, 2005) to teach a variety of skills-customer service, sales, and technical skills, to name a few-have employed various methods of guidance and feedback. Such simulations are based around carefully-constructed scenarios and situations, which are designed and scripted to force learners to make difficult, non-obvious decisions and perform the needed skills in a variety of contexts. Each successful simulation also included the following: • A "coaching guidance" component to help the learner through the tasks and to provide advice at various levels of detail along the way. This component can include specific suggestions about how the learner should proceed at a particular moment, as well as higher-level suggestions and answers to more general questions. It may include the use of relevant real-life stories when appropriate. More sophisticated coaching components can include asking the learner to engage in a Socratic dialog with the online system. • A feedback component that provides the learner with information on how well he or she performed the task. This can include both immediate and delayed feedback, and is designed to help learners foster the correct abstractions. This component generally includes both feedback from the simulated world itself (such as a video-based customer getting angry at the learner for providing poor service) and feedback from a tutoring component (to explain what went wrong and help learners abstract from their mistakes). In addition, real-life stories are often useful at moments in which the learner has made a mistake (Schank, 1997;Guralnick, 2005).
The use of coaching guidance and feedback, in each of the simulation-based learning programs, is carefully designed to suit the audience and the skills being taught.

Examples and Success Stories
Over the course of a large number of projects, we have used a variety of related guidance and feedback methods, which have evolved over the years. In this section, we examine several past, successful projects and the guidance and feedback style we used in each.

411: Directory Assistance Operator Training
Description: This course was a significant part of the training for directory assistance operators, teaching them how to find telephone listings in a fast and accurate way. Learners using this program took calls from simulated customers, via audio, and needed to answer their information requests as quickly and accurately by looking up listings using a specialized keyboard and by following "keying strategies" for each type of request.
Guidance: At any point in the simulation, the learner can ask the questions "Now What?, and the follow-up questions "How?" and "Why?". "Now What" provides some assistance regarding what the learner should do next, at a conceptual level. "How?" explains specifically how to perform the action, and "Why?" provides and explanation about what just happened or what the learner is asking about. There is also a small amount of general, noncontext-dependent guidance, mainly in the form of the "Tour of the Operator Screen," which shows the learner what each symbol on the screen means.
Feedback: When a learner makes a mistake, the tutoring component intervenes immediately and explains what the learner did wrong. If the error would not be trivial to recover from in real life-for example, if the learner moves the cursor to the wrong field-then the learner must "recover" from this error in the simulation. For other errors, the tutoring component jumps in but the learner then simply can continue.
Sample Screens:

Guest Service Training for Retail Employees
Description: This project taught customer service skills to "service desk" staff members at a large U.S. retailer. Service desk staff members need to learn how to handle customers' returns-the customers are returning merchandise that they had purchased-and understand how to appropriately treat a customer, particularly one who may be angry. Learners using this simulation played the role of a "service desk" staff member and had to handle customers who "approached" them in video. Guidance: As in 411, guidance is available via "Now What?," "How?," and "Why?". "Now What?" advice is intended to be at the conceptual level, while "How?" gets down to the level of what specific things the learner might say to the customer. It is more difficult than in 411 to keep the "How?" tutoring from giving away the answer though; in 411, "How?" could take the form of a rule or a visual, while in Guest Service, "How?" tends to be a bit more direct. Feedback: When the learner takes a wrong action, three things may happen: • The customer reacts badly, providing the learner with realistic feedback.
• The tutoring component provides feedback in the form of structured text, in order to help the learner make the appropriate generalizations. • An advice or story video clip is provided, with either an expert explaining a concept in more detail, or a peer telling a story of a similar experience.
Text feedback is always provided, whether or not there is a story or advice clip, or a customer reaction.

1-800-FOR-SERVICE
Description: This project also taught customer service skills, but aimed for a wider audience: it was created as a generic product that was licensed to companies in various industries. The focus here was even more on politeness and appropriate language than anything else, since the specific policies would vary from one organization to another. Learners using this simulation played the role of a customer service agent at a call center and had to handle customers who called in via audio.
Guidance: At any point in the simulation, the learner can ask a question from the list on the screen. These questions vary depending on context, and some refer to what the learner should do (e.g., "Should I try to convince the caller that our records are accurate?") and some with what has happened (e.g., "Why did the customer get so angry?"). Feedback: Similar to Guest Service Training, when he learner takes a wrong action, three things may happen: • The customer reacts badly, providing the learner with realistic feedback.
• The tutoring component provides feedback in the form of structured text, in order to help the learner make the appropriate generalizations. • An audio story is provided, with a peer telling a story of a similar experience.
Text feedback is always provided, whether or not there is a story or a customer reaction.

Radio Frequency Unit Technical Training
Description: Our client for this project was a U.S. retailer with over 1200 stores, who use the handheld devices shown above for a myriad of purposes. The applications on the device, and the tasks the device were used for, were many and varied. Under the simulation method used here, learners are given realistic tasks and a simulated version of the "RF unit" online. They must quickly and accurately perform the requested tasks using the simulated RF unit.
Guidance: The guidance for the learner in this simulation is very simple: there's a "Hint" button that is always available for the learner to click. Text guidance then appears in the box in the lower left-hand corner of the screen. Feedback: When the learner makes a mistake, text appears in the box in the lower left-hand corner. This text will explain any likely misconception that the learner may have, if any, and may also help the learner decide how to correctly perform the action.
Sample Screen:

Anti-Bias Training for High School Teachers
Description: This course, designed primarily for high school teachers, raises teachers' awareness of their own biases and provides them with the knowledge and skills to improve their own classrooms and schools. The course is delivered entirely online, making substantial use of video and with a variety of interactive exercises. The course includes twelve modules and takes approximately twenty hours to complete. The centerpiece module uses a method called character-based simulation, in which learners control the main character, a high-school teacher who is forced to deal with difficult situations of bias involving students, faculty, parents, and administrators. At each "turn" in this simulation, the learner must decide what the "teacher" character in the simulation should do. Along the way, learners can get advice from experts via context-sensitive questions, learn what the simulation's characters are thinking at any given time via "character insights" (a feature that is not available in a learn-by-doing simulation, or in real life!), and explore a print-based guide for handling situations of bias. At the end of the scenario, the learner receives a summary of his or her performance.
Guidance: Learners have three sections that can provide in-context guidance of some sort: "character insights," which let the learner get inside the head of a character to help them decide what to do; questions which are answered in text; and stories from peers who have encountered similar situations. In addition, learners can refer to the context-independent "zero-indifference guide" for general rules of behavior.
Feedback: When a learner takes an action, he or she sees the playout of that action-the story continues. Since these situations are complex, the notion of "failure" or "success" is not quite as defined as in the other examples discussed above; however, the learner can see if the situation is improving (sometimes time elapses), and if not, the learner is faced with a new set of choices and must again make a decision. In addition to the immediate feedback of seeing how a decision worked, the learner receives a summary at the end of a scenario, recapping the good and bad decisions.
Sample Screens:

Component Summary
As seen in the examples above, we have used several different guidance and feedback components. Those include: • Text-based guidance: This comes at a conceptual level (e.g., answering a "Now What?" question, specific level (such as in the answer to a "How?" question or in some of the RF hints), or as an explanation (as in "Why?" answers). Whenever possible, this is presented in a very structured form, clearly written and with examples, to aid learners in making appropriate generalizations. • Visual guidance: This was shown in the second "How?" example in 411, in which the guidance system demonstrates the location of a hard-to-find key. • Character insights: These are not strictly a form of guidance, but do provide additional information that helps a learner think through the situation and decide what to do next. • Text-based immediate feedback: When a learner makes a mistake, sometimes a feedback component intervenes immediately with structured text feedback, again intended to aid generalization. • Video stories: These have been used both as guidance (in the anti-bias model), and as part of immediate feedback (in the Guest Service model). Relevant stories are particularly effective when delivered in-context at a "failure" point (Schank, 1982). • Video advice clips: These clips work best when the person giving the advice is either well-known to the learners (e.g., someone famous, or someone known and respected within the learner's company), or is particularly charismatic on-camera. • Realistic simulation consequences: One of the most-effective types of feedback in a simulation, learners get to see the real consequences of their actions. But such feedback is most effective when paired with a feedback component (whether immediate or at a later point) that helps the learner understand his or her actions, reflect on them, and make appropriate generalizations. • Scenario summary: Summaries of the learners' actions and failures are especially useful if they are diagnostic-they provide some categorization of the learner's strengths and weaknesses-and prescriptive-they suggest what the learner can do to shore up his or her weaknesses, such as redoing a particular part of the training.
In the next section, we discuss some guidelines regarding when and how to use these different components.

Choosing Components and Methods
The above projects, and their guidance and feedback methods, have been considered to be "successful" in that they performed well on usability tests and received very positive responses from the end-users once the product was in use. The 411 product also involved a study which demonstrated improved performance for those learners who took the online course as opposed to the classroom-only course it replaced; it was not feasible for budget reasons for the other products to be studied in such detail. Learners did find the coaching guidance and feedback relevant and helpful on all projects.
One question, then, is when designing a learn-by-doing simulation, and assuming that we want to include coaching guidance and feedback in some form, how do we determine which specific methods to use-questions, stories, etc.? Some of the key factors are: • Teaching "soft skills," such as customer service or sales, vs. teaching technical skills: Technical skills may lend themselves better to a Now What/How/Why style, while providing specific questions seems to work better for soft skills. • The existence and relevance of good stories: Stories are generally better suited to more-complex tasks, and usually to soft skills more than to technical skills. This is primarily because the consequences of mistakes are much more interesting, rich, and memorable-though some technical skills could also have good stories (for example, an error that brought down a server, or caused an electrical fire). Even if the content area seems well-suited for stories, sometimes such stories simply cannot be found (for example, on some projects, we simply have not been able to elicit great stories in our interviews). • The complexity of the decisions the learner must make on the job: If the job, and therefore the simulation, involves simple decisions, immediate intervention by a tutoring component often works well (as seen in 411 and most of the customer service examples). More-complex decisions such as those in the anti-bias training example generally benefit from deferred feedback, in which the learner has time to explore and reflect deeper into a storyline before the coach or tutor steps in.
The series of design decisions in any learn-by-doing simulation project involves these factors and others.

Conclusion and Future Work
We plan to continue to extend our simulation work in two primary ways. First, we are currently rolling out a simulation authoring tool to corporations, which includes guidance and feedback structures and therefore encourages, and in some ways enforces, the use of pedagogical constructs in simulation design. Second, we continue to explore more sophisticated forms of guidance and feedback, such as more realistic Socratic dialogues. Our hope is that educationally-effective simulations can gain more widespread use within corporations.