{'fraction_non_alphanumeric': 0.027729112547574458, 'fraction_numerical': 0.01211260798646771, 'mean_word_length': 5.1720730797912005, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Formative feedback within plagiarism education: Is there a role for text-matching software?

The need for effective education to enhance students’ understanding and development of academic integrity has been well established, particularly for international students new to tertiary study in English-speaking countries. Most research recommends the use of methods such as instruction and warnings, as part of a plagiarism education programme. Few studies have looked at the role of formative feedback through tutorial intervention in the process of academic writing, in which use of the text-matching tool Turnitin is made in ways which support learning, rather than guide assessment. This study addresses that gap using data gathered over three years from cohorts of international students in the United Kingdom (UK) with regard to four identified areas of development: avoidance of plagiarism, decrease in over-reliance on some sources, correct use of citation and appropriate paraphrasing. Individual tutorials were held to give formative feedback on students’ own writing, with particular regard to their use of sources. A supportive environment was created in which questions about references could be discussed, by using the Turnitin originality reports directly. The tutorial feedback appeared to have a positive effect on students’ understanding of academic integrity reflected in improved drafts. This implies that tutorial feedback using Turnitin could be a key factor in plagiarism education. Recommendations for future use are given at the end of the study.


Introduction
This paper explores ways of providing plagiarism education through individual feedback on students' own work during a formative stage of students' writing practice. It also examines the place of text-matching software in supporting and evaluating the impact of tutor interventions designed to teach students academic citation skills. The tool used in this study, marketed under the trade name of Turnitin, is referred to with a variety of claims ("the standard in online plagiarism prevention", iParadigms, 2007; "anti-plagiarism software", Sutherland-Smith, 2008), but in general, has the purpose of highlighting students' unoriginal work. In brief, the tool functions in the following way: once a text is uploaded to the tool's software, it provides what the makers call an "originality report". The report suggests an overall percentage of the student's text which matches sources on its database, and indicates the level of match with a coloured icon (0=blue, 1-24=green, 25-49=yellow, 50-74=orange 75-100=red). The software can also indicate the percentage of matching text for each named source and presents these as a ranked list in order of quantity. With each student's text, a reviewer can click on a matched section and then see the suggested 'original' text side-by-side for an instant comparison between the two texts.
The authors present a three-year case study in the UK where they investigate the effect of feedback on students' attention to important aspects of academic writing, in particular to avoidance of plagiarism. The approach also examined three features of writing that may be connected with plagiarism: over-reliance on some sources, incorrect or incomplete citation, and insufficient paraphrasing. The underlying approach supports the argument for shifting the focus away from a 'catch-and-punish' approach to student plagiarism (Carroll, 2005), and towards one designed to support learning, through formative feedback to students on the use of academic conventions.
Debates continue in higher education as to an appropriate role for text-matching software. Levin (2006) probably speaks for many when he describes those who use Turnitin as the "plagiarism police" since in his view, it polarises student work into two categories: original or potentially plagiarised. In contrast, advocates of using Turnitin, such as the authors, believe it to be a useful support to other traditional teaching methods when used formatively. It is likely that the dichotomy of views between those in favour and against using Turnitin with students reflects much wider beliefs about student learning, and the role of teachers. This study offers evidence of positive impacts from Turnitin on students' learning and a key role for teachers.

Literature review
The literature on teaching appropriate writing techniques and deterring student plagiarism is extensive and continues to grow. A cursory search of the Internet will reveal good practice guides for deterring plagiarism (for example, , welldeveloped websites (for example, the Acknowledgements site based at Monash University), and sites where writing and deterrence are closely combined (for example, the statement by the US Council of Writing Programme Administrators). All such sources offer guidance to teachers on how to help students avoid plagiarism. Key elements include: students must draw from a range of sources, adopt acceptable paraphrasing practices, and correctly attribute others' texts when including them in student work, using an agreed referencing system. Whilst these aspects of writing can seem to be straightforward matters to academics experienced in academic writing, the growing literature on academic writing demonstrates that students find them problematic (such as Chanock's 2008 study of why students may plagiarise). Pecorari (2003) confirms "most students will use sources inappropriately before they learn how to use them appropriately" (p. 342). Advice on addressing their difficulties emphasises the importance of practice (Emerson, Rees & MacKay, 2005), detailed feedback (Barrett & Malcolm, 2006) and using examples (Biggs, 1999). The importance of addressing students' difficulties with writing can not be over-estimated, as where students fail to comply with the requirements, their work is judged to be plagiarised.
In many countries, including the UK, where the case study in this paper was undertaken, policies on plagiarism have been revised to include statements regarding institutional responsibilities to ensure students are able to comply with requirements to 'do your own work' before awarding punishments for plagiarism should they not do so (Macdonald & Carroll, 2006). Therefore, the recognition that institutions must ensure students have the required skills to avoid plagiarism has lessened the tendency to assume students arrive with them already in place and/or acquire the necessary academic skills informally. Explicit teaching aimed specifically at plagiarism avoidance is increasingly important since students enter higher education from ever more diverse backgrounds, particularly as international students, in English-language contexts such as the UK. Many authors have discussed the difficulties students experience when moving to a new academic culture in general terms (for example, Carroll & Ryan, 2005) and an increasing number of studies focus on international students' needs when learning new writing conventions and trying to avoid plagiarism (for example, Handa & Power, 2005). International students are frequently described as lacking confidence in writing and many researchers link this to greater reliance on textual borrowing (Ryan, 2000;Pecorari, 2003). As one lecturer in Sutherland-Smith's 2008 study exclaims: "students who are coming to us from overseas just don't have the grounding before they start…Thirteen weeks is not enough time to really understand plagiarism and be able to overcome it. Some students have a life-time of just copying textbooks to get over!" (p. 12). All students, but arguably, some international students in particular, may find it especially difficult to engage with the learning outcomes and may need particular help to learn to do so in acceptable ways, rather than unacceptable ones such as plagiarism.
In understanding how to avoid plagiarism, the concept of authorship is also discussed in the literature. A recent study involving five universities in the UK (Pittam, Elander, Lusher, Fox & Payne, 2009) analysed student views which confirmed students' problems with understanding the concept of 'authorial identity'. The students in this study tended to feel they were not authors since they saw authorship as something done by professional writers showing original thinking, whereas assignments were viewed as tasks which required them to keep to established literature. Thus they saw their own writing as more akin to 'editing' than authorship. Pittam et al. (2009) conclude that problems with establishing authorial identity may be key to understanding why some students plagiarise. Similarly, Sutherland-Smith (2008) found the concept of authorship to be blurred for many students going through a written process involving starting and stopping and absorbing many kinds of other influences before a finished 'product' eventually emerged.
Studies such as those cited show that within plagiarism education, students are learning new skills and adapting previous beliefs. One way to encourage students' learning is by offering clear, helpful formative feedback, in line with the reminder to UK academics of the Quality Assurance Agencies' principles (General Principle 12) that "Tutors in higher education today are encouraged to ensure that appropriate feedback is provided to students on assessed work in a way that promotes learning and facilitates improvement" (cited in Orsmond, Merry & Reiling, 2002). Tutors may recognise the importance of 'appropriate feedback', yet in relation to the serious issue of plagiarism, Hyland (2001) found that written feedback sometimes is seen by students as vague and ambiguous, and that students may be confused as to what they have to do to improve their drafts. Meanwhile, spoken feedback given on a oneto-one basis may be one of the most useful means to discuss issues such as plagiarism, since by its very communicative nature, involving both student and tutor, it can have a clear formative purpose (Gardner, 2004). This was confirmed by McGarrell and Verbeem (2007) who also stressed the importance of formative feedback in the process of acquiring academic literacy to enable more competent drafting. Orsmond et al. (2002) also advocated feedback clinics as a way to discuss the wider aspects of tutors' comments, thus bringing greater benefit to students.
As well as dealing with plagiarism by increasing students' skills and knowledge, there has also been widespread discussion in the relevant literature of the use of tools to detect text copying which could be judged to be plagiarism. Many such tools have been created and comparisons made as to their utility (see, for example, Bull, Coughlin, Collins & Sharp, 2001) as well as specific investigations of particular tools such as the one used in this study, Turnitin, which analyse the systems' benefits and flaws (for example Pearson, 2002;Goddard & Rudzki, 2005). Makers' claims such as the statement which describes Turnitin's 'originality report' as "allow[ing] for easy and unambiguous interpretation of document source analysis results" (iThenticate, 2007) are challenged by studies such as that by Peacock, Sharp and Anderson (2006), which found that many academics have not found the interpretation so easy and stress the need for clear guidelines and training to ensure tutors use the reports correctly. This level of ongoing interest confirms the need to note the limitations of any text-matching tool, rather than assuming its abilities to identify and deal with plagiarism.
The impact of Turnitin on students' work and tutors' practice has also been explored. Turnitin's capacity to deter students from plagiarism has been asserted by lecturers (Bennett, 2005;Sutherland-Smith & Carr, 2005), by those promoting use of the tool (Murray, 2006) and by students themselves (Davis, 2007). For example, Murray (2006) claimed that 76% of students responding to a survey "felt that Turnitin could discourage plagiarism". Contrasting views on its use as a deterrent are often shared via informal networks (for example, the UK government -funded Plagiarism Advisory Service mailbase discussions). Concerns include the danger that introducing students to Turnitin may simply teach them better strategies for avoiding detection, or what one mailbase contributor called "making the colour go away" (removing the colour-coded matches). There is a concern among academics that it could take away responsibility from the tutors to support student learning (e.g. Sutherland-Smith & Carr, 2005), but many institutes (such as the University of Maryland University College) actively promote Turnitin as an approach to improve students' awareness of plagiarism. However, there are few studies which investigate how best to use Turnitin to encourage learning, and furthermore, even fewer which investigate the central role of formative feedback in this process. This study addresses this apparent gap.

Case study: Oxford Brookes University, UK
The case study was made at Oxford Brookes University, UK, of international students on a year-long pre-Master's diploma for language and study skills, known as a programme of English for Academic Purposes (EAP). A total of 66 students participated in the research over three years (2007:19, 2008:23, 2009:24). The investigation followed the intervention of a pre-assessment tutorial feedback session using Turnitin on a second semester academic writing module, entitled the Extended Writing Project, towards the end of the programme. The module required students to produce a 3,000 word assignment, in preparation for their Master's level dissertation, on a topic related to their future study. Students were preparing for postgraduate study in a range of disciplines but a core majority were planning to take Business. The students were approximately 60% East Asian (Chinese and Japanese), while the remaining 40% were from the Middle East, North Africa and Eastern Europe; all had a non-UK accredited undergraduate degree. Students were admitted via an English language competency exam (the International English Language Testing System) with a band of 5.5 or above.
Plagiarism education formed a central theme of the module. At least six weeks of instruction was devoted to information-giving, examples and awareness-raising activities, supplemented by online discussion facilities and institutional warnings in the student handbook. To be consistent with the approach across the university, the Harvard system was taught, practised and discussed with particular emphasis on issues related to accurate use of sources. Following one trial of Turnitin in 2006, its use was made an integral part of the educational process during the module each year. Students submitted an electronic version of their first drafts to their tutor, who uploaded them to Turnitin and studied the resulting originality reports, before using the screen-based information as part of 15 minute individual tutorial feedback interviews. Tutors asked questions during the tutorial to promote student thinking about source use.
Data on the impact of the above teaching strategy was gathered by analysing issues that arose from perceived changes in student work from first draft to final draft in the tables below using similar measures of severe, moderate, minor and none, for consistency. The Turnitin originality reports (excluding the matches to bibliographies) were analysed for changes between the drafts. At the end of the module, all students in the research also completed a questionnaire to evaluate the usefulness of Turnitin for different areas of learning (for which the mean percentages have been calculated below), and a number of students (16) contributed comments in a focus group in 2007. Four key areas emerged from the data:  reduction in the amount of plagiarism  reduction in over-reliance on one source  reduction in citation errors  reduction in insufficient paraphrasing.

Avoiding plagiarism
For the purpose of this study, levels of plagiarism were categorised as severe, (10% or more of unattributed text in the draft), moderate (5-9.9%), minor (0.1-4.9%) and no plagiarism (complete absence of unattributed text). To produce a representative result, particularly with the varying lengths of first drafts compared with second drafts, the number of unattributed words from a matched source was calculated as a percentage of the word count (excluding bibliography). Table 1.

Percentages of plagiarism in first and final drafts 2007-2009
The table indicates a reduction in levels of severe and moderate plagiarism to the majority ending in the minor or no plagiarism categories. The results were affected by one student in 2008 submitting a shorter first draft with little source use, and a final draft with long sections of copied and pasted unattributed text; as an example of severe plagiarism at the final draft stage, this is clearly a case where the tutorial feedback was unable to help. Nevertheless, most identified plagiarism was at a very minor level of 1-2% of text, and a mean of almost half (45.5%) reduced plagiarism between drafts over the period.
Overall, from the questionnaire results, a mean of 73% stated that Turnitin was useful for understanding how to avoid plagiarism and talked about the important learning opportunity they had experienced: The software shows your mistakes. I would like to use it again because sometimes we do plagiarism without noticing it (Hospitality student).
The software is to find plagiarism, so it does not teach how to avoid it. The teacher teaches us that (Technology student).
As perceived by the students, Turnitin offers support to help them avoid 'mistakes' or unintentional plagiarism, but importantly, they also recognise its limitations and the importance of the tutor's role.

Over-reliance on sources
From the list of matched sources Turnitin supplies, it was possible to examine how much students were relying on one source or a number of sources. Similar descriptors are used for levels of the highest rank source use (severe, moderate, minor, none), though the percentages are scaled lower to suit this category.

Percentages of use of highest ranked source in first and final drafts 2007-2009
During the three year period, approximately half (mean: 45.5%) reduced the percentage of the top ranked source in their final drafts. In first drafts, the highest percentage from one source was 31% in 2007, 6% in 2008 and 11% in 2009, while the highest result for second drafts was over 5% in only three cases over the period, two of which were presentations of government data. In fact, the majority (70% over the period) of top-ranked sources in final drafts were 1-2%, which is generally regarded as an insignificant amount of matching (Davis, 2007). Those who increased the top source did so by a small percentage (66% of those who increased the top source only did so by 1%), and this especially occurred following the submission of a much shorter first draft (in 60% of cases of an increase in the percentage of a top source, the first draft was under 1,000 words). In addition, questionnaires revealed that a high percentage of 82% believed Turnitin was useful for avoiding over-reliance on sources; they seemed to find the visual representation clear and memorable, as shown by the comments: Turnitin is useful for avoiding over-reliance…because there is a sort of pressure, from the software to make aware of use of sources (Business student).

Correct and complete citation
Attempts at using citation within the text (in particular, the use of author's surname, year and where appropriate, page number) were recorded as problems when students' efforts were incorrect or incomplete (note: for this study, errors in the bibliography were not analysed). Differences between the instances of incorrect or incomplete citation in the first and final drafts were evident, especially as Turnitin highlights different references in colour. Numbers of errors were separated into four levels with similar descriptors: severe as ten or more errors, moderate at five to nine errors, minor as one to four errors and no errors. Table 3.

Percentages of levels of citation errors in first and final drafts 2007-2009
The table indicates a reduction of higher levels of citation errors, with none at a severe level in final drafts, and a great reduction of those at moderate level, thus a much greater number at a lower level, with more than half of final drafts showing no citation errors. The mean reduction in the number of errors was 62%, showing evidence of greater understanding of how to use citation accurately in their final drafts. This shows they have learned and used their learning, not just 'made the colour go away' from the original instances. Furthermore, the type of error changed in ways that indicate students began attending to significant aspects of the referencing process. Common errors in first drafts were use of first names rather than surnames, use of authors' surnames with an initial for first names (copying a bibliographical style), absence of page numbers for direct quotations, problems with punctuation and spacing. In contrast, the small number of errors that persisted in final drafts tended to be relatively minor inaccuracies of punctuation and spacing. Errors of format, such as using a bibliographical style reference for in-text citation, were eliminated.
Questionnaire responses revealed a mean of 69% of students found Turnitin reports useful for learning to write citations accurately. Students themselves described their errors as arising from both the process of writing and from incomplete knowledge. It was very notable that there were no cases in any year of the number of citation errors increasing, even where first drafts were very short (under 1,000 words) compared to the required 3,000 word final draft; this also indicates that students seem to have more knowledge of accurate citation use by the final draft: The software helped me to be more cautious about citation (Business student).
Sometimes we make a mistake and don't use citation or not correctly… this way gives us a second chance (Business student).
For me, I will never ever forget about efficient citations according to my first experience of Turnitin (Business student).
Therefore, students seem to recognise that the tutorial using Turnitin reports could be an important and memorable learning opportunity to take more care with citation.

Paraphrasing
Unlike generic search engines such as Google, Turnitin is able to identify continued textual borrowing that includes gaps. This makes it possible to study attempts at paraphrasing in detail and to consider whether the alterations made by the student were appropriate, for example where they are adopting a synonym substitution strategy (e.g. changing one in every four words) for large parts of a text. This study paid particular attention to instances where text highlighted by Turnitin showed an extract of ten words or more, with one to four words in grey text, which signals places where students have altered the original. These instances have been calculated as insufficient paraphrasing, using the same levels as for plagiarism above, for ease of understanding. Table 4.

Percentages of insufficient paraphrasing in first and final drafts 2007-2009
It is striking that the severe and moderate levels of insufficient paraphrasing largely disappeared by the final draft (despite the one example in 2008, the same outlying example as in the plagiarism table above). Most drafts with examples of insufficient paraphrasing were at a minor level, with less than 5%. The overall number of final drafts with a reduced level of insufficient paraphrasing was not as high as the result above for plagiarism, and in some cases the percentage rose. One explanation could be the number of shorter drafts (59% producing first drafts of less than 2,000 words) because longer final drafts meant more attempts at paraphrasing. It is also recognised that paraphrasing is a complex skill needing practice; if few attempts are made at the first draft stage, students could not benefit from tutor feedback.
Nevertheless, a high proportion of students overall (76%) stated that Turnitin was useful for thinking about appropriate paraphrasing. Students made comments such as: International students are not good at paraphrasing. So Turnitin is useful for students thinking about it (Business student).
It highlights some part of bad paraphrasing, then we can correct it (Social Sciences student).

Turnitin was useful for thinking about paraphrasing… well that is what the software is all about (Business student).
It seems that students were therefore connecting the software with learning about paraphrasing and found Turnitin helpful for finding 'bad' paraphrasing. However, unlike the more 'technical' issues with applying citation conventions, the more complex writing skills needed for paraphrasing clearly take practice and repeated feedback, presumably beyond the limits of this case study.

Discussion
Unlike a similar study by Barrett and Malcolm (2006) in which heavily plagiarised first drafts could only achieve a bare pass in final results, in this study no penalties were imposed at the formative stage, even if there was evidence of serious plagiarism. This was to encourage open discussion and to reduce the possible threat students might feel about assessment at the drafting stage. The feedback session included questions about the writing process, such as how it would be possible to use other author's words in a different way and whether a certain percentage of use of others' words represented appropriate or too extensive support to arguments. The students could use the feedback to amend their use of sources, re-draft their text and submit a final version for assessment four weeks later. The intervention through guided tutorial feedback using Turnitin reports seems to have encouraged an improvement in the areas under review.

Avoiding plagiarism
Much of the learning about academic writing is evidently focused on acquiring good practices and avoiding unacceptable ones such as plagiarism. However, research has shown that students often do not internalise instruction about plagiarism until it refers to their own work (Barrett & Malcolm, 2006), thus the discussion of sources in the feedback is an opportunity to achieve this, in a clear and direct way (Hyland, 2001). Many students seemed to have a kind of 'eureka' moment, when faced with the onscreen evidence of how they had used sources, where they understood more fully about issues related to academic integrity, as they connected to their own work. These realisations were clearly useful learning opportunities, which bears out the high number of students believing Turnitin helped them understand plagiarism found by Murray (2006).

Over-reliance on sources
The visual display seems to have been a factor in students' awareness of how much they were using sources, and whether they were, at times, relying too heavily on some sources. This seemed a useful and unexpected finding of the study, especially as there are few studies of the appropriate amount of source use in the literature. Students who need help to use sources well are often not specifically advised to use a wide range of texts and sources. This relates to Pecorari's (2003) study, where some confusion was reported on the amount of source use, and student respondents who were concerned that too many quotations may be seen as "something like plagiarism" (p. 337). Howard (2000) has made a very important distinction here, by identifying the practice of putting together sections of text from other authors to form a new text without a clear new voice as "patch writing". This may be seen as an expected and important stage of learning (Pecorari, 2008). The highlighting by Turnitin of these patches helped to show students where there was greater need for original voice in academic writing, sometimes an unfamiliar skill for students from different academic cultures (Pennycook, 1996;Hayes & Introna, 2005). This also confirms the findings of Pittam et al. (2009) regarding the difficulty for students to establish authorial identity.

Correct and complete citation
Similarly, international students are frequently new to the convention of in-text citation. It can be difficult to know when, how and how much to cite, as Shi's (2008) study of undergraduate writers shows. As Turnitin highlights matched citations and quotations, it makes it easy to see errors or lack of appropriate citation. Students seem to have noted its usefulness in understanding good practice with citation. Furthermore, as Chanock (2008) found, students may copy sections of text and put a citation but not acknowledge them as a quotation; Turnitin reports can help to show evidence of this, and formative feedback may encourage students to rectify an area which they may have misjudged to be acceptable, thus avoiding any serious consequences should this be linked to academic malpractice.

Paraphrasing
In contrast to the other areas, there seemed to be less reduction in the cases of inappropriate paraphrasing in final drafts. The result is perhaps surprising, given the students' perception that Turnitin was helping them effectively with paraphrasing, as shown in their comments above. The increase in errors may be a consequence of the methodology used for collecting data, linked to the length of students' draft submissions. Shorter drafts, particularly those of less than 1,000 words, permitted few attempts at paraphrasing to be analysed and given feedback on. Subsequent 3,000word final drafts included more attempts and therefore, more opportunities for error. However, the finding does indicate a need for more instruction and practice of paraphrasing in class, as found by Davis (2007). Furthermore, since the skill of paraphrasing is a complex one, which takes repeated practice and extensive reading (as opposed to the more technical and straightforward issues of applying citation conventions to punctuation), it seems unlikely that Turnitin could do more than stimulating discussion about paraphrasing.

Conclusions
This study indicates that formative feedback on a one-to-one basis using Turnitin originality reports seemed to have a positive impact on students' final submissions. Explanations for this effect include offering time, encouraging students to feel less distanced from their work and/or more engaged, all of which have been shown to be relevant. As suggested by Hyland (2001), giving direct advice on plagiarism in a nonthreatening, formative way in spoken feedback seems an effective approach.
The results show that the tutorial feedback had some specific effects on drafts including a noticeable reduction in plagiarism, mainly to a very minor level each year. Furthermore, the learning opportunities could be seen in other areas related to plagiarism education. After the intervention, there were reductions in the numbers of citation errors, a reduced amount of over-reliance on sources and some, though limited, effects on insufficient paraphrasing. Thus the potential for using Turnitin as an educational tool can be seen, in common with the study by Peacock et al. (2006), who found that "when used in non-policing mode, Turnitin acted as a form of goal-oriented learning for good referencing practice". It is this learning opportunity for good practice that Turnitin may support which is the most important result. It is acknowledged that numbers are small for each year (19-24), but as the same data was collected over three years involving a total of 66 students, the results may be considered more convincing. Overall, it does seem that the tutorial feedback may have helped students to improve their academic integrity, although we can only claim that these results are one of several means to determine whether levels of plagiarism have decreased.

Recommendations
Following the study, a major recommendation is to include an individual tutorial feedback session at the formative stage using Turnitin for important learning opportunities about avoiding plagiarism. Where this is impossible (or is seen to be impractical) due to large class size, small group tutorials, or a large group session providing examples of reports and typical errors may be worthwhile ways to offer some of the benefits found in this study. A student's overall programme can be created to offer this one-to-one opportunity at a strategic point, or points in their learning.
A limitation of this study was that the students' continuing performance has not been monitored in their further studies. A follow-up investigation is necessary to assess how much students are able to transfer this knowledge to a new learning context. Further research could also be carried out by collecting tutors' views on the usefulness of tutorial feedback using Turnitin, which would help to evaluate the benefits and issues still to be addressed.