{'fraction_non_alphanumeric': 0.02833423569752752, 'fraction_numerical': 0.010497503459062744, 'mean_word_length': 5.235371342835709, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Using Portfolio Items to Assess Information Literacy

In 1998, the American Association of School Librarians (AASL) developed nine standards for information literacy skills. Students with these skills are equipped to recognize their learning objectives, identify their information needs, acquire information, evaluate information, and share the results of their effort. These skills are keys to lifelong learning. Standard assessment tools, such as select response, closed-constructed response, and even open-ended-constructed response questions are sufficiently dynamic to align with the real-world experiences of learners exercising information literacy skills. In this study, an information structure was designed for students to use to describe learning activities. These written, student-generated items become part of a student’s portfolio. It is proposed that this information structure can serve as an alternative, authentic tool to assess students’ information literacy skills. Two student portfolio items are presented in this report along with a description of the process used to create assessments.


Introduction
In 1998, the American Association of School Librarians (AASL) set nine standards for information literacy skills. AASL proposed that information literacy is a key to lifelong learning. Students with information literacy skills are equipped to recognize their learning objectives, identify their information needs, acquire information, evaluate information, and share the results of their effort. They take responsibility for their own learning, making it possible for their teacher and librarian to focus on guiding and mentoring them through curriculum content (Eisenberg & Berkowitz, 1990;Wolf, Brush & Saye, 2003;Kuhlthau, 2003). Nations and states have created similar standards for information literacy and incorporated them into content and performance standards (Eisenberg, Lowe & Spitzer, 2004;Missouri, 1996). Yet, in the era of standards-based accountability, there is still little research in assessing information literacy skills in a manner that can be consistently applied across grades and disciplines (O'Connor, Radcliff & Gedeon, 2001).
Expectations for information literacy skills vary by grade level. AASL provides indicators and examples, but not objectives and benchmarks for the nine standards. Instead, the standards address skills and dispositions that become evident during curriculum related learning activities. Therefore, rather than using traditional means for assessing information literacy skills, an alternative assessment tool that is authentic (embedded within the activity) is indicated. From 2001From -2003, the researchers worked with middle school teachers to create a webbased, student portfolio system based on short written assignments or quick writes (Bean, Drenk, & Lee, 1982). An analysis of these portfolios items provided evidence of information literacy skills. This paper proposes that information literacy can be assessed using portfolio items similar to the ones created in this study. These assessments can be used in formative, summative and longitudinal evaluations of students' skills.

Alternative, Authentic Assessment
The standards movement in the United States is directed at producing a fair and equitable education for all public school students through the creation of consistent subject content and academic performance guidelines. However, the ideals of the standards movement might not be compatible with a system of rewards and punishment resulting from standardized tests that are comprised mainly of multiplechoice and short answer questions (Shepard, 2000). There is mounting evidence that these scores are little more than proxies of actual learning goals and achievements (Shepard, 1991). These large-scale standardized assessments instruments are the foundation for high-stakes decision-making, in the form of student promotion/graduation, class rank, program placement or tracking, teacher promotion, and school funding. The effect on teaching and learning in the classroom has been a pedagogical shift towards "teaching to the test" (Soloway, et al., 2000). Thus, highstakes accountability frequently inhibits schools and teachers from providing meaningful learning (Darling-Hammond, 1991, 1992Falk, Pecheone, & Darling-Hammond, 1996). The shift towards reliance on standardized assessments for decision-making can also have an adverse affect on students who learn in diverse ways. This is because the students learn at different rates and from various experiential perspectives (Darling-Hammond, Ancess & Falk, 1995;Garcia & Pearson, 1994).
Multiple-choice and short-answer tasks require "on-demand" responses without reflection. Yet reflection is essential for higher order thinking. Thus, assessment tools based on multiple-choice questions and short answers cannot measure higher order thinking skills or students' abilities to perform real world tasks (Sternberg, 1985). In the same way, they cannot measure information literacy skills. In order for educators to create meaningful learning experiences that foster higher-order thinking skills and information literacy, curriculum, instruction and assessment need to go hand in hand. Moreover, assessments must be rich enough and dynamic enough to align with the experiences of learners (Darling-Hammond, Ancess & Falk, 1995;Falk, Pencheon & Darling-Hammond, 1996;National Research Council, 2001;Shepard, 1995).
There has been an intense search for valid, alternative assessments that can complement or replace standardized tests (Arter & Spandel, 1992;Darling-Hammond and Falk, 1995;Falket al., 1996;Moss 1994;National Research Council, 2001). As Mabry (1999) has found, personalized alternative assessment tools, like portfolio systems tend to promote more valid inferences of the achievement of individuals than standardized testing. The kind of customized, concentrated focus on the individual that portfolios provide creates a more complete portrait of individual achievement. Also, this type of assessment is expansionist in nature, presenting a collection of student work that tells the story of the student's efforts, achievements, and progress (Arter & Spandel,1992).

The Study
This research explores the use of information structures as a tool for building an alternative, authentic portfolio-based assessment instrument. (The research platform is described in detail in the methods section of this paper). Information structures are ubiquitous in the world, from the use of an octagonal, red traffic sign containing the word 'Stop' in the United States to the placing of a table of contents in the beginning of a book and the index at the end. Information that is structured can provide greater meaning than information that is unstructured.
Information structures organize and direct expectations regarding the creation, storage, and presentation of information. They can be made uniform based on published standards or protocols. Examples include the labels on food products or catalog records used in libraries. But information structures can also be uniform as a result of conventions-of-use. Conventions-of-use can be defined for any context, global or local. For example, the structure of an essay is universally established but is not as the result of a published standard. On the other hand, the form of a lesson guide created by a teacher can be used in an individual classroom to help students understand classroom procedures.
Information structures can be created through the use of meta-data. Meta-data is data about data. For example, column and row headings in a table are a type of meta-data that help explain the data in the table's cells. As illustrated in Figure 1 below, metadata can provide a method to encapsulate text in ways that make it easy for humans to read and easy for computers to parse and process. The meta-data below is encapsulated in eXtensible Markup Language (XML) tags, which delimit and describe areas of text.
The StoneSoup information structure is derived from the structure used by the Inquiry Page. The Inquiry Page was created in 1996. It was first used in a professional development class for teachers. Inquiry units are used by instructors to track, record, and journal their own inquiries. <?xml version '1.0' ?><Unit> <Background>The StoneSoup information structure is derived from the structure used by the Inquiry Page. The Inquiry Page was created in 1996. It was first used in a professional development class for teachers. Inquiry units are used by instructors to track, record, and journal their own inquiries. </Background></Unit> Plain Text Plain Text with Meta-data Figure 1: How meta-data transforms plain text into XML documents.
Research on meta-data often focuses on practices that lead to the standardization of meta-data vocabularies and display schemas like the Dublin Core or the IMS Learning Resource Meta-data Standard. However, prior study on StoneSoup, the research platform used in this study, explored the affordances of the flexibility inherent in basic, two-dimensional meta-data structures similar to the one presented in Figure 1.
This research was informed by a study of the International Classification of Disease (ICD), the 100-year-old information structure used to collect global data on illnesses and death. The ICD is a transnational system where the languages, beliefs, practice, and knowledge representations of its user communities differ. Data are prepared by people with differing levels of expertise: doctors, nurses, and administrative staff. Yet the information gathered is a rich source for several communities of practice, such as community health organizations, governments, medical practitioners, medical statisticians, health insurance companies, and pharmaceutical companies. The reason the ICD has worked for 100 years is not because of its level of standardization or its flexibility, but because it balances these two characteristics in an effective way (Bowker & Star, 1999). Bowker and Star (1999) have compiled suggestions for creating information structures that coordinate information within communities. Because their research focused on ways to accommodate situational differences, their suggestions are appropriate for creating meta-data that can be used to mediate communication in school environments. StoneSoup uses five of their suggestions. The first is to create heterogeneous classification systems, employing diverse categories for things like "create: writing" and "create: new questions" along with categories for things like "discuss: informal" (See Table 1 in the methods section for the complete set of metadata labels used in the research platform.) Attempts to create categories with a homogeneous classification scheme or a single level of granularity impose restrictions on the information structure that make it less flexible for adaptations at the local school level. A second suggestion is that item labels have somewhat ambiguous definitions. The more precise the definition, the more tailored the answer. Ambiguity invites users to expand label definitions, which, in turn fuels the evolution of the structure. The third suggestion is that labels should always be specific enough to collect targeted information. Thus in selecting item labels, ambiguity and specificity must be balanced well to be effective. A fourth suggestion is that information structures include a category labeled something like "other" in order to encourage people to contribute information they see as relevant, but that doesn't fit in the predefined categories. This also gives the information structure room for change as specific school environments change. Fifth, Bowker and Star highlight the importance of creating rules regarding how and when changes are made. This creates a level of supervisory control that guaranties the structure continues to perform its intended function.

Method
For this study, a research platform, StoneSoup, was created that could support an information structure based on an XML form. Second, authentic classroom settings were found. Third, teachers who could use the research platform were self-selected. Finally, classroom activities were designed and performed using the research platform.
This design-based research was conducted using a hybrid methodology that included techniques from ethnography and participatory design. Ethnographic practices included taking the subject's perspective of phenomena and generating field notes from observations and interviews as a means of data collection. Attention was paid to how subjects constructed the social environment (Shapiro, 1994). These practices also included a willingness to follow a phenomenon wherever it led, rather than to direct participation toward research objectives. Thus, as is common in design-based research, the fieldwork was iterative and heuristic (Hoadley, 2002). Design-based research is also known as development research, which is a more precise label for the type of investigation done here (Van der Akker, 1999; Wang & Hannafin, 2004). To illustrate, the research platform evolved throughout the study based on field observations and formal and informal interviews with teachers and students.
The other methodology used in this study is participatory design (PD), which focuses on tools-in-use (Spinuzzi, 2000). The intent of PD is to make direct and immediate improvements in working conditions. As the name suggests, users are invited to be co-designers of the systems that affect their work practices. PD methods consist of laboratory exercises and fieldwork. What characterizes PD as a research method is the use of scenarios, prototypes, or active systems to direct subject participation. Thus, as a field method, PD is more intrusive than other ethnographic and ethnomethodological methods. For example, in interactions with teachers and students, although participation was not orchestrated, their attention was frequently directed to features of the research platform. However, the discussions that resulted focused on tasks where the participants were the experts. Thus, the participants drove the conversations and activities observed in this study (Carroll, 1995;O'Day, Bobrow & Shirley, 1998;Shapiro, 1994;Spinuzzi, 2000).
The study was conducted at four middle schools. A trade-off was made between indepth and cross-site data collection, favoring the latter. The use of multiple sites ensures multiple contexts. Consistent with an ethno-methodological mode of data collection, each site visit influenced activities at the next site. Being able to present an evolving view of the research platform to users in different contexts provided an opportunity to repeatedly test understanding of users' perceptions of its purpose and function. Robins (2003) undertook a participant observation study at three of the middle schools during 35 site visits. Data from these visits were collected in field notes and in interview transcripts. Snow (2002) conducted a semester long participant observation study at a fourth school. Subjects included four administrators, thirteen teachers, and three library media specialists. Student generated writing was captured by the research platform, StoneSoup. Two of the portfolio items created are presented as data in the Findings section of this paper. Data from portfolio units created by students were not initially a focus of analysis, but because the question of assessment arose frequently, student writings were analyzed for evidence of information literacy skills. The results presented here are preliminary. A separate study directed at using StoneSoup portfolio items for the assessment of information literacy is recommended.

The research platform
StoneSoup is an open-source program that emulates the functions of Xforms, part of the XHTML standard (Dubinko, 2004 A default question set is presented below in Figure 2. Teachers select one or more questions from this guide to direct their students or they can create their own questions to fit the category and subcategory headings in Table 1. Students write answers to their teacher's questions, and can create hyperlinks to resources used and to the products they create, such as essays, study aids, and media files. Students and teachers access portfolios through individual portals. The teachers' portals provide access by class period and lesson. Teachers can also access all of the portfolio units created by individual students. Student portals provide access to portfolio items arranged in chronological order.

Explore
• Background: What's the background for your portfolio unit? What did you know before you started? (and/or) Why is your question important?

Findings
Two units have been picked as representing beginning and advanced information literacy skills. The findings are presented as two figures consisting of a reconstruction of the teachers' oral questions and a transcription of students' written responses. This method of presentation emphasizes the brevity of the writing sample needed for assessment. Table 2 below presents the standards and indicators designed by AASL (1998) and used throughout this section: The student who is information literate accesses information efficiently and effectively. Indicator 1. Recognizes the need for information Indicator 2. Recognizes that accurate and comprehensive information is the basis for intelligent decision making Indicator 3. Formulates questions based on information needs Indicator 4. Identifies a variety of potential sources of information Indicator 5. Develops and uses successful strategies for locating information Standard 2: The student who is information critically and competently. Indicator 1. Determines accuracy, relevance, and comprehensiveness Indicator 2. Distinguishes among fact, point of view, and opinion Indicator 3. Identifies inaccurate and misleading information Indicator 4. Selects information appropriate to the problem or question at hand Standard 3: The student who is information literate uses information accurately and creatively. Indicator 1. Organizes information for practical application Indicator 2. Integrates new information into one's own knowledge Indicator 3. Applies information in critical thinking and problem solving Indicator 4. Produces and communicates information and ideas in appropriate formats Standard 4: The student who is an independent learner is information literate and pursues information related to personal interests. Indicator 1. Seeks information related to various dimensions of personal well-being, such as career interests, community involvement, health matters, and recreational pursuits Indicator 2. Designs, develops, and evaluates information products and solutions related to personal interests Standard 5: The student who is an independent learner is information literate and appreciates literature and other creative expressions of information. Indicator 1. Is a competent and self-motivated reader Indicator 2. Derives meaning from information presented creatively in a variety of formats Indicator 3. Develops creative products in a variety of formats Standard 6: The student who is an independent learner is information literate and strives for excellence in information seeking and knowledge generation. Indicator 1. Assesses the quality of the process and products of personal information seeking Indicator 2. Devises strategies for revising, improving, and updating self-generated knowledge Standard 7: The student who contributes positively to the learning community and to society is information literate and recognizes the importance of information to a democratic society. Indicator 1. Seeks information from diverse sources, contexts, disciplines, and cultures Indicator 2. Respects the principle of equitable access to information Standard 8: The student who contributes positively to the learning community and to society is information literate and practices ethical behavior in regard to information and information technology. Indicator 1. Respects the principles of intellectual freedom Indicator 2. Respects intellectual property rights Indicator 3. Uses information technology responsibly Standard 9: The student who contributes positively to the learning community and to society is information literate and participates effectively in groups to pursue and generate information. Indicator 1. Shares knowledge and information with others Indicator 2. Respects others' ideas and backgrounds and acknowledges Indicator 3. Collaborates with others, both in person and through technologies, to identify information problems and to seek their solutions Indicator 4. Collaborates with others, both in person and through technologies, to design, develop, and evaluate information products and solutions (AASL, 1998) Figure 3 below is a portfolio item created by an eighth-grade student working on a service-learning project for his middle school's Earth Day celebration. The student's role was to work with other students to create the mural that would decorate the school grounds. A public version of this item, without questions and with original spelling and punctuation is available at: http://www.stonesoup.info/cgibin/schema.cgi?1000122.

Explore: Background
Teacher: What did you know about the project before you started? Student: I have not done anything that is this size before. I have organized much smaller things like class meetings but nothing this big.

Create: Presentations
Teacher: How did the assignment affect your learning? When I attended one of our weekly meetings, we were discussing the mural that would be right next to the field [during the Earth Day Celebration]. I quickly volunteered but I was in deeper, more than I thought I was. When I volunteered, I thought I would only help paint it, not plan out a drawing.
I was in trouble early because I couldn't draw hands very well. I searched online and I found this perfect web site: http://susanooh.anime.net/Art/Class/03/. It showed me how to use simple shapes and turn [them] into a complex hand. Our plan was to draw an outline and let the people at Earth Day finish it. It worked out, but they sort of added too much. Before Earth Day, though, we needed to make the outline. Our first plan was to use a projector, but that didn't work because of the light. We learned the hard way that a slide projector didn't work either. Gridding took too long, so we just free-hand drew it.  Information literacy indicators were evident in Figure 3. The student recognized his deficiency as an artist (Standard 1, Indicator 1). He realized that he could use information to enhance his ability to create a drawing (Standard 5, Indicator 2). He used an Internet search to locate appropriate information (Standard 1, Indicator 5). He successfully sought a high quality, appropriate information source (Standard 2, Indicator 4). He used the information to solve the problem of drawing hands on the mural in a practical and appropriate way (Standard 3, Indicators 1 and 4). He used information to benefit his community (Standard 4, Indicator 1).

Discuss: Informal
The student also recognized the problem of producing a large drawing to scale. He investigated a number of solutions (Standard 4, Indicator 2), then selected the one that worked best (Standard 6, Indicator 1) using information in a creative fashion (Standard 5, Indicator 3). He tried iterative plans for successfully completing his task. There might have been information sources available at this step to guide the trial and error process, but they are not sought. However, he made a critical evaluation of three plans before a satisfactory solution was found, which is evidence of a pursuit of excellence in generating visual information (Standard 6, Indicator 2).
The student also shows evidence that he appreciates the democratic use of information in planning a mural as a creative project for the community to enjoy (Standard 7, Indicator 2). The student's behavior was ethical in that he realized the Earth Day Celebration was for the whole school community and he let others take over his Earth Day creation, even if he wasn't entirely pleased with the result (Standard 8, Indicators 1 and 2). He stuck to his group's plan for community involvement (Standard 9, Indicator 2). From this portfolio item, we also see the student is very adept at brainstorming with peers to plan, solve problems, and execute a project (Standard 9, Indicator 3), and receiving guidance from experts when appropriate (Standard 9, Indicator 4).
The rubric below is used to chart the indicators reported here. There is a row for each of the standards. If the writing sample has no evidence of a skill or a lack of skill, a mark is placed in the 'NE' column. If there is evidence that a skill is lacking, a mark is placed the 'Low' column on the rubric. If skill is present according to at least one of the indicators, 'Avg' (for average) is marked. If more than half the indicators for standard have been identified, a 'High' mark is given. Table 3: Information literacy skills assessment rubric for Figure 3. Information Literacy Assessment Matrix (Figure 3 -word count 239)

Number of Indicators Information Literacy Skills NE Low Avg High
Standard 1: Access information efficiently and effectively X Standard 2: Evaluate information critically and competently X Standard 3: Use information accurately and creatively X Standard 4: Pursue information related to personal interests X Standard 5: Appreciate creative expressions of information X Standard 6: Strive for excellence in information seeking and knowledge creation X Standard 7: Recognize the importance of information to a democratic society X Standard 8: Practice ethical behavior using information & information technology X Standard 9: Participate effectively in groups to pursue and generate information X Figure 4 below is a portfolio item created by a sixth-grade student working on a history project related to the United States' Civil War. Learning activities included readings, role-playing, the creation of a mural and re-enacting well-known speeches from that period. Once again, the teacher's oral questions are recreated. A public version of this item, without questions and with original spelling and punctuation is available at: http://www.stonesoup.info/cgi-bin/schema.cgi?1000120.

Discuss: Teachers
Teacher: What did you learn from the class discussion How they are living in Sommerville differently from my group.

Reflect: Description
Teacher: Describe the activities you did for this unit of study. I worked on the mural. I worked on the biography. I'm going to do art. and I watched the movie and took notes.

Reflect: Responses
Teacher: How did your teachers respond to the work you did? They think I need to get more descriptive and more voice and more work Figure 4: Sixth-grader's portfolio item describing his Civil War learning unit activities.
The sixth-grade student who created the portfolio item in Figure 4 lacked the writing skills of the student whose item explored in Figure 3. Though brief, this writing sample tells us many things about the student's information literacy skills. For example, the student did recognize a need for information. He knew he did not know the name of the President at that time (Standard 1, Indicator 1) and he was able to form the need into a question (Standard 1, Indicator 2). However, he was not able recognize a source for the information (Standard 1, Indicator 4) or use a strategy for locating the information (Standard 1, Indicator 5). According to examples provided in Information Power (AASL, 1998, p. 12) these skills should be exercised by fifth grade. Thus for Standard 1, this student shows evidence of a deficit.
During a role-play activity, the student took the role of a 42 year-old father with married children. This activity was effective in providing the student with a new, alternate point of view (Standard 2, Indicator 2), which is reflected in the student's response to the question about the overall affect of the learning activities. This was revealed in his reflection over his written exercises and in his discussion with other students. Learning about a person's life helped him understand what he was learning about this historic period (Standard 5, Indicator 2). Another activity related to the Civil War Project was the creation of the mural. The student's role was to create a train station for the mural. This activity led the student to develop a personal interest based on information acquired. From pictures he looked at, the student considered the geographic area of the South. It piqued his interest to find that trains were the way to travel during the Civil War (Standard 4, Indicator 2). Understanding how inquiry was sparked in this student might help in individualizing learning activities to meet his needs. The rubric below graphs the indicators evident in this student's portfolio item

Discussion
The interpretation of the information in Figures 3 and 4 contained an element of arbitrariness based on the inferences of the researchers. The assessments required personal judgments. But, the same can be said whenever rubrics are used. However, the use of the widely accepted AASL standards and indicators added uniformity to the effort. The findings suggest that the standards and indicators in Information Power are a viable tool for assessing information literacy. The rubric used in Tables 2 and 3 was developed as an attempt to visually represent the information in the assessment. Through future use, it can be refined.
Both of the portfolio items presenting in the findings section contained less than 250 words. It takes only minutes for a student to create these short writing samples, and they can be read in minutes. Because of this, the assessment of a student's information literacy skills can be conducted in very little time. The uniform structure of the portfolio items also has advantages. As the assessment task becomes more familiar, it becomes simpler. For example, for the researchers doing this analysis, it was not necessary to read the teacher's questions to understand the students' responses, only the category and subcategory headings are needed.
In the findings presented here, single portfolio items were assessed in isolation. In authentic practice, student portfolios would contain several items collected over time.
These items could create a body of evidence of information literacy skills. Future research can determine if it is possible to observe skills develop over time. It will also provide opportunity to further refine the portfolio's information structure and the assessment rubric.

Conclusion
Student learning is highly individualized, developing from experiential perspectives. Activities that create learning experiences produce not only content knowledge, they foster information literacy and critical thinking skills. For curriculum, instruction, and assessment to go hand in hand with learning activities, assessments need to be embedded within them. The assessment is then authentic; matched to the rich, dynamic experiences of students as they are led by effective teachers and librarians. Alternative assessment tools like portfolio systems can promote more valid inferences of the achievement of individuals than answers to test questions. Portfolios provide a tight focus on individual students, creating a more complete portrait of achievement. This type of assessment is also expansive since portfolio items can be collected over time. Portfolio items become the data that tells the story of the student's efforts, progress, and achievements.
The information structure used for StoneSoup portfolio items is unique in that it permits a portrayal of more than the products of student learning. The structure makes it possible to demonstrate the processes by which learning is accomplished, including the student's information literacy skills. This study indicates that these portfolio items capture evidence of what students know and what they can do. Because the items are accessible over the Internet, they communicate to all the members of the school community; students, teachers, parents and administrators. Moreover, because they consist of short written pieces they can be assessed in a time frame that fits within the busy schedules of teachers and librarians. Also, these student-generated items become data, an official record of student learning. Portfolio items can link to samples of student work in all disciplines and in a variety of media. In addition, they can include feedback from teachers, thus supporting formative assessment.
Findings from this research indicate that portfolio items based on an information structure similar to the one used in this study can be useful in assessing information literacy skills. It was possible to align the information literacy standards and indicators identified by the American Association of School Librarians with the student items presented in this report. Thus the items are data, evidence of student information literacy skills, and can form the basis of authentic, data-driven assessment.