{'fraction_non_alphanumeric': 0.028100475373407218, 'fraction_numerical': 0.006807122162527074, 'mean_word_length': 5.404611781660962, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Comparison of Beginning and Advanced Engineering Students’ Description of Information Skills

Abstract 
 
Objective – The purpose of this research was to examine how beginning and advanced level engineering students report use of information when completing an engineering design process. This information is important for librarians seeking to develop information literacy curricula in the context of engineering design. 
 
Methods – Researchers conducted semi-structured interviews about information strategies used in engineering design with 21 engineering students (10 first and second year; 11 senior and graduate). Researchers transcribed interviews and developed an inductive coding scheme. Then, from the coding scheme, researchers extracted broader themes. 
 
Results – Beginning level engineering students interviewed: (a) relied primarily on the parameters explicitly given in the problem statement; (b) primarily used general search strategies; (c) were documentation oriented; and (d) relied on external feedback to determine when they had found enough information. Advanced level engineering students interviewed: (a) relied on both their own knowledge and the information provided in the problem statement; (b) utilized both general and specific search strategies; (c) were application oriented; and (d) relied on self-reflection and problem requirements to determine when they had found enough information. 
 
Conclusion – Beginning level students describe information gathering as externally motivated tasks to complete, rather than activities that are important to inform their design. Advanced level students describe more personal investment in their use of information through consideration of information based on their prior knowledge and questioning information. Future research should consider how to best support beginning level engineering students’ personal engagement with information.


Introduction
With the vast amount of information readily available to current students on the open web and through library resources, the skillset necessary to navigate through information and use it appropriately is arguably one of the most important factors for educational success (Bruce, 2004). All students, both undergraduate and graduate, need to possess information literacy skills to manage the rapidly changing technological environment. In particular, future and current engineers are challenged to be adept at information literacy as rapid research and technological advances in their fields generate new and changing information that directly impacts their daily work. Throughout their careers, engineers must stay current within their field and incorporate new information to inform their own professional development (Fosmire & Radcliffe, 2014). Innovation and future technology are, at least in part, influenced by information literacy skills (Fosmire & Radcliffe, 2014).
In this qualitative study, we report data collected through interviews with students rather than relying on surveys or other quantitative measures of students' thoughts. This approach is especially helpful in uncovering assumptions, conceptions, and strategies students might employ as they solve open ended engineering problems that might be difficult to capture, for example, in a more formal survey or analysis of work products. By discovering the information strategies students use in an engineering context, the results will provide insight into the misconceptions that need to be corrected, as well as areas of strength that can be built upon in instructional interventions.
Information literacy skills are an important part of undergraduate and graduate education. For example, the Association of College and Research Libraries (ACRL) standards state an information literate person should be able to effectively search for, identify, evaluate, use, and document information (ACRL, 2000). In addition, the document Criteria for Accrediting Engineering Programs, created by the body which accredits engineering programs, ABET (Accreditation Board for Engineering and Technology), includes a requirement that students demonstrate life-long learning skills (2013). As we prepare future engineers, it is necessary to cultivate habits of information literacy that will serve as the prerequisites for life-long learning. According to a 2006 report, 91% of responding engineering employers rated life-long learning as either essential, highly important, or moderately important (Lattuca, Terenzini, & Volkwein, 2006). Yet, Lattuca and colleagues note that in the years since this criterion was originally released, growth in preparing engineering students in life-long learning skills has been relatively stagnant.
Research directed specifically at engineering students' information literacy is sparse. Of the research conducted, some inconsistency has been noted between first-year engineering students' self-assessment of their skills and actual task performance Atman, Chimka, Bursic, & Nachtmann, 1999;Douglas, Wertz, Fosmire, Purzer, & Van Epps, 2014;Wertz, Purzer, Fosmire, & Cardella, 2013). It can be difficult to ascertain whether inconsistencies are genuine self-inflation or artifacts of the surveys used. As Davidson (2005) noted, there is a sea of qualitative assumptions behind every quantitative measure. The purpose of this current research is to gain a deeper understanding into how engineering students discuss use of information literacy principles in the context of engineering design. The implications of this study include specific recommendations to inform decision-making among academic librarians and areas to target for curricular interventions. Dym, Little, Orwin, and Spjut (2004) define engineering design as "a systematic, intelligent process in which designers generate, evaluate and specify designs for devises, systems or processes whose form(s) and function(s) achieve clients' objectives and users' needs while satisfying a specified set of constraints" (p. 6). While there are many models of the engineering design process, it is commonly understood to result in the creation or adaptation of a product, system, or service after a process of feedback loops and iterations (Fosmire & Radcliffe, 2014).

Information Strategies of Engineers
The use of information in engineering design is context dependent. One of the roles of the engineer is to determine the type and depth of resources needed to complete the project (Ellis & Haugan, 1997;Tenopir & King, 2004). Engineers have been characterized as using a "least effort" approach to information gathering. That is, their goal is to find "reliable answers to specific questions," in contrast to scientists who are motivated more by deeper understanding of concepts (Pinelli, 1991). Further, engineers try to minimize loss (e.g., of time or performance) rather than maximize gain of finding the perfect solution (Pinelli, 1991). Engineers value accessibility above all else and rely on colleagues, personal knowledge, and personal collections as the most desirable sources of information (e.g., Engel, Robbins, & Kulp, 2011;Gerstberger & Allan, 1968;Leckie, Pettigrew, & Sylvain, 1996). However, engineers with advanced degrees (Kwasitsu, 2003) and exposure to information literacy coursework (Holland & Powell, 1995) consult formal information sources (e.g., journals, standards, patents) at a higher rate than those without. In recent years, Google has become an important first-resort method of professional engineers as well (Allard, Levine, & Tenopir, 2009). Ingwersen and Jarvelin (2005) conceptualize information use as centered on work tasks, where an engineer determines knowledge gaps between the task and their ability to complete it. Ellis and Haugan's (1997) behavioral model for engineers includes processes of Surveying, Chaining, Monitoring, Browsing, Distinguishing, Filtering, Extracting, and Ending. Recently, there has been a movement to combine information literacy and engineering design processes in an effort to cultivate information literacy skills and life-long learning among engineering students (Fosmire & Radcliffe, 2014). Fosmire (2012) applied Kuhlthau's (2004) Information Search Process (ISP), which comprises the stages of Initiation, Selection, Exploration, Formulation, Collection, Presentation, and Reflection to engineering design. Certainly, the engineering design process extends beyond information literacy; however, the skills needed to seek, evaluate, apply, and document information are essential to successful design . Johnson and Simonsen (2015) condense the above (and additional) information process models into the core tasks of Start, Survey, Evaluate, Gather, and Finish. The above process models show a great deal of similarity, which gave the current authors confidence in using Kuhlthau's ISP model as the basis of interview questions. While all the process models show similar stages, Kuhlthau's model is more highly tuned to the information components, so provides the opportunity for more focused questions.

Engineering Students' Information Literacy Skills
Prior research related to undergraduate students' information literacy has shown that while students report an overall confidence in skills associated with IL, such as evaluating information, there are aspects in which students do not feel adequately prepared. For example, Head and Eisenberg (2010) found 84% of undergraduates surveyed had difficulty getting started in the search process and synthesizing information. Similarly, in a study of first-year students, 74% struggled with online searches and 43% reported problems making sense of all of the information gathered (Head, 2013). Engineering students tend to use library resources less frequently than in other disciplines (Collins & Stone, 2014) yet, is unclear why, highlighting the need for more research to understand engineering students' information literacy skills and habits.
Emerging engineering education research has pointed to areas where librarians and instructors could further support engineering students in information literacy. For example, researchers have found differences between novice engineers and expert engineers in how they use information to make design decisions. In particular, expert engineers seem place a higher value on the role of information in solving design problems than do beginning students. Mosberg et al., (2005) found that when engineering experts were asked to select the most important aspect of design, they ranked gathering information as the fourth highest of 24 design activities. Atman, Adams, Cardella, Turns, Mosborg, & Saleem (2007) found experts collected substantially more information, over a larger breadth of topics, than students when engaged in design. Although engineering students have a strong self-perception of their information literacy skills (Ross, Fosmire, Wertz, Cardella, & Purzer, 2011), their actual performance was poor when asked to identify reliable sources and appropriate use of information to support a design decision (Wertz, Purzer, Fosmire, & Cardella, 2013). Even Masters-level engineering students did not demonstrate an effective search process or awareness of library sources or services (Johnson & Simonsen, 2015).
Most studies of the information skills of engineering students have focused on citation analysis, specific search strategies and database use (Bhatt, Dennick, & Layton, 2010;Hensel, Brown, & Strife, 2012;McAlpine & Uddin, 2009;Welker, McCarthy, Komlos, & Fry, 2012;Wertz et al., 2013). These research studies, when taken together, speak not only to the challenges associated with methods of self-report measurement, but also to challenges associated with engineering students' development and recognition of their own information literacy skillset. Without recognition that their own skills could be further developed, students are unlikely to be motivated to actively engage in increasing their information literacy. This study, on the other hand, asks less directed questions regarding library services than what has previously been done (George et al., 2006) and concentrates on the students' general conception of their processes (see Appendix).

Aims
While there have been some quantitative studies of engineering students' information literacy, a deeper look into the varying skill levels between advanced and beginning students is needed. Furthermore, there is limited research directly related to how engineering students describe their use of information when faced with a design project. Before curriculum can be developed to target engineering students' information literacy needs, there must be an exploration into how students report using information specifically when faced with engineering design. Therefore, a qualitative approach is needed to inform the conceptualization of how beginning and advanced students approach information when faced with an engineering design project. According to Peshkin (1993), one of the key outcomes of qualitative research is the ability to make interpretations that "explain or create generalizations, develop new concepts, elaborate existing concepts, provide insights, clarify complexity, and develop theory" (p. 25). Our study seeks to provide insight into how engineering students describe their use of information, for the purpose of informing future curricular efforts that target information literacy in engineering design. In particular, we are interested in informing first-year and senior level curriculum for engineering students.

Setting, Participants, and Interview Structure
We developed the research design based on the responsive interviewing method of qualitative research, as discussed by Rubin and Rubin (2012). To inform how to support information literacy for first-year and senior engineering students, we purposefully chose to interview two groups of students. "Beginning" level students are defined as being in their first or second year of their undergraduate engineering program and "advanced" level students are defined as being either a senior or a graduate student in an engineering field. We included second year and graduate engineering students in order to capture skill development gained through the current academic year. Therefore, a purposeful sampling strategy was used to create comparable groups, in an effort to fairly examine beginning and advanced engineering students with regard to information literacy.
We conducted the study at a large mid-western university with a large College of Engineering. After obtaining human subjects research approval, we recruited interviewees through posted fliers in campus buildings where firstyear and advanced engineering courses are routinely held. The flier stated that students would be paid $15 for participating in the study. Students went to an online survey and entered their year in school and email information. Only students that fit our study criteria (first-year, Other 0  2 second year, senior, and graduate engineering student) were interviewed. We described to interviewees the purpose of the interview, the procedures that would occur during the interview, and reiterated the voluntary nature of the interview. Interviewees were asked to give informed consent prior to interviews. Qualitative studies do not have rules on sample size. Patton (2002) states the information richness of the cases selected is more important than the number. We interviewed a total of 21 engineering students, 10 beginning and 11 advanced, for approximately 30 minutes each. All engineering students at the University are admitted as first-year engineering students, then they apply to engineering disciplines in the sophomore year. The students' majors represented include: first-year, aerospace, chemical, civil, computer and technology, electrical, industrial, materials, and mechanical engineering. Throughout this work, we refer two groups of students. "Beginning" refers to the eight first-year and 2 second-year engineers while the "advanced" group refers to four seniors and seven graduate engineering students. The interviewee demographics are shown in Table 1.
We created a semi-structured, responsive interview protocol based on guidelines from Rubin and Rubin (2012). Responsive interviewing allows researchers to probe for deeper insights based on information provided by the interviewee, compared to a semistructured or structured interview. In responsive interviewing, the emphasis is on hearing what the interviewee is saying and in the moment having the flexibility to ask follow-up questions that may not have been predetermined, as well as to foster an informal, conversational environment where students felt comfortable discussing their information literacy practices without feeling like they need to provide a "correct" answer.
The initial set of questions were written to include each stage of the Information Search Process (ISP) (Kuhlthau, 2004) in the context of an engineering design project (see Appendix).
The ISP is an evidence-based process model for research, and thus it was used as the underlying framework to make sure all phases of research were addressed in the interview. Multiple follow-up questions were asked to encourage students to think deeper about each topic. As part of the interview protocol development process and training for the undergraduate research assistants, two pilot interviews were conducted and revisions were made to the protocol based on feedback from the interviewees. The first author and two research assistants, trained in the responsive interviewing technique, conducted interviews. None of the interviewees had interacted with the research team prior to this research project. Interviews were conducted in a private conference room located in a campus engineering building. Student interviewees were instructed to think about a recent class-related design project when answering the questions. Each interview was audio recorded and transcribed.

Analysis
We followed Patton's (2002) guidelines for qualitative data analysis following a thematic analysis approach, which allows findings to emerge from the data.

Results and Discussion
In this section the five themes are broken out below, compared and contrasted between beginning and advanced groups, then each group is further characterized with supporting examples.

Awareness of Information Needs
Beginning students in our sample limited their discussion of information needs to those needs given to them through assignment guidelines or talking with instructors. Advanced level engineering students more frequently discussed the need to identify information for the purpose of completing a successful design.

Beginning
Beginning level engineering students were aware of their information needs based on information given to them by instructors. Students discussed reading and re-reading the design problem to determine what information to find. For example, one beginning level student said "I look over, I guess, all the guidelines and make sure I'm clear on everything". The students discussed that they were new to engineering and instructors must guide them into design projects by providing detailed instructions related to the design problem. For example one beginning student said, "Right, well, I'm just a freshman obviously so I'm in first year so a lot of the times they will pretty much just lay it out and say 'these are the constraints'." Another beginning student phrased knowing what information to find "mostly using the guidelines given to me."

Advanced
Advanced level students recognized information needs based on information provided by instructors, but they were also able to identify what to learn and then used information they found to identify additional gaps. In this way, they involved themselves in raising their awareness of information needs beyond what instructors had explicitly stated.
So when I'm given a problem, I try to understand what it means…what background is required for the project. Like….I had to work…..not right now but before this I had to work on an engine related project. And I didn't understand much about it so I went on to read about engines. Try to figure out what my part is in that and then see if I can understand it properly so that I can take it up as a project and go ahead with it.

Strategies for Searching
Beginning engineering students in our sample almost exclusively focused on general search strategies, whereas advanced level students discussed specific information sources used to locate technical information. In addition, advanced level students used their own personal judgment to determine the appropriateness of a specific information source, rather than only relying on taught heuristics.

Beginning
Both in terms of locating and evaluating information, beginning level students identified efficiency as guiding their search strategies. They utilized general search engines to locate sources (e.g., Google). One beginning level student said, "we don't really have too well of a process for initially just coming up with a good source. It's more a trial and error, I guess", and that they "tend to just search for the main keywords that [they're] trying to look for". In evaluating sources, they discussed readily available information, such as URLs, to determine the quality of a source. The beginning students placed a high value on where the information originated: "…usually try to look for like reputable sources like anything with like a .gov or a .org tend to be better than like a blog or something". Another student explained: You can look at the URL and determine like if it's .gov then it's usually credible compared to a .com and we also look at the author information and determine if they're a nobody… and they're talking about things that a PhD student could be talking about and it's not necessarily as credible as it could be.

Advanced
Advanced level students discussed reliance on general search engines and readily available evaluation criteria (e.g., domain name, host), similar to beginning level students, but they also reported utilization of their prior knowledge to make decisions about the usefulness of information. For example, they considered their knowledge gained from previous designs projects. They reported consulting forums, filtering information they found through their own experience to see if it was reasonable, and cross-checking information from multiple sources. Some students reported techniques to evaluate reliability such as replication of the source's information through modeling. A student said a source was credible if you can replicate the results. So, if you get an equation that says this and then they give like a sample…..some other places will give you a sample……if you use the equation and you get the same value and you can find the same equation somewhere else, it's good. It's basically a two-step check. So, if you can find the information somewhere else, then you know it's good.
Another student said: …if you like come across any website or page that you haven't seen up to now, you try to get some sort of a feedback about that website…..maybe through the reviews given in the bottom or ask somebody about the credibility of the paper.

Extraction of Information
When it comes to extracting information from sources to apply to a design problem, beginning level students utilized less sophisticated reading strategies, and they had a utilitarian purpose for using information sources. Advanced level students tended to report advanced reading strategies and to independently replicate results before using the information in their project.

Beginning
In terms of actual use of information, students described using information as a way to justify their solutions, for example, fulfilling an assignment's requirements for sources, rather than as a way to generate new ideas to solve a problem. One student described citing as a way of increasing credibility: I feel like that if we would just do that as much as possible [cite sources] then that helps the audience know that we're not just like throwing stuff out there. We're not just making up stuff and that we're actually using sources and that we try to base our entire project off of credibility of sources and stuff like that.
Beginning level students also discussed reading entire sources to understand information, rather than reading pertinent parts, indicating that they are having a hard time understanding the structure and perhaps content of papers, using brute force measures to extract meaning. "I try to read the entire source at least two to three times so I thoroughly can understand everything they're saying and make sure that I understand it so I can actually apply it without overlooking little details in it."

Advanced
Advanced level students discussed scanning for information in pertinent paragraphs, rather than reading straight through. Advanced level students also discussed attempts to replicate the results or use data from sources for calculations.
The results of those activities inform whether the information was used in the design. Students described pulling out the relevant parts of an information source: Generally I'll read probably half of it. Well see, like, if it's a good source I'll read the first two paragraphs and then continue reading word for word almost. If it's not exactly what I need I'll skim the first couple sentences in each paragraph and skip until I find something that's relevant. and, For this year I did project [and found] 60 or more articles that total 400 pages. There's not enough time. So I, like…..you know you find…..you know "ctrl + F" for the terms you want and you read the page above and below it and then move on with your life.

Sufficiency of Information
Students in both groups discussed some level of uncertainty in describing how they knew they had found enough information; however, how they determine the amount of information needed varied. Beginning level students more frequently discussed external factors to terminate their search for information, such as time constraints or lack of new results from a search engine, while advanced level students discussed determining whether their questions were completely answered as the mechanism for deciding when to stop looking for information.

Beginning
Beginning level students mentioned external feedback as a way of knowing when they had found enough information. Students discussed ending the search process based on pragmatic issues such as an answer found, group agreement, and time constraints. One student said, "I think when the information that we have starts to overlap more and more and we don't find anything new in, I don't know, ten….fifteen minutes then….well that's just saying that we run out of things to find." Another student described getting search results as ending the process, rather than thinking about other search terms or strategies, "I guess it's more when I've exhausted all the sources I can find and because if I type something in Google, first couple of pages will be real useful but after a while it's completely irrelevant to what I'm doing."

Advanced
Advanced level students reported the use of internally constructed measures to determine whether they had sufficient information. They discussed internal decision-making related to sufficiency of information. For example, one advanced level student said, "It's hard to say. It's very hard to say. Maybe when I try my idea, I find I'm not….I have not enough information so I go back to search again." Other students described the internal process they go through, such as "When you're not asking yourself questions, I guess, when you are at a point when you have gone far enough to continue selfreliantly, I think, is a point where you can say you have enough information."

Organization of Information
Beginning level students more frequently reported organization of their sources in a simple Word file, although both advanced and beginning level students did describe using that technique. Advanced level students also reported the use of additional methods, such as bibliographic management tools or filing papers by subject within folders. None of the respondents utilized advanced strategies such as rating or tagging sources within bibliographic systems. Sharing information among project team members was also mentioned more frequently by advanced level students than beginning level students.

Beginning
Beginning level students focused on the documentation aspects of the information search process, such as securing citations and references, rather than focusing on how they interact with information (e.g. collecting, using, and applying information). Strategies for maintaining information were not as clearly described as the role of citation in presentation. For example, in response to the question, "What do you do with information?" one student replied: Copy down what I think would be really useful into a Google doc, bookmark or just find the link and write down the link so what I think are the more useful sites that I can just refer back to them.

Advanced
Advanced level students discussed systems for saving the information about sources throughout the duration of the project, through tools such as electronic folders, bibliographies, and notes. One student describes one such organizational method as: "I keep a little folder for the specific project on Firefox or whatever browser I'm using. Professors, I write down their name, what the information was, the date…I keep specific tabs inside the folder for online ones..."

Summary
Through the analysis process several themes regarding similarities and differences between beginning and advanced level engineering students' information literacy strategies emerged.
All beginning level engineering students interviewed described: (a) reliance on the problem statement and parameters as explicitly given, (b) use of primarily basic search strategies, (c) orientation toward documentation purposes of information, and (d) reliance on external feedback to determine when they had found enough information.
All advanced level engineering students interviewed described: (a) integration of both their own knowledge and information provided in the problem statement, (b) use of both basic and advanced search strategies, (c) application purposes of information, and (d) integration of self-reflection and problem requirements to determine when they had found enough information.

Conclusion
In this study, we were able to identify several differences between the information habits of beginning and advanced level students. As might be expected, beginning level students exhibited less sophisticated strategies, relying more on rote or external operations and measures of success. They treated information gathering more as an activity to be completed as a mechanistic part of an assignment (e.g., collect five sources), rather than as a means of improving their work products or increasing their understanding. Indeed, beginning level students treated the projects themselves more as externally motivated tasks to complete rather than activities in which they were personally invested. Advanced level students, on the other hand, had internalized the purpose of information gathering. They compared information found to their prior knowledge and asked questions of the information, for example, whether it met their needs and whether unanswered questions remained.
In this respect, the advanced level students do seem to be acting more like experts than novices (Wankat & Oreovicz, 1993). Among other characteristics, Wankat and Oreovicz (1993) observe that experts can "chunk" information and identify fundamental information. Rather than treating each bit of information as separate and independent, they can identify relevant information and draw inferences from incomplete information, and they take time to define (and redefine) and explore a problem, rather than jump to a conclusion. Novice students use trial and error instead of coherent strategies.
It should be noted that expertise is typically limited to a particular field (e.g., an expert chess player is not necessarily a good dancer), so there is no a priori reason expert engineers would be expert information gatherers. Thus, it is encouraging that the advanced engineering students did in fact show well-developed information abilities and attitudes. Some of advanced skills follow from a greater disciplinary knowledge, i.e., students with a well-developed personal knowledgebase of engineering principles can more easily identify relevant and reasonable information from a source. Other skills, for example, information organization, are less obviously related to disciplinary knowledge, and they do show weaker improvement between the two samples, i.e., some advanced level students used advanced knowledge management strategies, but others showed similar strategies as the beginning level students.
The implications for instructors are clear.
Beginning students need to develop the "expert" information literacy skills they will need as professionals. They do not demonstrate an awareness or internalization of the importance and purpose of information in solving of engineering problems, so problems should be posed and feedback given that provides direct guidance where unsupported claims or suboptimal solutions could be strengthened by the search for and appropriate application of additional information. This study identified several areas where there are marked differences in the quality of information strategies used by beginning and advanced students: awareness of information need, search strategies, extraction of information, organization of information, and determining the sufficiency of information.
The qualitative summaries of these concepts show the thinking processes of beginning students, so instruction can directly target those conceptions. For example, discussions of why information is needed in problem solution, or practice creating mind-maps of a topic to identify what information students know and what is not, and what questions they want to explore further, will help students engage with the concept of being aware of their information need. In that way, they can internalize the importance and use of that concept. Similarly, for the other concepts, exercises that reinforce higher quality strategies and the better results that follow from using them will allow for the development of expert information literacy skills in tandem with expert disciplinary skill development. Ideally, these concepts need to be reinforced consistently over several courses so that students practice transfer of information literacy concepts across different problem contexts.
As with all research, this study has limitations. As a qualitative study, we interviewed a small number of beginning and advanced level engineering students to develop a deeper characterization of how they differ in their information literacy. The students interviewed were from one large research university, and it is unknown how similar engineering students at other campuses are to those in this sample. However, only findings that could represent every student interviewed were included in this study to strengthen the implications of the results. Future research should consider whether these findings are consistent with engineering students at different institutions.