{'fraction_non_alphanumeric': 0.024661815695658545, 'fraction_numerical': 0.0021487522586316355, 'mean_word_length': 5.203574674341109, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Need for New Guidelines and Training for Remote/Online Testing and Proctoring Due to COVID-19

INTRODUCTION Following the emergence of COVID-19, colleges/schools of pharmacy (C/SOP) quickly shifted delivery and assessment of the curriculum online. The rapid shift to students testing off-campus and away from traditional in-person proctored environments required institutions to evaluate and modify current examination policies and procedures to address new methods of test administration and proctoring. In fact, challenges associated with online testing and proctoring have been frequent topics of discussion through the Assessment SIG listserv, with 24 separate postings/replies to topic threads titled “exam day policies,” “Exam Monitor” policies, and “remote/exam monitoring” appearing on the listserv since spring 2020. Given the frequency of posts and the engagement with these threads, it is evident that guidance is needed. While previous guidance exists for administering and proctoring paper and electronic exams, these recommendations lack guidance on administration of exams without an in-person proctor. Therefore, it is important for C/SOPs to explore strategies for maintaining academic integrity in this new online testing environment, since examinations assess mastery of content for entry into the profession. Additionally, ACPE Standard 10.17 requires C/SOPs to “ensure that assignments and examinations take place under circumstances that minimize opportunities for academic misconduct.” The purpose of this article is to review options that exist for remote testing and proctoring, as well as suggest new guidelines and training that should be developed to accommodate this new testing method.


INTRODUCTION
Following the emergence of COVID-19, colleges/schools of pharmacy (C/SOP) quickly shifted delivery and assessment of the curriculum online. The rapid shift to students testing off-campus and away from traditional in-person proctored environments required institutions to evaluate and modify current examination policies and procedures to address new methods of test administration and proctoring. In fact, challenges associated with online testing and proctoring have been frequent topics of discussion through the Assessment SIG listserv, with 24 separate postings/replies to topic threads titled "exam day policies," "Exam Monitor" policies, and "remote/exam monitoring" appearing on the listserv since spring 2020. 1 Given the frequency of posts and the engagement with these threads, it is evident that guidance is needed. While previous guidance exists for administering and proctoring paper and electronic exams 2 , these recommendations lack guidance on administration of exams without an in-person proctor. Therefore, it is important for C/SOPs to explore strategies for maintaining academic integrity in this new online testing environment, since examinations assess mastery of content for entry into the profession. Additionally, ACPE Standard 10.17 requires C/SOPs to "ensure that assignments and examinations take place under circumstances that minimize opportunities for academic misconduct." 3,4 The purpose of this article is to review options that exist for remote testing and proctoring, as well as suggest new guidelines and training that should be developed to accommodate this new testing method.

Online Testing and Proctoring Software/Technology Options
A variety of third-party online testing and proctoring technologies are available, including Respondus Lockdown Browser and Respondus Lockdown Monitor (Respondus Inc., Redmond, WA), Exam Monitor (ExamSoft Inc., Dallas, TX), Honorlock (Honorlock Inc., Boca Raton, FL), and ProctorU (ProctorU Inc, Birmingham, AL). Online testing and proctoring software provide the ability to administer online tests while the student is observed or recorded through webcam video and audio. Some programs offer additional features, such as securing the user's device to block access to outside resources (ie, websites, class notes, other programs) or recording the computer screen through active screen capture. The four main online proctoring technologies include (1) live, (2) record and review, (3) artificial intelligence (AI), and (4) web-conferencing platforms.
First, live online proctoring allows a proctor to actively observe the student in real-time through the webcam. The proctor observes the student through the entire exam and may ask the student to show their surroundings and confirm student identification prior to the exam start. ProctorU is an example of live online proctoring.
Second, record and review proctoring records the student via webcam during the exam and the video is later reviewed by a proctor for any abnormalities. The video may be reviewed by faculty or staff, or by a third-party organization hired by the proctoring software company. Exam Monitor is an example of record and review proctoring.
Third, automated or AI proctoring is used to monitor the recorded video and audio during the exam and flag any suspicious behaviors, such as persistent off-screen eye gaze, or additional people in the room. HonorLock is an example of an AI proctoring program, which also allows for live proctor "pop-in", should irregularities be noted. With AI proctoring, the post-exam report may indicate different flags (ie, red, yellow, green, depending on the severity of the irregularity) and the ability to navigate to the point in the exam video where the flag occurred. With some programs, a

A J P E
A c c e p t e d D r a f t 2 human proctor reviews flagged videos to confirm breaches or false flags exist. If human proctor review is not part of the contract, the review may be performed by faculty or designated assessment staff, making it similar to the above "record and review", but with the addition of flags. Respondus Lockdown Monitor is another example of AI proctoring.
Fourth, outside of commercial online proctoring products, some C/SOPs have implemented proctoring through web conferencing platforms such as Zoom (Zoom Video Communications, Inc., San Jose, CA) or Go to Meeting (LogMeIn Inc., Boston, MA). With this method, faculty or staff proctor students through the web conferencing software. A smaller proctor-to-student ratio may be required versus in-person proctoring to allow effective oversight of test takers. Students may take the exam on a secure online testing platform (eg, ExamSoft) and connect to the assigned proctor using a secondary device since the exam software may lock the students out of applications like Zoom .
Overall, when evaluating the various online proctoring programs, C/SOPs should consider the compatibility of the program with existing online testing software, available campus supported platforms, minimum computer system requirements, the need for an active internet connection during the exam, explicit and implicit cost of the system, and the degree of faculty and staff support needed or the level of support provided by the proctoring program. For example, some institutions may have an existing contract for a web conferencing platform and this option may be a lower-cost alternative than the purchase of online proctoring software, although it may be more resource intensive with regards to the number of C/SOP personnel needed for proctoring.

Developing Guidelines for Optimal Testing Behavior
After the adoption of an online testing and proctoring software, and prior to the administration of any remote exams, C/SOPs should first outline optimal testing behavior guidelines and expectations and distribute these to faculty, staff, and students. These guidelines may be developed by appropriate administrators, offices, or committees involved in assessment (eg, Assessment Dean, Assessment Committee, or Office of Curricular Affairs) and then approved by the faculty and administration. It is helpful for a C/SOP to employ the same testing procedures in their institution for all courses in order to promote testing consistency and reduce confusion. C/SOPs may have existing testing guidelines and procedures in place, which can serve as the foundation when adapting them for the online testing and proctoring setting. While online proctoring technologies may provide instructions to prepare students for optimal video capture during assessments, C/SOPs should consider supplementing and enhancing these guidelines. One important item to note is that since the exam taker is recorded, the new testing policies should give notice to the students that video capture will occur and can be used in cases of academic misconduct. This notice may be placed in the course syllabus, on the course management system, or provided to students in an email and C/SOPs may require students to sign the document acknowledging the notice.
Guidelines should include recommendations for the testing environment (well-lit room, computer device on a flat clean surface, testing surface clear of any notes or resources) and appropriate dress during the assessment. Guidelines should also inform students of prohibited items such as cell phones, headphones, hats, or smart watches, and provide guidance on minimizing distractions that may generate a flag by the proctoring system (eg, looking away frequently, talking, or leaving the room). C/SOPs need to address potential inequities in student access to an optimal remote testing environment. Some students may lack access to reliable internet, preventing test access or completion. C/SOPs may need to help students identify reliable WIFI access so they can complete exams. Some students may lack a suitable environment for remote testing (taking test on a chair, in bed, or in a car) or may have difficulty preventing interruptions (school-aged children who are also remote learners and need assistance, family members or roommates who may be sick or loud, animals/pets who are noisy/distracting). Students facing these challenges should discuss them with appropriate administrators or faculty (eg, Office of Assessment or Student Affairs) to explore solutions.
C/SOPs should also include guidance on who to contact for exam issues (eg, exam access, WIFI connection issues, or technical difficulties), exam absences, bathroom breaks, and emergencies and/or interruptions (eg, fire alarm, child or family interruption, or loss of power). A process to support students during these challenges is recommended such as establishing a "help room" via Zoom that is maintained by the course coordinator and/or assessment staff that students can access on a secondary device (eg, cell phone) that is not being used for administration of the exam. Another support option is to have a faculty or staff member on call during the exam through email, an office phone, or a cell phone (depending on the comfort level of the support person). For externally delivered and proctored exams, some proctoring software companies provide immediate technical support via phone or chat feature, while others provide an email or ticket-based support system. Local faculty and staff support should be sure to have the software contact information handy in order to quickly refer students if the C/SOP proctor is unable to resolve the issue. Additionally, C/SOPs should consider adjusting guidelines related to exam settings, such as extending test upload deadlines and test time limits to allow for variable or slow internet speeds in the remote testing environment.

A J P E
A c c e p t e d D r a f t Scratch paper is another item that should be addressed in the guidelines, because it may have been previously used during in-person exams to perform calculations, access formulas, organize personal thoughts/ideas, or alert faculty to concerns about a question(s) on the exam. Prior to COVID-19, proctors collected and reviewed scratch paper at the end of the exam and discarded or stored it in an effort to protect the integrity of the exam for future years. When testing remotely, the same level of security with scratch paper cannot be enforced. Two alternatives for formulas previously provided on scratch paper are 1) the formulas can be embedded into the test questions (students may select from a few options) or 2) some online testing programs allow for attachment of a PDF document that could contain a formula sheet. For C/SOPs who want to provide students the opportunity to calculate or write down thoughts, some online testing programs allow students to type into a notepad within the testing software. However, this may not be well-received by students who appreciate the "long-hand" nature of writing out calculation steps or organizing their thoughts during an exam. If scratch paper with formulas or instructions is permitted, C/SOPs should be aware that students may lack access to a home printer and outline alternatives. Guidelines for scratch paper should also instruct students to display the paper on camera at the start (a blank paper) and end of the exam (what students wrote) and also ask them to upload it into the course management tool dropbox or assignment folder for faculty inspection following completion of the exam, even if the scratch paper is blank. Alternatively, C/SOPs may allow students to use a dry erase board in lieu of scratch paper. In this case, C/SOPs should provide guidelines that outline acceptable white board/dry erase board features such as size, and they should give students advanced notice to purchase the white board, unless the C/SOP chooses to purchase it for them. If scratch paper was previously used as method to provide feedback and alert faculty to problematic exam questions, faculty may consider adding an open-ended question on the exam or offering a link to a survey form (eg, Qualtrics (SAP, Provo, UT)) that allows students to submit concerns about a test question(s).
One last consideration is that the new guidelines should include specific consequences for student nonadherence to the online testing procedures. For example, a student's first infraction may result in an email warning with corrective recommendations, followed by the submission of a professional concerns report by the faculty proctor, and subsequently progress to a grade penalty (eg, 10% reduction) on the exam for continued infractions. Training Faculty and Staff on the Testing and Proctoring Software When the guidelines are finalized, they should be communicated to all stakeholders. The guidelines should be shared with students through course syllabi and other modes, such as email and/or announcements/postings in course management system. Once the guidelines are communicated to all stakeholders, the roles of faculty, staff or assessment personnel should be clearly outlined prior to any administration of exams in order to reduce mistakes and confusion. The number of faculty or staff assigned to administer the exam, proctor, or review proctoring videos needs to be determined in advance based on class size and existing responsibilities. Then, faculty and staff who are assigned and will serve as proctors need to receive training on the testing/proctoring software and the new procedures for administering and proctoring exams. The training may be conducted by the C/SOP assessment or instructional design staff, the software company, or an academic technology department at the university, and should include a demonstration of the technology, a review of the exam recordings and reports generated, instructions for how to access them, and reaffirmation of the role of the proctor in upholding academic integrity.
One important piece of training proctors should receive if their C/SOP uses record and review software is how to review the exam video recordings and/or flagged reports for academic misconduct. The training should help proctors determine if there are false flags (eg, poor lighting) and if they need to further evaluate suspicious behaviors for possible breaches in academic integrity. This is the most time-consuming step with online proctoring, especially in C/SOPs with large class sizes, as it may take several minutes per student to review videos and evaluate potential flags. It is important the proctor training include examples of potential academic misconduct, such as students reading test questions and answers aloud, and looking at areas other than their screen (lap, wall, next to or above screen) as they may be looking at notes on the wall, floor, or desk. Training should also outline what parts of the recording proctors should review entirely versus spot check. Specifically, proctors should review the entire environment scan; the exam start (in case students forget to show something in the environment scan); and the last five minutes of the exam since students may be erasing a white board on camera or showing scratch paper. They should then be trained how to spot check the video/thumbnails throughout the recording at a faster speed (maximum 4x speed) to listen for audio recording and to look for breaches in academic integrity, since the flags may not identify all areas of misconduct. Proctor training should conclude with reviewing the process for reporting suspected academic misconduct, including consulting the C/SOP's faculty handbook for specific guidance. For C/SOPs that use ProctorU or another live third-party proctor, faculty and staff should have awareness of the system, but understand they are not involved in the exam delivery or proctoring and cannot assist students if they experience technical difficulties. Students also need to receive training with the new online proctoring software, such as conducting a thorough environment scan, announcing the location of their cell phone, removing smart watches, understanding timing for receiving any exam passwords or access codes prior to the start of the exam, providing a baseline photo to confirm student identity via webcam capture prior to future assessments (if applicable), showing personal calculators (if permitted), displaying and erasing white board or blank and completed scratch paper (if used). Students should also be trained about expectations for stating the honor code and attesting that they have not received outside help with the assessment (if applicable). After students review the new guidelines and receive training, they should be provided with a mock examination or quiz to familiarize themselves with the use of the software. A mock examination also allows faculty to correct any student behaviors that may be flagged as a breach of integrity by the software, including poor positioning of the webcam, insufficient lighting, excessive movement (ie, getting up/walking around), or reading questions aloud during the assessment. Overall, students should be reminded of the importance of adhering to academic integrity expectations and online testing policies and procedures during all assessments.

Final Recommendations for Test Bank Refreshing
Despite faculty and staff efforts to promote academic integrity during an exam, faculty should acknowledge that exam questions may have been compromised and commit to refreshing the test bank prior to the next administration of the test. Although C/SOPs may have existing guidelines on refreshing test banks (eg, rewrite 30% of the test questions each year), they should consider a possible increased percent of questions be refreshed for the next academic year. The specific percentage should be discussed and agreed upon by the faculty.

CONCLUSION
The rapid transition from traditional in-person proctored environments to remote testing has precipitated the need for C/SOPs to create revised guidelines and training to address new methods of test administration and proctoring. New guidelines and procedures should be intentionally designed and documented and clearly communicated to all stakeholders. In this new environment, proctors should reaffirm the importance of their role and their commitment to maintaining academic integrity in the testing environment. Students should confirm their ability to uphold the honor code, adhere to academic integrity expectations, and comply with remote testing policies and procedures. These efforts are vitally important as C/SOP testing is one of the most important gateways to the profession and proctors are the gatekeepers. Without robust guidelines and training, C/SOPs can inadvertently invite academic misconduct, permitting students and the institution to compromise the ethical standards of the profession.