{'fraction_non_alphanumeric': 0.02947950253339475, 'fraction_numerical': 0.013933671119299862, 'mean_word_length': 4.713815789473684, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ABC of Response Process Validation and Face Validity Index Calculation

Validity evidence can be supported by five sources that are content, response process, internal structure, relation to other variables, and consequences. Response process validity measures the thought processes of users of the tested inventory as they respond to the assessment tool. These are commonly evaluated in the form of clarity of instructions and language used in the assessment tool, as well as the comprehension of instruction after training or an observation session. Response process validity contributes to the overall validity of an assessment tool; therefore, it should be quantified systematically based on the evidence and best practice. This paper describes a systematic approach to quantify response process validity in the form of face validity index based on the evidence.


INTRODUCTION
In 1947, Mosier analysed the various definitions of face validity concept (1). Commonly, response process validity evidence is performed after content validity has been established (2,3) and response process validity is also known as the face validity that refers to the degree to which test respondents view the content of a test and its items as relevant to the context in which the test is being administered (4). Similarly, other researchers define face validity as the degree raters judge the items of an assessment instrument as appropriate to the targeted construct and assessment objectives (5,6). The raters of face validity include: (a) the person who actually takes the test; (b) the nonprofessional users who work with the results of the test; and (c) the general public (6). In other words, the people who are involved with the test taking should be asked to do the rating, in which they cannot be replaced by professional, experts or psychometricians (6). The raters' understanding and interpretation about the items will determine the accuracy of an assessment tool to measure the targeted construct. People with a similar background rate test face validity similarly, and they rate the face validity of different tests differently (6). Due to so much concern about the face validity concept, Cook and Beck (2006) have avoided using the face validity term, instead, the researchers use the response process evidence of validity as the term to reflect the thought processes of users of

Step 1: Preparing response process validation form
The first step of response process validation (also known as face validation) is to prepare the response process validation form to ensure that the panel of raters, who are the intended respondents, will have a clear expectation and understanding about the task. An example for the instruction and rating scale is provided in Figure 1. The rating scales of clarity and comprehension have been used for scoring individual items (12-15) ( Figure 2).

Step 2: Selecting a panel of raters
The selection of raters to review and critique an assessment tool (e.g., questionnaire) is usually based on the target user of the tool, for example, students, public, and teachers. Table 1 summarises the number of raters with its implication on the acceptable cut-off score of FVI based on previous studies (8 -11, 16-18).
It can be concurred that for response process validation, the minimum acceptable number of raters is 10; however, most studies had at least 30 raters. Considering the previous studies (Table 1) and the author's experience, the number of experts for content validation should not be less than 10 raters.

Step 3: Conducting response process validation
The response process validation can be conducted through face-to-face or online survey (Table 1). For the face-to-face survey, the researcher facilitates the response process validation process by holding a meeting with the raters followed by Step 4 and Step 5 (as elaborated further). For the online survey, an online response process validation form is sent to the raters and clear instructions are provided ( Figure 1) to facilitate the validation process. Based on the author's experience, the face-to-face approach is very efficient to increase the response rate, whereas the online survey is efficient in terms of cost and time.
the tested assessment as they respond to the tool (7,8) and it can be quantified by face validity index (FVI) (8-11). These are commonly evaluated in the form of clarity and comprehensibility of instructions and language used in the assessment tool by the raters (7,8). According to Cook and Beck (2006), validity evidence can be supported by content, response process, internal structure, relation to other variables, and consequence of an assessment tool (7). The clarity of instructions and language refers to whether there were ambiguities or multiple ways to interpret the items, whereas the comprehensibility of instructions and language refers to whether words and sentences of the constructed items can be understood easily by raters. It is important to establish response process validity to support the overall validity of an assessment tool such as questionnaires, especially for research purposes. The response process validity can be represented by FVI and several studies (8-11) have calculated it to support the validity of an assessment tool. Based on the evidence, this paper describes the best practice to perform response process validation and calculate the FVI of an assessment tool.

Response Process Validation Procedure
The following are the six steps of response process validation: 1. Preparing response process validation form.
2. Selecting a panel of raters.
4. Reviewing items for clarity and comprehension.
5. Providing score for each item based on the clarity and comprehensibility rating scale.
Each step will be elaborated in the subsequent sections.  The definition and formula of the FVI indices are summarised in Table 2.
Prior to the calculation of FVI, the clarity and comprehension rating must be recoded as 1 (the scale of 3 or 4) or 0 (the scale of 1 or 2) as shown in Table 3. To illustrate the calculation of different FVI indices, the clarity and comprehension ratings on item scale by 10 raters are provided in Table 3.
To illustrate the calculation for the FVI indices (please refer to Table 2), the following are examples of calculation based on the data provided in Table 3: i. Raters in agreement: just sum up the relevant rating provided by all raters for each item, for example, the raters in agreement for Q2 (1 + 0 + 1 +1 + 1 + 1 + 1+ 1 + 1 + 1) = 9.
ii. Universal agreement: score '1' is assigned to the item that achieved 100% raters in agreement, for example, Q1 obtained 1 because all the raters provided rating of 1, while Q2 obtained 0 because not all raters provided rating of 1.
iii. I-FVI: the raters in agreement divided by the number of raters, for example I-FVI of Q2 is 9 divided by 10 raters that is equal to 0.9.

Step 4: Reviewing items for clarity and comprehension
In the response process validation form, the domain and its items are provided to the raters as shown in Figure 2. The raters are requested to review all items before providing score for each item. The raters are encouraged to provide verbal comment or written comment to improve the clarity and comprehension of the items. All comments are taken into consideration to refine items.

Step 5: Providing score for each item based on the clarity and comprehensibility rating scale
Upon completion of reviewing all items, the raters are requested to provide score for each item independently based on the clarity and comprehension scale (Figures 1 and  2). The raters are required to submit their responses to the researcher once they have provided a score for all the items.

Step 6: Calculating FVI
There are two forms of FVI, i.e. FVI for item (I-FVI) and FVI for scale (S-FVI). Two methods for calculating S-FVI, in which the average of the I-FVI scores for all the items on the scale (S-FVI/Ave) and the proportion of items on the scale that achieve a clarity and comprehension scale of 3 or 4 by all raters (S-FVI/UA) ( Table 2).  The proportion of items on the scale that achieve a clarity and comprehension scale of 3 or 4 by all raters. Universal agreement (UA) score is given as 1 when the item achieved 100% raters in agreement, otherwise the UA score is given as 0.

CONCLUSION
Response process validity is vital to ensure the overall validity of an assessment, therefore, a systematic approach for validating response process should be done based on the best evidence. This paper has provided a systematic and evidence-based approach to conduct a proper response process validation through face validity index.