{'fraction_non_alphanumeric': 0.02360589131538852, 'fraction_numerical': 0.011132554596241746, 'mean_word_length': 5.28764848639673, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Insights from Washington State’s COVID-19 Response: A Mixed-Methods Evaluation of WIC Remote Services and Expanded Food Options Using the RE-AIM Framework

Background In response to the COVID-19 pandemic, Washington State’s Supplemental Nutrition Program for Women, Infants, and Children (WA WIC) adopted federal waivers to transition to remote service delivery for certification and education appointments. WA WIC also expanded the approved food list without using federal waivers, adding more than 600 new items to offset challenges participants experienced accessing foods in stores. Objective This study aimed to assess the reach and effectiveness of the programmatic changes instituted by WA WIC during the COVID-19 pandemic; the processes, facilitators, and challenges involved in their implementation; and considerations for their continuation in the future. Design A mixed-methods design, guided by the RE-AIM framework, including virtual, semi-structured focus groups and interviews with WA WIC staff and participants, and quantitative programmatic data from WIC agencies across the state. Participants/setting This study included data from 52 state and local WIC staff and 40 WIC participants across the state of Washington and from various WA WIC programmatic records (2017-2021). The research team collected data and conducted analyses between January 2021 and August 2021. Analysis An inductive thematic analysis approach with Dedoose software was used to code qualitative data, generate themes, and interpret qualitative data. Descriptive statistics were calculated for quantitative programmatic data, including total participant count, percent increase and decrease in participation, percent of food benefits redeemed monthly, and appointment completion rates. Results All WA WIC participants (n = 125,279 in May 2020) experienced the programmatic changes. Participation increased by 2% from March to December 2020 after WA WIC adopted programmatic changes in response to the COVID-19 pandemic. Certification and nutrition education completion rates increased by 5% and 18% in a comparison of June 2019 with June 2020. Food benefit redemption also increased immediately after the food list was expanded in April 2020. Staff and participants were highly satisfied with remote service delivery, predominantly via the phone, and participants appreciated the expanded food options. Staff and participants want a remote service option to continue and suggested various changes to improve service quality. Conclusions Participation in WIC and appointment completion rates increased after WA WIC implemented service changes in response to the COVID-19 pandemic. Staff and participants were highly satisfied with remote services, and both desire a continued hybrid model of remote and in-person WIC appointments. Some of the suggested changes to WIC, especially the continuation of remote services, would require federal policy change, and others could be implemented under existing federal regulations.

participants are Black, Indigenous, and People of Color 2 and the percent of eligible families participating in WIC is higher in rural than urban communities. 3 WIC participation is associated with healthier diets and improved infant morbidity and mortality rates. 4,5 Despite program benefits, only approximately one-half of eligible individuals enroll in WIC. 6 The barriers to using WIC services are well-documented and include required physical presence at WIC offices for appointments, limited WIC-approved foods, and negative shopping experiences. [7][8][9][10] The COVID-19 pandemic initially exacerbated these barriers, as people were encouraged or required to stay home to limit coronavirus transmission, and grocery stores experienced shortages of WIC-approved foods. 1,11 The outbreak also created economic conditions that increased rates and severity of food insecurity among vulnerable families. 12 In March 2020, Washington State WIC (WA WIC) began offering WIC certifications and other appointments remotely (together referred to as "remote services"), using waivers provided through the Families First Coronavirus Response Act. 13 WA WIC also expanded the list of allowable foods by more than 600 items in April 2020; WA WIC intentionally expanded the list without the use of federal waivers to ensure the expanded access to allowable foods would continue beyond the COVID-19 pandemic. These changes allowed eligible households to participate in WIC when clinics were closed to the public and improved access to WIC foods during retail shortages.
COVID-19 pandemicerelated changes to WIC services provide an opportunity to gain insight into improving WIC services over the long-term. This study aimed to assess the reach and effectiveness of the programmatic changes instituted by WA WIC during the COVID-19 pandemic; the processes, facilitators, and challenges involved in their implementation; and considerations for their continuation in the future.

METHODS
This mixed-methods study used semi-structured interviews with WIC participants, focus groups and key informant interviews with WIC staff, and participant-related programmatic data to evaluate WA WIC's transition to remote services and expanded food options during the COVID-19 pandemic. Study protocols were determined exempt from human subjectsreview by the University of Washington Human Subjects Division and Washington State Institutional Review Board.

Framework
The RE-AIM framework (see Figure 1) guided the research aims and design, providing a structure for evaluating the programmatic changes' reach, effectiveness, adoption, implementation, and maintenance. 14 Using both qualitative and quantitative data within the RE-AIM dimensions generated a thorough examination of programmatic changes 15 because qualitative data provided context for quantitative trends. 15 Sample and Recruitment WIC Staff. Convenience sampling was used to recruit state and local WIC staff to participate in focus groups and key informant interviews. Existing contacts within WA WIC identified state-level staff involved with programmatic changes. To recruit local WIC staff, the research team selected 10 of the 57 local WIC agencies from which to recruit staff based on rurality, caseload, region, and service status (eg, if offering curbside services as of May 2020). The lead staff at the state WIC office e-mailed study information to coordinators at each local agency and asked them to share study information with other WIC staff at their agency. The final sample included 52 WIC staff, 10 who worked at the state level and 42 who worked at the local level. In addition to interest in sampling WIC staff from the state agency and a diverse array of local WIC agencies as detailed above, recruitment intentionally focused on staff roles in the program; specifically, the team sought perspectives of staff involved in program coordination, certification, nutrition education, or breastfeeding education at the local level and staff involved in developing the food list and supporting the transition to remote services at the state level. Recruitment continued until invitations of at least 1 staff member in each role were sought from each of the identified agencies. Race and ethnicity, sex or gender, and age information for WIC staff participants were not collected.
WIC Participants. WIC participants were recruited through a banner placed on the WICShopper app in English and Spanish for 1 day, as the WICShopper app was used by approximately 90% of Washington WIC households at the time of recruitment. 16 In that time, 147 WIC participants volunteered by clicking on the banner, which brought them to a webpage with the study procedures and a link to provide contact information and self-report demographic and other key characteristics, including race and ethnicity, length of time on WIC, breastfeeding status, how many (if any) infants and children enrolled in WIC, and the WIC agency they visit. No data on sex and gender or age of WIC participants were collected.
Seventy-two survey respondents were contacted using a maximum variation approach to aim for diverse perspectives of the WA WIC participant population regarding race, ethnicity, rurality, and WIC participant enrollment category (whether they were enrolled, their infant was enrolled, or their child/children were enrolled). The study team made a particular effort to contact participant volunteers identifying as American Indian and Black because these racial groups experience the largest nutrition 17 and infant and maternal health disparities in the state. 18 In addition, the aim was to recruit 10 WIC participants whose primary language was Spanish to ensure that the perspectives of participants who speak a language other than English were included. Spanish is the most commonly spoken language after English in Washington State, and an estimated 6.5% of households receiving cash, food, and medical benefits in the state consider Spanish their primary language. 19 Participant recruitment continued until the goal of 40 total interviewees, 10 of whom spoke Spanish, was reached; the goal was based on prior similar work that indicated 40 interviews would be sufficient to reach data saturation. 20 Of the 72 respondents contacted, 40 completed interviews, 6 declined participation, 11 were scheduled but did not complete interviews, and 15 were unreachable after up to 4 contact attempts.
Data Collection Programmatic Data. Nonidentifiable programmatic data were provided by WA WIC to examine reach and effectiveness through temporal trends in participation and food benefit redemption. WA WIC also shared reports from local WIC agencies from May and August 2020 with data on WIC service status (eg, number of staff and number of sites offering curbside or face-to-face services), supports needed, and challenges experienced. These data were explored so emergent quantitative trends could inform questions during qualitative data collection and to help triangulate focus group and interview findings.
Staff Focus Groups and Interviews. Ten focus groups were organized to include staff members with similar roles within WIC as able. One focus group included state staff managing the transition to remote services (n ¼ 5), 1 included state staff managing the food list expansion (n ¼ 5), 2 included local program coordinators (n ¼ 7, n ¼ 6), 2 included breastfeeding support staff (n ¼ 3, n ¼ 2), 1 included nutrition educators (n ¼ 5), 1 included certification staff (n ¼ 4), and 2 included staff from a variety of positions (n ¼ 7, n ¼ 6). In addition, 2 local staff participants (program coordinator and nutrition educator) were not able to participate in focus groups due to scheduling constraints, so key informant interviews were conducted with these individuals. The study team developed the semi-structured focus group guide ( Figure 2; available at www.jandonline.org) using the study aims and the RE-AIM framework; this guide was also used for the key informant interviews. The questions in the focus group guide asked staff about processes, facilitators, and challenges that accompanied the switch to remote services and the expanded food list. Staff were also asked about the feasibility and support needed to continue aspects of remote services. Four team members engaged in an iterative process of independently coding a small subset of the transcripts and then comparing code applications to refine code definitions. Coding inter-rater reliability between 2 pairs of coders was assessed in Dedoose; on reaching k-statistics signifying "good" inter-rater reliability (ie, .79 or greater), team members coded independently. Data saturation, meaning no new themes emerged from the data, 24 occurred before all transcripts were coded; however, the remaining transcripts were coded because the staff and participant samples intentionally captured a diverse set of experiences with WIC. After coding, team members summarized coded text.
Reflexivity, an evaluation of researcher's experiences and identities that could influence the research is important in qualitative research. 25 There were ways in which the study team's experiences and identities differed from the study participants, which had the ability to influence study design, data collection, and interpretation. To account for these differences, the team debriefed regularly throughout the research process; wrote reflections after focus groups and interviews; consulted with co-researchers to ensure precise analysis; and discussed findings with WA WIC staff to cross-check interpretation, a process known as "memberchecking." 26

Sample Characteristics
Characteristics collected for WIC staff and WIC participants are detailed in the Table. The majority of WIC participants interviewed for this study were enrolled with 1 or more children and had experience with WIC before the beginning of the COVID-19 pandemic. The majority of WIC staff who participated in interviews or focus groups were employed at the local level.

Reach
Based on data reported by agencies and focus group respondents, all WIC participants who had an appointment or were certified during the COVID-19 pandemic experienced some aspect of remote services. Because the food list expansion was implemented statewide, all WIC participants also experienced the additions to the allowable food list. By May 2020, the WIC participant population reached with these changes included 26,891 women, 26,378 infants, and 72,010 children in Washington State; the population was predominantly White (70%), followed by multi-race (10%), Black or African American (8%), American Indian or Alaskan Native (5%), Asian (5%), and Native Hawaiian or Pacific Islander (2%); 42% of WIC participants identified as Hispanic.
Effectiveness WIC Participation. As seen in Figure 4, participation had been declining steadily since 2017. Participation began increasing just before the start of the COVID-19 pandemic and continued to increase after the programmatic changes were made to WIC in March 2020. There was a 5.9% increase in participation from December 2019 to December 2020, and participation increased by 2% from March 2020 (when the COVID-19 pandemic began) to December 2020.

Appointment Completions
Local and state staff described increased appointment completion rates after implementing the programmatic changes, hypothesizing that the increase related to the convenience of remote services. As described by 1 WIC staff, "Our show rates are much higher because there isn't the barrier of gathering all of your kids and your things and either driving or taking public transportation to get to WIC" (staff 02). Programmatic data from sample months before and during the implementation period corroborated this statement; nutrition education completion rates increased from 78% in June 2019 to 96% in June 2020, and the certification completion rate increased from 72% in June 2019 to 77% in June 2020.

Redemption of Food Benefits
Staff reported that the expansion of the allowable foods helped increase access to approved foods during COVID-19 pandemicerelated food shortages. Participant interviewees echoed this statement. One participant shared that "at the beginning, it was hard to get milk, cheese, and cereal, but then WIC updated the list of available foods and now I am able to get them with no problem" (participant 30S).  Categories are not mutually exclusive. c 2020 Washington state race and ethnicity statistics: 78% were White, 13% were Hispanic, 10% were Asian, 5% were 2 or more races, 4% were Black, 2% were American Indian and Alaskan Native, and 1% were Native Hawaiian or other Pacific Islander. 27 Food benefit redemption data throughout 2020 ( Figure 5) demonstrated that the average percent of all food benefits redeemed by WIC participants declined at the start of the COVID-19 pandemic from March to April, then increased almost back to pre-COVID-19 pandemic levels right after the food list was expanded. Redemption rates then dipped once more before returning to rates near those experienced in early 2020.

Adoption
State staff described 3 federal waivers that made remote services possible during the COVID-19 pandemic, listed in Figure 6. [30][31][32] These waivers allowed WIC services to continue while offices were closed to the public and reduced the administrative burden on clinics that had staff pulled away to COVID-19 pandemic response. In addition, WA WIC had transitioned to an electronic benefit transfer (EBT)ebased system for benefit issuance throughout 2019 and had just started loading benefits remotely in the beginning of March 2020; WA WIC was able to continue remote benefit issuance throughout the transition to remote appointments. The other major programmatic change was a large-scale expansion of the approved WIC food list. The food list is updated annually, but typically 40 to 60 items are added; in response  to COVID-19 pandemicerelated food shortages and challenges finding WIC-allowed foods in the store, WA WIC staff approved approximately 600 new foods. WA WIC staff explained that they chose not to use the waivers related to food items so that added foods would not be revoked when the waivers expire.

Implementation
Remote Services Implementation. Phone appointments were mentioned by all staff as the most common way of reaching clients remotely. Although some local staff had attended state or agency-led training on using video conferencing, nearly all local staff said that video appointments had not yet been offered to WIC participants. Perceived challenges to offering video appointments included hesitancy from staff related to embarrassment interacting on camera, a lack of training, and limited access to video equipment. To help maintain contact with WIC participants, local staff, especially breastfeeding support staff, reported texting participants more frequently between appointments.
To supplement remote services, many local staff reported mailing handouts to participants either before some appointments. so participants could follow along with the content while on the call, or after appointments when participants requested additional information. Staff also discussed implementing pick-up services for breast pumps and educational materials. A small subset of staff mentioned putting infant scales outside of homes so parents could measure their child if growth was concerning.

Supports and Facilitators of the Implementation of Remote Services
Transitioning to remote services required numerous supports from federal, state, and local agency staff. A federal waiver to defer measurements combined with state-level policy to defer submitting proofs and signatures on documents improved the flexibility of the certifications. As a result, staff reported that appointment efficiency increased; one said, "when you're doing everything over the phone things are just a bit quicker. You're talking to them, but you can kind of write your notes simultaneously . . . so that saves a lot of time" (staff 28). State staff also initially provided biweekly webinars for agency staff to share information and assess needs related to virtual appointments; these webinars tapered to monthly by December 2020. At the local level, common requests from staff included cell phones, laptops with cameras, and attachable webcams for desktops.

Staff and Participant Satisfaction with Remote Services
Both staff and participants expressed high satisfaction with remote service delivery. Figure 7 summarizes a subset of the most common advantages and disadvantages of remote services from staff and participant perspectives and presents a selection of illustrative quotes. The advantages of remote services were mentioned more often and outweighed the disadvantages presented.
Both local and state staff appreciated remote services as a way to reduce longstanding participation barriers. Phone calls worked well for most appointment types, according to most local staff. The appointment types that were mentioned as more difficult remotely included enrollment appointments (especially for first-time WIC participants) and some breastfeeding appointments addressing latch and positioning support. For example, one staff member shared that "it's been very challenging offering these breastfeeding support services remotely. It's hard to . . . convey the different positions or movements just, you know, over the phone" (staff 29). Although enrollments and breastfeeding support appointments were brought up by some staff as challenging, these opinions were not unanimous. For example, one breastfeeding educator said "I feel like even for breastfeeding they're more relaxed when they are talking, because . . . we have the luxury to choose any time at their convenience and they can call me" (staff 39); in this way, remote services helped the participant feel comfortable and engage with breastfeeding staff when they needed it. The most reported element of remote services that detracted from staff satisfaction was the inability to consistently obtain height, weight, and hemoglobin measurements from participants. However, these challenges were mentioned less often than the success of increasing convenience for participants with remote services.
Participants viewed phone appointments as easy to complete, convenient, and sufficient for both sharing and receiving information. Participants reported they could connect, hear, and complete the appointments over the phone and felt as, or more, comfortable receiving services on the phone compared with in-person. Approximately one-half of the participants interviewed indicated they occasionally took appointments away from home, including at work and while running errands. One participant noted that phone

Adaptation Explanation
Physical Presence Waiver 30 Participants are allowed to certify and recertify for WA WIC services remotely (via phone or video appointment) and defer height, weight, and hemoglobin measurements.
Remote Benefit Issuance Waiver 31 Staff are allowed to issue a participant's benefits remotely.
Separation of Duties Waiver 32 A single employee is allowed to evaluate a participant's eligibility for all certification criteria, and issue food benefits. Figure 6. Key federal waivers that the Washington State Special Supplemental Program for Women, Infants, and Children (WA WIC) adopted during the COVID-19 pandemic.

Theme
Quote WA WIC staff focus groups

Advantages of remote services
Increased convenience for working parents, large families, those with transportation barriers "I think the majority of our clients like it, because it's quicker and they don't have to come in."-staff 28 Decreased barriers to participation such as: Having to take time off work for appointments Having to bring multiple children to office for one child's appointment Having to secure access to reliable transportation Having to travel a long way to reach the WIC office in remote/rural areas "All of the clients that I've talked to really appreciate that they don't have to get in the car, load all the kids up, and go to the clinic."-staff 33 Increased ease of sharing resources virtually with participants "They find [links] very helpful because after they had the education with the peer counselors or the breastfeeding staff the mom is able to go back and review that video again for anything that she had missed."-staff 02 Increased amount of communication between staff and participants (ie, between appointments and for questions) "People used to save up their little questions for the end of an appointment and go 'Oh, there was something else I wanted to ask' that they couldn't remember and now they're texting me more than they used to with just little questions intermittently."-staff 29 Increased richness in conversations over the phone "I think in part of the certification they have richer discussions. Participants are willing to share a little bit more over the phone versus face to face. -staff 02 Increased ability to adapt services to address participants needs in the moment "I was able to save that breastfeeding relationship by being able to meet that mom in that moment when she needed help right then with feeding. So that was really great for remote services."-staff 29

Disadvantages of remote services
Difficulty obtaining consistent height, weight, and hemoglobin measurements for infants and children "It just feels like you're flying without wings on not having accurate measurements and timely measurements."-staff 25 Missing in-person relationships and interactions with participants "Seeing clients in person you get to see how they're feeling in their expressions when they're talking to us and you don't pick that up remotely when you're just talking on the phone or even in a text and so we tend to miss out on a lot of information that we could pick up if we were with them in person."-staff 29 (continued on next page)  Noticing at times participants seem distracted when taking appointments over the phone "Being on the phone it's kind of hard to try to get like the client's needs because they're just kind of wanting to either just hurry up with the appointment because their kids are loud or they can't hear us."-staff 25 Increased difficulty scheduling and dividing the caseload fairly for select agencies "Sometimes a certifier will get four of those five appointments at the same time versus, you know, somebody else who will get a nice spread-out schedule.

Theme Quote
Feeling safer during the pandemic "It's actually better, it's safer for people to be at home and especially because going to the WIC office, the office is really small, so they wouldn't be able to keep the social distancing."-participant 30S Appointments seeming more focused and personalized "Most of the time, it's the same lady that calls me. So I mean . . . it kind of gets a little personal because we kind of get to know each other over the phone."participant 87E

Disadvantages of remote services
Missing information on height, weight, and iron for infants/children "Before, when we went in in person, we would know how the baby what was thriving and his weight and right now on the phone we wouldn't be able to know."-participant 25S Missing interactions between staff, other parents, and children "She misses the 'Hello, how are you? How's your daughter?' Those interactions that she enjoyed while being in person."-interpreter for participant 9S Feeling some appointments seemed rushed or brief "On the phone, it is kind of just quick like, 'Well, we're going to add this, and this is going to happen like this, and we'll see you talk to you in like 3 months or something.' So, I don't know if we miss anything, but it's just really different."-participant 70E Feeling that some appointments might not allow enough space and time to observe and interact with children, especially for new parents "I feel like they give more information when the child is in the room, and then they can see, 'Oh your child can do this. Your child can do that.' So, they can provide more information about specifically my child."participant 36E Figure 7. (continued) Perceived advantages and disadvantages of conducting appointments remotely during the COVID-19 pandemic as reported by Washington State Special Supplemental Program for Women, Infants, and Children (WA WIC) staff focus group participants and WA WIC interview participants. This figure includes a subset of the most common advantages and disadvantages of remote services noted by both staff and participants; it is not an exhaustive list. Key quotes have been selected to represent the theme presented. Participants with S after their ID number were interviewed in Spanish, participants with an E after their ID number were interviewed in English.
appointments "just made it really easy. It was convenient that as I'm doing my normal everyday things that I can still do my appointment" (participant 36E). None of the participants interviewed had conducted a video appointment and thus did not discuss satisfaction regarding video appointments.

Satisfaction with the Expanded Food List
When asked about their experience with the expanded food list, approximately one-half of participants were aware of the additional foods. Those participants were happy with the expansion and especially appreciated the increased variety of kid-friendly foods like string cheese, yogurt, cereal, milk, and juice. One participant said "Yes, I like them, because we have more and more of an option of how to get cheese and yogurt because that's what [my son] likes" (participant 25S). The WICShopper app was mentioned by participants as helpful when planning for and actively shopping, especially alongside the expanded food list. The scanning feature made it easier for participants to confirm allowable foods. Although the WICShopper app was useful for a majority of the participants interviewed, some still noted difficulties around determining which foods were added to the food list and having items deemed not allowable at the register.

Maintenance
Interest in Maintaining Remote Services. Both staff and participants expressed hope that remote services would continue beyond the COVID-19 pandemic. Almost all state and local staff advocated for a hybrid model, including both remote and in-person appointments. They recognized the increased convenience of remote appointments, and also noted that an in-person option could be useful for certain services and for parents needing to share sensitive information, including disclosures of domestic violence. One staff member shared that "moms sometimes like to come and share what's happening in their homes and . . . the kind of abuse they were facing or things like that . . . so a hybrid fashion, I think, is a better idea" (staff 12). Although an inperson option is desired, many staff anticipated a decline in caseload if reverting back to all in-person appointments. WIC participants generally said they would be fine with returning to in-person services, although one-third of interviewees said they would want to know returning was safe in terms of COVID-19 pandemic protocols and another one-third said they would return if it was required.
Nearly all participants interviewed hoped that remote services would continue after the COVID-19 pandemic. The most common reason given was the improved convenience of the phone appointments. Most participants supported an option for in-person appointments. For example, one participant suggested that "as far as reloading the card, then yeah, phones fine. But if it has anything to do with checkups or with growth information, then I would like those to be inperson" (participant 25S). Nearly one-quarter of WIC participants proposed that in-person appointments be provided for, or limited to, appointments with specific purposes like certifying, breastfeeding support, and/or discussing sensitive topics. Four participants-2 English-speaking participants and 2 Spanish-speaking participants-said that they would prefer all in-person services. Participants indicated that inperson appointments would allow more opportunities for socializing and observation of breastfeeding skills or child health. On this latter point, a participant said "To be honest, after COVID, [I] would like to go to the clinic because then they could assess the children. They could actually physically see the children and be able to evaluate them" (participant 57S).
Staff Suggestions for Improving Remote Services. The staff proposed many ideas for creating a hybrid service delivery model, including drop-in days for height and weight measurements and conducting the rest of the certification appointment remotely, and/or conducting biannual certifications in-person and all other appointments remotely. Staff also had ideas for capturing measurements without requiring physical presence, including improved data sharing among health providers and programs that capture measurements and technology for at-home measuring. Staff expressed strongly that consistent and accurate measurements are essential to assessing nutritional risk and, further, that WIC is unique in its ability to collect and use this information, especially among families that do not have insurance.
Supports Needed to Maintain Remote Services. Staff described federal-and state-level supports that would be necessary to continue providing remote services. All staff want to see flexibility in federal physical presence regulations. State staff also recognized the importance of further training on video conferencing, describing a significant learning curve for both staff and participants. Approximately one-quarter of staff expressed interest in conducting more video appointments, and participants had mixed and speculative views on video appointments because none had yet completed a video appointment. From the participant perspective, perceived advantages included seeing demonstrations, having staff interact with children, and feeling more comfortable seeing the staff member. The most common perceived disadvantage of video calls described by participants was that they would not be able to take calls "hands free," which was an important convenience of phone appointments. Staff also suggested developing video orientations for using the WICShopper app and WIC EBT card translated into numerous languages to help supplement the education they give to new participants during their first appointment.

Interest in and Improvements Recommended for a Maintained Expanded Food List
Participants made clear that they would like the list of allowable food to remain expanded, with even more foods added. Participants said they would appreciate additional approved food brands, more organic options, and more flexibility in allowable food package sizes. Some WA WIC participants said that although the number of approved brands of certain foods increased, the allowable sizes of food items remained a constraint because food packaging sizes in stores differed. Participants also suggested improvements to the WICShopper app, including improved search features, increased scanning reliability, and additional meal planning features. Both staff and participants mentioned that participants should be able to order WIC groceries online and pick up curbside.

DISCUSSION
Public health nutrition experts have called for evaluation of WIC service changes implemented in response to the COVID-19 pandemic to strengthen WIC and inform the upcoming Child Nutrition Reauthorization process. 33 This study was designed to generate evidence-based suggestions for implementing, maintaining, and improving the changes adopted by WA WIC for future practice. Overall, these results illustrate that appointment show rates improved after the programmatic changes were implemented and that WIC participants and staff were highly satisfied. Staff and participants communicated strongly that they want to see both changes maintained in some form. The impact of these changes on WIC participation trends and food benefit redemption is less clear, given the COVID-19 pandemic context influencing WIC eligibility and food shopping experiences; increased unemployment, 34 increased food insecurity, 34 and decreased access to other food-related safety net programs, 35 make it difficult to determine the extent to which increases in participation and food benefit redemption are attributable to WIC programmatic changes as opposed to increased need. 1 However, the changes demonstrate promise for optimizing WIC services beyond the COVID-19 pandemic.
This study aligns with other research on WIC participants' experiences with remote services during the COVID-19 pandemic and delves deeper into perceived advantages of remote appointments. Participants in Tennessee and now Washington State have indicated that phone appointments have been more convenient than in-person appointments. 36 Similar to participants in California, WA WIC participants felt comfortable with phone appointments and sharing documentation remotely. 37 Uniquely, participants in this study highlighted specific advantages of the remote appointments that address longstanding barriers to WIC; the most notable benefit was not having to secure childcare for appointments or bring multiple children to appointments. Past research has demonstrated that bringing children to appointments is one of the most common barriers to recertifying. 7 Other advantages mentioned by participants in this study that address barriers to using WIC services 7 include saving on travel time and costs, taking appointments on breaks from work, and feeling like appointments were more personalized on the phone. Similar to findings from other states, 37,38 WA WIC participants were highly satisfied with remote services and explicitly suggested their continuation. [36][37][38] Findings from this research also showcase satisfaction with the expanded food options during the COVID-19 pandemic, and suggest that the expanded food list likely offset a further decline in food benefit redemption rate due to COVID-19 pandemicerelated food shortages. 11 Food benefit redemption increased from April to May 2020, after the food list was expanded; while it did decrease again from May to June and July, this could have been related to the increase in Supplemental Nutrition Assistance Program benefits 39 and the COVID-19 pandemic EBT disbursements that began in late June and early July in Washington, 40 which likely impacted the purchasing patterns of many WIC participants. This study also provides unique insight into participants' opinions on expanded foods that were most appreciated during the COVID-19 pandemic, including string cheese, yogurt, milk, cereal, and juice. WA WIC expanded these categories without using waivers, so these additional items would not be revoked when waivers expire. To help determine which new items in the store were WIC-allowed, both California and WA WIC participants found the WICShopper app particularly helpful. 41 Despite the expanded food list and WICShopper app, participants from the current study noted that package size constraints were magnified during the COVID-19 pandemic. Package size availability has been a longstanding challenge reported by WIC participants, 7 and this current study indicated it has continued to impede WIC benefit redemption even within the context of expanded food options. Current study findings add further support for policy change regarding package sizes of WIC approved foods. 42 This is the first study that presents the WIC staff perspective on how services changed during the COVID-19 pandemic and related suggestions for future practice. Staff perspectives were similar to WIC participant perspectives when describing the ease and convenience of remote services, aspects of remote services that decrease participation barriers, and the helpfulness of the WICShopper app. The main drawbacks to remote services outlined by staff also aligned with the participants' perspectives; they wanted a way to capture consistent measurements remotely and they missed interacting with WIC participants in the clinic. Although there were similarities between staff and participant perspectives, they also diverged in several areas. First, some staff reported participants occasionally seemed distracted while taking phone appointments away from home. Participants did discuss taking appointments away from home, but within the context of convenience, highlighting the ability to multitask or maximize breaks from work as a benefit of remote services. Second, staff anticipated attrition once in-person services were re-established. Participants did report wanting to continue receiving WIC services remotely after the COVID-19 pandemic, but none said they would leave the program if in-person services were required. Staff and participants had similar suggestions for programmatic changes in the future, including transitioning to a hybrid model and including online ordering and pick-up options. Staff perspectives should be incorporated into future research, when possible, as it is valuable to know where WIC staff and participants' priorities align in order to make recommendations for changes.
Study findings add to the evidence supporting continued access to remote WIC services in addition to in-person services. 1,36,43 However, maintaining an option for remote certifications beyond the COVID-19 pandemic would require federal policy changes. For participants to be able to choose to certify and recertify remotely, either by video or phone, the federal physical presence requirement would need to be amended to allow either hybrid or full-remote options in addition to in-person. Some of the rationale for the physical presence requirement are to observe WIC participants, 44 identify health concerns, 44 personalize service delivery, 44 and protect against fraud. 45 Video appointments could be a substitute for physical presence because they allow for participant observation and providers can successfully identify health concerns, as seen with the increasing use of videoconferencing in telemedicine. 46 Furthermore, some participants in this current study felt that remote appointments were more personalized compared with in-person appointments and would like this option to continue in addition to in-person services.
To facilitate the continued success of remote appointments, additional means of collecting height, weight, and hemoglobin measurements should be piloted and evaluated; one option to research is drop-in hours for collecting measurements. Staff also suggested implementing data-sharing protocols between WIC and health care providers to minimize the duplication of services among participants whose measurements are being captured elsewhere. In addition, training and technological resources to maximize remote service quality in the future should be strengthened. 1,11,47 WA WIC staff advocated for additional video trainings for both staff and participants to feel comfortable providing and receiving services via video, as well as prerecorded video orientations in multiple languages about shopping with the WIC EBT card and WICShopper app. To improve the WIC shopping experience and food security, state programs should expand their approved food lists as much as possible within the current federal regulations. However, federal policy changes will be needed to address food package size discrepancies. Moreover, any change to WIC service delivery should be tracked and evaluated for both successes and unintended consequences.
There are multiple strengths of the design, methods, and analysis of this study. Key strengths include the mixedmethods design; participation of a diverse participant sample, including staff perspectives; the iterative approach to coding qualitative data; and the member-checking process with WA WIC staff. Integration of programmatic data with qualitative data allowed for contextualization and explanation of quantitative trends 48 ; use of both quantitative and qualitative data within the RE-AIM framework has been recommended and both were employed in this study for a rigorous exploration of the changes. 14 Furthermore, having multiple coders code a subset of the transcripts, discuss discrepancies, and collaborate on themes enhanced the data triangulation process 49,50 and confirmability of these results. 26 The participant sample also captured a diversity of perspectives about the programmatic changes. 36 Finally, in line with principles of rigor in qualitative research, the research team shared a preliminary synthesis of the results with WA WIC staff participants and discussed whether the findings resonated with their experiences. 26 There are also limitations to this study. First, the study focused on one state's WIC program, limiting the national generalization of the results. Second, interviews were only conducted in English and Spanish due to cost constraints; adding more languages could have increased the WIC participant sample diversity. Third, some trends, such as increases in participation, may have already been under way before COVID-19 pandemic shutdown orders; the team did not collect information that would elucidate whether these trends were due to growing awareness of the pandemic in late 2019/ early 2020 or to simultaneous programmatic changes, such as changes in electronic enrollment systems. Fourth, the team engaged in member-checking with WA WIC staff; however, there was not time to conduct a member-checking process with the participant sample. 50 Fifth, demographic data on race and ethnicity, gender, or age of WIC staff participants were not collected and WIC staff from diverse racial and ethnic groups were not purposively sampled. The interpretation of study findings may have been enhanced in the context of WIC staff participant race and ethnicity, gender, or age. Sixth, and finally, the WIC participant sample was recruited through the WIC-Shopper app, which may have generated a sample that would be more comfortable with the technology and experience of remote service delivery; however, nearly all WIC participants in Washington State use the WICShopper app and all have experienced remote services, so the potential for a biased sample is low.

CONCLUSIONS
This study explored the reach, effectiveness, adoption, implementation, and maintenance of remote services and the expanded food list executed by the WA WIC program in response to the COVID-19 pandemic. After these changes were implemented, participation, appointment show rates, and food redemption increased. Both staff and participants were highly satisfied with remote services and envision a combination of remote and in-person services offered in the future. Participants appreciated the increased variety of kidfriendly food options. Some of the suggested WIC changes, especially the continuation of remote certifications, would require federal policy change, and others, including increased training for video appointments and additions to the app and food list, would not.

1) Please briefly introduce yourself and describe your role in the WIC program.
2) Federal law currently requires certifications to be conducted in person, but a waiver during COVID allowed this to happen remotely or to postpone certain parts of the certification process. What does remote certification look like in Washington during the COVID pandemic? iii. Would any policy changes be required? iv. How feasible or likely does it seem that this could happen? 9) How has offering WIC certification and services remotely impacted service quality, convenience, and outcomes?
(continued on next page)

WIC STAFF INTERVIEW AND FOCUS GROUP QUESTIONS
10) In addition to the ability to conduct certifications and services remotely, what other WIC changes or flexibilities would you like to see in Washington State to make it easier for clients to use the program during or after the COVID pandemic? a. Are there any priorities that the agency is currently exploring or planning? 11) How have changes to the certification, nutrition education, health screenings and referrals, and breastfeeding support/peer counseling been communicated to WIC clients? 12) We plan to conduct focus groups with local staff and interviews with WIC clients about remote certification, remote services, and the recent expansion of Washington WIC's list of allowable foods. What would you be most interested in hearing from local staff and clients about these topics? (continued on next page) b. If so, how important or urgent do you feel these changes are? 8) We know other states also expanded their food list in response to COVID. What have your heard from them, and have you reconsidered or refined your approach based on this? Have you noticed any differences in how these changes have been experienced by specific groups of clients? For example, has the experience been different for tribal clients, clients that speak a language other than English, working parents, families of different sizes, or families who live in rural areas? 5) What changes or supports would you recommend to improve the quality, convenience, or impact of remote WIC [certification, nutrition education, breastfeeding support] services?
(continued on next page)  To what extent do you think the change has impacted benefit redemption or client food security? Do you think this may have been influenced by COVID-19 and changes in how people have been shopping for food or the availability of food items? 8) We also plan to conduct interviews with WIC clients about changes made to WIC certification and services since COVID started. What questions you would suggest that we ask? 9) Please think about 1 or 2 successful client interactions you have had. Did they happen remotely? If not, could they have happened if services were offered remotely? Why or why not? 10) Those are all the questions we have today. Can you think of anything we did not talk about today that would be important to consider in relation to remote WIC services or the expanded food package?