{'fraction_non_alphanumeric': 0.027679843972695222, 'fraction_numerical': 0.010764383767159253, 'mean_word_length': 5.090223846505253, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Initial Assessment of Unmanned Aircraft System Characteristics Required to Fill Data Gaps for Short-term Forecasts: Results from Focus Groups and Interviews

The integration of unmanned aircraft systems (UAS) into the weather surveillance network must be guided by the data needs of the principal stakeholders. This work aims to assess data needs/gaps for short-term forecasts (<1-day lead time) issued by the National Weather Service (NWS) and then identify UAS characteristics required to fill these gaps. Results from focus groups and interviews of forecasters in the central United States are presented. Participant verbal responses were coded and then categorized into a set of 25 unique features. Each feature was classified according to four characteristics: 1) environmental properties that need to be measured to represent a given feature, 2) flight type (vertical profile, horizontal transect, and/or survey) 3) flight height required to measure the environmental properties, and 4) relevance of feature to the forecasting of deep convection. Findings indicate the majority of identified features require measurement of typical state variables (temperature, moisture, and wind), but more than a third require visual imagery. Almost all of the features require either survey flight operations or vertical profiles. Additionally, 96% of the features require observations collected below 1000 m. Nearly two-thirds of the features are associated with deep convection. This work represents the first step towards establishing how UAS could be used to fill data gaps that exist for short-term forecasts issued by the NWS. The results stand alone in demonstrating the potential applications of UAS from the perspective of operational forecasters and have also informed ongoing efforts to develop a nationwide survey of forecasters. ABSTRACT (Manuscript received 21 August 2019; review completed 6 April 2020)


Introduction
Unmanned aircraft systems (UAS) have the potential to revolutionize weather surveillance (NRC 2009;NASEM 2018;Vömel et al. 2018). However, their integration into the United States weather surveillance network needs to be guided by the data requirements of the principal stakeholders. One of the primary users of these data is the National Oceanic and Atmospheric Administration's National Weather Service (NWS). In an effort to assess the data gaps that exist for short-term forecasts (<1-day lead time), which might be filled via UAS applications, we are adopting a mixed-methods approach consisting of 1) qualitative focus groups and one-on-one interviews of forecasters in the central United States and 2) a quantitative national survey. Results from the focus groups and interviews are presented here.
In general, focus groups and interviews are an appropriate and often-used methods for exploring and documenting expert perspectives on existing or new problems (e.g., Krueger and Casey 2000;Morgan and Krueger 2013;Royle and Laing 2014). They previously have been used to understand how meteorologists use uncertainty information and their preferences for forecast uncertainty information (Demuth et al. 2009).
Prior work that specifically focused on identifying data gaps for short-term forecasting has principally been the result of studies commissioned by national boards (e.g., NRC 2009) and or federal agencies (e.g., Dabberdt et al. 2005). Findings from these studies were informed by the experience of the designated committee of experts and vetted through peer-review of the resulting reports but were rarely (if ever) directly based on surveys of primary users. The research presented here aims to complement this prior work by reporting results from direct engagement with primary users. The need for this kind of work is reflected in Observing Weather and Climate from the Ground Up: A Nationwide Network of Networks (NRC 2009): "The stakeholders should commission an independent team of social and physical scientists to conduct an end-user assessment for selected sectors. The assessment should quantify further the current use and value of mesoscale data in decisionmaking and also should project future trends and the value associated with proposed new observations." (p. 13, NRC 2009).
A comparison of the results from the present research and the results from prior reports is included in Section 3.
The focus of the work presented here differs somewhat from prior work designed to assess the value of data to NWS forecasters (e.g., Morss and Ralph 2007;Heinselman et al. 2012;Evans et al. 2014;Heinselman et al. 2015;Bowden and Heinselman 2016). In this prior work, the impact of data was evaluated in a simulated and/or real forecasting environment. As such, the data and platforms were preselected, though the method of data delivery may have been altered (e.g., Bowden and Heinselman 2016). In contrast, the focus of our ongoing work is to answer the question "what data do forecasters need?" and then to explore the possible role of a particular platform (UAS) on delivering the needed data. Specifically, the objectives of the focus groups and interviews reported here were to 1) identify critical data gaps that exist for short-term forecasting in the central United States, 2) map out the potential applications of UAS for filling these gaps, and 3) translate these applications into the operational characteristics of UAS (e.g., instrumentation, platform type, flight ceiling). With the exception of the focus on UAS, these objectives are nearly identical to those of NRC (2009).
Although the data in this report are not representative of all NWS perspectives, they represent a first-of-itskind attempt to establish how UAS could be used to fill data gaps that exist for short term forecasts issued by the NWS. The results already have informed ongoing efforts to develop a nationwide survey of forecasters, and also stand alone in demonstrating the potential applications of UAS from the perspective of operational forecasters. This article proceeds with a description of the methods in section 2, results from analysis of the focus groups and interviews in section 3, and a summary and discussion of future work in section 4.

Methods
The present study and all of its methods and measures were reviewed and approved by the University of Nebraska Institutional Review Board for the Ethical Treatment of Human Subjects (RII Track-2 FEC: Unmanned Aircraft System for Atmospheric Physics, IRB Approval #: 20151115696 EX). Participants were required to be adults the age of majority (age 19 in Nebraska) and passive consent procedure was used, for which consent information was included on the first page of the survey and participants were informed that completion of the survey indicated consent to have their data analyzed and reported in non-identifiable form. Participants in this study were recruited from existing contacts at the NWS and through open invitations sent by division chiefs at NWS regional offices to forecast office Meteorologists in Charge encouraging them to invite their staff to participate.
Participants completed a short pre-survey to provide contact information, job title, forecast region, and willingness to participate. Of the 17 individuals who completed the pre-survey, 10 participated in a focus group or interview. Following each focus group or interview, participants were asked to complete a postsurvey assessing participant demographics. Nine of the 10 participants completed the post survey. Results from the pre-survey are not reported here because relevant information (job title and forecast region) was collected during the focus groups and interviews.
Focus groups enabled observation of meaningful group interactions to see how participants discuss and understand the complex ideas under discussion (Morgan and Krueger 1993), whereas the interviews enabled an in-depth evaluation of participant perspectives, data needs, and understanding of UAS technology (Rubin and Rubin 2011). Focus groups and interviews were conducted with a facilitator and an atmospheric science expert (the same facilitator and expert were used in the focus groups and interviews). Two 45 min focus groups were held online via video conferencing, each with four participants (three NWS offices were represented in the first focus group while four offices were represented in the second). In addition, two NWS employees (from two NWS offices not represented amongst the focus group participants) who could not join one of the focus group because of schedule conflicts, participated in ~40 min phone interviews.
The script used by the facilitator to guide the focus groups and interviews appears in the Appendix. Even though the focus of this work was to begin to define data gaps for short-term forecasting, the discussion began by asking for perspectives on challenging types of forecasts. Following this discussion of challenging forecasts, participants were asked to consider the data gaps that might help with these challenges. Importantly, initial discussions of challenges and relevant gaps were done without specifically considering how UAS might play a role. A discussion of the role that UAS might play in filling these gaps was last. Participants were also asked to consider the potential obstacles to the envisioned implementation of UAS.
This strategy of considering challenging forecast phenomena first before explicitly considering data gaps or the instrumentation capable of filling these gaps was also adopted by the NRC (2009) who used a "phenomenological approach" wherein "the hazardous weather events most important to detect, monitor, and predict" (p. 24, NRC 2009) were identified first followed by an examination of the relevant data gaps and then the state variables and the spatiotemporal sampling granularity required to characterize the phenomenon. For the work presented herein this approach was adopted because it allows participants to make concrete connections between data needs and the forecast elements they impact. It also enables an exploration of data gaps that is largely platform agnostic. Nevertheless, participants were aware from the recruitment material as well as the introductory information provided before each focus group/interview that the investigators wanted to ultimately determine if UAS could be used to fill these data gaps. As such, this prior knowledge may have influenced participants' cognitive frames and thus their responses.

Results
Each focus group and interview were transcribed verbatim. Transcripts were then coded by the members of the research team using a deductive approach (Braun and Clarke 2006). After responses were initially coded, responses were validated using exemplar quotes (Suter 2009), investigator triangulations (Suter 2009), and peer review with atmospheric scientists to ensure that the correct terminology was used in the coding (Creswell 2014). Responses were then categorized into a set of unique features ( Table 1). Some of these features were offered by forecasters as standalone meteorological/ environmental phenomena that are challenging to diagnose or predict, ostensibly because of data gaps (e.g., precipitation outside existing radar coverage, storm damage, levee breaches).
Other features were contributed by forecasters as meteorological/ environmental characteristics that, if observed better, could improve diagnosis and/or prediction of other phenomena (e.g., moisture gradients for convection initiation (CI), storm appearance, cap strength). Given that the primary goal of the research is to establish the breadth of extant qualitative features, not to establish the quantitative importance of features, no attempt is made to catalog the number of times that a particular feature was mentioned. Future research can build upon the present work to ascertain the importance or priority of such individual features. Here, the focus is instead on surveying the breadth and nature of the characteristics of the features.
As reflected in Table 1, each feature is classified according to four characteristics: 1) environmental properties that would need to be measured to represent a given feature (selected from temperature, water vapor content, wind, visual imagery, and/or "other"), 2) flight type (vertical profile, horizontal transect, and/or survey), 3) flight levels (AGL) at which operations would need to be conducted [low (<120 m), middle, (120-1000 m), and high (>1000 m)], and 4) whether a particular feature is relevant to the forecasting of deep convection. A survey flight is defined as a flight in which the aircraft collects observations from a fixed location (e.g., observations of snow depth or storm damage) or from a location "fixed" in a phenomenon-relative frame of reference (e.g., thunderstorm appearance observed while following the storm). By this definition, multiple locations can be observed but the horizontal/vertical interconnectedness is less the focus than the observations at a given point in space. The selection of height classifications has been  Table 1 also includes a comparison to prior work aimed toward identifying data gaps. Specifically, comparison is made to the NRC (2009) report, that identified data gaps relevant to "hazardous weather events most important to detect, monitor, and predict" as well as several "national priorities" (p. 21, NRC 2009) and the 2005 United States Weather Research Program report (Dabberdt et al. 2005), that focused on data gaps that needed to be filled to improve nowcasting (forecasting with 0-6 hr lead times) of high-impact weather and "acute air quality, hydrology, chemical emergency response, and other applications" (p. 962, Dabberdt et al. 2005).
Of the 25 features identified by our respondents, 32% would require measurements of temperature, 28% would require water vapor content, 24% would require wind, 36% would require visual imagery, and 20% would require a property falling in the category "other." Not surprisingly, the majority of features (52%) would require measurements of a typical state variable (temperature, moisture, or wind). What may be somewhat more surprising is that more than a third of the features would require visual imagery. Examples of features that require visual imagery are storm appearance, storm damage, and ice jams. This finding suggests that camera-equipped UAS technology may be very important for filling extant data gaps. Moreover, of the features that require visual imagery, 56% would require imagery of the ground. Given that privacy concerns of the general public become more acute when public UAS are tasked to collect images of the surface (Walther et al. 2019), this finding suggests that the collection of observations for nearly 1/5 of the features identified in this study could be viewed as threatening privacy by the general public.
Unlike the environmental properties' category, flight types (vertical profile, horizontal transect, or survey) are mutually exclusive. The "survey" flight type would be required of more of the features (48%) than vertical profiles (44%) or horizontal transects (8%).
Flight level classification was based on the expected flight levels at which data collection must occur. It does not consider the flight levels through which the aircraft must pass to begin data collection (otherwise 100% of the features would require flight levels in the "low" category). It is also assumed that surveying the ground (e.g., for damage surveying, monitoring ice jams, monitoring levee breaches) necessitates flights both below and above 120 m (the former for detailed interrogation, the latter for "larger" scale context) but no flights above 1000 m. Moreover, it is assumed that surveys of above-surface conditions (e.g., radar gaps, storm appearance, cloud field of the preconvective environment) require observations collected above 120 m and possibly above 1000 m. Based on these assumptions, and noting the significant overlap between the three observation flight levels, 96% of the features would require observations collected below 1000 m with a little over 1/4 of the features (28%) requiring observations collected above 1000 m. Clearly, data collection within (or from) the planetary boundary layer would be required for nearly all of the features identified.
As noted above and discussed further below, the features identified by NWS participants are likely to be strongly dependent on the region they represent such that the set of features recommended will be tied to the meteorological hazards common to a given forecast office. Because this work focused on the central United States where a variety of hazards are found, it was uncertain from the outset whether hazards associated with deep convection would be dominant. However, nearly 2/3 of the features identified (64%) were associated with deep convection. Because this dominance is not a measure of priority but instead a measure of the number of applicable features in the list, the dominance of features tied to deep convection does not necessarily mean that deep convection is the phenomenon in the central United States that is most plagued by data gaps. This result may instead mean that there are a number of data gaps that, if filled, could contribute to more accurate forecasts of deep convection and associated phenomena.
Besides providing a list of features and data gaps, focus group and interview participants offered valuable insight regarding the integration of UAS into the weather surveillance network. Concerning the responsibility for operating UAS, several participants felt that local forecast offices should not be solely in charge of UAS operation. Instead they thought that forecast offices should be involved in discussions regarding UAS deployments but that "somebody else" should run the program: "it would be better served to have somebody else probably running that program and then us being able to have some kind of collaboration, 'hey, this would be great if we could sample this area today' and then we benefit from the data." One participant noted the following: "I think the problem with having a local office have too much control is you're too parochial: it may be important to us but it may not be the most important thing happening in the region. I think offices should have input, certainly, and the discussion should be very open, but alternately, I think someone else is going to have to make the decision on what the best utilization of that technology is for that particular day." One of the participants noted that other agencies that engage with the NWS would likely be interested in data collected by UAS as well: "there's going to be a lot of different partner needs if you expand this past the Weather Service, USGS (United States Geological Survey), would be a good example of that, Fire Service would be another good example of that, Emergency Management community, everybody just having access, essentially, to that information, that it's readily available." In response to this another participant noted "this is probably not going to be something that just the weather service can utilize if we're going to be able to leverage this technology it's going to have to be in multiple partners." Although the relative importance of fixed-site profiling vs. targeted surveillance was not explicitly probed by the facilitator or offered voluntarily by the participants, one participant commented on the potential value of exploiting the targeted surveillance capability of UAS: "my pie-in-the-sky opinion on this it would be that the soundings or the upper air data above the surface could be concentrated in areas where we have [a] severe weather outlook for that day, whether it's automated or whether we draw a line and send it and say 'hey, you know, this is where we'd like some data,' it would benefit us a lot more to have the data from areas where severe weather is possible." One participant emphasized the importance of minimizing data latency: "the key really is making sure that data can at least probably reach the hands of a forecaster within at the most a five, ten min period. Once we start having latency of probably >30 min then it's still useful but certainly the usefulness goes down the longer it takes to get any observations in." The importance of observation accuracy for any new observing platform like UAS was also noted: "for forecasting, it's all about being able to trust the data, so if we can trust the data and it's coming in, then it will be useful." Although participants noted the potential value of assimilating UAS data into numerical weather prediction models, the prevailing sentiment across the focus groups and interviews was that UAS data should also be disseminated directly to the forecasters. Emphasis was also placed on the importance of visualizing these data in a format with which they are familiar, e.g., skew-T/log-p charts for vertical profiling UAS. Multiple participants also expressed interest in using UAS for verification of both numerical weather prediction model output and their own forecasts, particularly in data sparse regions.
Several participants noted the importance of making UAS data available via the web, instead of solely via integration into AWIPS: "it needs to be on a webpage of some sort that can be accessible from the field also. Sometimes we're doing remote forecasting, we're on site and we need to be able to access it through the web. It can't just come internally through our system". In response to this comment, another participant noted "now, it would be nice to have it in AWIPS but it's not a necessity. We do have PCs right next to our AWIPS workstations so we can look at any website. So if you had to prioritize something, I would agree with those guys and say, some sort of web platform first and foremost." Nevertheless, several participants noted that integration of UAS data with existing data is very important. One participant noted, "so when we start talking about real-time or near real-time observations, if that could be placed into like an AWIPS type system where it could be integrated with a lot of other sets of data, that could become extremely useful." One participant cautioned that, as with any new data stream that offers a more precise characterization of the environment or of a particular phenomenon, training is very important: "forecasters would just need some training to really understand how to best digest that type of data and realize that you're going to see things that you normally didn't expect to see with an increased spatial and temporal resolution that we wouldn't have before".
The issue of public perception of NWS operation of UAS was brought up by several forecasters as a concern: "so I've always been told that privacy issues are the main concern for us using drones. It's that, you're going to be flying over someone's property that's been damaged by a tornado and maybe they don't want photographs taken of their property." In response to this comment another participant noted "it might help to make [the drone] easily identified as something from the National Weather Service or something from [the University of Nebraska-Lincoln] so people know that it's not just some creeper trying to look at their stuff. Then again they may want to shoot it down even more at that point, if they see if it's from the government."

Summary and future work
Results from focus groups and interviews involving NWS personnel principally from the central United States have been presented with the aim of identifying key data gaps for short-term forecasting and the characteristics of UAS required to fill these gaps. Twenty-five unique features were identified by respondents. The majority of these features would require measurement of typical state variables (temperature, moisture, and wind) but more than a third required visual imagery (56% of these features required imagery of the ground). Nearly all of the features would require either survey flight operations or vertical profiles, yet only 8% required horizontal transects. Similarly, nearly all of the features would require observations collected below 1000 m. Nearly 2/3 of the features were associated with deep convection. Respondent comments indicated the importance of low-latency, direct dissemination of UAS data to forecasters in a format with which they were familiar. Trust in the accuracy of the data and training to interpret the higher precision observations were also important to participants. The potential value of using UAS for targeted surveillance was also noted. Although the possibility of UAS being operated by forecast offices met with some resistance, consultation with local offices regarding UAS operations was thought to be important. It was also noted that UAS tasked for meteorological data collection might have value to other federal agencies as well.
These results are being integrated into a survey for nationwide distribution to NWS employees. Chief among the aims of this survey will be a prioritization of the data gaps that exist for short term forecasting. This prioritization is necessary in order to then prioritize the UAS technology to fill the gaps. To achieve such prioritization, participants will be asked to rate the importance of each of the data gaps identified from the focus groups. They will also have an opportunity to add (and rate) data gaps they feel are missing from the original list.
Participants in the nationwide survey will also be asked to comment on several issues related to UAS operations that emerged from this work but that cannot be inferred from analysis of the data needs alone: • Is the operation of UAS at fixed sites (either profiling above these sites or executing horizontal transects between them) sufficient or is data collection in dynamically-defined regions-of-interest (i.e., targeted surveillance) necessary?
• If targeted surveillance occurs, how much control should the forecast offices have in the targeting decisions?
• If data collection is performed at fixed sites only (either as profilers or in the form of transects between sites), should the frequency of operation (within the technological limits of the platforms) be at the discretion of the forecast offices much like radiosonde launches?
Ultimately, the aim of this nationwide survey of NWS forecasters is to contribute to a broader determination of where technological strengths and shortcomings, public perception hopes and concerns, and/or regulatory supports or obstacles, either facilitate or stand in the way of filling the highest priority data gaps using UAS technologies.