title: how to tidy Standard Hydrologic Exchange Format (SHEF) data
body: <p>The U.S. National Oceanic and Atmospheric Administration (NOAA) has a large amount of data in Standard Hydrologic Exchange Format (SHEF) (eg, as in links below). The linked data has four main pieces of information: location name, location ID, reported value (either numeric or "NE" - not estimated), and elevation zone. I'm hoping to convert the SHEF data into four column <code>data.frame</code>s. The SHEF format, though it has "exchange" in its name, does not seem straightforward to work with but I might be missing something. </p>
  
  <p>Both pages of linked data below have 1137 lines of text of snow data for identical locations and times but for different snow parameters.  </p>
  
  <p>There are two chunks of code, one for each webpage. They're identical except for their urls that point to the respective parameters. </p>
  
  <p>The code below outputs an almost intended <code>data.frame</code> for one of the parameters, <code>swe</code>, but for the other, <code>sub</code>, the resultant <code>data.frame</code> comes out obviously partially complete with respect to the original data, and with wrong values (see tibbles at bottom).  I'm thinking that because SHEF format is at least consistent, and because there might be functions/libraries just for this sort of thing, there might be a whole different angle/significantly fewer steps needed, for the conversion? </p>
  
  <p>snow parameter 1 ("swe") (snow water equivalent): 
  <a href="https://www.nohrsc.noaa.gov/shef_archive/index.html?rfc=cnrfc&amp;product=swe&amp;year=2019&amp;month=3&amp;day=27&amp;hour=12" rel="nofollow noreferrer">https://www.nohrsc.noaa.gov/shef_archive/index.html?rfc=cnrfc&amp;product=swe&amp;year=2019&amp;month=3&amp;day=27&amp;hour=12</a> (data in gray box)</p>
  
  <p>snow parameter 2 ("sub") (sublimation):
  <a href="https://www.nohrsc.noaa.gov/shef_archive/index.html?rfc=cnrfc&amp;product=sb&amp;year=2019&amp;month=3&amp;day=27&amp;hour=12" rel="nofollow noreferrer">https://www.nohrsc.noaa.gov/shef_archive/index.html?rfc=cnrfc&amp;product=sb&amp;year=2019&amp;month=3&amp;day=27&amp;hour=12</a> (data in gray box)</p>
  
  <p>I'm hoping for two <code>data.frames</code>, <code>swe</code> and <code>sub</code>, with 4 columns each. Below is the working example.</p>
  
  <pre><code>library(tidyverse)
  library(rvest)
  library(lubridate)
  
  
  # webpage to scrape data from, March27's parameter "swe"      
  march27_param_swe &lt;- 
   "https://www.nohrsc.noaa.gov/shef_archive/index.html?rfc=cnrfc&amp;product=swe&amp;year=2019&amp;month=3&amp;day=27&amp;hour=12"
  
  ####### snow water equivalent (swe) [inches] ##########
  
  # scrape
  scrapedtext &lt;- read_html(march27_param_swe) %&gt;% html_node(".notes") %&gt;%
                 html_text() 
  
  
  
  swe &lt;- tibble(txt = read_lines(scrapedtext)) %&gt;%
  
   mutate(
           row = row_number(),
           with_code = str_extract(txt, "^[A-z0-9]{5}\\s+\\d+(\\.)?\\d"),
           wo_code = str_extract(txt, "^:?\\s+\\d+(\\.)?\\d") %&gt;% 
           str_extract("[:digit:]+\\.?[:digit:]"),
           basin_desc = if_else(!is.na(with_code), lag(txt, 1), NA_character_) %&gt;% 
           str_sub(start = 2)
         ) 
  
  swe &lt;- swe %&gt;% separate(with_code, c("code", "val"), sep = "\\s+") %&gt;%  
         mutate(value = case_when(
                                  !is.na(val) ~ val,
                                  !is.na(wo_code) ~ wo_code,
                                  TRUE ~ NA_character_) %&gt;%
                                  as.numeric) %&gt;% filter(!is.na(value)) 
  
  swe &lt;- swe %&gt;% mutate(code = zoo::na.locf(code), basin_desc = zoo::na.locf(basin_desc) ,
                        elevz = gsub(".*(inches))","",txt))  %&gt;%
                        select(code, value, basin_desc, elevz) %&gt;%
                        mutate(elevz = trimws(elevz))
  
   dim(swe) 
   #[1] 643   4
  
   head(swe)  
   # # A tibble: 6 x 4
   # code    value basin_desc               elevz             
   # &lt;chr&gt;   &lt;dbl&gt; &lt;chr&gt;                    &lt;chr&gt;             
   # 1 ACSC1   0   San Antonio Ck - Sunol   "Entire Basin"   
   # 2 ADLC1   0   Arroyo De La Laguna      "Entire Basin"   
   # 3 ADOC1   0   Santa Ana R - Prado Dam  "Entire Basin"   
   # 4 AHOC1   0   Arroyo Honda nr San Jose "Entire Basin"   
   # 5 AKYC1  41.8 SF American nr Kyburz    "Entire Basin"   
   # 6 AKYC1   3.9 SF American nr Kyburz    "Base  to 5000'"
  
   #which is what I'm hoping for, except that I'd like the `value` to be 
   #&lt;chr&gt; to be able to accommodate the numbers and "NE" values reported, like this:
  
   # # A tibble: 6 x 4
   # code  value basin_desc               elevz             
   # &lt;chr&gt; &lt;chr&gt; &lt;chr&gt;                    &lt;chr&gt;          
  
   #######  surface sublimation (sub) ##########
  
  # same locations and day, different parameter, "sb", blowing snow 
  # sublimation [inches]
  
  march27_param_temp &lt;- "https://www.nohrsc.noaa.gov/shef_archive/index.html?rfc=cnrfc&amp;product=sb&amp;year=2019&amp;month=3&amp;day=27&amp;hour=12"
  
  scrapedtext &lt;- read_html(march27_param_temp) %&gt;%
                 html_node(".notes") %&gt;% html_text() 
  
  sub &lt;- tibble(txt = read_lines(scrapedtext)) %&gt;%
    mutate(
           row = row_number(),
           with_code = str_extract(txt, "^[A-z0-9]{5}\\s+\\d+(\\.)?\\d"),
           wo_code = str_extract(txt, "^:?\\s+\\d+(\\.)?\\d") %&gt;% 
           str_extract("[:digit:]+\\.?[:digit:]"),
           basin_desc = if_else(!is.na(with_code), lag(txt, 1), NA_character_) %&gt;%
           str_sub(start = 2)
  ) 
  
  sub &lt;- sub %&gt;% separate(with_code, c("code", "val"), sep = "\\s+") %&gt;%  
          mutate(value = case_when(
                                   !is.na(val) ~ val,
                                   !is.na(wo_code) ~ wo_code,
                                   TRUE ~ NA_character_) %&gt;%
                                   as.numeric) %&gt;% filter(!is.na(value)) 
  
  sub &lt;- sub %&gt;% mutate(code = zoo::na.locf(code), basin_desc = zoo::na.locf(basin_desc) ,
                        elevz = gsub(".*(inches))","",txt))  %&gt;%
                        select(code, value, basin_desc, elevz)  %&gt;%
                        mutate(elevz = trimws(elevz))
  
  dim(sub)
  #[1] 263   4    #dim[swe] was 643x4
  
  head(sub)
  
   # A tibble: 6 x 4
   #code     value   basin_desc                elevz             
   #&lt;chr&gt;    &lt;dbl&gt;   &lt;chr&gt;                     &lt;chr&gt;             
   #1 ADOC1     0    Santa Ana R - Prado Dam   "Entire Basin"   
   #2 ADOC1     0    Santa Ana R - Prado Dam   "Base  to 5000'"
   #3 ARCC1     0    Mad River - Arcata        "Entire Basin"   
   #4 ARCC1     0    Mad River - Arcata        "Base  to 5000'"
   #5 BCAC1     0    Little Truckee - Boca Dam "Entire Basin"   
  
  #So `sub` should be the same size `data.frame` as swe, and 
  #sub$value's are supposed to be (as per the source page above: 
  # https://www.nohrsc.noaa.gov/shef_archive/index.html?rfc=cnrfc&amp;product=sb&amp;year=2019&amp;month=3&amp;day=27&amp;hour=12 ):
  
   #head(desired_sub)
   # A tibble: 6 x 4
   #code        value  basin_desc                elevz             
   #&lt;chr&gt;       &lt;chr&gt;  &lt;chr&gt;                     &lt;chr&gt;             
   #1 ADOC1     NE     Santa Ana R - Prado Dam   "Entire Basin"   
   #2 ADOC1     NE     Santa Ana R - Prado Dam   "Base  to 5000'"
   #3 ARCC1     0.000  Mad River - Arcata        "Entire Basin"   
   #4 ARCC1     NE     Mad River - Arcata        "Base  to 5000'"
   #5 BCAC1    -0.016  Little Truckee - Boca Dam "Entire Basin"   
  </code></pre>