title: Identifying outliers in an event sequence using a Python Dataframe
body: <p>I'm experimenting with Machine Learning and LSTM models for river level prediction based on the current level and rainfall within the upstream catchment. I'm currently using TensorFlow and a Keras LSTM model.</p>
  <p>I have a hourly rainfall data in mm from eight sites within the catchment, and the river stage (level) in meters at a site downstream from the catchment area.  The problem I face is that every now and then the rainfall sites are tested by pouring water into them.  This creates a significant spike in the data that I would like to filter out.</p>
  <p>Here's an example of what a typical rainfall event looks like within the dataframe:</p>
  <p><a href="https://i.stack.imgur.com/eg8LF.png" rel="nofollow noreferrer">DataFrame showing typical seqeunce rainfall and river stage data</a></p>
  <p>And here is an example of what it looks like when two sites have been tested</p>
  <p><a href="https://i.stack.imgur.com/xTaVc.png" rel="nofollow noreferrer">DataFrame showing abnormal rainfall data due to two sites being tested</a></p>
  <p>I've come across several ways to statistically cluster data and identify outliers however none of these really worked on a timed sequence of events.  Also, the rainfall site columns are listed in the DataFrame in the order in which they are located with the catchment so there is a loose spatial coupling moving across the columns.</p>
  <p>I was thinking of using something a little like a 3x3 or 3x4 convolution matrix, but rather than calculating new cell values it would find outliers by comparing the values from the central cells with the values in the outer cells.  Any significant difference would indicate abnormal data.</p>
  <p>The Pandas DataFrame API is quite large and I'm still getting familiar with it.  Any suggestions on specific functions or methods I should be looking at would be much appreciated.</p>
  <p>In the following example the 10:00:00 reading for Site 2 would be an obvious anomaly.</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Timestamp</th>
  <th>Site 1</th>
  <th>Site 2</th>
  <th>Site 3</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>09:00:00</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  </tr>
  <tr>
  <td>10:00:00</td>
  <td>0</td>
  <td>20</td>
  <td>0</td>
  </tr>
  <tr>
  <td>11:00:00</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>20mm of rainfall at one site with zero rainfall at the adjacent sites, or at the same site for the hour before and hour after is a very clear and obvious case.</p>
  <p>This is what a normal rainfall pattern might look like:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Timestamp</th>
  <th>Site 1</th>
  <th>Site 2</th>
  <th>Site 3</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>09:00:00</td>
  <td>6</td>
  <td>4</td>
  <td>0</td>
  </tr>
  <tr>
  <td>10:00:00</td>
  <td>0</td>
  <td>20</td>
  <td>2</td>
  </tr>
  <tr>
  <td>11:00:00</td>
  <td>0</td>
  <td>0</td>
  <td>11</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>This is a less obvious example:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Timestamp</th>
  <th>Site 1</th>
  <th>Site 2</th>
  <th>Site 3</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>09:00:00</td>
  <td>1</td>
  <td>0</td>
  <td>0</td>
  </tr>
  <tr>
  <td>10:00:00</td>
  <td>0</td>
  <td>20</td>
  <td>2</td>
  </tr>
  <tr>
  <td>11:00:00</td>
  <td>0</td>
  <td>3</td>
  <td>1</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>One possibility might be to compare the central cell value to the maximum of the surrounding cell values and if the difference is greater than 15 (or some other arbitrary threshold value).</p>
  <p>The exact criteria will probably change as I discover more about the data.  The mechanism of how to apply that criteria to the dataframe is what I'm looking for.   For example, if the criteria were implemented as a lambda function that could reference adjacent cells, is there a way to apply that lambda function across DataFrame?'</p>
  <p>An extra complication is how to deal with checking values for Site 1 when there is preceding site to the left, and Site 3 where there is no  following site to the right.</p>