title: How to smooth/reduce/filter large amounts of sensor data?
body: <p>I'm building a visualization for some industrial devices that produce large amounts of time-based data like temperatures, currents or voltages. All data is constantly written to a SQL Server database (can't control that part).</p>
  
  <p>The HTML5 frontend consists of an interactive zoomable chart I made with d3.js. Data series can be added(loaded) to the chart on demand, in which case the frontend sends an ajax request, ASP.NET MVC and EF6 fetches the values from the DB and returns them as Json.</p>
  
  <p>Each data element simply consists of a DateTime and a value. Please note that the values do <em>not</em> get written in regular intervals (like every 2 seconds or so), but in irregular intervals. This is because the device doesn't get polled regularly but sends data on specific events, like a raise/drop of a temperature by a given change of 0.1 °C, for example.</p>
  
  <p>So far everything works really well and smooth, but the large amount of data becomes a problem. For example, when I want to show a line chart for a selected period of lets say 3 month, each data series already consists of appr. 500.000 values, so the Json response from the server also gets bigger and bigger and the request takes longer with growing time periods.</p>
  
  <p>So I am looking for a way to reduce the amount of data without losing relevant information, such as peaks in temperature curves etc., but at the same time I want to smoothen out the noise in the signal.</p>
  
  <p>Here's an example, please keep in mind that this is just a chosen period of some hours or days, usually the user would like to see data for several months or even years as well:
  <a href="https://i.stack.imgur.com/trEZn.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/trEZn.jpg" alt="enter image description here"></a></p>
  
  <p>The green lines are temperatures, the red bars are representations of digital states (in this case a heater that makes one of the temp curves go up).</p>
  
  <p>You can clearly see the noise in the signals, this is what I want to get rid of. At the same time, I want to keep characteristic features like the ones after the heater turns on and the temperature strongly rises and falls.</p>
  
  <p>I already tried chopping the raw data into blocks of a given length and then aggregating the data in them, so that I have a min, max and average for that interval. This works, but by doing so I the characteristic features of the curve get lost and everything gets kind of flattened or averaged. Here's a picture of the same period as above zoomed out a bit, so that the aggregating kicks in:
  <a href="https://i.stack.imgur.com/vPupQ.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vPupQ.jpg" alt="enter image description here"></a></p>
  
  <p>The average of the upper series is shown as the green line, the extent (min/max) of each chop is represented by the green area around the average line.</p>
  
  <p>Is there some kind of fancy algorithm that I can use to filter/smoothen/reduce my data right when it comes out of the DB and before it gets send to the frontend? What are the buzzwords here that I need to dig after? Any specific libraries, frameworks or techniques are highly appreciated, as well as general comments on this topic. I'm interested primarily in server-side solutions, but please feel free to mention client-side Javascript solutions as well as they might surely be of interest for other people facing the same problem.</p>