title: Using Tensorflow to detect falls and implementing the model in Android
body: <p>It's my first post on the forum, so please excuse any major errors or stupidities on my part.</p>
  
  <p>I'm currently doing a project on fall detection using machine learning. I've seen many examples online about ADL's (Activites of Daily Live) and Tensorflow and tried to use some of them as a basis to my work. For example, this one: <a href="https://medium.com/@curiousily/human-activity-recognition-using-lstms-on-android-tensorflow-for-hackers-part-vi-492da5adef64" rel="nofollow noreferrer">https://medium.com/@curiousily/human-activity-recognition-using-lstms-on-android-tensorflow-for-hackers-part-vi-492da5adef64</a>.</p>
  
  <p>What I did was use the known WISDM dataset (this one: <a href="http://www.cis.fordham.edu/wisdm/dataset.php" rel="nofollow noreferrer">http://www.cis.fordham.edu/wisdm/dataset.php</a>) as the "Not Falling" examples and found another dataset with 30 fall examples (this one: <a href="http://fenix.univ.rzeszow.pl/~mkepski/ds/uf.html" rel="nofollow noreferrer">http://fenix.univ.rzeszow.pl/~mkepski/ds/uf.html</a>).</p>
  
  <p>However, I'm encountering some problems regarding the use of the model on the smartphone. Even tough I see, in some cases, accuracy levels of 80-90% and up, when I test the model on an App, the results are <strong>horrible</strong>. Is it even possible to do it this way? What could be the problem?</p>
  
  <p>I've tried numerous ways to mix the datasets, changing the learn rate, the complexity of the model, etc. but still no luck...</p>
  
  <pre><code>import pandas as pd
  import numpy as np
  import pickle
  import matplotlib.pyplot as plt
  from scipy import stats
  import tensorflow as tf
  import seaborn as sns
  from pylab import rcParams
  from pylab import plot
  from sklearn import metrics
  from sklearn.model_selection import train_test_split
  
  sns.set(style='whitegrid', palette='muted', font_scale=1.5)
  
  rcParams['figure.figsize'] = 14, 8
  
  RANDOM_SEED = 42
  columns = ['user','activity','timestamp', 'x-axis', 'y-axis', 'z-axis']
  df_train = pd.read_csv('test_4.txt', header = None, names = columns)
  df_test = pd.read_csv('train_4.txt', header = None, names = columns)
  print('Data read.')
  df_train['activity'].value_counts().plot(kind='bar', title='Training examples by activity type');
  #df['user'].value_counts().plot(kind='bar', title='Training examples by user');
  plt.show()
  df_test['activity'].value_counts().plot(kind='bar', title='Training examples by activity type');
  #df['user'].value_counts().plot(kind='bar', title='Training examples by user');
  plt.show()
  
  ##def plot_activity(activity, df):
  ##    data = df[df['activity'] == activity][['x-axis', 'y-axis', 'z-axis']][:200]
  ##    axis = data.plot(subplots=True, figsize=(16, 12), 
  ##                     title=activity)
  ##    for ax in axis:
  ##        ax.legend(loc='lower left', bbox_to_anchor=(1.0, 0.5))
  ##plot_activity("Jogging", df)
  ##plt.show()
  
  ## TRAIN DATA - EACH GENERATED SEQUENCE MUST CONTAIN 200 TRAINING EXAMPLES
  N_TIME_STEPS = 200
  N_FEATURES = 3
  step = 1
  segments = []
  labels = []
  for i in range(0, len(df_train) - N_TIME_STEPS, step):
      xs = df_train['x-axis'].values[i: i + N_TIME_STEPS]
      ys = df_train['y-axis'].values[i: i + N_TIME_STEPS]
      zs = df_train['z-axis'].values[i: i + N_TIME_STEPS]
      label = stats.mode(df_train['activity'][i: i + N_TIME_STEPS])[0][0]
      segments.append([xs, ys, zs])
      labels.append(label)
      print(i)
  
  print('Train sequences generated.')    
  np.array(segments).shape
  reshaped_segments = np.asarray(segments, dtype= np.float32).reshape(-1, N_TIME_STEPS, N_FEATURES)
  labels = np.asarray(pd.get_dummies(labels), dtype = np.float32)
  X_train = reshaped_segments
  y_train = labels
  print(X_train)
  
  ## TEST DATA - EACH GENERATED SEQUENCE MUST CONTAIN 200 TRAINING EXAMPLES
  N_TIME_STEPS = 200
  N_FEATURES = 3
  step = 1
  segments = []
  labels = []
  for i in range(0, len(df_test) - N_TIME_STEPS, step):
      xs = df_test['x-axis'].values[i: i + N_TIME_STEPS]
      ys = df_test['y-axis'].values[i: i + N_TIME_STEPS]
      zs = df_test['z-axis'].values[i: i + N_TIME_STEPS]
      label = stats.mode(df_test['activity'][i: i + N_TIME_STEPS])[0][0]
      segments.append([xs, ys, zs])
      labels.append(label)
      print(i)
  
  print('Test sequences generated.')    
  np.array(segments).shape
  reshaped_segments = np.asarray(segments, dtype= np.float32).reshape(-1, N_TIME_STEPS, N_FEATURES)
  labels = np.asarray(pd.get_dummies(labels), dtype = np.float32)
  X_test = reshaped_segments
  y_test = labels
  
  
  ## SPLITING THE DATA INTO TRAINING AND TEST (20%) SET
  ##X_train, X_test, y_train, y_test = train_test_split(reshaped_segments, labels, test_size=0.5, random_state=RANDOM_SEED)
  
  
  
  ##train_size = int(0.7 * segments)
  ##test_size = int(0.3 * segments)
  ##
  ##full_dataset = tf.data.TFRecordDataset(FLAGS.input_file)
  ##full_dataset = full_dataset.shuffle()
  ##train_dataset = full_dataset.take(train_size)
  ##test_dataset = full_dataset.skip(train_size)
  ##test_dataset = test_dataset.take(test_size)
  
  
  ## THE LSTM MODEL DEFINITION
  N_CLASSES = 2
  N_HIDDEN_UNITS = 64
  def create_LSTM_model(inputs):
      W = {
          'hidden': tf.Variable(tf.random_normal([N_FEATURES, N_HIDDEN_UNITS])),
          'output': tf.Variable(tf.random_normal([N_HIDDEN_UNITS, N_CLASSES]))
      }
      biases = {
          'hidden': tf.Variable(tf.random_normal([N_HIDDEN_UNITS], mean=1.0)),
          'output': tf.Variable(tf.random_normal([N_CLASSES]))
      }
  
      X = tf.transpose(inputs, [1, 0, 2])
      X = tf.reshape(X, [-1, N_FEATURES])
      hidden = tf.nn.relu(tf.matmul(X, W['hidden']) + biases['hidden'])
      hidden = tf.split(hidden, N_TIME_STEPS, 0)
  
      # Stack 2 LSTM layers
      lstm_layers = [tf.contrib.rnn.BasicLSTMCell(N_HIDDEN_UNITS, forget_bias=1.0) for _ in range(2)]
      lstm_layers = tf.contrib.rnn.MultiRNNCell(lstm_layers)
  
      outputs, _ = tf.contrib.rnn.static_rnn(lstm_layers, hidden, dtype=tf.float32)
  
      # Get output for the last time step
      lstm_last_output = outputs[-1]
  
      return tf.matmul(lstm_last_output, W['output']) + biases['output']
  print('LSTM model defined.')
  
  ## CREATE PLACEHOLDERS FOR THE MODEL
  tf.reset_default_graph()
  X = tf.placeholder(tf.float32, [None, N_TIME_STEPS, N_FEATURES], name="input")
  Y = tf.placeholder(tf.float32, [None, N_CLASSES])
  print('LSTM model placeholders defined.')
  
  ## CREATING THE MODEL
  pred_Y = create_LSTM_model(X)
  pred_softmax = tf.nn.softmax(pred_Y, name="y_")
  print('LSTM model created.')
  
  
  ## USE OF L2 REGULARIZATION
  L2_LOSS = 0.0015
  l2 = L2_LOSS * \
      sum(tf.nn.l2_loss(tf_var) for tf_var in tf.trainable_variables())
  loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits = pred_Y, labels = Y))
  
  
  ## OPTIMIZER
  LEARNING_RATE = 5e-5
  optimizer = tf.train.AdamOptimizer(learning_rate=LEARNING_RATE).minimize(loss)
  correct_pred = tf.equal(tf.argmax(pred_softmax, 1), tf.argmax(Y, 1))
  accuracy = tf.reduce_mean(tf.cast(correct_pred, dtype=tf.float32))
  
  
  # TRAINING
  N_EPOCHS = 5
  BATCH_SIZE = 1024
  ##batch_size = 10
  ##total_batchs = X_train.shape[0] ## batch_size
  
  saver = tf.train.Saver()
  
  history = dict(train_loss=[], 
                       train_acc=[], 
                       test_loss=[], 
                       test_acc=[])
  
  sess=tf.InteractiveSession()
  sess.run(tf.global_variables_initializer())
  
  train_count = len(X_train)
  
  for i in range(1, N_EPOCHS + 1):
      for start, end in zip(range(0, train_count, BATCH_SIZE),
                            range(BATCH_SIZE, train_count + 1,BATCH_SIZE)):
  
          sess.run(optimizer, feed_dict={X: X_train[start:end],
                                         Y: y_train[start:end]})
  
      _, acc_train, loss_train = sess.run([pred_softmax, accuracy, loss], feed_dict={
                                              X: X_train, Y: y_train})
  
      _, acc_test, loss_test = sess.run([pred_softmax, accuracy, loss], feed_dict={
                                              X: X_test, Y: y_test})
  
      history['train_loss'].append(loss_train)
      history['train_acc'].append(acc_train)
      history['test_loss'].append(loss_test)
      history['test_acc'].append(acc_test)
  
      print ("epoch: %d test accuracy: %f loss: %f" %(i,acc_test,loss_test))
  
  ##history = dict(train_loss=[], 
  ##                     train_acc=[], 
  ##                     test_loss=[], 
  ##                     test_acc=[])
  ##saver = tf.train.Saver()
  ##session = tf.Session()
  ##tf.global_variables_initializer().run(session=session)
  ##for epoch in range(N_EPOCHS):
  ##    cost_history = np.empty(shape=[1],dtype=float)
  ##    for b in range(total_batchs):
  ##        offset = (b * batch_size) % (y_train.shape[0] - batch_size)
  ##        batch_x = X_train[offset:(offset + batch_size), :, :, :]
  ##        batch_y = y_train[offset:(offset + batch_size), :]
  ##        _, c = session.run([optimizer, loss],feed_dict={X: batch_x, Y : batch_y})
  ##        cost_history = np.append(cost_history,c)
  ##    print ("Epoch: %d Training Loss: %f Training Accuracy %f" %(epoch,np.mean(cost_history),session.run(accuracy, feed_dict={X: X_train, Y: y_train})))
  ##print ("Testing Accuracy: %f" %(session.run(accuracy, feed_dict={X: X_test, Y: y_test})))
  
  
  
  predictions, acc_final, loss_final = sess.run([pred_softmax, accuracy, loss], feed_dict={X: X_test, Y: y_test})
  
  print()
  print ("final results: accuracy: %f loss: %f" %(acc_final,loss_final))
  
  
  ## STORE MODEL TO DISK
  pickle.dump(predictions, open("predictions.p", "wb"))
  pickle.dump(history, open("history.p", "wb"))
  tf.train.write_graph(sess.graph_def, '.', './checkpoint/har.pbtxt')  
  saver.save(sess, save_path = "./checkpoint/har.ckpt")
  sess.close()
  
  
  ## LOADING THE MODEL
  history = pickle.load(open("history.p", "rb"))
  predictions = pickle.load(open("predictions.p", "rb"))
  
  
  # MODEL EVALUATION
  plt.figure(figsize=(12, 8))
  plt.plot(np.array(history['train_loss']), "r--", label="Train loss")
  plt.plot(np.array(history['train_acc']), "g--", label="Train accuracy")
  plt.plot(np.array(history['test_loss']), "r-", label="Test loss")
  plt.plot(np.array(history['test_acc']), "g-", label="Test accuracy")
  plt.title("Training session's progress over iterations")
  plt.legend(loc='upper right', shadow=True)
  plt.ylabel('Training Progress (Loss or Accuracy values)')
  plt.xlabel('Training Epoch')
  plt.ylim(0)
  plt.show()
  
  
  # CONFUSION MATRIX
  LABELS = ["Falling", "NotFalling"];
  ##LABELS = ["Downstairs", "Falling", "Jogging", "Sitting", "Standing", "Upstairs", "Walking"];
  max_test = np.argmax(y_test, axis=1)
  max_predictions = np.argmax(predictions, axis=1)
  confusion_matrix = metrics.confusion_matrix(max_test, max_predictions)
  plt.figure(figsize=(12, 10))
  sns.heatmap(confusion_matrix, xticklabels=LABELS, yticklabels=LABELS, annot=True, fmt="d");
  plt.title("Confusion matrix")
  plt.ylabel('True label')
  plt.xlabel('Predicted label')
  plt.show();
  </code></pre>
  
  <p>Here is the code I used to export the model:</p>
  
  <pre><code>from tensorflow.python.tools import freeze_graph
  
  MODEL_NAME = 'har'
  
  input_graph_path = 'checkpoint/' + MODEL_NAME+'.pbtxt'
  checkpoint_path = './checkpoint/' +MODEL_NAME+'.ckpt'
  restore_op_name = "save/restore_all"
  filename_tensor_name = "save/Const:0"
  output_frozen_graph_name = 'frozen_'+MODEL_NAME+'.pb'
  
  freeze_graph.freeze_graph(input_graph_path, input_saver="",
                            input_binary=False, input_checkpoint=checkpoint_path, 
                            output_node_names="y_", restore_op_name="save/restore_all",
                            filename_tensor_name="save/Const:0", 
                            output_graph=output_frozen_graph_name, clear_devices=True, initializer_nodes="")
  </code></pre>
  
  <p>And this is the code I used on the Android App:</p>
  
  <pre><code>package com.example.daniel.app03;
  
  import android.content.Context;
  
  import org.tensorflow.contrib.android.TensorFlowInferenceInterface;
  
  
  public class TensorFlowClassifier {
      static {
          System.loadLibrary("tensorflow_inference");
      }
  
      private TensorFlowInferenceInterface inferenceInterface;
      private static final String MODEL_FILE = "file:///android_asset/frozen_har.pb";
      private static final String INPUT_NODE = "input";
      private static final String[] OUTPUT_NODES = {"y_"};
      private static final String OUTPUT_NODE = "y_";
      private static final long[] INPUT_SIZE = {1, 200, 3};
      private static final int OUTPUT_SIZE = 2;
  
      public TensorFlowClassifier(final Context context) {
          inferenceInterface = new TensorFlowInferenceInterface(context.getAssets(), MODEL_FILE);
      }
  
      public float[] predictProbabilities(float[] data) {
          float[] result = new float[OUTPUT_SIZE];
          inferenceInterface.feed(INPUT_NODE, data, INPUT_SIZE);
          inferenceInterface.run(OUTPUT_NODES);
          inferenceInterface.fetch(OUTPUT_NODE, result);
  
          //Downstairs    Falling     Jogging   Sitting   Standing    Upstairs    Walking
          return result;
      }
  }
  </code></pre>
  
  <p>Here are the loss and the accuracy graphs. Using all 7 classes:</p>
  
  <p><a href="https://i.stack.imgur.com/L7x0u.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/L7x0u.png" alt="Loss and accuracy evolution with 7 classes"></a></p>
  
  <p><a href="https://i.stack.imgur.com/zI9Ih.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/zI9Ih.png" alt="Confusion matrix with 7 classes"></a></p>
  
  <p>With just 2 classes ("Falling" and "NotFalling"):</p>
  
  <p><a href="https://i.stack.imgur.com/jqGIz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jqGIz.png" alt="Loss and accuracy evolution with 2 classes"></a></p>
  
  <p><a href="https://i.stack.imgur.com/SCq8h.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/SCq8h.png" alt="Confusion matrix with 2 classes"></a></p>
  
  <p><strong>Note: In this case I tried to use less ADL examples ("NotFalling") so there is less of a discrepancy.</strong></p>