title: Correct practice and approach for reporting the training and generalization performance
body: <p>I am trying to learn the correct procedure for training a neural network for classification. Many tutorials are there but they never explain how to report for the generalization performance. Can somebody please tell me if the following is the correct method or not. I am using first 100 examples from the fisheriris data set that has labels 1,2 and call them as <code>X</code> and <code>Y</code> respectively. Then I split <code>X</code> into <code>trainData</code> and <code>Xtest</code> with a 90/10 split ratio. Using <code>trainData</code> I trained the NN model. Now the NN internally further splits <code>trainData</code> into tr,val,test subsets. My confusion is which one is usually used for generalization purpose when reporting the performance of the model to unseen data in conferences/Journals?
  The dataset can be found in the link: <a href="https://www.mathworks.com/matlabcentral/fileexchange/71468-simple-neural-networks-with-k-fold-cross-validation-manner" rel="nofollow noreferrer">https://www.mathworks.com/matlabcentral/fileexchange/71468-simple-neural-networks-with-k-fold-cross-validation-manner</a></p>
  
  <pre><code>rng('default')
  load iris.mat;
  
  X = [f(1:100,:) l(1:100)];
  
  numExamples = size(X,1);
  indx = randperm(numExamples);
  X = X(indx,:);
  Y = X(:,end);
  
  
  split1 = cvpartition(Y,'Holdout',0.1,'Stratify',true); %90% trainval 10% test
  
  istrainval = training(split1); % index for fitting
  istest = test(split1);      % indices for quality assessment
  
  trainData = X(istrainval,:);
  
  Xtest = X(istest,:);
  Ytest = Y(istest);
  
  
  numExamplesXtrainval = size(trainData,1);
  
  indxXtrainval = randperm(numExamplesXtrainval);
  trainData = trainData(indxXtrainval,:);
  Ytrain = trainData(:,end);
  
  hiddenLayerSize = 10;
  
  % data format = rows = number of dim, column = number of examples
  net  = patternnet(hiddenLayerSize);
  net  = init(net);
  net.performFcn = 'crossentropy';
  net.trainFcn = 'trainscg';
  net.trainParam.epochs=50;
  
  [net tr]= train(net,trainData', Ytrain');
  Trained = sim(net, trainData');  %outputs predicted labels
  
  train_predict = net(trainData');
  
  performanceTrain = perform(net,Ytrain',train_predict)
  lbl_train=grp2idx(Ytrain);
  Yhat_train = (train_predict &gt;= 0.5);
  Lbl_Yhat_Train = grp2idx(Yhat_train);   
  [cmMatrixTrain]=  confusionmat(lbl_train,Lbl_Yhat_Train )
  
  accTrain=sum(lbl_train ==Lbl_Yhat_Train)/size(lbl_train,1);
  disp(['Training Set:    Total Train Acccuracy by MLP = ',num2str(100*accTrain ), '%'])
  
  [confTest] =  confusionmat(lbl_train(tr.testInd),Lbl_Yhat_Train(tr.testInd) )
  
  
  %unknown test
  test_predict = net(Xtest');
  
  performanceTest = perform(net,Ytest',test_predict);
  Yhat_test = (test_predict &gt;= 0.5);
  test_lbl=grp2idx(Ytest);
  Lbl_Yhat_Test = grp2idx(Yhat_test);
  
  [cmMatrix_Test]=  confusionmat(test_lbl,Lbl_Yhat_Test )
  </code></pre>
  
  <p>This is the output. </p>
  
  <p>Problem1: There seems to be no prediction for the other class. Why? </p>
  
  <p>Problem2: Do I need a separate dataset like the one I created as <code>Xtest</code> for reporting generalization error or is it the practice to use the data <code>trainData(tr.testInd,:)</code> as the generalization test set? Did I create an unnecessary subset?</p>
  
  <pre><code>performanceTrain =
  
     2.2204e-16
  
  
  cmMatrixTrain =
  
      45     0
      45     0
  
  Training Set:    Total Train Acccuracy by MLP = 50%
  
  confTest =
  
       9     0
       5     0
  
  
  cmMatrix_Test =
  
       5     0
       5     0
  </code></pre>