Update README.md

README.md

@@ -13,7 +13,7 @@ base_model:
 #### Description
 The provided model was trained to respond to incorrect student answers in an interactive reading comprehension exercise setting. Incorrect student answers can become valuable learning opportunities, provided that the student understands where they went wrong and why. To this end, rather than being given the correct answer, students should receive elaborated feedback on how to correct a mistake on their own. Highlighting the complex demands that the generation of such feedback places on a model's input utilization abilities, we proposed two extensions to the training pipeline. Firstly, we employed a KL regularization term between a standard and enriched input format to achieve more targeted input representations. Secondly, we added a preference optimization step to encourage student answer-adaptive feedback generation.
 
-#### Evaluation Results
+#### Automatic Evaluation Results
 The final model was trained and evaluated on all feedback turns from the DIRECT and DIRECT-Feedback datasets partially available at https://github.com/DIRECTDataset/DIRECTFeedback/blob/main/data/feedback_data_partial.csv
 
 | BLEU | METEOR | ROUGE | BERTScore |
@@ -23,6 +23,14 @@ The final model was trained and evaluated on all feedback turns from the DIRECT
 
 For additional details we refer the reader to our paper.
 
+#### Manual Evaluation Results
+We sampled 250 items for the joined DIRECT+DIRECT-F feedback set and had one of the authors of this paper manually evaluate the generated feedback.
+
+| appropriate (verification, explanation and hint feedback) | direct (correct feedback) | irrelevant or ambigue | unfaithful (contradicting the passage or alluding to an incorrect answer) |
+| :---: | :---: | :---: | :---: |
+| | | | |
+| 43.6% | 23.6% | 22% | 10.8% |
+
 #### Execution
 Code and instructions on how to perform inference on the model are provided at https://github.com/DIRECTDataset/DIRECTFeedback