Update README.md

README.md

@@ -23,34 +23,64 @@ The two versions are designed for different application scenarios.
 Jellyfish-13B is suitable for integration into larger data management systems due to its simple and clear responses that can be easily transformed into code.
 On the other hand, Jellyfish-13B-Interpreter is more user-oriented, with responses that provide them with in-depth data insights without the necessity for advanced coding skills or an intricate grasp of statistics.
 
-|  Task  | Dataset | Non-LLM SoTA<sup>1</sup> | GPT-3.5<sup>2</sup> | GPT-4<sup>2</sup> | Jellyfish-13B| Jellyfish-13B-Interpreter | Jellyfish-13B-1.1<sup>3</sup> | 
-| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | 
-| Entity Matching  | Fodors-Zagats  | 100  | 100 | 100 | 100 | 100 | 100 | 
-| Entity Matching  | Beer           | 94.37| 96.30 | 100 | 93.33 | 100 | 96.55 | 
-| Entity Matching  | iTunes-Amazon  | 97.06| 96.43 | 100 | 96.30 | 96.15 | 100 | 
-| Entity Matching  | Walmart-Amazon | 86.76| 86.17 | 90.27 | 80.71 | 85.16 | 89.18 | 
-| Entity Matching  | DBLP-ACM       | 98.99| 96.99 | 97.44 | 97.35 | 95.74 | 99.32 | 
-| Entity Matching  | DBLP-GoogleScholar | 95.60| 76.12 | 91.87 | 92.83 | 89.45 | 95.16 | 
-| Entity Matching  | Amazon-Google  | 75.58| 66.53 | 74.21 | 72.69 | 56.64 | 80.25 | 
-| Data Imputation  |  Restaurant    | 77.20| 94.19 | 97.67 | 94.19 | 93.02 | 93.02 | 
-| Data Imputation  |  Buy           | 96.50| 98.46 | 100 | 100 | 100 | 100 | 
-| Error Detection  |  Hosptial      | 99.10| 90.74 | 90.74 | 92.21 | 65.66 | 86.59 |
-| Error Detection  |  Adult         | 94.40| 92.01 | 92.01 | 96.62 | 90.13 | 99.20 | 
-| Schema Matching  |  Sythea        | 38.50| 57.14 | 66.67 | 36.36 | 30.77 | NA | 
-
+More details about the model can be found in the [Jellyfish paper](linktobeadded).
+
+## Performance on seen tasks
+
+|  Task  | Type | Dataset | Non-LLM SoTA<sup>1</sup> | GPT-3.5<sup>2</sup> | GPT-4<sup>2</sup> | Jellyfish-13B-1.1<sup>3</sup>| Jellyfish-13B-Interpreter |  
+| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |  
+| Entity Matching  | Seen | Fodors-Zagats  | 100  | 100 | 100 | 100 | 100 | 
+| Entity Matching  | Seen | Beer           | 94.37| 96.30 | 100 | 96.77 | 100 |  
+| Entity Matching  | Seen | iTunes-Amazon  | 97.06| 96.43 | 100 | 98.11 | 96.15 | 
+| Entity Matching  | Seen | DBLP-ACM       | 98.99| 96.99 | 97.44 | 98.98 | 95.74 |  
+| Entity Matching  | Seen | DBLP-GoogleScholar | 95.60| 76.12 | 91.87 | 98.51 | 89.45 |  
+| Entity Matching  | Seen | Amazon-Google  | 75.58| 66.53 | 74.21 | 81.34 | 56.64 |  
+| Entity Matching  | Unseen | Walmart-Amazon | 86.76| 86.17 | 90.27 | 89.42 | 85.16 |  
+| Entity Matching  | Unseen | Abt-Buy | 89.33 | -- | 92.77 | 89.58 | -- |  
+| Data Imputation  | Seen |  Restaurant    | 77.20| 94.19 | 97.67 | 94.19 | 93.02 |  
+| Data Imputation  | Seen |  Buy           | 96.50| 98.46 | 100 | 100 | 100 |  
+| Data Imputation  | Unseen |  Filpkart    | 68.00 | -- | 89.94 | 81.68 | -- |  
+| Data Imputation  | Unseen |  Phone       | 86.70| -- | 90.79 | 87.21 | -- |  
+| Error Detection  | Seen |  Hosptial      | 94.40| 90.74 | 90.74 | 95.59 | 65.66 |  
+| Error Detection  | Seen |  Adult         | 99.10| 92.01 | 92.01 | 99.33 | 90.13 |  
+| Error Detection  | Unseen |  Flights     | 81.00 | -- | 83.48  | 82.52 | -- |  
+| Error Detection  | Unseen |  Rayyan      | 79.00| -- | 81.95 | 90.65 | -- |  
+| Schema Matching  | Seen |  Sythea        | 38.50| 57.14 | 66.67 | 36.36 | 30.77 |  
+| Schema Matching  | Seen |  MIMIC        | 20.00| -- | 40.00 | 40.00 | -- |  
+| Schema Matching  | Unseen |  CMS        | 50.00| -- | 19.35 | 59.29 | -- |  
+
+_Few-shot is disabled for Jellyfish-13B on seen datasets and enabled on unseen datasets._   
 _Accuracy as the metric for data imputation and the F1 score for other tasks._  
 _For GPT-3.5, GPT-4 we used the few-shot approach, while for Jellyfish and Jellyfish-Interpreter, the zero-shot approach was employed._
+
 1.  
   [Ditto](https://arxiv.org/abs/2004.00584) for Entity Matching  
   [SMAT](https://www.researchgate.net/publication/353920530_SMAT_An_Attention-Based_Deep_Learning_Solution_to_the_Automation_of_Schema_Matching) for Schema Matching  
-  [HoloDetect](https://arxiv.org/abs/1904.02285) for Error Detection  
+  [HoloDetect](https://arxiv.org/abs/1904.02285) for Error Detection seen datasets  
+  [RAHA](https://dl.acm.org/doi/10.1145/3299869.3324956) for Error Detection unseen datasets
   [HoloClean](https://arxiv.org/abs/1702.00820) for Data Imputation  
 2.  
   [Large Language Models as Data Preprocessors](https://arxiv.org/abs/2308.16361)
-3. Jellyfish-13B-1.1 is set to be the next iteration of Jellyfish-13B and is presently under development. We showcase its performance at this stage to highlight its impressive potential. As demonstrated in the table, it has already outperformed non-LLM methods on the majority of benchmark datasets. We've optimized the training data for this 1.1 version, and its release is on the horizon.
+3. We have updated the main branch with Jellyfish-13B version 1.1 .
+
+## Performance on unseen tasks
+
+### Column Type Annotation
+
+| Dataset | RoBERTa (159 shots)<sup>1</sup> | GPT-3.5<sup>1</sup> | GPT-4 | Jellfish-13B-1.1 |  
+| ---- | ---- | ---- | ---- | ---- |  
+| SOTAB | 79.20 | 89.47 | 91.55 | 82.00 | 
+
+1. Results from [Column Type Annotation using ChatGPT](https://arxiv.org/abs/2306.00745)
 
+### Attribute Value Extraction
 
-**Jellyfish paper will be coming soon!**
+| Dataset |Stable Beluga 2 70B<sup>1</sup> | SOLAR 70B<sup>1</sup> | GPT-3.5<sup>1</sup> | GPT-4 <sup>1</sup>| Jellfish-13B-1.1 |  
+| ---- | ---- | ---- | ---- | ---- | ---- |   
+| AE-110k | 52.10 | 49.20 | 61.30 | 55.50 | 58.12 |  
+| OA-Mine | 50.80 | 55.20 | 62.70 | 68.90 | 55.96 |
+
+1. Results from [Product Attribute Value Extraction using Large Language Models](https://arxiv.org/abs/2310.12537)
 
 - **Developed by:** Haochen Zhang, Yuyang Dong, Chuan Xiao, Masafumi Oyamada
 - **Contact: [email protected]**
@@ -136,6 +166,15 @@ Attribute B is [name: {value of name}, description: {value of description}].
 Are Attribute A and Attribute B semantically equivalent? Choose your answer from: [Yes, No].
 ```
 
+### For Column Type Annotation
+
+We follow the prompt in [Column Type Annotation using ChatGPT](https://arxiv.org/abs/2306.00745) (text+inst+2-step).  
+
+### For Attribute Value Extraction
+
+We follow the prompt in [Product Attribute Value Extraction using Large Language Models](https://arxiv.org/abs/2310.12537) (textual, w/o examples).  
+
+
 ### JellyFish-13B-Interpreter
 #### For Entity Matching
 ```