size_categories:1K<n<10K format:parquet modality:image modality:text library:datasets library:dask library:mlcroissant library:polars region:us
---
dataset_info:
  features:
  - name: label
    dtype: string
  - name: question
    dtype: string
  - name: answer
    dtype: string
  - name: image
    dtype: image
  splits:
  - name: Comprehensive_VQA_MME
    num_bytes: 582782493.636
    num_examples: 2066
  download_size: 369339159
  dataset_size: 582782493.636
configs:
- config_name: default
  data_files:
  - split: Comprehensive_VQA_MME
    path: data/Comprehensive_VQA_MME-*
---

Every item has 2 T/F questions (one True and one False) - only if these two are both correct, acc_score += 1.

Here is the function that you can use:

```python
from sklearn.metrics import accuracy_score, precision_score, recall_score, confusion_matrix


def process_result(preds, gts):
  """
  This func. is working for only one task (items with same labels)
  """
  cnt = 0
  acc_plus_correct_num = 0
  for pred, gt in zip(preds, gts):
    if pred == gt:
      cnt += 1
    if cnt == 2:
      acc_plus_correct_num += 1
      cnt = 0

  # cal TP precision acc, etc.
  metric_dict = compute_metric(preds, gts)
  acc_plus = acc_plus_correct_num / len(gts) * 2
  metric_dict["acc_plus"] = acc_plus
  
  return metric_dict


def compute_metric(preds, gts):
  assert len(gts) == len(preds)
  label_map = {
      "yes": 1,
      "no": 0,
      "other": -1,
  }
  
  gts = [label_map[x] for x in gts]
  preds = [label_map[x] for x in preds]
  
  acc = accuracy_score(gts, preds) 
  
  clean_gts = []
  clean_preds = []
  other_num = 0 
  for gt, pred in zip(gts, preds):
      if pred == -1:
          other_num += 1
          continue
      clean_gts.append(gt)
      clean_preds.append(pred)
  
  
  conf_mat = confusion_matrix(clean_gts, clean_preds, labels=[1,0])
  precision = precision_score(clean_gts, clean_preds, average='binary')
  recall = recall_score(clean_gts, clean_preds, average='binary')
  tp, fn = conf_mat[0]
  fp, tn = conf_mat[1]
  
  metric_dict = dict()
  metric_dict = {
      "TP": tp,
      "FN": fn,
      "TN": tn,
      "FP": fp,
      "precision": precision,
      "recall": recall,
      "other_num": other_num,
      "acc": acc,
  }

  return metric_dict


if __name__ == "__main__":
  score = 0

  eval_type_dict = {
      "Perception": ["existence", "count", "position", "color", "posters", "celebrity", "scene", "landmark", "artwork", "OCR"],
      "Cognition": ["commonsense_reasoning", "numerical_calculation", "text_translation", "code_reasoning"]
  }
  
  # Notice that we have different tasks (labels) here
  for preds, gts in zip(tasks["pred"], tasks["gts"]):
    metric_dict = process_result(preds, gts)
    for k, v in metric_dict.items():
      if k in ["acc", "acc_plus"]:
          score += v*100
  print(f"The total score is: {score}")
```