task_categories:text-generation language:en license:mit size_categories:10M<n<100M region:us Scanpath Explanation
---
license: mit
task_categories:
- text-generation
language:
- en
tags:
- Scanpath
- Explanation
size_categories:
- 10M<n<100M
---

# GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths
GazeXplain is a novel study that goes beyond predicting where people look; it demands models to explain them in natural language, weaving a narrative thread that connects fixations to their underlying meaning.

The repository contains the datasets of the explanations of visual scanpaths in three different scanpath datasets (OSIE, AiR-D, COCO-Search18).

- Free-viewing: the prediction of scanpath for looking at some salient or important object in the given image. (OSIE)
- Visual Question Answering:  the prediction of scanpath during human performing general tasks, e.g., visual question answering, to reflect their attending and reasoning processes. (AiR-D)
- Visual search: the prediction of scanpath during the search of the given target object to reflect the goal-directed behavior under target present and absent conditions. (COCO-Search18 Target-Present and Target-Absent)

Citation
------------------
If you use our code or data, please cite our paper:
```text
@inproceedings{xianyu:2024:gazexplain,
    Author         = {Xianyu Chen and Ming Jiang and Qi Zhao},
    Title          = {GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths},
    booktitle      = {Proceedings of the European Conference on Computer Vision (ECCV)},
    Year           = {2024}
}
```
