task_categories:summarization license:apache-2.0 size_categories:10K<n<100K region:us
---
license: apache-2.0
task_categories:
- summarization
size_categories:
- 10K<n<100K
---
Researchers have successfully applied large language models (LLMs) such as ChatGPT to reranking in an information retrieval context, but to date, such work has mostly been built on proprietary models hidden behind opaque API endpoints. This approach yields experimental results that are not reproducible and non-deterministic, threatening the veracity of outcomes that build on such shaky foundations. We have organized re-ranking data from [RankGPT](https://github.com/sunnweiwei/RankGPT) and released it onto ModelScope for supporting the training and testing of re-ranking LLMs.

|   File  | Note |
|:-|:-|
| marco-train-10k.jsonl | 10K queries sampled from MS MARCO |
| marco-train-10k-gpt3.5.json |  Permutations predicted by ChatGPT   |
|marco-train-100k.jsonl | 100K queries from MS MARCO |
| marco-train-100k-gpt3.5.json | Permutations by ChatGPT of the 100K queries |

Prompt template reference [rankllm](https://github.com/castorini/rank_llm/blob/main/src/rank_llm/rerank/rank_gpt.py).