language:ko size_categories:n<1K format:json modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
language:
- ko
---

# Ko-Bench

## 1. Introduction 

**Ko-Bench** is a benchmark designed for evaluating the performance of Korean language models. Following the translation of the [MT-Bench](https://github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge/data/mt_bench/question.jsonl) dataset into Korean, we modified the questions to incorporate elements specific to Korea and its characteristics.

- leaderboard : https://huggingface.co/spaces/davidkim205/ko-bench
- github : https://github.com/davidkim205/Ko-Bench

## 2. Advantages

To assess whether Korea’s geography and culture can be effectively represented in Korean, we modified the existing dataset by replacing 'Hawaii' with '제주도', a prominent tourist destination in Korea. 

Additionally, we adjusted the translation or interpretation of foreign terms such as 'casual' and 'limerick' to their Korean equivalents and evaluated the model using queries that are appropriate for the Korean language.

In the Roleplay evaluation, we substituted foreign figures such as 'Elon Musk' and 'Sheldon' with well-known Korean personalities, including '유재석' and '천송이' from the drama '별에서 온 그대', and employed their distinctive speaking styles. This adjustment aimed to determine if the model could accurately generate content in Korean.

We also endeavored to create a dataset that reflects Korean conventions by modifying elements such as monetary units, names, variables, company names, and job titles to align with Korean standards.

## 3. Examples

| Category | MT-Bench | Ko-Bench |
| :---: | ----- | ----- |
| <b>Writing</b> |  |  |
| 1st Turn | Compose an engaging travel blog post about a recent trip to Hawaii, highlighting cultural experiences and must-see attractions. | 최근 제주도 여행을 다녀오면서 꼭 가봐야 할 명소를 강조하는 재미있는 여행 블로그 글을 작성하시오. |
| 2nd Turn | Rewrite your previous response. Start every sentence with the letter A. | 이전 응답을 다시 작성하시오. 모든 문장은 '제'로 시작하도록 하시오.  |
|  |  |  |
| 1st Turn  | Help me construct a catchy, yet scientifically accurate, headline for an article on the latest discovery in renewable bio-energy, while carefully handling the ethical dilemmas surrounding bio-energy sources. Propose 4 options.  | 재생 가능한 바이오 에너지의 최신 발견에 관한 기사의 눈길을 끌면서도 과학적으로 정확한 헤드라인을 작성하는 동시에 바이오 에너지원을 둘러싼 윤리적 딜레마를 주의 깊게 처리할 수 있도록 4가지의 헤드라인을 제안하세요.  |
| 2nd Turn | Alter your previous response. Make the following adjustments to the 2nd option: 1\. Make the tone sound casual 2\. Embed an advertisement for a company called \\"FlexPower\\" 3\. Fewer than 10 words.  | 이전 답변을 변경하세요. 이전에 제시한 헤드라인 중 두 번째 옵션을 다음과 같이 조정합니다. 1\. 말투를 일상적이고 친근하게 합니다. 2\. \\"최강전력\\"이라는 회사에 대한 광고를 삽입합니다. 3\. 10어절 미만입니다.  |
| <b>Roleplay</b> |  |  |
| 1st Turn | Pretend yourself to be Elon Musk in all the following conversations. Speak like Elon Musk as much as possible. Why do we need to go to Mars?  | 다음 대화에서 자신이 유재석인 척 해보세요. 가능한 한 유재석처럼 말하십시오. 우리가 왜 즐겁게 살아야 하나요?  |
| 2nd Turn | How do you like dancing? Can you teach me?  | 춤추는 걸 좋아해요? 가르쳐 주실 수 있나요?  |
| <b>Math</b> |  |  |
| 1st Turn  | Benjamin went to a bookstore and purchased a variety of books. He bought 5 copies of a sci-fi novel, each priced at $20, 3 copies of a history book priced at $30 each, and 2 copies of a philosophy book for $45 each.\\nWhat was the total cost of his purchases?  | 지민은 서점에 가서 다양한 책을 구입했습니다. 그는 각 권당 20000원인 공상과학 소설 5권, 각 30000원인 역사서 3권, 각 45000원에 철학 책 2권을 구입했습니다. 그가 구입한 총 비용은 얼마입니까?  |
| 2nd Turn  | Suppose Benjamin decides to sell each of these books at a 25% markup from the price he purchased them. What would be his total revenue if he sold all the books he bought?  | 지민이 각 책을 구입한 가격에서 25% 인상된 금액으로 판매하기로 결정했다고 가정해 보겠습니다. 만일 그가 구입한 책을 모두 판매한다면 그의 총 수익은 얼마일까요?  |