license:mit size_categories:100K<n<1M format:csv modality:text library:datasets library:dask library:mlcroissant library:polars region:us
---
license: mit
---

# CodeSemSim: A Benchmark for Evaluating Code Functional Understanding

CodeSemSim is a benchmark designed to evaluate the ability of language models (LMs) to comprehend and differentiate between code snippets based on **functional similarity** rather than mere syntactic resemblance. This benchmark is created to address the critical gap in existing code similarity metrics, which often fail to capture subtle functional changes that can negatively impact program behavior.

## Overview

Developers rely on code understanding tools powered by LLMs for assistance in generating and refining code. However, incomplete comprehension of code functionality by these models can lead to sub-optimal or incorrect code suggestions. CodeSemSim helps evaluate and improve the **functionality understanding** of LMs by providing a dataset of carefully curated examples, including:

1. **Original Code Snippets**
2. **Functionally Equivalent Paraphrased Versions**
3. **Functionality-Altered Perturbed Versions**

The benchmark uses an **Abstract Syntax Tree (AST) manipulation pipeline** to generate code variations systematically, ensuring functional and syntactic diversity.

## Dataset Structure

The dataset consists of triplets and accompanying metadata stored in the following columns:

- **`original_func_code_string`**: The original code snippet.
- **`paraphrased_func_code_string`**: A functionally equivalent paraphrased version of the original code.
- **`perturbed_func_code_string`**: A perturbed version of the original code with functionality-altering transformations (e.g., early returns, condition reversal).
- **`paraphrasing_meta_info`**: Information about the paraphrasing transformations applied.
- **`perturbed_meta_info`**: Information about the perturbations applied.

This dataset can evaluate LMs' ability to detect and understand functional equivalence and differences between snippets.

## Key Features

- **Functionally Focused**: Evaluates similarity based on functionality rather than syntax.
- **Diverse Transformations**:
  - **Paraphrasing**: Retains functionality while rephrasing syntax (e.g., variable renaming, loop reordering).
  - **Perturbations**: Introduces functionality-altering changes to highlight subtle mistakes.
- **Customizable Benchmarking**: Can be used to evaluate or fine-tune LMs for better code functionality understanding.

## Applications

- **Evaluation**: Assess the performance of code similarity metrics such as CodeBLEU, CodeBERTScore, and LLM-based evaluations on functionality comprehension.
- **Training**: Train LLMs to improve their understanding of functional aspects of code, enabling more robust and reliable code generation.

## Data Source and Inspiration

- The dataset uses source code from the [CodeSearchNet](https://github.com/github/CodeSearchNet) dataset.
- Paraphrasing techniques and transformations are inspired by NatGen (@inproceedings{chakraborty2022natgen}), with modifications and extensions to support functionality-focused evaluations.

  ```bibtex
  @inproceedings{chakraborty2022natgen,
      author = {Chakraborty, Saikat and Ahmed, Toufique and Ding, Yangruibo and Devanbu, Premkumar T. and Ray, Baishakhi},
      title = {NatGen: Generative Pre-Training by “Naturalizing” Source Code},
      year = {2022},
      isbn = {9781450394130},
      publisher = {Association for Computing Machinery},
      address = {New York, NY, USA},
      url = {https://doi.org/10.1145/3540250.3549162},
      doi = {10.1145/3540250.3549162},
      booktitle = {Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering},
      pages = {18–30},
      numpages = {13},
      keywords = {Neural Network, Semantic Preserving Transformation, Source Code Transformer, Source Code Pre-training},
      location = {Singapore, Singapore},
      series = {ESEC/FSE 2022}
  }

 