license:cc-by-nc-nd-3.0 size_categories:n<1K format:json modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: cc-by-nc-nd-3.0
---
# Introduction

Limitus has developed an AI agent designed to tackle complex planning tasks that involve both sequential and parallel actions. To ensure its effectiveness, we aim to rigorously evaluate the agent through a carefully designed benchmarking process. This benchmark will assess the agent's ability to handle a wide variety of tasks requiring sophisticated planning and optimization strategies.

## Benchmark Design

To evaluate the agent's proficiency, we propose the following benchmark framework:

- **Task Complexity:**  
  The benchmark will include a diverse set of tasks with varying levels of complexity, characterized by the number of steps, their durations, and specific ordering constraints. This will test the agent’s ability to handle both simple and intricate task structures.

- **Optimal Plan Calculation:**  
  For each task, the agent will be required to compute the shortest possible time to completion, effectively managing sequential and parallel task execution. This ensures the agent can prioritize efficiency while maintaining logical coherence.

- **Dynamic Constraints:**  
  Tasks will incorporate dynamic elements, such as resource limitations, tool dependencies, and time-sensitive operations, to evaluate the agent's adaptability to changing scenarios.

## Key Evaluation Metrics

We will measure the agent's performance using the following metrics:

- **Node F1 Score:**  
  Assesses the precision and recall in aligning task nodes with the expected task graph.

- **Tool F1 Score:**  
  Evaluates the accuracy of tool selection and usage during task execution.

- **Structural Similarity Index (SSI):**  
  Quantifies the overall fidelity of task graphs, ensuring both node and edge similarities are preserved.

- **Node Label Similarity:**  
  Determines the semantic similarity of nodes between the actual and expected task graphs using cosine similarity.

- **Graph Edit Distance (GED):**  
  Calculates the number of edits required to transform one graph into another, providing insight into structural differences.

- **Time-to-Plan (TTP):**  
  Measures the time the agent takes to generate a complete task plan, reflecting its efficiency under varying task complexities.

## Conclusion

This benchmark framework will enable a rigorous evaluation of Limitus's AI agent, testing its ability to generate efficient, accurate, and adaptable plans across a wide range of scenarios. The insights gained will guide further refinements, ensuring the agent is robust and reliable for real-world applications.