license:apache-2.0 region:us finance fine-tuning conversational-ai named-entity-recognition sentiment-analysis topic-classification rag multilingual lightweight-llm
---
license: apache-2.0
tags:
  - finance
  - fine-tuning
  - conversational-ai
  - named-entity-recognition
  - sentiment-analysis
  - topic-classification
  - rag
  - multilingual
  - lightweight-llm
---

# Finance-Instruct-500k Dataset

## Overview

**Finance-Instruct-500k** is a comprehensive and meticulously curated dataset designed to train advanced language models for financial tasks, reasoning, and multi-turn conversations. Combining data from numerous high-quality financial datasets, this corpus provides over **500,000 entries**, offering unparalleled depth and versatility for finance-related instruction tuning and fine-tuning.

The dataset includes content tailored for financial reasoning, question answering, entity recognition, sentiment analysis, address parsing, and multilingual natural language processing (NLP). Its diverse and deduplicated entries make it suitable for a wide range of financial AI applications, including domain-specific assistants, conversational agents, and information extraction systems.

Most entries include system, user and assistant fields.

Recent additions include:
- **[BAAI/IndustryInstruction_Finance-Economics](https://huggingface.co/datasets/BAAI/IndustryInstruction_Finance-Economics)**: Broader financial instructions and **Chinese** language coverage.
- **[Josephgflowers/Financial-NER-NLP](https://huggingface.co/datasets/Josephgflowers/Financial-NER-NLP)**: Advanced **XBRL tagging** and named-entity recognition examples.

![image/png](https://cdn-uploads.huggingface.co/production/uploads/6328952f798f8d122ce62a44/MgkW6-hDXoZPYbpVbH5f4.png)

---

## Key Features

- **Extensive Coverage**: Over 500,000 entries spanning financial QA, reasoning, sentiment analysis, topic classification, multilingual NER, and conversational AI.
- **Multi-Turn Conversations**: Rich dialogues emphasizing contextual understanding and reasoning.
- **Diverse Data Sources**: Includes entries from **Cinder**, **Sujet-Finance-Instruct-177k**, **Phinance Dataset**, **BAAI/IndustryInstruction_Finance-Economics**, **Josephgflowers/Financial-NER-NLP**, and many other high-quality datasets.
- **RAG-Formatted Data**: Retrieval-augmented generation (RAG) tasks include external data prepended to the `user` field for enhanced contextual understanding.
- **Deduplicated and Preprocessed**: Eliminates overlaps and irregular entries for cleaner and higher-quality data.
- **XBRL Tagging**: Includes structured finance entity labeling from **Financial-NER-NLP** for advanced extraction tasks.

---

**Future Plans** 1M! Like my work? Want to see more? Custom request? Message me on discord: joseph.flowers.ra Donate here: https://buymeacoffee.com/josephgflowers

---


## Supported Tasks and Use Cases

1. **Financial Question Answering**:
   - Contextual and direct-answer financial QA.
   - Multilingual QA and financial terminology explanation.

2. **Reasoning Tasks**:
   - Symbolic and numeric reasoning.
   - Portfolio analysis and investment strategy simulation.

3. **Conversational AI**:
   - Multi-turn dialogues to develop finance-specific assistants and advisors.

4. **Named Entity Recognition (NER)**:
   - Multilingual financial entity recognition.
   - XBRL tagging for structured finance data (via **Financial-NER-NLP**).
   - Address parsing and PII handling.

5. **Sentiment Analysis**:
   - Text classification as bullish, bearish, neutral, positive, or negative.
   - Entity-level sentiment analysis.

6. **Topic Classification**:
   - Categorization of financial texts into topics such as market trends, risk analysis, and economic events.

7. **Lightweight LLM Training**:
   - Domain-specific fine-tuning for smaller models in resource-constrained environments.

8. **RAG Applications**:
   - Seamless integration with external data using prepended context in the `user` field.

---

## Dataset Composition

The dataset is a deduplicated combination of the following sources filtered for finance-related entries or tasks:

1. **[alvanlii/finance-textbooks](https://huggingface.co/datasets/alvanlii/finance-textbooks)**
2. **[glaiveai/RAG-v1](https://huggingface.co/datasets/glaiveai/RAG-v1)**
3. **[instruction-pretrain/ft-instruction-synthesizer-collection](https://huggingface.co/datasets/instruction-pretrain/ft-instruction-synthesizer-collection)** (NewsQA, ConvFinQA, WikiTableQA)
4. **[gretelai/gretel-pii-masking-en-v1](https://huggingface.co/datasets/gretelai/gretel-pii-masking-en-v1)**
5. **[CohereForAI/aya_dataset (HotpotQA)](https://huggingface.co/datasets/CohereForAI/aya_dataset)**
6. **[CohereForAI/aya_dataset](https://huggingface.co/datasets/CohereForAI/aya_dataset)**
7. **[nvidia/OpenMathInstruct-1](https://huggingface.co/datasets/Nvidia-OpenMathInstruct)**
8. **[TIGER-Lab/WebInstructSub](https://huggingface.co/datasets/TIGER-Lab/WebInstructSub)**
9. **[glaiveai/glaive-code-assistant-v3](https://huggingface.co/datasets/glaiveai/glaive-code-assistant-v3)**
10. **[Open-Orca/1million-gpt-4](https://huggingface.co/datasets/Open-Orca/1million-gpt-4)**
11. **[Norquinal/claude_evol_instruct_210k](https://huggingface.co/datasets/Norquinal/claude_evol_instruct_210k)**
12. **[migtissera/Synthia-v1.3](https://huggingface.co/datasets/migtissera/Synthia-v1.3)**
13. **[meta-math/MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA)**
14. **[HuggingFaceTB/cosmopedia](https://huggingface.co/datasets/HuggingFaceTB/cosmopedia)**
15. **[Josephgflowers/PII-NER](https://huggingface.co/datasets/Josephgflowers/PII-NER)**
16. **[gbharti/finance-alpaca](https://huggingface.co/datasets/gbharti/finance-alpaca)**
17. **[ugursa/Yahoo-Finance-News-Sentences](https://huggingface.co/datasets/ugursa/Yahoo-Finance-News-Sentences)**
18. **[AdaptLLM/finance-tasks_Headline](https://huggingface.co/datasets/AdaptLLM/finance-tasks_Headline)**
19. **[ceadar-ie/FinTalk-19k](https://huggingface.co/datasets/ceadar-ie/FinTalk-19k)**
20. **[zeroshot/twitter-financial-news-topic](https://huggingface.co/datasets/zeroshot/twitter-financial-news-topic)**
21. **[dylanalloy/ehc-contrived-financial](https://huggingface.co/datasets/dylanalloy/ehc-contrived-financial)**
22. **[zeroshot/twitter-financial-news-sentiment](https://huggingface.co/datasets/zeroshot/twitter-financial-news-sentiment)**
23. **[financial_phrasebank](https://huggingface.co/datasets/financial_phrasebank)**
24. **[AdiOO7/llama-2-finance](https://huggingface.co/datasets/AdiOO7/llama-2-finance)**
25. **[amphora/lmsys-finance](https://huggingface.co/datasets/amphora/lmsys-finance)**
26. **[AdaptLLM/finance-tasks_ConvFinQA](https://huggingface.co/datasets/AdaptLLM/finance-tasks_ConvFinQA)**
27. **[KennNguyenDev/FiQA_Financial_Phrasebank_Combined](https://huggingface.co/datasets/KennNguyenDev/FiQA_Financial_Phrasebank_Combined)**
28. **[AdaptLLM/finance-tasks_FPB](https://huggingface.co/datasets/AdaptLLM/finance-tasks_FPB)**
29. **[yixuantt/FinEntity](https://huggingface.co/datasets/yixuantt/FinEntity)**
30. **[AdaptLLM/finance-tasks_FiQA_SA](https://huggingface.co/datasets/AdaptLLM/finance-tasks_FiQA_SA)**
31. **[nickmuchi/financial-classification](https://huggingface.co/datasets/nickmuchi/financial-classification)**
32. **[lighthouzai/finqabench](https://huggingface.co/datasets/lighthouzai/finqabench)**
33. **[adityamavle/FinRiskAnalysis](https://huggingface.co/datasets/adityamavle/FinRiskAnalysis)**
34. **[sujet-ai/Sujet-Finance-Instruct-177k](https://huggingface.co/datasets/sujet-ai/Sujet-Finance-Instruct-177k)**
35. **Cinder (custom multi-turn reasoning and parsing data)**
36. **[BAAI/IndustryInstruction_Finance-Economics](https://huggingface.co/datasets/BAAI/IndustryInstruction_Finance-Economics)**
37. **[Josephgflowers/Financial-NER-NLP](https://huggingface.co/datasets/Josephgflowers/Financial-NER-NLP)**

---

## Dataset Structure

### Data Fields

- **system**: Instruction or system message providing context.
- **user**: Query or input from the user, with RAG data prepended if applicable.
- **assistant**: Model-generated response or ground truth answer.

### Format

All entries are structured as:

- `system`, `user`, `assistant`

### Example Entry

```json
{
  "system": "You are a financial assistant. Answer the user's question accurately but keep it brief.",
  "user": "What is the difference between stocks and bonds?",
  "assistant": "Stocks represent ownership in a company, while bonds are a form of debt where the investor lends money to an entity."
}
```

---

## Languages

- **English**: Primary language of the dataset.
- **Chinese**: Increased coverage thanks to the **BAAI/IndustryInstruction_Finance-Economics** subset.
- **Other Multilingual Data**: Includes content from multilingual NER sources and the Aya datasets.

---

## Collection and Preprocessing

1. **Deduplication**: Removed over 60,000 duplicate entries across all source datasets.
2. **Data Cleaning**: Eliminated non-ASCII characters, fixed punctuation and spelling, and ensured consistent formatting.
3. **Dataset Merging**: Unified entries from various sources into a single cohesive dataset.
4. **Annotation**: Enhanced metadata for clarity and usability, including task types and system prompts.
5. **XBRL Tagging**: Integrated from **Financial-NER-NLP** subset, enabling structured labeling of financial instruments.

---

## Ethical Considerations

- **User Privacy**: All PII is synthetic and anonymized to ensure compliance with privacy standards.
- **Professional Use Only**: This dataset is not a substitute for certified financial guidance or professional advice.

---

## Limitations

- **Bias**: Coverage may skew toward certain financial sectors or topics based on dataset distribution.
- **Accuracy**: Outputs trained on this dataset require validation for critical financial applications.
- **Multilingual Support**: Non-English entries vary in volume, though recent additions (BAAI dataset) increase Chinese content.

---

## Citation

If you use this dataset, please cite:

```bibtex
@dataset{josephgflowers2025financeinstruct,
  title={Finance-Instruct-500k},
  author={Joseph G. Flowers},
  year={2025},
  url={https://huggingface.co/datasets/Josephgflowers/Finance-Instruct-500k}
}
```

---

## How to Load the Dataset

```python
from datasets import load_dataset

dataset = load_dataset("Josephgflowers/Finance-Instruct-500k")
print(dataset["train"][0])
```

---

## License

This dataset is released under the Apache 2.0 license.
