size_categories:1K<n<10K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
dataset_info:
  features:
  - name: companyName
    dtype: string
  - name: description
    dtype: string
  splits:
  - name: train
    num_bytes: 561548
    num_examples: 2923
  download_size: 408522
  dataset_size: 561548
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
---

# List of Company name and Description from HackerNoon Tech Companh Dataset
[HackerNoon Tech Companh Dataset](https://huggingface.co/datasets/HackerNoon/tech-company-news-data-dump)

### Here is the code I use to transform
```
from datasets import load_dataset, Dataset

# Load the dataset
dataset = load_dataset('HackerNoon/tech-company-news-data-dump', split='train')

# Filter out rows where the "description" column is None or empty
filtered_dataset = dataset.filter(lambda x: x['description'] is not None and x['description'] != '')

# Extract the 'companyName' and 'description' columns
extracted_data = filtered_dataset.select_columns(['companyName', 'description'])

# Remove duplicates from the 'companyName' column while keeping corresponding 'description'
unique_data = {}
for row in extracted_data:
    if row['companyName'] not in unique_data:
        unique_data[row['companyName']] = row['description']

# Convert the unique data back to a Hugging Face Dataset
new_dataset = Dataset.from_dict({
    'companyName': list(unique_data.keys()),
    'description': list(unique_data.values())
})

# Push the new dataset to Hugging Face
new_dataset.push_to_hub('pacozaa/tech-company-news-data-dump-clean')

# Print the length of the new dataset
print(len(new_dataset))
```
