license:other size_categories:10K<n<100K format:json modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: other
license_name: other
license_link: LICENSE
---
## Base information

AEZAKMI V3 is build on top of AEZAKMI V2 but there are many new samples.

I removed all coding samples plus those with "BEGINCONTEXT ENDCONTEXT References:" as they were bugging out the training with longer sequence len. \
I included filtered no_robots_sharegpt dataset, which makes this dataset non-commercial only! From no_robots, I removed stories, mentions of AI, coding etc. \
I added wsb dataset, based on Sentdex/wsb_reddit_v001, but I removed all samples shorter than 300 or 500 chars (i forgot now)

Finally, I removed all samples longer than 10000 chars from the dataset - my thinking is that those 13k-15k chars samples would have been given bigger weight during training assuming you have sample packing enabled.
They would have taken up more space in one particular sample and could introduce some errors similar to what I noticed with coding and BEGINCONTEXT "contextual" parts of airoboros. I don't want my model writing code or starting some weird context out of the blue, and those longer samples did that.