How to clean 8217 pictures from the similar one
@John6666Dear John and other huggingface community members,Do you guys know where to quickly clean from the similar image and labeled your image for dataset purposes? Thanks in advance.
2024-11-28T03:25:45Z
I know how to automatically tag images in detail to a certain extent, but I don’t know how to process large image data sets concisely, so I searched around for a solution.It seems there are several specialized tools.How to clean/audit your image data?🤗DatasetsFor the folks working on real-world CV projects or generative AI, I think you will find this open-sourced Python library useful that detects common issues (in famous datasets too) such as images which are: blurry, under/over-exposed, low information, oddly sized, or (near) duplicates of others.CleanVisionis a new package you can use to quickly audit any image dataset (including HuggingFace Datasets) for a broad range of common issues lurking in real-world data. Instead of relying on manual in…Medium – 4 Sep 23Cleaning Image Classification Datasets With fastdup and Renumics SpotlightCombining quick, automatic detection of data issues with powerful visualization to get high-quality image classification datasets, fast.Reading time: 6 min readgithub.comGitHub - cleanlab/cleanvision: Automatically find issues in image datasets and...Automatically find issues in image datasets and practice data-centric computer vision.
2024-11-28T03:30:27Z
Exactly what I need, that for some reason didn’t appear in my search engine. Will try this and reply if I find difficulties, thanks as always John6666
2024-11-28T03:32:32Z
Is SafeSearch turned on, or are your personal settings being applied? I searched using Bing.
2024-11-28T03:32:55Z
Then probably it’s google that sucks these days or just not lucky. I almost want to clean these data manually
2024-11-28T03:37:32Z
That’s right. Google search used to be amazing…Even now, Google is the most powerful when it comes to setting detailed search options. (Like site:huggingface.co)I usually just use Bing or DuckDuckGo, but I sometimes use this when I’m in a bind.https://searx.bndkt.io/If even that doesn’t work, there are also paid services by someone out there, but I don’t have anything that I want to search for that much…
2024-11-28T04:12:03Z
One more thing tho, which part of thesehuggingface.convidia/segformer-b5-finetuned-cityscapes-1024-1024 at mainWe’re on a journey to advance and democratize artificial intelligence through open source and open science.that is the hf dataset identifier mentioned in this line of codeimport json
from huggingface_hub import REDACTED_TOKEN_download

repo_id = f"datasets/{REDACTED_TOKEN_identifier}"
filename = "id2label.json"
id2label = json.load(open(REDACTED_TOKEN_download(repo_id=REDACTED_TOKEN_identifier, filename=filename, repo_type="dataset"), "r"))
id2label = {int(k): v for k, v in id2label.items()}
label2id = {v: k for k, v in id2label.items()}

num_labels = len(id2label)in this tutorialFine-Tune a Semantic Segmentation Model with a Custom DatasetI some how an expert in making things in huggingface tutorial doesn’t work, is it because that dataset doesn’t have the label? Thanks in advance
2024-11-28T04:41:23Z
huggingface.cosegments/sidewalk-semantic · Datasets at Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.GATED AGAIN…
2024-11-28T04:44:46Z
Don’t worry I don’t want to use that gated dataset … is the cityscape useable?
2024-11-28T04:47:35Z
nvidia/segformer-b5-finetuned-cityscapes-1024-1024 is amodel. not a dataset… So, I guess you can’t use it from the datasets library?Edit:dataset one is this?huggingface.cohuggan/cityscapes · Datasets at Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2024-11-28T05:20:55Z
What is the difference though, and what should I do if I want to do pretrain with bigger model to train on smaller dataset to “fine tune it”. I think I got confused of it
2024-11-28T05:39:01Z
What is the difference thoughIn the extreme, the only difference is whether you choose a dataset or a model when creating a new repo in HF. You can basically put anything in either, soHowever, it is usually more convenient to put the models that have not been fine-tuned or have been fine-tuned in the model repo, and to put the datasets in the datasets repo, so most people do this.I think the datasets library is designed to use the datasets in the datasets repo. So you need to find the datasets in the datasets repo. Probably.
2024-11-28T06:31:53Z
But can I fine tune with my own dataset with either of them or only with models? Is that what is called pretraining?
2024-11-28T06:36:07Z
Of course you can. Basically, I think it’s a matter of training a model with a data set, saving it to a new model repo, and repeating the process. You can also overwrite it instead of creating a new one, but in any case, I think that’s the method HF is assuming.So you should be able to train it further using nvidia’s as a base model and save it for your own use.I don’t really know the definition of pretraining…
2024-11-28T06:51:12Z
segments.ai is way too slow, does anyone know any guide to finetune with unlabelled images?
2024-11-28T09:24:53Z
Idk why I edited previous message for this question. I’ll make it a reply to that.segments.ai is way too slow, does anyone know any guide to finetune with unlabelled images?
2024-11-28T11:58:31Z
Whether or not this is possible will depend on what you train the model on.For example, if you train the model for a classification task, you need to tell it what it is classifying…You could use the image folder name as a label, but it would be difficult to do without anything…
2024-11-28T12:16:34Z
I find guides that let me uses image processing mask instead of classification-neural-network way of masking. I might wanna try that, better than nothing…
2024-11-28T13:50:36Z
Well, unless you have a specific end goal in mind, I think it’s a good idea to try out different things.I’ve heard that the quality of the data set and its labels has a direct impact on the efficiency and accuracy of the model training.But that’s something you can think about when you’re training the main model at the end. There’s not much point in improving the performance of the model you’re training on a trial basis…