Dataset for fake news detection, fine tune or pre-train
Is there any dataset for fake news (different from sentiment analysis) detection? I have one NELA-GT but then I would need to pre-train that from scratch?Any methods, am I on the correct page ?https://huggingface.co/transformers/training.htmlI want to use BERT model,thanks
2020-10-07T16:18:48Z
You could try to get baseline with fine-tuning before going for pre-training and then make decision based on the results.This thread has nice pointer for pre-trainingPre-Train BERT (from scratch)ResearchBERT has been trained on MLM and NSP objective. I wanted to train BERT with/without NSP objective (with NSP in case suggested approach is different). I haven’t performed pre-training in full sense before. Can you please share how to obtain the data (crawl and tokenization details which were used) on which BERT was trained on ?. Since it takes a lot of time, I am looking for well tested code that can yield the BERT with/without NSP in one go. Any suggestions will be helpful. 
I know about some pr…
2020-10-07T18:32:08Z
thanks, so it seems I need to know the difference between pre-training and fine-tuning?
2020-10-08T10:46:09Z
This resources should helpjalammar.github.ioThe Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning)Discussions:
Hacker News (98 points, 19 comments), Reddit r/MachineLearning (164 points, 20 comments)


Translations: Chinese (Simplified), French 1, French 2, Japanese, Korean, Persian, Russian, Spanish

2021 Update: I created this brief and highly...
2020-10-08T13:23:46Z
thanks , yours information is always very useful
2020-10-08T14:22:34Z
if I am correct, the pre-training on any corpus is unsupervised, by that I mean the text is large amount without any labels, however in fine-tuning, we should have labels?
2020-10-12T15:38:41Z
Yes, in modern nlp, the models are pre-trained  using unspervised objective (maksed lanaguge modeling, auto-regressive LM, document denosing etc).And the downstream tasks (classification, QA) etc are supervised. Again, the above resources should help you understand the difference better.
2020-10-12T15:41:25Z
@valhallathanks, you have so much knowledge, thanks for sharing with newbies.