Repost: Wikipedia (or something else) text to input output
Repost because my original post is being hide by bots:Hey, I’m new to topics related to LLMs. I’m not sure if this is the right category, but I’ve come across many datasets containing texts from sources like Wikipedia, the web, books, and documentation. However, I’m unsure how to train a chatbot model using text that doesn’t include question-and-answer pairs.Additionally, I’m curious about how companies like OpenAI and Meta train their models using text from the internet.One more question (I know I’m asking a lot): how can I speed up training a text classifier model on a dataset with 441k rows?Apologies for any mistakes; English isn’t my native language (I speak French).
2024-11-18T01:56:11Z
That bot thing was a disaster.If you’re not sure about the category, it’s fine to use ‘Beginners’. I’ve never really paid attention to the category…I don’t know the details of the LLM training, so I’ll get someone else.@not-lainYou’re probably the right person for this question.
2024-11-18T12:59:47Z
Hi@Bigfoot302this blogpost might interest you :huggingface.coFine-Tuning Gemma Models in Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.Additionally, I’m curious about how companies like OpenAI and Meta train their models using text from the internet.these companies train their AI models on 2 to 3 steps :pretraining : feeding the AI vast amounts of text data (ie wikipedia, math, poems, facts, …) just to teach the AI how to talk and some common general knowledgeinstruction tuning : in here we examples of conversations to the AI to teach it how to reply to instructions or conversations, this also helps to teach the AI when to stop talking instead of generating a long paragraph (example when I sayHii expect the AI to sayHelloinstead of creating an wikipedia like reply, a bad example of a reply without finetuning is the followingHi is a word people use to greet each other ...preferance optimization and further enhancementsOne more question (I know I’m asking a lot): how can I speed up training a text classifier model on a dataset with 441k rows?I would say get a big GPU and increase the batch size, else get multiple GPUs and setup multiple GPU parallelism, the second method is a bit complex, so try to research more about it in advance or use the trainer API (already used in the blogpost above) which comes with multiple GPU support by default.Hope this helps
2024-11-18T17:47:55Z
Thank you for the reply.not-lain:Pretraining: feeding the AI vast amounts of text data (e.g., Wikipedia, math, poems, facts, etc.) to teach the AI how to communicate and gain some general knowledge.How can I “feed” a model with text? As far as I know, when training a model, I needinput_idsandattention_maskfor the inputs andlabelsfor the outputs. However, I don’t see how it’s possible to train a model using just text.Edit: I see that if I leave the input blank and put the text I want in the output, it works.