Double Headed Bloom for Dialogue Tasks
I’m trying to replicate this article using bloom as the model to fine-tune instead of a GPT model:Medium – 2 Sep 20🦄 How to build a State-of-the-Art Conversational AI with Transfer-LearningHow you can use Transfer Learning to build a State-of-the-Art chatbot based on OpenAI GPT and GPT-2 Transformer language modelsReading time: 12 min readHowever, bloom doesn’t seem to have a double head framework available.So I figured I’d need to create a custom head on top of the BloomForCausalLM, and the closest I’ve found to this is this thread and article:How do I change the classification head of a model?🤗TransformersThank you! This fixed my problem too! 
It was weird that this didn’t work: 
AutoModelForSequenceClassification.from_pretrained(“huggingface/CodeBERTa-language-id”, num_labels=15) 
but this did: 
config = AutoConfig.from_pretrained(“huggingface/CodeBERTa-language-id”) 
config.num_labels = 15 
model = AutoModelForSequenceClassification.from_config(config) 
Answering to tolgayan, the point is that this gets trained, you’re fine tuning the model.Medium – 26 Jan 22Adding Custom Layers on Top of a Hugging Face ModelLearn how to extract the hidden states from a Hugging Face model body, modify/add task-specific layers on top of it and train the whole…Reading time: 5 min readI understand that blenderbot and dialoGPT are available models, but from what I’ve seen and tested they’re not the most impressive. I’d like to see what I can do on a custom data set of my own design.I think what I’m doing should be feasible, but I’m entirely self-taught in programming → machine learning → deep learning, and my method is usually to start with a goal, then figure out how to get there. If someone could at minimum tell me if what I’m doing isn’t even possible, that would save me time continuously trying to figure it out.Any other help is welcome from learning materials to guidance on this project.Thanks!