How to load an old (ko)bert .pt file, and convert the code that was supposed to run the old model into modern transformers/huggingface code
hey everyone, I essentially have code that was written for the old kobert (korean bert) and gluonnlp python libraries. I also have a .pt model file.I would like to be able to run the trained model using huggingface.The details:The training seems to have been done on one of the kobert models (skt/kobert-base-v1 · Hugging Faceor maybemonologg/kobert · Hugging Face) with these hyper parameters:max_len = 512batch_size = 6warmup_ratio = 0.1num_epochs = 20max_grad_norm = 1log_interval = 20learning_rate = 5e-6num_workers = 2n_splits = 5and this category list:category_list = [“화장품”, “패션”, “요리음식”, “여행아웃도어”, “인테리어”, “엔터테인먼트”, “육아”, “아이티”, “자동차”, “헬스/피트니스”, “반려동물”]e.g. it has been trained to take a single sentence and return a relevant category.a dataset class has been defined like this:from torch.utils.data import Datasetimport gluonnlp as nlp…class BERTDataset(Dataset):definit(self, dataset, bert_tokenizer, max_len,pad, pair):transform = nlp.data.BERTSentenceTransform(bert_tokenizer, max_seq_length=max_len, pad=pad, pair=pair)self.sentences = []

    for data in dataset:
        if len(data)<=max_len:
            self.sentences.append(transform([data]))
        else:
            self.sentences.append(transform([data[:max_len]]))
    

def __getitem__(self, i):
    return (self.sentences[i])

def __len__(self):
    return (len(self.sentences))from what I can tell, beyond the ugly code, this is just equivalent to adding “truncation=True” in a tokenizer, but I’m not 100% sure.a classifier has been defined which seems to just create the equivalent of ‘token_type_ids’ in a tokenizer:from torch import nn…class BERTClassifier(nn.Module):definit(self,bert,hidden_size = 768,num_classes=11,dr_rate=None,params=None):super(BERTClassifier, self).init()self.bert = bertself.dr_rate = dr_rateself.classifier = nn.Linear(hidden_size , num_classes)
    if dr_rate:
        self.dropout = nn.Dropout(p=dr_rate)

def gen_attention_mask(self, token_ids, valid_length):
    attention_mask = torch.zeros_like(token_ids)
    for i, v in enumerate(valid_length):
        attention_mask[i][:v] = 1
    return attention_mask.float()

def forward(self, token_ids, valid_length, segment_ids):
    attention_mask = self.gen_attention_mask(token_ids, valid_length)
    
    _, pooler = self.bert(input_ids = token_ids, token_type_ids = segment_ids.long(), attention_mask = attention_mask.float().to(token_ids.device))
    if self.dr_rate:
        out = self.dropout(pooler)
    return self.classifier(out)and a tokenizer & initial model are loaded:from kobert.utils import get_tokenizerfrom kobert.pytorch_kobert import get_pytorch_kobert_model…bertmodel, vocab = get_pytorch_kobert_model()tokenizer = get_tokenizer()tok = nlp.data.BERTSPTokenizer(tokenizer, vocab, lower=False)threshold = 5.26device = torch.device(“cuda:0”)model_name=“…\kobertbest_512.pt”modelbest = torch.load(model_name, map_location=device)modelbest.to(device)modelbest.eval()then the model is used like this:def GetMediaCategory(captionlist):for i in range(len(captionlist)):print(captionlist[i])print(type(captionlist[i]))captionlist[i] = unicodedata.normalize(‘NFC’,captionlist[i])captionlist[i] = ’ ‘.join(re.compile(’[가-힣]+').findall(captionlist[i]))if len(captionlist[i]) == 0:captionlist[i] = ‘기타’datalist = BERTDataset(captionlist, tok, max_len, True, False)
test_dataloader = torch.utils.data.DataLoader(datalist, batch_size=batch_size, num_workers=num_workers)
gc.collect() 
wholeout=[]
wholevalue=[]

for batch_id,(token_ids, valid_length, segment_ids) in enumerate(notebook.tqdm(test_dataloader)):
    token_ids = token_ids.long().to(device)
    segment_ids = segment_ids.long().to(device)
    valid_length= valid_length
    outlist = []
    valuelist = []
    out = modelbest(token_ids, valid_length, segment_ids)
    for outi in out:
        valuelist.append(outi.max().tolist())
        if outi.max().tolist() > threshold:
            outlist.append(categorylist[outi.argmax()])
        else:
            outlist.append('기타')
    wholeout+=outlist
    wholevalue+=valuelist

return wholeout, wholevalueagain, please excuse the shitty code. I’ve refactored it, but since I couldn’t run the original or refactored code, it felt wrong to share undebugged code. presumably this was debugged by the original authorsanyhow, I hope this is enough information to be able to tell how to load and use the trained model, using the huggingface library ecosystem. If more of the original code would be useful, I’ll gladly upload it, I just thought that these pieces might be the most relevant ones, if at all.