BERT for NER output of only '0'
Hi everyone! I’d really appreciate your help with an issue I’m having with BERT for NER in a highly specialized domain. I’m using BERT for token classification and used much of the format from here (Named entity recognition with Bert) as inspiration. I had to generate my own B/I labeling for my dataset, which looks fine - some samples are below. Text lengths were also often longer than BERT max tokens of 512 so I had a chunk a given text piece into smaller chunks (of which at the start I’d place a [CLS] token. I didn’t really add [SEP] tokens on any of these chunked pieces as they’d have padding to reach the length of 512. The labels, input id’s, padding and attention masks appear fine and I’ve also pasted some samples below. I’ve also checked the output numerous times - again my code is below for the predict function. I had to aggregate the byte level predictions to come up with labels at the original word level and did this by averaging over the byte level logit outputs if a word was split by the tokenzier (this is done by the aggregate function). Another idiosyncratic part of my predict function is I had to know indices in the original text that correspond to tokens - the first part of the predict function is just storing those values for the original ‘words’. What’s weird is if I train the BERT model on a handful of data samples (for instance, <10), I’ll get some output that has none ‘0’ predictions. Training over a larger set seems to result in only ‘0’ predictions. Really don’t understand what’s going on at all and would appreciate any help - thank you so much![… ‘P’, ‘##H’, ‘##Y’, ‘##SI’, ‘##CI’, ‘##AN’, ‘IN’, ‘##TE’, ‘##RP’, ‘##RE’, ‘##TA’, ‘##TI’, ‘##ON’, ‘:’, ‘N’, ‘##eg’, ‘##ative’, ‘:’, ‘Tu’, ‘##mor’, ‘cells’, ‘showing’, ‘no’][…‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘B-HER2’, ‘B-HER2’, ‘B-HER2’, ‘B-HER2’, ‘I-HER2’, ‘I-HER2’, ‘I-HER2’, ‘I-HER2’, ‘I-HER2’]def predict(test_texts, test_note_ids, model, tokenizer):
    offset = 0
    model.eval()
    for test_text, test_note_id in zip(test_texts, test_note_ids):
        num_notes += 1
        chunkified_tokenized_texts, _, _ = chunkify_note(test_text, None, test_note_id)
        predictions = []
        for split_text in chunkified_tokenized_texts:
            tokenized_test_text = tokenizer.encode(split_text)
            tok_list = []
            for m in re.finditer(r'\S+', split_text):
                token = m.group(0)
                sub_tokens = tokenizer.tokenize(token)
                tok_list.append((token, offset+m.start(), offset+m.end() - 1))
            offset = len(split_text)

            if torch.cuda.is_available():
                input_ids = torch.tensor([tokenized_test_text]).cuda()
            else:
                input_ids = torch.tensor([tokenized_test_text])
            with torch.no_grad():
                output = model(input_ids)
            if torch.cuda.is_available():
                scores = output[0].detach().cpu().numpy()
            else:
                scores = output[0].detach().numpy()

            pred = aggregate(scores, tokenizer, tok_list)

            predictions.extend(pred) #then do some extra processing to store thisInput Id’s on the input:[  101  1239  1104   115   115   115   115   115   117   115   115   115115   115   115   115   115   115   115  4183  2524   115   115   115115   115   115   115   115   115   115   115   115   115   115   115117   115   115   115   115   115   115   115   115   115   115   115115   115   115   115   115   115   115   115   115   117  9292   1177735   115   115   115   115   115   115   115   115   115   115   117115   115   115   115   115   115   115   115   115   115   118   115115   115   115   115   115   115   115   115   115   115   115   115115   115  2524 11341   131   113   115   115   115   115   115   114115   115   115   115   115   118   115   115   115   115   115   115115   115   115   115   115   115   115   115   115   119   115   115115   115   115   117  9292   117  7735   143  7897   131   113   115115   115   115   115   114   115   115   115   115   115   118   115115   115   115   115  9666  9970 18653 11922   131   115   115   115115   115   137   115   115   115   115   115   119   115   115   115115   115   115   115   115   115   115   115   115   115   115   115115   115   115   115   115   117  7735   115   115   115   115   115115   115   115   115   115   117  9292   117  7735   150 13901  82312591 10783  2069  8544 24162 13901  2346  2349  3663   155 16668  95651942  7195  9080 10208   131   115   115   115   115   115   117   115115   115   115   115   115   115   115   115   115   108   131   115115   115   115   115   118   115   115   115   115   115  2508  1181119 11336  1665   119   108   131   115   115   115   115   115   15926868  1204   108   131   115   115   115   115   115  2516 14265   1311429   120  1407   120  1446 26316   131  1492  2162  9850   131  971411336 21437   131  1367   120  5037   120  1446   140 19526   131   115115   115   115   115   141  2346  2064   131  1429   120  1429   1202679   113  4936   131  4062   114  7642  6834 27989  1389   113   188114   131   115   115   115   115   115   115   115   115   115   115119   115   115   115   115   115   113   113   115   115   115   115115   114   115   115   115   115   115   118   115   115   115   115115   114   115   115   115   115   115   115   115   115   115   115113   113   115   115   115   115   115   114   115   115   115   115115   118   115   115   115   115   115   114  2687  7136  6902   13115075 10584  1179   118  4275   117 18311 16274   118 11783  7918  11132525 15766   119   115   115   115   115   115   118   115   115   115115   115   115   115   115   115   115   131  4114  7209 15961   1451775   131   115   115   115   115   115   119  5539  1580   115   115115   115   115   115   115   115   115   115  7277  1643 17489 272591118   143  6258  3048 10722 23904  9664 18082  2069  3663   153  30483663 13882 19747 14962 15969 12880 20336 16941  9159 21669 11414   131151 12606  5838   131 17037 26271  3652  4000  1185     0     0     00     0     0     0     0     0     0     0]Tagged values:[ 9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  9  99  9  9  9  9  9  9  9  9  9  9  9  6  6  6  6  4  4  4  4  4 27 27 2727 27 27 27 27 27 27 27]Attention mask:[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]