Returned Tensors and Hidden State
Hi, just quickly getting started with GPT2.fromhttps://huggingface.co/gpt2:from transformers import GPT2Tokenizer, GPT2Model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2Model.from_pretrained('gpt2')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)is said to yield the features of the text.Upon inspecting the output, it is an irregularly shaped tuple with nested tensors. Looking at the source code for GPT2Model, this is supposed to represent the hidden state. I can guess what some of these dimensions represent, for example the768dimension is obviously the word embedding, but in general I can’t find any documentation about interpreting the information inoutputI also tried adding:output = model(**encoded_input, output_attentions = True)but I do not know how to interpret the dimensions of this either.I am told to “Seeattentionsunder returned tensors for more detail.” in the docstring athttps://huggingface.co/transformers/_modules/transformers/modeling_gpt2.html#GPT2ModelBut I cannot find what this is referring to. Can someone help me interpret the dimensions of these nested tuples?
2020-08-27T06:37:24Z
Please refer toGPT2 docs. It will give a detailed description of whatGPT2Modelis supposed to return.It returns (in order of output):last_hidden_state:(batch_size, sequence_length, hidden_size)past:(2, batch_size, num_heads, sequence_length, embed_size_per_head)hidden_statesattentions
2020-08-28T01:11:31Z
Can’t believe I missed this
2020-08-28T15:48:40Z
Hey@azhx,if you are onmasterthen you can also use theModelOutputobject, which is adictlike object which lets you access the output asout.attentions,out.hidden_statesetc. ForGPT2Modelmodel, it returnsBaseModelOutputWithPast. You can find the docshere
2020-09-05T06:18:35Z
Yes, I’m on master now and using this, thanks!