Is ConvBertModel autoregressive?
Can I use ConvBertModel  as a decoder ?I setis_decoder  = True. So I assume it should apply autoregressive mask.It converged very quickly on a training dataset.But when I generate sequence inserting tokens one by one it gives wrong result. I suspect that autoregressive mask is either not aplied or disregarded upon convolution (information from consecutive tokens leaks regradless mask).