Parameter count on mamba models
Hi everyone, I am still learning about this model and there is something I havent managed to wrap my head around: How do you calculate the number of parameters of the model?From what I understand, the mamba block has 3ED^2 + D parameters from the linear projections and the RMSNorm per layer. Then, the embedding and LM head parts both have D * Vocab size.So from the config params we would get:layers * 3E * D^2 + 2D*V= 24 * 3 * 2 * 768^2 + 2 * 768 * 50277 = 162,178,560The model card shows 130M, what am I missing?