Explain about Mamba and Jamba
huggingface.coJambaSimpleChatBot - a Hugging Face Space by AI-basic-classA simple chat bot using JambaHi everyone!When I started writing the article, I didn’t know what category was for me. Because Mamba and Jamba are new tech.I try to understand the new tech, so watching videos and arxiv papaers. But I have some trouble to understand.Who can explain about Jamba and Mamba comparing transformer?I know these new tech will play new role in AI field instead of transformer.Here are some relevant urls.deeplearning.aiBuild Long-Context AI Apps with JambaBuild LLM apps that can process very long documents using the Jamba model.https://arxiv.org/pdf/2312.00752ai21.comFoundation modelsJamba Model | AI21: Powerful, efficient language model with a 256K context window, optimized for speed and enterprise applications.In jamba tutorial video, they mentioned that jamba is powerful to dealing with long context prompting. but I think RAG using vector DB is more powerful than jamba. Am I right?And I used jamba but as I mentioned above, RAG is more powerful than jamba.Here is the sample space.
2025-01-19T13:14:58Z
Hey !First and foremost, it’s completely normal to not understand those models in a one shot because they rely on very specific principles and it is impossible to clearly vulgarize them according to me.To give you some tools to further deepen the resaearch, Mamba and Jamba both share a common ground which is : Selective State Space Models (SSMs). The architecture differs from transformers as it does not rely on attention. It is a middle ground between RNN and CNN according to me.Like RNNs:They maintain an internal state that gets updated sequentiallyThey can handle variable-length sequencesThey have a form of memory that carries information forwardLike CNNs:They can process data in parallelThey use convolution-like operationsThey’re efficient to train and can leverage hardware accelerationThe key difference that makes SSMs special is how they handle state updates:Unlike RNNs, which can suffer from vanishing/exploding gradients, SSMs use a more stable state update mechanismUnlike CNNs, which have a fixed receptive field, SSMs can theoretically capture dependencies of any length through their stateThey can be implemented efficiently using parallel hardware while maintaining the ability to process sequential informationAn analogy might help: If you think of:RNNs as being like reading a book one word at a time and keeping notesCNNs as being like looking at different parts of an image through sliding windowsThen SSMs are like scanning a document while maintaining a running summary, but being able to do multiple scans in parallelThe problem is that SSMs don’t outperform transformers because they have a sequence length issue which alters their abilities to capture information between two very distant sentences. Hence, Jamba proposed to stack transformers and SSMs blocks. The mixture should be the best of both worlds. However, it is still not the best model as combining both advantages comes also with combining both drawbacks.Also, RAG is not a model so you cannot compare a RAG to any LLMs. A RAG is a framework which uses an LLM at some point but the whole retrieval part is done by using encoders which are not LLMs.Hope this helps !
2025-01-19T23:40:09Z
Help for your answering.I think you are misunderstanding of my question.I mean in retrieval problem, what is better what is useful? Vector DB using LLM(transformers) or Jamba?In this aspect, I think that Vector DB is better then Jamba.
2025-01-20T09:36:33Z
Vector DB is not better than Jamba because it’s simply two different things. A Vector DB is a storage where you feed in embeddings produced after encoding a lot of resources that will be later used for retrieval. Jamba does not produce such encoding because it is “just” an LLM, something trained to generate the response. To do good retrieval you need an encoder and those are for example : sentence bert, MB25, ModernBert…So VectorDB != Jamba
2025-01-20T13:12:47Z
Then you mean that if I want to chat with the large amount of data(e.g)private company’s data), at that time I should use jamba not vectorDB.I used two methods, Vector DB + openai is better than only Jamba & jamba + vector DB!
2025-01-20T13:44:25Z
Yes use a Vector DB and Open AI for a rag it will be probably better than Jamba as the benchmarks are saying.