Darshan Hiranandani : Implementing Website Search Functionality for Custom Chatbots Using RAG
Hi everyone,I’mDarshan Hiranandani, new to Hugging Face and have been exploring its capabilities. I’m really excited about the possibilities here!I’m wondering if it’s possible to replicate the “domain search” feature, similar to what’s available in HuggingChat, for my own custom chatbots. Essentially, I’d like to use a retrieval-augmented generation (RAG) approach.Is there an easy way to crawl or connect to data from a website URL for this purpose? If so, I’d appreciate it if you could share any relevant tools, examples, or guidance on how to set this up.Thanks so much for your help!RegardsDarshan Hiranandani
2025-01-13T09:08:48Z
Hi. The forum is currently under strict regulations, so I can’t post many links. Please search using keywords…In addition to the methods explained below, the framework called “smolagents” was recently released by HF and is proving popular. It’s probably closer to what you’re looking for.How to use website search functionality with my LLMBeginnersHi everyone, 
I’m a beginner who discovered Hugging Face a few days ago and I’m really impressed by what we can do here. 
I was wondering if it’s possible to replicate the “domain search” feature (like in HuggingChat) for my own custom chatbots, essentially using it as a RAG  approach. 
Is there a straightforward way to crawl or connect data from a website URL for that purpose? If so, could you please explain how or point me to any relevant tools or examples? 
Thanks a lot ! 
nyri[Capture d’e…
2025-01-14T08:48:42Z
Yes, it’s possible to replicate a “domain search” feature similar to HuggingChat by using a retrieval-augmented generation (RAG) approach. You can crawl or connect to data from a website URL using web scraping or APIs. Tools like BeautifulSoup and Scrapy for web scraping, or langchain for integrating external data sources into language models, can help with this. You can extract relevant information from the site, index it, and use it to enhance your chatbot’s responses. For RAG, combining a search index (like FAISS or ElasticSearch) with a language model (such as GPT) allows the chatbot to retrieve contextually relevant information before generating responses.