Powerful Knowledge Chatbot with Firecrawl and RAG

In this tutorial, you'll learn how to build a custom knowledge chatbot using Lamatic.ai, Firecrawl, and RAG (Retrieval-Augmented Generation).

What you'll Build

• A chatbot using Lamatic.ai Studio
• A chat widget powered by LLM
• RAG & Firecrawl for smart data retrieval

Understanding the Components

Before we begin, let's go over the key technologies:

• Firecrawl (opens in a new tab) A web crawling service that systematically browses and collects data from websites. It extracts structured content, follows links, and processes various content types.

• RAG (Retrieval-Augmented Generation): A technique that enhances Large Language Models (LLMs) by retrieving relevant knowledge from a database, improving response accuracy and relevance.

Architecture Overview

Our chatbot consists of three main flows:

1. Firecrawl Flow

This flow handles:

1. Accepting website URLs via an API
2. Setting crawling parameters (e.g., depth, filters)
3. Extracting content from web pages
4. Sending collected data to the Webhook Processing Flow

2. Webhook Processing Flow

This flow processes and stores the crawled data:

1. Receiving data from Firecrawl
2. Cleaning and structuring text content
3. Creating vector embeddings for efficient search
4. Storing processed data in a vector database

3. Chatbot Flow

This flow enables user interaction and intelligent responses:

1. Managing user input through a chat interface
2. Retrieving relevant context using RAG
3. Generating responses using the LLM
4. Handling errors and fallback scenarios

Getting Started

1. Project Setup

1. Sign up at Lamatic.ai (opens in a new tab) and log in.
2. Navigate to the dashboard and click Create New Flow.

2. Setting Up the Firecrawl Flow

1. Click Create New Flow in Lamatic.ai Studio.
2. Name your flow Website Crawler.
3. Select API Request as the trigger type Node and define the schema:

   { "trigger": "bool" }
4. Add a Crawler Node, then enter your Firecrawl API key.
5. Provide a website URL for crawling.
6. Configure the Webhook Notification (this will be set up in the next step).
7. Save the Node and Test the node.

3. Creating the Webhook Processing Flow

1. Create a new flow named Crawler Data Processor.

2. Set Webhook as the trigger type and fetch data from Firecrawl.

3. Add a Conditional Node to check page data.

4. Use a Logic Node to parse data before vectorization.

   Example

       let vectorData = "title: " + {{ triggerNode_1.output.data }}[0].metadata.title + " \n description: " + {{ triggerNode_1.output.data }}[0].metadata.description + " \n url: " + {{ triggerNode_1.output.data }}[0].metadata.url + " \n sourceURL: " + {{ triggerNode_1.output.data }}[0].metadata.sourceURL + " \n markdown: " + {{ triggerNode_1.output.data }}[0].markdown
    
       let MetaData = {
       rawHtml: {{ triggerNode_1.output.data }}[0].rawHtml,  
       markdown: {{ triggerNode_1.output.data }}[0].markdown,
       title: {{ triggerNode_1.output.data }} [0].metadata.title,
       description: {{ triggerNode_1.output.data }} [0].metadata.description,
       url: {{ triggerNode_1.output.data }}[0].metadata.url,
       sourceURL:{{ triggerNode_1.output.data }} [0].metadata.sourceURL,
       }
       output = {"vectorData":[vectorData],"MetaData":[MetaData]};

5. Add a Vectorize Node, selecting an embedding model for conversion.

6. Add an Indexing Node to store vectorized data in a vector database.

7. Save and publish the flow.

8. Link the Webhook to the Firecrawl Flow. Save and publish the Firecrawl Flow.

9. Check the Webhook Processing Flow to verify the data received from the Firecrawl Node.

4. Building the Chatbot Flow

1. Create a new flow named RAG Chatbot or use the RAG Chatbot Template.
2. Add a Chat Interface Node.
3. Add a RAG Node, configure the database, and connect it to an LLM.

5. Testing Chatbot

1. Run the Firecrawl Flow to collect website data.
2. Test the Chatbot Flow to ensure accurate responses.

6. Deployment and Integration

1. Click Deploy to make the chatbot live.
2. To integrate the chatbot into your website:
   1. Click Setup to access the embed code.
   2. Paste the code into your website’s HTML.
   3. Ensure allowed domains are configured before deployment.

Ensure that the allowed domains are specified before integrating this chat widget.

Congratulations! You've completed the tutorial and created your custom knowledge chatbot using Lamatic.ai!