Powerful Knowledge Chatbot with Firecrawl and RAG

In this tutorial, you'll learn how to build a custom knowledge chatbot using Lamatic.ai, Firecrawl, and RAG (Retrieval-Augmented Generation).

What you'll Build

• A chatbot using Lamatic.ai Studio
• A chat widget powered by LLM
• RAG & Firecrawl for smart data retrieval

Understanding the Components

Before we begin, let's go over the key technologies:

• Firecrawl (opens in a new tab) A web crawling service that systematically browses and collects data from websites. It extracts structured content, follows links, and processes various content types.

• RAG (Retrieval-Augmented Generation): A technique that enhances Large Language Models (LLMs) by retrieving relevant knowledge from a database, improving response accuracy and relevance.

Architecture Overview

Our chatbot consists of three main flows:

1. Firecrawl Flow

This flow handles:

1. Accepting website URLs via an API
2. Setting crawling parameters (e.g., depth, filters)
3. Extracting content from web pages
4. Sending collected data to the Webhook Processing Flow

2. Webhook Processing Flow

This flow processes and stores the crawled data:

1. Receiving data from Firecrawl
2. Cleaning and structuring text content
3. Creating vector embeddings for efficient search
4. Storing processed data in a vector database

3. Chatbot Flow

This flow enables user interaction and intelligent responses:

1. Managing user input through a chat interface
2. Retrieving relevant context using RAG
3. Generating responses using the LLM
4. Handling errors and fallback scenarios

Getting Started

1. Project Setup

1. Sign up at Lamatic.ai (opens in a new tab) and log in.
2. Navigate to the Projects and click New Project or select your desired project.
3. You'll see different sections like Flows, Context, and Connections

2. Setting Up the Firecrawl Flow

1. Click Create New Flow in Lamatic.ai Studio.
2. Name your flow Website Crawler.
3. Select API Request as the trigger type Node and define the schema:

   { "url": "string" }
4. Add a Crawler Node, then enter your Firecrawl API key.
5. Click the "+" icon on the API Request Node to select and provide the website URL for crawling.
6. Use Async mode for the crawler to fetch all pages sequentially, one by one.
7. Configure the Webhook Notification (this will be set up in the next step).
8. Ensure that the Webhook Events value is set to "page" in the crawler configuration options.
9. Save the Node and Test the node.

3. Creating the Webhook Processing Flow

1. Create a new flow named Firecrawl Webhook.

2. Set Webhook as the trigger type to fetch data from Firecrawl Website Flow.

3. Add a Conditional Node to check page data.

     {
       "operator": null,
       "operands": [
         {
           "name": "{{triggerNode_1.output.type}}",
           "operator": "==",
           "value": "crawl.page"
         }
       ]
     }

4. Use a Logic Node to parse data before vectorization. Example

   let triggerData = {{ triggerNode_1.output.data }}
    
   let vectorData = "title: " + triggerData[0].metadata.title + " \n description: " + triggerData[0].metadata.description + " \n url: " + triggerData[0].metadata.url + " \n sourceURL: " + triggerData[0].metadata.sourceURL + " \n markdown: " + triggerData[0].markdown
    
   let metaData = {
     rawHtml: triggerData[0].rawHtml,
     markdown: triggerData[0].markdown,
     title: triggerData [0].metadata.title,
     description: triggerData [0].metadata.description,
     url: triggerData[0].metadata.url,
     sourceURL:triggerData [0].metadata.sourceURL,
   }
    
   output = {"vectorData":[vectorData],"metaData":[metaData]};

5. Add a Vectorize Node by selecting an embedding model for conversion, then choose the vector data by clicking the plus icon.

6. Add an Indexing Node to store vectorized data in a vector database. Choose your preferred vector database and ensure that each entry includes the vector, metadata, and the primary key url.

7. Save and publish the flow.

8. Link the Webhook to the Firecrawl Flow. Save and publish the Firecrawl Flow.

9. Check the Webhook Processing Flow to verify the data received from the Firecrawl Node.

4. Building the Chatbot Flow

1. Create a new flow named RAG Chatbot
2. Add a Chat Interface Node.
3. Configure the Chat Widget and whitelist the Domain
4. Add a RAG Node, configure the database, and connect it to an LLM.

5. Testing

1. Run the Firecrawl Flow (the first flow) to collect data from the website.
2. Ensure the webhook flow is deployed before testing the first flow.
3. After execution, you should see the vectors stored in the vector database.
4. Test the Chatbot Flow to verify that it generates accurate responses.

6. Deployment and Integration

1. Click Deploy to make the chatbot live.
2. To integrate the chatbot into your website:
   1. Click Setup to access the embed code.
   2. Paste the code into your website’s HTML.
   3. Ensure allowed domains are configured before deployment Configure the Chat Widget.

7. Troubleshooting

• Make sure the apiUrl is set correctly to your pod's URL.
• Confirm that the workflowId matches the ID of your intended workflow.
• Inspect the browser console for any errors related to script loading or widget initialization.

Congratulations! You've completed the tutorial and created your custom knowledge chatbot using Lamatic.ai!