Firecrawl Integration

No sections found for this integration

The integration documentation may not have the expected structure

Overview

Firecrawl is a robust tool designed to transform websites into LLM-ready data by leveraging its Crawler and Scraper functionalities. Whether you need to map website structures or extract specific data, Firecrawl provides a seamless and customizable solution.

⚠️

To use the Firecrawl node, you need to create a separate flow for crawling and scraping data. You can then utilize a webhook and integrate it into this distinct flow in the async node, or get the results directly in the sync node.

Features

✅ Key Functionalities

Web Crawling: Systematically browse and index websites, discovering and mapping their structure.
Web Scraping: Extract targeted content from specific web pages using customizable rules.
Integration with Webhooks: Receive real-time updates about crawling and scraping activities.
Dynamic Content Handling: Support for waiting on dynamic page loads and simulating mobile devices.

✅ Benefits

Generate structured data for language models.
Customize inclusion and exclusion of website sections.
Handle both static and dynamic web content.

Prerequisites

Before using Firecrawl, ensure the following:

A valid Firecrawl API Key (opens in a new tab).
Access to the Firecrawl service host URL.
Properly configured credentials for Firecrawl.
A webhook endpoint for receiving notifications (required for the crawler).

⚠️

For Self Hosting,If the connection fails, whitelist the following IPs: https://www.cloudflare.com/ips/ (opens in a new tab)

Setup

Step 1: Obtain API Credentials

Register on Firecrawl (opens in a new tab).
Generate an API key from your account dashboard.
Note the Host URL and Webhook Endpoint.

Step 2: Configure Firecrawl Credentials

Use the following format to set up your credentials:

Key Name	Description	Example Value
Credential Name	Name to identify this set of credentials	`my-firecrawl-creds`
Firecrawl API Key	Authentication key for accessing Firecrawl services	`fc_api_xxxxxxxxxxxxx`
Host	Base URL where Firecrawl service is hosted	`https://api.firecrawl.dev`

Configuration Reference

Now, a crawler has two modes :

Async
Sync

In asynchronous mode, the crawler will run in the background and notify you via a webhook when the crawl is complete.

Whereas in synchronous mode, the crawler will run in the foreground and return results directly.

In case of sync mode, the results will be returned as follows :

{
    "success": true,
    "status": "completed",
    "completed": 1,
    "total": 1,
    "creditsUsed": 1,
    "expiresAt": "2025-05-15T08:12:44.000Z",
    "data": [
        {
        "markdown": "Docs\n\nContributing\n\n# Contributing to Lamatic.ai Documentation\n\nWe're thrilled that you're interested in contributing to the Lamatic.ai documentation! Your efforts help improve the experience for all users of our platform. This guide will walk you through the process of contributing.\n\n## Getting Started [Permalink for this section](https://lamatic.ai/docs/contributing\\#getting-started)\n\n1. **Understand Our Docs Structure**\n   - Our documentation is built using Nextra, a Next.js and MDX-powered static site generator. To get a better understanding of how our docs work, visit our repository at [https://github.com/lamatic/lamatic-docs (opens in a new tab)](https://github.com/lamatic/lamatic-docs) and learn more about Nextra at [https://nextra.site/docs (opens in a new tab)](https://nextra.site/docs).\n2. **Join Our Community**\n\n\n   - Have questions or need help? Join our Slack community. We're always happy to assist contributors!\n\n[Join our Slack](https://lamatic.ai/docs/slack)\n\n3. **Get Beta Access**\n   - As a token of our appreciation, we'd like to offer all contributors beta access to our platform. Use this link to sign up: [https://studio.lamatic.ai/signup?code=earlybetauser (opens in a new tab)](https://studio.lamatic.ai/signup?code=earlybetauser)\n\n## Steps to Contribute [Permalink for this section](https://lamatic.ai/docs/contributing\\#steps-to-contribute)\n\n1. **Fork the Repository**\n   - Visit the [Lamatic Docs GitHub repository (opens in a new tab)](https://github.com/lamatic/lamatic-docs).\n   - Click the \"Fork\" button in the top-right corner to create a copy of the repository in your GitHub account.\n2. **Clone Your Fork**\n   - Clone your forked repository to your local machine:\n\n\n\n     ```nx-border-black nx-border-opacity-[0.04] nx-bg-opacity-[0.03] nx-bg-black nx-break-words nx-rounded-md nx-border nx-py-0.5 nx-px-[.25em] nx-text-[.9em] dark:nx-border-white/10 dark:nx-bg-white/10\n     git clone https://github.com/your-username/lamatic-docs.git\n     cd lamatic-docs\n     ```\n3. **Create a New Branch**\n   - Create a new branch for your changes:\n\n\n\n     ```nx-border-black nx-border-opacity-[0.04] nx-bg-opacity-[0.03] nx-bg-black nx-break-words nx-rounded-md nx-border nx-py-0.5 nx-px-[.25em] nx-text-[.9em] dark:nx-border-white/10 dark:nx-bg-white/10\n     git checkout -b your-branch-name\n     ```\n\n   - Use a descriptive name for your branch, e.g., `add-new-feature-doc` or `fix-typo-in-quickstart`.\n4. **Install Dependencies and Run the Development Server**\n   - Install the necessary dependencies:\n\n\n\n     ```nx-border-black nx-border-opacity-[0.04] nx-bg-opacity-[0.03] nx-bg-black nx-break-words nx-rounded-md nx-border nx-py-0.5 nx-px-[.25em] nx-text-[.9em] dark:nx-border-white/10 dark:nx-bg-white/10\n     npm install\n     ```\n\n   - Start the development server:\n\n\n\n     ```nx-border-black nx-border-opacity-[0.04] nx-bg-opacity-[0.03] nx-bg-black nx-break-words nx-rounded-md nx-border nx-py-0.5 nx-px-[.25em] nx-text-[.9em] dark:nx-border-white/10 dark:nx-bg-white/10\n     npm run dev\n     ```\n\n   - Open your browser and go to `http://localhost:3333/docs` to see your changes in real-time.\n5. **Make Your Changes**\n   - Edit or add the necessary files in your local repository.\n   - Ensure your changes adhere to the existing style and formatting of the documentation.\n   - If you're adding new documentation, update the `_meta.json` file to include the new page.\n   - If you're modifying existing documentation, update the corresponding `.mdx` file.\n   - Refresh your browser to see your changes reflected immediately.\n6. **Commit Your Changes**\n   - Stage and commit your changes:\n\n\n\n     ```nx-border-black nx-border-opacity-[0.04] nx-bg-opacity-[0.03] nx-bg-black nx-break-words nx-rounded-md nx-border nx-py-0.5 nx-px-[.25em] nx-text-[.9em] dark:nx-border-white/10 dark:nx-bg-white/10\n     git add .\n     git commit -m \"Brief description of your changes\"\n     ```\n7. **Push Your Changes**\n   - Push your changes to your forked repository:\n\n\n\n     ```nx-border-black nx-border-opacity-[0.04] nx-bg-opacity-[0.03] nx-bg-black nx-break-words nx-rounded-md nx-border nx-py-0.5 nx-px-[.25em] nx-text-[.9em] dark:nx-border-white/10 dark:nx-bg-white/10\n     git push origin your-branch-name\n     ```\n8. **Create a Pull Request**\n   - Go to the [original Lamatic Docs repository (opens in a new tab)](https://github.com/lamatic/lamatic-docs).\n   - Click on \"Pull requests\" and then the \"New pull request\" button.\n   - Click \"compare across forks\" and select your fork and branch.\n   - Review your changes and click \"Create pull request\".\n   - Provide a clear title and description for your pull request.\n9. **Wait for Review**\n   - The Lamatic.ai team will review your contribution.\n   - They may request changes or provide feedback.\n   - Once approved, your changes will be merged into the main documentation.\n\n## Issue Types [Permalink for this section](https://lamatic.ai/docs/contributing\\#issue-types)\n\nWhen contributing or reporting issues, it's helpful to categorize them. Here are the main types of issues you might encounter or want to report:\n\n1. **🐛 Bug**: An error, flaw, or fault in the documentation that produces an incorrect or unexpected result. This could include broken links, incorrect information, or formatting issues.\n\n2. **🚀 Feature Request**: A suggestion for a new addition to the documentation. This could be a request for documentation on a new feature of Lamatic.ai or a proposal for a new section in the existing docs.\n\n3. **📈 Improvement**: A suggestion to enhance existing documentation. This could involve clarifying explanations, adding more examples, or restructuring content for better readability.\n\n4. **✏️ Typo**: A small error in the text, such as a misspelling or grammatical mistake.\n\n\nWhen creating an issue or pull request, please prefix your title with the appropriate issue type in square brackets or assign appropriate labels. For example:\n\n- \\[Bug\\] Broken link in Quick Start guide\n- \\[Feature Request\\] Add documentation for new API endpoint\n- \\[Improvement\\] Clarify explanation in Authentication section\n- \\[Typo\\] Fix misspelling in Contributing guide\n\nThis categorization helps the maintainers prioritize and address issues more efficiently.\n\nThank you for contributing to Lamatic.ai documentation! Your efforts are greatly appreciated and help make our platform better for everyone. If you have any questions during the process, don't hesitate to reach out on our Slack channel.\n\nLast updated on April 24, 2025\n\nPlatform Status ↗ [Bug Bounty Program](https://lamatic.ai/docs/bug-bounty-program \"Bug Bounty Program\")\n\n### Was this page useful?\n\nYesCould be better\n\n### Questions? We're here to help\n\n[FeedbackOpenAI](https://product.lamatic.ai/) [Email](mailto:[email protected]) [Talk to sales](https://lamatic.ai/docs/demo)\n\n### Subscribe to updates\n\nGet updates\n\njust-footer",
        "metadata": {
            "twitter:url": "https://lamatic.ai",
            "next-head-count": "22",
            "twitter:site:domain": "lamatic.ai",
            "ogImage": "https://lamatic.ai/api/og?title=Contributing%20to%20Lamatic.ai%20Documentation&description=Contributing%20to%20Lamatic.ai%20Documentation&section=Docs",
            "ogTitle": "Contributing to Lamatic.ai Documentation - Lamatic.ai Docs",
            "twitter:title": "just-footer",
            "ogDescription": "Contributing to Lamatic.ai Documentation",
            "description": [
            "Contributing to Lamatic.ai Documentation",
            "Contributing to Lamatic.ai Documentation"
            ],
            "og:url": "https://lamatic.ai/docs/contributing",
            "ogUrl": "https://lamatic.ai/docs/contributing",
            "og:description": [
            "Contributing to Lamatic.ai Documentation",
            "Contributing to Lamatic.ai Documentation"
            ],
            "viewport": [
            "width=device-width, initial-scale=1.0, viewport-fit=cover",
            "width=device-width, initial-scale=1"
            ],
            "favicon": "https://lamatic.ai/public/favicon-32x32.png",
            "title": "Contributing to Lamatic.ai Documentation - Lamatic.ai Docs",
            "robots": "index,follow",
            "twitter:card": "summary_large_image",
            "twitter:image": "https://lamatic.ai/api/og?title=Contributing%20to%20Lamatic.ai%20Documentation&description=Contributing%20to%20Lamatic.ai%20Documentation&section=Docs",
            "og:title": [
            "Contributing to Lamatic.ai Documentation - Lamatic.ai Docs",
            "just-footer"
            ],
            "theme-color": "#000",
            "og:image": "https://lamatic.ai/api/og?title=Contributing%20to%20Lamatic.ai%20Documentation&description=Contributing%20to%20Lamatic.ai%20Documentation&section=Docs",
            "scrapeId": "332e0d70-915e-44df-bc34-7c888543add8",
            "sourceURL": "https://lamatic.ai/docs/contributing",
            "url": "https://lamatic.ai/docs/contributing",
            "statusCode": 200
        }
        }
    ]
}

In case of async mode, the user will have the option to choose the following configurations :

Parameter	Description	Example Value
Credential Name	Select previously saved credentials	`my-firecrawl-creds`
Callback Webhook	URL to receive notifications about crawl completion	`https://example.com/webhook`
Webhook Headers	Headers to be sent to the webhook	`{'Content-Type:application/json'}`
Webhook Metadata	Metadata to be sent to the webhook	`{'status':'{{codeNode_540.status}}'}`
Webhook Events	A multiselect list of events to be sent to the webhook	`["completed", "failed", "page", "started"]`

The output in the case of async mode will be as follows :

{
    "success": true,
    "id": "8***************************7",
    "url": "https://api.firecrawl.dev/v1/crawl/8***************************7"
}

Crawler Configuration

Parameter	Description	Example Value
Credential Name	Select previously saved credentials	`my-firecrawl-creds`
URL	Starting point URL for the crawler	`https://example.com`
Exclude Path	URL patterns to exclude from the crawl	`"admin/", "private/"`
Include Path	URL patterns to include in the crawl	`"blog/", "products/"`
Crawl Depth	Maximum depth to crawl relative to the entered URL	`3`
Crawl Limit	Maximum number of pages to crawl	`1000`
Crawl Sub Pages	Toggle to enable or disable crawling sub pages	`true`
Max Discovery Depth	Maximum depth for discovering new URLs during the crawl	`5`
Ignore Sitemap	Ignore the sitemap.xml file for crawling	`false`
Allow Backward Links	Allow crawling backward links (e.g., from a blog post to the homepage)	`true`
Allow External Links	Allow crawling external links (e.g., links to other domains)	`false`
Ignore Query Parameters	Ignore specific query parameters in URLs	`false`
Delay	Delay between requests to avoid overloading the server	`2` (seconds)

Scraper Configuration

Parameter	Description	Example Value
Credential Name	Select previously saved credentials	`my-firecrawl-creds`
URL	Target URL to scrape	`https://example.com/page`
Main Content	Extract only the main content of the page, excluding headers, navs, footers, etc.	`true`
Skip TLS Verification	Bypass SSL certificate validation	`false`
Include Tags	HTML tags to include in extraction	`p, h1, h2, article`
Exclude Tags	HTML tags to exclude from extraction	`nav, footer, aside`
Emulate Mobile Device	Simulate mobile browser access	`true`
Wait for Page Load	Time to wait for dynamic content (ms)	`123`

Troubleshooting

Common Issues

Problem	Solution
Invalid API Key	Ensure the API key is correct and has not expired.
Connection Issues	Verify that the host URL is correct and reachable.
Webhook Errors	Check if the webhook endpoint is active and correctly configured.
Crawling Errors	Review the inclusion/exclusion paths for accuracy.
Dynamic Content Not Loaded	Increase the `Wait for Page Load` time in the configuration.

Debugging

Check Firecrawl logs for detailed error information.
Test the webhook endpoint to confirm it is receiving updates.
If the connection fails, whitelist the following IPs: https://www.cloudflare.com/ips/ (opens in a new tab)