S3 Node

Overview

The S3 Node is a file synchronization component that automates fetching and synchronizing files from Amazon S3 buckets. This node supports various file types, including text files, PDFs, Word Documents, and others stored in the bucket, enabling regular file synchronization to support vectorization and indexing for Retrieval-Augmented Generation (RAG) flows.

This node is a Batch Trigger node that streamlines file collection from Amazon S3 buckets and prepares files for vectorization and indexing to enhance RAG flows.

Features

Prerequisites

Before using S3 Node, ensure the following:

• AWS Account: An AWS account with appropriate bucket access permissions.
• S3 Bucket Access: The target S3 bucket name and configuration details.
• IAM Permissions: Understanding of IAM policies and credentials for S3 access.
• Network Access: Proper network connectivity to AWS S3 services.

Installation

Step 1: Set Up AWS Credentials

1. Create IAM Policy:

   • Navigate to the IAM console (opens in a new tab)
   • Create a new policy with required S3 permissions
   • Use the provided JSON policy template

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:ListBucket",
                   "s3:ListAllMyBuckets"
               ],
               "Resource": "*"
           }
       ]
   }
   Note: If you want to give permission only to specific buckets then add them to resource key, refer the below example

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:ListBucket"
               ],
               "Resource": [
                   "arn:aws:s3:::{your-bucket-name-1}/*",
                   "arn:aws:s3:::{your-bucket-name-1}",
                   "arn:aws:s3:::{your-bucket-name-2}/*",
                   "arn:aws:s3:::{your-bucket-name-2}"
               ]
           }
       ]
   }
   💡

   Note: At this time, object-level permissions alone are not sufficient to successfully authenticate the connection. Please ensure you include the bucket-level permissions as provided in the example above.

   If you want to restrict the usage to specific bucket then you can create the policy as follows:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:ListBucket"
               ],
               "Resource": [
                   "arn:aws:s3:::bucket-name",
                   "arn:aws:s3:::bucket-name/*"
               ]
           },
           {
               "Sid": "VisualEditor1",
               "Effect": "Allow",
               "Action": "s3:ListAllMyBuckets",
               "Resource": "*"
           }
       ]
   }
   💡

   Note: You need to make sure that you are giving ListAllMyBuckets permission with resource as * else you will not be able to see the available buckets in the node config.

2. Configure IAM User:

   • Create or select an IAM user
   • Attach the created policy
   • Generate and securely store access credentials
   🚫

   Caution: Your Secret Access Key will only be visible once upon creation. Be sure to copy and store it securely.

   For more information on managing your access keys, please refer to the official AWS documentation (opens in a new tab).

Step 2: Configure S3 Credentials

Use the following format to set up your credentials:

Step 3: Set Up Lamatic Flow

1. Add S3 Node: Drag the S3 node to your flow
2. Provide Credentials: Configure AWS credentials and bucket details
3. Configure Sync Settings: Set up sync mode, schedule, and file filters

Configuration Reference

Batch Trigger Configuration

Configuration Details

Credentials

These are the authentication details used to connect to the S3 bucket. You must configure IAM credentials with appropriate permissions (e.g., read access to the bucket).

Bucket

The S3 bucket acts as the data source for this trigger. Enter the name of the bucket where your files are stored.

Parsing Strategy

This option determines how the files will be interpreted.

• auto lets the system decide the best way to parse files, while other strategies might require manual setup for specific file types.
• fast extracts text directly from the document which doesn't work for all files.
• ocr_only is more reliable, but slower.

Globs (Path Patterns)

Use glob patterns to filter which files should be processed.

• **: match everything.
• **/*.csv: match all files with a specific extension.
• myFolder/**/*.csv: match all .csv files anywhere under myFolder.
• */**: match everything at least one folder deep.
• */*/*/**: match everything at least three folders deep.
• **/file.*|**/file: match every file called "file" with any extension (or no extension).
• x/*/y/*: match all files that sit in folder x -> any folder -> folder y.
• **/prefix*.csv: match all .csv files with a specific prefix.
• **/prefix*.parquet: match all .parquet files with a specific prefix.

This is helpful for excluding unnecessary files.

Days to Sync If History Is Full

If the system encounters a large backlog of files, this setting limits the synchronization to a defined number of recent days. It helps manage processing time and storage efficiently.

Start Date

Define a specific date from which the system should start processing files. This is particularly useful for incremental syncs where you only want to process data from a certain point in time.

Format: YYYY-MM-DDTHH:mm:ss.SSSSSSZ Example: 2025-01-03T00:00:00.000000Z

Sync Mode

The S3 source connector supports the following:

1. In Incremental mode, the system only processes files that have been added or updated since the last sync. This reduces redundancy and improves performance.
2. Full Refresh sync mode is a data replication method that copies all data from a source to a destination

Sync Schedule

This setting allows you to specify how often the synchronization process should run, such as every hour or every 24 hours. Regular intervals ensure that your data stays updated. You can Schedule the Sync on Every 3, 6, 8, 12 and 24 hours

Low-Code Example

triggerNode:
  nodeId: triggerNode_1
  nodeType: s3Node
  nodeName: S3
  values:
    credentials: "AWS"
    bucket: "TEST"
    strategy: auto
    globs:
      - "**"
    days_to_sync_if_history_is_full: "3"
    start_date: "2025-01-03T00:00:00.000000Z"
    syncMode: incremental_append
    cronExpression: 0 0 00 1/1 * ? * UTC

Event Trigger Output

The S3 node outputs file data in the following format:

Example Output

{
    "document_key": "example.pdf",
    "content": "Extracted text content from the S3 file",
    "document_url": "s3://bucket-name/example.pdf"
}

Output Schema

• document_key: String identifier for the document (filename)
• content: String containing the extracted text content from the file
• document_url: S3 URL of the processed file for reference

Output Schema

Batch Trigger Output

Troubleshooting

Common Issues

Debugging

• Review AWS CloudWatch logs for access issues
• Verify IAM policy permissions are correctly configured
• Test bucket accessibility using AWS CLI or console
• Check Lamatic Flow logs for detailed error information
• Validate glob patterns to ensure they match your files
• Monitor sync logs for specific file processing errors
• Test with a small bucket before processing large repositories
• If the connection fails, whitelist the following IPs: https://www.cloudflare.com/ips/ (opens in a new tab)

Best Practices

• Use incremental sync mode for better performance
• Implement specific glob patterns to avoid processing unnecessary files
• Schedule syncs during off-peak hours to minimize impact
• Use appropriate parsing strategies for different file types
• Regularly monitor sync logs for any issues
• Set appropriate days_to_sync_if_history_is_full to limit historical data
• Test with sample files before processing large buckets
• Ensure proper IAM permissions with least privilege principle

Example Use Cases

Document Intelligence Workflows

• Business Documents: Sync reports, contracts, and spreadsheets from S3 for automated processing
• Data Archives: Index historical documents and data files stored in S3
• Compliance Documents: Process audit trails and compliance-related content
• Backup Files: Automate processing of backup documents and files

RAG Applications

• Semantic Search: Enable natural language search across S3 documents
• Question Answering: Build AI assistants that can answer questions about stored documents
• Document Summarization: Automatically summarize lengthy reports and documents
• Content Discovery: Help users find relevant information across S3 repositories

Additional Resources

• AWS S3 Documentation (opens in a new tab)
• AWS IAM Documentation (opens in a new tab)
• AWS CLI Setup Guide (opens in a new tab)
• Cron Expression Guide (opens in a new tab)