AWS S3 Integration

Overview

The AWS S3 node connects Lamatic to Amazon S3, and works in two modes:

• Trigger mode: automatically detects file additions and modifications in a bucket and syncs them into a flow, most commonly for vectorization and indexing in Retrieval-Augmented Generation (RAG) flows. This is the original behavior of the node and is unchanged.
• Action mode: lets a flow directly upload, fetch, list, move, copy, or delete files, read file metadata, and create or delete folders, on demand, as one step in a larger flow.

Features

✅ Key Functionalities

• Batch Trigger: Automates file fetching and synchronization on a schedule or in real-time using S3 event notifications.
• File Type Support: Handles text files, PDFs, Word Documents, and other compatible formats.
• Scheduled Processing: Supports automated sync schedules with configurable intervals for regular file updates.
• Selective Filtering: Use glob patterns to filter specific file types and paths for targeted file processing.
• Multiple Sync Modes: Supports both incremental (new/updated files only) and full-refresh (all files) synchronization modes.
• Scalable Processing: Scales efficiently with growing data volumes and large S3 buckets.
• File Actions: Upload, fetch, list, move, copy, delete, and read metadata for individual files on demand from within a flow.
• Folder Management: Create folders and delete folders (including everything inside them) directly from a flow.
• Configurable Signed URLs: Choose how long returned file URLs stay valid, from 15 minutes up to 7 days.
• S3-Compatible Providers: Works with any S3-compatible storage, not just AWS, including Supabase Storage, MinIO, and Cloudflare R2, by setting a custom endpoint.

✅ Benefits

• Streamlined File Collection: Automates the process of collecting files from Amazon S3 buckets, reducing manual effort and ensuring consistency.
• RAG Flow Preparation: Prepares files for vectorization and indexing to enhance Retrieval-Augmented Generation workflows.
• Scalable Architecture: Scales efficiently with growing data volumes and large S3 repositories.
• Flexible Configuration: Supports various file types and configurable processing strategies.
• Cost-Effective: Leverages AWS S3’s cost-effective storage for large-scale document management.

Prerequisites

Before setting up the AWS S3 node, ensure you have the following:

• An AWS account with appropriate bucket access permissions.
• The target S3 bucket name and configuration.
• An understanding of IAM policies and credentials.

Setup

Step 1: Set Up AWS Credentials

1. Create IAM Policy:

   • Navigate to the IAM console.

   • Create a new policy with required S3 permissions.

   • Use the provided JSON policy template.

     ⚠️

     The read-only policy below is enough for Trigger mode. If you’re using the node in Action mode, see Action mode permissions further down, Upload, Copy, and Create Folder need s3:PutObject, and Delete File, Delete Folder, and Move need s3:DeleteObject too.

         {
             "Version": "2012-10-17",
             "Statement": [
                 {
                     "Effect": "Allow",
                     "Action": [
                         "s3:GetObject",
                         "s3:ListBucket",
                         "s3:ListAllMyBuckets"
                     ],
                     "Resource": "*"
                 }
             ]
         }
     ℹ️

     Note: If you want to given permission only to specific buckets then add them to resource key, refer the below example

     {
         "Version": "2012-10-17",
         "Statement": [
             {
             "Effect": "Allow",
             "Action": [
                 "s3:GetObject",
                 "s3:ListBucket"
             ],
             "Resource": [
                     "arn:aws:s3:::{your-bucket-name-1}/*",
                     "arn:aws:s3:::{your-bucket-name-1}",
                     "arn:aws:s3:::{your-bucket-name-2}/*",
                     "arn:aws:s3:::{your-bucket-name-2}"
                 ]
             }
         ]
     }
     💡

     Note: At this time, object-level permissions alone are not sufficient to successfully authenticate the connection. Please ensure you include the bucket-level permissions as provided in the example above.

     
     

     If you want to restrict the usage to specific bucket then you can create the policy as follows:

     {
         "Version": "2012-10-17",
         "Statement": [
             {
                 "Sid": "VisualEditor0",
                 "Effect": "Allow",
                 "Action": [
                     "s3:GetObject",
                     "s3:ListBucket"
                 ],
                 "Resource": [
                     "arn:aws:s3:::bucket-name",
                     "arn:aws:s3:::bucket-name/*"
                 ]
             },
             {
                 "Sid": "VisualEditor1",
                 "Effect": "Allow",
                 "Action": "s3:ListAllMyBuckets",
                 "Resource": "*"
             }
         ]
      
     }
     💡

     Note: You need to make sure that you are giving ListAllMyBuckets permission with resource as * else you will not be able to see the available buckets in the node config.

     Action mode permissions

     If you’re using the S3 node in Action mode, extend the policy with write and delete permissions. Upload File, Upload File from URL, Copy File, and Create Folder need s3:PutObject. Delete File, Delete Folder, and Move File (which deletes the source after copying) need s3:DeleteObject.

     {
         "Version": "2012-10-17",
         "Statement": [
             {
                 "Effect": "Allow",
                 "Action": [
                     "s3:GetObject",
                     "s3:PutObject",
                     "s3:DeleteObject",
                     "s3:ListBucket",
                     "s3:ListAllMyBuckets"
                 ],
                 "Resource": "*"
             }
         ]
     }
     Moving or copying a file to a different bucket (via Destination Bucket) needs s3:PutObject on that destination bucket too, not just the source bucket. If you’re scoping permissions to specific buckets, include both in the policy’s Resource list.

2. Configure IAM User:

   • Create or select an IAM user.

   • Attach the created policy.

   • Generate and securely store access credentials.

     ️⚠️

     Caution: Your Secret Access Key will only be visible once upon creation. Be sure to copy and store it securely.

     
     

     For more information on managing your access keys, please refer to the official AWS documentation.

Step 2: Configure S3 Credentials

Use the following format to set up your credentials:

Step 3: Set Up Lamatic Flow

1. Add the S3 node to your flow, either as a Trigger (for scheduled RAG sync) or as an Action (for on-demand file operations).
2. Provide the credentials and bucket details.
3. For a Trigger, configure sync settings, schedule, and file filters. For an Action, pick the operation you want from the Action dropdown and fill in its fields.

Trigger Configuration Reference

This applies when the S3 node is used as a Trigger for scheduled RAG sync. See Action Reference below for the on-demand file operations.

Credentials

These are the authentication details used to connect to the S3 bucket. You must configure IAM credentials with appropriate permissions (e.g., read access to the bucket).

Bucket

The S3 bucket acts as the data source for this trigger. Enter the name of the bucket where your files are stored.

Parsing Strategy

This option determines how the files will be interpreted.

• auto lets the system decide the best way to parse files, while other strategies might require manual setup for specific file types.
• fast extracts text directly from the document which doesn’t work for all files.
• ocr_only is more reliable, but slower.

Globs (Path Patterns)

Use glob patterns to filter which files should be processed.

• **: match everything.
• **/*.csv: match all files with a specific extension.
• myFolder/**/*.csv: match all .csv files anywhere under myFolder.
• */**: match everything at least one folder deep.
• */*/*/**: match everything at least three folders deep.
• **/file.*|**/file: match every file called “file” with any extension (or no extension).
• x/*/y/*: match all files that sit in folder x -> any folder -> folder y.
• **/prefix*.csv: match all .csv files with a specific prefix.
• **/prefix*.parquet: match all .parquet files with a specific prefix.

This is helpful for excluding unnecessary files. See Glob Patterns for the full syntax reference.

Supported File Types

Only specific file types are currently supported for vectorization and indexing in Lamatic. Using unsupported formats may result in parsing errors during synchronization.

✅ Allowed File Extensions

• .pdf — PDF Documents
• .txt — Plain Text Files
• .docx — Microsoft Word
• .pptx — Microsoft PowerPoint
• .md — Markdown Files

To avoid sync issues, ensure your glob patterns are configured to include only these types.

🔍 Recommended Glob Pattern

    globs:
    - "**/*.pdf"
    - "**/*.txt"
    - "**/*.docx"
    - "**/*.pptx"
    - "**/*.md"

Days to Sync If History Is Full

If the system encounters a large backlog of files, this setting limits the synchronization to a defined number of recent days. It helps manage processing time and storage efficiently.

Start Date

Define a specific date from which the system should start processing files. This is particularly useful for incremental syncs where you only want to process data from a certain point in time.

Format: YYYY-MM-DDTHH:mm:ss.SSSSSSZ

Example: 2025-01-03T00:00:00.000000Z

Sync Mode

The S3 source connector supports the following:

1. In Incremental mode, the system only processes files that have been added or updated since the last sync. This reduces redundancy and improves performance.

2. Full Refresh sync mode is a data replication method that copies all data from a source to a destination

Sync Schedule

This setting allows you to specify how often the synchronization process should run, such as every hour or every 24 hours. Regular intervals ensure that your data stays updated. You can Schedule the Sync on Every 3, 6, 8, 12 and 24 hours

Trigger Output

Example Output

{
  "document_key": "example.pdf",
  "content": "Extracted text content from the S3 file",
  "document_url": "https://bucket-name.s3.us-east-1.amazonaws.com/example.pdf?X-Amz-Expires=18000&..."
}

Action Reference

When used as an Action, the S3 node runs one of ten operations, picked from the Action dropdown. Each operation only shows the fields it needs.

Get File

Returns a signed URL for a single file, without downloading its contents into the flow.

{
  "key": "reports/quarterly-report.pdf",
  "bucket": "my-bucket",
  "url": "https://my-bucket.s3.us-east-1.amazonaws.com/reports/quarterly-report.pdf?..."
}

List Files in Folder

Folder placeholder objects (see Keep Empty Folder below) are automatically excluded from results.

{
  "bucket": "my-bucket",
  "folder": "invoices/2024/",
  "files": [
    { "key": "invoices/2024/jan.pdf", "size": 10432, "lastModified": "2025-01-04T10:00:00.000Z", "url": "https://..." }
  ],
  "count": 1
}

Upload File

Creates a new file from text content typed or mapped directly into the node.

{
  "bucket": "my-bucket",
  "key": "notes/summary.md",
  "fileType": "markdown",
  "etag": "d41d8cd98f00b204e9800998ecf8427e",
  "size": 128,
  "contentType": "text/markdown; charset=utf-8",
  "url": "https://..."
}

Upload File from URL

Downloads a file from a URL and stores it in the bucket as-is, keeping the source’s original content type.

{
  "bucket": "my-bucket",
  "key": "imports/photo.jpg",
  "etag": "d41d8cd98f00b204e9800998ecf8427e",
  "size": 204800,
  "contentType": "image/jpeg",
  "url": "https://..."
}

Move File / Copy File

Moving is a copy followed by deleting the source. Both actions share the same fields.

The node checks the source file exists (and the destination bucket, if different) before copying, so a missing source or bucket returns a clear error instead of a generic failure. Moving/copying a file onto itself (same bucket, same path) is rejected up front.

{
  "sourceBucket": "my-bucket",
  "sourceKey": "reports/draft.pdf",
  "bucket": "my-bucket",
  "key": "archive/2024/draft.pdf",
  "url": "https://...",
  "moved": true
}

Delete File

S3 normally reports success even when you delete a key that never existed. This action checks first, so existed and deleted tell you whether there was actually a file there.

{
  "bucket": "my-bucket",
  "key": "reports/draft.pdf",
  "existed": true,
  "deleted": true
}

Get File Metadata

Reads a file’s size, content type, and custom metadata without downloading it.

{
  "bucket": "my-bucket",
  "key": "reports/draft.pdf",
  "exists": true,
  "size": 10432,
  "contentType": "application/pdf",
  "lastModified": "2025-01-04T10:00:00.000Z",
  "etag": "d41d8cd98f00b204e9800998ecf8427e",
  "metadata": {},
  "url": "https://..."
}

If the file doesn’t exist, the action still succeeds and returns { "exists": false } rather than erroring.

Create Folder

S3 has no real folders, this writes a zero-byte marker object with a trailing slash, the same convention the AWS console uses.

{
  "bucket": "my-bucket",
  "folder": "invoices/2025/",
  "created": true
}

Delete Folder

Deletes every file under the given path.

{
  "bucket": "my-bucket",
  "folder": "invoices/2023/",
  "deletedCount": 842,
  "truncated": false
}

Low-Code Example

Trigger mode

triggerNode:
nodeId: triggerNode_1
nodeType: s3Node
nodeName: S3
values:
  credentials: "AWS"
  bucket: "TEST"
  strategy: auto
  globs:
    - "**"
  days_to_sync_if_history_is_full: "3"
  start_date: "2025-01-03T00:00:00.000000Z"
  syncMode: incremental_append
  cronExpression: 0 0 00 1/1 * ? * UTC

Action mode

- nodeId: s3Node_512
  nodeType: s3Node
  nodeName: S3
  values:
    credentials: "AWS"
    bucket: "TEST"
    action: S3_UPLOAD_FILE
    filePath: "notes/{{triggerNode_1.output.title}}"
    fileType: markdown
    content: "{{triggerNode_1.output.body}}"
    urlExpiry: "3600"
  needs:
    - triggerNode_1

Troubleshooting

Common Issues

Debugging

• Review AWS CloudWatch logs for access issues.
• Verify IAM policy permissions are correctly configured.
• Test bucket accessibility using AWS CLI or console.
• Check Lamatic Flow logs for detailed error information.
• If the connection fails, review IP Allowlisting and confirm the required Lamatic and Cloudflare ranges are allowed.

Best Practices

• Use incremental sync mode for better performance.
• Implement specific glob patterns to avoid processing unnecessary files.
• Schedule syncs during off-peak hours to minimize impact.
• Use appropriate parsing strategies for different file types.
• Regularly monitor sync logs for any issues.
• Set appropriate days_to_sync_if_history_is_full to limit historical data.
• Test with sample files before processing large buckets.
• Ensure proper IAM permissions with least privilege principle.

Example Use Cases

Document Intelligence Workflows

• Business Documents: Sync reports, contracts, and spreadsheets from S3 for automated processing.
• Data Archives: Index historical documents and data files stored in S3.
• Compliance Documents: Process audit trails and compliance-related content.
• Backup Files: Automate processing of backup documents and files.

RAG Applications

• Semantic Search: Enable natural language search across S3 documents.
• Question Answering: Build AI assistants that can answer questions about stored documents.
• Document Summarization: Automatically summarize lengthy reports and documents.
• Content Discovery: Help users find relevant information across S3 repositories.

Additional Resources

• AWS S3 Documentation
• AWS IAM Documentation
• AWS CLI Setup Guide
• Cron Expression Guide