AI Document Processing System: MCP, ECS & S3

Building an AI-Powered Document Processing System with MCP, ECS, and S3

How I built a serverless architecture that lets AI models intelligently fetch and summarize documents from S3 using the Model Context Protocol

The Problem That Started It All

You know that feeling when you have tons of documents scattered across S3 buckets and you wish you could just ask an AI:

“Hey, can you summarize that report from last week?”

Well, that’s exactly what got me started on this project.

I wanted to bridge the gap between AI models (like Amazon Bedrock) and my document storage in S3 — without building yet another custom API that would be a pain to maintain.

That’s where the Model Context Protocol (MCP) came into play.

What is MCP and Why Should You Care?

The Model Context Protocol is basically a standardized way for AI models to interact with external data sources.

Think of it as a universal translator between AI models and your data. Instead of building custom integrations for every AI service, you build one MCP server and any MCP-compatible AI can talk to it.

It’s like having a waiter at a restaurant who speaks multiple languages — your AI orders in “AI speak”, and the MCP server translates that into “S3 speak” to fetch your documents.

The Architecture: Serverless All The Way Down

Here’s what I ended up building:

S3 Bucket → Lambda Function → ECS Fargate → API Gateway → AI Model
     ↓           ↓              ↓             ↓
CloudWatch ← SNS Alerts ← Load Balancer ← Authentication

Let’s break it down:

1. S3 Bucket — The Document Vault

This is where all the magic starts. I configured the S3 bucket with:

Server-side encryption (because security matters)
Versioning enabled (because mistakes happen)
Event notifications that trigger our Lambda function

2. Lambda Function — The Trigger Happy Component

Whenever someone uploads a file to S3, this Lambda function wakes up and says:

“Hey ECS, we got a new file to process!”

It’s like having a doorbell for your S3 bucket.

import boto3
import json
import os

def lambda_handler(event, context):
    """
    Lambda function triggered when a new file is uploaded to S3.
    Updates the ECS service to ensure it's running to process the new file.
    """
    try:
        cluster = os.environ.get('ECS_CLUSTER')
        service = os.environ.get('ECS_SERVICE')

        if not all([cluster, service]):
            raise ValueError("Missing required environment variables")

        ecs = boto3.client('ecs')

        ecs.update_service(
            cluster=cluster,
            service=service,
            forceNewDeployment=True
        )

        return {
            "statusCode": 200,
            "body": json.dumps("Successfully triggered ECS service update"),
        }

    except Exception as e:
        print(f"Error: {str(e)}")
        raise e

3. ECS Fargate — The Workhorse

This is where the actual MCP server runs. I chose Fargate because:

No servers to manage (serverless FTW!)
Auto-scaling when things get busy
Pay only for what you use

The MCP server is a Flask app that implements the MCP protocol, with endpoints like:

/health — Monitoring
/summarize/<filename> — The main event

4. API Gateway — The Bouncer

Handles authentication, rate limiting, and makes sure no random internet person is summarizing my documents.

Infrastructure as Code — Because Manual is Painful

I used Terraform for everything. Clicking around the AWS console gets old fast.

Infrastructure modules include:

ECR repository for Docker images
ECS cluster and service definitions
S3 bucket with proper policies
API Gateway with authentication
CloudWatch logging and monitoring
SNS for alerts

ECS Task Definition in Terraform:

resource "aws_ecs_task_definition" "mcp_task" {
  family                   = "${var.project_name}-${var.environment}-task"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = var.ecs_task_cpu
  memory                   = var.ecs_task_memory
  execution_role_arn       = aws_iam_role.ecs_task_execution_role.arn
  task_role_arn            = aws_iam_role.ecs_task_role.arn

  container_definitions = jsonencode([
    {
      name      = "mcp-server"
      image     = var.container_image
      essential = true
      portMappings = [
        {
          containerPort = var.container_port
          hostPort      = var.container_port
          protocol      = "tcp"
        }
      ]
      environment = [
        { name = "AWS_REGION",      value = data.aws_region.current.name },
        { name = "S3_BUCKET_NAME",  value = var.s3_bucket_name }
      ]
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          awslogs-group         = aws_cloudwatch_log_group.mcp_logs.name
          awslogs-region        = data.aws_region.current.name
          awslogs-stream-prefix = "ecs"
        }
      }
    }
  ])
}

The Code That Makes It Tick

MCP server endpoint:

@app.route('/summarize/<filename>', methods=['GET'])
def summarize_file(filename):
    try:
        response = s3_client.get_object(Bucket=S3_BUCKET, Key=filename)
        content = response['Body'].read().decode('utf-8')
        
        return jsonify({
            "status": "success",
            "filename": filename,
            "content": content,
            "timestamp": time.time()
        })
    except Exception as e:
        logger.error(f"Error processing file {filename}: {str(e)}")
        return jsonify({"error": str(e)}), 500

CI/CD Pipeline

GitHub Actions handles:

Build & test code
Build & push Docker images to ECR
Deploy infrastructure with Terraform
Update ECS services

name: CI/CD Pipeline

on:
  push:
    branches: [ "main", "develop" ]
  pull_request:
    branches: [ "main", "develop" ]

(full pipeline code in repo)

Monitoring & Observability

CloudWatch Logs for everything
CloudWatch Metrics for API calls, processing times
SNS Alerts for failures
Health Checks for ECS services

Lessons Learned

Start monitoring early
Test for circular dependencies (S3 → Lambda → ECS)
Mock external services
Document APIs better

Real-World Use Cases

Legal contract analysis
Marketing document insights
Academic research workflows
Compliance reporting

Performance & Costs

Cold start: ~2–3s
Warm requests: ~200–500ms
Cost: ~$20–30/month moderate usage
Scaling: 100+ concurrent requests easily

Security

API Gateway auth
IAM least privilege
S3 restrictive policies
VPC isolation
Encryption at rest & transit

FAQ

**Q: Why use MCP instead of just building a REST API?**

A: MCP provides a standardized protocol that any compatible AI model can use. It’s like building to a standard instead of reinventing the wheel every time.

**Q: How much does this cost to run?**

A: For moderate usage (few hundred requests per day), you’re looking at $20–30/month. Most of that is ECS Fargate compute time.

**Q: Can this handle large files?**

A: Currently it’s designed for text files up to a few MB. For larger files, you’d want to implement streaming or chunking.

**Q: What about security?**

A: The setup includes API Gateway authentication, IAM roles, VPC isolation, and encrypted storage. It’s production-ready from a security standpoint.

**Q: How do I add support for different file types?**

A: You’d extend the MCP server to handle different MIME types and add appropriate parsers (PDF, DOCX, etc.).

**Q: Can I use this with OpenAI instead of Bedrock?**

A: Absolutely! The MCP server is AI-agnostic. You just need to point your AI client to the MCP endpoints.

**Q: What happens if ECS tasks crash?**

A: The service will automatically restart failed tasks, and the load balancer will route traffic away from unhealthy instances.

**Q: How do I monitor this in production?**

A: CloudWatch dashboards, SNS alerts, and the health check endpoints give you full visibility into system health.

**Q: Is this overkill for simple document processing?**

A: Maybe! But it’s designed to scale. If you just need to process a few files, a simple Lambda function might be enough.

**Q: Can I deploy this in multiple AWS regions?**

A: The Terraform code would need modifications for multi-region deployment, but the architecture supports it.

This project is open source on my GitHub. Fork it, break it, improve it!

🚀 Try it yourself: GitHub Repo 💬 Feedback? Connect with me

Written by Rajesh Sood

📢 Have questions or feedback? Drop a comment below or connect with me on Twitter/X@spysood!

Originally published on Medium.