🚀 Automating Twitter Data Fetching and Image Generation using AWS Fargate, Python, and S3

A step-by-step guide to building a serverless pipeline that retrieves tweets, converts them into images, and uploads them to Amazon S3 — all automatically!

---

🧠 Introduction

In this project, we’ll build a serverless workflow that automatically:

1. Fetches tweets using the Twitter (X) API

2. Converts tweet text into a quote-style image

3. Uploads the image to an Amazon S3 bucket

4. Runs the entire process in the cloud using AWS Fargate (ECS) — no servers, no EC2s!

This project is great for:

• Developers exploring AWS Fargate

• Beginners learning Docker + ECS

• Anyone curious about automating APIs and cloud storage

---

🧩 Architecture Overview

Here’s the high-level flow:

Twitter API → Python App → Docker → ECR → ECS Fargate → CloudWatch Logs → S3Components:

---

⚙️ Step 1 — Get a Twitter Developer Account

1. Go to developer.x.com

2. Create a free developer project

3. In the use-case description, write something like:

   “Educational project to fetch tweets and create images stored in S3 using AWS.”

4. Once approved, go to Projects & Apps → Keys and Tokens

5. Copy your Bearer Token

---

🪣 Step 2 — Create an S3 Bucket

In AWS Console → S3 → Create Bucket

• Name: tweet-images-yourname-demo

• Region: us-east-1

• Keep all defaults (Block Public Access = ON)

• Click Create Bucket

---

🐍 Step 3 — Build the Python App

Project structure:

twitter-python-aws/
 ├── main.py
 └── requirements.txt

requirements.txt

boto3
tweepy
Pillow

main.py

import os, io, textwrap, datetime
import boto3
from PIL import Image, ImageDraw, ImageFont
import tweepy

# Environment variables
BEARER = os.environ["X_BEARER_TOKEN"]
QUERY = os.environ.get("TWEET_QUERY", "aws fargate -is:retweet lang:en")
BUCKET = os.environ["S3_BUCKET"]
REGION = os.environ.get("AWS_REGION", "us-east-1")

# Fetch recent tweets
client = tweepy.Client(bearer_token=BEARER, wait_on_rate_limit=True)
resp = client.search_recent_tweets(query=QUERY, max_results=10, tweet_fields=["created_at","lang"])
tweets = resp.data or []
text = tweets[0].text if tweets else "No tweets found."

# Create image
W, H = 1200, 630
img = Image.new("RGB", (W, H), (245, 245, 255))
draw = ImageDraw.Draw(img)
font = ImageFont.load_default()

wrapped = textwrap.fill(text, width=50)
draw.text((50, 200), wrapped, font=font, fill=(0, 0, 0))
draw.text((50, 50), "Tweet Snapshot", font=font, fill=(90, 90, 90))
timestamp = datetime.datetime.now(datetime.UTC).strftime("%Y-%m-%d %H:%M UTC")
draw.text((50, 550), f"{timestamp}", font=font, fill=(120, 120, 120))

# Upload to S3
buf = io.BytesIO()
img.save(buf, format="PNG")
buf.seek(0)
key = f"tweets/{datetime.datetime.now(datetime.UTC).strftime('%Y%m%d_%H%M%S')}.png"

s3 = boto3.client("s3", region_name=REGION)
s3.put_object(Bucket=BUCKET, Key=key, Body=buf, ContentType="image/png")

print(f"✅ Uploaded s3://{BUCKET}/{key}")

---

🧱 Step 4 — Test Locally

Set your environment variables:

export X_BEARER_TOKEN="your-twitter-bearer-token"
export S3_BUCKET="tweet-images-yourname-demo"
export AWS_REGION="us-east-1"

Run it:

pip install -r requirements.txt
python3 main.py

✅ Output:

✅ Uploaded s3://tweet-images-yourname-demo/tweets/20251028_113456.png

---

🐳 Step 5 — Containerize with Docker

Dockerfile

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "main.py"]

Build and test locally:

docker buildx build --platform linux/amd64 -t twitter-fargate-demo .

---

🪣 Step 6 — Push to Amazon ECR

aws ecr create-repository --repository-name twitter-fargate-demo
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <account-id>.dkr.ecr.us-east-1.amazonaws.com
docker tag twitter-fargate-demo:latest <account-id>.dkr.ecr.us-east-1.amazonaws.com/twitter-fargate-demo:latest
docker push <account-id>.dkr.ecr.us-east-1.amazonaws.com/twitter-fargate-demo:latest

---

🧾 Step 7 — Create IAM Roles

1️⃣ Execution Role

• Name: ecsTaskExecutionRole

• Policy: AmazonECSTaskExecutionRolePolicy

2️⃣ Task Role

• Name: twitter-fargate-task-role

• Add this inline policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:PutObjectAcl"],
      "Resource": "arn:aws:s3:::tweet-images-yourname-demo/*"
    }
  ]
}

---

☁️ Step 8 — Create ECS Task Definition

• Launch type: Fargate

• Task size: 0.25 vCPU / 512 MB

• Task Role: twitter-fargate-task-role

• Execution Role: ecsTaskExecutionRole

Add Container

Click Add container → Create ✅

---

🚀 Step 9 — Run the Task

1. Go to ECS → Clusters → Create Cluster (Fargate)

2. Name it twitter-cluster

3. Click Run new task

   • Cluster: twitter-cluster

   • Launch type: Fargate

   • Task definition: twitter-fargate-demo

   • Subnet: default public

   • Public IP: ENABLED

4. Run Task ✅

---

🪶 Step 10 — Verify CloudWatch Logs

Open CloudWatch → Log groups → /ecs/twitter-fargate-demo

Expected output:

✅ Uploaded s3://tweet-images-yourname-demo/tweets/20251028_113456.png

Your image will be visible inside the S3 bucket folder /tweets/.

---

⏰ Bonus — Automate with EventBridge Scheduler

1. Go to Amazon EventBridge → Scheduler → Create schedule

2. Frequency:

    rate(1 hour)
   

3. Target: ECS RunTask

4. Choose:

   • Cluster: twitter-cluster

   • Task Definition: twitter-fargate-demo

5. Save!

Now AWS Fargate runs your container automatically every hour 🕒

---

🧩 Final Architecture Recap

---

🎯 Conclusion

With just a few components, we built a fully serverless pipeline that:

• Runs Python in AWS Fargate

• Fetches live tweets via API

• Generates styled quote images

• Uploads them to S3 automatically

• Can scale and schedule without managing servers!

This is a perfect example of combining APIs, Docker, and AWS Cloud to build automation pipelines.
You can easily extend it to:

• Generate dynamic social media graphics

• Archive trending tweets

• Power a dashboard or newsletter

---

🧠 Tech Stack Summary

• AWS Fargate (ECS)

• Amazon S3

• CloudWatch Logs

• Docker

• Python 3.11

• Tweepy + Pillow + Boto3

---

💬 Final Thoughts

Building this end-to-end pipeline was a fantastic way to explore how AWS Fargate lets developers run any containerized workload — no servers, no maintenance, just deploy and forget.

---

✍️ Author: Shikhar Shukla

A full-stack developer & cloud enthusiast passionate about automating cool ideas.

---