From Script to Serverless: Automating Our AI YouTube Digest with AWS Lambda

In the previous article (AI-Powered YouTube Digest with Python, Gemini, and the YouTube API), we built a powerful Python script that uses the YouTube and Gemini APIs to find, summarize, and email a curated list of videos. It’s an intelligent tool, but it has one major limitation: it only works when we manually run it on our local computer.

True automation means a system that runs reliably on a schedule, without any manual intervention. We want our digest to arrive in our inbox at 8 AM every morning, whether our computer is on or not.

To achieve this, we need to move our project to the cloud. In this article, I’ll show you how to take our Python script and deploy it as a serverless function using AWS Lambda. We’ll refactor our code, secure our secret keys, and use Amazon EventBridge to create a daily schedule. This is how you transform a clever script into a robust, cloud-native application.

What is “Serverless”? (And Why It’s Perfect for This Project)

Before we dive in, let’s demystify the term “serverless.” It doesn’t mean there are no servers; it just means you don’t have to manage them.

AWS Lambda is a “Functions as a Service” (FaaS) platform. You upload your code, and AWS handles everything required to run and scale it with high availability. For our project, this is perfect because our script only needs to run for a few minutes each day. Paying for a server to sit idle for 23.5 hours would be a waste of money. With Lambda, we only pay for the few seconds of computation we use each morning.

The New Architecture: A Cloud-Native Pipeline

Moving to the cloud changes our architecture. Our script is no longer running in isolation; it becomes a component within the AWS ecosystem.

Our new workflow looks like this:

The Scheduler (Amazon EventBridge): This is our cloud-based alarm clock. We’ll configure it to trigger at 8 AM every morning. Its sole job is to kickstart our process.
The Serverless Function (AWS Lambda): This is the new home for our Python code. When triggered by EventBridge, AWS spins up an environment, runs our script, and then shuts it down.
Secure Storage (AWS Secrets Manager): Instead of hardcoding our API keys and email passwords in our code (a major security risk), we will store them securely in AWS Secrets Manager. Our Lambda function will be granted permission to fetch these secrets at runtime.
External APIs & Delivery: The core logic remains the same. Our function, running in the cloud, will call the external YouTube and Gemini APIs, and then use SMTP to send the final email digest.

Adapting Our Code for the Cloud

We need to make a few important changes to our Python script to make it compatible with the Lambda environment.

1. The Handler Function

AWS Lambda needs a specific entry point to start the execution. This is called the handler function. We’ll move the logic from our if __name__ == “__main__”: block into a function that Lambda can call, typically named lambda_handler.

Before (local python code):

if __name__ == "__main__":
    # Main logic was here
    all_videos = find_youtube_videos(...)
    # ... etc ...
    send_email(...)

After (AWS Lambda):

def lambda_handler(event, context):
    # Main logic moves here
    all_videos = find_youtube_videos(...)
    # ... etc ...
    send_email(...)
    return {
        'statusCode': 200,
        'body': 'Email sent successfully!'
    }

2. Fetching Secrets Securely

Hardcoding secrets is a bad practice. The professional way is to use a service like AWS Secrets Manager. We’ll need the boto3 library (the AWS SDK for Python) to fetch them.

import boto3
import json

def get_secrets():
    secret_name = "prod/YoutubeDigest/ApiKeys"
    region_name = "us-east-1"
    # Create a Secrets Manager client
    session = boto3.session.Session()
    client = session.client(service_name='secretsmanager', region_name=region_name)
    # In a real-world scenario, you'd use a try/except block here
    get_secret_value_response = client.get_secret_value(SecretId=secret_name)
    secret = get_secret_value_response['SecretString']
    return json.loads(secret)

# In our handler:
def lambda_handler(event, context):
    secrets = get_secrets()
    GEMINI_API_KEY = secrets['GEMINI_API_KEY']
    YOUTUBE_API_KEY = secrets['YOUTUBE_API_KEY']
    # ... now use these keys in your functions

This change dramatically improves the security and professionalism of our application.

3. Packaging Dependencies

In a Lambda environment, you can’t just pip install libraries. You must package all your required libraries (like google-generativeai, google-api-python-client) along with your code into a single .zip file, which you then upload to AWS.

High-Level Deployment Guide

Here’s a summary of the steps you would take in the AWS Management Console to deploy our project.

Store Your Secrets: Go to AWS Secrets Manager, create a new secret, and store your API keys and email password as a JSON key-value pair.
Package Your Application:

Create a requirements.txt file listing your dependencies.
On your local machine, create a new folder. Install the dependencies into this folder using the command: pip install -r requirements.txt -t .
Add your updated Python script (e.g., lambda_function.py) to this folder.
Zip the contents of the folder. This .zip file is your deployment package.

Create the Lambda Function:

Go to the AWS Lambda console and click “Create function.”
Choose “Author from scratch,” give it a name (e.g., YoutubeDigestFunction), and select a Python runtime (e.g., Python 3.11).
Under “Permissions,” ensure the function has a role with permissions to access Secrets Manager and write logs to CloudWatch.
Upload your .zip file and save.

Create the Schedule:

Go to Amazon EventBridge and select “Rules.”
Click “Create rule.” Give it a name and select “Schedule.”
Use a cron expression to define the schedule. For 8 AM UTC every day, you would use: cron(0 8 * * ? *).
For the target, select “Lambda function” and choose the function you just created.

Test and Monitor: Use the “Test” tab in the Lambda console to manually trigger your function and check for errors. You can view all logs and print statements in Amazon CloudWatch.

Conclusion: True Automation Unlocked

We have successfully evolved our project from a simple, local script into a robust, secure, and fully automated cloud application. It will now work for us every morning without fail, delivering valuable content directly to our inbox.

By leveraging serverless technology, we’ve gained incredible benefits:

Cost-Effectiveness: We pay only for the seconds of execution time we use.
Reliability: We are running on AWS’s world-class, highly available infrastructure.
Scalability: If we wanted to send this digest to thousands of users, Lambda would scale automatically to handle the load.
Zero Maintenance: We never have to worry about patching servers or managing infrastructure.

You now have a powerful blueprint for automating any information-gathering task. You can adapt this project to monitor websites, track stock prices, or curate articles — the possibilities are endless. Welcome to the world of building truly “smart” applications in the cloud.

Search This Blog

The Automated AI Blog