Boto3

Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python, enabling developers to interact with AWS services programmatically. It allows you to automate tasks such as creating, configuring, and managing AWS resources like EC2 instances, S3 buckets, DynamoDB tables, Lambda functions, and more.
Author

Benedict Thekkel

Key Concepts in Boto3

Clients

Clients are used to make low-level service calls to AWS APIs.

The client gives you access to a set of methods for each AWS service, which correspond directly to AWS API operations.

Example

import boto3
client = boto3.client('s3')  # S3 client
response = client.list_buckets()  # List all S3 buckets
print(response)

Resources

Resources are higher-level abstractions that wrap around AWS service objects, providing an object-oriented way of interacting with AWS.

Resources hide much of the complexity of working with AWS APIs.

Example

s3 = boto3.resource('s3')
bucket = s3.Bucket('my-bucket')
for obj in bucket.objects.all():
    print(obj.key)

Sessions

A session manages your AWS credentials, region, and configurations. If no session is explicitly created, Boto3 creates a default one.

You can create a session to manage different profiles, regions, or credentials.

Example

session = boto3.Session(profile_name='myprofile')
s3 = session.resource('s3')

Paginator

Paginators automatically handle the pagination of results, which is required when an API call returns too many results to fit into a single response.

Example

client = boto3.client('s3')
paginator = client.get_paginator('list_objects_v2')
for page in paginator.paginate(Bucket='my-bucket'):
    for obj in page['Contents']:
        print(obj['Key'])

Waiters

Waiters provide a convenient way to wait for a resource to reach a specific state, such as an EC2 instance becoming available or an S3 object being fully uploaded.

Example

ec2 = boto3.client('ec2')
ec2.start_instances(InstanceIds=['i-1234567890abcdef0'])
waiter = ec2.get_waiter('instance_running')
waiter.wait(InstanceIds=['i-1234567890abcdef0'])

Common AWS Services with Boto3

Amazon S3 (Simple Storage Service)

S3 is used for scalable object storage. With Boto3, you can create, read, and manage S3 buckets and objects.

Create a bucket

s3 = boto3.client('s3')
s3.create_bucket(Bucket='my-bucket')

Upload a file

s3 = boto3.client('s3')
s3.upload_file('localfile.txt', 'my-bucket', 'remote.txt')

Download a file

s3 = boto3.client('s3')
s3.download_file('my-bucket', 'remote.txt', 'localfile.txt')

List buckets

s3 = boto3.client('s3')
response = s3.list_buckets()
for bucket in response['Buckets']:
    print(bucket['Name'])

Delete a bucket

s3 = boto3.client('s3')
s3.delete_bucket(Bucket='my-bucket')

Amazon EC2 (Elastic Compute Cloud)

EC2 is a service that provides resizable compute capacity. With Boto3, you can manage instances, security groups, and key pairs.

Launch an EC2 instance

ec2 = boto3.resource('ec2')
instance = ec2.create_instances(
    ImageId='ami-0abcdef1234567890',
    MinCount=1,
    MaxCount=1,
    InstanceType='t2.micro'
)

Stop an instance

ec2 = boto3.client('ec2')
ec2.stop_instances(InstanceIds=['i-1234567890abcdef0'])

Terminate an instance

ec2 = boto3.client('ec2')
ec2.terminate_instances(InstanceIds=['i-1234567890abcdef0'])

List all instances

ec2 = boto3.resource('ec2')
for instance in ec2.instances.all():
    print(instance.id, instance.state)

Amazon DynamoDB

DynamoDB is a fully managed NoSQL database service. Boto3 lets you interact with tables, items, and queries.

Create a table

dynamodb = boto3.resource('dynamodb')
table = dynamodb.create_table(
    TableName='Users',
    KeySchema=[
        {
            'AttributeName': 'user_id',
            'KeyType': 'HASH'
        }
    ],
    AttributeDefinitions=[
        {
            'AttributeName': 'user_id',
            'AttributeType': 'S'
        }
    ],
    ProvisionedThroughput={
        'ReadCapacityUnits': 10,
        'WriteCapacityUnits': 10
    }
)

Put an item in a table

table = boto3.resource('dynamodb').Table('Users')
table.put_item(Item={'user_id': '123', 'name': 'John Doe'})

Get an item from a table

table = boto3.resource('dynamodb').Table('Users')
response = table.get_item(Key={'user_id': '123'})
print(response['Item'])

Delete an item

table = boto3.resource('dynamodb').Table('Users')
table.delete_item(Key={'user_id': '123'})

AWS Lambda

AWS Lambda allows you to run code in response to events without provisioning servers. With Boto3, you can manage Lambda functions.

Create a Lambda function

lambda_client = boto3.client('lambda')
with open('my-deployment-package.zip', 'rb') as f:
    zipped_code = f.read()

lambda_client.create_function(
    FunctionName='MyLambdaFunction',
    Runtime='python3.8',
    Role='arn:aws:iam::123456789012:role/service-role/MyLambdaRole',
    Handler='lambda_function.lambda_handler',
    Code=dict(ZipFile=zipped_code),
    Timeout=300
)

Invoke a Lambda function

lambda_client = boto3.client('lambda')
response = lambda_client.invoke(
    FunctionName='MyLambdaFunction',
    InvocationType='RequestResponse',
    Payload=b'{"key": "value"}'
)
print(response['Payload'].read())

Delete a Lambda function

lambda_client = boto3.client('lambda')
lambda_client.delete_function(FunctionName='MyLambdaFunction')

Amazon SNS (Simple Notification Service)

SNS allows you to send notifications via email, SMS, or to other AWS services.

Create an SNS topic

sns = boto3.client('sns')
response = sns.create_topic(Name='MyTopic')
topic_arn = response['TopicArn']

Subscribe to an SNS topic

sns = boto3.client('sns')
sns.subscribe(
    TopicArn=topic_arn,
    Protocol='email',
    Endpoint='example@example.com'
)

Publish a message to an SNS topic

sns = boto3.client('sns')
sns.publish(
    TopicArn=topic_arn,
    Message='This is a test message.'
)

Handling Credentials in Boto3

Boto3 automatically looks for credentials in the following order:

  1. Environment Variables:
    • AWS_ACCESS_KEY_ID
    • AWS_SECRET_ACCESS_KEY
    • AWS_SESSION_TOKEN (optional)
  2. Shared Credentials File (~/.aws/credentials):
  • Profiles can be configured here, like [default].
  1. EC2 Instance Metadata:
    • For instances running in AWS, Boto3 can retrieve credentials from the metadata service.

You can also pass credentials explicitly

session = boto3.Session(
    aws_access_key_id='YOUR_ACCESS_KEY',
    aws_secret_access_key='YOUR_SECRET_KEY',
    region_name='us-west-2'
)
s3 = session.resource('s3')

Error Handling in Boto3

Boto3 includes exceptions that you can catch to handle errors gracefully.

import boto3
from botocore.exceptions import NoCredentialsError, PartialCredentialsError

try:
    s3 = boto3.client('s3')
    s3.list_buckets()
except NoCredentialsError:
    print("Credentials not available.")
except PartialCredentialsError:
    print("Incomplete credentials provided.")

Optimizations in Boto3

  1. Multi-threading with concurrent.futures: You can speed up Boto3 operations (especially with S3) by using multi-threading or multi-processing.
from concurrent.futures import ThreadPoolExecutor

def upload_file(bucket, filename):
    s3 = boto3.client('s3')
    s3.upload_file(filename, bucket, filename)

with ThreadPoolExecutor(max_workers=5) as executor:
    executor.map(upload_file, ['my-bucket'] * 10, ['file1.txt', 'file2.txt', ...])

  1. Session Caching: Reuse the same session across multiple Boto3 resource or client invocations to avoid repeatedly initializing the session.

  2. Using Paginators: For services like S3 that return paginated results, using paginators avoids issues with large datasets.

Back to top