S3: Object Storage

Amazon Simple Storage Service (S3) is one of the most widely used AWS services. It provides virtually unlimited, highly durable object storage for any type of data. In this lesson, we'll explore how S3 works, create your first bucket, and learn best practices for using S3 effectively.

What You'll Learn

By the end of this lesson, you'll understand S3 concepts like buckets and objects, storage classes, access control, versioning, and common use cases for object storage.

What is Object Storage?

Before diving into S3, let's understand object storage:

Traditional file storage organizes data in a hierarchy of folders and files (like your computer's file system).

Object storage stores data as objects in a flat structure. Each object has:

• Data - The actual content (file)
• Metadata - Information about the object (size, type, custom tags)
• Unique identifier - A key to access the object

Object storage excels at storing large amounts of unstructured data at scale.

What is Amazon S3?

Amazon S3 stores objects (files) in containers called buckets. It's designed for:

• Durability - 99.999999999% (11 nines) durability
• Availability - 99.99% availability
• Scalability - Virtually unlimited storage
• Security - Flexible access controls

S3 Use Cases

Use Case	Example
Static website hosting	Host HTML, CSS, JS files
Data backup and archive	Store database backups
Media storage	Store images, videos, audio
Data lake	Store analytics data
Application assets	Store user uploads
Log storage	Store application and access logs
Machine learning	Store training data and models

Core S3 Concepts

Buckets

A bucket is a container for objects. Key points:

• Globally unique name - Bucket names must be unique across ALL AWS accounts
• Region-specific - Created in a specific region (though names are global)
• Flat structure - No actual folders (prefixes simulate folders)

Naming rules:

• 3-63 characters
• Lowercase letters, numbers, hyphens
• Must start with letter or number
• Cannot be formatted as IP address

Objects

Objects are the files stored in buckets:

• Key - The object's name (including any "path" prefixes)
• Value - The actual data (up to 5 TB per object)
• Metadata - System and user-defined information
• Version ID - Unique ID if versioning is enabled

Keys and Prefixes

While S3 is a flat storage system, you can use prefixes to organize objects:

my-bucket/
├── images/
│   ├── photo1.jpg     ← Key: "images/photo1.jpg"
│   └── photo2.jpg     ← Key: "images/photo2.jpg"
├── documents/
│   └── report.pdf     ← Key: "documents/report.pdf"
└── index.html         ← Key: "index.html"

The console displays this as folders, but technically they're all just keys with / in the name.

Creating Your First S3 Bucket

Step 1: Open S3 Console

1. Go to AWS Console
2. Search for "S3"
3. Click "Create bucket"

Step 2: Configure Bucket

1. Bucket name: Choose a unique name (e.g., my-first-bucket-abc123)
2. Region: Select your preferred region
3. Object Ownership: Keep "ACLs disabled" (recommended)
4. Block Public Access: Keep all options checked (recommended for security)
5. Bucket Versioning: Disable for now (we'll discuss later)
6. Click "Create bucket"

Step 3: Upload Objects

1. Click on your bucket name
2. Click "Upload"
3. Drag and drop files or click "Add files"
4. Click "Upload"

Step 4: Access Your Object

Each object has a URL:

https://my-first-bucket-abc123.s3.us-east-1.amazonaws.com/myfile.txt

By default, this URL won't work publicly due to access controls.

S3 Access Control

S3 provides multiple layers of access control:

Block Public Access

Account and bucket-level settings that prevent public access. Keep these enabled unless you specifically need public access.

Settings:

• Block all public access (recommended default)
• Block public ACLs
• Block public bucket policies
• Ignore public ACLs
• Restrict public bucket policies

Bucket Policies

JSON documents that define access rules for a bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

This policy makes all objects in my-bucket publicly readable.

IAM Policies

Control which IAM users/roles can access S3:

{
  "Effect": "Allow",
  "Action": [
    "s3:GetObject",
    "s3:PutObject"
  ],
  "Resource": "arn:aws:s3:::my-bucket/*"
}

Pre-signed URLs

Generate temporary URLs for private objects:

import boto3

s3_client = boto3.client('s3')

url = s3_client.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-bucket', 'Key': 'private-file.pdf'},
    ExpiresIn=3600  # URL valid for 1 hour
)

Use pre-signed URLs to share private content temporarily without making it fully public.

S3 Storage Classes

S3 offers different storage classes optimized for various access patterns and costs:

Frequently Accessed

Class	Use Case	Availability
S3 Standard	Frequently accessed data	99.99%
S3 Express One Zone	Low-latency access, single AZ	99.95%

Infrequently Accessed

Class	Use Case	Minimum Storage
S3 Standard-IA	Infrequent access, multi-AZ	30 days
S3 One Zone-IA	Infrequent access, single AZ	30 days

Archive

Class	Use Case	Retrieval Time
S3 Glacier Instant	Archive with instant access	Milliseconds
S3 Glacier Flexible	Archive, occasional access	1-12 hours
S3 Glacier Deep Archive	Long-term archive	12-48 hours

Intelligent-Tiering

S3 Intelligent-Tiering automatically moves objects between tiers based on access patterns. Great when you don't know access patterns upfront.

Choosing a Storage Class

• Unknown access patterns → Intelligent-Tiering
• Frequently accessed → Standard
• Infrequent, but needs fast access → Standard-IA
• Archive with occasional access → Glacier Flexible
• Compliance/long-term archive → Glacier Deep Archive

S3 Versioning

Versioning keeps multiple versions of an object:

• Protect against accidental deletion - Deleted objects can be restored
• Maintain object history - Access previous versions
• Required for replication - Cross-region replication needs versioning

Enabling Versioning

1. Go to bucket properties
2. Click "Edit" on Bucket Versioning
3. Select "Enable"
4. Save changes

How It Works

my-file.txt
├── Version: abc123 (current)
├── Version: xyz789 (previous)
└── Version: def456 (oldest)

When you upload a new version, the old version is kept. Deleting adds a "delete marker" but doesn't remove old versions.

Note: Versioning increases storage costs since all versions are stored.

S3 Lifecycle Policies

Automate transitioning objects between storage classes or deleting old objects:

Example Lifecycle Rule

{
  "Rules": [
    {
      "ID": "ArchiveOldFiles",
      "Status": "Enabled",
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 90,
          "StorageClass": "GLACIER"
        }
      ],
      "Expiration": {
        "Days": 365
      }
    }
  ]
}

This rule:

• Moves objects to Standard-IA after 30 days
• Moves to Glacier after 90 days
• Deletes after 365 days

Static Website Hosting

S3 can host static websites (HTML, CSS, JS, images):

Setting Up

1. Enable static website hosting in bucket properties
2. Set index document (e.g., index.html)
3. Set error document (optional, e.g., error.html)
4. Make bucket content public (via bucket policy)
5. Upload your website files

Bucket Policy for Website

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-website-bucket/*"
    }
  ]
}

Your website is available at:

http://my-website-bucket.s3-website-us-east-1.amazonaws.com

For HTTPS and custom domains, use CloudFront (covered later).

S3 Event Notifications

Trigger actions when events occur in your bucket:

• Object created
• Object deleted
• Object restored from Glacier

Destinations

• Lambda - Run code when files are uploaded
• SNS - Send notifications
• SQS - Queue messages for processing
• EventBridge - Route to many AWS services

Example: Process Uploads with Lambda

1. Configure S3 event notification for "Object created"
2. Set destination as Lambda function
3. When files are uploaded, Lambda automatically processes them

Use cases: Image resizing, file validation, metadata extraction.

S3 Pricing

S3 pricing has several components:

Component	What You Pay For
Storage	GB stored per month
Requests	PUT, GET, LIST operations
Data Transfer	Data transferred out of S3
Management	Features like analytics, inventory

Free Tier

• 5 GB storage (Standard)
• 20,000 GET requests
• 2,000 PUT requests

Cost Optimization Tips

• Use appropriate storage classes
• Set up lifecycle policies
• Delete unnecessary versions
• Use S3 Intelligent-Tiering for unpredictable access

Key Takeaways

• S3 provides virtually unlimited, highly durable object storage
• Buckets are containers with globally unique names; objects are your files
• Storage classes optimize for different access patterns and costs
• Block Public Access should stay enabled unless you specifically need public access
• Bucket policies and IAM policies control who can access your data
• Versioning protects against accidental deletion and maintains history
• Lifecycle policies automate transitioning and cleanup
• Static website hosting lets you host simple websites directly on S3

What's Next

Now that you understand S3 for object storage, let's look at structured data storage. In the next lesson, we'll explore Amazon RDS - managed relational databases in the cloud.