The Great S3 Showdown - Express One Zone vs Standard

In the vast expanse of cloud storage, we have a clear fan-favorite, known for its reliability and versatility, the champion of object storage division, the S3! But wait, the lights go dim… From Re:Invent 2023 Las Vegas, we have a challengerrrrrrrr… boasting unprecedented speed and performance, it’s AWS S3 Express One Zone

Hello, Internet, today we’ll explore a fun new AWS offering, the S3 Express One Zone, the with some practical usage scenarios, see whether it’s worth the buzz

Hopefully, no one will notice the spelling mistakes! :)

The Amazon S3 Express One Zone storage class is built to provide lowest latency and highest performance in object storage. AWS claims it will be ten times faster to standard storage class. It can scale to handle hundreds of thousands of requests per second while maintaining single-digit millisecond latency.

By the way, if this sparks your curiosity, I highly recommend this blog from Jeff Bar and the User Guide

S3 Express One Zone Architecture. Image Credits — AWS

As you might expect, this is designed specifically for storing frequently accessed datasets, training ML models, and performing data analytics. The objects are stored and replicated in a single Availability Zone that you choose, allowing you to co-locate storage and compute resources to further decrease latency.

With such bold claims, this could very well be the storage Data Engineers can only dream of…. or just another tale spun from the marketing! Well, there’s only one way to find out — by rolling up our sleeves and diving into the resources ourselves.

Section 1: Setting Up S3 Express One Zone Bucket

With S3 Express One Zone, you have a bunch of changes in the creation process — And you’ll notice the first change right away! We have a new bucket type called “Directory” that will only be utilized for S3 One Zone. In case you are wondering, the “General Purpose” allow objects that are stored across all storage classes except S3 Express One Zone

Create S3 Express One Zone S3 Bucket

Once you go with Directory type, you must now select the AZ in which the bucket will be created. Make sure to place it in the same AZ as your computing resources.

And for the third change, the bucket will have a new naming format too. bucket_name--region-az--x-s3

Next, we will discuss the object ownership and the Access. Standard S3 buckets allow us to set object ownership to either the current account or a separate account. However with Directory buckets, all objects will be owned by the current account.

Secondly, in Directory buckets, all Public access will be blocked by default and AWS do not provide the option to change and modify it like in Standard buckets.

Finally, the server side encryption is enabled by default and encrypted with Amazon S3 managed keys (SSE-S3). You do not have the same encryption options as in Standard.

Object ownership, Access and Encryption

Section 2: Setting up a SageMaker Notebook Instance & a Jupyter Notebook

Now that our S3 Buckets are ready, let’s get to the SageMaker. Now this experiment is purely for data retrieval speeds, so we don’t have to do anything fancy.
I’ve simply created a SageMaker Notebook instance. However, there is one key element to consider: place it in the same AZ as the One Zone bucket. Again, I’ve used very moderate specs as we are not intending to do much here.

Here’s my SageMaker Notebook

Now open the Jupyter Notebook with conda_python3, and we’re ready to begin! I utilized six datasets, ranging from 1MB to 16GB, to compare performance to file size.

Below is my python script to measure the download speeds

Section 3: Benchmarking the Performance

Ding! Ding! Ding! So, let the games begin! The two opponents clash head to head for the tile.

Round 1 : I didn’t notice much difference when the file sizes were under 5MB. For the modest 2MB dataset, here’s the result I’ve got

Standard S3 Read Time: 0.06330442428588867 seconds
S3 Express One Zone Download Time: 0.0615943832397461 seconds

Round 2 : We escalate to the mid-size datasets which hovers around 3GB. The results? A revelation, a turning point in our grand little experiment. S3 One Zone outpaced its opponent and showed a clear difference in speeds

Standard S3 Download Time: 40.768924713134766 seconds
S3 Express One Zone Download Time: 29.283695936203003 seconds

Round 3 : We move to the Heavyweight ̶C̶h̶a̶m̶p̶i̶o̶n̶s̶h̶i̶p̶ Dataset of 20GB. … And the results are significantly clearer, with S3 One Zone takes whopping 30 seconds less compared to standard S3.

Standard S3 Download Time: 195.38067507743835 seconds
S3 Express One Zone Download Time: 166.30217003822327 seconds

Bonus Round : Instead of a monstrous single 20GB file, this time we have 20GB worth of 2MB files! And the results blew me away. One Zone knocked this one out of the park and nearly 5x faster compared to Standard.

Total Standard S3 Download Time: 112.73444819450378 seconds
Total S3 Express One Zone Download Time: 22.653493642807007 seconds

Worth mentioning that aws s3 sync does not work with directory buckets yet :( You’ll have to write a script to upload the files individually via aws s3 cp

In this experiment thought, we can treat as a data migration. I imported the data from the standard S3 bucket to the Express One Zone directory Bucket, which was way faster

Curiosity got the better of me again! I wanted to test one more thing, but I promise, this would be the last!
What if the SageMaker Notebook instance was in a different AZ than the Express One Zone bucket?
Well, no much difference to be honest, and still beats Standard by a fair margin

Total Standard S3 Download Time: 119.41048288345337 seconds
Total S3 Express One Zone Download Time: 25.05690097808838 seconds

The dust settles on this Great S3 showdown. S3 Express One Zone has thrown down the gauntlet, challenging the reigning champ of object storage, Standard S3, for the championship of frequently accessed datasets; and emerged victorious!

Ding! Ding! Ding! And your new champion… S3 Express One Zone!

Amazon S3 Express One Zone shone with its low-cost, high-performance storage capabilities, proving that when it comes to lightning-fast data access, it’s in a league of its own. Its unique architecture, which includes innovations like directory buckets and session-based access, adds to remarkable efficiency, which can considerably boost your data-driven projects.
Moreover, the import functionality simplifies the dreaded task of data migration from general-purpose buckets. The batch operation dashboard keeps track of your progress, putting you in control at all times.

However, though S3 One Zone is tempting, it’s essential to design and plan for the limitations too. Bucket placement and the implications of storing your data in a single zone require careful consideration. Durability in a single zone, while sufficient for many use cases, may not fit all.