At some point, every engineer has to decide whether to write tests for something or just ship the feature and move on.
If a Bucket is Private, the ACL returned for the Bucket and any files within the bucket will be 'PRIVATE'. If a Bucket is Public, the ACL returned for the Bucket and any files within the bucket will be 'PUBLICREAD'. The S3 Compatible API supports the Put Bucket ACL call to change between the Get Object ACL and Get Bucket ACL calls only. Awesome backup replication storage s3 nfs s3-bucket s3-storage ceph awesome-list glusterfs high-availability san nas curated-lists s3-backup redundancy awesome-lists s3-client encription Updated Dec 3, 2020. Replace the BUCKETNAME and KEY values in the code snippet with the name of your bucket and the key for the uploaded file. Downloading a File ¶ The example below tries to download an S3 object to a file. Create AWS s3 bucket. Aws s3api create-bucket -bucket thedbadminbuk -region us-east-1. Indicates whether the copied object uses an S3 Bucket Key for server-side encryption with AWS KMS (SSE-KMS). RequestCharged (string) -If present, indicates that the requester was successfully charged for the request. S3.Client.exceptions.ObjectNotInActiveTierError; Examples. The following example copies an object from one bucket to.
Under a time crunch, I'll often write tests for the easy things (e.g. pure functions) or write the tests that provide the biggest bang for their buck (e.g. an end-to-end integration test for the service).
Testing code that interacts with external systems, like a database or S3, requires a bit more effort. However, important business logic often happens in this code and recently I've become more interested in testing it.
In this post, I'll explore three ways of testing S3 in Python.
Setup
Let's consider a simple CRUD app for recipes, backed by S3.
All tests below use pytest. All code is runnable and available on Github.
Option 1: moto
Moto is a Python library that makes it easy to mock out AWS services in tests. Let's use it to test our app.
First, create a pytest a fixture that creates our S3 bucket. All S3 interactions within the mock_s3 context manager will be directed at moto's virtual AWS account.
Next, we can test creating a new Recipe and fetching it.
If we try to fetch a Recipe that doesn't exist, an exception should be raised. This test covers that scenario.
We can also update a Recipe. This test confirms that the data is updated after save() is called.
Finally, we can delete a recipe and confirm that the data in S3 disappears.
Overall, moto does a great job of implementing the S3 API. It's easy to install, feels just like the real S3, and doesn't require any code changes.
Option 2: Botocore stubs
Botocore stubs allow you to mock out S3 requests with fake responses.
Below is a pytest fixture that creates an S3 stub. Since other S3 clients won't use this stub, we also need to patch get_s3 and replace its return value with the stub - thereby forcing all S3 clients in the Recipe class to use our stub.
Then, we can stub out responses for the put_object and get_object S3 APIs. With those stubs in place, we can run the test that creates and subsequently fetches a Recipe.
While botocore stubs are functional, I don't like working with them for several reasons:
They require a lot more prep. Creating stubs is time-consuming. Even if you run the real code interactively and copy the response, some things need to be replaced - such as the StreamingBody above.
They're fragile and fake. Responses are returned first in, first out - so if you call the S3 APIs in a different order than you added the responses, it will throw an error. If you have a bug in how you call the API, it might not be caught.
To make the stubs look somewhat realistic, you have to mock many fields that your code doesn't care about and bloat your tests with fake responses.
They leak implementation details from the module being tested. For example, if a module switched from using
s3.list_objectstos3.list_objects_v2, the test would fail because it depends on a specific API being called. This creates an unnecessary dependency on the private API of the module, instead of testing the public API.
Option 3: Localstack
A third option is localstack, which allows you to bring up an entire AWS cloud stack locally. Import excel into numbers ipad.
First, we need to bring up localstack. I choose to do this with docker-compose.
Next, we mock get_s3 again and this time replace it with an S3 client that is connected to localstack.
With this mock in place, we can run the same tests that we ran with moto.
Localstack is extremely easy to use, but it takes almost 30 seconds to spin up on my machine.
S3 Bucket Client Support
Recommendation
Both moto and localstack are very powerful and easy to work with. Both solutions do a good job of implementing the S3 API, and they also support other AWS services including EC2, RDS, Lambda, and more. They can both be used to test code in other languages in addition to Python.
Localstack is probably the closest thing to actually connecting to AWS, but for my simple use case presented above, I can't justify the extra overhead and time required to spin up the stack. Therefore, I recommend moto as it's the most lightweight solution that properly implements the S3 API.
For more complicated projects that are testing S3 performance, localstack could be a good choice. Botocore stubs don't make the cut.
What did I miss?
S3 Bucket Client App
Thanks for reading! If you have any feedback, I'd love to hear from you - follow me on Twitter or message me on LinkedIn.
