When you’re starting to use Amazon S3, it might seem not so intricate at first sight. But as soon as you delve into it any further, a range of problems arise. The Console, access key, buckets, objects… What else would you bump in instead of “normal” files & folders?
Yeah… AWS isn’t like shooting fish, especially for a new user. It has a learning curve & more advanced configuration options as compared to easy-peasy solutions like Google Drive. We’ve even written a detailed guide on how to upload and manage files in Amazon S3. But there is a lot more to it.
Buckets are the first you deal with when logging in to Amazon S3. And so we’d like to shed light on how to create & manage S3 buckets. But not so fast! We’ll start with the basic – what is an Amazon S3 bucket in the first place?
Amazon S3 bucket: Definition & key terms
What is the AWS S3 bucket?
When you first open Amazon S3, you might be surprised by its unusual terminology. And their documentation won’t be of great help to you: some sorts of buckets, objects, and many other unfamiliar terms.
But don’t let yourself be misled. An S3 bucket is a traditional root folder – just with a different principle of work.
A root folder means the highest level of hierarchy in folder structure i.e. you cannot place a bucket into another bucket. Still, you put objects (our traditional files, plus their metadata) into buckets, as usual.
You must understand that Amazon S3 isn’t anything different than other storages we know (Google Drive, OneDrive, etc.). It just calls traditional items for us different names.
Still, there are a couple of nuances you must know. An S3 bucket has its specifics like regions or naming conventions (we’ll talk about all this later). The storage itself has a bit different nature – it’s an object storage i.e. everything you upload there gets transformed and stored as objects.
But if you simplify it all, a bucket becomes a root folder that helps you store and organize your objects. And an object is simply the data you store. And just not to confuse you any further, we’ll use traditional Amazon terms in this post.
What can I do with my bucket?
Buckets aren’t a parent folder that we’re all used to. And so your bucket operations are somewhat limited. You can create a bucket, upload & download objects from there. You can also ask for the bucket location & a list of all available buckets in your S3 storage. Deleting and emptying your bucket is also possible.
What you cannot do:
- Move your buckets. Again, they’re NOT traditional directories so you cannot place one bucket into another, for example.
- Change the name of your bucket. These are unique & chosen once and forever. We’ll talk about it more.
- Rearrange the order. By default, your buckets go in alphabetical order. You can make them go in reversed order, but that’s it.
What are bucket size limits?
Fortunately, there aren’t any size limits for your buckets. You can store any number of objects you need – and Amazon claims to pose no limits on your storage space.
The only you should remember is the 100 bucket limit per account. Though, you could adjust this restriction by revisiting your account settings.
What other terms should I know?
If we’re speaking about buckets, there exist so-called bucket policies. Chosen individually for your S3 buckets, these specify who can do what within your bucket. Of course, no need to grant admin rights to all your users. Sometimes, read-only permissions are enough, and this is where bucket policies will come in handy to you.
Access control lists (ACLs) also affect the safety level in your S3 storage. These allow you to change bucket access settings. For example, you could specify a user or a group of users you’d like to grant access to your bucket. Or you point out the type of access you want to honor them with.
But you’d better be very cautious about your ACLs. Misconfigured ACLs are a common reason for data breaches in S3 storage. And you rarely need them in practice unless you’re an experienced S3 user. As a rule, those settings chosen by default must be enough for you at the beginning.
Create an S3 bucket
Steps to create an Amazon S3 bucket
Creating an S3 bucket is actually very easy. You just go to your AWS Console, select Services. In the storage section, you choose S3. And then when you open the storage, there is a big orange Create bucket button in the right upper corner of your screen.
Overall, there are two main things you need to know to create buckets in Amazon S3 – names & regions.
Naming conventions for S3 buckets
Unlike other storage solutions, S3 relies on globally unique names for its buckets. This means you cannot use the same name across different buckets. You cannot rename an S3 bucket after you created it as well.
For AWS storage, your bucket name is very important. This is a sort of identification for your data as you’ll use bucket names in URLs to access your objects. So be very attentive when choosing the name & adhere to AWS naming rules and practices. Here are a few of them for you to follow:
- Use a unique bucket name across your S3 account;
- Don’t go over 63 characters, but use not less than 3 characters as well.
- Avoid uppercase letters, underscores, and the use of dashes/hyphens at the end or next to periods.
- Begin & end your bucket name with a lowercase letter and/or a number.
- Refrain from using any sensitive info in your bucket names like your IP address or account numbers. As said, your bucket name will be reflected in its URL.
As a result, a valid name for you S3 bucket could look like this:
Variants like myexamplebucket.com or my.example.bucket are Ok but not recommended by Amazon. In turn, what that Amazon won’t definitely allow you to use are the next names:
- MyExampleBucket (uppercase letters);
- my-example-bucket- (a hyphen at the end);
- My_example_bucket (underscores).
And what about regions?
An opportunity to choose the region where your bucket will reside is another peculiarity of Amazon, which differs it from its competitors.
Yes, you must specify the region where you want to store your objects. And try to be as selective as possible since this will impact the costs, latency, and even regulatory requirements to your storage.
As a general guideline, Amazon recommends choosing the region that is the closest to you geographically. Following this logic, if you locate in Europe, choose between Europe (Ireland) and Europe (Frankfurt) regions. And so you’ll reduce your expenses & improve the performance of your AWS storage.
Secure your Amazon S3 bucket
During your bucket creation, you’ll meet a few blocks of security settings. The safety & security of your storage will depend on how you configure your buckets.
Fortunately, your S3 storage provides strong security settings by default, similar to any other Amazon-driven service.
Data leakage still happens as nobody is above making a mistake. 198 million US voter profiles that were stored in Amazon S3 went public in 2017 because of the insecure S3 bucket. In February, 2020, the English care home informed about the leak of 10,000 resident records caused by the same mistake.
If you don’t want to hit another scandal headline in cybersecurity, you’d better secure your AWS bucket. And here we talk about three specific ways of how to achieve this.
1. Manage your bucket permissions
We slightly touched on permissions in the beginning. But we still need to cover all the hierarchy so you had a better understanding of it. Generally, there are four methods to apply permissions in Amazon S3 storage:
- IAM (Identity and Access Management) policies are the most general access rules applied to the whole AWS Cloud. So they’ll impact your buckets rather indirectly.
- Then come bucket policies, which center around the data in your buckets. For example, you grant your users with read (= viewing) and/or write (editing, uploading, deleting, etc.) permissions.
- With Access Control Lists (ACLs), you manage access to your resources (buckets & objects). As said, you choose specific users who can access your buckets & the type of access they would have.
- Query string authentication & URLs-based access is a way to provide temporary access to your buckets. For example, you need to grant one-time access to your partner so they could upload the materials. And so you provide permissions based on a specific URL.
Fortunately, Amazon S3 has good security measures by default. So if you’re creating the first bucket in your life, you can omit the permission section before you learn it better.
2. Be cautious about public access
When creating an S3 bucket, you’re free to choose whether you need to set it as public or private. Public access means that literally everyone can access your bucket on the condition that they have its URL.
You might ask who on earth would want their private info to be publicly accessible. Sometimes, you still need it like when a photographer needs to share raw files with the client. And it’s much simpler to provide a link to the bucket where photos are stored than to send them one by one.
But this example is more an exception rather than the rule. In most cases, companies want their materials to be private & secure, while public access equals great damage to their reputation.
So be attentive when it comes to choosing private vs. public access. Again by default, all Amazon buckets are private:
And even if you uncheck the boxes above, you’ll have to go through a range of steps to set up a publicly accessible bucket. For one thing, AWS will give a warning message concerning public access to your bucket:
According to S3 policy, the storage recommends avoiding public access unless you have very specific goals. For another thing, you’ll have to be very careful what public access settings you need. For instance, you could restrict public access to new ACLs only.
By the way, it's easy to check whether your bucket is private or public. Just see the access column in your S3 window:
In your Console, you can also edit public/private access if needed. Just choose the needed bucket in the list. Then click Permissions > Edit > Block all public access (or tick the one you need) > Save.
3. Set up encryption
One more awesome thing about Amazon S3 is default encryption. This is one more layer of protection for your data, preventing hackers from accessing your info. Thanks to encryption, your info is converted into a code. So no one can see the objects without unauthorized access.
But don’t be confused by the “default” term. In S3 storage, you have to enable default encryption to make it work. This is one more thing you set up when creating a bucket:
How do I know my S3 buckets are secure enough?
One of the primary benefits of S3 is its compatibility with other tools. And security isn’t an exception here. Wanna be sure that your buckets are safe & private – use the tools available to you.
For example, Amazon offers you several cloud-native options to track access to your S3 buckets:
- AWS CloudTrail will help you record & see your bucket-level actions.
- AWS Config is a service to monitor your AWS configurations, such as ACLs and bucket policies to prevent any policy violations.
- Developed for more tech-savvy people, AWS Cloudwatch shows you resource utilization across your AWS account, Amazon S3 included.
- AWS Trusted Advisor provides you with real-time data regarding your AWS infrastructure, security, costs, and performance. But the best news here is available AWS Trusted Advisor S3 Bucket Permissions Check, which became free of charge in 2018.
Also, consider incorporating third-party tools, which won’t cost you an arm and a leg as compared to official Amazon services. Different cloud management platforms like Arcus or Dynatrace will scan your buckets & inform you about any potential threats timely.
Upload objects to your Amazon S3 bucket
Your bucket is ready. Our next step is to upload objects there. Just returning to S3 terminology, there are no files in your S3 storage – only objects. These are composed of files, plus their metadata (optionally) and could be of any format you know: text file, image, video, audio, etc.
So go to your bucket list > select the bucket > click on Upload (a big orange button). Now you have a couple of options: you can drag’n’drop your files or point-and-select them with the Add files button.
You can also upload your materials selectively or a whole folder at once. The good news is that Amazon S3 mirrors your folder structure. So you won’t spend all day putting your objects in proper order.
During the upload process, you can check the destination of your files one more time as well as the details. Destination details depend on those settings configured for your buckets originally, for example, chosen encryption or versioning. Don’t forget to save your results with the Upload button in the bottom part of your page.
Read also about Amazon S3 acceleration tools to upload files faster to the AWS storage.
Access your S3 bucket & download the objects
How to access your S3 bucket?
As you’ve probably understood, Amazon S3 Console is your main tool to manage your buckets. There you can perform any bucket operations without writing a line of code. So you just open the console & see all your buckets.
Still, there are alternative ways to access your buckets. And these are less time-consuming & save you from repetitive tasks.
For example, you might choose to access your buckets through so-called bucket URLs. Amazon S3 supports both virtual-hosted & path-style URLs so use whatever is more convenient to you.
A virtual-hosted style allows you to customize your URL as your bucket name goes first. In this case, your bucket URL will be website-like:
- http://testbucket.aws.s3-us-west-2.amazonaws.com, where testbucket.aws is your bucket name & us-west-2 is your region.
In path-style access, the subdomain is fixed (s3.amazonaws.com) & you cannot change it to your own needs:
- http://s3.amazonaws.com/testbucketl.aws, where testbucket.aws is your bucket name.
- http://s3-us-west-2.amazonaws.com/testbucket.aws, where testbucket.aws is your bucket name & us-west-2 is your region.
If you’re using other AWS services, there is even an easier way to access your bucket through S3://bucket. Just use the following URL:
- s3://testbucket.aws, where testbucket.aws is your bucket name.
How to download from an Amazon S3 bucket?
The same as uploading, downloading objects should be as easy as ABC for you. Go to your bucket list & select the one you need. Then multi-select the object(s) > click Actions > Download / Download as.
As usual, you’ll have to specify where you want to download the object & then save it.
You cannot download batch objects from the AWS Console. Downloading multiple files is possible only through the AWS Command Line Interface (AWS CLI). This is an open-source tool, enabling you to run commands to interact with AWS services. But as you understand, you’ll need some basic knowledge of technology in this case.
NB: Pay attention that you cannot preview objects in Amazon S3 like in similar cloud storage solutions. Stored as objects, your materials look all the same, regardless of their content or format.
Empty and/or delete your bucket
To empty the bucket, you choose the right bucket in your bucket list & then click Empty. Then, you’ll be asked to type permanently delete in the field to be able to empty the bucket. Press Empty – and all your objects will be deleted. Don’t forget that versions will also disappear if you enabled versioning for your bucket.
The deletion procedure is very similar. Still, there are a few things to remember:
- You won’t be able to delete the bucket unless it’s empty;
- You can reuse the bucket name when the bucket is deleted. But if you’re planning to use the same bucket name any further, it’s better not to delete the bucket. Just empty it.
- Amazon warns you that some issues could appear preventing you from reusing the name. For example, it may take some time before the name will become available to you again.
To delete the bucket, go to the list of buckets > select the one you need > press Delete. Then you’ll be asked to type the bucket name to confirm the deletion. Finally, click Delete.
Configure advanced settings for your bucket
There is a whole list of additional features you can set up for your Amazon S3 bucket. You add these right away during the creation process. Or just go to the Proprieties section when your bucket is ready.
So what can you add/change here?
- Versioning will be a real finding for you in case of accidental overwrites or deletes. This is also a good help to avoid duplicates.
- Server access logging will help record & monitor access requests to your buckets. With this functionality, you get updates on who requested access to your bucket, what bucket it was, when it happened, and the response status.
- You can use your bucket for static website hosting to store your website content.
- Default encryption will help you secure your data.
- Object locking is a useful feature to prevent your objects from being mistakenly deleted.
- Allocate your costs, categorize, and manage them with tagging.
- Transfer acceleration is a relatively new feature added to provide you with fast & secure data transfers to & from your buckets.
- Enable events if you want to receive notifications about any updates in your buckets.
- By default, the bucket owner takes responsibility for any payments in Amazon S3. But you can change it with requestPayment so the storage will charge users for downloads from your bucket.
- Plan the lifecycle of your objects ahead. With this feature, you can archive your objects after some time, delete them in 5 years after creation, etc.
- Replication will make your life easier if you need to copy objects across buckets. This configuration will help you do it automatically & effortless.
Amazon S3 makes up a perfect solution to scale your files & don’t overpay for space. Plus, it’s highly compatible with other Amazon services and its infrastructure, which many businesses choose for developing their applications.
A steep learning curve is its only disadvantage. But we’re not afraid of difficulties, are we? Amazon S3 is a perfect solution – just many things that we’re used to work a bit differently here.
This is actually why we prepared this post. If you follow our instructions & go point by point in bucket creation or management, you won’t meet any difficulty. And you can always peep at our detailed Amazon S3 guide to familiarize yourself with S3 infrastructure.
And don’t forget that you can power up your S3 storage with an advanced digital management tool. Tagging & metadata, shareable assets & public websites, advanced search capabilities, and anything you need to manage your data the most productively!