A complete guide to uploading and managing your files in Amazon S3

Updated August 2022

No doubt, we’re spoiled for choice with today’s variety of file storage. Cloud services are the most popular on the market thanks to their accessibility and ease of use. As estimated, there exist over 2300 million cloud storage users across the globe. This figure is expected to grow even further.

Scalable infrastructure and security measures make Amazon S3 a top media library for many.

The top choices come with caveats though. Amazon's object system can complicate the onboarding process for new users. With this article, we’re going to tell you things to be on the lookout for when going for Amazon S3.

As we have recently built DAM integration with Amazon at Pics.io, uploading and managing files on S3 became really important for our users. So here we are, bringing benefit to everyone who needs this info to make working with S3 a piece of cake for you.

Amazon S3 Terminology

As a new Amazon user, you may be puzzled when you first open your account. Where is the traditional file and folder organization? What is a secret key and why are my precious files stored in buckets?

Here is a short list of terms you might want to know before even signing in to your account:

AWS (Amazon Web Services) Management Console. Web-based application through which you access and manage cloud storage. You’ll need your user name & password to sign in to your account.

Root user vs. IAM (Identity and Access Management) user. There are two types of users in AWS. The owner (root user) and users with certain roles and access privileges (IAM users). For security purposes, Amazon recommends reducing the use of root user credentials. Instead, you can create an IAM user and grant them full access.

Access Key ID and Secret Key. Besides console access, there is also programmatic access. To make those calls you'll need WS access keys.

Bucket. In your Amazon S3 Console, you create buckets - parent folder for assets and their metadata. Amazon S3 gives 100 buckets per account, but you can increase this limit by up to 1000 buckets for an extra charge.

Bucket = Object 1 + Object 2 + Object 3

Object. We store objects in buckets that consist of files and their metadata. An object can be any kind of file you need to upload: a text file, an image, video, audio, and so on. The size limit for uploads is 160 GB.

Object = file + metadata (optionally)

Folders. You can group your objects by folders. Amazon S3 has a flat file system. A flat hierarchy is different from a traditional one with directories and subdirectories. For example, you add a project name + client name + due date so you won’t meet the same name across the storage.

Region. Amazon S3 buckets are region-specific. This means you choose the location where you want the company to store your assets. Objects in the bucket won’t leave their location unless you transfer them to a different region.

Key names & prefixes. Key names refer to object names. Together with prefixes, they help you access the needed file quicker and easier. Let’s say you store photo1 in folder1 in your bucket. You can search for files by entering bucket/folder1/photo1 instead of opening folders and buckets.

Getting started with Amazon S3

Creating Amazon S3 Bucket

After you’ve signed in to your AWS Console, it’s time to explore your user account. The first thing you do is to create an Amazon S3 bucket.

Here you need to state your bucket name and location where you want Amazon to store your bucket and its content. The bucket name must:

have a unique name;
be between 3 and 63 characters;
contain only lowercase characters.

As for the region, the storage allows you to create a bucket in the location you want. And the best idea is to choose the one that is the closest to you. In this way, you won’t only reduce response time but will cut costs and meet regulatory requirements.

What else can I do with my bucket?

1) Permissions

In the same menu, you also set permissions and configure options. Depending on the roles in your team, you decide who will create, edit, and delete objects in your bucket.

2) Public vs. individual access

Don't choose public access unless you need to share files with many clients or partners. You can always make particular files publicly accessible to others.

3) Versioning

Enable versioning if you’re planning to store different revisions of the same object. Let’s say you’re designing a new logo for your marketing campaign. There will be many updates to your file when you experiment with the color palette or elaborate on the font.

With versioning, revisions have one key. You access them all at once when accessing the object. Versioning can also help if somebody deleted or edited a file by mistake, as you can revert to the correct version.

4) Server vs. object access logging

Check server access logging if you want to track requests made in a bucket. Access log reports come in handy to you in times of audits and as a safety precaution.

You can also try more advanced object-level logging. On this occasion, you’re free to filter events to be logged, and you track them in CloudTrail - a separate AWS auditing service.

5) Encryption

Encrypt your files if you want to additionally secure your data. When you encrypt data, users can only access it with a password and decryption key. It's a good measure for those that security concerns or requirements. S3 lets you choose the default encryption when you create a new bucket.

Getting inside the bucket

How to upload your files to Amazon S3?

We store our objects in the bucket and use folders if we need to group our files. To upload data to S3 bucket, click upload and select the files that you need. Click on create a folder if you need to group your objects in folders.

Mind that if you want to upload an entire folder, you can only do it with drag and drop. It still simplifies the task if you need to upload a broad scope of files and reflect their structure. With folder upload, Amazon S3 mirrors its structure and uploads all the subfolders.

What else should I know when uploading objects to S3 storage?

As there is no traditional filesystem we won’t speak about names as filenames anymore. This is why when you upload a new object, you won’t even have the possibility to choose a name for it.

But to compensate for non-existing filenames, the service uses an object key (or key name) which uniquely defines an object in the bucket.

What are other configuration options during the upload? As with buckets, you can use encryption to secure your data and manage public permissions. You can also make a particular file accessible to a certain user or user.

Choose storage classes based on how often you’re planning to access your data. S3 Standard (the default type) is for critical, non-reproducible data you’re going to manage on a regular basis.

S3 Tags vs. Metadata

Apart from a key (and data), each S3 object has metadata you set when uploading it. In brief, this is extra information about the object like creation data or author. Metadata storage uses a key-value system. Key helps to identify an object, and value is the object itself.

Content length or file type are the keys when we’re referring to these kinds of metadata. Their values will be the object size in bytes and different file types. PDF, text, video, audio, or any other format you can think about.

You can add tags to your files that help to search, organize, and manage access to your objects. Tags are the same key-value pairs, and they’re like metadata, but with some differences.

An object in S3 is invariable, the same as its metadata. The AWS Console allows you “to edit” metadata, but it doesn’t actually do that. What happens is that each time you change an object, you create its new version.

The situation is different with tags. Tags are extra, “subresource” information about an object. Since they’re managed separately, you won’t change a file when adding tags to it. You can choose up to 10 tags per object in S3.

How to Upload Object that has Metadata?

Uploading an object with metadata to your S3 may feel a bit tricky as you need to do it without GUI.

One way to upload such an object is through the AWS management console. You can find the code snippet you would need to input on AWS's official website.

Folders as a means of grouping objects

How do we use folders in S3?

Buckets and objects play central roles in S3 storage. But this is not the case with folders. Folders compensate for the absent file hierarchy to improve file management and access.

In Amazon S3, folders help you to find your files thanks to prefixes (located before the key name). Let’s say you create a folder named Images, and there you store an object with the key name images/photo1.jpg. “Images” is the prefix in this case. “/” is the delimiter, automatically added by the system (avoid them in your folder names). The more folders and subfolders you create, the more prefixes your file will get.

And so you can use these prefixes to access your data. Just type one or more prefixes into the S3 search engine to filter your searches.

Actions with folders and objects

What you can do with your files and folders is pretty standard in Amazon S3 storage. You can create new folders, delete them, make them public, copy, and move them. You can also change their metadata, encryption, storage class, and tags. No renaming option is available.

Your interaction with objects won’t be very different. With Amazon S3, you’ll have no problem with uploading and copying objects. Plus, you can open your assets, move, download, and delete them (in different formats if needed).

Recovering deleted objects can be especially useful in case of system failures. Mind that “undeleting” objects is possible only in buckets with enabled versioning.

S3 Folder Structure Best Practices

Although it's ultimately up to you, there are a few things we would recommend when using folders.

First, you need to group related files together in separate subfolders. It would be best to avoid having multiple top-level folders. Still, it's best not to go too crazy on the subfolders as it doesn't add too much efficiency; only extra clicks.

Using descriptive names, meanwhile, can help with organizational structure and help you find folders you need long after you've forgotten why you have made them.

Getting back to upload again: Moving big data to Amazon S3

Uploading assets to Amazon S3 should not cause any difficulties. And it is so if we’re speaking about small-scale data. But what if your digital library extends to 1000 files or 10 000, or a million? Can you imagine you drag’n’drop these files or point-and-click them?

What a waste of time it could be! Fortunately, there are other, easier and faster ways to move massive data to your S3 storage…

Online tools

1) Direct Connect is an excellent solution for transferring large amounts of data. Its idea is to create a direct connection between your on-premise data sources and Amazon’s network. In this way, you bypass any obstacles created by your internet provider and web traffic and move your data quicker and easier.

You can request a connection in the AWS Console. Choose the region you want to use, set the number of ports, and their speed - and you can apply the solution.