Download CloudTier S3 Intelligent Tiering Demo Setup File
Download CloudTier S3 Intelligent Tiering Demo Zip File

CloudTier S3 Intelligent-Tiering Demo demonstrates how to connect the Amazon S3 Intelligent-Tiering as a second tier for on-premises storage automatically. CloudTier SDK makes it possible to automatically tier infrequently used on-premises storage data to more cost effective S3 Intelligent-Tiering and move it back automatically when required.

What is CloudTier Storage Tiering SDK?

The CloudTier Storage Tiering SDK is a Hierarchical Storage Management (HSM) file system filter driver development kit. It is a data storage technique that automatically moves data between high-cost and low-cost storage media. It allows you to free up on-premises storage capacity transparently, by moving out cooler data to the cloud storage, thereby reducing capital and operational expenditures.

What Is Intelligent Tiering in S3?

The Amazon S3 Intelligent-Tiering storage class is designed to optimize storage costs by automatically moving data to the most cost-effective access tier when access patterns change. For a small monthly object monitoring and automation charge, S3 Intelligent-Tiering monitors access patterns and automatically moves objects that have not been accessed to lower-cost access tiers. S3 Intelligent-Tiering delivers automatic storage cost savings in three low-latency and high-throughput access tiers.

Automatically File Archive in S3

S3 Intelligent-Tiering automatically stores objects in three access tiers: one tier (frequent access tier) optimized for frequent access, a lower-cost tier(infrequent access tier) optimized for objects not accessed for 30 consecutive days, and a very-low-cost tier(archive instant access tier) optimized for objects not accessed for 90 consecutive days. For infrequent access files, you can easily to implement soultion for file archive in S3.

To save even more on data that doesn’t require immediate retrieval, you can activate the optional asynchronous Archive Access and Deep Archive Access tiers within S3 Intelligent-Tiering. When turned on, objects not accessed for 90 days are moved directly to the Archive Access tier for savings of 71%, and to the Deep Archive Access tier after 180 days with up to 95% in storage cost savings. If the objects are accessed later, S3 Intelligent-Tiering automatically moves the objects back to the Frequent Access tier. If the objects you are retrieving are stored in the optional Archive Access or Deep Archive tiers, before you can retrieve them you must first restore a copy using RestoreObject.

Manage The S3 Objects Lifecycle

To manage the lifecycle of S3 objects,  you can create an S3 Lifecycle configuration for your bucket. An S3 Lifecycle configuration is a set of rules that define actions that Amazon S3 applies to a group of objects. With S3 Lifecycle configuration rules you can tell Amazon S3 to transition objects to less-expensive storage classes, archive or delete them. You can create an S3 Lifecycle rule in which you specify the initial transition action to S3 Intelligent-Tiering, S3 Standard-IA, or S3 One Zone-IA storage, another transition action to S3 Glacier Flexible Retrieval storage for archiving, and an expiration action. As you move the objects from one storage class to another, you save on storage costs.

The CloudTier S3 Intelligent-Tiering Storage Example

The CloudTier SDK integrates S3 Intelligent-Tiering cloud storage with on-premise storage systems seamlessly, creating a hybrid storage environment.  So it allows on-premise applications to access S3 file objects transparently, just as they would access on-premise regular files. There is no interruption to move or restore your on-premise files to/from the cloud storage, so you don’t need to change your existing applications and infrastructure to be compatible with the cloud environment.

The CloudTier S3 demo is a C# demo project, it demonstrates how to connect the S3 storage from your on-premise storage, and how to generate the test files in on-premise storage and S3 storage. It demonstrates how to browse and read the S3 files as a local regular files.

When you launch the CloudTier S3 demo, it will generate a few test stub files in your application folder’s sub folder “TestStubFolder”, the file’s content was stored in the sub folder “TestSourceFolder”, when you read the stub file, the file content will be retrieved from the source file, if your file content was stored in the S3 storage, then it will be retrieved from the S3, it can demonstrate how the storing tiering is working with this simple project.

A stub file looks and acts like a regular file. It has the same file attributes with the original physical file (file size, creation time, last write time, last access time). It also keeps the original file’s security. The difference between the stub file and the normal physical file is the stub file doesn’t take any physical space, looks like a 0 kb file.

To run the demo, you need to configure the CloudTier Filter Driver settings, as below:

1. Excluded the process Id from the filter driver: the excluded processes can’t access the stub file. It can prevent the unwanted processes from downloading the data from the cloud storage.
2. Filter connection threads: the number of the threads to handle the read requests of the stub files.
3. Connection timeout: the maximum time of the filter driver waiting for the data return from your application.
4. Cache folder path: the folder to store the cache files which download from the cloud storage.
5. The return data type: a. return block data on read: return the block of the requested read data if you have the block of the data. b. return cache file on read: return the cache file name if the whole file was downloaded. c. rehydrate file on first read: the file was downloaded to the cache folder, and the stub file will be rehydrated to the regular file.
6. Amazon S3 site settings: to test the S3 connection, you need to have an S3 account with the access key, then you can create a test bucket with the intelligent tiering enabled in Amazon S3 console.

7. To browse the files in s3 or upload/download the files to/from S3, you can go to the S3 explorer. After the files were uploaded to the S3, then you can create the stub file from the S3 explorer:

8.Read the S3 files: after the files were uploaded to the S3 cloud storage, you can replace your original file with the stub file based on your retention policies. You can create S3 stub file in S3 explorer. For example, to read the stub file by copying the stub file to another folder, you will see the S3 file will be downloaded to the cache folder from the S3 storage, after that the cache file will be returned to your local file system through the CloudTier filter driver, then the read request to the S3 file will be completed successfully.