AWS Open Source Blog

If you are hosting your own server productivity suite or file storage software, you cannot overlook Nextcloud, an open source platform to store, share and manage files over the web.

In our previous blog post, we demonstrated how to securely connect your Nextcloud server with Amazon Simple Storage Service (Amazon S3), allowing you to scale storage flexibly to the cloud. In this blog post, we build on this idea and demonstrate advanced functionalities and optimizations, such as starting workflows when you upload a file and optimizing your Amazon S3 usage for cost.

Integration with AWS services

If you are using the setup from Part 1 of this blog, then you have successfully extended your Nextcloud to Amazon S3 using the Nextcloud connector for S3. It will allow you to use your Amazon S3 storage as any other Nextcloud storage to share and receive files that are directly transferred to Amazon S3.

However, there are a lot of great features to make even more of this combination, save costs, and stay on top of your data: First, Amazon S3 offers advanced functionalities to optimize costs by helping to place data into different (hot and cold) storage classes that suit your use case. For instance, you can use Amazon S3 lifecycle rules to manage storage lifecycle explicitly or let Amazon S3 Intelligent-Tiering optimize storage classes based on your usage patterns.

Second, the existing integration of Nextcloud with Amazon S3 is an entry point to other services offered by AWS. For instance, file arrivals or changes also allow you to trigger functions or workflows, such as starting AWS Lambda functions to process files managed by Nextcloud or you can use Amazon CloudWatch to monitor files and access.

Finally, there are advanced considerations depending on your security needs. For instance, you can use different methods of encryption from Nextcloud, Amazon S3, or AWS Key Management Service (AWS KMS) to encrypt your data on the cloud. Here, we will present these advanced topics in more detail.

Cost & Cost-efficiency optimization

One reason to run your own server and use Amazon S3 as storage is that you only pay for what you use, for as long as you use it. This is often cheaper and less hassle than buying a long-term plan or provisioning the maximum capacity you might need beforehand. This is especially true for use cases where you need a temporary extension of your Nextcloud, not a permanent one. However, in addition to the competitive pay-as-you go prices, there are three features that will help you make the most of Amazon S3 for almost no effort: S3 has a Free Tier offering free base storage, the AWS cost calculator allows you to obtain a cost estimate before you add your data, and S3 Intelligent-Tiering is available to optimize storage classes.

Free tier

Amazon S3 offers a “free tier” to new users of 5GB or less of storage and including a corresponding amount of monthly access transactions. You can try the setup presented here under the free tier and even run smaller setups entirely for free.

AWS cost calculator

The AWS calculator allows you to quickly obtain a cost estimate. For instance, when selecting a computation for S3 on the website, we can enter the relevant inputs for S3 cost as shown in this image. For illustration, we estimate in the screenshot the cost for storing 10GB of pictures for a month with a reasonable access frequency of 10,000 read and 1,000 write access. This results in a cost estimate of about 24 cents.

Intelligent Tiering & optimizing storage classes

Optimizing the baseline cost of using Amazon S3 Standard for storage can be done by selecting a storage class tailored to your specific usage (https://aws.amazon.com/s3/pricing/) and creating lifecycle policies to move the files between them (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html). However, there is an easier way that is better suited to the use of Amazon S3 with Nextcloud, where access frequencies change over time and are often unpredictable: Amazon S3 Intelligent-Tiering (https://aws.amazon.com/about-aws/whats-new/2018/11/s3-intelligent-tiering/). Once activated on your bucket, S3 Intelligent-Tiering automatically moves objects between storage classes optimized for you in terms of access and storage costs for frequent access and infrequent access.

To activate Intelligent-Tiering on your bucket, navigate to Amazon S3 under the AWS Management Console, select your bucket, and under properties select “Intelligent-Tiering Archive configurations.” Enable it for all objects in the bucket. In addition, if you know that most of the data in the bucket will be rarely accessed after an initial phase, you can even activate the Archive Access Tiers. This option offers even higher savings on storage, yet comes at a slightly higher price for data access. After you select create, Amazon S3 will optimize storage classes and cost for you.

Data Lifecycle and automatic expiration of files

Apart from using the right storage class, you can also use Amazon S3 lifecycle policies to automatically delete files that are no longer needed. For instance, when sharing pictures with friends, pictures might only be relevant while everyone uploads and downloads them, but can be deleted after a few months. To enable lifecycle policies, navigate to Amazon S3 in your console, select your S3 bucket, go to the “Management” tab and select “Create lifecycle rule.” This will allow you to move files to a specific storage class or delete them entirely, once they reach a specific age.

Advanced considerations for managing and securing your data

In addition to extending and optimizing storage, the combination of Nextcloud and AWS opens a whole series of great functionalities. Here is a selection of typical options that show how to make the most of your Amazon S3 connection to Nextcloud.

Share an entire external Amazon S3 storage between users

When creating a connection to an external storage service using the Nextcloud frontend, you can decide who should have access to that storage. More precisely, you can select users or user groups that will share that storage. This is useful, for instance, when you have users belonging to the same team that should share storage, rather than storage that is accessible to a single user.

To use this feature it is a good practice to first create a separate user group in Nextcloud (let us call it “S3Users”). Next, using the Nextcloud external storage configuration we allow access to the external storage for that group instead of for individual users. Managing access to the shared storage then becomes a matter of adding or removing users from that group.

Sharing files and folders on external with individual users or via link

Sharing an entire external storage between users is an independent feature from the Nextcloud standard features for sharing files and folders with internal users (via shares) or external users (via links). However, for external storage, sharing files and folders is deactivated by default. To enable that, navigate to the setup page of your external storage and open the advanced menu (three dots) and select “Enable Sharing.” Now you are able to share the folders located on Amazon S3 as you would with any other Nextcloud folders.

Server- side Encryption using AWS KMS vs client-side encryption using Nextcloud

Both AWS and Nextcloud offer encryption for your files. To give you an idea when to use server-side encryption on AWS (using Amazon S3 or AWS KMS), client-side encryption in Nextcloud or even both at the same time, let us consider their purpose.

Server-side encryption on the cloud using Amazon S3 or AWS KMS allows you to encrypt files when they are stored or retrieved. These files are protected against attackers that do not have the correct encryption key or access to the key as an additional layer of security to the storage access rights. In addition, when using AWS KMS managed keys, you can control on a granular level which user or service should have access to your files. For the present case this is useful to grant access to the encryption key to those AWS services (such as AWS Lambda) that should read or manipulate the files, for instance to generate reports or convert file formats. Encryption on AWS using S3 managed encryption serves as a fundamental security feature that you should always activate and will not limit how you can read or process your files. Encryption using AWS KMS managed keys gives you an additional more granular way to manage the underlying encryption keys.

To activate encryption on AWS, go to Amazon S3 on the console, select your bucket and activate the encryption type (Amazon S3 or AWS KMS managed) on the bucket. Note that if you use AWS KMS managed keys, any user accessing and writing the files needs the AWS Identity and Access Management (IAM) permissions to use the key you select.

Client-side encryption provided by Nextcloud encrypts files with a key that is located on the Nextcloud server and that is therefore not available in AWS. In other words, using this method, files can only be decrypted when using Nextcloud. This can be an advantage when the AWS account is shared by many users because the encryption prevents users with access to your AWS account from decrypting the files. At the same time, it means that AWS services cannot help to recover your files when you lose access to your encryption key. It also means AWS services can only act on encrypted files (e.g. moving files), preventing processing of their content. Hence the decision to use this client-side encryption needs to consider the use of such integrations.

Integrate with AWS Tools & Workflows

We have already mentioned that the arrival (or deletion) of a file may be used to trigger follow-up actions. Since AWS offers hundreds of services that could be relevant depending on the use case, we will only showcase one example here that demonstrates the use of this powerful feature: picture conversion and distribution.

Imagine you want to receive uploaded pictures from multiple friends and allow all of them to download the pictures that others uploaded. Also, you want to standardize naming and convert their format.

To solve this problem, you can create one folder on Nextcloud that is located inside of the external source that connects to Amazon S3, similar to what we would do with Nextcloud’s internal storage. Then share a (write-only) link to that folder with your friends to collect their uploads. Also, you create another folder and share a (read-only) link to that folder such that files placed inside it can be retrieved by your friends. With the two folders set up via Nextcloud, you only need to move and convert the files using AWS.

Working with the uploaded files is simple on Amazon S3. Whenever a file arrives in the upload folder (i.e. is uploaded though Nextcloud by one of our friends), it starts an AWS Lambda function using Amazon S3 Bucket Notifications or Amazon EventBridge Rules. The AWS Lambda function then processes the file (e.g. standardizes and converts the picture), and moves it to the folder with the read-only access link. For more information on the conversion itself, consider this tutorial, where uploaded pictures are converted into thumbnails. Hence, whenever a file is now uploaded to the write-only folder, it is automatically converted and made available for download in the read-only folder in Nextcloud within seconds.

Monitoring

To keep you on top of your data, AWS allows you to set up monitoring for your storage in seconds. More precisely, Amazon S3 will automatically deliver to Amazon CloudWatch crucial metrics such as total daily storage or number of objects. Other metrics, such as access frequency, can be opted into at a low cost. If you are using Amazon S3 as external storage and require transparency on the different operations happening in your bucket, these metrics can be used to create monitoring dashboards, trigger alarms, or automate follow-up actions, such as removing access entirely. Hence, CloudWatch allows you to keep on top of all the activities that happen with your data on Amazon S3.

Summary

In this blog, we described how Amazon S3 creates the link between your Nextcloud and all of the features and services that AWS provides. We shared how to use cost optimization features, data lifecycle management tools, and encryption to make the most of Nextcloud. If you are not using AWS yet, you now have a first impression of all the possibilities this will open up. With the AWS Free Tier allowance, you can even start for free.

Share this post