Amazon S3 · Finder + guided fix
S3 Consolidate Files
ConsolidateFiles
Consolidate small S3 objects into larger files to reduce request costs for Athena queries. CloudFix identifies buckets with many small files.
What happens, in order.
Finder · read-only
Finds the opportunity
Scans your connected accounts with a read-only role. It can see resource metadata and usage; it can’t change anything.
You
Review in Recommendations
It appears in your Recommendations view with what it would save.
Guided fix
You make the change
You apply it yourself, following the fix’s step-by-step instructions; the write-up below explains it.
Finder role
Read-only. It can see resource metadata and usage. It can’t change anything.
Fixer role
Not used here: this one is a guided fix, so the change is yours to make.
Saves by
Amazon S3 buckets often store numerous small files, leading to significant request costs, especially for use cases such as log storage for Athena queries. This Finder identifies buckets where consolidating small files into larger, logically grouped files would reduce billable requests while maintaining data accessibility and integrity.
Amazon S3 buckets often store numerous small files, leading to significant costs associated with frequent requests, especially for use cases such as log storage for Athena queries. By consolidating these small files into larger, logically grouped files, this Finder identifies opportunities to reduce the number of billable requests and optimize S3 costs while maintaining data accessibility and integrity.
Overview
Problem Statement
Amazon S3 buckets can accumulate large numbers of small files over time, especially in data lake or log storage scenarios. Each of these small files generates individual API requests when accessed, resulting in significant request costs that can exceed the actual storage costs by an order of magnitude. This inefficiency is particularly noticeable in analytics workloads such as Athena queries that need to scan many small files.
Solution Impact
By identifying S3 buckets with high request costs relative to storage costs and recommending file consolidation strategies, this CloudFix feature can help organizations reduce S3 request costs by up to 90%. File consolidation not only lowers operational expenses but can also improve query performance in analytics workloads.
AWS Services Affected
How It Works
CloudFix uses your cost and usage data to find, read-only, the S3 buckets where request costs are far higher than storage costs because of large numbers of small files. Each opportunity appears in your Recommendations with the estimated savings and a link to the bucket in the S3 console.
This feature is recommendation-only. You consolidate the files yourself, manually or with tools such as AWS Glue, grouping small files into larger objects in a way that suits your workload.
FAQ
Is it possible to roll back once CloudFix implements the fixer?
Rollback is not applicable as the feature only identifies opportunities and provides recommendations. Users control the implementation of file consolidation.
Can CloudFix implement the fix automatically once I accept the recommendation?
No, this feature is recommendation-only. Implementation requires manual execution by the user or an external automated process.
Does the fix require downtime?
No downtime is required for identifying and recommending file consolidation. The actual implementation may require careful planning to avoid disrupting dependent workloads.
What typical savings can I expect?
Up to 90% of GET request costs can be saved for eligible buckets by consolidating small files into larger, more efficient objects.
How do I implement the file consolidation recommendations?
You can implement file consolidation using AWS Glue jobs, Lambda functions, or other ETL processes to combine small files into larger objects according to logical groupings (e.g., by date, customer, or data type).
Related Resources
- Optimizing Performance Guidelines for Amazon S3
- Amazon S3 Cost Optimization Best Practices
- S3 File Consolidation Support Article
Ready to start saving on AWS? See how much you could cut from your cloud bill with a free cost optimization assessment, or explore CloudFix automated Finder/Fixers that eliminate waste across 30+ AWS services.
Related Articles
See whether this one applies to your account.
A free savings assessment reports which fixers can be applied to your environment. Results typically within 24 hours.