Fixer
Save big on Amazon Kendra by deleting idle indices
Amazon Kendra indices incur costs even when idle. Find and delete unused Kendra indices to save on intelligent search costs. Identify idle indices automatically.
Information is a source of learning. But unless it is organized, processed, and available to the right people in a format for decision making, it is a burden, not a benefit.
_– William Pollard, English writer
_
Humans have always sought out knowledge, but in the last 30-some years, the search for information has become exponentially easier. Lycos, Alta Vista, Ask Jeeves, Yahoo, Google, Bing, Duck Duck Go!… search engines have transformed our lives and put virtually endless learnings at our fingertips.
Just because search engines are easy to use, however, doesn’t mean they’re easy to run. As the internet, data, and networks have grown, so has the amount of data that’s not public. This includes data held by corporations in the form of knowledge bases, internal wikis and documentation, etc. Being able to quickly and efficiently search this data is important to businesses, which makes it fertile ground for innovation.
Which brings us to Amazon Kendra. Kendra is an extremely powerful enterprise search tool that solves the challenge of internal search beautifully. The only issue: its costs can add up fast. In this fixer blog, we’ll take a look at how Kendra costs impact your AWS bill, what to do about it, and of course, how to reduce Kendra costs easily and automatically with CloudFix.
Table of Contents
- How idle Kendra indices add up – and how much they’re costing you
- How CloudFix finds and removes idle Kendra indices
- Stop paying for idle Kendra indices automatically with CloudFix
1. How idle Kendra indices add up – and how much they’re costing you
Amazon has tackled the challenge of enterprise search in a number of ways over the years. The first was with AWS CloudSearch, a managed service that can make it easy to search a website. CloudSearch is powered by Apache Lucene and is a very mature service that’s been on the market for over ten years. Next came Amazon OpenSearch. Launched as the ElasticSearch Service, Amazon OpenSearch forked from the open source ElasticSearch project after a licensing change. OpenSearch is the first layer in the famous ELK Stack, which combines ElasticSearch, Logstash, and Kibana into a log ingestion and visualization solution. (If you’re an OpenSearch user, check out this fixer blog on how to right-size Amazon OpenSearch instances to cut costs by 50% or more.)
Finally, there is Amazon Kendra. Launched in December 2019 and powered by machine learning, Kendra is the relatively new kid on the block that’s built for enterprise search. The key feature of Kendra is its flexibility. It can easily search SharePoint, S3, DropBox, structured and unstructured databases, and more. Even a collection of PDFs can be effectively indexed and searched with Kendra. We’re not gonna lie: it’s an impressive piece of technology.
Kendra’s hefty capabilities, of course, also come with a hefty price tag. Kendra is charged per index (a logical collection of searchable documents), with surcharges for additional documents indexed or queries per day. The developer edition of Kendra starts at $810 per index per month and is not meant for production use. The Enterprise Edition, which is suitable for production use, costs $1,008 per index per month, or $1.40 per hour. The pricing examples on the Kendra pricing page show various scenarios for pricing.
Clearly, these numbers aren’t small, and even a few rogue indices can significantly impact your budget. This is particularly true if you have multiple teams of developers who are encouraged to explore new AWS services. This is a good thing, and has led to many innovations across countless organizations, but can also lead to the age-old problem of playing around with a shiny new object and then forgetting about it. And that, my friends, is how we end up paying for resources that we’re not actually using.
We’ve been down this road before in fixer blog land: Let’s find idle Kendra indices and (thoughtfully, securely) get rid of them so we can stop overspending on AWS.
_Quick aside:
_Removing idle Kendra indices does not remove the underlying data, called a “datasource” in Kendra parlance. If that is stored in AWS, there will be costs associated with that too. For example, many users of Kendra use it to search data located in S3 or in RDS. This article is only focused on the Kendra indices themselves. Whether or not the underlying data is worth storing is a different question altogether.
2. How CloudFix finds and removes idle Kendra indices
Spotting an idle index from the bill alone is harder than it sounds. Kendra’s pricing includes a daily query allowance in each edition, so many indices show up at exactly the same daily cost whether they served thousands of queries or none at all. To tell an idle index apart from an infrequently used one, you need more than billing data, and you need a cautious definition of idle.
That’s the work CloudFix does for you. Finding is read-only: CloudFix uses your cost and usage data and Kendra’s own utilization metrics to find indices that have gone unused for an extended period, and it skips indices that are not in a safe state to change. Each idle index appears in your Recommendations, and nothing happens until you approve it.
Once approved, the fix runs as an AWS Systems Manager Automation runbook in your own account. Because Kendra indexes data rather than holding it, the fixer first saves the index’s data source configuration so the index can be reconstructed down the road if you ever need it, then deletes the idle index. Every execution is logged.
3. Stop paying for idle Kendra indices automatically with CloudFix
When it comes to Amazon Kendra, idle resources aren’t costing you pennies – they’re adding up to a real drain on your budget. Yet while finding and removing them by hand isn’t the most cumbersome of cleanups, it’s also not top priority. With so many opportunities to innovate and add business value through AWS, your engineers probably aren’t going to spend their time hunting down some unused Kendra indices.
That’s where CloudFix comes in. Finding and deleting idle resources is one of our specialties. We automate the process of identifying idle indices (using appropriately cautious criteria) and, with just a couple clicks, make it easy to get rid of them. It’s one less thing on your team’s to-do list and one more way that CloudFix helps you spend less on AWS.
Humans will never stop searching for knowledge, but with CloudFix, you can stop searching for idle Kendra indices – and stop paying for resources you don’t need.
Curious how much you can save with CloudFix? Take our fast, free savings assessment to discover exactly which fixers apply to your organization and precisely how much your company can save.
