Cut OpenSearch Costs 15-20%: Right-Size Your Clusters (Free Assessment)

Part of our complete AWS Storage Cost Optimization guide.

Search me!

_– An idiom in English meaning “I don’t know” that first appeared in a Washington D.C. newspaper in 1898.
_

In English, we often say “Search me!” when we don’t know the answer to a question. Its origin seems fairly intuitive: if you don’t believe that I don’t know or have something, search me to find out! Thankfully, at least these days, most people will take you at your theoretical word and leave the searching to the software.

“Search me,” of course, is also the most succinct summary of why we use Amazon OpenSearch (or technically Amazon OpenSearch Service, but as we explained here, plain old “OpenSearch” usually suffices.) We’ve talked about OpenSearch before – how to right-size EBS volumes for OpenSearch, why you should run OpenSearch on Graviton – and for good reason: it’s one of the most powerful suites of search and analytics capabilities on the market. 

via GIPHY

What’s the best way to right-size OpenSearch clusters to maximize performance and efficiency? Glad you asked. Read on for our approach to right-sizing OpenSearch clusters and to learn how CloudFix automatically optimizes instance size in just a few clicks. 

Table of contents#

1. OpenSearch pricing: Why it’s worth your while to right-size OpenSearch clusters
2. How to right-size Amazon OpenSearch clusters
3. Automatically right-size OpenSearch instances with CloudFix

1. OpenSearch pricing: Why it’s worth your while to right-size OpenSearch clusters#

OpenSearch can power both internal and external facing search applications. It’s easy to get started and scale up (more powerful individual instances) and out (more instances). This multidimensional flexibility can be great for meeting your specific requirements, but also makes it very challenging to appropriately size OpenSearch clusters. 

The variables don’t stop there, either. OpenSearch clusters comprise both data nodes and master nodes, which can both be resized and have different usage profiles. In addition, there are a number of different instance types to choose from, and many sizes for each type. All of these dials can be adjusted to optimize OpenSearch costs, but today we’re going to take a “first pass” approach and focus solely on right-sizing cluster instances. This is a great way to quickly reduce OpenSearch spend, and a good place to start before taking more drastic steps like changing the number of nodes in the cluster, the architecture of nodes within the cluster, or even the instance family. 

Let’s kick off our right-sizing conversation by taking a look at OpenSearch pricing, which – spoiler alert – is not insignificant. First of all, every OpenSearch cluster (also referred to as a domain) requires:

1. Master nodes: control the cluster, manage the list of indexes, maintain routing information to the data nodes, etc.
2. Data nodes: store part of the indexes and execute the search functions
3. EBS volumes: attached to the data nodes to store the indexes

In production workflows, Amazon recommends a 2 + 1 dedicated master node setup. This means having two operational master nodes and a standby node, spread across at least two availability zones within the same region. (Check out the Creating and managing domains document for further discussion on provisioning clusters.) For testing workflows, a single-node configuration is supported.

In terms of pricing, the key drivers are the instances (or nodes, in OpenSearch parlance) and the EBS volumes attached to the data nodes. The choices for OpenSearch nodes are a particular subset of EC2 instances, and they are priced with a substantial markup over vanilla EC2. Using us-east-1 in July 2023 as a reference, comparing EC2 pricing and OpenSearch pricing for the m6g.xlarge Graviton2 instance type, we can see a 66% markup on the same hardware.

EBS storage pricing, when used for OpenSearch, also has a substantial markup over standard EBS.

What does this mean in practice? Although there is no “standard” OpenSearch cluster size, a production cluster provisioned to Amazon’s best practices would contain at least three master nodes and several data nodes with EBS volumes. Let’s look at a back-of-the-envelope calculation for a small OpenSearch Cluster based on m6g.4xlarge instances:

This is a simplified example that doesn’t account for network traffic, IOPS, or throughput. It should illustrate, however, that OpenSearch clusters can be expensive, especially when there are multiple nodes involved.

Just how expensive? Cluster sizes will vary depending on load, the amount of data to be indexed, and performance requirements. The size described above would be considered sufficient for a small production-level workload. Larger workloads can add up much more quickly. In this article from the AWS Big Data blog, Amazon describes a “large” cluster composed of 200 data nodes of type I3.16xlarge.search, with each node containing 15.2 TB of storage. These instances run $7.987 per hour. For a cluster of this size, that amounts to $1597 per hour just for the data nodes! 

The net-net: OpenSearch costs can add up fast, so it’s worth right-sizing them for your current traffic. Remember – the cloud is elastic. If and when your usage goes up, you can easily scale up. For now, it’s time to stop paying for power we don’t need.

2. How to right-size Amazon OpenSearch clusters#

At the highest level, the goal is to find the best balance of performance and efficiency for each OpenSearch cluster: enough CPU and memory to meet historical demand, adequate headroom for surges, and no paying for a size you don’t need.

Within an OpenSearch domain, all of the data nodes have the same instance type and EBS volume configuration. Therefore, the savings you realize for the data nodes are multiplied by the number of data nodes. The same is true for the master nodes, although we wouldn’t expect (and it’s not recommended) to have more than three.

The two dimensions that matter for instance sizing are CPU and memory. Memory is the tricky one. Since ElasticSearch / OpenSearch run on the Java Virtual Machine (JVM), you can’t simply check the amount of free memory reported by the OS, because the JVM pre-allocates memory for itself. What matters is JVM memory pressure, and AWS explains in its knowledge center how high memory pressure triggers garbage collection and, eventually, blocked writes. Right-sizing has to respect both constraints.

Here’s what that looks like in practice. Say we’re using the largest of the Graviton2 instances, the m6g.12xlarge.search, and the cluster’s CPU and memory use turn out to be a small fraction of what those nodes provide. Staying within the M6g family (the sizes are listed below), a much smaller size such as the m6g.4xlarge.search can comfortably handle the workload with room to spare.

**Prophetic aside: Peer inside Stephen’s crystal ball
**I predict that one day we will be able to request instance sizes with the precise amount of vCPU and memory that we need. Some future version of Nitro will be able to allocate exactly the instance sizes we would like, and a sophisticated “bin packing” algorithm will make it straightforward for AWS to make it happen.

In general, you want the smallest instance size that satisfies both the CPU and memory requirements. In this example, switching to m6g.4xlarge.search means paying 1/3rd the cost of the m6g.12xlarge.search instances we were running before. If we had a production cluster with seven data nodes, this would take our monthly spend from $15,462.72 down to $5,040!

Once again for the folks in the back: We just cut OpenSearch costs by two-thirds. Not too shabby.

A quick note on changing instance families

The most cautious approach to OpenSearch cost optimization would be to only change sizes, not instance families. For the ambitious, however, there is room for additional optimization by switching instance families within the same architecture. For example, if the particulars of this cluster are more biased towards compute than memory, then switching from an m6g to a c6g may deliver additional savings. It’s really up to you and your risk profile. 

Whatever you do, be mindful that not all versions of OpenSearch run on all architectures. Check the OpenSearch version guide for details. For example, you should be able to freely pick from the C6g, R6g, and M6g instance types as they all run the Graviton2 processor and all support the same versions of OpenSearch.

How CloudFix does it#

The finder works read-only: CloudFix uses your cost and usage data and the cluster’s own CPU and memory utilization metrics to find OpenSearch domains whose nodes are consistently underused, and recommends a smaller size in the same instance family. Each opportunity appears in your Recommendations with the projected savings, and nothing changes until you approve it. Once you do, the fixer runs as an AWS Systems Manager Automation runbook in your own account and resizes the nodes, and every execution is logged.

3. Automatically right-size OpenSearch instances with CloudFix#

The numbers have spoken: Right-sizing OpenSearch instances is an excellent way to pay less for AWS. There is enormous potential for serious savings, especially considering the markup you pay on OpenSearch compute instances. If we look back to the pricing table, dropping each instance size by just one level will reduce the spend by 50% on those node charges. That’s some real money back in your pocket.

Even if you run this fix manually, it’s definitely worth your time. But you also don’t have to. CloudFix has automated this process and made it as simple as approving a fix. Our OpenSearch right-sizing fixer, like all of our automated processes, reduces costs with a change you approve before it runs. We only propose instance resizes within an instance family, and always stick with AWS recommendations for target CPU and memory utilization. As a result, you can trust that your instances are up to the task without being overkill for the job at hand. We even take care of monitoring and automated rollback if the workload suddenly changes. All you need to do is figure out what to do with all the money that you’ll save.

So, the question is, why wouldn’t you use CloudFix to right-size OpenSearch instances? Search me.