Planning indexes in OpenSearch and ElasticSearch is a critical factor for performance and scalability. Poorly configured indexes can create bottlenecks and reduce system efficiency. In this guide, you’ll learn the best practices for index planning to achieve an optimal balance between write and read performance.

Understanding OpenSearch/ElasticSearch Basics

OpenSearch and ElasticSearch are powered by Apache Lucene, a high-performance search engine. Lucene stores documents as key-value pairs and leverages inverted indexing to process search queries efficiently. This means that terms from a document are stored in a structured index, allowing fast retrieval instead of scanning documents sequentially.

A single Lucene index can easily grow several gigabytes in size. The optimal size is 5 to 10 GB, while indexes larger than 50 GB can become expensive to update. OpenSearch and ElasticSearch use shards as the smallest units of storage, each corresponding to a Lucene index. A single OpenSearch/ElasticSearch index consists of one or more of these shards.

Structuring an Index Cluster Properly

Each OpenSearch/ElasticSearch index can have multiple shards distributed across different servers. Grouping multiple indexes with a common prefix enables elastic scalability. For time-series data, it is common to create daily indexes. When the system load increases, the mapping of future indexes can be adjusted accordingly.

Example Adjustments:

• Adjust the number of shards to improve load distribution.
• Convert fields from text to keyword to optimize search queries.

Key Considerations for Shard Sizing

1. Efficient Data Deletion

Deleting individual documents in Lucene is costly since the ID is only marked as deleted. The actual removal happens later through file merging. For this reason, deleting entire shards or indexes is more efficient.

Best Practices:

• Use daily indexes to simplify historical data deletion.
• If a daily index is too small, use a weekly index to reduce shard count.
• If long-term searches become more important, merging indexes may be beneficial — however, this means only entire weeks can be deleted instead of single days.

2. Optimizing Data Writing

A single shard can handle multiple gigabytes of data. Write performance is limited by the I/O capacity of the disk or SSD. Increasing the number of shards on a single machine does not help; instead, load distribution through horizontal scaling is essential:

• Use multiple servers or separate disks.
• Distribute shards across different nodes.

3. Efficient Searching with Fewer Shards

The more shards an index has, the more parallel queries must be executed and aggregated, increasing query overhead.

Example:

• An index with one shard per day results in 30 parallel queries for a 30-day search.
• If each daily index had two shards, that would mean 60 parallel queries.

Thus, fewer shards improve search performance. Fewer shards reduce the number of parallel queries, decreasing query overhead. This also minimizes CPU and memory consumption needed for result aggregation, improving response times.

Best Practices for Index Planning

A well-structured index strategy should align with typical use cases:

• Writing occurs continuously and is independent of queries.
• Search queries focus on the last few days.
• Long-term analytics (30+ days) are less frequent but should be accounted for.

Conclusion: Monitoring and Continuous Optimization

Index planning is always a trade-off between write and read performance. A well-thought-out strategy allows both high write loads and efficient searches. The key factors to consider are:

• Avoid making shards too small — a shard can handle multiple GB of data.
• Use horizontal scaling when write loads exceed hardware limits.
• Delete entire shards or indexes instead of individual documents.
• Optimize search performance by keeping the number of shards per index low.

Monitoring and Long-Term Optimization

Regular index monitoring is essential. Tools like Kibana or OpenSearch Dashboards help identify bottlenecks early. Continuously adjusting the number of shards, index structure, and query optimizations ensures sustained performance improvements.

By following these best practices, you can keep your OpenSearch/ElasticSearch cluster efficient and scalable.