Composite Aggregations in Elasticsearch

Introduction

Composite aggregations are a powerful feature in Elasticsearch that allow you to paginate over aggregated data. Unlike traditional bucket aggregations, composite aggregations are specifically designed to handle large sets of data efficiently by paginating through them. This is particularly useful when you need to retrieve all buckets of a specific aggregation and not just the top N results.

Basic Concepts

Before diving into composite aggregations, it's important to understand some basic concepts:

• Bucket: A collection of documents that meet a certain criterion.
• Aggregation: A collection of documents or buckets based on a specific query.
• Pagination: The process of dividing a large set of results into smaller, manageable chunks.

Composite Aggregation Structure

A composite aggregation is composed of multiple sources, each defining a different criterion for creating buckets. The general structure of a composite aggregation query looks like this:

POST /index_name/_search?size=0
{
  "aggs": {
    "composite_agg": {
      "composite": {
        "sources": [
          { "field1": { "terms": { "field": "field1" } } },
          { "field2": { "terms": { "field": "field2" } } }
        ]
      }
    }
  }
}

Example: Composite Aggregation on Multiple Fields

Let's consider an example where we have an index of e-commerce data, and we want to group the data by two fields: category and brand. Here's how you would construct a composite aggregation query for this scenario:

POST /ecommerce/_search?size=0
{
  "aggs": {
    "by_category_and_brand": {
      "composite": {
        "sources": [
          { "category": { "terms": { "field": "category.keyword" } } },
          { "brand": { "terms": { "field": "brand.keyword" } } }
        ]
      }
    }
  }
}

The response will include buckets for each unique combination of category and brand found in the data:

{
  "aggregations": {
    "by_category_and_brand": {
      "buckets": [
        {
          "key": { "category": "Electronics", "brand": "Sony" },
          "doc_count": 10
        },
        {
          "key": { "category": "Electronics", "brand": "Samsung" },
          "doc_count": 15
        },
        // More buckets...
      ]
    }
  }
}

Pagination with Composite Aggregations

Composite aggregations support pagination, allowing you to retrieve all buckets in chunks. To paginate through the results, use the after parameter, which takes the last key from the previous response:

POST /ecommerce/_search?size=0
{
  "aggs": {
    "by_category_and_brand": {
      "composite": {
        "sources": [
          { "category": { "terms": { "field": "category.keyword" } } },
          { "brand": { "terms": { "field": "brand.keyword" } } }
        ],
        "after": { "category": "Electronics", "brand": "Sony" }
      }
    }
  }
}

This will continue retrieving the next set of buckets after the specified key.

Use Case: Monthly Sales Data

Consider a use case where we have sales data, and we want to aggregate this data by month and by product category. Here's how you can construct the composite aggregation query:

POST /sales_data/_search?size=0
{
  "aggs": {
    "monthly_sales": {
      "composite": {
        "sources": [
          { "month": { "date_histogram": { "field": "sale_date", "calendar_interval": "month" } } },
          { "category": { "terms": { "field": "category.keyword" } } }
        ]
      }
    }
  }
}

The response will give you buckets for each month and each product category:

{
  "aggregations": {
    "monthly_sales": {
      "buckets": [
        {
          "key": { "month": "2023-01-01T00:00:00.000Z", "category": "Electronics" },
          "doc_count": 100
        },
        {
          "key": { "month": "2023-01-01T00:00:00.000Z", "category": "Apparel" },
          "doc_count": 50
        },
        // More buckets...
      ]
    }
  }
}

Optimizing Composite Aggregations

Composite aggregations are designed to be efficient, but there are a few tips to optimize their performance:

• Use size parameter to control the number of buckets returned in each response. A reasonable value can help in managing memory and response size.
• Always use the after key for pagination to avoid missing any buckets.
• Use date_histogram for date fields to efficiently group data by time intervals.

Conclusion

Composite aggregations are a powerful tool in Elasticsearch for handling large sets of aggregated data. They provide an efficient way to paginate through buckets and can be used in various use cases, such as aggregating sales data by time and category. By understanding and utilizing composite aggregations, you can unlock more advanced and efficient querying capabilities in Elasticsearch.