> ## Documentation Index
> Fetch the complete documentation index at: https://test-8ad8522e-feat-ai-sre.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# ElasticSearch

> Configure alert rules for ElasticSearch data sources with SQL aggregate query support

Monitors implements log and metric data monitoring through ElasticSearch SQL feature, supporting flexible aggregate queries and alert evaluation.

## Core Concepts

<Warning>
  Due to SQL feature dependency, only **ElasticSearch 6.3** and above versions are supported.
</Warning>

| Config Item          | Description                                                                                                                                  |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| **Query Language**   | Currently only supports SQL syntax                                                                                                           |
| **Field Processing** | Preserves the column-name casing returned by Elasticsearch SQL. Use the actual returned column names when configuring label and value fields |

<Tip>
  Use explicit, stable column aliases in SQL and match their casing exactly in field mappings. This prevents mappings from breaking when query fields change.
</Tip>

## 1. Threshold Evaluation Mode

This mode is suitable for scenarios requiring threshold comparison on aggregated values, such as monitoring "error log count in the last 5 minutes".

### Configuration

1. **Query Statement**: Write SQL aggregate query, returning value columns and (optional) grouping columns.

* Example: Count error log quantity by service in the last 5 minutes.
  ```sql theme={null}
  SELECT service_name, count(*) AS error_cnt 
  FROM "app-logs-*" 
  WHERE "@timestamp" > now() - INTERVAL 5 MINUTES AND log_level = 'ERROR'
  GROUP BY service_name
  ```

2. **Field Mapping**:

* **Label fields**: Select `service_name` to identify the alert object. After you select label fields, other non-value fields are carried with the alert as additional information.
* **Value fields**: Select `error_cnt` for threshold evaluation.
* If Label fields is empty, every returned field except the value fields becomes a label. See [Query Result Field Mapping](/en/monitors/alert-rules/query-result-fields) for the complete behavior.

3. **Threshold Conditions**:

* Use `$A.field_name` to reference values.
* Example: `Critical: $A.error_cnt > 50`, `Warning: $A.error_cnt > 10`.
* Shorthand: If only one value field is configured, you can directly use `$A`, like `$A > 50`.

### How It Works

The engine executes SQL query and gets two-dimensional table data. It groups data by "label fields", then extracts "value fields" values to compare against threshold expressions.

<Warning>
  Label field combination uniquely identifies an alert object. Query results cannot have multiple rows with the same label field value combination.
</Warning>

### Recovery Logic

| Strategy                        | Description                                                                               |
| ------------------------------- | ----------------------------------------------------------------------------------------- |
| **Auto Recovery**               | When values no longer satisfy any alert threshold, automatically generates recovery event |
| **Specific Recovery Condition** | Configure recovery expression (e.g., `$A.error_cnt < 5`) to prevent alert flapping        |
| **Recovery Query**              | Write independent SQL for recovery evaluation, supports `${label_name}` variables         |
| **Manual Close**                | Keep the alert active until it is closed manually                                         |

<Accordion title="Recovery Query Example">
  If the alert SQL found that network card with `network_host="a", interface="b"` is down, the recovery SQL can be written as:

  ```sql theme={null}
  SELECT network_host, interface, status FROM "network-status-*" 
  WHERE "@timestamp" > now() - INTERVAL 5 MINUTES 
    AND network_host = '${network_host}' 
    AND interface = '${interface}' 
    AND status = 'UP'
  ```

  The engine will replace variables with actual values before executing the query. If data is found, recovery is determined.
</Accordion>

## 2. Data Exists Mode

This mode is suitable for scenarios where filter logic is written directly in SQL, or when you only care about "whether data is returned".

### Configuration

1. **Query Statement**: Use `HAVING` clause in SQL to directly filter out anomalous data.

* Example: Directly query services with error count exceeding 50.
  ```sql theme={null}
  SELECT service_name, count(*) AS error_cnt 
  FROM "app-logs-*" 
  WHERE "@timestamp" > now() - INTERVAL 5 MINUTES AND log_level = 'ERROR'
  GROUP BY service_name
  HAVING count(*) > 50
  ```

2. **Field Mapping**:

* Label fields and value fields are **optional** in this mode. If both are empty, Monitors treats every returned field as a label.
* To identify alerts with only stable fields and keep other columns as troubleshooting context, explicitly select Label fields. See [Query Result Field Mapping](/en/monitors/alert-rules/query-result-fields).

### Recovery Logic

* **Recovery When Data Disappears**: When SQL query result is empty (i.e., no longer satisfies HAVING condition), the engine determines incident recovery. This is the most commonly used recovery method.
* **Recovery Query**:
  * **Scenario**: Sometimes "no data found" doesn't mean recovery (might be log collection down), or need stricter recovery conditions (like no errors for N consecutive minutes).
  * **Configuration**: Write an independent SQL statement for recovery evaluation. As long as that query can find data, the incident is considered recovered.
  * **Variable Support**: Supports using `${label_name}` in recovery SQL to reference alert event label values for precise recovery detection.
* **Manual Close**: Keep the alert active until it is closed manually.

### Pros and Cons Analysis

| Type     | Description                                                                                                          |
| -------- | -------------------------------------------------------------------------------------------------------------------- |
| **Pros** | Leverages ES cluster's computing power for filtering, reducing network transmission with better performance          |
| **Cons** | Cannot differentiate multi-level alerts (like Info/Warning); SQL can only return data satisfying specific conditions |

## 3. No Data Mode

This mode is used to monitor scenarios where "data is expected but actually missing", commonly used to monitor log collection pipeline interruption or periodic task non-execution.

### Configuration

1. **Query Statement**: Write a SQL query that is expected to continuously return data.

* Example: Query log reporting heartbeat from all hosts.
  ```sql theme={null}
  SELECT host_name 
  FROM "heartbeat-logs-*" 
  WHERE "@timestamp" > now() - INTERVAL 5 MINUTES 
  GROUP BY host_name
  ```

2. **Evaluation Rules**:

* The engine periodically executes this SQL.
* If a `host_name` appeared in previous cycles but no longer appears in current cycle (and N consecutive cycles) query results, triggers "No Data" alert.
* Note: This is opposite to Data Exists mode. Data Exists means "alert when data found", No Data means "alert when data not found".

### Recovery Logic

No-data alerts support configuring the **alert ending mode**, which decides how the alert ends:

| Ending Mode                                        | Description                                                                                                                                                                                                              |
| -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **End automatically when data reappears**          | Default. Once that `host_name` reappears in query results, the alert ends automatically                                                                                                                                  |
| **End when data reappears or the timeout expires** | The alert ends when the data reappears or the automatic close timeout is reached. The timeout is in seconds with a minimum of 1. Available only when the "Alert if previously found data is now missing" mode is enabled |
| **Manual close only**                              | The alert stays active until you close it manually                                                                                                                                                                       |

## 4. Use Case

Log alerting often encounters this requirement: count ERROR logs in the last 5 minutes, alert if exceeding threshold, and display the most recent ERROR log as a sample in the alert message. Configuration approach:

* **Main Alert Condition**: Use Threshold mode, SQL statement counts ERROR logs in the last 5 minutes, configure threshold conditions.
* **Related Query**: Configure a related query, SQL statement queries the most recent ERROR log, using `${service_name}` and other variables to limit to specific service.
* **Rule Notes Description**: Reference related query results in alert rule's notes description, using `$relates` variable to render the original log.