> ## Documentation Index
> Fetch the complete documentation index at: https://test-8ad8522e-feat-ai-sre.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Loki

> Configure alert rules for Loki data sources with LogQL query syntax support

Monitors supports Loki's LogQL query syntax, enabling aggregate analysis on log data and triggering alerts.

## Core Concepts

Loki's query language LogQL is divided into two types:

| Type               | Description                                                               |
| ------------------ | ------------------------------------------------------------------------- |
| **Log Queries**    | Returns log line content (Stream)                                         |
| **Metric Queries** | Counts or aggregates logs, like `count_over_time` returns values (Vector) |

<Tip>
  When writing LogQL, the query input autocompletes label names and label values from the data source, helping you locate log streams quickly.
</Tip>

## 1. Threshold Evaluation Mode

This mode is suitable for scenarios requiring multi-level threshold evaluation on log aggregate values (e.g., Info/Warning/Critical).

### Configuration

* **Query Statement (LogQL)**: Write LogQL that returns numeric vectors (select "Do Stats" query mode)

**Example**: Count log lines containing `error` keyword in `mysql` job in the last 5 minutes:

```text theme={null}
count_over_time({job="mysql"} |= "error" [5m])
```

* **Threshold Conditions**:
  * **Critical**: `$A > 50` (Error logs exceed 50 in 5 minutes)
  * **Warning**: `$A > 10` (Error logs exceed 10 in 5 minutes)

### How It Works

The engine executes LogQL query and gets time series data with labels (Vector). The engine iterates through each series, extracting values to compare against configured threshold expressions.

### Recovery Logic

| Strategy                        | Description                                                           |
| ------------------------------- | --------------------------------------------------------------------- |
| **Auto Recovery**               | When query result value falls below threshold, automatically recovers |
| **Specific Recovery Condition** | Configurable like `$A < 5` to avoid oscillation near threshold        |
| **Recovery Query**              | Supports independent LogQL for recovery evaluation                    |
| **Manual Close**                | Keep the alert active until it is closed manually                     |

## 2. Data Exists Mode

This mode is suitable for users who prefer writing filter conditions directly in LogQL, or scenarios that only care about "whether anomalous data exists". Recommended for log anomaly detection alerts.

### Configuration

* **Query Statement (LogQL)**: Write LogQL containing comparison operators, returning only data satisfying conditions

**Example**: Directly filter services with error rate exceeding 5%:

```text theme={null}
count_over_time({job="ingress"} |= "error-code-500" [5m]) / count_over_time({job="ingress"} [5m]) * 100 > 5
```

* **Evaluation Rules**: As long as LogQL query returns data, triggers alert

### Pros and Cons Analysis

| Type     | Description                                                              |
| -------- | ------------------------------------------------------------------------ |
| **Pros** | Computation logic pushed down to Loki server, reducing data transmission |
| **Cons** | Cannot differentiate alert levels; can only trigger single-level alerts  |

### Recovery Logic

* **Recovery When Data Disappears**: When LogQL query result is empty, determines recovery
* **Recovery Query**: Supports configuring additional query statements to assist in determining recovery status
* **Manual Close**: Keep the alert active until it is closed manually

## 3. No Data Mode

This mode is used to monitor whether log reporting pipeline is interrupted, or whether logs that should be continuously generated have stopped.

### Configuration

* **Query Statement (LogQL)**: Write a query that is expected to always have data

**Example**: Count log reporting rate from all hosts:

```text theme={null}
rate({job="node-logs"} [1m])
```

* **Evaluation Rules**: If a Series (uniquely identified by labels, like `instance="host-1"`) existed in previous cycles but cannot be found in current and N consecutive cycles, triggers "No Data" alert

### Recovery Logic

No-data alerts support configuring the **alert ending mode**, which decides how the alert ends:

| Ending Mode                                        | Description                                                                                                                                                                                                              |
| -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **End automatically when data reappears**          | Default. The alert ends automatically once the data reappears                                                                                                                                                            |
| **End when data reappears or the timeout expires** | The alert ends when the data reappears or the automatic close timeout is reached. The timeout is in seconds with a minimum of 1. Available only when the "Alert if previously found data is now missing" mode is enabled |
| **Manual close only**                              | The alert stays active until you close it manually                                                                                                                                                                       |

### Typical Applications

* Monitor whether Promtail/Fluentd and other collection Agents have stopped working
* Monitor whether critical business logs (like order creation logs) are abnormally interrupted

## 4. Using a raw log query as the primary query

To alert whenever matching log rows exist, select **Raw log** and use **Data exists** mode. Suppose the query returns `job`, `service`, `__time__`, and `__log__`:

* **Label fields**: Select stable dimensions such as `job` and `service` to identify the alert object.
* **Value fields**: Leave this empty in Data exists mode. If you use Threshold evaluation, select a field that can be converted to a number.
* **Additional information**: `__time__`, `__log__`, and any other unselected non-value fields are automatically carried with the alert.

<Warning>
  Do not use frequently changing fields such as `__time__`, `__log__`, or a trace ID as labels. Otherwise, each log row may create a different alert object.
</Warning>

See [Query Result Field Mapping](/en/monitors/alert-rules/query-result-fields) for default behavior, threshold references, and multi-query matching requirements.

## 5. Getting raw logs through a related query

Original logs can be obtained through related queries during alerts. But typically not recommended to get too many; just get 1 as a log sample to include in alert message.

![](https://docs-cdn.flashcat.cloud/imges/mon/ca5ea15fcfca066f260e09d0f8d91cf2.png)

Related query results can be rendered in "Notes Description", example:

```text theme={null}
{{- if eq $status "firing" }}
error log count: {{ $value | printf "%.3f" }}
{{- range $x := $relates.R1}}
Loki log time: {{(nanoTime $x.Fields.__time__ 8).Format "2006-01-02T15:04:05Z07:00"}}
Loki Log line: {{$x.Fields.__log__}}
{{- end}}
{{- end}}
```