Copy as Markdown[Open in ChatGPT](https://chatgpt.com/?q=Read%20https%3A%2F%2Fdocs-docusaurus.kinsta.page%2Fdataprime%2Fuser-guide%2Fusing-dataprime%2Fpatterns.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.)[Open in Claude](https://claude.ai/new?q=Read%20https%3A%2F%2Fdocs-docusaurus.kinsta.page%2Fdataprime%2Fuser-guide%2Fusing-dataprime%2Fpatterns.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.) # How to use DataPrime to detect patterns and anomalies in your data ## Goal[​](#goal "Direct link to Goal") By the end of this guide you should be able to use DataPrime to filter, group, and statistically analyze your data to uncover frequent patterns and rare anomalies. You'll learn how to apply operators like [`matches`](https://docs-docusaurus.kinsta.page/dataprime/language-reference/functions-reference/string/matches/.md), [`countby`](https://docs-docusaurus.kinsta.page/dataprime/language-reference/commands-reference/countby/.md), [`percentile`](https://docs-docusaurus.kinsta.page/dataprime/language-reference/functions-reference/aggregation/percentile/.md), and [`bottom`](https://docs-docusaurus.kinsta.page/dataprime/language-reference/commands-reference/bottom/.md) to identify expected and unexpected behavior in your logs. ## Why it matters[​](#why-it-matters "Direct link to Why it matters") Every system has a baseline of "normal" behavior—and detecting when things deviate is critical. Whether you're tracking user behavior, spotting error spikes, or monitoring rare edge cases, DataPrime gives you powerful tools to surface these patterns in real time or retroactively. *** ## `matches` – Filter logs that match a regular expression[​](#matches--filter-logs-that-match-a-regular-expression "Direct link to matches--filter-logs-that-match-a-regular-expression") ### Description[​](#description "Direct link to Description") Use [`matches`](https://docs-docusaurus.kinsta.page/dataprime/language-reference/functions-reference/string/matches/.md) to test if a string field matches a given regular expression. This is useful for detecting structured patterns in log messages. ### Syntax[​](#syntax "Direct link to Syntax") ``` matches(string: string, pattern: regexp): bool ``` ### Example: Detect well-formed user purchase logs[​](#example-detect-well-formed-user-purchase-logs "Direct link to Example: Detect well-formed user purchase logs") **Description**: Filter logs where the `msg` field matches the pattern `"User [Name] bought [Number] [Item]"`. **Sample data** ``` { "msg": "User Chris bought 10 sunglasses" } { "msg": "User James bought 1 bed" } { "msg": "User X bo" } ``` **Query** ``` filter matches(msg, /User [A-Za-z]+ bought \d+ [A-Za-z]+/) ``` **Result** ``` { "msg": "User Chris bought 10 sunglasses" } { "msg": "User James bought 1 bed" } ``` *** ### Example: Detect malformed or truncated messages[​](#example-detect-malformed-or-truncated-messages "Direct link to Example: Detect malformed or truncated messages") **Description**: Flip the logic using `!matches` to find logs that deviate from the expected pattern. **Query** ``` filter !matches(msg, /User [A-Za-z]+ bought \d+ [A-Za-z]+/) ``` **Result** ``` { "msg": "User X bo" } ``` *** ## **countby** – Count grouped values[​](#countby--count-grouped-values "Direct link to countby--count-grouped-values") ### Description[​](#description-1 "Direct link to Description") Use [`countby`](https://docs-docusaurus.kinsta.page/dataprime/language-reference/commands-reference/countby/.md) to group documents by a key and count how many times each value appears. Ideal for identifying common or rare categories. ### Syntax[​](#syntax-1 "Direct link to Syntax") ``` countby [as ] [into ] ``` ### Example: Count logs by HTTP status code[​](#example-count-logs-by-http-status-code "Direct link to Example: Count logs by HTTP status code") **Sample data** ``` { "status_code": 200 } { "status_code": 404 } { "status_code": 500 } { "status_code": 200 } { "status_code": 403 } ``` **Query** ``` countby status_code into status_count ``` **Result** | status\_code | status\_count | | ------------ | ------------- | | 200 | 2 | | 404 | 1 | | 500 | 1 | | 403 | 1 | *** ### Example: Group status codes into buckets[​](#example-group-status-codes-into-buckets "Direct link to Example: Group status codes into buckets") **Query** ``` filter responseStatus.code != null | create status_range from case_greaterthan { responseStatus.code, 499 -> '5xx', 399 -> '4xx', 299 -> '3xx', 199 -> '2xx', 99 -> '1xx', _ -> 'other' } | countby status_range into status_count ``` **Result** | status\_range | status\_count | | ------------- | ------------- | | 2xx | 6619 | | 4xx | 172 | | 5xx | 101 | *** ## **percentile** – Find statistical outliers[​](#percentile--find-statistical-outliers "Direct link to percentile--find-statistical-outliers") ### Description[​](#description-2 "Direct link to Description") Use [`percentile`](https://docs-docusaurus.kinsta.page/dataprime/language-reference/functions-reference/aggregation/percentile/.md) to calculate the Nth percentile of a numeric field within a group. Useful for identifying slowest responses or largest payloads. ### Syntax[​](#syntax-2 "Direct link to Syntax") ``` percentile(percent: number, value: number): number ``` ### Example: Find 99th percentile latency by path[​](#example-find-99th-percentile-latency-by-path "Direct link to Example: Find 99th percentile latency by path") **Sample data** ``` { "operationName": "SELECT", "duration": 120 } { "operationName": "SELECT", "duration": 350 } { "operationName": "DELETE", "duration": 90 } { "operationName": "SELECT", "duration": 900 } ``` **Query** ``` source spans | groupby path agg percentile(0.99, latency) as latency_p99 ``` **Result** | path | latency\_p99 | | ------ | ------------ | | /home | 900 | | /about | 90 | *** ## **bottom** – Return the least frequent items[​](#bottom--return-the-least-frequent-items "Direct link to bottom--return-the-least-frequent-items") ### Description[​](#description-3 "Direct link to Description") Use [`bottom`](https://docs-docusaurus.kinsta.page/dataprime/language-reference/commands-reference/bottom/.md) to return the bottom N values based on count or another metric. This is ideal for spotting rarely used accounts, endpoints, or hosts. ### Syntax[​](#syntax-3 "Direct link to Syntax") ``` bottom by ``` *** ### Example: Least active users by count[​](#example-least-active-users-by-count "Direct link to Example: Least active users by count") **Sample data** ``` { "user": "Alice" } { "user": "Bob" } { "user": "Alice" } { "user": "Charlie" } ``` **Query** ``` bottom 2 user by count() ``` **Result** | user | count | | ------- | ----- | | Bob | 1 | | Charlie | 1 | *** ### Example: Least active users by total time[​](#example-least-active-users-by-total-time "Direct link to Example: Least active users by total time") **Sample data** ``` { "user": "Alice", "duration_ms": 500 } { "user": "Alice", "duration_ms": 800 } { "user": "Bob", "duration_ms": 100 } { "user": "Charlie", "duration_ms": 200 } ``` **Query** ``` bottom 2 user, count() as activity_count by sum(duration_ms) as total_time ``` **Result** | user | activity\_count | total\_time | | ------- | --------------- | ----------- | | Bob | 1 | 100 | | Charlie | 1 | 200 | *** ## Visualize result[​](#visualize-result "Direct link to Visualize result") Each of the above commands transforms your dataset and gives you focused insight: * `matches` filters for structural conformity. * `countby` helps visualize dominant categories. * `percentile` highlights statistical outliers. * `bottom` finds rare or underused values. You should see logs, tables, or metrics showing these patterns clearly—ready for dashboards or alerts. *** ## Common pitfalls[​](#common-pitfalls "Direct link to Common pitfalls") * **Too narrow filters**: If your regular expressions or filters are too strict, you might exclude valuable data. Try starting broad, then refining. * **Misusing `bottom`**: Without an explicit `by` clause, `bottom` may return confusing results. Always pair it with `count()` or another aggregation.