# Processing and routing

Copy as Markdown[Open in ChatGPT](https://chatgpt.com/?q=Read%20https%3A%2F%2Fdocs-docusaurus.kinsta.page%2Fuser-guides%2Fdata-layer%2Fdata-processing.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.)[Open in Claude](https://claude.ai/new?q=Read%20https%3A%2F%2Fdocs-docusaurus.kinsta.page%2Fuser-guides%2Fdata-layer%2Fdata-processing.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.)

When data enters Coralogix, it goes through a structured lifecycle: received from shippers or agents, transformed with DataPrime rules, routed based on attributes like region or team, and directed into the appropriate dataspace and dataset. If a dataset doesn't already exist, it's created automatically and inherits configuration from the parent dataspace.

## High-level flow[​](#high-level-flow "Direct link to High-level flow")

<!-- -->

1. **Ingress**

   Data enters the platform through a shipper or agent. Customers can pre-define a `targetDatabase` (dataspace) and `targetDataset` via the shipper config.

2. **Pre-processing**

   Coralogix applies DataPrime transformation rules:

   For example, fields can be removed or recalculated before finalizing the data structure:

   ```
   remove derived_metric

   | replace raw_value with normalized_value

   | create derived_metric from quantity * 232
   ```

3. **Routing**

   A set of conditions (e.g., region, team, environment) determine where data goes:

   For example, different regions or teams in the data can determine the target dataspace or dataset.

   ```
   <region == 'us2'>       ->      [targetDataspace = bu1, targetDataset = logs-us]

   <team == 'neptune'>     ->      [targetDataspace = planet, targetDataset = gassy]

   <team == 'venus'>       ->      [targetDataspace = planet, targetDataset = rocky]
   ```

   Routing is fully **data-driven** and can include dynamic elements:

   ```
   <region>                 ->     [targetDataspace = bu2, targetDataset = logs-{$l.applicationname}]
   ```

4. **Dataset creation**

   If a dataset does not already exist, it will be created automatically under the target dataspace.

5. **Configuration inheritance**

   The dataset inherits configuration from its dataspace, including:

   * Storage prefix (e.g., `s3://bucket/my-dataspace/logs-regionX`)
   * Retention and archive rules
   * Access control policies
   * Metadata enrichment

6. **Final storage & query**

   Once routed and processed, the data is written to object storage and made available for querying.

[![data ingestion](/assets/images/ingestion-dec6a190e860148feba562d3dfdc23ef.webp)](https://docs-docusaurus.kinsta.page/assets/images/ingestion-dec6a190e860148feba562d3dfdc23ef.webp)

***

## Example dataset structure[​](#example-dataset-structure "Direct link to Example dataset structure")

```
default/

  └── logs

  └── spans



business-unit1/

  └── logs



business-unit2/

  └── logs-cx510

  └── logs-euprod2

  └── logs-production

  └── ...

  └── <datasets created dynamically as data arrives>



security/

  └── ...
```

***

<!-- -->

<!-- -->

## Handling quota and duplication[​](#handling-quota-and-duplication "Direct link to Handling quota and duplication")

* Duplicating data across datasets (e.g., routing the same event to multiple targets) **will count against your quota**.
* You can monitor dataset-level usage in **Dataset Management**.
* Dataset quotas can be enforced per team, space, or workload.
* The [data usage](https://docs-docusaurus.kinsta.page/user-guides/account-management/payment-and-billing/data-usage/.md) page shows detailed breakdowns to help you understand where and how your data is being consumed.