# Tail sampling with OpenTelemetry using AWS ECS EC2

Copy as Markdown[Open in ChatGPT](https://chatgpt.com/?q=Read%20https%3A%2F%2Fdocs-docusaurus.kinsta.page%2Fopentelemetry%2Ftail-sampling%2Ftail-sampling-with-opentelemetry-using-aws-ecs-ec2.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.)[Open in Claude](https://claude.ai/new?q=Read%20https%3A%2F%2Fdocs-docusaurus.kinsta.page%2Fopentelemetry%2Ftail-sampling%2Ftail-sampling-with-opentelemetry-using-aws-ecs-ec2.md%20and%20help%20me%20with%20my%20question%20about%20this%20Coralogix%20documentation%20page.)

This tutorial demonstrates how to configure an AWS ECS EC2 cluster, deploy OpenTelemetry to collect logs, metrics, and traces, and enable intelligent trace sampling. We will cover four deployment approaches:

1. **[ECS Cluster with Tail Sampling](#ecs-cluster-with-tail-sampling)**: Deploy agents and gateway directly to your application cluster
2. **[Central collector cluster for Tail Sampling](#central-collector-cluster-for-tail-sampling)**: Deploy a dedicated telemetry cluster separate from your applications
3. **[Verification using telemetrygen](#verification-using-telemetrygen)**: Test and verify your deployment with sample traces
4. **[Individual templates](#individual-templates)**: Deploy components individually for granular control

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

* **Existing AWS ECS cluster**: EC2-based cluster with instances running
* **S3 Bucket**: For storing OpenTelemetry configuration files
* **Send-Your-Data API Key**: Your Coralogix [Send-Your-Data API key](https://docs-docusaurus.kinsta.page/user-guides/account-management/api-keys/send-your-data-api-key/.md)
* **Coralogix domain**: Your [Coralogix domain](https://docs-docusaurus.kinsta.page/user-guides/account-management/account-settings/coralogix-domain/.md) (e.g., `eu2.coralogix.com`, `us2.coralogix.com`)
* **Application and Subsystem names**: [Application and subsystem names](https://docs-docusaurus.kinsta.page/user-guides/account-management/account-settings/application-and-subsystem-names/.md) for organizing your data in Coralogix
* **Image tag**: This implementation utilises the **Coralogix OpenTelemetry Collector** image [coralogixrepo/coralogix-otel-collector](https://hub.docker.com/r/coralogixrepo/coralogix-otel-collector/tags). This image is an enhanced version of the official OpenTelemetry Contrib distribution, featuring custom components and extended configuration loading options, including environment variable support. You must explicitly select a version when pulling. Tags can be found [here](https://hub.docker.com/r/coralogixrepo/coralogix-otel-collector/tags).

## ECS cluster with Tail Sampling[​](#ecs-cluster-with-tail-sampling "Direct link to ECS cluster with Tail Sampling")

### How it works[​](#how-it-works "Direct link to How it works")

The AWS ECS EC2 OpenTelemetry integration consists of the following components:

**OpenTelemetry Agent.** The Agent is deployed as a daemon service on each ECS instance within the cluster and collects telemetry data from the applications running on that instance. The agent is configured to send the logs and metrics to Coralogix and Traces to OpenTelemetry Gateway. The agent ensures that traces with the same ID are sent to the same gateway. This allows tail sampling to be performed on the traces correctly, even if they span multiple applications and instances.

**OpenTelemetry Gateway.** The Gateway is responsible for receiving telemetry data from the agents and forwarding it to the Coralogix backend. The Gateway is also responsible for performing tail sampling decisions and load balancing the telemetry data to the Coralogix backend.

### Deployment steps[​](#deployment-steps "Direct link to Deployment steps")

#### 1. Prepare configuration files[​](#1-prepare-configuration-files "Direct link to 1. Prepare configuration files")

A working example for OpenTelemetry configurations for the agent and gateway can be found here:

* [configs](https://github.com/coralogix/cloudformation-coralogix-aws/blob/master/opentelemetry/ecs-ec2-tail-sampling/examples)

Upload your OpenTelemetry configuration files to S3:

```
# Agent configuration (for daemon)

aws s3 cp agent-config.yaml s3://your-bucket/configs/agent-config.yaml



# Sampling configuration (for gateway agents)

aws s3 cp gateway-config.yaml s3://your-bucket/configs/gateway-config.yaml
```

#### 2. Deploy using a CloudFormation template ([`otel-tail-sampling.yaml`](https://github.com/coralogix/cloudformation-coralogix-aws/blob/master/opentelemetry/ecs-ec2-tail-sampling/otel-tail-sampling.yaml))[​](#2-deploy-using-a-cloudformation-template-otel-tail-samplingyaml "Direct link to 2-deploy-using-a-cloudformation-template-otel-tail-samplingyaml")

```
aws cloudformation deploy \

  --stack-name otel-tail-sampling \

  --template-file otel-tail-sampling.yaml \

  --capabilities CAPABILITY_NAMED_IAM \

  --parameter-overrides \

    ClusterName=your-existing-cluster \

    VpcId=vpc-xxxxxxxxx \

    SubnetIds="subnet-xxxxxxxxx,subnet-yyyyyyyyy" \

    SecurityGroupId=sg-xxxxxxxxx \

    CoralogixRegion=EU2 \

    CoralogixApiKey=your-api-key \

    CDOTImageVersion=image-version\

    S3ConfigBucket=your-bucket \

    AgentS3ConfigKey=configs/agent-config.yaml \

    SamplingS3ConfigKey=configs/sampling-config.yaml \

    ApplicationName=my-app \

    SubsystemName=production \

    SamplingTaskCount=2 \

    TaskExecutionRoleArn=if-not-provided-template-creates-it
```

**Alternatively, use the [ECS EC2 Tail Sampling Terraform module](https://github.com/coralogix/terraform-coralogix-aws/blob/master/modules/ecs-ec2-tail-sampling/README.md) documentation for usage instructions.**

## Central collector cluster for Tail Sampling[​](#central-collector-cluster-for-tail-sampling "Direct link to Central collector cluster for Tail Sampling")

This section describes how to deploy OpenTelemetry services to an existing ECS cluster for centralized tail sampling. This approach separates the telemetry collection infrastructure from your application workloads, providing centralized telemetry processing and intelligent sampling capabilities.

### Architecture[​](#architecture "Direct link to Architecture")

The central collector deployment consists of:

* **Receiver services**: Collect telemetry data from applications and route traces to the gateway
* **Gateway services**: Perform tail sampling and forward to Coralogix
* **CloudMap**: Service discovery for dynamic load balancing

**Trace flow**: Applications → Receiver → Gateway → Coralogix

#### The central collector deployment does not deploy agents as deamon on the cluster, if needed you can deploy this [CloudFormation template](https://github.com/coralogix/cloudformation-corlaogix-aws/tree/main/opentelemetry/ecs-ec2), with the necessary parameters[​](#the-central-collector-deployment-does-not-deploy-agents-as-deamon-on-the-cluster-if-needed-you-can-deploy-thiscloudformation-template-with-the-necessary-parameters "Direct link to the-central-collector-deployment-does-not-deploy-agents-as-deamon-on-the-cluster-if-needed-you-can-deploy-thiscloudformation-template-with-the-necessary-parameters")

Tip

**Span metrics configuration**: When using the central collector approach, span metrics should be configured on either the agents OR the receivers, but not both. Configuring span metrics on both components can lead to duplicate metrics and incorrect data.

### Deployment steps[​](#deployment-steps-1 "Direct link to Deployment steps")

#### 1. Prepare configuration files[​](#1-prepare-configuration-files-1 "Direct link to 1. Prepare configuration files")

A working example for OpenTelemetry configurations for the receiver and gateway can be found here:

* [configs](https://github.com/coralogix/cloudformation-coralogix-aws/blob/master/opentelemetry/ecs-ec2-tail-sampling/examples)

Upload your OpenTelemetry configuration files to S3:

```
# Receiver configuration with span metrics

aws s3 cp examples/receiver-config.yaml s3://your-bucket/configs/receiver-config-with-spanmetrics.yaml



OR:

# Receiver configuration without span metrics

aws s3 cp examples/receiver-config.yaml s3://your-bucket/configs/receiver-config-no-spanmetrics.yaml



# Gateway configuration (with tail sampling)

aws s3 cp examples/gateway-config.yaml s3://your-bucket/configs/gateway-config.yaml
```

#### 2. Deploy to an existing cluster ([`otel-complete-cluster.yaml`](https://github.com/coralogix/cloudformation-coralogix-aws/blob/master/opentelemetry/ecs-ec2-tail-sampling/otel-complete-cluster.yaml))[​](#2-deploy-to-an-existing-cluster-otel-complete-clusteryaml "Direct link to 2-deploy-to-an-existing-cluster-otel-complete-clusteryaml")

```
aws cloudformation deploy \

  --stack-name otel-complete-cluster \

  --template-file otel-complete-cluster.yaml \

  --capabilities CAPABILITY_NAMED_IAM \

  --parameter-overrides \

    ClusterName=otel-cluster \

    VpcId=vpc-xxxxxxxxx \

    SubnetIds="subnet-xxxxxxxxx,subnet-yyyyyyyyy" \

    SecurityGroupId=sg-xxxxxxxxx \

    KeyName=your-key-pair \

    ECSAMI=ami-xxxxxxxxx \

    CoralogixRegion=EU2 \

    CoralogixApiKey=your-api-key \

    CDOTImageVersion=Image-version \

    S3ConfigBucket=your-bucket \

    ReceiverS3ConfigKey=configs/receiver-config.yaml \

    GatewayS3ConfigKey=configs/gateway-config.yaml \

    ApplicationName=my-app \

    SubsystemName=production \

    ReceiverTaskCount=2 \

    GatewayTaskCount=1 \

    TaskExecutionRoleArn=if-not-provided-template-creates-it
```

**Alternatively, use the [ECS EC2 Tail Sampling Terraform module](https://github.com/coralogix/terraform-coralogix-aws/blob/master/modules/ecs-ec2-tail-sampling/README.md) documentation for usage instructions.**

### Service discovery[​](#service-discovery "Direct link to Service discovery")

The solution uses AWS CloudMap for service discovery:

* **Namespace**: `cx-otel`
* **Receiver Service**: `grpc-receiver.cx-otel`
* **Gateway Service**: `grpc-gateway.cx-otel`

### Connecting applications[​](#connecting-applications "Direct link to Connecting applications")

To connect your applications to the central collector cluster:

1. **Update Application Configuration**: Point your application's OTLP exporter to the receiver service
2. **Use Service Discovery**: Applications can discover receivers using the CloudMap DNS name
3. **Load Balancing**: Multiple receivers provide high availability and load distribution

### Enabling other clusters to send data[​](#enabling-other-clusters-to-send-data "Direct link to Enabling other clusters to send data")

To enable other ECS clusters or applications to send telemetry data to the central collector cluster:

#### Method 1: Cross-cluster service discovery[​](#method-1-cross-cluster-service-discovery "Direct link to Method 1: Cross-cluster service discovery")

1. **Share CloudMap Namespace**: Ensure both clusters can access the same CloudMap namespace

2. **Update Application Configuration**: Point applications to the receiver service:

   ```
   --otlp-endpoint=grpc-receiver.cx-otel:4317
   ```

3. **Network Connectivity**: Ensure VPC peering or transit gateway for cross-cluster communication

#### Method 2: Load balancer endpoint[​](#method-2-load-balancer-endpoint "Direct link to Method 2: Load balancer endpoint")

1. **Create Application Load Balancer**: Expose the receiver service via ALB
2. **Update Security Groups**: Allow traffic from other clusters
3. **Use ALB DNS Name**: Point applications to the load balancer endpoint

#### Method 3: VPC endpoints[​](#method-3-vpc-endpoints "Direct link to Method 3: VPC endpoints")

1. **Create VPC Endpoints**: For ECS and CloudMap services
2. **Cross-Account Access**: If clusters are in different AWS accounts
3. **IAM Permissions**: Ensure proper permissions for cross-cluster access

#### Configuration example[​](#configuration-example "Direct link to Configuration example")

For applications in other clusters, update their OpenTelemetry configuration:

```
exporters:

  otlp:

    endpoint: grpc-receiver.cx-otel:4317

    tls:

      insecure: true
```

## Verification using telemetrygen[​](#verification-using-telemetrygen "Direct link to Verification using telemetrygen")

To verify that your OpenTelemetry deployment is working correctly and traces are reaching Coralogix, you can deploy a telemetrygen task that generates test traces.

### Deploying a telemetrygen task[​](#deploying-a-telemetrygen-task "Direct link to Deploying a telemetrygen task")

#### 1. Create a task definition[​](#1-create-a-task-definition "Direct link to 1. Create a task definition")

Create a task definition for telemetrygen that connects to your receiver service:

```
{

    "compatibilities": [

        "EXTERNAL",

        "EC2"

    ],

    "containerDefinitions": [

        {

            "command": [

                "traces",

                "--otlp-endpoint=grpc-receiver.cx-otel:4317",

                "--otlp-insecure",

                "--rate=10",

                "--duration=1h"

            ],

            "cpu": 256,

            "environment": [],

            "essential": true,

            "image": "ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest",

            "logConfiguration": {

                "logDriver": "awslogs",

                "options": {

                    "awslogs-group": "/ecs/telemetrygen",

                    "awslogs-region": "us-east-1",

                    "awslogs-stream-prefix": "ecs"

                }

            },

            "memory": 512,

            "mountPoints": [],

            "name": "telemetrygen",

            "portMappings": [

                {

                    "containerPort": 8080,

                    "hostPort": 0,

                    "protocol": "tcp"

                }

            ],

            "systemControls": [],

            "volumesFrom": []

        }

    ],

    "cpu": "256",

    "executionRoleArn": "arn:aws:iam::ACCOUNT:role/ecsTaskExecutionRole",

    "family": "telemetrygen-task",

    "memory": "512",

    "networkMode": "bridge",

    "placementConstraints": [],

    "requiresCompatibilities": [

        "EC2"

    ],

    "volumes": []

}
```

#### 2. Register the task definition[​](#2-register-the-task-definition "Direct link to 2. Register the task definition")

```
aws ecs register-task-definition --cli-input-json file://telemetrygen-task-definition.json
```

#### 3. Run the telemetrygen task[​](#3-run-the-telemetrygen-task "Direct link to 3. Run the telemetrygen task")

```
aws ecs run-task \

  --cluster your-cluster-name \

  --task-definition telemetrygen-task \

  --launch-type EC2 \

  --count 1
```

### Configuration options[​](#configuration-options "Direct link to Configuration options")

#### For a complete cluster deployment[​](#for-a-complete-cluster-deployment "Direct link to For a complete cluster deployment")

When using the complete cluster template (`otel-complete-cluster.yaml`), use the receiver service:

```
--otlp-endpoint=grpc-receiver.cx-otel:4317
```

#### For a tail sampling deployment[​](#for-a-tail-sampling-deployment "Direct link to For a tail sampling deployment")

When using the tail sampling template (`otel-tail-sampling.yaml`), use the agent service:

```
--otlp-endpoint=grpc-gateway.cx-otel:4317
```

**Verify in Coralogix**: Check your Coralogix dashboard for incoming traces

* Look for traces with service name matching your configuration
* Verify trace sampling is working as expected

## Individual templates[​](#individual-templates "Direct link to Individual templates")

For more granular control over your deployment, you can use individual CloudFormation templates located in the `individual-templates/` directory:

### Available templates[​](#available-templates "Direct link to Available templates")

* **`cloudmap-namespace.yaml`**: Creates the CloudMap namespace for service discovery
* **`otel-daemon-template.yaml`**: Deploys the OpenTelemetry agent as a daemon service
* **`load-balancer-agents.yaml`**: Deploys the receiver service (for central collector approach)
* **`sampling-agents.yaml`**: Deploys the gateway service for tail sampling

### Example deployment[​](#example-deployment "Direct link to Example deployment")

These commands will deploy a complete telemetry infrastructure with agents sending logs and metrics directly to Coralogix, and traces to the Receiver for load balancing, then to the Gateway for tail sampling decisions.

1. **CloudMap namespace** (if not using complete template):

   ```
   aws cloudformation deploy \

     --stack-name otel-cloudmap \

     --template-file individual-templates/cloudmap-namespace.yaml \

     --capabilities CAPABILITY_NAMED_IAM \

     --parameter-overrides \

       VpcId=vpc-xxxxxxxxx
   ```

   **Template**: [`individual-templates/cloudmap-namespace.yaml`](https://github.com/coralogix/cloudformation-coralogix-aws/blob/master/opentelemetry/ecs-ec2-tail-sampling/individual-templates/cloudmap-namespace.yaml)

2. **Daemon agent**:

   ```
   aws cloudformation deploy \

     --stack-name otel-daemon \

     --template-file individual-templates/otel-daemon-template.yaml \

     --capabilities CAPABILITY_NAMED_IAM \

     --parameter-overrides \

       ClusterName=your-cluster \

       S3ConfigBucket=your-bucket \

       S3ConfigKey=configs/agent-config-to-receiver.yaml \

       CoralogixRegion=EU2 \

       CoralogixApiKey=your-api-key \

       CDOTImageVersion=v0.5.0
   ```

   **Template**: [`individual-templates/otel-daemon-template.yaml`](https://github.com/coralogix/cloudformation-coralogix-aws/blob/master/opentelemetry/ecs-ec2-tail-sampling/individual-templates/otel-daemon-template.yaml)

3. **Receiver** (for the central collector approach):

   ```
   aws cloudformation deploy \

     --stack-name otel-receiver \

     --template-file individual-templates/load-balancer-agents.yaml \

     --capabilities CAPABILITY_NAMED_IAM \

     --parameter-overrides \

       ClusterName=your-cluster \

       NamespaceId=ns-xxxxxxxxx \

       SubnetIds="subnet-xxxxxxxxx,subnet-yyyyyyyyy" \

       SecurityGroupId=sg-xxxxxxxxx \

       CDOTImageVersion=v0.5.0 \

       CoralogixRegion=EU2 \

       CoralogixApiKey=your-api-key \

       S3ConfigBucket=your-bucket \

       ReceiverS3ConfigKey=configs/receiver-config-with-span-metrics.yaml \

       ReceiverTaskCount=2
   ```

   **Template**: see [`individual-templates/`](https://github.com/coralogix/cloudformation-coralogix-aws/tree/master/opentelemetry/ecs-ec2-tail-sampling/individual-templates) in the cloudformation-coralogix-aws repository.

4. **Gateway**:

   ```
   aws cloudformation deploy \

     --stack-name otel-gateway \

     --template-file individual-templates/sampling-agents.yaml \

     --capabilities CAPABILITY_NAMED_IAM \

     --parameter-overrides \

       ClusterName=your-cluster \

       NamespaceId=ns-xxxxxxxxx \

       Subnets="subnet-xxxxxxxxx,subnet-yyyyyyyyy" \

       SecurityGroupId=sg-xxxxxxxxx \

       CDOTImageVersion=v0.5.0 \

       CoralogixRegion=EU2 \

       CoralogixApiKey=your-api-key \

       S3ConfigBucket=your-bucket \

       GatewayS3ConfigKey=configs/gateway-config.yaml \

       GatewayTaskCount=2
   ```

   **Template**: [`individual-templates/sampling-agents.yaml`](https://github.com/coralogix/cloudformation-coralogix-aws/blob/master/opentelemetry/ecs-ec2-tail-sampling/individual-templates/sampling-agents.yaml)

### Template features[​](#template-features "Direct link to Template features")

All individual templates support:

* **External IAM Roles**: Use existing roles via `TaskExecutionRoleArn` parameter
* **Conditional Resource Creation**: IAM roles are only created when external roles are not provided
* **Flexible Configuration**: Each component can be deployed independently
* **Service Discovery**: Automatic CloudMap integration for load balancing

## Sampling policies[​](#sampling-policies "Direct link to Sampling policies")

For detailed information about available sampling policies and configuration options, see the [OpenTelemetry Tail Sampling Processor documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor).

## Service discovery[​](#service-discovery-1 "Direct link to Service discovery")

The solution uses AWS CloudMap for service discovery:

* **Namespace**: `cx-otel`
* **Service**: `grpc-gateway`
* **DNS**: `grpc-gateway.cx-otel`

The agent automatically discovers sampling agents using the loadbalancing exporter.

### Scaling[​](#scaling "Direct link to Scaling")

Adjust the number of sampling agents based on your workload:

```
aws ecs update-service \

  --cluster your-cluster \

  --service coralogix-otel-sampling \

  --desired-count 4
```

### External IAM roles[​](#external-iam-roles "Direct link to External IAM roles")

Use existing IAM roles instead of creating new ones. The role must have the following permissions:

**Required policies:**

* `AmazonECSTaskExecutionRolePolicy` (AWS managed policy)
* Custom policy for S3 read access to your configuration bucket
* Custom policy for CloudMap service discovery

**Example role policy:**

```
{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Action": [

                "s3:GetObject",

                "s3:GetObjectVersion"

            ],

            "Resource": "arn:aws:s3:::your-config-bucket/*"

        },

        {

            "Effect": "Allow",

            "Action": [

                "servicediscovery:DiscoverInstances",

                "servicediscovery:ListInstances",

                "servicediscovery:ListServices",

                "servicediscovery:ListNamespaces"

            ],

            "Resource": "*"

        }

    ]

}
```

**Usage:**

```
--parameter-overrides \

  TaskExecutionRoleArn=arn:aws:iam::ACCOUNT:role/existing-role
```