Understanding logs

Goal

By the end of this guide, you should be able to:

• Copy the full nested JSON path for any field
• Understand how user data is stored inside the $d, $m, $l objects
• Use the choose and filter functions to view and narrow down logs by specific fields

Why It Matters

Before you can investigate issues, correlate events, or create dashboards, you need to know how to explore raw logs. This guide will help you extract meaningful structure from JSON events and use DataPrime’s core tools to begin transforming and filtering logs effectively.

---

Use choose to select specific fields

If you want to see just what you need, use the choose command:

choose k8s

Or to access multiple items

choose k8s, user_id

This will return JSON objects with only the k8s and user_id fields.

Pro tip: You can also rename fields or compute new ones using as:

choose (response_bytes / 1024) as kb_returned

---

Understand the structure: $d is your user data

All user log data lives under the $d namespace. This includes fields like user_id, cluster_name, and nested structures like http.request.headers.

In this log, the "root" level of the log is $d.

{
    "user_id": 11,
    "cluster_name": "frank",
    "http": {
        "request": {
            "headers": {
                "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
                "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
                "accept_language": "en-US,en;q=0.5",
                "accept_encoding": "gzip, deflate, br",
                ...
            }
        }
    }
}

So you can access user_agent with $d.http.request.headers.user_agent.** But $d is the default namespace, so it can be omitted entirely**, http.request.headers.user_agent

Think of $d as the core payload of each log. Metadata like severity or timestamp is stored separately under $m.

note

Your field names in $d are your own, so they don't necessarity match what are in these docs.

Don’t forget that $d is not strictly needed in most cases. If it’s left out you will query your user data. To query metadata $m, labels $l, and parameters $p, those access mechanisms are required.

Copy a full nested JSON path

Want to build a query on a deeply nested field?

• Right-click on a key in the expanded log.
• Select Copy complete JSON path.
• This will copy the full path (e.g., http.request.headers.user_agent) which you can use in your query.

This is especially useful when dealing with structured logs or observability pipelines.

Accessing data

When writing queries in DataPrime, you’ll often reference keys inside your logs. These can be accessed in two ways:

Use dot notation for clean, readable paths when your key names don't contain special characters.

Say your JSON object looks like this:

{
  "k8s": {
    "container": {
      "id": 1
    }
  }
}

You can access the id using the flat dot notation and the choose command.

source logs
| choose k8s.container.id

Or you can use bracket notation

source logs
| choose $d['k8s']['container']['id']

Using bracket notation is required to access a key containing spaces, symbols, dots (.), or any other special character.

This log contains characters in its keys that break dot notation:

{
  "k8s": {
    "container": {
      "id": 1
    }
  },
  "k8s.container.id": 2,
  "user name": "Rick",
  "DataPrime 🤘": "Rocks"
}

query

source logs
| choose $d['k8s.container.id'], $d['user name'], $d['DataPrime 🤘']

result

{
   DataPrime 🤘: Rocks
   k8s.container.id: 2
   user name: Rick
}

You can also mix and match both styles as needed

source logs
| choose $d['k8s'].container['id']

or

source logs
| choose k8s.container['id']

These examples just prove the point, but use whatever best suits your use cases and preferences.

Use filter to focus your search

Once you know the path to a key, narrow your logs with a filter:

source logs
| filter http.status_code == 500

This returns only logs where the HTTP status code equals 500.

You can combine with choose for a compact view:

source logs
| filter http.status_code == 500
| choose timestamp, http.path, http.status_code

---

Common Pitfalls

• Forgetting the $d context: If your field isn’t found, make sure it’s inside the $d object unless it’s a label ($l) or metadata ($m).
• Copying partial paths: Always use “Copy complete JSON path” to avoid malformed queries.
• Too many fields? Use choose aggressively to reduce noise.