Querying the Metric Store

This topic describes configuring and using Metric Store.

Overview

Metric Store implements the Prometheus Query Language for querying metrics for which you have access.

Prerequisites

Using Metric Store

Authorization & Authentication

When querying the API via HTTPS, each request must have the Authorization header set with a UAA provided token.

For Metric Store admins

As a Metric Store admin, you will have access to all recorded metrics (platform and application) and can execute Label and Series queries. You will need to add either doppler.firehose or logs.admin scope to your admin account.

You then should be able to run the following query:

cf login
curl -vvv -H "Authorization: $(cf oauth-token)" -G https://metric-store.SYSTEM-DOMAIN/api/v1/label/source_id/values

The status code will be 200 is everything is working properly.

For PAS developers

As a PAS developer, you can query metrics for applications you have access to. When querying with PromQL, you must specify the source_id label with the application guid as the label value.

For example:

curl -H "Authorization: $(cf oauth-token)" -G "https://metric-store.SYSTEM-DOMAIN/api/v1/query" \
  --data-urlencode "query=cpu{source_id=\"$(cf app --guid your-app-name)\"}"
Standard PAS application metrics
Metric Name Description PromQL Example
cpu CPU usage percentage avg(cpu{source_id="$app_guid"})
memory Memory usage in bytes avg(memory{source_id="$app_guid"})
memory_quota Memory Quota allocated in bytes memory{source_id="$app_guid"} / memory_quota{source_id="$app_guid"}
disk Disk usage in bytes avg(disk{source_id="$app_guid"}) / 1024 / 1024 / 1024 # in gigabytes
disk_quota Disk quota allocated in bytes disk{source_id="$app_guid"} / disk_quota{source_id="$app_guid"}
`http_total` HTTP request counts includes status `sum(rate(http_total{source_id=“$app_guid”})) by (status)`
`http_duration_seconds_*` HTTP request request latencies bucketed by duration `histogram_quantile(0.95, sum(rate(http_duration_seconds_bucket[5m])) by (le))`

Querying via Prometheus-Compatible HTTP Endpoints

Notes on PromQL

The ultimate goal of these endpoints is to create a fully-compliant, Prometheus-compatible interface. This should allow tools such as Grafana to work directly with Metric Store without any additional translation.

A valid PromQL metric name consists of the character [a-Z][0-9] and underscore. Names can begin with [a-Z] or underscore. Names cannot begin with [0-9]. As a measure to work with existing metrics that do not comply with the above format a conversion process takes place when matching on metric names. Any character that is not in the set of valid characters is converted to an underscore. The metric is not changed in the store.

e.g., to match on a metric name http.latency use the name http_latency as a search term.

GET /api/v1/query

Issues a PromQL instant query against Metric Store data. You can read more detail in the Prometheus documentation here.

Query Parameters:

Note: Admins (`doppler.firehose` or `logs.admin`) are permitted to use this query without specifying a source_id in the query parameter. Non-admins cannot use regex matchers on source_id.

$ curl -H "Authorization: $(cf oauth-token)" -G "https://metric-store.SYSTEM-DOMAIN/api/v1/query" --data-urlencode 'query=metric_name_0{source_id="source_id_0"}'
Response Body
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      { 
        "metric": {"__name__": "metric_name"}, 
        "value": [ "<timestamp>", "<value>" ] 
      },
      "..."
    ]
  }
}

GET /api/v1/query_range

Issues a PromQL range query against Metric Store data. You can read more detail in the Prometheus documentation here.

Query Parameters:

  • query is a Prometheus expression query string.
  • start is a UNIX timestamp in seconds or RFC3339. (e.g. date -d '24 hours ago' +%s). Start time is inclusive. [start..end)
  • end is a UNIX timestamp in seconds or RFC3339. (e.g. date +%s). End time is exclusive. [start..end)
  • step is a query resolution step width in duration format or float number of seconds.

Note: Admins (`doppler.firehose` or `logs.admin`) are permitted to use this query without specifying a source_id in the query parameter. Non-admins cannot use regex matchers on source_id.

$ curl -H "Authorization: $(cf oauth-token)" -G "https://metric-store.SYSTEM-DOMAIN/api/v1/query_range" \
    --data-urlencode 'query=metric_name_0{source_id="source_id_0"}' \
    --data-urlencode "start=$(date -d '24 hours ago' +%s)" \
    --data-urlencode "end=$(date +%s)" \
    --data-urlencode 'step=1h'
Response Body
{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {"__name__": "metric_name"},
        "values": [
          [ "<timestamp>", "<value>" ],
          "..."
        ]
      },
      "..."
    ]
  }
}

GET /api/v1/series

Issues a PromQL series query against Metric Store data. You can read more detail in the Prometheus documentation here.

Note: Non-admins (`doppler.firehose` or `logs.admin`) are not permitted to use this query.

Query Parameters:

  • start is a UNIX timestamp in seconds or RFC3339. (e.g. date -d '24 hours ago' +%s). Start time is inclusive. [start..end)
  • end is a UNIX timestamp in seconds or RFC3339. (e.g. date +%s). End time is exclusive. [start..end)
  • match[] is a series selector.
$ curl -H "Authorization: $(cf oauth-token)" -G "https://metric-store.SYSTEM-DOMAIN/api/v1/series" \
    --data-urlencode 'match[]=metric_name_0' \
    --data-urlencode 'match[]=metric_name_1' \
    --data-urlencode "start=$(date -d '24 hours ago' +%s)" \
    --data-urlencode "end=$(date +%s)"
Response Body
{
  "status": "success",
  "data": [
    {
      "__name__": "metric_name_0",
      "source_id": "source_id_0",
      "...": ".."
    },
    "..."
  ]
}

GET /api/v1/labels

Retrieve all label names for authorized source ids from the store.

Note: Non-admins (`doppler.firehose` or `logs.admin`) are not permitted to use this query.

Request
$ curl -H "Authorization: $(cf oauth-token)" "https://metric-store.SYSTEM-DOMAIN/api/v1/labels"
Response Body
{
  "status": "success",
  "data": [
    "__name__",
    "deployment",
    "..."
    "ip",
    "origin",
    "request_type",
    "source_id",
    "status_code",
    "unit"
  ]
}

GET /api/v1/label/<label_name>/values

Retrieve label values by label_name for all authorized source ids. The special label __name__ can be used to retrieve metric names from the store.

Note: Non-admins (`doppler.firehose` or `logs.admin`) are not permitted to use this query.

Request
$ curl -H "Authorization: $(cf oauth-token)" "https://metric-store.SYSTEM-DOMAIN/api/v1/label/<label-name>/values"
Response Body
{
    "status":"success",
    "data":["10", "1"]
}
Request
$ curl -H "Authorization: $(cf oauth-token)" "https://metric-store.SYSTEM-DOMAIN/api/v1/label/__name__/values"
Response Body
{
    "status":"success",
    "data":["metric_name_0", "metric_name_1", "..." ]
}