Reporting App, Task, and Service Instance Usage

Page last updated:

This topic describes how to use the Cloud Foundry Command Line Interface (cf CLI) to retrieve historical system- and org-level usage information about your apps, tasks, and service instances through the Cloud Controller and Usage Service APIs.

You can also access usage information by using Apps Manager. For more information, see Reporting Instance Usage with Apps Manager.

To retrieve current app and event information from the Cloud Controller, see Retrieving App and Event Information in the open-source Cloud Foundry documentation.

Prerequisite

Before performing the procedures in this topic, ensure that your user is a member of the cloud_controller.admin or usage_service.audit group. For example, a UAAC admin can add a user to the usage_service.audit group with the following command:

uaac member add usage_service.audit USERNAME

Monitor Usage Service

Pivotal strongly recommends monitoring Usage Service at regular intervals with a recurring CI task or third-party monitoring tool.

One important reason to monitor Usage Service is that reports are generated based on app and service events fetched from the Cloud Controller. Events are dumped from the Cloud Controller database after 31 days. This means that if the Usage Service fails to fetch events for a period greater than 31 days, events can be permanently lost, and Usage Service will be unable to construct a complete history of usage on the foundation. Event fetching could fail for a number of reasons, including platform downtime and revoked credentials. Monitoring can help to catch these errors early and prevent incorrect data from propagating.

Use the following endpoints to monitor Usage Service:

https://app-usage.YOUR-SYSTEM-DOMAIN/heartbeat/failed_jobs?error_threshold=1

Usage Service continuously processes events and computes usage in the background using asynchronous jobs. This endpoint fails if the number of failed jobs exceeds a threshold.

Accepts a query parameter error_threshold. Set to 1 by default.

  • Succeeds with HTTP status 200 if the number of failed jobs is less than the error_threshold
  • Fails with HTTP status 500 if the number of failed jobs is greater than or equal to the error_threshold

Recommended error_threshold: 1

The current number of failed jobs is also emitted as a metric: usage_service.delayed_job_failures.

https://app-usage.YOUR-SYSTEM-DOMAIN/heartbeat/db

This endpoint checks if Usage Service is able to access its database in the internal PAS cluster.

  • Succeeds with HTTP status 200 if Usage Service successfully reads from its database
  • Fails with HTTP status 500 if Usage Service fails to read from its database

https://app-usage.YOUR-SYSTEM-DOMAIN/heartbeat/workers

This endpoint confirms that Usage Service workers are operating normally, and that background jobs are being scheduled.

  • Succeeds with HTTP status 200 if any Usage Service worker has checked in within the last 15 minutes
  • Fails with HTTP status 500 if no Usage Service worker has ever checked in OR if no Usage Service worker has checked in within the last 15 minutes

https://app-usage.YOUR-SYSTEM-DOMAIN/heartbeat/server_status

This endpoint confirms that the Usage Service server is operating normally.

  • Succeeds with HTTP status 200 if the Usage Service server is operating normally
  • Fails with HTTP status 500 if the Usage Service server is failing.

Obtain System Usage Information

You can obtain the following system-wide usage information:

App Usage

Use curl to make a request to /system_report/app_usages on the Usage Service endpoint to show system-wide app usage data:

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/system_report/app_usages" -k -v \
-H "authorization: `cf oauth-token`"
{
"report_time": "2017-04-11 22:28:24 UTC",
"monthly_reports": [
    {
      "month": 4,
      "year": 2017,
      "average_app_instances": 17855.256153713308,
      "maximum_app_instances": 18145,
      "app_instance_hours": 4686533.080277303
    }
  ],
"yearly_reports": [
    {
      "year": 2017,
      "average_app_instances": 16184.9,
      "maximum_app_instances": 18145,
      "app_instance_hours": 39207433.0802773
    }
  ]
}

The following table describes the data generated by this report:

Value Description Calculation Method
app_instance_hours The total number of hours app instances have been running on the foundation for a specific month or year. App start and stop events are used to create a rolling record of each single app’s runtime per month. Monthly records are then aggregated to calculate total app hours for a year.
average_app_instances The average number of app instances simultaneously running at any given time during the specified period.

app_instance_hours divided by the number of hours for a month or year.

Example: For the month of January (744 hours), 100 apps run for 300 hours each, generating 30,000 app_instance_hours. Dividing by 744 hours returns a value of 40.3 for average_app_instances.
maximum_app_instances The highest concurrent number of app instances running on the foundation for the month or year. Each time an app is started or stopped, a new record of the current number of running apps is created. The maximum concurrent value then comes from the record with the maximum value over the desired time range.

Task Usage

Use curl to make a request to /system_report/task_usages on the Usage Service endpoint to show system-wide task usage data:

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/system_report/task_usages" -k -v \
-H "authorization: `cf oauth-token`"
{
"report_time": "2017-04-11T22:33:48.971Z",
"monthly_reports": [
    {
      "month": 4,
      "year": 2017,
      "total_task_runs": 235,
      "maximum_concurrent_tasks": 7,
      "task_hours": 43045.201944444445
    }
  ],
"yearly_reports": [
    {
      "year": 2017,
      "total_task_runs": 2894,
      "maximum_concurrent_tasks": 184,
      "task_hours": 45361.26694444445
    }
  ]
}
Value Description Calculation Method
total_task_runs The total number of tasks that have run on a foundation for the specified month or year. Task events are summarized into monthly and yearly reports.
maximum_concurrent_tasks The maximum number of tasks simultaneously running at any given time during the specified period. Each time a task is started or stopped, a new record of the current number of running tasks is created. The maximum concurrent value then comes from the record with the maximum value over the desired time range.
task_hours The total number of hours task instances have been running on the foundation for a specific month or year. Task start and stop events are processed to create daily task summaries. Daily task summaries are then summed to compute the total duration in hours for the specified period.

Service Usage

Use curl to make a request to /system_report/service_usages on the Usage Service endpoint to show system-wide service usage data:

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/system_report/service_usages" -k -v \
-H "authorization: `cf oauth-token`"
{
  "report_time": "2017-05-11 18:29:14 UTC",
  "monthly_service_reports": [
    {
      "service_name": "fake-service-0507f1fd-2340-49a6-9d43-a347a5f5f6be",
      "service_guid": "177dcfde-cd51-4058-bd86-b98015c295f5",
      "usages": [
        {
          "month": 1,
          "year": 2017,
          "duration_in_hours": 0,
          "average_instances": 0,
          "maximum_instances": 0
        },
        {
          "month": 2,
          "year": 2017,
          "duration_in_hours": 0,
          "average_instances": 0,
          "maximum_instances": 0
        },
        {
          "month": 3,
          "year": 2017,
          "duration_in_hours": 4.182222222222227,
          "average_instances": 0,
          "maximum_instances": 2
        },
        {
          "month": 4,
          "year": 2017,
          "duration_in_hours": 2176.962222222186,
          "average_instances": 3,
          "maximum_instances": 7
        },
        {
          "month": 5,
          "year": 2017,
          "duration_in_hours": 385.61388888888854,
          "average_instances": 1.5,
          "maximum_instances": 3
        }
      ],
      "plans": [
        {
          "usages": [
            {
              "month": 1,
              "year": 2017,
              "duration_in_hours": 0,
              "average_instances": 0,
              "maximum_instances": 0
            },
            {
              "month": 2,
              "year": 2017,
              "duration_in_hours": 0,
              "average_instances": 0,
              "maximum_instances": 0
            },
            {
              "month": 3,
              "year": 2017,
              "duration_in_hours": 4.182222222222227,
              "average_instances": 0,
              "maximum_instances": 2
            },
            {
              "month": 4,
              "year": 2017,
              "duration_in_hours": 1465.6388888888941,
              "average_instances": 2,
              "maximum_instances": 5
            },
            {
              "month": 5,
              "year": 2017,
              "duration_in_hours": 385.61388888888854,
              "average_instances": 1.5,
              "maximum_instances": 3
            }
          ],
          "service_plan_name": "fake-plan",
          "service_plan_guid": "ac09f607-f4e5-4807-af16-e95856061bd7"
        }
      ]
    }
  ]
}
Value Description Calculation Method
usages.duration_in_hours The duration in hours service instances ran within a month and year. Service instance start and stop events are processed into summary records, grouped by month and year, and summed.
usages.average_instances The average number of service instances ran within a month and year. duration_in_hours divided by the number of hours for a month or year.
usages.maximum_instances The maximum number of simultaneously running service instances within a month and year. Usage Service walks through all service instance records for a month and year.
plans.usages.duration_in_hours The duration in hours service instances of a plan type ran within a month and year. usages.duration_in_hours filtered by plan type.
plans.usages.average_instances The average number of service instances of a plan type that ran within a month and year. usages.average_instances filtered by plan type.
plans.usages.maximum_instances The maximum number of simultaneously running service instances of a plan type within a month and year. usages.maximum_instances filtered by plan type.

All Orgs Usage (Beta)

The app_usages endpoint directly under your system domain returns system-wide app usage for all orgs that you have access to. The command works like the org-specific App Usage command, except you do not supply an org GUID. You can set order_by to either total_instance_hours or megabyte_hours.

The order_by ordering defaults to ascending, which lists orgs with the most usage at the end of the output. To change the sort ordering to descending, prepend the order_by argument with a -, as in order_by=-megabyte_hours.

Use curl to make a request to /app_usages on the Usage Service endpoint to show system-wide organization usage data:

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/app_usages?start=2018-010-01&end=2018-11-01&order_by=total_instance_hours" -k -v \
-H "authorization: `cf oauth-token`"
{
   "period_start":"2018-01-10T00:00:00Z",
   "period_end":"2018-11-01T23:59:59Z",
   "Organizations":[ 
      {
         "guid":"d5c9f947-df85-45af-abab-00a2174598f9",
         "total_instance_hours":"91535.6",
         "megabyte_hours":"1830712.2"
      },
      {
         "guid":"a9605fa4-99da-4932-84f5-3db8698c1fc1",
         "total_instance_hours":"99440.8",
         "megabyte_hours":"96698582.2"
      }
   ]
}

This beta API endpoint may change; please try it out and help Pivotal make it better.

Obtain Org Usage Information

You can obtain the following org-specific usage information:

You must have the GUID of the org you want to obtain information about in order to perform the procedures in this section. To retrieve your org GUID, run the cf org command:

$ cf org YOUR-ORG --guid

App Usage

Use curl to make a request to /app_usages on the Usage Service endpoint to show app usage in an org. You must complete the placeholders in start=YYYY-MM-DD&end=YYYY-MM-DD to define a date range.

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/organizations/YOUR-ORG-GUID/app_usages?start=YYYY-MM-DD&end=YYYY-MM-DD" \
-k -v \
-H "authorization: `cf oauth-token`"
   
{
  "organization_guid": "88e485fa-2222-4993-8ef5-bac3711adfee",
  "period_start": "2019-01-16T00:00:00Z",
  "period_end": "2019-01-17T23:59:59Z",
  "app_usages": [
    {
      "space_guid": "d6988d4d-6175-4dee-b219-a6eabbc05514",
      "space_name": "development",
      "app_name": "static2",
      "app_guid": "1afc2b20-dd41-4ba4-8e6e-c894b5a2cb23",
      "parent_application_name": "static2",
      "parent_application_guid": "1afc2b20-dd41-4ba4-8e6e-c894b5a2cb23",
      "instance_count": 1,
      "memory_in_mb_per_instance": 1024,
      "duration_in_seconds": 172800
    },
    {
      "space_guid": "d6988d4d-6175-4dee-b219-a6eabbc05514",
      "space_name": "development",
      "app_name": "static",
      "app_guid": "54e767ff-4f83-45b1-bf06-46c501217845",
      "parent_application_name": "static",
      "parent_application_guid": "54e767ff-4f83-45b1-bf06-46c501217845",
      "instance_count": 1,
      "memory_in_mb_per_instance": 1024,
      "duration_in_seconds": 172800
    }
  ]
}

You may see results with “instance_count”: 0. This happens when a v3 app is pushed but has processes that aren’t scaled up. The memory configuration and duration are still stored and can be useful for resource management. Apps with ‘0’ instances are not counted in the App Usages System Report above.

Task Usage

Use curl to make a request to /task_usages on the Usage Service endpoint to show task usage in an org. You must complete the placeholders in start=YYYY-MM-DD&end=YYYY-MM-DD to define a date range.

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/organizations/YOUR-ORG-GUID/task_usages?start=YYYY-MM-DD&end=YYYY-MM-DD" \
-k -v \
-H "authorization: `cf oauth-token`"
{
     "organization_guid": "8f88362f-547c-4158-808b-4605468387d5",
     "period_start": "2014-01-01",
     "period_end": "2017-04-04",
     "spaces": {
       "e6445eb3-fdac-4049-bafc-94d1703d5e78": {
         "space_name": "smoketest",
         "task_summaries": [
           {
             "parent_application_guid": "04cd29d5-1f9e-4900-ac13-2e903f6582a9",
             "parent_application_name": "task-dummy-app",
             "memory_in_mb_per_instance": 256,
             "task_count_for_range": 54084,
             "concurrent_task_count_for_range": 5,
             "total_duration_in_seconds_for_range": 37651415
           }
          ]
       },
       "b66665e4-873f-4482-acf1-89307ba9c6e4": {
         "space_name": "smoketest-experimental",
         "task_summaries": [
         {
             "parent_application_guid": "d941b689-4a27-44ec-91d3-1f97434dbed9",
             "parent_application_name": "console-blue",
             "memory_in_mb_per_instance": 256,
             "task_count_for_range": 14,
             "concurrent_task_count_for_range": 2,
             "total_duration_in_seconds_for_range": 20583
           }
          ]
       }
     }
}

Note: In the /task_usages endpoint, memory_in_mb_per_instance is the memory of the task.

Service Usage

Use curl to make a request to /service_usages on the Usage Service endpoint to show service usage in an org. You must complete the placeholders in start=YYYY-MM-DD&end=YYYY-MM-DD to define a date range.

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/organizations/`cf org YOUR-ORG --guid`/service_usages?start=YYYY-MM-DD&end=YYYY-MM-DD" -k -v -H "authorization: `cf oauth-token`"
{
     "organization_guid": "8d83362f-587a-4148-806b-4407428887b5",
     "period_start": "2016-06-01T00:00:00Z",
     "period_end": "2016-06-13T23:59:59Z",
     "service_usages": [
        {
          "deleted": false,
          "duration_in_seconds": 1377982.52,
          "service_guid": "02802293-b769-44cc-807f-eee331ba9b2b",
          "service_instance_creation": "2016-01-20T18:48:16.000Z",
          "service_instance_deletion": null,
          "service_instance_guid": "b25b4481-19aa-4504-82c9-f303e7e9ed6e",
          "service_instance_name": "something-usage-db",
          "service_instance_type": "managed_service_instance",
          "service_name": "my-mysql-service",
          "service_plan_guid": "70915a70-7311-4f3e-ab0d-4a7dfd3ef2d9",
          "service_plan_name": "20gb",
          "space_guid": "e6445eb3-fdac-4049-bafc-94d1703d5e78",
          "space_name": "outer-space"
        }
    ]
}

Usage Availability

The app, task, and service instance usage data returned from the Usage Service is delayed to ensure accuracy. The /usage_availability endpoint can be used to retrieve the date which the Usage Service data is accurate to. The Usage Service is able to provide data for time periods up to and including this date.

Use the following curl command to retrieve the current usage availability timestamp:

curl "https://app-usage.YOUR-SYSTEM-DOMAIN/usage_availability" -k -v
{"date":"2019-01-01"}

Where YOUR-SYSTEM-DOMAIN is the system domain of your PAS environment.

Usage Data Retention

Usage service retains all summarized monthly data and never deletes it. The endpoints listed below return data for all years:

  • /system_report/app_usages
  • /system_report/task_usages
  • /system_report/service_usages

For examples of how to curl these endpoints, see Obtain System Usage Information.

Usage service deletes old records of granular data, such as usage per app, per day. By default, granular data is deleted after 365 days, so the following endpoints return data for the last 365 days. The retention period is configurable in the PAS tile in Advanced Features.

The endpoints listed below return granular data:

  • /organizations/ORG-GUID/app_usages
  • /organizations/ORG-GUID/task_usages
  • /organizations/ORG-GUID/service_usages
  • /app_usages

Example: Autogenerated Database Usage Reports

For security compliance and record keeping, some Pivotal Platform customers create Concourse CI/CD pipelines that regularly run JavaScript scripts to do the following:

  1. Call the app-usage APIs in the above sections of this topic, as well as the Security Event Logging topic, to collect data about service instances from a Pivotal Platform foundation. This data includes database instance information such as tile or database type, database creator, and date of creation or deletion.

  2. Merge and format the data about services instances into a JSON output file and save it to an S3 bucket.

  3. Input the file to another process that does the following:

    1. Parses the data
    2. Eliminates duplicate entries from previous reports
    3. Sends the data to a database compliance system through APIs