Reporting App, Task, and Service Instance Usage

Page last updated:

Warning: Pivotal Cloud Foundry (PCF) v2.4 is no longer supported because it has reached the End of General Support (EOGS) phase as defined by the Support Lifecycle Policy. To stay up to date with the latest software and security updates, upgrade to a supported version.

This topic describes how to use the Cloud Foundry Command Line Interface (cf CLI) to retrieve historical system- and org-level usage information about your apps, tasks, and service instances through the Cloud Controller and Usage Service APIs.

Usage reports are compiled from the /v2/app_usage_events endpoint. For more information, see List all App Usage Events in the Cloud Foundry API documentation.

You can also access usage information by using Apps Manager. For more information, see the Reporting Instance Usage with Apps Manager topic.

To retrieve current app and event information from the Cloud Controller, see Retrieving App and Event Information in the open-source Cloud Foundry documentation.

Prerequisite

Before performing the procedures in this topic, ensure that your user is a member of the cloud_controller.admin or usage_service.audit group. For example, a UAAC admin can add a user to the usage_service.audit group with the following command:

uaac member add usage_service.audit USERNAME

Obtain System Usage Information

You can obtain the following system-wide usage information:

App Usage

Use curl to make a request to /system_report/app_usages on the Usage Service endpoint to show system-wide app usage data:

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/system_report/app_usages" -k -v \
-H "authorization: `cf oauth-token`"
{
"report_time": "2017-04-11 22:28:24 UTC",
"monthly_reports": [
    {
      "month": 4,
      "year": 2017,
      "average_app_instances": 17855.256153713308,
      "maximum_app_instances": 18145,
      "app_instance_hours": 4686533.080277303
    }
  ]
"yearly_reports": [
    {
      "year": 2017,
      "average_app_instances": 16184.9,
      "maximum_app_instances": 18145,
      "app_instance_hours": 39207433.0802773
    }
  ]
}

The following table describes the data generated by this report:

Value Calculation Method
app_instance_hours The total number of hours app instances have been running on the foundation for the month or year.
average_app_instances The total app_instance_hours on the foundation for the month or year divided by the total hours for that time period.

Example: For the month of January, 100 apps run for 300 hours each, generating a value of 30000 for app_instance_hours. When divided by 744, the total number of hours in January, you get a value of 40.3 for average_app_instances.
maximum_app_instances The highest concurrent number of app instances running on the foundation for the month or year.

Task Usage

Use curl to make a request to /system_report/task_usages on the Usage Service endpoint to show system-wide task usage data:

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/system_report/task_usages" -k -v \
-H "authorization: `cf oauth-token`"
{
"report_time": "2017-04-11T22:33:48.971Z",
"monthly_reports": [
    {
      "month": 4,
      "year": 2017,
      "total_task_runs": 235,
      "maximum_concurrent_tasks": 7,
      "task_hours": 43045.201944444445
    }
  ]
"yearly_reports": [
    {
      "year": 2017,
      "total_task_runs": 2894,
      "maximum_concurrent_tasks": 184,
      "task_hours": 45361.26694444445
    }
  ]
}

Service Usage

Use curl to make a request to /system_report/service_usages on the Usage Service endpoint to show system-wide service usage data:

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/system_report/service_usages" -k -v \
-H "authorization: `cf oauth-token`"
{
  "report_time": "2017-05-11 18:29:14 UTC",
  "monthly_service_reports": [
    {
      "service_name": "fake-service-0507f1fd-2340-49a6-9d43-a347a5f5f6be",
      "service_guid": "177dcfde-cd51-4058-bd86-b98015c295f5",
      "usages": [
        {
          "month": 1,
          "year": 2017,
          "duration_in_hours": 0,
          "average_instances": 0,
          "maximum_instances": 0
        },
        {
          "month": 2,
          "year": 2017,
          "duration_in_hours": 0,
          "average_instances": 0,
          "maximum_instances": 0
        },
        {
          "month": 3,
          "year": 2017,
          "duration_in_hours": 4.182222222222227,
          "average_instances": 0,
          "maximum_instances": 2
        },
        {
          "month": 4,
          "year": 2017,
          "duration_in_hours": 2176.962222222186,
          "average_instances": 3,
          "maximum_instances": 7
        },
        {
          "month": 5,
          "year": 2017,
          "duration_in_hours": 385.61388888888854,
          "average_instances": 1.5,
          "maximum_instances": 3
        }
      ],
      "plans": [
        {
          "usages": [
            {
              "month": 1,
              "year": 2017,
              "duration_in_hours": 0,
              "average_instances": 0,
              "maximum_instances": 0
            },
            {
              "month": 2,
              "year": 2017,
              "duration_in_hours": 0,
              "average_instances": 0,
              "maximum_instances": 0
            },
            {
              "month": 3,
              "year": 2017,
              "duration_in_hours": 4.182222222222227,
              "average_instances": 0,
              "maximum_instances": 2
            },
            {
              "month": 4,
              "year": 2017,
              "duration_in_hours": 1465.6388888888941,
              "average_instances": 2,
              "maximum_instances": 5
            },
            {
              "month": 5,
              "year": 2017,
              "duration_in_hours": 385.61388888888854,
              "average_instances": 1.5,
              "maximum_instances": 3
            }
          ],
          "service_plan_name": "fake-plan",
          "service_plan_guid": "ac09f607-f4e5-4807-af16-e95856061bd7"
        }

All Orgs Usage (Beta)

The app_usages endpoint directly under your system domain returns system-wide app usage for all orgs that you have access to. The command works like the org-specific App Usage command, except you do not supply an org GUID. You can set order_by to either total_instance_hours or megabyte_hours.

The order_by ordering defaults to ascending, which lists orgs with the most usage at the end of the output. To change the sort ordering to descending, prepend the order_by argument with a -, as in order_by=-megabyte_hours.

Use curl to make a request to /app_usages on the Usage Service endpoint to show system-wide organization usage data:

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/app_usages?start=2018-010-01&end=2018-11-01&order_by=total_instance_hours" -k -v \
-H "authorization: `cf oauth-token`"
{
   "period_start":"2018-01-10T00:00:00Z",
   "period_end":"2018-11-01T23:59:59Z",
   "Organizations":[ 
      {
         "guid":"d5c9f947-df85-45af-abab-00a2174598f9",
         "total_instance_hours":"91535.6",
         "megabyte_hours":"1830712.2"
      },
      {
         "guid":"a9605fa4-99da-4932-84f5-3db8698c1fc1",
         "total_instance_hours":"99440.8",
         "megabyte_hours":"96698582.2"
      }
   ]
}

This beta API endpoint may change; please try it out and help Pivotal make it better.

Obtain Org Usage Information

You can obtain the following org-specific usage information:

You must have the GUID of the org you want to obtain information about in order to perform the procedures in this section. To retrieve your org GUID, run the cf org command:

$ cf org YOUR-ORG --guid

App Usage

Use curl to make a request to /app_usages on the Usage Service endpoint to show app usage in an org. You must complete the placeholders in start=YYYY-MM-DD&end=YYYY-MM-DD to define a date range.

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/organizations/YOUR-ORG-GUID/app_usages?start=YYYY-MM-DD&end=YYYY-MM-DD" \
-k -v \
-H "authorization: `cf oauth-token`"
{
       "organization_guid": "8d83362f-587a-4148-806b-4407428887b5",
     "period_start": "2016-06-01T00:00:00Z",
     "period_end": "2016-06-13T23:59:59Z",
     "app_usages": [
        {
          space_guid: "44435fd6-fbac-5049-bbfc-92d1603a5e98"
          space_name: "outer-space"
          app_guid: "00ecd7ce-1dd0-4b3f-63b9-744c9de42afc"
          app_name: "your-app"
          instance_count: 6
          memory_in_mb_per_instance: 64
          duration_in_seconds: 76730
        }
      ]
   }

Task Usage

Use curl to make a request to /task_usages on the Usage Service endpoint to show task usage in an org. You must complete the placeholders in start=YYYY-MM-DD&end=YYYY-MM-DD to define a date range.

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/organizations/YOUR-ORG-GUID/task_usages?start=YYYY-MM-DD&end=YYYY-MM-DD" \
-k -v \
-H "authorization: `cf oauth-token`"
{
     "organization_guid": "8f88362f-547c-4158-808b-4605468387d5",
     "period_start": "2014-01-01",
     "period_end": "2017-04-04",
     "spaces": {
       "e6445eb3-fdac-4049-bafc-94d1703d5e78": {
         "space_name": "smoketest",
         "task_summaries": [
           {
             "parent_application_guid": "04cd29d5-1f9e-4900-ac13-2e903f6582a9",
             "parent_application_name": "task-dummy-app",
             "memory_in_mb_per_instance": 256,
             "task_count_for_range": 54084,
             "concurrent_task_count_for_range": 5
             "total_duration_in_seconds_for_range": 37651415
           },
          ]
       },
       "b66665e4-873f-4482-acf1-89307ba9c6e4": {
         "space_name": "smoketest-experimental",
         "task_summaries": [
         {
             "parent_application_guid": "d941b689-4a27-44ec-91d3-1f97434dbed9",
             "parent_application_name": "console-blue",
             "memory_in_mb_per_instance": 256,
             "task_count_for_range": 14,
             "concurrent_task_count_for_range": 2
             "total_duration_in_seconds_for_range": 20583
           }
          ]
       }
     }
}

Note: In the /task_usages endpoint, memory_in_mb_per_instance is the memory of the task.

Service Usage

Use curl to make a request to /service_usages on the Usage Service endpoint to show service usage in an org. You must complete the placeholders in start=YYYY-MM-DD&end=YYYY-MM-DD to define a date range.

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/organizations/`cf org YOUR-ORG --guid`/service_usages?start=YYYY-MM-DD&end=YYYY-MM-DD" -k -v -H "authorization: `cf oauth-token`"
  {
     "organization_guid": "8d83362f-587a-4148-806b-4407428887b5",
     "period_start": "2016-06-01T00:00:00Z",
     "period_end": "2016-06-13T23:59:59Z",
     "service_usages": [
        {
          deleted: false
          duration_in_seconds: 1377982.52
          service_guid: "02802293-b769-44cc-807f-eee331ba9b2b"
          service_instance_creation: "2016-01-20T18:48:16.000Z"
          service_instance_deletion: null
          service_instance_guid: "b25b4481-19aa-4504-82c9-f303e7e9ed6e"
          service_instance_name: "something-usage-db"
          service_instance_type: "managed_service_instance"
          service_name: "my-mysql-service"
          service_plan_guid: "70915a70-7311-4f3e-ab0d-4a7dfd3ef2d9"
          service_plan_name: "20gb"
          space_guid: "e6445eb3-fdac-4049-bafc-94d1703d5e78"
          space_name: "outer-space"
        }
    ]
 }

Monitor Usage Service

Pivotal recommends monitoring usage service at regular intervals with a recurring CI task or 3rd party monitoring tool. Usage service runs asynchronous jobs such as fetching usage events from the Cloud Controller API.

Use the following curl command to check for failed jobs:

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/heartbeat/failed_jobs" -k -v
Ok.

By default, the error_threshold is set to 1 and the command does the following:

  • Succeeds with HTTP status 200 if the number of failed jobs is lower than the threshold
  • Fails with HTTP status 500 if it exceeds the threshold

The following example specifies a threshold value:

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/heartbeat/failed_jobs?error_threshold=5" -k -v
Ok.

Example: Autogenerated Database Usage Reports

For security compliance and recordkeeping, some PCF customers create Concourse CI/CD pipelines that regularly run JavaScript scripts to do the following:

  1. Call the app-usage APIs in the above sections of this topic, as well as the Security Event Logging topic, to collect data about service instances from a PCF foundation. This data includes database instance information such as tile or database type, database creator, and date of creation or deletion.

  2. Merge and format the data about services instances into a JSON output file and save it to an S3 bucket.

  3. Input the file to another process that does the following:

    1. Parses the data
    2. Eliminates duplicate entries from previous reports
    3. Sends the data to a database compliance system through APIs