Reporting App, Task, and Service Instance Usage

Page last updated:

This topic describes how to use the Cloud Foundry Command Line Interface (cf CLI) with Pivotal Application Service to retrieve historical system- and org-level usage information about your apps, tasks, and service instances through the Cloud Controller and Usage Service APIs.

You can also access usage information by using Apps Manager. For more information, see Reporting Instance Usage with Apps Manager.

To retrieve current app and event information from the Cloud Controller, see Retrieving App and Event Information in the open-source Cloud Foundry documentation.

Note: PAS v2.7.18 fixes an issue where foundations report inaccurate or missing data. Pivotal recommends upgrading to a recent version.

Prerequisite

Before performing the procedures in this topic, ensure that your user is a member of the cloud_controller.admin or usage_service.audit group. For example, a UAAC admin can add a user to the usage_service.audit group with the following command:

uaac member add usage_service.audit USERNAME

Monitor Usage Service

Pivotal strongly recommends monitoring Usage Service at regular intervals with a recurring CI task or third-party monitoring tool.

One important reason to monitor Usage Service is that reports are generated based on app and service events fetched from the Cloud Controller. Events are dumped from the Cloud Controller database after 31 days. This means that if the Usage Service fails to fetch events for a period greater than 31 days, events can be permanently lost, and Usage Service cannot construct a complete history of usage on the foundation. Event fetching could fail for a number of reasons, such as platform downtime and revoked credentials. Monitoring can help to catch these errors early and prevent incorrect data from propagating.

Use the following endpoints to monitor Usage Service:

https://app-usage.YOUR-SYSTEM-DOMAIN/heartbeat/failed_jobs?error_threshold=1

Usage Service continuously processes events and computes usage in the background using asynchronous jobs. This endpoint fails if the number of failed jobs exceeds a threshold.

Accepts a query parameter error_threshold. Set to 1 by default.

• Succeeds with HTTP status 200 if the number of failed jobs is less than the error_threshold
• Fails with HTTP status 500 if the number of failed jobs is greater than or equal to the error_threshold

Recommended error_threshold: 1

The current number of failed jobs is also emitted as a metric: usage_service.delayed_job_failures.

https://app-usage.YOUR-SYSTEM-DOMAIN/heartbeat/db

This endpoint checks if Usage Service is able to access its database in the internal Pivotal Application Service (PAS) cluster.

• Succeeds with HTTP status 200 if Usage Service successfully reads from its database
• Fails with HTTP status 500 if Usage Service fails to read from its database

https://app-usage.YOUR-SYSTEM-DOMAIN/heartbeat/workers

This endpoint confirms that Usage Service workers are operating normally, and that background jobs are being scheduled.

• Succeeds with HTTP status 200 if any Usage Service worker has checked in within the last 15 minutes
• Fails with HTTP status 500 if no Usage Service worker has ever checked in or if no Usage Service worker has checked in within the last 15 minutes

https://app-usage.YOUR-SYSTEM-DOMAIN/heartbeat/server_status

This endpoint confirms that the Usage Service server is operating normally.

• Succeeds with HTTP status 200 if the Usage Service server is operating normally
• Fails with HTTP status 500 if the Usage Service server is failing.

Obtain System Usage Information

You can obtain the following system-wide usage information:

• App Usage
• Task Usage
• Service Usage
• All Orgs App Usage

App Usage

Use curl to make a request to /system_report/app_usages on the Usage Service endpoint to show system-wide app usage data:

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/system_report/app_usages" -k -v \
-H "authorization: `cf oauth-token`"
{
"report_time": "2017-04-11 22:28:24 UTC",
"monthly_reports": [
    {
      "month": 4,
      "year": 2017,
      "average_app_instances": 17855.256153713308,
      "maximum_app_instances": 18145,
      "app_instance_hours": 4686533.080277303
    }
  ],
"yearly_reports": [
    {
      "year": 2017,
      "average_app_instances": 16184.9,
      "maximum_app_instances": 18145,
      "app_instance_hours": 39207433.0802773
    }
  ]
}

The following table describes the data generated by this report:

Task Usage

Use curl to make a request to /system_report/task_usages on the Usage Service endpoint to show system-wide task usage data:

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/system_report/task_usages" -k -v \
-H "authorization: `cf oauth-token`"
{
"report_time": "2017-04-11T22:33:48.971Z",
"monthly_reports": [
    {
      "month": 4,
      "year": 2017,
      "total_task_runs": 235,
      "maximum_concurrent_tasks": 7,
      "task_hours": 43045.201944444445
    }
  ],
"yearly_reports": [
    {
      "year": 2017,
      "total_task_runs": 2894,
      "maximum_concurrent_tasks": 184,
      "task_hours": 45361.26694444445
    }
  ]
}

Service Usage

Use curl to make a request to /system_report/service_usages on the Usage Service endpoint to show system-wide service usage data:

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/system_report/service_usages" -k -v \
-H "authorization: `cf oauth-token`"
{
  "report_time": "2017-05-11 18:29:14 UTC",
  "monthly_service_reports": [
    {
      "service_name": "fake-service-0507f1fd-2340-49a6-9d43-a347a5f5f6be",
      "service_guid": "177dcfde-cd51-4058-bd86-b98015c295f5",
      "usages": [
        {
          "month": 1,
          "year": 2017,
          "duration_in_hours": 0,
          "average_instances": 0,
          "maximum_instances": 0
        },
        {
          "month": 2,
          "year": 2017,
          "duration_in_hours": 0,
          "average_instances": 0,
          "maximum_instances": 0
        },
        {
          "month": 3,
          "year": 2017,
          "duration_in_hours": 4.182222222222227,
          "average_instances": 0,
          "maximum_instances": 2
        },
        {
          "month": 4,
          "year": 2017,
          "duration_in_hours": 2176.962222222186,
          "average_instances": 3,
          "maximum_instances": 7
        },
        {
          "month": 5,
          "year": 2017,
          "duration_in_hours": 385.61388888888854,
          "average_instances": 1.5,
          "maximum_instances": 3
        }
      ],
      "plans": [
        {
          "usages": [
            {
              "month": 1,
              "year": 2017,
              "duration_in_hours": 0,
              "average_instances": 0,
              "maximum_instances": 0
            },
            {
              "month": 2,
              "year": 2017,
              "duration_in_hours": 0,
              "average_instances": 0,
              "maximum_instances": 0
            },
            {
              "month": 3,
              "year": 2017,
              "duration_in_hours": 4.182222222222227,
              "average_instances": 0,
              "maximum_instances": 2
            },
            {
              "month": 4,
              "year": 2017,
              "duration_in_hours": 1465.6388888888941,
              "average_instances": 2,
              "maximum_instances": 5
            },
            {
              "month": 5,
              "year": 2017,
              "duration_in_hours": 385.61388888888854,
              "average_instances": 1.5,
              "maximum_instances": 3
            }
          ],
          "service_plan_name": "fake-plan",
          "service_plan_guid": "ac09f607-f4e5-4807-af16-e95856061bd7"
        }
      ]
    }
  ]
}

All Orgs Usage (Beta)

The app_usages endpoint directly under your system domain returns system-wide app usage for all orgs that you have access to. The command works like the org-specific App Usage command, except you do not supply an org GUID. You can set order_by to either total_instance_hours or megabyte_hours.

The order_by ordering defaults to ascending, which lists orgs with the most usage at the end of the output. To change the sort ordering to descending, prepend the order_by argument with a -, as in order_by=-megabyte_hours.

Use curl to make a request to /app_usages on the Usage Service endpoint to show system-wide organization usage data:

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/app_usages?start=2018-010-01&end=2018-11-01&order_by=total_instance_hours" -k -v \
-H "authorization: `cf oauth-token`"
{
   "period_start":"2018-01-10T00:00:00Z",
   "period_end":"2018-11-01T23:59:59Z",
   "Organizations":[ 
      {
         "guid":"d5c9f947-df85-45af-abab-00a2174598f9",
         "total_instance_hours":"91535.6",
         "megabyte_hours":"1830712.2"
      },
      {
         "guid":"a9605fa4-99da-4932-84f5-3db8698c1fc1",
         "total_instance_hours":"99440.8",
         "megabyte_hours":"96698582.2"
      }
   ]
}

This beta API endpoint may change.

Obtain Org Usage Information

You can obtain the following org-specific usage information:

• App Usage
• Task Usage
• Service Usage

You must have the GUID of the org you want to obtain information about in order to perform the procedures in this section. To retrieve your org GUID, run the cf org command:

$ cf org YOUR-ORG --guid

App Usage

Use curl to make a request to /app_usages on the Usage Service endpoint to show app usage in an org. You must complete the placeholders in start=YYYY-MM-DD&end=YYYY-MM-DD to define a date range.

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/organizations/YOUR-ORG-GUID/app_usages?start=YYYY-MM-DD&end=YYYY-MM-DD" \
-k -v \
-H "authorization: `cf oauth-token`"
   
{
  "organization_guid": "88e485fa-2222-4993-8ef5-bac3711adfee",
  "period_start": "2019-01-16T00:00:00Z",
  "period_end": "2019-01-17T23:59:59Z",
  "app_usages": [
    {
      "space_guid": "d6988d4d-6175-4dee-b219-a6eabbc05514",
      "space_name": "development",
      "app_name": "static2",
      "app_guid": "1afc2b20-dd41-4ba4-8e6e-c894b5a2cb23",
      "parent_application_name": "static2",
      "parent_application_guid": "1afc2b20-dd41-4ba4-8e6e-c894b5a2cb23",
      "instance_count": 1,
      "memory_in_mb_per_instance": 1024,
      "duration_in_seconds": 172800
    },
    {
      "space_guid": "d6988d4d-6175-4dee-b219-a6eabbc05514",
      "space_name": "development",
      "app_name": "static",
      "app_guid": "54e767ff-4f83-45b1-bf06-46c501217845",
      "parent_application_name": "static",
      "parent_application_guid": "54e767ff-4f83-45b1-bf06-46c501217845",
      "instance_count": 1,
      "memory_in_mb_per_instance": 1024,
      "duration_in_seconds": 172800
    }
  ]
}

You may see results with "instance_count": 0. This happens when a v3 app is pushed but has processes that are not scaled up. The memory configuration and duration are still stored and can be useful for resource management. Apps with ‘0’ instances are not counted in the App Usages System Report above.

Task Usage

Use curl to make a request to /task_usages on the Usage Service endpoint to show task usage in an org. You must complete the placeholders in start=YYYY-MM-DD&end=YYYY-MM-DD to define a date range.

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/organizations/YOUR-ORG-GUID/task_usages?start=YYYY-MM-DD&end=YYYY-MM-DD" \
-k -v \
-H "authorization: `cf oauth-token`"
{
     "organization_guid": "8f88362f-547c-4158-808b-4605468387d5",
     "period_start": "2014-01-01",
     "period_end": "2017-04-04",
     "spaces": {
       "e6445eb3-fdac-4049-bafc-94d1703d5e78": {
         "space_name": "smoketest",
         "task_summaries": [
           {
             "parent_application_guid": "04cd29d5-1f9e-4900-ac13-2e903f6582a9",
             "parent_application_name": "task-dummy-app",
             "memory_in_mb_per_instance": 256,
             "task_count_for_range": 54084,
             "concurrent_task_count_for_range": 5,
             "total_duration_in_seconds_for_range": 37651415
           }
          ]
       },
       "b66665e4-873f-4482-acf1-89307ba9c6e4": {
         "space_name": "smoketest-experimental",
         "task_summaries": [
         {
             "parent_application_guid": "d941b689-4a27-44ec-91d3-1f97434dbed9",
             "parent_application_name": "console-blue",
             "memory_in_mb_per_instance": 256,
             "task_count_for_range": 14,
             "concurrent_task_count_for_range": 2,
             "total_duration_in_seconds_for_range": 20583
           }
          ]
       }
     }
}

Note: In the /task_usages endpoint, memory_in_mb_per_instance is the memory of the task.

Service Usage

Use curl to make a request to /service_usages on the Usage Service endpoint to show service usage in an org. You must complete the placeholders in start=YYYY-MM-DD&end=YYYY-MM-DD to define a date range.

$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/organizations/`cf org YOUR-ORG --guid`/service_usages?start=YYYY-MM-DD&end=YYYY-MM-DD" -k -v -H "authorization: `cf oauth-token`"
{
     "organization_guid": "8d83362f-587a-4148-806b-4407428887b5",
     "period_start": "2016-06-01T00:00:00Z",
     "period_end": "2016-06-13T23:59:59Z",
     "service_usages": [
        {
          "deleted": false,
          "duration_in_seconds": 1377982.52,
          "service_guid": "02802293-b769-44cc-807f-eee331ba9b2b",
          "service_instance_creation": "2016-01-20T18:48:16.000Z",
          "service_instance_deletion": null,
          "service_instance_guid": "b25b4481-19aa-4504-82c9-f303e7e9ed6e",
          "service_instance_name": "something-usage-db",
          "service_instance_type": "managed_service_instance",
          "service_name": "my-mysql-service",
          "service_plan_guid": "70915a70-7311-4f3e-ab0d-4a7dfd3ef2d9",
          "service_plan_name": "20gb",
          "space_guid": "e6445eb3-fdac-4049-bafc-94d1703d5e78",
          "space_name": "outer-space"
        }
    ]
}

Usage Availability

The app, task, and service instance usage data returned from the Usage Service is delayed to ensure accuracy. The /usage_availability endpoint can be used to retrieve the date which the Usage Service data is accurate to. The Usage Service is able to provide data for time periods up to and including this date.

Use the following curl command to retrieve the current usage availability timestamp:

curl "https://app-usage.YOUR-SYSTEM-DOMAIN/usage_availability" -k -v
{"date":"2019-01-01"}

Where YOUR-SYSTEM-DOMAIN is the system domain of your PAS environment.

Usage Data Retention

Usage Service returns data indefinitely for the following endpoints:

• /system_report/app_usages
• /organizations/ORG-GUID/app_usages
• /app_usages
• /system_report/task_usages
• /organizations/ORG-GUID/task_usages

Usage Service returns summarized data for service instances indefinitely unless they are deleted, in which case they are only retained for 93 days. This affects the following endpoints:

• /system_report/service_usages
• /organizations/ORG-GUID/service_usages

Example: Autogenerated Database Usage Reports

For security compliance and record keeping, some Pivotal Platform customers create Concourse CI/CD pipelines that regularly run JavaScript scripts to do the following:

1. Call the app-usage APIs in the above sections of this topic, as well as the Security Event Logging topic, to collect data about service instances from a Pivotal Platform foundation. This data includes database instance information such as tile or database type, database creator, and date of creation or deletion.

2. Merge and format the data about services instances into a JSON output file and save it to an S3 bucket.

3. Input the file to another process that does the following:

   1. Parses the data
   2. Eliminates duplicate entries from previous reports
   3. Sends the data to a database compliance system through APIs