Reporting App, Task, and Service Instance Usage
Page last updated:
This topic describes how to use the Cloud Foundry Command Line Interface (cf CLI) with Pivotal Application Service to retrieve historical system- and org-level usage information about your apps, tasks, and service instances through the Cloud Controller and Usage Service APIs.
You can also access usage information by using Apps Manager. For more information, see Reporting Instance Usage with Apps Manager.
To retrieve current app and event information from the Cloud Controller, see Retrieving App and Event Information in the open-source Cloud Foundry documentation.
Note: PAS v2.7.18 fixes an issue where foundations report inaccurate or missing data. Pivotal recommends upgrading to a recent version.
Prerequisite
Before performing the procedures in this topic, ensure that your user is a member of the cloud_controller.admin
or usage_service.audit
group. For example, a UAAC admin can add a user to the usage_service.audit
group with the following command:
uaac member add usage_service.audit USERNAME
Monitor Usage Service
Pivotal strongly recommends monitoring Usage Service at regular intervals with a recurring CI task or third-party monitoring tool.
One important reason to monitor Usage Service is that reports are generated based on app and service events fetched from the Cloud Controller. Events are dumped from the Cloud Controller database after 31 days. This means that if the Usage Service fails to fetch events for a period greater than 31 days, events can be permanently lost, and Usage Service cannot construct a complete history of usage on the foundation. Event fetching could fail for a number of reasons, such as platform downtime and revoked credentials. Monitoring can help to catch these errors early and prevent incorrect data from propagating.
Use the following endpoints to monitor Usage Service:
https://app-usage.YOUR-SYSTEM-DOMAIN/heartbeat/failed_jobs?error_threshold=1
Usage Service continuously processes events and computes usage in the background using asynchronous jobs. This endpoint fails if the number of failed jobs exceeds a threshold.
Accepts a query parameter error_threshold
. Set to 1 by default.
- Succeeds with HTTP status
200
if the number of failed jobs is less than theerror_threshold
- Fails with HTTP status
500
if the number of failed jobs is greater than or equal to theerror_threshold
Recommended error_threshold
: 1
The current number of failed jobs is also emitted as a metric: usage_service.delayed_job_failures
.
https://app-usage.YOUR-SYSTEM-DOMAIN/heartbeat/db
This endpoint checks if Usage Service is able to access its database in the internal Pivotal Application Service (PAS) cluster.
- Succeeds with HTTP status
200
if Usage Service successfully reads from its database - Fails with HTTP status
500
if Usage Service fails to read from its database
https://app-usage.YOUR-SYSTEM-DOMAIN/heartbeat/workers
This endpoint confirms that Usage Service workers are operating normally, and that background jobs are being scheduled.
- Succeeds with HTTP status
200
if any Usage Service worker has checked in within the last 15 minutes - Fails with HTTP status
500
if no Usage Service worker has ever checked in or if no Usage Service worker has checked in within the last 15 minutes
https://app-usage.YOUR-SYSTEM-DOMAIN/heartbeat/server_status
This endpoint confirms that the Usage Service server is operating normally.
- Succeeds with HTTP status
200
if the Usage Service server is operating normally - Fails with HTTP status
500
if the Usage Service server is failing.
Obtain System Usage Information
You can obtain the following system-wide usage information:
App Usage
Use curl
to make a request to /system_report/app_usages
on the Usage Service endpoint to show system-wide app usage data:
$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/system_report/app_usages" -k -v \ -H "authorization: `cf oauth-token`" { "report_time": "2017-04-11 22:28:24 UTC", "monthly_reports": [ { "month": 4, "year": 2017, "average_app_instances": 17855.256153713308, "maximum_app_instances": 18145, "app_instance_hours": 4686533.080277303 } ], "yearly_reports": [ { "year": 2017, "average_app_instances": 16184.9, "maximum_app_instances": 18145, "app_instance_hours": 39207433.0802773 } ] }
The following table describes the data generated by this report:
Value | Description | Calculation Method |
---|---|---|
app_instance_hours |
The total number of hours app instances have been running on the foundation for a specific month or year. | App start and stop events are used to create a rolling record of each single app’s runtime per month. Monthly records are then aggregated to calculate total app hours for a year. |
average_app_instances |
The average number of app instances simultaneously running at any given time during the specified period. |
app_instance_hours divided by the number of hours for a month or year. Example: For the month of January (744 hours), 100 apps run for 300 hours each, generating 30,000 app_instance_hours . Dividing by 744 hours returns a value of 40.3 for average_app_instances . |
maximum_app_instances |
The highest concurrent number of app instances running on the foundation for the month or year. | Each time an app is started or stopped, a new record of the current number of running apps is created. The maximum concurrent value then comes from the record with the maximum value over the desired time range. |
Task Usage
Use curl
to make a request to /system_report/task_usages
on the Usage Service endpoint to show system-wide task usage data:
$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/system_report/task_usages" -k -v \ -H "authorization: `cf oauth-token`" { "report_time": "2017-04-11T22:33:48.971Z", "monthly_reports": [ { "month": 4, "year": 2017, "total_task_runs": 235, "maximum_concurrent_tasks": 7, "task_hours": 43045.201944444445 } ], "yearly_reports": [ { "year": 2017, "total_task_runs": 2894, "maximum_concurrent_tasks": 184, "task_hours": 45361.26694444445 } ] }
Value | Description | Calculation Method |
---|---|---|
total_task_runs |
The total number of tasks that have run on a foundation for the specified month or year. | Task events are summarized into monthly and yearly reports. |
maximum_concurrent_tasks |
The maximum number of tasks simultaneously running at any given time during the specified period. | Each time a task is started or stopped, a new record of the current number of running tasks is created. The maximum concurrent value then comes from the record with the maximum value over the desired time range. |
task_hours |
The total number of hours task instances have been running on the foundation for a specific month or year. | Task start and stop events are processed to create daily task summaries. Daily task summaries are then summed to compute the total duration in hours for the specified period. |
Service Usage
Use curl
to make a request to /system_report/service_usages
on the Usage Service endpoint to show system-wide service usage data:
$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/system_report/service_usages" -k -v \ -H "authorization: `cf oauth-token`" { "report_time": "2017-05-11 18:29:14 UTC", "monthly_service_reports": [ { "service_name": "fake-service-0507f1fd-2340-49a6-9d43-a347a5f5f6be", "service_guid": "177dcfde-cd51-4058-bd86-b98015c295f5", "usages": [ { "month": 1, "year": 2017, "duration_in_hours": 0, "average_instances": 0, "maximum_instances": 0 }, { "month": 2, "year": 2017, "duration_in_hours": 0, "average_instances": 0, "maximum_instances": 0 }, { "month": 3, "year": 2017, "duration_in_hours": 4.182222222222227, "average_instances": 0, "maximum_instances": 2 }, { "month": 4, "year": 2017, "duration_in_hours": 2176.962222222186, "average_instances": 3, "maximum_instances": 7 }, { "month": 5, "year": 2017, "duration_in_hours": 385.61388888888854, "average_instances": 1.5, "maximum_instances": 3 } ], "plans": [ { "usages": [ { "month": 1, "year": 2017, "duration_in_hours": 0, "average_instances": 0, "maximum_instances": 0 }, { "month": 2, "year": 2017, "duration_in_hours": 0, "average_instances": 0, "maximum_instances": 0 }, { "month": 3, "year": 2017, "duration_in_hours": 4.182222222222227, "average_instances": 0, "maximum_instances": 2 }, { "month": 4, "year": 2017, "duration_in_hours": 1465.6388888888941, "average_instances": 2, "maximum_instances": 5 }, { "month": 5, "year": 2017, "duration_in_hours": 385.61388888888854, "average_instances": 1.5, "maximum_instances": 3 } ], "service_plan_name": "fake-plan", "service_plan_guid": "ac09f607-f4e5-4807-af16-e95856061bd7" } ] } ] }
Value | Description | Calculation Method |
---|---|---|
usages.duration_in_hours |
The duration in hours service instances ran within a month and year. | Service instance start and stop events are processed into summary records, grouped by month and year, and summed. |
usages.average_instances |
The average number of service instances ran within a month and year. | duration_in_hours divided by the number of hours for a month or year. |
usages.maximum_instances |
The maximum number of simultaneously running service instances within a month and year. | Usage Service walks through all service instance records for a month and year. |
plans.usages.duration_in_hours |
The duration in hours service instances of a plan type ran within a month and year. | usages.duration_in_hours filtered by plan type. |
plans.usages.average_instances |
The average number of service instances of a plan type that ran within a month and year. | usages.average_instances filtered by plan type. |
plans.usages.maximum_instances |
The maximum number of simultaneously running service instances of a plan type within a month and year | usages.maximum_instances filtered by plan type. |
All Orgs Usage (Beta)
The app_usages
endpoint directly under your system domain returns system-wide app usage for all orgs that you have access to. The command works like the org-specific App Usage command, except you do not supply an org GUID.
You can set order_by
to either total_instance_hours
or megabyte_hours
.
The order_by
ordering defaults to ascending, which lists orgs with the most usage at the end of the output. To change the sort ordering to descending, prepend the order_by
argument with a -
, as in order_by=-megabyte_hours
.
Use curl
to make a request to /app_usages
on the Usage Service endpoint to show system-wide organization usage data:
$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/app_usages?start=2018-010-01&end=2018-11-01&order_by=total_instance_hours" -k -v \ -H "authorization: `cf oauth-token`" { "period_start":"2018-01-10T00:00:00Z", "period_end":"2018-11-01T23:59:59Z", "Organizations":[ { "guid":"d5c9f947-df85-45af-abab-00a2174598f9", "total_instance_hours":"91535.6", "megabyte_hours":"1830712.2" }, { "guid":"a9605fa4-99da-4932-84f5-3db8698c1fc1", "total_instance_hours":"99440.8", "megabyte_hours":"96698582.2" } ] }
This beta API endpoint may change.
Obtain Org Usage Information
You can obtain the following org-specific usage information:
You must have the GUID of the org you want to obtain information about in order to perform the procedures in this section. To retrieve your org GUID, run the cf org
command:
$ cf org YOUR-ORG --guid
App Usage
Use curl
to make a request to /app_usages
on the Usage Service endpoint to show app usage in an org. You must complete the placeholders in start=YYYY-MM-DD&end=YYYY-MM-DD
to define a date range.
$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/organizations/YOUR-ORG-GUID/app_usages?start=YYYY-MM-DD&end=YYYY-MM-DD" \ -k -v \ -H "authorization: `cf oauth-token`" { "organization_guid": "88e485fa-2222-4993-8ef5-bac3711adfee", "period_start": "2019-01-16T00:00:00Z", "period_end": "2019-01-17T23:59:59Z", "app_usages": [ { "space_guid": "d6988d4d-6175-4dee-b219-a6eabbc05514", "space_name": "development", "app_name": "static2", "app_guid": "1afc2b20-dd41-4ba4-8e6e-c894b5a2cb23", "parent_application_name": "static2", "parent_application_guid": "1afc2b20-dd41-4ba4-8e6e-c894b5a2cb23", "instance_count": 1, "memory_in_mb_per_instance": 1024, "duration_in_seconds": 172800 }, { "space_guid": "d6988d4d-6175-4dee-b219-a6eabbc05514", "space_name": "development", "app_name": "static", "app_guid": "54e767ff-4f83-45b1-bf06-46c501217845", "parent_application_name": "static", "parent_application_guid": "54e767ff-4f83-45b1-bf06-46c501217845", "instance_count": 1, "memory_in_mb_per_instance": 1024, "duration_in_seconds": 172800 } ] }
You may see results with "instance_count": 0
. This happens when a v3 app is pushed but has processes that are not scaled up. The memory configuration and duration are still stored and can be useful for resource management. Apps with ‘0’ instances are not counted in the App Usages System Report above.
Task Usage
Use curl
to make a request to /task_usages
on the Usage Service endpoint to show task usage in an org. You must complete the placeholders in start=YYYY-MM-DD&end=YYYY-MM-DD
to define a date range.
$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/organizations/YOUR-ORG-GUID/task_usages?start=YYYY-MM-DD&end=YYYY-MM-DD" \ -k -v \ -H "authorization: `cf oauth-token`" { "organization_guid": "8f88362f-547c-4158-808b-4605468387d5", "period_start": "2014-01-01", "period_end": "2017-04-04", "spaces": { "e6445eb3-fdac-4049-bafc-94d1703d5e78": { "space_name": "smoketest", "task_summaries": [ { "parent_application_guid": "04cd29d5-1f9e-4900-ac13-2e903f6582a9", "parent_application_name": "task-dummy-app", "memory_in_mb_per_instance": 256, "task_count_for_range": 54084, "concurrent_task_count_for_range": 5, "total_duration_in_seconds_for_range": 37651415 } ] }, "b66665e4-873f-4482-acf1-89307ba9c6e4": { "space_name": "smoketest-experimental", "task_summaries": [ { "parent_application_guid": "d941b689-4a27-44ec-91d3-1f97434dbed9", "parent_application_name": "console-blue", "memory_in_mb_per_instance": 256, "task_count_for_range": 14, "concurrent_task_count_for_range": 2, "total_duration_in_seconds_for_range": 20583 } ] } } }
Note: In the /task_usages
endpoint, memory_in_mb_per_instance
is the memory of the task.
Service Usage
Use curl
to make a request to /service_usages
on the Usage Service endpoint to show service usage in an org. You must complete the placeholders in start=YYYY-MM-DD&end=YYYY-MM-DD
to define a date range.
$ curl "https://app-usage.YOUR-SYSTEM-DOMAIN/organizations/`cf org YOUR-ORG --guid`/service_usages?start=YYYY-MM-DD&end=YYYY-MM-DD" -k -v -H "authorization: `cf oauth-token`" { "organization_guid": "8d83362f-587a-4148-806b-4407428887b5", "period_start": "2016-06-01T00:00:00Z", "period_end": "2016-06-13T23:59:59Z", "service_usages": [ { "deleted": false, "duration_in_seconds": 1377982.52, "service_guid": "02802293-b769-44cc-807f-eee331ba9b2b", "service_instance_creation": "2016-01-20T18:48:16.000Z", "service_instance_deletion": null, "service_instance_guid": "b25b4481-19aa-4504-82c9-f303e7e9ed6e", "service_instance_name": "something-usage-db", "service_instance_type": "managed_service_instance", "service_name": "my-mysql-service", "service_plan_guid": "70915a70-7311-4f3e-ab0d-4a7dfd3ef2d9", "service_plan_name": "20gb", "space_guid": "e6445eb3-fdac-4049-bafc-94d1703d5e78", "space_name": "outer-space" } ] }
Usage Availability
The app, task, and service instance usage data returned from the Usage Service is delayed to ensure accuracy. The /usage_availability
endpoint can be used to retrieve the date which the Usage Service data is accurate to. The Usage Service is able to provide data for time periods up to and including this date.
Use the following curl command to retrieve the current usage availability timestamp:
curl "https://app-usage.YOUR-SYSTEM-DOMAIN/usage_availability" -k -v
{"date":"2019-01-01"}
Where YOUR-SYSTEM-DOMAIN
is the system domain of your PAS environment.
Usage Data Retention
Usage Service returns data indefinitely for the following endpoints:
/system_report/app_usages
/organizations/ORG-GUID/app_usages
/app_usages
/system_report/task_usages
/organizations/ORG-GUID/task_usages
Usage Service returns summarized data for service instances indefinitely unless they are deleted, in which case they are only retained for 93 days. This affects the following endpoints:
/system_report/service_usages
/organizations/ORG-GUID/service_usages
Example: Autogenerated Database Usage Reports
For security compliance and record keeping, some Pivotal Platform customers create Concourse CI/CD pipelines that regularly run JavaScript scripts to do the following:
Call the
app-usage
APIs in the above sections of this topic, as well as the Security Event Logging topic, to collect data about service instances from a Pivotal Platform foundation. This data includes database instance information such as tile or database type, database creator, and date of creation or deletion.Merge and format the data about services instances into a JSON output file and save it to an S3 bucket.
Input the file to another process that does the following:
- Parses the data
- Eliminates duplicate entries from previous reports
- Sends the data to a database compliance system through APIs