Troubleshooting Anti-Virus

Note: Pivotal Platform is now part of VMware Tanzu. In v2.2 and later, Pivotal Anti-Virus is named Anti-Virus for VMware Tanzu.

Page last updated:

This topic provides instructions for troubleshooting Anti-Virus for VMware Tanzu and verifying that it is protecting your Ops Manager deployment.

Installation Issues

Ops Manager Fails to Apply Changes

Symptom

Applying changes in Ops Manager fails. The bottom of the changelog contains an error message similar to:

...
Started updating job nats > nats/0 (12bfae02-b4af-4104-b2bd-227ff07b2d92) (canary). Done (00:02:31)
  Failed updating job etcd_server > etcd_server/0 (f8e492bf-db09-4d38-8a73-5cf69d7b8a11) (canary): 'etcd_server/0 (f8e492bf-db09-4d38-8a73-5cf69d7b8a11)' is not running after update. Review logs for failed jobs: clamd (00:05:53)

Error 400007: 'etcd_server/0 (f8e492bf-db09-4d38-8a73-5cf69d7b8a11)' is not running after update. Review logs for failed jobs: clamd

Explanation

The Anti-Virus Mirror for VMware Tanzu server was unavailable during initial deployment.

Solution

Review the manifest file, and replace the database_mirror key with the address of a stable mirror server. If you do not have a stable mirror server for reliable initial deployment, use the S3-based mirror: pivotal-anti-virus-mirror.s3.amazonaws.com


Virus Database Update Issues

Invalid Database Definitions

Symptom

Updating virus definitions writes an error like the following to the Anti-Virus Mirror log destination:

 2019/07/03 20:28:30 file /var/vcap/data/antivirus-mirror/unvalidated/main.cvd rejected: /var/vcap/data/antivirus-mirror/unvalidated/main.cvd is an invalid cvd file: exit status 1

Explanation

The Anti-Virus Mirror database verifier detected that a virus database file downloaded from the external database is invalid.

Solution

Check that the database files downloaded properly and re-download if necessary.

Old Database Definitions

Symptom

Updating virus definitions writes an error like the following to the Anti-Virus Mirror log destination:

   2019/07/03 20:35:34 file /var/vcap/data/antivirus-mirror/unvalidated/daily.cvd rejected: /var/vcap/data/antivirus-mirror/unvalidated/daily.cvd is not newer than /var/vcap/store/antivirus-mirror/validated/daily.cvd

Explanation

The Anti-Virus Mirror database verifier detected that a virus database file downloaded from the external database is older than the one most recently processed by the internal mirror.

Solution

Check that the latest version of the database files were downloaded. If the internal Anti-Virus Mirror has the latest files, no action is required.

Runtime Issues

Anti-Virus Is Not Detecting Malware

Symptom

Malware signature or sample malware is not detected, even though the ClamAV daemon is properly configured.

Explanation

Virus signatures are not up-to-date.

Solution

To resolve this issue, verify that:

  • The mirror server is correctly configured.
  • The mirror server is available within the private subnet.
  • At least one hour has elapsed. One hour is the default scan schedule interval.

If the local mirror is up-to-date and Anti-Virus is still failing to detect a malware sample, you might have encountered a new threat. VMware recommends alerting the community using existing channels and reporting the suspicious file directly to the ClamAV team.

Note: VMware does not provide support for ClamAV detection failures, mirror coordination, or threat tracking activity.


Anti-Virus Reports False Positives

Symptom

Anti-Virus reports a false positive result such as non-malicious file is reported to be a virus.

Explanation

Anti-Virus compares files to its database of known malicious patterns. Anti-Virus might detect a non-malicious file as a virus due to a coincidental similarity to those patterns.

Solution

Submit false positive reports to ClamAV. You can also subscribe to the ClamAV email list to be kept up-to-date with ClamAV database changes. It takes about a week for ClamAV to verify and publish a new database.


CPU Spikes While Using Anti-Virus

Symptom

Anti-Virus is taking more CPU resources than assigned in its configuration.

Explanation

Anti-Virus resource consumption is restricted using cgroups. Anti-Virus is resource-limited whenever other processes are active. However, cgroups enables Anti-Virus to occupy more CPU resources when all other processes are idle, because it does not impact their performance.

Solution

Set the Enforce CPU limit field to Always in the Anti-Virus tile. For instructions, see Configure Anti-Virus.


Out of Memory While Using Anti-Virus

Symptom

Anti-Virus fails to start and /var/log/syslog reports Memory cgroup out of memory: Kill process on the clamd process similar to:

2019-02-20T19:35:40.249205+00:00 localhost kernel: [  254.669948] Memory cgroup out of memory: Kill process 7493 (clamd) score 586 or sacrifice child
2019-02-20T19:35:40.249205+00:00 localhost kernel: [  254.679053] Killed process 7527 (clamd) total-vm:786136kB, anon-rss:626692kB, file-rss:1592kB

Explanation

Anti-Virus resource consumption is restricted by cgroups. The clamd process is terminated if the memory usage limit is exceeded. When memory swapping is disabled by other BOSH jobs, the Anti-Virus resource requires a larger memory limit.

Solution

This is expected behavior from cgroups. To configure the memory limit, configure Memory limit (in bytes) in the Anti-Virus tile.

Warning: When updating the memory limit, ensure that all VMs, including errand VMs, have sufficient memory resources.


Insufficient CPU Limit While Using Anti-Virus

Symptom

Anti-Virus fails to start during deployment. However, the clamd and freshclam processes eventually run.

The deployment failure log looks similar to:

Task 1071 | 19:40:49 | Updating instance clamav_1: clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c (0) (canary) (00:05:26)
                L Error: 'clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c (0)' is not running after update. Review logs for failed jobs: clamd, freshclam


When you run bosh -d DEPLOYMENT instances --ps, you see that the the clamd and freshclam processes are running successfully after the failed deployment.

For example:

$ bosh -d clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c instances --ps
Instance                                       Process    Process State  AZ  IPs
clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c  -          running        z1  10.0.0.7
~                                              clamd      running        -   -
~                                              freshclam  running        -   -

Explanation

Anti-Virus startup is CPU intensive and, if restricted, can prevent Anti-Virus from starting up correctly.

Solutions

  • Ensure cpu_limit is set high enough for Anti-Virus to execute normally. If the limit is too strict, Anti-Virus fails to start. To make changes to this limit, configure CPU limit (percentage) in the Anti-Virus tile.

  • Set enforce_cpu_limit to false. This allocates more CPU cycles to ClamAV if other processes are not using CPU resources.
    To disable this limit, set the Enforce CPU limit field to When other processes are using CPU resources in the Anti-Virus tile.

  • From the Ops Manager Installation Dashboard, navigate to the tile with the failing antivirus job. On Resource Config, adjust the VM Type for the Anti-Virus job to have sufficient CPU resources.

Too Many Open Files Error While Using Anti-Virus Mirror

Symptom

The Anti-Virus Mirror log reports that too many log files are open:

2019/07/29 20:02:41 10.0.0.72 is requesting main.cvd
2019/07/29 20:02:41 http: Accept error: accept tcp 0.0.0.0:80: accept4: too many open files; retrying in 10ms
2019/07/29 20:02:41 http: Accept error: accept tcp 0.0.0.0:80: accept4: too many open files; retrying in 20ms
2019/07/29 20:02:41 http: Accept error: accept tcp 0.0.0.0:80: accept4: too many open files; retrying in 40ms
2019/07/29 20:02:42 http: Accept error: accept tcp 0.0.0.0:80: accept4: too many open files; retrying in 5ms

Explanation

Anti-Virus Mirror opens files when a database is requested. There is a limit to how many files it can open at a time.

Solution

Increase the number of Anti-Virus Mirror instances. VMware recommends that there is one Anti-Virus Mirror for every 250 instances where Anti-Virus is installed. For more information, see Scale the Number of Deployed Mirrors.

Restoring with BOSH Backup and Restore Fails

Symptom

When using Anti-Virus Mirror, errors occur when you redeploy VMware Tanzu Application Service for VMs while restoring with BOSH Backup and Restore (BBR). For information about redeploying VMware Tanzu Application Service for VMs (TAS for VMs), see Step 12: Redeploy TAS for VMs.

Explanation

Anti-Virus Mirror must be running before you install Anti-Virus on other VMs in your deployment. Otherwise, Anti-Virus Mirror might not deploy before other tiles and dependencies deploy.

If Anti-Virus Mirror is not running, VMs with Anti-Virus installed cannot download the required database signature files. If this happens, errors and failed deployments occur.

Solution

To resolve this issue, you must ensure that Anti-Virus Mirror is deployed before restoring your deployment.

To do this:

  1. Follow the procedures before Step 12: Redeploy TAS for VMs in Restoring Deployments from Backup with BBR. Do not apply changes.

  2. Exclude Anti-Virus Mirror from the Anti-Virus deployment by following the procedure in Exclude Anti-Virus Mirror during Apply Changes. This ensures that Anti-Virus is not deployed on the Anti-Virus Mirror

  3. Remove the Anti-Virus Mirror exclusion from the Anti-Virus configurations by following the procedure in Remove the Exclusion.

  4. Continue to restore your deployment by following the remaining procedures in Restoring Deployments from Backup with BBR.

CPU Spikes While Enforcing a CPU Limit in Anti-Virus

Symptom

Anti-Virus is using more CPU resources than assigned in its configuration, even with the Enforce CPU limit field set to Always.

Explanation

Anti-Virus resource consumption is restricted using cgroups. If the VM does not have enough CPU or memory resources, the clamd PID is removed from the cgroup.procs file. This causes Anti-Virus to ignore the Enforce CPU limit setting.

Solution

Increase the VM size. VMware recommends a minimum VM size of micro.cpu using 2 CPU and 2 GB RAM.