Troubleshooting Anti-Virus

Note: Pivotal Platform is now part of VMware Tanzu. In v2.2 and later, Pivotal Anti-Virus is named Anti-Virus for VMware Tanzu.

Page last updated:

This topic provides instructions for troubleshooting Anti-Virus for VMware Tanzu and verifying that it is protecting your Ops Manager deployment.

Installation Issues

Ops Manager etcd_server Not Running after Update

Symptom

Applying changes in Ops Manager fails. The bottom of the changelog contains an error message similar to:

...
Started updating job nats > nats/0 (12bfae02-b4af-4104-b2bd-227ff07b2d92) (canary). Done (00:02:31)
  Failed updating job etcd_server > etcd_server/0 (f8e492bf-db09-4d38-8a73-5cf69d7b8a11) (canary): 'etcd_server/0 (f8e492bf-db09-4d38-8a73-5cf69d7b8a11)' is not running after update. Review logs for failed jobs: clamd (00:05:53)

Error 400007: 'etcd_server/0 (f8e492bf-db09-4d38-8a73-5cf69d7b8a11)' is not running after update. Review logs for failed jobs: clamd

Explanation

The Anti-Virus Mirror for VMware Tanzu server was unavailable during initial deployment.

Solution

Review the manifest file, and replace the database_mirror key with the address of a stable mirror server. The official supported mirror is database.clamav.net.

Ops Manager Antivirus Job Fails To Start

Symptom

Applying changes in Ops Manager fails. The bottom of the changelog contains an error message similar to:

...
Error: Action Failed get_task: Task d5b87522-c8b2-4870-7855-73d50bff0748 result: 1 of 6 pre-start scripts failed. Failed Jobs: antivirus. Successful Jobs: bpm, syslog_forwarder, bosh-dns, ipsec, pxc-mysql.

Explanation

The antivirus job can fail to start because it does not get the virus definitions from the antivirus-mirror. The antivirus-mirror fails to supply the virus definitions if it has failed to correctly obtain the following files: main.cvd, bytecode.cvd, and daily.cvd. If you manually get the ClamAV Virus Database, using curl or similar tools can return a file with an error instead of the virus definitions. For example:

...
$ curl -L -O database.clamav.net/main.cvd

$ cat main.cvd
error code: 1020

Solution

Configure the tile to use either the official mirror or an existing mirror. For information, see Configure Anti-Virus Mirror in Installing and Configuring Anti-Virus Mirror.

For use cases where CVD files are manually obtained, a supported method must be used. For information about error codes and supported methods, see ClamAV documentation.

Virus Database Update Issues

Invalid Database Definitions

Symptom

Updating virus definitions writes an error like the following to the Anti-Virus Mirror log destination:

 2019/07/03 20:28:30 file /var/vcap/data/antivirus-mirror/unvalidated/main.cvd rejected: /var/vcap/data/antivirus-mirror/unvalidated/main.cvd is an invalid cvd file: exit status 1

Explanation

The Anti-Virus Mirror database verifier detected that a virus database file downloaded from the external database is invalid.

Solution

Check that the database files downloaded properly and re-download if necessary.

Old Database Definitions

Symptom

Updating virus definitions writes an error like the following to the Anti-Virus Mirror log destination:

   2019/07/03 20:35:34 file /var/vcap/data/antivirus-mirror/unvalidated/daily.cvd rejected: /var/vcap/data/antivirus-mirror/unvalidated/daily.cvd is not newer than /var/vcap/store/antivirus-mirror/validated/daily.cvd

Explanation

The Anti-Virus Mirror database verifier detected that a virus database file downloaded from the external database is older than the one most recently processed by the internal mirror.

Solution

Check that the latest version of the database files were downloaded. If the internal Anti-Virus Mirror has the latest files, no action is required.

Runtime Issues

Anti-Virus Is Not Detecting Malware

Symptom

Malware signature or sample malware is not detected, even though the ClamAV daemon is properly configured.

Explanation

Virus signatures are not up-to-date.

Solution

To resolve this issue, verify that:

  • The mirror server is correctly configured.
  • The mirror server is available within the private subnet.
  • At least one hour has elapsed. One hour is the default scan schedule interval.

If the local mirror is up-to-date and Anti-Virus is still failing to detect a malware sample, you might have encountered a new threat. VMware recommends alerting the community using existing channels and reporting the suspicious file directly to the ClamAV team.

Note: VMware does not provide support for ClamAV detection failures, mirror coordination, or threat tracking activity.

Anti-Virus Reports False Positives

Symptom

Anti-Virus reports a false positive result such as non-malicious file is reported to be a virus.

Explanation

Anti-Virus compares files to its database of known malicious patterns. Anti-Virus might detect a non-malicious file as a virus due to a coincidental similarity to those patterns.

Solution

Submit false positive reports to ClamAV. You can also subscribe to the ClamAV email list to be kept up-to-date with ClamAV database changes. It takes about a week for ClamAV to verify and publish a new database.

CPU Spikes While Using Anti-Virus

Symptom

Anti-Virus is taking more CPU resources than assigned in its configuration.

Explanation

Anti-Virus resource consumption is restricted using cgroups. Anti-Virus is resource-limited whenever other processes are active. However, cgroups enables Anti-Virus to occupy more CPU resources when all other processes are idle, because it does not impact their performance.

Solution

Set the Enforce CPU limit field to Always in the Anti-Virus tile. For instructions, see Configure Anti-Virus.

Out of Memory While Using Anti-Virus

Symptom

Anti-Virus fails to start and /var/log/syslog reports Memory cgroup out of memory: Kill process on the clamd process similar to:

2019-02-20T19:35:40.249205+00:00 localhost kernel: [  254.669948] Memory cgroup out of memory: Kill process 7493 (clamd) score 586 or sacrifice child
2019-02-20T19:35:40.249205+00:00 localhost kernel: [  254.679053] Killed process 7527 (clamd) total-vm:786136kB, anon-rss:626692kB, file-rss:1592kB

Explanation

Anti-Virus resource consumption is restricted by cgroups. The clamd process is terminated if the memory usage limit is exceeded. When memory swapping is disabled by other BOSH jobs, the Anti-Virus resource requires a larger memory limit.

Solution

This is expected behavior from cgroups. To configure the memory limit, configure Memory limit (in bytes) in the Anti-Virus tile.

Warning: When updating the memory limit, ensure that all VMs, including errand VMs, have sufficient memory resources.

Insufficient CPU Limit While Using Anti-Virus

Symptom

Anti-Virus fails to start during deployment. However, the clamd and freshclam processes eventually run.

The deployment failure log looks similar to:

Task 1071 | 19:40:49 | Updating instance clamav_1: clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c (0) (canary) (00:05:26)
                L Error: 'clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c (0)' is not running after update. Review logs for failed jobs: clamd, freshclam


When you run bosh -d DEPLOYMENT instances --ps, you see that the the clamd and freshclam processes are running successfully after the failed deployment.

For example:

$ bosh -d clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c instances --ps
Instance                                       Process    Process State  AZ  IPs
clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c  -          running        z1  10.0.0.7
~                                              clamd      running        -   -
~                                              freshclam  running        -   -

Explanation

Anti-Virus startup is CPU intensive and, if restricted, can prevent Anti-Virus from starting up correctly.

Solutions

  • Ensure cpu_limit is set high enough for Anti-Virus to execute normally. If the limit is too strict, Anti-Virus fails to start. To make changes to this limit, configure CPU limit (percentage) in the Anti-Virus tile.

  • Set enforce_cpu_limit to false. This allocates more CPU cycles to ClamAV if other processes are not using CPU resources.
    To disable this limit, set the Enforce CPU limit field to When other processes are using CPU resources in the Anti-Virus tile.

  • From the Ops Manager Installation Dashboard, navigate to the tile with the failing antivirus job. On Resource Config, adjust the VM Type for the Anti-Virus job to have sufficient CPU resources.

Too Many Open Files Error While Using Anti-Virus Mirror

Symptom

The Anti-Virus Mirror log reports that too many log files are open:

2019/07/29 20:02:41 10.0.0.72 is requesting main.cvd
2019/07/29 20:02:41 http: Accept error: accept tcp 0.0.0.0:80: accept4: too many open files; retrying in 10ms
2019/07/29 20:02:41 http: Accept error: accept tcp 0.0.0.0:80: accept4: too many open files; retrying in 20ms
2019/07/29 20:02:41 http: Accept error: accept tcp 0.0.0.0:80: accept4: too many open files; retrying in 40ms
2019/07/29 20:02:42 http: Accept error: accept tcp 0.0.0.0:80: accept4: too many open files; retrying in 5ms

Explanation

Anti-Virus Mirror opens files when a database is requested. There is a limit to how many files it can open at a time.

Solution

Increase the number of Anti-Virus Mirror instances. VMware recommends that there is one Anti-Virus Mirror for every 250 instances where Anti-Virus is installed. For more information, see Scale the Number of Deployed Mirrors.

Restoring with BOSH Backup and Restore Fails

Symptom

When using Anti-Virus Mirror, errors occur when you redeploy VMware Tanzu Application Service for VMs while restoring with BOSH Backup and Restore (BBR). For information about redeploying VMware Tanzu Application Service for VMs (TAS for VMs), see Step 12: Redeploy TAS for VMs.

Explanation

Anti-Virus Mirror must be running before you install Anti-Virus on other VMs in your deployment. Otherwise, Anti-Virus Mirror might not deploy before other tiles and dependencies deploy.

If Anti-Virus Mirror is not running, VMs with Anti-Virus installed cannot download the required database signature files. If this happens, errors and failed deployments occur.

Solution

To resolve this issue, you must ensure that Anti-Virus Mirror is deployed before restoring your deployment.

To do this:

  1. Follow the procedures before Step 12: Redeploy TAS for VMs in Restoring Deployments from Backup with BBR. Do not apply changes.

  2. Exclude Anti-Virus Mirror from the Anti-Virus deployment by following the procedure in Exclude Anti-Virus Mirror during Apply Changes. This ensures that Anti-Virus is not deployed on the Anti-Virus Mirror

  3. Remove the Anti-Virus Mirror exclusion from the Anti-Virus configurations by following the procedure in Remove the Exclusion.

  4. Continue to restore your deployment by following the remaining procedures in Restoring Deployments from Backup with BBR.

CPU Spikes While Enforcing a CPU Limit in Anti-Virus

Symptom

Anti-Virus is using more CPU resources than assigned in its configuration, even with the Enforce CPU limit field set to Always.

Explanation

Anti-Virus resource consumption is restricted using cgroups. If the VM does not have enough CPU or memory resources, the clamd PID is removed from the cgroup.procs file. This causes Anti-Virus to ignore the Enforce CPU limit setting.

Solution

Increase the VM size. VMware recommends a minimum VM size of micro.cpu using 2 CPU and 2 GB RAM.

Freshclam Logs Show “can’t query *.ping.clamav.net”

Symptom

The freshclam logs show the following warning messages:

Can't query main.IP-ADDRESS.ping.clamav.net
Can't query daily.IP-ADDRESS.ping.clamav.net
Can't query bytecode.IP-ADDRESS.ping.clamav.net

Explanation

Freshclam is the process which downloads virus definitions. Freshclam queries these endpoints to give ClamAV information about the current definitions being used and the version of the ClamAV binary. Failure to query these endpoints indicates one of the ClamAV servers is experiencing network difficulty, but this is unrelated to downloading virus definitions and does not affect updates.

Solution

No action is required. This issue does not impact the functionality of Anti-Virus.