LATEST VERSION: 1.4 - RELEASE NOTES
ClamAV Add-on for PCF v1.4

Troubleshooting ClamAV Add-on for PCF

Page last updated:

This topic provides instructions to verify that the ClamAV-based antivirus add-on works with your Pivotal Cloud Foundry (PCF) deployment and provides general recommendations for troubleshooting and ensuring that the deployment is being protected as you expect.

ClamAV Installation Issues

Ops Manager Fails to Apply Changes

Symptom

Applying changes in Ops Manager fails. The bottom of the changelog contains an error message similar to the following:

...
Started updating job nats > nats/0 (12bfae02-b4af-4104-b2bd-227ff07b2d92) (canary). Done (00:02:31)
  Failed updating job etcd_server > etcd_server/0 (f8e492bf-db09-4d38-8a73-5cf69d7b8a11) (canary): 'etcd_server/0 (f8e492bf-db09-4d38-8a73-5cf69d7b8a11)' is not running after update. Review logs for failed jobs: clamd (00:05:53)

Error 400007: 'etcd_server/0 (f8e492bf-db09-4d38-8a73-5cf69d7b8a11)' is not running after update. Review logs for failed jobs: clamd

Explanation

The ClamAV mirror server was unavailable during initial deployment.

Solution

Review the manifest file, and replace the database_mirror key with the address of a stable mirror server. If you do not have a stable mirror server for reliable initial deployment, use the S3-based mirror: pivotal-clamav-mirror.s3.amazonaws.com


ClamAV Runtime Issues

ClamAV Is Not Detecting Malware

Symptom

Malware signature or sample malware is not detected, even though the ClamAV daemon is properly configured.

Explanation

Virus signatures are not up-to-date.

Solution

First, ensure that the configuration checks have been done, that the mirror server is correctly configured and is available on the network from within the PCF private subnet, and that at least one hour has elapsed. One hour is the default scan schedule interval.

If the local mirror is up-to-date and ClamAV is still failing to detect a malware sample, you might have encountered a new threat. Pivotal recommends alerting the community via existing channels and reporting the suspicious file directly to the ClamAV team.

Note: Pivotal does not provide support for ClamAV detection failures, mirror coordination, or threat tracking activity.


ClamAV Reports False Positives

Symptom

ClamAV reports a false postive result; a non-malicious file is reported to be a virus.

Explanation

ClamAV compares files to its database of known malicious patterns. ClavAV may detect a non-malicious file as a virus due to a coincidental similarity to those patterns.

Solution

Submit false positive reports to ClamAV. You can also subscribe to the ClamAV email list to be kept up-to-date with ClamAV database changes.


CPU Spikes While Using ClamAV

Symptom

ClamAV is taking more CPU resources than assigned in its configuration.

Explanation

ClamAV resource consumption is restricted using cgroups. ClamAV is resource-limited whenever other processes are active. However, cgroups allows ClamAV to occupy more CPU resources when all other processes are idle, because it does not impact their performance.

Solution

This is expected behavior from cgroups. If a hard limit is required, configure the enforce_cpu_limit Linux property. For more information, see clamav.yml Template for Linux Linux property.


Out of Memory While Using ClamAV

Symptom

ClamAV fails to start and /var/log/syslog reports Memory cgroup out of memory: Kill process on the clamd process similar to the following:

2019-02-20T19:35:40.249205+00:00 localhost kernel: [  254.669948] Memory cgroup out of memory: Kill process 7493 (clamd) score 586 or sacrifice child
2019-02-20T19:35:40.249205+00:00 localhost kernel: [  254.679053] Killed process 7527 (clamd) total-vm:786136kB, anon-rss:626692kB, file-rss:1592kB

Explanation

ClamAV resource consumption is restricted by cgroups. The clamd process is terminated if the memory usage limit is exceeded. When memory swapping is disabled by other BOSH jobs, the ClamAV resource requires a larger memory limit.

Solution

This is expected behavior from cgroups. To configure the memory limit, configure the memory_limit Linux property. For more information, see clamav.yml Template for Linux.

Warning: When updating the memory limit, ensure that all VMs, including errand VMs, have sufficient memory resources.


Insufficient CPU Limit While Using ClamAV

Symptom

ClamAV fails to start during deployment, however the clamd and freshclam processes eventually run.

The deployment failure log looks similar to the following:

Task 1071 | 19:40:49 | Updating instance clamav_1: clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c (0) (canary) (00:05:26)
                L Error: 'clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c (0)' is not running after update. Review logs for failed jobs: clamd, freshclam


When you run bosh -d DEPLOYMENT instances --ps, the clamd and freshclam processes are successfully running after the failed deployment.

For example:

$ bosh -d clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c instances --ps
Instance                                       Process    Process State  AZ  IPs
clamav_1/d5cfe4bd-b606-4372-8481-187f4cf57e6c  -          running        z1  10.0.0.7
~                                              clamd      running        -   -
~                                              freshclam  running        -   -

Explanation

ClamAV startup is CPU intensive and if restricted, can prevent ClamAV from starting up correctly.

Solutions

  • Ensure cpu_limit is set high enough for ClamAV to execute normally. If the limit is too strict, ClamAV fails to start. For example, an n1-standard VM on GCP requires cpu_limit to be > 45. For more information, see clamav.yml Template for Linux.

  • Set enforce_cpu_limit to false. This allocates more CPU cycles to ClamAV if other processes are not using CPU resources.
    For more information, see clamav.yml Template for Linux.

  • From the Ops Manager Installation Dashboard, navigate to the tile with the failing clamav_1 job. On the Resource Config pane, adjust the VM Type for the ClamAV job to have sufficient CPU resources.

Create a pull request or raise an issue on the source for this page in GitHub