Linux Stemcell Hardening
Page last updated:
Note: This document applies to stemcell v3263 and later.
Customers and prospects often ask for details on stemcell hardening, i.e., the process by which we secure Ops Manager by reducing its vulnerability surface from outside access. This document provides responses to some commonly-asked questions regarding the security configuration enhancements and hardening tests that Pivotal applies to the Cloud Foundry (“CF”) stemcell. This information will be helpful to customer accreditation teams who are responsible for running configuration scans of a Cloud Foundry deployment, and also to auditors who need a documentation artifact to feed into the customers’ existing security assessment processes.
WHAT IS A STEMCELL? A stemcell is a versioned Operating System (“OS”) image wrapped with IaaS specific packaging. A typical stemcell contains a bare minimum OS skeleton with a few common utilities pre-installed, a BOSH Agent, and a few configuration files to securely configure the OS by default. For example: with vSphere, the official stemcell for Ubuntu Trusty is an approximately 500MB VMDK file. With AWS, official stemcells are published as MIs that can be used in an AWS account. Stemcells do not contain any specific information about any software that will be installed once that stemcell becomes a specialized machine in the cluster; nor do they contain any sensitive information which would make them unable to be shared with other BOSH users. This clear separation between base OS and later-installed software is what makes stemcells a powerful concept. In addition to being generic, stemcells for one OS (e.g. all Ubuntu Trusty and Xenial stemcells) are exactly the same for all infrastructures. This property of stemcells allows BOSH users to quickly and reliably switch between different infrastructures without worrying about the differences between OS images. The CF BOSH team is responsible for producing and maintaining an official set of stemcells. Cloud Foundry currently supports Ubuntu Trusty and Xenial on vSphere, AWS, OpenStack, Google, and Azure infrastructures.
WHAT IS STEMCELL HARDENING? Stemcell hardening is the process of securing a stemcell by reducing its surface of vulnerability, which is larger when a system performs more functions; in principle a single-function system is more secure than a multipurpose one. There are various methods of hardening Linux systems. Common techniques include reducing available methods of attack by implementing more restrictive and/or conservative configurations of the OS kernel and system services, changing default passwords, the removal of unnecessary software, unnecessary usernames and logins, and the disabling or removal of unnecessary services.
WHAT IS OUR GENERAL APPROACH TO STEMCELL HARDENING? The CF stemcell is essentially a distinct Linux distribution. As such, industry-standard benchmarks are not entirely appropriate when assessing the security posture of the stemcell, but Pivotal has considered and incorporated hardening guidance from various sources both commercial and government. Some parts of the existing recommended industry-standard hardening configurations will certainly apply, but some other parts do not apply. In addition, because each stemcell is a unique Linux distribution, existing industry-standard benchmarks are silent on some important aspects of hardening the stemcell configurations. The following paragraphs describe the different categories of stemcell hardening configurations, and provide a count of the number of tests currently in each category. Note: The most current description of what has been delivered is always available in the BOSH public Pivotal Trackers.
- Baseline Passing: common hardening tests that pass without any changes to the stemcell or to test procedures. (130 tests)
- Test Amended: Stemcells are optimized for cloud deployment and some configuration settings are not stored in traditionally-expected locations. The industry standard test was changed to conform with stemcell design to accurately check the recommended setting. This new test reflects the changes to the industry standard test but the stemcell adheres to commonly accepted guidance. (36 tests)
- Additional Hardening:Configuration hardening improvements that have been made to the stemcell. As with most software, a stemcell’s security improves over time and every stemcell release is tested to ensure that it is suitable for use with its associated CF release. Later releases of a stemcell may include additional security features that were not present in earlier releases. (86 tests)
- New CF-specific Tests: New tests that have been added to check CF stemcell-specific configurations. These tests are not yet part of any industry standard Ubuntu benchmark. This category of tests is still under development and additional tests will be added over time. (20 tests)
WHAT ARE THE MAJOR FOCUS AREAS FOR OUR STEMCELL HARDENING APPROACH?
Maintenance, Updates, and Patching
- Regular patches and feature enhancements are delivered via routine BOSH deployments of updated stemcells (obviates apt-get upgrade).
File System Hardening
- The /tmp directory is configured to be on a separate partition.
- Users cannot create character or block special devices in the /tmp filesystem.
- Users cannot create set userid files in the /tmp filesystem.
- Users cannot run executable binaries from the /tmp filesystem.
- The temporary storage directories such as /tmp and /var/tmp are mounted on a dedicated partition, and configured with appropriately limiting options such as nodev, nosuid, and noexec.
- Each of the following directories is in a separate partition, with mount options managed via BOSH agent:
- File system mount options for users’ home directories are limited via appropriate mount options including nodev.
- Removable media may not be mounted as character or block special device.
- Executable programs may not run from removable media.
- setuid and setgid are not allowed on removable media.
- Users cannot create special devices in shared memory partitions.
- Users cannot put privileged programs onto shared memory partitions.
- Users cannot execute programs from shared memory partitions.
- Users cannot delete or rename files in world-writable directories such as /tmp that are owned by other users.
- Supplementary and exotic Linux file systems that are unused in CF have been disabled.
- Additional supplementary and exotic Linux file systems that are unused in CF have been disabled.
- Automount of USB drives or disks is not permitted.
- The owner and group for the bootloader config (/boot/grub/grub.cfg) is set to root. Only root has read and write access to this file.
- Boot loader has been configured so that a password is required to reboot the system.
- Unauthorized users cannot reboot the system into single user mode.
- Users cannot override the soft limit for core dumps.
- Randomized virtual memory region placement is enabled.
- Prelinking of shared libraries is disabled.
Minimization of Attack Surface
- The Network Information Service (“NIS”) is not used in CF and is not installed.
- The Berkeley rsh-server package is not used in CF and is disabled.
- Classic rsh-related tools are not used in CF and are not installed.
- The following servers are not used on CF stemcells and are disabled:
- talk server
- telnet server
- The talk client is not used in CF and is not installed.
- The eXtended InterNET Daemon (xinetd) is not used in CF and is disabled.
- The following network services are not used in CF and are disabled:
- The X Window system is not used in CF and is not installed.
- NTP time setting is synchronized on the stemcell via the ntpdate utility.
- The Samba daemon is not used in CF and is disabled.
- The Mail Transfer Agents (MTA) process only local mail.
- The rsync service is not used in CF and is disabled.
- The biosdevname tool is disabled.
- IPv4 networking is configured such that IP forwarding is disabled.
- The IPv4 networking has been configured such that the host cannot send ICMP redirects.
- IPv4 networking has been configured such that the system does not accept source routed packets.
- IPv4 networking is configured such that ICMP redirects are not accepted.
- ICMP echo and timestamp requests with broadcast or multicast destinations will be ignored.
- The stemcell will ignore malformed ICMP error responses.
- IPv4 networking is configured for source route validation.
- TCP SYN cookies are enabled.
- Stemcells are set to refuse IPv6 router advertisements.
- The /etc/hosts.allow file exists and is empty.
- The /etc/hosts.allow and /etc/hosts.deny files are protected from unauthorized write access.
- The /etc/hosts.deny file exists and is empty.
- The following protocols are not used in CF and are disabled:
- Wireless interfaces are disabled.
- IPv6 is not used in CF deployments and the IPv6 protocol is disabled.
- Audit log file size is configured for a manageable maximum size of 6 MB.
- The system auditd logs have been configured such that the system is resilient in the event of a denial of service attack on the auditd daemon.
- Auditd daemon is configured such that all auditd logs are kept after rotation.
- The auditd service is enabled.
- Auditing of successful and failed login/logout events is enabled.
- The Linux auditing subsystem has been configured in accordance with best practice industry guidance to capture all security-relevant events. The /etc/audit/audit.rules configuration now contains more than 50 monitoring rules.
- Audit records are created for loading and unloading of kernel modules and for system calls.
- File Integrity Monitoring can be done on the stemcell (via a BOSH Add-on).
Authentication and Authorization
- The cron daemon is enabled.
- Access to the /etc/crontab file is limited to root.
- Access to the cron utility configuration via the hourly, daily, weekly, and monthly directories is limited.
- User authorization to schedule cron jobs is limited.
- Only the vcap user is whitelisted to use the cron and at utilities.
- Password requirements follow industry best practice guidance and enforce a minimum length of 14 characters, with at least one each of: digit, uppercase, lowercase and special characters.
- Password reuse: users cannot reuse their twenty most recent passwords.
- SSH protocol version is configured for SSH-2.
- Logging level for SSH event is INFO.
- Minimum permissions are set on /etc/ssh/sshd_config.
- SSH X11 forwarding is disabled.
- The MaxAuthTries parameter for SSH is set to 3 attempts per connection.
- SSH is configured to require passwords and ignore host-based authentication.
- Root logins are not allowed over SSH.
- Users cannot set environment variables through the SSH daemon.
- SSH has been configured to use strong ciphers:
- Idle SSH sessions are terminated after 15 minutes, and no client “keep alive” messages are sent.
- Idle SSH sessions are terminated after 15 minutes. No client “keep alive” messages are sent.
- The SSH login banner may be configured to display site-specific text before user authentication is permitted (via BOSH Add-on).
- Root login is only permitted via console, not via tty devices.
- Only the vcap user is authorized in the sudo group.
- Only users in the root group (a.k.a. wheel) are authorized to run the su command.
- Contents of /etc/issue and /etc/issue.net have been configured to the phrase: “‘Unauthorized use is strictly prohibited. All access and activity is subject to logging and monitoring.” This may be amended if and as necessary via a BOSH Add-on.
- The Message of the Day file /etc/motd is not used, but may be populated via a BOSH Add-on if needed.
- Identification of the OS and/or version information about the OS does not appear in any login banners.
File System Permissions
- The /etc/passwd, /etc/shadow, and /etc/group files are protected from unauthorized write access.
- Use and/or presence of any world-writable files has been audited, and minimized to the extent possible for CF.
- By default, all stemcell files are owned by a known user and group, and may not belong to a non-existent user or group.
- Use of SUID and GUID is restricted, and only the /usr/bin/sudo and /bin/su programs are authorized as SUID and/or GUID programs.
User Account Management
- Users cannot change their password more than once a day.
- Users are notified 7 days before their passwords expire.
- Interactive logins are disabled for system accounts.
- The GID for the root account is 0.
- User accounts may not have empty passwords.
- NIS is not used in CF, and integration of OS security configuration with legacy NIS permissioning is not enabled (e.g., for /etc/passwd, shadow, and group).
- By default, the only UID 0 account present is root.
- By default, the root PATH does not include any risky directory such as the current working (.) or any writable directory.
- Minimum privileges are applied to all users’ hidden configuration (“dot”) files.
- The .netrc and .rhosts and .forward files are not used in CF and are not present in any user home directory.
- Any group present in the /etc/passwd file must also exist in the /etc/group file.
- Users defined in /etc/password must have a valid home directory.
- Users must own their home directories.
- All references to user and group names, as well as UID and/or GID identifiers, are self consistent, with no duplicates or orphans allowed.
- By default, the shadow group is not used in CF and must be empty.