Installing Velero and Restic
Page last updated:
This topic describes how to install Velero and Restic for the purposes of backing up and restoring Kubernetes workloads. This topic also describes how to install Minio as the storage for Velero with Restic backups.
Prerequisites
Read the workload backup and restore requirements before proceeding.
Prepare a Linux VM with sufficient storage to store several workload backups. You will install Minio on this VM.
It is assumed that you have TKGI Client VM (Linux) where CLI tools are installed, such as the TKGI CLI, kubectl, and others. You will install the Velero CLI on this client VM. If you do not have such a VM, you can install the Velero CLI on your laptop, but you will need to adjust the installation steps accordingly.
It is assumed that the Kubernetes environment has internet access and can be reached by the client VM. If the environment has no internet access, refer to the air-gapped instructions.
Deploy an Object Store
Restic requires an object store as the backup destination for for workload backups. Complete the following tasks to install and configure Minio as the backend object store. The instructions describe how to deploy the Minio Server on a Linux Ubuntu VM. For more information, see the Minio quick start guide.
Install Minio
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
Create directory where Minio data will be stored:
mkdir /DATA-MINIO
Start Minio
Start the Minio server:
./minio server /DATA-MINIO
Once the Minio server is started, you are provided with the endpoint URL, AccessKey, and ScretKey for the data store instance. Record this information.
Browse to the Minio data store using the endpoint URL, for example http://http://10.199.17.63:9000/minio/login/.
Provide the AccessKey and ScretKey and log in.
Enable Minio as a Service
Configure Minio for automatic startup.
Download the minio.service
script:
curl -O https://raw.githubusercontent.com/minio/minio-service/master/linux-systemd/minio.service
Edit the minio.service
script and add the following value for ExecStart
:
ExecStart=/usr/local/bin/minio server /DATA-MINIO path
Run the following series of commands to configure the Minio service:
cp minio.service /etc/systemd/system
cp minio /usr/local/bin/
systemctl daemon-reload
systemctl start minio
systemctl status minio
systemctl enable minio
Create Minio Bucket
Create a Minio bucket for TKGI workload backup and restore.
- Launch the Mino browswer and log in to your object store.
- Click the Create bucket icon.
- Enter the bucket name, for example:
tkgi-velero
- Verify that the bucket is created.
Configure Minio Bucket
By default, the policy for the new tkgi-velero
bucket is read-only. Change it to read-write:
- Select the bucket and click on the dots link.
- Select Edit Policy.
- Change the policy to Read and Write.
- Click Add.
- Click the X button to close the dialog box.
Install the Velero CLI on Your Workstation
Install the Velero CLI on your workstation.
Download the Velero CLI Binary
Download the supported version of the signed Velero binary from the TKGI product downloads page at myVMware.
Note: You must use the Velero binary signed by VMware to be eligible for support from VMware.
Install the Velero CLI
Install the Velero CLI on the TKIG client or on your local machine.
CD to the Velero CLI download and unzip the file.
gunzip velero-linux-v1.4.2_vmware.1.gz
Check for the Velero binary:
ls -l
-rw-r--r-- 1 root root 57819586 Sep 14 16:19 velero-linux-v1.4.2_vmware.1
Change the permissions and put the Velero CLI in the system path.
chmod +x velero-linux-v1.4.2_vmware.1
Put the CLI in the system path so it is globally available.
cp velero-linux-v1.4.2_vmware.1 /usr/local/bin/velero
Verify the installation:
velero version
Client:
Version: v1.4.2
Install Velero and Restic on the Target Kubernetes Cluster
Install the Velero and Restic pods on each Kubernetes cluster whose workloads you want to backup. The following instructions assume the use of Minio and that the Kubernetes cluster has internet access.
The Velero CLI context follows the kubectl CLI context. Before running the Velero CLI command to install Velero and Restic on the target cluster, you first set the kubectl context. The Velero CLI context will automatically follow the kubectl context. For example, running kubectl config use-context <cluster-name>
or pks get-credentials <cluster-name>
tells the Velero CLI which cluster to work on.
Prerequisites
Before installing Velero and Restic on the Kubernetes cluster:
- Get the name of the Minio bucket; it is
tkgi-velero
in our example - Get the keys for the Minio bucket:
- AccessKey:
0XXNO8JCCGV41QZBV0RQ
(for example) - SecretKey: SecretKey:
clZ1bf8Ljkvkmq7fHucrKCkxV39BRbcycGeXQDfx
(for example)
- AccessKey:
- Make sure
kubectl
works against the cluster (usetkgi get-credentials
if needed)
Procedure
Get the context for the target Kubernetes cluster so that the Velero CLI knows what cluster to work on. Replace cluster-1
with the name of the target cluster.
pks get-credentials cluster-1
Fetching credentials for cluster cluster-1.
Password: ********
Context set for cluster cluster-1.
You can now switch between clusters by using:
$kubectl config use-context <cluster-name>
Create a secrets file named credentials-minio
with the credentials to access the Minio server.
aws_access_key_id = 0XXNO8JCCGV41QZBV0RQ
aws_secret_access_key = clZ1bf8Ljkvkmq7fHucrKCkxV39BRbcycGeXQDfx
Verify:
ls
credentials-minio
Run the following command to install Velero and Restic on the target Kubernetes cluster:
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.0.0 \
--bucket tkgi-velero \
--secret-file ./credentials-minio \
--use-volume-snapshots=false \
--use-restic \
--backup-location-config \
region=minio,s3ForcePathStyle=“true”,s3Url=http://10.199.17.63:9000,publicUrl=http://10.199.17.63:9000
Sample expected result:
CustomResourceDefinition/backups.velero.io: created
...
Waiting for resources to be ready in cluster...
...
DaemonSet/restic: created
Velero is installed! Use 'kubectl logs deployment/velero -n velero' to view the status.
Verify
Verify the installation of Velero and Restic.
kubectl logs deployment/velero -n velero
Verify the velero
namespace.
kubectl get ns
NAME STATUS AGE
default Active 13d
kube-node-lease Active 13d
kube-public Active 13d
kube-system Active 13d
pks-system Active 13d
velero Active 2m38s
Verify the velero
and restic
pods.
kubectl get all -n velero
NAME READY STATUS RESTARTS AGE
pod/restic-25chx 0/1 CrashLoopBackOff 1 30s
pod/restic-rpxcp 0/1 CrashLoopBackOff 1 30s
pod/restic-wfxmg 0/1 CrashLoopBackOff 1 30s
pod/velero-8dc7498d9-9v7x4 1/1 Running 0 30s
Modify the Host Path
To run the 3-pod Restic DaemonSet on a Kubernetes cluster in TKGI, you must modify the Restic DaemonSet spec.
For more information as to why, see the Restic documentation.
In addition, the hostPath should be changed from /var/lib/kubelet/pods
to /var/vcap/data/kubelet/pods
.
The instructions for both of these configurations are provided below.
Verify the 3-pod Restic DaemonSet:
kubectl get pod -n velero
NAME READY STATUS RESTARTS AGE
pod/restic-p5bdz 0/1 CrashLoopBackOff 4 3m8s
pod/restic-rbmnd 0/1 CrashLoopBackOff 4 3m8s
pod/restic-vcpjm 0/1 CrashLoopBackOff 4 3m8s
pod/velero-68f47744f5-lb5df 1/1 Running 0 3m8s
Run the following command:
kubectl edit daemonset restic -n velero
Change hostPath from /var/lib/kubelet/pods
to /var/vcap/data/kubelet/pods
:
- hostPath:
path: /var/vcap/data/kubelet/pods
Save the file.
Verify the 3-pod Restic DaemonSet:
kubectl get pod -n velero
NAME READY STATUS RESTARTS AGE
restic-5jln8 1/1 Running 0 73s
restic-bpvtq 1/1 Running 0 73s
restic-vg8j7 1/1 Running 0 73s
velero-68f47744f5-lb5df 1/1 Running 0 10m
Adjust Velero Memory Limits (if necessary)
By default Velero sets limits and request memory at 256 and 128 by default. In case a Velero backup get status=‘InProgress’ for many hours, increase these limits.
kubectl edit deployment/velero -n velero
Change limits and request memory to 512 and 256.
Install Velero and Restic in an Air-Gapped Environment
If you are working in an air-gapped environment, you can install Velero and Restic using an internal registry.
Prerequisites
- A private container registry is installed and configured. The instructions use Harbor.
- Docker is installed on the workstation or TKGI jump host.
- Get the name of the Minio bucket; it is
tkgi-velero
in our example - Get the keys for the Minio bucket:
- AccessKey:
0XXNO8JCCGV41QZBV0RQ
(for example) - SecretKey: SecretKey:
clZ1bf8Ljkvkmq7fHucrKCkxV39BRbcycGeXQDfx
(for example)
- AccessKey:
- Make sure
kubectl
works against the cluster (usetkgi get-credentials
if needed)
Procedure
Download the Velero CLI and Velero with Restic Dockers images from VMware at https://www.vmware.com/.
- velero-plugin-for-aws-v1.1.0-beta.1.tar.gz
- velero-v1.4.2_vmware.1.tar.gz
- velero-restic-restore-helper-v1.4.2_vmware.1.tar.gz
Note: You must use the container images signed by VMware to be eligible for support from VMware.
Push the Docker images into the internal registry. Adjust the variables as needed for your registry instance and preferences.
docker login harbor.example.com docker load -i velero-plugin-for-aws-v1.1.0-beta.1.tar.gz docker tag vmware-tanzu/velero-plugin-for-aws:v1.1.0-beta.1 harbor.example.com/vmware-tanzu/velero-plugin-for-aws:v1.1.0-beta.1 docker load -i velero-restic-restore-helper-v1.4.2_vmware.1.tar.gz docker tag vmware-tanzu/velero-restic-restore-helper:v1.4.2_vmware.1 harbor.example.com/vmware-tanzu/velero-restic-restore-helper:v1.4.2_vmware.1 docker load -i velero-v1.4.2_vmware.1.tar.gz docker tag vmware-tanzu/velero:v1.4.2_vmware.1 harbor.example.com/vmware-tanzu/harbor.example.com/vmware-tanzu/velero:v1.4.2_vmware.1 docker push harbor.example.com/harbor-ci/vmware-tanzu/velero-plugin-for-aws:v1.1.0-beta.1 docker push harbor.example.vmware.com/vmware-tanzu/velero-restic-restore-helper:v1.4.2_vmware.1 docker push harbor.example.vmware.com/vmware-tanzu/harbor.example.com/vmware-tanzu/velero:v1.4.2_vmware.1
Create a secrets file named
credentials-minio
with the credentials to access the Minio server.aws_access_key_id = 0XXNO8JCCGV41QZBV0RQ aws_secret_access_key = clZ1bf8Ljkvkmq7fHucrKCkxV39BRbcycGeXQDfx
Install Velero with Restic. Refer to the On-Premises Environments instructions in the Velero documentation, and the Restic installation instructions.
- You can install Velero using the following command. Adjust the variables for your environment.
velero install --image harbor.example.com/vmware-tanzu/harbor.example.com/vmware-tanzu/velero:v1.4.2_vmware.1 --plugins harbor.example.com/vmware-tanzu/velero-plugin-for-aws:v1.1.0-beta.1 --provider aws --bucket velero --secret-file ./credentials-minio --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http:span>//20.20.224.27:9000,publicUrl=http://20.20.224.27:9000 --use-restic
- Expected result:
Velero is installed! Use 'kubectl logs deployment/velero -n velero' to view the status.
- You can install Velero using the following command. Adjust the variables for your environment.
After installing, configure the Restic post-installation settings:
- Customize the Restic helper container and make it the init container
for the pod during the restore process.
This can be done by creating a configmap and applying it in the velero namespace, such as
kubect apply -f restic-cm.yaml -n velero
. Download the example configmap restic-cm.yaml provided for such purposes. - Modify the host path by editing the Restic daemonset manifest. Replace
/var/lib/kubelet/pods
with/var/vcap/data/kubelet/pods
. Verify that the Restic pods are running. For more information, see Modify the Host Path above. - (Optional) Increase your Restic timeout: You can increase the Restic timeout
for backups 1 TB or larger by editing the Velero deployment manifest
and adding
'- --restic-timeout=900m'
tospec.template.spec.containers
. (Optional) Adjust your Restic Pod CPU and memory reserves: Depending on your requirements, you can adjust the CPU and memory reserves and limits for your Velero and Restic Pods. For more information, see Adjust Velero Memory Limits (if necessary) above.
restic podresources: limits: cpu: "1" memory: 1Gi requests: cpu: 500m memory: 512Mi
velero pod
resources: limits: cpu: "1" memory: 256Mi requests: cpu: 500m memory: 128Mi
- Customize the Restic helper container and make it the init container
for the pod during the restore process.
This can be done by creating a configmap and applying it in the velero namespace, such as
Please send any feedback you have to pks-feedback@pivotal.io.