Installing Velero and Restic

Prerequisites
Deploy an Object Store
Install the Velero CLI on Your Workstation
- Download the Velero CLI Binary
- Install the Velero CLI
Install Velero and Restic on the Target Kubernetes Cluster
Install Velero and Restic in an Air-Gapped Environment
- Prerequisites
- Procedure

Page last updated:

This topic describes how to install Velero and Restic for backing up and restoring Tanzu Kubernetes Grid Integrated Edition (TKGI)-provisioned Kubernetes workloads. This topic also describes how to install MinIO for Velero with Restic backup storage.

Prerequisites

Ensure the following before installing Velero and Restic for backing up and restoring TKGI:

You have read: Tanzu Kubernetes Workload Backup and Restore Requirements in Backing Up and Restoring Tanzu Kubernetes Workloads Using Velero with Restic.
You have a Linux VM with sufficient storage to store several workload backups. You will install MinIO on this VM. For more information, see Quick start evaluation install with Minio in the Velero documentation.
You have a TKGI Client VM (Linux) where CLI tools are installed, such as the TKGI CLI, kubectl, and others. You will install the Velero CLI on this client VM. If you do not have such a VM, you can install the Velero CLI locally, but you must adjust the following installation steps accordingly.
The Kubernetes environment has internet access and can be reached by the client VM. If the environment does not have internet access, refer to Install Velero and Restic in an Air-Gapped Environment below.

Deploy an Object Store

Restic requires an object store as the backup destination for workload backups.

To deploy and configure a MinIO Server on a Linux Ubuntu VM as the Velero and Restic backend object store:

Install MinIO
Enable MinIO as a Service
Create MinIO Bucket
Configure MinIO Bucket

For more information about MinIO, see the MinIO Quick Start Guide.

Install MinIO

To install MinIO:

To install the MinIO app:

wget https://dl.min.io/server/minio/release/linux-amd64/minio

To grant execute permissions:
```
chmod +x minio
```
To create a directory where MinIO data will be stored:
```
mkdir /DATA-MINIO
```

Start MinIO

To prepare the MinIO server:

To start the MinIO server:
```
./minio server /DATA-MINIO
```
After the MinIO server has started, you are provided with the data store instance endpoint URL, AccessKey, and SecretKey.
Record the MinIO server endpoint URL, AccessKey, and SecretKey information for the data store instance.
To browse to the MinIO data store, open a browser to the MinIO server endpoint URL. For example: http://10.199.17.63:9000/minio/login/.
To log in to the MinIO server, provide the AccessKey and ScretKey.

Enable MinIO as a Service

To enable MinIO as a service, configure MinIO for automatic startup:

To download the minio.service script:

curl -O https://raw.githubusercontent.com/minio/minio-service/master/linux-systemd/minio.service

Edit the minio.service script and add the following value for ExecStart:
```
ExecStart=/usr/local/bin/minio server /DATA-MINIO path
```
Save the revised script.

To configure the MinIO service, run the following series of commands:

cp minio.service /etc/systemd/system
cp minio /usr/local/bin/
systemctl daemon-reload
systemctl start  minio
systemctl status minio
systemctl enable minio

Create MinIO Bucket

To create a MinIO bucket for TKGI workload backup and restore:

Launch the Mino browswer and log in to your object store.
Click the Create Bucket icon.
Enter the bucket name, for example: tkgi-velero.
Verify that the bucket was created.

Configure MinIO Bucket

By default, a new MinIO bucket is read-only, but for our Velero backup and restore the MinIO bucket must be read-write.

To change the new tkgi-velero bucket to read-write:

Select the bucket and click on the dots link.
Select Edit Policy.
Change the policy to Read and Write.
Click Add.
To close the dialog box, click X.

Install the Velero CLI on Your Workstation

To install the Velero CLI on your workstation:

Download the Velero CLI Binary
Install the Velero CLI

Download the Velero CLI Binary

To download the Velero CLI Binary:

Download the supported version of the signed Velero binary for your version of TKGI from the TKGI product downloads page at myVMware. For more information on the currently supported Velero version(s), see Product Snapshot in Release Notes.

Note: You must use the Velero binary signed by VMware to be eligible for support from VMware.

Install the Velero CLI

To install the Velero CLI on the TKGI client or on your local machine:

Open a command line and change directory to the Velero CLI download.
To unzip the download file:
```
gunzip velero-linux-v1.4.2_vmware.1.gz
```

To check for the Velero binary:

ls -l

For example:

ls -l

-rw-r--r-- 1 root root 57819586 Sep 14 16:19 velero-linux-v1.4.2_vmware.1

To change the permissions and put the Velero CLI in the system path:
```
chmod +x velero-linux-v1.4.2_vmware.1
```
To make the Velero CLI globally available, move the CLI to the system path:
```
cp velero-linux-v1.4.2_vmware.1 /usr/local/bin/velero
```

Verify the installation:

velero version

For example:

velero version

Client:
    Version: v1.4.2

Install Velero and Restic on the Target Kubernetes Cluster

To install the Velero and Restic pods on each Kubernetes cluster whose workloads you want to backup, complete the following:

Prerequisites
Set Up the kubectl Context
Install Velero and Restic

Prerequisites

The following steps require:

You have installed MinIO as your backup object store. For more information, see Deploy an Object Store above.
Your Kubernetes cluster has internet access.

Set Up the kubectl Context

The Velero CLI context will automatically follow the kubectl context. Before running Velero CLI commands to install Velero and Restic on the target cluster, set the kubectl context:

Retrieve the name of the MinIO bucket. For example, tkgi-velero.
Get the AccessKey and SecretKey for the MinIO bucket. For example, AccessKey: 0XXNO8JCCGV41QZBV0RQ and SecretKey: clZ1bf8Ljkvkmq7fHucrKCkxV39BRbcycGeXQDfx.
Verify kubectl works against the cluster. If needed, use tkgi get-credentials.

Get the context for the target Kubernetes cluster so that the Velero CLI knows which cluster to work on.

tkgi get-credentials CLUSTER-NAME

Where CLUSTER-NAME is the name of the cluster.

tkgi get-credentials cluster-1

Fetching credentials for cluster cluster-1.
Password: ********
Context set for cluster cluster-1.

You can now switch between clusters by using:
$kubectl config use-context <cluster-name>

You could also run kubectl config use-context CLUSTER-NAME to set context.

To create a secrets file, create a file named credentials-minio. Update the file with the MinIO server access credentials you collected above:
```
aws_access_key_id = ACCESS-KEY

aws_secret_access_key = SECRET-KEY
```
Where:
- ACCESS-KEY is the AccessKey you collected above.
- SECRET-KEY is the SecretKey you collected above.
For example:
```
aws_access_key_id = 0XXNO8JCCGV41QZBV0RQ

aws_secret_access_key = clZ1bf8Ljkvkmq7fHucrKCkxV39BRbcycGeXQDfx
```
Save the file.
To verify the file is in place:
```
ls
```
For example:
```
ls

credentials-minio
```

Install Velero and Restic

To install Velero and Restic:

Run the following command to install Velero and Restic on the target Kubernetes cluster:

velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.0.0 \
--bucket tkgi-velero \
--secret-file ./credentials-minio \
--use-volume-snapshots=false \
--use-restic \
--backup-location-config \
region=minio,s3ForcePathStyle=“true”,s3Url=http://10.199.17.63:9000,publicUrl=http://10.199.17.63:9000

For example:

velero install  --provider aws  --plugins velero/velero-plugin-for-aws:v1.0.0 \
--bucket tkgi-velero  --secret-file ./credentials-minio  --use-volume-snapshots=false \
--use-restic  --backup-location-config \
region=minio,s3ForcePathStyle=“true”,s3Url=http://10.199.17.63:9000,publicUrl=http://10.199.17.63:9000

CustomResourceDefinition/backups.velero.io: created
...
Waiting for resources to be ready in cluster...
...
DaemonSet/restic: created
Velero is installed! Use 'kubectl logs deployment/velero -n velero' to view the status.

To verify the installation of Velero and Restic:
```
kubectl logs deployment/velero -n velero
```

To verify the velero namespace:

kubectl get ns

For example:

kubectl get ns

NAME              STATUS   AGE
default           Active   13d
kube-node-lease   Active   13d
kube-public       Active   13d
kube-system       Active   13d
pks-system        Active   13d
velero            Active   2m38s

To verify the velero and restic pods.

kubectl get all -n velero

For example:

kubectl get all -n velero

NAME READY STATUS RESTARTS AGE pod/restic-25chx 0/1 CrashLoopBackOff 1 30s pod/restic-rpxcp 0/1 CrashLoopBackOff 1 30s pod/restic-wfxmg 0/1 CrashLoopBackOff 1 30s pod/velero-8dc7498d9-9v7x4 1/1 Running 0 30s

Modify the Host Path

To run the three-pod Restic DaemonSet on a Kubernetes cluster in TKGI, you must modify the Restic DaemonSet spec and modify the hostpath.

To modify the Restic DaemonSet:

Verify the three-pod Restic DaemonSet:

kubectl get pod -n velero

For example:

kubectl get pod -n velero

NAME READY STATUS RESTARTS AGE pod/restic-p5bdz 0/1 CrashLoopBackOff 4 3m8s pod/restic-rbmnd 0/1 CrashLoopBackOff 4 3m8s pod/restic-vcpjm 0/1 CrashLoopBackOff 4 3m8s pod/velero-68f47744f5-lb5df 1/1 Running 0 3m8s

Run the following command:

kubectl edit daemonset restic -n velero

Change hostPath from /var/lib/kubelet/pods to /var/vcap/data/kubelet/pods:
```
 - hostPath:

      path: /var/vcap/data/kubelet/pods
```
Save the file.

To verify the three-pod Restic DaemonSet:

kubectl get pod -n velero

For example:

kubectl get pod -n velero

NAME READY STATUS RESTARTS AGE restic-5jln8 1/1 Running 0 73s restic-bpvtq 1/1 Running 0 73s restic-vg8j7 1/1 Running 0 73s velero-68f47744f5-lb5df 1/1 Running 0 10m

For information on why you must modify the Restic DaemonSet spec, see Restic Integration in the velero documentation.

Adjust Velero Memory Limits (if necessary)

If your Velero backup returns status=InProgress for many hours, increase the limits and requests memory settings.

To increase limits and requests memory settings:

Run the following command:

kubectl edit deployment/velero -n velero

Change the limits and request memory settings from the default of 256Mi and 128Mi to 512Mi and 256Mi:

ports:
- containerPort: 8085
  name: metrics
  protocol: TCP
resources:
  limits:
    cpu: "1"
    memory: 512Mi
  requests:
    cpu: 500m
    memory: 256Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File

Install Velero and Restic in an Air-Gapped Environment

If you are working in an air-gapped environment, you can install Velero and Restic using an internal registry. For more information, see Air-gapped deployments in the Velero documentation.

Prerequisites

A private container registry is installed and configured. The instructions use Harbor.
Docker is installed on the workstation or TKGI jump host.
kubectl context has been set and the MinIO credentials-minio file exists. For more information, see Set Up the kubectl Context above.

Procedure

Open the VMware Velero downloads page for your version of TKGI linked-to from the Product Snapshot of the Release Notes topic.
Download the Velero CLI and Velero with Restic Dockers images for your version of TKGI.
- velero-plugin-for-aws-v1.1.0-beta.1.tar.gz
- velero-v1.4.2_vmware.1.tar.gz
- velero-restic-restore-helper-v1.4.2_vmware.1.tar.gz
  Note: You must use the container images signed by VMware to be eligible for support from VMware.

Push the Docker images into the internal registry. Adjust the variables as needed for your registry instance and preferences.

docker login harbor.example.com
docker load -i velero-plugin-for-aws-v1.1.0-beta.1.tar.gz 
docker tag vmware-tanzu/velero-plugin-for-aws:v1.1.0-beta.1 harbor.example.com/vmware-tanzu/velero-plugin-for-aws:v1.1.0-beta.1
docker load -i velero-restic-restore-helper-v1.4.2_vmware.1.tar.gz 
docker tag vmware-tanzu/velero-restic-restore-helper:v1.4.2_vmware.1 harbor.example.com/vmware-tanzu/velero-restic-restore-helper:v1.4.2_vmware.1
docker load -i velero-v1.4.2_vmware.1.tar.gz
docker tag vmware-tanzu/velero:v1.4.2_vmware.1 harbor.example.com/vmware-tanzu/harbor.example.com/vmware-tanzu/velero:v1.4.2_vmware.1
docker push harbor.example.com/harbor-ci/vmware-tanzu/velero-plugin-for-aws:v1.1.0-beta.1
docker push harbor.example.vmware.com/vmware-tanzu/velero-restic-restore-helper:v1.4.2_vmware.1
docker push harbor.example.vmware.com/vmware-tanzu/harbor.example.com/vmware-tanzu/velero:v1.4.2_vmware.1

Install Velero with Restic:

velero install --image harbor.example.com/vmware-tanzu/harbor.example.com/vmware-tanzu/velero:v1.4.2_vmware.1 \
--plugins harbor.example.com/vmware-tanzu/velero-plugin-for-aws:v1.1.0-beta.1 \
--provider aws --bucket velero --secret-file ./credentials-minio \
--use-volume-snapshots=false    \
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://20.20.224.27:9000,publicUrl=http://20.20.224.27:9000 --use-restic

For example:

velero install --image harbor.example.com/vmware-tanzu/harbor.example.com/vmware-tanzu/velero:v1.4.2_vmware.1 --plugins harbor.example.com/vmware-tanzu/velero-plugin-for-aws:v1.1.0-beta.1 --provider aws --bucket velero --secret-file ./credentials-minio --use-volume-snapshots=false    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://20.20.224.27:9000,publicUrl=http://20.20.224.27:9000 --use-restic
Velero is installed! Use 'kubectl logs deployment/velero -n velero' to view the status.

For more information on installing Velero and Restic, see On-Premises Environments and Restic installation in the Velero documentation.

After installing, configure the Restic post-installation settings:
- Customize the Restic helper container and make it the init container for the pod during the restore process. This can be done by creating a configmap and applying it in the velero namespace, such as kubect apply -f restic-cm.yaml -n velero. Download the example configmap restic-cm.yaml provided for such purposes.
- Modify the host path by editing the Restic daemonset manifest. Replace /var/lib/kubelet/pods with /var/vcap/data/kubelet/pods. Verify that the Restic pods are running. For more information, see Modify the Host Path above.
- (Optional) Increase your Restic timeout: You can increase the Restic timeout for backups 1 TB or larger by editing the Velero deployment manifest and adding '- --restic-timeout=900m' to spec.template.spec.containers.
- (Optional) Adjust your Restic Pod CPU and memory reserves: Depending on your requirements, you can adjust the CPU and memory reserves and limits for your Velero and Restic Pods. For more information, see Adjust Velero Memory Limits (if necessary) above.
  
  restic pod
```
resources:
limits:
cpu: "1"
memory: 1Gi
requests:
cpu: 500m
memory: 512Mi
```
  velero pod
```
resources:
limits:
cpu: "1"
memory: 256Mi
requests:
cpu: 500m
memory: 128Mi
```

Please send any feedback you have to pks-feedback@pivotal.io.

Create a pull request or raise an issue on the source for this page in GitHub