Client-side Deduplication Documentation

Dedup Server and Client Setup Guide:

This guide provides users with a comprehensive walkthrough for establishing and configuring the Dedup server and client components.

Dedup Server Setup:

The Dedup server is installed as part of the backup-server binary installation. Follow these steps to configure the Dedup server.

Verify Dedup server services by executing the following command:
```
setup-dedup-server
```

Adding Dedup server to the cluster:
- The default port for Dedup server is 8004.

Update config.linux.json file in /etc/zmanda:
- Update the host in rabbitMQ and the server.
- Update CommandQueue and CommandQueueRoutingkey parameters with the server IP.
- Ensure rabbitMQ port is set to 5681, ServerPort, and VaultPort are set to 8004.
Change directory into the /etc/zmanda folder and modify the following JSON keys in the config.linux.json file.

{
   //…  
      “agent” : {
              //… 
              “rabbitMQ”: {
                      //… 
                      “host”: “<server IP address>”,
           "Port": 5681,
                       “CommandQueue”: “<IP/hostname of client>”,
                       “CommandQueueRoutingKey”: “<IP/hostname of client >”
        //…
               }
             "server": {
            	"Host": “<server IP address >”,
            	"ServerPort": 8004,
            	"VaultPort": 8004
       	 }
       }
}

The Dedup server setup is complete!

Dedup Client Setup:

Follow these steps to configure the Dedup client.

Linux - Zmanda Backup Agent:

Install Zmanda backup agent by executing the following command:
- RPM-based: yum install ./<binary.rpm>
- Debian-based: apt-get install ./<binary.deb>
Update config.linux.json file in /etc/zmanda:
- Update the host in rabbitMQ and the server with the server IP.
- Update CommandQueue and CommandQueueRoutingkey parameters with the client IP.
- Ensure rabbitMQ port is set to 5681, ServerPort, and VaultPort are set to 8004.

 {
   //…  
      “agent” : {
              //… 
              “rabbitMQ”: {
                      //… 
                      “host”: “<server IP address>”,
           "Port": 5681,
                       “CommandQueue”: “<IP/hostname of client>”,
                       “CommandQueueRoutingKey”: “<IP/hostname of client >”
        //…  
               }
             "server": {
            	"Host": “<server IP address>”,
            	"ServerPort": 8004,
            	"VaultPort": 8004
       	 }
       }
}

Restart Zmanda backup agent service by executing the following command:
```
systemctl restart zmanda-backup-agent
```
Verify service status by executing the following command:
```
systemctl status zmanda-backup-agent
```

The Zmanda backup agent (Dedup client for Linux) is now set up and registered with the Dedup server. Backups and restores can proceed as usual within ZMC.

Log files are located in the directory /var/log/amanda/zmanda-backup-agent/zmanda-backup-agent.log

Windows - Zmanda Backup Agent:

Zmanda backup agent installation:
- Verify the installed build in the installed applications.

Update config.windows.json file in C:\ProgramData\Zmanda\ZmandaBackupAgent:
- Update the host in rabbitMQ and the server with the server IP.
- Update CommandQueue and CommandQueueRoutingkey parameters with the client IP.
- Ensure rabbitMQ port is set to 5681, ServerPort, and VaultPort are set to 8004.

 {
   //…  
      “agent” : {
              //… 
              “rabbitMQ”: {
                      //… 
                      “host”: “<server IP address>”,
           "Port": 5681,
                       “CommandQueue”: “<IP/hostname of client>”,
                       “CommandQueueRoutingKey”: “<IP/hostname of client >”
      //…
               }
             "server": {
            	"Host": “<server IP address>”,
            	"ServerPort": 8004,
            	"VaultPort": 8004
       	 }
       }
}

Restart Zmanda backup agent service in Services.

The Zmanda Backup Agent (Dedup client for Windows) is now set up and registered with the Dedup server. Backups and restores can proceed as usual within ZMC.

Log files are located in the directory C:\ProgramData\Zmanda\ZmandaBackupAgent\Logs\BackupAgent

Notes:

Only Linux and Windows NTFS/ReFS workloads are supported as part of Dedup MVP.
Backup and Restore reports are not available.
Vaulting is not supported.
Immutable backups and Glacier LifeCycle features on AWS S3 are not supported.
To backup localhost, CommandQueue and CommandQueueRoutingkey parameters should be updated with "localhost".

{
   //…  
      “agent” : {
              //… 
              “rabbitMQ”: {
                      //… 
         “CommandQueue”: “localhost”,
                      “CommandQueueRoutingKey”: “localhost”
               }
       }
}

User Guide for Dedup Server Source Creation:

This user guide aims to explain all the necessary fields for creating a source when utilizing the dedup server. The following are the supported sources for deduplication:

Linux
Windows NTFS/REFS

Creating a Source:

There are certain fields exclusive to the AE server that will not function as expected when using the dedup server. Despite this, these fields are available since the source creation page remains the same regardless of the server type being utilized. These fields include:

Data Deduplication (optional): When using an AE server, the user may decide whether they want data deduplication by using this checkbox. The encryption and compression fields are disabled if this item is checked.
Encryption and Compression Strategy (optional): While using an AE server, the user can select the type of encryption and compression strategy they want for backing up the source.

Creating a Linux Source:

Below are the steps for creating a Linux source.

Type (Dropdown): Select the type of file system.

Hostname: Provide the hostname of the Linux server. Validations are in place to ensure the correct hostname format. It accepts only alphanumeric letters or the symbol - and should not begin or end with a white space. An error highlight stating Invalid Hostname appears if the user enters an invalid hostname.

Directory Path: Specify the path of the directory to be backed up.
Exclude Paths (optional): Specify any directories to be excluded from the backup. Each entry should start on a new line. For instance, if users wish to back up the entire disk excluding the var directory, they can enter /var/ in this field.
Global Exclude Paths: Selecting this option enables the exclusion of paths specified in the Policies tab.

Comments (optional): Add any comments for future reference.
Backup Sets: Select the Backup Sets to link this source to for data to be backed up.

After clicking Save, a toast message indicates that the source creation is in process.

Upon completion, another toast message confirms the successful creation of the source.

Creating a Windows NTFS/REFS Source:

Follow these steps for creating a Windows NTFS/REFS source.

Type (Dropdown): Select the type of the file system.

Hostname: Enter the IP address of the source where the agent is installed.
Directory Path: Enter the path of the directory to be backed up in MS-DOS format (forward slashes).
Exclude Paths (optional): Add any directories to be excluded from the backup. Each entry should be on a separate line. For instance, if users wish to back up the C drive but not the ProgramData directory, they can enter C:ProgramData in this field.
Global Exclude Paths: Selecting this option enables the exclusion of paths specified in the Policies tab.

Backup Sets: Choose the backup sets with which you want to link this source to back up your data.
Comments (optional): Add any comments for future reference.

After clicking Save, a toast message indicates that the source creation is in process.

Upon completion, another toast message confirms the successful creation of the source.

User Guide for Dedup Server Storage Creation:

This user guide aims to explain all the necessary fields for creating a storage when utilizing the dedup server. The supported storage options include:

Disk
AWS
S3 Compatible
Azure

Creating a Disk Storage:

When creating a disk storage, ensure that only devices with an active SMB service are used, particularly when selecting dedup server as the backup server. Follow these steps:

Name: Provide a name for the disk storage.
SMB IP: Enter the SMB IP and validate it for accuracy. SMB IP addresses must consist of four octets, each ranging from 0 to 255, and should exclusively contain digits. In the event of the user entering an incorrect IP address, the system will present an error indicating Invalid Server IP.

SMB Port: Specify the SMB Port number and validate it for accuracy. This field allows only digits (up to five) and no letters or special characters.

SMB Share: Enter the SMB storage share installed on the machine.
Username: Specify the username for the SMB storage.
Password: Provide the password for the SMB storage.
Directory (optional): Specify the SMB storage directory path. If left blank, the machine's home directory will be used as the default location.
Domain (optional): Provide the domain name of the SMB storage. If left blank, the default value is WORKGROUP.
Comments (optional): Add any comments for future reference.

If the Advanced option is chosen, additional fields appear, such as Output Buffer Size and Output Buffer Abbr, which are set to 2 MiB by default and are not editable.

When Advanced Toggle is off:

When Advanced Toggle is on:

After clicking Save, a toast message indicates that the storage creation is in process.

Upon completion, another toast message confirms the successful creation of the storage.

Creating an AWS Storage:

Follow these steps for creating an AWS storage:

Name: Provide a name for the AWS storage.
Storage Option: This field is disabled, preventing the user from making modifications.
Cloud URL Path: Specify the service URL of the cloud storage. This is a required field with the default value of “http://s3.amazon.com”.
Access Key: Enter the AWS cloud storage access key, allowing only alphanumeric characters. Validation tests have been implemented to ensure that the user enters the correct access key.

Secret Key: Specify the AWS storage's secret key, allowing both alphanumeric and special characters.
AWS User Token (optional): Specify the user token for AWS storage, allowing alphanumeric and special characters.
Comments (optional): Add any comments for future reference.
Secure Connection: Choose the storage connection type (HTTP or HTTPS).

The following fields are disabled during AWS storage creation with default settings:

Cloud Object Dimensions - 512
Cloud Object Abbreviation - MiB
AES-256 Cloud Encryption
Total Media Maximum - 2000

The sole modification on the storage screen when choosing an AE server over a dedup server is the disabling of the Bucket Names model. Instead, a new Secure Connection modal appears, providing the user with the option to select either HTTP or HTTPS.

After clicking Save, a toast message indicates that the storage creation is in process.

Upon completion, another toast message confirms the successful creation of the storage.

Creating an S3 Compatible Storage:

The types of S3 compatible storages supported by the AMBAR server are Wasabi, Oracle, Zadara, and MinIO. Follow these steps for creating S3-compatible storage:

Name: Provide a name for the S3-compatible storage.
Auth Service Hostname: Specify the hostname of the S3-compatible storage, automatically trimming http or https.
Access Key: Enter the AWS cloud storage access key, allowing only alphanumeric characters. Validation tests have been implemented to ensure that the user enters the correct access key.
Cloud URL Path (optional): Specify the S3 storage's service URL.
Region (optional): Provide the S3 storage region.
Comments (optional): Add any comments for future reference.
Secure Connection: Choose the storage connection type (HTTP or HTTPS).

The following fields are disabled during S3 compatible storage creation with default settings:

Cloud Object Size - 512
Cloud Object Abbr - MiB
Maximum Total Media - 2000

After clicking Save, a toast message indicates that the storage creation is in process.

Upon completion, another toast message confirms the successful creation of the storage.

Creating an Azure Storage:

Follow these steps for creating an Azure storage:

Name: Provide a name for the Azure storage.
Account Name: Specify the Azure cloud storage account name.
Account Key: Enter the account key for Azure cloud storage, allowing alphanumeric and special characters.
Storage Options: Choose the Azure storage access tier. It include:
- Hot Tier: An online storage tier tailored for data with frequent access or modifications.
- Cool Tier: An online storage tier tailored for data with rare access or modifications.
Comments (optional): Add any comments for future reference.

If the Advanced toggle is selected, additional fields appear for Secure Connection and Bucket Names, which are disabled and cannot be modified. The fields below also have their respective default values.

Cloud Object Size - 100
Cloud Object Abbr - MiB
Maximum Total Media - 2000

When Advanced Toggle is on:

When Advanced Toggle is off:

User Guide for Dedup Server Backup Set Creation:

This user guide aims to explain all the necessary fields for creating a backup set when utilizing the dedup server.

Creating a Backup Set:

Follow these steps to create a backup set:

Select Backup Server: Choose a backup server from the menu to handle backups. All registered backup servers in the Zmanda Management Console (ZMC) are listed here.
Select Storage: Choose a storage device where you want to save your backups.
Select Schedule (optional): If desired, select an ideal schedule plan for backup runs. This option enables you to specify the backup start time.
Name: Specify a unique name for the backup set. Alphanumeric characters are permitted, with no spaces. The name must have five characters and cannot exceed 64 characters. The only special characters allowed are '-', '_', and '.'. If any of these validations fail, an error message displaying Invalid Backupset name will appear.
Description (optional): Provide a description to serve as a reminder of the included backups or configuration details.
Comments (optional): Add any comments for future reference.