Backup and Restore

Administrators can back up and restore an TQ Data Foundation workspace either by using the built-in Backup and Restore Utility or by manually backing up and restoring the workspace files. These two methods are described in more detail in the sections below.

The Backup and Restore Utility can back up Data Foundation while the system is online and operational; while the manual backup process requires Data Foundation to be shut down while its files are backed up. Both methods require the Data Foundation web application to be restarted when being restored from a backup.

Although these two methods can both be used to back up and restore the same Data Foundation server at different times, their artifacts are not interchangeable. That is, backups created by the Backup and Restore Utility cannot be used by the manual restore process, and vice versa.

Administrators can also back up and restore Explorer using these same options.

Backup and Restore Utility

Administrators can use the Backup and Restore Utility to back up and restore the Data Foundation workspace and its collections. Please test this utility in a development environment to get familiar with the process before using it in a production environment. This utility should not replace your organization’s existing backup strategy; it should be used to supplement and enhance the existing strategy.

Important notes

  • When restoring a workspace, the workspace performing the restore must match the workspace that originally generated the backup in two ways:

    • The two workspaces must be configured to use the same database type (e.g. Shared TDB or Data Platform)

    • The two workspaces must have the same major and minor version numbers; but the patch version numbers can differ. Please do not upgrade to a new version of Data Foundation prior to restoring from a backup, this will not work. Upgrade after the restore.

  • Each backup is created as a zip file. The zip file can be either downloaded to the user’s local machine or stored as an object in an Amazon S3 bucket. See External System Integration Management Admin Page and S3 Configuration Section.

  • Everything in the workspace is backed up and restored, with the exception of the read-only Data Foundation system files.

  • During the restore process, all existing TDB databases in the workspace will be deleted and replaced by only those TDB databases present in the backup file.

  • Data Platform-backed workspaces can be backed up and restored using this utility, but the restore process requires manual intervention. See the discussion below.

  • Once a restore is in progress, do not navigate away from the Backup/Restore page or shut down the server. It may take a while to perform the restore, so allow it to finish. If an error occurs, it may be necessary to restart the web server.

Utility Backup

Administrators can use the Backup and Restore Utility to back up the Data Foundation workspace via either the Backup/Restore page or an API call; and either method can be used to download a backup file or store the file in an Amazon S3 bucket.

The backup process is straightforward: An administrator requests a backup and the utility generates a zip file containing the requested backup.

Backup/Restore Page

Administrators can use the Backup/Restore page to generate a backup using the Data Foundation UI.

Download

To download the backup file, leave the backup-related Amazon S3 settings unconfigured and press the Create Local Backup button. The backup file will be created and downloaded like any other browser download.

Amazon S3 Bucket

To store the backup file in an Amazon S3 bucket, configure the backup-related Amazon S3 settings. (See External System Integration Management Admin Page and S3 Configuration Section.) If the appropriate Amazon S3 settings are configured, the Backup/Restore page will display the name of the target Amazon S3 bucket. If the settings are not completely configured, the Backup/Restore page will display links to the appropriate Data Foundation pages for configuring those settings. Once the settings are configured, press the Create Backup button. The backup file will be created and uploaded to the configured Amazon S3 bucket with a time-stamped object name that will look something like this: TopBraid-EDG-Studio-Backup-20220615T195844Z.zip.

To verify the checksum of a downloaded backup zip file, execute one of the following commands, substituting the appropriate filename, and compare the result to the checksum displayed alongside the file on the Server Administration Backup/Restore page:

Windows Command Prompt:

certutil -hashfile <filename> SHA256

Linux shell:

sha256sum <filename>

API

Administrators can use the Backup and Restore API to generate a backup using any HTTP client. This allows backups to be created using scripts that can be executed automatically by scheduled jobs and the like.

More details about the Backup and Restore API can be found in the Data Foundation online OpenAPI documentation.

Download

To download a newly-generated backup file, use an HTTP GET request similar to this curl command:

curl --request GET \
  http://localhost/tbl/backup \
  --header "Accept: application/zip" \
  --header "Authorization: Basic xxxxxxxxxxxxxx" \
  --output backup.zip

Amazon S3 Bucket

To store a newly-generated backup file in an Amazon S3 bucket, use an HTTP POST request similar to this curl command:

curl --request POST \
  http://localhost/tbl/s3-backups \
  --header "Accept: application/json" \
  --header "Authorization: Basic xxxxxxxxxxxxxx"

Utility Restore

Administrators can use the Backup and Restore Utility to restore a previously backed up Data Foundation workspace via either the Backup/Restore page or an API call; and either method can be used to upload the backup file to be used for the restore or fetch the file from an Amazon S3 bucket.

To restore an Data Foundation workspace, an administrator initiates the restore with a specified backup file, either an uploaded file or a file stored in an Amazon S3 bucket. This backup file is staged on the Data Foundation server and, in most cases, a restart of the Data Foundation web application is automatically initiated. Once Data Foundation is restarted, it will wipe clean the existing workspace and replace it with the contents of the staged backup file. When Data Foundation has completed its initialization, it will have been restored to the state it was in when the backup file was created.

In some cases, initiating a restore will not automatically restart the Data Foundation web application, and the entire web server must be manually restarted:

  • EDG Studio: EDG Studio does not support restarting the web application. The entire web server must be manually restarted.

  • Windows server: Data Foundation cannot be restarted when the web server is running on Windows. The entire web server must be manually restarted.

  • Data Platform: The restoration of a Data Foundation workspace running on Data Platform requires some manual intervention; so it cannot be automatically restarted. See the discussion below.

Note

When initiating a restore with the Backup and Restore Utility, since the user-specified backup file must be staged on the Data Foundation server, there must be enough disk space available to the server to hold the backup file. If there is not enough disk space, the restore will fail.

Backup/Restore Page

Administrators can use the Backup/Restore page to initiate a restore using the Data Foundation UI.

Upload

To upload a backup zip file for restore, press the Choose File button and select the desired file from the local workstation; then press the Restore button. The backup file will be uploaded and staged on the Data Foundation server and a system restart initiated.

Amazon S3 Bucket

To use a backup zip file stored in an Amazon S3 bucket for restore, configure the backup-related Amazon S3 settings. Once the appropriate Amazon S3 settings are configured, the Backup/Restore page will display a list of the objects in the configured Amazon S3 bucket. Press the Restore button next to the desired file in the list. That backup file will be downloaded from Amazon S3 and staged on the Data Foundation server and a system restart initiated.

API

Administrators can use the Backup and Restore API to initiate a restore using any HTTP client. This allows restores to be initiated using scripts that can be executed automatically by scheduled jobs and the like.

More details about the Backup and Restore API can be found in the Data Foundation online OpenAPI documentation.

Upload

To upload a backup file for restore and initiate a system restart, use an HTTP POST request similar to this curl command:

curl --request POST \
  http://localhost/tbl/backup \
  --header "Accept: */*" \
  --header "Content-Type: multipart/form-data" \
  --header "Authorization: Basic xxxxxxxxxxxxxx" \
  --include \
  --form "file=@TopBraid-EDG-Studio-Backup-20220621T041100Z.zip;type=application/zip"

Amazon S3 Bucket

To use a backup zip file stored in an Amazon S3 bucket for restore and initiate a system restart, use an HTTP POST request similar to this curl command:

curl --request POST \
  http://localhost/tbl/s3-backups/TopBraid-EDG-Studio-Backup-20220621T041100Z.zip/backup \
  --header "Accept: application/json" \
  --header "Authorization: Basic xxxxxxxxxxxxxx"

To list the backup zips file stored in an Amazon S3 bucket, use an HTTP GET request similar to this curl command:

curl --request GET \
  http://localhost/tbl/s3-backups \
  --header "Accept: application/json" \
  --header "Authorization: Basic xxxxxxxxxxxxxx"

The keys of the objects in the returned list can be used in the path of the POST request above to indicate which backup file is to be used in the restore.

Data Platform

Administrators can use either the Backup and Restore Utility or the Backup and Restore API to initiate a restore of a set of clustered Data Platform Data Foundation workspaces; but this will only stage the backup-related files on the Data Foundation web server from which the restore was initiated. It will not restart any of the clustered Data Foundation workspaces, because Data Platform restores must be completed manually.

When a Data Platform workspace is backed up using the Backup Utility or API, the produced zip file contains two nested zip files:

  • backup.zip contains the files from the Data Foundation node that originally performed the backup. These files will be the same across all the Data Foundation nodes.

  • dp-server-patch-logs.zip contains the Data Coordinator patch logs, which will be used to restore the Data Coordinator server.

Use one of the current Data Foundation nodes (or a freshly-installed Data Foundation node paired with a freshly-installed Data Coordinator server) to initiate a restore, which will stage the necessary backup files; then follow these steps:

  1. Shut down all the Data Foundation nodes; while leaving the Data Coordinator server running.

  2. Restart the Data Foundation node that initiated the restore, automatically triggering its restoration from the locally staged backup.zip file.

  3. Once this Data Foundation node is successfully restored, shut it down again. (At this point, the Data Coordinator server will also have been restored.)

  4. Replace the other Data Foundation nodes’ edg/workspace directory trees with the restored node’s edg/workspace directory tree.

  5. Restart all the Data Foundation nodes.

See also

For additional details, see Data Platform.

Manual Backup and Restore

Administrators can manually back up and restore an Data Foundation workspace’s files. The steps necessary depend on the workspace’s configured datastore:

  • TDB (Shared or Separate)

  • Data Platform

Manual Backup

TDB

Steps to back up an Data Foundation workspace that is configured to use Separate or Shared TDB:

  1. Shut down the Data Foundation web server.

  2. Back up the entire Data Foundation workspace directory tree, edg/workspace; e.g. by adding it to a zip or tar. This backup will include the workspace’s TDB databases.

  3. Store this backup as appropriate.

  4. Once the backup is complete, restart the Data Foundation web server.

Data Platform

Steps to back up a cluster of Data Foundation workspaces that are configured to use Data Platform:

  1. Shut down all the Data Foundation nodes.

  2. Shut down the Data Coordinator server.

  3. Back up the entire Data Foundation workspace directory tree, edg/workspace, of one Data Foundation node; e.g. by adding it to a zip or tar.

  4. Back up the Data Coordinator server’s --base (patch logs) directory; e.g. by adding it to a zip or tar.

  5. Store these two backups together as appropriate.

  6. Once the backup is complete, restart the Data Coordinator server.

  7. Restart all the Data Foundation nodes.

Manual Restore

TDB

Steps to restore an Data Foundation workspace that is configured to use Separate or Shared TDB:

  1. Shut down the Data Foundation web server.

  2. Replace the entire Data Foundation workspace directory tree, edg/workspace, with the contents of a TDB workspace backup.

  3. Restart the Data Foundation web server.

Data Platform

Steps to restore a cluster of Data Foundation workspaces that are configured to use Data Platform:

  1. Shut down all the Data Foundation nodes.

  2. Shut down the Data Coordinator server.

  3. Replace the entire Data Foundation workspace directory tree, edg/workspace, of each Data Foundation node with the contents of a Data Platform node workspace backup.

  4. Replace the Data Coordinator server’s --base (patch logs) directory with the contents of the corresponding Data Coordinator backup.

  5. Restart the Data Coordinator server.

  6. Restart all the Data Foundation nodes.