↔️

Form Data Transfer

💡
Launch Form Data Transfer to transfer data from Box, DropBox, or Google Drive to the Form Bio platform.
The Form Data Transfer tool can be used to transfer files from a user’s cloud-based data storage to the From Bio platform. Form Data Transfer incorporates Rclone to manage access to files stored in the cloud. Use this tool to transfer compatible data files from Box, DropBox, or Google Drive.

Form Bench Walkthrough

  1. Navigate to the Form Data Transfer tab in the Form Bench launcher.
  2. To launch a Form Data Transfer instance, you must provide the following fields. Default values (where provided) are suitable for most use-cases.
    1. Name: Provide a name for your instance.
    2. Session Duration: Determine how long your session will run for.
      3. image

Connect to public S3 bucket

To connect to a public S3 bucket e.g. s3://genome-idx that does not require credentials.

  1. Add a remote Configuration - Click “Configurations” in the left nav panel
    1. Name: s3-public
    2. Type: Amazon S3 Compliant Storage Providers
  2. Click “Next” Configuration options (all other configuration can be left blank/default)
    1. Choose S3 Provider: AWS
image
image

Transfer S3 bucket Data

  1. Open “Explorer” and click “+” on the right side
  2. Enter remote s3-public:${public-s3-bucket-name} e.g. s3-public:genome-idx
  3. Browse and explore the bucket and select the data you want to transfer
image
image

Connect to Box / DropBox / Google Drive

  1. Open a browser window and log into your Box account, DropBox account, or Google Drive.
  2. Install RClone on your computer:
    1. On Windows, RClone can be installed using Windows package manager (Wignet) - https://rclone.org/install/#windows
    2. ‘Script Installation’ method is recommended on Linux, MacOS - https://rclone.org/install/#script-installation
  3. Open a command window
  4. Run rclone authorize <cloud_service> and follow the instructions.
    1. Run rclone authorize dropbox if using DropBox, or rclone authorize drive if using Google Drive.
      2. image
    2. Copy the JSON token string from the command line. It will be used later in step 8.
      1. For example, it will look something like this: {"access_token":"7vnKRaYPncpvXaaSZqXcEdABCcC82HAn","token_type":"bearer","refresh_token":"1UWHvT06JTJOoh9YyEOIGHtajFgoXYvrY9lKhk6BwIjlwh5lIpQWEGkOOxbWyQXU","expiry":"2023-12-08T14:51:01.033697-08:00"}
  5. Navigate to the Form Data Transfer tab in the Form Bench launcher and launch a session.
  6. Once the session has launched, click on ‘Configs’ on the left side and then click ‘Create a New Config’.
    7. image
  7. Name the config and then select ‘box’ if using Box, ‘DropBox’ if using DropBox, or ‘drive’ if using Google Drive. Click next.
    8. image
  8. Leave all fields empty. Select the ‘Edit Advanced Options’ checkbox at the bottom. Click next.
    9. image
  9. Paste the JSON token string (from the command line, see step 4) into the first field, ‘Auth Access Token as a JSON blob’. This step is the same whether you’re using Box, DropBox, or Google Drive. The rest of the fields do not need to be modified. Click next.
    10. image
  10. Click on ‘+’ on the right and enter the name of your Box config (see step 7). Click open. This loads your Box (or DropBox or Google Drive) files.
    11. image
  11. Drag and drop any files between the two sources.
image

Error Handling

  • When running a Form Data Transfer instance for the first time, there is a possibility that you may be prompted to sign in to an account. No sign in is required, click “Login” without filling in any of the credentials.
image
  • Ignore any error messages shown during the configuration, such as this one. If you come across an error such as this one, ignore it and click on ‘Explorer’ to continue.
    • image

Connect to Google Drive (using Google Cloud Platform)

  1. Create a Cloud project if you don’t have one (this doesn't require any credit card information). Search for Google Drive API and enable it.
  2. Go to APIs & Services and CREATE CREDENTIALS, select OAuth client ID, select Desktop app for Application type (CONFIGURE CONSENT SCREEN if you haven’t)
    1. Select External for User Type, then Create
    2. In App name, you can put anything but it’s recommended to choose an easy to remember name
    3. Enter your email into User support email
    4. Enter your email into Developer contact information
    5. SAVE AND CONTINUE until Test users
    6. ADD USERS for Test users, use your email (this must be done to ensure authentication)
  3. Download the credential JSON
  4. Open the credential JSON and copy the client_id value, the credential JSON should look like this when opened:
{"installed":{"client_id":"xxxxxx.apps.googleusercontent.com","project_id":"xxx-xxx-xxx","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_secret":"xxx-xxxxxx","redirect_uris":["http://localhost"]}}
  1. Paste the client_id to the below URL
https://accounts.google.com/o/oauth2/auth?
    client_id=<client_id>&
    redirect_uri=http://localhost&
    response_type=code&
    scope=https://www.googleapis.com/auth/drive
  1. Open the URL, allow the app to access to your Google account, it will automatically redirect to the localhost:
http://localhost/?code=<authorization_code>&scope=https://www.googleapis.com/auth/cloud-platform
  1. Copy the authorization code from the above localhost URL that goes after http://localhost/?code= and before the &scope
  2. Open Cloud Shell
image
  1. Copy the client_id, client_secret from the credential JSON, and the authorization_code from the localhost URL into the below curl command:
curl -d "client_id=xxxxxx.apps.googleusercontent.com" \
    -d "client_secret=xxx-xxxxxx" \
    -d "code=<authorization_code>" \
    -d "redirect_uri=http://localhost" \
    -d "grant_type=authorization_code" \
    https://oauth2.googleapis.com/token
  1. Copy the curl command and execute it in the Cloud Shell, you will receive an access token that should look something like:

{"access_token":"xxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxx","expiry":"2024-03-30T01:41:08.10626+07:00"}

  1. Copy the Project number in the screen of Step 7 and the access token in Step 9.
    1. Paste the Project number to the option Google Application Client Id
    2. Check Edit Advanced Options and click Next
    3. Paste the access token to the OAuth Access Token as a JSON blob.
  2. Now, you should be able to see the Google Drive directory in Form Bench RClone

Connect to Google Cloud Storage (GCS)

  1. Go to https://console.cloud.google.com/ and navigate to the project that has the Cloud Storage you want to connect via RClone
image
  1. Copy the Project number, this will be used for GCS authentication in RClone configuration
  2. Run rclone config and select Google Cloud Storage (option 16 for v1.66.0) when creating a new remote
  3. Leave all blanks except project_number . Enter the Project number in Step 2.
image
  1. When Edit advanced config? appears, enter n
  2. Next, authenticate via web browser with y when Use web browser to automatically authenticate rclone with remote? appears
image
  1. Select the account that holds the Google Cloud Storage and allow RClone to access your Google Account, you should see a Success! after this
  2. In your terminal, RClone should show something like this:
image
  1. In your Form Bench RClone, copy the project_number in the terminal and paste in to the Project number. Optional - needed only for list/create/delete buckets - see your developer console.
  2. Next, check the Edit Advanced Options and click Next
  3. Paste the token {"access_token":"xxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxx","expiry":"2024-03-30T01:41:08.10626+07:00"} to OAuth Access Token as a JSON blob.
image
  1. Now, you should be able to see the Google Cloud Storage directory in Form Bench RClone

Connect to GCS (using Google Cloud Platform)

This is very similar to how you connect your Google Drive, the only difference is how you generate the authorization code, thus change the scope accordingly.

  1. Go to APIs & Services and CREATE CREDENTIALS, select OAuth client ID, select Desktop app for Application type (CONFIGURE CONSENT SCREEN should be already done, if not please go through the below)
    1. Select External for User Type, then Create
    2. In App name, you can put anything but it’s recommended to choose an easy to remember name
    3. Enter your email into User support email
    4. Enter your email into Developer contact information
    5. SAVE AND CONTINUE until Test users
    6. ADD USERS for Test users, use your email (this must be done to ensure authentication)
  2. Download the credential JSON
  3. Open the credential JSON and copy the client_id value, the credential JSON should look like this when opened:
{"installed":{"client_id":"xxxxxx.apps.googleusercontent.com","project_id":"xxx-xxx-xxx","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_secret":"xxx-xxxxxx","redirect_uris":["http://localhost"]}}
  1. Paste the client_id to the below URL
https://accounts.google.com/o/oauth2/auth?
    client_id=<client_id>&
    redirect_uri=http://localhost&
    response_type=code&
    scope=https://www.googleapis.com/auth/cloud-platform
  1. Open the URL, allow the app to access to your Google account, it will automatically redirect to the localhost:
http://localhost/?code=<authorization_code>&scope=https://www.googleapis.com/auth/cloud-platform
  1. Copy the authorization code from the above localhost URL that goes after http://localhost/?code= and before the &scope
  2. Open Cloud Shell
image
  1. Copy the client_id, client_secret from the credential JSON, and the authorization_code from the localhost URL into the below curl command:
curl -d "client_id=xxxxxx.apps.googleusercontent.com" \
    -d "client_secret=xxx-xxxxxx" \
    -d "code=<authorization_code>" \
    -d "redirect_uri=http://localhost" \
    -d "grant_type=authorization_code" \
    https://oauth2.googleapis.com/token
  1. Copy the curl command and execute it in the Cloud Shell, you will receive an access token that should look something like:

{"access_token":"xxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxx","expiry":"2024-03-30T01:41:08.10626+07:00"}

  1. Copy the Project number in the screen of Step 7 and the access token in Step 9.
    1. Paste the Project number to the option Project number. Optional - needed only for list/create/delete buckets - see your developer console.
    2. Check Edit Advanced Options and click Next
    3. Paste the access token to the OAuth Access Token as a JSON blob.
  2. Now, you should be able to see the GCS directory in Form Bench RClone