Bring Your Own Software With Rescale Software Publisher (Beta)

Overview

Rescale is happy to provide Rescale Software Publisher in Beta Release. We welcome customers to try out this new capability and provide us feedback. Please contact your Rescale account team for details if you are interested.

Rescale Software Publisher is a new product feature that allows organizations to bring their own software to the Rescale cloud platform. It provides customers the following benefits:

On-demand publishing to Rescale’s software catalog- fast without waiting on other teams to process your request
You can now bring a wide variety of software to use in Rescale- custom, proprietary, commercial, open source, and personal utilities you may have developed.
Easy to use and does not require advanced IT skills, coding, or special cloud administrator access to infrastructure
Centralized visibility, access, and administration for all your HPCHPC stands for High-Performance Computing. It refers to the ... software applications and packages

The following new features and capabilities are highlighted for Rescale Software Publisher:

The Sandbox is a new concept introduced by Publisher, a temporary working environment to install and test software prior to publishing.
The command line interface extends the existing rescale-cli, with a new ‘spub’ option to perform publishing commands.
There are also new visibility settings when publishing software, that are described later in this document. Basically when you are ready to share your published software with others, you can share it at the workspace level, or at the organization level which would be visible to all workspaces.

The diagram below shows an overview of the publishing process:

There are known limitations and missing functionality in the current Beta version of Rescale Software Publisher. These are outlined at the end of this document.

Pre-requisites:

The user you log in as, will need admin role privileges- either at the Workspace level or Organizational level.
- Organizational admins can publish and set visibility within any workspace within their organization
- Workspace admins can only publish and set visibility to workspaces they manage
You will need someone from Rescale to enable your company / organization to have access to Rescale Software Publisher. This is part of the on-boarding process for new Beta customer users. Reach out to your Rescale team for details.

UI access:

As an admin user, go to the admin portal and you will see a “Tools” menu at the top. Under there, select “Software Publisher”.

If you do not see this new Tools menu from the admin portal, your Rescale user account may not have the necessary permissions configured.

You will see a list of local published software. Or it may be empty. The early release version does not currently have an option to search or sort columns. To find your software in a long list, use the browser find / search option on the page.

New Software Tile

To publish new software, select the “+ add software” button above the list table. You also need to create the software tile or analysis first before you can create any versions. The CLI is an exception (see CLI section below). You can pick any version method you want, as long as the version name does not already exist for that software. (examples – “1.0”, “2a”, “version1”, “20”, etc.)

Under the configuration section, fill out the fields for the name and description of the new software to add to the catalog. Select batch or workstation for the software you will publish, if your intended use of the published software is via jobs or via interactive Workstations. The thumbnail image is optional to provide- the size must be under 100K.

Here is an example from the UI after creating the initial software tile, before publishing any versions:

New Software Version

Once you add your software tile, then you can publish one or more versions. Select the software tile you created from the list, and then click on the “Versions” icon. Select “+ Add Software Version”. A new window will appear with two selection options on the left for “Version Properties” and “Sandbox Setup”. Beginning with the Version Properties page, fill in the required fields. Optionally you can provide environment variables that your published software can reference:

Now navigate to the “Sandbox Setup” page. You can upload your software from this page, specify the operating system, and indicate if your software has any MPI dependencies or not. For the sandbox size, try to pick a size that is enough to install and test your software, but not too large as to waste unused space.

There is an optional command field on this page that is different from the command on the version properties page. This is meant to run a script to automate installing the software after it is uploaded. If you leave this command field empty, you can access the sandbox interactively to perform any necessary commands and configurations before publishing.

Create

When you complete filling out the information on both pages, click the “create” button:

You will see a new “status” window appear under versions. If you select it on the left side, you can monitor the sandbox creation progress:

Active

When sandbox setup is complete, you will see a new SSH section appear, and also a new “connect” button at the top. You can directly access your sandbox clicking on this button:

This virtual desktop environment is using Workstations underneath. To access a terminal window for shell / CLI options, select the terminal emulator button on the bottom:

If you chose to upload software when creating a new sandbox, the software should be placed in the mount base or mount point path location specified. There is an option in the UI when uploading files to select automatically uncompressing and extracting TAR files.

In this interactive mode, you can log in to your sandbox and perform commands via a SSH or terminal session.

You can run git() directly from your sandbox to retrieve software or other material, such as test data. Here is an example retrieving Tutorial data for a HPC software package (SU2):

You also have other options to move or transfer software to your Rescale sandbox environment, such as sftp().

It is highly recommended that software be tested prior to publishing! Here is an example via interactive model starting a sample test:

Publish

When you are ready to publish, you should copy or move all of the necessary software files under the mount point base before publishing, and clean up / remove any temporary files not needed to run the software.

From the UI, select the “Publish” button at the bottom of the status page. You will see the status value at the top of the page change to “Publishing” :

NOTE: this step can take a while, as it is doing several steps behind the scenes to make this software version available for use.

When your software version publish action is completed, you will see the version status updated to “Completed”:

Visibility

There is also a “Visibility” column when viewing a published version. There are multiple visibility settings that administrators can set to promote the software between stages after it is initially published. They are as follows:

Visibility Settings / Stages	Description
Draft	Initial state of starting a new version publishing session. Sandboxes exist in this publishing state. No physical snapshot exists yet, and software is not visible in the Rescale Catalog for the organization.
Admin Private	A physical snapshot is first created in this stage of publishing. The sandbox is removed. Software can only be viewed and accessed by Administrators from the catalog.
Workspace Private	This is one option to share your published software version with others. Promoting the visibility of published software to Workspace Private makes it available to the named workspaces you select. You must have either org admin privileges, or be a workspace admin for the workspaces you wish to share.
Organization Private	This is another option to share your published software version with others. Promoting the visibility of published software to Organization Private make the software visible to all members in your organization, across all workspaces.

From the UI, here is how you set the visibility of a published software from its initial state, which by default always starts with “Admin Private”. On the version -> status page:

Admin Private

When your software is published to the initial state, Admin Private, you as an administrator, and other administrators with the proper access in the workspace published, can access and use this new software tile or version from the Rescale software catalog when creating jobs. The visibility at this stage is limited to just the admins:

Details of new published tile:

After you test and validate this software version is working ok, you can now share it with others in your organization.

REMINDER: Always test and validate before making published software available to others!

There are two visibility options when you want to share your newly published version with others:

Publish visibility of the version to specific workspaces (workspace private)
Publish visibility across all workspaces in the organization (organization private)

Workspace Private

From the Versions page, select the version number you just published. At the top of the page next to the “Completed” status, select the menu and pick “workspace private” to promote the visibility of your published software version to a named list of workspaces. On the left side, select “Allowed Workspaces” and pick the workspaces you wish to share this software version with:

You can only select workspaces that you as an Administrator have access to.

Organization Private

You would follow similar steps as previously described in changing the visibility of a published software version to “organization private”. Note that you need to be in the role of an organizational administrator for this option to be selectable in the UI. If your user is just granted admin access to a workspace, you cannot set visibility of published software to organization private.

Command Line Interface

Publisher is accessible from the Rescale CLI via a new argument / option , ‘spub’. You first need to download the latest CLI package as a user and install it on your local machine.

Within the product UI, from your account login menu on the top right, select “User Profile Settings”. Then on the left side rail, select “APIApplication program interface (API) is a set of routines, pr...”. If you do not have an API key already generated, do that now by selecting “+ Add API Key”. You will need to provide this key value when using the CLI for publishing.

In the CLI section on that same page, click on the “Beta Release” tab, and select the platform you plan to run the CLI from:

CLI Install- Mac

When you download the CLI file from the UI, it is packaged as a *.dmg file for Mac OSX. OPen that file to begin the installation process. Follow the instructions provided:

The Rescale CLI is by default installed under /applications/Rescale CLI as the binary “rescale-cli”. You can either add this directory to your PATH, or run the rescale-cli within the installed directory location.

TIP: copy and paste your Rescale API key value, and software tile name, to a handy location for use with the CLI.

CLI Install – Windows

When you download the CLI file from the UI, it is packaged as an *.exe file for Windows x64 or x86. Open that file to begin the installation process. Follow the instructions provided:

The Rescale CLI is by default installed under C:\Users\[username]\AppData\Local\rescale-cli as the binary “rescale-cli.exe”. Run the command from Command Prompt or Windows Powershell

CLI Usage

There is no syntax usage difference in the Rescale CLI between the different client platforms.

Some getting started tips when using the Rescale CLI:

Each publisher command will start with this syntax: “rescale-cli -X https://yourplatform.rescale.com spub …”. Adjust the URL accordingly to the Rescale platform your organization is hosted from.
Each command line execution needs to include your Rescale API key, typically the end part of your command in the format: “-p <reallyLongHexaDecimalValue>”.
To operate on a software tile or version, use the “tile” argument to spub. There is no “version” option for spub to use.
The spub sub-command and the options to it both support help options via “-h” or “–help”

Here is a summary table showing the sub-command and options broken out in categories – object or “noun” and action or “verb” The logs option doesn’t quite fit in as a verb :

Root Command:	Top level sub-command:	Object level argument / option / parameter(noun)	Action level argument / option / parameter(verb)
rescale-cli	spub	tile	create
		sandbox	publish
		help	list
			connect
			logs
			upload-file
			suspend \| resume
			terminate
			delete (coming later!)

The command line interface for spub has help information that is context sensitive. For example you can view help content at a general level for ‘tile’, and view more detailed help content for a specific argument that tile supports, such as “tile create”.

Here is sample output of the help for the two main object options supported- tile and sandbox :

% ./rescale-cli -X https://platform-stage.rescale.com spub -h
usage: Rescale Client App spub [-h] {tile,sandbox} ...
positional arguments:
  {tile,sandbox}
named arguments:
  -h, --help             show this help message and exit
% ./rescale-cli -X https://platform-stage.rescale.com spub tile -h 
usage: Rescale Client App spub tile [-h] {create,publish} ...
positional arguments:
  {create,publish}
named arguments:
  -h, --help             show this help message and exit
% ./rescale-cli -X https://platform-stage.rescale.com spub sandbox -h
usage: Rescale Client App spub sandbox [-h] {connect,list,logs,resume,suspend,terminate,upload-file} ...
positional arguments:
  {connect,list,logs,resume,suspend,terminate,upload-file}
named arguments:
  -h, --help             show this help message and exit
%

There is also help context for the command line options that are actions or verbs. Here is one example showing help for “create” being used with the tile option:

% ./rescale-cli -X https://platform-stage.rescale.com spub tile create -h
usage: Rescale Client App spub tile create [-h]
                          [--visibility {admin-private,workspace-private,organization-private}]
                          [-s SETTINGS-FILE] [--silent-mode] [-f INPUT-FILE-PATHS [INPUT-FILE-PATHS ...]]
                          [-fid INPUT-FILE-IDS [INPUT-FILE-IDS ...]] [-p API-TOKEN]
named arguments:
  -h, --help             show this help message and exit
  --visibility {admin-private,workspace-private,organization-private}
                         admin-private|workspace-private|organization-private
  -s SETTINGS-FILE
  --silent-mode
  -f INPUT-FILE-PATHS [INPUT-FILE-PATHS ...]
  -fid INPUT-FILE-IDS [INPUT-FILE-IDS ...]
creds:
  -p API-TOKEN, --api-token API-TOKEN
                         The Rescale API token to use for authentication
%

When you create a new software version via the spub CLI, for either an existing software tile or a brand new one, you will be asked to provide answers to a series of prompts:

Software code
Software name
Software description
Software version name
Software version code
Software mount base
Software command
Number of environment variables to add
Sandbox coreA core processor, also known as a central processing unit (C... count
Sandbox OS family
Sandbox distro
MPI type
Installation command
Installation volume size in GB
Installation mount point

These are more options than you see via the UI. Some notes about these CLI prompts for creation:

Software code, and software version code, are internally used values in the product. Publisher will take those values and make them unique. We may remove these from the CLI in the future. For now, you can provide the same values the following pairs of fields:
- Software code = Software name
- Software version code = Software version name
Installation mount point can be the same value used as software mount base. Only provide a different value for installation mount point if you want the software in your sandbox to be installed in a different directory than the mount base. It is the software mount base that is used for the final publish of the software version, so be sure your software is placed under this mount base in your sandbox prior to publishing.

Here is example output of a spub create command:

you@rescale.com@You Rescale CLI % rescale-cli -X https://platform-stage.rescale.com spub tile create -p 9298f4861c760156b87b94c664f026141dcb3ddf

2023-04-17 16:45:12,391 – Authenticated as dfierro@rescale.com

Please enter the software code:

You-spub-tst01-SU2

Please enter the software name:

You-spub-tst01-SU2

Please enter the software description:

My software description for this version

Please enter the software version name:

1.0

Please enter the software version code:

1.0

Please enter the software mount base:

/program/spubtst/SU2

Please enter the software command:

echo test software command

Please enter the number of environment variables you’d like to add:

Please enter the sandbox core count:

Please enter the sandbox walltime:

Please enter the sandbox os family:

linux

Please enter the sandbox os distro:

centos

Please enter the sandbox mpi type or leave blank for none:

none

Please enter the installation command:

echo test installation command

Please enter the installation volume size in GB:

Please enter the installation mount point:

/program/spubtst/SU2

2023-04-17 16:50:03,017 – Sandbox: Syncing software

2023-04-17 16:50:03,668 – created dev analysis wQVseb

2023-04-17 16:50:04,107 – created dev analysis version iXVkhb

2023-04-17 16:50:04,108 – created dev analysis version iXVkhb env

2023-04-17 16:50:04,130 – Uploading Files

2023-04-17 16:50:04,136 – Uploading: /var/folders/vx/6777877n4p517_ymy_224gwc0000gn/T/rescale-cli-work8858473315941349202/settings.json

2023-04-17 16:50:04,137 – Uploading settings.json:

2023-04-17 16:50:05,290 – ##############################| 848B / 848B

2023-04-17 16:50:08,285 – Sandbox: Saving Sandbox

2023-04-17 16:50:11,586 – Sandbox Ysnugb: Saved

2023-04-17 16:50:15,455 – Sandbox Ysnugb: Waiting for status INSTALLATION_COMPLETE

2023-04-17 16:51:15,840 – Sandbox Ysnugb: Status – SANDBOX_STARTING

2023-04-17 16:52:16,237 – Sandbox Ysnugb: Status – SANDBOX_STARTING

2023-04-17 16:53:16,681 – Sandbox Ysnugb: Status – SANDBOX_STARTING

2023-04-17 16:54:17,053 – Sandbox Ysnugb: Status – SANDBOX_STARTING

2023-04-17 16:55:17,455 – Sandbox Ysnugb: Status – SANDBOX_STARTING

2023-04-17 16:56:17,827 – Sandbox Ysnugb: Status – SANDBOX_STARTING

2023-04-17 16:57:18,232 – Sandbox Ysnugb: Status – INSTALLATION_COMPLETE

2023-04-17 16:57:18,232 – Sandbox creation completed.

Please perform any manual installation steps on the sandbox before running `spub tile publish –sandboxId Ysnugb` to publish the software.

Sandbox connection info can be retrieved with `spub sandbox connect –sandbox-id Ysnugb`.

you@rescale.com@You Rescale CLI %

Software Portability and Helper Functions

When bringing your own software to the Rescale platform, a best practice is to make your software application, package, or tool as portable as possible. This is because the Rescale platform is architected so that the software which runs on our managed compute clusters is de-coupled from the underlying hardware and operating system. This allows customers maximum flexibility when choosing core hardware options and target environments to run their high performance compute workloads. If your HPC software becomes too intertwined with the underlying operating system, or has other hard-coded dependencies, over time your software solution will become more brittle, and simple updates like an OS patch may require you to re-install and test your software each time such changes occur.

This is why Rescale sandboxes by design do not allow super user or administrator privileges such as the root user account or sudo access, as well as for security reasons. However it is still possible you can install, run, and test your software within the Rescale controlled hosted environment successfully without requiring such elevated access. With every sandbox that is created, Rescale provides a set of utilities that help you with installing and testing your software so it is portable and flexible. These helper functions address many common install commands and packages that typically require root or sudo access, from yum install to python pip . Below is an overview of the Rescale helper functions and utilities that you can use to ensure your published software is portable and follows recommended best practices.

Cookie	Duration	Description
AWSALBCORS	7 days	This cookie is managed by Amazon Web Services and is used for load balancing.
cookielawinfo-checkbox-advertisement	1 year	Set by the GDPR Cookie Consent plugin, this cookie is used to record the user consent for the cookies in the "Advertisement" category .
cookielawinfo-checkbox-analytics	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional	11 months	The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy	11 months	The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.

Cookie	Duration	Description
__cf_bm	30 minutes	This cookie, set by Cloudflare, is used to support Cloudflare Bot Management.
bcookie	2 years	LinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser ID.
lang	session	LinkedIn sets this cookie to remember a user's language setting.
lidc	1 day	LinkedIn sets the lidc cookie to facilitate data center selection.
player	1 year	Vimeo uses this cookie to save the user's preferences when playing embedded videos from Vimeo.

Cookie	Duration	Description
AWSALB	7 days	AWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.
sync_active	never	This cookie is set by Vimeo and contains data on the visitor's video-content preferences, so that the website remembers parameters such as preferred volume or video quality.

Cookie	Duration	Description
_ga	2 years	The _ga cookie, installed by Google Analytics, calculates visitor, session and campaign data and also keeps track of site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognize unique visitors.
_gat_UA-32985745-1	1 minute	A variation of the _gat cookie set by Google Analytics and Google Tag Manager to allow website owners to track visitor behaviour and measure site performance. The pattern element in the name contains the unique identity number of the account or website it relates to.
_gcl_au	3 months	Provided by Google Tag Manager to experiment advertisement efficiency of websites using their services.
_gid	1 day	Installed by Google Analytics, _gid cookie stores information on how visitors use a website, while also creating an analytics report of the website's performance. Some of the data that are collected include the number of visitors, their source, and the pages they visit anonymously.
CONSENT	2 years	YouTube sets this cookie via embedded youtube-videos and registers anonymous statistical data.
utm_campaign	past	Google Ad Services sets this cookie to store session campaign value if present.
utm_content	past	This cookie is used for storing the session content value if present.
utm_source	past	This cookie is used to record from where the visitor came to the website orginally. This information is used by the website operator to know the efficiency of their marketing.
utm_term	past	This cookie is used to record from where the visitor came to the website orginally. This information is used by the website operator to know the efficiency of their marketing.
vuid	2 years	Vimeo installs this cookie to collect tracking information by setting a unique ID to embed videos to the website.

Cookie	Duration	Description
_fbp	3 months	This cookie is set by Facebook to display advertisements when either on Facebook or on a digital platform powered by Facebook advertising, after visiting the website.
_mkto_trk	2 years	This cookie, provided by Marketo, has information (such as a unique user ID) that is used to track the user's site usage. The cookies set by Marketo are readable only by Marketo.
fr	3 months	Facebook sets this cookie to show relevant advertisements to users by tracking user behaviour across the web, on sites that have Facebook pixel or Facebook social plugin.
IDE	1 year 24 days	Google DoubleClick IDE cookies are used to store information about how the user uses the website to present them with relevant ads and according to the user profile.
personalization_id	2 years	Twitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.
test_cookie	15 minutes	The test_cookie is set by doubleclick.net and is used to determine if the user's browser supports cookies.
utm_medium	past	This cookie is used to record from where the visitor came to the website orginally. This information is used by the website operator to know the efficiency of their marketing.
VISITOR_INFO1_LIVE	5 months 27 days	A cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.
YSC	session	YSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.
yt-remote-connected-devices	never	YouTube sets this cookie to store the video preferences of the user using embedded YouTube video.
yt-remote-device-id	never	YouTube sets this cookie to store the video preferences of the user using embedded YouTube video.
yt.innertube::nextId	never	This cookie, set by YouTube, registers a unique ID to store data on what videos from YouTube the user has seen.
yt.innertube::requests	never	This cookie, set by YouTube, registers a unique ID to store data on what videos from YouTube the user has seen.

Platform Overview

Bring Your Own Software With Rescale Software Publisher (Beta) – Getting Started Guide