Using the EMBL-EBI Cloud Portal¶
Setting up up your Cloud Profile¶
Your Cloud Profile contains all the information required to deploy your applications to a given cloud provider. It’s subdivided in three sections: Configurations , Cloud Credentials, and Deployment Parameters. Cloud Credentials are combined with a set of Deployment Parameters and a SSH public key to give the Configuration required to deploy a certain application in the cloud provider of choice.
Interacting with clouds, being them private or public, usually requires authentication. Many of the tools used to interact with cloud APIs rely on environment variables to source authentication tokens, with the clear advantage of not requiring credentials to lie in some file with your filesystem. Terraform, the open-source tools that the EMBL-EBI Cloud Portal uses to provision the required virtual infrastructure, is not an exception to this.
Each cloud provider requires a set of cloud credentials. While it’s possible somebody has shared some credentials with you already via a Team, you can find the instructions to insert you personal Cloud Credentials below.
Amazon Web Service (AWS)¶
Add your Cloud Credentials like this:
You can find this information in you AWS user page
under the section
USERS (or via the official docs).
AWS_DEFAULT_REGION is optional and can be omitted if the
region is selected in another way (i.e. via a Terraform variable specificied in the Deployment Parameters). A list of the
AWS regions can be found here.
Cloud Credentials like this:
In order to use Terraform with Azure it is required to create a
Service Principal via the Azure portal.
The Terraform documentation provide an extensive explanation on how to obtain this in its official documentation.
Cloud Credentials like this:
The specific set of values for your OpenStack provider might be slightly different and is contained in the
OpenStack RC file, which is project-specific and contains the credentials used by all the OpenStack services.
You can download the
OpenStack RC file file from the OpenStack
dashboard as an administrative user or any other user.
- Log in to the OpenStack dashboard.
- Choose the
Projectfor which you want to download the OpenStack RC file
API Access. (In older version of OpenStack click
Compute—>``Access & Security``)
Download OpenStack RC Fileand save the file.
Google Cloud Platform (GCP)¶
Authenticating with Google Cloud services requires the set up of the
GOOGLE_CREDENTIALS variable, as described
in the official GCP documentation.
GOOGLE_CREDENTIALS can simply point to a JSON file containing your access keys, in the EMBL-EBI Cloud Portal you’re required
to upload the JSON string directly.
The JSON file can be downloaded directly from the Google Developers Console following these steps:
- Log into the Google Developers Console and select a
- The API Manager view should be selected, click on
Credentialson the left, then
Create credentials, and finally “Service account key”.
Compute Engine default service accountin the
Service accountdrop-down, and select
JSONas the key type.
Createwill download your
Once you have your credentials you can add them to
in this form:
Deployment parameters represent a set of inputs specific that are related to
the cloud provider and eventually to a specific application. In general, they
provide information about the shared instances that you can have in place in
your cloud provider or just information that you prefer to set up just the first
time and avoid to repeat every time you deploy the instance.
The deployment parameters required by an appliance are expressed in the documentation page of the git repository of the same appliance.
For your convenience you can use a single
configuration for different appliances: it will make use only of the
share inputs ignoring the ones that are not relevant.
A deployment parameter can also be used to overwrite any of the
variables defined in the
terraform.tfvars file even when it is not
reported as input in the
Configurations represent a way to link a set of Cloud Credentials with a set of Deployment Parameters and an SSH public key. The use of a configuration simplifies the deployment of the applications, allowing to store and reuse as much configuration as possible.
Specify a new configuration is very easy:
- click on the
- assign a name of your choice;
- choose one of the
Cloud Providerthat you have previously defined;
- choose one of the
Deployment parametersthat you have previously defined;
- (optionally) add a public SSH key.
Inputs parameters represent a set of parameters that are likely going to
change per deployment and thus cannot determined in advance. The best example
of this is the number of nodes a compute cluster you’re about to deploy will
need to have.
Inputs can also refer to variables defined in Deployment Parameters, and thus allow to override them only when required.
Managing the Registry¶
How to add an Application to the Registry¶
Adding a new Application is very simple: you just need to know the URL of the git repository where the Applications is stored. As a test, you can add one of the applications maintained by the TSI team: https://github.com/EMBL-EBI-TSI/cpa-instance
Starting from the EMBL-EBI Cloud Portal Home:
- Click Application Repository in
the menu on the left-hand side;
- Click on the
- Enter the
URL of the git repository;
Your new application is now included in your Repository!
EMBL-EBI Cloud Portal requires the presence of a well-formed
Manifest file in the root directory of each git repository containing
an Application. This file is is a simple
manifest.json file contains a simple dictinary specifying, for example,
the Application name and mainainer along with the supported Cloud Providers.
Trying to add an Application repository that does not contain - or contains a
malformed manifest file, will result in an error.
Programmatic access (ecp-cli)¶
To facilitate automated deployment and application tested, a command line tool is available to interact with the portal: [ecp-cli](https://github.com/EMBL-EBI-TSI/ecp-cli).
To use, clone the repo; ecp.py is the main executable. First login by running
This will take you through the Elixir login process that will authenticate you through the portal. Your token will be stored automatically. Next, you can e.g. list your applications by running:
ecp.py get deployments
The general syntax is ecp.py *verb* *resource* [*name*], where verb is one of
stop (deployments only) and resource is one of
For more examples see the repo README.