Everest Tutorial

Check that your computer meets the following requirements:

Open https://everest.distcomp.org/ in a browser. Ignore SSL certificate warnings.

Log in with your username and password. If you don’t have an Everest user account, register yourself using Sign Up link.

Download tutorial.zip and unpack it to some directory.

Applications are the main entities in Everest - any computation is performed in the context of some application. Clients interact with applications by sending requests and receiving back results.

All Everest applications follow the same abstract model:

You can think of an application as a "black box" with some input and output ports or as a "function" with some arguments and return values. Just like pure functions, applications usually process each request independently from other requests in a stateless fashion.

In order to add an application to Everest you should provide an application description that consists of two parts:

In this part of tutorial we will add to Everest a well-known program for doing  molecular docking called AutoDock Vina.

To add an application to Everest navigate to Applications, click New application button on the top right corner and select Create application. You will be presented with a form divided into several tabbed sections:

Normally when adding a new application to Everest you should fill out this form manually. But to save time we will take a shortcut and import our application description from a prepared JSON file. Click Cancel to close the form.

Click again New application button, but this time select Import from JSON item. Choose file TUTORIAL_HOME/vina/vina.json. After the import is completed you should see the following message:

Select My apps checkbox on top of the applications list and check that AutoDock Vina is present in the list.

Let’s examine application description. Click on AutoDock Vina and then on Edit button on the top right corner. You will see the same form as before, but this time it is filled out. Let’s walk through the form sections.

You are returned to the main page of our application. You can always navigate to this page by clicking on the application name in Applications list. The application page has four tabbed sections.

From the user’s viewpoint running an application basically means sending it a request containing input values and waiting for a result containing corresponding output values. For each request Everest creates a new job which can be used to track the status of the request and to collect the result. So "running an application" and "submitting a job" have the same meaning.

Internally each job is composed of one or more tasks that represent units of computation sent for execution to resources. Tasks are generated from job inputs in accordance with the application configuration. Currently Everest users can create only applications with jobs consisting of a single task. However there is a special Parameter Sweep application described in Chapter 8 that runs jobs containing multiple tasks.

Open AutoDock Vina application and switch to Submit Job section.

Specify your job name, e.g. "First Vina Run". By default jobs are named as the application, so custom job names help to distinguish the different runs.

Specify the following values for input parameters:

Since we didn’t specify default resources in the application configuration, we should manually select at least one resource to run our job. In Resources input select test.

Click Submit button.

After the job submission you will be redirected to the job page which has the following sections.

Everest doesn’t provide its own computing infrastructure to run application tasks, nor does it provide access to some fixed external infrastructure like grid. Instead Everest enables users to attach to it any external computing resources and to run applications on arbitrary sets of these resources. From this point of view Everest can be seen as a multitenant metascheduling service.

Currently the preferred method for attaching a resource to Everest is based on using a special program called agent. The agent runs on the resource and acts as a mediator between it and Everest. This method has one drawback - it requires deployment of the agent on the resource. However, it also brings a number of advantages in comparison to plain SSH access like supporting resources behind a firewall and more strict security policies. Also, as we will see soon, the agent has minimal requirements and is easy to install.

The agent supports integration with various types of resources via adapter mechanism. At the moment the following adapters are implemented:

Agent repository contains agent’s source code, while detailed instructions on installing and using agent can be found in Agent User Manual.

Before attaching resource to Everest you should register it and obtain a resource token which is used for agent authentication.

In Web UI navigate to Resources section and click on New resource button. In Name field enter the name of your new resource. Click Save button.

You are redirected to the resource page. Note that the Token field contains an alphanumeric string. This string should be passed to the agent running on this resource.

Make sure the server satisfies the following requirements:

Open SSH connection to the server.

Repeat the same steps as for the server with the following differences.

When editing the agent configuration file change the resource.taskHandler.type parameter value to torque or slurm. Also add the resource.taskHandler.queue parameter with the name of the cluster queue you want the agent to use. For example:

You can also adjust the value of resource.maxTasks parameter to set the maximum number of tasks submitted by the agent to the queue.

Everest supports flexible binding of resources to applications. It is possible to configure a static set of resources that should be used by Everest to run application tasks. Application owner can also enable dynamic resource binding when user manually selects resources for running a job.

In both scenarios it is also possible to specify multiple resources and let Everest to schedule application tasks across these resources.

In this section we will try out different options for resource binding supported by Everest.

Running Everest applications via Web UI is easy and convenient, but it has some limitations. For example, if you want to run an application many times with different inputs, it is inconvenient to submit many jobs manually via web form. In other case, if you want to produce some result by using multiple applications, you will have to manually copy data between several jobs. Finally, Web UI is not suitable if you want to run Everest application from your program or some other external application.

For all these cases, from automation of repetitive tasks to application integration, Everest provides a REST API. It is the platform’s application programming interface implemented as a RESTful web service. REST API includes operations for accessing and managing applications, jobs, resources and other entities. This interface serves as a single entry point for all Everest clients, including Web UI which uses REST API under the hood.

REST API can be used to access Everest from any program that can speak HTTP protocol and parse JSON format. However it is too low level for most users, so it is nice to have ready-to-use client libraries built on top of REST API. At the moment, Everest provides one such client library for Python programming language called Python API. In this part of tutorial we will examine and run some programs written with Python API.

Python API supports all operating systems. Just check that you have Python 2.7 installed (Python 3 is not supported).

Python API depends on Requests library. You can install it with pip:

Python API consists of a single Python file everest.py. Download this file and place it in TUTORIAL_HOME/api/ so that our example programs can find it.

Before we start running programs that access Everest it is essential to obtain a client token. Each client accessing Everest should present such token with its request in order to authenticate itself. A token can be obtained from Everest by any user. A client using the token will be authenticated as the user who requested the token.

In order to obtain a client token open terminal in TUTORIAL_HOME/api/ and run the following command (replace USERNAME with your Everest username):

This command will prompt you for your Everest password and, if all went well, will create a file .token with the new token inside.

You can find the list all your tokens in Web UI. Click on your username in the top right corner and select Access Tokens. Valid tokens are marked with green circles. Each token has a label. This helps to distinguish tokens used by different clients. For example, tokens used by Web UI have label everest-webui. When you create a token with token_get.py script you can specify a token label via option -l. Check that the newly created token with label tutorial is present in the list.

Let’s start with a simple example of submitting a single job to Autodock Vina.

In Web UI open Autodock Vina application and copy application ID (long alphanumeric string).

Open TUTORIAL_HOME/api/vina_run.py in text editor.

After you replaced INSERT VINA APP ID with application ID, save the file.

Now it’s time to run our program as follows:

After the program is completed you can examine job outputs in TUTORIAL_HOME/api/results/run/.

In Web UI open Jobs list and check the corresponding job called "Vina Run - Job 0". The job name is produced by adding the job number to the session name.

Let’s consider a more complex case when we want to submit many jobs to an application. For example, we want to use Autodock Vina for virtual screening of many ligands against some protein.

Open TUTORIAL_HOME/api/vina_screening.py in text editor.

After you replaced INSERT VINA APP ID with application ID, save the file.

Now it’s time to run our program as follows:

In Web UI open Jobs list and check the corresponding jobs called "Vina Screening - Job X". You can enable automatic update of job states by selecting Auto Update checkbox on the top of the list. Since we delete the jobs in our program, the jobs will disappear from the list after they are completed.

After the program is completed you can examine job outputs in TUTORIAL_HOME/api/results/screening/.

Let’s consider another typical use case when one needs to combine multiple applications to produce a desired result. For example, suppose we want to produce a nice animation showing the results of molecular docking. We can do this in the following steps:

In this example we will perform these steps by using three different Everest applications:

Our program will run each of these applications one after another in a chain by passing results of one job to the next one.

Open TUTORIAL_HOME/api/vina_visualize.py in text editor.

After you replaced INSERT VINA APP ID with application ID, save the file.

Now it’s time to run our program as follows:

In Web UI open Jobs list and check the corresponding jobs called "Vina Visualize - Job X".

After the program is completed you can examine the produced animation in TUTORIAL_HOME/api/results/visualize/movie.gif.

Parameter sweep experiments represent an important class of scientific applications that require a large amount of computing resources in order to run a large number of similar computations across different combinations of parameter values. Parameter sweep application consists of many independent tasks (such applications are also called bag-of-tasks applications). The number of tasks is defined by a number of parameter combinations in the experiment. As a rule, each task runs the same executable but with different arguments and input files that depend on parameter values.

Everest provides a native support for such applications via generic Parameter Sweep application. This is an Everest application that can be used to run arbitrary parameter sweep experiments described by means of the so called plan file. This is a plain text file that contains parameter definitions and other directives that together define rules for generation of parameter sweep tasks and processing of their results. Besides a plan file a user can also provide an archive with executables, scripts and input files referred in the plan file.

After a user submitted his plan file and files archive to Parameter Sweep it creates a new job that can be used to monitor the status of the experiment. Parameter Sweep parses plan file, generates required tasks, submits them for execution, processes task results and generates the job output. The output is an archive that contains the results of the individual tasks taking into account filtering rules specified in the plan file.

In this section we will revisit example from the Python API tutorial where we used Autodock Vina for simplified virtual screening of many ligands against some protein. We used Python API to submit multiple jobs to Vina application and then collected and processed job results. Now let’s take a different approach and use Parameter Sweep application.