Cloudbus Workflow Engine

Contents

Introduction

Cloudbus Workflow Engine (WFE) facilitates users to link standalone applications and execute their workflow applications on distributed network environments, such as Cluster, Grid, and Clouds. The WFE provides an XML-based workflow language for the users to define tasks and dependencies. It supports several scheduling algorithms as plug-in components, thus allowing the resource allocation decision to be made using several techniques, both in online and offline mode. We also use tuple spaces approach to enable an event-driven scheduling architecture for simplifying workflow execution. The latest version of the WFE has been integrated into the Cloudbus Broker and supports multiple computing platforms.

The Cloudbus Workflow Engine is a significant extension to the Gridbus Workflow Engine (GWFE), which was initially developed under the Gridbus Project, Grid Computing and Distributed Systems (GRIDS) Lab., Dept. of Computer Science and Software Engineering, the University of Melbourne, Australia. The project is partially supported by Australian Research Council Discovery Project grant, Storage Technology and the University of Melbourne.

 

Components of the Workflow Engine

Users interact with the Workflow Management System (WMS) through a web-portal. The following key components facilitae users to execute and manage scientific applications.


Grid Portal: The primary user interface for any application is a Web Portal that encompasses the following functionalities:

1. A workflow editor, which enables users to compose new workflows and modify existing ones.
2. A submission page, through which users can upload to the system, all necessary input files to run a workflow including the workflow description file, credentials, and services files.
3. A monitoring and output visualization page, which allows users to monitor multiple workflow executions in progress. The most common monitoring activity consists of keeping track the status of each task through the workflow monitor, which provides a real-time updated graphical representation of workflow tasks. The application’s output is presented in the form of images where applicable.
4. A resource information page, which shows the characteristics of all available computing resources.
5. An application specific page, which in the current implementation provides generation of application workflow description files by taking application specific parameters as input.

Workflow Editor: The workflow editor provides a Graphical User Interface (GUI) and allows users to create new and modify existing workflows, based on an XML-based workflow language (xWFL) utilizing the drag and drop facilities. Using the editor, expert users can design and create the workflows for complex scientific procedures following the workflow language and the nature of application, whereas primitive users can reuse these workflows with some minor changes. In the editor, workflows are created graphically as a Directed acyclic Graph (DAG) with some nodes and links. The node represents the computational activities of a particular task in the workflow and a link is used to specify the data flow between two tasks.

Workflow Monitor: The Workflow Monitor provides a GUI for viewing the status of each task in the workflow. Users can easily view the ready, executing, stage-in, and completed tasks. Task status is represented using di!erent colors. Users can also view the site of execution of each task, the number of tasks being executed (in case of a parameter sweep type of application) and the failure history of each task. The workflow structure is editable such that users can drag tasks and group or separate tasks of interest when there are numerous tasks in the workflow. The monitor interacts with the workflow engine (WFE) using an event mechanism by using the tuple space model. In the backend, a database server stores the states of each task for each application workflow. Whenever any task changes state, the monitoring interface is notified and the status values are stored. This enables multiple users to access the monitoring interface from di!erent locations. The monitoring interface does not have support for deletion and insertion of individual tasks at run-time. However, users can add and delete tasks at the time of construction using the workflow editor.

Workflow Engine: Scientific application portals submit task definitions along with their dependencies in the form of the workflow language to WFE. Then the WFE schedules the tasks in the workflow application through the middleware services and manages the execution of tasks on distributed resources. The key components of the WFE are: workflow submission, workflow language parser, resource discovery, dispatcher, data movement and workflow scheduler.

The WFE is designed to support an XML-based WorkFlow Language (xWFL). This facilitates user level planning at the submission time. The workflow language parser converts workflow description from XML format to Tasks, Parameters, Data Constraint (workflow dependency), Conditions, etc., that can be accessed by workflow scheduler. The resource discovery component of the WFE can query Information Services such as Globus MDS, directory service, and replica catalogues, to locate suitable resources for execution of the tasks in the workflow by coordinating with middleware technologies such as the Cloudbus Broker. The WFE uses the Cloudbus Broker for deploying and managing task execution on various middlewares as a dispatcher component. The Cloudbus Broker as a middleware mediates access to distributed resources by (a) discovering resources, (b) deploying and monitoring task execution on selected resources, (c) accessing data from local or remote data source during task execution, and (d) collating and presenting results.

The WFE is designed to be loosely-coupled and flexible using a tuple spaces model, eventdriven mechanism, and subscription/notification approach in the workflow scheduler, which is managed by the workflow coordinator component. The data movement component of the WFE enables data transfers between distributed resources by using SFTP and GridFTP protocols. The workflow executor is the central component in WFE. With the help from dispatcher component it interacts with the resource discovery component to find suitable compute resources at run time (depends on the scheduling algorithm used), submits a task to resources, and controls input data transfer between task execution nodes.

Gridbus Workflow Engine Documentation

For architecture, APIs and installation documentations, refer to the following source:


Cloudbus Workflow Engine related Documentation

 

Software and License

The WFE software and documents are released as "open source" under the GPL license. Copyright The Cloudbus Project, CLOUDS Lab, The University of Melbourne, 2011.

LATEST CODE: WorkFlow Engine code (released in 2016)

Compatibility

For compability, install the following software:

Bug Reports and Feedback

For WFE bugs and feedback, please email to raj@csse.unimelb.edu.au,

Credits

This release is developed by:

Assistance received from:

The Gridbus Workflow Engine (previous releases) was developed by:

Links

For additional information, please refer to the following websites:

       
Cloud Computing and Distributed Systems (CLOUDS) Laboratory
School of Computing and Information Systems
The University of Melbourne, Australia