ADES - Application Deployment and Execution Service

AMC - Application Management Client

API - Application Programming Interface

APP - User-define application

COTS - Commercial Off The Shelf

CRUD - Create, Replace, Update, Delete

EMS - Execution Management Service

IdP - Identity Provider

MOAW - Modular OGC API Workflows

OAProc - OGC API Processes API

OGC - Open Geospatial Consortium

STS - Security Token Service

TEP - Thematic Exploitation Platform

The app developer is a user who have created an application in their local environment and now want to deploy that application to the TEP marketplace. The following figure illustrates, at a high level, the basis steps taken by the user and the TEP marketplace to deploy the application to the ADES and make it available for execution. Initially the application is only available to the user for execution but eventually, the user can submit the application for public deployment at which point other user will be able to discover and executed the application within the TEP environment.

Figure - App Developer Scenario

For a typical TEP user, interaction with the system begins by logging into the TEP marketplace and performing a search for data of interest and/or applications of interest. Since associations are maintained in the catalogue between TEP resources find one (say data) leads to discovering the other (i.e. applications suitable to operate on that data). Having discovered data and application of interest, the user can then proceed to execute the application. This following figure illustrates, at a high level, this workflow.

Figure - App User Scenario

In simple terms, the basic requirement of the TEP marketplace is to enable data processing algorithms uploaded by developers to be run by users, at scale, on very large spatial and non-spatial datasets hosted on the local cloud infrastructure.

All interaction with applications, data and services must be protected by a role-based, spatially-aware authentication and access control system.

The web front-end is the primary entry point for the TEP. The web front-end must satisfy the following requirements:

All aspect of the TEP must be governed by authentication and access control. TEP users must be authenticated in order to access the TEP. Once authenticated, access control must be applied to control what resources the user can access.

The authentication and access control requirements for the TEP include:

Having logged into a TEP, the user is placed in a virtual environment called a "workspace". All the resources of the TEP that the user owns or has access to are available in their workspace. This includes:

Each processes executed on a TEP will, either explicitly or implicitly, create a job that can be used to:

It is anticipated that many resources available on a TEP will have associated with then a license. The function of the license manager is to:

The TEP is a collaborative environment where "events" are constantly occurring. The notification and analytics module allows these events to be logged and that log, in turn, to be used as the basis of a notification system and analytics. The requirements for the notification and analytics module include:

The execution and management service (EMS) is the primary hub of the TEP marketplace. The EMS interacts with all the other TEP components to enable and coordinate the functionality of the TEP. The functions of the EMS include:

The TEP shall include a central catalogue that:

The ADES is responsible for executing user-defined app in the underlying cloud computing environment. As such the ADES must:

In order to deploy applications, an ADES must accept an "application package" which is a JSON document that:

The execution unit can be a binary executables bundled in a container (e.g. Docker container) but it can also be code written in some interpreted language such as Python or R.

User-defined app executed on a TEP run in a cloud computing environment and as such consume chargeable resources. As a result, users will want to be able to get a cost estimate of executing an application on a specified set of inputs. The cost estimation module must:

The function of the parallel execution module is to analyzes an execute request and the process description of a deployed user-defined app and determine the extent to which the execute request can be parallelized. For example, consider an execute request that references N scenes of satellite imagery as input. If the ADES determines that the N scenes can be processed independently, then the ADES could decide to group the input scenes into M batches and then execute the batches in parallel.

The result of executing an app is some output. The function of the result manager is to present the results as requested in the execute request. Results can be:

Results deployed as an OGC API can be updated and added to via subsequent runs of the process of other data. Subsequent process runs can be ad-hoc but also scheduled and periodic.

The storage layer is the repository of data in the TEP. The storage layer:

Data in the storage layer can be stored to:

Data in the storage layer can be accessed:

Assuming that a trust agreement exists between two TEPs one TEP should be able to call a process on another TEP and retrieve the results of that processing.

The web from end is the primary entry point for users of the TEP. It is generated by and interacts with the Execution Management Service. It provides interfaces for both app developers and app consumers. It allows developers to upload and register their applications with the system and allows consumers to search for applications and data, execute those applications, access and publish the results.

The EMS could fairly be considered the “hub” of the system. It acts as the back end for the Web Client. As such, it manages all interactions between the user and other parts of the system.

It is responsible for:

This service provides secure access and rule-based access control to TEP resources. It also provides user management capabilities (i.e. creating users, password recovery, user roles, api keys, etc.)

Each TEP user is provided with a workspace. A workspace is a virtual environment that contains all the resources that a TEP user owns or has access to. The workspace manager module implements this virtual environment.

The job manager, manages the lifecycle of jobs created on the TEP. It allows jobs to be monitored, cancelled, deleted, etc. It also allows results to be associated with a job and retrieved once a job is completed.

The license manager is responsible for maintaining and enforcing licensing requirements for TEP resources. When necessary, the license manager will (for example) trigger a click-through license to access licensed resources on the TEP.

The Notification & Analytics allow TEP users to subscribe to and receive notifications about various events that can occur in the TEP. For example, the queuing of a job for execution, the completion of the job, exception events, additions of new data to the TEP, etc.

The module also provides analytic tools to view and analyze the event data.

As its name implies, the ADES is responsible for deploying and executing the app developers’ application packages in the cloud provider’s systems. It interfaces with the cloud provider’s proprietary system for executing Docker containers and manages their execution and monitoring. It is worthy of note that this communication comprises the only cloud-provider-specific interface of the entire architecture. All other components would be identical, regardless which cloud provider is used.

The ADES also interacts with the other modules of the TEP to:

The catalogue is a central, searchable repository for both data and applications. It is an OGC-Compliant Service that will serve both app developer and consumer requirements through the EMS.

Docker is a tool designed to make it easier to create, deploy, and run applications by using containers. Containers allow a developer to package up an application with all of the parts it needs, such as libraries and other dependencies, and ship it all out as one package. By doing so, thanks to the container, the developer can rest assured that the application will run on any other machine regardless of any customized settings that machine might have that could differ from the machine used for writing and testing the code.

A Docker image is a light-weight virtual machine. But unlike a virtual machine, rather than creating a entire virtual operating system, Docker allows applications to use the same kernel as the system that they’re running on and only requires applications be shipped with things not already running on the host computer. This gives a significant performance boost and reduces the size of the application.

Applications in the TEP marketplace will be packaged into Docker containers for execution by the cloud service provider’s system. The Docker registry is a managed repository of Docker containers with API access both for uploading packages (through the EMS) and for accessing and execution them (by the cloud service provider’s execution service).

The TEP uses Amazon’s Elastic Kubernetes Service (EKS) to simplify the Docker execution environment and provide scalability for large jobs.

EKS is described by Amazon as “a compute engine that lets you run containers at scale on AWS infrastructure without having to manage clusters or individual virtual servers – This removes the need to choose server types, decide when to scale your clusters, or optimize cluster packing. EKS removes the need for customers to interact with or think about servers or clusters. EKS lets you focus on designing and building your applications instead of managing the infrastructure that runs them.”

Containers running in EKS access TEP data via the storage layer either directly or mediated by the ADES.

The storage layer is where TEP data is stored. TEP data includes the thematic data that the TEP offers, data uploaded into user workspaces and processing results.

TEP data can be stored in:

Apps deployed on the TEP can access TEP data directly (i.e. the app knows, for example, how to read data from a S3 bucket) or mediated by the ADES (i.e. the ADES stages the data for the application in a form that the application can read).

TEP data case be accesses directly for storage or it can be accesses via an OGC API.

CubeWerx is building some components of the TEP architecture by also extending and enhancing our suite of standards-compliant geospatial web services. Over the past 10 years, we have built out a sophisticated infrastructure for securing such services, based around a cryptographically secure identity token, and “spatially-aware” web services that can use this token to enforce access control.

The Security and access control component is presented in some detail, since 3^rd party applications may need to use these authentication methods to access services generated by the TEP.

A sign-in request to the TEP authentication server will return a secure token encoded (in the case of Web applications) in the form of an HTTP cookie called CW_CREDENTIALS. Encoded within the value of this cookie is a small JSON document containing an authenticated set of credentials. These credentials are packaged in an OpenPGP message (RFC 4880) and are signed with the PGP private key of a CubeWerx Authentication Server that is set up to provide authentication for the TEP. All other servers within the TEP are equipped with the public key(s) of the trusted authentication server(s) of that domain. This allows the servers to verify the authenticity of the credentials, since a successful decryption with the public key of one of the trusted authentication servers implies that the credentials must have originated from that authentication server. Note, therefore, that it is not necessary for a server to contact a CubeWerx Authentication Server in order to decrypt or verify the authenticity of a set of credentials.

A user is authenticated by making a login request to a CubeWerx Authentication Server. This can be done directly by having the user visit the CubeWerx Authentication Server with his/her browser, at which point the user will be prompted for a username and password. The authentication server will then pass a CW_CREDENTIALS cookie containing the appropriate credentials back to the browser. Alternatively, a browser-based application can collect the user’s username and password and contact the authentication server on behalf of the user (via either an Ajax request or a redirection to the authentication server with a callback back to the application). Either way, the web browser receives a CW_CREDENTIALS cookie containing the user’s authenticated credentials for that domain.

Since the CwAuth credentials for a particular domain are communicated via an HTTP cookie, web browsers (and thus browser-based applications) will automatically pass along such credentials when communicating with servers within that domain. Furthermore, properly equipped servers can cascade such credentials to other servers further down the service chain by recognizing the CW_CREDENTIALS cookie in the request and passing it along in the HTTP headers of any requests that it makes.

A web service can determine the credentials of the current user in one of two ways:

A decrypted CW_CREDENTIALS cookie contains an JSON representation of the user’s credentials, and has the following form:

All of the properties of this credentials object other than domain and username are optional. If the value of the hasMaxSeats property is false, it’s usually omitted. The originServer property indicates the URL of the CubeWerx Authentication Server that authenticated the credentials.

The access control mechanism to be deployed for the TEP and employed by the CubeSERV OGC Web Services is an internal software module built into all CubeWerx Web services that controls access to services, data and map layers in a number of ways, based on the identity of the user. It is configured as a set of rules (expressed as XML), where each rule has an appliesTo attribute indicating which user(s) or roles the rule applies to. An individual user may trigger more than one applicable rule, or none. If more than one rule is applicable, then the user is granted access to the union of what the applicable rules grant. Rules can be configured to expire after a certain time, allowing subscription times to be configured.

A TEP Marketplace user must have the ability to upload their own data. This area will be called "the user’s TEP workspace". To support this, the TEP Marketplace will have an S3 bucket, accessible only by the TEP Marketplace server, with the following directory structure:

Under a user’s workspace/ directory is a free-form directory structure of files that’s under the user’s complete control. It consists of:

The user should be presented with this directory as a complete filesystem through the TEP Marketplace, manipulatable in the standard filesystem ways. The only way the user can access this directory structure is through the TEP Marketplace (authenticated via CwAuth). There will be no direct access.

The reason for the extra workspace/ directory level is to allow the possibility of other TEP-managed directories to exist in S3 per username.

The user should be able to select one or more files or directories in their workspace and do things like download or delete the files. If the files and/or directories are data files, the user should also be given the option to expose them as a web service (see the Exposing workspace data as web services section). The user should also be able up upload new files and directories.

There will be per-user quotas on the size of a user’s workspace. The user should be given a visual indication of how full their workspace is (e.g., with a progress bar and/or "xGB of xGB used (x%)". We can determine the number of bytes taken up by an S3 directory with the command:

The TEP workspace will use a MariaDB database (on the same database system where cw_auth exists) called cw_tep, which has the following table:

This table can be JOINed with the cw_auth.users table to determine the user’s full name and e-mail address, etc.

The TEP Marketplace GUI will have some sort of workflow editor, and will store the workflows in this table. The script will likely be CQL, so the GUI will need to know how to parse and generate CQL.

App management within the TEP Marketplace is described in the License manager section of this document.

The user should be presented with a list of their current jobs (in whatever state they’re in - in progress, completed, etc.). This information can be provided to the TEP Marketplace GUI by the ADES and/or the jobs table (see section ???). If possible, a progress bar (with an estimated time to completion) should be presented for in-progress jobs.

Upon completion of a process, the results (and logs, etc.) will automatically be stored (by the WPS itself) in the user’s workspace under a jobs/{jobId}/ directory.

A user should be able to select any data file (e.g., a GeoTIFF or GeoJSON file) or directory (e.g., jobs/{jobId}/Output) in their workspace and expose it as a CubeSERV web service (which provides several OGC web service endpoints). The user should be given the option to "expose as new web service" or "add to existing web service". Perhaps there can also be a shortcut way of doing this upon completion of a job.

The TEP will expose a single CubeWerx Suite web deployment and use only the default instance. To expose a set of one or more data files through CubeSERV, it must be first loaded as a CubeSTOR MariaDB feature set. Each TEP user will be assigned exactly one CubeSTOR data store, whose name is the user’s username. When a TEP user is created, the following steps must be taken to support this:

Conversely, these things must be removed when the TEP user is deleted.

The steps to expose a set of one or more data files as a new web service are:

To add one or more data files to an existing web service, it’s simply a matter of registering the specified source file(s) as new data sources.

The TEP GUI will need to present the user with the list of their currently-exposed web services. This list can be derived simply by querying the user’s CubeSTOR data store for the list of feature sets that are loaded into it. From this list, the user should be able to:

With this arrangement, the following web-service URLs will exist:

All of these URLs, except for the last one, will always exist for all TEP users. The last one is feature-set specific, so there will be zero or more of them per user.

In the first version of the TEP Marketplace, it has been tentatively agreed that the only way to share data with others will be by exposing it as a web service, marking it as "public", and sharing the URL(s) to this web service.

In a future version, we can allow a TEP user to share arbitrary content in their workspace with other users. To accommodate this, we can add the following table to the cw_tep database:

username is the username of the TEP user granting access to something in their workspace. path is the path of the file or directory being shared, and is relative to the root of the user’s workspace. grant_user is the username of the TEP user who’s being granted access, or NULL to grant all users access. grant_type is the type of access granted. In the first implementation of this mechanism, this will probably be limited to "r" for read-only access.

This table can be JOINed with the cw_auth.users table to determine the user’s full name and e-mail address, etc.

A user should be able to grant or revoke access to arbitrary files and directories in their workspace by selecting the appropriate option in the file-or-directory-specific menu in their workspace view.

Each user’s workspace view will have a directory called shared/ (that is not renameable or removable) whose subdirectories are the usernames of users who have granted the user access to at least one thing. Under these subdirectories will be the files and directories of the things being shared. In this way the user can access (and refer to in process inputs) other user’s shared files in exactly the same way as they refer to their own. This shared/ directory wouldn’t actually exist in S3; it would be a virtual directory that is grafted on by the GUI.

Many of the components of the TEP marketplace are driven by the catalogue. The function of the catalogue is to:

The catalogue for the TEP marketplace is based on the OGC API - Records standard.

The atomic unit of information in the catalogue is the record. The following table lists the properties of a record:

The OGC API Records specification defines a GeoJSON encoding for a record. The following is an example of the GeoJSON-encoded catalogue record describing a process that computes NDVI and operates on RCM2 and Sentinel2 data:

The following list of resource types that are expected to be catalgued in the TEP:

Associations are expressed in the form <source resource> <association> <target resource>. The following list of associations and expected to exist among catalogues resources:

The Records API allows a subset of records to be retrieved from a catalogue using a logically connected set of predicates that may include spatial and/or temporal predicates.

The Records API extends OGC API - Features - Core: Part 1 to provide modern API patterns and encodings to facilitate to facilitate flexible discovery of resource available on the TEP.

The OGC API Features API that has been:

The following tables summarizes the access paths and relation types of the TEP catalogue search API:

Where:

The following tables lists the query parameter that may be used at the search endpoint to query the TEP catalogue:

By default the catalogue will access the internal CubeWerx Suite data dictionary to make all the metadata about available features sets, sources, data products, etc. accessible through the catalogue’s API. Thus, whenever a new resource is added to the TEP through CubeWerx Suite, it will automatically become discoverable via the catalogue’s API.

S3 triggers will be used to keep the catalogue’s information about the TEP holdings and user’s TEP workspace data up to date. Whenever data is added to the TEP holdings or the user’s TEP workspace, an S3 trigger will invoke a Lambda function that will generate a new catalogue record describing the newly added data and then HTTP POST that record to the TEP’s catalogue server. If data is removed from the TEP holdings or the user’s TEP workspace, an S3 trigger will invoke a Lambda function that will remove the corresponding record from the catalogue.

The catalogue also supports a Harvest API and a "CRUD" API so that other metadata can be explicitly loaded into the catalogue if necessary. Using the Harvest API, a reference to a knows resource type can be provided to the catalogue which will then read the metadata about the resource and populate the catalogue. Similarly a catalogue records can be externally prepared to describe a resource and pushed to the catalogue using the standard "CRUD" methods.

Whenever a process is executed, whether synchronously or asynchronously, a job resource is implicitly created. These job resources are managed through the jobs API describe is this clause.

The TEP Marketplace API supports the HTTP GET operation at the /jobs endpoint. The body of the GET response contains a list of jobs. The following schema fragment defines the schema of the response:

By default all jobs on the system are returned. However, the following tables defined a set of query parameters that can be used to filter the subset of jobs that are listed in the response.

The type parameter allows filtering of the results based on the type of service that created the job. For the TEP marketplace the value process is the default value indicated that only jobs created as a result of executing an app will be listed.

The processID parameter allows filtering of the results based on a list of one or more app identifiers (i.e. property processID).

The status parameter allows filtering of the results based on the specified list of job statuses. Valid status values include accepted, running, successful, failed and dismissed.

The datetime parameter allows filtering of the results based on when a job was created. The datetime parameter is an ISO 8601 interval defined by the following BNF:

If the value of the datetime parameter is a specific time instance than only jobs created at time time instance are listed in the response. If the value of the datetime parameter is an interval then only jobs created during the specific interval are listed in the response.

The limit parameter defines the number of jobs that are presented is a single response. If there are more jobs than the value of the limit parameter, links (rel="next" or rel="prev") are included in the response to navigate to subsequent or previous response pages.

The TEP Marketplace API supports the HTTP GET operation at the /jobs/{jobsID} endpoint. The body of the GET response contains a status of the specified job identified by its jobID. The following schema fragment defines the schema of the response:

The following schema defines the possible statuses that a job can assume:

As indicated above, the status response includes a links section. This section can contain links to an endpoint for ongoing monitoring of the job (i.e. rel="monitor") or an endpoint to cancel a running job (i.e. rel="cancel") or a link to delete the job (i.e. rel="edit") and thus free any storage resources that the job might be consuming.

The TEP Marketplace API supports the HTTP GET operation at the /jobs/{jobID}/runlog endpoint. The body of the GET response contains a runlog for that job that is current as of the moment the GET operation was executed. The response of the operation is a text/plain document that contains a run log of the job.

The TEP Marketplace API supports the HTTP DELETE operation at the /jobs/{jobsID} endpoint. The result of this operation depends on the status of the job. If the job status is currently accepted or running then the operation will cancel the job and set the job’s status to dismissed. If the job status is successful, failed or dismissed then the operation will remove the job and delete any artifacts associated with the job.

There will be many different types of events. Each event type will have:

The following is a tentative list of the types of events that the TEP marketplace should be able to handle:

A potential set of notification methods is:

For the first version of the TEP Marketplace, we will probably support only e-mail and the news feed.

Certain notification types (notably for workspace quotas and job activity) are for a user’s own events. Users should be automatically subscribed to receive such notifications through their news feed.

The following MariaDB database tables (in the cw_tep database) can be used to record the events that occur in the TEP Marketplace.

Whenever an event record is read from the database, a corresponding read from the event_properties table should also be performed WHERE event_id={eventId} in order to determine the full set of property values for that event.

Event records will always be added, never deleted. In a future version of the TEP Marketplace, it may be worthwhile to add an event_summaries table that provides long-term summary information (similar to the request_summaries and served_summaries tables of the cw_audit database). It may also be necessary to cull the events table (but not the event_summaries table!) after so many records or months to keep it from growing too large.

The following MariaDB database tables (in the cw_tep database) can be used to keep track of the current subscriptions:

Whenever a subscription record is read from the database, a corresponding read from the subscription_filters table should also be performed WHERE subscription_id={subscriptionId} in order to determine the full set of filters for that subscription.

The events and subscriptions tables can be JOINed with the cw_auth.users table to determine the user’s full name and e-mail address, etc.

TBD: Do we need another table to keep track of every notification that’s sent?

The adding of an event to the events table should be done as low-level in the code as possible to reduce code duplication and to limit the number of execution paths which fail to trigger the appropriate event notifications. In Javascript this function call might look like:

and in C this function call might look like:

Either way, all these functions do is add an entry to the events table (with the appropriate entries to the event_properties table).

Note that it may be a bit tricky to properly handle the quotaWarning1, quotaWarning2 and quotaExceeded events. Such events should only occur when the corresponding threshold is crossed to avoid bombarding the user with redundant warnings.

A notification daemon will be constantly running, and will wake up every few seconds to deliver the latest batch of subscribed-to event notifications. Its algorithm will look something like this:

The feed notification method will be handled differently. It will make use of the Atom feed.

Much of the above assumes that we will be building our own notification system from scratch. However, there’s still the possibility that there’s a third-party solution out there that we can make use of. Amazon EventBridge and Apache Kafka may be possibilities.

The TEP will initially be deployed with a set of built-in processes. Over time, more processes will be added by users. Eventually, users will want to chain one or more of these offered processes into a workflow in order to accomplish some task. This clause describes the chaining and workflow capabilities of the TEP.

The TEP shall implement the OGC Modular Workflows (MOAW) extension to the OGC API - Processes - Part 1: Core standad and shall also support the Common Workflow Language (CWL) as the means of expressing workflows.

The MOAW extension is not an OGC standard but CubeWerx is actively involved in the development of this standard and input from this projects shall be fed into that effort.

CWL is developed by a multi-vendor working group consisting of organizations and individuals aiming to enable scientists to share data analysis workflows.

Common Workflow Language (CWL) is an open standard for describing how to run application and connect them to create workflows. This clause discusses the details of the integration of CWL into the TEP environment.

A CWL script is file that is interpreted and executed by a runner. A runner is simply an application that knows how to parse and execute a CWL script. There are many implementations of CWL runners listed here https://www.commonwl.org/implementations/.

For the TEP, the chosen runner shall be bundled into a Docker container. Whenever a CWL workflow is presented to the ADES, that Docker container will be executed to process the workflow. Although CWL is a generic workflow processing engine, for the TEP, CWL will be used to combine TEP process invocations into workflows.

Using the same multiply-add example as presented above, the following CWL file will execute a workflow that multiplies two number (using the "Mul" process) and add the result to a third number (using the "Add" process).

This ad-hoc workflow could, as was the case in the above example, be deployed as a persistent process using the same application package as that used above. The only difference would be that the executionUnit would be CWL in this case rather the OGC Modular Workflow used in the above example.

In order to deploy an application to the TEP, the application must be defined in an application package. An application package is a JSON document that contains the description of the application; that is the definition of the application’s inputs, outputs and the execution unit for the application. The application package may also contains additional metadata about the resources required by the execution unit (e.g. number of CPUs, memory requirements, disk space requirements, GPU access, etc.).

The EMS is the central, coordinating component of the TEP. Although a critical component in this architecture, it is not the subject of standardization efforts currently, since all of the other components that it interacts with already provide standard or emerging standard interfaces. It could more properly be considered an application, built around the components of the TEP, to tie them all together. It is responsible for:

Given the nature of the EMS, we will cover it in less detail in this document than the components that publish public APIs or may need to integrate with other tools and systems. Its basic architecture will be described, as well as its interfaces with other components.

The EMS is being implemented in PHP, using the Slim Micro Framework (https://www.slimframework.com/) for its REST APIs and basic structure. PHP was chosen for its relatively complete set of packages and functionality required to handle the various interactions that will be necessary between other components (XML, ATOM, HTML/JavaScript, OpenSearch, AWS etc.).

All communication between the EMS and Amazon Web Services are handled either with Amazon’s PHP API, or directly from the system shell using PHP merely as a scripting language. Composer (https://getcomposer.org/ ) is used to manage 3^rd party packages and libraries and keep them up to date.

Although the EMS Web Client will communicate directly with the Identity Token Service to authenticate and obtain valid credentials, the EMS back end will validate those credentials and insure that they are passed along to any downstream services in order to allow for proper access control. A secure token, in the form of an encrypted HTTP cookie, is attached to all back-channel Web Service requests executed by the EMS.

All search functionality, both for data and for available applications are handled by an catalogue using the OpenSearch API. To enable proper discovery of imagery, the OpenSearch Time and Geo extensions are implemented by the catalogue. The EMS is responsible for formulating proper OpenSearch queries and processing the results for display by the Web Client.

The EMS posts Docker images to the Amazon Docker registry on behalf of the client. Docker is manipulated using a set of command-line tools, but this complexity will be hidden from the user and taken care of on their behalf by the EMS client and server.

The Slim Framework supports several templating packages to generate dynamic HTML. We are using the simple php-view package (https://github.com/slimphp/PHP-View) to render the web pages necessary for the EMS Web Client. This allows customization of the Web Client based on user credentials, to display a different interface for developers vs users, for instance.

The Application Deployment and Execution Service (ADES) is responsible for execution of developers’ containers on the cloud service providers architecture. It provides a standard OGC API Processes (OAPIP) endpoint. The EMS communicates with the ADES via that protocol.

The EMS is responsible for constructing the appropriate WPS requests (as detailed in chapter 8 below) in order to deploy and execute applications, monitor their execution and deliver the results.

The normal method for monitoring execution of a process executed by a Web Processing Server is using a polling mechanism defined by the WPS standard, and the GetStatus operation. This is inefficient for long running jobs, for that reason the ADES implements a push notification mechanism based on the work done during OGC Testbed 12 and described in "Testbed-12 Implementing Asynchronous Services Response Engineering Report (see OGC 15-023r3, 7.2).

This allows the server to report to the client (the EMS) when a job has completed. The ADES sends its notifications to a Webhook as per http://www.webhooks.org implemented in the EMS. The EMS will then report to the client via email, SMS etc.

The ADES also implements the standard GetStatus operation, which may also be used in the Web Client to provide progress indicators etc. to the user.

This chapter describes the mechanism for the deployment and execution of applications built in the context of the TEP. The TEP refers to a computing platform that follows a given set of scenarios for users, data and ICT (Information and Communications Technology) provision aggregated around an Earth Science thematic area. The TEP defines three canonical user scenarios covering the EO data exploitation (users discover and manipulate data), new EO service development (users deploy a new processor) and new EO product development (users publish new results). The TEP implements these canonical scenarios and moves the processing to the data, rather than the data to the users, thereby enabling ultra-fast data access and processing.

Algorithms are initially developed by a third-party developer and subsequently deployed in the TEP as an application package. An application package contains all the information necessary to allow applications to be specified and deployed in federated resource providers.

The application package encoding conveys all the information needed for encapsulating user applications and enabling their automatic management (upload, deploy, run) in diverse platforms. The application package is described in detailed in the The OGC Application Package.

As ADES is an OGC API - Processes deployment that was been extended to accept an application package as input. The ADES processes the application package and deploys the process described therein. Once deployed, the process becomes part of the API of the ADES and can be invoked using an execute request.

In the TEP the application package is create by the EMS service, on behalf of developers. The execution unit of the application is then deployed on the TEP’s cloud infrastructure by the ADES. At that point, the application is ready to be executed on the TEP-holdings or the a TEP user’s workspace data.

Referring to App Developer Scenario, the application package allows a developer (e.g. Marco) to register an application that will be deployed at a given cloud provider site. The application is first built and registered in a third-party Application Hub (e.g., DockerHub). The developer then directly interacts with the EMS Client that registers the application on the ADES.

The sequence of actions is described in the following sequence diagram image.

Figure 4 - Sequence of Deploy and Register Steps

With the application registered, a user (e.g., Jim) can directly interact with the EMS Client that will allow the execution of the application on the ADES.

The sequence of actions for the execution is described in the following sequence diagram image.

Figure 5 - Sequence of Execution Steps

The following diagram illustrates the process of deploying a user-defined application to an ADES.

The request body contains an application package that is HTTP POSTed to the /processes endpoint of the ADES. The application package contains a machine-readable description of the process to be deployed. The ADES reads the application package, deploys the process and responds with the identifier (i.e. /process/{processID}) for the newly added process.

The following JSON fragment is an example of an application package.

The following JSON fragment is an example of a process summary generated as the content of a deployment operation.

The following diagram illustrates how a deployed process can be executed:

The body of the HTTP post is a JSON-encoded execute request. The following JSON fragment is an example of a JSON-encoded execute request:

The normal workflow of executing an application includes using the GET operation at the /jobs/{jobID} endpoint to poll the server to determine the execution status of a running job. The server offers this polling method of monitoring the process BUT it also oofers a more efficient notification approach where the server notifies the client when the operation has completed processing.

The notification is triggered by specifying a value for the responseHandler parameter. The value of the response handler can be something like mailto:user@server.com or sms:4165555555. The details of the response handler parameter can be found in OGC 16-023r3, clause 7.2.

The value of any specified response handler is ignored by the ADES unless the value of the mode parameter is set to async indicating asynchronous execution. The response to an asynchronous request is a status information message such as the one shown below:

If an error occurs, the ADES will generate an exception report as shown below:

In this example, the execute request body was invalid.

The following table summaries the resources implemented by the CubeWerx OAProc service:

The server also implements the OPTIONS method (see RFC2616, clause 9) which allows clients to ascertain the methods and representations that are supported for each resource.

The following sequence diagram illustrates the sequence of operations performed by the ADES on the back end to execute a deployed application.

Figure 6 - Sequence Diagram for Application Execution

Keys:

In order to leverage the elasticity of the cloud infrastructure upon which the TEP is built, the ADES must analyze each execution request and determine the extent to which that execution can be parallelized. This analysis is performed using the description of the application and the inputs provided in the execute request. The process description can also include hints or directives to let the ADES know the certain aspects of the execution of a process can be parallelized.

For example, consider the following process description:

This processes takes a single input with an unlimited cardinality. Without any processing directives, the ADES cannot assume that N instances of this process can be executed in parallel if M inputs (M>=N) are specified in an execution request. This process description, however, includes a processing directive named correlatedInputs for the productRef input. This directive indicates that if multiple inputs are specified, they are not correlated or dependent on each other and so the ADES can decide to execute parallel instances of this process to speed up processing of all the provided inputs.

The schema for an execute request from OGC API - Processes - Part 1: Core is:

This schema shall be extended for the CSA project with a parameter named repeat as specified be the following schema:

The value of the repeat parameter is an ISO8601 date-time string. The value can be a time instant or a recurring time interval. If the value of the repeat parameter is a time instant in the future then the process shall be queued for execution at that time; otherwise the time instant value is ignored. If the value of the repeat parameter is a recurring time interval then the process is executed and re-executed at each repeating interval.

The following example illustrates an execute request, using the dsi process described above, that is automatically re-executed once a week by the ADES. This is specified using the repeat parameter with a value of P1W.

Triggered execution is a function of the notification module. A subscription is created for a specific event and if the even occurs, one of the notification methods can be process execution. For example, a user can subscribe to notified when new TEP-holding data is added to the TEP and specify that the notification method should be the execution of a specified process.

Every application that executes on the TEP will generate some result(s). Processing results are always available via the OGC API - Processes - Part 1: Core interface via the `/jobs/{jobID}/results' endpoint. However, this interface only presents the raw results. In many cases it is desirable to publish results using an OGC API endpoint so that the results can be easily visualized or integrated with other results or other data.

How the ADES responds to a request is controlled by the response parameter in the execute request. The currently defines values for the response parameter are raw and document. The value raw indicates that the raw output generated by the process should be returned (e.g. a GeoTIFF image). The value document indicates that a JSON response document should be generated that contains all the outputs generated by a process. The output can be encoded in-line in the response document or referenced via a link.

For the CSA project, the schema of the execute request will be extended with a new parameter named publish as indicated in the following schema fragment:

A value of publish for the response parameter will trigger the ADES to publish the results of processing to an OGC API endpoint. The value of the endpoint parameter is the URI of a collections to which processing results should be published. If the specified endpoint URI does not exist, the ADES will first create the endpoint before publishing the results.

This section describes the quotation mechanism that extends capabilities of an OGC API - Processes service to estimate the resource requirements for executing a process, and the relevant costs that would result from allocating those resources to perform the actual process execution. The result from a quotation would provide an estimate of the expected amount that would be billed to the user for processing its operations if he deems the costs are acceptable to proceed with its execution.

Following sections will provide further context about quotation estimates, details about the operations implied by these new capabilities as well as presenting proposed implementation details.

Because every submitted quotation request for a given process contains all the necessary details to run this process, it is possible to employ the provided contents directly to request its execution. For convenience, the request endpoint /quotations/{quoteID}/execution is used to POST the details stored as processExecuteParameters.yaml since they contain the corresponding fields required for fulfilling the typical execute.yaml request contents. In addition than for convenience purposes, this also guarantees that the process that would be executed through the quotation mechanism will match with the estimated quote results, as the exact same parameters would be submitted for execution. This reduces chances of manipulation errors in contrast to relying on an external entity that would otherwise have to extract the contents themselves from the quote and supply them accordingly without alteration in a subsequent request for execution on the /processes/{processID}/execution endpoint. This would produce unnecessary data handling steps that can be avoided, as well leaving opportunities to falsify the information of the originally estimated quote. Using the parameters already stored in the quote definition, this alleviates these issues. The last and most important advantage of this endpoint is that it allows triggering the billing mechanism, which is further detailed below.

As presented in the figure, most of the billing operations first consist of extracting the relevant execution parameters from the submitted quote, and then proceed to the usual execution of the process using them. One additional step required prior to the process execution is the evaluation of the user profile which requests this execution. Effectively, because someone, or an organization, will have to be billed for allocating the necessary resources to execute the estimated process requirements with the quote, the user must be identified. This identification mechanism could be implemented with various methods, and is therefore left open to the decision of the service provider. It is generally recommended to employ traditional HTTP authentication (RFC-7235) or OAuth (RFC-6749) techniques to facilitate interactions with web services using well known identity resolution approaches. Once identity has been resolved, it should be carried over along with the process execution parameters to run the process, resulting in a job (as per OGC API - Processes: Core) to monitor and obtain execution status information.

Following process execution, the task handling the job being processed can proceed to additional steps involved specifically for the billing operations. One optional step to consider accomplishing here is to collect metadata about real execution parameters and allocated resources by the job in order to better estimate future quote submissions. The Quotation System could then take advantage of better use-case metadata to improve its estimation algorithm. Using the quote details retrieved at the start of the billing process, such as the estimated total and currency information, the bill can then be created and charged to the identified user. The intricate details about the billing process itself, such as retrieving account holder data and sending transaction information toward a financial institution are omitted to simplify the figure, but would be involved during this step.

Finally, after successfully billing the customer for the execution of the process, the job results are published. More specifically, the outputs generated by the process execution are made available to the user using the relevant delivery system selected by the platform hosting the service. This would be done through a catalogue or simply a file server accessible by an URL inscribed along with the job details. Results are updated only after billing was validated to ensure that, even when running the process asynchronously, it is not possible to obtain them in advance when monitoring the job status information endpoint. After job completion, either synchronously or asynchronously, the typical /jobs/{jobId}/results endpoint from OGC API - Processes: Core can then be used to retrieve the outputs the user paid for.

In order to track bills created by these operations, it is recommended to store billing metadata using the schema defined in bill.yaml. This definition should minimally contain details regarding the associated quote ID, the executed process ID, the job that processed it to link to the relevant outputs, and a user identification reference that was billed. Furthermore, the billed price and currency, as well as the date the billed was charged should be tracked in the billed. Any additional details that jugged necessary for tracking billing by the system can also be added. Because billing can contain sensitive data, such as the user details, it is not strictly required to expose this information through API endpoints. A protected API could still provide this information, using for example a GET request to /bills/{billID} to retrieve this data for management purposes. The following sample contents provide an example of what a bill could look like using the specified schema.

The EMS Client supports several modes of operation that are automatically set depending on the credentials available to a user when they log in. These may be implemented as subsets of each other, i.e. Developer Mode may include all the functionality of User Mode, plus the Developer-only portions, etc.