This section goes into more detail about how you can create Streams, which are collections of Spring Cloud Stream applications. It covers topics such as creating and deploying Streams.

If you are just starting out with Spring Cloud Data Flow, you should probably read the Getting Started guide before diving into this section.

A Stream is a collection of long-lived Spring Cloud Stream applications that communicate with each other over messaging middleware. A text-based DSL defines the configuration and data flow between the applications. While many applications are provided for you to implement common use-cases, you typically create a custom Spring Cloud Stream application to implement custom business logic.

The general lifecycle of a Stream is:

For deploying Streams, the Data Flow Server has to be configured to delegate the deployment to a new server in the Spring Cloud ecosystem named Skipper .

Furthermore, you can configure Skipper to deploy applications to one or more Cloud Foundry orgs and spaces, one or more namespaces on a Kubernetes cluster, or to the local machine. When deploying a stream in Data Flow, you can specify which platform to use at deployment time. Skipper also provides Data Flow with the ability to perform updates to deployed streams. There are many ways the applications in a stream can be updated, but one of the most common examples is to upgrade a processor application with new custom business logic while leaving the existing source and sink applications alone.

16.1. Stream Pipeline DSL

A stream is defined by using a Unix-inspired Pipeline syntax . The syntax uses vertical bars, known as “pipes”, to connect multiple commands. The command ls -l | grep key | less in Unix takes the output of the ls -l process and pipes it to the input of the grep key process. The output of grep is, in turn, sent to the input of the less process. Each | symbol connects the standard output of the command on the left to the standard input of the command on the right. Data flows through the pipeline from left to right.

In Data Flow, the Unix command is replaced by a Spring Cloud Stream application and each pipe symbol represents connecting the input and output of applications over messaging middleware, such as RabbitMQ or Apache Kafka.

Each Spring Cloud Stream application is registered under a simple name. The registration process specifies where the application can be obtained (for example, in a Maven Repository or a Docker registry). In Data Flow, we classify the Spring Cloud Stream applications as Sources, Processors, or Sinks.

As a simple example, consider the collection of data from an HTTP Source and writing to a File Sink. Using the DSL, the stream description is:

http | file

A stream that involves some processing would be expressed as:

http | filter | transform | file

Stream definitions can be created by using the shell’s stream create command, as shown in the following example:

dataflow:> stream create --name httpIngest --definition "http | file"

The Stream DSL is passed in to the --definition command option.

The deployment of stream definitions is done through the Shell’s stream deploy command, as follows:

dataflow:> stream deploy --name ticktock

The Getting Started section shows you how to start the server and how to start and use the Spring Cloud Data Flow shell.

Note that the shell calls the Data Flow Server’s REST API. For more information on making HTTP requests directly to the server, see the REST API Guide .

16.2. Stream Application DSL

You can use the Stream Application DSL to define custom binding properties for each of the Spring Cloud Stream applications. See the Stream Application DSL section of the microsite for more information.

16.3. Application Properties

Each application takes properties to customize its behavior. As an example, the http source module exposes a port setting that lets the data ingestion port be changed from the default value:

This port property is actually the same as the standard Spring Boot server.port property. Data Flow adds the ability to use the shorthand form port instead of server.port . You can also specify the longhand version:

This shorthand behavior is discussed more in the section on Stream Application Properties . If you have registered application property metadata , you can use tab completion in the shell after typing -- to get a list of candidate property names.

The shell provides tab completion for application properties. The app info --name <appName> --type <appType> shell command provides additional documentation for all the supported properties.

Skipper is a server that lets you discover Spring Boot applications and manage their lifecycle on multiple cloud platforms.

Applications in Skipper are bundled as packages that contain the application’s resource location, application properties, and deployment properties. You can think of Skipper packages as being analogous to packages found in tools such as apt-get or brew .

When Data Flow deploys a Stream, it generates and upload a package to Skipper that represents the applications in the Stream. Subsequent commands to upgrade or roll back the applications within the Stream are passed through to Skipper. In addition, the Stream definition is reverse-engineered from the package, and the status of the Stream is also delegated to Skipper.

17.1. Register a Stream Application

You can register a versioned stream application by using the app register command. You must provide a unique name, an application type, and a URI that can be resolved to the application artifact. For the type, specify source , processor , or sink . The version is resolved from the URI. Here are a few examples:

dataflow:>app register --name mysource --type source --uri maven://com.example:mysource:0.0.1
dataflow:>app register --name mysource --type source --uri maven://com.example:mysource:0.0.2
dataflow:>app register --name mysource --type source --uri maven://com.example:mysource:0.0.3
dataflow:>app list --id source:mysource
╔═══╤══════════════════╤═════════╤════╤════╗
║app│      source      │processor│sink│task║
╠═══╪══════════════════╪═════════╪════╪════╣
║   │> mysource-0.0.1 <│         │    │    ║
║   │mysource-0.0.2    │         │    │    ║
║   │mysource-0.0.3    │         │    │    ║
╚═══╧══════════════════╧═════════╧════╧════╝
dataflow:>app register --name myprocessor --type processor --uri file:///Users/example/myprocessor-1.2.3.jar
dataflow:>app register --name mysink --type sink --uri https://example.com/mysink-2.0.1.jar

dataflow:>app register --name http --type source --uri maven://org.springframework.cloud.stream.app:http-source-rabbit:3.2.1
dataflow:>app register --name log --type sink --uri maven://org.springframework.cloud.stream.app:log-sink-rabbit:3.2.1

If you would like to register multiple applications at one time, you can store them in a properties file, where the keys are formatted as <type>.<name> and the values are the URIs.

For example, to register the snapshot versions of the http and log applications built with the RabbitMQ binder, you could have the following in a properties file (for example, stream-apps.properties ):

Registering an application by using --type app is the same as registering a source , processor or sink . Applications of the type app can be used only in the Stream Application DSL (which uses double pipes || instead of single pipes | in the DSL) and instructs Data Flow not to configure the Spring Cloud Stream binding properties of the application. The application that is registered using --type app does not have to be a Spring Cloud Stream application. It can be any Spring Boot application. See the Stream Application DSL introduction for more about using this application type.

You can register multiple versions of the same applications (for example, the same name and type), but you can set only one as the default. The default version is used for deploying Streams.

The first time an application is registered, it is marked as default. The default application version can be altered with the app default command:

dataflow:>app default --id source:mysource --version 0.0.2
dataflow:>app list --id source:mysource
╔═══╤══════════════════╤═════════╤════╤════╗
║app│      source      │processor│sink│task║
╠═══╪══════════════════╪═════════╪════╪════╣
║   │mysource-0.0.1    │         │    │    ║
║   │> mysource-0.0.2 <│         │    │    ║
║   │mysource-0.0.3    │         │    │    ║
╚═══╧══════════════════╧═════════╧════╧════╝

dataflow:>app unregister --name mysource --type source --version 0.0.1
dataflow:>app list --id source:mysource
╔═══╤══════════════════╤═════════╤════╤════╗
║app│      source      │processor│sink│task║
╠═══╪══════════════════╪═════════╪════╪════╣
║   │> mysource-0.0.2 <│         │    │    ║
║   │mysource-0.0.3    │         │    │    ║
╚═══╧══════════════════╧═════════╧════╧════╝

All applications in a stream should have a default version set for the stream to be deployed. Otherwise, they are treated as unregistered application during the deployment. Use the app default command to set the defaults.

app default --id source:mysource --version 0.0.3
dataflow:>app list --id source:mysource
╔═══╤══════════════════╤═════════╤════╤════╗
║app│      source      │processor│sink│task║
╠═══╪══════════════════╪═════════╪════╪════╣
║   │mysource-0.0.2    │         │    │    ║
║   │> mysource-0.0.3 <│         │    │    ║
╚═══╧══════════════════╧═════════╧════╧════╝

The stream deploy necessitates default application versions being set. The stream update and stream rollback commands, though, can use all (default and non-default) registered application versions.

The following command creates a stream that uses the default mysource version (0.0.3):

17.1.1. Register Out-of-the-Box Applications and Tasks

For convenience, we have the static files with application-URIs (for both Maven and Docker) available for all the out-of-the-box stream and task applications. You can point to this file and import all the application-URIs in bulk. Otherwise, as explained previously, you can register them individually or have your own custom property file with only the required application-URIs in it. We recommend, however, having a “focused” list of desired application-URIs in a custom property file.

Out-of-the-Box Stream Applications

The following table includes the dataflow.spring.io links to the stream applications based on Spring Cloud Stream 3.2.x and Spring Boot 2.7.x .

RabbitMQ + Maven

dataflow.spring.io/rabbitmq-maven-latest

dataflow.spring.io/rabbitmq-maven-latest-snapshot

RabbitMQ + Docker

dataflow.spring.io/rabbitmq-docker-latest

dataflow.spring.io/rabbitmq-docker-latest-snapshot

Apache Kafka + Maven

dataflow.spring.io/kafka-maven-latest

dataflow.spring.io/kafka-maven-latest-snapshot

Apache Kafka + Docker

dataflow.spring.io/kafka-docker-latest

dataflow.spring.io/kafka-docker-latest-snapshot

For more information about the available out-of-the-box stream applications see the Spring Cloud Stream Applications project page.

For more information about the available out-of-the-box task applications see timestamp-task and timestamp-batch docs.

As an example, if you would like to register all out-of-the-box stream applications built with the Kafka binder in bulk, you can use the following command:

You can also pass the --local option (which is true by default) to indicate whether the properties file location should be resolved within the shell process itself. If the location should be resolved from the Data Flow Server process, specify --local false .

When you use either app register or app import , if an application is already registered with the provided name and type and version, it is, by default, not overridden. If you would like to override the pre-existing application uri or metadata-uri coordinates, include the --force option.

Note, however, that, once downloaded, applications may be cached locally on the Data Flow server, based on the resource location. If the resource location does not change (even though the actual resource bytes may be different), it is not re-downloaded. When using maven:// resources, on the other hand, using a constant location may still circumvent caching (if using -SNAPSHOT versions).

Moreover, if a stream is already deployed and uses some version of a registered app, then (forcibly) re-registering a different application has no effect until the stream is deployed again.

17.1.2. Register Custom Applications

While Data Flow includes source, processor, sink applications, you can extend these applications or write a custom Spring Cloud Stream application. You can follow the Stream Development guide on the Microsite to create your own custom application. Once you have created a custom application, you can register it, as described in Register a Stream Application .

17.2. Creating a Stream

The Spring Cloud Data Flow Server exposes a full RESTful API for managing the lifecycle of stream definitions, but the easiest way to use is it is through the Spring Cloud Data Flow shell. The Getting Started section describes how to start the shell.

New streams are created with the help of stream definitions. The definitions are built from a simple DSL. For example, consider what happens if we run the following shell command:

This defines a stream named ticktock that is based off of the DSL expression time | log . The DSL uses the “pipe” symbol ( | ), to connect a source to a sink.

The stream info command shows useful information about the stream, as shown (with its output) in the following example:

dataflow:>stream info ticktock
╔═══════════╤═════════════════╤═══════════╤══════════╗
║Stream Name│Stream Definition│Description│  Status  ║
╠═══════════╪═════════════════╪═══════════╪══════════╣
║ticktock   │time | log       │           │undeployed║
╚═══════════╧═════════════════╧═══════════╧══════════╝

17.2.1. Stream Application Properties

Application properties are the properties associated with each application in the stream. When the application is deployed, the application properties are applied to the application through command-line arguments or environment variables, depending on the underlying deployment implementation.

The following stream can have application properties defined at the time of stream creation:

The app info --name <appName> --type <appType> shell command displays the exposed application properties for the application. For more about exposed properties, see Application Metadata .

The following listing shows the exposed properties for the time application:

dataflow:> app info --name time --type source
Information about source application 'time':
Version: '3.2.1':
Default application version: 'true':
Resource URI: maven://org.springframework.cloud.stream.app:time-source-rabbit:3.2.1
╔══════════════════════════════╤══════════════════════════════╤══════════════════════════════╤══════════════════════════════╗
║         Option Name          │         Description          │           Default            │             Type             ║
╠══════════════════════════════╪══════════════════════════════╪══════════════════════════════╪══════════════════════════════╣
║spring.integration.poller.max-│Maximum number of messages to │<none>                        │java.lang.Integer             ║
║messages-per-poll             │poll per polling cycle.       │                              │                              ║
║spring.integration.poller.fixe│Polling rate period. Mutually │<none>                        │java.time.Duration            ║
║d-rate                        │exclusive with 'fixedDelay'   │                              │                              ║
║                              │and 'cron'.                   │                              │                              ║
║spring.integration.poller.fixe│Polling delay period. Mutually│<none>                        │java.time.Duration            ║
║d-delay                       │exclusive with 'cron' and     │                              │                              ║
║                              │'fixedRate'.                  │                              │                              ║
║spring.integration.poller.rece│How long to wait for messages │1s                            │java.time.Duration            ║
║ive-timeout                   │on poll.                      │                              │                              ║
║spring.integration.poller.cron│Cron expression for polling.  │<none>                        │java.lang.String              ║
║                              │Mutually exclusive with       │                              │                              ║
║                              │'fixedDelay' and 'fixedRate'. │                              │                              ║
║spring.integration.poller.init│Polling initial delay. Applied│<none>                        │java.time.Duration            ║
║ial-delay                     │for 'fixedDelay' and          │                              │                              ║
║                              │'fixedRate'; ignored for      │                              │                              ║
║                              │'cron'.                       │                              │                              ║
║time.date-format              │Format for the date value.    │MM/dd/yy HH:mm:ss             │java.lang.String              ║
╚══════════════════════════════╧══════════════════════════════╧══════════════════════════════╧══════════════════════════════╝

dataflow:> app info --name log --type sink
Information about sink application 'log':
Version: '3.2.1':
Default application version: 'true':
Resource URI: maven://org.springframework.cloud.stream.app:log-sink-rabbit:3.2.1
╔══════════════════════════════╤══════════════════════════════╤══════════════════════════════╤══════════════════════════════╗
║         Option Name          │         Description          │           Default            │             Type             ║
╠══════════════════════════════╪══════════════════════════════╪══════════════════════════════╪══════════════════════════════╣
║log.name                      │The name of the logger to use.│<none>                        │java.lang.String              ║
║log.level                     │The level at which to log     │<none>                        │org.springframework.integratio║
║                              │messages.                     │                              │n.handler.LoggingHandler$Level║
║log.expression                │A SpEL expression (against the│payload                       │java.lang.String              ║
║                              │incoming message) to evaluate │                              │                              ║
║                              │as the logged message.        │                              │                              ║
╚══════════════════════════════╧══════════════════════════════╧══════════════════════════════╧══════════════════════════════╝

Note that, in the preceding example, the fixed-delay and level properties defined for the time and log applications are the “short-form” property names provided by the shell completion. These “short-form” property names are applicable only for the exposed properties. In all other cases, you should use only fully qualified property names.

17.2.2. Common Application Properties

In addition to configuration through DSL, Spring Cloud Data Flow provides a mechanism for setting common properties to all the streaming applications that are launched by it. This can be done by adding properties prefixed with spring.cloud.dataflow.applicationProperties.stream when starting the server. When doing so, the server passes all the properties, without the prefix, to the instances it launches.

For example, all the launched applications can be configured to use a specific Kafka broker by launching the Data Flow server with the following options:

--spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.brokers=192.168.1.100:9092
--spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.zkNodes=192.168.1.100:2181

Doing so causes the spring.cloud.stream.kafka.binder.brokers and spring.cloud.stream.kafka.binder.zkNodes properties to be passed to all the launched applications.

17.3. Deploying a Stream

This section describes how to deploy a Stream when the Spring Cloud Data Flow server is responsible for deploying the stream. It covers the deployment and upgrade of Streams by using the Skipper service. The description of how to set deployment properties applies to both approaches of Stream deployment.

Consider the ticktock stream definition:

The Data Flow Server delegates to Skipper the resolution and deployment of the time and log applications.

The stream info command shows useful information about the stream, including the deployment properties:

dataflow:>stream info --name ticktock
╔═══════════╤═════════════════╤═════════╗
║Stream Name│Stream Definition│  Status ║
╠═══════════╪═════════════════╪═════════╣
║ticktock   │time | log       │deploying║
╚═══════════╧═════════════════╧═════════╝
Stream Deployment properties: {
  "log" : {
    "resource" : "maven://org.springframework.cloud.stream.app:log-sink-rabbit",
    "spring.cloud.deployer.group" : "ticktock",
    "version" : "2.0.1.RELEASE"
  "time" : {
    "resource" : "maven://org.springframework.cloud.stream.app:time-source-rabbit",
    "spring.cloud.deployer.group" : "ticktock",
    "version" : "2.0.1.RELEASE"
There is an important optional command argument (called --platformName) to the stream deploy command.
Skipper can be configured to deploy to multiple platforms.
Skipper is pre-configured with a platform named default, which deploys applications to the local machine where Skipper is running.
The default value of the --platformName command line argument is default.
If you commonly deploy to one platform, when installing Skipper, you can override the configuration of the default platform.
Otherwise, specify the platformName to be one of the values returned by the stream platform-list command.
In the preceding example, the time source sends the current time as a message each second, and the log sink outputs it by using the logging framework.
You can tail the stdout log (which has an <instance> suffix). The log files are located within the directory displayed in the Data Flow Server’s log output, as shown in the following listing:
$ tail -f /var/folders/wn/8jxm_tbd1vj28c8vj37n900m0000gn/T/spring-cloud-dataflow-912434582726479179/ticktock-1464788481708/ticktock.log/stdout_0.log
2016-06-01 09:45:11.250  INFO 79194 --- [  kafka-binder-] log.sink    : 06/01/16 09:45:11
2016-06-01 09:45:12.250  INFO 79194 --- [  kafka-binder-] log.sink    : 06/01/16 09:45:12
2016-06-01 09:45:13.251  INFO 79194 --- [  kafka-binder-] log.sink    : 06/01/16 09:45:13
However, it is not common in real-world use cases to create and deploy the stream in one step.
The reason is that when you use the stream deploy command, you can pass in properties that define how to map the applications onto the platform (for example, what is the memory size of the container to use, the number of each application to run, and whether to enable data partitioning features).
Properties can also override application properties that were set when creating the stream.
The next sections cover this feature in detail.
17.3.1. Deployment Properties
When deploying a stream, you can specify properties that can control how applications are deployed and configured. See the Deployment Properties section of the microsite for more information.
17.6. Validating a Stream
Sometimes, an application contained within a stream definition contains an invalid URI in its registration.
This can caused by an invalid URI being entered at application registration time or by the application being removed from the repository from which it was to be drawn.
To verify that all the applications contained in a stream are resolve-able, a user can use the validate command:
dataflow:>stream validate ticktock
╔═══════════╤═════════════════╗
║Stream Name│Stream Definition║
╠═══════════╪═════════════════╣
║ticktock   │time | log       ║
╚═══════════╧═════════════════╝
ticktock is a valid stream.
╔═══════════╤═════════════════╗
║ App Name  │Validation Status║
╠═══════════╪═════════════════╣
║source:time│valid            ║
║sink:log   │valid            ║
╚═══════════╧═════════════════╝
In the preceding example, the user validated their ticktock stream. Both the source:time and sink:log are valid.
Now we can see what happens if we have a stream definition with a registered application with an invalid URI:
dataflow:>stream validate bad-ticktock
╔════════════╤═════════════════╗
║Stream Name │Stream Definition║
╠════════════╪═════════════════╣
║bad-ticktock│bad-time | log   ║
╚════════════╧═════════════════╝
bad-ticktock is an invalid stream.
╔═══════════════╤═════════════════╗
║   App Name    │Validation Status║
╠═══════════════╪═════════════════╣
║source:bad-time│invalid          ║
║sink:log       │valid            ║
╚═══════════════╧═════════════════╝
17.7. Updating a Stream
To update the stream, use the stream update command, which takes either --properties or --propertiesFile as a command argument.
Skipper has an important new top-level prefix: version.
The following commands deploy http | log stream (and the version of log which registered at the time of deployment was 3.2.0):
dataflow:> stream create --name httptest --definition "http --server.port=9000 | log"
dataflow:> stream deploy --name httptest
dataflow:>stream info httptest
╔══════════════════════════════╤══════════════════════════════╤════════════════════════════╗
║             Name             │             DSL              │          Status            ║
╠══════════════════════════════╪══════════════════════════════╪════════════════════════════╣
║httptest                      │http --server.port=9000 | log │deploying                   ║
╚══════════════════════════════╧══════════════════════════════╧════════════════════════════╝
Stream Deployment properties: {
  "log" : {
    "spring.cloud.deployer.indexed" : "true",
    "spring.cloud.deployer.group" : "httptest",
    "maven://org.springframework.cloud.stream.app:log-sink-rabbit" : "3.2.0"
  "http" : {
    "spring.cloud.deployer.group" : "httptest",
    "maven://org.springframework.cloud.stream.app:http-source-rabbit" : "3.2.0"
Then the following command updates the stream to use the 3.2.1 version of the log application.
Before updating the stream with the specific version of the application, we need to make sure that the application is registered with that version:
dataflow:>app register --name log --type sink --uri maven://org.springframework.cloud.stream.app:log-sink-rabbit:3.2.1
Successfully registered application 'sink:log'
dataflow:>stream info httptest
╔══════════════════════════════╤══════════════════════════════╤════════════════════════════╗
║             Name             │             DSL              │          Status            ║
╠══════════════════════════════╪══════════════════════════════╪════════════════════════════╣
║httptest                      │http --server.port=9000 | log │deploying                   ║
╚══════════════════════════════╧══════════════════════════════╧════════════════════════════╝
Stream Deployment properties: {
  "log" : {
    "spring.cloud.deployer.indexed" : "true",
    "spring.cloud.deployer.count" : "1",
    "spring.cloud.deployer.group" : "httptest",
    "maven://org.springframework.cloud.stream.app:log-sink-rabbit" : "3.2.1"
  "http" : {
    "spring.cloud.deployer.group" : "httptest",
    "maven://org.springframework.cloud.stream.app:http-source-rabbit" : "3.2.1"
17.8. Forcing an Update of a Stream
When upgrading a stream, you can use the --force option to deploy new instances of currently deployed applications even if no application or deployment properties have changed.
This behavior is needed for when configuration information is obtained by the application itself at startup time — for example, from Spring Cloud Config Server.
You can specify the applications for which to force an upgrade by using the --app-names option.
If you do not specify any application names, all the applications are forced to upgrade.
You can specify the --force and --app-names options together with the --properties or --propertiesFile options.




    

17.9. Stream Versions
Skipper keeps a history of the streams that were deployed.
After updating a Stream, there is a second version of the stream.
You can query for the history of the versions by using the stream history --name <name-of-stream> command:
dataflow:>stream history --name httptest
╔═══════╤════════════════════════════╤════════╤════════════╤═══════════════╤════════════════╗
║Version│        Last updated        │ Status │Package Name│Package Version│  Description   ║
╠═══════╪════════════════════════════╪════════╪════════════╪═══════════════╪════════════════╣
║2      │Mon Nov 27 22:41:16 EST 2017│DEPLOYED│httptest    │1.0.0          │Upgrade complete║
║1      │Mon Nov 27 22:40:41 EST 2017│DELETED │httptest    │1.0.0          │Delete complete ║
╚═══════╧════════════════════════════╧════════╧════════════╧═══════════════╧════════════════╝
17.10. Stream Manifests
Skipper keeps a “manifest” of the all of the applications, their application properties, and their deployment properties after all values have been substituted.
This represents the final state of what was deployed to the platform.
You can view the manifest for any of the versions of a Stream by using the following command:
spec:
  resource: maven://org.springframework.cloud.stream.app:log-sink-rabbit
  version: 3.2.0
  applicationProperties:
    spring.cloud.dataflow.stream.app.label: log
    spring.cloud.stream.bindings.input.group: httptest
    spring.cloud.dataflow.stream.name: httptest
    spring.cloud.dataflow.stream.app.type: sink
    spring.cloud.stream.bindings.input.destination: httptest.http
  deploymentProperties:
    spring.cloud.deployer.indexed: true
    spring.cloud.deployer.group: httptest
    spring.cloud.deployer.count: 1
# Source: http.yml
apiVersion: skipper.spring.io/v1
kind: SpringCloudDeployerApplication
metadata:
  name: http
spec:
  resource: maven://org.springframework.cloud.stream.app:http-source-rabbit
  version: 3.2.0
  applicationProperties:
    spring.cloud.dataflow.stream.app.label: http
    spring.cloud.stream.bindings.output.producer.requiredGroups: httptest
    server.port: 9000
    spring.cloud.stream.bindings.output.destination: httptest.http
    spring.cloud.dataflow.stream.name: httptest
    spring.cloud.dataflow.stream.app.type: source
  deploymentProperties:
    spring.cloud.deployer.group: httptest
17.13. Skipper’s Upgrade Strategy
Skipper has a simple “red/black” upgrade strategy. It deploys the new version of the applications, using as many instances as the currently running version, and checks the /health endpoint of the application.
If the health of the new application is good, the previous application is undeployed.
If the health of the new application is bad, all new applications are undeployed, and the upgrade is considered to be not successful.
The upgrade strategy is not a rolling upgrade, so, if five instances of the application are running, then, in a sunny-day scenario, five of the new applications are also running before the older version is undeployed.
This section covers additional features of the Stream DSL not covered in the  Stream DSL introduction.
18.1. Tap a Stream
Taps can be created at various producer endpoints in a stream. See the Tapping a Stream section of the microsite for more information.
18.2. Using Labels in a Stream
When a stream is made up of multiple applications with the same name, they must be qualified with labels.
See the Labeling Applications section of the microsite for more information.
18.3. Named Destinations
Instead of referencing a source or sink application, you can use a named destination.
See the Named Destinations section of the microsite for more information.
18.4. Fan-in and Fan-out
By using named destinations, you can support fan-in and fan-out use cases.
See the Fan-in and Fan-out section of the microsite for more information.
Instead of using the shell to create and deploy streams, you can use the Java-based DSL provided by the spring-cloud-dataflow-rest-client module.
See the Java DSL section of the microsite for more information.
In some cases, a stream can have its applications bound to multiple spring cloud stream binders when they are required to connect to different messaging
middleware configurations. In those cases, you should make sure the applications are configured appropriately with their binder
configurations. For example, a multi-binder transformer that supports both Kafka and Rabbit binders is the processor in the following stream:
The Multi-Binder Transform processor receives events from RabbitMQ (rabbit1) and sends the processed events into Kafka (kafka1).
The log sink receives events from Kafka (kafka1).
Here, rabbit1 and kafka1 are the binder names given in the Spring Cloud Stream application properties.
Based on this setup, the applications have the following binders in their classpaths with the appropriate configuration:
The spring-cloud-stream binder configuration properties can be set within the applications themselves.
If not, they can be passed through deployment properties when the stream is deployed:
dataflow:>stream create --definition "http | multibindertransform --expression=payload.toUpperCase() | log" --name mystream
dataflow:>stream deploy mystream --properties "app.http.spring.cloud.stream.bindings.output.binder=rabbit1,app.multibindertransform.spring.cloud.stream.bindings.input.binder=rabbit1,
app.multibindertransform.spring.cloud.stream.bindings.output.binder=kafka1,app.log.spring.cloud.stream.bindings.input.binder=kafka1"
With Spring Cloud Stream 3.x adding functional support, you can build Source, Sink and Processor applications merely by implementing the Java Util’s Supplier, Consumer, and Function interfaces respectively.
See the Functional Application Recipe of the SCDF site for more about this feature.
dataflow:>stream create --name words --definition "http --server.port=9900 | splitter --expression=payload.split(' ') | log"
Created new stream 'words'
dataflow:>stream deploy words --properties "app.splitter.producer.partitionKeyExpression=payload,deployer.log.count=2"
Deployed stream 'words'
dataflow:>http post --target http://localhost:9900 --data "How much wood would a woodchuck chuck if a woodchuck could chuck wood"
> POST (text/plain;Charset=UTF-8) http://localhost:9900 How much wood would a woodchuck chuck if a woodchuck could chuck wood
> 202 ACCEPTED
dataflow:>runtime apps
╔════════════════════╤═══════════╤═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗
║App Id / Instance Id│Unit Status│                                                               No. of Instances / Attributes                                                               ║
╠════════════════════╪═══════════╪═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╣
║words.log-v1        │ deployed  │                                                                             2                                                                             ║
╟┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈╢
║                    │           │       guid = 24166                                                                                                                                        ║
║                    │           │        pid = 33097                                                                                                                                        ║
║                    │           │       port = 24166                                                                                                                                        ║
║words.log-v1-0      │ deployed  │     stderr = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/words-1542803461063/words.log-v1/stderr_0.log     ║
║                    │           │     stdout = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/words-1542803461063/words.log-v1/stdout_0.log     ║
║                    │           │        url = https://192.168.0.102:24166                                                                                                                   ║
║                    │           │working.dir = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/words-1542803461063/words.log-v1                  ║
╟┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈╢
║                    │           │       guid = 41269                                                                                                                                        ║
║                    │           │        pid = 33098                                                                                                                                        ║
║                    │           │       port = 41269                                                                                                                                        ║
║words.log-v1-1      │ deployed  │     stderr = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/words-1542803461063/words.log-v1/stderr_1.log     ║
║                    │           │     stdout = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/words-1542803461063/words.log-v1/stdout_1.log     ║
║                    │           │        url = https://192.168.0.102:41269                                                                                                                   ║
║                    │           │working.dir = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/words-1542803461063/words.log-v1                  ║
╟────────────────────┼───────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╢
║words.http-v1       │ deployed  │                                                                             1                                                                             ║
╟┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈╢
║                    │           │       guid = 9900                                                                                                                                         ║
║                    │           │        pid = 33094                                                                                                                                        ║
║                    │           │       port = 9900                                                                                                                                         ║
║words.http-v1-0     │ deployed  │     stderr = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/words-1542803461054/words.http-v1/stderr_0.log    ║
║                    │           │     stdout = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/words-1542803461054/words.http-v1/stdout_0.log    ║
║                    │           │        url = https://192.168.0.102:9900                                                                                                                    ║
║                    │           │working.dir = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/words-1542803461054/words.http-v1                 ║
╟────────────────────┼───────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╢
║words.splitter-v1   │ deployed  │                                                                             1                                                                             ║
╟┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈╢
║                    │           │       guid = 33963                                                                                                                                        ║
║                    │           │        pid = 33093                                                                                                                                        ║
║                    │           │       port = 33963                                                                                                                                        ║
║words.splitter-v1-0 │ deployed  │     stderr = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/words-1542803437542/words.splitter-v1/stderr_0.log║
║                    │           │     stdout = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/words-1542803437542/words.splitter-v1/stdout_0.log║
║                    │           │        url = https://192.168.0.102:33963                                                                                                                   ║
║                    │           │working.dir = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/words-1542803437542/words.splitter-v1             ║
╚════════════════════╧═══════════╧═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝
2016-06-05 18:35:47.047  INFO 58638 --- [  kafka-binder-] log.sink                                 : How
2016-06-05 18:35:47.066  INFO 58638 --- [  kafka-binder-] log.sink                                 : chuck
2016-06-05 18:35:47.066  INFO 58638 --- [  kafka-binder-] log.sink                                 : chuck
2016-06-05 18:35:47.047  INFO 58639 --- [  kafka-binder-] log.sink                                 : much
2016-06-05 18:35:47.066  INFO 58639 --- [  kafka-binder-] log.sink                                 : wood
2016-06-05 18:35:47.066  INFO 58639 --- [  kafka-binder-] log.sink                                 : would
2016-06-05 18:35:47.066  INFO 58639 --- [  kafka-binder-] log.sink                                 : a
2016-06-05 18:35:47.066  INFO 58639 --- [  kafka-binder-] log.sink                                 : woodchuck
2016-06-05 18:35:47.067  INFO 58639 --- [  kafka-binder-] log.sink                                 : if
2016-06-05 18:35:47.067  INFO 58639 --- [  kafka-binder-] log.sink                                 : a
2016-06-05 18:35:47.067  INFO 58639 --- [  kafka-binder-] log.sink                                 : woodchuck
2016-06-05 18:35:47.067  INFO 58639 --- [  kafka-binder-] log.sink                                 : could
2016-06-05 18:35:47.067  INFO 58639 --- [  kafka-binder-] log.sink                                 : wood
23.3. Other Source and Sink Application Types
This example shows something a bit more complicated: swapping out the time source for something else. Another supported source type is http, which accepts data for ingestion over HTTP POST requests. Note that the http source accepts data on a different port from the Data Flow Server (default 8080). By default, the port is randomly assigned.




    

To create a stream that uses an http source but still uses the same log sink, we would change the original command in the Simple Stream Processing example to the following:
dataflow:>runtime apps
╔══════════════════════╤═══════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗
║ App Id / Instance Id │Unit Status│                                                                    No. of Instances / Attributes                                                                    ║
╠══════════════════════╪═══════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╣
║myhttpstream.log-v1   │ deploying │                                                                                  1                                                                                  ║
╟┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈╢
║                      │           │       guid = 39628                                                                                                                                                  ║
║                      │           │        pid = 34403                                                                                                                                                  ║
║                      │           │       port = 39628                                                                                                                                                  ║
║myhttpstream.log-v1-0 │ deploying │     stderr = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/myhttpstream-1542803867070/myhttpstream.log-v1/stderr_0.log ║
║                      │           │     stdout = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/myhttpstream-1542803867070/myhttpstream.log-v1/stdout_0.log ║
║                      │           │        url = https://192.168.0.102:39628                                                                                                                             ║
║                      │           │working.dir = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/myhttpstream-1542803867070/myhttpstream.log-v1              ║
╟──────────────────────┼───────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╢
║myhttpstream.http-v1  │ deploying │                                                                                  1                                                                                  ║
╟┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┼┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈╢
║                      │           │       guid = 52143                                                                                                                                                  ║
║                      │           │        pid = 34401                                                                                                                                                  ║
║                      │           │       port = 52143                                                                                                                                                  ║
║myhttpstream.http-v1-0│ deploying │     stderr = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/myhttpstream-1542803866800/myhttpstream.http-v1/stderr_0.log║
║                      │           │     stdout = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/myhttpstream-1542803866800/myhttpstream.http-v1/stdout_0.log║
║                      │           │        url = https://192.168.0.102:52143                                                                                                                             ║
║                      │           │working.dir = /var/folders/js/7b_pn0t575l790x7j61slyxc0000gn/T/spring-cloud-deployer-6467595568759190742/myhttpstream-1542803866800/myhttpstream.http-v1             ║
╚══════════════════════╧═══════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝
See the Stream Developer Guides on the microsite for more about how to create, test, and run Spring Cloud Stream applications on your local machine.
Stream Monitoring
See the Stream Monitoring Guide on the microsite for more about how to monitor the applications that were deployed as part of a Stream.
Tasks
This section goes into more detail about how you can orchestrate Spring Cloud Task applications on Spring Cloud Data Flow.
If you are just starting out with Spring Cloud Data Flow, you should probably read the Getting Started guide for  “Local” , “Cloud Foundry”, or “Kubernetes” before diving into this section.
A task application is short-lived, meaning that it stops running on purpose and can be run on demand or scheduled for later.
One use case might be to scrape a web page and write to the database.
The Spring Cloud Task framework is based on Spring Boot and adds the ability for Boot applications to record the lifecycle events of a short-lived application, such as when it starts, when it ends, and the exit status.
The TaskExecution documentation shows which information is stored in the database.
The entry point for code execution in a Spring Cloud Task application is most often an implementation of Boot’s CommandLineRunner interface, as shown in this example.
The Spring Batch project is probably what comes to mind for Spring developers writing short-lived applications.
Spring Batch provides a much richer set of functionality than Spring Cloud Task and is recommended when processing large volumes of data.
One use case might be to read many CSV files, transform each row of data, and write each transformed row to a database.
Spring Batch provides its own database schema with a much more rich set of information about the execution of a Spring Batch job.
Spring Cloud Task is integrated with Spring Batch so that, if a Spring Cloud Task application defines a Spring Batch Job, a link between the Spring Cloud Task and Spring Cloud Batch execution tables is created.
When running Data Flow on your local machine, Tasks are launched in a separate JVM.
When running on Cloud Foundry, tasks are launched by using Cloud Foundry’s Task functionality. When running on Kubernetes, tasks are launched by using either a Pod or a Job resource.
25.1. Creating a Task Application
Spring Cloud Dataflow provides a couple of out-of-the-box task applications (timestamp-task and timestamp-batch) but most task applications require custom development.
To create a custom task application:
With this class, you need one or more CommandLineRunner or ApplicationRunner implementations within your application.  You can either implement your own or use the ones provided by Spring Boot (there is one for running batch jobs, for example).
Packaging your application with Spring Boot into an über jar is done through the standard Spring Boot conventions.
The packaged application can be registered and deployed as noted below.
When launching a task application, be sure that the database driver that is being used by Spring Cloud Data Flow is also a dependency on the task application.
For example, if your Spring Cloud Data Flow is set to use Postgresql, be sure that the task application also has Postgresql as a dependency.
When you run tasks externally (that is, from the command line) and you want Spring Cloud Data Flow to show the TaskExecutions in its UI, be sure that common datasource settings are shared among them both.
By default, Spring Cloud Task uses a local H2 instance, and the execution is recorded to the database used by Spring Cloud Data Flow.
25.2. Registering a Task Application
You can register a Task application with the App Registry by using the Spring Cloud Data Flow Shell app register command.
You must provide a unique name and a URI that can be resolved to the application artifact. For the type, specify task.
The following listing shows three examples:
dataflow:>app register --name task1 --type task --uri maven://com.example:mytask:1.0.2
dataflow:>app register --name task2 --type task --uri file:///Users/example/mytask-1.0.2.jar
dataflow:>app register --name task3 --type task --uri https://example.com/mytask-1.0.2.jar
If you would like to register multiple applications at one time, you can store them in a properties file where the keys are formatted as <type>.<name> and the values are the URIs.
For example, the following listing would be a valid properties file:
You can also pass the --local option (which is TRUE by default) to indicate whether the properties file location should be resolved within the shell process itself.
If the location should be resolved from the Data Flow Server process, specify --local false.
When using either app register or app import, if a task application is already registered with
the provided name and version, it is not overridden by default. If you would like to override the
pre-existing task application with a different uri or uri-metadata location, include the --force option.
In some cases, the resource is resolved on the server side.
In other cases, the URI is passed to a runtime container instance, where it is resolved.
Consult the specific documentation of each Data Flow Server for more detail.
25.3. Creating a Task Definition
You can create a task definition from a task application by providing a definition name as well as
properties that apply to the task execution. You can create a task definition through
the RESTful API or the shell.  To create a task definition by using the shell, use the
task create command to create the task definition, as shown in the following example:
You can obtain a listing of the current task definitions through the RESTful API or the shell.
To get the task definition list by using the shell, use the task list command.
25.3.1. Maximum Task Definition Name Length
The maximum character length of a task definition name is dependent on the platform.
25.3.2. Automating the Creation of Task Definitions
As of version 2.3.0, you can configure the Data Flow server to automatically create task definitions by setting spring.cloud.dataflow.task.autocreate-task-definitions to true.
This is not the default behavior but is provided as a convenience.
When this property is enabled, a task launch request can specify the registered task application name as the task name.
If the task application is registered, the server creates a basic task definition that specifies only the application name, as required. This eliminates a manual step similar to:
25.4. Launching a Task
An ad hoc task can be launched through the RESTful API or the shell.
To launch an ad hoc task through the shell, use the task launch command, as shown in the following example:
You can pass in additional properties meant for a TaskLauncher itself by using the --properties option.
The format of this option is a comma-separated string of properties prefixed with app.<task definition name>.<property>.
Properties are passed to TaskLauncher as application properties.
It is up to an implementation to choose how those are passed into an actual task application.
If the property is prefixed with deployer instead of app, it is passed to TaskLauncher as a deployment property, and its meaning may be TaskLauncher implementation specific.
This timestamp property is actually the same as the timestamp.format property specified by the timestamp application.
Data Flow adds the ability to use the shorthand form format instead of timestamp.format.
You can also specify the longhand version as well, as shown in the following example:
This shorthand behavior is discussed more in the section on Stream Application Properties.
If you have registered application property metadata, you can use tab completion in the shell after typing -- to get a list of candidate property names.
The shell provides tab completion for application properties. The app info --name <appName> --type <appType> shell command provides additional documentation for all the supported properties. The supported task <appType> is task.
25.4.2. Common application properties
In addition to configuration through DSL, Spring Cloud Data Flow provides a mechanism for setting properties that are common to all the task applications that are launched by it.
You can do so by adding properties prefixed with spring.cloud.dataflow.applicationProperties.task when starting the server.
The server then passes all the properties, without the prefix, to the instances it launches.
For example, you can configure all the launched applications to use the prop1 and prop2 properties by launching the Data Flow server with the following options:
Properties configured by using this mechanism have lower precedence than task deployment properties.
They are overridden if a property with the same key is specified at task launch time (for example, app.trigger.prop2
overrides the common property).
25.5. Limit the number concurrent task launches
Spring Cloud Data Flow lets a user limit the maximum number of concurrently running tasks for each configured platform to prevent the saturation of IaaS or hardware resources.
By default, the limit is set to 20 for all supported platforms. If the number of concurrently running tasks on a platform instance is greater than or equal to the limit, the next task launch request fails, and an error message is returned through the RESTful API, the Shell, or the UI.
You can configure this limit for a platform instance by setting the corresponding deployer property, spring.cloud.dataflow.task.platform.<platform-type>.accounts[<account-name>].maximumConcurrentTasks, where <account-name> is the name of a configured platform account (default if no accounts are explicitly configured).
The <platform-type> refers to one of the currently supported deployers: local or kubernetes. For cloudfoundry, the property is spring.cloud.dataflow.task.platform.<platform-type>.accounts[<account-name>].deployment.maximumConcurrentTasks.  (The difference is that deployment has been added to the path).
The TaskLauncher implementation for each supported platform determines the number of currently running tasks by querying the underlying platform’s runtime state, if possible. The method for identifying a task varies by platform.
For example, launching a task on the local host uses the LocalTaskLauncher. LocalTaskLauncher runs a process for each launch request and keeps track of these processes in memory. In this case, we do not query the underlying OS, as it is impractical to identify tasks this way.
For Cloud Foundry, tasks are a core concept supported by its deployment model. The state of all tasks ) is available directly through the API.
This means that every running task container in the account’s organization and space is included in the running execution count, whether or not it was launched by using Spring Cloud Data Flow or by invoking the CloudFoundryTaskLauncher directly.
For Kubernetes, launching a task through the KubernetesTaskLauncher, if successful, results in a running pod, which we expect to eventually complete or fail.
In this environment, there is generally no easy way to identify pods that correspond to a task.
For this reason, we count only pods that were launched by the KubernetesTaskLauncher.
Since the task launcher provides task-name label in the pod’s metadata, we filter all running pods by the presence of this label.




    

25.6. Reviewing Task Executions
Once the task is launched, the state of the task is stored in a relational database. The state
includes:
You can check the status of your task executions through the RESTful API or the shell.
To display the latest task executions through the shell, use the task execution list command.
To get a list of task executions for just one task definition, add --name and
the task definition name — for example, task execution list --name foo. To retrieve full
details for a task execution, use the task execution status command with the ID of the task execution,
for example task execution status --id 549.
25.7. Destroying a Task Definition
Destroying a task definition removes the definition from the definition repository.
This can be done through the RESTful API or the shell.
To destroy a task through the shell, use the task destroy command, as shown in the following example:
By default, the cleanup option is set to false (that is, by default, the task executions are not cleaned up when the task is destroyed).
To destroy all tasks through the shell, use the task all destroy command as shown in the following example:
task destroy <task-name> deletes only the definition and not the task deployed on Cloud Foundry.
The only way to do delete the task is through the CLI in two steps:
. Obtain a list of the apps by using the cf apps command.
. Identify the task application to be deleted and run the cf delete <task-name> command.

25.8. Validating a Task
Sometimes, an application contained within a task definition has an invalid URI in its registration.
This can be caused by an invalid URI being entered at application-registration time or the by the application being removed from the repository from which it was to be drawn.
To verify that all the applications contained in a task are resolve-able, use the validate command, as follows:
dataflow:>task validate time-stamp
╔══════════╤═══════════════╗
║Task Name │Task Definition║
╠══════════╪═══════════════╣
║time-stamp│timestamp      ║
╚══════════╧═══════════════╝
time-stamp is a valid task.
╔═══════════════╤═════════════════╗
║   App Name    │Validation Status║
╠═══════════════╪═════════════════╣
║task:timestamp │valid            ║
╚═══════════════╧═════════════════╝
In the preceding example, the user validated their time-stamp task. The task:timestamp application is valid.
Now we can see what happens if we have a stream definition with a registered application that has an invalid URI:
dataflow:>task validate bad-timestamp
╔═════════════╤═══════════════╗
║  Task Name  │Task Definition║
╠═════════════╪═══════════════╣
║bad-timestamp│badtimestamp   ║
╚═════════════╧═══════════════╝
bad-timestamp is an invalid task.
╔══════════════════╤═════════════════╗
║     App Name     │Validation Status║
╠══════════════════╪═════════════════╣
║task:badtimestamp │invalid          ║
╚══════════════════╧═════════════════╝
25.9. Stopping a Task Execution
In some cases, a task that is running on a platform may not stop because of a problem on the platform or the application business logic itself.
For such cases, Spring Cloud Data Flow offers the ability to send a request to the platform to end the task.
To do this, submit a task execution stop for a given set of task executions, as follows:
dataflow:>task execution list
╔══════════╤══╤════════════════════════════╤════════════════════════════╤═════════╗
║Task Name │ID│         Start Time         │          End Time          │Exit Code║
╠══════════╪══╪════════════════════════════╪════════════════════════════╪═════════╣
║batch-demo│5 │Mon Jul 15 13:58:41 EDT 2019│Mon Jul 15 13:58:55 EDT 2019│0        ║
║timestamp │1 │Mon Jul 15 09:26:41 EDT 2019│Mon Jul 15 09:26:41 EDT 2019│0        ║
╚══════════╧══╧════════════════════════════╧════════════════════════════╧═════════╝
When stopping a task execution that has a running Spring Batch job, the job is left with a batch status of STARTED.
Each of the supported platforms sends a SIG-INT to the task application when a stop is requested. That allows Spring Cloud Task to capture the state of the app. However, Spring Batch does not handle a SIG-INT and, as a result, the job stops but remains in the STARTED status.
25.9.1. Stopping a Task Execution that was Started Outside of Spring Cloud Data Flow
You may wish to stop a task that has been launched outside of Spring Cloud Data Flow. An example of this is the worker applications launched by a remote batch partitioned application.
In such cases, the remote batch partitioned application stores the external-execution-id for each of the worker applications. However, no platform information is stored.
So when Spring Cloud Data Flow has to stop a remote batch partitioned application and its worker applications, you need to specify the platform name, as follows:
You can also tap into various task and batch events when the task is launched.
If the task is enabled to generate task or batch events (with the additional dependencies of spring-cloud-task-stream and, in the case of Kafka as the binder, spring-cloud-stream-binder-kafka), those events are published during the task lifecycle.
By default, the destination names for those published events on the broker (Rabbit, Kafka, and others) are the event names themselves (for instance: task-events, job-execution-events, and so on).
dataflow:>task create myTask --definition "myBatchJob"
dataflow:>stream create task-event-subscriber1 --definition ":task-events > log" --deploy
dataflow:>task launch myTask
The following table lists the default task and batch event and destination names on the broker:
Table 2. Task and Batch Event Destinations
Spring Cloud Data Flow lets you create a directed graph, where each node of the graph is a task application.
This is done by using the DSL for composed tasks.
You can create a composed task through the RESTful API, the Spring Cloud Data Flow Shell, or the Spring Cloud Data Flow UI.
27.1. The Composed Task Runner
Composed tasks are run through a task application called the Composed Task Runner. The Spring Cloud Data Flow server automatically deploys the Composed Task Runner when launching a composed task.
27.1.1. Configuring the Composed Task Runner
The composed task runner application has a dataflow-server-uri property that is used for validation and for launching child tasks.
This defaults to localhost:9393. If you run a distributed Spring Cloud Data Flow server, as you would if you deploy the server on Cloud Foundry or Kubernetes, you need to provide the URI that can be used to access the server.
You can either provide this by setting the dataflow-server-uri property for the composed task runner application when launching a composed task or by setting the spring.cloud.dataflow.server.uri property for the Spring Cloud Data Flow server when it is started.
For the latter case, the dataflow-server-uri composed task runner application property is automatically set when a composed task is launched.
Configuration Options
The ComposedTaskRunner task has the following options:
composed-task-arguments
The command line arguments to be used for each of the tasks. (String, default: <none>).
increment-instance-enabled
Allows a single ComposedTaskRunner instance to be run again without changing the parameters by adding a incremented number job parameter based on run.id from the previous execution. (Boolean, default: true).
ComposedTaskRunner is built by using Spring Batch. As a result, upon a successful execution, the batch job is considered to be complete.
To launch the same ComposedTaskRunner definition multiple times, you must set  either increment-instance-enabled or uuid-instance-enabled property to true or change the parameters for the definition for each launch.
When using this option, it must be applied for all task launches for the desired application, including the first launch.
uuid-instance-enabled
Allows a single ComposedTaskRunner instance to be run again without changing the parameters by adding a UUID to the ctr.id job parameter. (Boolean, default: false).
ComposedTaskRunner is built by using Spring Batch. As a result, upon a successful execution, the batch job is considered to be complete.
To launch the same ComposedTaskRunner definition multiple times, you must set  either increment-instance-enabled or uuid-instance-enabled property to true or change the parameters for the definition for each launch.
When using this option, it must be applied for all task launches for the desired application, including the first launch. This option when set to true will override the value of increment-instance-id.
Set this option to true when running multiple instances of the same composed task definition at the same time.
interval-time-between-checks
The amount of time, in milliseconds, that the ComposedTaskRunner waits between checks of the database to see if a task has completed. (Integer, default: 10000).
ComposedTaskRunner uses the datastore to determine the status of each child tasks. This interval indicates to ComposedTaskRunner how often it should check the status its child tasks.
transaction-isolation-level
Establish the transaction isolation level for the Composed Task Runner.
A list of available transaction isolation levels can be found  here.
Default is ISOLATION_REPEATABLE_READ.
max-wait-time
The maximum amount of time, in milliseconds, that an individual step can run before the execution of the Composed task is failed (Integer, default: 0).
Determines the maximum time each child task is allowed to run before the CTR ends with a failure. The default of 0 indicates no timeout.
split-thread-allow-core-thread-timeout
Specifies whether to allow split core threads to timeout. (Boolean, default: false)
Sets the policy governing whether core threads may timeout and terminate if no tasks arrive within the keep-alive time, being replaced if needed when new tasks arrive.
split-thread-core-pool-size
Split’s core pool size. (Integer, default: 1)
Each child task contained in a split requires a thread in order to execute. So, for example, a definition such as <AAA || BBB || CCC> && <DDD || EEE> would require a split-thread-core-pool-size of 3.
This is because the largest split contains three child tasks. A count of 2 would mean that AAA and BBB would run in parallel, but CCC would wait until either AAA or BBB finish in order to run.
Then DDD and EEE would run in parallel.
split-thread-keep-alive-seconds
Split’s thread keep alive seconds. (Integer, default: 60)
If the pool currently has more than corePoolSize threads, excess threads are stopped if they have been idle for more than the keepAliveTime.
split-thread-max-pool-size
Split’s maximum pool size. (Integer, default: Integer.MAX_VALUE).
Establish the maximum number of threads allowed for the thread pool.
split-thread-queue-capacity
Capacity for Split’s BlockingQueue. (Integer, default: Integer.MAX_VALUE)
If fewer than corePoolSize threads are running, the Executor always prefers adding a new thread rather than queuing.
If corePoolSize or more threads are running, the Executor always prefers queuing a request rather than adding a new thread.
If a request cannot be queued, a new thread is created unless this would exceed maximumPoolSize. In that case, the task is rejected.
split-thread-wait-for-tasks-to-complete-on-shutdown
Whether to wait for scheduled tasks to complete on shutdown, not interrupting running tasks and running all tasks in the queue. (Boolean, default: false)
dataflow-server-uri
The URI for the Data Flow server that receives task launch requests. (String, default: localhost:9393)
dataflow-server-username
The optional username for the Data Flow server that receives task launch requests.
Used to access the the Data Flow server by using Basic Authentication. Not used if 	dataflow-server-access-token is set.
dataflow-server-password
The optional password for the Data Flow server that receives task launch requests.
Used to access the the Data Flow server by using Basic Authentication. Not used if 	dataflow-server-access-token is set.
dataflow-server-access-token
This property sets an optional OAuth2 Access Token.
Typically, the value is automatically set by using the token from the currently logged-in user, if available.
However, for special use-cases, this value can also be set explicitly.
A special boolean property, dataflow-server-use-user-access-token, exists for when you want to use the access token of the currently logged-in user and propagate it to the Composed Task Runner. This property is used
by Spring Cloud Data Flow and, if set to true, auto-populates the dataflow-server-access-token property. When using dataflow-server-use-user-access-token, it must be passed for each task execution.
In some cases, it may be preferred that the user’s dataflow-server-access-token must be passed for each composed task launch by default.
In this case, set the Spring Cloud Data Flow spring.cloud.dataflow.task.useUserAccessToken property to true.
To set a property for Composed Task Runner you will need to prefix the property with app.composed-task-runner..
For example to set the dataflow-server-uri property the property will look like app.composed-task-runner.dataflow-server-uri.
dataflow:> app register --name timestamp --type task --uri maven://org.springframework.cloud.task.app:timestamp-task:
dataflow:> app register --name mytaskapp --type task --uri file:///home/tasks/mytask.jar
dataflow:> task create my-composed-task --definition "mytaskapp && timestamp"
dataflow:> task launch my-composed-task
In the preceding example, we assume that the applications to be used by our composed task have not yet been registered.
Consequently, in the first two steps, we register two task applications.
We then create our composed task definition by using the task create command.
The composed task DSL in the preceding example, when launched, runs mytaskapp and then runs the timestamp application.
But before we launch the my-composed-task definition, we can view what Spring Cloud Data Flow generated for us.
This can be done by using the task list command, as shown (including its output) in the following example:
dataflow:>task list
╔══════════════════════════╤══════════════════════╤═══════════╗
║        Task Name         │   Task Definition    │Task Status║
╠══════════════════════════╪══════════════════════╪═══════════╣
║my-composed-task          │mytaskapp && timestamp│unknown    ║
║my-composed-task-mytaskapp│mytaskapp             │unknown    ║
║my-composed-task-timestamp│timestamp             │unknown    ║
╚══════════════════════════╧══════════════════════╧═══════════╝
In the example, Spring Cloud Data Flow created three task definitions, one for each of the applications that makes up our composed task (my-composed-task-mytaskapp and my-composed-task-timestamp) as well as the composed task (my-composed-task) definition.
We also see that each of the generated names for the child tasks is made up of the name of the composed task and the name of the application, separated by a hyphen - (as in my-composed-task - mytaskapp).




    

Task Application Parameters
The task applications that make up the composed task definition can also contain parameters, as shown in the following example:
dataflow:>task execution list
╔══════════════════════════╤═══╤════════════════════════════╤════════════════════════════╤═════════╗
║        Task Name         │ID │         Start Time         │          End Time          │Exit Code║
╠══════════════════════════╪═══╪════════════════════════════╪════════════════════════════╪═════════╣
║my-composed-task-timestamp│713│Wed Apr 12 16:43:07 EDT 2017│Wed Apr 12 16:43:07 EDT 2017│0        ║
║my-composed-task-mytaskapp│712│Wed Apr 12 16:42:57 EDT 2017│Wed Apr 12 16:42:57 EDT 2017│0        ║
║my-composed-task          │711│Wed Apr 12 16:42:55 EDT 2017│Wed Apr 12 16:43:15 EDT 2017│0        ║
╚══════════════════════════╧═══╧════════════════════════════╧════════════════════════════╧═════════╝
In the preceding example, we see that my-compose-task launched and that the other tasks were also launched in sequential order.
Each of them ran successfully with an Exit Code as 0.
Passing Properties to the Child Tasks
To set the properties for child tasks in a composed task graph at task launch time,
use the following format: app.<child task app name>.<property>.
The following listing shows a composed task definition as an example:
task launch my-composed-task --properties "deployer.mytaskapp.memory=2048m,app.mytimestamp.timestamp.format=HH:mm:ss"
Launched task 'a1'
dataflow:>task create my-composed-task --definition "<aaa: timestamp || bbb: timestamp>"
Created new task 'my-composed-task'
dataflow:>task launch my-composed-task --arguments "--increment-instance-enabled=true --max-wait-time=50000 --split-thread-core-pool-size=4" --properties "app.bbb.timestamp.format=dd/MM/yyyy HH:mm:ss"
Launched task 'my-composed-task'
If no ExitMessage is present and the ExitCode is set to zero, the ExitStatus for the step is COMPLETED.
If no ExitMessage is present and the ExitCode is set to any non-zero number, the ExitStatus for the step is FAILED.
27.2.3. Destroying a Composed Task
The command used to destroy a stand-alone task is the same as the command used to destroy a composed task.
The only difference is that destroying a composed task also destroys the child tasks associated with it.
The following example shows the task list before and after using the destroy command:
dataflow:>task list
╔══════════════════════════╤══════════════════════╤═══════════╗
║        Task Name         │   Task Definition    │Task Status║
╠══════════════════════════╪══════════════════════╪═══════════╣
║my-composed-task          │mytaskapp && timestamp│COMPLETED  ║
║my-composed-task-mytaskapp│mytaskapp             │COMPLETED  ║
║my-composed-task-timestamp│timestamp             │COMPLETED  ║
╚══════════════════════════╧══════════════════════╧═══════════╝
dataflow:>task destroy my-composed-task
dataflow:>task list
╔═════════╤═══════════════╤═══════════╗
║Task Name│Task Definition│Task Status║
╚═════════╧═══════════════╧═══════════╝
To stop a composed task through the dashboard, select the Jobs tab and click the *Stop() button next to the job execution that you want to stop.
The composed task run is stopped when the currently running child task completes.
The step associated with the child task that was running at the time that the composed task was stopped is marked as STOPPED as well as the composed task job execution.
27.2.5. Restarting a Composed Task
In cases where a composed task fails during execution and the status of the composed task is FAILED, the task can be restarted.
You can do so through the:
28.1. Conditional Execution
Conditional execution is expressed by using a double ampersand symbol (&&).
This lets each task in the sequence be launched only if the previous task
successfully completed, as shown in the following example:
When the composed task called my-composed-task is launched, it launches the task called task1 and, if task1 completes successfully, the task called task2 is launched.
If task1 fails, task2 does not launch.
You can also use the Spring Cloud Data Flow Dashboard to create your conditional execution, by using the designer to drag and drop applications that are required and connecting them together to create your directed graph, as shown in the following image:
The preceding diagram is a screen capture of the directed graph as it being created by using the Spring Cloud Data Flow Dashboard.
You can see that four components in the diagram comprise a conditional execution:
28.2. Transitional Execution
The DSL supports fine-grained control over the transitions taken during the execution of the directed graph.
Transitions are specified by providing a condition for equality that is based on the exit status of the previous task.
A task transition is represented by the following symbol ->.
28.2.1. Basic Transition
A basic transition would look like the following:
In the preceding example, foo would launch, and, if it had an exit status of FAILED, the bar task would launch.
If the exit status of foo was COMPLETED, baz would launch.
All other statuses returned by cat have no effect, and the task would end normally.
Using the Spring Cloud Data Flow Dashboard to create the same “basic transition” would resemble the following image:
The preceding diagram is a screen capture of the directed graph as it being created in the Spring Cloud Data Flow Dashboard.
Notice that there are two different types of connectors:
Dashed line: Represents transitions from the application to one of the possible destination applications.
Solid line: Connects applications in a conditional execution or a connection between the application and a control node (start or end).
When creating a transition, link the application to each possible destination by using the connector.
Once complete, go to each connection and select it by clicking it.
A bolt icon appears.
Click that icon.
Enter the exit status required for that connector.
The solid line for that connector turns to a dashed line.
In the preceding example, foo would launch, and, if it had an exit status of FAILED, bar task would launch.
For any exit status of cat other than FAILED, baz would launch.
Using the Spring Cloud Data Flow Dashboard to create the same “transition with wildcard” would resemble the following image:
28.2.3. Transition With a Following Conditional Execution
A transition can be followed by a conditional execution, so long as the wildcard
is not used, as shown in the following example:
In the preceding example, foo would launch, and, if it had an exit status of FAILED, the bar task would launch.
If foo had an exit status of UNKNOWN, baz would launch.
For any exit status of foo other than FAILED or UNKNOWN, qux would launch and, upon successful completion, quux would launch.
Using the Spring Cloud Data Flow Dashboard to create the same “transition with conditional execution” would resemble the following image:
28.2.4. Ignoring Exit Message
If any child task within a split returns an ExitMessage other than COMPLETED the split
will have an ExitStatus of FAILED.  To ignore the  ExitMessage of a child task,
add the ignoreExitMessage=true for each app that will return an ExitMessage
within the split.  When using this flag, the ExitStatus of the task will be
COMPLETED if the ExitCode of the child task is zero.  The split will have an
ExitStatus of FAILED if the ExitCode`s is non zero.  There are 2 ways to
set the `ignoreExitMessage flag:
Setting the property for each of the apps that need to have their exitMessage
ignored within the split.  For example a split like <AAA || BBB> where BBB
will return an exitMessage, you would set the ignoreExitMessage property like
app.BBB.ignoreExitMessage=true
You can also set it for all apps using the composed-task-arguments property,
for example: --composed-task-arguments=--ignoreExitMessage=true.
28.3. Split Execution
Splits let multiple tasks within a composed task be run in parallel.
It is denoted by using angle brackets (<>) to group tasks and flows that are to be run in parallel.
These tasks and flows are separated by the double pipe || symbol, as shown in the following example:
In the preceding example, the foo, bar, and baz tasks are launched in parallel.
Once they all complete, then the qux and quux tasks are launched in parallel.
Once they complete, the composed task ends.
However, if foo, bar, or baz fails, the split containing qux and quux does not launch.
Using the Spring Cloud Data Flow Dashboard to create the same “split with multiple groups” would resemble the following image:
Tasks that are used in a split should not set the their ExitMessage. Setting the ExitMessage is only to be used
with  transitions.
In the preceding example, we see that foo and baz are launched in parallel.
However, bar does not launch until foo completes successfully.
Using the Spring Cloud Data Flow Dashboard to create the same " split containing conditional execution " resembles the following image:
28.3.2. Establishing the Proper Thread Count for Splits
Each child task contained in a split requires a thread in order to run. To set this properly, you want to look at your graph and find the split that has the largest number of child tasks. The number of child tasks in that split is the number of threads you need.
To set the thread count, use the split-thread-core-pool-size property (defaults to 1). So, for example, a definition such as <AAA || BBB || CCC> && <DDD || EEE> requires a split-thread-core-pool-size of 3.
This is because the largest split contains three child tasks. A count of two would mean that AAA and BBB would run in parallel but CCC would wait for either AAA or BBB to finish in order to run.
Then DDD and EEE would run in parallel.
You can launch a task from a stream by using the task-launcher-dataflow sink which is provided as a part of the Spring Cloud Data Flow project.
The sink connects to a Data Flow server and uses its REST API to launch any defined task.
The sink accepts a JSON payload representing a task launch request, which provides the name of the task to launch and may include command line arguments and deployment properties.
The task-launch-request-function component, in conjunction with Spring Cloud Stream functional composition, can transform the output of any source or processor to a task launch request.
Adding a dependency to task-launch-request-function auto-configures a java.util.function.Function implementation, registered through Spring Cloud Function as a taskLaunchRequest.
For example, you can start with the time source, add the following dependency, build it, and register it as a custom source.
<dependency>
    <groupId>org.springframework.cloud.stream.app</groupId>
    <artifactId>app-starters-task-launch-request-common</artifactId>
</dependency>
This will create an apps directory that contains  time-source-rabbit and time-source-kafka directories in the <stream app project>/applications/source/time-source directory.   In each of these you will see a target directory that contains a time-source-<binder>-<version>.jar. Now  register the time-source jar (use the appropriate binder jar) with SCDF as a time source named timestamp-tlr.
Next, register the task-launcher-dataflow sink with SCDF and create a task definition timestamp-task.   Once this is complete create the stream definition as shown below:
The preceding stream produces a task launch request every minute. The request provides the name of the task to launch: {"name":"timestamp-task"}.
The following stream definition illustrates the use of command line arguments. It produces messages such as {"args":["foo=bar","time=12/03/18 17:44:12"],"deploymentProps":{},"name":"timestamp-task"} to provide command-line arguments to the task:
stream create --name task-every-second --definition 'timestamp-tlr --task.launch.request.task-name=timestamp-task --spring.cloud.function.definition=\"timeSupplier|taskLaunchRequestFunction\" --task.launch.request.args=foo=bar --task.launch.request.arg-expressions=time=payload | tasklauncher-sink'   --deploy
Note the use of SpEL expressions to map each message payload to the time command-line argument, along with a static argument (foo=bar).
You can then see the list of task executions by using the shell command task execution list, as shown (with its output) in the following example:
dataflow:>task execution list
╔══════════════╤═══╤════════════════════════════╤════════════════════════════╤═════════╗
║  Task Name   │ID │         Start Time         │          End Time          │Exit Code║
╠══════════════╪═══╪════════════════════════════╪════════════════════════════╪═════════╣
║timestamp-task│581│Thu Sep 08 11:38:33 EDT 2022│Thu Sep 08 11:38:33 EDT 2022│0        ║
║timestamp-task│580│Thu Sep 08 11:38:31 EDT 2022│Thu Sep 08 11:38:31 EDT 2022│0        ║
║timestamp-task│579│Thu Sep 08 11:38:29 EDT 2022│Thu Sep 08 11:38:29 EDT 2022│0        ║
║timestamp-task│578│Thu Sep 08 11:38:26 EDT 2022│Thu Sep 08 11:38:26 EDT 2022│0        ║
╚══════════════╧═══╧════════════════════════════╧════════════════════════════╧═════════╝
In this example, we have shown how to use the time source to launch a task at a fixed rate.
This pattern may be applied to any source to launch a task in response to any event.
29.1. Launching a Composed Task From a Stream
A composed task can be launched with the task-launcher-dataflow sink, as discussed here.
Since we use the ComposedTaskRunner directly, we need to set up the task definitions for the composed task runner itself, along with the composed tasks, prior to the creation of the composed task launching stream.
Suppose we wanted to create the following composed task definition: AAA && BBB.
The first step would be to create the task definition, as shown in the following example:
Now that the task definition we need for composed task definition is ready, we need to create a stream that launches composed-task-sample.
We create a stream with:
As discussed in the Tasks documentation, Spring
Cloud Data Flow lets you view Spring Cloud Task application executions. So, in
this section, we discuss what is required for a task application and Spring
Cloud Data Flow to share the task execution information.




    

30.1. A Common DataStore Dependency
Spring Cloud Data Flow supports many databases out-of-the-box,
so all you typically need to do is declare the spring_datasource_* environment variables
to establish what data store Spring Cloud Data Flow needs.
Regardless of which database you decide to use for Spring Cloud Data Flow, make sure that your task also
includes that database dependency in its pom.xml or gradle.build file. If the database dependency
that is used by Spring Cloud Data Flow is not present in the Task Application, the task fails
and the task execution is not recorded.
30.2. A Common Data Store
Spring Cloud Data Flow and your task application must access the same datastore instance.
This is so that the task executions recorded by the task application can be read by Spring Cloud Data Flow to list them in the Shell and Dashboard views.
Also, the task application must have read and write privileges to the task data tables that are used by Spring Cloud Data Flow.
Given this understanding of the datasource dependency between Task applications and Spring Cloud Data Flow, you can now review how to apply them in various Task orchestration scenarios.
30.2.1. Simple Task Launch
When launching a task from Spring Cloud Data Flow, Data Flow adds its datasource
properties (spring.datasource.url, spring.datasource.driverClassName, spring.datasource.username, spring.datasource.password)
to the application properties of the task being launched. Thus, a task application
records its task execution information to the Spring Cloud Data Flow repository.
30.2.2. Composed Task Runner
Spring Cloud Data Flow lets you create a directed graph where each node
of the graph is a task application. This is done through the
composed task runner.
In this case, the rules that applied to a simple task launch
or task launcher sink apply to the composed task runner as well.
All child applications must also have access to the datastore that is being used by the composed task runner.
Also, all child applications must have the same database dependency as the composed task runner enumerated in their pom.xml or gradle.build file.
30.2.3. Launching a Task Externally from Spring Cloud Data Flow
You can launch Spring Cloud Task applications by using another method (scheduler, for example) but still track the task execution in Spring Cloud Data Flow.
You can do so, provided the task applications observe the rules specified here and here.
If you want to use Spring Cloud Data Flow to view your
Spring Batch jobs, make sure that
your batch application uses the @EnableTask annotation and follow the rules enumerated here and here.
More information is available here.
Spring Cloud Data Flow lets you schedule the execution of tasks with a cron expression.
You can create a schedule through the RESTful API or the Spring Cloud Data Flow UI.
31.1. The Scheduler
Spring Cloud Data Flow schedules the execution of its tasks through a scheduling agent that is available on the cloud platform.
When using the Cloud Foundry platform, Spring Cloud Data Flow uses the PCF Scheduler.
When using Kubernetes, a CronJob will be used.
dataflow:>task schedule create --definitionName mytask --name mytaskschedule --expression '*/1 * * * *'
Created schedule 'mytaskschedule'
Maximum Length for a Schedule Name
The maximum character length of a schedule name is dependent on the platform.
Table 3. Maximum Schedule Name Character Length by Platform
dataflow:>task schedule list
╔══════════════════════════╤════════════════════╤════════════════════════════════════════════════════╗
║      Schedule Name       │Task Definition Name│                     Properties                     ║
╠══════════════════════════╪════════════════════╪════════════════════════════════════════════════════╣
║mytaskschedule            │mytask              │spring.cloud.scheduler.cron.expression = */1 * * * *║
╚══════════════════════════╧════════════════════╧════════════════════════════════════════════════════╝
As task applications evolve, you want to get your updates to production. This section walks through the capabilities that Spring Cloud Data Flow provides around being able to update task applications.
When a task application is registered (see Registering a Task Application), a version is associated with it.  A task application can have multiple versions associated with it, with one selected as the default. The following image illustrates an application with multiple versions associated with it (see the timestamp entry).
Versions of an application are managed by registering multiple applications with the same name and coordinates, except the version. For example, if you were to register an application with the following values, you would get one application registered with two versions (2.1.0.RELEASE and 2.1.1.RELEASE):
Besides having multiple versions, Spring Cloud Data Flow needs to know which version to run on the next launch. This is indicated by setting a version to be the default version. Whatever version of a task application is configured as the default version is the one to be run on the next launch request. You can see which version is the default in the UI, as this image shows:
32.1. Task Launch Lifecycle
In previous versions of Spring Cloud Data Flow, when the request to launch a task was received, Spring Cloud Data Flow would deploy the application (if needed) and run it. If the application was being run on a platform that did not need to have the application deployed every time (CloudFoundry, for example), the previously deployed application was used. This flow has changed in 2.3. The following image shows what happens when a task launch request comes in now:
There are three main flows to consider in the preceding diagram. Launching the first time or launching with no changes is one. The other two are launching when there are changes but the appliction is not currently and launching when there are changes and the application is running. We look at the flow with no changes first.
32.1.1. Launching a Task With No Changes
A launch request comes into Data Flow. Data Flow determines that an upgrade is not required, since nothing has changed (no properties, deployment properties, or versions have changed since the last execution).
On platforms that cache a deployed artifact (CloudFoundry, at this writing), Data Flow checks whether the application was previously deployed.
If the application needs to be deployed, Data Flow deploys the task application.
Data Flow launches the application.
A launch request comes into Data Flow. Data Flow determines that an upgrade is required, since there was a change in the task application version, the application properties, or the deployment properties.
Data Flow checks to see whether another instance of the task definition is currently running.
If there is no other instance of the task definition currently running, the old deployment is deleted.
On platforms that cache a deployed artifact (CloudFoundry, at this writing), Data Flow checks whether the application was previously deployed (this check evaluates to false in this flow, since the old deployment was deleted).
Data Flow does the deployment of the task application with the updated values (new application version, new merged properties, and new merged deployment properties).
Data Flow launches the application.
32.1.3. Launch a Task With Changes While Another Instance Is Running
The last main flow is when a launch request comes to Spring Cloud Data Flow to do an upgrade but the task definition is currently running. In this case, the launch is blocked due to the requirement to delete the current application. On some platforms (CloudFoundry, at this writing), deleting the application causes all currently running applications to be shut down. This feature prevents that from happening. The following process describes what happens when a task changes while another instance is running:
A launch request comes into Data Flow. Data Flow determines that an upgrade is required, since there was a change in the task application version, the application properties, or the deployment properties.
Data Flow checks to see whether another instance of the task definition is currently running.
Data Flow prevents the launch from happening, because other instances of the task definition are running.
See the Batch Developer section of the microsite for more about how to create, test, and run Spring Cloud Task applications on your local machine.
Task Monitoring
See the Task Monitoring Guide of the microsite for more about how to monitor the applications that were deployed as part of a task.
Dashboard
This section describes how to use the dashboard of Spring Cloud Data Flow.
Apps: The Apps tab lists all available applications and provides the controls to register and unregister them.
Runtime: The Runtime tab provides the list of all running applications.
Streams: The Streams tab lets you list, design, create, deploy, and destroy Stream Definitions.
Tasks: The Tasks tab lets you list, create, launch, schedule, and destroy Task Definitions.
Jobs: The Jobs tab lets you perform batch job related functions.
For example, if Spring Cloud Data Flow is running locally, the dashboard is available at localhost:9393/dashboard.
If you have enabled HTTPS, the dashboard is available at localhost:9393/dashboard.
If you have enabled security, a login form is available at localhost:9393/dashboard/#/login.
The Applications tab of the dashboard lists all the available applications and provides the controls to register and unregister them (if applicable).
You can import a number of applications at once by using the Bulk Import Applications action.
The following image shows a typical list of available applications within the dashboard:
34.1. Bulk Import of Applications
Applications can be imported in numerous ways which are available on the "Applications" page.
For bulk import, the application definitions are expected to be expressed in a properties style, as follows:
In the "Import application coordinates from an HTTP URI location" section, you can specify a URI that points to a properties file stored elsewhere, it should contain properties formatted as shown in the previous example.
Alternatively, by using the Apps as Properties textbox in the "Import application coordinates from a properties file" section , you can directly list each property string. Finally, if the properties are stored in a local file, the Import a File option opens a local file browser to select the file.
After setting your definitions through one of these routes, click Import Application(s).
The following image shows an example page of one way to bulk import applications:
The Runtime tab of the Dashboard application shows the list of all running applications.
For each runtime applicaiton, the state of the deployment and the number of deployed instances is shown.
A list of the used deployment properties is available by clicking on the application ID.
The following image shows an example of the Runtime tab in use:
36.1. Working with Stream Definitions
The Streams section of the Dashboard includes the Definitions tab that provides a listing of stream definitions.
There you have the option to deploy or undeploy those stream definitions.
Additionally, you can remove the definition by clicking on Destroy.
Each row includes an arrow on the left, which you can click to see a visual representation of the definition.
Hovering over the boxes in the visual representation shows more details about the applications, including any options passed to them.
In the following screenshot, the timer stream has been expanded to show the visual representation:
If you click the details button, the view changes to show a visual representation of that stream and any related streams.
In the preceding example, if you click details for the timer stream, the view changes to the following view, which clearly shows the relationship between the three streams (two of them are tapping into the timer stream):
36.2. Creating a Stream
The Streams section of the Dashboard includes the Create Stream tab, which makes the Spring Flo designer available. The designer is a canvas application that offers an interactive graphical interface for creating data pipelines.
In this tab, you can:
You should watch this screencast that highlights some of the "Flo for Spring Cloud Data Flow" capabilities.
The Spring Flo wiki includes more detailed content on core Flo capabilities.
The following image shows the Flo designer in use:
36.3. Deploying a Stream
The stream deploy page includes tabs that provide different ways to set up the deployment properties and deploy the stream.
The following screenshots show the stream deploy page for foobar (time | log).
You can define deployments properties by using:
Form builder tab: a builder that helps you to define deployment properties (deployer, application properties, and so on)
Free text tab: a free text area (for key-value pairs)
36.5. Creating Fan-In and Fan-Out Streams
In the Fan-in and Fan-out chapter, you can learn how to support fan-in and fan-out use cases by  using named destinations.
The UI provides dedicated support for named destinations as well:
In this example, we have data from an HTTP Source and a JDBC Source that is being sent to the
sharedData channel, which represents a fan-in use case.
On the other end we have a Cassandra Sink and a File Sink subscribed to the sharedData channel, which represents a fan-out use case.
36.6. Creating a Tap Stream
Creating taps by using the Dashboard is straightforward.
Suppose you have a stream consisting of an HTTP Source and a File Sink and you would like to tap into the stream
to also send data to a JDBC Sink.
To create the tap stream, connect the output connector of the HTTP Source to the JDBC Sink.
The connection is displayed as a dotted line, indicating that you created a tap stream.
The primary stream (HTTP Source to File Sink) will be automatically named, in case you did not provide a name for the stream, yet.
When creating tap streams, the primary stream must always be explicitly named.
In the preceding image, the primary stream was named HTTP_INGEST.
By using the Dashboard, you can also switch the primary stream so that it becomes the secondary tap stream.
Hover over the existing primary stream, the line between HTTP Source and File Sink.
Several control icons appear, and, by clicking on the icon labeled Switch to/from tap,
you change the primary stream into a tap stream.
Do the same for the tap stream and switch it to a primary stream.
When interacting directly with named destinations,
there can be "n" combinations (Inputs/Outputs). This allows you to create complex topologies involving a
wide variety of data sources and destinations.
36.7. Import and Export Streams
The Import/Export tab of the Dashboard includes a page that provides the option to import and export streams.
The following image shows the streams export page:
Each application encapsulates a unit of work into a reusable component.
Within the Data Flow runtime environment, applications let you create definitions for streams as well as tasks.
Consequently, the Apps tab within the Tasks tab lets you create task definitions.




    

37.2. Definitions
This page lists the Data Flow task definitions and provides actions to launch or destroy those tasks.
The following image shows the Definitions page:
On this page, you can also specify various properties that are used during the deployment of the application.
Once you are satisfied with the task definition, you can click the CREATE TASK button.  A dialog box then asks for a task definition name and description. At a minimum, you must provide a name for the new definition.
37.2.2. Creating Composed Task Definitions
The dashboard includes the Create Composed Task tab, which provides an interactive graphical interface for creating composed tasks.
In this tab, you can:
37.2.3. Launching Tasks
Once the task definition has been created, you can launch the tasks through the dashboard.
To do so, click the Tasks tab and select the task you want to launch by pressing Launch.
The following image shows the Task Launch page:
37.2.4. Import/Export Tasks
The Import/Export page  provides the option to import and export tasks.  This is done by clicking the Import/Export option on the left side of page.  From here, click the Export task(s): Create a JSON file with the selected tasks option. The Export Tasks(s) page appears.
The following image shows the tasks export page:
Similarly, you can import task definitions. To do so, click the Import/Export option on the left side of page. From here, click the Import task(s): Import tasks from a JSON file option to show the Import Tasks page.  On the Import Tasks page, you have to import from a valid JSON file. You can either manually draft the file or export the file from the Tasks Export page.
37.3. Executions
The Task Executions tab shows the current running and completed task executions. From this page, you can drill down into the Task Execution details page. Furthermore, you can relaunch a Task Execution or stop a running execution.
Finally, you can clean up one or more task executions. This operation removes any associated task or batch job from the underlying persistence store. This operation can only be triggered for parent task executions and cascades down to the child task executions (if there are any).
The following image shows the Executions tab:
Job Execution IDs links (Clicking the Job Execution Id will take you to the Job Execution Details for that Job Execution ID.)
Task Execution Duration
Task Execution Exit Message
Logging output from the Task Execution
37.4.1. Stop Executing Tasks
To submit a stop task execution request to the platform, click the drop down button next to the task execution that needs to be stopped.
Now click the Stop task option. The dashboard presents a dialog box asking if you are sure that you want to stop the task execution. If so, click Stop Task Execution(s).
The Job Executions tab of the Dashboard lets you inspect batch jobs.
The main section of the screen provides a list of job executions.
Batch jobs are tasks that each execute one or more batch jobs.
Each job execution has a reference to the task execution ID (in the Task ID column).
The list of job executions also shows the state of the underlying Job Definition.
Thus, if the underlying definition has been deleted, “No definition found” appears in the Status column.
You can take the following actions for each job:
38.1. Job Execution Details
After you have launched a batch job, the Job Execution Details page shows information about the job.
The following image shows the Job Execution Details page:
38.2. Step Execution Details
The Step Execution Details page provides information about an individual step within a job.
The following image shows the Step Execution Details page:
For exceptions, the Exit Description field contains additional error information.
However, this field can have a maximum of 2500 characters.
Therefore, in the case of long exception stack traces, trimming of error messages may occur.
When that happens, check the server log files for further details.
38.3. Step Execution History
Under Step Execution History, you can also view various metrics associated with the selected step, such as duration, read counts, write counts, and others across all of its executions.
For each metric there are 5 attributes:
Count - The number of step executions that the metric could have participated.  It is not a count for the number of times the event occurred during each step execution.
Min - The minimum value for the metric across all the executions for this step.
Max - The maximum value for the metric across all the executions for this step.
Mean - The mean value for the metric across all the executions for this step.
Standard Deviation - The standard deviation for the metric across all the executions for this step.
Commit Count - The max, min, mean, and standard deviation for the number of commits of all the executions for the given step.
Duration - The max, min, mean, and standard deviation for the duration of all the executions for the given step.
Duration Per Read - The max, min, mean, and standard deviation for the duration per read of all the executions for the given step.
FilterCount - The max, min, mean, and standard deviation for the number of filters of all the executions for the given step.
Process Skip Count - The max, min, mean, and standard deviation for the process skips of all the executions for the given step.
Read Count - The max, min, mean, and standard deviation for the number of reads of all the executions for the given step.
Read Skip Count - The max, min, mean, and standard deviation for the number of read skips of all the executions for the given step.
Rollback Count - The max, min, mean, and standard deviation for the number of rollbacks of all the executions for the given step.
Write Count - The max, min, mean, and standard deviation for the number of writes of all the executions for the given step.
Write Skip Count - The max, min, mean, and standard deviation for the number of skips of all the executions for the given step.
The Auditing page of the Dashboard gives you access to recorded audit events. Audit events
are recorded for:
By clicking the show details icon (the “i” in a circle on the right), you can obtain further details regarding
the auditing details:
The written value of the audit data property depends on the performed audit operation and the action type.
For example, when a schedule is being created, the name of the task definition,
task definition properties, deployment properties, and command line arguments are written
to the persistence store.
Sensitive information is sanitized prior to saving the Audit Record, in a best-effort manner.
Any of the following keys are being detected and their sensitive values are
masked:
Spring Cloud Data Flow provides a REST API that lets you access all aspects of the server.
In fact, the Spring Cloud Data Flow shell is a first-class consumer of that API.
41.1. HTTP Version
Spring Cloud Data Flow establishes a RESTful API version that is updated when there is a breaking change to the API.
The API version can be seen at the end of the home page of Spring Cloud Data Flow as shown in the example below:
Used to update an existing resource, including partial updates. Also used for
resources that imply the concept of restarts, such as tasks.
DELETE
Used to delete an existing resource.
201 Created
A new resource has been created successfully. The resource’s URI is available from the response’s Location header.
204 No Content
An update to an existing resource has been applied successfully.
400 Bad Request
The request was malformed. The response body includes an error description that provides further information.
404 Not Found
The requested resource did not exist.
409 Conflict
The requested resource already exists. For example, the task already exists or the stream was already being deployed
422 Unprocessable Entity
Returned in cases where the job execution cannot be stopped or restarted.
41.6. Hypermedia
Spring Cloud Data Flow uses hypermedia, and resources include links to other resources
in their responses.
Responses are in the Hypertext Application from resource-to-resource Language (HAL) format.
Links can be found beneath the _links key.
Users of the API should not create URIs themselves.
Instead, they should use the above-described links to navigate.
_links.streams/definitions/definition.href
String
Link to the streams/definitions/definition
_links.streams/definitions/definition.templated
Boolean
Link streams/definitions/definition is templated
_links.runtime/apps.href
String
Link to the runtime/apps
_links.runtime/apps/{appId}.href
String
Link to the runtime/apps/{appId}
_links.runtime/apps/{appId}.templated
Boolean
Link runtime/apps is templated
_links.runtime/apps/{appId}/instances.href
String
Link to the runtime/apps/{appId}/instances
_links.runtime/apps/{appId}/instances.templated
Boolean
Link runtime/apps/{appId}/instances is templated
_links.runtime/apps/{appId}/instances/{instanceId}.href
String
Link to the runtime/apps/{appId}/instances/{instanceId}
_links.runtime/apps/{appId}/instances/{instanceId}.templated
Boolean
Link runtime/apps/{appId}/instances/{instanceId} is templated
_links.runtime/apps/{appId}/instances/{instanceId}/post.href
String
Link to the runtime/apps/{appId}/instances/{instanceId}/post
_links.runtime/apps/{appId}/instances/{instanceId}/post.templated
Boolean
Link runtime/apps/{appId}/instances/{instanceId}/post is templated
_links.runtime/apps/{appId}/instances/{instanceId}/actuator[].href
String
Link to the runtime/apps/{appId}/instances/{instanceId}/actuator
_links.runtime/apps/{appId}/instances/{instanceId}/actuator[].templated
Boolean
Link runtime/apps/{appId}/instances/{instanceId}/actuator is templated
_links.runtime/streams.href
String
Link to the runtime/streams
_links.runtime/streams.templated
Boolean
Link runtime/streams is templated
_links.runtime/streams/{streamNames}.href
String
Link to the runtime/streams/{streamNames}
_links.runtime/streams/{streamNames}.templated
Boolean
Link runtime/streams/{streamNames} is templated
_links.streams/logs.href
String
Link to the streams/logs
_links.streams/logs/{streamName}.href
String
Link to the streams/logs/{streamName}
_links.streams/logs/{streamName}/{appName}.href
String
Link to the streams/logs/{streamName}/{appName}
_links.streams/logs/{streamName}.templated
Boolean
Link streams/logs/{streamName} is templated
_links.streams/logs/{streamName}/{appName}.templated
Boolean
Link streams/logs/{streamName}/{appName} is templated
_links.streams/deployments
Object
Link to streams/deployments
_links.streams/deployments.href
String
Link to streams/deployments
_links.streams/deployments/{name}
Object
Link streams/deployments/{name} is templated
_links.streams/deployments/{name}.href
String
Link streams/deployments/{name} is templated
_links.streams/deployments/{name}.templated
Boolean
Link streams/deployments/{name} is templated
_links.streams/deployments/{name}{?reuse-deployment-properties}.href
String
Link streams/deployments/{name} is templated
_links.streams/deployments/{name}{?reuse-deployment-properties}.templated
Boolean
Link streams/deployments/{name} is templated
_links.streams/deployments/deployment.href
String
Link to the streams/deployments/deployment
_links.streams/deployments/deployment.templated
Boolean
Link streams/deployments/deployment is templated
_links.streams/deployments/manifest/{name}/{version}.href
String
Link to the streams/deployments/manifest/{name}/{version}
_links.streams/deployments/manifest/{name}/{version}.templated
Boolean
Link streams/deployments/manifest/{name}/{version} is templated
_links.streams/deployments/history/{name}.href
String
Link to the streams/deployments/history/{name}
_links.streams/deployments/history/{name}.templated
Boolean
Link streams/deployments/history is templated
_links.streams/deployments/rollback/{name}/{version}.href
String
Link to the streams/deployments/rollback/{name}/{version}
_links.streams/deployments/rollback/{name}/{version}.templated
Boolean
Link streams/deployments/rollback/{name}/{version} is templated
_links.streams/deployments/update/{name}.href
String
Link to the streams/deployments/update/{name}
_links.streams/deployments/update/{name}.templated
Boolean
Link streams/deployments/update/{name} is templated
_links.streams/deployments/platform/list.href
String
Link to the streams/deployments/platform/list
_links.streams/deployments/scale/{streamName}/{appName}/instances/{count}.href
String
Link to the streams/deployments/scale/{streamName}/{appName}/instances/{count}
_links.streams/deployments/scale/{streamName}/{appName}/instances/{count}.templated




    

Boolean
Link streams/deployments/scale/{streamName}/{appName}/instances/{count} is templated
_links.streams/validation.href
String
Link to the streams/validation
_links.streams/validation.templated
Boolean
Link streams/validation is templated
_links.tasks/platforms.href
String
Link to the tasks/platforms
_links.tasks/definitions.href
String
Link to the tasks/definitions
_links.tasks/definitions/definition.href
String
Link to the tasks/definitions/definition
_links.tasks/definitions/definition.templated
Boolean
Link tasks/definitions/definition is templated
_links.tasks/executions.href
String
Link to the tasks/executions
_links.tasks/executions/name.href
String
Link to the tasks/executions/name
_links.tasks/executions/name.templated
Boolean
Link tasks/executions/name is templated
_links.tasks/executions/current.href
String
Link to the tasks/executions/current
_links.tasks/executions/execution.href
String
Link to the tasks/executions/execution
_links.tasks/executions/execution.templated
Boolean
Link tasks/executions/execution is templated
_links.tasks/info/executions.href
String
Link to the tasks/info/executions
_links.tasks/info/executions.templated
Boolean
Link tasks/info is templated
_links.tasks/logs.href
String
Link to the tasks/logs
_links.tasks/logs.templated
Boolean
Link tasks/logs is templated
_links.tasks/schedules.href
String
Link to the tasks/executions/schedules
_links.tasks/schedules/instances.href
String
Link to the tasks/schedules/instances
_links.tasks/schedules/instances.templated
Boolean
Link tasks/schedules/instances is templated
_links.tasks/validation.href
String
Link to the tasks/validation
_links.tasks/validation.templated
Boolean
Link tasks/validation is templated
_links.jobs/executions.href
String
Link to the jobs/executions
_links.jobs/thinexecutions.href
String
Link to the jobs/thinexecutions
_links.jobs/executions/name.href
String
Link to the jobs/executions/name
_links.jobs/executions/name.templated
Boolean
Link jobs/executions/name is templated
_links.jobs/executions/status.href
String
Link to the jobs/executions/status
_links.jobs/executions/status.templated
Boolean
Link jobs/executions/status is templated
_links.jobs/thinexecutions/name.href
String
Link to the jobs/thinexecutions/name
_links.jobs/thinexecutions/name.templated
Boolean
Link jobs/executions/name is templated
_links.jobs/thinexecutions/jobInstanceId.href
String
Link to the jobs/thinexecutions/jobInstanceId
_links.jobs/thinexecutions/jobInstanceId.templated
Boolean
Link jobs/executions/jobInstanceId is templated
_links.jobs/thinexecutions/taskExecutionId.href
String
Link to the jobs/thinexecutions/taskExecutionId
_links.jobs/thinexecutions/taskExecutionId.templated
Boolean
Link jobs/executions/taskExecutionId is templated
_links.jobs/executions/execution.href
String
Link to the jobs/executions/execution
_links.jobs/executions/execution.templated
Boolean
Link jobs/executions/execution is templated
_links.jobs/executions/execution/steps.href
String
Link to the jobs/executions/execution/steps
_links.jobs/executions/execution/steps.templated
Boolean
Link jobs/executions/execution/steps is templated
_links.jobs/executions/execution/steps/step.href
String
Link to the jobs/executions/execution/steps/step
_links.jobs/executions/execution/steps/step.templated
Boolean
Link jobs/executions/execution/steps/step is templated
_links.jobs/executions/execution/steps/step/progress.href
String
Link to the jobs/executions/execution/steps/step/progress
_links.jobs/executions/execution/steps/step/progress.templated
Boolean
Link jobs/executions/execution/steps/step/progress is templated
_links.jobs/instances/name.href
String
Link to the jobs/instances/name
_links.jobs/instances/name.templated
Boolean
Link jobs/instances/name is templated
_links.jobs/instances/instance.href
String
Link to the jobs/instances/instance
_links.jobs/instances/instance.templated
Boolean
Link jobs/instances/instance is templated
_links.tools/parseTaskTextToGraph.href
String
Link to the tools/parseTaskTextToGraph
_links.tools/convertTaskGraphToText.href
String
Link to the tools/convertTaskGraphToText
_links.apps.href
String
Link to the apps
_links.about.href
String
Link to the about
_links.completions/stream.href
String
Link to the completions/stream
_links.completions/stream.templated
Boolean
Link completions/stream is templated
_links.completions/task.href
String
Link to the completions/task
_links.completions/task.templated
Boolean
Link completions/task is templated
      "href" : "http://localhost:9393/streams/definitions"
    "streams/definitions/definition" : {
      "href" : "http://localhost:9393/streams/definitions/{name}",
      "templated" : true
    "streams/validation" : {
      "href" : "http://localhost:9393/streams/validation/{name}",
      "templated" : true
    "runtime/streams" : {
      "href" : "http://localhost:9393/runtime/streams{?names}",
      "templated" : true
    "runtime/streams/{streamNames}" : {
      "href" : "http://localhost:9393/runtime/streams/{streamNames}",
      "templated" : true
    "runtime/apps" : {
      "href" : "http://localhost:9393/runtime/apps"
    "runtime/apps/{appId}" : {
      "href" : "http://localhost:9393/runtime/apps/{appId}",
      "templated" : true
    "runtime/apps/{appId}/instances" : {
      "href" : "http://localhost:9393/runtime/apps/{appId}/instances",
      "templated" : true
    "runtime/apps/{appId}/instances/{instanceId}" : {
      "href" : "http://localhost:9393/runtime/apps/{appId}/instances/{instanceId}",
      "templated" : true
    "runtime/apps/{appId}/instances/{instanceId}/actuator" : [ {
      "href" : "http://localhost:9393/runtime/apps/{appId}/instances/{instanceId}/actuator?endpoint={endpoint}",
      "templated" : true
      "href" : "http://localhost:9393/runtime/apps/{appId}/instances/{instanceId}/actuator",
      "templated" : true
    "runtime/apps/{appId}/instances/{instanceId}/post" : {
      "href" : "http://localhost:9393/runtime/apps/{appId}/instances/{instanceId}/post",
      "templated" : true
    "streams/deployments" : {
      "href" : "http://localhost:9393/streams/deployments"
    "streams/deployments/{name}{?reuse-deployment-properties}" : {
      "href" : "http://localhost:9393/streams/deployments/{name}?reuse-deployment-properties=false",
      "templated" : true
    "streams/deployments/{name}" : {
      "href" : "http://localhost:9393/streams/deployments/{name}",
      "templated" : true
    "streams/deployments/history/{name}" : {
      "href" : "http://localhost:9393/streams/deployments/history/{name}",
      "templated" : true
    "streams/deployments/manifest/{name}/{version}" : {
      "href" : "http://localhost:9393/streams/deployments/manifest/{name}/{version}",
      "templated" : true
    "streams/deployments/platform/list" : {
      "href" : "http://localhost:9393/streams/deployments/platform/list"
    "streams/deployments/rollback/{name}/{version}" : {
      "href" : "http://localhost:9393/streams/deployments/rollback/{name}/{version}",
      "templated" : true
    "streams/deployments/update/{name}" : {
      "href" : "http://localhost:9393/streams/deployments/update/{name}",
      "templated" : true
    "streams/deployments/deployment" : {
      "href" : "http://localhost:9393/streams/deployments/{name}",
      "templated" : true
    "streams/deployments/scale/{streamName}/{appName}/instances/{count}" : {
      "href" : "http://localhost:9393/streams/deployments/scale/{streamName}/{appName}/instances/{count}",
      "templated" : true
    "streams/logs" : {
      "href" : "http://localhost:9393/streams/logs"
    "streams/logs/{streamName}" : {
      "href" : "http://localhost:9393/streams/logs/{streamName}",
      "templated" : true
    "streams/logs/{streamName}/{appName}" : {
      "href" : "http://localhost:9393/streams/logs/{streamName}/{appName}",
      "templated" : true
    "tasks/platforms" : {
      "href" : "http://localhost:9393/tasks/platforms"
    "tasks/definitions" : {
      "href" : "http://localhost:9393/tasks/definitions"
    "tasks/definitions/definition" : {
      "href" : "http://localhost:9393/tasks/definitions/{name}",
      "templated" : true
    "tasks/executions" : {
      "href" : "http://localhost:9393/tasks/executions"
    "tasks/executions/name" : {
      "href" : "http://localhost:9393/tasks/executions{?name}",
      "templated" : true
    "tasks/executions/current" : {
      "href" : "http://localhost:9393/tasks/executions/current"
    "tasks/executions/execution" : {
      "href" : "http://localhost:9393/tasks/executions/{id}",
      "templated" : true
    "tasks/validation" : {
      "href" : "http://localhost:9393/tasks/validation/{name}",
      "templated" : true
    "tasks/info/executions" : {
      "href" : "http://localhost:9393/tasks/info/executions{?completed,name}",
      "templated" : true
    "tasks/logs" : {
      "href" : "http://localhost:9393/tasks/logs/{taskExternalExecutionId}{?platformName}",
      "templated" : true
    "tasks/schedules" : {
      "href" : "http://localhost:9393/tasks/schedules"
    "tasks/schedules/instances" : {
      "href" : "http://localhost:9393/tasks/schedules/instances/{taskDefinitionName}",
      "templated" : true
    "jobs/executions" : {
      "href" : "http://localhost:9393/jobs/executions"
    "jobs/executions/name" : {
      "href" : "http://localhost:9393/jobs/executions{?name}",
      "templated" : true
    "jobs/executions/status" : {
      "href" : "http://localhost:9393/jobs/executions{?status}",
      "templated" : true
    "jobs/executions/execution" : {
      "href" : "http://localhost:9393/jobs/executions/{id}",
      "templated" : true
    "jobs/executions/execution/steps" : {
      "href" : "http://localhost:9393/jobs/executions/{jobExecutionId}/steps",
      "templated" : true
    "jobs/executions/execution/steps/step" : {
      "href" : "http://localhost:9393/jobs/executions/{jobExecutionId}/steps/{stepId}",
      "templated" : true
    "jobs/executions/execution/steps/step/progress" : {
      "href" : "http://localhost:9393/jobs/executions/{jobExecutionId}/steps/{stepId}/progress",
      "templated" : true
    "jobs/instances/name" : {
      "href" : "http://localhost:9393/jobs/instances{?name}",
      "templated" : true
    "jobs/instances/instance" : {
      "href" : "http://localhost:9393/jobs/instances/{id}",
      "templated" : true
    "tools/parseTaskTextToGraph" : {
      "href" : "http://localhost:9393/tools"
    "tools/convertTaskGraphToText" : {
      "href" : "http://localhost:9393/tools"
    "jobs/thinexecutions" : {
      "href" : "http://localhost:9393/jobs/thinexecutions"
    "jobs/thinexecutions/name" : {
      "href" : "http://localhost:9393/jobs/thinexecutions{?name}",
      "templated" : true
    "jobs/thinexecutions/jobInstanceId" : {
      "href" : "http://localhost:9393/jobs/thinexecutions{?jobInstanceId}",
      "templated" : true
    "jobs/thinexecutions/taskExecutionId" : {
      "href" : "http://localhost:9393/jobs/thinexecutions{?taskExecutionId}",
      "templated" : true
    "apps" : {
      "href" : "http://localhost:9393/apps"
    "about" : {
      "href" : "http://localhost:9393/about"
    "completions/stream" : {
      "href" : "http://localhost:9393/completions/stream{?start,detailLevel}",
      "templated" : true
    "completions/task" : {
      "href" : "http://localhost:9393/completions/task{?start,detailLevel}",
      "templated" : true
  "api.revision" : 14
about
Access meta information, including enabled features, security info, version information
dashboard
Access the dashboard UI
audit-records
Provides audit trail information
Handle registered applications
completions/stream
Exposes the DSL completion features for Stream
completions/task
Exposes the DSL completion features for Task
jobs/executions
Provides the JobExecution resource
jobs/thinexecutions
Provides the JobExecution thin resource with no step executions included
jobs/executions/execution
Provides details for a specific JobExecution
jobs/executions/execution/steps
Provides the steps for a JobExecution
jobs/executions/execution/steps/step
Returns the details for a specific step
jobs/executions/execution/steps/step/progress
Provides progress information for a specific step
jobs/executions/name
Retrieve Job Executions by Job name
jobs/executions/status
Retrieve Job Executions by Job status
jobs/thinexecutions/name




    

Retrieve Job Executions by Job name with no step executions included
jobs/thinexecutions/jobInstanceId
Retrieve Job Executions by Job Instance Id with no step executions included
jobs/thinexecutions/taskExecutionId
Retrieve Job Executions by Task Execution Id with no step executions included
jobs/instances/instance
Provides the job instance resource for a specific job instance
jobs/instances/name
Provides the Job instance resource for a specific job name
runtime/streams
Exposes stream runtime status
runtime/streams/{streamNames}
Exposes streams runtime status for a given stream names
runtime/apps
Provides the runtime application resource
runtime/apps/{appId}
Exposes the runtime status for a specific app
runtime/apps/{appId}/instances
Provides the status for app instances
runtime/apps/{appId}/instances/{instanceId}
Provides the status for specific app instance
runtime/apps/{appId}/instances/{instanceId}/actuator
EXPERIMENTAL: Allows invoking Actuator endpoint on specific app instance
runtime/apps/{appId}/instances/{instanceId}/post
EXPERIMENTAL: Allows POST on http sink
tasks/definitions
Provides the task definition resource
tasks/definitions/definition
Provides details for a specific task definition
tasks/validation
Provides the validation for a task definition
tasks/executions
Returns Task executions and allows launching of tasks
tasks/executions/current
Provides the current count of running tasks
tasks/info/executions
Provides the task executions info
tasks/schedules
Provides schedule information of tasks
tasks/schedules/instances
Provides schedule information of a specific task
tasks/executions/name
Returns all task executions for a given Task name
tasks/executions/execution
Provides details for a specific task execution
tasks/platforms
Provides platform accounts for launching tasks.  The results can be filtered to show the platforms that support scheduling by adding a request parameter of 'schedulesEnabled=true
tasks/logs
Retrieve the task application log
schema/versions
List of Spring Boot related schemas
schema/targets
List of schema targets
streams/definitions
Exposes the Streams resource
streams/definitions/definition
Handle a specific Stream definition
streams/validation
Provides the validation for a stream definition
streams/deployments
Provides Stream deployment operations
streams/deployments/{name}
Request deployment info for a stream definition
streams/deployments/{name}{?reuse-deployment-properties}
Request deployment info for a stream definition
streams/deployments/deployment
Request (un-)deployment of an existing stream definition
streams/deployments/manifest/{name}/{version}
Return a manifest info of a release version
streams/deployments/history/{name}
Get stream’s deployment history as list or Releases for this release
streams/deployments/rollback/{name}/{version}
Rollback the stream to the previous or a specific version of the stream
streams/deployments/update/{name}
Update the stream.
streams/deployments/platform/list
List of supported deployment platforms
streams/deployments/scale/{streamName}/{appName}/instances/{count}
Scale up or down number of application instances for a selected stream
streams/logs
Retrieve application logs of the stream
streams/logs/{streamName}
Retrieve application logs of the stream
streams/logs/{streamName}/{appName}
Retrieve a specific application log of the stream
tools/parseTaskTextToGraph
Parse a task definition into a graph structure
tools/convertTaskGraphToText
Convert a graph format into DSL text format
42.2. Server Meta Information
The server meta information endpoint provides more information about the server itself.
The following topics provide more details:
      "name" : "Spring Cloud Data Flow Shell",
      "version" : "2.11.0",
      "url" : "https://repo.maven.apache.org/maven2/org/springframework/cloud/spring-cloud-dataflow-shell/2.11.0/spring-cloud-dataflow-shell-2.11.0.jar"
  "securityInfo" : {
    "authenticationEnabled" : false,
    "authenticated" : false,
    "username" : null,
    "roles" : [ ]
  "runtimeEnvironment" : {
    "appDeployer" : {
      "deployerImplementationVersion" : "Test Version",
      "deployerName" : "Test Server",
      "deployerSpiVersion" : "2.11.0",
      "javaVersion" : "1.8.0_372",
      "platformApiVersion" : "",
      "platformClientVersion" : "",
      "platformHostVersion" : "",
      "platformSpecificInfo" : {
        "default" : "local"
      "platformType" : "Skipper Managed",
      "springBootVersion" : "2.7.10",
      "springVersion" : "5.3.26"
    "taskLaunchers" : [ {
      "deployerImplementationVersion" : "unknown",
      "deployerName" : "LocalTaskLauncher",
      "deployerSpiVersion" : "unknown",
      "javaVersion" : "1.8.0_372",
      "platformApiVersion" : "Linux 5.15.0-1036-azure",
      "platformClientVersion" : "5.15.0-1036-azure",
      "platformHostVersion" : "5.15.0-1036-azure",
      "platformSpecificInfo" : { },
      "platformType" : "Local",
      "springBootVersion" : "2.7.10",
      "springVersion" : "5.3.26"
      "deployerImplementationVersion" : "unknown",
      "deployerName" : "LocalTaskLauncher",
      "deployerSpiVersion" : "unknown",
      "javaVersion" : "1.8.0_372",
      "platformApiVersion" : "Linux 5.15.0-1036-azure",
      "platformClientVersion" : "5.15.0-1036-azure",
      "platformHostVersion" : "5.15.0-1036-azure",
      "platformSpecificInfo" : { },
      "platformType" : "Local",
      "springBootVersion" : "2.7.10",
      "springVersion" : "5.3.26"
  "monitoringDashboardInfo" : {
    "url" : "",
    "refreshInterval" : 15,
    "dashboardType" : "NONE",
    "source" : "default-scdf-source"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/about"
42.3. Registered Applications
The registered applications endpoint provides information about the applications that are registered with the Spring Cloud Data Flow server.
The following topics provide more details:
42.3.1. Listing Applications
A GET request lists all of the applications known to Spring Cloud Data Flow.
The following topics provide more details:
GET /apps?search=&type=source&defaultVersion=true&page=0&size=10&sort=name%2CASC HTTP/1.1
Accept: application/json
Host: localhost:9393
Restrict the returned apps to the type of the app. One of [app, source, processor, sink, task]
defaultVersion
The boolean flag to set to retrieve only the apps of the default versions (optional)
The zero-based page number (optional)
The sort on the list (optional)
The requested page size (optional)
$ curl 'http://localhost:9393/apps?search=&type=source&defaultVersion=true&page=0&size=10&sort=name%2CASC' -i -X GET \
    -H 'Accept: application/json'
      "name" : "http",
      "type" : "source",
      "uri" : "maven://org.springframework.cloud.stream.app:http-source-rabbit:1.2.0.RELEASE",
      "version" : "1.2.0.RELEASE",
      "defaultVersion" : true,
      "bootVersion" : "2",
      "versions" : [ "1.2.0.RELEASE" ],
      "label" : null,
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/apps/source/http/1.2.0.RELEASE"
      "name" : "time",
      "type" : "source",
      "uri" : "maven://org.springframework.cloud.stream.app:time-source-rabbit:1.2.0.RELEASE",
      "version" : "1.2.0.RELEASE",
      "defaultVersion" : true,
      "bootVersion" : "2",
      "versions" : [ "1.2.0.RELEASE" ],
      "label" : null,
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/apps/source/time/1.2.0.RELEASE"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/apps?page=0&size=10&sort=name,asc"
  "page" : {
    "size" : 10,
    "totalElements" : 2,
    "totalPages" : 1,
    "number" : 0
42.3.2. Getting Information on a Particular Application
A GET request on /apps/<type>/<name> gets info on a particular application.
The following topics provide more details:
  "name" : "http",
  "type" : "source",
  "uri" : "maven://org.springframework.cloud.stream.app:http-source-rabbit:1.2.0.RELEASE",
  "version" : "1.2.0.RELEASE",
  "defaultVersion" : true,
  "bootVersion" : "2",
  "versions" : null,
  "label" : null,
  "options" : [ {
    "id" : "http.path-pattern",
    "name" : "path-pattern",
    "type" : "java.lang.String",
    "description" : "An Ant-Style pattern to determine which http requests will be captured.",
    "shortDescription" : "An Ant-Style pattern to determine which http requests will be captured.",
    "defaultValue" : "/",
    "hints" : {
      "keyHints" : [ ],
      "keyProviders" : [ ],
      "valueHints" : [ ],
      "valueProviders" : [ ]
    "deprecation" : null,
    "deprecated" : false
    "id" : "http.mapped-request-headers",
    "name" : "mapped-request-headers",
    "type" : "java.lang.String[]",
    "description" : "Headers that will be mapped.",
    "shortDescription" : "Headers that will be mapped.",
    "defaultValue" : null,
    "hints" : {
      "keyHints" : [ ],
      "keyProviders" : [ ],
      "valueHints" : [ ],
      "valueProviders" : [ ]
    "deprecation" : null,
    "deprecated" : false
    "id" : "http.secured",
    "name" : "secured",
    "type" : "java.lang.Boolean",
    "description" : "Secure or not HTTP source path.",
    "shortDescription" : "Secure or not HTTP source path.",
    "defaultValue" : false,
    "hints" : {
      "keyHints" : [ ],
      "keyProviders" : [ ],
      "valueHints" : [ ],
      "valueProviders" : [ ]
    "deprecation" : null,
    "deprecated" : false
    "id" : "server.port",
    "name" : "port",
    "type" : "java.lang.Integer",
    "description" : "Server HTTP port.",
    "shortDescription" : "Server HTTP port.",
    "defaultValue" : null,
    "hints" : {
      "keyHints" : [ ],
      "keyProviders" : [ ],
      "valueHints" : [ ],
      "valueProviders" : [ ]
    "deprecation" : null,
    "deprecated" : false
  "shortDescription" : null,
  "inboundPortNames" : [ ],
  "outboundPortNames" : [ ],
  "optionGroups" : { }
42.3.3. Registering a New Application
A POST request on /apps/<type>/<name> allows registration of a new application.
The following topics provide more details:
POST /apps/source/http HTTP/1.1
Host: localhost:9393
Content-Type: application/x-www-form-urlencoded
uri=maven%3A%2F%2Forg.springframework.cloud.stream.app%3Ahttp-source-rabbit%3A1.1.0.RELEASE
42.3.4. Registering a New Application with version
A POST request on /apps/<type>/<name>/<version> allows registration of a new application.
The following topics provide more details:
POST /apps/source/http/1.1.0.RELEASE HTTP/1.1
Host: localhost:9393
Content-Type: application/x-www-form-urlencoded
uri=maven%3A%2F%2Forg.springframework.cloud.stream.app%3Ahttp-source-rabbit%3A1.1.0.RELEASE
The type of application to register. One of [app, source, processor, sink, task] (optional)
The name of the application to register
version
The version of the application to register
42.3.5. Registering Applications in Bulk
A POST request on /apps allows registering multiple applications at once.
The following topics provide more details:
URI where a properties file containing registrations can be fetched. Exclusive with apps.
Inline set of registrations. Exclusive with uri.
force
Must be true if a registration with the same name and type already exists, otherwise an error will occur
      "name" : "http",
      "type" : "source",
      "uri" : "maven://org.springframework.cloud.stream.app:http-source-rabbit:1.1.0.RELEASE",
      "version" : "1.1.0.RELEASE",
      "defaultVersion" : true,
      "bootVersion" : "2",
      "versions" : null,
      "label" : null,
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/apps/source/http/1.1.0.RELEASE"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/apps?page=0&size=20"
  "page" : {
    "size" : 20,
    "totalElements" : 1,
    "totalPages" : 1,
    "number" : 0
42.3.6. Set the Default Application Version
For an application with the same name and type, you can register multiple versions.
In this case, you can choose one of the versions as the default application.
The following topics provide more details:
42.3.7. Unregistering an Application
A DELETE request on /apps/<type>/<name> unregisters a previously registered application.
The following topics provide more details:
42.3.8. Unregistering all Applications
A DELETE request on /apps unregisters all the applications.
The following topics provide more details:
42.4.1. List All Audit Records
The audit records endpoint lets you retrieve audit trail information.
The following topics provide more details:
GET /audit-records?page=0&size=10&operations=STREAM&actions=CREATE&fromDate=2000-01-01T00%3A00%3A00&toDate=2099-01-01T00%3A00%3A00 HTTP/1.1
Host: localhost:9393
      "createdBy" : null,
      "correlationId" : "timelog",
      "auditData" : "time --format='YYYY MM DD' | log",
      "createdOn" : "2023-05-04T00:03:10.201Z",
      "auditAction" : "CREATE",
      "auditOperation" : "STREAM",
      "platformName" : null,
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/audit-records/5"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/audit-records?page=0&size=10"
  "page" : {
    "size" : 10,
    "totalElements" : 1,
    "totalPages" : 1,
    "number" : 0
  "createdBy" : null,
  "correlationId" : "timelog",
  "auditData" : "time --format='YYYY MM DD' | log",
  "createdOn" : "2023-05-04T00:03:10.201Z",
  "auditAction" : "CREATE",
  "auditOperation" : "STREAM",
  "platformName" : null,
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/audit-records/5"
  "description" : "Create an Entity",
  "key" : "CREATE",
  "nameWithDescription" : "Create (Create an Entity)"
  "id" : 200,
  "name" : "Delete",
  "description" : "Delete an Entity",
  "key" : "DELETE",
  "nameWithDescription" : "Delete (Delete an Entity)"
  "id" : 300,
  "name" : "Deploy",
  "description" : "Deploy an Entity",
  "key" : "DEPLOY",
  "nameWithDescription" : "Deploy (Deploy an Entity)"
  "id" : 400,
  "name" : "Rollback",
  "description" : "Rollback an Entity",
  "key" : "ROLLBACK",
  "nameWithDescription" : "Rollback (Rollback an Entity)"
  "id" : 500,
  "name" : "Undeploy",
  "description" : "Undeploy an Entity",
  "key" : "UNDEPLOY",
  "nameWithDescription" : "Undeploy (Undeploy an Entity)"
  "id" : 600,
  "name" : "Update",
  "description" : "Update an Entity",
  "key" : "UPDATE",
  "nameWithDescription" : "Update (Update an Entity)"
  "id" : 700,
  "name" : "SuccessfulLogin",
  "description" : "Successful login",
  "key" : "LOGIN_SUCCESS",
  "nameWithDescription" : "SuccessfulLogin (Successful login)"
42.5. Stream Definitions
The registered applications endpoint provides information about the stream definitions that are registered with the Spring Cloud Data Flow server.
The following topics provide more details:




    

42.5.1. Creating a New Stream Definition
Creating a stream definition is achieved by creating a POST request to the stream definitions endpoint.
A curl request for a ticktock stream might resemble the following:
A stream definition can also contain additional parameters.
For instance, in the example shown under “Request Structure”, we also provide the date-time format.
The following topics provide more details:
POST /streams/definitions HTTP/1.1
Host: localhost:9393
Content-Type: application/x-www-form-urlencoded
name=timelog&definition=time+--format%3D%27YYYY+MM+DD%27+%7C+log&description=Demo+stream+for+testing&deploy=false
  "name" : "timelog",
  "dslText" : "time --format='YYYY MM DD' | log",
  "originalDslText" : "time --format='YYYY MM DD' | log",
  "status" : "undeployed",
  "description" : "Demo stream for testing",
  "statusDescription" : "The app or group is known to the system, but is not currently deployed",
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/streams/definitions/timelog"
42.5.2. List All Stream Definitions
The streams endpoint lets you list all the stream definitions.
The following topics provide more details:
      "status" : "undeployed",
      "description" : "",
      "statusDescription" : "The app or group is known to the system, but is not currently deployed",
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/streams/definitions/mysamplestream"
      "name" : "timelog",
      "dslText" : "time --format='YYYY MM DD' | log",
      "originalDslText" : "time --format='YYYY MM DD' | log",
      "status" : "undeployed",
      "description" : "Demo stream for testing",
      "statusDescription" : "The app or group is known to the system, but is not currently deployed",
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/streams/definitions/timelog"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/streams/definitions?page=0&size=10&sort=name,asc"
  "page" : {
    "size" : 10,
    "totalElements" : 2,
    "totalPages" : 1,
    "number" : 0
42.5.3. List Related Stream Definitions
The streams endpoint lets you list related stream definitions.
The following topics provide more details:
GET /streams/definitions/timelog/related?page=0&sort=name%2CASC&search=&size=10&nested=true HTTP/1.1
Host: localhost:9393
nested
Should we recursively findByTaskNameContains for related stream definitions (optional)
The zero-based page number (optional)
search
The search string performed on the name (optional)
The sort on the list (optional)
The requested page size (optional)
    "streamDefinitionResourceList" : [ {
      "name" : "timelog",
      "dslText" : "time --format='YYYY MM DD' | log",
      "originalDslText" : "time --format='YYYY MM DD' | log",
      "status" : "undeployed",
      "description" : "Demo stream for testing",
      "statusDescription" : "The app or group is known to the system, but is not currently deployed",
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/streams/definitions/timelog"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/streams/definitions/timelog/related?page=0&size=10&sort=name,asc"
  "page" : {
    "size" : 10,
    "totalElements" : 1,
    "totalPages" : 1,
    "number" : 0
42.5.4. Retrieve Stream Definition Detail
The stream definition endpoint lets you get a single stream definition.
The following topics provide more details:
  "name" : "timelog",
  "dslText" : "time --format='YYYY MM DD' | log",
  "originalDslText" : "time --format='YYYY MM DD' | log",
  "status" : "undeployed",
  "description" : "Demo stream for testing",
  "statusDescription" : "The app or group is known to the system, but is not currently deployed",
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/streams/definitions/timelog"
42.5.5. Delete a Single Stream Definition
The streams endpoint lets you delete a single stream definition.
(See also: Delete All Stream Definitions.)
The following topics provide more details:
42.5.6. Delete All Stream Definitions
The streams endpoint lets you delete all single stream definitions.
(See also: Delete a Single Stream Definition.)
The following topics provide more details:
42.6. Stream Validation
The stream validation endpoint lets you validate the apps in a stream definition.
The following topics provide more details:
  "appName" : "timelog",
  "dsl" : "time --format='YYYY MM DD' | log",
  "description" : "Demo stream for testing",
  "appStatuses" : {
    "source:time" : "valid",
    "sink:log" : "valid"
42.7. Stream Deployments
The deployment definitions endpoint provides information about the deployments that are registered with the Spring Cloud Data Flow server.
The following topics provide more details:
42.7.1. Deploying Stream Definition
The stream definition endpoint lets you deploy a single stream definition.
Optionally, you can pass application parameters as properties in the request body.
The following topics provide more details:
POST /streams/deployments/timelog HTTP/1.1
Content-Type: application/json
Content-Length: 36
Host: localhost:9393
{"app.time.timestamp.format":"YYYY"}
/streams/deployments/{timelog}
$ curl 'http://localhost:9393/streams/deployments/timelog' -i -X POST \
    -H 'Content-Type: application/json' \
    -d '{"app.time.timestamp.format":"YYYY"}'
42.7.2. Undeploy Stream Definition
The stream definition endpoint lets you undeploy a single stream definition.
The following topics provide more details:
42.7.3. Undeploy All Stream Definitions
The stream definition endpoint lets you undeploy all single stream definitions.
The following topics provide more details:
POST /streams/deployments/update/timelog1 HTTP/1.1
Content-Type: application/json
Content-Length: 196
Host: localhost:9393
{"releaseName":"timelog1","packageIdentifier":{"repositoryName":"test","packageName":"timelog1","packageVersion":"1.0.0"},"updateProperties":{"app.time.timestamp.format":"YYYYMMDD"},"force":false}
/streams/deployments/update/{timelog1}
$ curl 'http://localhost:9393/streams/deployments/update/timelog1' -i -X POST \
    -H 'Content-Type: application/json' \
    -d '{"releaseName":"timelog1","packageIdentifier":{"repositoryName":"test","packageName":"timelog1","packageVersion":"1.0.0"},"updateProperties":{"app.time.timestamp.format":"YYYYMMDD"},"force":false}'
42.7.9. Scale Stream Definition
The stream definition endpoint lets you scale a single app in a stream definition.
Optionally, you can pass application parameters as properties in the request body.
The following topics provide more details:
POST /streams/deployments/scale/timelog/log/instances/1 HTTP/1.1
Content-Type: application/json
Content-Length: 36
Host: localhost:9393
{"app.time.timestamp.format":"YYYY"}
/streams/deployments/scale/{streamName}/{appName}/instances/{count}
$ curl 'http://localhost:9393/streams/deployments/scale/timelog/log/instances/1' -i -X POST \
    -H 'Content-Type: application/json' \
    -d '{"app.time.timestamp.format":"YYYY"}'
42.8. Task Definitions
The task definitions endpoint provides information about the task definitions that are registered with the Spring Cloud Data Flow server.
The following topics provide more details:
42.8.1. Creating a New Task Definition
The task definition endpoint lets you create a new task definition.
The following topics provide more details:
POST /tasks/definitions HTTP/1.1
Host: localhost:9393
Content-Type: application/x-www-form-urlencoded
name=my-task&definition=timestamp+--format%3D%27YYYY+MM+DD%27&description=Demo+task+definition+for+testing
  "name" : "my-task",
  "dslText" : "timestamp --format='YYYY MM DD'",
  "description" : "Demo task definition for testing",
  "composed" : false,
  "composedTaskElement" : false,
  "lastTaskExecution" : null,
  "status" : "UNKNOWN",
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/tasks/definitions/my-task"
42.8.2. List All Task Definitions
The task definition endpoint lets you get all task definitions.
The following topics provide more details:
    "taskDefinitionResourceList" : [ {
      "name" : "my-task",
      "dslText" : "timestamp --format='YYYY MM DD'",
      "description" : "Demo task definition for testing",
      "composed" : false,
      "composedTaskElement" : false,
      "lastTaskExecution" : null,
      "status" : "UNKNOWN",
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/tasks/definitions/my-task"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/tasks/definitions?page=0&size=10&sort=taskName,asc"
  "page" : {
    "size" : 10,
    "totalElements" : 1,
    "totalPages" : 1,
    "number" : 0
42.8.3. Retrieve Task Definition Detail
The task definition endpoint lets you get a single task definition.
The following topics provide more details:
  "name" : "my-task",
  "dslText" : "timestamp --format='YYYY MM DD'",
  "description" : "Demo task definition for testing",
  "composed" : false,
  "composedTaskElement" : false,
  "lastTaskExecution" : null,
  "status" : "UNKNOWN",
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/tasks/definitions/my-task"
42.8.4. Delete Task Definition
The task definition endpoint lets you delete a single task definition.
The following topics provide more details:
42.9. Task Scheduler
The task scheduler endpoint provides information about the task schedules that are registered with the Scheduler Implementation.
The following topics provide more details:
42.9.1. Creating a New Task Schedule
The task schedule endpoint lets you create a new task schedule.
The following topics provide more details:
POST /tasks/schedules HTTP/1.1
Host: localhost:9393
Content-Type: application/x-www-form-urlencoded
scheduleName=myschedule&taskDefinitionName=mytaskname&platform=default&properties=scheduler.cron.expression%3D00+22+17+%3F+*&arguments=--foo%3Dbar
      "taskDefinitionName" : "BAR",
      "scheduleProperties" : {
        "scheduler.AAA.spring.cloud.scheduler.cron.expression" : "00 41 17 ? * *"
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/tasks/schedules/FOO"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/tasks/schedules?page=0&size=10"
  "page" : {
    "size" : 10,
    "totalElements" : 1,
    "totalPages" : 1,
    "number" : 0
42.9.3. List Filtered Schedules
The task schedules endpoint lets you get all task schedules that have the specified task definition name.
The following topics provide more details:
      "taskDefinitionName" : "BAR",
      "scheduleProperties" : {
        "scheduler.AAA.spring.cloud.scheduler.cron.expression" : "00 41 17 ? * *"
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/tasks/schedules/FOO"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/tasks/schedules/instances/FOO?page=0&size=1"
  "page" : {
    "size" : 1,
    "totalElements" : 1,
    "totalPages" : 1,
    "number" : 0
42.9.4. Delete Task Schedule
The task schedule endpoint lets you delete a single task schedule.
The following topics provide more details:
42.10. Task Validation
The task validation endpoint lets you validate the apps in a task definition.
The following topics provide more details:
42.11. Task Executions
The task executions endpoint provides information about the task executions that are registered with the Spring Cloud Data Flow server.
The following topics provide more details:
42.11.1. Launching a Task
Launching a task is done by requesting the creation of a new task execution.
The following topics provide more details:
POST /tasks/executions HTTP/1.1
Host: localhost:9393
Content-Type: application/x-www-form-urlencoded
name=taskA&properties=app.my-task.foo%3Dbar%2Cdeployer.my-task.something-else%3D3&arguments=--server.port%3D8080+--foo%3Dbar
42.11.2. Stopping a Task
Stopping a task is done by posting the id of an existing task execution.
The following topics provide more details:
POST /tasks/executions/1 HTTP/1.1
Host: localhost:9393
Content-Type: application/x-www-form-urlencoded
platform=default
42.11.3. List All Task Executions
The task executions endpoint lets you list all task executions.
The following topics provide more details:
      "jobExecutionIds" : [ ],
      "errorMessage" : null,
      "externalExecutionId" : "taskB-751ce929-7aed-49e3-977d-e08ced094236",
      "parentExecutionId" : null,
      "resourceUrl" : "org.springframework.cloud.task.app:timestamp-task:jar:1.2.0.RELEASE",
      "appProperties" : {
        "management.metrics.tags.service" : "task-application",
        "timestamp.format" : "yyyy MM dd",
        "spring.datasource.username" : null,
        "spring.datasource.url" : null,
        "spring.datasource.driverClassName" : null,
        "management.metrics.tags.application" : "${spring.cloud.task.name:unknown}-${spring.cloud.task.executionid:unknown}",
        "spring.cloud.task.name" : "taskB"
      "deploymentProperties" : {
        "app.my-task.foo" : "bar",
        "deployer.my-task.something-else" : "3"
      "platformName" : "default",
      "taskExecutionStatus" : "UNKNOWN",
      "_links" : {
        "tasks/logs" : {
          "href" : "http://localhost:9393/tasks/logs/taskB-751ce929-7aed-49e3-977d-e08ced094236?platformName=default"
        "self" : {
          "href" : "http://localhost:9393/tasks/executions/2"
      "executionId" : 1,
      "exitCode" : null,
      "taskName" : "taskA",
      "startTime" : null,
      "endTime" : null,
      "exitMessage" : null,
      "arguments" : [ ],
      "jobExecutionIds" : [ ],
      "errorMessage" : null,
      "externalExecutionId" : "taskA-7d9b84f8-1192-4910-92e5-bb8e3ea6474d",
      "parentExecutionId" : null,
      "resourceUrl" : "org.springframework.cloud.task.app:timestamp-task:jar:1.2.0.RELEASE",
      "appProperties" : {
        "management.metrics.tags.service" : "task-application",
        "timestamp.format" : "yyyy MM dd",
        "spring.datasource.username" : null,
        "spring.datasource.url" : null,
        "spring.datasource.driverClassName" : null,
        "management.metrics.tags.application" : "${spring.cloud.task.name:unknown}-${spring.cloud.task.executionid:unknown}",
        "spring.cloud.task.name" : "taskA"
      "deploymentProperties" : {
        "app.my-task.foo" : "bar",
        "deployer.my-task.something-else" : "3"
      "platformName" : "default",
      "taskExecutionStatus" : "UNKNOWN",
      "_links" : {
        "tasks/logs" : {
          "href" : "http://localhost:9393/tasks/logs/taskA-7d9b84f8-1192-4910-92e5-bb8e3ea6474d?platformName=default"
        "self" : {
          "href" : "http://localhost:9393/tasks/executions/1"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/tasks/executions?page=0&size=10"
  "page" : {
    "size" : 10,
    "totalElements" : 2,
    "totalPages" : 1,
    "number" : 0
42.11.4. List All Task Executions With a Specified Task Name
The task executions endpoint lets you list task executions with a specified task name.
The following topics provide more details:




    

      "jobExecutionIds" : [ ],
      "errorMessage" : null,
      "externalExecutionId" : "taskB-751ce929-7aed-49e3-977d-e08ced094236",
      "parentExecutionId" : null,
      "resourceUrl" : "org.springframework.cloud.task.app:timestamp-task:jar:1.2.0.RELEASE",
      "appProperties" : {
        "management.metrics.tags.service" : "task-application",
        "timestamp.format" : "yyyy MM dd",
        "spring.datasource.username" : null,
        "spring.datasource.url" : null,
        "spring.datasource.driverClassName" : null,
        "management.metrics.tags.application" : "${spring.cloud.task.name:unknown}-${spring.cloud.task.executionid:unknown}",
        "spring.cloud.task.name" : "taskB"
      "deploymentProperties" : {
        "app.my-task.foo" : "bar",
        "deployer.my-task.something-else" : "3"
      "platformName" : "default",
      "taskExecutionStatus" : "UNKNOWN",
      "_links" : {
        "tasks/logs" : {
          "href" : "http://localhost:9393/tasks/logs/taskB-751ce929-7aed-49e3-977d-e08ced094236?platformName=default"
        "self" : {
          "href" : "http://localhost:9393/tasks/executions/2"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/tasks/executions?page=0&size=10"
  "page" : {
    "size" : 10,
    "totalElements" : 1,
    "totalPages" : 1,
    "number" : 0
42.11.5. Task Execution Detail
The task executions endpoint lets you get the details about a task execution.
The following topics provide more details:
  "jobExecutionIds" : [ ],
  "errorMessage" : null,
  "externalExecutionId" : "taskA-7d9b84f8-1192-4910-92e5-bb8e3ea6474d",
  "parentExecutionId" : null,
  "resourceUrl" : "org.springframework.cloud.task.app:timestamp-task:jar:1.2.0.RELEASE",
  "appProperties" : {
    "management.metrics.tags.service" : "task-application",
    "timestamp.format" : "yyyy MM dd",
    "spring.datasource.username" : null,
    "spring.datasource.url" : null,
    "spring.datasource.driverClassName" : null,
    "management.metrics.tags.application" : "${spring.cloud.task.name:unknown}-${spring.cloud.task.executionid:unknown}",
    "spring.cloud.task.name" : "taskA"
  "deploymentProperties" : {
    "app.my-task.foo" : "bar",
    "deployer.my-task.something-else" : "3"
  "platformName" : "default",
  "taskExecutionStatus" : "UNKNOWN",
  "_links" : {
    "tasks/logs" : {
      "href" : "http://localhost:9393/tasks/logs/taskA-7d9b84f8-1192-4910-92e5-bb8e3ea6474d?platformName=default"
    "self" : {
      "href" : "http://localhost:9393/tasks/executions/1"
The cleanup implementation (first option) is platform specific. Both operations can be triggered
at once or separately.
You must provide task execution IDs that actually exist. Otherwise, a 404 (Not Found) HTTP status is returned.
In the case of submitting multiple task execution IDs, the invalidity of a single task execution ID causes the entire request to fail,
without performing any operation.
Request Parameters
This endpoint supports one optional request parameter named action. It is an enumeration and supports the following
values:
42.11.7. Deleting Task Execution Data
Not only can you clean up resources that were used to deploy tasks but you can also delete the data associated with
task executions from the underlying persistence store. Also, if a task execution is associated with one or
more batch job executions, these are removed as well.
The following example illustrates how a request can be made using multiple task execution IDs and multiple actions:
$ curl 'http://localhost:9393/tasks/executions/1,2?action=CLEANUP,REMOVE_DATA' -i -X DELETE
/tasks/executions/{ids}
When deleting data from the persistence store by using the REMOVE_DATA action parameter, you must provide
task execution IDs that represent parent task executions. When you provide child task executions (executed as part of a composed task),
a 400 (Bad Request) HTTP status is returned.
When deleting large number of task executions some database types limit the number of entries in the IN clause (the method Spring Cloud Data Flow uses to delete relationships for task executions).
Spring Cloud Data Flow supports the chunking of deletes for Sql Server (Maximum 2100 entries) and Oracle DBs (Maximum 1000 entries).
However, Spring Cloud Data Flow allows users to set their own chunking factor.  To do this set the spring.cloud.dataflow.task.executionDeleteChunkSize property to the appropriate chunk size.
Default is 0 which means Spring Cloud Data Flow will not chunk the task execution deletes (except for Oracle and Sql Server databases).
42.11.8. Task Execution Current Count
The task executions current endpoint lets you retrieve the current number of running executions.
The following topics provide more details:
42.12. Job Executions
The job executions endpoint provides information about the job executions that are registered with the Spring Cloud Data Flow server.
The following topics provide more details:
List All Job Executions For A Specified Task Execution Id Without Step Executions Included
Job Execution Detail
Stop Job Execution
Restart Job Execution
42.12.1. List All Job Executions
The job executions endpoint lets you list all job executions.
The following topics provide more details:
        "stepExecutions" : [ ],
        "status" : "STOPPED",
        "startTime" : "2023-05-04T00:02:53.589+0000",
        "createTime" : "2023-05-04T00:02:53.588+0000",
        "endTime" : null,
        "lastUpdated" : "2023-05-04T00:02:53.589+0000",
        "exitStatus" : {
          "exitCode" : "UNKNOWN",
          "exitDescription" : ""
        "executionContext" : {
          "dirty" : false,
          "empty" : true,
          "values" : [ ]
        "failureExceptions" : [ ],
        "jobConfigurationName" : null,
        "allFailureExceptions" : [ ]
      "jobParameters" : { },
      "jobParametersString" : "",
      "restartable" : true,
      "abandonable" : true,
      "stoppable" : false,
      "defined" : true,
      "timeZone" : "UTC",
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/jobs/executions/2"
      "executionId" : 1,
      "stepExecutionCount" : 0,
      "jobId" : 1,
      "taskExecutionId" : 1,
      "name" : "DOCJOB",
      "startDate" : "2023-05-04",
      "startTime" : "00:02:53",
      "duration" : "00:00:00",
      "jobExecution" : {
        "id" : 1,
        "version" : 2,
        "jobParameters" : {
          "parameters" : { }
        "jobInstance" : {
          "id" : 1,
          "jobName" : "DOCJOB",
          "version" : null
        "stepExecutions" : [ ],
        "status" : "STOPPING",
        "startTime" : "2023-05-04T00:02:53.583+0000",
        "createTime" : "2023-05-04T00:02:53.569+0000",
        "endTime" : null,
        "lastUpdated" : "2023-05-04T00:02:53.775+0000",
        "exitStatus" : {
          "exitCode" : "UNKNOWN",
          "exitDescription" : ""
        "executionContext" : {
          "dirty" : false,
          "empty" : true,
          "values" : [ ]
        "failureExceptions" : [ ],
        "jobConfigurationName" : null,
        "allFailureExceptions" : [ ]
      "jobParameters" : { },
      "jobParametersString" : "",
      "restartable" : false,
      "abandonable" : true,
      "stoppable" : false,
      "defined" : false,
      "timeZone" : "UTC",
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/jobs/executions/1"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/jobs/executions?page=0&size=10"
  "page" : {
    "size" : 10,
    "totalElements" : 2,
    "totalPages" : 1,
    "number" : 0
42.12.2. List All Job Executions Without Step Executions Included
The job executions endpoint lets you list all job executions without step executions included.
The following topics provide more details:
      "startDate" : "2023-05-04",
      "startTime" : "00:02:53",
      "startDateTime" : "2023-05-04T00:02:53.589+0000",
      "duration" : "00:00:00",
      "jobParameters" : { },
      "jobParametersString" : "",
      "restartable" : true,
      "abandonable" : true,
      "stoppable" : false,
      "defined" : true,
      "timeZone" : "UTC",
      "status" : "STOPPED",
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/jobs/thinexecutions/2"
      "executionId" : 1,
      "stepExecutionCount" : 0,
      "jobId" : 1,
      "taskExecutionId" : 1,
      "instanceId" : 1,
      "name" : "DOCJOB",
      "startDate" : "2023-05-04",
      "startTime" : "00:02:53",
      "startDateTime" : "2023-05-04T00:02:53.583+0000",
      "duration" : "00:00:00",
      "jobParameters" : { },
      "jobParametersString" : "",
      "restartable" : false,
      "abandonable" : false,
      "stoppable" : true,
      "defined" : false,
      "timeZone" : "UTC",
      "status" : "STARTED",
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/jobs/thinexecutions/1"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/jobs/thinexecutions?page=0&size=10"
  "page" : {
    "size" : 10,
    "totalElements" : 2,
    "totalPages" : 1,
    "number" : 0
42.12.3. List All Job Executions With a Specified Job Name
The job executions endpoint lets you list all job executions.
The following topics provide more details:
        "stepExecutions" : [ ],
        "status" : "STOPPING",
        "startTime" : "2023-05-04T00:02:53.583+0000",
        "createTime" : "2023-05-04T00:02:53.569+0000",
        "endTime" : null,
        "lastUpdated" : "2023-05-04T00:02:53.775+0000",
        "exitStatus" : {
          "exitCode" : "UNKNOWN",
          "exitDescription" : ""
        "executionContext" : {
          "dirty" : false,
          "empty" : true,
          "values" : [ ]
        "failureExceptions" : [ ],
        "jobConfigurationName" : null,
        "allFailureExceptions" : [ ]
      "jobParameters" : { },
      "jobParametersString" : "",
      "restartable" : false,
      "abandonable" : true,
      "stoppable" : false,
      "defined" : false,
      "timeZone" : "UTC",
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/jobs/executions/1"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/jobs/executions?page=0&size=10"
  "page" : {
    "size" : 10,
    "totalElements" : 1,
    "totalPages" : 1,
    "number" : 0
42.12.4. List All Job Executions With a Specified Job Name Without Step Executions Included
The job executions endpoint lets you list all job executions.
The following topics provide more details:
      "startDate" : "2023-05-04",
      "startTime" : "00:02:53",
      "startDateTime" : "2023-05-04T00:02:53.583+0000",
      "duration" : "00:00:00",
      "jobParameters" : { },
      "jobParametersString" : "",
      "restartable" : false,
      "abandonable" : true,
      "stoppable" : false,
      "defined" : false,
      "timeZone" : "UTC",
      "status" : "STOPPING",
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/jobs/thinexecutions/1"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/jobs/thinexecutions?page=0&size=10"
  "page" : {
    "size" : 10,
    "totalElements" : 1,
    "totalPages" : 1,
    "number" : 0
42.12.5. List All Job Executions For A Specified Date Range Without Step Executions Included
The job executions endpoint lets you list all job executions.
The following topics provide more details:
GET /jobs/thinexecutions?page=0&size=10&fromDate=2000-09-24T17%3A00%3A45%2C000&toDate=2050-09-24T18%3A00%3A45%2C000 HTTP/1.1
Host: localhost:9393
      "startDate" : "2023-05-04",
      "startTime" : "00:02:53",
      "startDateTime" : "2023-05-04T00:02:53.589+0000",
      "duration" : "00:00:00",
      "jobParameters" : { },
      "jobParametersString" : "",
      "restartable" : true,
      "abandonable" : true,
      "stoppable" : false,
      "defined" : true,
      "timeZone" : "UTC",
      "status" : "STOPPED",
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/jobs/thinexecutions/2"
      "executionId" : 1,
      "stepExecutionCount" : 0,
      "jobId" : 1,
      "taskExecutionId" : 1,
      "instanceId" : 1,
      "name" : "DOCJOB",
      "startDate" : "2023-05-04",
      "startTime" : "00:02:53",
      "startDateTime" : "2023-05-04T00:02:53.583+0000",
      "duration" : "00:00:00",
      "jobParameters" : { },
      "jobParametersString" : "",
      "restartable" : false,
      "abandonable" : true,
      "stoppable" : false,
      "defined" : false,
      "timeZone" : "UTC",
      "status" : "STOPPING",
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/jobs/thinexecutions/1"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/jobs/thinexecutions?page=0&size=10"
  "page" : {
    "size" : 10,
    "totalElements" : 2,
    "totalPages" : 1,
    "number" : 0
42.12.6. List All Job Executions For A Specified Job Instance Id Without Step Executions Included
The job executions endpoint lets you list all job executions.
The following topics provide more details:
      "startDate" : "2023-05-04",
      "startTime" : "00:02:53",
      "startDateTime" : "2023-05-04T00:02:53.583+0000",
      "duration" : "00:00:00",
      "jobParameters" : { },
      "jobParametersString" : "",
      "restartable" : false,
      "abandonable" : true,
      "stoppable" : false,
      "defined" : false,
      "timeZone" : "UTC",
      "status" : "STOPPING",
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/jobs/thinexecutions/1"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/jobs/thinexecutions?page=0&size=10"
  "page" : {
    "size" : 10,
    "totalElements" : 1,
    "totalPages" : 1,
    "number" : 0
42.12.7. List All Job Executions For A Specified Task Execution Id Without Step Executions Included
The job executions endpoint lets you list all job executions.
The following topics provide more details:
      "startDate" : "2023-05-04",
      "startTime" : "00:02:53",
      "startDateTime" : "2023-05-04T00:02:53.583+0000",
      "duration" : "00:00:00",
      "jobParameters" : { },
      "jobParametersString" : "",
      "restartable" : false,
      "abandonable" : true,
      "stoppable" : false,
      "defined" : false,
      "timeZone" : "UTC",
      "status" : "STOPPING",
      "_links" : {
        "self" : {
          "href" : "http://localhost:9393/jobs/thinexecutions/1"
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/jobs/thinexecutions?page=0&size=10"
  "page" : {
    "size" : 10,
    "totalElements" : 1,
    "totalPages" : 1,
    "number" : 0
42.12.8. Job Execution Detail
The job executions endpoint lets you get the details about a job execution.
The following topics provide more details:
    "stepExecutions" : [ ],
    "status" : "STOPPED",
    "startTime" : "2023-05-04T00:02:53.589+0000",
    "createTime" : "2023-05-04T00:02:53.588+0000",
    "endTime" : null,
    "lastUpdated" : "2023-05-04T00:02:53.589+0000",
    "exitStatus" : {
      "exitCode" : "UNKNOWN",
      "exitDescription" : ""
    "executionContext" : {
      "dirty" : false,
      "empty" : true,
      "values" : [ ]
    "failureExceptions" : [ ],
    "jobConfigurationName" : null,
    "allFailureExceptions" : [ ]
  "jobParameters" : { },
  "jobParametersString" : "",
  "restartable" : true,
  "abandonable" : true,
  "stoppable" : false,
  "defined" : true,
  "timeZone" : "UTC",
  "_links" : {
    "self" : {
      "href" : "http://localhost:9393/jobs/executions/2"
Accept: application/json
Host: localhost:9393
Content-Type: application/x-www-form-urlencoded
stop=true
/jobs/executions/{id}
$ curl 'http://localhost:9393/jobs/executions/1' -i -X PUT \
    -H 'Accept: application/json' \
    -d 'stop=true'
42.12.10. Restart Job Execution
The job executions endpoint lets you restart a job execution.
The following topics provide more details:
Accept: application/json
Host: localhost:9393
Content-Type: application/x-www-form-urlencoded
restart=true

/jobs/executions/{id}

$ curl 'http://localhost:9393/jobs/executions/2' -i -X PUT \
    -H 'Accept: application/json' \
    -d 'restart=true'

42.13. Job Instances

The job instances endpoint provides information about the job instances that are registered with the Spring Cloud Data Flow server. The following topics provide more details:

42.13.1. List All Job Instances

The job instances endpoint lets you list all job instances. The following topics provide more details:

42.14. Job Step Executions

The job step executions endpoint provides information about the job step executions that are registered with the Spring Cloud Data Flow server. The following topics provide more details:

42.14.1. List All Step Executions For a Job Execution

The job step executions endpoint lets you list all job step executions. The following topics provide more details:

42.14.2. Job Step Execution Detail

The job step executions endpoint lets you get details about a job step execution. The following topics provide more details:

42.14.3. Job Step Execution Progress

The job step executions endpoint lets you get details about the progress of a job step execution. The following topics provide more details:

42.15. Runtime Information about Applications

You can get information about running apps known to the system, either globally or individually. The following topics provide more details:

42.15.1. Listing All Applications at Runtime

To retrieve information about all instances of all apps, query the /runtime/apps endpoint by using GET . The following topics provide more details:

42.15.2. Querying All Instances of a Single App

To retrieve information about all instances of a particular app, query the /runtime/apps/<appId>/instances endpoint by using GET . The following topics provide more details:

42.15.3. Querying a Single Instance of a Single App

To retrieve information about a particular instance of a particular application, query the /runtime/apps/<appId>/instances/<instanceId> endpoint by using GET . The following topics provide more details:

42.16. Stream Logs

You can get the application logs of the stream for the entire stream or a specific application inside the stream. The following topics provide more details:

42.16.1. Get the applications' logs by the stream name

Use the HTTP GET method with the /streams/logs/<streamName> REST endpoint to retrieve all the applications' logs for the given stream name. The following topics provide more details:

42.16.2. Get the logs of a specific application from the stream

To retrieve the logs of a specific application from the stream, query the /streams/logs/<streamName>/<appName> endpoint using the GET HTTP method. The following topics provide more details:

42.17.1. Get the task execution log

To retrieve the logs of the task execution, query the /tasks/logs/<ExternalTaskExecutionId> endpoint by using the HTTP GET method.. The following topics provide more details:

The Springdoc library is integrated with the server in an opt-in fashion. Once enabled, it provides OpenAPI3 documentation and a Swagger UI.

To enable, set the following properties in your application.yml prior to launching the server:

springdoc:
  api-docs:
    enabled: true
  swagger-ui:
    enabled: true

The properties can also be set on the command line:

-Dspringdoc.api-docs.enabled=true -Dspringdoc.swagger-ui.enabled=true

or as environment variables:

SPRINGDOC_APIDOCS_ENABLED=true
SPRINGDOC_SWAGGERUI_ENABLED=true

Once enabled, the OpenAPI3 docs and Swagger UI are available at the /v3/api-docs and /swagger-ui/index.html URIs, respectively (eg. localhost:9393/v3/api-docs ).

Ask a question. We monitor stackoverflow.com for questions tagged with spring-cloud-dataflow .

Report bugs with Spring Cloud Data Flow at github.com/spring-cloud/spring-cloud-dataflow/issues .

As described in API Guide chapter, Spring Cloud Data Flow’s functionality is completely exposed through REST endpoints. While you can use those endpoints directly, Spring Cloud Data Flow also provides a Java-based API, which makes using those REST endpoints even easier.

The central entry point is the DataFlowTemplate class in the org.springframework.cloud.dataflow.rest.client package.

This class implements the DataFlowOperations interface and delegates to the following sub-templates that provide the specific functionality for each feature-set:

A.1. Using the Data Flow Template

When you use the Data Flow Template, the only needed Data Flow dependency is the Spring Cloud Data Flow Rest Client, as shown in the following Maven snippet:

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-dataflow-rest-client</artifactId>
  <version>2.11.0</version>
</dependency>

With that dependency, you get the DataFlowTemplate class as well as all the dependencies needed to make calls to a Spring Cloud Data Flow server.

When instantiating the DataFlowTemplate , you also pass in a RestTemplate . Note that the needed RestTemplate requires some additional configuration to be valid in the context of the DataFlowTemplate . When declaring a RestTemplate as a bean, the following configuration suffices:

  @Bean
  public static RestTemplate restTemplate() {
    RestTemplate restTemplate = new RestTemplate();
    restTemplate.setErrorHandler(new VndErrorResponseErrorHandler(restTemplate.getMessageConverters()));
    for(HttpMessageConverter<?> converter : restTemplate.getMessageConverters()) {
      if (converter instanceof MappingJackson2HttpMessageConverter) {
        final MappingJackson2HttpMessageConverter jacksonConverter =
            (MappingJackson2HttpMessageConverter) converter;
        jacksonConverter.getObjectMapper()
            .registerModule(new Jackson2HalModule())
            .addMixIn(JobExecution.class, JobExecutionJacksonMixIn.class)
            .addMixIn(JobParameters.class, JobParametersJacksonMixIn.class)
            .addMixIn(JobParameter.class, JobParameterJacksonMixIn.class)
            .addMixIn(JobInstance.class, JobInstanceJacksonMixIn.class)
            .addMixIn(ExitStatus.class, ExitStatusJacksonMixIn.class)
            .addMixIn(StepExecution.class, StepExecutionJacksonMixIn.class)
            .addMixIn(ExecutionContext.class, ExecutionContextJacksonMixIn.class)
            .addMixIn(StepExecutionHistory.class, StepExecutionHistoryJacksonMixIn.class);
    return restTemplate;
PagedResources<AppRegistrationResource> apps = dataFlowTemplate.appRegistryOperations().list();
System.out.println(String.format("Retrieved %s application(s)",
    apps.getContent().size()));
for (AppRegistrationResource app : apps.getContent()) {
  System.out.println(String.format("App Name: %s, App Type: %s, App URI: %s",
    app.getName(),
    app.getType(),
    app.getUri()));
A.2. Data Flow Template and Security
When using the DataFlowTemplate, you can also provide all the security-related
options as if you were using the Data Flow Shell. In fact, the Data Flow Shell
uses the DataFlowTemplate for all its operations.
To let you get started, we provide a HttpClientConfigurer that uses the builder
pattern to set the various security-related options:
		.create(targetUri)                                             (1)
		.basicAuthCredentials(username, password)                      (2)
		.skipTlsCertificateVerification()                              (3)
		.withProxyCredentials(proxyUri, proxyUsername, proxyPassword)  (4)
		.addInterceptor(interceptor)                                   (5)
		.buildClientHttpRequestFactory()                               (6)

Add a custom interceptor e.g. to set the OAuth2 Authorization header. This allows
you to pass an OAuth2 Access Token instead of username/password credentials.
Builds the ClientHttpRequestFactory that can be set on the RestTemplate.
Once the HttpClientConfigurer is configured, you can use its buildClientHttpRequestFactory
to build the ClientHttpRequestFactory and then set the corresponding
property on the RestTemplate. You can then instantiate the actual DataFlowTemplate
using that RestTemplate.
To configure Basic Authentication, the following setup is required:
	RestTemplate restTemplate = DataFlowTemplate.getDefaultDataflowRestTemplate();
	HttpClientConfigurer httpClientConfigurer = HttpClientConfigurer.create("http://localhost:9393");
	httpClientConfigurer.basicAuthCredentials("my_username", "my_password");
	restTemplate.setRequestFactory(httpClientConfigurer.buildClientHttpRequestFactory());
	DataFlowTemplate dataFlowTemplate = new DataFlowTemplate("http://localhost:9393", restTemplate);
This section provides answers to some common ‘how do I do that…​’ questions that often arise when people use Spring Cloud Data Flow.
If you have a specific problem that we do not cover here, you might want to check out stackoverflow.com to see if someone has already provided an answer.
That is also a great place to ask new questions (use the spring-cloud-dataflow tag).
We are also more than happy to extend this section. If you want to add a “how-to”, you can send us a pull request.
B.1. Configure Maven Properties
You can set the Maven properties, such as the local Maven repository location, remote Maven repositories, authentication credentials, and proxy server properties through command-line properties when you start the Data Flow server.
Alternatively, you can set the properties by setting the SPRING_APPLICATION_JSON environment property for the Data Flow server.




    

The remote Maven repositories need to be configured explicitly if the applications are resolved by using the Maven repository, except for a local Data Flow server.
The other Data Flow server implementations (which use Maven resources for application artifacts resolution) have no default value for remote repositories.
The local server has repo.spring.io/libs-snapshot as the default remote repository.
To pass the properties as command-line options, run the server with a command similar to the following:
$ java -jar <dataflow-server>.jar --maven.localRepository=mylocal
--maven.remote-repositories.repo1.url=https://repo1
--maven.remote-repositories.repo1.auth.username=repo1user
--maven.remote-repositories.repo1.auth.password=repo1pass
--maven.remote-repositories.repo2.url=https://repo2 --maven.proxy.host=proxyhost
--maven.proxy.port=9018 --maven.proxy.auth.username=proxyuser
--maven.proxy.auth.password=proxypass
export SPRING_APPLICATION_JSON='{ "maven": { "local-repository": "local","remote-repositories": { "repo1": { "url": "https://repo1", "auth": { "username": "repo1user", "password": "repo1pass" } },
"repo2": { "url": "https://repo2" } }, "proxy": { "host": "proxyhost", "port": 9018, "auth": { "username": "proxyuser", "password": "proxypass" } } } }'
DEPS_FOLDER should be a full filename or path expression for files to copy to the container.
CONTAINER_REPO the source docker image name.
CONTAINER_TAG the tag of source image.
PRIVATE_REGISTRY the host name of the private registry.
export CONTAINER_REPO="springcloud/spring-cloud-dataflow-server"
export CONTAINER_TAG="2.9.5-jdk17"
export PRIVATE_REGISTRY="our.private.registry"
export DEPS_FOLDER="./extra-libs/"
docker build -f Dockerfile -t "$PRIVATE_REGISTRY/$CONTAINER_REPO:$CONTAINER_TAG"
docker push "$PRIVATE_REGISTRY/$CONTAINER_REPO:$CONTAINER_TAG"
B.3.2. JAR File
When using CloudFoundry or local deployment you will need to update jar before publishing it to a private registry or Maven Local.
Example
This example adds the dependencies and then installs the jar to Maven local.
./gradlew -i publishToMavenLocal \
    -P appFolder="." \
    -P appGroup="org.springframework.cloud" \
    -P appName="spring-cloud-dataflow-server" \
    -P appVersion="2.9.5" \
    -P depFolder="./extra-libs"
B.4. Create containers for architectures not supported yet.
In the case of macOS on M1 the performance of amd64/x86_64 is unacceptable.
We provide a set of scripts that can be used to download specific versions of published artifacts.
We also provide a script that will create a container using the downloaded artifact for the host platform.
In the various projects you will find then in src/local or local folders.
Download or create container for: spring-cloud-dataflow-server,

spring-cloud-dataflow-composed-task-runner,

spring-cloud-dataflow-single-step-batch-job,

spring-cloud-dataflow-tasklauncher-sink-kafka,

spring-cloud-dataflow-tasklauncher-sink-rabbit

Skipper
local/download-app.sh

local/create-container.sh
Download or create container for: spring-cloud-skipper-server
Stream Applications
local/download-apps.sh

local/create-containers.sh

local/pack-containers.sh
create-containers.sh uses jib

pack-containers.sh uses pack
src/local/download-apps.sh
Downloads all applications needed by create-containers.sh from Maven repository.
If the timestamp of snapshots matches the download will be skipped.
Usage: download-apps.sh [version]
src/local/create-containers.sh
Creates all containers and pushes to local docker registry.
This script requires jib-cli
Usage: create-containers.sh [version] [jre-version]
local/download-app.sh
Downloads all applications needed by create-containers.sh from Maven repository.
If the timestamp of snapshots matches the download will be skipped.
Usage: download-app.sh [version]
local/download-apps.sh
Downloads all applications needed by create-containers.sh from Maven repository.
If the timestamp of snapshots matches the download will be skipped.
Usage: download-apps.sh [version] [broker] [filter]
B.5.1. Prerequisites
You will need to install kubectl and then kind or minikube for a local cluster.
All the examples assume you have cloned the spring-cloud-dataflow repository and are executing the scripts from src/local/k8s.
On macOS you may need to install realpath from Macports or brew install realpath
Kubernetes Provider
How do I choose between minikube and kind? kind will generally provide quicker setup and teardown time than Minikube. There is little to choose in terms of performance between the 2 apart from being able to configure limits on CPUs and memory when deploying minikube. So in the case where you have memory constraints or need to enforce memory limitations Minikube will be a better option.
Kubectl
You will need to install kubectl in order to configure the Kubernetes cluster
Kind is Kubernetes in docker and ideal for local development.
B.5.3. Building and loading containers.
For local development you need control of the containers used in the local environment.
In order to ensure to manage the specific versions of data flow and skipper containers you can set SKIPPER_VERSION and DATAFLOW_VERSION environmental variable and then invoke ./pull-dataflow.sh and ./pull-skipper.sh or if you want to use a locally built application you can invoke ./build-skipper-image.sh and ./build-dataflow.sh
B.5.4. Configure k8s environment
You can invoke one of the following scripts to choose the type of installation you are targeting:
use-kind.sh [<namespace>] [<database>] [<broker>]
use-mk-docker.sh [<namespace>] [<database>] [<broker>]
use-mk-kvm2.sh [<namespace>] [<database>] [<broker>]
use-mk.sh <driver> [<namespace>] [<database>] [<broker>] (1)
use-tmc.sh <cluster-name> [<namespace>] [<database>] [<broker>]
use-gke.sh <cluster-name> [<namespace>] [<database>] [<broker>]

For kind follow instruction to update src/local/k8s/yaml/metallb-configmap.yaml and then apply using kubectl apply -f src/local/k8s/yaml/metallb-configmap.yaml
For minikube launch a new shell and execute minikube tunnel
build-scdf-pro-image.sh
Build a docker image from the local repo of Dataflow Pro. Set USE_PRO=true in environment to use Dataflow Pro
build-skipper-image.sh
Build a docker image from the local repo of Skipper.
configure-k8s.sh
Configure the Kubernetes environment based on your configuration of K8S_DRIVER.
delete-scdf.sh
Delete all Kubernetes resources create by the deployment.
destroy-k8s.sh
Delete cluster, kind or minikube.
export-dataflow-ip.sh
Export the url of the data flow server to DATAFLOW_IP
export-http-url.sh
Export the url of an http source of a specific flow by name to HTTP_APP_URL
install-scdf.sh
Configure and deploy all the containers for Spring Cloud Dataflow
load-images.sh
Load all container images required by tests into kind or minikube to ensure you have control over what is used.
load-image.sh
Load a specific container image into local kind or minikube.
local-k8s-acceptance-tests.sh
Execute acceptance tests against cluster where DATAFLOW_IP is pointing.
register-apps.sh
Register the Task and Stream apps used by the unit tests.
B.6. Frequently Asked Questions
In this section, we review the frequently asked questions for Spring Cloud Data Flow.
See the Frequently Asked Questions section of the microsite for more information.
This appendix contains information how specific providers can be set up to work
with Data Flow security.
At this writing, Azure is the only identity provider.
C.1. Azure
Azure AD (Active Directory) is a fully fledged identity provider that provide a wide range of features
around authentication and authorization. As with any other provider, it has its
own nuances, meaning care must be taken to set it up.
In this section, we go through how OAuth2 setup is done for AD and
Spring Cloud Data Flow.
C.1.1. Creating a new AD Environment
To get started, create a new Active Directory environment. Choose a
type as Azure Active Directory (not the b2c type) and then pick your organization name and
initial domain. The following image shows the settings:
C.1.2. Creating a New App Registration
App registration is where OAuth clients are created to get used by OAuth
applications. At minimum, you need to create two clients, one for the
Data Flow and Skipper servers and one for the Data Flow shell, as these two have
slightly different configurations. Server applications can be considered to be
trusted applications while shell is not trusted (because users can see its full
configuration).
NOTE:
We recommend using the same OAuth client for both the Data Flow and the Skipper servers. While
you can use different clients, it currently would not provide any value, as the
configurations needs to be the same.
The following image shows the settings for creating a a new app registration:
C.1.4. Creating a Privileged Client
For the OAuth client, which is about to use password grants, the same API permissions need
to be created for the OAuth client as were used for the server (described in the previous section).
Privileged client needs a client secret, which needs to be exposed to a client
configuration when used in a shell. If you do not want to expose that secret, use the
Creating a Public Client public client.
C.1.5. Creating a Public Client
A public client is basically a client without a client secret and with its type set to public.
The following image shows the configuration of a public client:
C.1.6. Configuration Examples
This section contains configuration examples for the Data Flow and Skipper servers and the shell.
To starting a Data Flow server:
                ROLE_DESTROY: dataflow.destroy
                ROLE_MODIFY: dataflow.modify
                ROLE_SCHEDULE: dataflow.schedule
  security:
    oauth2:
      client:
        registration:
          dataflow-server:
            provider: azure
            redirect-uri: '{baseUrl}/login/oauth2/code/{registrationId}'
            client-id: <client id>
            client-secret: <client secret>
            scope:
            - openid
            - profile
            - email
            - offline_access
            - api://dataflow-server/dataflow.view
            - api://dataflow-server/dataflow.deploy
            - api://dataflow-server/dataflow.destroy
            - api://dataflow-server/dataflow.manage
            - api://dataflow-server/dataflow.modify
            - api://dataflow-server/dataflow.schedule
            - api://dataflow-server/dataflow.create
        provider:
          azure:
            issuer-uri: https://login.microsoftonline.com/799dcfde-b9e3-4dfc-ac25-659b326e0bcd/v2.0
            user-name-attribute: name
      resourceserver:
          jwk-set-uri: https://login.microsoftonline.com/799dcfde-b9e3-4dfc-ac25-659b326e0bcd/discovery/v2.0/keys

$ java -jar spring-cloud-dataflow-shell.jar \
  --spring.config.additional-location=dataflow-azure-shell.yml \
  --dataflow.username=<USERNAME> \
  --dataflow.password=<PASSWORD>

  security:
    oauth2:
      client:
        registration:
          dataflow-shell:
            provider: azure
            client-id: <client id>
            client-secret: <client secret>
            authorization-grant-type: password
            scope:
            - offline_access
            - api://dataflow-server/dataflow.create
            - api://dataflow-server/dataflow.deploy
            - api://dataflow-server/dataflow.destroy
            - api://dataflow-server/dataflow.manage
            - api://dataflow-server/dataflow.modify
            - api://dataflow-server/dataflow.schedule
            - api://dataflow-server/dataflow.view
        provider:
          azure:
            issuer-uri: https://login.microsoftonline.com/799dcfde-b9e3-4dfc-ac25-659b326e0bcd/v2.0

$ java -jar spring-cloud-dataflow-shell.jar \
  --spring.config.additional-location=dataflow-azure-shell-public.yml \
  --dataflow.username=<USERNAME> \
  --dataflow.password=<PASSWORD>

spring:
  security:
    oauth2:
      client:
        registration:
          dataflow-shell:
            provider: azure
            client-id: <client id>
            authorization-grant-type: password
            client-authentication-method: post
            scope:
            - offline_access
            - api://dataflow-server/dataflow.create
            - api://dataflow-server/dataflow.deploy
            - api://dataflow-server/dataflow.destroy
            - api://dataflow-server/dataflow.manage
            - api://dataflow-server/dataflow.modify
            - api://dataflow-server/dataflow.schedule
            - api://dataflow-server/dataflow.view
        provider:
          azure:
            issuer-uri: https://login.microsoftonline.com/799dcfde-b9e3-4dfc-ac25-659b326e0bcd/v2.0