|
|
Creating a Batch Service
This guide walks you through the process of creating a basic batch-driven solution.
What You Will build
You will build a service that imports data from a CSV spreadsheet, transforms it with custom code, and stores the final results in a database.
Java 17 or later
You can also import the code straight into your IDE:
Like most Spring Getting Started guides , you can start from scratch and complete each step or you can bypass basic setup steps that are already familiar to you. Either way, you end up with working code.
To start from scratch , move on to Starting with Spring Initializr .
To skip the basics , do the following:
Download
and unzip the source repository for this guide, or clone it using
Git
:
git clone
https://github.com/spring-guides/gs-batch-processing.git
cd into
gs-batch-processing/initial
Jump ahead to Create a Business Class .
When you finish
, you can check your results against the code in
gs-batch-processing/complete
.
This spreadsheet contains a first name and a last name on each row, separated by a comma. This is a fairly common pattern that Spring can handle without customization.
Next, you need to write an SQL script to create a table to store the data. You can find such a script in
src/main/resources/schema-all.sql
:
You can use this pre-initialized project and click Generate to download a ZIP file. This project is configured to fit the examples in this tutorial.
To manually initialize the project:
Navigate to https://start.spring.io . This service pulls in all the dependencies you need for an application and does most of the setup for you.
Choose either Gradle or Maven and the language you want to use. This guide assumes that you chose Java.
Click Dependencies and select Spring Batch and HyperSQL Database .
Click Generate .
Download the resulting ZIP file, which is an archive of a web application that is configured with your choices.
A common paradigm in batch processing is to ingest data, transform it, and then pipe it out somewhere else. Here, you need to write a simple transformer that converts the names to uppercase. The following listing (from
src/main/java/com/example/batchprocessing/PersonItemProcessor.java
) shows how to do so:
Now you need to put together the actual batch job. Spring Batch provides many utility classes that reduce the need to write custom code. Instead, you can focus on the business logic.
To configure your job, you must first create a Spring
@Configuration
class like the following example in
src/main/java/com/exampe/batchprocessing/BatchConfiguration.java
. This example uses a memory-based database, meaning that, when it is done, the data is gone. Now add the following beans to your
BatchConfiguration
class to define a reader, a processor, and a writer:
@Bean
public FlatFileItemReader<Person> reader() {
return new FlatFileItemReaderBuilder<Person>()
.name("personItemReader")
.resource(new ClassPathResource("sample-data.csv"))
.delimited()
.names(new String[]{"firstName", "lastName"})
.fieldSetMapper(new BeanWrapperFieldSetMapper<Person>() {{
setTargetType(Person.class);
.build();
@Bean
public PersonItemProcessor processor() {
return new PersonItemProcessor();
@Bean
public JdbcBatchItemWriter<Person> writer(DataSource dataSource) {
return new JdbcBatchItemWriterBuilder<Person>()
.itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
.sql("INSERT INTO people (first_name, last_name) VALUES (:firstName, :lastName)")
.dataSource(dataSource)
.build();
reader() creates an ItemReader. It looks for a file called sample-data.csv and parses each line item with enough information to turn it into a Person.
processor() creates an instance of the PersonItemProcessor that you defined earlier, meant to convert the data to upper case.
writer(DataSource) creates an ItemWriter. This one is aimed at a JDBC destination and automatically gets a copy of the dataSource created by @EnableBatchProcessing. It includes the SQL statement needed to insert a single Person, driven by Java bean properties.
The last chunk (from src/main/java/com/example/batchprocessing/BatchConfiguration.java) shows the actual job configuration:
@Bean
public Job importUserJob(JobRepository jobRepository,
JobCompletionNotificationListener listener, Step step1) {
return new JobBuilder("importUserJob", jobRepository)
.incrementer(new RunIdIncrementer())
.listener(listener)
.flow(step1)
.end()
.build();
@Bean
public Step step1(JobRepository jobRepository,
PlatformTransactionManager transactionManager, JdbcBatchItemWriter<Person> writer) {
return new StepBuilder("step1", jobRepository)
.<Person, Person> chunk(10, transactionManager)
.reader(reader())
.processor(processor())
.writer(writer)
.build();
The first method defines the job, and the second one defines a single step. Jobs are built from steps, where each step can involve a reader, a processor, and a writer.
In this job definition, you need an incrementer, because jobs use a database to maintain execution state. You then list each step, (though this job has only one step). The job ends, and the Java API produces a perfectly configured job.
In the step definition, you define how much data to write at a time. In this case, it writes up to ten records at a time. Next, you configure the reader, processor, and writer by using the beans injected earlier.
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.core.BatchStatus;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.JobExecutionListener;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Component;
@Component
public class JobCompletionNotificationListener implements JobExecutionListener {
private static final Logger log = LoggerFactory.getLogger(JobCompletionNotificationListener.class);
private final JdbcTemplate jdbcTemplate;
@Autowired
public JobCompletionNotificationListener(JdbcTemplate jdbcTemplate) {
this.jdbcTemplate = jdbcTemplate;
@Override
public void afterJob(JobExecution jobExecution) {
if(jobExecution.getStatus() == BatchStatus.COMPLETED) {
log.info("!!! JOB FINISHED! Time to verify the results");
jdbcTemplate.query("SELECT first_name, last_name FROM people",
(rs, row) -> new Person(
rs.getString(1),
rs.getString(2))
).forEach(person -> log.info("Found <{{}}> in the database.", person));
Although batch processing can be embedded in web apps and WAR files, the simpler approach demonstrated below creates a standalone application. You package everything in a single, executable JAR file, driven by a good old Java main() method.
The Spring Initializr created an application class for you. For this simple example, it works without further modification. The following listing (from src/main/java/com/example/batchprocessing/BatchProcessingApplication.java) shows the application class:
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class BatchProcessingApplication {
public static void main(String[] args) throws Exception {
System.exit(SpringApplication.exit(SpringApplication.run(BatchProcessingApplication.class, args)));
@Configuration: Tags the class as a source of bean definitions for the application context.
@EnableAutoConfiguration: Tells Spring Boot to start adding beans based on classpath settings, other beans, and various property settings. For example, if spring-webmvc is on the classpath, this annotation flags the application as a web application and activates key behaviors, such as setting up a DispatcherServlet.
@ComponentScan: Tells Spring to look for other components, configurations, and services in the com/example package, letting it find the controllers.
The main() method uses Spring Boot’s SpringApplication.run() method to launch an application. Did you notice that there was not a single line of XML? There is no web.xml file, either. This web application is 100% pure Java and you did not have to deal with configuring any plumbing or infrastructure.
Note that SpringApplication.exit() and System.exit() ensure that the JVM exits upon job completion. See the Application Exit section in Spring Boot Reference documentation for more details.
For demonstration purposes, there is code to create a JdbcTemplate, query the database, and print out the names of people the batch job inserts.
Build an executable JAR
You can run the application from the command line with Gradle or Maven. You can also build a single executable JAR file that contains all the necessary dependencies, classes, and resources and run that. Building an executable jar makes it easy to ship, version, and deploy the service as an application throughout the development lifecycle, across different environments, and so forth.
If you use Gradle, you can run the application by using ./gradlew bootRun. Alternatively, you can build the JAR file by using ./gradlew build and then run the JAR file, as follows:
Converting (firstName: Jill, lastName: Doe) into (firstName: JILL, lastName: DOE)
Converting (firstName: Joe, lastName: Doe) into (firstName: JOE, lastName: DOE)
Converting (firstName: Justin, lastName: Doe) into (firstName: JUSTIN, lastName: DOE)
Converting (firstName: Jane, lastName: Doe) into (firstName: JANE, lastName: DOE)
Converting (firstName: John, lastName: Doe) into (firstName: JOHN, lastName: DOE)
Found <firstName: JILL, lastName: DOE> in the database.
Found <firstName: JOE, lastName: DOE> in the database.
Found <firstName: JUSTIN, lastName: DOE> in the database.
Found <firstName: JANE, lastName: DOE> in the database.
Found <firstName: JOHN, lastName: DOE> in the database.
Want to write a new guide or contribute to an existing one? Check out our contribution guidelines.