Recently, I watched Quarkus presentations by @burrsutter. The topic interested me a lot because although on a daily basis I don’t work with systems that requires running many instances of microservices, but as a developer I am always interested when someone says that something works faster and needs less resources. Burr in his presentation showed very quickly the comparison of Quarkus and Spring Boot, but I wanted to do it a little more accurately myself while playing with a new technology.
During presentation, an important question was asked about comparing the Spring Boot application that is running on a “warmed up” JVM and thus could be faster than the native Quarkus application…let’s answer this question with some data.
I use Kotlin in both apps and I will give additional thoughts on the use of Kotlin with Quarkus.
Description of both projects
First I will show code of both applications and describe it a bit, and then we will go to performance testing and comparison.
This is Quarkus project build.gradle.kts file:
import io.quarkus.gradle.tasks.QuarkusNative
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
group = "pl.codeaddict"
version = "1.0.0-SNAPSHOT"
plugins {
java
id("io.quarkus") version "1.3.1.Final"
kotlin("jvm") version "1.3.71"
kotlin("plugin.allopen") version "1.3.71"
}
repositories {
mavenCentral()
}
dependencies {
implementation(kotlin("stdlib-jdk8"))
implementation(enforcedPlatform("io.quarkus:quarkus-bom:1.3.1.Final"))
implementation("io.quarkus:quarkus-kotlin")
implementation("io.quarkus:quarkus-resteasy")
implementation("io.quarkus:quarkus-resteasy-jackson")
implementation("io.quarkus:quarkus-agroal")
implementation("io.quarkus:quarkus-jdbc-h2")
testImplementation("io.quarkus:quarkus-junit5")
testImplementation("io.rest-assured:rest-assured")
}
tasks {
named<QuarkusNative>("buildNative") {
isEnableHttpUrlHandler = true
}
}
allOpen {
annotation("javax.ws.rs.Path")
annotation("javax.enterprise.context.ApplicationScoped")
annotation("io.quarkus.test.junit.QuarkusTest")
}
java {
sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_11
}
val compileKotlin: KotlinCompile by tasks
compileKotlin.kotlinOptions {
jvmTarget = "11"
}
val compileTestKotlin: KotlinCompile by tasks
compileTestKotlin.kotlinOptions {
jvmTarget = "11"
}
Some things need to be explained.
First of all, remember that if you want to build a native application in Quarkus, you must use Quarkus extensions. The list of extensions is available here, and I think there are so many of them that they can meet a lot of developers’ needs. So I used the extensions that were required to create a simple application that provides REST API and connects to the database. For the latter, I used the simplest agroal library which provide JDBC connection pool and provide interfaces to communicate with the database.
Another important thing is isEnableHttpUrlHandler option, you need to set it to true if you creating web application and http url handler should be enabled in native build.
As you can see I used the allopen plugin which sets the compilation so that all classes are not final but only if they have specific annotations (check allOpen code block). This allows to create proxy objects and is required by many libraries including Quarkus when using Kotlin.
To create project configuration you can also use an initilizer similar to the one offered by Spring.
Let’s have look on Spring Boot project build.gradle.kts file:
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
plugins {
id("org.springframework.boot") version "2.2.6.RELEASE"
id("io.spring.dependency-management") version "1.0.9.RELEASE"
kotlin("jvm") version "1.3.71"
kotlin("plugin.spring") version "1.3.71"
}
group = "pl.codeaddict"
version = "1.0.0-SNAPSHOT"
repositories {
mavenCentral()
}
dependencies {
implementation("org.springframework.boot:spring-boot-starter")
implementation("org.springframework.boot:spring-boot-starter-web")
implementation("org.springframework.boot:spring-boot-starter-data-jdbc")
runtimeOnly("com.h2database:h2:1.4.200")
implementation("org.jetbrains.kotlin:kotlin-reflect")
implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")
testImplementation("org.springframework.boot:spring-boot-starter-test") {
exclude(group = "org.junit.vintage", module = "junit-vintage-engine")
}
testImplementation("io.rest-assured:rest-assured")
}
tasks.withType<Test> {
useJUnitPlatform()
}
java {
sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_11
}
val compileKotlin: KotlinCompile by tasks
compileKotlin.kotlinOptions {
jvmTarget = "11"
freeCompilerArgs = listOf("-Xjsr305=strict")
}
val compileTestKotlin: KotlinCompile by tasks
compileTestKotlin.kotlinOptions {
jvmTarget = "11"
}
As you can see in above file I used very similar libraries so that the projects differ as little as possible. Access to the database will be done through the JDBC template.
Ok, time to show API controller (Quarkus project):
import javax.ws.rs.*
import javax.ws.rs.core.MediaType
@Path("/api")
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
class ApiResource(private val service: ApiResourceService) {
@GET
@Path("/{data}")
fun apiGet(@PathParam("data") data: String): DataObject {
return service.readDataObject(data)
.orElseThrow { RuntimeException("No data object with data: $data") }
}
@POST
fun apiPost(dataObject: DataObject): Int {
return service.storeDataObject(dataObject)
}
}
I don’t think there is much to explain here. In Quarkus we use JAX-RS annotations to configure context of the application. I created two endpoints, one for returning data from DB and one for saving new data in DB.
API controller in Spring Boot is very similar:
import org.springframework.http.MediaType
import org.springframework.web.bind.annotation.*
@RestController
class ApiResource(private val service: ApiResourceService) {
@GetMapping(path = ["/api/{name}"], produces = [MediaType.APPLICATION_JSON_VALUE])
fun apiGet(@PathVariable name: String): DataObject {
return service.readDataObject(name)
.orElseThrow { RuntimeException("No data object with name: $name") }
}
@PostMapping(path = ["/api"], consumes = [MediaType.APPLICATION_JSON_VALUE], produces = [MediaType.APPLICATION_JSON_VALUE])
fun apiPost(@RequestBody dataObject: DataObject): Int {
return service.storeDataObject(dataObject)
}
}
Let’s have a look at main application service in Quarkus project:
import io.agroal.api.AgroalDataSource
import java.sql.Connection
import java.sql.ResultSet
import java.util.*
import javax.inject.Singleton
@Singleton
open class ApiResourceService(private val dataSource: AgroalDataSource) {
fun readDataObject(data: String): Optional<DataObject> {
synchronized(this) {
val sql = "SELECT * FROM DATA_OBJECTS WHERE DATA = '$data'"
val connection = getConnection()
val statement = connection.createStatement()
val results: ResultSet = statement.executeQuery(sql)
var dataResult: DataObject? = null
while (results.next()) {
val resultData: String = results.getString("DATA")
dataResult = DataObject(resultData)
break
}
statement.close()
connection.close()
return Optional.ofNullable(dataResult)
}
}
fun storeDataObject(data: DataObject): Int {
synchronized(this) {
val connection = getConnection()
val statement = connection.createStatement()
statement.execute("DROP TABLE DATA_OBJECTS IF EXISTS")
statement.execute("CREATE TABLE DATA_OBJECTS(" +
"ID INT, DATA VARCHAR(100))")
val result: Int =
statement.executeUpdate("INSERT INTO DATA_OBJECTS(DATA) VALUES ('${data.data}')")
statement.close()
connection.close()
return result
}
}
private fun getConnection(): Connection {
println("JDBC Metrics:" + dataSource.metrics)
return dataSource.connection
}
}
One important thing worth mention. Quarkus gives us hot reload when we use ./gradlew quarkusDev without any additional requirements. It’s cool feature. Unfortunately, when we use Kotlin we can encounter many problems with the initialization or injection of beans. For example, the annotation @ApplicationScope which can be found in many Quarkus examples, makes the dataSource object not initializing. When we use this annotation and we change something in the ApiResourceService.class and hot reload occurs dataSource object will be null. I used the @Singleton annotation here, but its limitations should be taken into account (check StackOverflow question) or do not use the Kotlin with Quarkus 
(Kotlin is still marked as beta on Quarkus initializer site).
Another problem I found is that we can’t use @Synchronized annotation because Quarkus won’t start in dev mode although I wouldn’t use them too often 
.
Let’s see same service but for Spring Boot project:
import org.springframework.jdbc.core.JdbcTemplate
import org.springframework.stereotype.Service
import java.sql.ResultSet
import java.util.*
@Service
open class ApiResourceService(private val jdbcTemplate: JdbcTemplate) {
fun readDataObject(data: String): Optional<DataObject> {
synchronized(this) {
val sql = "SELECT * FROM DATA_OBJECTS WHERE DATA = ?"
return Optional.ofNullable(jdbcTemplate.queryForObject(sql, arrayOf<Any>(data)) { rs: ResultSet, _: Int ->
DataObject(rs.getString("DATA"))
})
}
}
fun storeDataObject(dataObject: DataObject): Int {
synchronized(this) {
jdbcTemplate.execute("DROP TABLE DATA_OBJECTS IF EXISTS")
jdbcTemplate.execute("CREATE TABLE DATA_OBJECTS(" +
"ID INT, DATA VARCHAR(100))")
return jdbcTemplate.update("INSERT INTO DATA_OBJECTS(DATA) VALUES (?)", dataObject.data)
}
}
}
In both project I use same entity model:
data class DataObject (val data: String = "")
Last but not least, properties files. First Quarkus:
quarkus.datasource.db-kind=h2
quarkus.datasource.username=sa
quarkus.datasource.jdbc.url=jdbc:h2:tcp://localhost:1521/test
quarkus.datasource.jdbc.initial-size=10
quarkus.datasource.jdbc.min-size=10
quarkus.datasource.jdbc.max-size=100
quarkus.datasource.jdbc.enable-metrics=true
quarkus.datasource.metrics.enabled=true
I added properties which turn on metrics for datasource so I can check if connections are created and closed porperly. It’s worth to mention that Quarkus use one property file for every profiles (configuration entries for each profile are marked with prefix), so I thinking that it can be quite messy sometimes. In the first versions it was not possible to use the YAML file but now it is possible after adding the appropriate extension.
Spring Boot property file:
spring.datasource.url=jdbc:h2:tcp://localhost:1521/test
spring.datasource.username=sa
spring.datasource.password=
spring.datasource.hikari.minimumIdle=10
spring.datasource.hikari.maximumPoolSize=100
logging.level.com.zaxxer.hikari=TRACE
I’ve turned on logging for HikariCP to see status of connection pool.
The only visible difference between projects is the parameter quarkus.datasource.jdbc.initial-size. The connection pool in Spring Boot is created in the amount specified by the spring.datasource.hikari.minimumIdle parameter so here I had to set it to have Quarkus application have pool initialized same as Spring Boot.
Running the applications
We can proceed to running the application. Let’s start by running the H2 database in a separate docker container. We can do it using the prepared docker-compose.yml configuration which can be found in the root project directory. Just use below command:
docker-compose up -d h2
…this will start H2 database. Now we have few options when it comes to running Quarkus project:
- Start in JVM (dev mode) - just run
./gradlew quarkusDev. This mode is made for development and it has hot reload turn on by default. - Build the application uber-JAR and run so that it use JVM. You can build uber JAR using
./gradlew clean build -x test -Dquarkus.package.uber-jar=truecommand and run builded JAR withjava -jar ./build/QuarkusSimpleAPI-1.0.0-SNAPSHOT-runner.jar. I ommited tests because the only ones that I created are integration tests and applciation need to be up and running before. - Native build (using Docker) - I prepared docker configuration in order to simplify build process. You can use
./build_native.shscript from QuarkusSimpleAPI project dir and then./run_native.shscript. - Native (without docker). You need to install GrallVM and run
./gradlew clean build -x test -Dquarkus.package.type=nativeand you can runQuarkusSimpleAPI-1.0.0-SNAPSHOT-runneras normal application.
With Spring Boot project we have two options. You can simply run ./gradlew bootRun or build JAR using ./gradlew build -x test command and then run it using: java -jar ./build/SpringSimpleAPI-1.0.0-SNAPSHOT.jar
To run everything you can use docker-compose up command in project root directory. It runs Quarkus application, both in native and JVM mode, Spring Boot app and H2 in separate container.
If you run the applications, you will immediately see the difference in the start time of each version. But it’s time to run deep dive into tests 
.
Tests
Application launch time
- Test conditions: I run applications one at a time in docker containers with the H2 container already running. The values are taken from the application log.
- Number of attempts: 11
- Results:
app-quarkus-jvm,3.665,1.737,1.716,1.715,1.667,1.877,1.811,1.676,1.895,1.793,1.921 app-spring-boot,3.807,2.724,2.856,2.891,2.819,2.760,2.732,2.744,2.758,2.818,2.908 app-quarkus-native,0.892,0.013,0.016,0.014,0.014,0.012,0.012,0.014,0.014,0.012,0.013The first launch of each container is noticeably slower but I have kept this data in results.
- Plot:
As you can see, Quarkus (native) wins with a significant advantage.
Artifact size
- Results:
app-quarkus-jvm,23.388866 app-spring-boot,26.882397 app-quarkus-native,47.413896 - Plot:
The native application built using Quarkus has the largest size, but keep in mind that we don’t need Java Runtime with Quarkus Native, so it can really save us a lot.
Application memmory usage
- Test conditions: I start all applications on docker and read
dockerstats. First value is memmory usage when application is at rest and second is maximal usage when receiving requests (I used the wrk tool to load the application.wrk -t2 -c10 -d1m $urlcommand) - Results:
app-quarkus-jvm,97.34,418.2 app-spring-boot,287.3,601.5 app-quarkus-native,6.488,282.5 - Plot:
As you can see, Quarkus also wins the competition.
Testing performance
To make testing easier, I added an Dockerfile with the wrk tool that I will be using to test performance. If you want to run tests by yourself first you should build image using test-scripts/perfBuildDocker.sh and then you can run any test script (scripts are prefixed with perf).
I set the connection pool size for both applications to min 10, max 100. An I run wrk with 2 simultaneous threads and 10 connections with duration of 10 seconds. This can also be described as ten users that request repeatedly for ten seconds.
Application has two endpoints GET which reads from database and POST to write to it.
Quarkus on JVM
- GET ‘/api/{data}’
- Script used to run tests:
perfGetObjQuarkusJVM.sh - Results:
Running 10s test @ http://quarkus-vs-spring-app-quarkus-jvm:8080/api/SlimShady 2 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 15.57ms 33.22ms 343.49ms 97.10% Req/Sec 526.93 251.38 1.50k 65.46% 10219 requests in 10.02s, 0.89MB read Requests/sec: 1020.09 Transfer/sec: 90.65KB
- Script used to run tests:
- POST ‘/api/’
- Script used to run tests:
perfPostObjQuarkusNative.sh - Results:
Running 10s test @ http://quarkus-vs-spring-app-quarkus-jvm:8080/api/ 2 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 22.45ms 40.84ms 387.82ms 96.29% Req/Sec 332.73 112.97 580.00 67.53% 6443 requests in 10.01s, 446.73KB read Requests/sec: 643.53 Transfer/sec: 44.62KB
- Script used to run tests:
Quarkus as native application
- GET ‘/api/{data}’
- Script used to run tests:
perfGetObjQuarkusNative.sh - Results:
Running 10s test @ http://quarkus-vs-spring-app-quarkus-native:8080/api/SlimShady 2 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 4.64ms 2.15ms 26.13ms 78.92% Req/Sec 1.09k 103.47 1.36k 72.50% 21780 requests in 10.00s, 1.89MB read Requests/sec: 2177.64 Transfer/sec: 193.52KB
- Script used to run tests:
- POST ‘/api/’
- Script used to run tests:
perfPostObjQuarkusNative.sh - Results:
Running 10s test @ http://quarkus-vs-spring-app-quarkus-native:8080/api/ 2 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 9.02ms 2.99ms 45.99ms 74.27% Req/Sec 560.27 67.75 696.00 66.00% 11160 requests in 10.01s, 773.79KB read Requests/sec: 1114.74 Transfer/sec: 77.29KB
- Script used to run tests:
Spring Boot
- GET ‘/api/{data}’
- Script used to run tests:
perfGetObjSpring.sh - Results:
Running 10s test @ http://quarkus-vs-spring-app-spring-jvm:8080/api/SlimShady 2 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 11.41ms 45.57ms 441.14ms 96.85% Req/Sec 1.69k 649.85 2.70k 58.85% 32407 requests in 10.00s, 4.49MB read Requests/sec: 3239.68 Transfer/sec: 459.34KB
- Script used to run tests:
- POST ‘/api/’
- Script used to run tests:
perfPostObjSpring.sh - Results:
Running 10s test @ http://quarkus-vs-spring-app-spring-jvm:8080/api/ 2 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 14.41ms 46.06ms 429.64ms 96.80% Req/Sec 847.83 303.67 1.80k 71.35% 16240 requests in 10.00s, 1.94MB read Requests/sec: 1623.40 Transfer/sec: 198.48KBSpring Boot long run test (warming up JVM)
Test conditions are similar to those above –> 2 simultaneous threads with 10 connections, but this time I use duration of 15 min.
- Script used to run tests:
- GET ‘/api/{data}’
- Script used to run tests:
perfGetObjSpringLong.sh - Results:
Running 15m test @ http://quarkus-vs-spring-app-spring-jvm:8080/api/SlimShady 2 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 1.55ms 4.86ms 402.72ms 99.50% Req/Sec 3.59k 361.49 4.72k 74.17% 6423216 requests in 15.00m, 0.87GB read Requests/sec: 7136.70 Transfer/sec: 0.99MB
- Script used to run tests:
- POST ‘/api/’
- Script used to run tests:
perfPostObjSpringLong.sh - Results:
Running 15m test @ http://quarkus-vs-spring-app-spring-jvm:8080/api/ 2 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 3.77ms 5.83ms 438.59ms 99.22% Req/Sec 1.40k 146.65 2.01k 73.86% 2500953 requests in 15.00m, 298.59MB read Requests/sec: 2778.57 Transfer/sec: 339.70KB
- Script used to run tests:
Lets summarize and plot data!
First let’s see latency for our requests. I take average latencies:
Requests per second:
As you can see Quarkus is doing well, however, the warm-up Spring Boot came out best. To sum up, if we have a microservice environment in which we often launch new instances and we care about immediate performance, it is worth using Quarkus. On the other hand, if we have long-lived service, you can easily stay with the good old Spring Boot. Of course, the size of the application and the need for RAM are also important considerations - these can be important indicators when choosing a framework. Quarkus is still a young project but it is worth paying attention to it.
This is it! You can find all the source code in my repository GitHub account. I also share the jupyter notebook in it. Have fun and thanks for reading!
