Distributed Scheduling with Spring Boot: the challenges & pitfalls of implementing a background job

Spring I/O 2024 - 30-31 May, Barcelona

Speaker: Rafael Ponte

Sooner or later a developer will implement his/her first background job using Java and Spring Boot, and what usually is a simple task for the majority of systems might become a nightmare in scenarios that need to deal with high performance, parallelism, distributed systems and a large volume of data. Scenarios like those hide several issues which many developers are not used to, such as large volumes of data, network failures, data inconsistency, out-of-memory errors and even taking the whole system down.

Although it seems controversial, dealing with many of these problems does not require hype technologies or services, but solid distributed systems fundamentals. This talk will present how an experienced developer implements a background job with Java and Spring Boot taking into consideration the main challenges and pitfalls it brings along, and how he/she designs a solution for high-performance, resilience and horizontal scalability at the same time he/she takes advantage of many modules of Spring Boot, Hibernate and the relational database.

If you still believe that a background job is a simple task, so this talk is for you!

[gustadev]

Que talk boa, aprendi tanta coisa que fiquei até perdido!

[eduardo]

Congratulations on your presentation! You absolutely nailed it. Your thorough research and confident delivery captivated everyone in the room. Your ability to explain complex ideas so clearly is truly impressive. Keep up the fantastic work!

[codeisma]

Great talk! There are a few Java libraries that already solve these challenges (db-scheduler, JobRunr or Quartz). At JobRunr we'd love to share your talk as it explains JobRunr's architecture well and can help our users understand the challenges of distributed scheduling even better!

[RaphaelDeLio]

Parabéns, Rafael! Foi um prazer assistir sua apresentação pessoalmente!

[linhvudev]

Thanks Rafael! especially for the SKIP_LOCKED feature, new knowledge learnt

[popastefan]

36:48 Actually in our example, each instance will fight for first 50 records, not one record as it is illustrated in the slide.

[felipedossantos]

I've seen this presentation in portuguese before of Rafael Pontes in Zup Channel, and I could implement something similar it in my job. Great work, Bro! Thank you so much

[danielponte]

Parabéns meu irmão, você deu um show na apresentação, impecável! show de top!

[terteseamos]

this for me is the best presentation. Great job

[balazsklezli]

It was pretty cool talk, thank you for it!

[YZ-ixdn]

Thank you for clear and well-structured presentation. It's very useful and important information even for people with lots years of experience. I wish every developer should watch this video when every time they put @Transactional onto theirs method.

[knoppix]

26:34, 31:59, 32:14, 36:04, 40:31 - key moments

[jesprotech]

I really like the way you explained short running transactions. Nice addition to the jobs! Parabéns pela excelente apresentação! É muito útil!

[ferlezcano]

Excellent topic! Have some background jobs running here and there and I definitely going to check them again.

[hirenpandit]

Great talk!! so much learnings and addressed real life problems I faced while writing background scheduled jobs... btw we used ShedLock library but this is real good insight.

[matthewmoore]

Great talk! A couple of thoughts. Your statement about entity state and transactions is only true if Spring's "open session in view" is not enabled. I find there is a lot of confusion out there about the Hibernate session, transaction state, OSIV, and entity state. Along similar lines, the call to a repository save() method is unnecessary when updating an attached entity because of Hibernate change tracking, and calling save() leads people to assume that it persists changes, which (counterintuitively) it doesn't. (It adds/merges detached entities to the session/persistence context.) Regarding transaction scope, I would argue it is still too broad. Work for a single user/card should generally happen in its own transaction, at least in an OLTP context.

[mindcontrolkmc.]

Really great talk!
But I am curious that if 2 save statements already wrap in 1 small transaction how can it combine with the hibernate batch with another save statement process

[gjperes]

O Rafael é fera demais!! Great presentation

[aleksandrS]

Does a single @Transactional annotation for Scheduled method (in case of JPA framework) fix the original code right away?

[bkavun]

Great presentation, great work. Thanks a lot for sharing this knowledge with us!