Synchronizing Concurrent Workflows | Replay 2023

At Yum Brands, we initially implemented the former approach for our menu publishing process since it was somewhat less code & slightly more intuitive.

However, it came with a notable caveat that wasn’t immediately apparent to us as new Temporal users- additional complexity to tracking the success and failure of the synchronized activities. That complexity is due to the synchronized activities running in a workflow entirely independent of the parent workflow where we’re aggregating successes & failures.

This is of particular concern for us since: - Our parent workflow performs a large batch of these synchronized activities - There is a distinct possibility that these activities will have non-recoverable failures that we must track.

The latter approach resolves that complexity by keeping the batch of activities executing directly in the parent workflow, where we can easily track their successes and failures with standard TS try/catch flow control. This change significantly simplified the code for our use case despite requiring somewhat more code overall.

---

Temporal is the simple, scalable, open source way to write and run reliable cloud applications.

Learn more

Developer resources

[joebowbeer]

Deadlock is avoided by acquiring locks in a strict order? How often are locks contended in practice? Would the reservation pattern be applicable here? (just wondering)

[scottamolinari]

Sounds like a version control process. If you think about it, this issue of concurrent workflows with making sure who has priority to set a version of a menu sounds a lot like the issues one has with devs working in a git repository concurrently. So modeling a git workflow might have fitted?

Another possible path to a solution for this could be optimistic concurrency control, (maybe?) where the store menus are versioned and before a mutation process is started, the version number is pulled from the store. If the version during the update had changed, it causes a failure, meaning, someone else's changes were taken before the update could occur. There are no locks. This is only considering race conditions of course. To me though, it also sounds like there is a business decision consideration too. The store seems to also wish to override the global menu?

At any rate, this was a very interesting talk and I appreciate the use case description (and the humor) at the beginning.