Long-running Cloud Run Jobs

Cloud Run Jobs is a great way of running unattended workloads on a schedule, like data conversions or cleanups. And your jobs can now run for up to 24 hours. Follow along as Karolína Netolická and Martin Omander deploy a data import job and try out various options.

Chapters:
0:00 - Intro
1:11 - Deploy a job
1:53 - Schedule the job
2:22 - Parallelize the job
3:02 - Recap

#ServerlessExpeditions

[banzai]

Cool! I’m glad I found this channel. ❤️👏

[scorpdevnull]

WoW, that's fantastic. I was waiting for the time!!!!

[zhalie]

Awesome video GCT Team!!!
Really like this type of video that simple and straightforward. Keep it up 👍👍

[dzivojelic]

Now we need a video how to build debug and run those jobs locally

[eklok]

Cool Video! But why wouldn’t I use cloud function for that? Isn’t there a huge product Feature overlap? What’s the adv of using this

[SidneiBecker]

Nice!
I have a question..
Thinking about non-concurrent processing, like large database processing, which can't have two jobs running at the same time, is there a way in Google Cloud to have a queue of jobs, which executes my "new" job only when the "old" job finishes?

[dgucyzx]

Some questions i wish were addressed. What occurs if the schedule calls the job while its still running (assuming somehow the timeout is greater than the schedule occurrence). It would also be cool if you showed the logging in how it looks on a job running successfully vs not. How does the job pass in the index for parallel operations ? Is there a way to stop the schedule from triggering if the previous failed. Such as a pause ?

[sandeepkulkarni]

Would be great to see App engine covered in both of its types...standard and flexible.

[mjad]

Hi, I am part of Google Cloud for Startup. I rely heavily on Cloud Run and Cloud Run Jobs. The only thing I don't like is I can't deploy Cloud Run Jobs just like Cloud Run. Can you add Cloud Build triggers that can deploy to Cloud Run Jobs?

[swapnilbodhe]

Query - if it's going to run for max 24 hours & parallel as well, what's the difference cloud run & dataflow. And in which scenario, would you use which service. Thank you

[davidspiess]

I would like to trigger a long running sync job by clicking a button on the frontend. Would it be possible to create and execute a cloud run job from my API and pass the job some parameters?

[adoniskomplex]

Would be nice to have long-running jobs for cloud run services, too.

[vinoprime]

Cool, I have a doubt, how to use containerized jobs?
How to run a cloud run from a private docker image repository.

[mohamedabdelkader]

Hello can this job be triggered from applications with passed parameters ?

[uhstube]

I have 2 questions based on my current understanding. Let me know if there is disconnect or reply on my query.
1)cloud run job is not on http protocol but cloud scheduler is able to invoke via http protocol using run.googleapi?
2)for cloud scheduler [attempt-deadline] param max value is 30 min for http. So for cloud run job which run more than 30 min, cloud scheduler will treat as timeout & mark it as failed? What will happen to cloud run job process in this case?

[mohamedshafvan]

need more topics about multi=cluster management in k8s.

[Melgazar]

Hey I used cloud run for a long running ELT process, but the job kept getting killed after one hour. I read that cloud run has a max request limit of 1hr. Am i doing something wrong? Is there another service i should use? I only get a 60 minute request timeout.

[JZL]

But for batch jobs, no preemtibility/spot is a huge cost hit, often for no reason. If you're doing big batches, repeating every few for such a huge cost savings is huge.

[zgnlgho]

Can I make sure that only one instance of a job is running? I have a python script (it's a Telegram client), so in NO CIRCUMSTANCE i can allow two processes to be at the same time. Otherwise Telegram bans my Session ID, and I have to re-log and redeploy. That's what happened today when a trigger started a new execution while the old one was still running. Ideally, I want to have some number of retries (in case it fails), and a trigger in the morning, but so that the trigger would kill the previous execution. Is Cloud Run Jobs a good choice at all?