Dagger: The Missing Ingredient for Your Disastrous CI/CD Pipeline

In this video, we will take a look at some of the common mistakes that people make when building CI/CD pipelines, and we will show you how Dagger can help you to avoid these mistakes.

Dagger is a set of libraries enable us to write CI (not CI/CD) pipelines in a variety of languages (NodeJS, Python, Elixir, etc.), that can run anywhere (locally, remotely, in other pipeline tools), and that is based on Docker or other container runtimes.

It replaces many of the tasks we normally write in Jenkins, GitHub Actions, Argo Workflows, Tekton, CircleCI, and other remote pipeline solutions.

#CICD #Dagger #SDLC

▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬

▬▬▬▬▬▬ 💰 Sponsoships 💰 ▬▬▬▬▬▬

▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬

▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬

▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬
00:00 introduction
02:02 CI Pipelines Should NOT Be Remote Only
05:56 Pipelines Should NOT Do CI Orchestration
09:20 Pipelines Should NOT Be Declarative
14:50 Pipelines Should NOT Depend On Specific Tools
17:04 Dagger: CI/CD With Code That Runs Everywhere
27:28 Dagger Pros and Cons

[DevOpsToolkit]

What do you think of Dagger? Can it change how we do CI (NOT CD)?

[suporteking]

Viktor just sending a message here from a brazilian devops engineer to thank you so much for your contents.God bless you and your life!You are a good person to provide such a content for free to us, you are helping people on their jobs!Congrats...and please..keep doing your videos!

[trento]

I love this-- I've been looking into Dagger and I'm glad someone made a very digestible intro to Dagger!

[twistplay]

We utilize Taskfile, which is a modern equivalent of a makefile. We are delighted with its capabilities, as it covers nearly everything you mentioned in a format combining declarative YAML and Bash. Taskfile provides us with a unified interface for both local and remote CI.

[PierfrancescoZamana-jwng]

Once again, a fantastic video! I became enamored with Dagger right after watching it. However, I later discovered that it doesn't support rootless, and in my company, privileged containers are akin to shutting down a datacenter, haha.
It would be fantastic if Dagger could, in the future, utilize pods instead of containers, especially considering that all developers these days have a local Kubernetes cluster.

[coocoobau]

"Glorified cronjobs"... made my day. I've been trying to explain this to my fellow devs and devops for some years now :) And yes, we had the huge shell script that did orchestration and was just triggered by basic pipelines in a CI system that eventually became redundant.

[LeakMemory]

This sounds great in theory (and is super cool!), but 15s cycle time for no changes is too slow. Devs' inner loop cycle time is extremely precious, so while I totally agree it would be great if all those things ran locally, I wouldn't trade 15s to get it. I'm former Netflix DevEx and we initially built something like this based on the mistake of conflating what was good for Platform with what's actually good for app devs -- it's an easy mistake to make and I believe you might've partially fallen into the same trap. We ended up abandoning containers for localdev (except for some sidecar services they didn't touch directly) a few years ago and everyone was happier. Well, everyone except Platform who had to support it!

[paulogodinho]

This reminded me of Nuke, another tool to accomplish similar results. Great video!

[KsaweryKarwacki]

I work in java based environment and used the same principle, but instead I used gradle. Jenkins just uses "sh ./gradlew .." to trigger but whole logic is in gradle and custom plugins. It does not matter if your running on Windows laptop, WSL, virtual machine, Jenkins agent. It always works the same. Heck we even have plugin to bundle JS apps to have consistency. Additionally I do not need to force everyone to learn quirks of Jenkins because most of them are JVM devs so they can just debug gradle runtime on their machine.

[davemeech]

"Today I will continue complaining about CI/CD pipelines..." Already sold.

[MrTheGurra]

Very good points! Use declarative tooling when you are describing an end state. Use imperative tooling when you are describing a process. Declarative pipelines always felt counterintuitive to me

[esantix]

Declarative and imperative are not that distinct concepts. Declarative is not necessarily describing a state. It simply a way of "not describing how". Meanwhile imperative does describe the how. But declarative can define a set of insctructions and not necessarily a state. So YAML can represent a set of instructions to run and not a state.

I agree on the discussion about whether a pipeline should be defined by state or by instructions. But its not about declarative or imperative from my point of view.

[dmitriinadein]

Виктор, ты бесспорно профессионал, но тут я не соглашусь - важно не забывать про простоту - KISS метод. Представь, что ты накодил все круто на Go и уволился, а потом компания долго пытается найти девопса который в этом сможет разобраться. И очень повезет, если есть документация! Отчасти поэтому стали везде делать декларативное описание и лично мне это нравится.

[scottamolinari]

I thought I watched everything, but I seemed to miss a note about Dagger modules i.e. code reuse for the win! The even cooler thing is, someone who needs to use a module in Go, and they are using Dagger with TypeScript, can still call on the Go module to use it. How awesome is that?

[mrcaique]

Not sure if i read it correctly, obviously we can run github pipelines locally. The same is with Jenkins, they provide slim runner for that. So in that aspect both solutions are quite similar

[CostisPan]

Thanks for the video.

Any idea which software Victor uses to create these lovely animated diagrams?

[ZiggleFingers]

dagger seems like an attempt to reimplement a very small part of Nix when they don't understand (or they don't want to fill you in on) the full problem. each step of daggers development will come to the same conclusion Nix has to move it further towards reproducible builds. they will find they need a nixpkgs equivalent (every* software package defined with inputs and outputs) to be able to build their DAG (which Nix has done for decades). then find that it makes sense to use a language to simply define inputs and outputs (hmm Nix?). Why ship a monolithic 1GB container when Nix can build a DAG based container (because it knows each *files* inputs) with upto 128 layers? If all your containers are Nix built, they will share most of the same layers because they're ALL reproduceable from the same inputs! Need to swap out one 500KB shared dependency on 5 pods that run on a 5000 node K8S? One 500KB layer pull - theres no special logic to define this in your bespoke pipeline. Nix knows this.

[iamunknownperiod]

I agree with your point on imperative CI but there's something about seeing the actual code, calling shell via go instead of just writing it in just shell seems too cumbersome. It might be very flexible but at the same time, it seems like not very straightforward and would require a longer time to start when compared to writing it in just shell scripts or lines in a YAML file. YAML is declarative but maybe there is merit on using it for pipelines because of its human-readable nature. I guess it will depend on which you value more.

[ZiggleFingers]

What if I told you there is another tool that does everything dagger does but doesn't require using containers. Outputs containers, ami's, kubevirt images, iso's, hyperv, gce, azure, virtualbox, qemu and more. Built-in distributed building, hot rebuild, sandboxed dev environments with all tooling included. Can be run equally locally or remote with one command.

[dgkaba]

Hello, Nice video again thanks, great case on declarative. would you wrap dagger task into cicd task so u could benefit parallelism for example ?