Pulumi - Infrastructure as Code (IaC) Using Programming Languages

Pulumi allows us to manage infrastructure as code (IaC) using familiar programming languages. It is like Terraform, but without HCL.

#pulumi #infrastructure #iac

Timecodes ⏱:
00:00 Intro
01:40 Creating a new project
06:39 Creating a cluster
15:46 Final thoughts

[DevOpsToolkit]

At 17:10 I said that it's great that Pulumi stores state in Cloud. Terraform does that as well. I did not mention it since I thought that it is a paid feature. It's not. Terraform allows Cloud state storage for up to 5 users.

[dinoscheidt]

As a dev, I just did an evaluation of terraform etc and pulumi. And I see no reason for having one thousand and one yaml files or learn some proprietary DSL. 🤜 Nope. All this is (async) life cycle management. Programming languages are made for life cycle management and here I love that I can use the full power of my typescript IDE to create infrastructure in code that looks both familiar, is testable, is statically typed and allows to manage entities (resources) with all the best practices like the actually product is being build with. Very smart - 🧠 pulumi is so far great for me / us to eliminate the Ops from Dev*. 💡And btw: YAML vs Programming languages has nothing to do with imperative vs declarative programming. If you think really hard and write 100 lines of yaml config while I create a 3 line loop that instantiates 10 resources: its still a declarative output. Only that I encoded the imperative part into the code base while nobody else understands why there are 100 lines of yaml.

[taytzehao]

I am relatively new to the IAC space and Pulumi is my first IAC stack. Never used Terraform before. So, let me know if there are any arguments that you have. I beg to differ from your comments and I find it to be amazing. The following are my points:

1) Pulumi allows the use of SWE practices as it has the full power of programming language. It allows us to code less by manipulating minimal amount of configuration and reduces the amount of boilerplate code. For example, if we would want to create multiple VPC endpoints, we do not need to copy and paste the same declaration multiple times, we can just create an array of services that we want to VPC to be connecting eg ["s3", "dynamoDB", "rds"]. Just with this array, depending on the way the template/component resource is made, we can just loop through this array and create 3 endpoints without duplicating the code multiple times. This can be especially useful for services that come in huge quantities like IAM roles, security groups, SSM etc. There are quite many more ways that SWE practices can reduce amount of code significantly.

2) Automation. It can be used on multiple cloud providers and be used in inline programming language. That means we can say goodbye to things like boto3 or any cloud specific SDKs or language SDKs and instead Pulumi as the universal Infrastructure Provider. This helps to reduce the learning curve of using multiple tech stack significantly.

3) I have been using s3 buckets to store the state rather than using Pulumi Service and it works fine. This free method is a bit rough on the edges, but once the configuration for s3 is made. I think it works well or even better. The only downside is the lack of UI to visualise the resources.

4) It can also replace the feature of Cloudmapping by accessing the Outputs stored in the state, making variables to be very dynamic.

[idsvandermolen]

I haven’t got much experience with Pulumi, but I think k8s YAML and Terraform are not the complete picture.. When you want to define a k8s deployment and compare the syntax of YAML vs Pulumi (or even Terraform) the YAML wins for me. However, if you need to define multiple infrastructure components in various environments (dev, staging, qa, prod, ephemeral stacks) with variation, modularization and packaging I think YAML and Terraform start to break down.. That’s when it is probably a good idea to use a real programming language with it’s large ecosystem of tools instead of band-aids like kustomize, helm, jsonnet, ksonnet or terraform modules. Things like IDE support, linting, static analysis, packaging and (unit) testing tools for generic programming languages are much better.
Also with generic programming languages it will be easier to build higher-level abstractions.

[miran]

17:10 use terraform cloud - you can run terraform locally or remotely, that way only one team member (or ci) can do the changes at the given point in time. You can also load the remote state from one workspace in another, which can be very powerful. I have one workspace, which creates the gcp project and another which manages the k8s cluster using the data from the first workspace - because of that i don't need any env variables (credentials, locations, ...)

[eriklundevall]

I agree with your sentiment that it not any clear advantage with using a programming language to describe something that will be easier to read in YAML. After all, the main purpose of a programming language (and configuration languages) is to convey information to human readers.

The power of using programming languages to describe infrastructure etc comes when you are building better and more (re)usable abstractions on top of the lower level native interfaces. In that sense, Pulumi is only at the start of the journey towards this path, building a foundation mainly.

Compare that to AWS Cloud Development Kit (AWS CDK), which has come further in terms of building higher-level components on top of the low-level CloudFormation-based primitives. The AWS CDK uses a convention-over-configuration approach to set mostly sane defaults, which you can override if needed. It also has a pretty nice approach for IAM permissions, which makes it pretty easy to get the somewhat minimal permissions in place.
I think Pulumi could get to where AWS CDK is in that regard, but the challenge is that this has to be done for many different platforms, while AWS CDK has it much easier in that regard given that it only has to care about AWS.
The Terraform CDK (CDKTF) is also evolving and that will be interesting to see what happens there as well.

I see these tools as potentially better options than plain Terraform, CloudFormation, ARM to define re-usable components to facilitate their usage for multiple target audiences. But they need a critical mass of higher-level re-usable components.

[matscloud]

Great video, as always. Not sure about Pulumi, but we've been using AWS CDK for a while now, and it highly simplifies the relation between resources. For example, referencing a database when creating Fargate ECS creates a service role for the container to consume the database, and so on. Not sure how Pulumi handles all this, but I kinda hope it's something similar...

[luxiax]

I haven't used pulumi, but we usually combine terraform or cloduformation with jenkins to achive something similar, why? complex deployments and pipelines automation. we want our deployments to be click and wait, so we automate as much as we can, that means decision, config files, input gathering is done by code, and that needs a programming language, I can definately see polumi being useful for that.

[bogdanbliznuk]

1. Ansible is not declarative in any way. It is imperative procedural execution of an awful turning complete YAML language.
2. Pulumi is declarative. You were able to run destroy, without defining explicitly the steps, it builds and resolves the DAG for you. With Ansible, you'd explicitly run a destroy command for each resource in a particular sequence.
3. You can write code in a declarative style using ANY general programming language or functional language. Pure functions are declarative (source: Wikipedia)
4. You can write unit tests on your infra and utility functions (edit: this is actually mentioned)
5. Please use IDE next time, it will resolve references, add completion, highlight syntax.
6. It IS solving the problem of sharing knowledge and communication in the DevOps space. Awful DSLs separate ops vs dev even more. SW practices were developed and barely changed 60 years ago, including declarative programming. Awful tools and DSLs lead to bad SW practices.

[hirenkamat]

Is terraform or pulumi better for on demand IAC - i.e. creating infrastructure which is consistently changing/creating say on demand from user on pipeline runs? I tend to think pulumi will give an edge because of programming languages and the ease with which we can build logic. But would like to know your thoughts?

[CuongTranQ]

Could you do a video about AWS CDK and its various cousins (cdktf, cdk8s, etc)?

[joebowbeer]

Pulumi can consume terraform state and/or convert: tf2pulumi

[jodanxx]

I’m trying to understand why, if I wanted to define my infrastructure in a programming language, I would use the pulumi wrapper around the cloud provider API rather than just using the cloud provider APIs...

[ducktsmt]

The advantage and future of Pulumi is to enable converged programming in a unified code stack. It is not born for infra only, at all.

[zeenmc]

Maybe Pulumi has nice idea with using popular languages, but their documentation is very bad. Maybe is issue in me, who knows, but I am guessing is not. Terraform has much better documentation.