Serving and Managing Reproducible Conda Environments via Conda-store | PyData Global 2021

Serving and Managing Reproducible Conda Environments via Conda-store
Speaker: Chris Ostrouchov

Summary
Conda-Store is an open source tool being developed by Quansight to help manage, build, and serve Conda environments. It enables IT to provide controls over environments, users the flexibility of quickly updating environments, all while serving these environments in a reproducible manner in a pinned YAML, Conda lockfile, Conda-Pack tarball, and Docker images.

Description
End users think in terms of environments not packages. The core philosophy of conda-store is to serve reproducible conda environments in as many ways as possible to users. Conda-store was developed due to a significant need we found in enterprise and hpc architectures. There are many ways to serve environments and each plays an important role. Thus conda-store serves the same environment via a filesystem, lockfile, pinned yaml specification, conda pack archive, and docker image. This logic could easily be extended to also support the creation of VM iso's and singularity containers for HPC environments.

During this talk I will highlight some common problems with environments we have seen while consulting and show how conda-store aims to solve them: - Friction between system administrators and end users in controlled environments where new packages are needed - Enabling a given notebook developed within jupyterlab to be reproducibly run in workflows reliably for years to come - Helping to removing the need for specially crafted docker and singularity containers

This talk will be full of demos along with a site that everyone in the talk can try out. Additionally I will showcase initial work on a jupyterlab extension gator that allows for conda environment management.

Chris Ostrouchov's Bio
Developer passionate about numerical computing and making open source sustainable. Scientific Software Developer for QHub and Conda-Store. When I have spare time I enjoy running and contributing to reproducible software via nixpkgs and nixos.

PyData Global 2021

PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.

PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.

00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.

[SanataniAryavrat]

Thanks Chris. This was great learning.