Script Layering for large-scale number crunching in Python

Catherine Moroney

Python's combination of number-crunching and plotting prowess, combined with scripting makes it easy to submit a massive number of scientific jobs in a fully-automated fashion. I often need to run hundreds or thousands of orbits through the processing software and by dividing the duty into multiple scripts, each calling the one below, I have a customizable and automated way of finding the input files, caching them to a local disk, creating the input files, submitting each orbit to our batch-processing system and monitoring their progress so the entire chain consisting of 10-15 executables can all be run in order with a single command.

Python's scripting capabilities, combined with its number-crunching utility make it very easy to gather up all the data, submit thousands of jobs and then monitor their progress, all with a single command. One script submits a single orbit, another one calls the first to run a month or year worth of data, and then a third calls the second to run the entire processing chain. Script layering!

A Python conference north of the Golden Gate


North Bay Python is a single-track conference with a carefully curated set of talks representing the diverse Python community and their different areas of interest.

If a topic is less to your interest, or you've met some people you really want to sit down and chat with, we'll have plenty of areas away from the main theatre to catch up and chat.

Our goal is to keep prices as low as possible. That means we won't be catering lunch. Instead, you can look forward to extra-long lunch breaks you can use to explore all of the great food options around the venue.

Sun Nov 3 11:25:29 2019 at Mystic Theatre