filmov
tv
Using pandas and pyspark to address challenges in processing and storing time series instrument data
Показать описание
Aaron Wiegel
Time series data from scientific instruments for fermentation, environmental sensors, or spectroscopy often comes in proprietary or unusual formats that are require custom logic to process. In addition, processing data at scale is challenge since enterprise laboratory information management systems (LIMS) typically rely on transactional, row-oriented databases that are not designed to handle millions of records at a time. However, with clever use of pandas for unusually formatted files or pyspark (via Databricks) for large numbers of records, this data can be processed into cleaner, more useful forms for further analysis.
PyBay features the most influential speakers presenting the most crucial technologies to help beginners and seasoned developers alike get up-to-date quickly, in a single-track format. Whether you’re interested in web technologies, data, devops, Python internals, or performance, PyBay will help you stay on top of your game AND network with engineers at companies that are hiring!
Working remotely and want to meet your teammates to boost team cohesiveness? Leverage the platform we’ve built. There are great talks, yummy food, fresh air, vitamin D... all the elements developers crave for these days. If there are talks that don’t interest your team, take the opportunity to talk to speakers, create your own team activities or book a tee-time at the adjacent miniature golf course!
PyBay is the regional Python conference for the San Francisco Bay Area, bringing together Pythonistas from around the Bay Area and beyond. It is a volunteer-run organization dedicated to building a stronger Python community. PyBay offers deep-dive talks and networking opportunities that aim to enrich and empower the Python community. PyBay is part of BAPyA (Bay Area Python Association). BAPyA member organizations are the SF Python, Pyninsula, and BayPIGgies meetups.
Sun Oct 8 15:45:00 2023 at Bungalo West
Time series data from scientific instruments for fermentation, environmental sensors, or spectroscopy often comes in proprietary or unusual formats that are require custom logic to process. In addition, processing data at scale is challenge since enterprise laboratory information management systems (LIMS) typically rely on transactional, row-oriented databases that are not designed to handle millions of records at a time. However, with clever use of pandas for unusually formatted files or pyspark (via Databricks) for large numbers of records, this data can be processed into cleaner, more useful forms for further analysis.
PyBay features the most influential speakers presenting the most crucial technologies to help beginners and seasoned developers alike get up-to-date quickly, in a single-track format. Whether you’re interested in web technologies, data, devops, Python internals, or performance, PyBay will help you stay on top of your game AND network with engineers at companies that are hiring!
Working remotely and want to meet your teammates to boost team cohesiveness? Leverage the platform we’ve built. There are great talks, yummy food, fresh air, vitamin D... all the elements developers crave for these days. If there are talks that don’t interest your team, take the opportunity to talk to speakers, create your own team activities or book a tee-time at the adjacent miniature golf course!
PyBay is the regional Python conference for the San Francisco Bay Area, bringing together Pythonistas from around the Bay Area and beyond. It is a volunteer-run organization dedicated to building a stronger Python community. PyBay offers deep-dive talks and networking opportunities that aim to enrich and empower the Python community. PyBay is part of BAPyA (Bay Area Python Association). BAPyA member organizations are the SF Python, Pyninsula, and BayPIGgies meetups.
Sun Oct 8 15:45:00 2023 at Bungalo West
Комментарии