Vault '20 - Scaling Databases and File APIs with Programmable Ceph Object Storage

Scaling Databases and File APIs with Programmable Ceph Object Storage

Jeff LeFevre and Carlos Maltzahn, University of California, Santa Cruz

In this talk, we will present three of our latest developments on the SkyhookDM project since Vault '19. First, SkyhookDM can be used to also offload operations of access libraries that support plugins for backends, such as HDF5 and its Virtual Object Layer. Second, in addition to row-oriented data format using Google's Flatbuffers, we have added support for column-oriented data formats using the Apache Arrow library within our Ceph extensions. Third, we added dynamic switching between row and column data formats within Ceph objects, a first step towards physical design management in storage systems, similar to physical design tuning in database systems.