ASPLOS '21 | Rhythmic Pixel Regions | Lightning Talk

Rhythmic Pixel Regions: Multi-resolution Visual Sensing System towards High-Precision Visual Computing at Low Power
Architectural Support for Programming Languages and Operating Systems 2021 (ASPLOS '21)

Authors:
Venkatesh Kodukula
Alexander Shearer
Van Nguyen
Srinivas Lingutla
Yifei Liu
Robert LiKamWa

Abstract:
High spatiotemporal resolution can offer high precision for vision
applications, which is particularly useful to capture the nuances of
visual features, such as for augmented reality. Unfortunately, capturing and processing high spatiotemporal visual frames generates
energy-expensive memory traffic. On the other hand, low resolution frames can reduce pixel memory throughput, but reduce also
the opportunities of high-precision visual sensing. However, our
intuition is that not all parts of the scene need to be captured at a
uniform resolution. Selectively and opportunistically reducing resolution for different regions of image frames can yield high-precision
visual computing at energy-efficient memory data rates.
To this end, we develop a visual sensing pipeline architecture
that flexibly allows application developers to dynamically adapt
the spatial resolution and update rate of different “rhythmic pixel
regions” in the scene. We develop a system that ingests pixel streams
from commercial image sensors with their standard raster-scan
pixel read-out patterns, but only encodes relevant pixels prior to
storing them in the memory. We also present streaming hardware
to decode the stored rhythmic pixel region stream into traditional
frame-based representations to feed into standard computer vision
algorithms. We integrate our encoding and decoding hardware
modules into existing video pipelines. On top of this, we develop
runtime support allowing developers to flexibly specify the region
labels. Evaluating our system on a Xilinx FPGA platform over three
vision workloads shows 43 − 64% reduction in interface traffic and
memory footprint, while providing controllable task accuracy.