A Brief History of Data by Tim Berglund

Data professionals and software developers of every stripe are moving rapidly into the world of real time, streaming data. There's no question that this is our future, and that its promises are already being realized, but how did we get here? Is real-time really the natural evolution of batch processing? When did we start thinking of what we call "data" itself, and has has that influenced the way we build systems how have our data management tools evolved from the earliest days?We'll start with a frustrated Belgian astronomer who went on a quest to compute the statistics of the "average man." Then we'll follow automated computing technology as it proceeds from the Difference Engine to tabulating machines to the first digital computers to mainframes to data warehouses to Hadoop to Kafka to Apache Pinot. As we tell the stories of the main technologies along the way, we'll look at how developments in data storage, information management, and computation all conspired to give us not just just the data stack we have now, but the very ideas we use to think about data, put it to use, and move its associated technologies forward.