Dealing With Big Data - Computerphile

Big Data sounds like a buzz word, and is hard to quantify, but the problems with large data sets are very real. Dr Isaac Triguero explains some of the challenges.


This video was filmed and edited by Sean Riley.

[griof]

Developer: we use a 3Gb database to plot some dashboards with statistical information about the customer's behavior

Marketing team: we use big data, machine learning and artificial intelligence to analyze and predict customer's action at any given time.

[RealityIsNot]

The problem with the word big data is that it went from a technical jargon to a marketing one.. and marketing department don't care what the word means.. they create their own meaning 😀. Other examples include AI ML

[Jamiered]

It's interesting, because at my company we deal with petabytes of data. Yet, I'm not sure you could call that "big data", because it's not complex and it doesn't require multiple nodes to process.

[letsburn]

I never realised just how much information there was to store until I tried downloading half a decade of satellite images from a single satellite at a fairly low resolution. It was a quarter Terabyte per Channel and it was producing over a dozen channels.

Then I had to process it....

[kevinhayes]

"Big Data" is talked about everywhere now. Really great to hear an explanation of it's fundamentals.

[SlackWi]

i work in bioinformatics and i would totally agree that 'big data' is anything i have to run on our university cluster

[iammaxhailme]

I used to work in computational chemistry... I had to use large GPU-driven compute clusters to do my simulations, but I wouldn't call it big data. I'd call it "big calculator that crunches molecular dynamics for a week and then pops out a 2 mb results .txt file" lol

[mokopa]

"If you're using Windows, that's your own mistake" INSTANT LIKE + FAVORITE

[gubbin]

Would love to see some future videos on Apache Spark!

[sandraviknander]

Freaky! I had this exact need of data locality on our cluster for the first time in my work this week.

[leahshitindi]

We had three hours lecture with Isaac last month. It was very interesting

[evilsqirrel]

As someone who works more on the practical side of this field, it really is a huge problem to solve. I work with data sets where we feed in multiple terabytes per day, and making sure the infrastructure stays healthy is a huge undertaking. It's cool to see it broken down in a digestible manner like this.

[NoEggu]

@3:23 "...the digital universe was estimated to be 44 zeta-bytes", and half of that is adult videos.

[chsyank]

Interesting video. I worked on and designed big data building large databases for litigation in the early 1980... that was big at the time. Then a few years later creating big data for shopping analysis. The key is that big data is big for the years that you are working on it and not afterwards as storage and processing gets bigger and faster. I think that while analysis and reporting is important, (otherwise there is no value to the data) I do believe that designing and building proper ingestion and storage designs are as important. My two cents from over 30 years of building big data.

[nandafprado]

"If you are using windows that is your own mistake" ...well that is the hard truth for data scientists lol

[lightspiritblix]

I'm actually studying these concepts at college, this video could not have come at a more convenient time!

[GloriousSimplicity]

The industry is moving away from having long-term storage on compute nodes. Since data storage needs grow at a different rate than compute needs, the trend is to have a storage cluster and a compute cluster. This means that applications start a bit slower as the data must be transferred from the storage cluster to the compute cluster. However it allows for more efficient spending on commodity hardware.

[nikhilPUD]

In few years "Super big data."

[shiolei]

Awesome simple explanation and diagrams. Loved this breakdown!

[quanta]

Take a drink everytime they say data for the ultimate experience