Big Data Problems: Crash Course Statistics #39

There is a lot of excitement around the field of Big Data, but today we want to take a moment to look at some of the problems it creates. From questions of bias and transparency to privacy and security concerns, there is still a lot to be done to manage these problems as Big Data plays a bigger role in our lives.

Special thanks to Dr. Sameer Singh, the University of Washington, and the University of California Irvine for the content provided in this video.

Thanks to the following Patrons for their generous monthly contributions that help keep Crash Course free for everyone forever:

Sam Buck, Mark Brouwer, James Hughes, Kenneth F Penttinen, Trevin Beattie, Satya Ridhima Parvathaneni, Erika & Alexa Saur, Glenn Elliott, Justin Zingsheim, Jessica Wode, Eric Prestemon, Kathrin Benoit, Tom Trval, Jason Saslow, Nathan Taylor, Brian Thomas Gossett, Khaled El Shalakany, Indika Siriwardena, SR Foxley, Sam Ferguson, Yasenia Cruz, Eric Koslow, Caleb Weeks, Tim Curwick, D.A. Noe, Shawn Arnold, Malcolm Callis, Advait Shinde, William McGraw, Andrei Krishkevich, Rachel Bright, Mayumi Maeda, Kathy & Tim Philip, Jirat, Ian Dundore
--

Want to find Crash Course elsewhere on the internet?

[ruchaabhyankar]

Very informative! Real life examples make your videos more interesting.

[MichaelTrystorm]

My wife is getting her PHD in BIg Data:3 Im so proud of her

[xeroxyde]

That's why curating data is an important part of statistical analysis.

[theraphachukwu]

I didn't know how much I needed to here this until right now... thank you so much. It'll definitely give my research more meat

[ValerieJeanLove]

You guys are really helping me understanding different subjects for my media-exam. Will defiantly use your videos for my other subjects as well. Thank you!

[mustafaparekh]

I am so sorry Adriene Hill. I might seem a little ungrateful but your enthusiasm on the Economics crash course was so contagious. It really made me interested in Economics and made the videos so much fun. Really loved them! These are great as well!

[Zikiss]

Started statistics chapter this week, currently on correlation.

[ZijnShayatanica]

One of the biggest issues in big data is outsourcing algorithms, which gives companies plausible deniability when the program does something that could be considered unfair with the data.

[mightykimo]

This video deserves much much more views..

[avi]

0:25 Well, depends on the algorithm maker, really
Some developers are more experienced than others
Some developers are more creative than others in their solution-solving ability
So really, there is no "perfect" algorithm, only "what algorithm will best manage this set of data"
And I'm talking as a software developer myself

[RangerRuby]

I'm not taking Statistics but I still enjoy watching their videos...Anyone else?

[saijeyaprakash]

This video actually helped me with AP computer science principles... Thnx a lot!

[mpilosov]

I would say that using an algorithm as black-box is extremely problematic ... 5:00 ... I constantly see people throwing a dozen algorithms at a problem before stepping back to think about the problem they are trying to solve first.
Study MATH instead of Stats or CS and you'll be better equipped to assess what's happening between input and output.

[avi]

The thing is, lots of big companies prefer to not deal with the security themselves, but rather hand it over to cyber security company, by using their products
Because, it's easier to focus on one thing - the data, than on the security as well
It's like, for example, to use Adobe Photoshop grandma and advanced image editor, to edit your images, instead of a program like Paint to edit individual pixels, because Photoshop has lots of tools to edit the pixels more efficiently, more realistically
So, the company would generally prefer to ‏hire an expert Photoshop designer, then tell an employee to use paint to edit individual pixels to create the complete image - and if we're talking about a 1920×1080 image, we're talking about more than 2 million pixels (just multiply the numbers) - a work that wo6 take ages. ‏That's why Adobe created Photoshop

[lodgin]

I had to re-open this tab and re-listen to the credits... the Chad and Stacy? HAH

[jeremiasrobinson]

I often hear conspiracy theorist types talking about how scary Big Data is and yet, it seems when Google, Netflix, Youtube, or Facebook try to suggest things to me based off my interests, they get it wrong more than right. I'll be more scared of Big Data being all knowing and tracking my every move when it can figure out what videos I actually want to watch on Youtube.

[shane]

I think I was being tracked but after examining the information gathered a determination was made that I’m boring AF😞

[rkpetry]

*_...there's an interesting perspective on university science, that should call attention to the persistent belief than science can be advanced pointwise, instead of uniformly: it's called "functional analysis" where the researcher assigns data values to a metric field basis e.g. EMF(x, y, z, t), whereas 'reality→reality' outputs the same as the inputs, to be understood..._*

[Unhacker]

Best episode in series for ML Transparency nuts. But you /should/ watch them all. DFTBAQ! ;)

[brucewilson]

I noticed some slippery speech here. Was compass wrong about different races offending at different rates ? It wasn't clear if compass was 200% actual rates or if black defendants actually were offending at twice the rate.