Data-driven vs hypothesis-driven science (from Livestream #11)

Clip taken from DarkHorse Podcast Livestream #11 (originally streamed live on July 21, 2020):

Q&A:

Become a member of the DarkHorse LiveStreams, and get access to an additional Q&A livestream every month. Join at Heather's Patreon.

[anesig]

Being a data scientist, I always get goosebumps when I hear "data-driven". In the business world, the term often serves as a wild card for "you should accept what I'm saying, I've analyzed the data".
Data and logic are completely different things.

[christophersurnname]

Wow this is seriously good stuff man! Thank you

[tiaanfourie]

Wonderful explanation. You both have assisted me in my understanding of science and my lecturing to students the past 2 years. Thank you.

[maxprivus]

These concerns are becoming increasingly relevant as the data driven approach is increasingly favored in public policy circles. This is especially true with the expanding definition of public health and the intense focus on equity and so disparities research. If you ever want to present the issues you've highlighted in a manner that's accessible to lay people, and won't be politically controversial, consider interviewing someone with expertise in computer vision & machine learning.

This area addresses a range of applications that will be familiar to people,
- provides visual examples of inputs and outcomes that will be appreciable,
- employs many of the same techniques that data scientists do,
- utilizes both a priori and generated hypotheses
- demonstrates the risks of over-fitting models and data and how easily specious relationships can be discovered

Basically the development lifecycle in this domain is similar in relevant ways to a data driven research lifecycle, and so is a good metaphor for data driven research that can be presented in a way that people can understand IMO. Lex Fridman would be a good person to talk to about that.

[manfredduley]

Look to Kurt Godel's incompleteness theorems for the necessity of non-algorithmic input.

There is an argument of diminishing returns in a real world application and fleshing out the "provable" space by algorithmic means; but it is something that needs to be addressed in exclusive big-data research.

[djolds]

Curve fitting. The term you were looking for is curve fitting.

[theludovicotechnique]

"Therefore what you get out of such queries is a hypothesis."

Sounds like a classical case of confusing inputs with outputs.

[primetimedurkheim]

"I like your data but watch me smash your computer, dork."

[charlesjohnson]

An algorithm is just a human thought process rendered in code, or even physically manefested (digital versus analog) with gears for that matter, so that the process can be recalled and executed on demand. So, when you say that a method can be automated, you're just saying that somebody's idea can be executed without a person's physical presence with every execution, not that a machine is doing anything independently.

[someguy]

We should be careful about the distinction but I think both are very useful tools. Data is data; it doesn't lie. It is however subject to interpretation, and equally importantly it only reflects the conditions that generated it. This is a strength and a weakness. Many housing market models used in 2007 were unable to make predictions in novel conditions. On the flip side, in medicine, we sometimes rely too much on models rather than data (proxies or markers vs end results) and the models can turn out to be flawed.

Doing proper science is expensive. The gains may sometimes offset the risks if we shortcut it, but we should be clear that we're doing so and try to backfill some rigor over time.

[pcfreakx]

Just wanted to say how grateful I am that Heather and Bret pour out their big brains for me to learn great insights. I'll miss you both when the woke mobs round you up into a struggle session and crucify you in public. You'll have my support even then . . . I mean private support of course. I have a young kid - I hope you'll understand me being a coward ;)

[genkimachina]

I like to get into debates with a friend of mine who is a bit more of a futurist than myself and thinks that machine-learning is really going to push scientific progress at a faster and faster pace. I have a more pessimistic view because I don't believe that data, in and of itself, produces discovery. As an educator I see more and more of a push to be data-driven. This results in a lot more spread sheets, but little actual progress with students. Any educator who wants the appearance of progress can simply manipulate the data in easy ways to give the appearance that higher learning is occurring.

[bohrora]

Data driven science and science driven by data are two different things. They way toy describe it, it doesn’t exist. This is sophistry talk for housewives at best.

[jamesarthur]

Science is only as good as the person doing it.

[marshacd]

"The Scientific Method"??? I detest this simplistic litany. There are many methods used in doing various parts of science, many more or less standard, some novel. I wonder what Eric has to say about "The Scientific Method"? At about 7 minutes you touch on this, on the ART and artistry in doing science.

[axel]

Who is the intended audience? This is all very basic stuff. And a bit meandering. People who need to listen, I don't think they would like the very "academic" format

[nealfffj]

Is that a strange looking toupe or what?