Neural Networks are Decision Trees (w/ Alexander Mattick)

#neuralnetworks #machinelearning #ai

Alexander Mattick joins me to discuss the paper "Neural Networks are Decision Trees", which has generated a lot of hype on social media. We ask the question: Has this paper solved one of the large mysteries of deep learning and opened the black-box neural networks up to interpretability?

OUTLINE:
0:00 - Introduction
2:20 - Aren't Neural Networks non-linear?
5:20 - What does it all mean?
8:00 - How large do these trees get?
11:50 - Decision Trees vs Neural Networks
17:15 - Is this paper new?
22:20 - Experimental results
27:30 - Can Trees and Networks work together?

Abstract:
In this manuscript, we show that any feedforward neural network having piece-wise linear activation functions can be represented as a decision tree. The representation is equivalence and not an approximation, thus keeping the accuracy of the neural network exactly as is. We believe that this work paves the way to tackle the black-box nature of neural networks. We share equivalent trees of some neural networks and show that besides providing interpretability, tree representation can also achieve some computational advantages. The analysis holds both for fully connected and convolutional networks, which may or may not also include skip connections and/or normalizations.

Author: Caglar Aytekin

Links:

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n

[YannicKilcher]

OUTLINE:
0:00 - Introduction
2:20 - Aren't Neural Networks non-linear?
5:20 - What does it all mean?
8:00 - How large do these trees get?
11:50 - Decision Trees vs Neural Networks
17:15 - Is this paper new?
22:20 - Experimental results
27:30 - Can Trees and Networks work together?

[sunchax]

Thanks for an interesting video, just wanted to say that I am thoroughly impressed by Alexander Mattick, sounds like he has a deeper understanding of the subject than many professors and professionals out there and is still a student. Impressive.

[ragnarherron]

Absolutely excellent. Please provide more of this!!!

[fiNitEarth]

Would be interesting to see a graph of all the "x are y" papers and conclude from that xD

[MarcosScheeren]

recent sub here. just wanted to say thanks for the info your're bringing, hard to find channels that dig sorta deep into the new papers without being boring.

[sean_vikoren]

Nice to meet you Alexander.
Your enthusiasm kept my heart beating fast the whole show! 😆

[brll]

Problem is they are unlabeled decision trees.
Doesn't really help with understanding when you now have millions of nodes and edges and no idea what they mean.

[nintishia]

Thanks for this wonderfully enlightening conversation. It exposes so much more than the content of the paper itself. Look forward to more such discussions in the future.

[nitinbommi]

I HAVE BEEN WAITING FOR THIS PAPER! THANK YOU

[ryanbaten]

Yes this is known. It's worth pointing out that part of the claim that decision trees are interpretable comes from the fact that the training algorithm for them uses axis aligned splits. This is in contrast to the nn case which typically has no such constraint in its training algorithm. In the NN case looking at a single FF layer, what does being in some region of space defined by what sides it lies on a bunch of non-axis aligned hyperplanes mean? Other than similarity to inputs that land in the same region, there's no simple "human" interpretation I can think of.

[albert-ggbd]

I have met Alexander in the discord and he is just amazingly intelligent and humble! But to be totally honest here, I would have preferred the typical easy to follow Yannick explanation style and video structure for the Paper Review. I found this video confusing.

[DRKSTRN]

Appreciate this one very much. Reaffirmed my path forward. Y'all have a good one

[Neomadra]

How to be a successful AI researcher: Rephrase something that was already well known and then give it a cool title.

[bautibunge]

This is a great video. It's a very interesting and fairly comprehensible discussion

[alenasenie]

I mean, yeah, small neural networks are known by what it is done, we know it is a decision tree, the fact we knew that doesn't mean they aren't black boxes, we don't know how a small change will affect the rest of the network. But this paper is extremely interesting for optimizations, it might help to reduce significantly the computation time or to find layers that are kinda stable for some functionality, meaning accelerated training time, this is extremely interesting.

[zerotwo]

Rational and Logical level of most interviewers: 💪
Their mic: 💥

[Kram]

13:20 Sounds like what you'd really want is a Decision A-Cyclic directed Graph (where equivalent decisions are collapsed, so going ->Small->Dark and going ->Dark->Small lands you at the same point)

[Xraid]

Great video. Alexander Mattick needs a mic made from this decade though. Maybe use some Machine learning from Nvidia's RTX voice to clean up the audio. =)

[johnkost]

It would be very useful to have computational equivalence to a neural network. It's almost as if we are headed in a direction where a large dataset uses training to shake out a 'rule-of-thumb' or generalization. Hmmm..

[samselvaraj]

I wanna be honest and say that I literally thought about this a month back!! I definitely am not in a position to write a research paper but I had abstractly thought about how decision trees and ANN are very similar in structure and how activations functions act as gini indices in Neural Networks.