Tai-Danae Bradley | Category Theory and Language Models | The Cartesian Cafe with Timothy Nguyen

Tai-Danae Bradley is a mathematician who received her Ph.D. in mathematics from the CUNY Graduate Center. She was formerly at Alphabet and is now at Sandbox AQ, a startup focused on combining machine learning and quantum physics. Tai-Danae is a visiting research professor of mathematics at The Master’s University and the executive director of the Math3ma Institute, where she hosts her popular blog on category theory. She is also a co-author of the textbook Topology: A Categorical Approach that presents basic topology from the modern perspective of category theory.

In this episode, we provide a compressed crash course in category theory. We provide definitions and plenty of basic examples for all the basic notions: objects, morphisms, categories, functors, natural transformations. We also discuss the first basic result in category theory which is the Yoneda Lemma. We conclude with a discussion of how Tai-Danae has used category-theoretic methods in her work on language modeling, in particular, in how the passing from syntax to semantics can be realized through category-theoretic notions.

#TaiDanaeBradley #maths #mathematics #categorytheory #language #machinelearning #naturallanguageprocessing #topology #TimothyNguyen

--------------------------
The Cartesian Cafe is the podcast in which an expert guest and Timothy Nguyen map out scientific and mathematical subjects in detail.
--------------------------

Timestamps:
00:00:00 : Introduction
00:03:07 : How did you get into category theory?
00:06:29 : Outline of podcast
00:09:21 : Motivating category theory
00:11:35 : Analogy: Object Oriented Programming
00:12:32 : Definition of category
00:18:50 : Example: Category of sets
00:20:17 : Example: Matrix category
00:25:45 : Example: Preordered set (poset) is a category
00:33:43 : Example: Category of finite-dimensional vector spaces
00:37:46 : Forgetful functor
00:39:15 : Fruity example of forgetful functor: Forget race, gender, we're all part of humanity!
00:40:06 : Definition of functor
00:42:01 : Example: API change between programming languages is a functor
00:44:23 : Example: Groups, group homomorphisms are categories and functors
00:47:33 : Resume definition of functor
00:49:14 : Example: Functor between poset categories = order-preserving function
00:52:28 : Hom Functors. Things are getting meta (no not the tech company)
00:57:27 : Category theory is beautiful because of its rigidity
01:00:54 : Contravariant functor
01:03:23 : Definition: Presheaf
01:04:04 : Why are things meta? Arrows, arrows between arrows, categories of categories, ad infinitum.
01:07:38 : Probing a space with maps (prelude to Yoneda Lemma)
01:12:10 : Algebraic topology motivated category theory
01:15:44 : Definition: Natural transformation
01:19:21 : Example: Indexing category
01:21:54 : Example: Change of currency as natural transformation
01:25:35 : Isomorphism and natural isomorphism
01:27:34 : Notion of isomorphism in different categories
01:30:00 : Yoneda Lemma
01:33:46 : Example for Yoneda Lemma: Identity functor and evaluation natural transformation
01:42:33 : Analogy between Yoneda Lemma and linear algebra
01:46:06 : Corollary of Yoneda Lemma: Isomorphism of objects = Isomorphism of hom functors
01:50:40 : Yoneda embedding is fully faithful. Reasoning about this.
01:55:15 : Language Category
02:03:10 : Tai-Danae's paper: "An enriched category theory of language: from syntax to semantics"
02:04:25 : Motivation from large language models and machine learning
02:07:44 : Language modeling: Conditional probability of next word
02:09:21 : Syntax, statistics, semantics
02:18:28 : Yoneda embedding adds more structure (limits and colimits)
02:20:42 : Products and coproducts give logical or semantical "and" and "or"
02:24:01 : Topos
02:24:25 : Wrap up

Further Reading:

Twitter:
@iamtimnguyen

Webpage:

[horothesun]

This is pure gold! Thank you very much for sharing this amazing content.
P.S.: I'm a software engineer in love with pragmatic functional programming.

[effy]

as a programmer who likes abstract math, i enjoy this conversation with interviewer who is expertise at category and the host who could explain the topic by using a metaphor of interface in programming.

[paxdriver]

Great job you two, thank you very very much for the thorough whiteboarding it helps a ton.

[ua]

This formalization seems oddly similar to initializing some ensemble on Fock space. Given Tai-Danae's work on generalizing entropy, I wonder if she's already made the connection (Edit: Yes she has, glad to know people are working on this!). I've seen a paper by Wang et al use Hilbert spaces for semantics and a lot of other work using category theory for semantics, but this was such a clean approach. I look forward to seeing her future applied category theory work (although most upper ontologies seem to run into the same problems these days). Thank you so much!

[alexandersanchez]

46:00 You guys left out the best part of this example; it turns out that the action of a [group, G] on a [[set]] (aka a G-set) is just a functor from the [group] regarded as a category, as in the example, to the category of [[set]]s! In particular, we can generalize this by changing the domain and codomain categories. In that sense, a functor is just "the action of a [category] on a [[category]]" (regarded as an object in CAT, the category of [[categorie]]s). Indeed, functors generalize the notion of group actions to an awesome extent, and allow/formalize profoundly concise and canonical-feeling definitions, like "a vector space is a field acting on an abelian group."

[jsmdnq]

The power of category theory is that it couples with on those objects. The morphisms are "first-class citizens" and, in some sense, are actually more important than objects(mainly because the morphisms must specify the objects so we don't need to be track objects explicitly). In any case, set theory only has objects and morphisms are something "outside" of sets. Recursion is, ultimately, what makes category theory powerful. With sets, we can have sets of sets(elements can be sets) but there is not a lot of power in this since we can't track, in the recursive process, how all those subsets of ... of subsets relate. Category theory solves this problem by "attaching" morphisms to the subjects so that when we recurse we can keep track of the morphisms. The power of category theory isn't so much the objects and morphisms... it's the fact that objects can be categories and so when we have categories of categories we have, in some sense morphisms of morphisms.

By having such a structure that is very simple and so unconstrained it can represent a fast number of structures. By working on the recursive nature we can move to macro and micro views of that structure. So, in some sense, it gives us a sort of microscope in to structure allowing us to zoom in and out of different levels of the structure. Category theory establishes common patterns that show up in structure at different levels so that we have common features to recognize across many different scales of complexity.

It is important that one realize, though, the only way to truly understand category theory is to learn it, and that is true of all things. One learns it and then has it as a tool. It is pointless to just know it is exists and have some extremely vague idea of what it might be. One should simply start learning it even if it is done very slowly and takes years so that one eventually gets the benefit of being able to use it to understand the world around them more easily.

[mementomori]

Love the conversation. Do more of these please!

[CyberneticOrganism]

That connection of NLP to a category is excellent. It shows where LLMs live in. I posited a similar categorification of LLMs based on symbolic logic. My morphisms are logical inferences. The composition of words / symbols is via (Cartesian) products in my case, which is also true in your case. But our mophisms are different ☺️

[DestroManiak]

I wish the example at 20:00 was clearer. I have no idea how we use a 3x2 matrix on a single integer and get 3 as a result. Seems like nonsense. I guess we are working not in row or column space, but rather "how many rows there are"-space.

[MichaelJonn]

So the way I’m understanding what category theory is, is sort of the inversion of how traditional mathematics is done. Whereas traditional mathematics is about first having these math objects, and then figuring out the relationships between them, and how they are connected, like the langlands program, category theory seems to be about first giving you the relations and connections, and asking you to figure out what types of math objects would actually fit between the relations.

[lfossati]

Cool stuff! I wish there was a video like this when I did my PhD. At the time, MacLane’s book was pretty much my only resource (which is great, but you have to learn a whole bunch of other stuff first). QUESTION: Has there been any work done on quotienting L, to get a category whose objects are lexical categories (nouns, verbs, adjectives, etc…)?

[Smoothspin]

@24:30, you state that identity morphism is the (an ?) identity matrix, but i would rather say that identity morphism is a square matrix, no ?

[salashtolan]

currency exchange as a natural transformation is super-cool! It only works as an analogy though and it is not an actual natural transformation per the setup. In fact, any functor between C and D would by default do a currency exhange (i.e maps dollar-arrows in C, to euro--arrows in D). Different exchange rates would be examples of functors between C and D (assuming composition is summing) . And even if you thought of C as a category of people and transcations in a third currency/undefined units, you couldnt have D as a category with both euro-arrows and dollar-arrows in it because then composing arrows wouldnt make sense (assuming composition in your setup of C and D was summing/any associative arithmetic operation with identity); and for the analogy to be an actual example the former is a requirement.

[ryan_c_letsgo]

wow this was so good and very inspiring!!

[thegrey]

This is nice, please what is the name of the software you used for this presentation?

[blackestbill]

to me, category theory is the study of pattern based structures. The patterns that describe these structures consist of the patterns described by the emergence of numbers in set theory and all their potential combinations. This describes all math. Math describes all existence. Everything within the domain of existence can be modeled by math and must have been created according to the hierarchical patterns that emerge from numbers and their potential

[harakara]

Thank you guys! Wonderfull information

[djgreyjoy]

Does the example used to illustrate natural transformations really work? Monetary transactions don't compose, do they? If a transaction exists from x to y and another from y to z, does that mean one must exist from x to z? Moreover, does the definition of the categories C and D having objects that are people and $, and people and euros, respectively, make any sense? How do the functors F and G map people and dollars to people and euros differently? Wouldn't this be the currency conversion altogether?

[tomctutor]

I cant see anything in Category Theory that's not already covered by Group Theory? Please enlighten me someone if I am wrong!
PS: I am aware of Category called Grp😗

[axeljacobsen]

Great stuff! Is this podcast on apple podcasts?