Harvard Presents NEW Knowledge-Graph AGENT (MedAI)

Harvard Unveils New Knowledge Graph Agent for improved AI in Medicine. Called KGARevion, it combines the knowledge from knowledge graphs with the knowledge of LLMs.

Since RAG suffers from inaccurate and incomplete retrieval problems in medicine, Harvard et al present a new and improved methodology to significantly increase the reasoning performance of medical AI systems. Special focus on complex medical human interactions.

New insights and new methods to combine the non-codified knowledge of LLMs with the structural codified knowledge of medical knowledge graphs.

Detailed explanation of the new methods in this AI research pre-print (also for beginners in AI).

All rights w/ authors:
KNOWLEDGE GRAPH BASED AGENT FOR COMPLEX,
KNOWLEDGE-INTENSIVE QA IN MEDICINE

00:00 Harvard has a problem w/ LLMs and RAG
04:20 Harvard Univ develops a new solution
07:24 The Generate Phase (medical triplets)
09:50 Review Phase of KGARevion
12:30 Multiple embeddings from LLM and Graphs
15:40 Alignment of all embeddings in common math space
20:48 Dynamic update of the Knowledge graph
21:52 Update LLM with grounded graph knowledge
23:15 Revise phase to correct incomplete triplets
25:20 Answer phase brings it all together
26:07 Summary
29:52 Performance analysis
33:39 All prompts for KGARevion in detail

#airesearch
#aiagents
#harvarduniversity

[Scientist]

The idea of projecting the LLM embeddings to the same geometric space as KG embeddings and then aligning them through optimization reminds me a LOT of how Llava works… it’s really a very similar idea, in Llava you project the image into the LLM space to learn the aspects of the image which increase classification performance when fine tuning on (image+text, completion) pairs

[johnkintree]

The concept of "operating principles" was mentioned repeatedly in the book, Building Knowledge Graphs, by Jesuś Barrasa and Jim Webber, published by O'Reilly in 2023.

Trustworthiness is an essential operating principle. Measures of trustworthiness, based on evidence and reason, can be stored as properties of entities and relationships within graphs.

The Graphiti project incorporates the temporal dimension into the graph representation.

[medseekai]

I agree with the premise that LLMs could be inadequate in terms of the medical knowledge required. Quality of training data is more important than quantity of data. However, the 3 test questions appear to be ones that an advanced physician/academic researcher in the world of - omics would ask, and not the common questions that most of the physicians would ask in the day to day practice of medicine.

[stephenterry]

I always enjoy the first half of the content, but always anticipate a more conclusive summary of findings.

[Nick_With_A_Stick]

THANK YOU ❤! I was skimming through deep-minds ICML paper over knowledge graphs usage in LLM’s and this helps understand it a ton!

[goodtothinkwith]

We need a fuzzy method for synthesizing and exploring knowledge graphs, somewhere in between discrete and creative association for retrieval

[sonnygmony]

Interesting. Semantic processing and "working memory" housed in the LLM, long term "crystallized" memory in the Knowledge Graph. Combined with other processing centers and RAG, could easily cross the singularity threshold.

[goodtothinkwith]

Does this imply that we need to have an open source knowledge graph for medicine that the entire world works to improve? New LLMs could then use it instead of having to reinvent the wheel each time. That’s terrific. Has this been applied to Google’s AIME?

[goodtothinkwith]

This work is more important than it may seem at first. It combines LLM creativity with discrete knowledge. There are a lot of fields that have such discrete knowledge. I do wonder how this may relate to AlphaGeometry and math theorem provers

[BorderAgent]

Seems to me to be a rather simple garbage in / garbage out problem. Just restrict model training to the highest quality sources, and only respond to queries within this knowledge or give a quality grade number for responses outside this training set.

[Matx]

In the current state of the art, you need to know your subject in order to question AI. It answers in layers of limited depth, to save resources. You have to dig to get the answer. It can be very useful in science, if you're aware of that. For it to have the value of a real expert, it needs exponential memory.

[warpdrive]

Hello Sir. Please make more videos where you are explaining research papers. You are doing God's work! Much love from India :)

[깐돌엄마-ge]

I love your Text animations in your video :)

[itmomotitimo]

I love the stereotypical "hey Siri, wie spät ist es denn" at the very end 😂

[hakimsenane]

Very clear presentation and interesting work. Thank you! One question: why using a non-directed graph? (of course for some relationships direction does not matter and may be represent those twice inverting tail and head, but for others it does and I wonder if this could explain performance.)

[andrejuha]

OK. But if you don't have information from llm in knowledge graph? You are where you been before.🙂And how you know, if its not failing mapping the knowledge between kg and llm?

[lesptitsoiseaux]

Do a follow up where you walk through doing it yourself

[MusingsAndIdeas]

So basically they created a method for synthesizing an ontology so that the LLM and the KG make sense to each other. This is hilarious

[EvanGoodwin-blzq]

It is very interesting but seems very limited. How many domains have the equivalent of the UMLS. If your q & a domain doesn't have something like this, you can't use this at all. I don't know if science or finance domains have something as robust as the UMLS. Plus the technique doesn't even get past 80% - which seems unacceptable for medicine or any other high stakes domain. It would be interesting to see specific question it got wrong and how it got.them wrong.

[KCMNJL]

I fell asleep listening to this video last which I have just woken from, in which my mind conjured up this mad dream about building a synthetic dataset comprising of LLM Prompts, Knowledgebase edges containing trust indicators and a bunch of other things which didn't make sense in the dream (it was largely symbolic as all dreams I'm just relating some of the symbolic ideas back to what I heard in the video prior to going Zzzzz. The whole concept of the dream though, was developing agents that were derived from EP (Evolutionary Programming), where better LLM prompts were derived over time for targeted problem spaces, guided by trust indicators.

Now I'm no ML so I haven't the foggiest if this makes any sense. But it was a cool dream :)