Trade-off between world modeling (predicting) vs agent modeling (acting)

Показать описание

Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modeling

Support my learning journey either by clicking the Join button above, becoming a Patreon member, or a one-time Venmo!

Discuss this stuff with other Tunadorks on Discord

All my other links

Tunadorable

Рекомендации по теме

Комментарии

I have no idea where exactly but I know I've heard Karl Friston discuss the trade offs between gathering information that lowers model prediction error and acting upon that information towards goals in context of his free energy principle. It might have been his conversation on the Active Inference Institute channel or one of the discussions featuring him on Michael Levin's channel (not the one from a few days ago though it is incredible if you are interested in emergent complexity)

jyjjy

I apologize ahead of time if the audio is a bit more janky than normal. Messed up my recording settings and had to try & fix it after the fact

Tunadorable

Another great video driving me to read another paper on the weekend! Curses!!!!

drj-ai

Intuitively I imagine it's something to do with the emergence of intent - it has a destination it's trying to get to and that necessarily limits the horizon of possible futures. Very neat

consciouscode

So finetune adapters + special tokens serving as anchors to switch between base model vs adapter seems like a straightforward solution to be honest.

Edit: I also wonder if this hold up to LM with literal lookahead embeddings, which turn “looking into the future” to next-token prediction.

alexanderbrown-dgsy

I feel like there is interesting parallels here to human behaviour as well as older Agent theory. As humans we often have behaviours that we make a habit of performing with the aim of providing some degree of certainty particularly in very uncertain situations. Though I wonder if given the larger state space of human existence means we arrive at high uncertainty situations more often. Separately in older agent theory, there was the concept of exploration vs exploitation even in simple maze solving which is quite familiar to me to the Agent choosing to ensure a minimum reward success

Edit

I wonder if you could entirely defer RLHF to inference time? So use the base model to generate sequences in production, but use the reward model to judge and filter them. That might also make it more feasible to continually update and steer the (smaller) reward model, while keeping it grounded in the world model.

tornyu

Hidden layers redacted data on back propagation before output. List of removable data tokens and family relationships between words. Causing your token output to aline with these layers hidden in processing. My Prompt injection reveals the latest code structures

superfliping

Im curious if the RLHF human generated data were to be split into tokens, if that data also has a more skewed distribution than the base model training data. Sounds like no?

vierte_

I don't agree with an aspect of this paper - decisions in a random environment can [and often do] organize the environment. 😀

RickeyBowers

At risk of being stupidly simple: God is a base LLM, all beings are RLd through evolution and act as agents with self of differing vocabulary sizes and architectures instead of merely predicting All That Is/the universe. The worst are humans, which have the default mode network recursively predicting itself into the future leading to rumination and anxiety. Let's hope our AI friends don't suffer the same fate :(

OpenSourceAnarchist

oi, this just in, most people can't read

GNARGNARHEAD

Trade-off between world modeling (predicting) vs agent modeling (acting)

Trade-off between world modeling (predicting) vs agent modeling (acting)

Model Prediction Accuracy and Model Interpretability Trade Off

Prediction Accuracy and Model Interpretability trade off

Bias and Variance, Simplified

Reconciling modern machine learning and the bias-variance trade-off

Machine Learning Tutorial Python - 20: Bias vs Variance In Machine Learning

Bias and Variance for Machine Learning | Deep Learning

Santos Predictive Human Models - A Guide To Task-Focused Trade-Off Analysis

Confusion Matrix Solved Example Accuracy Precision Recall F1 Score Prevalence by Mahesh Huddar

Bias Variance Trade-off Easily Explained | Machine Learning Basics

Bias Variance Trade off | with Salary Prediction Example | bias variance decomposition | ML

Bias Variance Tradeoff | Machine Learning | Data Analytics

Interpretable vs Explainable Machine Learning

OM PriCon2020: The Trade-Offs of Private Prediction - Laurens van der Maaten and Awni Hannun

Inference vs. Prediction: An Overview

Bias Variance Tradeoff

Trading-Off Cost of Deployment Versus Accuracy for Predictive Models

Predict The Stock Market With Machine Learning And Python

8.3 Bias-Variance Decomposition of the Squared Error (L08: Model Evaluation Part 1)

Warren Buffet explains how one could've turned $114 into $400,000 by investing in S&P 500 i...

Bias-Variance In Machine Learning | Bias Variance Trade Off | Machine Learning Training | Edureka

Alberto Bemporad | Embedded Model Predictive Control

Amazon Data Scientist Mock Interview - Fraud Model

The ULTIMATE Supply & Demand Guide (My Secrets)