Evaluating Large Language Models for Cybersecurity Tasks: Challenges and Best Practices

preview_player
Показать описание
How can we effectively use large language models (LLMs) for cybersecurity tasks? In this podcast from the Carnegie Mellon Software Engineering Institute, Jeff Gennari and Sam Perl discuss applications for LLMs in cybersecurity, potential challenges, and recommendations for evaluating LLMs.

#LLMs, #AI, #cybersecurity, @TheSEICMU

  1. Software Engineering Institute | Carnegie Mellon University
Рекомендации по теме
How to evaluate 0:03:17

How to evaluate and choose a Large Language Model (LLM)

How Large Language 0:05:34

How Large Language Models Work

Evaluating the Output 0:01:42

Evaluating the Output of Your LLM (Large Language Models): Insights from Microsoft & LangChain

Master LLMs: Top 0:08:42

Master LLMs: Top Strategies to Evaluate LLM Performance

Petr Polezhaev – 0:42:21

Petr Polezhaev – Advancements in Evaluating Large Language Model Applications

Evaluation for Large 1:16:49

Evaluation for Large Language Models and Generative AI - A Deep Dive

Evaluating LLM-based Applications 0:33:50

Evaluating LLM-based Applications

LLM Module 4: 0:02:23

LLM Module 4: Fine-tuning and Evaluating LLMs | 4.9 Evaluating LLMs

There are no 0:00:34

There are no standardized evaluation criteria to measure the responsible behavior of LLMs.

Evaluating Large Language 0:59:09

Evaluating Large Language Models: Simple and Easy Techniques for Ensuring Generative AI Reliability

Evaluating Large Language 0:43:05

Evaluating Large Language Models for Cybersecurity Tasks: Challenges and Best Practices

EvalLM: Interactive Evaluation 0:11:40

EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria

Evaluating Large Language 0:31:48

Evaluating Large Language Models on Clinical & Biomedical NLP Benchmarks

Evaluating Large Language 0:09:57

Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study

Evaluating Large Language 0:21:09

Evaluating Large Language Models Trained on Code - OpenAI Codex Paper

Evaluating Large Language 0:26:19

Evaluating Large Language Models Trained on Code

Evaluating Large Language 0:51:05

Evaluating Large Language Models Trained on Code

Evaluating Large Language 0:04:25

Evaluating Large Language Models: 30 Common Metrics

Yann Dubois: Scalable 1:37:47

Yann Dubois: Scalable Evaluation of Large Language Models

Read TWO papers: 0:07:37

Read TWO papers: How to evaluate LLM performance

Evaluating large language 0:28:59

Evaluating large language models with Ray in hybrid cloud

Can AI Really 0:08:47

Can AI Really Plan? Evaluating Large Language Models and Reasoning Models

How to evaluate 0:04:02

How to evaluate large language models using Prompt Engineering | Testing and Improving with PyTorch

Evaluation Approaches for 0:02:50

Evaluation Approaches for Your LLM (Large Language Model): Insights from Microsoft & LangChain

Комментарии
Автор

At 36:45 a key point is made about the specialization within the profession. It's already a problem within #cybersecurity that specialization across vulnerability management, platform security, app sec, cryptography, secrets management -- become de facto siloes. So far, it's unclear whether #LLMs can help address this problem, by summarization and narrative-making perhaps. But I doubt it can fully overcome it. In fact, the LLM may add a layer of, for instance, generated "code" or templating on top of what is already a tower of subcomponents and services -- thereby adding still more complexity.

That said, I doubt there will be any turning back.

MarkUnderwood-knowlengr