Artwork

Content provided by Daniel Filan. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Daniel Filan or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.
Player FM - Podcast App
Go offline with the Player FM app!

23 - Mechanistic Anomaly Detection with Mark Xu

2:05:52
 
Share
 

Manage episode 372320078 series 2844728
Content provided by Daniel Filan. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Daniel Filan or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.

Is there some way we can detect bad behaviour in our AI system without having to know exactly what it looks like? In this episode, I speak with Mark Xu about mechanistic anomaly detection: a research direction based on the idea of detecting strange things happening in neural networks, in the hope that that will alert us of potential treacherous turns. We both talk about the core problems of relating these mechanistic anomalies to bad behaviour, as well as the paper "Formalizing the presumption of independence", which formulates the problem of formalizing heuristic mathematical reasoning, in the hope that this will let us mathematically define "mechanistic anomalies".

Patreon: patreon.com/axrpodcast

Ko-fi: ko-fi.com/axrpodcast

Episode art by Hamish Doodles: hamishdoodles.com/

Topics we discuss, and timestamps:

- 0:00:38 - Mechanistic anomaly detection

- 0:09:28 - Are all bad things mechanistic anomalies, and vice versa?

- 0:18:12 - Are responses to novel situations mechanistic anomalies?

- 0:39:19 - Formalizing "for the normal reason, for any reason"

- 1:05:22 - How useful is mechanistic anomaly detection?

- 1:12:38 - Formalizing the Presumption of Independence

- 1:20:05 - Heuristic arguments in physics

- 1:27:48 - Difficult domains for heuristic arguments

- 1:33:37 - Why not maximum entropy?

- 1:44:39 - Adversarial robustness for heuristic arguments

- 1:54:05 - Other approaches to defining mechanisms

- 1:57:20 - The research plan: progress and next steps

- 2:04:13 - Following ARC's research

The transcript: axrp.net/episode/2023/07/24/episode-23-mechanistic-anomaly-detection-mark-xu.html

ARC links:

- Website: alignment.org

- Theory blog: alignment.org/blog

- Hiring page: alignment.org/hiring

Research we discuss:

- Formalizing the presumption of independence: arxiv.org/abs/2211.06738

- Eliciting Latent Knowledge (aka ELK): alignmentforum.org/posts/qHCDysDnvhteW7kRd/arc-s-first-technical-report-eliciting-latent-knowledge

- Mechanistic Anomaly Detection and ELK: alignmentforum.org/posts/vwt3wKXWaCvqZyF74/mechanistic-anomaly-detection-and-elk

- Can we efficiently explain model behaviours? alignmentforum.org/posts/dQvxMZkfgqGitWdkb/can-we-efficiently-explain-model-behaviors

- Can we efficiently distinguish different mechanisms? alignmentforum.org/posts/JLyWP2Y9LAruR2gi9/can-we-efficiently-distinguish-different-mechanisms

  continue reading

38 episodes

Artwork
iconShare
 
Manage episode 372320078 series 2844728
Content provided by Daniel Filan. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Daniel Filan or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.

Is there some way we can detect bad behaviour in our AI system without having to know exactly what it looks like? In this episode, I speak with Mark Xu about mechanistic anomaly detection: a research direction based on the idea of detecting strange things happening in neural networks, in the hope that that will alert us of potential treacherous turns. We both talk about the core problems of relating these mechanistic anomalies to bad behaviour, as well as the paper "Formalizing the presumption of independence", which formulates the problem of formalizing heuristic mathematical reasoning, in the hope that this will let us mathematically define "mechanistic anomalies".

Patreon: patreon.com/axrpodcast

Ko-fi: ko-fi.com/axrpodcast

Episode art by Hamish Doodles: hamishdoodles.com/

Topics we discuss, and timestamps:

- 0:00:38 - Mechanistic anomaly detection

- 0:09:28 - Are all bad things mechanistic anomalies, and vice versa?

- 0:18:12 - Are responses to novel situations mechanistic anomalies?

- 0:39:19 - Formalizing "for the normal reason, for any reason"

- 1:05:22 - How useful is mechanistic anomaly detection?

- 1:12:38 - Formalizing the Presumption of Independence

- 1:20:05 - Heuristic arguments in physics

- 1:27:48 - Difficult domains for heuristic arguments

- 1:33:37 - Why not maximum entropy?

- 1:44:39 - Adversarial robustness for heuristic arguments

- 1:54:05 - Other approaches to defining mechanisms

- 1:57:20 - The research plan: progress and next steps

- 2:04:13 - Following ARC's research

The transcript: axrp.net/episode/2023/07/24/episode-23-mechanistic-anomaly-detection-mark-xu.html

ARC links:

- Website: alignment.org

- Theory blog: alignment.org/blog

- Hiring page: alignment.org/hiring

Research we discuss:

- Formalizing the presumption of independence: arxiv.org/abs/2211.06738

- Eliciting Latent Knowledge (aka ELK): alignmentforum.org/posts/qHCDysDnvhteW7kRd/arc-s-first-technical-report-eliciting-latent-knowledge

- Mechanistic Anomaly Detection and ELK: alignmentforum.org/posts/vwt3wKXWaCvqZyF74/mechanistic-anomaly-detection-and-elk

- Can we efficiently explain model behaviours? alignmentforum.org/posts/dQvxMZkfgqGitWdkb/can-we-efficiently-explain-model-behaviors

- Can we efficiently distinguish different mechanisms? alignmentforum.org/posts/JLyWP2Y9LAruR2gi9/can-we-efficiently-distinguish-different-mechanisms

  continue reading

38 episodes

All episodes

×
 
Loading …

Welcome to Player FM!

Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.

 

Quick Reference Guide