Transformers

TagLast edit: 24 Feb 2022 11:01 UTC by Vivek Hebbar

Research agenda—Building a multi-modal chess-language model

p.b.7 Apr 2022 12:25 UTC

8 points

2 comments2 min readLW link

How fast can we perform a forward pass?

jsteinhardt10 Jun 2022 23:30 UTC

53 points

9 comments15 min readLW link

(bounded-regret.ghost.io)

Searching for Modularity in Large Language Models

NickyP and Stephen Fowler

8 Sep 2022 2:25 UTC

44 points

3 comments14 min readLW link

Brief Notes on Transformers

Adam Jermyn26 Sep 2022 14:46 UTC

46 points

3 comments2 min readLW link

Building a transformer from scratch—AI safety up-skilling challenge

Marius Hobbhahn12 Oct 2022 15:40 UTC

42 points

1 comment5 min readLW link

Concrete Steps to Get Started in Transformer Mechanistic Interpretability

Neel Nanda25 Dec 2022 22:21 UTC

54 points

7 comments12 min readLW link

(www.neelnanda.io)

[Question] Are Mixture-of-Experts Transformers More Interpretable Than Dense Transformers?

simeon_c31 Dec 2022 11:34 UTC

7 points

5 comments1 min readLW link

Tracr: Compiled Transformers as a Laboratory for Interpretability | DeepMind

DragonGod13 Jan 2023 16:53 UTC

62 points

12 comments1 min readLW link

(arxiv.org)

No Really, Attention is ALL You Need—Attention can do feedforward networks

Robert_AIZI31 Jan 2023 18:48 UTC

29 points

7 comments6 min readLW link

(aizi.substack.com)

Addendum: More Efficient FFNs via Attention

Robert_AIZI6 Feb 2023 18:55 UTC

10 points

2 comments5 min readLW link

(aizi.substack.com)

Google’s PaLM-E: An Embodied Multimodal Language Model

SandXbox7 Mar 2023 4:11 UTC

86 points

7 comments1 min readLW link

(palm-e.github.io)

How Do Induction Heads Actually Work in Transformers With Finite Capacity?

Fabien Roger23 Mar 2023 9:09 UTC

27 points

0 comments5 min readLW link

So, just why do GPTs have to operate by continuing an existing string?

Bill Benzon24 Mar 2023 12:08 UTC

−4 points

0 comments3 min readLW link

We Need To Know About Continual Learning

michael_mjd22 Apr 2023 17:08 UTC

29 points

14 comments4 min readLW link

Residual stream norms grow exponentially over the forward pass

StefanHex and TurnTrout

7 May 2023 0:46 UTC

72 points

24 comments11 min readLW link

Understanding mesa-optimization using toy models

tilmanr, rusheb, Guillaume Corlouer, Dan Valentine, afspies, mivanitskiy and Can

7 May 2023 17:00 UTC

42 points

2 comments10 min readLW link

An Analogy for Understanding Transformers

CallumMcDougall13 May 2023 12:20 UTC

80 points

5 comments9 min readLW link

Transformer Architecture Choice for Resisting Prompt Injection and Jail-Breaking Attacks

RogerDearnaley21 May 2023 8:29 UTC

9 points

1 comment4 min readLW link

Neuroevolution, Social Intelligence, and Logic

vinnik.dmitry0731 May 2023 17:54 UTC

1 point

0 comments10 min readLW link

[Question] Killing Recurrent Memory Over Self Attention?

Del Nobolo6 Jun 2023 23:02 UTC

3 points

0 comments1 min readLW link

GPT-2′s positional embedding matrix is a helix

AdamYedidia21 Jul 2023 4:16 UTC

42 points

18 comments4 min readLW link

How LLMs are and are not myopic

janus25 Jul 2023 2:19 UTC

122 points

14 comments8 min readLW link

The positional embedding matrix and previous-token heads: how do they actually work?

AdamYedidia10 Aug 2023 1:58 UTC

26 points

4 comments13 min readLW link

Google DeepMind’s RT-2

SandXbox11 Aug 2023 11:26 UTC

9 points

1 comment1 min readLW link

(robotics-transformer2.github.io)

World, mind, and learnability: A note on the metaphysical structure of the cosmos [& LLMs]

Bill Benzon5 Sep 2023 12:19 UTC

4 points

1 comment5 min readLW link

New Tool: the Residual Stream Viewer

AdamYedidia1 Oct 2023 0:49 UTC

32 points

7 comments4 min readLW link

(tinyurl.com)

The Method of Loci: With some brief remarks, including transformers and evaluating AIs

Bill Benzon2 Dec 2023 14:36 UTC

6 points

0 comments3 min readLW link

Has anyone experimented with Dodrio, a tool for exploring transformer models through interactive visualization?

Bill Benzon11 Dec 2023 20:34 UTC

4 points

0 comments1 min readLW link

Exploring the Residual Stream of Transformers for Mechanistic Interpretability — Explained

Zeping Yu26 Dec 2023 0:36 UTC

7 points

1 comment11 min readLW link

AGI will be made of heterogeneous components, Transformer and Selective SSM blocks will be among them

Roman Leventov27 Dec 2023 14:51 UTC

33 points

9 comments4 min readLW link

Striking Implications for Learning Theory, Interpretability — and Safety?

RogerDearnaley5 Jan 2024 8:46 UTC

35 points

4 comments2 min readLW link

Attention SAEs Scale to GPT-2 Small

Connor Kissane, robertzk, Arthur Conmy and Neel Nanda

3 Feb 2024 6:50 UTC

76 points

4 comments8 min readLW link

Skepticism About DeepMind’s “Grandmaster-Level” Chess Without Search

Arjun Panickssery12 Feb 2024 0:56 UTC

53 points

13 comments3 min readLW link

Deconfusing In-Context Learning

Arjun Panickssery25 Feb 2024 9:48 UTC

37 points

1 comment2 min readLW link

Modern Transformers are AGI, and Human-Level

abramdemski26 Mar 2024 17:46 UTC

196 points

89 comments5 min readLW link

Decompiling Tracr Transformers—An interpretability experiment

Hannes Thurnherr27 Mar 2024 9:49 UTC

3 points

0 comments14 min readLW link

[Question] Barcoding LLM Training Data Subsets. Anyone trying this for interpretability?

right..enough?13 Apr 2024 3:09 UTC

7 points

0 comments7 min readLW link

Transformers Represent Belief State Geometry in their Residual Stream

Adam Shai16 Apr 2024 21:16 UTC

300 points

63 comments12 min readLW link