RSS

Transformers

TagLast edit: 24 Feb 2022 11:01 UTC by Vivek Hebbar

Strik­ing Im­pli­ca­tions for Learn­ing The­ory, In­ter­pretabil­ity — and Safety?

RogerDearnaley5 Jan 2024 8:46 UTC
35 points
4 comments2 min readLW link

How LLMs are and are not myopic

janus25 Jul 2023 2:19 UTC
122 points
14 comments8 min readLW link

AGI will be made of het­ero­ge­neous com­po­nents, Trans­former and Selec­tive SSM blocks will be among them

Roman Leventov27 Dec 2023 14:51 UTC
33 points
9 comments4 min readLW link

Tracr: Com­piled Trans­form­ers as a Lab­o­ra­tory for In­ter­pretabil­ity | Deep­Mind

DragonGod13 Jan 2023 16:53 UTC
62 points
12 comments1 min readLW link
(arxiv.org)

How fast can we perform a for­ward pass?

jsteinhardt10 Jun 2022 23:30 UTC
53 points
9 comments15 min readLW link
(bounded-regret.ghost.io)

How Do In­duc­tion Heads Ac­tu­ally Work in Trans­form­ers With Finite Ca­pac­ity?

Fabien Roger23 Mar 2023 9:09 UTC
27 points
0 comments5 min readLW link

Resi­d­ual stream norms grow ex­po­nen­tially over the for­ward pass

7 May 2023 0:46 UTC
72 points
24 comments11 min readLW link

Google’s PaLM-E: An Em­bod­ied Mul­ti­modal Lan­guage Model

SandXbox7 Mar 2023 4:11 UTC
86 points
7 comments1 min readLW link
(palm-e.github.io)

Con­crete Steps to Get Started in Trans­former Mechanis­tic Interpretability

Neel Nanda25 Dec 2022 22:21 UTC
54 points
7 comments12 min readLW link
(www.neelnanda.io)

Trans­former Ar­chi­tec­ture Choice for Re­sist­ing Prompt In­jec­tion and Jail-Break­ing Attacks

RogerDearnaley21 May 2023 8:29 UTC
9 points
1 comment4 min readLW link

Neu­roevolu­tion, So­cial In­tel­li­gence, and Logic

vinnik.dmitry0731 May 2023 17:54 UTC
1 point
0 comments10 min readLW link

[Question] Killing Re­cur­rent Me­mory Over Self At­ten­tion?

Del Nobolo6 Jun 2023 23:02 UTC
3 points
0 comments1 min readLW link

GPT-2′s po­si­tional em­bed­ding ma­trix is a helix

AdamYedidia21 Jul 2023 4:16 UTC
42 points
18 comments4 min readLW link

Has any­one ex­per­i­mented with Do­drio, a tool for ex­plor­ing trans­former mod­els through in­ter­ac­tive vi­su­al­iza­tion?

Bill Benzon11 Dec 2023 20:34 UTC
4 points
0 comments1 min readLW link

Google Deep­Mind’s RT-2

SandXbox11 Aug 2023 11:26 UTC
9 points
1 comment1 min readLW link
(robotics-transformer2.github.io)

World, mind, and learn­abil­ity: A note on the meta­phys­i­cal struc­ture of the cos­mos [& LLMs]

Bill Benzon5 Sep 2023 12:19 UTC
4 points
1 comment5 min readLW link

New Tool: the Resi­d­ual Stream Viewer

AdamYedidia1 Oct 2023 0:49 UTC
32 points
7 comments4 min readLW link
(tinyurl.com)

Re­search agenda—Build­ing a multi-modal chess-lan­guage model

p.b.7 Apr 2022 12:25 UTC
8 points
2 comments2 min readLW link

Search­ing for Mo­du­lar­ity in Large Lan­guage Models

8 Sep 2022 2:25 UTC
44 points
3 comments14 min readLW link

Brief Notes on Transformers

Adam Jermyn26 Sep 2022 14:46 UTC
44 points
3 comments2 min readLW link

Build­ing a trans­former from scratch—AI safety up-skil­ling challenge

Marius Hobbhahn12 Oct 2022 15:40 UTC
42 points
1 comment5 min readLW link

[Question] Are Mix­ture-of-Ex­perts Trans­form­ers More In­ter­pretable Than Dense Trans­form­ers?

simeon_c31 Dec 2022 11:34 UTC
7 points
5 comments1 min readLW link

No Really, At­ten­tion is ALL You Need—At­ten­tion can do feed­for­ward networks

Robert_AIZI31 Jan 2023 18:48 UTC
26 points
7 comments6 min readLW link
(aizi.substack.com)

Ad­den­dum: More Effi­cient FFNs via Attention

Robert_AIZI6 Feb 2023 18:55 UTC
10 points
2 comments5 min readLW link
(aizi.substack.com)

The po­si­tional em­bed­ding ma­trix and pre­vi­ous-to­ken heads: how do they ac­tu­ally work?

AdamYedidia10 Aug 2023 1:58 UTC
26 points
4 comments13 min readLW link

Ex­plor­ing the Resi­d­ual Stream of Trans­form­ers for Mechanis­tic In­ter­pretabil­ity — Explained

Zeping Yu26 Dec 2023 0:36 UTC
7 points
1 comment11 min readLW link

At­ten­tion SAEs Scale to GPT-2 Small

3 Feb 2024 6:50 UTC
69 points
1 comment8 min readLW link

Skep­ti­cism About Deep­Mind’s “Grand­mas­ter-Level” Chess Without Search

Arjun Panickssery12 Feb 2024 0:56 UTC
48 points
13 comments3 min readLW link

“In-Con­text” “Learn­ing”

Arjun Panickssery25 Feb 2024 9:48 UTC
27 points
0 comments2 min readLW link

So, just why do GPTs have to op­er­ate by con­tin­u­ing an ex­ist­ing string?

Bill Benzon24 Mar 2023 12:08 UTC
−4 points
0 comments3 min readLW link

We Need To Know About Con­tinual Learning

michael_mjd22 Apr 2023 17:08 UTC
29 points
14 comments4 min readLW link

The Method of Loci: With some brief re­marks, in­clud­ing trans­form­ers and eval­u­at­ing AIs

Bill Benzon2 Dec 2023 14:36 UTC
6 points
0 comments3 min readLW link

An Anal­ogy for Un­der­stand­ing Transformers

CallumMcDougall13 May 2023 12:20 UTC
80 points
5 comments9 min readLW link
No comments.