irradiated@radiation.partyMB to TechNews@radiation.party · 2 years ago

[HN] Transformer Attention is off by one

www.evanmiller.org

0

cross-posted to:
mltheory@lemmy.intai.tech

1

[HN] Transformer Attention is off by one

www.evanmiller.org

irradiated@radiation.partyMB to TechNews@radiation.party · 2 years ago

0

cross-posted to:
mltheory@lemmy.intai.tech

Attention Is Off By One

www.evanmiller.org

Let’s fix these pesky Transformer outliers using Softmax One and QuietAttention.

[ comments | sourced from HackerNews ]

You must log in or register to comment.

Chat