Hyena Hierarchy

This code trains a character-level language model using a variant of a convolutional neural network architecture called Hyena.

Requirements

To run this code, you need to have Python 3 and the following packages installed:

• torch

Run

python hyena-split-memory.py

Credits

This code is inspired by the papers

A Convolutional Neural Network for Modelling Sentences"

by Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom.

Hyena Hierarchy: Towards Larger Convolutional Language Models

by: Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, Christopher Ré

Affiliations:

1. Department of Computer Science, Stanford University, Stanford, CA, USA
2. Mila - Quebec AI Institute and DIRO, Université de Montréal, Montréal, QC, Canada