GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch
- Andonian, Alex
- Biderman, Stella
- Black, Sid
- Gali, Preetham
- Gao, Leo
- Hallahan, Eric
- Levy-Kramer, Josh
- Leahy, Connor
- Nestler, Lucas
- Parker, Kip
- Pieler, Michael
- Purohit, Shivanshu
- Songz, Tri
- Phil, Wang
- Weinbach, Samuel
Publication date
March 2023
DOIAbstract
This is the legacy GPT-NeoX relying on old DeeperSpeed (0.3.15). We only recommend using this release under circumstance that you're loading a model based on old DeeperSpeed (e.g. GPT-J, GPT-NeoX20B, the Pythia suite, etc). The primary difference between this release and v2.x is the DeepSpeed version supported. If you're using 2.x, we're assuming that you're using either the latest release of DeepSpeed or DeeperSpeed 2.x.If you use this software, please cite it using these metadata
Add to ORKG