A PyTorch implementation of Perceiver, Perceiver IO and Perceiver AR with PyTorch Lightning scripts for distributed training.

Data preprocessing and documentation enhancements, major refactorings Functional enhancements: Support for static word masking in addition to dynamic word masking. Support for individual token masking in addition to whole word masking. Task-specific data preprocessing for all supported text datasets. Constant learning rate scheduler with warmup now used by default. Documentation enhancements: All training examples now provided as command line and Python script. Better overview of official models and example training checkpoints. Example training checkpoints can now be downloaded individually. Minor enhancements to all other documentation sections. Refactorings and breaking changes: Rename image package to vision. TextDataModule base class now implements complete preprocessing logic. TextDataModule subclasses only convert source dataset to a common structure. Abstraction over cross-attention query creation (QueryProvider). Decouple OutputAdapter interface from trainable cross-attention query. Implement learned positions encodings as nn.Embedding. Move adapters to separate perceiver.model.core.adapter module. Rename PerceiverConfig to PerceiverIOConfig Rename LitModel base class to LitPerceiverIO. LitClassifier.forward now behaves like the wrapped model's forward. Object-oriented design of conversion from Hugging Face Perceiver models. Major refactoring of PerceiverAR and CausalLanguageModel. Move FourierPositionEncoding to perceiver.model.core.position` module.If you use this software, please cite it as below