CarperAI/trlx: v0.5.0: Initial NeMo integration, HH example, and improved Hugging Face integration

Highlights Initial NeMo ILQL integration leading way to large-scale RLHF efforts. See https://github.com/CarperAI/trlx/blob/main/trlx/models/README.md to get started. In-depth example showcasing trlx usage on AnthropicAI's Helpful & Harmless dataset https://github.com/CarperAI/trlx/tree/main/examples/hh Improved ILQL modeling integration with Hugging Face transformers. Users can now work with AutoModelForCausalLMWithILQLHeads objects to generate samples and save/load fine-tuned ILQL models that can be quickly pushed to the Hub. What's Changed Add wandb group naming by @jon-tow in https://github.com/CarperAI/trlx/pull/188 Update reward_fn signatures in examples by @jon-tow in https://github.com/CarperAI/trlx/pull/190 Add tokenizer config by @reciprocated in https://github.com/CarperAI/trlx/pull/189 Fix extraction of mixed_precision option for deepspeed by @reciprocated in https://github.com/CarperAI/trlx/pull/197 Fix summarize_rlhf inference checkpoint paths by @jon-tow in https://github.com/CarperAI/trlx/pull/194 Make the config loading consistent across all example scripts. by @shermansiu in https://github.com/CarperAI/trlx/pull/192 Make Trainer.save_pretrained sub-directory optional by @jon-tow in https://github.com/CarperAI/trlx/pull/201 Update Readme to include T5 models by @aaronrmm in https://github.com/CarperAI/trlx/pull/198 Make make_head accept dtype parameter by @reciprocated in https://github.com/CarperAI/trlx/pull/213 Enable training with Tensorboard tracking by @marcobellagente93 in https://github.com/CarperAI/trlx/pull/209 Support nested updates in merge by @cat-state in https://github.com/CarperAI/trlx/pull/219 Fix typo reward normalize summarize by @PhungVanDuy in https://github.com/CarperAI/trlx/pull/221 Update stale comment from results table by @jon-tow in https://github.com/CarperAI/trlx/pull/222 Fix undefined trackers property by @alan-cooney in https://github.com/CarperAI/trlx/pull/224 Fix tokenizer missing form config.to_dict() by @alan-cooney in https://github.com/CarperAI/trlx/pull/228 Make experiment tracking optional by @jon-tow in https://github.com/CarperAI/trlx/pull/226 read tokenizer path from config correctly by @JustinAWei in https://github.com/CarperAI/trlx/pull/230 Add devcontainer support by @alan-cooney in https://github.com/CarperAI/trlx/pull/196 fix: change lora_a:float to lora_r:int by @aaronrmm in https://github.com/CarperAI/trlx/pull/235 Bump isort to hotfix CI code quality workflow by @jon-tow in https://github.com/CarperAI/trlx/pull/237 Fix optional tracking in accelerator.log by @jon-tow in https://github.com/CarperAI/trlx/pull/233 Improve documentation/comments on the random walk example by @alan-cooney in https://github.com/CarperAI/trlx/pull/208 Update link to "Learning to Summarize from Human Feedback" by @jon-tow in https://github.com/CarperAI/trlx/pull/241 Fix deepspeed state saving under save_best condition by @reciprocated in https://github.com/CarperAI/trlx/pull/242 added colab notebook by @smellslikeml in https://github.com/CarperAI/trlx/pull/244 [style] Increase black's line length by @reciprocated in https://github.com/CarperAI/trlx/pull/250 Add help string to get_advantages_and_returns by @pesvut in https://github.com/CarperAI/trlx/pull/225 Filter out empty responses by @reciprocated in https://github.com/CarperAI/trlx/pull/265 NeMo Integrate by @cat-state in https://github.com/CarperAI/trlx/pull/125 Add multi-process logger utility for status monitoring by @jon-tow in https://github.com/CarperAI/trlx/pull/254 Add NeMo support info to README by @jon-tow in https://github.com/CarperAI/trlx/pull/275 Fix distributed dataloaders & deduplicate eval by @reciprocated in https://github.com/CarperAI/trlx/pull/276 Improve PPO readability by @alan-cooney in https://github.com/CarperAI/trlx/pull/210 Add T5 to delta modifier map by @aaronrmm in https://github.com/CarperAI/trlx/pull/234 [fix] Set deepspeed's fp16 auto_cast to false by @reciprocated in https://github.com/CarperAI/trlx/pull/279 Rename remaining logprobs_from_logits call by @jon-tow in https://github.com/CarperAI/trlx/pull/281 [feat] Add Accelerate SFT Trainer by @reciprocated in https://github.com/CarperAI/trlx/pull/280 Add Colab Notebook for Sentiment by @zswitten in https://github.com/CarperAI/trlx/pull/285 Remove pylance installs from devcontainer by @jon-tow in https://github.com/CarperAI/trlx/pull/296 Move notebooks to examples dir by @jon-tow in https://github.com/CarperAI/trlx/pull/294 [fix] Summarize config discrepancy by @reciprocated in https://github.com/CarperAI/trlx/pull/293 Make Git check optional by @cat-state in https://github.com/CarperAI/trlx/pull/299 refactor: remove orchestrator abstraction from API by @jon-tow in https://github.com/CarperAI/trlx/pull/289 Set add_special_tokens=False to not add EOS unexpectedly by @cat-state in https://github.com/CarperAI/trlx/pull/287 [feat] Gather experience samples by @reciprocated in https://github.com/CarperAI/trlx/pull/305 [fix] Make gather_for_metrics usage more strict by @reciprocated in https://github.com/CarperAI/trlx/pull/315 Add helpful and harmless example by @reciprocated in https://github.com/CarperAI/trlx/pull/128 Adopt PreTrainedModelWrapper for Hugging Face models by @jon-tow in https://github.com/CarperAI/trlx/pull/215 New Contributors @shermansiu made their first contribution in https://github.com/CarperAI/trlx/pull/192 @aaronrmm made their first contribution in https://github.com/CarperAI/trlx/pull/198 @marcobellagente93 made their first contribution in https://github.com/CarperAI/trlx/pull/209 @alan-cooney made their first contribution in https://github.com/CarperAI/trlx/pull/224 @JustinAWei made their first contribution in https://github.com/CarperAI/trlx/pull/230 @smellslikeml made their first contribution in https://github.com/CarperAI/trlx/pull/244 @pesvut made their first contribution in https://github.com/CarperAI/trlx/pull/225 @zswitten made their first contribution in https://github.com/CarperAI/trlx/pull/285 Full Changelog: https://github.com/CarperAI/trlx/compare/v0.4...v0.5.