CarperAI/trlx: v0.7.0: NeMO PPO, PEFT Migration, and Fixes

The v0.7.0 release includes several new features, bug fixes, and overall improvements to the codebase. Here are the key changes: NeMO PPO and SFT support This release introduces NeMO-backed PPO and SFT implementations for capabilities and improved system performance under large-scale training. NeMo PPO by @cat-state in https://github.com/CarperAI/trlx/pull/472 Add Supervised Fine-Tuning (SFT) support for NeMo backend by @jon-tow in https://github.com/CarperAI/trlx/pull/353 PEFT Migration trlx now supports parameter-efficient tuning methods via the peft library, which we hope will provide greater access to RLHF training in low-resource settings. peft to opendelta migration (#434) + memory optimization (#320) by @glerzing in https://github.com/CarperAI/trlx/pull/486 Fixes and mores! Set pad_token for all tokenizers in tests by @cat-state in https://github.com/CarperAI/trlx/pull/414 Convert tensors in the stats dict into scalars by @ZHAOTING in https://github.com/CarperAI/trlx/pull/417 Add Translation Finetuning Example with T5 by @alexandremuzio in https://github.com/CarperAI/trlx/pull/392 set torch dependency to version 2.0.0 for CUDA in installation instru… by @cauyxy in https://github.com/CarperAI/trlx/pull/409 [fix] add position_ids to LlamaModelBranch by @jon-tow in https://github.com/CarperAI/trlx/pull/418 fix(CI): use pinned deps for CI testing by @jon-tow in https://github.com/CarperAI/trlx/pull/423 Minibatch impl by @Dahoas in https://github.com/CarperAI/trlx/pull/364 [feat] Support tying metadata to each prompt by @maxreciprocate in https://github.com/CarperAI/trlx/pull/421 feat(examples): revamp simulacra example by @maxreciprocate in https://github.com/CarperAI/trlx/pull/430 [fix] update pairwise dataloader. by @Chen9154 in https://github.com/CarperAI/trlx/pull/395 fix(sft_trainer): total_steps calculation when running distributed by @maxreciprocate in https://github.com/CarperAI/trlx/pull/432 fix(base_trainer): gather weights in save_pretrained under zero3 by @maxreciprocate in https://github.com/CarperAI/trlx/pull/429 fix(offline_pipeline): ILQL negative indexing under truncation by @maxreciprocate in https://github.com/CarperAI/trlx/pull/435 fix(ppo_trainer): compute mean KL sequence-wise by @maxreciprocate in https://github.com/CarperAI/trlx/pull/441 Create Example training scripts to run in Stability cluster by @alexandremuzio in https://github.com/CarperAI/trlx/pull/419 Upgrade official released Ray instead of an unstable one. by @jovany-wang in https://github.com/CarperAI/trlx/pull/455 Pin transformers<=4.27.1 by @jovany-wang in https://github.com/CarperAI/trlx/pull/458 fix(ppo_gpt): prevent position_ids being None by @li-plus in https://github.com/CarperAI/trlx/pull/451 fix(trainer): init self.generate_sweep_kwarg at self.init by @mymusise in https://github.com/CarperAI/trlx/pull/460 Ensure trailing EOS token is added correctly for shorter generated outputs by @mikljohansson in https://github.com/CarperAI/trlx/pull/420 Pad prompts to the right in T5 examples and add EOS token to seq2seq prompts by @mikljohansson in https://github.com/CarperAI/trlx/pull/422 docs(base_trainer): fill in missing prepare_learning method by @maxreciprocate in https://github.com/CarperAI/trlx/pull/449 fix(modeling_ppo): invert padding percentage calculation by @maxreciprocate in https://github.com/CarperAI/trlx/pull/450 fix(base_trainer): flatten tag list for tensorboard hparams logging by @maxreciprocate in https://github.com/CarperAI/trlx/pull/444 feat(requirements.txt): upgrade dependencies by @maxreciprocate in https://github.com/CarperAI/trlx/pull/465 fix(offline_pipeline): force drop_last only for distributed by @maxreciprocate in https://github.com/CarperAI/trlx/pull/475 hotfix(bnb): install scipy with bitsanbytes to avoid ModuleNotFoundError by @jon-tow in https://github.com/CarperAI/trlx/pull/492 fix type hint in PromptPipeline.init by @g-simmons in https://github.com/CarperAI/trlx/pull/496 fix(modeling_ilql): single q-head indexing by @maxreciprocate in https://github.com/CarperAI/trlx/pull/471 Fix deprecated arguments for Accelerate >= v0.20.0 by @iwiwi in https://github.com/CarperAI/trlx/pull/506 Fix PPO log_ratio bug by @TobiasNorlund in https://github.com/CarperAI/trlx/pull/509 fix(ppo_trainer): default gen kwargs by @maxreciprocate in https://github.com/CarperAI/trlx/pull/510 New Contributors @ZHAOTING made their first contribution in https://github.com/CarperAI/trlx/pull/417 @cauyxy made their first contribution in https://github.com/CarperAI/trlx/pull/409 @Chen9154 made their first contribution in https://github.com/CarperAI/trlx/pull/395 @jovany-wang made their first contribution in https://github.com/CarperAI/trlx/pull/455 @li-plus made their first contribution in https://github.com/CarperAI/trlx/pull/451 @mymusise made their first contribution in https://github.com/CarperAI/trlx/pull/460 @mikljohansson made their first contribution in https://github.com/CarperAI/trlx/pull/420 @g-simmons made their first contribution in https://github.com/CarperAI/trlx/pull/496 @iwiwi made their first contribution in https://github.com/CarperAI/trlx/pull/506 @TobiasNorlund made their first contribution in https://github.com/CarperAI/trlx/pull/509 @glerzing made their first contribution in https://github.com/CarperAI/trlx/pull/486 Full Changelog: https://github.com/CarperAI/trlx/compare/v0.6.0...v0.7.