Uproot

Uproot version 5.0.0 <p>Uproot version 5 has a few major new features, one removal (<code>uproot.lazy</code>), and is based on <a href="https://github.com/scikit-hep/awkward/releases/tag/v2.0.0">Awkward Array version 2</a> instead of version 1.</p> uproot.lazy → uproot.dask <p>@kkothari2001 upgraded Uproot from Awkward version 1 to version 2, the major part of which was replacing <code>uproot.lazy</code>, which is based on Awkward 1's virtual and partitioned lazy arrays, with the new Dask collection, <a href="https://github.com/dask-contrib/dask-awkward">dask-awkward</a>. The entry point for this function is <code>uproot.dask</code>.</p> <p>@kkothari2001 also simplified Uproot's Pandas backend, which used to "explode" ragged arrays from ROOT into Pandas DataFrames with a non-trivial MultiIndex. Now, it takes advantage of <a href="https://github.com/intake/awkward-pandas">awkward-pandas</a> to put ragged (and more complex) Awkward Arrays directly into Pandas columns.</p> <p>If you want the old behavior, you can read data using <code>library="ak"</code> to get an Awkward Array, and use <a href="https://awkward-array.org/doc/main/reference/generated/ak.to_dataframe.html">ak.to_dataframe</a> to "explode" the data into a MultiIndex.</p> TTree-reading with AwkwardForth <p>@aryan26roy added a new code path to the TTree-reading routines to read them with AwkwardForth instead of pure Python. Users won't see any <em>interface</em> changes due to this code, but the performance of reading TBranches with <code>AsObject</code> or <code>AsStrings</code> Interpretations should be orders of magnitude faster. For example, <code>std::vector<std::vector<float>></code> reading is now 400× faster.</p> Reading RNTuples <p>@Moelf added a complete reader of RNTuple data with most of an RNTuple-writer in an unmerged pull request (#705). Although the RNTuple format is still in development, this is a very good start at reading RNTuple data, whose structure is a close match to Awkward Arrays (so the translation is more one-to-one than it is for TTrees, for instance).</p> New features <ul> <li>feat: move to hatchling by @henryiii in <a href="https://github.com/scikit-hep/uproot5/pull/688">https://github.com/scikit-hep/uproot5/pull/688</a></li> <li>feat: <code>from_map</code> like optimization for dask arrays by @kkothari2001 in <a href="https://github.com/scikit-hep/uproot5/pull/679">https://github.com/scikit-hep/uproot5/pull/679</a></li> <li>feat: Finalizing AwkwardForth reader for Uproot by @aryan26roy in <a href="https://github.com/scikit-hep/uproot5/pull/644">https://github.com/scikit-hep/uproot5/pull/644</a></li> <li>feat: implemented NON-memberwise deserialization for AsMap. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/746">https://github.com/scikit-hep/uproot5/pull/746</a></li> <li>feat: Added column_projection optimization by @kkothari2001 in <a href="https://github.com/scikit-hep/uproot5/pull/755">https://github.com/scikit-hep/uproot5/pull/755</a></li> <li>feat: support categorical axes on boost histograms by @lobis in <a href="https://github.com/scikit-hep/uproot5/pull/764">https://github.com/scikit-hep/uproot5/pull/764</a></li> <li>feat: warn about TBranch name, alias name conflict. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/776">https://github.com/scikit-hep/uproot5/pull/776</a></li> <li>feat: any Mapping assigned to a WritableDirectory is interpreted as a TTree or failure, no fall-through. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/779">https://github.com/scikit-hep/uproot5/pull/779</a></li> <li>feat: add 'interp_options' mechanism and ak_add_doc. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/784">https://github.com/scikit-hep/uproot5/pull/784</a></li> <li>feat: Use awkward pandas, instead of the existing code that explodes Pandas Dataframes by @kkothari2001 in <a href="https://github.com/scikit-hep/uproot5/pull/734">https://github.com/scikit-hep/uproot5/pull/734</a></li> <li>feat: made 'very optional' arguments keyword-only by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/787">https://github.com/scikit-hep/uproot5/pull/787</a></li> <li>feat: adjust for name change in scikit-hep/awkward#1919. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/788">https://github.com/scikit-hep/uproot5/pull/788</a></li> </ul> Bug-fixes and performance <ul> <li>fix: depend on packaging, not setuptools vendored packaging by @henryiii in <a href="https://github.com/scikit-hep/uproot5/pull/684">https://github.com/scikit-hep/uproot5/pull/684</a></li> <li>fix: Avoid triggering temporary dask-awkward/awkward incompatibility. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/694">https://github.com/scikit-hep/uproot5/pull/694</a></li> <li>fix: Do not write incorrect fSumw2 in histograms (v5). by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/698">https://github.com/scikit-hep/uproot5/pull/698</a></li> <li>fix: Fixes uproot.dask bug with empty branches by @kkothari2001 in <a href="https://github.com/scikit-hep/uproot5/pull/700">https://github.com/scikit-hep/uproot5/pull/700</a></li> <li>fix: Use <code>from_map</code> optimization for delayed numpy arrays and add tests with empty branches for the same by @kkothari2001 in <a href="https://github.com/scikit-hep/uproot5/pull/703">https://github.com/scikit-hep/uproot5/pull/703</a></li> <li>fix: use ctx manager to ensure resources are freed by @agoose77 in <a href="https://github.com/scikit-hep/uproot5/pull/713">https://github.com/scikit-hep/uproot5/pull/713</a></li> <li>fix: ReadOnlyDirectory should provide the largest abs(cycle) when cycle is unspecified, not the largest cycle. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/715">https://github.com/scikit-hep/uproot5/pull/715</a></li> <li>fix: regularize ROOT type aliases to C fundamental type names. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/717">https://github.com/scikit-hep/uproot5/pull/717</a></li> <li>fix: avoid empty TBasket issue in embedded TBasket by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/751">https://github.com/scikit-hep/uproot5/pull/751</a></li> <li>fix: don't use Awkward in test_0751 that doesn't need it by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/753">https://github.com/scikit-hep/uproot5/pull/753</a></li> <li>fix: working TList serialization by @lobis in <a href="https://github.com/scikit-hep/uproot5/pull/763">https://github.com/scikit-hep/uproot5/pull/763</a></li> <li>fix: histogram weights not handled correctly in hist / boost conversion by @lobis in <a href="https://github.com/scikit-hep/uproot5/pull/774">https://github.com/scikit-hep/uproot5/pull/774</a></li> <li>perf: streamline metadata handling for TBranch name lookup and uproot.dask by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/772">https://github.com/scikit-hep/uproot5/pull/772</a></li> <li>fix: ensure AwkwardForth fallback path is tested without history. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/780">https://github.com/scikit-hep/uproot5/pull/780</a></li> <li>fix: all AwkwardForth Forms now agree with awkward_form method output. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/790">https://github.com/scikit-hep/uproot5/pull/790</a></li> <li>fix: Uproot tests now work with Awkward 2.0.0. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/795">https://github.com/scikit-hep/uproot5/pull/795</a></li> </ul> Other <ul> <li>Manually add a Model for TMatrixTSym<double>. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/484">https://github.com/scikit-hep/uproot5/pull/484</a></li> <li>Updating docs and test in response to the removal of uproot.lazy in Uproot5 by @kkothari2001 in <a href="https://github.com/scikit-hep/uproot5/pull/615">https://github.com/scikit-hep/uproot5/pull/615</a></li> <li>changed arguments for awkward_form by @aryan26roy in <a href="https://github.com/scikit-hep/uproot5/pull/617">https://github.com/scikit-hep/uproot5/pull/617</a></li> <li>Completed the Forth based AsStrings reader. by @aryan26roy in <a href="https://github.com/scikit-hep/uproot5/pull/616">https://github.com/scikit-hep/uproot5/pull/616</a></li> <li>Actually pass user-specified 'awkward_form' arguments into context. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/622">https://github.com/scikit-hep/uproot5/pull/622</a></li> <li>Fix annoying gaps in test files. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/625">https://github.com/scikit-hep/uproot5/pull/625</a></li> <li>Cleaning up string generation in streamers.py by @aryan26roy in <a href="https://github.com/scikit-hep/uproot5/pull/629">https://github.com/scikit-hep/uproot5/pull/629</a></li> <li>chore: add dependabot for actions by @henryiii in <a href="https://github.com/scikit-hep/uproot5/pull/631">https://github.com/scikit-hep/uproot5/pull/631</a></li> <li>Bump actions/checkout from 2 to 3 by @dependabot in <a href="https://github.com/scikit-hep/uproot5/pull/632">https://github.com/scikit-hep/uproot5/pull/632</a></li> <li>Bump actions/upload-artifact from 2 to 3 by @dependabot in <a href="https://github.com/scikit-hep/uproot5/pull/633">https://github.com/scikit-hep/uproot5/pull/633</a></li> <li>Bump pypa/gh-action-pypi-publish from 1.4.2 to 1.5.0 by @dependabot in <a href="https://github.com/scikit-hep/uproot5/pull/634">https://github.com/scikit-hep/uproot5/pull/634</a></li> <li>Bump actions/download-artifact from 2 to 3 by @dependabot in <a href="https://github.com/scikit-hep/uproot5/pull/635">https://github.com/scikit-hep/uproot5/pull/635</a></li> <li>Forth based ROOT reader (revised) by @aryan26roy in <a href="https://github.com/scikit-hep/uproot5/pull/636">https://github.com/scikit-hep/uproot5/pull/636</a></li> <li>Set up tests for AsObjects, for the AwkwardForth reader by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/637">https://github.com/scikit-hep/uproot5/pull/637</a></li> <li>Iterate over objects in TDirectory in linear time. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/638">https://github.com/scikit-hep/uproot5/pull/638</a></li> <li>[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in <a href="https://github.com/scikit-hep/uproot5/pull/641">https://github.com/scikit-hep/uproot5/pull/641</a></li> <li>Awkward v2 update by @kkothari2001 in <a href="https://github.com/scikit-hep/uproot5/pull/620">https://github.com/scikit-hep/uproot5/pull/620</a></li> <li>docs: add aryan26roy as a contributor for code by @allcontributors in <a href="https://github.com/scikit-hep/uproot5/pull/645">https://github.com/scikit-hep/uproot5/pull/645</a></li> <li>docs: add kkothari2001 as a contributor for code by @allcontributors in <a href="https://github.com/scikit-hep/uproot5/pull/646">https://github.com/scikit-hep/uproot5/pull/646</a></li> <li>docs: add Moelf as a contributor for code by @allcontributors in <a href="https://github.com/scikit-hep/uproot5/pull/647">https://github.com/scikit-hep/uproot5/pull/647</a></li> <li>chore: cleanup flake8 by @henryiii in <a href="https://github.com/scikit-hep/uproot5/pull/527">https://github.com/scikit-hep/uproot5/pull/527</a></li> <li>[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in <a href="https://github.com/scikit-hep/uproot5/pull/650">https://github.com/scikit-hep/uproot5/pull/650</a></li> <li>[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in <a href="https://github.com/scikit-hep/uproot5/pull/654">https://github.com/scikit-hep/uproot5/pull/654</a></li> <li>Bump pypa/gh-action-pypi-publish from 1.5.0 to 1.5.1 by @dependabot in <a href="https://github.com/scikit-hep/uproot5/pull/656">https://github.com/scikit-hep/uproot5/pull/656</a></li> <li>Primitive Support for RNTuple by @Moelf in <a href="https://github.com/scikit-hep/uproot5/pull/630">https://github.com/scikit-hep/uproot5/pull/630</a></li> <li>Set ReadOnlyDirectory attributes when fSeekKeys == 0 by @kakwok in <a href="https://github.com/scikit-hep/uproot5/pull/660">https://github.com/scikit-hep/uproot5/pull/660</a></li> <li>docs: add kakwok as a contributor for code by @allcontributors in <a href="https://github.com/scikit-hep/uproot5/pull/663">https://github.com/scikit-hep/uproot5/pull/663</a></li> <li>Dask awkward support for uproot.dask by @kkothari2001 in <a href="https://github.com/scikit-hep/uproot5/pull/652">https://github.com/scikit-hep/uproot5/pull/652</a></li> <li>pathlib.Path drops '//' (naturally), but it's sometimes used for URLs by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/670">https://github.com/scikit-hep/uproot5/pull/670</a></li> <li>Gets the number of overflow bins for hist.axis.IntCategory, at least. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/671">https://github.com/scikit-hep/uproot5/pull/671</a></li> <li>Prevent std::pair from being AsStridedObjects. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/673">https://github.com/scikit-hep/uproot5/pull/673</a></li> <li>Implement transformed axis from boost-histogram/hist. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/675">https://github.com/scikit-hep/uproot5/pull/675</a></li> <li>AsDynamic has no self._header. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/674">https://github.com/scikit-hep/uproot5/pull/674</a></li> <li>Fixed TTree write_anew in a subdirectory (consistent caches). by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/677">https://github.com/scikit-hep/uproot5/pull/677</a></li> <li>[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in <a href="https://github.com/scikit-hep/uproot5/pull/657">https://github.com/scikit-hep/uproot5/pull/657</a></li> <li>Implement stl containers for RNTuple by @Moelf in <a href="https://github.com/scikit-hep/uproot5/pull/662">https://github.com/scikit-hep/uproot5/pull/662</a></li> <li>Multiple clusters support for RNTuple by @Moelf in <a href="https://github.com/scikit-hep/uproot5/pull/682">https://github.com/scikit-hep/uproot5/pull/682</a></li> <li>ci: Get test dependencies from one source by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/686">https://github.com/scikit-hep/uproot5/pull/686</a></li> <li>ci: autocancel repeated runs by @henryiii in <a href="https://github.com/scikit-hep/uproot5/pull/685">https://github.com/scikit-hep/uproot5/pull/685</a></li> <li>ci: use mamba by @henryiii in <a href="https://github.com/scikit-hep/uproot5/pull/683">https://github.com/scikit-hep/uproot5/pull/683</a></li> <li>ci: Lint PR titles according to conventional commits by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/689">https://github.com/scikit-hep/uproot5/pull/689</a></li> <li>docs: Installation requirements and error text for dask/dask-awkward in extras.py by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/690">https://github.com/scikit-hep/uproot5/pull/690</a></li> <li>ci: use concurrency group for <code>semantic-pr-title</code> by @agoose77 in <a href="https://github.com/scikit-hep/uproot5/pull/691">https://github.com/scikit-hep/uproot5/pull/691</a></li> <li>docs: Add image for dask docs by @kkothari2001 in <a href="https://github.com/scikit-hep/uproot5/pull/708">https://github.com/scikit-hep/uproot5/pull/708</a></li> <li>ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in <a href="https://github.com/scikit-hep/uproot5/pull/706">https://github.com/scikit-hep/uproot5/pull/706</a></li> <li>chore: dask_awkward.test_utils moved in ContinuumIO/dask-awkward#76 by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/714">https://github.com/scikit-hep/uproot5/pull/714</a></li> <li>test: adjust for boost-histogram 1.3.2's _storage_type deprecation. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/719">https://github.com/scikit-hep/uproot5/pull/719</a></li> <li>ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in <a href="https://github.com/scikit-hep/uproot5/pull/716">https://github.com/scikit-hep/uproot5/pull/716</a></li> <li>add veprbl as a contributor by @allcontributors in <a href="https://github.com/scikit-hep/uproot5/pull/728">https://github.com/scikit-hep/uproot5/pull/728</a></li> <li>add nikoladze as a contributor by @allcontributors in <a href="https://github.com/scikit-hep/uproot5/pull/729">https://github.com/scikit-hep/uproot5/pull/729</a></li> <li>add klieret as a contributor by @allcontributors in <a href="https://github.com/scikit-hep/uproot5/pull/730">https://github.com/scikit-hep/uproot5/pull/730</a></li> <li>add dcervenkov as a contributor by @allcontributors in <a href="https://github.com/scikit-hep/uproot5/pull/731">https://github.com/scikit-hep/uproot5/pull/731</a></li> <li>add beojan as a contributor by @allcontributors in <a href="https://github.com/scikit-hep/uproot5/pull/732">https://github.com/scikit-hep/uproot5/pull/732</a></li> <li>add agoose77 as a contributor by @allcontributors in <a href="https://github.com/scikit-hep/uproot5/pull/733">https://github.com/scikit-hep/uproot5/pull/733</a></li> <li>docs: add a CITATION.cff for Uproot. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/726">https://github.com/scikit-hep/uproot5/pull/726</a></li> <li>chore: drop Python 3.6. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/742">https://github.com/scikit-hep/uproot5/pull/742</a></li> <li>ci: remove GHA workaround for macOS Python 3.11 by @henryiii in <a href="https://github.com/scikit-hep/uproot5/pull/743">https://github.com/scikit-hep/uproot5/pull/743</a></li> <li>chore: some cleanup inpsired by refurb by @henryiii in <a href="https://github.com/scikit-hep/uproot5/pull/745">https://github.com/scikit-hep/uproot5/pull/745</a></li> <li>chore(deps): update pre-commit hooks by @pre-commit-ci in <a href="https://github.com/scikit-hep/uproot5/pull/737">https://github.com/scikit-hep/uproot5/pull/737</a></li> <li>chore(deps): bump amannn/action-semantic-pull-request from 4 to 5 by @dependabot in <a href="https://github.com/scikit-hep/uproot5/pull/757">https://github.com/scikit-hep/uproot5/pull/757</a></li> <li>chore: update Uproot to require Awkward 2.0.0rc1. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/765">https://github.com/scikit-hep/uproot5/pull/765</a></li> <li>ci: update to Python 3.11 final by @henryiii in <a href="https://github.com/scikit-hep/uproot5/pull/766">https://github.com/scikit-hep/uproot5/pull/766</a></li> <li>docs: add lobis as a contributor for code by @allcontributors in <a href="https://github.com/scikit-hep/uproot5/pull/771">https://github.com/scikit-hep/uproot5/pull/771</a></li> <li>refactor: refactor Forth generation by @aryan26roy in <a href="https://github.com/scikit-hep/uproot5/pull/710">https://github.com/scikit-hep/uproot5/pull/710</a></li> <li>chore: remove Identifier and <code>"uproot"</code> parameter. by @jpivarski in <a href="https://github.com/scikit-hep/uproot5/pull/770">https://github.com/scikit-hep/uproot5/pull/770</a></li> <li>ci: [pre-comm