The LC Python Wheelhouse is transitioning to a shared Nexus instance (https://wci-repo.llnl.gov). Nexus is "a repository manager that organizes, stores and distributes artifacts needed for development." Nexus caches any pip installed packages and shares these installs amongst all users. It is also replicated to the SRD / SCF network once a day, allowing for the same packages available on both networks. In addition to supporting Python pip packages, this cache will serve NPM, Docker, Maven, and other package indexes.
The Nexus instance is a joint effort between LC and the SD Shared Services team. For more information about making use of the other package proxies supported by the WCI Shared Services team, check out their Proxy Usage Configuration documentation.
Workflow Changes
NOTE: If you currently have any configuration in your `~/.config/pip/pip.conf`, `~/.pip/pip.conf`, or similar file, you may need to delete that file or comment out the configuration to avoid issues going forward.
Python users should see minimal changes to their workflow. There is no longer any need to specifically search the LC wheelhouse. Instead, LC has changed system-level pip configuration (/etc/pip.conf) to default to pointing to this Nexus instance. The Nexus instance acts as a "transparent proxy", first serving up packages directly from its cache, before fetching (and caching) from Pypi.org.
Cached packages are mirrored to the SRD network approximately every weekday morning, providing a better turn around time than we previously had with the LC Python Wheelhouse.
Said another way, the new workflow looks a bit like this:
- A user requests a package via a pip install command on the unclassified network (CZ / RZ). If the package is already in the cache, then it’s returned from the Nexus cache.
- If the package is NOT already in the Nexus cache, then the request is transparently (to you) proxied to pypi.org, and the results are cached on Nexus.
- Approximately daily, the entire content of Nexus will be mirrored from the unclassified to SRD networks, and are available to LC SCF systems.
Everything that is currently in the LC Python Wheelhouse has already been synced to Nexus, and we've had several users testing things out to make sure everything is working as expected. If you want to look around for your favorite package specifically, or else just see what is cached, check out the Python group in Nexus.
As always, if you have any questions, please reach out to the LC Hotline so we can assist you.
Thank You to Wheelhouse Contributors
Thank you to all those who have contributed to the Python Wheelhouse over the years. The Python Wheelhouse was originally started mainly for LC to help support our users on LC systems for faster downloads, and also for a standard “not having the user do it” method of getting python packages over to the SCF. We believe that system very much helped our users in their Python development workflows, but it is time that we take a step forward to modernize and consolidate processes.
It’s been a labor of love, but it’s time for something newer and better.