JupyterHub and Jupyter Notebooks

Overview

JupyterHub is a tool for launching multiple notebooks as different users. Changes were made to the default JupyterHub in order to launch Jupyter Notebooks that adhere to LC security policies. That said, only launch notebooks through the LC JupyterHub deployment at one of the following links, not on your own:

CZ: https://lc.llnl.gov/jupyter

RZ: https://rzlc.llnl.gov/jupyter

SCF: https://lc.llnl.gov/jupyter

The LC JupyterHub deployment allows you to select a machine on which you have an account and to remotely spawn a notebook on a login node of that machine. Bear in mind that launching a notebook requires an accessible python3 install on the target machine and if none is available, spawning will fail (this mostly applies to some EA systems, for now). Please see the section "Custom Kernels" for more info.

Caveats

Notebook lifetimes are tied to your credential lifetimes (12 hours) and must be relaunched after the time has expired. Work is saved in ".ipynb" files (in your home directory by default) and can be restored on launch.

Notebooks are files essentially equivalent to python files additionally containing output from when it was last run. You can run notebooks that you obtain with other users, but be advised, as with running any code not written by you, exercise caution.

Notebooks are currently launched on LC Login nodes (there are plans to launch kernels in allocations instead). For the moment, please refrain from doing any heavy processing in the notebook itself and instead opt for letting a batch job do that for you.

Again, the JupyterHub deployments that we expose at the URLs above are what you should use to run a notebook. Do not launch notebooks on your own on LC systems.

Custom kernels

JupyterHub, as deployed in LC, ships with a read-only python 3 kernel that includes all installed system packages. For some notebooks, that might be sufficient. If you'd like to install other python packages for a project that you're working on, you'll need a custom kernel. There are 2 options for installing a custom kernel: installing in your home directory or installing into a team project directory (to be shared with a group).

Custom kernel caveats:

By default, custom kernel invocation entails launching the ipykernel_launcher with the python specified in your virtualenv. This does not source your virtualenv (adding the virtualenv /bin directory to your path). As such, referring to "binaries" in your virtualenv or to site-packages from a subprocess launched from a notebook will not work. To remedy this, after installing your custom kernel following the instructions below, open the relevant kernelspec file under `.local/share/jupyter/kernels` and add the "env" option to the kernel spec as shown in the sample JSON file below.

Home directory install:

To set up a kernel in your home directory, run the following from the command line while in $HOME:

# Set up a virtualenv in your home directory (python 2 or 3)
/usr/apps/python-2.7.16/bin/virtualenv --system-site-package my_personal_env
# or
/usr/apps/python-3.7.2/bin/virtualenv --system-site-packages my_personal_env

# Activate your environment (bash)
source my_personal_env/bin/activate

# Or activate your environment (csh)
source my_personal_env/bin/activate.csh

# Install ipython kernel
# Note on the SCF, you may need to specify ipykernel==4.10.0
python -m pip install ipykernel
# or
python3 -m pip install ipykernel

# Install your custom kernel to .local in your home directory
python -m ipykernel install --prefix=$HOME/.local/ --name 'some-unique-name' --display-name 'My awesome kernel'
# or
python3 -m ipykernel install --prefix=$HOME/.local/ --name 'some-unique-name' --display-name 'My awesome kernel'

# "--name" must be unique among your kernels and "--display-name" is what you'll see when selecting a new kernel

 

Custom Kernel ENV

Ensure that a JSON file with the following format exists in `.local/share/jupyter/kernels` for your new custom kernel:

{
 "display_name": "Python 2", 
 "language": "python",
 "_comment": "NOTE: variable expansion _not_ supported",
 "env": {
  "PATH": "/g/g0/user/python2-venv/bin:/existing/path/here:..."
 },
 "argv": [
  "/g/g0/user/python2-venv/bin/python", 
  "-m", 
  "ipykernel_launcher", 
  "-f", 
  "{connection_file}"
 ]
}

Once you finish installing your custom kernel, it will be available as on option on JupyterHub when you create a new notebook.

Nonstandard kernels requiring additional/different steps:

Matlab kernel

Julia kernel

R kernel

For powerai installations and tensorflow, please refer to IBM Power AI in LC - JupyterHub Custom Kernel

Troubleshooting

As spawning on various hosts relies on ssh, which will source files like your `.bashrc` file, be sure to guard against output producing commands as they will affect notebook startup. For example you could create an interactive bash guard in your  `.bashrc` file via

if [[ $- == *i* ]]
then
    # PLACE YOUR OUTPUT PRODUCING COMMANDS HERE
fi

Other common issues

Problem: "Service Unavailable"

Explanation: JupyterHub is trying to connect to a notebook that is no longer available or may have errored out in startup

Solution: Change your url to https://lc.llnl.gov/jupyter/hub/home or https://rzlc.llnl.gov/jupyter/hub/home to get to the notebook management page, click the red "Stop My Server" button, and try spawning on a different host by clicking on the green "My Server" button.

Problem: Experiencing 504 gateway timeout when spawning

Explanation: This is either due to the notebook being unable to complete the OAuth flow as a result of a firewall block or an unresponsive notebook server.

Solution: Change your url to https://lc.llnl.gov/jupyter/hub/home or https://rzlc.llnl.gov/jupyter/hub/home to get to the notebook management page, click the red "Stop My Server" button, and try spawning on a different host by clicking on the green "My Server" button.

Problem: SSHSpawnException: jupyterhub-singleuser command not found

Explanation: Some hosts do not have an appropriate python environment yet.

Solution: For now, launching on that host will be unavailable until a global python3 environment is available to install the necessary libraries to run our custom Jupyter notebook.

Problem: SSHSpawnException: connection refused, too many failed login attempts for user <user>

Explanation: The Jupyter login cookie and LC SSO token are out of sync.

Solution: This will be a short-lived issue. The production Jupyter instances have been updated to tie Jupyter login cookies to the same lifetime and session as your standard LC login. If you come across this, clear your cookies for either https://[rz]lc.llnl.gov and log in again.

LLNL-WEB-822441