Hello LC Web Services users!

Welcome to the first of what will be several updates per year from the LC Web Services team. This first bulletin hopes to clear the air on a number of topics, some of which you may already know, or know parts of.

LC Web Services

First of all, what are all of the LC Web Services?

LC's Web Services consist of all of the applications that live under the https://lc.llnl.gov (CZ / SCF) and https://rzlc.llnl.gov (RZ) web domains. These include:

  • LC Apache
    • Content Directories
    • Lorenz / MyLC
  • Atlassian
    • Confluence
    • Jira
    • Bitbucket / Bamboo (* we'll come back to these in a bit)
  • GitLab
  • JupyterHub

What's New?

Atlassian

At the beginning of November, the team upgraded the Atlassian tools, bringing new functionality and improved support to those tools. The new versions are:

The next planned Atlassian upgrade is targeted for early 2022.

GitLab

GitLab currently is on a monthly release cadence, with the latest version, GitLab 14.4 being deployed across LC in late October.

There are a number of new features in the GitLab space that we are happy to announce, and we encourage you to check them out and provide feedback as you're able.

  • Web IDE: After some initial issues, the Web IDE in GitLab is now working. Check it out and let us know what you think!
  • Git over HTTPS: There were some outstanding issues with Git over HTTPS which have been resolved recently. With these fixed, you should now be able to use your LC Username and RSA Pin/Token with Git using HTTPS style URLs.
  • Issue Importer: Our newest team member, James Taliaferro, has been working hard on an easy to use web interface to expand upon the simple issue importer provided by GitLab, with feedback provided by you, our users. This tool is useful for projects that wish to move out of Jira, and track their issues in GitLab along side their repositories. For more information, check out our how to guide in Confluence. (NOTE: This feature is currently only deployed to the CZ and RZ, it will be coming soon to the SCF as well.)
  • Service Users: At long last, and with significant effort and leadership from Thomas Mendoza and Neil O'Neill, LC's GitLab instances now support CI jobs that run as a service user! This is a long requested feature from our users, and involved collaboration with GitLab and other partners to develop a custom "Jacamar" runner for our HPC use case. For more details about how to configure your CI jobs to use this new feature, check out the documentation in Confluence.

On that note, it is worth mentioning for those that don't know, that the URL https://gitlab.llnl.gov is an alias for the LC Confluence space where information about GitLab can be found.

October SCF GitLab Outage

As no doubt some of you are aware, the SCF GitLab service experienced intermittent outages between October 22 - 26. These were due to a rapid increase in server disk usage as projects move to GitLab, coupled with a few projects that have been storing large files in their git repositories, causing some repos to balloon to as much as 400GB on disk.

What are we doing about this for the future?

  1. In the short term, we have take steps to grow the server storage available in all GitLab instances where storage was even close to becoming an issue.
  2. The team continues to investigate and deploy new monitoring capabilities to alert us before any issues arise in the future, so that we can take corrective action before you as our users are impacted.
  3. We are investigating placing repository size limitations on GitLab repositories. This is a tricky issue since some projects are quite large by necessity. In general though, GitLab (and really any git repository) is not well designed for things like binary files (especially large ones) due to how it tracks changes and stores those files internally.
    • Note that for comparison, GitHub.com's documentation states that:
      "Ideally, Git repositories should be under 1 GiB, and (without special handling) they start to get unwieldy over 5 GiB. Big repositories take a long time to clone and repack, and take a lot of disk space."

Interested in taking a look at the data about your git repository? Check out the open source tool git-sizer from GitHub themselves that can help analyze your repository. The README in particular includes some excellent documentation to consider when dealing with or contemplating large git repositories.

Deprecating Bitbucket/Bamboo and moving to GitLab

And now for the news that many of you will be the most interested in. This bulletin serves as the official announcement that LC has decided to deprecate its Bitbucket and Bamboo services in favor of moving to GitLab and GitLab CI.

This decision was not made lightly, and GitLab was chosen for a number of compelling reasons including:

  • Overall maturity of the product
  • A better, more sustainable CI solution for our developer model and environment
  • Willingness on the part of GitLab to support the needs of our developers
  • Open source tool, allowing us to directly contribute back to the features we care most about

Path Forward

We have put together a Migrating to GitLab primer on Confluence for those of you that have existing content you will need to migrate forward.

The short version is:

  • If you are needing new code repository or CI services, please start those efforts in GitLab going forward.
  • If you are already using Bitbucket and / or Bamboo today, please start identifying the projects that you would like to carry forward, and migrate them to GitLab.
    • The Confluence page linked above has information about how to accomplish the migration, and the LC Web Services team will support you in making the transition as smooth as possible.

We have worked extensively with code teams to identify any hurdles with transitioning to GitLab, and continue to prioritize any blockers. At the moment though, we are not aware of any issues that should prevent teams from moving to GitLab.

Therefore, it is our hope that teams will be able to migrate off of Bitbucket and Bamboo and over to GitLab and GitLab CI, by the end of CY 2021.

LC will turn off Bitbucket and Bamboo, on all networks (CZ, RZ, SCF) on January 17, 2022. All data left in Bitbucket / Bamboo at that time will be archived at that time.

What about Confluence and Jira?

Many of you have heard a rumor that the lab (including LC) was moving away from Atlassian entirely. We can confirm that this is not true (at least for Livermore Computing). LC is committed to supporting our Confluence and Jira instances, on all networks, into the future.

We will of course continue to monitor the software landscape, and evaluate any new products that could provide better service to our users, however for now, Confluence and Jira are the reigning champions in their domains.

Wrapping Up

And that brings us to the end of the news for the LC Web Services for this article. Again, we hope to make this a more regular (and hopefully shorter) bulletin going forward, but we thought it was important to cover a lot of this ground this first time out.

If you have any questions about anything you've seen here, feel free to reach out to any of the service support lists:

Or, as always, to the LC Hotline at large.

Thank you!

~ LC Web Services Team