In July 2025, a bug was found that caused some newly created files in Lustre to be ~10x their expected size. This caused issues with users consuming far more than their expected space and not being able to write. No other issue happens to the file except for the unexpected size mismatch; however, the bug did cause increased load and instability on the Lustre file systems. The LC Hotline granted temporary increases for the users who were over quota due to this issue and who contacted the LC Hotline. The bug was found, a fix was applied, and the Lustre systems were updated in the last TOSS update. Following that update (per the dates listed below), no new files should have been created that exhibit the bug.
Zone | Lustre | Date Fixed |
---|---|---|
CZ | lustre1 | 7/29/2025 |
CZ | lustre2 | 7/30/2025 |
CZ | lustre3 | 7/29/2025 |
CZ | lustre5 | 7/30/2025 |
RZ | lustre1 | 7/30/2025 |
SCF | lustre1 | 7/31/2025 |
SCF | lustre2 | 7/31/2025 |
SCF | lustre4 | 7/29/2025 |
Users can request a list of affected files that are in their Lustre directories by sending an email to lc-hotline@llnl.gov requesting this data. Please provide both the zone and explicit path of interest (i.e. please give me a file with the list of file mismatches for SCF /p/lustre1/ines). If a user needs to retain a file and it is causing quota issues, they can copy the file to a new name, delete the original file, then move the new file to the original name
>cp originalFile tempFile >rm originalFile >mv tempFile originalFile
NOTE There is a parallel version of copy that is far faster than cp: https://lc.llnl.gov/confluence/spaces/deg/pages/36831420/mpiFileUtils
LC is working on a script that will allow users to identify their own file mismatches and information will be sent when it is available.