Fix JRA-55 downloader

[LisaBock]

Description

The downloading works now without RDA login. And the link to the data changed.

• Closes #issue_number
• Link to documentation:

---

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🛠 This pull request has a descriptive title
• 🛠 Code is written according to the code quality guidelines
• 🛠 Documentation is available
• 🛠 Tests run successfully
• 🛠 The list of authors is up to date
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 All checks below this pull request were successful

New or updated data reformatting script

• 🛠 Documentation is available
• 🛠 The dataset has been added to the CMOR check recipe
• 🛠 The dataset has been added to the shared data pools of DKRZ and Jasmin by the @ESMValGroup/OBS-maintainers team
• 🧪 Numbers and units of the data look physically meaningful

---

[jlenh]

I think there might be changes coming as they are migrating their website today (cf. https://rda.ucar.edu/news/rda-gdex-relaunch/). I will have a look tomorrow!

[jlenh]

Thanks Lisa, the new link works perfectly!

I think there are few things to remove still from the old code that are not relevant anymore:

• Removing the temporary files is not needed anymore
  

  ESMValTool/esmvaltool/cmorizers/data/downloaders/datasets/jra_55.py

  Lines 123 to 126 in c31a664

  	if os.path.exists("Authentication.log"):
  	os.remove("Authentication.log")
  	if os.path.exists("auth.rda_ucar_edu"):
  	os.remove("auth.rda_ucar_edu")

• The use of the download_options parameter is not needed anymore
  

  ESMValTool/esmvaltool/cmorizers/data/downloaders/datasets/jra_55.py

  Lines 110 to 112 in c31a664

  	downloader.download_file(
  	url + f"/{channel}/{year}/" + fname, download_options
  	)

[LisaBock]

@jlenh I still have problems with the downloaded data, as the formatter is not working with them. The files are too small. Did you experience the same issue?

[jlenh]

@jlenh I still have problems with the downloaded data, as the formatter is not working with them. The files are too small. Did you experience the same issue?

The formatter raises PermissionError for some of the variables for me on Levante but I still get cmorised outputs. I will try to run it fully in my own data directory to avoid permission issues w/ the data folders on Levante.

[valeriupredoi]

many thanks Lisa 🍻 Ah just noticed the network comms issues - Julien is on it, I see

[jlenh]

I had another look and I cannot open the newly downloaded GRIB files as I always get the following error:
"ERROR: unreadable message"
It indeed looks like the files on the new system are corrupted or incomplete in a way because their size is greatly reduced (a few kb compared to a few Mb).
Maybe we need to switch the downloader to access the files through DIAS at https://search.diasjp.net/en/dataset/JRA55 instead? (but one needs an account there I think)

On the other hand, I think I couldn't run the formatter on the already downloaded data on Levante because of the index files that are written when opening the GRIB files which cannot be done on my side as I do not have writing permission in the ESMValTool data folders.

[jlenh]

Made it work using the following address taken from the web downloader https://osdf-director.osg-htc.org/ncar/gdex/d628001 but I doubt that it is ideal as it always redirects to different addresses.

I have also sent a message to the data host at NSF/NCAR Geoscience Data Exchange (GDEX) for further recommendations.

[schlunma]

Hello, this pull request has been marked with the v2.14.0 milestone. The release of version 2.14.0 is currently scheduled for February 2026. To get this into the new release, it would be great to get this merged by the end of January.

If you won't be able to finish this in time, don't worry - just unassign the milestone v2.14.0. If you need any support, ping myself (@schlunma; the release manager for v2.14.0) or the @ESMValGroup/technical-lead-development-team. Please note that I won't be available until the beginning of February, though.

[jlenh]

Made it work using the following address taken from the web downloader https://osdf-director.osg-htc.org/ncar/gdex/d628001 but I doubt that it is ideal as it always redirects to different addresses.

I have also sent a message to the data host at NSF/NCAR Geoscience Data Exchange (GDEX) for further recommendations.

I would be in favor of proceeding with this option in order to make the data downloadable again, keeping in mind that it would need an update later on (by opening a new issue probably?) as it does not use a supported download address. Haven't gotten an answer from the GDEX support yet.

Thoughts from @ESMValGroup/technical-lead-development-team so that this can potentially be included in the 2.14 release? (I haven't pushed the commit with the modified download URL yet, waiting for feedback first)

Here is an example of the downloading output using the aforementioned address:

--2026-02-09 15:36:30--  https://osdf-director.osg-htc.org/ncar/gdex/d628001/anl_surf125/1958/anl_surf125.011_tmp.195801_195812
Resolving osdf-director.osg-htc.org (osdf-director.osg-htc.org)... 2607:f388:2200:c3::3, 128.105.82.132
Connecting to osdf-director.osg-htc.org (osdf-director.osg-htc.org)|2607:f388:2200:c3::3|:443... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://amst-fiona.nationalresearchplatform.org:8443/ncar/gdex/d628001/anl_surf125/1958/anl_surf125.011_tmp.195801_195812 [following]
--2026-02-09 15:36:31--  https://amst-fiona.nationalresearchplatform.org:8443/ncar/gdex/d628001/anl_surf125/1958/anl_surf125.011_tmp.195801_195812
Resolving amst-fiona.nationalresearchplatform.org (amst-fiona.nationalresearchplatform.org)... 145.100.133.50
Connecting to amst-fiona.nationalresearchplatform.org (amst-fiona.nationalresearchplatform.org)|145.100.133.50|:8443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 752976 (735K)
anl_surf125.011_tmp.195801_195812       100%[===============================================================================>] 735.33K   739KB/s    in 1.0s 

--2026-02-09 15:36:33--  https://osdf-director.osg-htc.org/ncar/gdex/d628001/anl_p125/1958/anl_p125.039_vvel.195801_195812
Resolving osdf-director.osg-htc.org (osdf-director.osg-htc.org)... 2607:f388:2200:c3::3, 128.105.82.132
Connecting to osdf-director.osg-htc.org (osdf-director.osg-htc.org)|2607:f388:2200:c3::3|:443... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://osg-cache.ms4.surfsara.nl:8443/ncar/gdex/d628001/anl_p125/1958/anl_p125.039_vvel.195801_195812 [following]
--2026-02-09 15:36:34--  https://osg-cache.ms4.surfsara.nl:8443/ncar/gdex/d628001/anl_p125/1958/anl_p125.039_vvel.195801_195812
Resolving osg-cache.ms4.surfsara.nl (osg-cache.ms4.surfsara.nl)... 145.38.219.74
Connecting to osg-cache.ms4.surfsara.nl (osg-cache.ms4.surfsara.nl)|145.38.219.74|:8443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27860112 (27M)
anl_p125.039_vvel.195801_195812         100%[===============================================================================>]  26.57M  8.82MB/s    in 3.0s

[jlenh]

Changing the link as discussed, so it's good to go now!

[jlenh]

@schlunma If you want to test this quickly before merging, let me know! Otherwise I will merge later today 😄

[schlunma]

Thanks @jlenh, fine for me if everything works fine and files on Levante have been updated (if necessary) 🚀

[jlenh]

Thanks @jlenh, fine for me if everything works fine and files on Levante have been updated (if necessary) 🚀

I tested again for a few years and it works fine. I just bumped the end_date to 2023 since it is available. I cannot update the files on Levante though

[schlunma]

I can copy them over for you if you point me to your path!