[New Notebook]: Reconcile the RA datasets at NCEI vs data available through IOOS Data Catalog

[MathewBiddle]

Programming Language

Python

Module/Software/IOOS data service?

IOOS Data Catalog CKAN, NCEI Geoportal REST API, ISO xml metadata

Module/Software/IOOS data service?

See the following notebooks:

• https://ioos.github.io/ioos_code_lab/content/code_gallery/data_access_notebooks/2017-06-12-NCEI_RA_archive_history.html
• https://ioos.github.io/ioos_code_lab/content/code_gallery/data_access_notebooks/2024-09-17-CKAN_API_Query.html

Provide a description.

Do some matching between records at NCEI and records available through the IOOS data catalog. Essentially come up with a list of datasets that aren't at NCEI.

@ocefpaf we have published two notebooks in the CodeLab which is a start to this analysis. Linked below. I would just need to smash them together to see what we get.

1. https://ioos.github.io/ioos_code_lab/content/code_gallery/data_access_notebooks/2017-06-12-NCEI_RA_archive_history.html
2. https://ioos.github.io/ioos_code_lab/content/code_gallery/data_access_notebooks/2024-09-17-CKAN_API_Query.html

I think collecting the metadata will be easy. But, trying to match what is at NCEI to what's available from the RAs in the IOOS data catalog will be tricky. At first glance, metadata doesn't match across the two. And the frequency at which we harvest and records change in the data catalog is a moving target.

[MathewBiddle]

This is going to be messy. Jotting down some notes from an initial dive into tracing one dataset from GCOOS.

• Record in OneStop - https://data.noaa.gov/onestop/collections/details/8f999470-ece1-4272-8e64-d2f1f1e1609f
• which is harvested from IOOS Data Catalog - https://data.ioos.us/dataset/walton-smith-ctd-ws22215-ws22215_2022_08_weatherbird_smith_ctd_rp2-2022-08-06-26-5532n-82-2562w
• which is harvested from GCOOS ERDDAP - https://gcoos5.geos.tamu.edu/erddap/info/WS22215_WS22215_2022_08_Weatherbird_Smith_CTD_2/index.html

Full stop. Are those data archived at NCEI?? Maybe?

• Those data might be archived at NCEI in https://www.ncei.noaa.gov/archive/accession/0248626. Which was submitted via OCADS.
• Data might also be archived at NCEI via R2R or OMAO.
• AOML could have already archived the data previously as well.
• When searching NCEI via GeoPortal, we also get the IOOS Data Catalog hit - https://www.ncei.noaa.gov/metadata/geoportal/rest/metadata/item/WS22215_WS22215_2022_08_Weatherbird_Smith_CTD_2/html
• AOML also has their website for these data https://www.aoml.noaa.gov/ocd/ocdweb/wsmith/wsmith_2022.html

For reference, this was digging around for one record. There are >43k records in the IOOS Data catalog...

[MathewBiddle]

CC @ksauby, @apkrelling

[MathewBiddle]

This is somewhat nearly impossible. There is no consistency between what is at NCEI and what is on the IOOS Data Catalog...

I've tried to do some fuzzy matching between titles and that is hit or miss. I've tried to trace one dataset that I know exists in both and it's still a ~90% confidence that they are 1:1.

Not sure where to go from here.

[mwengren]

H @MathewBiddle I'm not sure this is going to help with your immediate goal here, but here's a step in the right direction from my standpoint, as far as IOOS' data providers go anyway:

1. IOOS data providers include both id and naming_authority attributes across their datasets in a consistent way, following IOOS Metadata Profile guidelines in the Dataset Description section (which are identical to ACDD guidelines for both attributes).
2. ERDDAP's ISO output code is updated (xref: erddap/erddap#349) to combine both naming_authority and id when generating the <gmd:fileIdentifier></gmd:fileIdentifier> content of resulting ISO XML (for the 'current' version of ISO 19115-2 NOAA endorses, anyway, which includes this field). This concatenated ID still meets the ISO 19115-2 ISO Metadata Workbook guidance NOAA published in 2012 that specifies "fileIdentifier – A unique phrase or string which uniquely identifies the metadata file". Presently, I believe ERDDAP just uses the dataset name as fileIdentifier, which leads to some collisions in metadata catalogs for IOOS anyway (see below).
3. NCEI archival datasets somehow reference or incorporate some or all of the above concatenated unique identifier in their archival datasets and associated metadata. Not quite sure how this would work, but maybe it would help? At least for datasets that were 1:1 between IOOS Catalog and NCEI/OneStop, you could infer a source dataset ID if the naming_authority + id identifier were preserved in the archived data. I'm sure a 1:1 ratio is definitely not always the case, so how would you map IDs from multiple source datasets aggregated together into a single archived dataset? Comma-separated ID list attribute?

FWIW, these changes would also help avoid collisions of duplicated ISO fileIdentifier values in ERDDAP datasets in metadata catalogs (an issue we have currently within our own IOOS Catalog with so many providers). This may be a catalog-software specific issue - I'm not sure if OneStop cares if it has two datasets with the same fileIdentifier, but I know CKAN does, which affects Data.gov inventories.

There's probably a better way to do all of this involving DOIs I'm not aware of, but I thought the above knowledge dump might be useful anyway.

The main challenge, in any case, rolling out a solution across 43K+ datasets...

[MathewBiddle]

@apkrelling now that you understand the NCEI pipelines a little more, do you think it's possible to identify records that appear in the IOOS Data Catalog (https://data.ioos.us/) which are not yet archived at NCEI?

I was previously using the NCEI geoportal API to gather the metadata records from NCEI to compare with what's found in the IOOS Data Catalog.

The results of this would, ideally, be a list of datasets we need to archive at NCEI.