This is the data package which forms a part of the PyEarthTools package. It contains code for fetching, loading, transforming and working with a wide variety of data sources. It has support for industry standard data sources of common interest, and also has code to aid users in managing their own data in their own projects.
Many research facilities have pre-existing data holdings on disk, and it is not necessary for users to fetch data. On the other hand, many users do need to fetch their own data for their projects. Both situations are catered for, but it's important to bear in mind that this package is catering to a broad and diverse set of user requirements.
The use of the data package within PyEarthTools includes:
- Fetching known data sources (such as ERA5 or ISD)
- Indexing into either pre-existing or newly-fetched data that has been downloaded
- Subsetting and reprocessing that data for efficient storage, access and reprocessing
- Loading that data into memory for efficient use in machine learning
- Performing scientific operations on that data as part of data pre-processing
These tasks are aided by the API presented by the data package. Users looking for "how-to guides" or worked examples should review the Tutorial Gallery.
The rest of this page contains reference information for the components of the Data package. The data API docs can be viewed at Data API Docs.