adding technical integration details for platforms

Author: sunnydean

pages/Integrating New Platforms to EarthCODE/Technical Requirements.md

@@ -0,0 +1,26 @@
+---
+order: 6
+---
+
+# Access
+
+Resource discovery and access are built on a common set of open APIs (see figure below) while remaining adaptable to different platform implementations.
+
+![Resource Discovery & Access](/img/integration/data_access_ecstandards.png)
+
+Catalog search uses the STAC API to perform spatial, temporal and attribute‑based queries against collections and items. Feature‑level retrieval is provided by OGC API – Features (or WFS), and multi‑dimensional data can be accessed via OGC WCS. Cloud‑optimized assets (COG, Zarr) are available for direct download, and visualization layers are exposed through OGC WMS and WMTS. 
+
+### Product Hosting
+
+Data products can be hosted on the ESA Project Results Repository (PRR) or an external repository. Each product must be described using STAC metadata, with stable asset links and appropriate media types for clients to resolve.
+
+For details on the process of storing data, refer to the [Publish](./Publish.md) page.
+
+### Accessing Data from the ESA Projects Results Repository
+
+To see examples of accessing EarthCODE assets from the PRR please see the [relevant tutorial pages](https://esa-earthcode.github.io/tutorials/prr-stac-download-example/).
+
+**Platform Integration**
+The Open Science Catalogue is accessible through open standard interfaces through which the Portal and Platforms integrate for discovery (mainly STAC API) and publishing - note that the integration for publishing currently happens directly via Github. Platforms can make requests to obtain items using their unique Experiment Id or unique Product Id.
+
+![Open Science Interfaces](/img/integration/openscienceinterfaces.png)

pages/Integrating New Platforms to EarthCODE/Technical Requirements/Access.md

@@ -0,0 +1,38 @@
+---
+order: 3
+---
+
+# Develop
+
+### FAIR Open Science Platforms Detailed Requirements
+
+FAIR Open Science Platforms within EarthCODE must provide comprehensive tooling and environments that support researchers across the entire workflow development lifecycle. This includes the following capabilities:
+
+* **Integrated Workflow Development Tools**
+    Platforms must offer environments for creating and editing scientific workflows, such as JupyterLab notebooks, IDEs, or dedicated graphical user interfaces (GUIs). These tools should allow researchers to seamlessly write code, manage software dependencies, and interact with data, ultimately producing formal "Experiment" definitions that link workflows to their specific inputs and configurations.
+
+* **Source Code Management and Automation**
+    To ensure versioning and reproducibility, platforms must integrate with source code management systems like Git. They should provide Continuous Integration/Continuous Deployment (CI/CD) pipelines to automate the process of:
+    * **Building:** Packaging the workflow and its software environment into a standardized container image (e.g., Docker).
+    * **Testing:** Running automated tests to verify the correctness of the code.
+    * **Deploying:** Making the versioned, containerized workflow ready for execution on FAIR Infrastructure Platforms.
+
+* **Workflow Validation**
+    Before execution, platforms must perform validation checks to prevent runtime errors and ensure interoperability. This includes:
+    * **Syntax and Semantic Validation:** Verifying that the workflow definition is valid according to its specified standard (e.g., an openEO Process Graph).
+    * **Compatibility Checks:** Ensuring the workflow is compatible with the target FAIR Infrastructure Platform where it is intended to be run.
+
+* **Reproducibility Checks**
+    To guarantee that experiments can be reproduced, platforms must validate the inputs and configuration before a workflow is published or executed. This involves confirming that all input data references are valid and accessible, and that all configuration parameters are complete and correct for the specified workflow.
+
+* **Monitoring and Observability**
+    Researchers need clear feedback during the development process. Platforms should provide tools for real-time logging, monitoring the status of builds and tests, and observing the behavior of workflows during development runs.
+
+* **Documentation and Support**
+    Platform providers must supply comprehensive documentation covering both user-facing guides and technical integration details. This includes operational guides for scientists and clearly defined Service Level Agreements (SLAs) for the services provided.
+
+
+
+
+
+

pages/Integrating New Platforms to EarthCODE/Technical Requirements/Develop.md

@@ -0,0 +1,66 @@
+---
+order: 7
+---
+
+# Execute
+
+### Workflows and Experiments
+In EarthCODE, a published product is not just a dataset—it is the outcome of a defined process, captured as an **Experiment**. In addition to the definitions above, a product includes a reference to the experiment that produced it, making it possible to understand, reuse, and reproduce the results.
+
+![Workflows](/img/terms/workflow-components.svg)
+
+An Experiment describes everything needed to generate a product. This includes:
+- A human-readable description of the experiment’s purpose and context  
+- A machine-executable workflow used to transform inputs into outputs
+- A definition of the **input data** used, including references to other published products  
+- A **configuration** describing any parameters or settings used during execution  
+
+These components ensure that published results are reproducible. They are defined using structured metadata and described using STAC and Record Items, which together provide the semantic and technical context needed for discovery and reuse. 
+
+A **workflow** defines the set of processing steps used in an experiment. It is not just source code—it is something that can be formally executed within a platform using a defined interface. Workflows may take different forms, including:
+- openEO Process Graphs
+- OGC API Processes (e.g. CWL, Application Package)
+- MLflow models
+- Jupyter Notebooks
+
+The source code that supports a workflow may be referenced, but the workflow itself must be described in a way that allows it to be executed by integrated EarthCODE platforms. This distinction enables reproducibility and compatibility across platforms. Platforms handle these definitions.
+
+Workflows are typically stored in the **EarthCODE GitHub organization**, and referenced in the EarthCODE Open Science Catalog as part of the workflow metadata.
+
+Experiments also declare the **input datasets** used and a **configuration** that defines any parameters passed to the workflow at runtime. Inputs are referenced using unique identifiers, making it easier to validate and re-run experiments with the same data. Configuration values are usually a set of simple name–value pairs, but can vary depending on workflow complexity.
+
+---
+
+In summary, EarthCODE combines the concepts of workflows and products are combined. A **product** is the result of a successfully run experiment. The product metadata links back to the experiment metadata, which in turn references the workflow, input, and config. Together, this structure ensures reproducibility, FAIRness and Openness.
+
+![all together](/img/terms/metadata-components.svg)
+
+
+### Executing Experiments
+
+EarthCODE integrates compute capabilities through standard interfaces that support a variety of processing paradigms. Experiments can be executed as containerized application packages, data‑cube workflows, machine learning inference services or full model training runs—all discoverable and shareable under FAIR principles. Platforms are expected to provide an interface for requesting the execution of one or many of the following.
+
+![alt text](/img/integration/processes_and_analytics_integration.png)
+
+**OGC Application Packages** ([Best Practice for EOAP](https://docs.ogc.org/bp/20-089r1.html)) 
+General purpose algorithms and workflows are delivered as OGC Application Packages. Each package bundles code, dependencies and a defined entry point, and can be deployed on any platform supporting [OGC API – Processes](https://ogcapi.ogc.org/processes/).
+
+**openEO Processing Graphs**  ([openEO API process graphs](https://api.openeo.org/v/0.3.0/processgraphs/))  
+Data cube analytics leverage the openEO API, which accepts a processing graph describing operations on multidimensional datacubes. Platforms that implement the openEO specification execute these graphs close to the hosted data, enabling scalable, server‑side computation.
+
+**Machine Learning Model Execution**  
+Inference workloads can be exposed via OGC API Processes or served through Seldon Core [Seldon Core](https://docs.seldon.io/projects/seldon-core/en/latest/). Models packaged as reusable services respond to standardized process calls, allowing EarthCODE experiments to incorporate AI inference without bespoke integration effort.
+
+**Machine Learning Training**  
+Model development and training runs are managed with ([MLflow](https://mlflow.org/)), which tracks experiments, parameters, metrics and artifacts. By invoking MLflow’s REST API within workflows, platforms provide end‑to‑end support for training, versioning and deployment of machine learning models.
+
+---
+
+To better understand what is expected for making submissions to the catalog, it helps to look at the following examples for Experiments and Products.
+
+:::details Examples of an Experiment Submission to the Open Science Catalog
+
+https://opensciencedata.esa.int/workflows/worldcereal-workflow/record
+
+https://github.com/ESA-EarthCODE/open-science-catalog-metadata/blob/main/experiments/worldcereal-experiment/record.json
+:::

pages/Integrating New Platforms to EarthCODE/Technical Requirements/Execute.md

@@ -0,0 +1,16 @@
+---
+order: 5
+---
+
+# Find
+
+Discoverability in EarthCODE is provided by the Open Science Catalog, which exposes all published resources through a suite of open, standardized APIs for both human and machine access.
+
+### Discovery and Access
+Catalog search uses the STAC API to perform spatial, temporal and attribute‑based queries against collections and items. Feature‑level retrieval is provided by OGC API – Features (or WFS), and multi‑dimensional data can be accessed via OGC WCS. Visualization layers may be exposed through OGC WMS and WMTS.
+
+![Resource Discovery & Access](/img/integration/data_access_ecstandards.png)
+
+#### Core Discovery via STAC API
+
+The primary programmatic interface for searching the catalog is the **STAC API**. This is the main endpoint for machine-to-machine queries. It enables complex spatial, temporal, and attribute-based queries to find relevant data products and experiments. The effectiveness of these searches relies on the rich, standardized metadata described in the [Metadata Definitions](./Metadata-Definitions.md) page.

pages/Integrating New Platforms to EarthCODE/Technical Requirements/Find.md

@@ -0,0 +1,76 @@
+---
+order: 2
+---
+
+# Metadata Definitions
+
+### Data
+In EarthCODE, the final outputs of your research—referred to as Products or data products—are stored, described, and published in a way that ensures long-term FAIRness and availability.
+
+Each product is described using STAC (SpatioTemporal Asset Catalog) metadata through Collections that capture key attributes like the spatial and temporal extent, scientific context, provenance, and more.
+
+With common dictionary and unified metadata across heterogeneous sources, product discovery is facilitated. Published Products are associated to Projects and additional information is provided by tags. Themes and Variables are used as tags to facilitate Products discovery. Earth Observation Satellite Missions used to generate the Product are provided by EO‑Mission description in Product metadata.
+
+![metadata-stac](https://github.com/EOEPCA/open-science-catalog-metadata/assets/120453810/71b8e8a7-9a86-491b-ae54-1fb4de9ccf32)
+
+Platform Integration
+- The formal definition and requirements of the schema can be found at https://github.com/stac-extensions/osc.
+- EarthCODE provides a validation library: https://github.com/ESA-EarthCODE/open-science-catalog-validation
+- Guide for manual submissions (platforms automate these steps): [Contributing to the Open Science Catalog](../../Technical%20Documentation/Open%20Science%20Catalog/Contributing%20to%20the%20Open%20Science%20Catalog.md)
+
+### Workflows and Experiments
+In EarthCODE, a published product is the outcome of a defined process, captured as an Experiment. A product includes a reference to the experiment that produced it.
+
+An Experiment describes everything needed to generate a product, including:
+- A human‑readable description of the experiment’s purpose and context
+- A machine‑executable workflow used to transform inputs into outputs
+- A definition of the input data used, including references to other published products
+- A configuration describing any parameters or settings used during execution
+
+Workflows may take different forms, including openEO Process Graphs, OGC API Processes (e.g. CWL, Application Package), MLflow models, and Jupyter Notebooks. The source code may be referenced, but the workflow must be described in a way that allows execution by integrated EarthCODE platforms.
+
+Workflows are typically stored in the EarthCODE GitHub organization and referenced in the Open Science Catalog. Experiments also declare the input datasets and configuration values used at runtime.
+
+### Validate Products and Experiments
+Before any publication to the Open Science Catalog, products and experiment definitions must pass validation using the open‑science‑catalog‑validation package. See https://github.com/ESA-EarthCODE/open-science-catalog-validation for up‑to‑date documentation.
+
+
+---
+
+
+
+# Metadata Definitions
+
+EarthCODE uses a linked-metadata approach to ensure all scientific outputs are FAIR. This relies on two core, community-driven standards: **STAC** for describing data products and **OGC API - Records** for describing the workflows and experiments that create them. Integrated platforms are responsible for generating metadata that is compliant with these standards and the specific EarthCODE profiles.
+
+### Data Products (STAC)
+
+In EarthCODE, the final outputs of your research—referred to as **Products**—are described and published in a way that ensures long-term FAIRness and availability.
+
+Each product must be described using a **STAC (SpatioTemporal Asset Catalog) Collection**. This metadata captures key attributes like the spatial and temporal extent, scientific context, licensing, and provenance, making the data discoverable and understandable.
+
+To facilitate discovery across diverse datasets, product metadata is enriched using a common dictionary of tags, including `Projects`, `Themes`, `Variables`, and the `EO-Mission` used to generate the data.
+
+![STAC Metadata Structure](https://github.com/EOEPCA/open-science-catalog-metadata/assets/120453810/71b8e8a7-9a86-491b-ae54-1fb4de9ccf32)
+
+#### Platform Requirements for Data Products
+- **Schema Compliance:** Platforms must generate STAC Collection JSON that is compliant with the EarthCODE definitions and terms, which builds upon the Open Science Catalog (OSC) STAC extension. The formal definition can be found at: [https://github.com/stac-extensions/osc](https://github.com/stac-extensions/osc).
+- **Validation:** All generated STAC metadata **must** pass validation using the official EarthCODE validation library before it can be published. See: [https://github.com/ESA-EarthCODE/open-science-catalog-validation](https://github.com/ESA-EarthCODE/open-science-catalog-validation).
+- **Manual Submissions:** While platforms are expected to automate these steps, a guide for manual submissions is available here: [Contributing to the Open Science Catalog](../../Technical%20Documentation/Open%20Science%20Catalog/Contributing%20to%20the%20Open%20Science%20Catalog.md).
+
+### Workflows and Experiments (OGC API - Records)
+
+In EarthCODE, a published product is not just a dataset—it is the outcome of a defined and reproducible process, captured as an **Experiment**. The product's metadata must include a reference to the experiment that produced it.
+
+An Experiment record describes everything needed to regenerate a product, including:
+- A human-readable description of its purpose and context.
+- A link to a machine-executable **Workflow** used to transform inputs into outputs.
+- A definition of the **Input Data** used, including references to other published products.
+- The **Configuration** describing parameters or settings used during execution.
+
+Workflows may take different forms, such as openEO Process Graphs, OGC API Processes (e.g., CWL, Application Package), MLflow models, or Jupyter Notebooks. The source code may be referenced, but the workflow itself must be described in a way that allows it to be formally executed by an integrated EarthCODE platform.
+
+#### Platform Requirements for Workflows and Experiments
+- **Schema Compliance:** Platforms must generate metadata for workflows and experiments that is compliant with the **OGC API - Records** standard and the specific EarthCODE profiles.
+- **Validation:** As with data products, all workflow and experiment metadata **must** pass validation using the [`open-science-catalog-validation`](https://github.com/ESA-EarthCODE/open-science-catalog-validation) package before publication.
+- **Reproducibility:** The generated metadata must contain all necessary information to re-run the experiment, including links to versioned code, container images, input data identifiers, and configuration parameters.