diff --git a/docs/develop/.pages b/docs/develop/.pages index a11a2dfb9..1d1dbc14b 100644 --- a/docs/develop/.pages +++ b/docs/develop/.pages @@ -2,6 +2,7 @@ nav: - Develop: index.md - Accessing Graphs with Java Applications: accessing-graphs-with-java-applications - Python Plugins: python-plugins + - Marketplace Packages: packages - cmempy - Python API: cmempy-python-api - cmemc - Python Scripts: cmemc-scripts - Build (DataIntegration) APIs: dataintegration-apis diff --git a/docs/develop/index.md b/docs/develop/index.md index c31d88f00..460c35a45 100644 --- a/docs/develop/index.md +++ b/docs/develop/index.md @@ -21,6 +21,12 @@ API documentation and programming recipes. For Python developers, we offer a [Plugin SDK](python-plugins/index.md) as well as an API for accessing and manipulating Corporate Memory Instances ([cmem-cmempy](cmempy-python-api/index.md)). +- :material-package: Packages + + --- + + [Create your own packages](packages/development) to allow easy distribution and reuse of existing content (data, workflows, ...) through [Marketplace Packages](packages/index.md). + - :material-api: OpenAPI specification --- diff --git a/docs/develop/packages/.pages b/docs/develop/packages/.pages new file mode 100644 index 000000000..fd7aaa43e --- /dev/null +++ b/docs/develop/packages/.pages @@ -0,0 +1,5 @@ +nav: + - Packages: index.md + - Installation and Usage: installation + - Development: development + diff --git a/docs/develop/packages/development/index.md b/docs/develop/packages/development/index.md new file mode 100644 index 000000000..9012d7f57 --- /dev/null +++ b/docs/develop/packages/development/index.md @@ -0,0 +1,152 @@ +--- +title: "Marketplace Package Development" +icon: material/code-json +tags: + - Marketplace + - Package +--- +# Marketplace Package Development + +## Introduction + +In order to support the development of packages, we published a [package template](https://github.com/eccenca/cmem-package-template). +Please have a look at this project to get started. + +This page walks you through a basic example of creating a new package, adding different type of content inside it and finally building it into a package archive ready for distribution. + +## Creating your own package + +## Initializing + +Using [described instructions](https://github.com/eccenca/cmem-package-template/tree/main#usage) have a local folder contain your newly templated project (`my-package-id`), for our example we will use : +```shell +🎤 Type of package + Project Package +🎤 Package ID (e.g., 'eccenca-supply-chain-vocab', 'w3c-org-vocab') + my-package-id +🎤 Human-readable package name (e.g., 'My Awesome Vocabulary', 'My Great Project' ..) + My own package +🎤 Short description of the package (e.g., 'A vocabulary for ...', 'A project ...') + My project and graphs +``` + +You should now have a folder with two level of files : + +- Top level — generic package information such as the changelog, README, CI instructions, and licensing. +- Nested folder (my-package-id) — the actual package content, along with a manifest. + +## Package content + +The nested folder `my-package-id` represents your working directory for developing the package. + +To add content to the package simply drag and drop files you want added into this folder or extract existing content from a live Corporate Memory instance to the working directory. + +!!! Example "Extracting Corporate Memory content to add to the package " + + ```shell + cmemc graph export https://my-company.org/queries/ --output-file my-package-id/queries.ttl + + cmemc project export MyProject_78e981443900a761 --output-dir my-package-id + Export project 1/1: MyProject_78e981443900a761 to my-package-id/2026-07-08-unnamed-MyProject_78e981443900a761.project.zip ... done + + mv my-package-id/2026-07-08-unnamed-MyProject_78e981443900a761.project.zip my-package-id/project.zip + ``` + +## Declaring the files in the manifest + +In order for the package to know about these added files, the `cpa-manifest.json` needs to be edited. + +The `"files":[]` section of the manifest references the files the package needs to bundle. +Complete information about the [package manifest can be found here](https://github.com/eccenca/cmem-package-template/tree/main#package-manifest) and more specifically [how to declare new files](https://github.com/eccenca/cmem-package-template/tree/main#adding-files). + +For our example we will be adding a query graph and a project file, make sure you use valid `file_path` relative to your package working directory (nested folder). + +### Adding graph and project + +```json +"files": [ + {...}, + { + "file_path": "queries.ttl", // inside my-package-id nested folder + "file_type": "graph", + "graph_iri": "https://my-company.org/queries/", + "import_into": [], + "register_as_vocabulary": false + }, + { + "file_path": "project.zip", // inside my-package-id nested folder + "file_type": "project", + "project_id": "MyProject_78e981443900a761" + } +] +``` + +## Testing your package + +To ensure the package correctly detects your added files, you can try to import it in a Corporate Memory instance. + +The package template comes with a predefined `Taskfile.yaml` allowing you to wrap your development steps in single commands. +```shell +task: Available tasks for this project: +* build: Build package archive +* check: Run whole test suite +* clean: Removes dist, *.cpa, ... +* delete: Delete (uninstall) package from Corporate Memory +* export: Export package content from Corporate Memory +* import: Import (install) package to Corporate Memory +* publish: Publish package archive to the marketplace +``` + +To tell the package system to take files from the local working directory and to import them inside Corporate Memory we use **task import**. +An import will always try to uninstall previously installed version of the same package, to ensure it is correctly replaced. + +```shell +task import +task: [delete] cmemc package uninstall $package_id +Package 'my-package-id' is not installed. +task: [import] cmemc package install --input $package_dir +Installing package 'my-package-id' from 'my-package-id' ... done +``` + +!!! Warning "Importing duplicated content" + + If you extracted from Corporate Memory already existing content and added it to your package with the same identifiers (graph URIs, projects IDs, ...), and try to import it back in the form of a new package, the instance might raise a `MarketplacePackagesImportError` due to conflicting elements : (Repository item 'https://my-company.org/queries/' already exists.) . + + In this case, you can simply delete your duplicated content inside Corporate Memory (make sure you do backups in case) before importing them back as a package content. + The difference will be that now Corporate Memory will know this content is part of a managed package, and will handle import/export on that file from now on. + +## Updating the package file content + +If you make modifications to your package content in Corporate Memory, the files will not automatically sync back with your local working directory. + +To extract all the updated content from Corporate Memory into your package working directory in a managed way, simply run **task export** + +```shell +task export +``` + +!!! Warning "Exporting without installing first" + + The platform can only export updated versions of package files that were imported at least once before. If you create new information directly in Corporate Memory that the package manifest doesn't yet declare, such as new graphs, you need to manually add them to your working directory and to your manifest and then import them. + + The rule of thumb is : if you need to make structural change to your package that requires for you to edit your manifest, then make sure to run `import` right after to let Corporate Memory keep track of new files. + + Adding a workflow inside a project is not impacted by this limitation, since it is part of the "project" that is managed and tracked by the package. + +## Building your package + +To generate a `.cpa` file ready to be distributed and installed in different Corporate Memory instances you can run **task build**. + +Make sure your local package folder is a git repository with a clean state (the commit hash is used to generate the cpa). +```shell +task build +``` + +To check how this output `.cpa` file can be imported in different places refer to the [Installation and Usage section](../installation/index.md). + +!!! Success "Next steps" + + There are many improvements you can add to your package, such as declaring dependencies to other plugins or packages, to ensure your `.cpa` file can be installed with all it's requirements everywhere, for that you can refer to existing package examples or the template documentation. + + The final step is usually publishing a version of the package to a remote Marketplace repository, to avoid having to manually transfert the `.cpa` archive. This requires you to have publishing permissions on an eccenca Marketplace instance (either a public or private instance). + This can be done with task publish, either manually or from a CI runner. diff --git a/docs/develop/packages/index.md b/docs/develop/packages/index.md new file mode 100644 index 000000000..2d3ae816b --- /dev/null +++ b/docs/develop/packages/index.md @@ -0,0 +1,33 @@ +--- +title: "Marketplace Packages: Overview" +icon: material/package +tags: + - Marketplace + - Package +hide: + - toc +--- +# Marketplace Packages + +A package is a bundle that groups different parts of Corporate Memory configuration — data, workflows and dependencies — letting you share and reuse them across projects, teams, and different Corporate Memory instances. + +A package is packed into a `.cpa` file, a zip-based archive you can either hand off directly or distribute through a Marketplace, a central repository that supports pushing and +pulling packages. + +The following pages give an overview about this feature: + +
+ +- :material-download-circle-outline: [Installation and Usage](installation/index.md) + + --- + + Intended for Linked Data Experts and Deployment Engineers, this page outlines how to install and use existing marketplace packages. + +- :material-code-json: [Development](development/index.md) + + --- + + Intended for Developers, this page explains how to start developing your own packages. + +
diff --git a/docs/develop/packages/installation/example-project.png b/docs/develop/packages/installation/example-project.png new file mode 100644 index 000000000..753ed3baf Binary files /dev/null and b/docs/develop/packages/installation/example-project.png differ diff --git a/docs/develop/packages/installation/example-vocabulary.png b/docs/develop/packages/installation/example-vocabulary.png new file mode 100644 index 000000000..013961a18 Binary files /dev/null and b/docs/develop/packages/installation/example-vocabulary.png differ diff --git a/docs/develop/packages/installation/index.md b/docs/develop/packages/installation/index.md new file mode 100644 index 000000000..62b7c5f67 --- /dev/null +++ b/docs/develop/packages/installation/index.md @@ -0,0 +1,53 @@ +--- +title: "Marketplace Packages: Installation and Usage" +icon: material/download-circle-outline +tags: + - Package + - Marketplace +--- +# Installation and Usage of Marketplace Packages + +## Installation + +If you want to install a marketplace package, you need to use cmemc's package command group. + +You can either install from a remote marketplace or from a locally obtained `.cpa` file + +```shell-session title="Install a package from marketplace" +$ cmemc package install w3c-xsd-vocab +Installing package 'w3c-xsd-vocab' from marketplace ... done +``` + +```shell-session title="Install a package from cpa file:" +$ cmemc package install --replace --input my-package-v0.0.0-4b7516f.cpa +Installing package 'my-package' from 'my-package-v0.0.0-4b7516f.cpa' +done +``` + +## Usage + +Depending on the content types inside it, an installed package can appear in different places in Corporate Memory, with each item (graph, workflows, projects, ...) surfacing in its respective component. + +
+ +!!! info inline "" + + ![Example: Graphs](example-vocabulary.png "Example: Graphs") + +**Graphs** such as data graphs but also **Vocabularies** or **Shapes Catelog** are listed in [**Explore > Graphs**](../../../explore-and-author/graph-exploration/#graphs). + +
+ +
+ +!!! info inline "" + + ![Example: Projects](example-project.png "Example: Projects") + +**Projects** are imported inside [**Build**](../../../build/introduction-to-the-user-interface#projects). +When you install your first project package, Corporate Memory also creates a special project to store all installed files. +This folder is automatically managed by the package system, and removed once the last package is uninstalled. + +
+ +
diff --git a/docs/develop/python-plugins/installation/index.md b/docs/develop/python-plugins/installation/index.md index 7b4a73896..9cfd0f98c 100644 --- a/docs/develop/python-plugins/installation/index.md +++ b/docs/develop/python-plugins/installation/index.md @@ -7,7 +7,7 @@ tags: --- # Installation and Usage of Python Plugins -Plugins are a released as parts of Python packages. +Plugins are released as parts of Python packages. They can but do not need to be open-source and published on [pypi.org](https://pypi.org/search/?q=%22cmem-plugin-%22) (a widely used Python Package Index). One package can contain multiple plugins. ## Installation