diff --git a/docs/source/DeveloperGuide/APIs/index.rst b/docs/source/DeveloperGuide/APIs/index.rst deleted file mode 100644 index dfbabefff8b..00000000000 --- a/docs/source/DeveloperGuide/APIs/index.rst +++ /dev/null @@ -1,67 +0,0 @@ -.. _apis: - -Systems APIs -============= - -Currently, client-server interaction with DIRAC systems is provided by the HTTP services that are coded using the :py:class:`~DIRAC.Core.Tornado.Server.TornadoService` module. -This module implements a fixed interface via HTTP POST requests meant to implement RPC calls, see :ref:`httpsTornado`. -Starting with 8.0, an additional :py:class:`~DIRAC.Core.Tornado.Server.TornadoREST` module was created, which expands the possibilities for writing access interfaces to DIRAC systems by implementing a REST interface. - -``TornadoService`` and ``TornadoREST`` inherit from :py:class:`~DIRAC.Core.Tornado.Server.private.BaseRequestHandler` class the basic logic for authorizing the request, search and run the target method in the thread. - -.. image:: /_static/Systems/APIs/APIClass.png - :width: 400px - :alt: Inheritance of classes. - :align: center - -From the scheme it is intuitively clear that on the basis of BaseRequestHandler it is possible to implement other frameworks. - -What is the purpose of this? ----------------------------- - -The purpose of this component is to build DIRAC system status access interfaces based on HTTP requests outside the fixed RPC request interface described in TornadoServices. - -The main reason for the implementation of this feature is the need to implementation of API for DIRAC Authorization Server based on OAuth2 framework, -which in turn is an integral part of the implementation of OAuth2 DIRAC authorization. -So currently, there is only one API implementation for the Framework system, which is :py:class:`~DIRAC.FrameworkSystem.API.AuthHandler`. -For the same reason, the ability to authorize with an access token (see https://datatracker.ietf.org/doc/html/rfc6750#section-2) was added to the authorization steps. - -How to write APIs ------------------ - -You need to choose the system for which you want to write an API, then create a file with the name of your API in the following path /src/DIRAC//API/Handler.py - -.. literalinclude:: /../../src/DIRAC/Core/Tornado/Server/TornadoREST.py - :start-after: ### Example - :end-before: """ - :caption: Example of the simple system API handler - -.. note:: If you need to implement the interface on a standard 443 https port, you will need to use a balancer, such as nginx - -The example described is likely to be sufficient for most writing cases. But here are some additional features, see :py:class:`~DIRAC.Core.Tornado.Server.private.BaseRequestHandler`: - - - ``USE_AUTHZ_GRANTS`` set the list and order of steps to authorize the request. For example, set ``USE_AUTHZ_GRANTS = ["JWT"]`` to allow access to your endpoint only with a valid access token. - - ``DEFAULT_AUTHORIZATION`` set the authorization requirements. For example, ``DEFAULT_AUTHORIZATION = ['authenticated']`` will allow access only to authenticated users. - - in addition to standard S_OK/S_ERROR you can return text, whether the dictionary for example or nothing, the result will be sent with a 200 status. - - If your API is complex enough and may include, for example, redirection or additional headers, you can use :py:class:`~DIRAC.Core.Tornado.Server.private.BaseRequestHandler.TornadoResponse` - to add all these necessary things, which is thread-safe because TornadoResponse will call your actions outside the thread in which this method is executed: - - .. code-block:: python - - from DIRAC.Core.Tornado.Server.private.BaseRequestHandler import TornadoResponse - - class MyClass(TornadoREST): - - # Describe the class as in the example. - - def web_myMethod(self, my_option): - # Do some thing - response = TornadoResponse(data) - response.set_status(201) - return response - - -The framework takes into account the target method annotations when preparing arguments using the ``inspect`` module, see https://docs.python.org/3/library/inspect.html#inspect.signature. -This means that if you specify an argument type in the annotation, the framework will try to convert the received argument from the request to the specified type. - -.. note:: This component is still quite poor and can be improved by subsequent PRs. diff --git a/docs/source/DeveloperGuide/DevelopmentModel/index.rst b/docs/source/DeveloperGuide/DevelopmentModel/index.rst index 14bcee40be3..e07c9feae37 100644 --- a/docs/source/DeveloperGuide/DevelopmentModel/index.rst +++ b/docs/source/DeveloperGuide/DevelopmentModel/index.rst @@ -1,58 +1,24 @@ .. _development_model: ================================== -Development Model +Development Model for DIRAC ================================== -The DIRACGrid Project is a pure open source project, advanced collectively by a distributed team of -developers working in parallel on the core software as well as on various -extensions. Everybody is welcome to participate. +NB: This document covers DIRAC developments. For DiracX, please check `its documentation `_. + +The DIRACGrid Project is an open source project, advanced collectively by a distributed team of developers. The DIRACGrid project includes several repositories, all hosted in `Github `_: - `DIRAC `_ is the main repository: contains the client and server code - `WebAppDIRAC `_ is the repository for the web portal - `Pilot `_ is *not* a DIRAC extension, but a new version of the DIRAC pilots (dubbed Pilots 3.0) - - `DIRACOS `_ is the repository for the old (py2) DIRAC dependencies - `DIRACOS2 `_ is the repository for the DIRAC dependencies (py3) - - `COMDIRAC `_ is a DIRAC extension of its CLI - `DB12 `_ is *not* a DIRAC extension, but a self-contained quick benchmark - `management `_ is *not* a DIRAC extension, but a repository for creating docker images used for tests and for creating releases - `diraccfg `_ is a stand-alone utility for parsing DIRAC cfg files The content of the other repositories at `https://github.com/DIRACGrid` have either been included in those above, or became obsolete. -This work must be supported by a suitable development model which -is described in this chapter. - -The DIRAC code development is done with the help of the Git code management system. -It is inherently distributed and is well suited for the project. It is outlined in this `guide on github `_ - -The DIRAC Development Model relies on the excellent Git capability for managing -code branches which is mandatory for a distributed team of developers. - DIRAC releases nomenclature follow `PEP 440 `_. - - -Release branches -------------------------- - -The following branches are used in managing DIRAC releases: - -*integration branch* - this branch is used to assemble new code developments that will be eventually released as a new major or - minor release. - -*release branches* - the release branches contain the code corresponding to a given major or minor release. This is the production - code which is distributed for the DIRAC installations. The release branches are created when a new minor - release is done. The patches are incorporated into the release branches. The release branch names have the - form ``rel-vXrY``, where ``vXrY`` part corresponds to the branch minor release version. - -*master branch* - the master branch corresponds to the current stable production code. It is a copy of the corresponding - release branch. - -These branches are the only ones maintained in the central Git repository -by the release managers. They are used to build DIRAC releases. They also serve -as reference code used by developers as a starting point for their work. +DIRAC uses `Trunk Based Development `_ and uses the `integration` branch as trunk. diff --git a/docs/source/DeveloperGuide/OAuth2Authorization/index.rst b/docs/source/DeveloperGuide/OAuth2Authorization/index.rst deleted file mode 100644 index 9ddeb68a65f..00000000000 --- a/docs/source/DeveloperGuide/OAuth2Authorization/index.rst +++ /dev/null @@ -1,285 +0,0 @@ -.. _oauth2_authorization: - -#################### -OAuth2 authorization -#################### - -`OAuth2 `_ authorization is being implemented in DIRAC for transitioning to using access tokens to manage jobs and data, and for simplifying the authorization process for users. - -.. contents:: - -************* -The main goal -************* - -The main goal is to expand the capabilities of DIRAC in terms of interoperability with third-party systems that support OAuth 2 authorization. -Enabling users to access DIRAC resources not only through a proxy certificate, but also with access tokens obtained through Identity Providers. - -******************* -OAuth 2.0 framework -******************* - -The OAuth 2.0 authorization framework is a protocol that allows a user to grant a third-party web site or application access to the user's protected resources, without necessarily revealing their long-term credentials or even their identity. -There are already many articles to familiarize yourself with this framework, for example `Auth0 Docs `_ or `RFCs `_ - -The following diagram shows the main OAuth 2.0 roles in DIRAC. - -.. image:: /_static/Systems/FS/OAuth2/OAuth2Roles.png - :alt: OAuth 2.0 roles in DIRAC (source https://github.com/TaykYoku/DIRACIMGS/raw/main/OAuth2_Roles.ai) - -A feature of DIRAC is the ability to perform user tasks asynchronously on behalf of the user, i.e. using their access token or proxy certificate. - -.. image:: /_static/Systems/FS/OAuth2/DIRACComponentsInteractionRoles.png - :alt: OAuth 2.0 roles in context of the DIRAC components interation (source https://github.com/TaykYoku/DIRACIMGS/raw/main/OAuth2_Roles_ServiceAsClient.ai) - -As shown in the figure, DIRAC server components, such as service or agent, may have sufficient privileges to request a user access token (or proxy). Upon receiving it, the component can access the protected resource on behalf of the user. - -.. warning:: The OAuth 2.0 scheme does not involve the use of X509 certificates, but since their usage is still mandatory in DIRAC, the scheme is more complicated: - the protected resource request may contain the X509 proxy user certificate instead of the user access token. - -OAuth 2.0 roles -=============== - -An OAuth 2.0 flow has the following `roles `_: - - - **Resource Owner** - Entity that can grant access to a protected resource. In the context of DIRAC, these are DIRAC users. - - **Resource Server** - Server hosting the protected resources. In the context of DIRAC, this is DIRAC backend components like a DIRAC services. - - **Client** - Application requesting access to a protected resource on behalf of the *Resource Owner*. In the context of DIRAC, these are DIRAC client installations. The client may also be a DIRAC component, such as a service or agent, that uses a user access token to access DIRAC services. - - **Authorization Server** - Server that authenticates the *Resource Owner* and issues access tokens after getting proper authorization. In the context of DIRAC, this is DIRAC Authorization Server. - -OAuth 2.0 grants -================ - -OAuth 2.0 defines flows to get an access token, called `grant types `_. We use the following flows: - - - `Device Flow `_ to authorize with DIRAC client installation. - - `Authorization Code Flow `_ to authorize with browser. - - `Client Credentials `_ to authorize Web portal and to interact with third party authorization services. - - `Refresh Token `_ to implement long sessions for DIRAC clients and to refresh users access tokens. - - `Token Exchange `_ to get access tokens from third party Identity Providers with scope needed for a particular case. - -.. warning:: DIRAC components can use the host certificate as Client Credentials, which goes beyond the OAuth 2.0 scheme. - -******************* -Involved components -******************* - - - DIRAC **Authorization Server** (AS) - acts as an authorization server for DIRAC clients by providing users with access tokens and proxies - - command line interface (CLI) - commands for creating a DIRAC work session, see :ref:`dirac-login`, :ref:`dirac-logout`, :ref:`dirac-configure`. - - **Authorization API** endpoints - OAuth2 endpoints for authorization, receiving a response from Identity Provider, obtaining an access token, etc., see :py:class:`~DIRAC.FrameworkSystem.API.AuthHandler.AuthHandler`. - - **Token Manager** (TM) service - Service that takes care of storing, updating and obtaining new user access tokens. Similar to the Proxy Manager service, but differs in the specifics of working with tokens. - - **Identity Provider** (IdP) - a type of DIRAC resource that allows you to describe the interaction with third-party services that manage user accounts. - - also the **tornado framework** containing the logic of authorizing client requests to DIRAC components, which in turn act as a resource. - - -.. _dirac_as: - -DIRAC Authorization Server -========================== - -This component is based on the popular `authlib `_ python3 library. -The necessary components for DIRAC Authorization Server to work are collected in a :py:mod:`~DIRAC.FrameworkSystem.private.authorization` subpackage. - -.. image:: /_static/Systems/FS/OAuth2/AuthorizationServerPackage.png - :alt: DIRAC Authorization Server structure in a subpackage (source https://github.com/TaykYoku/DIRACIMGS/raw/main/Authorization_server_structure.ai) - -Components ----------- - - - :py:class:`~DIRAC.FrameworkSystem.private.authorization.grants` contains helper classes with descriptions of the flows to get and revoke an access token. - - :py:class:`~DIRAC.FrameworkSystem.private.authorization.utils` contains helper classes with main OAuth2 object descriptions and helper methods. - - :py:class:`~DIRAC.FrameworkSystem.private.authorization.AuthServer` inherits from `authlib.oauth2.AuthorizationServer` and simulates the operation of OAuth 2 authorization server. - - -Configuration -------------- - -*Authorization Server metadata*: - - DIRAC AS should contain a `metadata `_ that an OAuth client can use to obtain the information needed to interact with DIRAC AS, including its endpoint locations and authorization server capabilities. - But you don't have to worry about that, just define the `/DIRAC/Security/Authorization/issuer` option in the DIRAC configuration, and everything else will be determined for you by the :py:meth:`~DIRAC.FrameworkSystem.private.authorization.utils.Utilities.collectMetadata` method. - -*Authorization clients*: - - OAuth defines two types of `clients `_: - - - confidential clients - - public clients - - DIRAC AS takes both into account and already has a default *public client* (see :py:class:`~DIRAC.FrameworkSystem.private.authorization.utils.Clients`) configured to authorize DIRAC client installations via the device code authorization flow mentioned earlier. - The new `authorization client metadata `_ can be described in the `/DIRAC/Security/Authorization/Clients` section in format:: - - CLIENT_NAME - { - client_id=MY_CLIENT_ID - client_secret=MY_CLIENT_SECRET - scope=supported scopes separated by a space - response_types=device, - grant_types=refresh_token, - } - -*Supported scopes*: - - For DIRAC-specific authorization, support for the following scopes is implemented: - - - `g:` this parametric scope allows you to notify which group the user selects when logging in. - - `proxy` scope informs that the user expects to receive a proxy certificate instead of a token after successful authorization. - - `lifetime:` scope informs how long the proxy should be. - - -Commands -======== - -Two commands were created for interaction with DIRAC AS: - - - :ref:`dirac-login` - - :ref:`dirac-logout` - -Also added the ability to authorize without a certificate while configuring the DIRAC client with the :ref:`dirac-configure` command and a special ``--login`` flag. - - -Authorization API -================= - -With a new system component - :ref:`APIs `, was created Authorization API for *Framework* system (see :py:class:`~DIRAC.FrameworkSystem.API.AuthHandler`) which provides the necessary endpoints for interaction with DIRAC AS. - - -Token Manager -============= - -The Token Manager service aims to capture access tokens and refresh user tokens upon successful authorization and manage them, issue access tokens upon request of DIRAC services or user-owners. - - -Identity Provider -================= - -Since DIRAC is not going to perform the function of user account management, it delegates this function as much as possible to third parties services where VOs should be registered and where there are VO administrators who will deal with it. -Such resources are described as `IdProviders`, see :ref:`resourcesIdProvider`. - - -Tornado Framework -================= - -The framework has also been modified, adding the ability to access DIRAC services using access tokens, see :py:class:`~DIRAC.Core.Tornado.Client.private.TornadoBaseClient.TornadoBaseClient` and :py:class:`~DIRAC.Core.Tornado.Server.private.BaseRequestHandler.BaseRequestHandler`. - -.. note:: to use the received access token to access DIRAC services, you need to add ``/DIRAC/Security/UseTokens=true`` or ``export DIRAC_USE_ACCESS_TOKEN=true``. - - -********** -Logging in -********** - -Consider process by which an user gains access to a DIRAC resources by identifying and authenticating themselves. - -DIRAC CLI -========= - -The ``dirac-login`` command will help us with this. There are three main ways to authorize: - -- using a local user certificate to obtain a proxy certificate -- logging in with DIRAC AS to obtain a proxy certificate -- logging in with DIRAC AS to obtain an access token - - -Using ``dirac-login my_group --use-certificate``: - -.. image:: /_static/Systems/FS/OAuth2/certificateFlow.png - :alt: DIRAC CLI login with certificate flow (source https://raw.githubusercontent.com/TaykYoku/DIRACIMGS/main/component_schema_flows.drawio) - -Using the local certificate ``dirac-login`` makes a similar algorithm as :ref:`dirac-proxy-init`: - 1) Generate a proxy certificate locally on the user's machine from a locally installed user certificate. - #) Try to connect to the DIRAC Configuration Server (CS) with this proxy certificate. - #) If the connection was successful, a command generate a proxy certificate with the required extensions. - #) A proxy certificate without extensions upload to :py:class:`~DIRAC.FrameworkSystem.DB.ProxyDB.ProxyDB` using :py:class:`~DIRAC.FrameworkSystem.Service.ProxyManagerHandler.ProxyManagerHandler`. - -Using ``dirac-login my_group --use-diracas --token``: - -.. image:: /_static/Systems/FS/OAuth2/diracasTokenFlow.png - :alt: DIRAC CLI login DIRAC AS flow and obtaining an access token (source https://raw.githubusercontent.com/TaykYoku/DIRACIMGS/main/component_schema_flows.drawio) - -User do not need to have a locally installed certificate if logging in through DIRAC AS. - - 1) ``dirac-login`` initializes **OAuth 2.0 Device flow** by passing DIRAC client ID to DIRAC AS. - #) DIRAC AS responds with a ``device_code``, ``user_code``, ``verification_uri``, ``verification_uri_complete``, ``expires_in`` (lifetime in seconds for device_code and user_code), and polling ``interval``. - #) The command asks the user to log in using a device that has a browser(e.g.: their computer, smartphone) or if the device running ``dirac-login`` has a browser installed, a new tab with the received URL will open automatically. - - a) The command begins polling DIRAC AS for an access token sending requests to token endpoint until either the user completes the browser flow path or the user code expires. - - 4) After receiving this request from the browser, DIRAC AS will initialize **OAuth 2.0 Authorization Code flow** with choosed IdP. If several IdPs are registered in DIRAC and it is not clear from the requested group which one to choose, DIRAC AS will ask the user to choose one. - #) DIRAC AS prepare authorization URL for the corresponding IdP and redirects the user to the login and authorization prompt. - #) When the user has successfully logged in, IdP redirects him back to the DIRAC AS with an authorization code. - #) DIRAC AS sends this code to the IdP along with the client credentials and recieve an ID token, access token and refresh token. - #) DIRAC AS try to parse received tokens to get the user profile and its ID. - #) Check whether the ID is registered in the DIRAC CS Registry, if not then the authorization process is interrupted and administrators receive a message about an unregistered user. - - a) If the user is registered, :py:class:`~DIRAC.FrameworkSystem.Service.TokenManagerHandler.TokenManagerHandler` stores tokens in :py:class:`~DIRAC.FrameworkSystem.DB.TokenDB.TokenDB`. - #) If ``TokenDB`` already contains tokens for the user, then the extra tokens are revoked (just one refresh token in Token Manager for the user is enough). - - 10) DIRAC AS update authorization session status. - 11) Here we **back to OAuth 2.0 Device flow**. Upon receipt of a request for an access token, DIRAC AS requests :py:class:`~DIRAC.FrameworkSystem.Service.TokenManagerHandler.TokenManagerHandler` to provide a fresh access token to the requested user and group. - - a) Token Manager forms a scope that corresponds to the selected group. - #) After that Token Manager makes aexchange token request to get new access and refresh tokens. - #) DIRAC AS encrypts the refresh token and stores it in :py:class:`~DIRAC.FrameworkSystem.DB.AuthDB.AuthDB`. - #) DIRAC AS responds with an access and encripted refresh token. - -Using ``dirac-login my_group --use-diracas --proxy``: - -.. image:: /_static/Systems/FS/OAuth2/diracasProxyFlow.png - :alt: DIRAC CLI login DIRAC AS flow and obtaining a proxy (source https://raw.githubusercontent.com/TaykYoku/DIRACIMGS/main/component_schema_flows.drawio) - -In this case, the process differs only in that when the user successfully completes the browser flow path, DIRAC AS responds with a proxy: - 11) Upon receipt of a request for a proxy, DIRAC AS requests :py:class:`~DIRAC.FrameworkSystem.Service.ProxyManagerHandler.ProxyManagerHandler` to provide a proxy to the requested user and group. - - a) Proxy Manager see if you need a VOMS extension for the selected group. - #) Proxy Manager makes ``voms-proxy-init`` with the required flags if a VOMS extension is required and add DIRAC group extension. - #) DIRAC AS responds with a proxy. - -Web portal -========== - -.. image:: /_static/Systems/FS/OAuth2/WebAppLoginFlow.png - :alt: DIRAC web login flow (source https://raw.githubusercontent.com/TaykYoku/DIRACIMGS/main/component_schema_flows.drawio) - -The diagram shows the following steps: - 1) The user selects an identity provider for authorization in the web portal. - #) After receiving this request from the browser, the web server creates an authorization session, and redirects the user to DIRAC AS by initiating the **OAuth 2.0 Authorization Code flow**. - #) DIRAC AS will initialize **OAuth 2.0 Authorization Code flow** with choosed IdP. - #) When the user has successfully logged in, DIRAC AS redirects him back to the web server with an authorization code. - #) Web server sends this code to the DIRAC AS along with the client credentials and recieve an access and refresh tokens. - #) The web server creates an http only secure cookie with the received tokens and store an access token in sessionStorage (see https://www.w3schools.com/jsref/prop_win_sessionstorage.asp for more details). This token can be used by JS code from the user's browser (currently not used). - -This scheme is being revised to simplify it. - -*********** -Logging out -*********** - -Consider process by which an user end work session with DIRAC. - -DIRAC CLI -========= - -Using ``dirac-logout``: - -.. image:: /_static/Systems/FS/OAuth2/revokeToken.png - :alt: DIRAC logout flow (source https://raw.githubusercontent.com/TaykYoku/DIRACIMGS/main/component_schema_flows.drawio) - -If it is a long session, i.e. with a refresh token, which allows you to update the access token and thus continue the working session, then to end the session it is necessary to revoke the refresh token: - 1) :ref:`dirac-logout` sends a revoke request to DIRAC AS. - - a) DIRAC AS decrypts the refresh token and reads to whom it belongs. - #) DIRAC AS makes a revoke request to the appropriate IdP. - #) DIRAC AS removes the record about this refresh token from the ``AuthDB`` database. - - 2) Delete the token file. - -Web portal -========== - -Click on the username to select "Log out". - -.. image:: /_static/Systems/FS/OAuth2/revokeTokenWeb.png - :alt: DIRAC web logout flow (source https://raw.githubusercontent.com/TaykYoku/DIRACIMGS/main/component_schema_flows.drawio) - -The web server receives a request from the user's browser to end the session and made revoke refresh token request to DIRAC AS. After that cleans cookies. diff --git a/docs/source/DeveloperGuide/REST/index.rst b/docs/source/DeveloperGuide/REST/index.rst deleted file mode 100644 index 7e0ae489584..00000000000 --- a/docs/source/DeveloperGuide/REST/index.rst +++ /dev/null @@ -1,159 +0,0 @@ -.. _rest_interface: - -REST Interface -================ - -DIRAC has been extended to provide the previously described language agnostic API. -This new API follows the *REST* style over *HTML* using *JSON* as the serialization format. -*OAuth2* is used as the credentials delegation mechanism to the applications. All three -technologies are widely used and have bindings already made for most of today's modern languages. -By providing this new API DIRAC can now be interfaced to any component written in most of -today's modern languages. - -The *REST* interface enpoint is an *HTTPS* server provided in the *RESTDIRAC* module. This -*HTTPS* server requires `Tornado `_. If you don't have it installed just do:: - - pip install -U "tornado>=2.4" - -All requests to the *REST* API are *HTTP* requests. For more info about *REST* take a look -`here `_. From here on a basic -understanding of the HTTP protocol is assumed. - -OAuth2 authentication ------------------------ - -Whenever an application wants to use the API, DIRAC needs to know on behalf of which user -the application is making the request. Users have to grant privileges to the application so -DIRAC knows what to do with the request. Apps have to follow a `OAuth2 `_ -flow to get a token that has user assigned privileges. There are two different flows to get a -token depending on the app having access to the user certificate. Both flows are one or more -*HTTP* queries to the *REST* server. - -* If the app has access to the user certificatea it has to *GET* request to */oauth2/token* using the user certificate as the client certificate. That request has to include as *GET* parameters: - - * *grant_type* set to *client_credentials* - * *group* set to the dirac group the token is being request for. - - * To retrieve a list of valid groups for a certificate, make a *GET* request to */oauth2/groups* using the certificate. - - * *setup* set to the dirac setup the token is being request for. - - * To retrieve a list of valid setups for a certificate, make a *GET* request to */oauth2/setups* using the certificate. - - -* If the app does not have access to the user certificate (for instance a web portal) it has to: - - 1. Redirect the user to */oauth2/auth* passing as *GET* parameters: - - * *response_type* set to *code*. This is a mandatory parameter. - * *client_id* set to the identifier given yo you when the app was registered in DIRAC. This is a mandatory parameter. - * *redirect_uri* set to the URL where the user will be redirected after the request has been authorized. Optional. - * *state* set to any value set by the app to maintain state between the request and the callback. - - 2. Once the user has authorized the request, it will be redirected to the *redirect_uri* defined either in the - request or in the app - registration in DIRAC. The user request will carry the following parameters: - - * *code* set to a temporal token - * *state* set the the original value - - 3. Exchange the *code* token for the final one. Make a *GET* request to */oauth2/token* with: - - * *grant_type* set to *authorization_code*. Mandatory. - * *code* set to the temporal token received by the client. - * *redirect_uri* set to the original *redirect_uri* if it was defined in step 1 - * *client_id* set to the identifier. Same as in step 1. - - 4. Receive access token :) - -From now on. All requests to the *REST* API have to bear the access token either as: - -* *GET* *access_token* parameter -* *Authorization* header with form "tokendata Bearer" - -For more info check out the `OAuth2 draft `_. - -REST API Resources -------------------- - -Once the app has a valid access token, it can use the *REST* API. All data sent or received will be serialized in JSON. - -Job management -*************** - -**GET /jobs** - Retrieve a list of jobs matching the requirements. Parameters: - - * *allOwners*: Show jobs from all owners instead of just the current user. By default is set to *false*. - * *maxJobs*: Maximum number of jobs to retrieve. By default is set to *100*. - * *startJob*: Starting job for the query. By default is set to *0*. - * Any job attribute can also be defined as a restriction in a HTTP list form. For instance:: - - Site=DIRAC.Site.com&Site=DIRAC.Site2.com&Status=Waiting - -**GET /jobs/** - Retrieve info about job with id=*jid* - - -**GET /jobs//manifest** - Retrieve the job manifest - -**GET /jobs//inputsandbox** - Retrieve the job input sandbox - -**GET /jobs//outputsandbox** - Retrieve the job output sandbox - -**POST /jobs** - Submit a job. The API expects a manifest to be sent as a *JSON* object. Files can also be sent as a multipart request. - If files are sent, they will be added to the input sandbox and the manifest will be modified accordingly. An example - of manifest can be:: - - { - Executable: "/bin/echo", - Arguments: "Hello World", - Sites: [ "DIRAC.Site.com", "DIRAC.Site2.com" ] - } - -**DELETE /jobs/** - Kill a job. The user has to have privileges over a job. - -File catalogue -*************** - -All directories that have to be set in a URL have to be encoded in url safe base 64 (RFC 4648 Spec where '+' is -encoded as '-' and '/' is encoded as '_'). There are several implementations for different languages already. - -An example in python of the url safe base 64 encoding would be: - - >>> import base64 - >>> base64.urlsafe_b64encode( "/" ) - 'Lw==' - -Most of the search queries accept a metadata condition. This condition has to be coded as a GET query string of key value pairs. Each key -can be a metadata field and its value has to have the form 'operation|value'. The operation depends on the type of metadata field. For -integers valid operations are '<', '>', '=', '<=', '>=' and the value has to be a number. For string fields the operation has to be 'in' and -the value has to be a comma separared list of possible values. An example would be: - - someNumberField=>|4.2&someStrangeName=in|name1,name2 - -**GET /filecatalogue/metadata** - Retrieve all metadata keys with their type and possible values that are compatible with the metadata restriction. - *Accepts metadata condition* - -**GET /filecatalogue/directory/** - Retrieve contents of the specified directory. Set parameter *verbose* to true to get extended information. - -**GET /filecatalogue/directory//metadata** - Retrieve metadata values for this directory compatible with the metadata condition. - *Accepts metadata condition* - -**GET /filecatalogue/directory//search** - Search from this directory subdirectories that match the requested metadata search. Each directory will also have the amount of files it contains and their total size. - *Accepts metadata condition* - -**GET /filecatalogue/file//attributes** - Get the file information - -**GET /filecatalogue/file//metadata** - Get the file metadata diff --git a/docs/source/DeveloperGuide/index.rst b/docs/source/DeveloperGuide/index.rst index 9d49f03a466..67562c428a1 100644 --- a/docs/source/DeveloperGuide/index.rst +++ b/docs/source/DeveloperGuide/index.rst @@ -33,10 +33,7 @@ For questions, comments, or operational issues, use `GitHub discussions