diff --git a/docs/user-guide/assignments/Sail_the_ship.ipynb b/docs/user-guide/assignments/Sail_the_ship.ipynb index 1e1f439d0..b4b94f3dd 100644 --- a/docs/user-guide/assignments/Sail_the_ship.ipynb +++ b/docs/user-guide/assignments/Sail_the_ship.ipynb @@ -157,7 +157,7 @@ "The next step is to finalise the expedition schedule plan, including setting times and instrument selection choices for each waypoint, as well as configuring the ship (including any underway measurement instruments). \n", "\n", "
\n", - "**NOTE**: This section describes the process of finalising the expedition schedule and instrument selection using the `virtualship plan` application. For expeditions with many waypoints, it can become cumbersome to use the planning tool (note, using VirtualShip in a remote terminal / cloud-based environment can also introduce lag in the user-interface). **In this case, you may prefer to edit the** `expedition.yaml` **file directly (see [here](../tutorials/working_with_expedition_yaml.md) for more details on how to do so)**.\n", + "**Note**: This section describes the process of finalising the expedition schedule and instrument selection using the `virtualship plan` application. For expeditions with many waypoints, it can become cumbersome to use the planning tool (note, using VirtualShip in a remote terminal / cloud-based environment can also introduce lag in the user-interface). **In this case, you may prefer to edit the** `expedition.yaml` **file directly (see [here](../tutorials/working_with_expedition_yaml.md) for more details on how to do so)**.\n", "
\n", "\n", "\n", @@ -177,6 +177,22 @@ "\n", "For the underway ADCP, there is a choice of using the 38 kHz OceanObserver or the 300 kHz SeaSeven version (see [here](https://virtualship.readthedocs.io/en/latest/user-guide/assignments/Research_proposal_intro.html#ADCP) for more detail on the two ADCP types).\n", "\n", + "### Instrument/sensor configuration\n", + "\n", + "The most important instrument configuration setting to consider is the list of **sensors** for each instrument, which controls what type of measurements/variables the instrument records in the simulation and therefore what output data you will receive for each instrument.\n", + "\n", + "Sensor lists can be configured for each instrument under _Ship Config Editor_ > _Instrument Configurations_. For example, for the CTD instrument, you can specify which sensors to include in the simulation (e.g., `TEMPERATURE`, `SALINITY`, `OXYGEN`, etc.) by toggling the respective switches on or off.\n", + "\n", + "
\n", + "**Note**: Sensor choices are only relevant for the instruments you plan to deploy as [underway measurements](#underway-measurements) or at waypoints across your expedition schedule [(see below)](#instrument-selection). For example, if you do not select to deploy a CTD at any of your waypoints, the CTD sensor choices will not affect any output data.\n", + "
\n", + "\n", + "
\n", + "**TIP**: See [here](../documentation/full_sensor_list.md) for more information on the sensors available for each instrument.\n", + "
\n", + "\n", + "There are other instrument configurations settings that can be adjusted in the editor as well (e.g. `max_depth` for the CTD), but these are more advanced and in most cases do not need to be changed from the default values.\n", + "\n", "### Waypoint datetimes\n", "\n", "
\n", @@ -200,7 +216,7 @@ "You should now consider which measurements are to be taken at each sampling site (think about those required for your chosen research question), and therefore which instruments need to be selected in the planning tool at each waypoint.\n", "\n", "
\n", - "**Tip**: Click [here](https://virtualship.readthedocs.io/en/latest/user-guide/assignments/Research_proposal_intro.html#Measurement-Options) for more information on what measurement options are available, and a brief introduction to each instrument.\n", + "**Tip**: Click [here](https://virtualship.readthedocs.io/en/latest/user-guide/assignments/Research_proposal_intro.html#Measurement-Options) for more information on which instruments are available in VirtualShip, and a brief introduction to each.\n", "
\n", "\n", "You can make instrument selections for each waypoint in the same sub-panels as the [waypoint time](#waypoint-datetimes) selection by simply switching each on or off. Multiple instruments are allowed at each waypoint.\n", diff --git a/docs/user-guide/documentation/copernicus_products.md b/docs/user-guide/documentation/copernicus_products.md index fec42d442..54737fc6d 100644 --- a/docs/user-guide/documentation/copernicus_products.md +++ b/docs/user-guide/documentation/copernicus_products.md @@ -1,4 +1,4 @@ -# Copernicus Marine products +# Copernicus Marine Service products VirtualShip supports running experiments anywhere in the global ocean from 1993 through to the present day (and approximately two weeks into the future), using the suite of products available from the [Copernicus Marine Data Store](https://data.marine.copernicus.eu/products). @@ -60,7 +60,7 @@ For biogeochemical variables `ph` and `phyc`, monthly products are required for * Certain BGC variables (`ph`, `phyc`) are only available as monthly products in hindcast and hindcast interim periods. ``` -##### CMEMS variables legend +##### Copernicus Marine Service variables legend | Variable Code | Full Variable Name | Category | | :------------ | :------------------------------------------------------------ | :------------- | diff --git a/docs/user-guide/documentation/example_copernicus_download.ipynb b/docs/user-guide/documentation/example_copernicus_download.ipynb index 1e630520f..ae5774f7f 100644 --- a/docs/user-guide/documentation/example_copernicus_download.ipynb +++ b/docs/user-guide/documentation/example_copernicus_download.ipynb @@ -7,7 +7,7 @@ "source": [ "# Example Copernicus data download \n", "\n", - "This notebook provides a rough, non-optimised example of how to download Copernicus Marine data using the `copernicusmarine` Python package.\n", + "This notebook provides a rough, non-optimised example of how to download Copernicus Marine Service data using the `copernicusmarine` [toolbox](https://github.com/mercator-ocean/copernicus-marine-toolbox?tab=readme-ov-file).\n", "\n", "This will download:\n", "- Global bathymetry data (static)\n", diff --git a/docs/user-guide/documentation/full_sensor_list.md b/docs/user-guide/documentation/full_sensor_list.md new file mode 100644 index 000000000..9b22c05d6 --- /dev/null +++ b/docs/user-guide/documentation/full_sensor_list.md @@ -0,0 +1,30 @@ +# Full list of available instrument sensors + +The following table provides a comprehensive list of available sensors for each instrument. These sensors can be specified via the `virtualship plan` tool (see the [Quickstart guide](../quickstart.md)) or in the `sensors` section of the respective instrument configuration in the `expedition.yaml` file (see the [working with expedition.yaml tutorial](../tutorials/working_with_expedition_yaml.md)). + +The source data associated with these instrument sensor variables are described in the [Copernicus Marine products](copernicus_products.md) page. + +```{note} +Trying to add a sensor to an instrument that does not support it will result in errors in VirtualShip. Always refer to this table to check which sensors are available for each instrument. +``` + + + +| Instrument | Sensor Name | Description | Units | Category | +| :------------------------------------- | :----------------- | :---------------------------------------------------------------------------------------- | :----------------------------------- | :------------- | +| **ADCP** | VELOCITY | Current velocities (Eastward Sea Water Velocity (U) and Northward Sea Water Velocity (V)) | m/s | Physical | +| **UNDERWATER_ST** (Ship Underwater ST) | TEMPERATURE | Sea Water Potential Temperature | °C | Physical | +| | SALINITY | Sea Water Salinity | psu | Physical | +| **CTD** | TEMPERATURE | Sea Water Potential Temperature | °C | Physical | +| | SALINITY | Sea Water Salinity | psu | Physical | +| | OXYGEN | Mole Concentration of Dissolved Molecular Oxygen in Sea Water | mmol m-3 | Biogeochemical | +| | CHLOROPHYLL | Mass Concentration of Chlorophyll a in Sea Water | mmol m-3 | Biogeochemical | +| | NITRATE | Mole Concentration of Nitrate in Sea Water | mmol m-3 | Biogeochemical | +| | PHOSPHATE | Mole Concentration of Phosphate in Sea Water | mmol m-3 | Biogeochemical | +| | PH | Sea Water pH | - | Biogeochemical | +| | PHYTOPLANKTON | Mole Concentration of Phytoplankton | mmol m-3 | Biogeochemical | +| | PRIMARY_PRODUCTION | Net Primary Production of Biomass | mmol m-3 day-1 | Biogeochemical | +| **ARGO_FLOAT** | TEMPERATURE | Sea Water Potential Temperature | °C | Physical | +| | SALINITY | Sea Water Salinity | psu | Physical | +| **DRIFTER** | TEMPERATURE | Sea Water Potential Temperature | °C | Physical | +| **XBT** | TEMPERATURE | Sea Water Potential Temperature | °C | Physical | diff --git a/docs/user-guide/documentation/pre_download_data.md b/docs/user-guide/documentation/pre_download_data.md index f37912e9b..d58100cc4 100644 --- a/docs/user-guide/documentation/pre_download_data.md +++ b/docs/user-guide/documentation/pre_download_data.md @@ -1,6 +1,6 @@ # Pre-downloading data -By default, VirtualShip will automatically 'stream' data from the Copernicus Marine Service via the [copernicusmarine toolbox](https://github.com/mercator-ocean/copernicus-marine-toolbox?tab=readme-ov-file). However, for users who wish to manage data locally, it is possible to pre-download the required datasets and feed them into VirtualShip simulations. +By default, VirtualShip will automatically 'stream' data from the Copernicus Marine Service via the `copernicusmarine` [toolbox](https://github.com/mercator-ocean/copernicus-marine-toolbox?tab=readme-ov-file). However, for users who wish to manage data locally, it is possible to pre-download the required datasets and feed them into VirtualShip simulations. @@ -74,7 +74,7 @@ The following assumptions are also made about the data: 1. All pre-downloaded data files must be in NetCDF format (`.nc`). 2. Physical data files must contain the following variables: `uo`, `vo`, `so`, `thetao` - Or these strings must appear as substrings within the variable names (e.g. `uo_glor` is acceptable for `uo`). -3. If using BGC instruments (e.g. `CTD_BGC`), the relevant biogeochemical data files must contain the following variables: `o2`, `chl`, `no3`, `po4`, `nppv`, `ph`, `phyc`. +3. If using BGC-enabled instruments (e.g. BGC variables on the `CTD`), the relevant biogeochemical data files must contain the following variables: `o2`, `chl`, `no3`, `po4`, `nppv`, `ph`, `phyc`. - Or these strings must appear as substrings within the variable names (e.g. `o2_glor` is acceptable for `o2`). 4. Bathymetry data files must contain a variable named `deptho`. diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md index fac1c26cf..24eee0c89 100644 --- a/docs/user-guide/index.md +++ b/docs/user-guide/index.md @@ -14,6 +14,7 @@ assignments/index ```{toctree} :maxdepth: 1 +documentation/full_sensor_list.md documentation/copernicus_products.md documentation/pre_download_data.md documentation/example_copernicus_download.ipynb diff --git a/docs/user-guide/quickstart.md b/docs/user-guide/quickstart.md index cc43b8a0a..7d9841116 100644 --- a/docs/user-guide/quickstart.md +++ b/docs/user-guide/quickstart.md @@ -56,7 +56,7 @@ This will create a folder/directory called `EXPEDITION_NAME` with a single file: For advanced users: it is also possible to run the expedition initialisation step without an MFP .xlsx export file. In this case you should simply run `virtualship init EXPEDITION_NAME` in the CLI. This will write an example `expedition.yaml` file in the `EXPEDITION_NAME` folder/directory. This file contains example waypoints, timings, instrument selections, and ship configuration, but can be edited or propagated through the rest of the workflow unedited to run a sample expedition. ``` -## 3) Expedition scheduling & ship configuration +## 3) Expedition scheduling & configuration ```{important} This section describes the process of finalising the expedition schedule and instrument selection using the `virtualship plan` application. This is the recommended way for most users but when expeditions become larger with many waypoints, it can become cumbersome to use the planning tool (note, using VirtualShip in a remote terminal / cloud-based environment can also introduce lag in the user-interface). **In this case, you may prefer to edit the `expedition.yaml` file directly (see [here](./tutorials/working_with_expedition_yaml.md) for more details on how to do so)**. @@ -82,6 +82,22 @@ VirtualShip is capable of taking underway temperature and salinity measurements, For the underway ADCP, there is a choice of using the 38 kHz OceanObserver or the 300 kHz SeaSeven version (see [here](https://virtualship.readthedocs.io/en/latest/user-guide/assignments/Research_proposal_intro.html#ADCP) for more detail on the two ADCP types). +### Instrument/sensor configuration + +The most important instrument configuration setting to consider is the list of **sensors** for each instrument, which controls what type of measurements/variables the instrument records in the simulation and therefore what output data you will receive for each instrument. + +Sensor lists can be configured for each instrument under _Ship Config Editor_ > _Instrument Configurations_. For example, for the CTD instrument, you can specify which sensors to include in the simulation (e.g., `TEMPERATURE`, `SALINITY`, `OXYGEN`, etc.) by toggling the respective switches on or off. + +```{note} +Sensor choices are only relevant for the instruments you plan to deploy as [underway measurements](#underway-measurements) or at waypoints across your expedition schedule [(see below)](#instrument-selection). For example, if you do not select to deploy a CTD at any of your waypoints, the CTD sensor choices will not affect any output data. +``` + +```{tip} +See [here](../documentation/full_sensor_list.md) for more information on the sensors available for each instrument. +``` + +There are other instrument configurations settings that can be adjusted in the editor as well (e.g. `max_depth` for the CTD), but these are more advanced and in most cases do not need to be changed from the default values. + ### Waypoint datetimes @@ -107,15 +123,11 @@ The MFP route planning tool will give estimated durations of sailing between sit You should now consider which measurements are to be taken at each sampling site, and therefore which instruments need to be selected in the planning tool. ```{tip} -Click [here](https://virtualship.readthedocs.io/en/latest/user-guide/assignments/Research_proposal_intro.html#Measurement-Options) for more information on what measurement options are available, and a brief introduction to each instrument. +Click [here](https://virtualship.readthedocs.io/en/latest/user-guide/assignments/Research_proposal_intro.html#Measurement-Options) for more information on what instrument options are available in VirtualShip, and a brief introduction to each. ``` You can make instrument selections for each waypoint in the same sub-panels as the [waypoint time](#waypoint-datetimes) selection by simply switching each on or off. Multiple instruments are allowed at each waypoint. -```{note} -For advanced users: you can also make further customisations to behaviours of all instruments under _Ship Config Editor_ > _Instrument Configurations_. -``` - ### Save changes When you are happy with your ship configuration and schedule plan, press _Save Changes_. diff --git a/docs/user-guide/tutorials/working_with_expedition_yaml.md b/docs/user-guide/tutorials/working_with_expedition_yaml.md index af059b3f8..6bbf181e6 100644 --- a/docs/user-guide/tutorials/working_with_expedition_yaml.md +++ b/docs/user-guide/tutorials/working_with_expedition_yaml.md @@ -49,8 +49,13 @@ instruments_config: # <-- 2. instrument configuration section num_bins: 40 max_depth_meter: -1000.0 period_minutes: 5.0 + sensors: + - VELOCITY ship_underwater_st_config: period_minutes: 5.0 + sensors: + - TEMPERATURE + - SALINITY argo_float_config: ... ctd_config: ... drifter_config: ... @@ -88,6 +93,27 @@ You can do multiple `DRIFTER` deployments at the same waypoint by adding multipl This section defines the configuration settings for each instrument used in the expedition. Each instrument has its own subsection where specific parameters can be set. +##### Sensors + +For most users, the most important instrument configuration setting to consider is the list of **sensors** for each instrument, which controls what type of measurements/variables the instrument records in the simulation. For example, for the `CTD` instrument, you can specify which sensors to include in the simulation (e.g., `TEMPERATURE`, `SALINITY`, `OXYGEN`, etc.) by adding or removing entries from the `sensors` list in the `ctd_config` section. These must be added on _new lines_ and be in _uppercase_, for example: + +```yaml +ctd_config: + max_depth_meter: -2000.0 + min_depth_meter: -11.0 + stationkeeping_time_minutes: 50.0 + sensors: + - TEMPERATURE + - SALINITY + - OXYGEN +``` + +```{important} +See [here](../documentation/full_sensor_list.md) for a full list of available sensors for each instrument. Trying to add a sensor to an instrument that does not support it will result in errors in VirtualShip. +``` + +##### Underway Instruments + Because **underway instruments** (e.g., ADCP, Ship Underwater ST) collect data continuously while the ship is moving, their deployment is not tied to specific waypoints. Instead, the presence of their configuration sections in `instruments_config` indicates that they will be active throughout the expedition. This means that if you wish to turn off an underway instrument, you can remove its configuration section or simply set it to `null`, for example: ```yaml @@ -96,7 +122,7 @@ instruments_config: ship_underwater_st_config: null ``` -For **all other instruments**, e.g. CTD, ARGO_FLOAT etc., the parameters can often be left as the default values unless advanced customisations are required. +For **all other instruments**, e.g. CTD, ARGO_FLOAT etc., the parameters can often be left as the default values unless further, advanced customisations are required. #### 3. `ship_config` diff --git a/src/virtualship/static/expedition.yaml b/src/virtualship/static/expedition.yaml index 3be45c4d7..acb16dcf0 100644 --- a/src/virtualship/static/expedition.yaml +++ b/src/virtualship/static/expedition.yaml @@ -1,7 +1,5 @@ # see https://virtualship.readthedocs.io/en/latest/user-guide/tutorials/working_with_expedition_yaml.html for more details on how to edit this file # -# TODO: add a link to docs where lists what sensors are supported for each instrument -# schedule: waypoints: - instrument: diff --git a/tests/test_utils.py b/tests/test_utils.py index fde6796f9..2628793df 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -1,4 +1,5 @@ import datetime +import re from pathlib import Path import numpy as np @@ -399,3 +400,71 @@ def test_build_particle_class_scipy_base(): ParticleClass = build_particle_class_from_sensors(sensors, nonsensor, ScipyParticle) assert issubclass(ParticleClass, ScipyParticle) + + +def test_allowed_sensors_matches_docs(): + """Test that SUPPORTED_SENSORS_MAP (sensors allowed for each instrument) matches the sensor table in full_sensor_list.md.""" + # local imports to trigger instrument registration and avoid potential circular imports + import virtualship.instruments # noqa: F401 - ensures all @register_instrument decorators run + from virtualship.utils import INSTRUMENT_CLASS_MAP, SUPPORTED_SENSORS_MAP + + docs_path = ( + Path(__file__).parent.parent + / "docs/user-guide/documentation/full_sensor_list.md" + ) + content = docs_path.read_text(encoding="utf-8") + + display_name_to_instrument_type: dict[str, InstrumentType] = { + instrument_type.value: instrument_type + for instrument_type in INSTRUMENT_CLASS_MAP + if isinstance(instrument_type, InstrumentType) + } # all instruments should use their enum value as the bold display name in the markdown table + + # parse markdown table rows + row_pattern = re.compile(r"^\|([^|]*)\|([^|]*)\|.*$", re.MULTILINE) + + expected: dict[InstrumentType, set[SensorType]] = {} + current_instrument: InstrumentType | None = None + + for match in row_pattern.finditer(content): + instrument_cell = match.group(1).strip() + sensor_cell = match.group(2).strip() + + # extract only the **bold** text from the cell (e.g. "**UNDERWATER_ST** (Ship Underwater ST)" -> "UNDERWATER_ST") + bold_match = re.search(r"\*\*(.+?)\*\*", instrument_cell) + instrument_name = bold_match.group(1).strip() if bold_match else "" + + if instrument_name and instrument_name in display_name_to_instrument_type: + current_instrument = display_name_to_instrument_type[instrument_name] + if current_instrument not in expected: + expected[current_instrument] = set() + + # skip irrelevant cells + if ( + not sensor_cell + or sensor_cell.startswith(":") + or sensor_cell == "Sensor Name" + ): + continue + + sensor_name = sensor_cell.strip() + if current_instrument is not None and sensor_name: + expected[current_instrument].add(SensorType(sensor_name)) + + # verify each instrument in the docs is registered and has matching sensors + for instrument_type, doc_sensors in expected.items(): + assert instrument_type in SUPPORTED_SENSORS_MAP, ( + f"{instrument_type} is listed in full_sensor_list.md but not found in SUPPORTED_SENSORS_MAP." + ) + registered_sensors = set(SUPPORTED_SENSORS_MAP[instrument_type]) + assert registered_sensors == doc_sensors, ( + f"Sensor mismatch for {instrument_type}:\n" + f" In docs: {sorted(s.value for s in doc_sensors)}\n" + f" In code: {sorted(s.value for s in registered_sensors)}\n" + ) + + # verify each instrument registered in code is also covered in the docs + for instrument_type in SUPPORTED_SENSORS_MAP: + assert instrument_type in expected, ( + f"{instrument_type} is registered in SUPPORTED_SENSORS_MAP but not listed in full_sensor_list.md." + )