diff --git a/docs/user-guide/assignments/Sail_the_ship.ipynb b/docs/user-guide/assignments/Sail_the_ship.ipynb index 1e1f439d0..b4b94f3dd 100644 --- a/docs/user-guide/assignments/Sail_the_ship.ipynb +++ b/docs/user-guide/assignments/Sail_the_ship.ipynb @@ -157,7 +157,7 @@ "The next step is to finalise the expedition schedule plan, including setting times and instrument selection choices for each waypoint, as well as configuring the ship (including any underway measurement instruments). \n", "\n", "
\n", - "**NOTE**: This section describes the process of finalising the expedition schedule and instrument selection using the `virtualship plan` application. For expeditions with many waypoints, it can become cumbersome to use the planning tool (note, using VirtualShip in a remote terminal / cloud-based environment can also introduce lag in the user-interface). **In this case, you may prefer to edit the** `expedition.yaml` **file directly (see [here](../tutorials/working_with_expedition_yaml.md) for more details on how to do so)**.\n", + "**Note**: This section describes the process of finalising the expedition schedule and instrument selection using the `virtualship plan` application. For expeditions with many waypoints, it can become cumbersome to use the planning tool (note, using VirtualShip in a remote terminal / cloud-based environment can also introduce lag in the user-interface). **In this case, you may prefer to edit the** `expedition.yaml` **file directly (see [here](../tutorials/working_with_expedition_yaml.md) for more details on how to do so)**.\n", "
\n", "\n", "\n", @@ -177,6 +177,22 @@ "\n", "For the underway ADCP, there is a choice of using the 38 kHz OceanObserver or the 300 kHz SeaSeven version (see [here](https://virtualship.readthedocs.io/en/latest/user-guide/assignments/Research_proposal_intro.html#ADCP) for more detail on the two ADCP types).\n", "\n", + "### Instrument/sensor configuration\n", + "\n", + "The most important instrument configuration setting to consider is the list of **sensors** for each instrument, which controls what type of measurements/variables the instrument records in the simulation and therefore what output data you will receive for each instrument.\n", + "\n", + "Sensor lists can be configured for each instrument under _Ship Config Editor_ > _Instrument Configurations_. For example, for the CTD instrument, you can specify which sensors to include in the simulation (e.g., `TEMPERATURE`, `SALINITY`, `OXYGEN`, etc.) by toggling the respective switches on or off.\n", + "\n", + "
\n", + "**Note**: Sensor choices are only relevant for the instruments you plan to deploy as [underway measurements](#underway-measurements) or at waypoints across your expedition schedule [(see below)](#instrument-selection). For example, if you do not select to deploy a CTD at any of your waypoints, the CTD sensor choices will not affect any output data.\n", + "
\n", + "\n", + "
\n", + "**TIP**: See [here](../documentation/full_sensor_list.md) for more information on the sensors available for each instrument.\n", + "
\n", + "\n", + "There are other instrument configurations settings that can be adjusted in the editor as well (e.g. `max_depth` for the CTD), but these are more advanced and in most cases do not need to be changed from the default values.\n", + "\n", "### Waypoint datetimes\n", "\n", "
\n", @@ -200,7 +216,7 @@ "You should now consider which measurements are to be taken at each sampling site (think about those required for your chosen research question), and therefore which instruments need to be selected in the planning tool at each waypoint.\n", "\n", "
\n", - "**Tip**: Click [here](https://virtualship.readthedocs.io/en/latest/user-guide/assignments/Research_proposal_intro.html#Measurement-Options) for more information on what measurement options are available, and a brief introduction to each instrument.\n", + "**Tip**: Click [here](https://virtualship.readthedocs.io/en/latest/user-guide/assignments/Research_proposal_intro.html#Measurement-Options) for more information on which instruments are available in VirtualShip, and a brief introduction to each.\n", "
\n", "\n", "You can make instrument selections for each waypoint in the same sub-panels as the [waypoint time](#waypoint-datetimes) selection by simply switching each on or off. Multiple instruments are allowed at each waypoint.\n", diff --git a/docs/user-guide/documentation/full_sensor_list.md b/docs/user-guide/documentation/full_sensor_list.md new file mode 100644 index 000000000..7e108b67d --- /dev/null +++ b/docs/user-guide/documentation/full_sensor_list.md @@ -0,0 +1,28 @@ +# Full list of available instrument sensors + +The following table provides a comprehensive list of available sensors for each instrument. These sensors can be specified via the `virtualship plan` tool (see the [Quickstart guide](../quickstart.md)) or in the `sensors` section of the respective instrument configuration in the `expedition.yaml` file (see the [working with expedition.yaml tutorial](../tutorials/working_with_expedition_yaml.md)). + +```{note} +Trying to add a sensor to an instrument that does not support it will result in errors in VirtualShip. Always refer to this table to check which sensors are available for each instrument. +``` + + + +| Instrument | Sensor Name | Description | Units | Category | +| :------------------------------------- | :----------------- | :-------------------------------------------------- | :----------------------------------- | :-------------- | +| **ADCP** | VELOCITY | Current velocities (eastward (u) and northward (v)) | m/s | Physical | +| **UNDERWATER_ST** (Ship Underwater ST) | TEMPERATURE | Temperature | °C | Physics | +| | SALINITY | Salinity | psu | Physics | +| **CTD** | TEMPERATURE | Temperature | °C | Physics | +| | SALINITY | Salinity | psu | Physics | +| | OXYGEN | Oxygen concentration | mmol m-3 | Biogeochemistry | +| | CHLOROPHYLL | Chlorophyll concentration | mmol m-3 | Biogeochemistry | +| | NITRATE | Nitrate concentration | mmol m-3 | Biogeochemistry | +| | PHOSPHATE | Phosphate concentration | mmol m-3 | Biogeochemistry | +| | PH | pH | - | Biogeochemistry | +| | PHYTOPLANKTON | Phytoplankton concentration in carbon | mmol m-3 | Biogeochemistry | +| | PRIMARY_PRODUCTION | Net primary production | mmol m-3 day-1 | Biogeochemistry | +| **ARGO_FLOAT** | TEMPERATURE | Temperature | °C | Physics | +| | SALINITY | Salinity | psu | Physics | +| **DRIFTER** | TEMPERATURE | Temperature | °C | Physics | +| **XBT** | TEMPERATURE | Temperature | °C | Physics | diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md index fac1c26cf..d5d4a124d 100644 --- a/docs/user-guide/index.md +++ b/docs/user-guide/index.md @@ -17,4 +17,5 @@ assignments/index documentation/copernicus_products.md documentation/pre_download_data.md documentation/example_copernicus_download.ipynb +documentation/full_sensor_list.md ``` diff --git a/docs/user-guide/quickstart.md b/docs/user-guide/quickstart.md index cc43b8a0a..7d9841116 100644 --- a/docs/user-guide/quickstart.md +++ b/docs/user-guide/quickstart.md @@ -56,7 +56,7 @@ This will create a folder/directory called `EXPEDITION_NAME` with a single file: For advanced users: it is also possible to run the expedition initialisation step without an MFP .xlsx export file. In this case you should simply run `virtualship init EXPEDITION_NAME` in the CLI. This will write an example `expedition.yaml` file in the `EXPEDITION_NAME` folder/directory. This file contains example waypoints, timings, instrument selections, and ship configuration, but can be edited or propagated through the rest of the workflow unedited to run a sample expedition. ``` -## 3) Expedition scheduling & ship configuration +## 3) Expedition scheduling & configuration ```{important} This section describes the process of finalising the expedition schedule and instrument selection using the `virtualship plan` application. This is the recommended way for most users but when expeditions become larger with many waypoints, it can become cumbersome to use the planning tool (note, using VirtualShip in a remote terminal / cloud-based environment can also introduce lag in the user-interface). **In this case, you may prefer to edit the `expedition.yaml` file directly (see [here](./tutorials/working_with_expedition_yaml.md) for more details on how to do so)**. @@ -82,6 +82,22 @@ VirtualShip is capable of taking underway temperature and salinity measurements, For the underway ADCP, there is a choice of using the 38 kHz OceanObserver or the 300 kHz SeaSeven version (see [here](https://virtualship.readthedocs.io/en/latest/user-guide/assignments/Research_proposal_intro.html#ADCP) for more detail on the two ADCP types). +### Instrument/sensor configuration + +The most important instrument configuration setting to consider is the list of **sensors** for each instrument, which controls what type of measurements/variables the instrument records in the simulation and therefore what output data you will receive for each instrument. + +Sensor lists can be configured for each instrument under _Ship Config Editor_ > _Instrument Configurations_. For example, for the CTD instrument, you can specify which sensors to include in the simulation (e.g., `TEMPERATURE`, `SALINITY`, `OXYGEN`, etc.) by toggling the respective switches on or off. + +```{note} +Sensor choices are only relevant for the instruments you plan to deploy as [underway measurements](#underway-measurements) or at waypoints across your expedition schedule [(see below)](#instrument-selection). For example, if you do not select to deploy a CTD at any of your waypoints, the CTD sensor choices will not affect any output data. +``` + +```{tip} +See [here](../documentation/full_sensor_list.md) for more information on the sensors available for each instrument. +``` + +There are other instrument configurations settings that can be adjusted in the editor as well (e.g. `max_depth` for the CTD), but these are more advanced and in most cases do not need to be changed from the default values. + ### Waypoint datetimes @@ -107,15 +123,11 @@ The MFP route planning tool will give estimated durations of sailing between sit You should now consider which measurements are to be taken at each sampling site, and therefore which instruments need to be selected in the planning tool. ```{tip} -Click [here](https://virtualship.readthedocs.io/en/latest/user-guide/assignments/Research_proposal_intro.html#Measurement-Options) for more information on what measurement options are available, and a brief introduction to each instrument. +Click [here](https://virtualship.readthedocs.io/en/latest/user-guide/assignments/Research_proposal_intro.html#Measurement-Options) for more information on what instrument options are available in VirtualShip, and a brief introduction to each. ``` You can make instrument selections for each waypoint in the same sub-panels as the [waypoint time](#waypoint-datetimes) selection by simply switching each on or off. Multiple instruments are allowed at each waypoint. -```{note} -For advanced users: you can also make further customisations to behaviours of all instruments under _Ship Config Editor_ > _Instrument Configurations_. -``` - ### Save changes When you are happy with your ship configuration and schedule plan, press _Save Changes_. diff --git a/docs/user-guide/tutorials/working_with_expedition_yaml.md b/docs/user-guide/tutorials/working_with_expedition_yaml.md index af059b3f8..6bbf181e6 100644 --- a/docs/user-guide/tutorials/working_with_expedition_yaml.md +++ b/docs/user-guide/tutorials/working_with_expedition_yaml.md @@ -49,8 +49,13 @@ instruments_config: # <-- 2. instrument configuration section num_bins: 40 max_depth_meter: -1000.0 period_minutes: 5.0 + sensors: + - VELOCITY ship_underwater_st_config: period_minutes: 5.0 + sensors: + - TEMPERATURE + - SALINITY argo_float_config: ... ctd_config: ... drifter_config: ... @@ -88,6 +93,27 @@ You can do multiple `DRIFTER` deployments at the same waypoint by adding multipl This section defines the configuration settings for each instrument used in the expedition. Each instrument has its own subsection where specific parameters can be set. +##### Sensors + +For most users, the most important instrument configuration setting to consider is the list of **sensors** for each instrument, which controls what type of measurements/variables the instrument records in the simulation. For example, for the `CTD` instrument, you can specify which sensors to include in the simulation (e.g., `TEMPERATURE`, `SALINITY`, `OXYGEN`, etc.) by adding or removing entries from the `sensors` list in the `ctd_config` section. These must be added on _new lines_ and be in _uppercase_, for example: + +```yaml +ctd_config: + max_depth_meter: -2000.0 + min_depth_meter: -11.0 + stationkeeping_time_minutes: 50.0 + sensors: + - TEMPERATURE + - SALINITY + - OXYGEN +``` + +```{important} +See [here](../documentation/full_sensor_list.md) for a full list of available sensors for each instrument. Trying to add a sensor to an instrument that does not support it will result in errors in VirtualShip. +``` + +##### Underway Instruments + Because **underway instruments** (e.g., ADCP, Ship Underwater ST) collect data continuously while the ship is moving, their deployment is not tied to specific waypoints. Instead, the presence of their configuration sections in `instruments_config` indicates that they will be active throughout the expedition. This means that if you wish to turn off an underway instrument, you can remove its configuration section or simply set it to `null`, for example: ```yaml @@ -96,7 +122,7 @@ instruments_config: ship_underwater_st_config: null ``` -For **all other instruments**, e.g. CTD, ARGO_FLOAT etc., the parameters can often be left as the default values unless advanced customisations are required. +For **all other instruments**, e.g. CTD, ARGO_FLOAT etc., the parameters can often be left as the default values unless further, advanced customisations are required. #### 3. `ship_config` diff --git a/pixi.toml b/pixi.toml index ba0249687..b86b8f60a 100644 --- a/pixi.toml +++ b/pixi.toml @@ -18,13 +18,12 @@ setuptools = "*" setuptools_scm = "*" [package.run-dependencies] # Keep in sync with `pyproject.toml` and feedstock recipe -python = ">=3.10" +python = "3.11.*" click = "*" -parcels = ">3.1.0" pyproj = ">=3,<4" sortedcontainers = "==2.4.0" opensimplex = "==0.4.5" -numpy = ">=1,<2" +numpy = ">=2.1.0" pydantic = ">=2,<3" pyyaml = "*" copernicusmarine = ">=2.2.2" @@ -33,15 +32,23 @@ textual = "*" [dependencies] virtualship = { path = "." } +# Pre-install as conda packages to avoid PyPI source builds +netcdf4 = "*" +numpy = ">=2.1.0" +dask = "*" -[feature.py310.dependencies] -python = "3.10.*" +[pypi-dependencies] +parcels = { git = "https://github.com/Parcels-code/Parcels", branch = "main" } -[feature.py311.dependencies] -python = "3.11.*" +# Commented out whilst parcels v4 alpha only supports Python 3.11 +# [feature.py310.dependencies] +# python = "3.10.*" + +# [feature.py311.dependencies] +# python = "3.11.*" -[feature.py312.dependencies] -python = "3.12.*" +# [feature.py312.dependencies] +# python = "3.12.*" [feature.test.dependencies] pytest = "*" @@ -98,11 +105,8 @@ lxml = "*" typing = "mypy src/virtualship --install-types" [environments] -default = { features = ["test", "notebooks", "typing", "pre-commit", "analysis"] } +default = { features = ["test", "notebooks", "typing", "pre-commit", "analysis"] } test-latest = { features = ["test"], solve-group = "test" } -test-py310 = { features = ["test", "py310"] } -test-py311 = { features = ["test", "py311"] } -test-py312 = { features = ["test", "py312"] } test-notebooks = { features = ["test", "notebooks"], solve-group = "test" } analysis = { features = ["analysis"], solve-group = "analysis" } docs = { features = ["docs"], solve-group = "docs" } diff --git a/pyproject.toml b/pyproject.toml index 7f9a2108a..e19024d3b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,7 +7,7 @@ name = "virtualship" description = "Code for the Virtual Ship Classroom, where Marine Scientists can combine Copernicus Marine Data with an OceanParcels ship to go on a virtual expedition." readme = "README.md" dynamic = ["version"] -authors = [{ name = "oceanparcels.org team" }] +authors = [{ name = "parcels-code.org team" }] requires-python = ">=3.10" license = { file = "LICENSE" } classifiers = [ @@ -26,11 +26,11 @@ classifiers = [ ] dependencies = [ "click", - "parcels >3.1.0", + "parcels >=4.0.0alpha", "pyproj >= 3, < 4", "sortedcontainers == 2.4.0", "opensimplex == 0.4.5", - "numpy >=1, < 2", + "numpy >=2.1.0", "pydantic >=2, <3", "PyYAML", "copernicusmarine >= 2.2.2", @@ -40,7 +40,7 @@ dependencies = [ ] [project.urls] -Homepage = "https://oceanparcels.org/" # TODO: Update this to just be repo? +Homepage = "https://virtualship.parcels-code.org/" Repository = "https://github.com/OceanParcels/virtualship" Documentation = "https://virtualship.readthedocs.io/" "Bug Tracker" = "https://github.com/OceanParcels/virtualship/issues" diff --git a/src/virtualship/cli/_run.py b/src/virtualship/cli/_run.py index f2622be36..10afc5c5a 100644 --- a/src/virtualship/cli/_run.py +++ b/src/virtualship/cli/_run.py @@ -35,11 +35,9 @@ get_instrument_class, ) -# parcels logger (suppress INFO messages to prevent log being flooded) -external_logger = logging.getLogger("parcels.tools.loggers") -external_logger.setLevel(logging.WARNING) - -# copernicusmarine logger (suppress INFO messages to prevent log being flooded) +# suppress INFO messages from copernicusmarine and parcels loggers; prevent log flooding +parcels_logger = logging.getLogger("parcels._logger") +parcels_logger.setLevel(logging.WARNING) logging.getLogger("copernicusmarine").setLevel("ERROR") @@ -202,6 +200,7 @@ def _run( ) # execute simulation + # TODO: outpath will be Parquet with v4... instrument.execute( measurements=measurements, out_path=expedition_dir.joinpath(RESULTS, f"{itype.name.lower()}.zarr"), diff --git a/src/virtualship/instruments/adcp.py b/src/virtualship/instruments/adcp.py index b2da6582c..626ca3540 100644 --- a/src/virtualship/instruments/adcp.py +++ b/src/virtualship/instruments/adcp.py @@ -3,7 +3,7 @@ from typing import ClassVar import numpy as np -from parcels import ParticleSet, ScipyParticle +from parcels import ParticleSet from virtualship.instruments.base import Instrument from virtualship.instruments.sensors import SensorType @@ -35,9 +35,13 @@ class ADCP: # ===================================================== -def _sample_velocity(particle, fieldset, time): - particle.U, particle.V = fieldset.UV.eval( - time, particle.depth, particle.lat, particle.lon, applyConversion=False +def _sample_velocity(particles, fieldset): + particles.U, particles.V = fieldset.UV.eval( + particles.time, + particles.z, + particles.lat, + particles.lon, + applyConversion=False, ) @@ -96,7 +100,7 @@ def simulate(self, measurements, out_path) -> None: # build dynamic particle class from the active sensors adcp_config = self.expedition.instruments_config.adcp_config _ADCPParticle = build_particle_class_from_sensors( - adcp_config.sensors, _ADCP_NONSENSOR_VARIABLES, ScipyParticle + adcp_config.sensors, _ADCP_NONSENSOR_VARIABLES ) bins = np.linspace(MAX_DEPTH, MIN_DEPTH, NUM_BINS) diff --git a/src/virtualship/instruments/argo_float.py b/src/virtualship/instruments/argo_float.py index 8c90cfb2c..4eae63504 100644 --- a/src/virtualship/instruments/argo_float.py +++ b/src/virtualship/instruments/argo_float.py @@ -1,11 +1,11 @@ -import math from collections.abc import Callable from dataclasses import dataclass from datetime import timedelta from typing import ClassVar import numpy as np -from parcels import AdvectionRK4, JITParticle, ParticleSet, StatusCode, Variable +from parcels import ParticleSet, StatusCode, Variable +from parcels.kernels import AdvectionRK2 from virtualship.instruments.base import Instrument from virtualship.instruments.sensors import SensorType @@ -53,103 +53,125 @@ class ArgoFloat: # SECTION: Kernels # ===================================================== - -def _argo_float_vertical_movement(particle, fieldset, time): - if particle.cycle_phase == 0: - # Phase 0: Sinking with vertical_speed until depth is drift_depth - particle_ddepth += ( # noqa - particle.vertical_speed * particle.dt +# TODO: need to add back in the shallow bathymetry checks (to phases 0 and 2?!) +# TODO: can this be refactored as well to a helper function? + + +def _argo_float_vertical_movement(particles, fieldset): + # Split particles based on their current cycle_phase + ptcls0 = particles[particles.cycle_phase == 0] + ptcls1 = particles[particles.cycle_phase == 1] + ptcls2 = particles[particles.cycle_phase == 2] + ptcls3 = particles[particles.cycle_phase == 3] + ptcls4 = particles[particles.cycle_phase == 4] + + # Phase 0: Sinking with vertical_speed until depth is driftdepth + ptcls0.dz += particles.vertical_speed * ptcls0.dt + loc_bathy = fieldset.bathymetry.eval(ptcls0.time, ptcls0.z, ptcls0.lat, ptcls0.lon) + driftdepth_mask = ptcls0.z + ptcls0.dz >= particles.drift_depth + bathy_mask = ptcls0.z + ptcls0.dz >= loc_bathy + next_phase = np.logical_and( + driftdepth_mask, bathy_mask + ) # combined mask; not at drift depth yet and not hitting bathymetry + ptcls0.cycle_phase[next_phase] = 1 + ptcls0.dz[next_phase] = ( + particles.drift_depth - ptcls0.z[next_phase] + ) # avoid overshoot + + # Phase 0.5: Check for grounding at bathymetry and raise if necessary + ptcls0.grounded[~bathy_mask] = 1 + if np.any(~bathy_mask): + print( + "Shallow bathymetry warning: Argo float grounded at bathymetry depth during sinking to drift depth. Raising by 50m above bathymetry and continuing cycle." ) - - # bathymetry at particle location - loc_bathy = fieldset.bathymetry.eval( - time, particle.depth, particle.lat, particle.lon + ptcls0.dz[~bathy_mask] = ( + loc_bathy[~bathy_mask] - ptcls0.z[~bathy_mask] + 50.0 + ) # raise to 50m above bathymetry + ptcls0.cycle_phase[~bathy_mask] = 1 + + # Phase 1: Drifting at depth for drifttime seconds + ptcls1.drift_age += ptcls1.dt + next_phase = ptcls1.drift_age >= particles.drift_days * 86400 # [seconds] + ptcls1.cycle_phase[next_phase] = 2 + ptcls1.drift_age[next_phase] = 0 # reset drift_age for next cycle + + # Phase 2: Sinking further to maxdepth + ptcls2.dz += particles.vertical_speed * ptcls2.dt + loc_bathy = fieldset.bathymetry.eval(ptcls2.time, ptcls2.z, ptcls2.lat, ptcls2.lon) + maxdepth_mask = ptcls2.z + ptcls2.dz >= particles.max_depth + bathy_mask = ptcls2.z + ptcls2.dz >= loc_bathy + next_phase = np.logical_and( + maxdepth_mask, bathy_mask + ) # combined mask; not at max depth yet and not hitting bathymetry + ptcls2.cycle_phase[next_phase] = 3 + ptcls2.dz[next_phase] = ( + particles.max_depth - ptcls2.z[next_phase] + ) # avoid overshoot + + # Phase 2.5: Check for grounding at bathymetry and raise if necessary + ptcls2.grounded[~bathy_mask] = 1 + if np.any(~bathy_mask): + print( + "Shallow bathymetry warning: Argo float grounded at bathymetry depth during sinking to max depth. Raising by 50m above bathymetry and continuing cycle." ) - if particle.depth + particle_ddepth <= loc_bathy: - particle_ddepth = loc_bathy - particle.depth + 50.0 # 50m above bathy - particle.cycle_phase = 1 - particle.grounded = 1 - print( - "Shallow bathymetry warning: Argo float grounded at bathymetry depth during sinking to drift depth. Raising by 50m above bathymetry and continuing cycle." - ) + ptcls2.dz[~bathy_mask] = ( + loc_bathy[~bathy_mask] - ptcls2.z[~bathy_mask] + 50.0 + ) # raise to 50m above bathymetry + ptcls2.cycle_phase[~bathy_mask] = 3 - elif particle.depth + particle_ddepth <= particle.drift_depth: - particle_ddepth = particle.drift_depth - particle.depth - particle.cycle_phase = 1 - - elif particle.cycle_phase == 1: - # Phase 1: Drifting at depth for drifttime seconds - particle.drift_age += particle.dt - if particle.drift_age >= particle.drift_days * 86400: - particle.drift_age = 0 # reset drift_age for next cycle - particle.cycle_phase = 2 - - elif particle.cycle_phase == 2: - # Phase 2: Sinking further to max_depth - particle_ddepth += particle.vertical_speed * particle.dt - loc_bathy = fieldset.bathymetry.eval( - time, particle.depth, particle.lat, particle.lon - ) - if particle.depth + particle_ddepth <= loc_bathy: - particle_ddepth = loc_bathy - particle.depth + 50.0 # 50m above bathy - particle.cycle_phase = 3 - particle.grounded = 1 - print( - "Shallow bathymetry warning: Argo float grounded at bathymetry depth during sinking to max depth. Raising by 50m above bathymetry and continuing cycle." - ) - elif particle.depth + particle_ddepth <= particle.max_depth: - particle_ddepth = particle.max_depth - particle.depth - particle.cycle_phase = 3 - - elif particle.cycle_phase == 3: - # Phase 3: Rising with vertical_speed until at surface - particle_ddepth -= particle.vertical_speed * particle.dt - particle.cycle_age += ( - particle.dt - ) # solve issue of not updating cycle_age during ascent - particle.grounded = 0 - if particle.depth + particle_ddepth >= particle.min_depth: - particle_ddepth = particle.min_depth - particle.depth - particle.cycle_phase = 4 + # Phase 3: Rising with vertical_speed until at surface + ptcls3.dz -= particles.vertical_speed * ptcls3.dt + ptcls3.temp = fieldset.thetao[ptcls3.time, ptcls3.z, ptcls3.lat, ptcls3.lon] + next_phase = ptcls3.z + ptcls3.dz <= particles.min_depth + ptcls3.cycle_phase[next_phase] = 4 + ptcls3.dz[next_phase] = ( + particles.min_depth - ptcls3.z[next_phase] + ) # avoid overshoot - elif particle.cycle_phase == 4: - # Phase 4: Transmitting at surface until cycletime is reached - if particle.cycle_age > particle.cycle_days * 86400: - particle.cycle_phase = 0 - particle.cycle_age = 0 + # Phase 4: Transmitting at surface until cycletime is reached + next_phase = ptcls4.cycle_age >= particles.cycle_days * 86400 + ptcls4.cycle_phase[next_phase] = 0 + ptcls4.cycle_age[next_phase] = 0 # reset cycle_age for next cycle + ptcls4.temp = np.nan # no temperature measurement when at surface - if particle.state == StatusCode.Evaluate: - particle.cycle_age += particle.dt # update cycle_age + particles.cycle_age += particles.dt # update cycle_age -def _keep_at_surface(particle, fieldset, time): - # Prevent error when float reaches surface - if particle.state == StatusCode.ErrorThroughSurface: - particle.depth = particle.min_depth - particle.state = StatusCode.Success +def _keep_at_surface(particles, fieldset): + through_surface = particles.state == StatusCode.ErrorThroughSurface + particles.z[through_surface] = particles.min_depth[through_surface] + particles.state[through_surface] = StatusCode.Success -def _check_error(particle, fieldset, time): - if particle.state >= 50: # This captures all Errors - particle.delete() +def _check_error(particles, fieldset): + errors = particles.state >= 50 # captures all Errors + particles.state[errors] = StatusCode.Delete -def _argo_sample_temperature(particle, fieldset, time): +def _argo_sample_temperature(particles, fieldset): # Phase 3: ascending — sample temperature; NaN otherwise - if particle.cycle_phase == 3 and particle.depth < particle.min_depth: - particle.temperature = fieldset.T[ - time, particle.depth, particle.lat, particle.lon - ] - else: - particle.temperature = math.nan - - -def _argo_sample_salinity(particle, fieldset, time): + phase_mask = particles.cycle_phase == 3 + depth_mask = particles.depth < particles.min_depth + sampling_particles = particles[np.logical_and(phase_mask, depth_mask)] + sampling_particles.temperature = fieldset.T[ + sampling_particles.time, + sampling_particles.depth, + sampling_particles.lat, + sampling_particles.lon, + ] + + +def _argo_sample_salinity(particles, fieldset): # Phase 3: ascending — sample salinity; NaN otherwise - if particle.cycle_phase == 3 and particle.depth < particle.min_depth: - particle.salinity = fieldset.S[time, particle.depth, particle.lat, particle.lon] - else: - particle.salinity = math.nan + phase_mask = particles.cycle_phase == 3 + depth_mask = particles.depth < particles.min_depth + sampling_particles = particles[np.logical_and(phase_mask, depth_mask)] + sampling_particles.salinity = fieldset.S[ + sampling_particles.time, + sampling_particles.depth, + sampling_particles.lat, + sampling_particles.lon, + ] # ===================================================== @@ -229,9 +251,7 @@ def simulate(self, measurements, out_path) -> None: # build dynamic particle class from the active sensors argo_float_config = self.expedition.instruments_config.argo_float_config _ArgoParticle = build_particle_class_from_sensors( - argo_float_config.sensors, - _ARGO_NONSENSOR_VARIABLES, - JITParticle, + argo_float_config.sensors, _ARGO_NONSENSOR_VARIABLES ) # define parcel particles @@ -272,7 +292,7 @@ def simulate(self, measurements, out_path) -> None: [ _argo_float_vertical_movement, *sampling_kernels, - AdvectionRK4, + AdvectionRK2, _keep_at_surface, _check_error, ], diff --git a/src/virtualship/instruments/base.py b/src/virtualship/instruments/base.py index d4e078e61..f6afd3e02 100644 --- a/src/virtualship/instruments/base.py +++ b/src/virtualship/instruments/base.py @@ -8,8 +8,8 @@ from typing import TYPE_CHECKING, ClassVar import copernicusmarine +import parcels import xarray as xr -from parcels import FieldSet from yaspin import yaspin from virtualship.errors import CopernicusCatalogueError @@ -86,7 +86,7 @@ def __init__( self.min_lat, self.max_lat = min(wp_lats), max(wp_lats) self.min_lon, self.max_lon = min(wp_lons), max(wp_lons) - def load_input_data(self) -> FieldSet: + def load_input_data(self) -> parcels.FieldSet: """Load and return the input data as a FieldSet for the instrument.""" try: fieldset = self._generate_fieldset() @@ -97,7 +97,7 @@ def load_input_data(self) -> FieldSet: # interpolation methods for var in (v for v in self.variables if v not in ("U", "V")): - getattr(fieldset, var).interp_method = "linear_invdist_land_tracer" + getattr(fieldset, var).interp_method = parcels.interpolators.XLinear # depth negative for g in fieldset.gridset.grids: @@ -183,11 +183,11 @@ def _get_copernicus_ds( coordinates_selection_method="outside", ) - def _generate_fieldset(self) -> FieldSet: + def _generate_fieldset(self) -> parcels.FieldSet: """ Create and combine FieldSets for each variable, supporting both local and Copernicus Marine data sources. - Per variable avoids issues when using copernicusmarine and creating directly one FieldSet of ds's sourced from different Copernicus Marine product IDs, which is often the case for BGC variables. + N.B. Per variable avoids issues when using copernicusmarine and creating directly one FieldSet of ds's sourced from different Copernicus Marine product IDs (which can also have different temporal resolutions), which is often the case for BGC variables. """ fieldsets_list = [] keys = list(self.variables.keys()) @@ -217,12 +217,11 @@ def _generate_fieldset(self) -> FieldSet: [data_dir.joinpath(f) for f in files] ) # using: ds --> .from_xarray_dataset seems more robust than .from_netcdf for handling different temporal resolutions for different variables ... - fs = FieldSet.from_xarray_dataset( - ds, - variables={key: full_var_name}, - dimensions=self.dimensions, - mesh="spherical", - ) + # TODO: do docs on pre-downloading data need to be updated for these changes? Anything about conventions etc.? + fields = {key: ds[full_var_name]} + ds_fset = parcels.convert.copernicusmarine_to_sgrid(fields=fields) + fs = parcels.FieldSet.from_sgrid_conventions(ds_fset) + else: # stream via Copernicus Marine Service physical = var in COPERNICUSMARINE_PHYS_VARIABLES ds = self._get_copernicus_ds( @@ -230,9 +229,11 @@ def _generate_fieldset(self) -> FieldSet: physical=physical, var=var, ) - fs = FieldSet.from_xarray_dataset( - ds, {key: var}, self.dimensions, mesh="spherical" - ) + + fields = {key: ds[var]} + ds_fset = parcels.convert.copernicusmarine_to_sgrid(fields=fields) + fs = parcels.FieldSet.from_sgrid_conventions(ds_fset) + fieldsets_list.append(fs) base_fieldset = fieldsets_list[0] diff --git a/src/virtualship/instruments/ctd.py b/src/virtualship/instruments/ctd.py index 583a099cc..d8f229c12 100644 --- a/src/virtualship/instruments/ctd.py +++ b/src/virtualship/instruments/ctd.py @@ -4,13 +4,13 @@ from typing import TYPE_CHECKING, ClassVar import numpy as np -from parcels import JITParticle, ParticleSet, Variable +from parcels import ParticleSet, Variable +from parcels._core.statuscodes import StatusCode from virtualship.instruments.base import Instrument from virtualship.instruments.sensors import SensorType from virtualship.instruments.types import InstrumentType from virtualship.utils import ( - add_dummy_UV, build_particle_class_from_sensors, register_instrument, ) @@ -52,60 +52,87 @@ class CTD: ## physical variables -def _sample_temperature(particle, fieldset, time): - particle.temperature = fieldset.T[time, particle.depth, particle.lat, particle.lon] +def _sample_temperature(particles, fieldset): + particles.temperature = fieldset.T[ + particles.time, particles.z, particles.lat, particles.lon + ] -def _sample_salinity(particle, fieldset, time): - particle.salinity = fieldset.S[time, particle.depth, particle.lat, particle.lon] +def _sample_salinity(particles, fieldset): + particles.salinity = fieldset.S[ + particles.time, particles.z, particles.lat, particles.lon + ] ## bgc variables -def _sample_o2(particle, fieldset, time): - particle.o2 = fieldset.o2[time, particle.depth, particle.lat, particle.lon] +def _sample_o2(particles, fieldset): + particles.o2 = fieldset.o2[ + particles.time, particles.z, particles.lat, particles.lon + ] -def _sample_chlorophyll(particle, fieldset, time): - particle.chl = fieldset.chl[time, particle.depth, particle.lat, particle.lon] +def _sample_chlorophyll(particles, fieldset): + particles.chl = fieldset.chl[ + particles.time, particles.z, particles.lat, particles.lon + ] -def _sample_nitrate(particle, fieldset, time): - particle.no3 = fieldset.no3[time, particle.depth, particle.lat, particle.lon] +def _sample_nitrate(particles, fieldset): + particles.no3 = fieldset.no3[ + particles.time, particles.z, particles.lat, particles.lon + ] -def _sample_phosphate(particle, fieldset, time): - particle.po4 = fieldset.po4[time, particle.depth, particle.lat, particle.lon] +def _sample_phosphate(particles, fieldset): + particles.po4 = fieldset.po4[ + particles.time, particles.z, particles.lat, particles.lon + ] -def _sample_ph(particle, fieldset, time): - particle.ph = fieldset.ph[time, particle.depth, particle.lat, particle.lon] +def _sample_ph(particles, fieldset): + particles.ph = fieldset.ph[ + particles.time, particles.z, particles.lat, particles.lon + ] -def _sample_phytoplankton(particle, fieldset, time): - particle.phyc = fieldset.phyc[time, particle.depth, particle.lat, particle.lon] +def _sample_phytoplankton(particles, fieldset): + particles.phyc = fieldset.phyc[ + particles.time, particles.z, particles.lat, particles.lon + ] -def _sample_primary_production(particle, fieldset, time): - particle.nppv = fieldset.nppv[time, particle.depth, particle.lat, particle.lon] +def _sample_primary_production(particles, fieldset): + particles.nppv = fieldset.nppv[ + particles.time, particles.z, particles.lat, particles.lon + ] ## cast -def _ctd_cast(particle, fieldset, time): +def _ctd_cast(particles, fieldset): + particles_lowering = particles[particles.raising == 0] + particles_raising = particles[particles.raising == 1] + + # TODO: change to boolean masking, like with Argo Floats? + # lowering - if particle.raising == 0: - particle_ddepth = -particle.winch_speed * particle.dt - if particle.depth + particle_ddepth < particle.max_depth: - particle.raising = 1 - particle_ddepth = -particle_ddepth + particles_lowering.dz = -particles_lowering.winch_speed * particles_lowering.dt + particles_lowering.raising = np.where( + particles_lowering.z + particles_lowering.dz < particles_lowering.max_depth, + 1, + particles_lowering.raising, + ) + # raising - else: - particle_ddepth = particle.winch_speed * particle.dt - if particle.depth + particle_ddepth > particle.min_depth: - particle.delete() + particles_raising.dz = particles_raising.winch_speed * particles_raising.dt + particles_raising.state = np.where( + particles_raising.z + particles_raising.dz > particles_raising.min_depth, + StatusCode.Delete, + particles_raising.state, + ) # ===================================================== @@ -162,9 +189,6 @@ def simulate(self, measurements, out_path) -> None: fieldset = self.load_input_data() - # add dummy U - add_dummy_UV(fieldset) # TODO: parcels v3 bodge; remove when parcels v4 is used - # use first active field for time reference _time_ref_key = next(iter(self.variables)) _time_ref_field = getattr(fieldset, _time_ref_key) @@ -208,7 +232,7 @@ def simulate(self, measurements, out_path) -> None: # build dynamic particle class from the active sensors ctd_config = self.expedition.instruments_config.ctd_config _CTDParticle = build_particle_class_from_sensors( - ctd_config.sensors, _CTD_NONSENSOR_VARIABLES, JITParticle + ctd_config.sensors, _CTD_NONSENSOR_VARIABLES ) # define parcel particles diff --git a/src/virtualship/instruments/drifter.py b/src/virtualship/instruments/drifter.py index 379334b3e..79a1e34be 100644 --- a/src/virtualship/instruments/drifter.py +++ b/src/virtualship/instruments/drifter.py @@ -4,7 +4,9 @@ from typing import ClassVar import numpy as np -from parcels import AdvectionRK4, JITParticle, ParticleSet, Variable +from parcels import ParticleSet, Variable +from parcels._core.statuscodes import StatusCode +from parcels.kernels import AdvectionRK2 from virtualship.instruments.base import Instrument from virtualship.instruments.sensors import SensorType @@ -46,15 +48,21 @@ class Drifter: # ===================================================== -def _sample_temperature(particle, fieldset, time): - particle.temperature = fieldset.T[time, particle.depth, particle.lat, particle.lon] +def _sample_temperature(particles, fieldset): + particles.temperature = fieldset.T[ + particles.time, particles.z, particles.lat, particles.lon + ] -def _check_lifetime(particle, fieldset, time): - if particle.has_lifetime == 1: - particle.age += particle.dt - if particle.age >= particle.lifetime: - particle.delete() +def _check_lifetime(particles, fieldset): + particles_wlifetime = particles[particles.has_lifetime == 1] + + particles_wlifetime.age += particles_wlifetime.dt + particles_wlifetime.state = np.where( + particles_wlifetime.age >= particles_wlifetime.lifetime, + StatusCode.Delete, + particles_wlifetime.state, + ) # ===================================================== @@ -123,7 +131,7 @@ def simulate(self, measurements, out_path) -> None: # build dynamic particle class from the active sensors drifter_config = self.expedition.instruments_config.drifter_config _DrifterParticle = build_particle_class_from_sensors( - drifter_config.sensors, _DRIFTER_NONSENSOR_VARIABLES, JITParticle + drifter_config.sensors, _DRIFTER_NONSENSOR_VARIABLES ) # define parcel particles @@ -169,7 +177,7 @@ def simulate(self, measurements, out_path) -> None: # execute simulation drifter_particleset.execute( - [AdvectionRK4, *sampling_kernels, _check_lifetime], + [AdvectionRK2, *sampling_kernels, _check_lifetime], endtime=endtime, dt=DT, output_file=out_file, diff --git a/src/virtualship/instruments/ship_underwater_st.py b/src/virtualship/instruments/ship_underwater_st.py index 6a564cc06..d844b7a27 100644 --- a/src/virtualship/instruments/ship_underwater_st.py +++ b/src/virtualship/instruments/ship_underwater_st.py @@ -3,13 +3,12 @@ from typing import ClassVar import numpy as np -from parcels import ParticleSet, ScipyParticle +from parcels import ParticleSet from virtualship.instruments.base import Instrument from virtualship.instruments.sensors import SensorType from virtualship.instruments.types import InstrumentType from virtualship.utils import ( - add_dummy_UV, build_particle_class_from_sensors, register_instrument, ) @@ -40,13 +39,13 @@ class Underwater_ST: # define function sampling Salinity -def _sample_salinity(particle, fieldset, time): - particle.salinity = fieldset.S[time, particle.depth, particle.lat, particle.lon] +def _sample_salinity(particles, fieldset): + particles.S = fieldset.S[particles.time, particles.z, particles.lat, particles.lon] # define function sampling Temperature -def _sample_temperature(particle, fieldset, time): - particle.temperature = fieldset.T[time, particle.depth, particle.lat, particle.lon] +def _sample_temperature(particles, fieldset): + particles.T = fieldset.T[particles.time, particles.z, particles.lat, particles.lon] # ===================================================== @@ -95,13 +94,10 @@ def simulate(self, measurements, out_path) -> None: fieldset = self.load_input_data() - # add dummy U - add_dummy_UV(fieldset) # TODO: parcels v3 bodge; remove when parcels v4 is used - # build dynamic particle class from the active sensors st_config = self.expedition.instruments_config.ship_underwater_st_config _ShipSTParticle = build_particle_class_from_sensors( - st_config.sensors, _ST_NONSENSOR_VARIABLES, ScipyParticle + st_config.sensors, _ST_NONSENSOR_VARIABLES ) particleset = ParticleSet.from_list( diff --git a/src/virtualship/instruments/xbt.py b/src/virtualship/instruments/xbt.py index 051bf1fa8..d02afb7a2 100644 --- a/src/virtualship/instruments/xbt.py +++ b/src/virtualship/instruments/xbt.py @@ -4,14 +4,14 @@ from typing import ClassVar import numpy as np -from parcels import JITParticle, ParticleSet, Variable +from parcels import ParticleSet, Variable +from parcels._core.statuscodes import StatusCode from virtualship.instruments.base import Instrument from virtualship.instruments.sensors import SensorType from virtualship.instruments.types import InstrumentType from virtualship.models.spacetime import Spacetime from virtualship.utils import ( - add_dummy_UV, build_particle_class_from_sensors, register_instrument, ) @@ -50,26 +50,32 @@ class XBT: # ===================================================== -def _sample_temperature(particle, fieldset, time): - particle.temperature = fieldset.T[time, particle.depth, particle.lat, particle.lon] +def _sample_temperature(particles, fieldset): + particles.temperature = fieldset.T[ + particles.time, particles.z, particles.lat, particles.lon + ] -def _xbt_cast(particle, fieldset, time): - particle_ddepth = -particle.fall_speed * particle.dt +def _xbt_cast(particles, fieldset): + particles.dz = -particles.fall_speed * particles.dt # update the fall speed from the quadractic fall-rate equation # check https://doi.org/10.5194/os-7-231-2011 - particle.fall_speed = ( - particle.fall_speed - 2 * particle.deceleration_coefficient * particle.dt + particles.fall_speed = ( + particles.fall_speed - 2 * particles.deceleration_coefficient * particles.dt ) # delete particle if depth is exactly max_depth - if particle.depth == particle.max_depth: - particle.delete() + particles.state = np.where( + particles.z == particles.max_depth, StatusCode.Delete, particles.state + ) # set particle depth to max depth if it's too deep - if particle.depth + particle_ddepth < particle.max_depth: - particle_ddepth = particle.max_depth - particle.depth + particles.dz = np.where( + particles.z + particles.dz < particles.max_depth, + particles.max_depth - particles.z, + particles.z, + ) # ===================================================== @@ -117,9 +123,6 @@ def simulate(self, measurements, out_path) -> None: fieldset = self.load_input_data() - # add dummy U - add_dummy_UV(fieldset) # TODO: parcels v3 bodge; remove when parcels v4 is used - # use first active field for time reference _time_ref_key = next(iter(self.variables)) _time_ref_field = getattr(fieldset, _time_ref_key) @@ -166,7 +169,7 @@ def simulate(self, measurements, out_path) -> None: # build dynamic particle class from the active sensors xbt_config = self.expedition.instruments_config.xbt_config _XBTParticle = build_particle_class_from_sensors( - xbt_config.sensors, _XBT_NONSENSOR_VARIABLES, JITParticle + xbt_config.sensors, _XBT_NONSENSOR_VARIABLES ) # define xbt particles diff --git a/src/virtualship/models/expedition.py b/src/virtualship/models/expedition.py index eef23c761..da0843281 100644 --- a/src/virtualship/models/expedition.py +++ b/src/virtualship/models/expedition.py @@ -17,7 +17,6 @@ _calc_sail_time, _calc_wp_stationkeeping_time, _get_bathy_data, - _get_waypoint_latlons, _validate_numeric_to_timedelta, get_supported_sensors, register_instrument_config, @@ -131,14 +130,7 @@ def verify( land_waypoints = [] if not ignore_land_test: try: - wp_lats, wp_lons = _get_waypoint_latlons(self.waypoints) - bathymetry_field = _get_bathy_data( - min(wp_lats), - max(wp_lats), - min(wp_lons), - max(wp_lons), - from_data=from_data, - ).bathymetry + bathymetry_field = _get_bathy_data(from_data=from_data).bathymetry except Exception as e: raise ScheduleError( f"Problem loading bathymetry data (used to verify waypoints are in water) directly via copernicusmarine. \n\n original message: {e}" diff --git a/src/virtualship/static/expedition.yaml b/src/virtualship/static/expedition.yaml index 3be45c4d7..acb16dcf0 100644 --- a/src/virtualship/static/expedition.yaml +++ b/src/virtualship/static/expedition.yaml @@ -1,7 +1,5 @@ # see https://virtualship.readthedocs.io/en/latest/user-guide/tutorials/working_with_expedition_yaml.html for more details on how to edit this file # -# TODO: add a link to docs where lists what sensors are supported for each instrument -# schedule: waypoints: - instrument: diff --git a/src/virtualship/utils.py b/src/virtualship/utils.py index 7ad275cd9..85b3ad750 100644 --- a/src/virtualship/utils.py +++ b/src/virtualship/utils.py @@ -13,9 +13,10 @@ import copernicusmarine import numpy as np +import parcels import pyproj import xarray as xr -from parcels import FieldSet, Variable +from parcels import FieldSet, Particle, Variable from virtualship.errors import CopernicusCatalogueError @@ -340,29 +341,6 @@ def _get_expedition(expedition_dir: Path) -> Expedition: ) from e -def add_dummy_UV(fieldset: FieldSet): - """Add a dummy U and V field to a FieldSet to satisfy parcels FieldSet completeness checks.""" - if "U" not in fieldset.__dict__.keys(): - for uv_var in ["U", "V"]: - dummy_field = getattr( - FieldSet.from_data( - {"U": 0, "V": 0}, {"lon": 0, "lat": 0}, mesh="spherical" - ), - uv_var, - ) - fieldset.add_field(dummy_field) - try: - fieldset.time_origin = ( - fieldset.T.grid.time_origin - if "T" in fieldset.__dict__.keys() - else fieldset.o2.grid.time_origin - ) - except Exception: - raise ValueError( - "Cannot determine time_origin for dummy UV fields. Assert T or o2 exists in fieldset." - ) from None - - def _select_product_id( physical: bool, schedule_start, @@ -448,13 +426,7 @@ def _start_end_in_product_timerange( ) -def _get_bathy_data( - min_lat: float, - max_lat: float, - min_lon: float, - max_lon: float, - from_data: Path | None = None, -) -> FieldSet: +def _get_bathy_data(from_data: Path | None = None) -> FieldSet: """Bathymetry data from local or 'streamed' directly from Copernicus Marine.""" if from_data is not None: # load from local data var = "deptho" @@ -466,11 +438,6 @@ def _get_bathy_data( f"\n\n❗️ Could not find bathymetry variable '{var}' in data directory '{from_data}/bathymetry/'.\n\n❗️ Is the pre-downloaded data directory structure compliant with VirtualShip expectations?\n\n❗️ See the docs for more information on expectations: https://virtualship.readthedocs.io/en/latest/user-guide/index.html#documentation\n" ) from e ds_bathymetry = xr.open_dataset(bathy_dir.joinpath(filename)) - bathymetry_variables = {"bathymetry": "deptho"} - bathymetry_dimensions = {"lon": "longitude", "lat": "latitude"} - return FieldSet.from_xarray_dataset( - ds_bathymetry, bathymetry_variables, bathymetry_dimensions - ) else: # stream via Copernicus Marine Service ds_bathymetry = copernicusmarine.open_dataset( @@ -478,12 +445,11 @@ def _get_bathy_data( variables=["deptho"], coordinates_selection_method="outside", ) - bathymetry_variables = {"bathymetry": "deptho"} - bathymetry_dimensions = {"lon": "longitude", "lat": "latitude"} - return FieldSet.from_xarray_dataset( - ds_bathymetry, bathymetry_variables, bathymetry_dimensions + ds_fset = parcels.convert.copernicusmarine_to_sgrid( + fields={var: ds_bathymetry[var]} ) + return FieldSet.from_sgrid_conventions(ds_fset) def expedition_cost(schedule_results: ScheduleOk, time_past: timedelta) -> float: @@ -677,14 +643,13 @@ def _make_hash(s: str, length: int) -> str: def build_particle_class_from_sensors( sensors: list[SensorConfig], nonsensor_variables: list[Variable], - particle_class: type, # generic type annotation needed for v3 particle class behaviour # TODO: Update with Parcels v4 ) -> type: - """Build a Particle class (JITParticle or ScipyParticle) from nonsensor variables and active sensors.""" + """Build a Particle class from nonsensor variables and active sensors.""" sensor_variables = [ variable for sc in sensors if sc.enabled for variable in sc.meta.particle_vars ] - return particle_class.add_variables(nonsensor_variables + sensor_variables) + return Particle.add_variables(nonsensor_variables + sensor_variables) # ===================================================== diff --git a/tests/test_utils.py b/tests/test_utils.py index fde6796f9..2628793df 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -1,4 +1,5 @@ import datetime +import re from pathlib import Path import numpy as np @@ -399,3 +400,71 @@ def test_build_particle_class_scipy_base(): ParticleClass = build_particle_class_from_sensors(sensors, nonsensor, ScipyParticle) assert issubclass(ParticleClass, ScipyParticle) + + +def test_allowed_sensors_matches_docs(): + """Test that SUPPORTED_SENSORS_MAP (sensors allowed for each instrument) matches the sensor table in full_sensor_list.md.""" + # local imports to trigger instrument registration and avoid potential circular imports + import virtualship.instruments # noqa: F401 - ensures all @register_instrument decorators run + from virtualship.utils import INSTRUMENT_CLASS_MAP, SUPPORTED_SENSORS_MAP + + docs_path = ( + Path(__file__).parent.parent + / "docs/user-guide/documentation/full_sensor_list.md" + ) + content = docs_path.read_text(encoding="utf-8") + + display_name_to_instrument_type: dict[str, InstrumentType] = { + instrument_type.value: instrument_type + for instrument_type in INSTRUMENT_CLASS_MAP + if isinstance(instrument_type, InstrumentType) + } # all instruments should use their enum value as the bold display name in the markdown table + + # parse markdown table rows + row_pattern = re.compile(r"^\|([^|]*)\|([^|]*)\|.*$", re.MULTILINE) + + expected: dict[InstrumentType, set[SensorType]] = {} + current_instrument: InstrumentType | None = None + + for match in row_pattern.finditer(content): + instrument_cell = match.group(1).strip() + sensor_cell = match.group(2).strip() + + # extract only the **bold** text from the cell (e.g. "**UNDERWATER_ST** (Ship Underwater ST)" -> "UNDERWATER_ST") + bold_match = re.search(r"\*\*(.+?)\*\*", instrument_cell) + instrument_name = bold_match.group(1).strip() if bold_match else "" + + if instrument_name and instrument_name in display_name_to_instrument_type: + current_instrument = display_name_to_instrument_type[instrument_name] + if current_instrument not in expected: + expected[current_instrument] = set() + + # skip irrelevant cells + if ( + not sensor_cell + or sensor_cell.startswith(":") + or sensor_cell == "Sensor Name" + ): + continue + + sensor_name = sensor_cell.strip() + if current_instrument is not None and sensor_name: + expected[current_instrument].add(SensorType(sensor_name)) + + # verify each instrument in the docs is registered and has matching sensors + for instrument_type, doc_sensors in expected.items(): + assert instrument_type in SUPPORTED_SENSORS_MAP, ( + f"{instrument_type} is listed in full_sensor_list.md but not found in SUPPORTED_SENSORS_MAP." + ) + registered_sensors = set(SUPPORTED_SENSORS_MAP[instrument_type]) + assert registered_sensors == doc_sensors, ( + f"Sensor mismatch for {instrument_type}:\n" + f" In docs: {sorted(s.value for s in doc_sensors)}\n" + f" In code: {sorted(s.value for s in registered_sensors)}\n" + ) + + # verify each instrument registered in code is also covered in the docs + for instrument_type in SUPPORTED_SENSORS_MAP: + assert instrument_type in expected, ( + f"{instrument_type} is registered in SUPPORTED_SENSORS_MAP but not listed in full_sensor_list.md." + )