What's clobbering my config.xml

[frizwi]

Hi, just wondering if this is under active development? Not necessarily looking for new features but more about understanding the code/setup a little more. This seems like a much more lightweight and easy to configure than relying on the WMS services from either Hyrax or THREDDS.

If this is not being supported, please let me know and I can try and work with an alternative server. But this does basically fill our needs so would really like to continue using it.

This particular issue is about the config.xml being overwritten. I've volume mounted it for custom datasets and it all works just fine. I'm now adding some <Styles> ... info inside the <dataset> element but at startup these are getting stripped out. Is it because my xml is invalid or something or some other reset mechanism?

Thanks!

[guygriffiths]

I'm not actively developing this at the moment, and it's been a while, but my first thought is that it's not a valid part of the XML schema for config.xml. ncWMS is loading it, and at some point saving it again. Anything which it doesn't recognise will disappear in the process.

But the root of your issue is that does not belong inside , and we don't support a general style renderer anyway. There is quite a lot of flexibility still though, depending upon what you want to do. I'm guessing you want a custom style of some sort? In which case you need to make a template file (https://reading-escience-centre.gitbooks.io/ncwms-user-guide/content/06-development.html#styles), drop it in your ~/.ncWMS2/.styles directory, restart, and it should become available. Easier said than done, obviously, but come back here if/when you have troubles.

[frizwi]

Thanks @guygriffiths

No nothing fancy like a custom style. All I want to do is to have GetCapabilities return a subset of the default-scaler styles. We have a web app client that automatically creates a dropdown list based on the available styles. I just want x-Rainbow, Magma and a couple of others as the only thing exposed.

I've tried setting the env ADVERTISED_PALETTES in docker-compose but there some sed/filesystem issue at startup, so then I've hardcoded and volume mounted ncWMS.xml as such:

<?xml version='1.0' encoding='utf-8'?>
<Context>
	<Parameter name="defaultPalette" value="x-Rainbow" override="true"/>
	<Parameter name="advertisedPalettes" value="x-Rainbow,x-Rainbow-inv,x-Sst,x-Sst-inv,psu-Magma,psu-Magma-inv" override="true"/>
</Context>

But that's not doing the trick either

[frizwi]

For example, THREDDS lists a whole bunch - see https://data-cbr.csiro.au/thredds/wms/catch_all/oa-sb-recom/tasse-hydro-1.5/test.ncml?service=WMS&version=1.3.0&request=GetCapabilities

But I'm having deployment issues with that, I'd prefer to use this repo instead

[guygriffiths]

This is from the web.xml:

 <!-- The following settings can be overridden by either changing this file in place (not recommended) or by defining a custom context in $CATALINA_BASE/conf/[enginename]/[hostname]/[webappname].xml Within this file, override the parameter with entries in a Context tag such as: -->
    <!-- <Parameter name="configDir" value="$HOME/.ncWMS2-testserver" override="false"/> -->
    <!-- Note that the override="false" is referring to the ability of THIS web.xml file to override the setting, so it NEEDS to be set to false to have any effect. -->

    <context-param>
        <!-- This specifies a list of palettes to advertise in the Capabilities document in addition to the default one. Other palettes are available, but advertising the full set in the Capabilities document makes for an excessively large XML document. All palettes must be predefined named palettes. -->
        <param-name>advertisedPalettes</param-name>
        <param-value></param-value>
        <!-- Only advertise the default. The following is an example of advertising 3 palettes in addition to the default: -->
        <!-- <param-value>div-RdBu,div-RdBu-inv,seq-cubeYF</param-value> -->
    </context-param>

Which looks more-or-less like what you're doing. Note this, however:

Note that the override="false" is referring to the ability of THIS web.xml file to override the setting, so it NEEDS to be set to false to have any effect.

So when you are setting override="true" it is saying "allow this setting to be overridden", which ncWMS is doing.

Possibly, anyway, it's been a long time since I dealt with this stuff. But yeah, my first guess would be change those trues to falses and restart it.

[frizwi]

Okay setting the override="false" didn't work. But the "not recommended" direct edits to web.xml did!

Atleast, I have a backup plan.

[ghansham]

We can set default value for palette/styles for every variable in
$HOME/.ncWMS2/config.xml

See here which all fields are available to configure variables:

https://reading-escience-centre.gitbooks.io/ncwms-user-guide/content/03-config.html#variables

For example,

<dataset id="TEST_DS" title="Test_DS" location="/usr1/john/*L2B_CO2.h5" queryable="true" downloadable="true" dataReaderClass="" copyrightStatement="" moreInfo="" disabled="false" updateInterval="-1">
    <variables>
        <variable id="CO2" title="CO2" description="Carbon dioxide concentration" palette="co2_palette" belowMinColor="#FF000000" aboveMaxColor="#FF000000" noDataColor="transparent" numColorBands="20" disabled="false" colorScaleRange="0.0 10.0" scaling="linear"/>
    </variables>
</dataset>

co2_palette is a custom palette placed in $HOME/.ncWMS2/.styles directory

This shall allow the variable to be loaded with that particular palette whenever it gets loaded. ncWMS offers restful api to do addition, editing, refreshing and deletion of multiple datasets and variables.

This way we can clobber config.xml. Please visit this link:

https://reading-escience-centre.gitbooks.io/ncwms-user-guide/content/03-config.html

We can use these rest api calls to configure metadata for datasets and their layers. Using these calls is a cleaner way as we allow ncWMS to edit the config.xml.

Even the xml based styles can be defined and stored anywhere and served via independent server. Those urls if we mention in our GetMap requests via client side code, we can get that working.

See part of sample sld xml screenshot attached.

The NamedLayer exists on ncWMS server and you can pass on the url of this xml served via independent server to the getMapRequest. Refer to edal and ncwms gitbook for versatile slds that ncwms offers like contours, coloured contours, rgb images, etc.

By the way, the getCapabilities url mentioned by you is not working right now.

[frizwi]

I'm not sure I understand - does the REST API modify config.xml or is there some other mechanism to hold the state? In a db or something?

So far, what I've been doing is adding a couple of datasets manually via the admin web interface and then replicating the template/structure with additional datasets. Having some other api mechanism would be a nice alternative.

Incidentally, the config docs that show how to use the api via curl seems to be having auth issues - is there some trick I need to be aware of when running under docker? I'm putting in the same default auth creds in the curl arguments as I do interactively when logging in via the web.

Finally, I'm using the guygriffiths/ncwms docker image from docker hub. Is this the most recent or should I be rebuilding the image from a later release? I do note that this image is throwing errors to do with unstructured meshes that I report back in 2017 which Guy has already fixed in 2020!!

[ghansham]

Try

curl --digest -u username:password -d "id=TEST_DS&title=TEST_DS_CO2&location=/usr1/john/*L2B_CO2.h5&downloadable=true" -X POST http://localhost:8080/ncWMS2/admin/addDataset

You can set some shell variables like senid, lop, num_colors, pal_name, etc and then set your preferences for each variable in the dataset.

curl --digest -u username:password -d "dataset.id=${senid}_${lop}_${layer_name}&${layer_name}.scaleMin=${scale_min}&${layer_name}.title=${layer_title}&${layer_name}.scaleMax=${scale_max}&${layer_name}.numColorBands=${num_colors}&${layer_name}.palette=${pal_name}&save=Save&${layer_name}.scaling=linear" -X POST http://localhost:80800/ncWMS2/admin/updateVariables

This configuration persists over server restarts.
And if new file gets added to the dataset, there is a rest api to refresh it to load the added files.

wget http://127.0.0.1:8080/ncWMS2/refresh/dataset?id=TEST_DS

Let me know if you are facing any issues.
I am not sure though if it works on docker setup. @guygriffiths can help us in that

[frizwi]

Yes, I did that but tomcat seems to be blocking it. The same username:password combo works fine in the browser

% curl --digest -u ncwms:ncwms -d "id=TEST_DS&title=TEST_DS_CO2&location=/usr1/john/*L2B_CO2.h5&downloadable=true" -X 
POST http://140.79.17.92:8080/ncWMS/admin/addDataset

And the response is a 401

<!doctype html><html lang="en"><head><title>HTTP Status 401 – Unauthorized</title><style type="text/css">h1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;} a {color:black;} a.name {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 401 – Unauthorized</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The request has not been applied because it lacks valid authentication credentials for the target resource.</p><hr class="line" /><h3>Apache Tomcat/8.5.27</h3></body></html>

[frizwi]

Actually, looks like I'm running ncWMS v2.4.1- the latest release seems to be 2.5.2 (not counting the dev version), I'd better build the latest docker image to see how that fares

[ghansham]

I am using these commands in 2.5.0. You can try upgrading. It should work. I am using it for last 5 years. Not even once I faced issue. It works 24x7. Mostly using rest api for adding datasets, configuring variables and refreshing datasets.

Seeing the release history, 2.4.1 is 7 years old. Better use 2.5.2/2.5.3 (latest ones).

Although I am not sure if it will solve the issue.

[ghansham]

Is the $HOME/.ncWMS2/config.xml getting overwritten on container restarts?

[frizwi]

Yes, that's right, config.xml gets overwrtten on container restarts - there is a .backup created. Seems like things are getting stripped out of there. The datasets that I saved from the admin web interface stay intact

[frizwi]

I am using these commands in 2.5.0. You can try upgrading. It should work. I am using it for last 5 years. Not even once I faced issue. It works 24x7. Mostly using rest api for adding datasets, configuring variables and refreshing datasets.

Seeing the release history, 2.4.1 is 7 years old. Better use 2.5.2/2.5.3 (latest ones).

Although I am not sure if it will solve the issue.

What tomcat + java version are you using? unidata/tomcat-docker-8.5 doesn't exist anymore and the new 9.0 one is built on jdk11 at the least. I had to tweak the code a little bit to make it build with the newer version, but still somethings not working. Will dig into it a little more

[ghansham]

The saved config.xml needs to be copied to $HOME/.ncWMS2/ so that it persists container restarts.

I am using jdk 1.8.0_241 and tomcat 8.5.53.
And ncwms 2.5.0.