Skip to content

0.12.5-6-osf

Choose a tag to compare

View all tags
@SentinelDataHub SentinelDataHub released this 14 Dec 17:03
· 9 commits to master since this release
0.12.5-6-osf
8350288

Release Notes v.012-5-6-osf.pdf

Release Notes v. 0.12.5-6-osf

This software release introduces a set of new features and performance improvements with respect to the previous open source distribution (version 0.9.1-osf).

Important notes

Please note that DHuS 0.12.5-6-osf introduces outstanding changes in the database and this implies:

  • installation process more time consuming than usual: the bigger the amount of data stored in DB/SolR, the more time consuming the first start will be.

  • the roll-back to previous version (0.9.1-osf) is not available: a back-up of DB/SolR is strongly recommended before migrating to DHuS 0.12.5-6-osf.

What’s new?

Mission related features
Sentinel-2
Compatibility with the following new product format:

  • Compact format
  • Level-2A
  • Sentinel 2 addon update:
    The following additional indexes are extracted at ingestion time for Sentinel 2 L2A products:
    o VEGETATION_PERCENTAGE
    o BARE_SOILS_PERCENTAGE
    o WATER_PERCENTAGE
    o LOW_PROBA_CLOUDS_PERCENTAGE
    o MEDIUM_PROBA_CLOUDS_PERCENTAGE
    o HIGH_PROBA_CLOUDS_PERCENTAGE
    o NOW_ICE_PERCENTAGE

Sentinel-3
Compatibility with Level-1 OLCI and SLSTR products

Security features

  • Fixed security issues on AJS client
  • Fixed Privacy Violation
  • Removed usercode stub
  • SQL injection via OData prevented
  • Database encryption configurability at the first start.

UI features

New features for users:

  • SolR suggester, typing in the search bar
  • Extensible search bar and list of results
  • Sorting criteria on the advances search panel
  • Latitude and longitude coordinates for mouse hovering visualization (Google Earth like)
  • Product sub-component download in product inspector

New features for administrators:

  • Beta version of the Feature shapefile support (set the "enable_shapefile" to true in appconfig.json file)
    This feature allow to use shapefile (file with extension .shp) to define geographical selection in the DHuS AJS GUI. The shapefile can be imported:
    • Dragging and dropping the file directly in the map (only modern browser support this functionality)
    • By means of manual upload, added in advanced search panel
  • Allow to update passwords (set the "show_admin_change_password" to true, in appconfig.json file)
    The administrator can now update the user password through a classic update on the User entity.
  • Bulk product deletion from the list of products
  • Configurable Terms and conditions
  • Possibility to configure an SMTP client from the UI without specifying any authentication
  • Search by User e-mail from GUI
  • OWC integration
    A first version of the Open Web Component is available under http://[dhus address]/new. Please note that to be able to access OWC, DHuS shall be configured such that it is accessible at the following address http://[IP or name]/dhus

API features

  • OData “Connections” entity: New filters implemented to Identify pending requests via OData
  • New filters available for the OData “User” entity to select user created in a certain time span
  • Product deletion via Odata

Functionality improvement
Synchronizer updates:

  • Target collection field has been turned in a drop down menu available in the UI synchronizer panel
  • Possibility to stop any synchronizer even if the BE is unresponsive
  • The downloads triggered by a synchronization with copy are performed in parallel

Ingestion and eviction updates:

  • The products failing the ingestion can be deleted or transferred to a folder configurable in the dhus.xml
  • The evicted products can be delete or moved in a folder configurable in the dhus.xml
  • Products Deletion performance improved

Bug Fixes and Implementations
Here follows the list of bug fixes and implementations released:

Mission related features

  • Sentinel-1 Acquisition Period Value: S1 product with sensing date between 12:00-12:59 are now indexed correctly
  • Sentinel-3: SLSTR Level-0 reconstructed footprint now computed correctly (please note that it remains a visualization problem due to Spatial4j/JTS library)

Major functionality updates

  • DHuS FTP Service: the DHuS FTP service has been restored
  • Fix corrupted data transfer
    This implementation fixes the bugged data transfer process causing downloads corruption in case of connection drop.
  • DHuS Scanner Stuck
    This implementation solves the block of the scanners which is experienced on some Operational Hubs
  • Remove action records: Statistics are not more stored in the Hub DB tables

UI features fixes
Fixes for users:

  • GUI search list supporting quicklooks with size different from 512x512
  • Feedback during registration or password reset: Now the message says something which make clear that the e-mail AND/OR the username do not exists.
  • Feature multi cart icon: Changed icons to add/remove a product to chart and also the icon to add multiple products to chart in products list

Fixes for administrators:

  • The FileScanner panel now shows messages on the status of the file scanner, the date of the next scanner run and, when the ingestion is ongoing, a loading icon is displayed.

API features fixes

Solr service

  • Added solr client connection timeout to avoid deadlocks when too much tomcat threads are used
  • Added socket and connection timeouts on SolrClient to avoid threads lock
  • Usage of a dedicated Tomcat connector for DHuS for isolating Solr requests
  • Introduction of a limit to the number of rows with the /search API

OData v1 service:

This contribution implements classes renaming and moving. Moreover it brings the following improvements:

  • Better class naming overall
  • Moved Metalink producing code in a utility class
  • Add code to work around missing features of Olingo2 (contained targets).
  • Do not display the UserSynchronisers EntitySet to all users in the service document
  • Collection access restored
  • Fix for wring link in OData Users/Cart entity
  • Fix for OData substring() filter

Functionality improvement
Queries response performance

  • OData and Solr query reponse performance improvements

Odata Synchronizer

  • The products synchronizer stability is ensured in case of updates
  • User Synchronizer stops when trying to synchronize users having forbidden character in username

Logs

  • Product deleted from the Hub are now logged in the DHuS log

System errors fixes

  • Fix too many open files: all "too many open files" issue sources are identified. All are fixed, the only remaining aspect to be managed is to close the Zip stream to remove the error: “Cleaning up unclosed ZipFile for archive /x/x/x “
  • Update OpenJpeg Native Library
    This contribution fixes the memory leak in native library OpenJpeg causing DHuS crashes in operations with unexpected signal receive error
  • Fixed errors during deletion : : avoid time out of SolR when deleting products in huge databases
  • Fix compatibility dependency drbx-impl-jp2k
  • Fix exception when browsing manifest/dataObjectSection nodes

Known issues and limitations:

  • The synchronizers are performed sequentially and use the ingestion date to create a synchronization catalogue
  • List of deleted product not available.
  • Eviction based on the ingestion date.
  • Creation of user accounts with malformed e-mail: it is possible to create user accounts with malformed e-mails like "alaanajeeb1965alaa@gmail". This problem causes issue in the user synchronizers.
  • Multiple log-in component popped-up (OWC GUI): If not logged, the search action calls a new login component (OWC GUI) to always appear (first time with an error message). Log in panel shall be unique.
  • First node inspection not possible for S3A_SR_1_SRA_A and S3A_SR_1_SRA_BS: The NetCdf driver does not use the correct implementation to decode the data.
  • Sentinel-3A SLSTR JTS footprints not well visualized: JTS footprints are not well visualized for S3A SLSTR Level 1 and SLSTR Level 2 NTC product types. In some cases it seems that, the footprint has a bottom line (on the pole) which must go from -45 to 160 passing over the 0, but the JTS algorithm is doing the opposite.
  • Error in the log at DHuS start: at every DHuS start some errors are logged.They seem to be infos rather than messages or wrong software behaviour.

-Example below:

[0.12.5-3][2016-11-24 13:49:18,973][INFO ] Server started. (DHuS.java:197 - main)
[0.12.5-3][2016-11-24 13:49:18,975][INFO ] Installing webapp SolrWebapp{name:solr; welcome:[]; allow:null; deny:null} (TomcatServer.java:265 - main)
[0.12.5-3][2016-11-24 13:49:19,271][ERROR] Nov 24, 2016 1:49:19 PM org.apache.catalina.core.ApplicationContext log (StreamEncoder.java:221 - main)
[0.12.5-3][2016-11-24 13:49:19,271][ERROR] INFO: No Spring WebApplicationInitializer types detected on classpath (StreamEncoder.java:221 - main)
[0.12.5-3][2016-11-24 13:49:19,290][ERROR] Nov 24, 2016 1:49:19 PM org.apache.catalina.core.ApplicationContext log (StreamEncoder.java:221 - main)
[0.12.5-3][2016-11-24 13:49:19,290][ERROR] INFO: Initializing Spring root WebApplicationContext (StreamEncoder.java:221 - main)
[0.12.5-3][2016-11-24 13:49:19,290][INFO ] Root WebApplicationContext: initialization started (ContextLoader.java:285 - main)
[0.12.5-3][2016-11-24 13:49:19,300][INFO ] Refreshing Root WebApplicationContext: startup date [Thu Nov 24 13:49:19 UTC 2016]; root of context hierarchy (AbstractApplicationContext.java:510 - main)
Assets 3
Loading