Post-hoc examination of baro data import errors after reimport is finished

[monica-water]

Referencing PWD-MARS/shinyDownloadTools#19, a reimport of the baro data is underway. Care will be taken to reimport data correctly in accordance with the fix outlined in PWD-MARS/shinyDownloadTools#19, but full assessment of the differences between the reimported data and the current data may need to wait for a subsequent analysis. This issue represents the need to make that assessment, and will be closed early if such an assessment is not deemed necessary.

[monica-water]

The recommendations from this analysis are as follows:

• Monica should examine the 4 files with parse errors to assess their suitability for reimportation.
• Following that examination, a sandbox DB should be created in which to trial the reimport.
• In the sandbox DB, test copies of admin.tbl_baro_rawfile and data.tbl_baro should be created to facilitate the reimporation process.
• Data from files to-be-purged should be deleted from the test baro table.
• Records from to-be-purged files should be deleted from the test baro_rawfile table.
• The to-be-reimported files should be sequentially read into an import script, written to the test rawfile table, and their data written to the test baro table.
• A new primary key should be created in the test baro data table to ensure that all data sorts correctly.
• The test tables should be renamed to match the real tables, and the real tables should be renamed to indicate their deprecation.
• Downloads of baro data from the sandbox DB should be effectuated to assess the correctness of the new data.
• If that test passes, the above steps should be repeated on prod after business hours.
• Following the prod implementation, the deprecated tables should be dropped and the team should be notified of the change.