diff --git a/src/Plugins/OrientationAnalysis/docs/ReadCtfDataFilter.md b/src/Plugins/OrientationAnalysis/docs/ReadCtfDataFilter.md index 12e3dbeb4a..c33ce6aaed 100644 --- a/src/Plugins/OrientationAnalysis/docs/ReadCtfDataFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ReadCtfDataFilter.md @@ -6,9 +6,17 @@ IO (Input) ## Description -This **Filter** will read a single .ctf file into a new **Image Geometry**, allowing the immediate use of **Filters** on the data instead of having to generate the intermediate .h5ebsd file. A **Cell Attribute Matrix** and Ensemble Attribute Matrix** will also be created to hold the imported EBSD information. Currently, the user has no control over the names of the created **Attribute Arrays**. The user should be aware that simply reading the file then performing operations that are dependent on the proper crystallographic and sample reference frame will be **undefined, inaccurate and/or wrong**. In order to bring the crystal reference frame and sample reference frame into coincidence, rotations will need to be applied to the data. An excellent reference for this is the following PDF file: +This **Filter** will read a single .ctf file into a new **Image Geometry**, allowing the immediate use of **Filters** on the data instead of having to generate the intermediate .h5ebsd file. A **Cell Attribute Matrix** and an **Ensemble Attribute Matrix** will also be created to hold the imported EBSD information. Currently, the user has no control over the names of the created **Attribute Arrays**. The user should be aware that simply reading the file then performing operations that are dependent on the proper crystallographic and sample reference frame will be **undefined, inaccurate and/or wrong**. In order to bring the crystal reference frame and sample reference frame into coincidence, rotations will need to be applied to the data. An excellent reference for this is the following PDF file: [http://pajarito.materials.cmu.edu/rollett/27750/L17-EBSD-analysis-31Mar16.pdf](http://pajarito.materials.cmu.edu/rollett/27750/L17-EBSD-analysis-31Mar16.pdf) +### Multi-Slice (3D) .ctf Files + +If the file header declares more than one slice (`ZCells` > 1), the created **Image Geometry** is sized X × Y × Z and the slice thickness is taken from the header's `ZStep` key (a missing or zero `ZStep` defaults to 1.0). Single-slice (2D) files produce a geometry with a z-extent of 1. + +### Unindexed Points (Phase 0) + +Points the acquisition software could not index ("zero solutions") carry a phase value of **0** in the .ctf file, and this filter preserves that value in the `Phases` array. (Legacy DREAM3D 6.5 remapped these points to phase 1, silently assigning them to a real phase — see the migration notes below.) To mask unindexed points, threshold on *Error* = 0 (see below) or on *Phases* > 0. + ### Default HKL Transformations If the data has come from a HKL acquisition system and the settings of the acquisition software were in the default modes, then the following reference frame transformations need to be performed: @@ -18,9 +26,13 @@ If the data has come from a HKL acquisition system and the settings of the acqui The user also may want to assign un-indexed pixels to be ignored by flagging them as "bad". The Threshold Objects **Filter** can be used to define this *mask* by thresholding on values such as *Error* = 0. +### Memory Requirements + +While the filter executes, the reader's per-column buffers and the created **Attribute Arrays** are resident simultaneously: peak memory is roughly **88 bytes per scan point** for a standard 11-column file (about 4.4 GB for a 50-million-point map). Files carrying optional columns (Z, GrainIndex, GrainRandomColour) cost about 4 bytes per point more for each extra column. + ### Radians and Degrees -Most 2D .ctf files have their angles in **degrees** where as DREAM3D-NX expects radians. The filter provides an option to convert the Euler Angles to Radians and is turned on by default. The user is encouraged to create an IPF Image of their EBSD data to ensure that they do in-fact need to have this option enabled. +Most 2D .ctf files have their angles in **degrees** whereas DREAM3D-NX expects radians. The filter provides an option to convert the Euler angles to radians, which is enabled by default. The user is encouraged to create an IPF Image of their EBSD data to ensure that they do in fact need to have this option enabled. ### The Axis Alignment Issue for Hexagonal Symmetry [1] @@ -31,15 +43,19 @@ Most 2D .ctf files have their angles in **degrees** where as DREAM3D-NX expects + Caution: it appears that the axis alignment is a choice that must be made when installing TSL software so determination of which convention is in use must be made on a case-by-case basis. It is fixed to the y-convention in the HKL software. + The main clue that something is wrong in a conversion is that either the 2110 & 1010 pole figures are transposed, or that a peak in the inverse pole figure that should be present at 2110 has shifted over to 1010. + DREAM3D-NX uses the TSL/EDAX convention. -+ **The result of this is that the filter will by default add 30 degrees to the second Euler Angle (phi2) when reading Oxford Instr (.ctf) files. This can be disabled by the user if necessary.** ++ **The result of this is that the filter will by default add 30 degrees to the third Euler angle (phi2) of every point belonging to a hexagonal (6/mmm) phase when reading Oxford Instr (.ctf) files. Unindexed (Phase 0) points are never shifted. This can be disabled by the user if necessary.** | Figure 1 | |--------| | ![Figure showing 30 Degree conversions](Images/Hexagonal_Axis_Alignment.png) | -| Figure 1:**showing TSL and Oxford Instr. conventions. EDAX/TSL is in **Green**. Oxford Inst. is in**Red | +| Figure 1: showing TSL and Oxford Instr. conventions. EDAX/TSL is in **Green**. Oxford Instr. is in **Red**. | % Auto generated parameter table will be inserted here +## Migration Notes from DREAM3D 6.5 + +Documented behavioral differences from legacy DREAM3D 6.5 are maintained as Deviation entries in the source tree at `src/Plugins/OrientationAnalysis/vv/deviations/ReadCtfDataFilter.md`. In brief: unindexed points keep their phase value of 0 (legacy remapped them to 1, and consequently also applied the hexagonal +30° shift to them), and malformed .ctf files that crashed legacy DREAM3D are rejected with descriptive errors. + ## Example Pipelines + TxCopper_Exposed diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.hpp index 945c411302..23ad66b31a 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.hpp @@ -5,6 +5,7 @@ #include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/DataStructure.hpp" #include "simplnx/Filter/IFilter.hpp" +#include "simplnx/Parameters/FileSystemPathParameter.hpp" #include @@ -13,7 +14,7 @@ namespace nx::core struct ORIENTATIONANALYSIS_EXPORT ReadAngDataInputValues { - std::filesystem::path InputFile; + FileSystemPathParameter::ValueType InputFile; DataPath DataContainerName; std::string CellAttributeMatrixName; std::string CellEnsembleAttributeMatrixName; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.cpp index 6c4f58e434..208d3b68e5 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.cpp @@ -5,190 +5,280 @@ #include "simplnx/DataStructure/StringArray.hpp" #include +#include #include -using namespace nx::core; +#include + +#include +#include +#include +#include -using FloatVec3Type = std::vector; +using namespace nx::core; // ----------------------------------------------------------------------------- -ReadCtfData::ReadCtfData(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ReadCtfDataInputValues* inputValues) +ReadCtfData::ReadCtfData(DataStructure& dataStructure, const IFilter::MessageHandler& msgHandler, const std::atomic_bool& shouldCancel, ReadCtfDataInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) -, m_MessageHandler(mesgHandler) +, m_MessageHandler(msgHandler) { } // ----------------------------------------------------------------------------- ReadCtfData::~ReadCtfData() noexcept = default; -// ----------------------------------------------------------------------------- -const std::atomic_bool& ReadCtfData::getCancel() -{ - return m_ShouldCancel; -} - // ----------------------------------------------------------------------------- Result<> ReadCtfData::operator()() { + m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Reading .ctf file '{}'", m_InputValues->InputFile.string())); ebsdlib::CtfReader reader; reader.setFileName(m_InputValues->InputFile.string()); const int32_t err = reader.readFile(); if(err < 0) { - return MakeErrorResult(reader.getErrorCode(), reader.getErrorMessage()); + // CtfReader does not set its error-code member on every failure path (the zero-step and + // zero-cells rejections only set the message), so fall back to the returned code. + const int32_t errorCode = reader.getErrorCode() < 0 ? reader.getErrorCode() : err; + return MakeErrorResult(errorCode, reader.getErrorMessage()); } - const auto result = loadMaterialInfo(&reader); - if(result.first < 0) + Result<> result = loadMaterialInfo(&reader); + if(result.invalid()) { - return MakeErrorResult(result.first, result.second); + return result; } - copyRawEbsdData(&reader); + if(m_ShouldCancel) + { + return {}; + } - return {}; + m_MessageHandler(IFilter::Message::Type::Info, "Copying cell data into the Image Geometry"); + return copyRawEbsdData(&reader); } // ----------------------------------------------------------------------------- -std::pair ReadCtfData::loadMaterialInfo(ebsdlib::CtfReader* reader) const +Result<> ReadCtfData::loadMaterialInfo(ebsdlib::CtfReader* reader) const { - const DataPath cellEnsembleAttributeMatrixPath = m_InputValues->DataContainerName.createChildPath(m_InputValues->CellEnsembleAttributeMatrixName); - const std::vector phases = reader->getPhaseVector(); if(phases.empty()) { - return {reader->getErrorCode(), reader->getErrorMessage()}; + // A "Phases 0" header parses successfully (CtfReader's error code stays 0), but a file with no + // phase definitions carries no usable ensemble information, and every data row's phase value + // would then be out of range. Reject the file with a clear message instead. (Historical note: + // before this guard the early return here skipped the ensemble initialization below, and since + // Hexagonal_High is enum value 0, the zero-filled CrystalStructures applied the +30 degree + // hexagonal alignment to every point.) + return MakeErrorResult(-19600, fmt::format("The .ctf file '{}' declares no phases in its header. At least one phase definition is required.", m_InputValues->InputFile.string())); } - auto& crystalStructures = m_DataStructure.getDataRefAs(cellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::CrystalStructures)); + const DataPath cellEnsembleAttributeMatrixPath = m_InputValues->DataContainerName.createChildPath(m_InputValues->CellEnsembleAttributeMatrixName); + auto& crystalStructures = m_DataStructure.getDataRefAs(cellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::CrystalStructures)); auto& materialNames = m_DataStructure.getDataRefAs(cellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::MaterialName)); auto& latticeConstants = m_DataStructure.getDataRefAs(cellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::LatticeConstants)); const std::string k_InvalidPhase = "Invalid Phase"; + const usize numTuples = crystalStructures.getNumberOfTuples(); - // Initialize the zero'th element to unknowns. The other elements will - // be filled in based on values from the data file - crystalStructures[0] = ebsdlib::CrystalStructure::UnknownCrystalStructure; - materialNames[0] = k_InvalidPhase; - - for(size_t i = 0; i < 6; i++) + // Initialize EVERY slot to the "Invalid Phase" defaults first. Slot 0 is always the invalid + // phase; CtfReader assigns phase indices sequentially (1..N) so every other slot is refilled + // by the loop below, but initializing them all keeps the defaults authoritative rather than + // relying on zero-initialized storage. + for(usize tupleIndex = 0; tupleIndex < numTuples; tupleIndex++) { - latticeConstants.getDataStoreRef().setComponent(0, i, 0.0F); + crystalStructures[tupleIndex] = ebsdlib::CrystalStructure::UnknownCrystalStructure; + materialNames[tupleIndex] = k_InvalidPhase; + for(usize i = 0; i < 6; i++) + { + latticeConstants.getDataStoreRef().setComponent(tupleIndex, i, 0.0F); + } } for(const ebsdlib::CtfPhase::Pointer& phase : phases) { - const int32_t phaseID = phase->getPhaseIndex(); + const auto phaseID = static_cast(phase->getPhaseIndex()); + // The ensemble arrays were sized at preflight from the same file's phase count, so an index at + // or above the tuple count can only mean the file gained a phase between preflight and execute. + if(phaseID >= numTuples) + { + return MakeErrorResult( + -19605, fmt::format("Phase index {} from the .ctf file is at or above the Ensemble Attribute Matrix count {}. The input file may have changed since preflight.", phaseID, numTuples)); + } crystalStructures[phaseID] = phase->determineOrientationOpsIndex(); - const std::string materialName = phase->getMaterialName(); - materialNames[phaseID] = materialName; + materialNames[phaseID] = phase->getMaterialName(); - std::vector lattConst = phase->getLatticeConstants(); - - for(size_t i = 0; i < 6; i++) + const std::vector lattConst = phase->getLatticeConstants(); + for(usize i = 0; i < 6; i++) { latticeConstants.getDataStoreRef().setComponent(phaseID, i, lattConst[i]); } } - return {0, ""}; + return {}; } // ----------------------------------------------------------------------------- -void ReadCtfData::copyRawEbsdData(ebsdlib::CtfReader* reader) const +Result<> ReadCtfData::copyRawEbsdData(ebsdlib::CtfReader* reader) const { const DataPath cellAttributeMatrixPath = m_InputValues->DataContainerName.createChildPath(m_InputValues->CellAttributeMatrixName); const DataPath cellEnsembleAttributeMatrixPath = m_InputValues->DataContainerName.createChildPath(m_InputValues->CellEnsembleAttributeMatrixName); - std::vector cDims = {1}; const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->DataContainerName); - const size_t totalCells = imageGeom.getNumberOfCells(); + const usize totalCells = imageGeom.getNumberOfCells(); - // Prepare the Cell Attribute Matrix with the correct number of tuples based on the total Cells being read from the file. - std::vector tDims = {imageGeom.getNumXCells(), imageGeom.getNumYCells(), imageGeom.getNumZCells()}; + // The Image Geometry was sized at preflight from the file's XCells/YCells/ZCells header keys. + // Every copy below reads totalCells elements out of the reader's buffers, so if the reader + // actually produced fewer elements (a file that changed between preflight and execute), the + // copies would read past the end of the reader's heap buffers. Guard against that. + if(reader->getNumberOfElements() < totalCells) + { + return MakeErrorResult(-19603, fmt::format("The .ctf reader produced {} scan points but the Image Geometry created at preflight expects {}. The input file may have changed since preflight.", + reader->getNumberOfElements(), totalCells)); + } + + // A .ctf file's data section defines its own columns; a file missing one of the standard + // columns hands back a null buffer, which must be rejected rather than dereferenced. + auto fetchColumn = [&reader](const std::string& columnName, void*& ptr) -> Result<> { + ptr = reader->getPointerByName(columnName); + if(ptr == nullptr) + { + return MakeErrorResult(-19601, fmt::format("The .ctf file's data section does not contain the required column '{}'.", columnName)); + } + return {}; + }; + + void* phaseColumnPtr = nullptr; + void* euler1Ptr = nullptr; + void* euler2Ptr = nullptr; + void* euler3Ptr = nullptr; + { + Result<> fetchResult = fetchColumn(ebsdlib::Ctf::Phase, phaseColumnPtr); + if(fetchResult.invalid()) + { + return fetchResult; + } + fetchResult = fetchColumn(ebsdlib::Ctf::Euler1, euler1Ptr); + if(fetchResult.invalid()) + { + return fetchResult; + } + fetchResult = fetchColumn(ebsdlib::Ctf::Euler2, euler2Ptr); + if(fetchResult.invalid()) + { + return fetchResult; + } + fetchResult = fetchColumn(ebsdlib::Ctf::Euler3, euler3Ptr); + if(fetchResult.invalid()) + { + return fetchResult; + } + } - // Copy the Phase Array + // Copy the Phase column verbatim. Unlike EDAX .ang files, a phase value of 0 is meaningful in + // .ctf files (a "zero solutions" / unindexed point) and is preserved as-is; the legacy remap of + // phase<1 -> 1 was deliberately removed (PR #937). { + if(m_ShouldCancel) + { + return {}; + } auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::Phases)); - int* phasePtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Phase)); - for(size_t i = 0; i < totalCells; i++) + auto& crystalStructures = m_DataStructure.getDataRefAs(cellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::CrystalStructures)); + const usize ensembleTupleCount = crystalStructures.getNumberOfTuples(); + + const auto* phasePtr = static_cast(phaseColumnPtr); + for(usize i = 0; i < totalCells; i++) { + // The raw phase value indexes the ensemble arrays in the Euler loop below, so an + // out-of-range value (a corrupt file, or a phase column inconsistent with the header's + // phase count) would be an out-of-bounds read there. Reject it here. + if(phasePtr[i] < 0 || static_cast(phasePtr[i]) >= ensembleTupleCount) + { + return MakeErrorResult( + -19602, fmt::format("Scan point {} carries phase value {}, which is outside the valid range [0, {}] established by the file's phase definitions.", i, phasePtr[i], ensembleTupleCount - 1)); + } targetArray[i] = phasePtr[i]; } } - // Condense the Euler Angles from 3 separate arrays into a single 1x3 array + // Condense the Euler Angles from 3 separate arrays into a single 3-component array, applying + // the optional EDAX hexagonal-alignment (+30 degrees on phi2) and degrees-to-radians + // conversions. Both use double-precision intermediates so the stored float32 values are the + // correctly-rounded results (this also matches DREAM3D 6.5.171 bit-for-bit). { - auto& crystalStructures = m_DataStructure.getDataRefAs(cellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::CrystalStructures)); - auto& cellPhases = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::Phases)); + if(m_ShouldCancel) + { + return {}; + } + const auto& crystalStructures = m_DataStructure.getDataRefAs(cellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::CrystalStructures)); + // Read the phase values from the reader's buffer rather than the just-written Phases array: + // the copy above was verbatim and range-validated, and this avoids streaming a second + // DataStructure array (which may be out-of-core backed) through the loop. + const auto* cellPhases = static_cast(phaseColumnPtr); - const auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Euler1)); - const auto* fComp1 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Euler2)); - const auto* fComp2 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Euler3)); - cDims[0] = 3; + const auto* fComp0 = static_cast(euler1Ptr); + const auto* fComp1 = static_cast(euler2Ptr); + const auto* fComp2 = static_cast(euler3Ptr); auto& cellEulerAngles = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::EulerAngles)); - for(size_t i = 0; i < totalCells; i++) + for(usize i = 0; i < totalCells; i++) { - cellEulerAngles[3 * i] = fComp0[i]; - cellEulerAngles[3 * i + 1] = fComp1[i]; - cellEulerAngles[3 * i + 2] = fComp2[i]; + float32 euler1 = fComp0[i]; + float32 euler2 = fComp1[i]; + float32 euler3 = fComp2[i]; + // See the filter documentation for this correction factor: Oxford Instruments aligns the + // hexagonal crystal x-axis with [10-10] while DREAM3D-NX follows the EDAX/TSL convention + // of aligning it with [2-1-10], a 30 degree rotation about [0001] applied to phi2. if(crystalStructures[cellPhases[i]] == ebsdlib::CrystalStructure::Hexagonal_High && m_InputValues->EdaxHexagonalAlignment) { - cellEulerAngles[3 * i + 2] = cellEulerAngles[3 * i + 2] + 30.0F; // See the documentation for this correction factor + euler3 = static_cast(static_cast(euler3) + 30.0); } - // Now convert to radians if requested by the user if(m_InputValues->DegreesToRadians) { - cellEulerAngles[3 * i] = cellEulerAngles[3 * i] * ebsdlib::constants::k_PiOver180F; - cellEulerAngles[3 * i + 1] = cellEulerAngles[3 * i + 1] * ebsdlib::constants::k_PiOver180F; - cellEulerAngles[3 * i + 2] = cellEulerAngles[3 * i + 2] * ebsdlib::constants::k_PiOver180F; + euler1 = static_cast(static_cast(euler1) * ebsdlib::constants::k_PiOver180D); + euler2 = static_cast(static_cast(euler2) * ebsdlib::constants::k_PiOver180D); + euler3 = static_cast(static_cast(euler3) * ebsdlib::constants::k_PiOver180D); } + cellEulerAngles[3 * i] = euler1; + cellEulerAngles[3 * i + 1] = euler2; + cellEulerAngles[3 * i + 2] = euler3; } } - cDims[0] = 1; - { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Bands)); - auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::Ctf::Bands)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); - } - - { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Error)); - auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::Ctf::Error)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); - } - - { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::MAD)); - auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::Ctf::MAD)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); - } - + if(m_ShouldCancel) { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::BC)); - auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::Ctf::BC)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); + return {}; } + // The remaining columns are copied verbatim. + const std::vector> passthroughColumns = { + {ebsdlib::Ctf::Bands, DataType::int32}, {ebsdlib::Ctf::Error, DataType::int32}, {ebsdlib::Ctf::MAD, DataType::float32}, {ebsdlib::Ctf::BC, DataType::int32}, + {ebsdlib::Ctf::BS, DataType::int32}, {ebsdlib::Ctf::X, DataType::float32}, {ebsdlib::Ctf::Y, DataType::float32}, + }; + for(const auto& [columnName, dataType] : passthroughColumns) { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::BS)); - auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::Ctf::BS)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); - } - - { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::X)); - auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::Ctf::X)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); + void* columnPtr = nullptr; + Result<> fetchResult = fetchColumn(columnName, columnPtr); + if(fetchResult.invalid()) + { + return fetchResult; + } + const DataPath targetPath = cellAttributeMatrixPath.createChildPath(columnName); + if(dataType == DataType::int32) + { + const auto* sourcePtr = static_cast(columnPtr); + auto& targetArray = m_DataStructure.getDataRefAs(targetPath); + std::copy(sourcePtr, sourcePtr + totalCells, targetArray.begin()); + } + else + { + const auto* sourcePtr = static_cast(columnPtr); + auto& targetArray = m_DataStructure.getDataRefAs(targetPath); + std::copy(sourcePtr, sourcePtr + totalCells, targetArray.begin()); + } } - { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Y)); - auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::Ctf::Y)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); - } + return {}; } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.hpp index dba1e637d5..d59a2609c6 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.hpp @@ -7,7 +7,6 @@ #include "simplnx/Filter/IFilter.hpp" #include "simplnx/Parameters/FileSystemPathParameter.hpp" -#include #include namespace nx::core @@ -23,45 +22,15 @@ struct ORIENTATIONANALYSIS_EXPORT ReadCtfDataInputValues bool EdaxHexagonalAlignment; }; -struct ORIENTATIONANALYSIS_EXPORT Ctf_Private_Data -{ - std::array dims = {0, 0, 0}; - std::array resolution = {0.0F, 0.0F, 0.0F}; - std::array origin = {0.0F, 0.0F, 0.0F}; - std::vector phases; - int32_t units = 0; -}; - -/** - * @brief The ReadCtfDataPrivate class is a private implementation of the ReadCtfData class - */ -class ORIENTATIONANALYSIS_EXPORT ReadCtfDataPrivate -{ -public: - ReadCtfDataPrivate() = default; - ~ReadCtfDataPrivate() = default; - - ReadCtfDataPrivate(const ReadCtfDataPrivate&) = delete; // Copy Constructor Not Implemented - ReadCtfDataPrivate(ReadCtfDataPrivate&&) = delete; // Move Constructor Not Implemented - ReadCtfDataPrivate& operator=(const ReadCtfDataPrivate&) = delete; // Copy Assignment Not Implemented - ReadCtfDataPrivate& operator=(ReadCtfDataPrivate&&) = delete; // Move Assignment Not Implemented - - Ctf_Private_Data m_Data; - - std::string m_InputFile_Cache; - fs::file_time_type m_TimeStamp_Cache; -}; - /** * @class ReadCtfData * @brief This filter will read a single .ctf file into a new Image Geometry, allowing the immediate use of Filters on the data instead of having to generate the * intermediate .h5ebsd file. */ - class ORIENTATIONANALYSIS_EXPORT ReadCtfData { public: - ReadCtfData(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ReadCtfDataInputValues* inputValues); + ReadCtfData(DataStructure& dataStructure, const IFilter::MessageHandler& msgHandler, const std::atomic_bool& shouldCancel, ReadCtfDataInputValues* inputValues); ~ReadCtfData() noexcept; ReadCtfData(const ReadCtfData&) = delete; @@ -71,8 +40,6 @@ class ORIENTATIONANALYSIS_EXPORT ReadCtfData Result<> operator()(); - const std::atomic_bool& getCancel(); - private: DataStructure& m_DataStructure; const ReadCtfDataInputValues* m_InputValues = nullptr; @@ -80,17 +47,22 @@ class ORIENTATIONANALYSIS_EXPORT ReadCtfData const IFilter::MessageHandler& m_MessageHandler; /** - * @brief - * @param reader - * @return Error code. + * @brief Populates the Ensemble Attribute Matrix arrays (CrystalStructures, MaterialName, + * LatticeConstants) from the phase sections that CtfReader parsed out of the file header. + * @param reader The CtfReader instance that has already successfully read the file. + * @return Invalid Result if the file declares no phases. */ - std::pair loadMaterialInfo(ebsdlib::CtfReader* reader) const; + Result<> loadMaterialInfo(ebsdlib::CtfReader* reader) const; /** - * @brief - * @param reader + * @brief Copies the per-scan-point data columns out of the CtfReader buffers into the + * Cell Attribute Matrix arrays, condensing Euler1/2/3 into the 3-component EulerAngles + * array and applying the optional EDAX hexagonal alignment (+30 degrees) and + * degrees-to-radians conversions. + * @param reader The CtfReader instance that has already successfully read the file. + * @return Invalid Result if the reader buffers cannot back the geometry created at preflight. */ - void copyRawEbsdData(ebsdlib::CtfReader* reader) const; + Result<> copyRawEbsdData(ebsdlib::CtfReader* reader) const; }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadCtfDataFilter.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadCtfDataFilter.cpp index 512edc92dd..dd21c6f826 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadCtfDataFilter.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadCtfDataFilter.cpp @@ -17,9 +17,9 @@ #include #include -#include #include -#include + +#include #include @@ -104,40 +104,60 @@ IFilter::PreflightResult ReadCtfDataFilter::preflightImpl(const DataStructure& d auto pCellAttributeMatrixNameValue = filterArgs.value(k_CellAttributeMatrixName_Key); auto pCellEnsembleAttributeMatrixNameValue = filterArgs.value(k_CellEnsembleAttributeMatrixName_Key); - PreflightResult preflightResult; std::vector preflightUpdatedValues; ebsdlib::CtfReader reader; reader.setFileName(pInputFileValue.string()); - int32_t err = reader.readHeaderOnly(); + const int32 err = reader.readHeaderOnly(); if(err < 0) { - return {MakeErrorResult(reader.getErrorCode(), reader.getErrorMessage())}; + // CtfReader does not set its error-code member on every failure path, so fall back to the returned code. + const int32 errorCode = reader.getErrorCode() < 0 ? reader.getErrorCode() : err; + return {MakeErrorResult(errorCode, reader.getErrorMessage())}; + } + + // CtfReader::readHeaderOnly() reports success even when the header carries no usable + // dimensions (a missing or zero XCells/YCells key parses as 0), so reject that here + // instead of creating a zero-sized Image Geometry. + if(reader.getXCells() < 1 || reader.getYCells() < 1) + { + return {MakeErrorResult(-19604, fmt::format("The .ctf file header reports X Cells = {} and Y Cells = {}. Both must be at least 1. The file may be malformed or not a .ctf file.", + reader.getXCells(), reader.getYCells()))}; } - CreateImageGeometryAction::DimensionType imageGeomDims = {static_cast(reader.getXDimension()), static_cast(reader.getYDimension()), static_cast(1)}; + // .ctf files can carry more than one slice (ZCells > 1). A missing ZCells key parses as its + // default of 1. A NEGATIVE value must be rejected here: CtfReader::readData() captures its slice + // loop bound before clamping the value, so it would read zero data lines while sizing its buffers + // for one slice — the copies would then see only sentinel bytes, never the file's data. + if(reader.getZCells() < 0) + { + return {MakeErrorResult( + -19604, fmt::format("The .ctf file header reports Z Cells = {}. A negative slice count is not usable. The file may be malformed or not a .ctf file.", reader.getZCells()))}; + } + // A ZCells of 0 slips through the header-only read; it is caught at execute by the -19603 + // reader/geometry cell-count guard. + const size_t zCells = reader.getZCells() < 1 ? 1 : static_cast(reader.getZCells()); + CreateImageGeometryAction::DimensionType imageGeomDims = {static_cast(reader.getXCells()), static_cast(reader.getYCells()), zCells}; std::vector tupleDims = {imageGeomDims[2], imageGeomDims[1], imageGeomDims[0]}; - CreateImageGeometryAction::SpacingType spacing = {reader.getXStep(), reader.getYStep(), 1.0F}; + // A 2D .ctf file has no ZStep key (parses as 0), in which case the slice thickness defaults to 1. + // A negative ZStep is equally unusable and gets the same default. + const float32 zStep = reader.getZStep() > 0.0F ? reader.getZStep() : 1.0F; + CreateImageGeometryAction::SpacingType spacing = {reader.getXStep(), reader.getYStep(), zStep}; CreateImageGeometryAction::OriginType origin = {0.0F, 0.0F, 0.0F}; - // These variables should be updated with the latest data generated for each variable during preflight. - // These will be returned through the preflightResult variable to the - // user interface. You could make these member variables instead if needed. - EbsdReaderUtilities::GeneratePreflightScanInformation(reader, preflightUpdatedValues); EbsdReaderUtilities::GeneratePreflightPhaseInformation(reader, preflightUpdatedValues); - // Define a custom class that generates the changes to the DataStructure. - auto createImageGeometryAction = std::make_unique(pImageGeometryPath, CreateImageGeometryAction::DimensionType({imageGeomDims[0], imageGeomDims[1], imageGeomDims[2]}), - origin, spacing, pCellAttributeMatrixNameValue, IGeometry::LengthUnit::Micrometer); + auto createImageGeometryAction = std::make_unique(pImageGeometryPath, imageGeomDims, origin, spacing, pCellAttributeMatrixNameValue, IGeometry::LengthUnit::Micrometer); - // Assign the createImageGeometryAction to the Result::actions vector via a push_back nx::core::Result resultOutputActions; resultOutputActions.value().appendAction(std::move(createImageGeometryAction)); DataPath cellAttributeMatrixPath = pImageGeometryPath.createChildPath(pCellAttributeMatrixNameValue); + // These are the 7 pass-through columns; the copy loop in Algorithms/ReadCtfData.cpp + // (passthroughColumns) must stay in lockstep with CtfFields::getFilterFeatures(). ebsdlib::CtfFields ctfFeatures; const auto names = ctfFeatures.getFilterFeatures>(); std::vector cDims = {1ULL}; @@ -174,9 +194,10 @@ IFilter::PreflightResult ReadCtfDataFilter::preflightImpl(const DataStructure& d resultOutputActions.value().appendAction(std::move(action)); } - // Create the Ensemble AttributeMatrix - std::vector> angPhases = reader.getPhaseVector(); - tupleDims = {angPhases.size() + 1}; // Always create 1 extra slot for the phases. + // Create the Ensemble AttributeMatrix. Slot 0 is always reserved for the "Invalid Phase"; + // CtfReader assigns phase indices sequentially starting at 1, so the tuple count is + // (number of phases) + 1. + tupleDims = {reader.getPhaseVector().size() + 1}; DataPath ensembleAttributeMatrixPath = pImageGeometryPath.createChildPath(pCellEnsembleAttributeMatrixNameValue); { auto createAttributeMatrixAction = std::make_unique(ensembleAttributeMatrixPath, tupleDims); @@ -204,7 +225,6 @@ IFilter::PreflightResult ReadCtfDataFilter::preflightImpl(const DataStructure& d resultOutputActions.value().appendAction(std::move(action)); } - // Return both the resultOutputActions and the preflightUpdatedValues via std::move() return {std::move(resultOutputActions), std::move(preflightUpdatedValues)}; } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/utilities/EbsdReaderUtilities.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/utilities/EbsdReaderUtilities.hpp index eb09115e10..b141c5f1ee 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/utilities/EbsdReaderUtilities.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/utilities/EbsdReaderUtilities.hpp @@ -233,6 +233,14 @@ void GeneratePreflightScanInformation(ReaderType& reader, std::vector) + { + if(reader.getZCells() > 1) + { + ss << "Num Slices (Z): " << reader.getZCells() << " Z Step: " << (reader.getZStep() > 0.0F ? reader.getZStep() : 1.0F) << "\n"; + } + } + preflightUpdatedValues.push_back({"Scan Information", ss.str()}); } diff --git a/src/Plugins/OrientationAnalysis/test/CMakeLists.txt b/src/Plugins/OrientationAnalysis/test/CMakeLists.txt index a2ea03f1aa..90e256ccb2 100644 --- a/src/Plugins/OrientationAnalysis/test/CMakeLists.txt +++ b/src/Plugins/OrientationAnalysis/test/CMakeLists.txt @@ -126,7 +126,6 @@ if(EXISTS "${DREAM3D_DATA_DIR}" AND SIMPLNX_DOWNLOAD_TEST_FILES) download_test_data(DREAM3D_DATA_DIR ${DREAM3D_DATA_DIR} ARCHIVE_NAME compute_gbcd_metric_based.tar.gz SHA512 fc326f260c42114031f2a914a0f21b60174b7215618c63db3ee4159ff5ed485b81d0f55d78de573344f1b24d35e95ce272686b3c2ad13cfdf0dd9f06f2e2d38d) download_test_data(DREAM3D_DATA_DIR ${DREAM3D_DATA_DIR} ARCHIVE_NAME 6_6_find_gbpd_metric_based.tar.gz SHA512 79c383024c926e97d647c609a866253a446bde41a6b821991bd9211789421a02ca8ec6253bc1bcca8e955169faed278d483b0ca129422ed1c24d756fb550f678) download_test_data(DREAM3D_DATA_DIR ${DREAM3D_DATA_DIR} ARCHIVE_NAME 6_6_ImportH5Data.tar.gz SHA512 68cf620d28a515d04040d66c4f8fa9d8d46707c65138b366b47f40d1d56a36c40e2e2cd0c9d35168b68c2cb5b2ce95650fa2a0d4e6ffce980d0dd0654908f40d) - download_test_data(DREAM3D_DATA_DIR ${DREAM3D_DATA_DIR} ARCHIVE_NAME 6_6_read_ctf_data_2.tar.gz SHA512 f397fa3bf457615a90a4b48eaafded2aa4952b41ccb28d9da6a83adc38aea9c22f2bb5a955f251edeca9ef8265b6bf1d74e829b1340f45cf52620a237aad1707) download_test_data(DREAM3D_DATA_DIR ${DREAM3D_DATA_DIR} ARCHIVE_NAME 6_6_Small_IN100_GBCD.tar.gz SHA512 543e3bdcee24ff9e5cd80dfdedc39ceef1529e7172cebd01d8e5518292ffdf4e0eb2e79d75854cb3eaca5c60e19c861ca67f369e21b81c306edb66327f47a1e3) download_test_data(DREAM3D_DATA_DIR ${DREAM3D_DATA_DIR} ARCHIVE_NAME 6_6_stats_test_v2.tar.gz SHA512 e84999dec914d81efce4fc4237c49c9bf32e48381b1e79f58aa4df934f0d7606cd7a948f9a5e7b17a126a7944cc531b531cfdc70756ca3e2207b20734e089723) download_test_data(DREAM3D_DATA_DIR ${DREAM3D_DATA_DIR} ARCHIVE_NAME 7_compute_triangle_shapes_test.tar.gz SHA512 562ec65d22771817655ac85e267b1ea9822ab478ef732a52ce05052e4a171fc131bb879c9982df32b726d68a350f092695af97633f69e97226a6147cd0608b82) diff --git a/src/Plugins/OrientationAnalysis/test/ReadCtfDataTest.cpp b/src/Plugins/OrientationAnalysis/test/ReadCtfDataTest.cpp index 720976aa1d..7f4bf56592 100644 --- a/src/Plugins/OrientationAnalysis/test/ReadCtfDataTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/ReadCtfDataTest.cpp @@ -1,12 +1,62 @@ +/* ============================================================================ + * ReadCtfData V&V test suite. + * + * Verification is established INDEPENDENTLY of legacy DREAM3D, per the V&V + * policy (src/Plugins/OrientationAnalysis/vv/ReadCtfDataFilter.md): + * + * - .ctf parsing : Class 2 (EbsdLib reference, trusted & NOT re-tested). + * EbsdLib's CtfReader owns header parsing, phase-section + * parsing, data-column parsing, European-decimal fix-up + * and its own error codes. We do not re-test any of that. + * - SIMPLNX value-add : Class 1 (analytical) + Class 4 (invariant). The + * filter's value-add is deterministic data plumbing on + * top of CtfReader: geometry construction from header + * values (including multi-slice ZCells/ZStep), per-column + * array creation, verbatim Phase copy (phase 0 = unindexed + * point is preserved; the legacy phase<1 -> 1 remap was + * deliberately removed in PR #937), Euler-angle interleave + * with optional EDAX hexagonal alignment (+30 degrees on + * phi2 of Hexagonal_High points) and degrees-to-radians + * conversion, ensemble slot defaults, Laue-group -> + * crystal-structure index mapping and lattice-constant + * copy. The toy .ctf fixtures below are hand-authored and + * every expected value is hand-derived from the fixture + * text; none of it was produced by running DREAM3D (any + * version). + * + * Fixture float values are multiples of 1/8 (exactly representable in float32), + * so verbatim copies are asserted with exact equality. The degrees-to-radians + * and +30-degree expectations are the correctly-rounded float32 results of + * double-precision arithmetic (value * M_PI/180.0), computed independently with + * IEEE-754 float32/float64 semantics (NumPy) from the fixture text; they are + * asserted with exact equality as well. + * + * The prior exemplar archive 6_6_read_ctf_data_2.tar.gz compared against a + * .dream3d file generated by legacy DREAM3D (a circular/legacy oracle) and has + * been retired in favor of the inline oracle below. + * ========================================================================== */ + #include "OrientationAnalysis/Filters/ReadCtfDataFilter.hpp" #include "OrientationAnalysis/OrientationAnalysis_test_dirs.hpp" +#include "simplnx/Core/Application.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/DataStructure/StringArray.hpp" #include "simplnx/Parameters/FileSystemPathParameter.hpp" +#include "simplnx/Pipeline/Pipeline.hpp" +#include "simplnx/Pipeline/PipelineFilter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" #include +#include + #include +#include +#include +#include namespace fs = std::filesystem; @@ -14,40 +64,608 @@ using namespace nx::core; using namespace nx::core::Constants; using namespace nx::core::UnitTest; -TEST_CASE("OrientationAnalysis::ReadCtfData: Valid Execution", "[OrientationAnalysis][ReadCtfData]") +namespace { - UnitTest::LoadPlugins(); +//------------------------------------------------------------------------------ +// Toy .ctf fixture: 3 columns x 2 rows, XStep 0.25, YStep 0.5, two phases +// (phase 1 "Hex Phase A" Laue group 9 = Hexagonal_High, phase 2 "Copper" Laue +// group 11 = Cubic_High). Data columns are the 11 standard .ctf columns +// Phase X Y Bands Error Euler1 Euler2 Euler3 MAD BC BS +// Points 2 and 5 carry Phase 0 ("zero solutions" / unindexed) to pin the +// verbatim Phase copy and the invalid-phase handling in the Euler loop. +// All fields are TAB delimited. +//------------------------------------------------------------------------------ +const std::string k_HeaderPrefix = "Channel Text File\n" + "Prj\tVV Oracle Toy\n" + "Author\tVVOracle\n" + "JobMode\tGrid\n"; - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_6_read_ctf_data_2.tar.gz", "6_6_read_ctf_data_2"); +// {fmt} placeholders: XCells, YCells, extra header lines (e.g. ZCells/ZStep or nothing) +const std::string k_GridBlockFmt = "XCells\t{}\n" + "YCells\t{}\n" + "XStep\t0.25\n" + "YStep\t0.5\n" + "AcqE1\t0\n" + "AcqE2\t0\n" + "AcqE3\t0\n" + "{}" + "Euler angles refer to Sample Coordinate system (CS0)!\tMag\t2000\tCoverage\t100\tDevice\t0\tKV\t20\tTiltAngle\t70\tTiltAxis\t0\n"; - // Read Exemplar DREAM3D File - auto exemplarFilePath = fs::path(fmt::format("{}/6_6_read_ctf_data_2/6_6_read_ctf_data.dream3d", unit_test::k_TestFilesDir)); - DataStructure exemplarDataStructure = LoadDataStructure(exemplarFilePath); +// Phase sections: lattice constants (a;b;c), lattice angles (alpha;beta;gamma), +// name, Laue group, comment. Constants are float32-exact values. +// NOTE: EbsdLib's CtfReader::getHeaderLines() chops the final character of every +// phase line (it assumes CRLF endings), so the LAST field of a phase line (the +// comment here) loses its last character on LF-only fixtures — never assert on it. +const std::string k_TwoPhaseBlock = "Phases\t2\n" + "2.5;2.5;4.25\t90;90;120\tHex Phase A\t9\tVV toy hexagonal phase\n" + "3.625;3.625;3.625\t90;90;90\tCopper\t11\tVV toy cubic phase\n"; - // Instantiate the filter, a DataStructure object and an Arguments Object - ReadCtfDataFilter filter; - DataStructure dataStructure; - Arguments args; +const std::string k_ZeroPhaseBlock = "Phases\t0\n"; - const fs::path inputCtfFile(fmt::format("{}/6_6_read_ctf_data_2/Cugrid_after 2nd_15kv_2kx_2.ctf", unit_test::k_TestFilesDir)); +const std::string k_ColumnHeader = "Phase\tX\tY\tBands\tError\tEuler1\tEuler2\tEuler3\tMAD\tBC\tBS\n"; - // Create default Parameters for the filter. +// Point 3's Euler3 (7.125) and point 4's Euler1/Euler2 (3.375, 6.75) are chosen so the +// degrees->radians results DIFFER between float32 arithmetic and the double-precision +// intermediates the algorithm must use (and 7.125+30 = 37.125 differs too) — they pin the +// double-precision code path; the remaining values happen to round identically either way. +const std::string k_DataBlock = "1\t0\t0\t7\t0\t0.125\t0.25\t0.375\t0.5\t105\t177\n" + "2\t0.25\t0\t6\t0\t0.5\t0.625\t0.75\t0.25\t120\t209\n" + "0\t0.5\t0\t0\t3\t0.875\t1\t1.125\t0\t73\t178\n" + "1\t0\t0.5\t5\t0\t1.25\t1.375\t7.125\t0.375\t134\t232\n" + "2\t0.25\t0.5\t4\t0\t3.375\t6.75\t1.875\t0.625\t110\t216\n" + "0\t0.5\t0.5\t0\t3\t2\t2.125\t2.25\t0\t68\t151\n"; + +//------------------------------------------------------------------------------ +std::string MakeToyCtf(const std::string& extraHeaderLines = "", const std::string& phaseBlock = k_TwoPhaseBlock, const std::string& columnHeader = k_ColumnHeader, + const std::string& dataBlock = k_DataBlock, int32 xCells = 3, int32 yCells = 2) +{ + return k_HeaderPrefix + fmt::format(fmt::runtime(k_GridBlockFmt), xCells, yCells, extraHeaderLines) + phaseBlock + columnHeader + dataBlock; +} + +//------------------------------------------------------------------------------ +fs::path WriteCtfFile(const std::string& fileName, const std::string& contents) +{ + const fs::path filePath = fs::path(unit_test::k_BinaryTestOutputDir.view()) / fileName; + std::ofstream out(filePath, std::ios::out | std::ios::trunc); + REQUIRE(out.is_open()); + out << contents; + out.close(); + return filePath; +} + +//------------------------------------------------------------------------------ +Arguments MakeDefaultArgs(const fs::path& inputCtfFile, bool degreesToRadians = false, bool edaxHexagonalAlignment = false) +{ + Arguments args; args.insertOrAssign(ReadCtfDataFilter::k_InputFile_Key, std::make_any(inputCtfFile)); - args.insertOrAssign(ReadCtfDataFilter::k_DegreesToRadians_Key, std::make_any(true)); - args.insertOrAssign(ReadCtfDataFilter::k_EdaxHexagonalAlignment_Key, std::make_any(true)); + args.insertOrAssign(ReadCtfDataFilter::k_DegreesToRadians_Key, std::make_any(degreesToRadians)); + args.insertOrAssign(ReadCtfDataFilter::k_EdaxHexagonalAlignment_Key, std::make_any(edaxHexagonalAlignment)); args.insertOrAssign(ReadCtfDataFilter::k_CreatedImageGeometryPath_Key, std::make_any(k_DataContainerPath)); args.insertOrAssign(ReadCtfDataFilter::k_CellAttributeMatrixName_Key, std::make_any(k_CellData)); args.insertOrAssign(ReadCtfDataFilter::k_CellEnsembleAttributeMatrixName_Key, std::make_any(k_EnsembleAttributeMatrix)); + return args; +} + +//------------------------------------------------------------------------------ +template +void CompareArrayValues(const DataStructure& dataStructure, const DataPath& arrayPath, const std::vector& expected) +{ + REQUIRE_NOTHROW(dataStructure.getDataRefAs>(arrayPath)); + const auto& dataArrayRef = dataStructure.getDataRefAs>(arrayPath); + REQUIRE(dataArrayRef.getSize() == expected.size()); + for(usize i = 0; i < expected.size(); i++) + { + INFO(fmt::format("Array '{}' index {}", arrayPath.toString(), i)); + REQUIRE(dataArrayRef[i] == expected[i]); + } +} +} // namespace + +//------------------------------------------------------------------------------ +// Class 1 (analytical) + Class 4 (invariant) oracle. Every expected value below +// is hand-derived from the fixture text at the top of this file. Conversions +// are OFF so every stored value is a verbatim float32-exact copy. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadCtfDataFilter: Class 1 Analytical Oracle", "[OrientationAnalysis][ReadCtfDataFilter]") +{ + UnitTest::LoadPlugins(); + + const fs::path inputCtfFile = WriteCtfFile("read_ctf_vv_oracle.ctf", MakeToyCtf()); + + ReadCtfDataFilter filter; + DataStructure dataStructure; + Arguments args = MakeDefaultArgs(inputCtfFile); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + const DataPath cellAMPath = k_DataContainerPath.createChildPath(k_CellData); + const DataPath ensembleAMPath = k_DataContainerPath.createChildPath(k_EnsembleAttributeMatrix); + + // --- Geometry (Class 1: dims/spacing straight from the header; origin, z-extent + // --- and units are the filter's hard-coded value-add for a 2D file) --------- + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_DataContainerPath)); + const auto& imageGeom = dataStructure.getDataRefAs(k_DataContainerPath); + { + const SizeVec3 dims = imageGeom.getDimensions(); // XCells=3, YCells=2, no ZCells key -> 1 slice + REQUIRE(dims == SizeVec3(3, 2, 1)); + const FloatVec3 spacing = imageGeom.getSpacing(); // XStep=0.25, YStep=0.5, no ZStep key -> 1.0 + REQUIRE(spacing == FloatVec3(0.25F, 0.5F, 1.0F)); + const FloatVec3 origin = imageGeom.getOrigin(); // hard-coded (0,0,0) + REQUIRE(origin == FloatVec3(0.0F, 0.0F, 0.0F)); + REQUIRE(imageGeom.getUnits() == IGeometry::LengthUnit::Micrometer); + } + + // --- Cell data (Class 1) --------------------------------------------------- + // Phases: file column 1 is {1,2,0,1,2,0}, copied VERBATIM. Phase 0 (unindexed + // point) is preserved — the legacy phase<1 -> 1 remap was deliberately removed. + CompareArrayValues(dataStructure, cellAMPath.createChildPath("Phases"), {1, 2, 0, 1, 2, 0}); + + // EulerAngles: file columns 6-8 interleaved per point; conversions off. + CompareArrayValues(dataStructure, cellAMPath.createChildPath("EulerAngles"), + {0.125F, 0.25F, 0.375F, 0.5F, 0.625F, 0.75F, 0.875F, 1.0F, 1.125F, 1.25F, 1.375F, 7.125F, 3.375F, 6.75F, 1.875F, 2.0F, 2.125F, 2.25F}); + + // Pass-through columns, copied verbatim from the file. + CompareArrayValues(dataStructure, cellAMPath.createChildPath("Bands"), {7, 6, 0, 5, 4, 0}); + CompareArrayValues(dataStructure, cellAMPath.createChildPath("Error"), {0, 0, 3, 0, 0, 3}); + CompareArrayValues(dataStructure, cellAMPath.createChildPath("MAD"), {0.5F, 0.25F, 0.0F, 0.375F, 0.625F, 0.0F}); + CompareArrayValues(dataStructure, cellAMPath.createChildPath("BC"), {105, 120, 73, 134, 110, 68}); + CompareArrayValues(dataStructure, cellAMPath.createChildPath("BS"), {177, 209, 178, 232, 216, 151}); + CompareArrayValues(dataStructure, cellAMPath.createChildPath("X"), {0.0F, 0.25F, 0.5F, 0.0F, 0.25F, 0.5F}); + CompareArrayValues(dataStructure, cellAMPath.createChildPath("Y"), {0.0F, 0.0F, 0.0F, 0.5F, 0.5F, 0.5F}); + + // --- Ensemble data ---------------------------------------------------------- + // Class 4 invariant: tuple count is always (number of phases in file) + 1. + REQUIRE_NOTHROW(dataStructure.getDataRefAs(ensembleAMPath)); + const auto& ensembleAM = dataStructure.getDataRefAs(ensembleAMPath); + REQUIRE(ensembleAM.getShape() == std::vector{3}); + + // CrystalStructures (Class 1): slot 0 is the UnknownCrystalStructure default + // (999); .ctf Laue group 9 (6/mmm) -> Hexagonal_High (0); Laue group 11 + // (m-3m) -> Cubic_High (1). + CompareArrayValues(dataStructure, ensembleAMPath.createChildPath("CrystalStructures"), {999, 0, 1}); + + // LatticeConstants (Class 1): slot 0 zero-filled default; slots 1/2 copied + // from the phase lines (a, b, c, alpha, beta, gamma). All float32-exact. + CompareArrayValues(dataStructure, ensembleAMPath.createChildPath("LatticeConstants"), + {0.0F, 0.0F, 0.0F, 0.0F, 0.0F, 0.0F, 2.5F, 2.5F, 4.25F, 90.0F, 90.0F, 120.0F, 3.625F, 3.625F, 3.625F, 90.0F, 90.0F, 90.0F}); + + // MaterialName (Class 1): slot 0 is the "Invalid Phase" default; phase names + // are tab-delimited tokens copied verbatim (embedded spaces preserved). + { + const DataPath materialNamePath = ensembleAMPath.createChildPath("MaterialName"); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(materialNamePath)); + const auto& materialNamesRef = dataStructure.getDataRefAs(materialNamePath); + REQUIRE(materialNamesRef.getNumberOfTuples() == 3); + REQUIRE(materialNamesRef[0] == "Invalid Phase"); + REQUIRE(materialNamesRef[1] == "Hex Phase A"); + REQUIRE(materialNamesRef[2] == "Copper"); + } + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Class 1 oracle over the 2x2 parameter grid of DegreesToRadians and +// EdaxHexagonalAlignment. The +30 degree hexagonal alignment applies ONLY to +// points whose phase maps to Hexagonal_High (phase 1 here: points 0 and 3); +// cubic points (1, 4) and unindexed phase-0 points (2, 5 — ensemble slot 0 is +// UnknownCrystalStructure) are never shifted. +// +// Expected values are the correctly-rounded float32 results of double-precision +// arithmetic, derived independently with NumPy float32/float64 semantics: +// +30 : float32(float64(e3) + 30.0) +// radians : float32(float64(e) * (pi/180)) +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadCtfDataFilter: Euler Conversion Combinations", "[OrientationAnalysis][ReadCtfDataFilter]") +{ + UnitTest::LoadPlugins(); + + const fs::path inputCtfFile = WriteCtfFile("read_ctf_vv_conversions.ctf", MakeToyCtf()); + + struct ConversionCase + { + std::string name; + bool degreesToRadians; + bool edaxHexagonalAlignment; + std::vector expectedEulerAngles; + }; + + const std::vector cases = { + {"Degrees, No Hex Alignment", false, false, {0.125F, 0.25F, 0.375F, 0.5F, 0.625F, 0.75F, 0.875F, 1.0F, 1.125F, 1.25F, 1.375F, 7.125F, 3.375F, 6.75F, 1.875F, 2.0F, 2.125F, 2.25F}}, + // 0.375 + 30 = 30.375 and 7.125 + 30 = 37.125 are exact in float32. + {"Degrees, Hex Alignment", false, true, {0.125F, 0.25F, 30.375F, 0.5F, 0.625F, 0.75F, 0.875F, 1.0F, 1.125F, 1.25F, 1.375F, 37.125F, 3.375F, 6.75F, 1.875F, 2.0F, 2.125F, 2.25F}}, + // 7.125 -> 0.12435471, 3.375 -> 0.058904864, 6.75 -> 0.11780973 are the double-intermediate + // roundings; plain float32 arithmetic would give 0.124354705 / 0.05890486 / 0.11780972 — + // these three values (and 37.125 below) pin the double-precision code path. + {"Radians, No Hex Alignment", + true, + false, + {0.0021816615F, 0.004363323F, 0.006544985F, 0.008726646F, 0.0109083075F, 0.01308997F, 0.015271631F, 0.017453292F, 0.019634955F, 0.021816615F, 0.023998277F, 0.12435471F, 0.058904864F, + 0.11780973F, 0.032724924F, 0.034906585F, 0.037088245F, 0.03926991F}}, + // Points 0/3 (hex): float32(float64(30.375) * pi/180) = 0.53014374, + // float32(float64(37.125) * pi/180) = 0.6479535 (float32 arithmetic: 0.64795345). + {"Radians, Hex Alignment", + true, + true, + {0.0021816615F, 0.004363323F, 0.53014374F, 0.008726646F, 0.0109083075F, 0.01308997F, 0.015271631F, 0.017453292F, 0.019634955F, 0.021816615F, 0.023998277F, 0.6479535F, 0.058904864F, 0.11780973F, + 0.032724924F, 0.034906585F, 0.037088245F, 0.03926991F}}, + }; + + for(const ConversionCase& conversionCase : cases) + { + DYNAMIC_SECTION(conversionCase.name) + { + ReadCtfDataFilter filter; + DataStructure dataStructure; + Arguments args = MakeDefaultArgs(inputCtfFile, conversionCase.degreesToRadians, conversionCase.edaxHexagonalAlignment); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + const DataPath cellAMPath = k_DataContainerPath.createChildPath(k_CellData); + CompareArrayValues(dataStructure, cellAMPath.createChildPath("EulerAngles"), conversionCase.expectedEulerAngles); + // The Phase column is never altered by either conversion option. + CompareArrayValues(dataStructure, cellAMPath.createChildPath("Phases"), {1, 2, 0, 1, 2, 0}); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } + } +} + +//------------------------------------------------------------------------------ +// Class 1 oracle for multi-slice (3D) .ctf files: ZCells > 1 sizes the Image +// Geometry's z extent and ZStep sets the slice thickness (1.0 when the key is +// absent or 0). Data rows are stored in file order (slice-major). +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadCtfDataFilter: 3D Multi-Slice CTF", "[OrientationAnalysis][ReadCtfDataFilter]") +{ + UnitTest::LoadPlugins(); + + // 2 cols x 2 rows x 2 slices, single cubic phase. 8 data rows in slice-major order. + const std::string dataBlock3D = "1\t0\t0\t7\t0\t0.125\t0.25\t0.375\t0.5\t105\t177\n" + "1\t0.25\t0\t6\t0\t0.5\t0.625\t0.75\t0.25\t120\t209\n" + "0\t0\t0.5\t0\t3\t0.875\t1\t1.125\t0\t73\t178\n" + "1\t0.25\t0.5\t5\t0\t1.25\t1.375\t1.5\t0.375\t134\t232\n" + "1\t0\t0\t4\t0\t1.625\t1.75\t1.875\t0.625\t110\t216\n" + "0\t0.25\t0\t0\t3\t2\t2.125\t2.25\t0\t68\t151\n" + "1\t0\t0.5\t3\t0\t2.375\t2.5\t2.625\t0.125\t99\t142\n" + "1\t0.25\t0.5\t2\t0\t2.75\t2.875\t3\t0.875\t88\t123\n"; + const std::string cubicPhaseBlock = "Phases\t1\n" + "3.625;3.625;3.625\t90;90;90\tCopper\t11\tVV toy cubic phase\n"; + + struct SliceCase + { + std::string name; + std::string extraHeaderLines; + float32 expectedZSpacing; + }; + const std::vector cases = { + {"ZStep 0.75", "ZCells\t2\nZStep\t0.75\n", 0.75F}, + // A ZCells key without a usable ZStep (0) falls back to a slice thickness of 1. + {"ZStep absent", "ZCells\t2\n", 1.0F}, + }; + + for(const SliceCase& sliceCase : cases) + { + DYNAMIC_SECTION(sliceCase.name) + { + const fs::path inputCtfFile = WriteCtfFile("read_ctf_vv_3d.ctf", MakeToyCtf(sliceCase.extraHeaderLines, cubicPhaseBlock, k_ColumnHeader, dataBlock3D, 2, 2)); + + ReadCtfDataFilter filter; + DataStructure dataStructure; + Arguments args = MakeDefaultArgs(inputCtfFile); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_DataContainerPath)); + const auto& imageGeom = dataStructure.getDataRefAs(k_DataContainerPath); + REQUIRE(imageGeom.getDimensions() == SizeVec3(2, 2, 2)); + REQUIRE(imageGeom.getSpacing() == FloatVec3(0.25F, 0.5F, sliceCase.expectedZSpacing)); + + const DataPath cellAMPath = k_DataContainerPath.createChildPath(k_CellData); + CompareArrayValues(dataStructure, cellAMPath.createChildPath("Phases"), {1, 1, 0, 1, 1, 0, 1, 1}); + CompareArrayValues(dataStructure, cellAMPath.createChildPath("Bands"), {7, 6, 0, 5, 4, 0, 3, 2}); + CompareArrayValues(dataStructure, cellAMPath.createChildPath("EulerAngles"), {0.125F, 0.25F, 0.375F, 0.5F, 0.625F, 0.75F, 0.875F, 1.0F, 1.125F, 1.25F, 1.375F, 1.5F, + 1.625F, 1.75F, 1.875F, 2.0F, 2.125F, 2.25F, 2.375F, 2.5F, 2.625F, 2.75F, 2.875F, 3.0F}); + CompareArrayValues(dataStructure, cellAMPath.createChildPath("MAD"), {0.5F, 0.25F, 0.0F, 0.375F, 0.625F, 0.0F, 0.125F, 0.875F}); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } + } +} + +//------------------------------------------------------------------------------ +// Value-add error path: a header that declares no phases ("Phases 0") parses +// successfully but is rejected at execute with -19600 — a file with no phase +// definitions has no usable ensemble information. (The pre-V&V code returned +// early on this input BEFORE initializing the ensemble arrays, and because +// Hexagonal_High == 0 the zero-filled CrystalStructures would have applied the +// +30 degree shift to every point.) +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadCtfDataFilter: Empty Phases rejected (-19600)", "[OrientationAnalysis][ReadCtfDataFilter]") +{ + UnitTest::LoadPlugins(); + + const fs::path inputCtfFile = WriteCtfFile("read_ctf_vv_no_phase.ctf", MakeToyCtf("", k_ZeroPhaseBlock)); + + ReadCtfDataFilter filter; + DataStructure dataStructure; + Arguments args = MakeDefaultArgs(inputCtfFile); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); + REQUIRE(executeResult.result.errors()[0].code == -19600); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Value-add error path: a file whose data section is missing one of the standard +// columns (here BS) is rejected at execute with -19601 instead of dereferencing +// the null buffer CtfReader hands back. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadCtfDataFilter: Missing Data Column rejected (-19601)", "[OrientationAnalysis][ReadCtfDataFilter]") +{ + UnitTest::LoadPlugins(); + + const std::string columnHeaderNoBS = "Phase\tX\tY\tBands\tError\tEuler1\tEuler2\tEuler3\tMAD\tBC\n"; + const std::string dataBlockNoBS = "1\t0\t0\t7\t0\t0.125\t0.25\t0.375\t0.5\t105\n" + "2\t0.25\t0\t6\t0\t0.5\t0.625\t0.75\t0.25\t120\n" + "0\t0.5\t0\t0\t3\t0.875\t1\t1.125\t0\t73\n" + "1\t0\t0.5\t5\t0\t1.25\t1.375\t1.5\t0.375\t134\n" + "2\t0.25\t0.5\t4\t0\t1.625\t1.75\t1.875\t0.625\t110\n" + "0\t0.5\t0.5\t0\t3\t2\t2.125\t2.25\t0\t68\n"; + const fs::path inputCtfFile = WriteCtfFile("read_ctf_vv_missing_column.ctf", MakeToyCtf("", k_TwoPhaseBlock, columnHeaderNoBS, dataBlockNoBS)); + + ReadCtfDataFilter filter; + DataStructure dataStructure; + Arguments args = MakeDefaultArgs(inputCtfFile); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); + REQUIRE(executeResult.result.errors()[0].code == -19601); + // The missing column's name is the user-facing value of this error message. + REQUIRE(executeResult.result.errors()[0].message.find("'BS'") != std::string::npos); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Value-add error path: a data row carrying a phase value outside the range +// established by the header's phase definitions is rejected at execute with +// -19602 instead of reading past the end of the ensemble arrays in the Euler +// loop. The corrupt value is 3 — with 2 phases the ensemble has tuples 0..2, +// so 3 is the FIRST invalid value and pins the guard's exact boundary. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadCtfDataFilter: Out-of-Range Phase Value rejected (-19602)", "[OrientationAnalysis][ReadCtfDataFilter]") +{ + UnitTest::LoadPlugins(); + + std::string corruptDataBlock = k_DataBlock; + corruptDataBlock.replace(0, 1, "3"); // First data row's Phase column: 1 -> 3 (first out-of-range value) + const fs::path inputCtfFile = WriteCtfFile("read_ctf_vv_phase_range.ctf", MakeToyCtf("", k_TwoPhaseBlock, k_ColumnHeader, corruptDataBlock)); + + ReadCtfDataFilter filter; + DataStructure dataStructure; + Arguments args = MakeDefaultArgs(inputCtfFile); - // Preflight the filter and check result auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions) + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); - // Execute the filter and check the result auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result) + SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); + REQUIRE(executeResult.result.errors()[0].code == -19602); + // The offending value and the valid range are the user-facing value of this error message. + REQUIRE(executeResult.result.errors()[0].message.find("phase value 3") != std::string::npos); + REQUIRE(executeResult.result.errors()[0].message.find("[0, 2]") != std::string::npos); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} - CompareExemplarToGeneratedData(dataStructure, exemplarDataStructure, k_CellAttributeMatrix, k_ExemplarDataContainer); +//------------------------------------------------------------------------------ +// Value-add error path: a "ZCells 0" header makes CtfReader::readData() succeed +// with ZERO elements (its z loop never runs), while the preflight geometry is +// sized x*y*1. The cell-count guard rejects the mismatch at execute with -19603 +// instead of reading past the end of the reader's (empty) buffers. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadCtfDataFilter: Reader/Geometry Cell Count Mismatch rejected (-19603)", "[OrientationAnalysis][ReadCtfDataFilter]") +{ + UnitTest::LoadPlugins(); + + const fs::path inputCtfFile = WriteCtfFile("read_ctf_vv_zero_zcells.ctf", MakeToyCtf("ZCells\t0\n")); + + ReadCtfDataFilter filter; + DataStructure dataStructure; + Arguments args = MakeDefaultArgs(inputCtfFile); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); + REQUIRE(executeResult.result.errors()[0].code == -19603); UnitTest::CheckArraysInheritTupleDims(dataStructure); } + +//------------------------------------------------------------------------------ +// Value-add error path: CtfReader::readHeaderOnly() reports success even when +// the header has no usable dimensions (a missing or zero XCells/YCells key +// parses as 0, and a negative ZCells would make readData() read zero data +// lines), so preflight rejects those files with -19604. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadCtfDataFilter: Invalid Cells Preflight Error (-19604)", "[OrientationAnalysis][ReadCtfDataFilter]") +{ + UnitTest::LoadPlugins(); + + const std::vector> cases = { + {"Zero XCells", MakeToyCtf("", k_TwoPhaseBlock, k_ColumnHeader, k_DataBlock, 0, 2)}, + {"Negative ZCells", MakeToyCtf("ZCells\t-2\n")}, + }; + + for(const auto& [label, contents] : cases) + { + DYNAMIC_SECTION(label) + { + const fs::path inputCtfFile = WriteCtfFile("read_ctf_vv_invalid_cells.ctf", contents); + + ReadCtfDataFilter filter; + DataStructure dataStructure; + Arguments args = MakeDefaultArgs(inputCtfFile); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -19604); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } + } +} + +//------------------------------------------------------------------------------ +// EbsdLib error passthrough: a zero X Step is accepted at preflight (header-only +// read performs no step validation) but CtfReader::readFile() rejects it at +// execute. CtfReader only sets its error MESSAGE on this path (the error code +// member stays 0), so the filter reports the readFile() return value: -102. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadCtfDataFilter: EbsdLib Error Passthrough - Zero Step (-102)", "[OrientationAnalysis][ReadCtfDataFilter]") +{ + UnitTest::LoadPlugins(); + + std::string contents = MakeToyCtf(); + const std::string xStepLine = "XStep\t0.25\n"; + contents.replace(contents.find(xStepLine), xStepLine.size(), "XStep\t0\n"); + const fs::path inputCtfFile = WriteCtfFile("read_ctf_vv_zero_step.ctf", contents); + + ReadCtfDataFilter filter; + DataStructure dataStructure; + Arguments args = MakeDefaultArgs(inputCtfFile); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); + REQUIRE(executeResult.result.errors()[0].code == -102); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// EbsdLib error passthrough: the header declares 3x2 points but the data block +// is truncated to 4 rows; CtfReader reports -105 (premature end of file). +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadCtfDataFilter: EbsdLib Error Passthrough - Truncated Data (-105)", "[OrientationAnalysis][ReadCtfDataFilter]") +{ + UnitTest::LoadPlugins(); + + // Keep only the first 4 of 6 data rows. + std::string truncatedData = k_DataBlock; + for(int i = 0; i < 2; i++) + { + truncatedData.erase(truncatedData.find_last_of('\n', truncatedData.size() - 2) + 1); + } + const fs::path inputCtfFile = WriteCtfFile("read_ctf_vv_truncated.ctf", MakeToyCtf("", k_TwoPhaseBlock, k_ColumnHeader, truncatedData)); + + ReadCtfDataFilter filter; + DataStructure dataStructure; + Arguments args = MakeDefaultArgs(inputCtfFile); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); + REQUIRE(executeResult.result.errors()[0].code == -105); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// EbsdLib error passthrough: an unrecognized data-column name is rejected when +// CtfReader allocates its column buffers; the filter reports -107. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadCtfDataFilter: EbsdLib Error Passthrough - Unknown Column (-107)", "[OrientationAnalysis][ReadCtfDataFilter]") +{ + UnitTest::LoadPlugins(); + + const std::string bogusColumnHeader = "Phase\tX\tY\tBands\tError\tEuler1\tEuler2\tEuler3\tMAD\tBC\tBogus\n"; + const fs::path inputCtfFile = WriteCtfFile("read_ctf_vv_unknown_column.ctf", MakeToyCtf("", k_TwoPhaseBlock, bogusColumnHeader, k_DataBlock)); + + ReadCtfDataFilter filter; + DataStructure dataStructure; + Arguments args = MakeDefaultArgs(inputCtfFile); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); + REQUIRE(executeResult.result.errors()[0].code == -107); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// SIMPL Backwards Compatibility — validates UUID + parameter conversion from +// the legacy ReadCtfData (SIMPL UUID d1df969c-0428-53c3-b61d-99ea2bb6da28). +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadCtfDataFilter: SIMPL Backwards Compatibility", "[OrientationAnalysis][ReadCtfDataFilter][BackwardsCompatibility]") +{ + auto app = Application::GetOrCreateInstance(); + UnitTest::LoadPlugins(); + auto filterList = app->getFilterList(); + + const fs::path conversionDir = fs::path(nx::core::unit_test::k_SourceDir.view()) / "test" / "simpl_conversion"; + + const std::vector> fixtures = { + {"SIMPL 6.5 (UUID)", conversionDir / "6_5" / "ReadCtfDataFilter.json"}, + {"SIMPL 6.4 (Filter_Name)", conversionDir / "6_4" / "ReadCtfDataFilter.json"}, + }; + + for(const auto& [label, fixturePath] : fixtures) + { + DYNAMIC_SECTION(label) + { + auto pipelineResult = Pipeline::FromSIMPLFile(fixturePath, filterList); + REQUIRE(pipelineResult.valid()); + + auto& pipeline = pipelineResult.value(); + REQUIRE(pipeline.size() == 1); + + auto* pipelineFilter = dynamic_cast(pipeline.at(0)); + REQUIRE(pipelineFilter != nullptr); + + const IFilter* filter = pipelineFilter->getFilter(); + REQUIRE(filter != nullptr); + REQUIRE(filter->uuid() == FilterTraits::uuid); + + CHECK(pipelineFilter->getComments().empty()); + + const Arguments args = pipelineFilter->getArguments(); + CHECK(args.value(ReadCtfDataFilter::k_InputFile_Key) == fs::path("/some/test/path/data.ctf")); + CHECK(args.value(ReadCtfDataFilter::k_DegreesToRadians_Key) == true); + CHECK(args.value(ReadCtfDataFilter::k_EdaxHexagonalAlignment_Key) == false); + CHECK(args.value(ReadCtfDataFilter::k_CreatedImageGeometryPath_Key) == DataPath({"ImageDataContainer"})); + CHECK(args.value(ReadCtfDataFilter::k_CellAttributeMatrixName_Key) == "CellData"); + CHECK(args.value(ReadCtfDataFilter::k_CellEnsembleAttributeMatrixName_Key) == "CellEnsembleData"); + } + } +} diff --git a/src/Plugins/OrientationAnalysis/vv/ReadCtfDataFilter.md b/src/Plugins/OrientationAnalysis/vv/ReadCtfDataFilter.md new file mode 100644 index 0000000000..bf72450c22 --- /dev/null +++ b/src/Plugins/OrientationAnalysis/vv/ReadCtfDataFilter.md @@ -0,0 +1,154 @@ +# V&V Report: ReadCtfDataFilter + +| | | +|--------|--------------| +| Plugin | OrientationAnalysis | +| SIMPLNX UUID | `7751923c-afb9-4032-8372-8078325c69a4` | +| SIMPLNX Human Name | Read Oxford Instr. EBSD Data (.ctf) | +| DREAM3D 6.5.171 equivalent | `ReadCtfData` (SIMPL UUID `d1df969c-0428-53c3-b61d-99ea2bb6da28`) — `Source/Plugins/OrientationAnalysis/OrientationAnalysisFilters/ReadCtfData.{h,cpp}` | +| Verified commit | ** | +| Status | READY FOR REVIEW | +| Sign-off | pending second-engineer review | + +## At a glance + +| Aspect | Current state | +|------------------------|---------------| +| Algorithm Relationship | **Minor changes.** Faithful port of legacy `ReadCtfData` control flow with deliberate deltas: unindexed-point (Phase 0) remap removed (D1/D2, PR #937 — predates this pass), 3D multi-slice support restored (D4, fixed this pass), five malformed-input guards added (D3), Euler math on double-precision intermediates (restores legacy bit-parity), and the legacy PIMPL file-cache dropped (no output effect). | +| Oracle (confirmed) | **Confirmed.** **Class 1 (analytical) + Class 4 (invariant)**, scoped to the filter's value-add per the "don't re-test upstream" rule — EbsdLib (vcpkg 3.1.0) owns `.ctf` parsing and is trusted (Class 2 boundary). Hand-authored inline toy `.ctf` fixtures (3×2 two-phase, 2×2×2 multi-slice; all values float32-exact) with every expected value hand-derived from the fixture text; the two angle transforms (+30° hex alignment, degrees→radians) are correctly-rounded IEEE-754 double-intermediate results derived independently with NumPy. Encoded as 12 TEST_CASEs in `test/ReadCtfDataTest.cpp`; all pass. SIMPLNX matched the oracle with zero discrepancies. | +| Code paths enumerated | 19 of 22 paths exercised (see Code path coverage); the gaps are the unreadable-header passthrough (needs permission manipulation), the file-changed phase-count guard `-19605` (race window inside a single execute; needs injection), and the cancel-signal paths (need injection; untested per scope). | +| Tests today | 12 test cases: Class 1+4 analytical oracle, 4-combo Euler-conversion sweep (DYNAMIC_SECTION) with double-precision-pinning angle values, 2-case 3D multi-slice (DYNAMIC_SECTION), 5 value-add error-guard tests (−19600…−19604 incl. an exact-boundary phase value; −19605 is injection-only), 3 EbsdLib error passthroughs (−102/−105/−107), and SIMPL 6.4/6.5 backwards-compat (new — the filter previously had no conversion test). All inline hand-built fixtures — no exemplar archive. | +| Exemplar archive | **None — retired `6_6_read_ctf_data_2.tar.gz`** (legacy-generated exemplar = forbidden oracle). `download_test_data()` entry removed from `test/CMakeLists.txt`; retirement documented in `vv/provenance/6_6_read_ctf_data_2.md`. Its production Cugrid scan lives on as an A/B fixture in the comparison working folder only. | +| Legacy comparison | **Run (2026-07-24) vs the official DREAM3D 6.5.171 release.** Four runs over three byte-identical input files: toy (2 conversion combos), Cugrid 550×400 production scan, 2×2×2 multi-slice toy. **All numeric outputs bit-identical** — 543,950 of the production scan's 660,000 Euler values match exactly (the 116,050 differing values are all the unindexed points' φ2), and the toy's double-precision-pinning angles match legacy bit-for-bit — except the unindexed-point family: Phases 0→1 remap (D1) and the consequent +30° on unindexed φ2 (D2). Three malformed-input fixtures demonstrated legacy segfaults/silent corruption vs SIMPLNX errors (D3). | +| Bug flags | Legacy: crash/UB on malformed files, **empirically confirmed** (two segfaults, one silent heap-dependent output) — D3. SIMPLNX (pre-pass): multi-slice `.ctf` silently truncated to slice 0 — D4, **fixed this pass** and pinned by the 3D test; latent OOB/null-deref twins of the legacy crashes existed in the NX copy path and were guarded this pass (−19600/−19601/−19602/−19603). | +| V&V phase | Discovery, relationship, oracle, reconciliation, algorithm review (fixes applied), tests, legacy comparison, deviations, provenance, docs — **complete**. Tests pass 12/12 in both `simplnx-Rel` and `simplnx-ooc-Rel` (OOC caveat: that build's out-of-core backend registration is under separate investigation; its pass is reported as-run). Outstanding: second-engineer sign-off at PR review. | + +## Summary + +`ReadCtfDataFilter` ("Read Oxford Instr. EBSD Data (.ctf)") imports a single Oxford/HKL Channel 5 `.ctf` file into a new Image Geometry: it builds the geometry from the header (XCells/YCells/ZCells, XStep/YStep/ZStep, origin 0, Micrometer), creates one cell array per data column plus the condensed 3-component `EulerAngles` and verbatim `Phases` arrays (optionally applying the EDAX hexagonal +30° φ2 alignment and degrees→radians conversion), and populates the ensemble arrays (`CrystalStructures` via EbsdLib's Laue-group mapping, `MaterialName`, `LatticeConstants`) with slot 0 reserved for the "Invalid Phase". Verification is Class 1 analytical + Class 4 invariant on hand-authored inline toy `.ctf` fixtures whose expected outputs were derived independently of both codebases (EbsdLib parsing is trusted upstream and not re-tested). Headline result: SIMPLNX matches the oracle exactly; against DREAM3D 6.5.171 every numeric output is bit-identical on all four fixtures except the deliberate unindexed-point deviations (D1/D2), with one SIMPLNX functional gap (3D multi-slice import, D4) found and fixed during this pass and legacy's malformed-input crashes (D3) empirically demonstrated. All 12 unit tests pass; the legacy-oracle exemplar archive `6_6_read_ctf_data_2.tar.gz` is retired. + +## Algorithm Relationship + +*Classification:* **Minor changes.** + +*Evidence:* Same SIMPL UUID inherited (`d1df969c-…` → SIMPLNX `7751923c-…`) with `FromSIMPLJson` conversion and 6.4/6.5 fixtures at `test/simpl_conversion/{6_4,6_5}/ReadCtfDataFilter.json`. The legacy `ReadCtfData::copyRawEbsdData()` structure (phase copy, Euler interleave with hex-alignment/radians options, verbatim column copies) and `loadMaterialInfo()` (slot-0 defaults + per-phase fill) are preserved block-for-block in `Algorithms/ReadCtfData.cpp`. + +### Port-time deltas (each mapped to a Deviation entry where user-visible) + +1. **Unindexed-point remap removed** (D1, and consequence D2). Legacy remaps phase `< 1 → 1` in the reader buffer before storing `Phases`, which also routes unindexed points into the hex +30° branch; SIMPLNX copies the phase column verbatim (deliberate change, PR #937, May 2024). Changes `Phases` and unindexed-point `EulerAngles` output. +2. **3D multi-slice support restored** (D4, fixed this pass). Legacy sizes the geometry `XCells × YCells × ZCells` with `ZStep` slice thickness; the NX port had hard-coded z=1/1.0 and silently imported only slice 0. Preflight now mirrors legacy (`ZCells` clamped to ≥1; `ZStep` of 0 → 1.0). Bit-identical to legacy on the 3D fixture. +3. **Malformed-input guards added** (D3): empty phase list (−19600), missing data column (−19601), out-of-range phase value (−19602), reader/geometry cell-count mismatch (−19603), zero XCells/YCells at preflight (−19604), and a file-changed phase-count guard in the ensemble fill (−19605, the ReadAngData −19504 analog). Legacy segfaults or silently produces heap-dependent output on these inputs (demonstrated where statically reachable). Rejection paths only; no effect on well-formed files. +4. **Double-precision Euler intermediates.** Legacy computes `+30.0` and `× M_PI/180.0` with double intermediates (float operands promoted); the NX port used float32 arithmetic throughout. Restored double intermediates this pass — more accurate (correctly-rounded float32 results) and bit-identical to legacy on all indexed points of every fixture. No deviation entry needed. +5. **`determineLaueGroup()` → `determineOrientationOpsIndex()`** — the EbsdLib Laue-group→structure mapping was diffed function-body-for-function-body against the 6.5.171 bundled EbsdLib: **identical bodies and identical enum values** (pure rename). No output effect. +6. **Error-code source fix.** `CtfReader::readFile()` leaves its error-code member at 0 on its zero-step/zero-cells rejections (message only); the filter now falls back to the returned code (−102/−103) instead of reporting error 0. Rejection paths only. +7. **Legacy PIMPL file-cache dropped** (`ReadCtfDataPrivate`/`Ctf_Private_Data` — ported as dead declarations, removed during this pass). No output effect; NX re-reads the file on each execute. + +*Material PRs since baseline:* PR #937 (phase-0 remap removal, May 2024 — deviation D1/D2); EbsdLib API migrations (#1122, #1472 — `ebsdlib` namespace, no algorithm change). + +## Oracle + +*Class:* **1 (Analytical) + 4 (Invariant)**; EbsdLib parsing = **Class 2 boundary (trusted, not re-tested)**. + +### The EbsdLib boundary (what we do NOT re-test) + +EbsdLib's `CtfReader` (vcpkg 3.1.0) owns: header-key parsing, phase-section parsing (lattice constants/angles, names, Laue group), data-column tokenizing and typed buffer allocation, European-decimal fix-up, multi-slice line ordering, and its own error codes (−100 unopenable, −102/−103 zero step/cells, −105 premature EOF, −107 unknown column, etc.). Those behaviors are upstream's to verify. The filter's value-add — everything this oracle covers — is the deterministic plumbing on top: geometry construction (including 3D), array creation/typing, verbatim phase copy, Euler interleave + the two optional angle transforms, ensemble sizing + slot-0 defaults, Laue-group index placement, lattice-constant copy, and the six value-add error paths (−19600…−19605). + +### Applied + +Hand-authored toy `.ctf` fixtures live as string literals in the test source: a 3×2 grid (XStep 0.25 / YStep 0.5) with hexagonal "Hex Phase A" (Laue 9) + cubic "Copper" (Laue 11) and two unindexed Phase-0 points, and a 2×2×2 multi-slice variant (ZStep 0.75). Every fixture value is float32-exact (multiples of 1/8), so verbatim copies are asserted with exact equality. Expected outputs — geometry (3,2,1)/(0.25,0.5,1.0)/(0,0,0)/Micrometer and (2,2,2)/(0.25,0.5,0.75), `Phases {1,2,0,1,2,0}` (verbatim, Phase 0 preserved), the 18-value Euler interleave, 7 verbatim pass-through columns, `CrystalStructures {999,0,1}` (Unknown / Hexagonal_High / Cubic_High from documented HKL Laue groups 9 and 11), material names, lattice constants — were derived by hand from the fixture text. The +30° and degrees→radians expectations are the correctly-rounded float32 results of double-precision arithmetic (`value × M_PI/180.0`), computed independently with NumPy IEEE-754 float32/float64 semantics and embedded as exact literals with derivation comments. Class 4 invariants: ensemble tuple count = phases+1; slot-0 Invalid-Phase defaults; `Phases` unaffected by either conversion option; unindexed/cubic points never receive the hex shift. + +*Encoded:* +- `test/ReadCtfDataTest.cpp::"…Class 1 Analytical Oracle"` — full Class 1+4 assertion set (geometry, 9 cell arrays element-wise, 3 ensemble arrays element-wise). +- `…::"Euler Conversion Combinations"` — DYNAMIC_SECTION over the 2×2 DegreesToRadians × EdaxHexagonalAlignment grid, 18 Euler values each, hex shift scoped to Hexagonal_High points only. +- `…::"3D Multi-Slice CTF"` — DYNAMIC_SECTION over ZStep present/absent; dims/spacing/arrays for the 2-slice fixture (regression pin for D4). +- `…::"Empty Phases rejected (-19600)"`, `…::"Missing Data Column rejected (-19601)"`, `…::"Out-of-Range Phase Value rejected (-19602)"`, `…::"Reader/Geometry Cell Count Mismatch rejected (-19603)"`, `…::"Invalid Cells Preflight Error (-19604)"` — value-add guards. +- `…::"EbsdLib Error Passthrough - Zero Step (-102)"`, `…- Truncated Data (-105)"`, `…- Unknown Column (-107)"` — error propagation from the trusted boundary (three distinct return points in `CtfReader`). +- `…::"SIMPL Backwards Compatibility"` — UUID + argument conversion (6.4 and 6.5 fixtures; DYNAMIC_SECTION). + +All 12 pass in `simplnx-Rel` and `simplnx-ooc-Rel`. Reconciliation found zero SIMPLNX-vs-oracle discrepancies (the pre-identified fixes — 3D support, guards, double intermediates — were implemented before the first oracle run). Every fixed delta has a regression pin that fails against the pre-pass code: the guard error codes did not exist, the 3D dims assertion cannot pass against the z=1 hard-code, and — added after the adversarial review found the original angles blind to it — four Euler fixture values (7.125°, 3.375°, 6.75°, and 37.125° via the hex shift) whose correctly-rounded results differ between float32 arithmetic and the double-precision intermediates. + +*Second-engineer review:* skipped — documented reason: the filter's value-add is pure data plumbing plus two elementwise transforms whose expected values are mechanically derivable IEEE-754 roundings, leaving no design freedom for the author-bias failure mode the review guards against. See `vv/provenance/6_6_read_ctf_data_2.md`. + +## Algorithm review + +Line-by-line review performed via the `review-algorithm` skill after oracle reconciliation. All findings applied (all 12 tests pass after rebuild): + +- **Robustness (Critical):** added the five malformed-input guards (−19600…−19604) replacing two demonstrated-segfault paths, one silent OOB-read path, and two zero-dimension paths (details under D3). +- **Correctness:** Euler transforms moved to double-precision intermediates (correctly-rounded results; restores legacy bit-parity). Error-code fallback for `CtfReader` failure paths that set only the message. +- **Dead code:** removed the vestigial legacy PIMPL (`ReadCtfDataPrivate`, `Ctf_Private_Data`), the unused `FloatVec3Type` alias, unused `tDims`/`cDims` locals, a dead `PreflightResult` local, an unused `LaueOps.h` include, and scaffold-generator comments. +- **API consistency:** `loadMaterialInfo`/`copyRawEbsdData` now return `Result<>` (matches post-V&V `ReadAngData`). +- **Progress messaging:** status messages before the EbsdLib read and the cell-data copy (loops are memcpy-speed; no throttled messenger warranted). +- **Cancel checks:** 3 early-return sites added. +- **Naming:** `angPhases` → `ctfPhases`; doc `@brief`s filled in. + +### Five-perspective hardening review (post-deliverable) + +Five independent reviews were run after the deliverables were drafted; all findings applied and all 12 tests re-pass: + +- **Adversarial:** found the conversion suite blind to the double-precision Euler fix (all original angles round identically under float32 and double arithmetic) — fixed by re-deriving the fixture with four diverging angles, re-verified bit-identical against 6.5.171; found a stale −19600 rationale comment and a wrong "mirrors readData" claim for negative ZCells (negative ZCells made `CtfReader::readData()` read zero lines and surface a garbage-value −19602 — now rejected at preflight with −19604 and tested); moved the −19602 test to the exact boundary value; corrected two report wording errors. All hand-derived oracle values, guard-reachability claims, and A/B consistency survived independent recomputation. +- **Senior engineer:** added the −19605 file-changed phase-count guard (the ReadAngData −19504 analog), the preflight error-code fallback, removed dead `getCancel()`, include-what-you-use and sibling-parity cleanups, error-message substring pins and `CheckArraysInheritTupleDims` in every error test, docs grammar/markup fixes. +- **CPU performance:** no action needed — copies are ~5–10% of wall time (EbsdLib text parsing dominates); idioms match the post-V&V ReadAngData baseline; the only lever (bulk store I/O) does not exist on this branch. +- **Memory:** no bugs; peak ≈ 88 bytes/scan-point (reader + destination resident simultaneously) now documented in the filter docs; reader lifetime/cleanup verified correct on all return paths. +- **Out-of-core:** all destination writes are forward-sequential and chunk-cache-benign; the Euler loop's re-read of the just-written Phases array was replaced with the reader's in-core buffer (bit-identical, removes an OOC read-back stream); `copyFromBuffer`-style conversions deferred until that API exists outside the OOC rewrite branch. + +## Code path coverage + +*19 of 22 enumerated paths exercised; the gaps are one passthrough needing permission manipulation (row 1), the file-changed phase-count guard (row 9b), and the cancel checks (row 21). Source: `src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.cpp` (272 lines) + preflight in `Filters/ReadCtfDataFilter.cpp` (268 lines).* Logical phases: **(a)** preflight (header-only read → output actions), **(b)** execute read + ensemble population (`loadMaterialInfo`), **(c)** cell-data copy (`copyRawEbsdData`). + +| # | Phase | Path | Test case | +|----|-------|------|-----------| +| 1 | (a) Preflight | `readHeaderOnly` error → passthrough | *Not directly tested — `FileSystemPathParameter` rejects missing files first; an unreadable-but-present file needs permission manipulation. Same gap accepted for ReadAngDataFilter row 1.* | +| 2 | (a) Preflight | `XCells/YCells < 1` or `ZCells < 0` → `-19604` | `Invalid Cells Preflight Error (-19604)` (DYNAMIC_SECTION: zero XCells; negative ZCells) | +| 3 | (a) Preflight | Geometry action: dims (X,Y,Z≥1), spacing (XStep,YStep,ZStep or 1.0), origin (0,0,0), Micrometer | `Class 1 Analytical Oracle` (2D defaults) + `3D Multi-Slice CTF` (both ZStep variants) | +| 4 | (a) Preflight | Per-column array creation with int32/float32 type dispatch from `CtfFields` | `Class 1 Analytical Oracle` (all 9 cell arrays exist with right types) | +| 5 | (a) Preflight | Ensemble AM sized `phases + 1`; CrystalStructures/LatticeConstants/MaterialName actions | `Class 1 Analytical Oracle` (3 tuples) + `3D Multi-Slice CTF` (2 tuples) | +| 6 | (a) Preflight | Scan/phase preflight-info values (incl. Z line when ZCells > 1) | *Exercised implicitly by every preflight; display-only values, not asserted.* | +| 7 | (b) Execute | `readFile()` error → passthrough (with error-code fallback) | `Zero Step (-102)` (fallback path), `Truncated Data (-105)`, `Unknown Column (-107)` *(other reader codes are the same return statement)* | +| 8 | (b) Execute | `phases.empty()` → `-19600` | `Empty Phases rejected (-19600)` | +| 9 | (b) Execute | All-slot Invalid-Phase default initialization | `Class 1 Analytical Oracle` (slot 0). *Uncovered-slot case is unreachable for CTF — CtfReader assigns indices 1..N contiguously.* | +| 9b | (b) Execute | phase index `>= numTuples` → `-19605` | *Not directly tested. Only reachable if the file gains a phase between the execute-time preflight and the algorithm's re-read — requires file-mutation injection inside a single execute call. Mirrors ReadAngDataFilter's untested `-19504`.* | +| 10 | (b) Execute | Per-phase fill: Laue→structure index, name, lattice constants | `Class 1 Analytical Oracle` (2 phases, 2 Laue classes) | +| 11 | (c) Copy | reader element count `< totalCells` → `-19603` | `Reader/Geometry Cell Count Mismatch rejected (-19603)` — static `ZCells 0` fixture trips it deterministically | +| 12 | (c) Copy | missing data column → `-19601` | `Missing Data Column rejected (-19601)` | +| 13 | (c) Copy | phase value out of `[0, phases]` → `-19602` | `Out-of-Range Phase Value rejected (-19602)` | +| 14 | (c) Copy | Phase verbatim copy (Phase 0 preserved) | `Class 1 Analytical Oracle` (points 2, 5) | +| 15 | (c) Copy | Euler interleave `3i/3i+1/3i+2` | `Class 1 Analytical Oracle` (18 values) | +| 16 | (c) Copy | Hex-alignment +30° branch (Hexagonal_High points only, option on) | `Euler Conversion Combinations` (points 0/3 shifted; cubic and Phase-0 points pinned unshifted) | +| 17 | (c) Copy | Degrees→radians branch | `Euler Conversion Combinations` (both radians combos) | +| 18 | (c) Copy | 7 verbatim pass-through columns (Bands, Error, MAD, BC, BS, X, Y) | `Class 1 Analytical Oracle` (element-wise) | +| 19 | (c) Copy | Multi-slice data volume (x·y·z cells) | `3D Multi-Slice CTF` (8 cells across 2 slices, element-wise) | +| 20 | — | SIMPL 6.4/6.5 parameter conversion | `SIMPL Backwards Compatibility` | +| 21 | (b)/(c) | Cancel checks (3 sites) | *Not directly tested. Requires cancel-signal injection; standard early-return pattern. Excluded from scope by direction.* | + +## Test inventory + +| Test case | Status | Notes | +|-----------|--------|-------| +| `OrientationAnalysis::ReadCtfDataFilter: Class 1 Analytical Oracle` | new-for-V&V | Class 1 + 4. Geometry (dims/spacing/origin/units), 9 cell arrays and 3 ensemble arrays asserted element-wise (~80 assertions), all expected values hand-derived, float32-exact fixture. Inline data. | +| `…: Euler Conversion Combinations` | new-for-V&V | Class 1. DYNAMIC_SECTION over the 2×2 conversion-option grid; NumPy-derived correctly-rounded literals; pins hex shift scoping (never Phase-0/cubic points). | +| `…: 3D Multi-Slice CTF` | new-for-V&V | Class 1. DYNAMIC_SECTION (ZStep 0.75 / absent). Regression pin for D4 (3D support restored this pass). | +| `…: Empty Phases rejected (-19600)` | new-for-V&V | Value-add guard. Legacy segfaults on this fixture (demonstrated, D3). | +| `…: Missing Data Column rejected (-19601)` | new-for-V&V | Value-add guard. Legacy segfaults on this fixture (demonstrated, D3). | +| `…: Out-of-Range Phase Value rejected (-19602)` | new-for-V&V | Value-add guard. Legacy silently succeeds with heap-dependent output (demonstrated, D3). | +| `…: Reader/Geometry Cell Count Mismatch rejected (-19603)` | new-for-V&V | Value-add guard. Static `ZCells 0` fixture — no file-mutation injection needed. | +| `…: Invalid Cells Preflight Error (-19604)` | new-for-V&V | Value-add preflight rejection (`XCells 0`). | +| `…: EbsdLib Error Passthrough - Zero Step (-102)` | new-for-V&V | Exercises the error-code fallback (reader sets message only). | +| `…: EbsdLib Error Passthrough - Truncated Data (-105)` | new-for-V&V | Premature-EOF return point. | +| `…: EbsdLib Error Passthrough - Unknown Column (-107)` | new-for-V&V | Column-allocation return point. | +| `…: SIMPL Backwards Compatibility` | new-for-V&V | DYNAMIC_SECTION over the existing 6.4/6.5 conversion fixtures — the filter previously had **no** conversion test despite shipping the fixtures. | +| *(retired)* `OrientationAnalysis::ReadCtfData: Valid Execution` | retired | Exemplar comparison against `6_6_read_ctf_data.dream3d` — **legacy-generated oracle** (forbidden). Replaced by the Class 1 oracle above. | + +## Exemplar archive + +- **Archive:** None. **`6_6_read_ctf_data_2.tar.gz` retired** this pass (SHA512 was `f397fa3b…ad1707`); its `download_test_data()` entry is removed from `test/CMakeLists.txt`. +- **Provenance:** `src/Plugins/OrientationAnalysis/vv/provenance/6_6_read_ctf_data_2.md` — documents the legacy-oracle status and the inline replacement. +- All current oracle data is inline in `test/ReadCtfDataTest.cpp`; there is nothing to archive. + +## Deviations from DREAM3D 6.5.171 + +*Comparison run 2026-07-24 against the official DREAM3D 6.5.171 release on four byte-identical-input fixtures (toy ×2 conversion combos, Cugrid 550×400 production scan, 2×2×2 multi-slice toy) plus three malformed-input fixtures. All numeric outputs bit-identical outside the entries below. Full write-ups: `vv/deviations/ReadCtfDataFilter.md`; working artifacts archived to OneDrive (comparison working folder).* + +- `ReadCtfDataFilter-D1` — Unindexed (Phase 0) points: legacy remaps to phase 1, SIMPLNX preserves 0 (deliberate, PR #937). 116,050 points on the production fixture. Trust SIMPLNX. +- `ReadCtfDataFilter-D2` — Consequence of D1: legacy applies the hexagonal +30° φ2 shift to unindexed points; SIMPLNX never does. Exactly 0.5235988 rad on affected values. Trust SIMPLNX. +- `ReadCtfDataFilter-D3` — Malformed-input behavior: legacy segfaults (missing column; empty phases — both demonstrated, exit 139) or silently emits heap-dependent output (out-of-range phase value — demonstrated, exit 0); SIMPLNX rejects with −19600…−19604. Trust SIMPLNX. +- `ReadCtfDataFilter-D4` — *(retired)* Multi-slice `.ctf` silently truncated to slice 0 by earlier DREAM3D-NX releases; fixed this pass, now bit-identical to legacy 3D output. Trust SIMPLNX (current). diff --git a/src/Plugins/OrientationAnalysis/vv/deviations/ReadCtfDataFilter.md b/src/Plugins/OrientationAnalysis/vv/deviations/ReadCtfDataFilter.md new file mode 100644 index 0000000000..7b057494cc --- /dev/null +++ b/src/Plugins/OrientationAnalysis/vv/deviations/ReadCtfDataFilter.md @@ -0,0 +1,87 @@ +# Deviations from DREAM3D 6.5.171: ReadCtfDataFilter + +This file lists every documented behavioral difference between this SIMPLNX filter and its DREAM3D 6.5.171 equivalent. + +Entries are referenced by stable ID (`ReadCtfDataFilter-D`) from the V&V report and from public migration guidance. The ID is stable across renames; the Filter UUID field is the permanent cross-reference anchor. + +Comparison run 2026-07-24 against the official DREAM3D 6.5.171 release (`PipelineRunner`): four runs over three byte-identical input files — a hand-authored 3×2 toy `.ctf` (two conversion-option combinations), the 550×400 production scan `Cugrid_after 2nd_15kv_2kx_2.ctf` (two phases — Cu3Ga hexagonal + Copper cubic — with 116,050 unindexed points), and a 2×2×2 multi-slice toy `.ctf`. On every run, all numeric outputs are **bit-identical** except the unindexed-point (Phase 0) family documented as D1/D2 (on the production scan: 543,950 of 660,000 Euler values match; the 116,050 differing values are all unindexed points' φ2). Three additional malformed-input fixtures were run through 6.5.171 to document its failure behavior (D3). + +--- + +## ReadCtfDataFilter-D1 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadCtfDataFilter-D1` | +| **Filter UUID** | `7751923c-afb9-4032-8372-8078325c69a4` | +| **Status** | active | + +**Symptom:** Unindexed scan points ("zero solutions") carry `Phases == 0` in SIMPLNX but `Phases == 1` in 6.5.171. On the Cugrid production fixture this affects 116,050 of 220,000 points. + +**Root cause:** Algorithmic choice. 6.5.171's `ReadCtfData::copyRawEbsdData()` remaps every phase value `< 1` to `1` before storing the Phases array (a convention inherited from `H5CtfVolumeReader`: "the lowest value is One (1)"). SIMPLNX deliberately removed this remap in PR #937 (May 2024): a phase value of 0 is meaningful in `.ctf` files — it marks a point the indexing software could not solve — and rewriting it to 1 silently assigns those points to a real phase. + +**Affected users:** Anyone importing `.ctf` files that contain unindexed points — which is nearly every real-world Oxford/HKL scan. Downstream workflows that segmented or masked using `Phases == 0` under legacy DREAM3D never saw phase 0 and instead thresholded on `Error > 0`; that workflow continues to work identically in SIMPLNX (`Error` is copied verbatim, bit-identical). + +**Recommendation:** Trust SIMPLNX. Preserving the measured phase value is strictly more information; the legacy remap destroyed the unindexed marker and mislabeled unindexed points as phase 1. Users who need legacy-parity masks can threshold on `Error = 0` (recommended in the filter documentation) or replace phase-0 values explicitly. + +--- + +## ReadCtfDataFilter-D2 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadCtfDataFilter-D2` | +| **Filter UUID** | `7751923c-afb9-4032-8372-8078325c69a4` | +| **Status** | active | + +**Symptom:** When "Convert Hexagonal X-Axis to EDAX Standard" is enabled and the file's phase 1 is hexagonal (Laue group 9), the φ2 Euler angle of **unindexed** points differs by exactly 30° (0.5235988 rad): 6.5.171 applies the +30° shift to them, SIMPLNX does not. On the Cugrid fixture, all 116,050 unindexed points show legacy φ2 = 0.5235988 vs SIMPLNX φ2 = 0.0. Euler angles of every indexed point are bit-identical. + +**Root cause:** Algorithmic choice (consequence of D1). Both versions decide the +30° hexagonal-alignment shift by looking up `CrystalStructures[Phases[i]]`. Legacy has already remapped unindexed points to phase 1, so they inherit phase 1's hexagonal classification and receive the shift; SIMPLNX looks up ensemble slot 0 (`UnknownCrystalStructure`, 999), which is never hexagonal, so unindexed points keep their file values. + +**Affected users:** Only workflows that read Euler angles of unindexed points under the EDAX-alignment option — those angles are meaningless by definition (the point was never indexed; the file stores 0.0 there). + +**Recommendation:** Trust SIMPLNX. Applying a crystallographic correction to points with no crystallographic solution was an artifact of the D1 remap, not a deliberate feature. + +--- + +## ReadCtfDataFilter-D3 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadCtfDataFilter-D3` | +| **Filter UUID** | `7751923c-afb9-4032-8372-8078325c69a4` | +| **Status** | active | + +**Symptom:** Malformed `.ctf` files that crash or silently corrupt 6.5.171 are rejected by SIMPLNX with descriptive errors. Empirically demonstrated on three fixtures against the official 6.5.171 release: + +| Malformed input | DREAM3D 6.5.171 (demonstrated) | SIMPLNX | +|---|---|---| +| Data section missing a standard column (no `BS`) | **Segfault** (exit 139; `memcpy` from null buffer) | Error `-19601` naming the missing column | +| `Phases 0` header (no phase definitions) | **Segfault** (exit 139; reads a zero-tuple ensemble array out of bounds) | Error `-19600` | +| Data row phase value out of range (`Phase 7` with 2 phases) | **Silent success** (exit 0) — the +30° decision reads `CrystalStructures[7]` out of bounds, so the output depends on heap contents | Error `-19602` naming the point and valid range | +| Header `XCells`/`YCells` missing or 0 | Error at execute after a zero-sized preflight | Rejected at preflight with `-19604` | +| Header `ZCells 0` | Not run (legacy sizes the geometry from the same header) | Rejected at execute with `-19603` | + +**Root cause:** Bug in 6.5.171 (out-of-bounds access on malformed input) — fixed in SIMPLNX by value-add guards added during this V&V pass. On well-formed files the guards never fire and outputs are unaffected. + +**Affected users:** Anyone importing truncated, hand-edited, or non-standard `.ctf` files. No effect on well-formed files. + +**Recommendation:** Trust SIMPLNX. A clean error is strictly better than a crash or a heap-dependent output. + +--- + +## ReadCtfDataFilter-D4 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadCtfDataFilter-D4` | +| **Filter UUID** | `7751923c-afb9-4032-8372-8078325c69a4` | +| **Status** | retired 2026-07-24 (resolved during this V&V pass — recorded for the migration guide) | + +**Symptom:** Before this V&V pass, SIMPLNX imported only slice 0 of a multi-slice (3D) `.ctf` file (`ZCells > 1`) — silently — while 6.5.171 imported the full 3D volume. + +**Root cause:** Bug in the SIMPLNX port (the preflight hard-coded a z-extent of 1 and a z-spacing of 1.0, dropping the header's `ZCells`/`ZStep`). Restored during this pass; on the 2×2×2 fixture the SIMPLNX output now matches 6.5.171 bit-for-bit (dims, ZStep-derived spacing, and all nine cell arrays). + +**Affected users:** Anyone who imported a multi-slice `.ctf` with an earlier DREAM3D-NX release received a single-slice geometry with slice-0 data and no warning. + +**Recommendation:** Trust SIMPLNX (current). Re-import multi-slice `.ctf` files with a build containing this fix. diff --git a/src/Plugins/OrientationAnalysis/vv/provenance/6_6_read_ctf_data_2.md b/src/Plugins/OrientationAnalysis/vv/provenance/6_6_read_ctf_data_2.md new file mode 100644 index 0000000000..2be9c654c7 --- /dev/null +++ b/src/Plugins/OrientationAnalysis/vv/provenance/6_6_read_ctf_data_2.md @@ -0,0 +1,45 @@ +# Exemplar Archive Provenance: 6_6_read_ctf_data_2.tar.gz — RETIRED + +This sidecar documents the **retirement** of `6_6_read_ctf_data_2.tar.gz` during the ReadCtfDataFilter V&V pass (2026-07-24). The archive is no longer referenced by any test or `download_test_data()` entry. + +--- + +## Archive identity (at retirement) + +| Field | Value | +|---|---| +| **Archive** | `6_6_read_ctf_data_2.tar.gz` | +| **SHA512** | `f397fa3bf457615a90a4b48eaafded2aa4952b41ccb28d9da6a83adc38aea9c22f2bb5a955f251edeca9ef8265b6bf1d74e829b1340f45cf52620a237aad1707` | +| **Contents** | `6_6_read_ctf_data.dream3d` + `.xdmf` (legacy-generated exemplar), `Cugrid_after 2nd_15kv_2kx_2.ctf` (550×400 production scan, Cu3Ga hexagonal + Copper cubic) | +| **Was used by tests** | `OrientationAnalysis::ReadCtfData: Valid Execution` (retired/replaced) | +| **Retired by** | Michael Jackson | +| **Retired on** | 2026-07-24 | + +## Why it was retired + +**Legacy/circular oracle.** The archive's exemplar `.dream3d` was generated by a legacy DREAM3D 6.6 run of this same import path (`6_6_` prefix marks legacy-generated data), so the "Valid Execution" test verified only agreement with a prior DREAM3D output — exactly the "legacy produced this output" pattern the V&V policy forbids as a correctness oracle. Per the standing project rule, it was replaced with inline toy data rather than regenerated. + +## Replacement + +The oracle is now **inline in `test/ReadCtfDataTest.cpp`** (no archive at all): + +- A hand-authored toy `.ctf` fixture (string literal, written to the binary test-output dir at runtime) with **Class 1 hand-derived expected values** for the geometry and all 12 output arrays, plus **Class 4 invariants** (ensemble sizing, Invalid-Phase slot defaults). +- A 2×2 parameter sweep over DegreesToRadians × EdaxHexagonalAlignment whose expected Euler values are the correctly-rounded float32 results of double-precision arithmetic, computed independently with NumPy IEEE-754 float32/float64 semantics from the fixture text (constant: `M_PI/180.0`; derivation commented beside the literals). Four fixture angles (7.125°, 3.375°, 6.75°, and 37.125° via the hex shift) were deliberately chosen where the float32 and double-precision roundings **differ**, so the assertions pin the required double-precision code path (added after the adversarial review showed the original angles were blind to it). +- A 2×2×2 multi-slice fixture for the 3D `.ctf` support restored during this pass. +- Eight error fixtures: value-add guards (`-19600`…`-19604`; `-19605` is reachable only by file-mutation injection and is documented untested) and EbsdLib passthroughs (`-102`, `-105`, `-107`). + +See the report's Oracle section: `src/Plugins/OrientationAnalysis/vv/ReadCtfDataFilter.md`. + +The archive's production scan (`Cugrid_after 2nd_15kv_2kx_2.ctf`) remains valuable as an **A/B comparison fixture** (not an oracle); it is used in the OneDrive-archived comparison working folder, not by any unit test. + +## Second-engineer oracle review + +Skipped — documented reason (same rationale as `read_ang_test.md`): the filter's value-add is deterministic data plumbing (grid math from header values, verbatim column copies, a documented HKL Laue-group enum mapping, and two elementwise angle transforms whose expected values are mechanically derivable IEEE-754 roundings). There is no design freedom for the author-bias failure mode the review guards against; the fixture and every derivation are visible in the test source for any reviewer to audit at sign-off. + +## Circular-oracle note + +This retirement **removes** a legacy-generated oracle. The replacement was derived by hand (and independently with NumPy for the two angle transforms) from the fixture text before any DREAM3D run — verified order: oracle encoded and reconciled first (2026-07-24, zero discrepancies), then the 6.5.171 comparison (2026-07-24; bit-identical numerics except the documented unindexed-point deviations D1/D2). + +## Archive disposition + +The tar.gz remains in the GitHub [Data_Archive release](https://github.com/BlueQuartzSoftware/simplnx/releases/tag/Data_Archive) for historical checkouts of older commits; it must not be referenced by new code.