files.json

[dbirman]

Coordinate with data infrastructure about how to enforce file standards using a new files.json core file.

[dbirman]

Some things to try:

• Can we just store the actual mlcroissant spec and use a validator for mlcroissant, to avoid needing our own validation code?
• If we store the spec, how do we do name validation for files?
• Should we store mlcroissant but make it easy to convert back-and-forth to pydantic?

[dbirman]

After quick discussion w/ David:

• files.json will be a Pydantic object built from a list of FileSet objects. We will define a standard enum that allows for certain kinds of folder/file names to be validated against. For now "modality" (from registry list) and "device_name" (from get_component_names())
• to_croissant() and from_croissant() functions, from_croissant we need to confirm if we can recover the enum values for validation? We won't create the croissant.json files directly.
• aind-file-standards will have the actual FileSet objects for each standard and a croissant dump that lives in a subfolder, aind-metadata-mapper will auto-generate based on modality if a new setting is turned on.