Our data is chunked, and we typically don't analyze all of the data at once. Currently, the timestamps that are generated during processing assume that the data is contiguous, but we would also like there to be a file that is generated with the absolute times (i.e. the global time of each data point). One solution we discussed was to parse the .json files at job dispatch and then generate absolute time files at the preprocessing and downstream processing stages. The goal here is allow us to better visualize our data at different temporal resolutions. Please let me know if you have any questions and thank you!
Our data is chunked, and we typically don't analyze all of the data at once. Currently, the timestamps that are generated during processing assume that the data is contiguous, but we would also like there to be a file that is generated with the absolute times (i.e. the global time of each data point). One solution we discussed was to parse the .json files at job dispatch and then generate absolute time files at the preprocessing and downstream processing stages. The goal here is allow us to better visualize our data at different temporal resolutions. Please let me know if you have any questions and thank you!