One has to be very careful at where to place self.metrics.add_metric calls.
The problem we had and which took us a while to understand was that:
We have devices which have a method submit. This submit method is called from within the postprocess method of our solvers. When we added a self.metrics.add_metric call to the device.submit method the corresponding entry in the results.json appears not as part of the current backlog item but as part of the next backlog item - if there is one.
This is because at the time the device submit method is called the postprocessing of the device has already been finished (the device is actually a device factory).
It might be helpful to throw an error (or warning?) if add_metric is called for a module which already has been postprocessed (this is probably possible only on the last repetition of the last backlog item - or one allows add_metric calls only inbetween calling preprocess and ending postprocess - but that might be too strict).
In any case should the user manual contain a paragraph about how to write metrics.
One has to be very careful at where to place self.metrics.add_metric calls.
The problem we had and which took us a while to understand was that:
We have devices which have a method submit. This submit method is called from within the postprocess method of our solvers. When we added a self.metrics.add_metric call to the device.submit method the corresponding entry in the results.json appears not as part of the current backlog item but as part of the next backlog item - if there is one.
This is because at the time the device submit method is called the postprocessing of the device has already been finished (the device is actually a device factory).
It might be helpful to throw an error (or warning?) if add_metric is called for a module which already has been postprocessed (this is probably possible only on the last repetition of the last backlog item - or one allows add_metric calls only inbetween calling preprocess and ending postprocess - but that might be too strict).
In any case should the user manual contain a paragraph about how to write metrics.