It is very important to know (and I think not documented) that QUARK module are instantiated once and the instance than is used for all backlog items and iterations. This can lead to hard to find problems if the module uses some state
(I had the concrete problem that I had an expensive helper method which I used several times in the postprocessing and which stored its result for reuse - but if the result depends on the backlog item or the repetition one has to be very careful to reset the cache appropriately)
I think there are situations where it can help to have the same instance during the whole QUARK run but these are probably exceptions. May be the default should be that the QUARK modules are newly instantiated on every backlog item/repetition but a module can somehow tell the benchmark manager to do no reinitialization ('singleton-mode')
I know this would be a problematic compatibility issue - may be one could force the module implementations to provide a method 'is_singleton'?)
But in any case the behaviour should be prominently documented for the developers.
It is very important to know (and I think not documented) that QUARK module are instantiated once and the instance than is used for all backlog items and iterations. This can lead to hard to find problems if the module uses some state
(I had the concrete problem that I had an expensive helper method which I used several times in the postprocessing and which stored its result for reuse - but if the result depends on the backlog item or the repetition one has to be very careful to reset the cache appropriately)
I think there are situations where it can help to have the same instance during the whole QUARK run but these are probably exceptions. May be the default should be that the QUARK modules are newly instantiated on every backlog item/repetition but a module can somehow tell the benchmark manager to do no reinitialization ('singleton-mode')
I know this would be a problematic compatibility issue - may be one could force the module implementations to provide a method 'is_singleton'?)
But in any case the behaviour should be prominently documented for the developers.