Reproduction Results Differ from Reported Results — Seeking Possible Causes

[huhotel]

First, thank you very much for this excellent work! The project is extremely valuable for image restoration research and provides strong inspiration for our ongoing work.
During reproduction, we did not modify any of the configurations or default parameters, and strictly followed the settings provided in the repository. However, when running the Multiple-Degradation IR task on the MiO100 dataset using the GenMIR-P Profile, our reproduced results exhibit noticeable gaps compared with the reported scores, especially on RP(Reference Point) related metrics.
In addition, due to limitations in our CUDA version and environment, we had to exclude MAXIM and RIDCP from our available toolset. To verify whether this caused inconsistencies, we compared the per-image plan generated in our run with the plan provided by the authors. Interestingly, besides skipping MAXIM and RIDCP, we observed:
The order of subtasks sometimes differs from the authors' plan
Even when the subtask order matches, the selected optimal tool for a subtask may differ
These inconsistencies may further contribute to the performance gaps we observed.
Our reproduced results screenshot:

Therefore, we would like to ask:
What might be the possible reasons for these discrepancies?
For example, could they be related to:
Environment versions (CUDA / cuDNN / PyTorch)
Model weight versions
Dataset preprocessing details
MiO100 dataset source or unpacking details
Inference script or internal parameters
etc.?
Thank you very much for your time and for providing such an impressive work!

[cuishuang99]

I encountered a similar problem during my reproduction, but due to the CUDA version and the lack of maxim, my results differed significantly from the author's.

[YushenZuo]

Thank you for your email and for your interest in our work! It is very encouraging to hear that the project has been valuable and inspiring for your ongoing research.

Regarding the discrepancies you observed when reproducing the Multiple-Degradation IR task on the MiO-100 dataset using the GenMIR-P profile, particularly the noticeable gaps in RP (Reference Point) related metrics and differences in the generated plans, I would like to explain the likely causes.

Under the GenMIR-P profile, the 4KAgent pipeline involves Perception (using DepictQA to detect degradations), Planning (using GPT-4 to formulate the restoration plan), Execution, and Reflection (including the rollback mechanism). The GenMIR-P profile follows a design similar to AgenticIR [1], and as shown in related experiments (e.g., Figure 13 in the AgenticIR paper), planning stability is generally high but not perfect. In certain complex degradation combinations, the generated plan can vary across runs, leading to differences in subtask order. This inherent variability in LLM/VLM outputs is expected and can explain the plan differences you observed, even when the input images are identical.

Additionally, you mentioned excluding MAXIM and RIDCP due to CUDA version and environment constraints. This will indeed impact results, as the reported scores assume a complete toolbox. Missing tools can alter the overall restoration pipeline, even if the plan simply skips them.

Another important factor is the Reflection and Rollback mechanism during execution. The model checkpoints used in the toolbox in 4KAgent are listed in https://github.com/taco-group/4KAgent/blob/main/Toolbox.md. Please verify that the checkpoints loaded at runtime match these exactly. Different versions of Pytorch, CUDA, and CuDNN may also alter the result image, even using the same checkpoint. After each tool generates a candidate restored image, 4KAgent computes a quality score to select the best one and to decide whether to proceed or rollback. This quality score is a weighted combination of no-reference metrics: NIQE, MANIQA, MUSIQ, and CLIPIQA, calculated using the pyiqa library (version 0.1.13). Any difference in the pyiqa version can alter these scores, affecting tool selection and rollback decisions.

Rollback is triggered based on the quality score and a predefined threshold (default: 0.5). If rollback occurs and replanning is invoked, the final execution sequence and restored image can change significantly. As the quality score relies entirely on no-reference metrics, the no-reference metrics (e.g., MANIQA, CLIPIQA, MUSIQ) in your reproduction are close to reported values. Finally, the script we used in evaluation is https://github.com/taco-group/4KAgent/blob/main/eval/test_metrics_mio.py.

To conclude, potential sources of discrepancy include:

1. Variability in Perception and Planning modules.
2. Model checkpoints and runtime environment: Differences in model checkpoints, PyTorch, CUDA, or CuDNN versions can also affect model outputs.
3. Incomplete toolbox.
4. pyiqa library version mismatch.
5. Modified rollback threshold.
6. Difference in the evaluation script.

I will also provide the package versions required in the environments for 4KAgent.

Please feel free to reach out with any further questions or updates. I look forward to hearing about your progress and wish you all the best with your research!

[1] Zhu, K., Gu, J., You, Z., Qiao, Y., & Dong, C. (2024). An intelligent agentic system for complex image restoration problems. arXiv preprint arXiv:2410.17809.

Best regards,
Yushen Zuo

[YushenZuo]

Hello, thank you for your interest.

I have checked the evaluation code in AgenticIR, which uses the default settings when computing PSNR and SSIM
(Code: https://github.com/Kaiwen-Zhu/AgenticIR/blob/main/utils/scorer.py).

In pyiqa, the default configurations differ: SSIM is computed in the Y channel in the YIQ color space, while PSNR is computed in RGB color space.
Please refer to:
• chaofengc/IQA-PyTorch#167
• https://github.com/chaofengc/IQA-PyTorch/blob/main/pyiqa/archs/psnr_arch.py
• https://github.com/chaofengc/IQA-PyTorch/blob/main/pyiqa/archs/ssim_arch.py

As a result, there is a misalignment in the evaluation settings: our implementation computes both PSNR and SSIM in Y channel in the YCbCr color space, whereas AgenticIR computes SSIM in the Y channel in the YIQ and PSNR in RGB.

Thank you for pointing out this issue. We will update the evaluation code accordingly and revise the corresponding results in the camera-ready version of our paper.

Best regards,
Yushen Zuo

[cuishuang99]

OK. Revising the evaluation code and the corresponding results is essential for further research in this field.