(closes #3262) Update lfric gpu script and collect aggregated transformation stats

[sergisiso]

No description provided.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (3e99eac) to head (b49b8f8).

Additional details and impacted files

@@            Coverage Diff            @@
##            master     #3450   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          393       393           
  Lines        54918     54918           
=========================================
  Hits         54918     54918           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sergisiso]

@arporter @LonelyCat124 This is ready for review, its a small modification to the gpu offloading script but also adds the aggregated stats that I have been reporting. The ITs now show:

Should we also add integration checks for the numbers, eg "if number of module-inlined successes is < 277: fail"?

[LonelyCat124]

I think having checks that we don't make this do worse is probably a good idea, if we have a good reason to make things worse then we can manually make that change as/when it happens.

[LonelyCat124]

I think having checks that we don't make this do worse is probably a good idea, if we have a good reason to make things worse then we can manually make that change as/when it happens.

I thought about this more - having an option to see which expected files are missed would be helpful for future PRs that cause issues.

[sergisiso]

@arporter @LonelyCat124 For now I added the suggested threshold checks. Listing which kernels diverged is harder because it requires keeping a full log of a working run, and this are not very stable: parallel runs, reflect any change, ... so for now I haven't done this second step but I maybe it can be a follow on PR.