Retry a learned artifact's pickle once, after a collection

[amburger66]

Validated by a controlled experiment

The green CI on this branch alone would NOT have been evidence: this PR adds two
tests, which changes pytest-split's packing -- the same mechanism that moved
the failure between shards 8, 5 and 6 across #138/#139/#140. A green run was
therefore consistent with "the retry worked" and with "the cluster relocated".

So the fix was tested against a fixed packing. Branch control-pickle-fix is
PR #138's exact failing head (901696d) plus only the production half of this
PR -- the helper and its two call sites, no test files touched, so the test
set, the packing and shard 6's contents are byte-identical to the run that
failed.

tree	shard 6
#138 head, no fix	failed -- test_oracle_samplers, test_main, test_skip_initial_test
#138 head, no fix (re-run of the untouched job)	failed -- same three
#138 head + retry only	green (8/8 shards)
#138 head + retry only (re-run)	green (8/8 shards)

Same tree, same tests, same packing; the retry is the only variable.

This also promotes the hypothesis to a finding. The body below describes the
weak-reference/collection-timing story as an untested guess, which is what it
was when written. A gc.collect() plus one retry converting a twice-reproduced
failure into two clean runs is what that hypothesis predicts, and nothing else
in the diff could produce it. The root cause -- which object graph reaches an
_abc_data, and why -- is still not identified, and this remains a mitigation
rather than a repair.

control-pickle-fix is an experiment branch marked "Not for merge"; it should
be deleted once this lands.

---

Saving learned NSRTs and GNN weights intermittently dies with
TypeError: cannot pickle '_abc._abc_data' object. It has blocked three PRs on
three different shards this week (#138, #139, #140) and it predates all of them
-- the same nine-test cluster reproduces on unmodified master.

What is established

Two call sites, both reached from the CI tracebacks:
nsrt_learning_approach.py:114 (pkl.dump(self._nsrts, f)) and
gnn_approach.py:207 (pkl.dump(info, f)). Every failure seen so far funnels
through one of them.

On CI it is deterministic for a given test set: shard 8 failed twice on the
same single test, shard 6 twice on the same three, across re-runs of untouched
jobs. Which shard is hit moves with pytest-split's packing, which is why
adding tests to an unrelated PR looks like it "causes" the failure.

Locally it is stochastic -- 3 of 12 identical runs, with code, execution
order and PYTHONHASHSEED all fixed. The likely difference is that CI
containers are uniform and a developer machine is not.

What is NOT established

The root cause. An _abc_data holds weak references, so whether dill
trips over one plausibly turns on collection timing -- the one thing that still
varies with everything else pinned. That is the hypothesis this mitigation
targets, and it is a hypothesis; I would rather say so than dress it up.

If the failures stop, that is the evidence for it. If they do not, it rules
the hypothesis out, which is worth knowing too. That is the main argument for
landing this: it converts an unfalsifiable annoyance into a decided question.

Ruled out along the way, none of them the cause:

candidate	why not
pytest-randomly	not installed -- -p no:randomly is a silent no-op here
execution order	progress lines match character-for-character, red run vs green
hash randomisation	PYTHONHASHSEED=0 is exported for every run
a single polluting test	a bisection appeared to find one; the control showed removing its neighbours worked equally well

That last one deserves a note: the bisection was invalid, because it used
single runs to measure a 25% event. Measuring the rate first would have shown
each step had a ~75% chance of a spurious pass.

Why the fix is shaped this way

Serialise to bytes, then write -- rather than retry into the open handle. A
dump that raises part-way has already written a prefix, and appending a retry to
that leaves a corrupt file which only fails at load time, long after the run
that produced it could have been repeated. A test pins the file being empty
after a persistent failure.

Catch any TypeError, not just the _abc_data one. Matching on the message
would break silently when it is reworded, and an object that is genuinely
unpicklable fails the second time too and raises exactly as before -- so the
broad catch costs one wasted attempt and hides nothing.

Tests

Two, both mutation-checked:

• removing the retry fails the first and not the second
• dumping straight into the handle fails both

mypy clean, pylint 10.00/10, yapf/docformatter/isort applied. The previously
failing shard passes locally -- though at a 25% local rate a single clean run is
weak evidence, which is exactly why CI is the test that matters here.

Unrelated thing noticed

Running the suite rewrites
tests/datasets/mock_vlm_datasets/ice_tea_making__demo+labelled_atoms__manual__1.txt
in the working tree. It is a test writing into the repo, it nearly rode along in
this commit twice, and it is not fixed here.