BUG: overriding intermediate data when no outputs asked

[ankostis]

In the following diagram, all data are given, and asked is different, depending on whether we expicitly ask it in the outputs:

• when asked output asked, it checks overriden is already given and dose not recompute it,
• without outputs, it overrides it.

Code to reproduce it:

def test_pruning_not_overrides_given_intermediate():
    # Test #25: not overriding intermediate data when an output is not asked
    graph = compose(name="graph")(
        operation(name="unjustly run", needs=["a"], provides=["overriden"])(lambda a: a),
        operation(name="op", needs=["overriden", "c"], provides=["asked"])(add),
    )
    graph.net.plot('t.png')
    assert graph({'a': 5, 'overriden': 1, "c": 2}, ['asked']) == {'asked': 3}  # that's ok
    assert graph({'a': 5, 'overriden': 1, "c": 2}) == {'a': 5, 'overriden': 1, "c": 2, 'asked': 3}  # FAILs

Root cause:

• The code that prunes the predecessors of intermediate data runs only when outputs asked.
• The existing "pruning" TCs were too simple to catch this.

Note that the pruning code in v1.2.4 is buggy (#24), so it cannot be used as is.

[ankostis]

Another "too pruning" but more complex example is when a multi-output operation MUST NOT run, not to override a given intermediate input, but it MUST run, to provide other outputs.

In the diagram below, must run is needed for e but must not override overriden.

Code to reproduce:

def test_pruning_multiouts_not_override_intermediates():
    # Test #25: v.1.2.4 overrides intermediate data when a previous operation
    # must run for its other outputs (outputs asked or not)
    netop = compose(name="netop")(
        operation(name="must run", needs=["a"], provides=["overriden", "e"])
        (lambda x: (x, 2 * x)),
        operation(name="op1", needs=["overriden", "c"], provides=["d"])(add),
        operation(name="op2", needs=["d", "e"], provides=["asked"])(lambda x, y: x * y),
    )

    # FAILs
    # - on v1.2.4 with KeyError: 'e',
    # - # - on #18(unsatisfied) + #23(ordered-sets) with empty result.
    assert netop({"a": 5, "overriden": 1, "c": 2}, ["asked"]) == {"asked": 3}
    # FAILs
    # - on v1.2.4 with (overriden, asked) = (5, 70) instead of (1, 13)
    # - # - on #18(unsatisfied) + #23(ordered-sets) like v1.2.4.
    assert (
        netop({"a": 5, "overriden": 1, "c": 2})
        ==
        {"a": 5, "overriden": 1, "c": 2, "asked": 3})

[ankostis]

Or this even simpler one:

def test_pruning_multiouts_not_override_intermediates1():
    # Test #25: v.1.2.4 overrides intermediate data when a previous operation
    # must run for its other outputs (outputs asked or not)
    netop = compose(name="netop")(
        operation(name="must run", needs=["a"], provides=["overriden", "calced"])
        (lambda x: (x, 2 * x)),
        operation(name="add", needs=["overriden", "calced"], provides=["asked"])(add),
    )
    netop.net.plot('t.png')
    # FAILs
    # - on v1.2.4 with KeyError: 'e',
    # - on #18(unsatisfied) + #23(ordered-sets) with empty result.
    assert netop({"a": 5, "overriden": 1, "c": 2}, ["asked"]) == {"asked": 3}
    # FAILs
    # - on v1.2.4 with (overriden, asked) = (5, 15) instead of (1, 1)
    # - on #18(unsatisfied) + #23(ordered-sets) like v1.2.4.
    assert (
        netop({"a": 5, "overriden": 1})
        ==
        {"a": 5, "overriden": 1, "calced": 10, "asked": 3})