feat: add input nodes as evaluation baseline#372
Conversation
ntalluri
changed the title
Current visualization on the egfr dataset
|
|
I thought this pull request was for something else. Could we rename it to be more like "feat: add input nodes as evaluation baseline"? |
| # TODO what if the node_ensemble is all frequency = 0.0, that will be the new source/target/prize/ baseline? | ||
|
|
||
| # Set frequency to 1.0 for matching nodes | ||
| prize_node_ensemble_df.loc[ |
There was a problem hiding this comment.
make the baseline 1 baseline per dataset not per algorithm. All the sources/prizes/targest/active = 1.0 and everything else is 0. But still calculate the precision and recall.
There was a problem hiding this comment.
this will be algorithm agnostic.
There was a problem hiding this comment.
Another option also maybe needed is to only evaluate the internal nodes (set the source/target/prizes to 0.0).
There was a problem hiding this comment.
we might need both of these. One will be a baseline on the current ensemble pr. New figures for all the evaluation for only evaluating the internal nodes.
ntalluri
changed the title
ntalluri
added
analysis
| # the Input_Nodes_Baseline PR curve highlights their overlap with the gold standard. | ||
| if prc_input_nodes_baseline_df is None: | ||
| input_nodes_set = set(input_nodes['NODEID']) | ||
| input_nodes_gold_intersection = input_nodes_set & gold_standard_nodes # TODO should this be all inputs nodes or the intersection with the gold standard for this baseline? I think it should be the intersection |
There was a problem hiding this comment.
Very good question. For a synthetic dataset like Panther pathways, the inputs are sampled from the pathway nodes that make up the gold standard so it doesn't matter.
For an omics input like EGFR, it matters substantially. What makes you prefer the intersection? I was inclined to say all input nodes because we cannot have a baseline algorithm that makes use of gold standard information as part of its ranking. I could create a valid pathway reconstruction algorithm that takes the input nodes and simply returns those. I can't use a gold standard in a valid pathway reconstruction algorithm, however.
There was a problem hiding this comment.
In my opinion, the only input nodes that matter for evaluation are those that overlap with the gold standard. While it’s true that an algorithm could trivially return all input nodes, our precision recall evaluation is only defined with respect to the gold standard. Input nodes that aren’t in the gold standard don’t contribute to true positives, so including them in the baseline wouldn’t be meaningful.
That said, I also see the case for using all input nodes as it represents a valid baseline algorithm where an algorithm could simply return the given inputs without any reconstruction.
Maybe the difference is that the intersection provides an upper bound, while all input nodes provides a lower bound on what you could do without doing any reconstruction and we should provide both?
There was a problem hiding this comment.
think through 3 baselines and explain why each of these can be used/needed
- the no intersection, but input nodes by itself
- the intersection of the gold standard and input nodes
- the the gold standard by itself (which I think is what baseline is)
- or do we want to make an ensemble of the gold standard as 1.0 and everything else 0.0 and do a PRC
There was a problem hiding this comment.
Deciding this is the last point of feedback and last decision to finalize. Then I can do a last careful review and we should be ready to merge.
In our meeting, we discussed how options 1 and 2 will give the same recall. The only difference is that one will have some precision value and the other always has precision of 1.0.
|
@ntalluri is this waiting for my review or your updates? We haven't touched it in a while. |
tristan-f-r
added
the
awaiting-author
Co-authored-by: Tristan F.-R. <pub.tristanf@gmail.com>
There was a problem hiding this comment.
revert back to what it was before
3 participants
Working on #370