Issues mixing shadow pricing and cost-based pricing

[tsmbland]

See below comment from #1319

---

I've figured out the problem with the marginal and marginal_average models. These were actually using shadow pricing for heat, because the patch wasn't configured properly (fixed in #1324). The problem with this is that if heat uses shadow pricing and electricity etc use cost-based pricing, then heat ends up underpriced compared to the other commodities in the system (shadow prices are set without any context of the cost-based prices we later calculate).

When it comes to investment appraisal, assets that were active in the previous year dispatch choose not to be active in the appraisal because they're not profitable with the price of heat that's given to them.

I've rebased on top of #1324 and these two models now work.

The circularity model still fails, possibly because it's still using shadow prices for some commodities, although hopefully that will be fixed up with #1322.* (Edit: still fails)

Overall, I think this reveals an inherent issue with mixing shadow pricing and cost-based pricing. I think we have to either use shadow pricing throughout or cost-based pricing throughout, but not mix and match (although it would probably be fine if all the shadow-priced commodities were upstream of the cost-based commodities in the price calculation order).

*I still have doubts about this though. I think what this shows is the importance of commodity prices being consistent with one another, and in the current approach of iterating just once, rather than aiming for an equilibrium, it's possible that prices won't be consistent with one another.

Originally posted by @tsmbland in #1319 (comment)

[tsmbland]

@ahawkes See above comment about mixing shadow prices and cost-based prices. Would it be reasonable just to block users from doing this (i.e. require either all commodities to use shadow prices, or all commodities to use cost-based prices)?

[ahawkes]

Thanks @tsmbland.
This might be solved when we fix the appraisal tools? The new approach in appraisal should always use shadow prices when figuring out asset dispatch for use in appraisal, because this is how the asset would have been operated in previous MSY conditions.
Then, dispatch is fixed, and we calculate the appraisal metrics using the chosen pricing strategy.
Realise this sounds convoluted, but I think it's the only way to have dispatch in appraisal be representative of dispatch in previous MSY conditions. Does it make sense?
I think fixing dispatch (based on shadow prices) like this should solve this issue?

[tsmbland]

Thanks @ahawkes. I had missed that we were planning to use shadow prices for the appraisal dispatch, but that makes sense

[ahawkes]

Thanks @ahawkes. I had missed that we were planning to use shadow prices for the appraisal dispatch, but that makes sense

We probably should make this clearer in the relevant issue. The idea is that shadow prices will invoke the same dispatch that one would have seen in the previous MSY (our own calculated prices possibly won't do this - and could lead to our own calc suggesting there would be dispatch where in fact there won't be - which could be bad). So, we fix dispatch using shadow prices (should be exactly the same dispatch for NPV and LCOX methods), and then we calculate metrics using own own prices...

[tsmbland]

Related to this, when we're doing the partial system dispatch after each commodity investment, and we have to explicitly pass in commodity prices for yet-to-be balanced commodities - which prices should be be using here? Guessing these should be shadow prices by the same logic?

[ahawkes]

Related to this, when we're doing the partial system dispatch after each commodity investment, and we have to explicitly pass in commodity prices for yet-to-be balanced commodities - which prices should be be using here? Guessing these should be shadow prices by the same logic?

Yes!

[alexdewar]

Related to this, when we're doing the partial system dispatch after each commodity investment, and we have to explicitly pass in commodity prices for yet-to-be balanced commodities - which prices should be be using here? Guessing these should be shadow prices by the same logic?

Yes!

I've opened an issue for this now: #1346

Out of interest, is this what we should always have been doing or just a change needed for #1199? I just want to know whether this is a bug affecting our previous releases.

[ahawkes]

Thanks @alexdewar - I think always should have been done like this - as it's more consistent with the overall approach. Though it is possible, as ever, that it'll cause problems and need revisiting - let's see.