Introduce recursive minimum fee calculation #1106
Conversation
7fd4c6e to
7f0d7a8
Compare
f42c341 to
93aa7a8
Compare
ab0d8fa to
8fabff7
Compare
There was a problem hiding this comment.
Pull request overview
This PR introduces a new experimental API function to compute a minimum transaction fee via a recursive “set fee → rebalance → repeat” approach, and adds property tests intended to validate the behavior.
Changes:
- Added
calcMinFeeRecursiveandRecursiveFeeCalculationErrorto the experimental fee API and re-exported them fromCardano.Api.Experimental. - Implemented recursive min-fee calculation and a helper to distribute surplus balance into outputs.
- Added Hedgehog properties covering “well-funded succeeds”, “idempotent fixpoint”, and “insufficient funds fails”, plus updated an existing fee comparison test.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| cardano-api/src/Cardano/Api/Experimental/Tx/Internal/Fee.hs | Adds the recursive fee calculation implementation and its error type. |
| cardano-api/src/Cardano/Api/Experimental.hs | Re-exports the new recursive fee API from the experimental surface. |
| cardano-api/test/cardano-api-test/Test/Cardano/Api/Experimental.hs | Adds new property tests for calcMinFeeRecursive and extends an existing fee test. |
| cardano-api/cardano-api.cabal | Adds test-suite dependencies used by the new tests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| | minFee == txBodyFee && L.isZero txBalanceCoin = | ||
| -- We have reached the minimum fee but there isn't a guarantee that | ||
| -- the inputs/outputs are balanced | ||
| return unSignTx | ||
| | minFee == txBodyFee && txBalanceCoin > 0 = do | ||
| -- We have a surplus balance so we modify the outputs to include it. | ||
| balancedOuts <- balanceTxOuts txBalanceValue unSignTx | ||
| let updatedTx = UnsignedTx (ledgerTx & L.bodyTxL . L.outputsTxBodyL .~ Seq.fromList balancedOuts) | ||
| in return updatedTx | ||
| | txBalanceCoin < 0 = Left $ NotEnoughAda txBalanceCoin | ||
| | otherwise = | ||
| let newTx = UnsignedTx (ledgerTx & L.bodyTxL . L.feeTxBodyL .~ minFee) | ||
| in calcMinFeeRecursive newTx utxo pparams nExtraWitnesses | ||
| where | ||
| minFee = obtainCommonConstraints (useEra @era) $ L.calcMinFeeTx utxo pparams ledgerTx nExtraWitnesses | ||
| txBodyFee = ledgerTx ^. L.bodyTxL . L.feeTxBodyL | ||
| txBalanceValue = evaluateTransactionBalance pparams mempty mempty mempty utxo unSignTx | ||
| txBalanceCoin = L.coin txBalanceValue |
There was a problem hiding this comment.
calcMinFeeRecursive only checks L.isZero txBalanceCoin / compares txBalanceCoin against 0, which ignores any non-ADA assets in txBalanceValue. A tx can have coin == 0 while being unbalanced in multi-assets. Consider checking L.isZero txBalanceValue and handling negative (or positive) non-ADA balance explicitly (and/or extending the error type accordingly).
There was a problem hiding this comment.
calcMinFeeRecursiveonly checksL.isZero txBalanceCoin/ comparestxBalanceCoinagainst 0, which ignores any non-ADA assets intxBalanceValue. A tx can havecoin == 0while being unbalanced in multi-assets. Consider checkingL.isZero txBalanceValueand handling negative (or positive) non-ADA balance explicitly (and/or extending the error type accordingly).
While this is true, balancing assets is much easier than balancing Ada, as long as assets cannot be used for paying fees. So we could just balance them before starting iterating (if we balance them at all). I would actually replace L.isZero with == 0
| -- We have a surplus balance so we modify the outputs to include it. | ||
| balancedOuts <- balanceTxOuts txBalanceValue unSignTx | ||
| let updatedTx = UnsignedTx (ledgerTx & L.bodyTxL . L.outputsTxBodyL .~ Seq.fromList balancedOuts) | ||
| in return updatedTx |
There was a problem hiding this comment.
In the surplus case (minFee == txBodyFee && txBalanceCoin > 0), the code updates outputs and returns immediately. Updating output values can change the serialized tx size and therefore the minimum fee, potentially making the returned tx underfunded. It would be safer to rerun the fee/balance step on updatedTx (i.e. continue recursion until both fee and full balance reach a fixpoint).
| -- We have a surplus balance so we modify the outputs to include it. | |
| balancedOuts <- balanceTxOuts txBalanceValue unSignTx | |
| let updatedTx = UnsignedTx (ledgerTx & L.bodyTxL . L.outputsTxBodyL .~ Seq.fromList balancedOuts) | |
| in return updatedTx | |
| -- We have a surplus balance so we modify the outputs to include it, | |
| -- then recurse to ensure the fee/balance reach a fixpoint. | |
| balancedOuts <- balanceTxOuts txBalanceValue unSignTx | |
| let updatedTx = UnsignedTx (ledgerTx & L.bodyTxL . L.outputsTxBodyL .~ Seq.fromList balancedOuts) | |
| calcMinFeeRecursive updatedTx utxo pparams nExtraWitnesses |
There was a problem hiding this comment.
In the surplus case (
minFee == txBodyFee && txBalanceCoin > 0), the code updates outputs and returns immediately. Updating output values can change the serialized tx size and therefore the minimum fee, potentially making the returned tx underfunded. It would be safer to rerun the fee/balance step onupdatedTx(i.e. continue recursion until both fee and full balance reach a fixpoint).
Yes. Also, if we are not iterating both fee and balance at the same time, it would simplify the logic to separate the iteration over fee from the iteration over balance completely: here we are checking that minFee == txBodyFee and that txBalanceCoin > 0 at the same time, for example
| oldFees H.=== L.Coin 236 | ||
|
|
||
| newFees H.=== L.Coin 236 | ||
|
|
||
| -- Recursive fee calculation appears result in fees that are ~ 20% lower | ||
| recFee H.=== L.Coin 193 | ||
|
|
||
| H.assert $ recFee < oldFees | ||
|
|
||
| H.assert $ recFee < newFees |
There was a problem hiding this comment.
These assertions hard-code exact fee values (Coin 236 / Coin 193). This makes the test brittle to any future change in protocol parameters, CBOR encoding, or ledger fee calculation. Prefer asserting relationships/invariants (e.g. oldFees == newFees, and recursive fee is a fixpoint / <= non-recursive fee) rather than pinning to specific constants.
| prop_calcMinFeeRecursive_fee_fixpoint = H.property $ do | ||
| (unsignedTx, utxo) <- H.forAll $ genFundedSimpleTx Exp.ConwayEra | ||
| case Exp.calcMinFeeRecursive unsignedTx utxo exampleProtocolParams 0 of | ||
| Left _ -> H.success |
There was a problem hiding this comment.
This property currently treats a failure of calcMinFeeRecursive as success (Left _ -> H.success), which can hide real regressions. Since the generator produces a well-funded transaction, the test should fail on Left (and annotate the error) rather than passing.
| Left _ -> H.success | |
| Left err -> H.annotateShow err >> H.failure |
There was a problem hiding this comment.
Not really, but a comment with clarification would be nice.
| minFee = obtainCommonConstraints (useEra @era) $ L.calcMinFeeTx utxo pparams ledgerTx nExtraWitnesses | ||
| txBodyFee = ledgerTx ^. L.bodyTxL . L.feeTxBodyL | ||
| txBalanceValue = evaluateTransactionBalance pparams mempty mempty mempty utxo unSignTx | ||
| txBalanceCoin = L.coin txBalanceValue |
There was a problem hiding this comment.
What about non-ada assets? Those have to be balanced as well.
| -- We have reached the minimum fee but there isn't a guarantee that | ||
| -- the inputs/outputs are balanced |
There was a problem hiding this comment.
I think this comment is not clear. What does it mean that there is no guarantee? Could you expand it?
There was a problem hiding this comment.
I think this comment is not clear. What does it mean that there is no guarantee? Could you expand it?
Yes, do you mean other than Ada? Ada should be balanced because txBalanceCoin == 0, right?
There was a problem hiding this comment.
I guess the author meant non-ada assets. This should be explained.
| balanceTxOuts txBalance (UnsignedTx tx) = | ||
| let outs = toList $ tx ^. L.bodyTxL . L.outputsTxBodyL | ||
| in case List.uncons outs of | ||
| Nothing -> Left NoTxOuts |
There was a problem hiding this comment.
I think this could be a valid case, no? I imagine user could ask to balance a minting transaction with no outputs to just get one output generated: with a change.
There was a problem hiding this comment.
I think you're right here. Let me push a change.
| in case List.uncons outs of | ||
| Nothing -> Left NoTxOuts | ||
| Just (h, rest) -> | ||
| let updatedout = h & L.valueTxOutL %~ (<> txBalance) |
There was a problem hiding this comment.
Why are you simply updating the first output with the balance difference? Shouldn't you add a change output at the end instead?
There was a problem hiding this comment.
I suppose that's the assumption of the function that it sends surplus change to the first output - but it's not explained anywhere. I understand that this also saves on the fees. I am not sure that everyone would want that. We should provide an option to return the change to the change address.
| where | ||
| minFee = obtainCommonConstraints (useEra @era) $ L.calcMinFeeTx utxo pparams ledgerTx nExtraWitnesses | ||
| txBodyFee = ledgerTx ^. L.bodyTxL . L.feeTxBodyL | ||
| txBalanceValue = evaluateTransactionBalance pparams mempty mempty mempty utxo unSignTx |
There was a problem hiding this comment.
What about transactions registering and deregistering stake?
| -> Int | ||
| -- ^ Number of extra key hashes for native scripts | ||
| -> Either RecursiveFeeCalculationError (UnsignedTx (LedgerEra era)) | ||
| calcMinFeeRecursive unSignTx@(UnsignedTx ledgerTx) utxo pparams nExtraWitnesses |
There was a problem hiding this comment.
I (w/ help of Claude) have found a counterexample for the idempotence test. See my latest commit REMOVEME: counterexample.
I think Copilot is right here: https://github.com/IntersectMBO/cardano-api/pull/1106/files#r2841432882
| prop_calcMinFeeRecursive_fee_fixpoint = H.property $ do | ||
| (unsignedTx, utxo) <- H.forAll $ genFundedSimpleTx Exp.ConwayEra | ||
| case Exp.calcMinFeeRecursive unsignedTx utxo exampleProtocolParams 0 of | ||
| Left _ -> H.success |
There was a problem hiding this comment.
Not really, but a comment with clarification would be nice.
palas
left a comment
There was a problem hiding this comment.
The limit to recursion is very nice. I have some suggestions, and I think I found a bug with help of Claude. Also I am still investigating one edge case that I think may also be an issue, but I am finding it hard to prove empirically
| | multiAssetIsNegative = | ||
| -- Case 1 | ||
| Left $ NonAdaAssetsUnbalanced (getMultiAssets (useEra @era) txBalanceValue) |
There was a problem hiding this comment.
Does this need to be iterated on? If I understand correctly it could just be checked once before iterating, right?
There was a problem hiding this comment.
Good catch it does not.
| multiAssetIsNegative = | ||
| obtainCommonConstraints (useEra @era) $ | ||
| not (L.pointwise (>=) txBalanceValue (L.inject txBalanceCoin)) |
There was a problem hiding this comment.
I think it would be good to explain why this is the same as checking that multi assets (and only multi-assets) are negative, it is not obvious at all
There was a problem hiding this comment.
It would be more self-documenting to compare only multi-assets to zero
| | lastOut ^. L.addrTxOutL == changeAddr -> | ||
| -- Update existing change output in place | ||
| let updatedOut = lastOut & L.valueTxOutL %~ (<> txBalance) | ||
| changeCoin = L.coin (updatedOut ^. L.valueTxOutL) | ||
| in if changeCoin < 0 | ||
| then Left $ NotEnoughAda changeCoin | ||
| else Right $ rest Seq.:|> updatedOut |
There was a problem hiding this comment.
| | lastOut ^. L.addrTxOutL == changeAddr -> | |
| -- Update existing change output in place | |
| let updatedOut = lastOut & L.valueTxOutL %~ (<> txBalance) | |
| changeCoin = L.coin (updatedOut ^. L.valueTxOutL) | |
| in if changeCoin < 0 | |
| then Left $ NotEnoughAda changeCoin | |
| else Right $ rest Seq.:|> updatedOut | |
| | lastOut ^. L.addrTxOutL == changeAddr -> | |
| let currentValue = lastOut ^. L.valueTxOutL | |
| newValue = currentValue <> txBalance | |
| changeCoin = L.coin newValue | |
| in if changeCoin < 0 | |
| then Left $ NotEnoughAda changeCoin | |
| else Right $ rest Seq.:|> (lastOut & L.valueTxOutL .~ newValue) |
Apparently setting a negative value into the TxOut (LedgerEra era) crashes, so it needs to be checked before setting it. This was something Claude discovered when investigating a different issue.
It produces this:
│ Illegal Value in TxOut: MaryValue (Coin (-21)) (MultiAsset (fromList []))
See the also Claude generated test here: 842edd7
Check the computed value for negativity before writing it into the TxOut. Previously the lens setter (%~) would trigger the ledger's TxOut invariant check and throw an exception before the changeCoin < 0 guard could run. Now we compute the new value on a plain MaryValue first, check it, and only write it with (.~) if non-negative. Also add HasCallStack to balanceTxOuts for better error diagnostics.
862bee6 to
8fe5e26
Compare
8fe5e26 to
c53efcd
Compare
|
@carbolymer I've opened a new pull request, #1119, to work on those changes. Once the pull request is ready, I'll request review from you. |
| let outs = tx ^. L.bodyTxL . L.outputsTxBodyL | ||
| in case outs of | ||
| rest Seq.:|> lastOut | ||
| | lastOut ^. L.addrTxOutL == changeAddr -> |
There was a problem hiding this comment.
What if I create outputs: ["addr1", "addr2", "addr3"], and ask to send change to addr1? I think this could be improved to look up the change address here from all addresses in the outputs and modify it.
Unless this is expected to have multiple outputs pointing to the same address. If that is the case, users may be surprised that the last output gets modified with change value.
There was a problem hiding this comment.
We're crystal-ball gazing a bit here. It depends on what we think the user wants to do. A user might create a specific output to addr1 and still want the transaction balance in a separate change output at addr1. My gut says: if a user is calling an automatic balancing function, knowing it creates a change output, and they explicitly construct an output to the same address, they likely want that output to have a specific value, with an additional change output generated separately.
There was a problem hiding this comment.
We're crystal-ball gazing a bit here. It depends on what we think the user wants to do.
Yes a bit. My point was to just bring this possible use case to your attention.
My gut says: if a user is calling an automatic balancing function, knowing it creates a change output, and they explicitly construct an output to the same address, they likely want that output to have a specific value, with an additional change output generated separately.
Fine by me. 👍🏻 Just wanted to be sure that this is by design.
| -- transaction. These are counted as refunds on the consumed side. | ||
| -> Int | ||
| -- ^ Number of extra key hashes for native scripts | ||
| -> Either FeeCalculationError (UnsignedTx (LedgerEra era)) |
There was a problem hiding this comment.
I would make FeeCalculationError constructors carry HasCallStack at this point. It would make pinpointing of an error easier. For example NotEnoughAda is returned in two cases:
- when modifying the output
- when adding new output.
So it's not super obvious where did error come from, until you analyse further.
There was a problem hiding this comment.
Good catch but I'll modify the names to make it more obvious.
| -> Map (Ledger.Credential Ledger.DRepRole) L.Coin | ||
| -> Int | ||
| -> Either FeeCalculationError (UnsignedTx (LedgerEra era)) | ||
| go 0 _ _ _ _ _ _ _ = Left FeeCalculationDidNotConverge |
There was a problem hiding this comment.
Is this actually an error? I think the result of the calculation can still be useful if the fee is somewhat reasonable.
@copilot figure out a definition of a "reasonable fee" which could be implemented here.
There was a problem hiding this comment.
I guess this is a bit hand-wavy, but not too unreasonable #1120
|
@carbolymer I've opened a new pull request, #1120, to work on those changes. Once the pull request is ready, I'll request review from you. |
| let updatedTx = UnsignedTx (ledgerTx & L.bodyTxL . L.outputsTxBodyL .~ balancedOuts) | ||
| go (n - 1) updatedTx utxo' pparams' poolids' stakeDelegDeposits' drepDelegDeposits' nExtraWitnesses' |
There was a problem hiding this comment.
| let updatedTx = UnsignedTx (ledgerTx & L.bodyTxL . L.outputsTxBodyL .~ balancedOuts) | |
| go (n - 1) updatedTx utxo' pparams' poolids' stakeDelegDeposits' drepDelegDeposits' nExtraWitnesses' | |
| go (n - 1) $ UnsignedTx (ledgerTx & L.bodyTxL . L.outputsTxBodyL .~ balancedOuts) |
| let newTx = UnsignedTx (ledgerTx & L.bodyTxL . L.feeTxBodyL .~ minFee) | ||
| in go (n - 1) newTx utxo' pparams' poolids' stakeDelegDeposits' drepDelegDeposits' nExtraWitnesses' |
There was a problem hiding this comment.
| let newTx = UnsignedTx (ledgerTx & L.bodyTxL . L.feeTxBodyL .~ minFee) | |
| in go (n - 1) newTx utxo' pparams' poolids' stakeDelegDeposits' drepDelegDeposits' nExtraWitnesses' | |
| go (n - 1) $ UnsignedTx (ledgerTx & L.bodyTxL . L.feeTxBodyL .~ minFee) |
| (first show . Api.deserialiseFromRawBytesHex . Text.encodeUtf8) | ||
|
|
||
| -- --------------------------------------------------------------------------- | ||
| -- Property tests for calcMinFeeRecursive |
There was a problem hiding this comment.
This file is too long at this point. This change should be moved to a separate module.
| Exp.obtainCommonConstraints era $ | ||
| Exp.TxOut $ | ||
| Exp.obtainCommonConstraints era $ |
There was a problem hiding this comment.
are those doubled constraints necessary? same in line 834
…constraints - Remove getMultiAssets helper and inline the MaryValue extraction, since EraCommonConstraints guarantees Value ~ MaryValue for all eras including Dijkstra - Remove unnecessary parameters from go's recursive loop — utxo, pparams, poolids, stakeDelegDeposits, drepDelegDeposits, and nExtraWitnesses are never modified and are already in scope from the enclosing function - Remove redundant obtainCommonConstraints calls in test generators where constraints are already in scope from the outer call
a994f03 to
1b5f63a
Compare
Changelog
Context
Additional context for the PR goes here. If the PR fixes a particular issue please provide a link to the issue.
How to trust this PR
Highlight important bits of the PR that will make the review faster. If there are commands the reviewer can run to observe the new behavior, describe them.
Checklist