Relocatable build tree

[LaurentRDC]

This PR adds a new flag, --relative-build-tree, to make all paths in the build tree relative to the build tree root. This means that the build tree (e.g. dist-newtyle) can be copied wholesale elsewhere (for example, to a git worktree) and promptly be re-used by cabal without re-building.

This behavior can be triggered with cabal build --relative-build-tree, or the project-level configuration option relative-build-tree: True.

Some notes:

• --relative-build-tree changes plan.json to relativize paths. It's not clear if that's too much of a breaking change.

• --relative-build-tree is not the default; it is opt-in.

• --relative-build-tree is backwards compatible, in a sense: older versions of cabal will see a stale build tree, and rebuild, instead of erroring out.

• --relative-build-tree is only needed for the initial build: the following does not trigger a rebuild:

  $ cabal build --relative-build-tree
  $ cp -R ./dist-newstyle ../some-worktree
  $ cd ../some-worktree
  $ cabal build
  < Up to date >

Fixes #12137

• Patches conform to the coding conventions.
• Any changes that could be relevant to users have been recorded in the changelog.
  • Is the change significant? If so, remember to add significance: significant in the changelog file.
• The documentation has been updated, if necessary.
• Manual QA notes have been included.
• Tests have been added. (Ask for help if you don’t know how to write them! Ask for an exemption if tests are too complex for too little coverage!)

[Bodigrim]

This PR re-purposes the --enable-relocatable flag to make all paths in the build tree relative to the build tree root.

In what sense does it "re-purpose" the existing option? What was the earlier purpose of this option?

[LaurentRDC]

In what sense does it "re-purpose" the existing option? What was the earlier purpose of this option?

It's not clear what relocatable does, according to the user guide:

I didn't want to add yet another flag if it could be avoided, so I suggest re-purposing the name

[Bodigrim]

I'm no expert in this area, but I'm not comfortable about such repurposing. AFAICT from brief grepping, the existing --relocatable option essentially relates to ld --relocatable, which is a different notion of relocatability. Despite what the manual says, the existing option is far from no-op (it might still be buggy, but nevertheless).

Back to your PR, is there a reason to make this behaviour configurable? Is there a reason not to use relative paths always?

[LaurentRDC]

Is there a reason not to use relative paths always?

It depends on whether plan.json is considered a public interface. I meant to look into this but I haven't yet. My understanding is that there's at least one external tool (cabal-plan) that consumes it, and so I would make an associated patch to support both absolute and relative paths.

[Bodigrim]

I hardly imagine dist-newstyle/cache/plan.json to be a public interface. It's under cache/ after all: it could exist or not exist, to be outdated or not, and whatever it contains is entirely up to the current Cabal executable.

Is plan.json the only place containing absolute filepaths and thus preventing dist-newstyle/ from being relocatable?

[Bodigrim]

(Is it possible to write a standalone executable cabal-worktree-copy DIR which would take the existing build tree from dist-newstyle, copy it into DIR and make whatever amendments to absolute filepaths are needed? This is somewthing everyone could start using tomorrow instead of waiting another year until Cabal 3.20)

[LaurentRDC]

It would be possible to write a standalone tool to do the copying. My colleague has created a script to do it entirely in Neovim.

I'd like to hear thoughts from maintainers; if this functionality is not gated behind a flag (for example, if plan.json is not considered a public interface), then I would be happy backport this patch

[andreabedini]

@LaurentRDC I haven't had the chance to review your PR but I want to say that I am very interested in this. In fact I have implemented something similar in stable-haskell/cabal¹. We build GHC's binary distribution with cabal so in the end we need something we can put into an archive and ship. I look forward to comparing our approaches.

Footnotes

1. commits are somewhere in here, search for "local store". I apologise for that mess, I am in the process of cleaning up. ↩

[adamgundry]

I'm no expert in this area, but I'm not comfortable about such repurposing. AFAICT from brief grepping, the existing --relocatable option essentially relates to ld --relocatable, which is a different notion of relocatability. Despite what the manual says, the existing option is far from no-op (it might still be buggy, but nevertheless).

I agree with this. I'd suggest coming up with a different flag/name for this PR and considering the future of relocatable separately. If nothing else, it will be confusing for a user if they either had previously enabled the old option and now accidentally get the new behaviour when they upgrade cabal, or if they are trying to enable the new behaviour but their cabal is too old so they accidentally get the old behaviour rather than an error message.

(See #462/#2255 for the original implementation of the feature. Given the discussion on #11221 (comment) perhaps it should be removed. I don't know anything about the details here but perhaps @christiaanb may have insight?)

[LaurentRDC]

I've changed the PR to use a new flag, --relative-build-tree, rather than re-purposing an old one. That was not a good idea of me.

The main question that remains is: is a flag even needed? Are changes to plan.json considered breaking?