Move the compiler to a new `compiler/` directory

[mark-i-m]

Proposal

(this is a successor of #316, primarily to restart the discussion with a smaller scope)

In MCP #298 and rust-lang/rust#73265, we moved the standard libraries into their own library/ directory at the root of the repo. In this proposal, we follow through to reorganize the compiler source into a new compiler/ directory:

- library/        // as in #298 
- compiler/
  - librustc_* -> rustc_*
  - rustc/rustc.rs -> rustc/src/main.rs
  - test/ (only compiler tests)
- llvm-project (submodule) // ongoing discussion about where to put this, rustllvm, rustc_llvm
- src/      // unchanged

The primary goals of this proposal are to

1. make it more obvious where things live by separating out distinct-ish parts of the project into different top-level directories
2. help (especially new) contributors to feel less overwhelmed by the size of the project by breaking the top-level directories down into more manageable and modular parts
3. work towards a world in which std/core and/or other parts of the project can be built independently of the compiler
4. make the repo a bit more ergonomic for contributors

Note that in particular, we move some things around:

• The src/test directory is split up to compiler/test and src/test/
• We remove the lib* prefixes for all the crates

Unresolved question: there was a proposal on the zulip thread for #316 to move the llvm-project submodule out of the compiler src directory (e.g. to the root of the repo). This would make it easier to grep in the compiler/ directory. (Personally, I like this idea).

Unresolved question: there was also some questions about what to do with the current rustc crate (effectively, the main.rs of the compiler), which is basically just a main function that calls rustc_driver. In the above proposal, I have it as bin/rustc.rs.

Implementation: I propose we do this move over multiple PRs to reduce conflicts. I think it should be less painful than moving the std libs, and should largely be editing Cargo.toml, but it would be nice to avoid the problems of rust-lang/rust#73265, where a bazillion things needed to be updated.

Approval: We anticipate bikeshedding over names, exact paths, and etc. The approval of this MCP will be construed as a green light to start working on PRs, but not as consensus on the exact final form. Rather, a separate FCP or nomination will be required to merge e.g. the PR that moves compiler/.

Followup: In followup MCPs similar to this one, we will move around other parts of src until we reach an acceptable state.

Mentors or Reviewers

@Mark-Simulacrum mentored/seconded #298, but I don't know if they wish to mentor/second this MCP. @Dylan-DPC also expressed interest in working on this.

Process

The main points of the Major Change Process is as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[rustbot]

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[petrochenkov]

I'm ready to second this part:

- compiler/
  - librustc_* -> rustc_*
  - rustc/ -> bin/rustc.rs

except that rustc.rs should probably use rustc/main.rs or whatever the default cargo layout for binaries is.

[mark-i-m]

• There is still some ongoing discussion on zulip about what to do with rustc_llvm, rustllvm and llvm-project.
• There is also a possibility of renaming the directories to drop the rustc_ prefix (but keep the crate names and transition those slowly over time).
• I updated the proposal to be rustc/src/main.rs, as you proposed.

[petrochenkov]

@rustbot second
If that's what is needed to formally unblock rust-lang/rust#74862.