Reduce circular dependencies between `utils.py` and `_compat/` by migrating pip-related utilities into a new `_pip_api`

[sirosen]

I've been bitten a couple of times by the circular relationship between "utils" and "_compat", and we have some old PRs which got snagged on it as well. (I was trying to port the changes in #2108 forward.) The current module structure is not right in terms of defining dependencies between components -- if you try to migrate any component from "utils" into "compat", you create cycles.
"utils" is also a generally disliked component name, since it doesn't contain important information about its contents.

Tinkering a bit in the repo today, and trying to decide on a path forward, I started to focus on the fact that utils and _compat.pip_compat are both concerned with providing an API for pip usages, like the PIP_VERSION constant.

I propose that we pursue a gradual migration:

1. We introduce piptools._pip_api, which is our API for usage of pip. It is, of course, internal-only.
2. piptools._pip_api is defined to not have any dependencies on any other parts of piptools for now. i.e. We aren't going to allow the reintroduction of this kind of circular problem -- we could add smaller modules which _pip_api would be able to consume in the future, but the one we have to strictly rule out is utils.
3. As much of utils as is reasonable is ported into _pip_api, and consumption is updated.
4. Testing is also dir structured, into tests/pip_api/test_{module_name}.py (we have too many tests in a flat dir at present, IMO. I want some better organization to the testsuite so that I can find things).
5. Once piptools._pip_api is defined, piptools._compat.pip_compat is migrated into it.
6. Celebrate! Then reassess and look for more ways to reduce circularity and improve module structure.

@webknjaz, looking for your feedback on this in particular. I have some initial work I'll turn into a draft PR to demo the idea.

[webknjaz]

1. We introduce piptools._pip_api, which is our API for usage of pip. It is, of course, internal-only.

I think I'd like a more agressive approach and would put it into piptools._private._pip_api or piptools._internal._pip_api.

2. We aren't going to allow the reintroduction of this kind of circular problem

For this, I'd like to have a port of https://github.com/aio-libs/aiohttp/blob/91108c9/tests/test_circular_imports.py in our test suite.

Additionally, may I ask you to play with https://import-linter.rtfd.io/en/stable/readme.html? I've been meaning to start using it but never got to actually playiing with it..

4. Testing is also dir structured, into tests/pip_api/test_{module_name}.py (we have too many tests in a flat dir at present, IMO. I want some better organization to the testsuite so that I can find things).

I usually mirroring the project layout into the test dir structure + maybe a layer w/ categories. For example, prj/mod1.py and prj/sub/mod2.py would be tested under tests/unit/mod1.py and tests/unit/sub/mod2.py respectively. And for more high-level tests, we'd have tests/integration/, for example (maybe also e2e, not sure).

I've also been wanting to try out pytest-bdd for acceptance / regression behaviors.

[sirosen]

I'll update the initial crack at this to use piptools._internal.pip_api, and work on those options for linting and testing imports. 👍 to all of that!

4. Testing is also dir structured, into tests/pip_api/test_{module_name}.py (we have too many tests in a flat dir at present, IMO. I want some better organization to the testsuite so that I can find things).

I usually mirroring the project layout into the test dir structure + maybe a layer w/ categories. For example, prj/mod1.py and prj/sub/mod2.py would be tested under tests/unit/mod1.py and tests/unit/sub/mod2.py respectively. And for more high-level tests, we'd have tests/integration/, for example (maybe also e2e, not sure).

I've also been wanting to try out pytest-bdd for acceptance / regression behaviors.

Yeah, I do similar things across projects.

Generally, tests/unit/ which fully mirrors the repo structure, as you say, and tests/acceptance/ (or tests/functional/, tests/integration/, tests/e2e/, tests/system/ ... largely depending on whether or not there is some history on a project which chooses one of these names for me 😉 ) for higher level behavioral tests. I used to try to do three tiers, for small units, bigger units, and then the whole system, but it never worked out in practice.

For the initial body of work, let's stick with introducing tests/unit/_internal/pip_api/test_{...}.py modules?

[webknjaz]

I'll update the initial crack at this to use piptools._internal.pip_api

So I was thinking piptools._internal._pip_api — this way, if somebody does from piptools._internal import _pip_api, they'd get a _private var in the module globals automatically (unless they override that explicitly via from piptools._internal import _pip_api as _pip_api, of course, which should be avoided in general).

and work on those options for linting and testing imports

I just realiazed the Gentoo devs made https://pypi.org/p/pytest-import-check — it may be complementary or even a replacement for the test module I linked above. I haven't checked the differences, though. I think I was considering turning that module into a plugin myself since I have variations of it copied into like 5 projects already.

For the initial body of work, let's stick with introducing tests/unit/_internal/pip_api/test_{...}.py modules?

Yep. tests/unit/_internal/_pip_api/test_{...}.py. Although, it'd be nice to eventually migrate to _test suffices (instead of test_ prefixes) in the module names — it's a bit better for autocompletion on CLI (pre-commit's default checks have something for this, actually).