Memory leak during import

[insani7y]

Long story short

I use aiofile to check health of consumers. Some file is being touched during consumer process and then in k8s liveness probe I compare modified time.
Unfortunately I noticed a memory leak that was caused by liveness probe.

Expected behavior

I expect no memory leaks :)

Actual behavior

There is one.

Steps to reproduce

Here code for reproduction:

# liveness_probe/__main__.py
import asyncio


async def liveness_probe() -> None:
    import aiofile

    aiofile


if __name__ == "__main__":
    asyncio.run(liveness_probe())

Aiofile should be imported inside function.

    livenessProbe:
      initialDelaySeconds: 30
      periodSeconds: 10
      timeoutSeconds: 10
      exec:
        command:
          - python
          - -m
          - liveness_probe

Just probe inside k8s Deployment.

Environment info

k8s version: 1.21.8
aiofile version: 3.8.0 - 3.8.8 (didn't check other)
python version: 3.11.3 - 3.11.6 (didn't check other)

I have been produced this problem with implementations:
✅ export CAIO_IMPL=linux
✅ export CAIO_IMPL=thread
✅ export CAIO_IMPL=python

Additional info

Here are graphs of the increase in the amount of memory consumed in pod:

[egorsmkv]

Why did you import aiofile in a function?

I just finished to test it with taskiq's worker and do not see a memory leak because of aiofile, memory usage is not changing.

[insani7y]

I didn't import it in a function. I launched a script several times, imported aiofile inside of it. It's just an example

[egorsmkv]

I meant why did you write the code like:

# liveness_probe/__main__.py
import asyncio


async def liveness_probe() -> None:
    import aiofile

    aiofile

but not like:

# liveness_probe/__main__.py
import asyncio
import aiofile

async def liveness_probe() -> None:
    aiofile

Also, are you seeing the memory leak in K8S only? Can you make a test script without K8S?

[insani7y]

Got it. You're right, that would be more correct.

# liveness_probe/__main__.py
import asyncio
import aiofile

async def liveness_probe() -> None:
    aiofile

And that's how we use it on our side. But this script is loaded dynamically and executed, so every time we run it, we import aiofile again.
My original example is just a simplification, but the problem persists even in your example.

Additionally, memory leak is confirmed with memray.

[egorsmkv]

Can you attach the memray log?

I also tried to check this issue with memray, but haven't found the leak.