High memory allocation in pendulum 3.0

[cody-scott]

• [x ] I am on the latest Pendulum version.

• [x ] I have searched the issues of this repo and believe that this is not a duplicate.

• OS version and name: Debian GNU/Linux 12 (bookworm)

• Pendulum version: 3.0.0

Issue

In pendulum 3.0.0 it is allocating a large amount of memory for the library (up to 1gb). In prior versions, memory allocation was usually less than 2 mb for the same import. Possibly due to rust re-write?

tested/validated using memray with both pendulum==3.0.0 and pendulum==2.1.2

flame_graphs.zip

[Secrus]

That is troubling. I don't know too much about Rust, but I am open to accepting PRs that implement improvements on that front.

[cody-scott]

I think one first step would be to see if anyone else sees the same results I get when profiling. I tested in windows/linux and saw similar results, but would be great to confirm its not just a "me" issue.

[protoss-player]

I think one first step would be to see if anyone else sees the same results I get when profiling

Actually, I got the exactly same memory consumption issue with memray and ended up here.

[s-rigaud]

I'm getting segfault inside a docker container as fastapi_cache is based over pendulum and memory consumption is way higher than before.

[Secrus]

@cody-scott any chance you could test the latest master branch? I have merged some changes to the Rust-Python code and I suspect it might be the answer to this issue (at least partially)

[rudolfix]

this is because mimalloc is used instead of of default memory allocator. but I bet memray is giving you wrong results.
bloomberg/memray#577

I can't see increased memory usage otherwise, not in the process stat, not when using valgrind.

[GriffinBoris]

I am seeing the same issue in version 3.0.0

I am running macOS Sequoia 15.3.1.
I am using python 3.12

I have tested with versions 3.0.0a1, 3.0.0b1, and 3.0.0. The high memory usage appears to correlate with the new Rust extension introduced in 3.0.0b1.

I've attached an archive file containing flamegraphs for each version, with comparisons across different malloc implementations, along with a test Python script used for profiling.

Archive.zip

For a quick reference here is 3.0.0a1 with malloc:

Here is 3.0.0b1 with malloc:

Here is the reference of the script:

def regular_pendulum_import():
    import pendulum
    pass


if __name__ == '__main__':
    regular_pendulum_import()

This continues into 3.0.0 regardless of allocator.

[AngerM]

Is disabling of mimalloc actually working? At least in the default mimalloc is just grabbing 1GB immediately:

> MIMALLOC_SHOW_STATS=1 uv run python
Python 3.11.8 (main, Jan  6 2025, 11:19:08) [Clang 16.0.0 (clang-1600.0.26.6)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pendulum
>>>
>>>
heap stats:     peak       total       freed     current        unit       count
  reserved:     1.0 GiB     1.0 GiB     0           1.0 GiB
 committed:     1.0 GiB     1.0 GiB     0           1.0 GiB
     reset:     0
    purged:     0
   touched:    64.2 KiB    64.2 KiB    39.2 KiB    25.0 KiB                          not all freed
  segments:     1           1           0           1                                not all freed!
-abandoned:     0           0           0           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0          10         -10                                ok
-abandoned:     0           0           0           0                                ok
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:     0
   threads:     0           0           0           0                                ok
  searches:     0.0 avg
numa nodes:     1
   elapsed:     2.022 s
   process: user: 0.154 s, system: 0.048 s, faults: 6, rss: 35.5 MiB, commit: 1.0 GiB

[levzem]

I am seeing similar numbers as well

[ashb]

Thanks we will take a look at this soon. (Myself: once Airflow 3.0 is out and things have calmed down a bit)

[ashb]

This should be fixed by #871 which removes mimalloc (as it's benefits don't make sense for our usage patterns anyway!)