Add performance benchmarks to the readme

[Archmonger]

Right now the performance difference between stdlib files, aiofiles, and aiofile is undocumented.

There should be at least two benchmarks:

1. Show the performance during single-threaded operations (single while loop where stdlib files is likely to be faster).
2. Show the performance improvement when using aiofile during high IO concurrency scenarios.

[mosquito]

The performance actually depends on what underlaying implementation of caio is currently used in the benchmark environment. The linux_aio-based implementation, of course, only for linux this have a better performance.
The thread_aio and python_aio implementations should not show much improvement over aiofiles in the general case, but should be slightly better on POSIX-compatible OSes.

I have no ideas how to make this benchmark fair enough, if you have any share it with me, preferably the code.

[Archmonger]

For environment specific libraries like this one, I recommend re-running tests per environment. It would likely require a fairly large text matrix though.

For example, for ReactPy we run each part of our code through IOPS tests. The concept is the same, although we focus on network IO which means we need a live web browser.

Since 99% percent of users only care about read/write performance, your test result matrix might look something like this:

---

Read

This test repeatedly reads 5MB of binary data from a file, and measures how many reads occurred within 60 seconds (higher is better).

	1 Worker	500 Workers	1 Worker (Linux)	500 Workers (Linux)
aiofile	x	x	x	x
aiofiles	x	x	x	x
stdlib	x	x	x	x

Write

This test repeatedly writes 5MB of binary data to a file, and measures how many writes occurred within 60 seconds (higher is better).

	1 Worker	500 Workers	1 Worker (Linux)	500 Workers (Linux)
aiofile	x	x	x	x
aiofiles	x	x	x	x
stdlib	x	x	x	x

---

Since I probably don't need to show you how to create the test cases themselves, I'll give some top-level tips.

I recommend reading/writing binary data in order to minimize variables. Also, you should use asyncio.gather to spawn workers. For example...

import asyncio

async def read_binary_test():
  ...

async def main():
  tasks = [read_test() for _ in range(500)]
  await asyncio.gather(*tasks)

if __name__ == "__main__":
  asyncio.run(main())

The write test should make sure to store BINARY_DATA_5MB as a global to avoid reconstructing it every time, which would impact test results

I arbitrarily chose 5MB as a value that gives Python enough "downtime" to actually do something else while IO is being executed.

The test cases should be written to stop recording results after 60 seconds has elapsed.

[rooterkyberian]

This would be quite valuable.
When starting new project I'm not even sure if I should bother with asyncio .

According to #18 (comment) performance before new implementation was atrocious (i.e. you will be better of having blocking stdlib write/reads most of the time).

#18 (comment) looks like a huge improvement have been achieved, but does not compare against stdlib, which is quite crucial.

As for which interface need the benchmark the most - even doing it for just for linux would be good enough, since most of the applications in which performance is critical, tend to be hosted on servers, which tend to be Linux-based.

I tried rerunning test from the aforementioned comment, but the results look bad. I hope this benchmark is somewhat botched ( I did not try to debug it, just changed it a little to produce table automatically): https://gist.github.com/rooterkyberian/a2c12fc6269c86bcf4e199149eb6b9ec .

Results I got:

Python version: 3.11.6
Platform: Linux-6.5.0-25-generic-x86_64-with-glibc2.35
aiofiles version: 23.2.1
aiofile version: 3.8.8
uvloop version: 0.19.0
aiofile default context <caio.linux_aio_asyncio.AsyncioContext object at 0x7e2eb3ac0f50>

iterations	sync	async executor 'dumb'	async executor w/ coroutines	async multiple executors	async aiofiles	async aiofile	aiofiles@uvloop	aiofile@uvloop
1000	0.006	0.01	0.007	0.01	0.748	0.303	0.455	0.293
10000	0.051	0.059	0.059	0.061	7.558	3.09	4.653	2.709
100000	0.481	0.49	0.666	0.518	74.853	190.855	47.254	185.326
1e+06	4.676	4.651	4.85	5.199	746.184

I left it running more than an hour after last result and still nothing, so it seems like either benchmarking script is broken or aiofile (caio backend) is.

[HaukurPall]

Thanks for this script. I ran it to test my fix in #93 and here are my results:

iterations	sync	async executor 'dumb'	async executor w/ coroutines	async multiple executors	async aiofiles	async aiofile	aiofiles@uvloop	aiofile@uvloop
1000	0.012	0.013	0.013	0.021	1.136	0.208	0.733	0.166
10000	0.11	0.107	0.07	0.074	9.942	1.122	3.372	0.755
100000	0.577	0.577	0.582	0.676	107.671	9.06	50.216	7.583

It seems to improve the performance quite a bit (and beat aiofiles!)

[Archmonger]

Perhaps I'm reading the results wrong, but it looks like the results suggest that using standard lib with asyncio.run_in_executor is more performant that both aiofiles and aiofile?

[Archmonger]

Okay I am seeing an issue with the testing methodology.

Fundamentally, all Python async file frameworks will perform slower for small reads/writes. This is because they add multiple additional layers of stack complexity compared to standard lib.

The only real benefit of aiofiles and aiofile is the ability to perform long-running IO independently of the Python interpreter. This allows the Python interpreter do other operations in the "IO downtime". So reading/writing four bytes (python integer) is far too small of an operation to actually gain any benefit. As seen above, a mixture of stack trace overhead and context-switching will compound run times exponentially.

The tests would need to be modified to read/write something much larger, like a 5MB text file.

[HaukurPall]

I think you are mostly right @Archmonger.

I have been doing some reading on the subject of async file reading on Linux via Python and would like to add to your point. From what I've learnt; The most "accepted/performant/non-blocking" async file I/O subsystems in Linux are the Linux AIO and the (more recent) IO-uring. This package implements an interface for Linux AIO. Both of them are setup around a queue and can service multiple requests at the same time, f.ex. the client can batch multiple requests at once, for different files etc. Setting up such a capable system for a single file read, is an overkill. Furthermore, to add a nice async/await interface for Python (mapping callbacks to futures), considerable amount of Python code is required. Going from a simple open/iterate over file object implemented in C to a multilayered Python code will always be slower.

The performance tests in the link above is not comparing the async capabilities equally. The "async *executor*" tests make the full file read async, not each individual read operation like this package does. New tests should be added to run each open() and readline() on a worker thread to have equivalent async granularity.

[Archmonger]

Got reminded of this while working on a project, so I decided to create my own tests.

Tests source code: https://gist.github.com/Archmonger/5f42613d7f23724f58eb2091a08ded5a

Methodology

Async tests are run with 500 concurrent tasks via asyncio.gather to simulate a pure asyncio concurrency environment (similar to what ASGI webservers would do). All tests are written with 5MB operations. Sync tests are run sequentially.

Note that 500 concurrent read/write operations is intentionally IO bound due to operating system IO interrupts.

Results

It seems that a simple asyncio.to_thread wrapper is more performant in the majority of scenarios.

The aiofile package seems to be the next runner-up in terms of performance.

Windows 11, WindowsProactorEventLoop, NTFS

Function	Time (s)
asyncio.to_thread(stdlib_write)	0.714191
asyncio.to_thread(stdlib_read)	1.05836
aiofile_write	1.09225
stdlib_write	1.25541
aiofile_read	1.3647
stdlib_read	1.42107
aiofiles_read	4.36606
anyio_read	4.39978
anyio_write	7.73693
aiofiles_write	7.7867

Windows 11, WindowsSelectorEventLoopPolicy, NTFS

Function	Time (s)
asyncio.to_thread(stdlib_write)	0.65146
asyncio.to_thread(stdlib_read)	1.03726
aiofile_write	1.08998
stdlib_write	1.21648
aiofile_read	1.32839
stdlib_read	1.43044
aiofiles_read	4.34663
anyio_read	4.4364
aiofiles_write	7.71415
anyio_write	7.76821

Unraid 6.12.10 (Linux), SelectorEventLoopPolicy, BTRFS

Function	Time (s)
stdlib_read	2.05419
asyncio.to_thread(stdlib_write)	2.33569
stdlib_write	2.39474
aiofiles_write	2.53117
anyio_write	2.77296
asyncio.to_thread(stdlib_read)	3.03362
anyio_read	3.92669
aiofiles_read	4.43138
aiofile_write	4.73931
aiofile_read	6.5278

Unraid 6.12.10 (Linux), Uvloop, BTRFS

Function	Time (s)
stdlib_read	1.98613
asyncio.to_thread(stdlib_write)	2.20811
aiofiles_write	2.30034
stdlib_write	2.36995
anyio_write	2.54212
asyncio.to_thread(stdlib_read)	2.96265
anyio_read	3.47736
aiofiles_read	4.14617
aiofile_write	5.01876
aiofile_read	6.60789

[Archmonger]

Rewrote my tests to average out 10 iterations, and also added tests for all the combinations of event loops.

[siddhpant]

I modified @Archmonger's script to add some more stuff (basically testing for flushing too), and here are the results on my machine:

(For aiofile I use fsync)

---

_UnixDefaultEventLoopPolicy

Function	Time (s)
stdlib_read_unbuffered	0.200348
stdlib_read	0.242624
aiofiles_read	0.415127
asyncio.to_thread(stdlib_read)	0.437746
asyncio.to_thread(stdlib_read_unbuffered)	0.490132
anyio_read	0.548908
aiofile_read	0.87412
asyncio.to_thread(stdlib_write_unbuffered)	0.875594
anyio_write_flush	0.876996
aiofiles_write_flush	0.890379
asyncio.to_thread(stdlib_write)	0.892612
asyncio.to_thread(stdlib_write_flush)	0.897756
anyio_write	0.910071
aiofiles_write	1.01746
stdlib_write	1.03949
stdlib_write_flush	1.0398
stdlib_write_unbuffered	1.03991
aiofile_write	1.19374
aiofile_write_flush	1.33136

---

uvloop

Function	Time (s)
stdlib_read_unbuffered	0.239012
stdlib_read	0.279507
asyncio.to_thread(stdlib_read)	0.381207
aiofiles_read	0.389197
asyncio.to_thread(stdlib_read_unbuffered)	0.442786
anyio_read	0.547405
asyncio.to_thread(stdlib_write)	0.811238
asyncio.to_thread(stdlib_write_unbuffered)	0.811762
asyncio.to_thread(stdlib_write_flush)	0.814089
aiofiles_write_flush	0.857403
anyio_write_flush	0.878293
anyio_write	0.909213
aiofiles_write	1.0003
stdlib_write_unbuffered	1.03255
stdlib_write_flush	1.04873
stdlib_write	1.07907
aiofile_read	1.16294
aiofile_write	1.23341
aiofile_write_flush	1.29143