The s3bench utility is a simple tool to test s3 server performance.
Upload and download runs report both transfer throughput in MB/s and object throughput in obj/s.
# export AWS_SECRET_ACCESS_KEY=user
# export AWS_ACCESS_KEY_ID=password
# ./s3bench -upload -bucket test -endpoint http://s3-server.example.com:12345 -objectsize 10737418240 -pathstyle -prefix host1/file. -region us-east-1 -concurrency 8 -disablechecksum -n 4
0: 10737418240 in 11.723064482s (873 MB/s)
1: 10737418240 in 11.536811023s (887 MB/s)
2: 10737418240 in 11.639824457s (879 MB/s)
3: 10737418240 in 11.67056153s (877 MB/s)
run perf: 42949672960 in 11.729466731s (3492 MB/s)
# ./s3bench -download -bucket test -endpoint http://s3-server.example.com:12345 -objectsize 10737418240 -pathstyle -prefix host1/file. -region us-east-1 -concurrency 8 -disablechecksum -n 4
0: 10737418240 in 6.92038851s (1479 MB/s)
1: 10737418240 in 6.928683753s (1477 MB/s)
2: 10737418240 in 6.953225327s (1472 MB/s)
3: 10737418240 in 6.952238442s (1472 MB/s)
: run perf: 42949672960 in 6.954156319s (5890 MB/s)
useful options:
- endpoint is the server listening endpoint, use https for ssl connections
- chunksize is the part size for the multipart upload
- concurrency is the max simultaneous part uploads in multipart upload or range gets for download
- n is the total number of objects to read/write in the run
- workers limits how many object uploads/downloads run at once; default is n
- prefixes spreads object names across multiple prefixes; with
-prefix bench/ -prefixes 3, keys becomebench/0/0,bench/1/0,bench/2/0,bench/0/1, ... - prefix is prepended to each object name
- rand uses randomly generated data instead of all 0s
- objectsize is how much total data to write per object
- disablechecksum sets x-amz-content-sha256 header to UNSIGNED-PAYLOAD instead of calculating payload hash values
- debug enables extra debugging output
The upload/download option specifies if this is an upload or download test. The download object must exist already, so it is suggest to start with upload then follow with download.
When downloading, data is retrieved from the server and then discarded so that there is no artifical bottleneck from local storage.
Required options are the standard -access -secret -region -bucket -endpoint for any s3 session. Requires read/write permissions ot the bucket depending on test.
To approximate a workload with many small objects spread across prefixes, keep -concurrency 1 because multipart concurrency does not help for small objects, then use -workers to control the number of in-flight objects and -prefixes to distribute keys.
Use the same naming options with -download after upload to replay reads against the same object set.