ci: remove test that doesn't work on GH Actions (#170)

This removes performance tests because they don't work reliably on GH
actions (which, fair, it's free-tier compute)

Author: crankynetman