There are a lot of exchanges for trading cryptocurrencies, which all have pros and cons. This lead to a comparison of APIs with respect to this project's use case. It is important that the API has a good and easy documentation and an extensive python library/wrapper.
The following table shows the full comparison of five cryptocurrency API's 10:
| Platform | Binance | CoinBase | Kucoin | Coinmarketcap | Kraken |
|---|---|---|---|---|---|
| Pros | Directly sell and buy with the API, Connector to diff. Languages (python, Ruby), Commission when using BNB coin, 1200 free Requests per Minute, Binance has one of the biggest markets |
API can be used as an Exchange or Wallet, Real-Time Notifications for Account Events |
Many currencies(>200), Telegram Group for Support, 1800 Free Requests per MinuteCrypto Lending available |
Good integration in many languages(python,node.js,php), 11 endpoints in real time for free |
Good python integration and code examples directly in documentation |
| Cons | There have been outages in the past. When exceeding the transaction limits, the IP will be temporarily banned |
Only etc, eth, btc cash, litecoin, Only 180 Requests per Minute |
Performance issues (payments & trades hold back) | No historical Data, Pay 29$/month for 1month historical data 60 Requests per Minute Not all REST API Calls available |
ID Verification needed, No test environment, Trading Volume and Trade Calls depending on account level (starter, intermediate, pro): When 15 requests are reached, a decay of -0.33/sec is applied and you need to wait |
| Historical Data | Yes | Yes | Yes | No | Yes |
| Fees | 0.1%, but when using BNB can be lowered to 0.075% | 1% | 3-5%, when buying with FIAT 0.1% for crypto, 0.08% when buying with Kucoin Token |
not found any data | 0.9% fee for any stablecoins 1.5% fee for any other crypto or FX pair |
| Rating of Documentation | Very good | Good | Very good | Okay | Very good |
| Availability of Python Libraries/Wrappers | Very good and updated. https://github.com/sammchardy/python-binancehttps://github.com/binance/binance-connector-python |
Last updated 8 years ago. https://github.com/resy/coinbase_python33 |
Very good and updated. https://github.com/sammchardy/python-kucoinhttps://github.com/Kucoin/kucoin-python-sdk |
Not really detailed. https://github.com/rsz44/python-coinmarketcap |
Not updated recently. https://github.com/veox/python3-krakenex |
When it came to the decision which API to use, it was quite easy to filter out the ones that weren't suitable. CoinBase was the first to drop because it has only five coins and no currently active python libraries available.
Coinmarketcap was also not very intriguing because it has limited Endpoints and no historical data. Paying 29$ a month for 1 month of historical data didn't seem worth it when others offer it for free. Their python wrapper was also not very detailed.
I personally use Kraken, so the ID Verification wouldn't even be a problem, but they do not offer a sandbox (test environment). Their way of working with request limits is also very different to the others and seemed to be a limiting factor. But the biggest problem with Kraken was that there is no recently updated python wrapper available.
Kucoin offers many currencies and the most free requests per minute, but reported some performance issues, which lead to the decision to Binance. At least at first.
Binance has one of the biggest markets with the highest liquidity and small fees. They have a very good documentation and an updated python wrapper with lots of tutorials. The only thing to keep in mind is the potential ban of my IP when the request limit is reached.
The verification process was long and even required to send some Bitcoin to the Binance Wallet. Afterwards, it was possible to create the API-Keys, which are needed to connect the Client with Binance. Getting live price data from the Binance API was working perfectly, so everything seemed fine. Visualisations of Charts were made to have a better overview and make a small simulation to test the strategy (see Papertrading).
Then it was time to implement the trading strategy into the Binance sandbox and the problem appeared. The Sandbox Testnet didn't accept the API-Keys and even after a while of debugging the error, it wouldn't work, leaving us with no choice but to drop Binance for Kucoin. Setting up Kucoin was faster and their sandbox worked perfectly.
After registration you need to go through a verification process. Typically, this is a call with a person that verifies your identity or uploading a photo of a personal ID-Card, but Kucoin only needs verification via Google Authenticator.
After that the API-Keys could be generated. These Keys are needed to connect and verify the Client with the API.
The python-kucoin wrapper helps with the Kucoin API, so there is no need to manually connect with the API via HTTP-Requests.
Acquiring data from the Kucoin API is fairly easy. Getting the latest price for BTCUSDT, the account balance, and making a market order are all done with a few lines of code. As can be seen below in the output, the current USDT Balance in the account is over 48k USDT. Kucoin offers every account a virtual amount of 250k USDT (~11 BTC) to trade with in their sandbox environment. They also offer Sub-Accounts, which is great to test a strategy for a specific amount of money, without risking to lose it all, when something goes wrong. This meant, papertrading could be performed and evaluated.
from kucoin.client import Client as kucoinClient
kClient = kucoinClient(kconf.KUCOIN_KEY, kconf.KUCOIN_SECRET,kconf.KUCOIN_PASS,sandbox=True)
tickers = kClient.get_ticker(symbol="BTC-USDT")
btc_price = tickers["bestAsk"]
print(f"Current BTC Price: {btc_price}")
accounts = kClient.get_accounts(account_type="trade")
usdt_balance = accounts[0]["balance"] #for Main Client
btc_balance = accounts[1]["balance"] #for Main client
btc_in_usdt = float(btc_balance) * float(btc_price)
print(f"USDT Balance: {usdt_balance} $")
print(f"BTC Balance: {btc_balance} ({btc_in_usdt} $)")
# Make a Market Order for 5% of the current USDT Balance
funds = re.match(r'\d+.\d{3}', str(usdt_balance*0.05)).group(0)
order = kClient.create_market_order('BTC-USDT', kClient.SIDE_BUY, funds = funds)
Output:
Current BTC Price: 22420.7
USDT Balance: 48942.07811348 $
BTC Balance: 0.00194057 (43.508937799 $)
Building the strategy to make signals when a trade (buy or sell) should be executed, was a real challenge. There are hundreds of different strategies on different technical indicators, but now the buy or sell signal should be based on the sentiment of tweets on Twitter [FR 30].
At first, the idea was to calculate the average of sentiment for a particular time period and if this average reaches a certain threshold, a trade should be executed.
In the left table in Figure 14 you see the timestamp, sentiment score and the corresponding meaning for each tweet.
Figure 15: Sentiment for a single tweet on the left, Average Sentiment and counted Tweets for a timeperiod of 1h
The right table shows the grouped and counted tweets for intervals of 60 min. Afterwards, the average sentiment is calculated and converted into a Buy or Sell Signal. It can be seen that there isn't a lot of tweets and some timestamps are missing. This was an error from SQL-Alchemy, which lead to the state of not committing the current Database-Session and thus leading to missing timestamps. This is fixed in the final version, but was kept here to correspond to the below chart.
The sentiment score for single tweets are all very different, but the average will naturally move towards neutral sentiment (0.0). This shift would have been even more if all the neutral tweets had not been removed beforehand.
It would have been easy to just define the buy signal when the average sentiment is positive and a sell signal when the average sentiment is negative. But this would mean a lot of buy signals for the above example and since the database is not quite big enough to see if this average is only positive at the moment or if the calculation just tends to be a little above neutral. The bigger the time intervals, the more tweets and the bigger the shift towards neutral. This is why the threshold is set to 0.2.
An average sentiment above 0.2 means Buy, below 0.2 means Sell.
Figure 16 shows the sentiment and Bitcoin price from August 5th till August 6th. If the sentiment is above 0.2 (positive or very positive) it is marked as a buy signal in the chart (green triangle). Since this looked promising it was implemented to real-time papertrading in the Kucoin Sandbox [FR 40].
The trade-file starts with the essential function to collect all the single tweets from the database and stores them in a pandas DataFrame. As explained in Data Acquisition, this is needed to delete the duplicates with the following methods:
duplicates = list(
df.index[
df.duplicated(
subset=["Tweet"], keep=False
)
]
)
df.drop_duplicates(
subset=["Tweet"], keep=False, inplace=True
)
Afterwards, the neutral tweets are filtered, and the rest are counted and the mean/avg is calculated for a resampled interval of 1h. This means all the timestamps from 1h (e.g. from 16:00 - 17:00) are grouped, and a function is applied to all of them. This function could be to summarize, count or calculate the mean/avg.
df = df[df["Sentiment Score"] != 0.0]
count_tweets = df.resample(f"1H").count()
mean_df = df.resample(f"1H").mean().sort_index(ascending=False)
These two series get concatenated together to have one table:
And these last three timestamps are important for the trading. The first row is showing the actual timestamp. So at the time this picture was taken it was somewhere between 05:00 and 06:00. Since the tweets are collected live and every few seconds this row will be updated frequently and the avg and total tweets will change.
This is why the second row will be selected as the trading decision. Firstly, it will be checked if a trade was already made for this timestamp. Then, if the signal indicates buy, a market order will be created with 5% of the available USDT Balance (after a check, if there are any funds available).
A sell order will sell a quarter of the current Bitcoin holdings.
At last, some metrics from the trade are uploaded into a separate table on Heroku for a better overview of the trades and later calculation of current Profit and Loss (PNL). An excerpt from this table is shown below in figure 18. It contains the sentiment average and the time period, as well as information about the trade itself. The last balances are actually the balances after the trade was made. So, it is possible, to look back at all the balances without gaps.
Figure 19 shows the Heroku Logs. At first, the runner is started, that listens to the tweets and adds them to the database. When you see the line Listening to tweets now..., you know, everything is working properly.
As explained in the Backend-Section, the scheduler is executing the trading script every hour.
The sentiment average is shown and the balances before and after the trade. It also says if a trade already exists and then either starts a new trade or does nothing.
The Heroku logs were an essential part of the development-phase, since they allowed for a good way of debugging. Usually, no print-statements are left in the final code, when it's entering the production phase. However, Heroku seems to only print the print-statements rather than from the logging-library, which is otherwise used.
Therefore, a few print statements are left in the final code.
The system was trading fine for a couple of days, but then, all of a sudden, it did not execute any trades for a couple timestamps. As can be seen in figure 20 below, the USDT and BTC Balance are switched, and the system tried to buy for an amount of 0 USDT, which did not work. Interestingly, in figure 20, you can see that this was not a consistent state since there are some trades at random timestamps. The problem lay with Kucoin. When acquiring the current account balances, USDT normally came before BTC. However, this order seems to be switched at random. This unforeseen coincidence needed to be checked by the system and be acted upon.
The biggest challenge came with the sandbox. All the trading worked perfectly, but it was weird to see that the first trade of 250 $ equalled a BTC amount of 0.06, which would be more than 1300 $. This didn't make any sense.
Since these different prices were not prone to Bitcoin alone, but all coins had different prices in the sandbox, the Kucoin Support was contacted.
The answer (figure 22), confirmed that the prices in the sandbox are just different, and another solution was needed.
The solution was to get the correct price from somewhere else and in this case: Binance.
To convert the current BTC Holdings a real-time price was needed.
This is done with a few lines of code and a typical HTML-Request:
url = "https://api.binance.com/api/v3/ticker/price?symbol=BTCUSDT"
data = requests.get(url)
data = data.json()
To get all the historical data from Binance, the python-binance wrapper was used:
frame = pd.DataFrame(binance_client.get_historical_klines(symbol,1,lookback))
frame = frame.iloc[:,:6]
frame.columns= ["Time","Open","High","Low","Close","Volume"]
frame = frame.set_index("Time")
frame.index = pd.to_datetime(frame.index,unit="ms")
frame.index = frame.index + timedelta(hours=2) #utc to local
frame
The result looked like this:
The method get_historical_klines() accepts arguments to look for intervals in a past time period. For example, in figure 23 we are looking for the past day ("1d") with intervals of one minute ("1m").
The timestamp and the closing price were then used for further calculations and visualisations.