{'fraction_non_alphanumeric': 0.031572481572481575, 'fraction_numerical': 0.01715806715806716, 'mean_word_length': 5.0002457002457, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Reinforcement Learning Applied to a Cryptocurrency Portfolio in a Complexity Environment

Recently, cryptocurrencies have been used as financial assets and have presented positive returns, albeit their volatility is high. This paper aims to elaborate a hypothetical cryptocurrency portfolio and to do so, employs machine learning and an optimization algorithm to define the ideal amount to be allocated in each asset. The results show the hypothetical portfolio presents superior returns and lesser volatility compared to other allocation strategies.


Introduction
defines a virtual coin as a chain of digital signatures, where two agents transfer assets by validating a public and a private key, both cryptographed. The entire ideology behind the creation of cryptocurrencies revolves around having a financial system which does not depend on trust in institutions and that safeguards the privacy ofagents involved. Chiu and Koeppl (2017) remark that in a scenario with virtual coins, means of payment are nothing but a set of bits.
Specifically, big volumes of cryptographed data compose a transaction, and they compose a block. When a block is validated, it becomes part of a chain (hence the name blockchain). Even though Bitcoin is the most famous cryptocurrency, there are many more, but always revolving around the same blockchain concept. According to CoinMarketCap, in March 2018 there were 1592 listed cryptocurrencies, moving around 25 billion dollars every day, with a total market value of more than 400 billion dollars.
Recently, cryptocurrencies have become more popular as a portfolio choice even though one of their defining features is high volatility. Yermack (2013) shows that correlation between Bitcoin and other coins is near zero, which mostly writes off its use as medium of exchange and makes it more of a speculative asset in practice. As high volatility is intrinsic to this market, there is a need to develop a good strategy to build a portfolio with positive returns. However, the classic approaches 1 to portfolio management in financial studies might face hardships in this market because of its instability.
Because of such instability, it might be said agents purporting to maximize their returns investing in cryptocurrencies find themselves inside a complex system. According to Arthur et al. (1997), in complex systems, interactions occur in a diffuse way between adaptative and heterogeneous economic agents. New features are introduced all the time, maintaining agents in a perpetual process of learning, adaptation and evolution, therefore being out of equilibrium.
Given this scenario, the methodology chosen is machine learningparticularly, reinforcement learningto find thebest weights of each asset in the portfolio and thus, the best allocation. In order for the results to be more credible, the portfolio will go through daily rebalancing. It will be composed by the five cryptocurrencies with the biggest capitalization at the time: Bitcoin (BTC), Ethereum (ETH), Ripple (XRP), Bitcoin Cash (BCH) and Litecoin (LTC).
The usage of machine learning in economics is related to the idea of complexity, whose precise definition in economics is still open to debate 2 .To Arthur (2010), complexity in economics is linked to a tendency towards disequilibrium. Complexity economics is not a temporary adjunct to static economic theory, but theory at a more general, out-ofequilibrium level (ARTHUR, 1999, p. 109). Stodder (1995) relates complexity to a web of complicated inter-relationships where the interaction between involved parts in this complex system affects the individual behavior of each one of them. Simon (1962) understands that in complex systems, the whole is more than the sum of all parts, therefore it is hard to infer the properties of the whole.
To model complex systems, a popular approach is machine learning. It is important to note that the pioneering of computational economics is Schelling´s (1971) model of racial segregation that could be understood as a classical reference on the study of computational economics in various scientific scenarios. Agent-based computational economics -ACEis the area of computational economics that allows the study of economies modeled as evolving systems of autonomous interacting agents (TESFATSION, 2003) 3 . Arthur (2006) argues that while neoclassical economics is used to studying actions, strategies and expectations of agents in equilibrium, reinforcement learning models allow the researchers to create environments and observe how agents react out of steady state. In dealing with problems of high complexity humans live in a world of learning and adaptation (ARTHUR, 1992, p. 3). Pyka and Fagiolo (2005) point out that computational models allow the insertion of heterogeneity between agents, emphasizing bounded rationality, bottom-up approaches and microfoundations of economics. Therefore, learning, adapting and evolving is a most important feature of these models.
For Holland;Miller (1991, p. 365), a critique on current theoretical constructs based on optimizing principles is that real agents lack the behavioral sophistication necessary to derive the proposed solutions. For them, this dilemma is resolved if it is postulated that adaptive mechanisms, driven by market forces, lead the agents to act as if they were optimizing. Artificial Adaptive Agents (AAA) models the link between adaptation and market forces, and can be used to analyze the conditions under which optimization behavior will (not) occur. For Holland;Miller (1990), a major feature of AAA is their ability to produce emergent behavior and a wide variety of behaviors can arise endogenously, but these behaviors are constrained by the initial structure. There are different computer-based adaptative algorithms for exploring AAAclassifier systems, genetic algorithms, neural networks, and reinforcement learning mechanisms (HOLLAND; MILLER, 1991).
According to Honda, Facure and Yaohao (2017), machine learning aims to enhance an agent's performance through conducting a task and acquiring experience. One method of machine learning is reinforcement learning, which employs trial and error, based in an entry function, and evaluates the results of each action taken. The process happens by way of attributing rewards and penalties to each decision, without a need to necessarily determine how to undertake the task. In other words, this is a method that allows the algorithm to work through experimentation, being punished when it is wrong and rewarded when it is right. This kind of algorithm was employed to develop the robots that play chess against chess masters.
Economia Ensaios, Uberlândia, 36(1), Jan./Jun. 2021 ISSN impresso: 0102-2482 / ISSN online: 1983-1994 31 In particular, Murphy (1998) defines reinforcement learning as a problem of maximizing results of an agent in a certain scenario. Sutton (1992) reveals that the idea of reinforcement learning is quite old:the first researches were published by Minsky (1961), followed by Waltz and Fu (1965). However, only in the near end of the 1980s relevant studies in the area were published again, including Werbos (1987) and Watkins (1989), which connected the subject to dynamic programming and showed the link between artificial intelligence and machine learning.
According to Dias Júnior (2012), in reinforcement learning, the agent is inserted into an environment and reacts to it with some possible actions. In a first moment, the agent finds him/herself in an initial state, before the action is taken. After the action, this state is altered and this new state generates a feedback determined by a certain value for the algorithm, in order for it to discern, according to predefined criteria, if that output was desirable or not, called reinforcement. The reinforcement normally is given by {0,1} or by real numbers. Depending on the result given by the reinforcement, the algorithm will define a bigger or smaller probability of taking that decision again when it is again in that initial state. Afterwards, the cycle repeats, so it keeps learning.
Literature regarding machine learning and cryptocurrencies is recent and has received various contributions. Among the remarkable works, we can mention the studies by Alessandretti et al. (2018) or Jang and Lee (2017). The first shows that simple negotiation strategies joined with machine learning are better than standard benchmarks. However, Jang and Lee (2017) use Bayesian neural network and show that those have a good performance predicting Bitcoin price time series and explain the high volatility of these assets.
Considering what is said above, this paper's contribution to the theme is the realization of a portfolio optimization that takes reinforcement learning in consideration to choose an optimalallocation between cryptocurrencies. Therefore, in this study we verify whether machine learning tools have merit in building a profitable cryptocurrency trading strategy, even though the model does not account forreplies of the market's other players to the algorithmic trader.
In addition to this introduction, the paper featuresthree more sections. Section two presents the structure and the main features of the adopted modelthat is, materials and methods. Section threeshows the results found and their discussion. The fourth and last section presents the concluding remarks.

Cryptocurrency Selection
The study is comprised of three months, starting from the first day of January 2018 until the last day of March 2018, with daily frequency. Consider an agent that aims to allocate his/her net worth, initially $1.000.000,00, among five different cryptocurrencies, whose eligibility criterion was their capitalization. According to CoinMarketCap (2018) On each time period t (that is, every day), the agent allocates his/her net worth in such a way that the sum of the modulus of the weight of each coin is equal to 100%. Precisely, the agent might, by using future contracts, realize buy (if s/he believes the price will rise) or sell (if s/he believes the price will fall) operations.
Besides choosing the position (buy/sell), the agent needs to define the intensity of each negotiation, which means choosing the percentage of the net worth that should be allocated in each asset. Assuming that the agent has the intent to diversify his/her portfolio, minimum and maximum ranges (limits) of the net worth percentage that can be allocated in each cryptocurrency are established. As seen next, the implemented reinforcement learning increases (reduces) the ranges in case the previous strategy has obtained success (failure). Lastly, it is worth mentioning that transaction costs associated to each portfolio reallocation are also added daily. Therefore, by the end of each day, it is possible to calculate the portfolio's total net worth.

Data, Portfolio Composition and the Reinforcement Learning Algorithm
This paper will observe the empirical series of real cryptocurrency prices from January 2018 to March 2018 with daily frequency 4 . After the price of the assets is obtained, it is possible to calculate the returns. Formally: in which R is the asset return; P is the price; Q is the quantity; and t is the period analyzed.
Aiming to make the model the most realistic possible, the exchange costs of the BitMex exchange, one of the biggest cryptocurrency brokers in the world, will be considered. The transaction cost (TC) of an order is 0,05% of the order's value. Therefore, TC is given by: In the initial period, the portfolio's net worth is expressed by: In turn, the portfolio's net worth in the (t≠1) period given by: The strategy review process occurs between the end of the t period and the t+1 period. While it happens, the agents evaluate the past performances of their strategies (allocations) and search for the strategies that offer the best rewards inside the possible ranges. In this adaptative process, the agents utilize the reinforcement learning mechanism to reevaluate and find the best allocation ranges.
In the reinforcement learning process, as stressed out by Roth and Erev (1995, p. 165), a decision strategy that presented a positive reward in the t period has its probability of being chosen in the the t+1 period reinforced. However, in the case of the payoff result being undesirable in the t period, the probability of this strategy being picked in the t+1 period is weakened. Formally, this relationship is expressed by: (1-0.10) if < ( −1) .
By the allocation range updating function, it can be inferred that given a strategy that results in a positive return in the t period, the allocation range will be increased for the t+1 period. But if the chosen asset's price is reduced in the t period, its allocation range will be reduced in t+1 period, thus the decisionmaker tends to allocate less resources in the asset with the lesser reward. It is worth reinforcing that price changes in the model are completely exogenous, with the causes of these changes not being the paper's focus.
In order to define the model's initial condition and avoid possible selection biases, for the t=0 period, it is assumed that the five cryptocurrencies are adopted in a uniform way (20% of the net worth in each asset) and only the buying position is able to be carried out in this initial period 5 . After the designation of initial conditions, it is possible to obtain the return of assets (equation 1) and subsequently calculate the operation costs (equation 2).

Computational Methods
To implement the agent's choice model, Jupyter Notebooka text editor for the Python programming languageis the choice. It must be reminded that for each t period, the agent will be faced with the same set of the five cryptocurrencies mentioned beforehand and has to find a combination (buy/sell) in such a way that all his/her net worth is utilized.
The method chosen to solve the optimization problem is a Sequential Least Squares Programming (SLSQP) algorithm, employed in the taskof finding the best weight distribution for each cryptocurrency in the portfolio in order to maximize the return in the present period. Afterwards, these weights will be allocated in the process of asset choice in the next period. Therefore, it is possible to calculate the portfolio's net worth (equation 4). 5 Though this initial condition is the presented, it is worth remarking that different initial conditions (fraction of agents using each cryptocurrency is randomly decided) were utilized and final results were not altered. Thus, the initial condition chosen was that each strategy is equally used by all agents. SLSQP is an iterative method employed in numerical optimization to find solutions to problemsof constrained nonlinear optimization. Precisely, the optimization mechanism adopted aims to obtain the largest net worth from the previous day and return the optimal weights of each asset. This result is found by the algorithm trying diverse weight distributions and analyzing their results, returning the best one afterwards. That is, it realizes a finite search between the allocation options and chooses the one with the biggest returns.
These tries follow two constraints. The first is related to the sum of the weights. The sum of the absolute value of the assets (bought/sold) must be equal to 1 (or 100%). The second is related to the maximum and minimum ranges (limits) of each asset's allocation. All the currencies start the study with limits of (-40%, 40%). However, these limits are altered due to the implemented reinforcement learning mechanism. Once the optimal allocation for each asset in the next period is defined, a new round is initiated. All the procedures described here for the optimal weight choice of assets in the t=1 period can be applied for any other t ≥2 period.

Portfolio Results
This section presents the portfolio results with an initial net worth of $1.000.000,00 and the analysis period from January 2018 to March 2018. Both for the theoretical portfolio and for Bitcoin (BTC), the cumulative financial returns, the cumulative percentage returns, the total costs, the variance, the largest drawdown (largest price fall in a day) and the average daily return were calculated. Figure 1 shows the net worth of the cryptocurrency portfolio proposed and all the net worth invested in Bitcoins.
It is possible to see that the theoretical portfolio is constantly above Bitcoin, except for a few days in the sample's beginning. At the period's end, the portfolio's cumulative financial return is $ 455.508,57 while Bitcoin's is a loss of $ 344.421,05. That means a return of 45,55% for the portfolio, against -34,44% for Bitcoin. It is worth mentioning that this return already accounts for transaction costs, which totaled $ 86.000,12. As there is no brokerage when acquiring only Bitcoin with the starting net worth, the transaction cost in that case is zero.
Not only against Bitcoin is the theoretical portfolio superior. Figure 2 shows that, even though Ethereum (ETH) was superior during some moments, at the period's end all of the cryptocurrencies lost out to the proposed portfolio.

Figure 2: Comparing with the Five Biggest Cryptocurrencies
Source: own elaboration.
With regard to the risk-associated variables, the chosen measures were the portfolio's variance and maximum drawdown against Bitcoin's. In the study's period, the proposed portfolio's variance was 0,59% and Bitcoin's was 0,41%, showing the portfolio had a bigger risk. However, the maximum daily drawdown was bigger in Bitcoin's case. While the portfolio's biggest loss in a day was 15,74%, Bitcoin's was 17,23%. To measure the daily returns of both assets, the average daily returns were utilized. Figure 3 shows this data in a visual way. The portfolio's average results were much superior to Bitcoin, as the portfolio had a 0,76% average daily return while Bitcoin had -0,33%. It is possible to see in the Figure  3that the returns of both assets were, to a certain extent, inversely correlated in most days.
The results obtained show that the value invested in the theoretical portfolio would have been superior to any cryptocurrency. The portfolio's biggest drawdown was inferior to Bitcoin and the portfolio's returns were much bigger than Bitcoin's returnseven considering transaction costs. The portfolio had a bigger variance though (hence, it was riskier).

Comparing to Other Strategies
For comparison purposes, two more strategies were tested, in order to understand how the resources directed to allocation decisions impact results. The first one was the same model as the portfolio but keeping the allocation limits fixed from -40% to 40%, therefore, without the reinforcement learning mechanism. This strategy's result compared to Bitcoin can be observed in Figure 4. This strategy had much worse returns, demonstrating reinforcement learning's value to improve allocation decisions as time goes on. This strategy's cumulative return was a loss of $ 206.582,75equivalent to -20,66%, which is closer to Bitcoin's result that was shown in the previous sub-section. The variance was also closer to Bitcoin, with a value of -0,43%. There was, too, a rather insignificant reduction in the maximum daily drawdown, which was 12,84%. Figure 5 shows the daily return for each day of the study for the model without reinforcement learning compared to Bitcoin. The average daily return was negative, of -0,08% against -0,33% from Bitcoin.  The second model for comparison is a portfolio with fixed weights (20% for each currency). This is the most ingenue model amongst the tests, as it not only does not learn with the right and wrong decisions from before but does not even take allocation decisions. Figure 6 shows the comparison to Bitcoin. The period's cumulative financial return is a loss of $ 410.371,29, superior even to the loss from allocating everything in Bitcoin for the same period. Even so, the variance was superior to Bitcoin, with a value of 0,46%. The portfolio's biggest daily loss was 22,59%, much superior to Bitcoin and the biggest among the analyzed strategies. Figure 7 shows the returns for each day of the series for the fixed weights strategy and Bitcoin.
This strategy's average return was -0,43%, the worst among all strategies, and the maximum drawdown was also the biggest. An interesting observation from these results is that with fixed weights of 20% for each currency, the daily returns are very close to Bitcoin's, which shows high correlation of the other currencies with it.
For a more comprehensive analysis of the more ingenue strategies and the proposed portfolio, the approach will be inspired on the Sharpe ratio, very famous in traditional financial market studies. Formally, it is given by: where RR is Risk/return ratio; PR is portfolio's return; BR is Bitcoin's return; PV is portfolio's variance. For all estimates, Bitcoin's return is -34,44%. For the theoretical portfolio proposed, in which the period's return is 45,55%, and variance is 0,59%, the ratio is: For the comparative portfolio without reinforcement learning, in which the period's return is -20,66%, and variance is 0,43%, the ratio is: For the comparative portfolio with fixed weights of 20%, in which the period's return is -41,03%, and variance is 0,46%, the ratio is: After these calculations, it can be seen that by analyzing risk-return through this approach, the proposed portfolio was much superior to comparative portfolios. As a matter of fact, the fixed weights model actually has negative risk-return. That happens because, even though this study does not focus on risk reduction but on return on asset maximization, there was low variation between the variances of the proposed portfolio and Bitcoin, but a big raise in expected returns.

Concluding Remarks
In the last few years, the advent of cryptocurrencies like Bitcoin caused a stir among financial market players. Cryptocurrencies are financial assets that might yield high returns, and it is worth to explore the subject from a portfolio optimization return, as envisioned by Markowitz (1952). To find the optimal results of a cryptocurrency portfolio, the chosen method was a reinforcement learning, which rewards successes and punishes mistakes in the market's direction. The made-up portfolio started with a net worth of $ 1.000.000,00, with the study taking place from 01/01/2018 to 19/03/2018.
At the chosen period's end, the portfolio's net worth was bigger than it would be if invested in any of the five individual cryptocurrencies. There were also comparisons to other strategies. Firstly, a portfolio without reinforcement learning, with 40% ranges. For this strategy, the results were worse than the proposed portfolio but still better than investing all the net worth in Bitcoin. Another model was an allocation with ingenue weightsfixed in 20% per cryptocurrency. This was the worst allocation by all measurements. An interesting fact is that this strategy's returns and daily variationwere very similar to Bitcoin's, showing high correlation among the cryptocurrencies.  Table 1 synthesizes the risk-return analysis, utilizing an own ratio (RR) based on the Sharpe ratio. Therefore, it can be observed that the intelligent portfolio which employs reinforcement learning boasts a risk-return ratio much better than the other portfolios (Bitcoin was used as the basis return for the ratio, so it does not have a RR). The portfolio with ingenue weights goes as far as having negative RR. Even though the paper focused on return maximization, it is easy to see that, based on available results, there is no big risk difference among the options, while returns are raised immensely.
This paper shows the utility in employing computational methods such as machine learning in the solving of problems from the financial area, allowing investors to better understand some mostly uncharted territories, as is the cryptocurrency market. The usage of an agent under reinforcement learning dramatically improves returns against the rest of market. However, as the model does not account for the responses of other agents to the main agent's strategy, it should not be taken as a recommendation for real-world investing, and a suggestion for future researches is trying to model the cryptocurrency market with many agents looking for the best response against the rest of the market.