期權定價

This post demonstrates how to use reinforcement learning to price an American Option. An option is a derivative contract that gives its owner the right but not the obligation to buy or sell an underlying asset. Unlike its European-style counterpart, American-style option may exercise at any time before expiry.

這篇文章演示了如何使用強化學習來定價美國期權。 期權是一種衍生合同，它賦予其所有者購買或出售基礎資產的權利，但沒有義務。 與歐式期權不同，美式期權可以在到期前的任何時間行使。

American Option is known to be an optimal control MDP (Markov Decision Process) problem where the underlying process is Geometric Brownian motion ([1]). The Markovian state is a price-time tuple and the control is a binary action that decides on each day whether to exercise the option or not.

眾所周知，美式期權是一個最優控制MDP(馬爾可夫決策過程)問題，其中基礎過程是幾何布朗運動([1])。 馬爾可夫狀態是一個價格時間元組，而控件是一個二進制操作，它每天決定是否執行該期權。

The optimal stopping policy looks like the figure below, where the x-axis is time and the y-axis is the stock price. The curve in red is commonly called the optimal exercise boundary. On each day, if the stock price falls in the exercise region that is located above the boundary for a call or below the boundary for a put, it is optimal to exercise the option and get paid by the amount between stock price and strike price.

最佳停止策略如下圖所示，其中x軸是時間，y軸是股票價格。 紅色的曲線通常稱為最佳運動邊界。 每天，如果股票價格位于看漲期權的邊界上方或看跌期權的邊界下方的行使區域中，則最好行使期權并按股票價格與行使價之間的金額獲得報酬。

One can imagine it as a discretized Q-table as illustrated in dotted grids. Every day the agent or the trader looks up the table and take action according to today’s’ price. The Q-table is monotonous in that all the grids above the boundary yield a go-decision and all the grids below yield a no-go decision. Therefore Q-learning suits well to find the optimal strategy that is defined by this boundary.

可以想象它是離散的Q表，如虛線網格所示。 代理商或交易員每天都會查表，并根據今天的價格采取行動。 Q表是單調的，因為邊界上方的所有網格均產生決策，而邊界下方的所有網格均產生決策。 因此，Q學習非常適合找到由該邊界定義的最佳策略。

The remainder contains three sections. In the first section, a baseline price is computed using classical models. In the second section, an OpenAI gym environment is constructed, similar to building an Atari game. and then in the third section, an agent is trained with DQN (Deep Q-Network) to play American options, similar to training computers to play Atari games. The full Python notebook is located here on Github.

其余部分分為三個部分。 在第一部分中，使用經典模型計算基準價格。 在第二部分中，構建了OpenAI體育館環境，類似于構建Atari游戲 。 然后在第三部分中，對代理商進行了DQN(深度Q網絡)培訓，以玩美式期權，類似于訓練計算機玩Atari游戲。 完整的Python筆記本位于Github上 。

第一節-基準 (Section One — Baseline)

There are many ways to price an American option, from for example binomial tree to Longstaff-Schwartz Monte Carlo methods. Here I use QuantLib package to price a one-year American put option.

有多種定價美式期權的方法，例如從二叉樹到Longstaff-Schwartz蒙特卡洛方法 。 在這里，我使用QuantLib packag e為一年期美國看跌期權定價。

pricing_dict = {}bsm73 = ql.AnalyticEuropeanEngine(bsm_process) european_option.setPricingEngine(bsm73) pricing_dict['BlackScholesEuropean'] = european_option.NPV()analytical_engine = ql.BaroneAdesiWhaleyEngine(bsm_process) american_option.setPricingEngine(analytical_engine) pricing_dict['BawApproximation'] = american_option.NPV()binomial_engine = ql.BinomialVanillaEngine(bsm_process, "crr", 100) american_option.setPricingEngine(binomial_engine) pricing_dict['BinomialTree'] = american_option.NPV()print(pricing_dict){'BlackScholesEuropean': 6.92786901829998, 'BawApproximation': 7.091254636695334, 'BinomialTree': 7.090924645858217}

The last line is the output, which says this American option is worth $7.091, while its European counterpart is worth $6.928. This implies an early exercise premium of $0.163.

最后一行是輸出，表示此美式期權價值$ 7.091，而其歐洲期權價值$ 6.928。 這意味著提前行使期權費為0.163美元。

第二部分-OpenAI體育館環境 (Section Two — OpenAI Gym Environment)

It is standard to derive from OpenAI gym environment. This makes our work expandable to further studies such as exotic options and stochastic volatilities. The underlying theory is the famous Black-Sholes framework and the underlying asset follows Geometric Brownian motion in the risk-neutral world. This is realized in the step function below,

它是從OpenAI體育館環境衍生而來的標準。 這使我們的工作可擴展到進一步的研究，例如外來期權和隨機波動率。 基本理論是著名的Black-Sholes框架 ，基本資產在風險中性世界中遵循幾何布朗運動 。 這在下面的步進功能中實現，

def step(self, action):if action == 1: # exercise reward = max(K-self.S1, 0.0) * np.exp(-self.r * self.T * (self.day_step/self.N))
done = Trueelse: # holdif self.day_step == self.N: # at maturity
reward = max(self.K-self.S1, 0.0) * np.exp(-self.r * self.T)
done = Trueelse: # move to tomorrow
reward = 0# lnS1 - lnS0 = (r - 0.5*sigma^2)*t + sigma * Wt
self.S1 = self.S1 * np.exp((self.r - 0.5 * self.sigma**2) * (self.T/self.N) + self.sigma * np.sqrt(self.T/self.N) * np.random.normal())
self.day_step += 1
done = False tao = 1.0-self.day_step/self.N # time to maturity, in unit of yearsreturn np.array([self.S1, tao]), reward, done, {}

The AmeriOptionEnv takes action 0 as to hold or not exercise, and action 1 as to exercise. If we stick to the no-exercise policy until expiry, this essentially becomes stock price simulation and serves as input to price European option as control variate.

AmeriOptionEnv采取行動0來決定是否進行鍛煉，采取行動1來進行鍛煉。 如果我們在到期前堅持不運動的政策，這實質上就是股票價格模擬，并作為控制變量作為歐洲期權價格的輸入。

import matplotlib.pyplot as pltenv = AmeriOptionEnv()
s = env.reset()sim_prices = []
sim_prices.append(s[0])for i in range(365):
action = 0 # hold until expiry
s_next, reward, done, info = env.step(action)
sim_prices.append(s_next[0])plt.xlabel('Date')
plt.ylabel('Stock Price')
plt.plot(sim_prices)Stock Price Gym Simulation股票價格模擬

第三節-DQN定價 (Section Three — Pricing with DQN)

Once the gym environment is constructed, we are ready to price the American option using reinforcement learning, specifically DQN (Deep Q-Network) in this post. Here I use the Tensorflow TF-Agents Library. Alternative choices are other OpenAI Gym compatible libraries such as Pytorch and OpenAI baseline.

構建好體育館環境后，我們準備使用強化學習(特別是本文中的DQN(深度Q網絡))為美國選項定價。 在這里我使用Tensorflow TF-Agents Library 。 其他的選擇是其他與OpenAI Gym兼容的庫，例如Pytorch和OpenAI基線 。

The code follows precisely the TF-Agents API document. The only changes I made are importing customized AmeriOption environment and adjusting hyper-parameters that are more pertinent to the one-year option than Cartpole game.

該代碼恰好遵循TF-Agents API文檔。 我所做的唯一更改是導入自定義的AmeriOption環境并調整與一年選項相比比Cartpole游戲更相關的超參數。

As labelled in Jupyter notebook, the RL model is constructed in the following steps:

如Jupyter筆記本中標記的那樣，RL模型是按照以下步驟構建的：

Import user-defined AmeriOptionEnv Environment.

導入用戶定義的AmeriOptionEnv環境。

Define deep Q-network

定義深度Q網絡

Employ a DQN agent

聘請DQN代理

Construct Experience Replay Buffer

構造體驗重播緩沖區

Train the agent in episodes

在情節中訓練特工

Use the trained policy to price option

使用經過培訓的政策為期權定價

According to API, a TF-agent is defined as

根據API，TF代理定義為

agent = dqn_agent.DqnAgent(
train_env.time_step_spec(),
train_env.action_spec(),
q_network=q_net,
optimizer=optimizer,
td_errors_loss_fn=common.element_wise_squared_loss,
train_step_counter=train_step_counter)agent.initialize()

to be aware of the environment states, action space, the deep neural network for policy evaluation, and an optimizer on temporal-difference loss function to do TD-optimization.

要了解環境狀態，行動空間，用于策略評估的深度神經網絡，以及針對時差損失函數的優化器以進行TD優化。

Policy Training Performance政策培訓績效

The training performance is shown as above. It is rather noisy because the evaluation step uses only 10 simulation paths and is subject to Monte Carlo randomness. For example, we know the option price is around $7 yet the average price can go as high as $12. Therefore, after learning the optimal stopping policy, it is essential to do a full-blown Monte Carlo to find the actual price as below.

訓練效果如上所示。 這很嘈雜，因為評估步驟僅使用10條仿真路徑，并且受到蒙特卡洛隨機性的影響。 例如，我們知道期權價格約為7美元，但平ASP格可能高達12美元。 因此，在學習了最佳停止策略之后，有必要做一個成熟的蒙特卡洛法以找到如下的實際價格。

import pandas as pdnpv = compute_avg_return(eval_env, agent.policy, num_episodes=2_000)
pricing_dict['ReinforcementAgent'] = npvpricing_df = pd.DataFrame.from_dict(pricing_dict, orient='index')
pricing_df.columns = ['Price']
print(pricing_df)

The Reinforcement learning agent values the price at $7.057, implying an early exercise premium $0.129, This result is in line with classical baseline models.

強化學習代理的價格為$ 7.057，這意味著早期行使價為$ 0.129，此結果與經典基準模型一致。

結論 (Conclusion)

In this post, we prepare a gym environment and then train a DQN TF-Agent to price an American option. The result is encouraging with a reasonably good price that is in line with classical baseline models. Some improvements include,

在這篇文章中，我們準備了一個健身房環境，然后訓練了DQN TF-Agent來定價美式期權。 其結果是令人鼓舞的，它的合理價格與經典基準模型相一致。 一些改進包括

For practitioners,

對于從業者

Use a mirror AmeriOption gym environment to provide antithetic variates.

在AmeriOption健身房環境中使用鏡子以提供對立變量 。

In function compute_avg_return, continue the simulation path to price European option as control variates.

在函數compute_avg_return中，隨著控制變量的變化，繼續進行模擬以對歐式期權定價。

For researchers,

對于研究人員

Add another MDP process to capture stochastic volatility.

添加另一個MDP流程以捕獲隨機波動率。

Instead of using the default network structure, design a specialized multi-layer network to enable transfer learning into other maturities as well as options on rates, futures, FX, and exotic products.

與其使用默認的網絡結構，不如設計一個專門的多層網絡以使學習轉移到其他到期日以及利率，期貨，外匯和奇異產品的選擇權。

翻譯自: https://medium.com/swlh/option-pricing-using-reinforcement-learning-ad2ddca7735b

期權定價

總結

以上是生活随笔為你收集整理的期权定价_强化学习的期权定价的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。