文章目錄

• 強化學習的基本概念
• • 1. 強化學習的算法步驟：
  • 2. 強化學習和其他機器學習范式的不同
  • 3. 強化學習的要素
  • • a. 智能體
    • b. 策略函數
    • c. 值函數
    • d. 模型
  • 4. 強化學習的環境類型
  • • （1）確定性環境
    • （2）隨機性環境
    • （3）完全可觀測環境
    • （4）部分可觀測環境
    • （5）離散環境
    • （6）連續環境
    • （7）情景和非情景環境
    • （8）單智能體和多智能體環境
  • 5. 強化學習的平臺
  • 6.強化學習的入門例子
  • • 小程序一：平衡擺游戲

強化學習的基本概念

1. 強化學習的算法步驟：

首先，智能體通過執行行為與環境進行交互。

智能體執行一個行為之后，從一個狀態轉移到另一個狀態

然后，智能體將會根據所執行的行為獲得相應的獎勵。

根據獲得的獎勵，智能體會知曉該行為是好是壞。

如果行為是好的，即如果智能體得到正面獎勵，那么將會傾向于執行該行為，否則智能體將會嘗試執行其他行為來獲得正面獎勵。因此，該方法本質上是一個試錯學習的過程。

2. 強化學習和其他機器學習范式的不同

在監督學習中，機器（智能體）從具有標記的輸入|輸出訓練數據集中學習。目的是使得模型對其學習進行外推和推廣，從而能夠很好適用于未知數據。在此，需要有一個對環境具備完備知識基礎的外部監督者，來監督智能體完成該任務。

在無監督學習中，將提供一個僅有一組輸入訓練數據的模型，通過模型來學習確定輸入數據中的隱藏模式。通常普遍會誤認為強化學習是一種無監督學習，其實并非如此。在無監督學習中，通過模型來學習隱藏的結構，而在強化學習中，通過最大化獎勵來學習模型。假設想要向用戶推薦新電影。無監督學習會通過分析用戶觀看過的類似電影來推薦，而強化學習則不斷接收來自用戶的反饋信息，了解其偏好電影，并在此基礎上構建一個知識庫，來推薦一部新電影。

另外還有一種稱為半監督的學習，其本質上是監督學習和無監督學習的結合。該方法涉及數據和未標記數據的函數估計，而強化學習的本質上智能體與其環境之間的交互。所以，強化學習與其他所有機器學習的范式完全不同。

3. 強化學習的要素

a. 智能體

智能體是指進行智能決策的軟件程序，在強化學習中通常是學習者。智能體通過與環境交互來執行行為，并根據其行為來獲得獎勵，比如說超級馬里奧在視頻游戲中的動作行為。

b. 策略函數

策略定義了智能體在環境中的行為。智能體決定執行何種行為取決于策略。假設從家到辦公室之間存在不同的路徑，其中有些路徑很短，而有些路徑相對很長。這些路徑可以稱為策略，因為代表了為達到目標而選擇執行的行為。策略通常用符號$\pi$ 表示。策略也可以是查找表形式或者復雜的搜索過程。

c. 值函數

值函數表示智能體在某一特定狀態下的程度。這與策略有關，通常用$v(s)$表示。值函數等價于智能體從開始狀態開始所獲得的總的期望獎勵。值函數有多種形式。最優值函數是與其他值函數相比，所有狀態下具有最大值的一種值函數。同理，最優策略是指具有最優值函數的策略。

d. 模型

模型是指智能體對環境的表示。學習可以分為基于模型的學習和無模型學習兩種類型，在基于模型的學習中，智能體利用先前學習到的信息來完成任務，而在無模型學習中，智能體僅是通過試錯經驗來執行正確行為。假設想要更快地從家到辦公室。在基于模型的學習中，只需利用先前學習的經驗（地圖）來快速地到達辦公室，而在無模型學習中，不會使用之前的經驗，而是嘗試所有不同的路線，并從中選擇較快的一種。

4. 強化學習的環境類型

與智能體交互的所有內容都稱為環境。環境是指外部世界，包括除智能體之外的一切事物。環境有不同類型：

（1）確定性環境

如果根據當前狀態就可以知道相應的結果，那么就稱為確定性環境。例如，在國際象棋中，會知道移動每個棋子侯的確切結果。

（2）隨機性環境

如果不能根據當前狀態來去確定相應的結果，那么這種環境就稱為隨機性環境。在這種環境中，會存在較大程度的不確定性。例如，在擲骰子時永遠不知道會出現什么數字。

（3）完全可觀測環境

如果智能體在任何時候都能確定系統的狀態，那么久稱為完全可觀測環境。例如，在國際象棋中，系統的狀態，即棋盤上所有棋子的位置，都是可以獲得的。因此，棋手可以做出最優決策。

（4）部分可觀測環境

如果智能體無法在任何時候都能確定系統的狀態，那么就稱為部分可觀測環境。例如，在玩撲克時，不知道對手的牌。

（5）離散環境

如果從一個狀態轉移到另一個狀態后只能有一個有限的行為狀態，那么就稱為離散環境。例如，在國際象棋中，只能有移動棋子后的有限集。

（6）連續環境

如果從一個狀態轉移到另一個狀態可以有無限的行為狀態，那么就稱為連續環境。例如，從出發地到目的地的旅行中可以有多條路線。

（7）情景和非情景環境

情景環境也稱為非序貫環境。在情景環境中，智能體的當前行為不會影響將來的行為，而在非情景環境中，智能體的當前行為會影響今后的行為，也稱為序貫環境。也就是說，智能體在情景環境中執行獨立的任務，而在非情景環境中，所有智能體的行為都是相關的。

（8）單智能體和多智能體環境

單智能體環境中只能有一個智能體，而多智能體中可以存在多個智能體。在執行復雜任務時，常用多智能體環境。在完全不同的環境中，有不同的智能體執行。不同環境下的智能體可以相互通信。多智能體環境大多是隨機的，因為其具有較大程度的不確定性。

5. 強化學習的平臺

OpenAI Gym是一個用于構建、評估和比較強化學習算法的工具包?？梢约嫒軹ensorFlow、Theano、keras等任何框架下所編寫的算法。該工具包簡單易懂，無需對智能體的結構進行假設，并對所有強化學習任務提供了接口。

OpenAI Universe是OpenAI Gym的擴展，提供了在從簡單到實時復雜的各種環境下訓練和評估智能體的功能。可以無限訪問許多游戲環境。利用Universe,任何程序都可以轉換為一個Gym環境，而無需訪問程序內部、源代碼或者API,這是因為Universe是通過一個計算遠程桌面的虛擬網絡來自動啟動程序的。

DeepMind Lab是基于AI智能體的另一個優秀的研究平臺，提供了豐富的模擬環境，可作為運行多種強化學習算法的實驗平臺。同時它具有高度的可定制化和可拓展性，可視化內容十分豐富，且具有科幻風格和逼真效果。

此外，還有RL-Glue、Project Malmo、VizDoom等平臺，因為沒有嘗試過這些，所以不再具體介紹。

6.強化學習的入門例子

Gym的安裝教程可以自行百度?？蓞⒖歼@個鏈接：三大操作系統中的環境配置方法 安裝好后，通過

from gym import envs print(envs.registry.all())

查看支持環境。
結果如下：

dict_values([EnvSpec(KangarooNoFrameskip-v4), EnvSpec(SkiingDeterministic-v0), EnvSpec(ChopperCommand-ramNoFrameskip-v0), EnvSpec(NameThisGame-ramDeterministic-v4), EnvSpec(Gopher-ramNoFrameskip-v4), EnvSpec(MontezumaRevenge-ramNoFrameskip-v0), EnvSpec(Skiing-ramDeterministic-v0), EnvSpec(Pooyan-ramNoFrameskip-v4), EnvSpec(ChopperCommand-ramDeterministic-v0), EnvSpec(Assault-ramNoFrameskip-v4), EnvSpec(ZaxxonNoFrameskip-v4), EnvSpec(JourneyEscape-ram-v0), EnvSpec(BankHeist-v4), EnvSpec(Gravitar-ramNoFrameskip-v0), EnvSpec(Berzerk-ramNoFrameskip-v0), EnvSpec(Pong-ram-v0), EnvSpec(DemonAttackNoFrameskip-v4), EnvSpec(BankHeist-v0), EnvSpec(AtlantisNoFrameskip-v0), EnvSpec(MsPacmanNoFrameskip-v0), EnvSpec(Bowling-v0), EnvSpec(ElevatorAction-v4), EnvSpec(Breakout-ramDeterministic-v4), EnvSpec(FishingDerbyDeterministic-v4), EnvSpec(BipedalWalkerHardcore-v2), EnvSpec(KungFuMaster-ramDeterministic-v0), EnvSpec(DemonAttackDeterministic-v4), EnvSpec(CentipedeDeterministic-v0), EnvSpec(TimePilot-ramDeterministic-v4), EnvSpec(SeaquestDeterministic-v4), EnvSpec(SkiingNoFrameskip-v0), EnvSpec(MountainCarContinuous-v0), EnvSpec(BattleZone-v4), EnvSpec(Krull-ramDeterministic-v4), EnvSpec(GravitarDeterministic-v4), EnvSpec(SpaceInvaders-ramNoFrameskip-v0), EnvSpec(HandManipulateEggFull-v0), EnvSpec(BowlingDeterministic-v4), EnvSpec(InvertedPendulum-v2), EnvSpec(Skiing-ram-v0), EnvSpec(Tutankham-ramDeterministic-v0), EnvSpec(Gopher-ramDeterministic-v4), EnvSpec(KellyCoinflipGeneralized-v0), EnvSpec(Asterix-ramDeterministic-v0), EnvSpec(BattleZone-ramNoFrameskip-v4), EnvSpec(DemonAttack-ramDeterministic-v4), EnvSpec(AirRaid-ramDeterministic-v4), EnvSpec(Robotank-ramDeterministic-v4), EnvSpec(Gravitar-v0), EnvSpec(VentureDeterministic-v0), EnvSpec(AsterixDeterministic-v0), EnvSpec(DemonAttackDeterministic-v0), EnvSpec(BeamRiderDeterministic-v0), EnvSpec(IceHockey-ramDeterministic-v4), EnvSpec(HandManipulateEggDense-v0), EnvSpec(Boxing-ramDeterministic-v0), EnvSpec(HandManipulatePenFullDense-v0), EnvSpec(KungFuMasterDeterministic-v4), EnvSpec(ElevatorAction-ram-v4), EnvSpec(SolarisDeterministic-v0), EnvSpec(JamesbondNoFrameskip-v0), EnvSpec(FrostbiteDeterministic-v0), EnvSpec(AmidarDeterministic-v4), EnvSpec(Enduro-ramDeterministic-v0), EnvSpec(CarnivalNoFrameskip-v4), EnvSpec(Tutankham-v4), EnvSpec(HeroNoFrameskip-v0), EnvSpec(MountainCar-v0), EnvSpec(YarsRevenge-ram-v4), EnvSpec(Centipede-ramNoFrameskip-v0), EnvSpec(VideoPinball-ramDeterministic-v4), EnvSpec(BattleZone-v0), EnvSpec(BankHeistNoFrameskip-v0), EnvSpec(MsPacman-ramNoFrameskip-v4), EnvSpec(Centipede-ramDeterministic-v0), EnvSpec(NameThisGameDeterministic-v0), EnvSpec(Swimmer-v2), EnvSpec(Skiing-ram-v4), EnvSpec(JourneyEscape-ramNoFrameskip-v4), EnvSpec(Assault-v4), EnvSpec(MsPacman-v0), EnvSpec(Frostbite-ramNoFrameskip-v4), EnvSpec(Breakout-ram-v4), EnvSpec(FrostbiteDeterministic-v4), EnvSpec(Qbert-ramNoFrameskip-v0), EnvSpec(CliffWalking-v0), EnvSpec(TimePilotNoFrameskip-v0), EnvSpec(BeamRiderNoFrameskip-v0), EnvSpec(Phoenix-ramDeterministic-v4), EnvSpec(IceHockey-ramNoFrameskip-v0), EnvSpec(IceHockeyDeterministic-v0), EnvSpec(Kangaroo-ram-v0), EnvSpec(HandManipulatePen-v0), EnvSpec(VideoPinball-v0), EnvSpec(ElevatorAction-ramNoFrameskip-v4), EnvSpec(KellyCoinflip-v0), EnvSpec(Centipede-v4), EnvSpec(CrazyClimber-ram-v4), EnvSpec(IceHockey-ramNoFrameskip-v4), EnvSpec(HandManipulateBlockRotateParallelDense-v0), EnvSpec(NameThisGameNoFrameskip-v0), EnvSpec(VideoPinballDeterministic-v0), EnvSpec(Krull-ram-v0), EnvSpec(Pong-ramDeterministic-v0), EnvSpec(IceHockeyNoFrameskip-v4), EnvSpec(TimePilot-ram-v0), EnvSpec(Enduro-ramDeterministic-v4), EnvSpec(CentipedeDeterministic-v4), EnvSpec(KangarooDeterministic-v4), EnvSpec(CrazyClimber-v4), EnvSpec(Bowling-ramNoFrameskip-v0), EnvSpec(Pendulum-v0), EnvSpec(BankHeistDeterministic-v0), EnvSpec(Phoenix-ramNoFrameskip-v4), EnvSpec(Alien-v0), EnvSpec(ChopperCommand-ram-v4), EnvSpec(FrostbiteNoFrameskip-v0), EnvSpec(PongDeterministic-v4), EnvSpec(Amidar-v4), EnvSpec(DoubleDunk-ramNoFrameskip-v4), EnvSpec(BankHeistDeterministic-v4), EnvSpec(Solaris-ramDeterministic-v0), EnvSpec(TimePilot-ramDeterministic-v0), EnvSpec(Freeway-ramNoFrameskip-v0), EnvSpec(Alien-ramNoFrameskip-v0), EnvSpec(CubeCrash-v0), EnvSpec(BankHeistNoFrameskip-v4), EnvSpec(BeamRider-v0), EnvSpec(CubeCrashSparse-v0), EnvSpec(PrivateEyeDeterministic-v0), EnvSpec(Pooyan-ramNoFrameskip-v0), EnvSpec(Bowling-ram-v4), EnvSpec(FetchPushDense-v1), EnvSpec(Pong-v0), EnvSpec(AirRaidNoFrameskip-v4), EnvSpec(DemonAttack-ramDeterministic-v0), EnvSpec(Riverraid-ram-v4), EnvSpec(ZaxxonDeterministic-v0), EnvSpec(FetchSlideDense-v1), EnvSpec(Skiing-ramNoFrameskip-v4), EnvSpec(Gopher-v0), EnvSpec(SpaceInvaders-ramDeterministic-v4), EnvSpec(Gravitar-v4), EnvSpec(BattleZoneDeterministic-v0), EnvSpec(SolarisDeterministic-v4), EnvSpec(BattleZone-ramDeterministic-v0), EnvSpec(Breakout-v0), EnvSpec(AlienDeterministic-v4), EnvSpec(Centipede-v0), EnvSpec(Freeway-ramNoFrameskip-v4), EnvSpec(Seaquest-ramDeterministic-v0), EnvSpec(AlienNoFrameskip-v0), EnvSpec(MontezumaRevenge-v0), EnvSpec(Robotank-v4), EnvSpec(VideoPinball-v4), EnvSpec(Hero-ramNoFrameskip-v4), EnvSpec(AsteroidsDeterministic-v0), EnvSpec(AirRaid-v0), EnvSpec(RoadRunner-v4), EnvSpec(Hero-ramDeterministic-v4), EnvSpec(SpaceInvaders-ramDeterministic-v0), EnvSpec(BeamRider-ramNoFrameskip-v4), EnvSpec(Gravitar-ramDeterministic-v4), EnvSpec(InvertedDoublePendulum-v2), EnvSpec(MontezumaRevenge-ram-v4), EnvSpec(Breakout-v4), EnvSpec(HandManipulateBlock-v0), EnvSpec(TennisDeterministic-v0), EnvSpec(WizardOfWorNoFrameskip-v4), EnvSpec(Berzerk-ramDeterministic-v0), EnvSpec(YarsRevengeNoFrameskip-v0), EnvSpec(FreewayDeterministic-v0), EnvSpec(MontezumaRevenge-ramNoFrameskip-v4), EnvSpec(Zaxxon-ramDeterministic-v0), EnvSpec(Hopper-v2), EnvSpec(TennisNoFrameskip-v4), EnvSpec(UpNDown-ram-v0), EnvSpec(PrivateEye-ramNoFrameskip-v4), EnvSpec(AssaultDeterministic-v0), EnvSpec(RobotankNoFrameskip-v0), EnvSpec(BoxingDeterministic-v0), EnvSpec(BattleZoneNoFrameskip-v4), EnvSpec(Breakout-ramNoFrameskip-v4), EnvSpec(Blackjack-v0), EnvSpec(BipedalWalker-v2), EnvSpec(ChopperCommandDeterministic-v4), EnvSpec(Robotank-ram-v0), EnvSpec(NameThisGameDeterministic-v4), EnvSpec(StarGunnerDeterministic-v0), EnvSpec(Solaris-ramDeterministic-v4), EnvSpec(UpNDown-ramDeterministic-v4), EnvSpec(RoadRunnerNoFrameskip-v0), EnvSpec(PrivateEye-ram-v0), EnvSpec(Pooyan-v4), EnvSpec(Qbert-v4), EnvSpec(DoubleDunk-ramDeterministic-v0), EnvSpec(MontezumaRevengeDeterministic-v0), EnvSpec(ElevatorAction-ramNoFrameskip-v0), EnvSpec(Copy-v0), EnvSpec(Gravitar-ramNoFrameskip-v4), EnvSpec(Alien-v4), EnvSpec(Walker2d-v2), EnvSpec(EnduroNoFrameskip-v0), EnvSpec(JourneyEscape-ram-v4), EnvSpec(Robotank-ram-v4), EnvSpec(Venture-v4), EnvSpec(BankHeist-ram-v4), EnvSpec(Amidar-v0), EnvSpec(NameThisGame-ram-v0), EnvSpec(AsteroidsDeterministic-v4), EnvSpec(Asteroids-ramDeterministic-v0), EnvSpec(TimePilotDeterministic-v4), EnvSpec(Centipede-ram-v4), EnvSpec(FishingDerby-ram-v4), EnvSpec(HandManipulateBlockRotateParallel-v0), EnvSpec(HeroDeterministic-v4), EnvSpec(AirRaid-ramNoFrameskip-v4), EnvSpec(UpNDown-ramDeterministic-v0), EnvSpec(RobotankNoFrameskip-v4), EnvSpec(FrostbiteNoFrameskip-v4), EnvSpec(WizardOfWor-ramNoFrameskip-v0), EnvSpec(GopherNoFrameskip-v4), EnvSpec(DoubleDunk-v4), EnvSpec(YarsRevengeNoFrameskip-v4), EnvSpec(HalfCheetah-v2), EnvSpec(CrazyClimberNoFrameskip-v4), EnvSpec(AsterixNoFrameskip-v0), EnvSpec(DoubleDunkNoFrameskip-v4), EnvSpec(BoxingNoFrameskip-v0), EnvSpec(NameThisGame-ramNoFrameskip-v4), EnvSpec(GravitarDeterministic-v0), EnvSpec(BattleZone-ramDeterministic-v4), EnvSpec(Pong-ramDeterministic-v4), EnvSpec(FishingDerby-ramDeterministic-v0), EnvSpec(PitfallDeterministic-v4), EnvSpec(Berzerk-ramNoFrameskip-v4), EnvSpec(RoadRunner-ramDeterministic-v0), EnvSpec(Atlantis-ramNoFrameskip-v0), EnvSpec(SpaceInvaders-ramNoFrameskip-v4), EnvSpec(PooyanNoFrameskip-v0), EnvSpec(Berzerk-ram-v4), EnvSpec(VideoPinball-ramDeterministic-v0), EnvSpec(UpNDown-ramNoFrameskip-v0), EnvSpec(Asteroids-ramNoFrameskip-v4), EnvSpec(PhoenixDeterministic-v0), EnvSpec(CrazyClimber-ramDeterministic-v0), EnvSpec(Jamesbond-ramDeterministic-v4), EnvSpec(CarnivalDeterministic-v0), EnvSpec(PrivateEyeDeterministic-v4), EnvSpec(Enduro-ram-v4), EnvSpec(Phoenix-ramNoFrameskip-v0), EnvSpec(NameThisGame-v0), EnvSpec(ElevatorActionNoFrameskip-v0), EnvSpec(MontezumaRevenge-v4), EnvSpec(Pong-v4), EnvSpec(Tennis-ramDeterministic-v4), EnvSpec(Krull-v4), EnvSpec(AmidarDeterministic-v0), EnvSpec(Pitfall-v0), EnvSpec(Gopher-ramDeterministic-v0), EnvSpec(RoadRunner-ram-v4), EnvSpec(Alien-ram-v4), EnvSpec(CentipedeNoFrameskip-v0), EnvSpec(Riverraid-ram-v0), EnvSpec(DoubleDunk-ram-v4), EnvSpec(Phoenix-v4), EnvSpec(Pitfall-ramNoFrameskip-v0), EnvSpec(Asterix-ramNoFrameskip-v4), EnvSpec(PhoenixNoFrameskip-v0), EnvSpec(Jamesbond-ramNoFrameskip-v4), EnvSpec(Berzerk-ramDeterministic-v4), EnvSpec(YarsRevenge-ramNoFrameskip-v4), EnvSpec(Riverraid-ramDeterministic-v4), EnvSpec(SolarisNoFrameskip-v0), EnvSpec(Kangaroo-v0), EnvSpec(IceHockeyDeterministic-v4), EnvSpec(Thrower-v2), EnvSpec(WizardOfWorDeterministic-v4), EnvSpec(Freeway-v4), EnvSpec(Zaxxon-ramNoFrameskip-v4), EnvSpec(Assault-ramDeterministic-v0), EnvSpec(Bowling-v4), EnvSpec(UpNDownDeterministic-v4), EnvSpec(Frostbite-v4), EnvSpec(TutankhamDeterministic-v0), EnvSpec(MsPacman-ramDeterministic-v0), EnvSpec(Centipede-ram-v0), EnvSpec(CartPole-v0), EnvSpec(VentureNoFrameskip-v4), EnvSpec(TennisDeterministic-v4), EnvSpec(Reverse-v0), EnvSpec(HandManipulateBlockFull-v0), EnvSpec(HumanoidStandup-v2), EnvSpec(Frostbite-ramDeterministic-v0), EnvSpec(DemonAttack-ramNoFrameskip-v0), EnvSpec(IceHockeyNoFrameskip-v0), EnvSpec(Gravitar-ram-v0), EnvSpec(RobotankDeterministic-v4), EnvSpec(RoadRunner-ramNoFrameskip-v0), EnvSpec(Pooyan-ramDeterministic-v0), EnvSpec(AtlantisNoFrameskip-v4), EnvSpec(Alien-ramDeterministic-v4), EnvSpec(Seaquest-ramNoFrameskip-v0), EnvSpec(HeroDeterministic-v0), EnvSpec(Robotank-ramNoFrameskip-v4), EnvSpec(Tutankham-ramNoFrameskip-v0), EnvSpec(MsPacman-ramNoFrameskip-v0), EnvSpec(PitfallNoFrameskip-v4), EnvSpec(Pong-ram-v4), EnvSpec(DoubleDunk-v0), EnvSpec(Pooyan-ram-v0), EnvSpec(Tennis-v0), EnvSpec(Solaris-v4), EnvSpec(KungFuMasterNoFrameskip-v4), EnvSpec(Carnival-v0), EnvSpec(BankHeist-ramDeterministic-v0), EnvSpec(HandManipulatePenDense-v0), EnvSpec(Seaquest-v4), EnvSpec(AirRaidDeterministic-v4), EnvSpec(VideoPinball-ram-v4), EnvSpec(SkiingDeterministic-v4), EnvSpec(CrazyClimber-v0), EnvSpec(BreakoutNoFrameskip-v4), EnvSpec(HandReachDense-v0), EnvSpec(Qbert-ramDeterministic-v0), EnvSpec(Hero-ramDeterministic-v0), EnvSpec(SpaceInvaders-v0), EnvSpec(FishingDerby-ramNoFrameskip-v0), EnvSpec(CrazyClimberDeterministic-v0), EnvSpec(AirRaid-ramDeterministic-v0), EnvSpec(YarsRevenge-v0), EnvSpec(MontezumaRevengeNoFrameskip-v4), EnvSpec(Freeway-ramDeterministic-v0), EnvSpec(IceHockey-v4), EnvSpec(KrullDeterministic-v4), EnvSpec(PrivateEye-ramNoFrameskip-v0), EnvSpec(JourneyEscape-ramDeterministic-v0), EnvSpec(KungFuMaster-ramNoFrameskip-v4), EnvSpec(Seaquest-ramNoFrameskip-v4), EnvSpec(FishingDerby-v0), EnvSpec(Gopher-ram-v4), EnvSpec(SeaquestNoFrameskip-v4), EnvSpec(ChopperCommandNoFrameskip-v4), EnvSpec(KungFuMaster-ramNoFrameskip-v0), EnvSpec(Enduro-v0), EnvSpec(Enduro-v4), EnvSpec(PooyanDeterministic-v0), EnvSpec(SpaceInvaders-ram-v0), EnvSpec(Boxing-ramDeterministic-v4), EnvSpec(Hero-ram-v0), EnvSpec(VentureDeterministic-v4), EnvSpec(RiverraidNoFrameskip-v0), EnvSpec(QbertDeterministic-v4), EnvSpec(BattleZone-ramNoFrameskip-v0), EnvSpec(Riverraid-v0), EnvSpec(Frostbite-ramDeterministic-v4), EnvSpec(NameThisGame-ramDeterministic-v0), EnvSpec(Solaris-ram-v4), EnvSpec(UpNDown-v4), EnvSpec(HandManipulateBlockRotateXYZDense-v0), EnvSpec(Assault-ram-v4), EnvSpec(Solaris-ramNoFrameskip-v0), EnvSpec(Boxing-v0), EnvSpec(Phoenix-v0), EnvSpec(Seaquest-ram-v0), EnvSpec(SpaceInvadersDeterministic-v4), EnvSpec(FetchSlide-v1), EnvSpec(Skiing-v0), EnvSpec(GravitarNoFrameskip-v0), EnvSpec(FetchPickAndPlace-v1), EnvSpec(StarGunner-ramNoFrameskip-v0), EnvSpec(PooyanDeterministic-v4), EnvSpec(Tutankham-ramDeterministic-v4), EnvSpec(MontezumaRevenge-ram-v0), EnvSpec(Skiing-v4), EnvSpec(KungFuMasterDeterministic-v0), EnvSpec(Hero-v0), EnvSpec(StarGunner-ramDeterministic-v0), EnvSpec(StarGunner-ram-v0), EnvSpec(EnduroDeterministic-v0), EnvSpec(SpaceInvaders-ram-v4), EnvSpec(Amidar-ramNoFrameskip-v0), EnvSpec(Boxing-v4), EnvSpec(AirRaid-ram-v0), EnvSpec(TimePilot-ram-v4), EnvSpec(WizardOfWor-ramNoFrameskip-v4), EnvSpec(RiverraidDeterministic-v0), EnvSpec(MemorizeDigits-v0), EnvSpec(RoadRunnerNoFrameskip-v4), EnvSpec(AlienNoFrameskip-v4), EnvSpec(UpNDown-ram-v4), EnvSpec(BerzerkDeterministic-v0), EnvSpec(Phoenix-ramDeterministic-v0), EnvSpec(Kangaroo-ramDeterministic-v4), EnvSpec(Humanoid-v2), EnvSpec(BankHeist-ramNoFrameskip-v4), EnvSpec(Qbert-ramNoFrameskip-v4), EnvSpec(BattleZoneNoFrameskip-v0), EnvSpec(DoubleDunk-ramNoFrameskip-v0), EnvSpec(WizardOfWor-v4), EnvSpec(BreakoutNoFrameskip-v0), EnvSpec(KungFuMaster-v0), EnvSpec(BeamRiderDeterministic-v4), EnvSpec(Hero-ramNoFrameskip-v0), EnvSpec(HandManipulateBlockRotateXYZ-v0), EnvSpec(Zaxxon-ram-v0), EnvSpec(RoadRunnerDeterministic-v0), EnvSpec(FetchPickAndPlaceDense-v1), EnvSpec(BreakoutDeterministic-v0), EnvSpec(Jamesbond-v0), EnvSpec(HandManipulateEgg-v0), EnvSpec(BankHeist-ramNoFrameskip-v0), EnvSpec(Bowling-ramDeterministic-v0), EnvSpec(Tennis-ramNoFrameskip-v0), EnvSpec(Kangaroo-ram-v4), EnvSpec(HotterColder-v0), EnvSpec(Atlantis-v4), EnvSpec(HandManipulateEggRotateDense-v0), EnvSpec(BeamRider-ramDeterministic-v4), EnvSpec(Ant-v2), EnvSpec(RoadRunner-ram-v0), EnvSpec(PhoenixNoFrameskip-v4), EnvSpec(Assault-ram-v0), EnvSpec(BankHeist-ramDeterministic-v4), EnvSpec(CrazyClimberDeterministic-v4), EnvSpec(Asteroids-ram-v0), EnvSpec(Krull-v0), EnvSpec(Amidar-ram-v4), EnvSpec(Bowling-ramDeterministic-v4), EnvSpec(FishingDerbyNoFrameskip-v4), EnvSpec(YarsRevenge-ram-v0), EnvSpec(RiverraidDeterministic-v4), EnvSpec(PongNoFrameskip-v0), EnvSpec(Hero-v4), EnvSpec(Seaquest-ramDeterministic-v4), EnvSpec(KangarooDeterministic-v0), EnvSpec(JamesbondDeterministic-v4), EnvSpec(Venture-ramDeterministic-v0), EnvSpec(HandManipulateBlockRotateZDense-v0), EnvSpec(GopherDeterministic-v4), EnvSpec(StarGunner-ramNoFrameskip-v4), EnvSpec(QbertNoFrameskip-v4), EnvSpec(Kangaroo-v4), EnvSpec(Carnival-ramNoFrameskip-v0), EnvSpec(HandManipulateBlockRotateZ-v0), EnvSpec(EnduroNoFrameskip-v4), EnvSpec(UpNDownNoFrameskip-v4), EnvSpec(FishingDerbyDeterministic-v0), EnvSpec(PrivateEye-ramDeterministic-v0), EnvSpec(UpNDown-ramNoFrameskip-v4), EnvSpec(AirRaidDeterministic-v0), EnvSpec(CrazyClimber-ramNoFrameskip-v4), EnvSpec(Venture-ramNoFrameskip-v4), EnvSpec(AssaultDeterministic-v4), EnvSpec(IceHockey-v0), EnvSpec(HandManipulateBlockDense-v0), EnvSpec(YarsRevenge-ramDeterministic-v0), EnvSpec(Pitfall-ramDeterministic-v4), EnvSpec(LunarLanderContinuous-v2), EnvSpec(Alien-ramNoFrameskip-v4), EnvSpec(ChopperCommand-ramNoFrameskip-v4), EnvSpec(JourneyEscape-ramNoFrameskip-v0), EnvSpec(RepeatCopy-v0), EnvSpec(TimePilot-ramNoFrameskip-v4), EnvSpec(FishingDerbyNoFrameskip-v0), EnvSpec(Asteroids-ram-v4), EnvSpec(AirRaid-ram-v4), EnvSpec(NameThisGameNoFrameskip-v4), EnvSpec(Seaquest-v0), EnvSpec(Freeway-v0), EnvSpec(VideoPinball-ramNoFrameskip-v0), EnvSpec(JamesbondNoFrameskip-v4), EnvSpec(Venture-ramDeterministic-v4), EnvSpec(Krull-ram-v4), EnvSpec(ChopperCommand-ramDeterministic-v4), EnvSpec(BoxingNoFrameskip-v4), EnvSpec(Tutankham-ram-v0), EnvSpec(Pitfall-ramDeterministic-v0), EnvSpec(YarsRevenge-ramNoFrameskip-v0), EnvSpec(VideoPinball-ram-v0), EnvSpec(Tennis-ramDeterministic-v0), EnvSpec(HandManipulateBlockFullDense-v0), EnvSpec(CarnivalNoFrameskip-v0), EnvSpec(Carnival-ramNoFrameskip-v4), EnvSpec(Frostbite-ramNoFrameskip-v0), EnvSpec(Carnival-ramDeterministic-v4), EnvSpec(Qbert-ramDeterministic-v4), EnvSpec(AsteroidsNoFrameskip-v0), EnvSpec(HandManipulatePenFull-v0), EnvSpec(DoubleDunkDeterministic-v4), EnvSpec(FreewayDeterministic-v4), EnvSpec(Qbert-ram-v0), EnvSpec(FetchReach-v1), EnvSpec(CarRacing-v0), EnvSpec(NameThisGame-ramNoFrameskip-v0), EnvSpec(GravitarNoFrameskip-v4), EnvSpec(Boxing-ramNoFrameskip-v0), EnvSpec(ZaxxonDeterministic-v4), EnvSpec(Zaxxon-ramDeterministic-v4), EnvSpec(KungFuMaster-v4), EnvSpec(BerzerkDeterministic-v4), EnvSpec(TimePilot-v4), EnvSpec(SeaquestDeterministic-v0), EnvSpec(AtlantisDeterministic-v0), EnvSpec(IceHockey-ram-v4), EnvSpec(PongNoFrameskip-v4), EnvSpec(YarsRevengeDeterministic-v4), EnvSpec(BreakoutDeterministic-v4), EnvSpec(JourneyEscapeNoFrameskip-v0), EnvSpec(UpNDown-v0), EnvSpec(Striker-v2), EnvSpec(Venture-v0), EnvSpec(Riverraid-v4), EnvSpec(ElevatorActionDeterministic-v0), EnvSpec(Frostbite-v0), EnvSpec(HandReach-v0), EnvSpec(Asterix-ramNoFrameskip-v0), EnvSpec(BattleZoneDeterministic-v4), EnvSpec(MsPacman-v4), EnvSpec(RobotankDeterministic-v0), EnvSpec(TimePilotDeterministic-v0), EnvSpec(JamesbondDeterministic-v0), EnvSpec(DemonAttack-ram-v0), EnvSpec(BeamRider-ram-v4), EnvSpec(Assault-ramDeterministic-v4), EnvSpec(PrivateEye-ram-v4), EnvSpec(SpaceInvaders-v4), EnvSpec(PitfallDeterministic-v0), EnvSpec(Asterix-ram-v0), EnvSpec(RoadRunner-ramDeterministic-v4), EnvSpec(VideoPinballNoFrameskip-v4), EnvSpec(Riverraid-ramDeterministic-v0), EnvSpec(AssaultNoFrameskip-v0), EnvSpec(CartPole-v1), EnvSpec(PrivateEye-ramDeterministic-v4), EnvSpec(AirRaid-ramNoFrameskip-v0), EnvSpec(Pooyan-ramDeterministic-v4), EnvSpec(Krull-ramDeterministic-v0), EnvSpec(Kangaroo-ramNoFrameskip-v4), EnvSpec(WizardOfWor-ramDeterministic-v4), EnvSpec(StarGunner-ram-v4), EnvSpec(Pong-ramNoFrameskip-v4), EnvSpec(Gopher-ram-v0), EnvSpec(Frostbite-ram-v4), EnvSpec(Jamesbond-ramNoFrameskip-v0), EnvSpec(CarnivalDeterministic-v4), EnvSpec(GuessingGame-v0), EnvSpec(PongDeterministic-v0), EnvSpec(Breakout-ramNoFrameskip-v0), EnvSpec(HandManipulatePenRotateDense-v0), EnvSpec(Tennis-ram-v4), EnvSpec(AirRaid-v4), EnvSpec(Assault-v0), EnvSpec(Berzerk-v0), EnvSpec(Riverraid-ramNoFrameskip-v4), EnvSpec(MsPacmanDeterministic-v4), EnvSpec(MsPacman-ram-v0), EnvSpec(AssaultNoFrameskip-v4), EnvSpec(KungFuMaster-ram-v0), EnvSpec(ElevatorAction-ramDeterministic-v4), EnvSpec(BeamRider-v4), EnvSpec(RoadRunner-ramNoFrameskip-v4), EnvSpec(TimePilot-ramNoFrameskip-v0), EnvSpec(SpaceInvadersNoFrameskip-v4), EnvSpec(Atlantis-ramDeterministic-v0), EnvSpec(MontezumaRevengeNoFrameskip-v0), EnvSpec(Seaquest-ram-v4), EnvSpec(PrivateEye-v4), EnvSpec(ElevatorAction-ramDeterministic-v0), EnvSpec(Tennis-ram-v0), EnvSpec(Pitfall-v4), EnvSpec(CrazyClimber-ramDeterministic-v4), EnvSpec(Enduro-ramNoFrameskip-v0), EnvSpec(Robotank-v0), EnvSpec(KangarooNoFrameskip-v0), EnvSpec(WizardOfWor-v0), EnvSpec(BerzerkNoFrameskip-v0), EnvSpec(ChopperCommandDeterministic-v0), EnvSpec(WizardOfWor-ram-v4), EnvSpec(HandManipulateEggRotate-v0), EnvSpec(GopherDeterministic-v0), EnvSpec(DemonAttackNoFrameskip-v0), EnvSpec(PrivateEyeNoFrameskip-v0), EnvSpec(Pong-ramNoFrameskip-v0), EnvSpec(Freeway-ramDeterministic-v4), EnvSpec(MsPacman-ramDeterministic-v4), EnvSpec(AsterixDeterministic-v4), EnvSpec(TutankhamDeterministic-v4), EnvSpec(Jamesbond-ramDeterministic-v0), EnvSpec(Skiing-ramNoFrameskip-v0), EnvSpec(Solaris-ram-v0), EnvSpec(Breakout-ram-v0), EnvSpec(SeaquestNoFrameskip-v0), EnvSpec(Jamesbond-ram-v4), EnvSpec(RoadRunner-v0), EnvSpec(VideoPinball-ramNoFrameskip-v4), EnvSpec(Bowling-ram-v0), EnvSpec(Venture-ram-v4), EnvSpec(FishingDerby-ram-v0), EnvSpec(YarsRevenge-v4), EnvSpec(Atlantis-ram-v0), EnvSpec(Frostbite-ram-v0), EnvSpec(Pooyan-v0), EnvSpec(AsterixNoFrameskip-v4), EnvSpec(Pitfall-ram-v4), EnvSpec(Roulette-v0), EnvSpec(Tutankham-ramNoFrameskip-v4), EnvSpec(Centipede-ramDeterministic-v4), EnvSpec(ChopperCommandNoFrameskip-v0), EnvSpec(Freeway-ram-v4), EnvSpec(Tennis-v4), EnvSpec(NameThisGame-v4), EnvSpec(BowlingNoFrameskip-v0), EnvSpec(HandManipulatePenRotate-v0), EnvSpec(KungFuMasterNoFrameskip-v0), EnvSpec(Solaris-v0), EnvSpec(Enduro-ramNoFrameskip-v4), EnvSpec(NameThisGame-ram-v4), EnvSpec(StarGunnerDeterministic-v4), EnvSpec(AmidarNoFrameskip-v4), EnvSpec(Gravitar-ram-v4), EnvSpec(CrazyClimberNoFrameskip-v0), EnvSpec(Riverraid-ramNoFrameskip-v0), EnvSpec(DemonAttack-v0), EnvSpec(MontezumaRevengeDeterministic-v4), EnvSpec(Krull-ramNoFrameskip-v0), EnvSpec(Amidar-ramDeterministic-v0), EnvSpec(Gopher-v4), EnvSpec(Amidar-ramDeterministic-v4), EnvSpec(TutankhamNoFrameskip-v0), EnvSpec(VentureNoFrameskip-v0), EnvSpec(StarGunner-v0), EnvSpec(Breakout-ramDeterministic-v0), EnvSpec(BeamRider-ram-v0), EnvSpec(FetchPush-v1), EnvSpec(ElevatorAction-v0), EnvSpec(ElevatorActionNoFrameskip-v4), EnvSpec(UpNDownNoFrameskip-v0), EnvSpec(FishingDerby-ramNoFrameskip-v4), EnvSpec(Alien-ramDeterministic-v0), EnvSpec(StarGunnerNoFrameskip-v4), EnvSpec(YarsRevenge-ramDeterministic-v4), EnvSpec(PrivateEyeNoFrameskip-v4), EnvSpec(BattleZone-ram-v4), EnvSpec(CentipedeNoFrameskip-v4), EnvSpec(DoubleDunkDeterministic-v0), EnvSpec(DoubleDunk-ramDeterministic-v4), EnvSpec(Zaxxon-ram-v4), EnvSpec(WizardOfWor-ram-v0), EnvSpec(AlienDeterministic-v0), EnvSpec(AtlantisDeterministic-v4), EnvSpec(KrullNoFrameskip-v0), EnvSpec(Phoenix-ram-v4), EnvSpec(TimePilotNoFrameskip-v4), EnvSpec(MsPacmanDeterministic-v0), EnvSpec(Carnival-ram-v4), EnvSpec(Asterix-v0), EnvSpec(TimePilot-v0), EnvSpec(EnduroDeterministic-v4), EnvSpec(Pitfall-ram-v0), EnvSpec(BeamRider-ramNoFrameskip-v0), EnvSpec(CrazyClimber-ram-v0), EnvSpec(Atlantis-ram-v4), EnvSpec(QbertNoFrameskip-v0), EnvSpec(MsPacmanNoFrameskip-v4), EnvSpec(SkiingNoFrameskip-v4), EnvSpec(VideoPinballNoFrameskip-v0), EnvSpec(FrozenLake8x8-v0), EnvSpec(Boxing-ramNoFrameskip-v4), EnvSpec(MontezumaRevenge-ramDeterministic-v0), EnvSpec(Zaxxon-v0), EnvSpec(ChopperCommand-v0), EnvSpec(DuplicatedInput-v0), EnvSpec(StarGunnerNoFrameskip-v0), EnvSpec(JourneyEscapeDeterministic-v0), EnvSpec(Berzerk-v4), EnvSpec(Skiing-ramDeterministic-v4), EnvSpec(ChopperCommand-v4), EnvSpec(RoadRunnerDeterministic-v4), EnvSpec(BeamRiderNoFrameskip-v4), EnvSpec(ZaxxonNoFrameskip-v0), EnvSpec(Pooyan-ram-v4), EnvSpec(Berzerk-ram-v0), EnvSpec(Zaxxon-v4), EnvSpec(KrullNoFrameskip-v4), EnvSpec(Asteroids-ramDeterministic-v4), EnvSpec(IceHockey-ram-v0), EnvSpec(Tutankham-v0), EnvSpec(PrivateEye-v0), EnvSpec(Venture-ram-v0), EnvSpec(Enduro-ram-v0), EnvSpec(KungFuMaster-ramDeterministic-v4), EnvSpec(Taxi-v2), EnvSpec(JourneyEscapeDeterministic-v4), EnvSpec(FreewayNoFrameskip-v0), EnvSpec(MontezumaRevenge-ramDeterministic-v4), EnvSpec(StarGunner-ramDeterministic-v4), EnvSpec(KungFuMaster-ram-v4), EnvSpec(CrazyClimber-ramNoFrameskip-v0), EnvSpec(Alien-ram-v0), EnvSpec(BowlingNoFrameskip-v4), EnvSpec(Kangaroo-ramDeterministic-v0), EnvSpec(BeamRider-ramDeterministic-v0), EnvSpec(JourneyEscape-v0), EnvSpec(FetchReachDense-v1), EnvSpec(ElevatorAction-ram-v0), EnvSpec(Zaxxon-ramNoFrameskip-v0), EnvSpec(ReversedAddition3-v0), EnvSpec(KrullDeterministic-v0), EnvSpec(AsteroidsNoFrameskip-v4), EnvSpec(StarGunner-v4), EnvSpec(Boxing-ram-v0), EnvSpec(JourneyEscape-ramDeterministic-v4), EnvSpec(VideoPinballDeterministic-v4), EnvSpec(Pusher-v2), EnvSpec(PooyanNoFrameskip-v4), EnvSpec(Centipede-ramNoFrameskip-v4), EnvSpec(Amidar-ram-v0), EnvSpec(TutankhamNoFrameskip-v4), EnvSpec(QbertDeterministic-v0), EnvSpec(WizardOfWorDeterministic-v0), EnvSpec(Atlantis-v0), EnvSpec(YarsRevengeDeterministic-v0), EnvSpec(Asteroids-v0), EnvSpec(Jamesbond-ram-v0), EnvSpec(Atlantis-ramNoFrameskip-v4), EnvSpec(FrozenLake-v0), EnvSpec(IceHockey-ramDeterministic-v0), EnvSpec(Tennis-ramNoFrameskip-v4), EnvSpec(SpaceInvadersDeterministic-v0), EnvSpec(MsPacman-ram-v4), EnvSpec(PitfallNoFrameskip-v0), EnvSpec(PhoenixDeterministic-v4), EnvSpec(Krull-ramNoFrameskip-v4), EnvSpec(Reacher-v2), EnvSpec(LunarLander-v2), EnvSpec(Asteroids-v4), EnvSpec(ElevatorActionDeterministic-v4), EnvSpec(Gopher-ramNoFrameskip-v0), EnvSpec(HeroNoFrameskip-v4), EnvSpec(ReversedAddition-v0), EnvSpec(FishingDerby-v4), EnvSpec(BoxingDeterministic-v4), EnvSpec(WizardOfWorNoFrameskip-v0), EnvSpec(Asteroids-ramNoFrameskip-v0), EnvSpec(Pitfall-ramNoFrameskip-v4), EnvSpec(Qbert-v0), EnvSpec(Bowling-ramNoFrameskip-v4), EnvSpec(JourneyEscapeNoFrameskip-v4), EnvSpec(Asterix-ramDeterministic-v4), EnvSpec(Assault-ramNoFrameskip-v0), EnvSpec(Robotank-ramDeterministic-v0), EnvSpec(AirRaidNoFrameskip-v0), EnvSpec(Atlantis-ramDeterministic-v4), EnvSpec(Phoenix-ram-v0), EnvSpec(HandManipulateEggFullDense-v0), EnvSpec(BowlingDeterministic-v0), EnvSpec(Carnival-ramDeterministic-v0), EnvSpec(SolarisNoFrameskip-v4), EnvSpec(Carnival-v4), EnvSpec(Jamesbond-v4), EnvSpec(BerzerkNoFrameskip-v4), EnvSpec(Robotank-ramNoFrameskip-v0), EnvSpec(SpaceInvadersNoFrameskip-v0), EnvSpec(Amidar-ramNoFrameskip-v4), EnvSpec(ChopperCommand-ram-v0), EnvSpec(Boxing-ram-v4), EnvSpec(Tutankham-ram-v4), EnvSpec(Gravitar-ramDeterministic-v0), EnvSpec(NChain-v0), EnvSpec(Venture-ramNoFrameskip-v0), EnvSpec(BankHeist-ram-v0), EnvSpec(DoubleDunkNoFrameskip-v0), EnvSpec(TennisNoFrameskip-v0), EnvSpec(DemonAttack-ram-v4), EnvSpec(DoubleDunk-ram-v0), EnvSpec(Kangaroo-ramNoFrameskip-v0), EnvSpec(JourneyEscape-v4), EnvSpec(CubeCrashScreenBecomesBlack-v0), EnvSpec(AmidarNoFrameskip-v0), EnvSpec(Solaris-ramNoFrameskip-v4), EnvSpec(GopherNoFrameskip-v0), EnvSpec(Hero-ram-v4), EnvSpec(WizardOfWor-ramDeterministic-v0), EnvSpec(UpNDownDeterministic-v0), EnvSpec(Qbert-ram-v4), EnvSpec(FishingDerby-ramDeterministic-v4), EnvSpec(DemonAttack-v4), EnvSpec(DemonAttack-ramNoFrameskip-v4), EnvSpec(Acrobot-v1), EnvSpec(BattleZone-ram-v0), EnvSpec(RiverraidNoFrameskip-v4), EnvSpec(FreewayNoFrameskip-v4), EnvSpec(Asterix-v4), EnvSpec(Asterix-ram-v4), EnvSpec(Carnival-ram-v0), EnvSpec(Freeway-ram-v0)])

小程序一：平衡擺游戲

cart pole即車桿游戲，游戲如下，很簡單，游戲里面有一個小車，上有豎著一根桿子。小車需要左右移動來保持桿子豎直。如果桿子傾斜的角度大于15°，那么游戲結束。小車也不能移動出一個范圍（中間到兩邊各2.4個單位長度）。
在gym的Cart Pole環境（env）里面，左移或者右移小車的action之后，env都會返回一個+1的reward。到達100個reward之后，游戲也會結束。

import gymenv = gym.make('CartPole-v0')for i_episode in range(100):observation = env.reset()for t in range(100):env.render() # 更新動畫action = env.action_space.sample()observation, reward, done, info = env.step(action) # 推進一步if done:env.reset()continue

運行結果：

總結

以上是生活随笔為你收集整理的强化学习入门系列一VS强化学习的基本概念的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。