{"id":1911,"date":"2019-07-25T05:48:47","date_gmt":"2019-07-24T20:48:47","guid":{"rendered":"http:\/\/141.164.34.82\/?p=1911"},"modified":"2019-08-04T15:24:33","modified_gmt":"2019-08-04T06:24:33","slug":"evolution-strategies-as-a-scalable-alternative-to-reinforcement-learning","status":"publish","type":"post","link":"http:\/\/ds.sumeun.org\/?p=1911","title":{"rendered":"Evolution Strategies as a Scalable Alternative to Reinforcement Learning"},"content":{"rendered":"

\ub4e4\uc5b4\uac00\uae30 \uc804\uc5d0 : \uc774 \uae00\uc740 \uac15\ud654\ud559\uc2b5 \uad00\ub828 \uae00\uc785\ub2c8\ub2e4.<\/strong> \ucc28\ud6c4 \ub2e4\ub978 \uc0ac\uc774\ud2b8\ub85c \uc774\ub3d9\ub420 \uac83\uc785\ub2c8\ub2e4. <\/p>\n

\ud575\uc2ec\uc801 \uc218\uc2dd<\/h2>\n

3\ucabd \uc0c1\ub2e8\uc758 \uc218\uc2dd\uc740 \ub2e4\uc74c\uacfc \uac19\ub2e4.<\/p>\n

\\[ \\nabla_\\theta \\mathbb{E}_{\\epsilon \\sim \\mathcal{N}(\\vec{0}, \\textbf{I})} F(\\theta + \\sigma \\epsilon) = \\frac{1}{\\sigma} \\mathbb{E}_{\\epsilon \\sim \\mathcal{N}(\\vec{0}, \\textbf{I})} \\{ F(\\theta + \\sigma \\epsilon) \\epsilon\\}\\]<\/p>\n

\uc774\ub294 2\ucabd\uc758 \uc77c\ubc18\uc801\uc778 \ub4f1\uc2dd\uc5d0 factored Gaussian\uc744 \uc801\uc6a9\ud55c \uac83\uc774\ub2e4. 2\ucabd\uc758 \uc77c\ubc18\uc801\uc778 \ub4f1\uc2dd\uc740 \ub2e4\uc74c\uacfc \uac19\ub2e4.<\/p>\n

\\[ \\nabla_{\\psi} \\mathbb{E}_{\\theta \\sim p_{\\psi}} F(\\theta) = \\mathbb{E}_{\\theta \\sim p_{\\psi}} \\{ F(\\theta) \\nabla_{\\psi} \\log p_{\\psi} (\\theta) \\}\\]<\/p>\n

\uc774\ub97c \ud558\ub098\ud558\ub098\uc529 \uc124\uba85\ud574\ubcf4\uc790.
\n\\(\\mathbb{E}_{\\theta \\sim p_{\\psi}} F(\\theta)\\) \uc740 \\(\\theta\\) \uac00 \\(\\psi\\) \ub97c \ubaa8\uc218\ub97c \uac00\uc9c0\ub294 \ud655\ub960\ubd84\ud3ec\ub97c \ub530\ub97c \ub54c, \\(F(\\theta)\\) \uc758 \ud3c9\uade0\uc744 \ub098\ud0c0\ub0b8\ub2e4. (\uc608\ub97c \ub4e4\uc5b4 \\(\\theta\\) \uac00 \ud3c9\uade0 0, \ubd84\uc0b0 1\uc778 \uc815\uaddc\ubd84\ud3ec\uc77c\ub54c, \\(p_\\psi(\\theta)\\) \ub294 \uc815\uaddc\ubd84\ud3ec\uc758 \ud655\ub960\ubc00\ub3c4\ud568\uc218\uc774\uace0, \\(\\psi\\) \ub294 \ud3c9\uade0\uacfc \ubd84\uc0b0\uc744 \ub098\ud0c0\ub0b8\ub2e4.) \uadf8\ub9ac\uace0 \\(\\nabla_{\\psi} \\mathbb{E}_{\\theta \\sim p_{\\psi}} F(\\theta)\\) \ub294 \\(F(\\theta)\\) \uc758 \ud3c9\uade0\uc744 \ub098\ud0c0\ub0b8\ub2e4( \\(\\theta\\) \uac00 \\(p_\\psi\\) \uc758 \ud655\ub960\ubc00\ub3c4\ud568\uc218\ub97c \uac00\uc9c0\ub294 \ubd84\ud3ec\uc77c \ub54c!). <\/p>\n

\uc774 \uac12\uc744 \uad6c\ud558\uae30\ub294 \uc27d\uc9c0 \uc54a\uc744 \uac83\uc774\ub2e4. \ud2b9\ud788 \\(F(\\theta)\\) \uc758 \uac12\uc744 \ubaa8\ub97c \ub54c. \ub9cc\uc57d \\(\\mathbb{E}_{\\theta \\sim p_{\\psi}} F(\\theta)\\) \ub97c \uad6c\ud558\uace0\uc790 \ud55c\ub2e4\uba74, \uc0d8\ud50c\ub9c1\uc744 \uc0ac\uc6a9\ud558\uc5ec \uadfc\uc0ac\uac12\uc744 \uc5bb\uc744 \uc218 \uc788\ub2e4. \ud558\uc9c0\ub9cc \\(\\mathbb{E}_{\\theta \\sim p_{\\psi}} F(\\theta)\\) \uc740…<\/p>\n

\ubc18\uba74 \uc6b0\ubcc0\uc744 \ubcf4\uba74 \\(\\{ F(\\theta) \\nabla_{\\psi} \\log p_{\\psi} (\\theta)\\) \uc758 \ud3c9\uade0\uc744 \ub098\ud0c0\ub0b4\uae30 \ub54c\ubb38\uc5d0 \uc0d8\ud50c\ub9c1\uc73c\ub85c \uacc4\uc0b0\uc774 \uac00\ub2a5\ud558\ub2e4. \uadf8\ub9ac\uace0 \\(\\nabla F(\\theta)\\) \ub97c \uad6c\ud558\uc9c0 \uc54a\uc544\ub3c4 \ub41c\ub2e4. \ub300\uc2e0 \\(\\nabla_\\psi p_\\psi(\\theta)\\) \uac00 \ud544\uc694\ud55c\ub370, \\(p_\\psi\\) \ub294 \ud655\ub960\ubc00\ub3c4\ud568\uc218\ub97c \ub098\ud0c0\ub0b4\uae30 \ub54c\ubb38\uc5d0 \ub300\ubd80\ubd84 \uc6b0\ub9ac\uac00 \uc54c\uace0 \uc788\ub294 \ud568\uc218\uc774\ub2e4.<\/p>\n

\uadf8\ub7f0\ub370 3\ucabd \uc0c1\ub2e8\uc758 \uc218\uc2dd\uc740 \ub2e4\uc18c \ub2e4\ub978 \ud615\ud0dc\ub97c \ub744\uace0 \uc788\uae30 \ub54c\ubb38\uc5d0 \ud63c\ub3d9\ud560 \uc218 \uc788\ub2e4. 3\ucabd \uc0c1\ub2e8\uc758 \uc218\uc2dd\uc744 \ub2e4\uc2dc \ubcf4\uc790.<\/p>\n

\\[ \\nabla_\\theta \\mathbb{E}_{\\epsilon \\sim \\mathcal{N}(\\vec{0}, \\textbf{I})} F(\\theta + \\sigma \\epsilon) = \\frac{1}{\\sigma} \\mathbb{E}_{\\epsilon \\sim \\mathcal{N}(\\vec{0}, \\textbf{I})} \\{ F(\\theta + \\sigma \\epsilon) \\epsilon\\}\\]<\/p>\n

\uc5ec\uae30\uc11c \uc77c\ubc18\uc801\uc778 \ub4f1\uc2dd\uc5d0 \\(p_\\psi\\) \ub294 \uc774 \ub17c\ubb38\uc5d0\uc11c\ub294 factored Gaussian( \\(\\mathcal{N}(\\vec{\\mu}, \\textbf{D})\\) )\uc744 \ub098\ud0c0\ub0b8\ub2e4. \ub17c\ubb38\uc5d0\uc11c\ub294 \\(\\textbf{D}\\) \ub97c \ubaa8\ub4e0 \ubc29\ud5a5\uc73c\ub85c \ubd84\uc0b0\uc774 \\(\\sigma\\) \uc774\uace0 \uacf5\ubd84\uc0b0\uc774 \\(0\\) \uc73c\ub85c \uac00\uc815\ud588\ub2e4.<\/p>\n

\ub530\ub77c\uc11c \\(p_\\psi\\) \ub294 \\(p_\\mu=\\mathcal{N}(\\vec{\\mu}, \\sigma \\textbf{I})\\) \uac00 \ub41c\ub2e4. <\/p>\n

\uc774\ub97c \uc77c\ubc18\uc801\uc778 \ub4f1\uc2dd\uc5d0 \uadf8\ub300\ub85c \uc801\uc6a9\ud558\uba74 \ub2e4\uc74c\uacfc \uac19\ub2e4. (\uc5ec\uae30\uc11c \ud3b8\uc758\uc0c1 \\(\\theta\\) \ub294 \\(\\mathcal{N}(\\vec{\\mu}, \\textbf{I})\\) \uc989, \\(\\sigma=1\\) \uc744 \uac00\uc815\ud588\ub2e4.)<\/p>\n

\\[\\nabla_{\\psi} \\mathbb{E}_{\\theta \\sim p_{\\psi}} F(\\theta) = \\mathbb{E}_{\\theta \\sim p_{\\psi}} \\{ F(\\theta) \\nabla_{\\psi} \\log p_{\\psi} (\\theta) \\} \\ \\]
\n\\[\\nabla_{\\vec{\\mu}} \\mathbb{E}_{\\theta \\sim p_{\\mu}} F(\\theta) = \\mathbb{E}_{\\theta \\sim p_{\\mu}} \\{ F(\\theta) \\nabla_{\\mu} \\log p_{\\mu} (\\theta) \\} \\]<\/p>\n

Gaussian \ubd84\ud3ec\ub97c \ub098\ud0c0\ub0b4\ub294 \\(p_{\\mu} = (2\\pi)^{-k\/2} \\exp(-\\frac{1}{2}(\\theta-\\mu)‘(\\theta-\\mu))\\) \uc774\ubbc0\ub85c \\(\\log p_{\\mu} = -(2\\pi)^{-k\/2} \\frac{1}{2}(\\theta-\\mu)’(\\theta-\\mu)\\) \uac00 \ub41c\ub2e4. <\/p>\n

\ub530\ub77c\uc11c \uc704 \uc804\uac1c\uc758 \ub9c8\uc9c0\ub9c9\uc740 \ub2e4\uc74c\uacfc \uac19\ub2e4.<\/p>\n

\\[\\nabla_{\\vec{\\mu}} \\mathbb{E}_{\\theta \\sim p_{\\mu}} F(\\theta) = \\mathbb{E}_{\\theta \\sim p_{\\mu}} \\{ F(\\theta) \\nabla_{\\mu} \\log p_{\\mu} (\\theta) \\} \\]
\n\\[\\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ = \\mathbb{E}_{\\theta \\sim p_{\\mu}} \\{ F(\\theta) \\nabla_{\\mu} [ -(2\\pi)^{-k\/2} \\frac{1}{2}(\\theta-\\mu)‘(\\theta-\\mu) ] \\} \\] <\/p>\n

\\(\\nabla_{\\mu} [ -(2\\pi)^{-k\/2} \\frac{1}{2}(\\theta-\\mu)’(\\theta-\\mu) ]\\) \ub97c \uad6c\ud55c\ub2e4\uba74 \uc0d8\ud50c\ub9c1\uc73c\ub85c \\(\\nabla_{\\vec{\\mu}} \\mathbb{E}_{\\theta \\sim p_{\\mu}} F(\\theta)\\) \ub97c \uad6c\ud560 \uc218 \uc788\uc74c\uc744 \uc54c \uc218 \uc788\ub2e4.<\/p>\n

\uc774\uc81c \ub2e4\uc2dc 3\ucabd \uc0c1\ub2e8\uc758 \uc218\uc2dd\uc744 \ubcf4\uc790.<\/p>\n

\\[ \\nabla_\\theta \\mathbb{E}_{\\epsilon \\sim \\mathcal{N}(\\vec{0}, \\textbf{I})} F(\\theta + \\sigma \\epsilon) = \\frac{1}{\\sigma} \\mathbb{E}_{\\epsilon \\sim \\mathcal{N}(\\vec{0}, \\textbf{I})} \\{ F(\\theta + \\sigma \\epsilon) \\epsilon\\}\\]<\/p>\n

\uc774 \uc2dd\uc5d0\uc11c \ub2e4\uc18c \ub2f9\ud669\uc2a4\ub7ec\uc6e0\ub358 \uac83\uc740 \\(\\nabla_\\theta \\mathbb{E}_{\\epsilon \\sim \\mathcal{N}(\\vec{0}, \\textbf{I})} F(\\theta + \\sigma \\epsilon)\\) \uc5d0\uc11c \uadf8\ub798\ub514\uc5b8\ud2b8\ub97c \\(\\mathbb{E}\\) \uc548\ucabd\uc73c\ub85c \ub123\uc5b4\ubcf4\uba74 \ub2e4\uc74c\uac00 \uac19\uc774 \uc4f8 \uc218 \uc788\ub2e4.<\/p>\n

\\[\\nabla_\\theta \\mathbb{E}_{\\epsilon \\sim \\mathcal{N}(\\vec{0}, \\textbf{I})} F(\\theta + \\sigma \\epsilon) = \\mathbb{E}_{\\epsilon \\sim \\mathcal{N}(\\vec{0}, \\textbf{I})} \\nabla_\\theta F(\\theta + \\sigma \\epsilon) \\]<\/p>\n

\uc5ec\uae30\uc11c \uac11\uc790\uae30 \uadf8\ub798\ub514\uc5b8\ud2b8\uac00 \uc0ac\ub77c\uc84c\uae30 \ub54c\ubb38\uc774\ub2e4. <\/p>\n

\ubb34\uc5c7\uc774 \ubb38\uc81c\uc600\uc744\uae4c\uc694?<\/p>\n

REINFORCE\uc640\uc758 \uad00\uacc4<\/h2>\n

\uc5b4\uca0b\ub4e0 \uc218\uc2dd\uc744 \uc774\ud574\ud588\ub2e4\uba74 REINFORCE\uc640 \uc815\ud655\ud788 \uac19\uc740 \ubc29\uc2dd\uc758 \uc804\uac1c\uc784\uc744 \uc54c \uc218 \uc788\ub2e4. \ub2e8\uc9c0 REINFORCE\ub294 \uc815\ucc45 \uc0c1\uc758 \ud655\ub960\uc5d0 \ub300\ud574\uc11c \ub2e4\uc74c\uc744 \uad6c\ud55c\ub2e4.<\/p>\n

\\[ \\nabla_\\theta \\mathbb{E}_{s_t} \\Big[ \\sum_t \\mathbb{E}_{a \\sim \\pi(\\cdot|s_t, \\theta)} \\Big[ \\gamma^t R(s_t,a) \\Big] \\Big]\\]<\/p>\n

\uc774\ub54c \\(\\nabla_{\\theta}\\) \ub97c \\(\\mathbb{E}_{s_t}\\) \uc548\ucabd\uc73c\ub85c \ubcf4\ub0b4\uba74 \ub2e4\uc74c\uacfc \uac19\ub2e4.<\/p>\n

\\[ \\mathbb{E}_{s_t} \\Big[ \\sum_t \\nabla_\\theta \\mathbb{E}_{a \\sim \\pi(\\cdot|s_t, \\theta)} \\Big[ \\gamma^t R(s_t,a) \\Big] \\Big]\\]<\/p>\n

\\(\\nabla_{\\theta}\\mathbb{E}_{a \\sim \\pi(\\cdot | s_t, \\theta)}\\) \uc5d0\uc11c \uc704\uc5d0\uc11c\uc640 \ub3d9\uc77c\ud55c \uc774\uc720\ub97c \\(\\nabla_\\theta\\) \ub97c \\(\\mathbb{E}\\) \uc548\ucabd\uc73c\ub85c \ub123\uc73c\uba74\uc11c \\(\\nabla_\\theta \\log \\pi(a|s_t, \\theta)\\) \ub85c \uc4f8 \uc218 \uc788\ub2e4.<\/p>\n

ES vs. REINFORCE<\/h2>\n

average return \ud568\uc218\uac00 local minima\uac00 \ub9ce\uc744 \uacbd\uc6b0\uc5d0 REINFORCE\uc640 \uac19\uc740 policy graident \ubc29\ubc95\uc740 local minima\uc5d0 \uac07\ud600 \ubc84\ub9b4 \uc218 \uc788\ub2e4. \ubc18\uba74 ES\ub294 \\(\\theta\\) \ubd84\ud3ec\uc758 \ubd84\uc0b0\uc774 \ud074 \uc218\ub85d \uadf8\ub7f0 \ubb38\uc81c\uc5d0\uc11c \uc790\uc720\ub85c\uc6b8 \uc218 \uc788\ub2e4. \ubb3c\ub860 \ub2f4\uae08\uc9c8 \uae30\ubc95(Simulated Annealing, SA)\uacfc \uac19\uc740 local minima\ub97c \ubc97\uc5b4\ub098\ub294 \ubc29\ubc95\uc774 \uc788\uae30\ub294 \ud558\ub2e4. <\/p>\n

\uadf8\uc678\uc5d0\ub3c4 \ubcf4\uc0c1\uc774 \ubcc4\ub85c \uc5c6\uace0, \uc989\uac01\uc801\uc774\uc9c0 \uc54a\uc740 \uacbd\uc6b0\uc5d0\ub294 ES\uac00 \ub098\uc744 \uac83 \uac19\ub2e4. <\/p>\n

\ub17c\ubb38\uc5d0\uc11c \ucd94\uac00\ub85c \uc81c\uc2dc\ud558\ub294 \uc7a5\uc810\uc740 average return \ud568\uc218\uac00 \ubbf8\ubd84 \uac00\ub2a5\ud558\uc9c0 \uc54a\uc740 \uacbd\uc6b0(\uc608. hard attention), parallel \ud558\uac8c \uc9c4\ud589\ud560 \uacbd\uc6b0 agent \uc0ac\uc774\uc758 \ud1b5\uc2e0 \uc6a9\ub7c9\uc774 \ud06c\uc9c0 \uc54a\ub2e4\ub294 \uc810 \ub4f1\uc774 \uc788\ub2e4. <\/p>\n

\uc758\uc678\uc758 \uc7a5\uc810<\/h2>\n

\uc774 \ub17c\ubb38\uc5d0\uc11c \uc8fc\uc7a5\ud558\ub294 ES\uc758 \uac00\ub2a5\uc131\uc740 \uc758\uc678\ub85c \ub9ce\uc740 \uacc4\uc0b0\ubd80\ud558<\/strong>\ub97c \uc694\uad6c\ud558\uc9c0 \uc54a\ub294\ub2e4\ub294 \uc810\uc774\ub2e4. \uc65c\ub0d0\ud558\uba74 \ubc31\ud504\ub85c\ud37c\uac8c\uc774\uc158\uc744 \ud558\uc9c0 \uc54a\uae30 \ub54c\ubb38\uc5d0 gpu\ub97c \uc4f0\uc9c0 \uc54a\uc544\ub3c4 \ub418\uba70, cpu\ub85c\ub3c4 \ucda9\ubd84\ud558\ub2e4\ub294 \uc810\uc774\ub2e4. \uc0ac\uc2e4 \ub098\ub294 \uc720\uc804\uc790 \uc54c\uace0\ub9ac\uc998\uc774 \uadf8\ub0e5 \uc5c4\uccad\ub098\uac8c \uacc4\uc0b0\uc744 \ud558\ub294 \uac83\uacfc \ub2e4\ub974\uc9c0 \uc54a\ub2e4\uace0 \uc0dd\uac01\ud588\ub294\ub370 \uc758\uc678\uc758 \uc7a5\uc810\uc778 \ub4ef \ud558\ub2e4(\uc9c0\uad6c \ucd5c\ucd08\uc758 \uc0dd\uba85\uccb4\ub294 30\uc5b5\ub144 \uc804\uc774\ub77c\ub294\ub370, \uc9c4\ud654\uc801 \uad00\uc810\uc5d0\uc11c \uace0\ub4f1 \uc0dd\uba85\uccb4\uac00 \ud0c4\uc0dd\ud558\uae30 \uc704\ud574 \ud544\uc694\ud588\ub358 \uac83\uc740 \ubb34\uc218\ud788 \uae34 \uc2dc\uac04\uacfc \uc801\uc808\ud55c \uc9c4\ud654\uc555\ub825\uc774\uc5c8\uc744 \ubfd0?).<\/p>\n

\uadf8\ub807\ub2e4\uba74 \ub124\ud2b8\uc6cc\ud06c\uc758 \ud06c\uae30\uac00 \uc99d\uac00\ud568\uc5d0 \ub530\ub77c ES\uc640 REINFORCE \uc911\uc5d0 \ube44\uad50 \uc6b0\uc704\ub97c \uc810\ud558\ub294 \ucabd\uc740 \uc5b4\ub290 \ucabd\uc77c\uae4c?<\/p>\n

\ucc38\uace0\ubb38\ud5cc<\/h2>\n