{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 中文汉化版说明\n",
    "\n",
    "本 notebook 是 Andrej Karpathy《Neural Networks: Zero to Hero》课程 **第 1 讲（micrograd，上半场）** 配套 notebook 的中文汉化版。\n",
    "\n",
    "- 原仓库：[karpathy/nn-zero-to-hero](https://github.com/karpathy/nn-zero-to-hero)（MIT License，Copyright (c) 2022 Andrej Karpathy）\n",
    "- 原文件：`lectures/micrograd/micrograd_lecture_first_half_roughly.ipynb`\n",
    "- 视频课程：[The spelled-out intro to neural networks and backpropagation: building micrograd](https://www.youtube.com/watch?v=VMj-3S1tku0)\n",
    "- 汉化：极客教育（jikejiao.com），依 MIT 许可翻译改编，完整保留原版权声明（见随附 LICENSE 文件）\n",
    "- 配套中文导学页：<https://jikejiao.com/karpathy/zero-to-hero/lecture-01-micrograd.html>\n",
    "\n",
    "**译者说明**：原 notebook 是课程视频的粗略代码记录（文件名中的 \"roughly\" 即此意），原版不含任何文字说明单元。为方便中文读者对照视频学习，我们在关键节点插入了标有【译者导读】的说明单元；**所有代码单元与原版完全一致，未做任何改动**。建议配合视频与导学页一起食用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 一、先从导数说起：单变量函数\n",
    "【译者导读】反向传播的一切都建立在导数 (derivative) 之上。下面先定义一个普通的标量函数 f(x) = 3x² − 4x + 5，画出它的曲线，然后用「数值法」逼近它在某一点的导数：取一个很小的 h，计算 (f(x+h) − f(x)) / h。这就是导数的定义式，也是我们后面校验梯度是否算对的终极手段。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(x):\n",
    "  return 3*x**2 - 4*x + 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20.0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f(3.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f9b98432ee0>]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAkRklEQVR4nO3dd3yV5f3/8dcnkwQCIRBmwg5hD4mIimhlSAXRWkXpF8E6UGtrbSutddTafq1YW7+ttQ6cqNRRxOLCURwIAhL2StgbQghkEMi+fn9w7I8qSNbJfcb7+XjwOOc+I/f76CNvLu5z3ddtzjlERCS0RHgdQERE6p/KXUQkBKncRURCkMpdRCQEqdxFREJQlNcBAFq2bOk6derkdQwRkaCybNmyg8655JM9FxDl3qlTJzIzM72OISISVMxsx6me02EZEZEQpHIXEQlBKncRkRCkchcRCUEqdxGREHTacjez58zsgJmtPeGxJDP7yMw2+W6bn/Dcr81ss5llm9lF/gouIiKnVp2R+wvA6K89dicwzzmXBszzbWNmvYCrgd6+9zxuZpH1llZERKrltOXunJsPHPraw5cCM3z3ZwCXnfD4q865UufcNmAzMLh+on5TTmEJv39nPYeLy/y1CxGRoFTbY+6tnXP7AHy3rXyPtwd2nfC63b7HvsHMpphZppll5ubm1ipE/tFynl2wjZcXn3Iev4hIWKrvL1TtJI+d9GogzrnpzrkM51xGcvJJz549rfQ2CVyQnsyMRdspKa+s1c8QEQlFtS33HDNrC+C7PeB7fDeQesLrUoC9tY93elOGdeHgkTLeWL7bn7sREQkqtS33t4DJvvuTgTknPH61mcWaWWcgDfiybhG/3dldWtAvpRnPfL6NyipdMlBEBKo3FfIVYBGQbma7zex6YBow0sw2ASN92zjn1gGvA+uB94FbnXN+PV5iZkwZ1oVtB4v5aH2OP3clIhI0TrsqpHNuwimeGn6K1z8APFCXUDU1uncbUpPimD5/C6P7tGnIXYuIBKSQOEM1KjKCG4Z2YfnOfDK3f33WpohI+AmJcge4MiOFxPhonpq/1esoIiKeC5lyj4+JYtKQjvx7Qw5bco94HUdExFMhU+4Ak87pRExkBM98rtG7iIS3kCr3lk1i+f6gFN5YtocDRSVexxER8UxIlTvAjed1obyqihlfbPc6ioiIZ0Ku3Du3bMyoXq15efFOiksrvI4jIuKJkCt3gCnDulJwrJzXlu46/YtFREJQSJb7oI7NObNTc55dsI2Kyiqv44iINLiQLHc4Pnrfk3+Md9fs8zqKiEiDC9lyH96jFV2TGzN9/lac04JiIhJeQrbcIyKOLyi2bm8hn2866HUcEZEGFbLlDnDZwPa0adqIxz7Z7HUUEZEGFdLlHhsVyZRhXfhy2yGWakExEQkjIV3uAFcPTiWpcQx/1+hdRMJIyJd7fEwU1w/tzKfZuazdU+B1HBGRBhHy5Q4wcUhHEmKjePxTjd5FJDyERbk3i4tm0jkdmbt2P5sPFHkdR0TE78Ki3AGuO7czsVERPPGplgMWkdAXNuXeokksEwZ34F8r97Dr0FGv44iI+FXYlDvAlGFdiDCYrkvxiUiIC6tyb9ssju+fkcJrmbs4UKiLeYhI6Aqrcge4+fyuVFRW8cyCbV5HERHxm7Ar904tG3NJ/3a8vHgH+UfLvI4jIuIXYVfuAD+6oBtHyyp5fuF2r6OIiPhFWJZ7epsERvZqzQtfbOeILsUnIiEoLMsd4NbvdKPgWDkzF+/wOoqISL0L23IfkJrI0G4tefrzbRwrq/Q6johIvQrbcge4bXgaB4+UMnOJRu8iElrCutwHd07i3G4tePKzLRq9i0hICetyB7h9RHcOHinjZR17F5EQEvblfmanJIZ2a8mTn23haJlmzohIaKhTuZvZz8xsnZmtNbNXzKyRmSWZ2Udmtsl327y+wvrLz0amkVdcxkuLNHoXkdBQ63I3s/bAbUCGc64PEAlcDdwJzHPOpQHzfNsBbVDHJM5La8lT87dq9C4iIaGuh2WigDgziwLigb3ApcAM3/MzgMvquI8GcfuI7hwqLuNFjd5FJATUutydc3uAPwE7gX1AgXPuQ6C1c26f7zX7gFYne7+ZTTGzTDPLzM3NrW2MejOoY3OGdU9m+vytFOusVREJcnU5LNOc46P0zkA7oLGZTazu+51z051zGc65jOTk5NrGqFc/G5Gm0buIhIS6HJYZAWxzzuU658qB2cA5QI6ZtQXw3R6oe8yGMbBDcy5IT2b6/C1ac0ZEglpdyn0nMMTM4s3MgOHABuAtYLLvNZOBOXWL2LBuH9Gdw0fLmfHFdq+jiIjUWl2OuS8BZgHLgTW+nzUdmAaMNLNNwEjfdtAYkJrId9KTefrzrRSVlHsdR0SkVuo0W8Y5d59zrodzro9z7hrnXKlzLs85N9w5l+a7PVRfYRvK7SO6k3+0XMfeRSRohf0ZqifTPzWR4T1aMX2+Ru8iEpxU7qfw0xFpFBwr5wVdrUlEgpDK/RT6pSQyomdrpn++VddaFZGgo3L/FlMvSudIaQVPfLbF6ygiIjWicv8W6W0SuGxAe15YuJ2cwhKv44iIVJvK/TR+NqI7Vc7x6LxNXkcREak2lftpdGgRz4TBHXht6S62Hyz2Oo6ISLWo3Kvhxxd2Izoygkc+2uh1FBGRalG5V0OrhEZcN7QTb63ay7q9BV7HERE5LZV7NU0Z1pVmcdH86YNsr6OIiJyWyr2amsVFc8sFXfkkO5cvtwXdigoiEmZU7jUw+exOtG4ayx/fz8I553UcEZFTUrnXQFxMJLcNTyNzx2E+yQ6aZepFJAyp3GtofEYqnVrE88f3s6mq0uhdRAKTyr2GoiMj+PmodLL2F/H26r1exxEROSmVey2M7duWXm2b8ucPN1JWUeV1HBGRb1C510JEhDF1dDo7Dx3ltaU7vY4jIvINKvdauqB7MkO6JPGXf2/SBT1EJOCo3GvJzLj74l7kFZfx+KdaElhEAovKvQ76pjTj8oHteXbBNnYfPup1HBGR/1C519EdF6VjwMNalkBEAojKvY7aJcZx43ldmLNyLyt35XsdR0QEULnXi5sv6ErLJrE88O56LUsgIgFB5V4PmsRG8fOR3Vm6/TAfrNvvdRwREZV7fRmfkUJaqyZMm5ulE5tExHMq93oSFRnBXWN6sj3vKC8t3uF1HBEJcyr3enRB92TOS2vJo/M2kX+0zOs4IhLGVO71yMy46+KeFJaU87ePN3sdR0TCmMq9nvVs25Txg1J5cdF2duQVex1HRMKUyt0PfjGqO9GRETz0fpbXUUQkTKnc/aBV00bcNKwr763Zz9Ltut6qiDQ8lbuf3DisM+2aNeK+Oeuo1BWbRKSB1anczSzRzGaZWZaZbTCzs80sycw+MrNNvtvm9RU2mMTHRHHXmJ6s31fIP77Umu8i0rDqOnL/K/C+c64H0B/YANwJzHPOpQHzfNthaUzftpzdpQV//jCbw8WaGikiDafW5W5mTYFhwLMAzrky51w+cCkww/eyGcBldYsYvMyM347rTVFJBX/6UKtGikjDqcvIvQuQCzxvZivM7Bkzawy0ds7tA/DdtjrZm81sipllmllmbm5uHWIEtvQ2CUw6uyP/+HIna/cUeB1HRMJEXco9CjgDeMI5NxAopgaHYJxz051zGc65jOTk5DrECHy3j+hOUnwM9721TqtGikiDqEu57wZ2O+eW+LZncbzsc8ysLYDv9kDdIga/ZnHR/Gp0D5btOMybK/Z4HUdEwkCty905tx/YZWbpvoeGA+uBt4DJvscmA3PqlDBEXDEohf6piTw4N0sX1BYRv6vrbJmfADPNbDUwAPgDMA0YaWabgJG+7bAXEWHcP643uUWlWndGRPwuqi5vds6tBDJO8tTwuvzcUDUgNZGrMlJ5bsE2xmek0q1VE68jiUiI0hmqDWzq6HTiYiK5/219uSoi/qNyb2Atm8Ty85Hd+XzTQT5Yl+N1HBEJUSp3D1wzpCPprRP4/TvrOVZW6XUcEQlBKncPREVGcP+lvdmTf4xHP97kdRwRCUEqd48M6dKCKwal8PT8rWTtL/Q6joiEGJW7h+6+uCdN46K5a/YaqrQssIjUI5W7h5o3juHui3uyfGe+lgUWCUOvL93Fsh2H/fKzVe4eu/yM9pzbrQUPvZ/FgcISr+OISAPZknuEe+as5bmF2/zy81XuHjMz/veyvpRWVHH/2+u9jiMiDaCqyvHr2WtoFBXBfZf08ss+VO4BoHPLxvzkO914d80+Ps7S3HeRUPfq0l18ue0Q94zpRauERn7Zh8o9QNx0fle6tWrCvf9ax9GyCq/jiIif5BSW8OB7GzinawuuzEjx235U7gEiJiqCBy/vy578Y/zl35r7LhKqfjNnLWWVVfzhe30xM7/tR+UeQM7slMSEwak8u2Ab6/bqqk0ioeb9tfv4YF0OPxvZnU4tG/t1Xyr3AHPn6J40jz8+971Sc99FQkbBsXLunbOO3u2acsPQzn7fn8o9wDSLj+besb1YtbuAlxZt9zqOiNSTaXM3cKi4jIe+34+oSP9Xr8o9AI3r345h3ZN5+INsdh066nUcEamjRVvyeOXLXdwwtDN92jdrkH2q3AOQmfGH7/UB4M7Zq7Xuu0gQKymv5K4319AhKZ7bR3RvsP2q3ANUSvN47h7Ti4Wb85i5REsTiASrR+dtYtvBYh68vC9xMZENtl+VewCbMDiV89Ja8of3NujwjEgQWr+3kKfmb+XKQSmc261lg+5b5R7AzIxp3+9HhBm/nLVaK0eKBJGyiiqmzlpF8/ho7h7Ts8H3r3IPcO0T47h7TE8Wbc1j5pIdXscRkWp67ONNrNtbyAPf60tifEyD71/lHgSuPvP44ZkH52bp8IxIEFi1K5+/f7qFy89oz0W923iSQeUeBE48PDN11iodnhEJYCXllfz89ZW0Sojlvkt6e5ZD5R4k2ifGcc+YnizeeoiXdXhGJGD98f1stuQW8/AV/WkWF+1ZDpV7ELnqzFSGdU/mwfey2JFX7HUcEfmaRVvyeG7hNiad3ZGhaQ07O+brVO5BxMx46Pt9iYowpmr2jEhAOVJawR3/XEWnFvHc+d0eXsdRuQebts3iuHdsL77cdogZWntGJGD87zvr2VdwjD+PH0B8TJTXcVTuwejKjBS+k57MtLlZbMwp8jqOSNj7OCuHV5fu4qbzuzKoY3Ov4wAq96BkZvzxiv40iY3itldWUFJe6XUkkbB1uLiMX72xhh5tErh9RJrXcf5D5R6kkhNiefjKfmTtL+Kh97O8jiMStu6ds5b8o2U8Mn4AsVENt3bM6ajcg9iFPVpz7TmdeH7hdj7NPuB1HJGwM2flHt5ZvY+fDk+jV7umXsf5Lyr3IHfnd3uQ3jqBO/65moNHSr2OIxI2duYd5e4313JGh0RuPr+r13G+oc7lbmaRZrbCzN7xbSeZ2Udmtsl3GxjfLoSoRtGRPDphIIUl5Uz95yqt/S7SAMoqqvjJK8uJMHh0wsAGubJSTdVHop8CG07YvhOY55xLA+b5tsWP0tskcPfFPfkkO5cXF+nsVRF/+/OH2azaXcAfr+hHSvN4r+OcVJ3K3cxSgDHAMyc8fCkww3d/BnBZXfYh1TPp7I5c2KMVD7y3gez9mh4p4i+fZh/gqflbmTikA6P7tPU6zinVdeT+F+CXQNUJj7V2zu0D8N22OtkbzWyKmWWaWWZubm4dY8jx6ZH9aNooWtMjRfzkQGEJv3h9FT3aJHDPmF5ex/lWtS53MxsLHHDOLavN+51z051zGc65jOTk5NrGkBO0bBLLn67sR3ZOEdPmanqkSH2qqnL87PWVFJdV8NgPBtIoOnCmPZ5MXUbu5wLjzGw78CpwoZm9DOSYWVsA363m6DWgC9Jbcd25nXnhi+18nJXjdRyRkPHEZ1tYuDmP+8f1plurBK/jnFaty90592vnXIpzrhNwNfCxc24i8BYw2feyycCcOqeUGvnVd9Pp1bYpP399FbsP6+IeInW1bMchHvloI5f0b8f4jFSv41SLP+bvTANGmtkmYKRvWxpQbFQkT0w8g8oqx49mLqe0QsffRWqr4Gg5t72yknaJjXjge30wM68jVUu9lLtz7lPn3Fjf/Tzn3HDnXJrv9lB97ENqpmOLxvz5yv6s3l3A795e73UckaDknOPO2avJKSzhbxPOoGkj7y6+UVOBN/Ne6s2o3m246fwuzFyykzdX7PY6jkjQeeGL7cxdu5+pF6UzIDXR6zg1onIPcVNHpXNW5yR+PXuN5r+L1MCSrXk88O4GRvRszY3ndfE6To2p3ENcVGQEf/vBQBIaRXPLy8soKin3OpJIwNtfUMKt/1hBh6R4HrmqPxERwXGc/UQq9zDQKqERj00YyI5DR/nlrNVaf0bkW5RWVHLLzGUcLavgqWsGBdVx9hOp3MPEWV1a8KvR6cxdu59nF2zzOo5IwPrd2+tZsTOfP13Zn7TWgT+f/VRU7mHkxvO6cFHv1kybm0Xmdk1iEvm615fuYuaSndx8flcu7hu468ZUh8o9jJgZD1/Zn5Tmcdz6j+XkFmn9d5GvrN6dzz1z1jK0W0vuGNXd6zh1pnIPM00bRfP4/wyi4Fg5N72UqQXGRIC8I6Xc/NIykpvEBuz67DUV/J9AaqxXu6Y8Mn4Ay3fmc9fsNfqCVcJaRWUVP3llBQeLy3hy4iCSGsd4HaleqNzD1MV92/Lzkd2ZvWIPT3621es4Ip754wfZfLEljz98ry99U5p5HafeRHkdQLzzkwu7senAEf74QRZdkxszqncbryOJNKhZy3Yz3XfhjSsGpXgdp15p5B7GzIyHr+hHv/bNuP21lazfW+h1JJEGs2hLHr+evZpzu7Xgvkt6ex2n3qncw1yj6EimT8qgaaNobpixVDNoJCxsyT3CzS8vo2OLxjz+P4OIDoEvUL8u9D6R1Fjrpo14elIGh46WaQaNhLxDxWVc98JSoiKM5689k2ZxwXkG6umo3AWAvinNNINGQl5JeSVTXsxkf0EJT0/OIDUp3utIfqNyl/+4uG9bfjbi+AyaJz7b4nUckXrlnOOXs1aTueMwj4wfwBkdmnsdya80W0b+y23Du7E59wgPf5BNSvN4xvVv53UkkXrxfx9t5K1Ve/nl6HTG9AvupQWqQ+Uu/+WrGTQ5BSX84vWVJMXHMDStpdexROrkjWW7efTjzVyVkcot53f1Ok6D0GEZ+YZG0ZE8PTmDrslNuOmlTFbvzvc6kkitLd6ax52zV3NO1xb8bxBdA7WuVO5yUs3ioplx3WAS42P44fNL2Xaw2OtIIjW2fm8hU17MpGOLxjwxMTSnPJ5K+HxSqbHWTRvx0vWDccA1zy7hQGGJ15FEqm3bwWImPbeExrFRvPDD0J3yeCoqd/lWXZKb8Ny1Z3KouIxJz31JoS7TJ0Fgb/4xJj6zhCoHL11/FinNQ3fK46mo3OW0BqQm8uTEQWw+cIQbZ+gkJwlseUdKmfjsEgqPlfPidYPp1qqJ15E8oXKXahnWPZk/j+/Pkm2H+OmrK6is0klOEngKS8qZ9NyX7Dl8jGevPZM+7UNnlceaUrlLtV06oD33ju3FB+tyuOdfOotVAsuxskpueCGT7P1FPHnNIAZ3TvI6kqc0z11q5Pqhnck7Usrjn24hOjKC+8f1DpupZRK4yiqquGXmMpbuOMTfJgzkO+mtvI7kOZW71NjUi9Ipr6zi6c+3YcBvVfDiocoqx89fX8mn2bk8eHlfxvbTWdWgcpdaMDPuurgnzsEzC7YBKnjxRmWV49ezV/PO6n3cdXEPJgzu4HWkgKFyl1oxM+4e0xMHPLtgG2bGfZf0UsFLg6morGLqrNW8uWIPtw1PY8qw8FhWoLpU7lJrZsY9Y3oCxwseUMFLgyivrOL2V1fy7pp93DGqOz++MM3rSAFH5S518lXBOwfPLdyGGfxmrApe/Ke0opIf/2MFH63P4e6Le3LjsC5eRwpIKnepMzPj3rE9cTieX7gdUMGLf5SUV3Lzy8v4NDuX313am0lnd/I6UsCqdbmbWSrwItAGqAKmO+f+amZJwGtAJ2A7MN45d7juUSWQmRm/GdsLgOcXbse54wUfEaGCl/pxtKyCG1/M5IsteTx4eV99eXoadTmJqQL4hXOuJzAEuNXMegF3AvOcc2nAPN+2hIGvCv6GoZ154Yvt3P7aSsoqqryOJSHgSGkF1z63lEVb8vjzlf1V7NVQ65G7c24fsM93v8jMNgDtgUuBC3wvmwF8CvyqTiklaHw1i6ZFk1geej+Lw0fLeGLiIJrE6gig1E7BsXKuff5LVu8u4NEJAzWPvZrqZfkBM+sEDASWAK19xf/VXwAnPVXMzKaYWaaZZebm5tZHDAkQZsYtF3Tl4Sv68cWWPCZMX8zBI6Vex5IgtCf/GFc++QVr9xTw+P+coWKvgTqXu5k1Ad4AbnfOFVb3fc656c65DOdcRnJycl1jSAC6MiOVpycNYtOBIq544gt25h31OpIEkbV7Crjs7wvZV1DCjB8O5qLebbyOFFTqVO5mFs3xYp/pnJvtezjHzNr6nm8LHKhbRAlmF/ZozT9uHEL+sXIuf+L4CEzkdD7JOsD4pxYRExnBG7ecwznddB3fmqp1udvxeW7PAhucc4+c8NRbwGTf/cnAnNrHk1BwRofmzLr5bGIijaunL+aLzQe9jiQBbOaSHdzwYiadWzbmzR+dQ/fWCV5HCkp1GbmfC1wDXGhmK31/LgamASPNbBMw0rctYa5bqwTe+NE5tEtsxLXPL2XOyj1eR5IAU1XlmDY3i7vfXMuwtJa8ftPZtGrayOtYQasus2UWAKeaxDy8tj9XQlfbZnH886ZzmPJSJj99dSXr9xbyy9E9iNRc+LBXUl7J1FmreXvVXn5wVgd+N643UWF0MWt/0H89aVDN4qN5+YazmHR2R56av5Vrn/+S/KNlXscSDx0qLmPSs1/y9qq93PndHjxwWR8Vez3Qf0FpcNGREfzu0j5Mu7wvS7YeYtxjC8naX+2JVhJClu88zJhHP2fl7nz+NmEgN5/fVctW1BOVu3jm6sEdeGXKEErKK7n88S+Yu2af15GkgTjneHHRdq56ahGREcbsW87hkv6aw16fVO7iqUEdm/P2T4aS3iaBW2Yu508fZFOli2+HtKNlFdz+2kp+M2cd56Ul8+5PzgvrC1n7i8pdPNe6aSNenTKEqzJSeeyTzdzwYiYFR8u9jiV+sPnAES59bCFvr9rL1IvSeWZSBs3io72OFZJU7hIQYqMimfb9vvz+sj7M35jLd/86n0Vb8ryOJfXo3dX7uPSxBeQVl/HidWdx63e6adVQP1K5S8AwM64Z0pE3bjmH2OhIfvDMYh6cu0ErSwa50opKfvf2em79x3LS2yTw7m1DGZqmM079TeUuAad/aiLv3jaUq8/swFOfbeV7jy9k84Eir2NJLazZXcAlf1vAcwu3ce05nXh1ytm0bRbndaywoHKXgBQfE8WDl/dl+jWD2FdQwphHF/DSou04py9bg0FZRRWPfJjNZY8vpOBYOc//8Ex+O643MVGqnIaiRbYloI3q3YYBHRKZ+s/V3DtnHZ9k5/LQ9/uRnBDrdTQ5hfV7C/nFP1exYV8hl5/RnvvG9taXph7QX6MS8FolNOKFH57Jby/pxYLNBxn9l/nMXr5bo/gAU15ZxaPzNjHusQXkFpXy9KQMHhk/QMXuEQuEX5CMjAyXmZnpdQwJAhtzipg6azWrduVzVuckfn9ZH60aGAA25hTxi9dXsWZPAeP6t+P+cb1p3jjG61ghz8yWOecyTvqcyl2CTVWV47XMXUybm0VxaQXXn9eZ2y5Mo7Eu5dfgCo6W85d5G3lp0Q6axkXzwGV9+G7ftl7HChvfVu76bZCgExFhTBjcgVG9WvPQ+1k89dlW3lq5l/su6cVFvdtobZIGUFFZxStLd/HIh9nkHyvn6jM7cMeo7rRoou9CAoVG7hL0Mrcf4p5/rSVrfxHnd0/m/nG96dSysdexQtbCzQf53dvryc4pYkiXJH4ztje92jX1OlZY0mEZCXkVlVXMWLSDRz7MpqyyivEZqfz4wm6aU12Pth8s5oH3NvDR+hxSmsdx98U9Gd1H/1LykspdwkZOYQmPztvE65m7MIwfnNWBH13QVVf0qYP9BSVMn7+VlxfvICrSuPU73bh+aGcaRUd6HS3sqdwl7Ow6dJTHPt7MrOW7iYo4vqzBzRd0paWOCVfb9oPFPDV/C7OW7abKweUD2zP1onT9RRlAVO4StnbkFfPXeZv414o9xEZFMvmcTkwZ1oUkTdM7paz9hTz+yRbeWb2XqMgIrspIZcqwLqQmxXsdTb5G5S5hb0vuEf767028vXov0ZERjO3XlolDOjIwNVHHjH2W7zzM459s5t8bDtA4JpKJZ3fk+qGdaZWgkXqgUrmL+GzKKWLGou28uXwPxWWV9G7XlIlDOnLpgHbEx4TfzOCCo+W8tXovs5btZtWufBLjo7nu3M5MPruTziwNAip3ka85UlrBmyv2MHPxDrL2F5EQG8XlZ7Rn4pCOpIX4Ga+VVY6Fmw/yz2W7+WDdfsoqqujRJoHxGalcdWaqTgYLIip3kVNwzrFsx2FeWryDuWv2U1ZZRb+UZozq1ZpRvduQ1qpJyBy22XawmFnLdjF7+R72FZTQLC6aywa044pBqfRp3zRkPmc4UbmLVEPekVJmLdvN3LX7WbkrH4BOLeIZ1bsNo3q1ZmCH5kQG0ZWDSsor+XLbIT7bmMv8jblsOnCECINh3ZO5clAqI3q1IjZK0xmDmcpdpIZyCkv4aH0OH67PYdGWg5RXOlo2iWF4j9YM6ZpE/5REOrdsHFCjXeccW3KLmb8xl8825rJ4ax6lFVXEREVwVuckzu+ezNh+7WjTTF+QhgqVu0gdFJaU82l2Lh+u289n2bkUlVYA0Cwumv6piQxIacaADon0T0lssLVVKqscO/KK2bCviKz9hWzYV8S6vQXsKygBoEtyY4alJXN+ejJDOrcgLkYj9FCkchepJ5VVjk0Hili5M5+Vu47/2ZhTRJXv16h9YhypSXG0S4yjXbPjt20TG9E+MY62zRqR0Kh6M1COllWQd6SMQ8XH/+QVl3GouJQtB4rJ2l9Idk4RJeXHry0bGWF0admYHm2bMqRLEsPSkjUnPUyo3EX8qLi0gjV7Cli5K591ewvZm3+MffnHyCkqpbLqv3+/YqMiiI6MICrSiIowIiOMqIgI361RWlFFXnHpf4r765Iax9CzbQI92jSlR5sEerZtSrdWTbQUQJjSkr8iftQ4NoohXVowpEuL/3q8orKKA0Wl7Cs4xt78EvbmHyOvuIyKSkdlVRUVVY6KSkdF1f/fjomMIKlxDElNYmjROIakxrEkNfbdbxJDQmxUQB3nl8Clchfxk6jIiOOHZxLjGNTR6zQSbnQNVRGREKRyFxEJQX4rdzMbbWbZZrbZzO70135EROSb/FLuZhYJ/B34LtALmGBmvfyxLxER+SZ/jdwHA5udc1udc2XAq8ClftqXiIh8jb/KvT2w64Tt3b7H/sPMpphZppll5ubm+imGiEh48le5n2wi7n+dzeGcm+6cy3DOZSQnJ/sphohIePJXue8GUk/YTgH2+mlfIiLyNX5ZfsDMooCNwHBgD7AU+IFzbt0pXp8L7Kj3IA2jJXDQ6xAeCNfPDeH72fW5A09H59xJD3345QxV51yFmf0Y+ACIBJ47VbH7Xh+0x2XMLPNUazuEsnD93BC+n12fO7j4bfkB59x7wHv++vkiInJqOkNVRCQEqdzrbrrXATwSrp8bwvez63MHkYBYz11EROqXRu4iIiFI5S4iEoJU7vXEzO4wM2dmLb3O0lDM7GEzyzKz1Wb2ppklep3Jn8J1pVMzSzWzT8xsg5mtM7Ofep2pIZlZpJmtMLN3vM5SEyr3emBmqcBIYKfXWRrYR0Af51w/jp+09muP8/hNmK90WgH8wjnXExgC3BpGnx3gp8AGr0PUlMq9fvwf8Eu+tn5OqHPOfeicq/BtLub4MhOhKmxXOnXO7XPOLffdL+J40bX/9neFBjNLAcYAz3idpaZU7nVkZuOAPc65VV5n8dh1wFyvQ/jRaVc6DQdm1gkYCCzxOEpD+QvHB25VHueoMV0guxrM7N9Am5M8dTdwFzCqYRM1nG/77M65Ob7X3M3xf7rPbMhsDey0K52GOjNrArwB3O6cK/Q6j7+Z2VjggHNumZld4HGcGlO5V4NzbsTJHjezvkBnYJWZwfHDEsvNbLBzbn8DRvSbU332r5jZZGAsMNyF9kkTYb3SqZlFc7zYZzrnZnudp4GcC4wzs4uBRkBTM3vZOTfR41zVopOY6pGZbQcynHOBuoJcvTKz0cAjwPnOuZC+4kpNVzoNJXZ85DIDOOScu93jOJ7wjdzvcM6N9ThKtemYu9TFY0AC8JGZrTSzJ70O5C++L46/Wul0A/B6OBS7z7nANcCFvv/PK32jWQlgGrmLiIQgjdxFREKQyl1EJASp3EVEQpDKXUQkBKncRURCkMpdRCQEqdxFRELQ/wOVnmci5YAbRAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xs = np.arange(-5, 5, 0.25)\n",
    "ys = f(xs)\n",
    "plt.plot(xs, ys)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.999378523327323e-06"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h = 0.000001\n",
    "x = 2/3\n",
    "(f(x + h) - f(x))/h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二、多个输入时的「偏导数」\n",
    "【译者导读】现实中的函数有很多输入。这里用 d = a·b + c 演示：固定其他变量、只微调其中一个，看输出 d 变化多少——这就是 d 对该变量的偏导数 (partial derivative)。比如 d 对 c 的斜率是 1，对 a 的斜率是 b（= −3）。请动手改改被微调的变量，验证一下直觉。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.0\n"
     ]
    }
   ],
   "source": [
    "# les get more complex\n",
    "a = 2.0\n",
    "b = -3.0\n",
    "c = 10.0\n",
    "d = a*b + c\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "d1 4.0\n",
      "d2 4.0001\n",
      "slope 0.9999999999976694\n"
     ]
    }
   ],
   "source": [
    "h = 0.0001\n",
    "\n",
    "# inputs\n",
    "a = 2.0\n",
    "b = -3.0\n",
    "c = 10.0\n",
    "\n",
    "d1 = a*b + c\n",
    "c += h\n",
    "d2 = a*b + c\n",
    "\n",
    "print('d1', d1)\n",
    "print('d2', d2)\n",
    "print('slope', (d2 - d1)/h)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 三、micrograd 的核心：Value 对象与表达式图\n",
    "【译者导读】接下来是本讲的主角——`Value` 类。它把一个标量包起来，同时记录：这个值是由哪些「子节点」(`_prev`)、通过什么运算 (`_op`) 得到的。这样一连串运算就自然连成一张**表达式图 (expression graph)**，前向传播 (forward pass) 算出的每个中间值都是图上的一个节点；`grad` 字段则准备存放反向传播算出的梯度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Value(data=-8.0)"
      ]
     },
     "execution_count": 257,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class Value:\n",
    "  \n",
    "  def __init__(self, data, _children=(), _op='', label=''):\n",
    "    self.data = data\n",
    "    self.grad = 0.0\n",
    "    self._backward = lambda: None\n",
    "    self._prev = set(_children)\n",
    "    self._op = _op\n",
    "    self.label = label\n",
    "\n",
    "  def __repr__(self):\n",
    "    return f\"Value(data={self.data})\"\n",
    "  \n",
    "  def __add__(self, other):\n",
    "    out = Value(self.data + other.data, (self, other), '+')\n",
    "    \n",
    "    def _backward():\n",
    "      self.grad += 1.0 * out.grad\n",
    "      other.grad += 1.0 * out.grad\n",
    "    out._backward = _backward\n",
    "    \n",
    "    return out\n",
    "\n",
    "  def __mul__(self, other):\n",
    "    out = Value(self.data * other.data, (self, other), '*')\n",
    "    \n",
    "    def _backward():\n",
    "      self.grad += other.data * out.grad\n",
    "      other.grad += self.data * out.grad\n",
    "    out._backward = _backward\n",
    "      \n",
    "    return out\n",
    "  \n",
    "  def tanh(self):\n",
    "    x = self.data\n",
    "    t = (math.exp(2*x) - 1)/(math.exp(2*x) + 1)\n",
    "    out = Value(t, (self, ), 'tanh')\n",
    "    \n",
    "    def _backward():\n",
    "      self.grad += (1 - t**2) * out.grad\n",
    "    out._backward = _backward\n",
    "    \n",
    "    return out\n",
    "  \n",
    "  def backward(self):\n",
    "    \n",
    "    topo = []\n",
    "    visited = set()\n",
    "    def build_topo(v):\n",
    "      if v not in visited:\n",
    "        visited.add(v)\n",
    "        for child in v._prev:\n",
    "          build_topo(child)\n",
    "        topo.append(v)\n",
    "    build_topo(self)\n",
    "    \n",
    "    self.grad = 1.0\n",
    "    for node in reversed(topo):\n",
    "      node._backward()\n",
    "\n",
    "\n",
    "a = Value(2.0, label='a')\n",
    "b = Value(-3.0, label='b')\n",
    "c = Value(10.0, label='c')\n",
    "e = a*b; e.label = 'e'\n",
    "d = e + c; d.label = 'd'\n",
    "f = Value(-2.0, label='f')\n",
    "L = d * f; L.label = 'L'\n",
    "L"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 可视化工具\n",
    "【译者导读】这段 graphviz 代码把表达式图画出来：每个节点显示 data（前向值）和 grad（梯度），箭头表示运算依赖。不用细读它的实现，它只是我们的「显微镜」。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [],
   "source": [
    "from graphviz import Digraph\n",
    "\n",
    "def trace(root):\n",
    "  # builds a set of all nodes and edges in a graph\n",
    "  nodes, edges = set(), set()\n",
    "  def build(v):\n",
    "    if v not in nodes:\n",
    "      nodes.add(v)\n",
    "      for child in v._prev:\n",
    "        edges.add((child, v))\n",
    "        build(child)\n",
    "  build(root)\n",
    "  return nodes, edges\n",
    "\n",
    "def draw_dot(root):\n",
    "  dot = Digraph(format='svg', graph_attr={'rankdir': 'LR'}) # LR = left to right\n",
    "  \n",
    "  nodes, edges = trace(root)\n",
    "  for n in nodes:\n",
    "    uid = str(id(n))\n",
    "    # for any value in the graph, create a rectangular ('record') node for it\n",
    "    dot.node(name = uid, label = \"{ %s | data %.4f | grad %.4f }\" % (n.label, n.data, n.grad), shape='record')\n",
    "    if n._op:\n",
    "      # if this value is a result of some operation, create an op node for it\n",
    "      dot.node(name = uid + n._op, label = n._op)\n",
    "      # and connect this node to it\n",
    "      dot.edge(uid + n._op, uid)\n",
    "\n",
    "  for n1, n2 in edges:\n",
    "    # connect n1 to the op node of n2\n",
    "    dot.edge(str(id(n1)), str(id(n2)) + n2._op)\n",
    "\n",
    "  return dot\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 3.0.0 (20220226.1711)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"1157pt\" height=\"156pt\"\n",
       " viewBox=\"0.00 0.00 1157.00 156.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 152)\">\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-152 1153,-152 1153,4 -4,4\"/>\n",
       "<!-- 140306649871024 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140306649871024</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"4.5,-111.5 4.5,-147.5 189.5,-147.5 189.5,-111.5 4.5,-111.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"16\" y=\"-125.8\" font-family=\"Times,serif\" font-size=\"14.00\">a</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"27.5,-111.5 27.5,-147.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"67.5\" y=\"-125.8\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"107.5,-111.5 107.5,-147.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"148.5\" y=\"-125.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad 6.0000</text>\n",
       "</g>\n",
       "<!-- 140306649871264* -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>140306649871264*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"257\" cy=\"-101.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"257\" y=\"-97.8\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140306649871024&#45;&gt;140306649871264* -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>140306649871024&#45;&gt;140306649871264*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M189.91,-113.21C200.76,-111.29 211.28,-109.43 220.61,-107.77\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"221.37,-111.19 230.6,-106 220.15,-104.3 221.37,-111.19\"/>\n",
       "</g>\n",
       "<!-- 140306649871072 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140306649871072</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"320,-28.5 320,-64.5 516,-64.5 516,-28.5 320,-28.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"331.5\" y=\"-42.8\" font-family=\"Times,serif\" font-size=\"14.00\">c</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"343,-28.5 343,-64.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"386.5\" y=\"-42.8\" font-family=\"Times,serif\" font-size=\"14.00\">data 10.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"430,-28.5 430,-64.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"473\" y=\"-42.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;2.0000</text>\n",
       "</g>\n",
       "<!-- 140306649871744+ -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>140306649871744+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"579\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"579\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140306649871072&#45;&gt;140306649871744+ -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>140306649871072&#45;&gt;140306649871744+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M516.07,-62.98C525.42,-64.57 534.43,-66.1 542.55,-67.48\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"542.21,-70.97 552.65,-69.2 543.38,-64.07 542.21,-70.97\"/>\n",
       "</g>\n",
       "<!-- 140306649871120 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140306649871120</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-56.5 0,-92.5 194,-92.5 194,-56.5 0,-56.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"11.5\" y=\"-70.8\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"23,-56.5 23,-92.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"65.5\" y=\"-70.8\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;3.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"108,-56.5 108,-92.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"151\" y=\"-70.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;4.0000</text>\n",
       "</g>\n",
       "<!-- 140306649871120&#45;&gt;140306649871264* -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>140306649871120&#45;&gt;140306649871264*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M194.01,-90.91C203.35,-92.5 212.37,-94.04 220.49,-95.43\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"220.16,-98.93 230.6,-97.16 221.34,-92.03 220.16,-98.93\"/>\n",
       "</g>\n",
       "<!-- 140306649871648 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>140306649871648</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"957,-27.5 957,-63.5 1149,-63.5 1149,-27.5 957,-27.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"969.5\" y=\"-41.8\" font-family=\"Times,serif\" font-size=\"14.00\">L</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"982,-27.5 982,-63.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"1024.5\" y=\"-41.8\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;8.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1067,-27.5 1067,-63.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"1108\" y=\"-41.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140306649871648* -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>140306649871648*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"894\" cy=\"-45.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"894\" y=\"-41.8\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140306649871648*&#45;&gt;140306649871648 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140306649871648*&#45;&gt;140306649871648</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M921.28,-45.5C928.78,-45.5 937.44,-45.5 946.67,-45.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"946.87,-49 956.87,-45.5 946.87,-42 946.87,-49\"/>\n",
       "</g>\n",
       "<!-- 140306649871744 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>140306649871744</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"642,-55.5 642,-91.5 831,-91.5 831,-55.5 642,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"653.5\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">d</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"665,-55.5 665,-91.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"705\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">data 4.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"745,-55.5 745,-91.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"788\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;2.0000</text>\n",
       "</g>\n",
       "<!-- 140306649871744&#45;&gt;140306649871648* -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>140306649871744&#45;&gt;140306649871648*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M831.1,-56.65C840.4,-54.97 849.38,-53.35 857.49,-51.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"858.37,-55.29 867.59,-50.07 857.13,-48.4 858.37,-55.29\"/>\n",
       "</g>\n",
       "<!-- 140306649871744+&#45;&gt;140306649871744 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140306649871744+&#45;&gt;140306649871744</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M606.03,-73.5C613.66,-73.5 622.52,-73.5 631.94,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"631.94,-77 641.94,-73.5 631.94,-70 631.94,-77\"/>\n",
       "</g>\n",
       "<!-- 140306649871264 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>140306649871264</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"321,-83.5 321,-119.5 515,-119.5 515,-83.5 321,-83.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"332.5\" y=\"-97.8\" font-family=\"Times,serif\" font-size=\"14.00\">e</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"344,-83.5 344,-119.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"386.5\" y=\"-97.8\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"429,-83.5 429,-119.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"472\" y=\"-97.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;2.0000</text>\n",
       "</g>\n",
       "<!-- 140306649871264&#45;&gt;140306649871744+ -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>140306649871264&#45;&gt;140306649871744+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M515.16,-84.57C524.82,-82.87 534.13,-81.23 542.49,-79.75\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"543.3,-83.17 552.54,-77.98 542.08,-76.27 543.3,-83.17\"/>\n",
       "</g>\n",
       "<!-- 140306649871264*&#45;&gt;140306649871264 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>140306649871264*&#45;&gt;140306649871264</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M284.26,-101.5C292.05,-101.5 301.1,-101.5 310.75,-101.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"310.99,-105 320.99,-101.5 310.99,-98 310.99,-105\"/>\n",
       "</g>\n",
       "<!-- 140306649871792 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>140306649871792</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"642.5,-0.5 642.5,-36.5 830.5,-36.5 830.5,-0.5 642.5,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"653\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">f</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"663.5,-0.5 663.5,-36.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"706\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;2.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"748.5,-0.5 748.5,-36.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"789.5\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad 4.0000</text>\n",
       "</g>\n",
       "<!-- 140306649871792&#45;&gt;140306649871648* -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>140306649871792&#45;&gt;140306649871648*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M830.65,-34.67C840.2,-36.33 849.43,-37.93 857.72,-39.37\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"857.24,-42.84 867.69,-41.11 858.44,-35.95 857.24,-42.84\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f9bb8244670>"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "draw_dot(L)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 四、梯度有什么用：一步优化的预览\n",
    "【译者导读】拿到梯度后，把每个叶子节点的 data 沿梯度方向微调一小步，再跑一遍前向传播，就能看到输出朝我们期望的方向变化——这就是**梯度下降 (gradient descent)** 的最小雏形。下面还有一个 `lol()` 函数，用数值法逐一核对我们手推的梯度是否正确。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-7.286496\n"
     ]
    }
   ],
   "source": [
    "a.data += 0.01 * a.grad\n",
    "b.data += 0.01 * b.grad\n",
    "c.data += 0.01 * c.grad\n",
    "f.data += 0.01 * f.grad\n",
    "\n",
    "e = a * b\n",
    "d = e + c\n",
    "L = d * f\n",
    "\n",
    "print(L.data)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-3.9999999999995595\n"
     ]
    }
   ],
   "source": [
    "def lol():\n",
    "  \n",
    "  h = 0.001\n",
    "  \n",
    "  a = Value(2.0, label='a')\n",
    "  b = Value(-3.0, label='b')\n",
    "  c = Value(10.0, label='c')\n",
    "  e = a*b; e.label = 'e'\n",
    "  d = e + c; d.label = 'd'\n",
    "  f = Value(-2.0, label='f')\n",
    "  L = d * f; L.label = 'L'\n",
    "  L1 = L.data\n",
    "  \n",
    "  a = Value(2.0, label='a')\n",
    "  b = Value(-3.0, label='b')\n",
    "  b.data += h\n",
    "  c = Value(10.0, label='c')\n",
    "  e = a*b; e.label = 'e'\n",
    "  d = e + c; d.label = 'd'\n",
    "  f = Value(-2.0, label='f')\n",
    "  L = d * f; L.label = 'L'\n",
    "  L2 = L.data\n",
    "  \n",
    "  print((L2 - L1)/h)\n",
    "  \n",
    "lol()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 五、一个神经元的手动反向传播\n",
    "【译者导读】现在把这些工具用到一个真实结构上：一个带 tanh 激活函数 (activation function) 的神经元 (neuron)，即 o = tanh(x1·w1 + x2·w2 + b)。先看看 tanh 的形状，然后沿着表达式图**从输出往输入**一个节点一个节点手推梯度——每一步都是链式法则 (chain rule)：本地导数 × 上游梯度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjgklEQVR4nO3de3zU9Z3v8dcnE8IlQe6EcNGggIoXVAJUbWtAcVFrqbZrrbvWunU5rbrb9rTbutu92N2ze2y3Z3u2W7eUtq60i7LdVo5gUVBqKtYqFxUkXAwiQkgId0Ig5DLzOX/MBMcQIGQm85vJ7/18POYxv8v3N/P5PiaZ9/zu5u6IiEh45QVdgIiIBEtBICIScgoCEZGQUxCIiIScgkBEJOTygy6gK4YOHeqlpaVBl3HWjh49SmFhYdBlZJz6HT5h7Xu293vt2rX73H1Y++k5GQSlpaWsWbMm6DLOWkVFBeXl5UGXkXHqd/iEte/Z3m8ze6+j6do0JCIScgoCEZGQUxCIiIScgkBEJOQUBCIiIZeWIDCzx8xsj5ltOMV8M7Pvm9lWM1tvZlclzZtlZlsS8x5KRz0iItJ56VojeByYdZr5NwHjE485wA8BzCwCPJqYPxH4jJlNTFNNIiLSCWk5j8DdXzKz0tM0mQ38zOPXvH7VzAaaWQlQCmx1920AZrYw0XZjOuoSkfBwd5qjMZpaY7S0xmiOxmhpdZqjUZpaY7RGndaY0xqNEY05LTEnGotPj7kTjUHUnVjMicbi09wh5k7sxHN8vsOJeSQNv7OtmU28gxNftq2utjYnaj1Rc9u4txv/QMc+0M/brhrN2KHpPWktUyeUjQJ2Jo1XJ6Z1NH1aRy9gZnOIr01QXFxMRUVFtxTanRoaGnKy7lSp3+HTlb43RZ36JudIs1PfHH8+0uI0tkJj23OrJx7QHHWaotAcc5qj0Bxt9wUalLc3p/0lLWk4//BOLh+W3q/uTAWBdTDNTzP95Inu84B5AGVlZZ7NZ++dSrafddhd1O/w6ajv+xua2L7/GNUHj1F9sDHpuZG6+uMca452+Fp5BkW98+nfpxf9++TT/5x8RvTOp19BPn16RejTK4++vSL0LYjQp1eE3vl59IrkUZCfR0Ekj175eRREjF6RPPIjeeTnWfwRMfLz8ojkGXlmRPKMSB4nhvPMyMsz8iw+zRLPeWYY8WGME9MNeHnlSj760Y9iiW+2eBN7fzypX5aYaLzfNnl6JmUqCKqBMUnjo4EaoOAU00UkR0VjTm1DjCXrathYW8+m2no21tSz50jTB9oNKSxg9KC+TBx5DjMuGs7Qot4MKSxgSFEBQxLDgwoLKCyIBPLl2BW9842+BZGgyzhrmQqCxcCDiX0A04DD7l5rZnuB8WY2FtgF3AnclaGaRCQN3J1t+46y8u29rKzax6vb9nO0OQq8QX6eMW54ER8eP5SJJedw/rBCRg/qx+hBfelXkJOXOuuR0vJJmNmTQDkw1Myqgb8DegG4+1xgKXAzsBU4BtybmNdqZg8Cy4AI8Ji7V6ajJhHpPo3NUV7csoeXEl/+uw41AlA6pB+fuHIUfY7u5vYZUxk3vIje+bn3Czls0nXU0GfOMN+BB04xbynxoBCRLLd5dz1PvLaDRW/s4sjxVvr3yefaC4Zy//QL+Mi4YZw7pB8AFRX7uWTkgICrlc7SupmInFZjc5Ql62t4ctUO3thxiIL8PG65rIQ7ysYwpXQQ+RFdoCDXKQhEpENHm1r5YcU7zP/9do4cb2Xc8CL+5mMT+eRVoxjYryDo8iSNFAQi8gGxmLPojV18+7nN7DnSxC2Xl3DP1aVMKR2UM0fvyNlREIjICW/sOMjDSzaybuchJo0ZyNy7J3PVuYOCLku6mYJARKirP863n93MU2/sYnj/3vyfP5zEbVeOIi9PawBhoCAQCbnfv7Of+xes5WhzlAemX8D95eMo7K2vhjDRpy0SYj9/9T2+tbiS0qGF/OruyZw/rCjokiQACgKREGqJxvjWkkr+89UdzLhoOP965xX079Mr6LIkIAoCkZA5eLSZ+xe8zu+37ed/XHc+X/+Di4hoX0CoKQhEQuTtuiN8fv5q6uqb+N6nJ3HblaODLkmygIJAJCQ21dZzx9zf07cgwn/N+RBX6rBQSVAQiIRA7eFG7v2P1RT2zudX91/DqIF9gy5JsoiCQKSHO3K8hXv/YzUNTa389xeuVgjISRQEIj1YSzTG/Qtep2pPA//xuSlcXHJO0CVJFtJlA0V6KHfnrxdtYGXVPv73bZfx0QnDgi5JspSCQKSHevTFrfzXmp38+Yxx3DFlzJkXkNBKSxCY2Swz22JmW83soQ7m/4WZvZl4bDCzqJkNTszbbmZvJeatSUc9ImG36I1qvrv8bW6/chRfmTkh6HIky6W8j8DMIsCjwEziN6lfbWaL3X1jWxt3/2fgnxPtbwW+4u4Hkl5murvvS7UWEYH11Yf4+i/Xc/X5Q3jkk5fr0tFyRulYI5gKbHX3be7eDCwEZp+m/WeAJ9PwviLSTks0xtd/uZ7BhQXM/ePJFORr66+cmcVvJ5zCC5h9Cpjl7vclxu8Gprn7gx207Ud8rWFc2xqBmb0LHAQc+JG7zzvF+8wB5gAUFxdPXrhwYUp1B6GhoYGiovBd1Ev9zpzF7zTzVFULX7qqN1cOD+6gQH3m2Wn69Olr3b2s/fR0/KV0tN55qnS5Ffhdu81C17p7jZkNB543s83u/tJJLxgPiHkAZWVlXl5enmLZmVdRUUEu1p0q9Tsztu5p4JnnV3LL5SV85Y6rMva+HdFnnlvSsd5YDSQfkjAaqDlF2ztpt1nI3WsSz3uARcQ3NYnIWYjFnL98aj19CyI8fOslQZcjOSYdQbAaGG9mY82sgPiX/eL2jcxsAHAd8HTStEIz6982DNwIbEhDTSKhsmDVDlZvP8hf33Ixw/r3DrocyTEpbxpy91YzexBYBkSAx9y90sy+kJg/N9H0NmC5ux9NWrwYWJQ4qiEfeMLdn0u1JpEwqT3cyLef3cxHxg/lU5N1NVE5e2nZm+TuS4Gl7abNbTf+OPB4u2nbgEnpqEEkjNrOHo7GnH+67TIdKipdomPLRHLYkvW1rNi8h6/eOIExg/sFXY7kKAWBSI46eLSZby2uZNLoAdx77digy5EcpquPiuSof11RxaHGFv7zvmm61aSkRGsEIjloT/1xnli1g09eNUqXlpaUKQhEctDc324jGnMemD4u6FKkB1AQiOSYPUeOs+C19/jEFaM4b0hh0OVID6AgEMkxP35pGy3RGA/O0NqApIeCQCSH7Gto4uevvsfsK0YxdqjWBiQ9FAQiOeTHK7fR3Kq1AUkvBYFIjjhwtJmf//49bp00kguGZe+ljiX3KAhEcsSPV26jsSXKn2ltQNJMQSCSAw4ebeZnr2znlstKGDe8f9DlSA+jIBDJAT99+V2OtUT58+vHB12K9EAKApEsd/hYC4+/sp2bLy1hQrHWBiT9FAQiWe6x371LQ1Mrf3a99g1I90hLEJjZLDPbYmZbzeyhDuaXm9lhM3sz8fjbzi4rEmYt0RgLXtvB9RcN56IRuqaQdI+Urz5qZhHgUWAm8fsXrzazxe6+sV3Tle7+sS4uKxJKKzbVsa+hiT/60LlBlyI9WDrWCKYCW919m7s3AwuB2RlYVqTHW/DaDkoG9OG6CcODLkV6sHQEwShgZ9J4dWJae1eb2Toze9bMLjnLZUVCZ+eBY6ys2senp4zR/QakW6XjxjQd/YV6u/HXgfPcvcHMbgb+HzC+k8vG38RsDjAHoLi4mIqKiq7WG5iGhoacrDtV6nfX/PLtZgw4t7WaioqatNWVCfrMc0s6gqAaGJM0Phr4wF+tu9cnDS81s383s6GdWTZpuXnAPICysjIvLy9PQ+mZVVFRQS7WnSr1++y1RGN87eXfcP3Fw7l91pT0FpYB+sxzSzo2Da0GxpvZWDMrAO4EFic3MLMRZmaJ4amJ993fmWVFwuiFjfGdxHdN005i6X4prxG4e6uZPQgsAyLAY+5eaWZfSMyfC3wK+KKZtQKNwJ3u7kCHy6Zak0iue2KVdhJL5qTl5vXuvhRY2m7a3KThHwA/6OyyImHWtpP4yzeM105iyQidWSySZZ5ctYM8g09PGXPmxiJpoCAQySIt0Ri/WFPNjIuGUzKgb9DlSEgoCESyiHYSSxAUBCJZ5IlVOxipncSSYQoCkSyxY3/bmcTnaiexZJSCQCRLLFwd30l8x5TRQZciIaMgEMkC0Zjzy7XVTL9QO4kl8xQEIlngtXf3s+dIE7dfpbUByTwFgUgWWLKuln4FEWZcpJ3EknkKApGAtURjPLuhlpkTi+lbEAm6HAkhBYFIwF7euo9Dx1q49fKRQZciIaUgEAnYknU1nNMnn49MGBp0KRJSCgKRAB1vibK8so5Zl46gd742C0kwFAQiAarYspeGplZunaTNQhIcBYFIgJasq2FoUQFXnz8k6FIkxBQEIgFpaGplxeY6br6shPyI/hUlOGn56zOzWWa2xcy2mtlDHcz/IzNbn3i8YmaTkuZtN7O3zOxNM1uTjnpEcsGKTXUcb4lps5AELuU7lJlZBHgUmEn8ZvSrzWyxu29MavYucJ27HzSzm4jfhH5a0vzp7r4v1VpEcsmSdTWUDOjD5HMHBV2KhFw61gimAlvdfZu7NwMLgdnJDdz9FXc/mBh9FdB59BJqh4+18Nu39/Kxy0vI05VGJWDpuGfxKGBn0ng1H/y1397ngWeTxh1YbmYO/Mjd53W0kJnNAeYAFBcXU1FRkUrNgWhoaMjJulOlfp/speoWWqLOyNZaKir2ZLawDNBnnlvSEQQd/ZzxDhuaTSceBB9Omnytu9eY2XDgeTPb7O4vnfSC8YCYB1BWVubl5eUpF55pFRUV5GLdqVK/T/bTn77GeUOO8bmPl2PW89YI9JnnlnRsGqoGku+yPRqoad/IzC4HfgLMdvf9bdPdvSbxvAdYRHxTk0iPta+hid9t3cetl4/skSEguScdQbAaGG9mY82sALgTWJzcwMzOBZ4C7nb3t5OmF5pZ/7Zh4EZgQxpqEslaz75VS8zR0UKSNVLeNOTurWb2ILAMiACPuXulmX0hMX8u8LfAEODfE7+AWt29DCgGFiWm5QNPuPtzqdYkks2WrKtlQnERF47oH3QpIkB69hHg7kuBpe2mzU0avg+4r4PltgGT2k8X6al2Hz7Oqu0H+OrMCUGXInKCTmcUyaDlG3cDcNNlJQFXIvI+BYFIBi2r3M0FwwoZN7wo6FJETlAQiGTIwaPNvLrtALMuHRF0KSIfoCAQyZAVm/cQjTl/cImCQLKLgkAkQ57bsJuRA/pw2agBQZci8gEKApEMONrUysqqvdx4yQidRCZZR0EgkgG/fXsvTa0x7R+QrKQgEMmA5zbsZnBhAVNKBwddishJFAQi3aypNcqLm/cw8+JiIrrktGQhBYFIN3vlnf0caWrlDy4tDroUkQ4pCES62fLK3RT1zueaC4YGXYpIhxQEIt0oGnOWV9ZRfuEw+vSKBF2OSIcUBCLdaO17B9l/tFlHC0lWUxCIdKPnNuymID+P8guHB12KyCkpCES6ibuzrHI3Hxk3lKLeabniu0i3UBCIdJP36mPsOtSoawtJ1ktLEJjZLDPbYmZbzeyhDuabmX0/MX+9mV3V2WVFctXaPVHyDG6YqMNGJbulHARmFgEeBW4CJgKfMbOJ7ZrdBIxPPOYAPzyLZUVy0tq6VqaOHczgwoKgSxE5rXSsEUwFtrr7NndvBhYCs9u1mQ38zONeBQaaWUknlxXJOdv2NlDT4MzSZiHJAenYgzUK2Jk0Xg1M60SbUZ1cFgAzm0N8bYLi4mIqKipSKjoIDQ0NOVl3qsLY72e2NQPQv/5dKireC7iazAvjZw652+90BEFHF0/xTrbpzLLxie7zgHkAZWVlXl5efhYlZoeKigpyse5UhbHf36v8HWMH1PPJm2YEXUogwviZQ+72Ox2bhqqBMUnjo4GaTrbpzLIiOaX2cCPrdh5i8nCdSSy5IR1BsBoYb2ZjzawAuBNY3K7NYuCziaOHPgQcdvfaTi4rklOWV9YBMLlY5w5Ibkj5L9XdW83sQWAZEAEec/dKM/tCYv5cYClwM7AVOAbce7plU61JJEjPbdjNuOFFlBR1uJVTJOuk5SeLuy8l/mWfPG1u0rADD3R2WZFcdeBoM6u2H+CL110A1AZdjkin6MxikTR6YVMd0ZjrbGLJKQoCkTRaXrmbUQP7cumoc4IuRaTTFAQiadLQ1MpLVfu48ZJizHRLSskdCgKRNPntlr00t8Z0NrHkHAWBSJo8V7mbIYUFlJUODroUkbOiIBBJg6bWKC9u3sPMicVE8rRZSHKLgkAkDV7Zup+GplYdLSQ5SUEgkgbPbdhNUe98rhk3JOhSRM6agkAkRdGY8/ymOqZfNJze+bq+kOQeBYFIilZvP8CBo806WkhyloJAJEXLKndTkJ9H+YXDgi5FpEsUBCIpcHeWV9bx0fFDKeytq41KblIQiKRgw656dh1q5EZtFpIcpiAQScGzG2qJ5Bk3XFwcdCkiXaYgEOkid2fJ+hquuWAIgwsLgi5HpMsUBCJdtK76MDsPNHLrpJFBlyKSkpSCwMwGm9nzZlaVeB7UQZsxZvaimW0ys0oz+1LSvIfNbJeZvZl43JxKPSKZtGRdDb0iprOJJeelukbwELDC3ccDKxLj7bUCX3X3i4EPAQ+Y2cSk+d9z9ysSD92pTHJCLOY8s76G6yYMZ0DfXkGXI5KSVINgNjA/MTwf+ET7Bu5e6+6vJ4aPAJuAUSm+r0igVm8/QF19Ex+/QpuFJPdZ/HbCXVzY7JC7D0waP+juJ20eSppfCrwEXOru9Wb2MPA5oB5YQ3zN4eAplp0DzAEoLi6evHDhwi7XHZSGhgaKioqCLiPjemK/f1bZxMs1rfzb9H70zu/4aqM9sd+dFda+Z3u/p0+fvtbdy06a4e6nfQAvABs6eMwGDrVre/A0r1MErAVuT5pWDESIr5n8I/DYmepxdyZPnuy56MUXXwy6hED0tH63tEb9yr9f7g8sWHvadj2t32cjrH3P9n4Da7yD79Qzngrp7jecap6Z1ZlZibvXmlkJsOcU7XoBvwIWuPtTSa9dl9Tmx8AzZ6pHJGivvLOfA0ebdbSQ9Bip7iNYDNyTGL4HeLp9A4vfvPWnwCZ3/5d280qSRm8jvqYhktUWr6uhf+98rpugawtJz5BqEDwCzDSzKmBmYhwzG2lmbUcAXQvcDczo4DDR75jZW2a2HpgOfCXFekS6VVNrlGUbdnPjJSPo00uXnJaeIaWrZLn7fuD6DqbXADcnhl8GOtyb5u53p/L+Ipn22y17OdLUqqOFpEfRmcUiZ2HJ+loGFxZwzQW6E5n0HAoCkU461tzKCxvruOnSEfSK6F9Heg79NYt00opNe2hsiepoIelxFAQinbRkXQ3F5/RmSungoEsRSSsFgUgn1B9voWLLXm65bCSRvI7PJBbJVQoCkU5YXllHczTGrZNKztxYJMcoCEQ64ek3dzFmcF+uGDMw6FJE0k5BIHIGO/YfY2XVPj511RjiJ8qL9CwKApEzWLh6B3kGn54yJuhSRLqFgkDkNFqiMX6xppoZFxUzYkCfoMsR6RYKApHTeGFjHfsamrhrmtYGpOdSEIicxhOrdjByQB+umzA86FJEuo2CQOQU2nYSf3rKuTp3QHo0BYHIKbTtJL5jyuigSxHpVgoCkQ4k7yQuGdA36HJEulVKQWBmg83seTOrSjx3eON6M9ueuAHNm2a25myXF8k07SSWMEl1jeAhYIW7jwdWJMZPZbq7X+HuZV1cXiRjtJNYwiTVIJgNzE8Mzwc+keHlRdJu5wHtJJZwSTUIit29FiDxfKqfTw4sN7O1ZjanC8uLZMyTq7STWMLF3P30DcxeAEZ0MOubwHx3H5jU9qC7n7Sd38xGunuNmQ0Hngf+zN1fMrNDnVk+MW8OMAeguLh48sKFC8/YuWzT0NBAUVFR0GVkXC71uzXm/M+KRs4fkMeXJ6d2JnEu9Tvdwtr3bO/39OnT17bbPB/n7l1+AFuAksRwCbClE8s8DHytq8u7O5MnT/Zc9OKLLwZdQiByqd9L19f4ed94xlds2p3ya+VSv9MtrH3P9n4Da7yD79RUNw0tBu5JDN8DPN2+gZkVmln/tmHgRmBDZ5cXySTtJJYwSjUIHgFmmlkVMDMxjpmNNLOliTbFwMtmtg5YBfza3Z873fIiQaisOczKqn3cNU07iSVc8lNZ2N33A9d3ML0GuDkxvA2YdDbLiwTh+yuq6N8nn7uvLg26FJGM0pnFIsCm2nqWVdZx77VjGdC3V9DliGSUgkAE+LffVNG/dz6fv3Zs0KWIZJyCQEJvy+4jLH1rN5+7tpQB/bQ2IOGjIJDQ+/5vqigsiPD5D2ttQMJJQSChVlV3hKVv1XLPNaUM7FcQdDkigVAQSKj922+20rdXhPs+cn7QpYgERkEgobV1TwNL1tfw2atLGVyotQEJLwWBhNYPflNFn/wIf/oR7RuQcFMQSCht29vA4nU13H31eQwp6h10OSKBUhBIKP3gxa0U5Ofxp9o3IKIgkPDZsvsIT79Zwx9NO49h/bU2IKIgkFCJxpxv/Go95/TJ5/7yC4IuRyQrKAgkVB5/ZTtv7jzE3916ifYNiCQoCCQ0dh44xneXbaH8wmHMvmJk0OWIZA0FgYSCu/NXi94iz+Afb7sMM91vQKSNgkBC4anXd7Gyah9fn3URowb2DbockaySUhCY2WAze97MqhLPHd24/kIzezPpUW9mX07Me9jMdiXNuzmVekQ6sq+hiX/49UYmnzeIuz90XtDliGSdVNcIHgJWuPt4YEVi/APcfYu7X+HuVwCTgWPAoqQm32ub7+5L2y8vkqqHF1dyrCnKtz95GXm6BaXISVINgtnA/MTwfOATZ2h/PfCOu7+X4vuKdMoLG+t4Zn0tD84Yx7jh/YMuRyQrmbt3fWGzQ+4+MGn8oLuftHkoaf5jwOvu/oPE+MPA54B6YA3wVXc/eIpl5wBzAIqLiycvXLiwy3UHpaGhgaKioqDLyLig+n2sxfnmy40U9oKHr+lLfobXBsL6eUN4+57t/Z4+ffpady9rP/2MQWBmLwAjOpj1TWB+Z4PAzAqAGuASd69LTCsG9gEO/ANQ4u5/cqbOlJWV+Zo1a87ULOtUVFRQXl4edBkZF0S/W6Mx7vvZGlZW7eNXX7yGK8YMzOj7Q3g/bwhv37O932bWYRDkn2lBd7/hNC9aZ2Yl7l5rZiXAntO81E3E1wbqkl77xLCZ/Rh45kz1iJyJu/M3T1dSsWUv/3TbZYGEgEguSXUfwWLgnsTwPcDTp2n7GeDJ5AmJ8GhzG7AhxXpE+OFv3+HJVTv4YvkF3DXt3KDLEcl6qQbBI8BMM6sCZibGMbORZnbiCCAz65eY/1S75b9jZm+Z2XpgOvCVFOuRkHv6zV1857kt3DppJH9x44VBlyOSE864aeh03H0/8SOB2k+vAW5OGj8GDOmg3d2pvL9Iste27ecv/ns9U0sH890/vFyHiop0ks4slh5h654G5vx8LaMH92XeZyfTOz8SdEkiOUNBIDlv75Em7n18Fb0ixuOfm8rAfrr/sMjZSGnTkEjQquqOcN/P1rD3SBML51zNuUP6BV2SSM5REEjO+s3mOv78yTfp0yvCgvum6TBRkS5SEEjOcXd+9NI2vv3cZiaWnMOPP1vGSF1RVKTLFASSU463RPnLp95i0Ru7uOXyEr77qUn0LdCOYZFUKAgkZ9TVH2fOz9eybuchvjpzAg/OGKcbzIikgYJAsl5LNMaCV9/jey9U0RKNMfePJzPr0o4ufyUiXaEgkKy2smovf79kI1V7Grh23BC+9fFLdDlpkTRTEEhW2r7vKP/r15t4YVMd5w7ux7y7JzNzYrE2BYl0AwWBZJWdB44x/5XtzP/9dgoieXxj1kX8yYdLdaawSDdSEEjgWqIxVmyqY8FrO3h56z4MuO3K0Xxj1oUMP6dP0OWJ9HgKAgnMzgPHWLh6B79YU83eI02UDOjDl64fz6enjKFkgM4LEMkUBYFkTGNzlFXbD7Dy7b2srNrHlroj5BnMuGg4d007l+smDCeiK4aKZJyCQLpN/fEWNtXUs/TdZn6y9TVWbT9Ac2uMgvw8ppYO5varRvHxK0bq179IwBQEkrLjLVF2HWrknT0NbKytZ1NtPRtr69l5oPFEmwnFx7n7Q+fx0QnDmFo6WGcDi2SRlILAzP4QeBi4GJjq7h3eUd7MZgH/CkSAn7h7253MBgP/BZQC24E73P1gKjVJ+rg79Y2t7DvaxP6GZg4cbWJfQzO1hxvZeaCR6oPHqD7YyJ4jTSeWMYOxQwq5fPRA7pxyLhNLzuHw9g18YtZ1AfZERE4n1TWCDcDtwI9O1cDMIsCjxG9VWQ2sNrPF7r4ReAhY4e6PmNlDifFvpFhTjxSLOVF3ojEn5k5rzIlGnZZYjGjMaY3Gp7VEYzS3xmhOPJ8Yb43R2BKlsSXK8ZYYx1uiNDZHOdYcpaGphSPHW+OPplaOHI+PHzrWTEvUT6olkmeMHNiH0QP7UX7hMEYP6sfoQX0pHVrIRSP606/gg39WFbt12wuRbJbqrSo3AWc6yWcqsNXdtyXaLgRmAxsTz+WJdvOBCroxCL6/oorF62qA+K/dZCd/3X1wRtv8tuXeH2+b//4ruCdN9/gcdzje1ESv372QmOe4QywxPxZ7fzx24jk+HI2dsrqU5Bn0K8inqHc+/fvEHwP69mL0oL70753P4MICBhcWMLSoN0OKChhS2PZcQH5EX+4iPUUm9hGMAnYmjVcD0xLDxe5eC+DutWY2/FQvYmZzgDkAxcXFVFRUnHUhB2taGJQXff81O7lcW7v2eWftBizpFZOXaRtu7RujoFc0Pm6J28Ml5uclns0Mw2g7eCbP2j1OTDMieRCx+CPP4r/U8w3y89oedmK4V55RkAcFESiIGL0j8eXeD3EHWhKPJFHgMEQPwx7ij7PV0NDQpc8r14W13xDevudqv88YBGb2AtDRFb6+6e5Pd+I9Ovq+PeufuO4+D5gHUFZW5uXl5Wf7EidWPYJSUVFBV+rOdep3+IS177na7zMGgbvfkOJ7VANjksZHAzWJ4TozK0msDZTQtR+cIiKSgkxs6F0NjDezsWZWANwJLE7MWwzckxi+B+jMGoaIiKRRSkFgZreZWTVwNfBrM1uWmD7SzJYCuHsr8CCwDNgE/MLdKxMv8Qgw08yqiB9V9Egq9YiIyNlL9aihRcCiDqbXADcnjS8FlnbQbj9wfSo1iIhIanQMoIhIyCkIRERCTkEgIhJyCgIRkZCz9pdayAVmthd4L+g6umAosC/oIgKgfodPWPue7f0+z92HtZ+Yk0GQq8xsjbuXBV1Hpqnf4RPWvudqv7VpSEQk5BQEIiIhpyDIrHlBFxAQ9Tt8wtr3nOy39hGIiISc1ghEREJOQSAiEnIKgoCY2dfMzM1saNC1ZIKZ/bOZbTaz9Wa2yMwGBl1TdzKzWWa2xcy2Ju7H3eOZ2Rgze9HMNplZpZl9KeiaMsnMImb2hpk9E3QtZ0tBEAAzG0P8sts7gq4lg54HLnX3y4G3gb8MuJ5uY2YR4FHgJmAi8BkzmxhsVRnRCnzV3S8GPgQ8EJJ+t/kS8Uvt5xwFQTC+B3ydLtyyM1e5+/LEvSkAXiV+p7qeaiqw1d23uXszsBCYHXBN3c7da9399cTwEeJfiqOCrSozzGw0cAvwk6Br6QoFQYaZ2ceBXe6+LuhaAvQnwLNBF9GNRgE7k8arCckXYhszKwWuBF4LuJRM+b/Ef9zFAq6jS1K6MY10zMxeAEZ0MOubwF8BN2a2osw4Xb/d/elEm28S34SwIJO1ZZh1MC00a39mVgT8Cviyu9cHXU93M7OPAXvcfa2ZlQdcTpcoCLqBu9/Q0XQzuwwYC6wzM4hvHnndzKa6++4MltgtTtXvNmZ2D/Ax4Hrv2SewVANjksZHAzUB1ZJRZtaLeAgscPengq4nQ64FPm5mNwN9gHPM7D/d/Y8DrqvTdEJZgMxsO1Dm7tl8tcK0MLNZwL8A17n73qDr6U5mlk98h/j1wC5gNXBX0r26eySL/7qZDxxw9y8HXE4gEmsEX3P3jwVcylnRPgLJlB8A/YHnzexNM5sbdEHdJbFT/EFgGfEdpr/o6SGQcC1wNzAj8Rm/mfiVLFlOawQiIiGnNQIRkZBTEIiIhJyCQEQk5BQEIiIhpyAQEQk5BYGISMgpCEREQu7/AyKbINjxDo2qAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(-5,5,0.2), np.tanh(np.arange(-5,5,0.2))); plt.grid();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inputs x1,x2\n",
    "x1 = Value(2.0, label='x1')\n",
    "x2 = Value(0.0, label='x2')\n",
    "# weights w1,w2\n",
    "w1 = Value(-3.0, label='w1')\n",
    "w2 = Value(1.0, label='w2')\n",
    "# bias of the neuron\n",
    "b = Value(6.8813735870195432, label='b')\n",
    "# x1*w1 + x2*w2 + b\n",
    "x1w1 = x1*w1; x1w1.label = 'x1*w1'\n",
    "x2w2 = x2*w2; x2w2.label = 'x2*w2'\n",
    "x1w1x2w2 = x1w1 + x2w2; x1w1x2w2.label = 'x1*w1 + x2*w2'\n",
    "n = x1w1x2w2 + b; n.label = 'n'\n",
    "o = n.tanh(); o.label = 'o'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 3.0.0 (20220226.1711)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"1575pt\" height=\"210pt\"\n",
       " viewBox=\"0.00 0.00 1575.00 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-206 1571,-206 1571,4 -4,4\"/>\n",
       "<!-- 140307056976896 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140307056976896</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"2.5,-165.5 2.5,-201.5 197.5,-201.5 197.5,-165.5 2.5,-165.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"19\" y=\"-179.8\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"35.5,-165.5 35.5,-201.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"75.5\" y=\"-179.8\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"115.5,-165.5 115.5,-201.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"156.5\" y=\"-179.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
       "</g>\n",
       "<!-- 140307056976608* -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>140307056976608*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"263\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"263\" y=\"-124.8\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140307056976896&#45;&gt;140307056976608* -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>140307056976896&#45;&gt;140307056976608*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M172.53,-165.44C181.84,-162.67 191.2,-159.67 200,-156.5 210.53,-152.71 221.75,-147.9 231.72,-143.33\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"233.25,-146.48 240.82,-139.07 230.28,-140.14 233.25,-146.48\"/>\n",
       "</g>\n",
       "<!-- 140307056979488 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140307056979488</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"326,-55.5 326,-91.5 546,-91.5 546,-55.5 326,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"352.5\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"379,-55.5 379,-91.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"421.5\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"464,-55.5 464,-91.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"505\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140307056977616+ -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>140307056977616+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"609\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"609\" y=\"-96.8\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140307056979488&#45;&gt;140307056977616+ -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>140307056979488&#45;&gt;140307056977616+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M546.27,-90.75C555.64,-92.23 564.6,-93.65 572.65,-94.92\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"572.23,-98.4 582.65,-96.5 573.32,-91.48 572.23,-98.4\"/>\n",
       "</g>\n",
       "<!-- 140307056979488* -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140307056979488*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"263\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"263\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140307056979488*&#45;&gt;140307056979488 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140307056979488*&#45;&gt;140307056979488</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M290.34,-73.5C297.77,-73.5 306.37,-73.5 315.6,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"315.84,-77 325.84,-73.5 315.84,-70 315.84,-77\"/>\n",
       "</g>\n",
       "<!-- 140307056975936 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>140307056975936</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"2,-55.5 2,-91.5 198,-91.5 198,-55.5 2,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"17\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"32,-55.5 32,-91.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"72\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"112,-55.5 112,-91.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"155\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;1.5000</text>\n",
       "</g>\n",
       "<!-- 140307056975936&#45;&gt;140307056979488* -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>140307056975936&#45;&gt;140307056979488*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M198.37,-73.5C208.05,-73.5 217.4,-73.5 225.8,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"225.91,-77 235.91,-73.5 225.91,-70 225.91,-77\"/>\n",
       "</g>\n",
       "<!-- 140307056975984 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>140307056975984</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"4,-110.5 4,-146.5 196,-146.5 196,-110.5 4,-110.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"19\" y=\"-124.8\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"34,-110.5 34,-146.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"74\" y=\"-124.8\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"114,-110.5 114,-146.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"155\" y=\"-124.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140307056975984&#45;&gt;140307056976608* -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>140307056975984&#45;&gt;140307056976608*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M196.05,-128.5C206.52,-128.5 216.65,-128.5 225.71,-128.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"225.79,-132 235.79,-128.5 225.79,-125 225.79,-132\"/>\n",
       "</g>\n",
       "<!-- 140307056976512 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>140307056976512</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1382,-54.5 1382,-90.5 1567,-90.5 1567,-54.5 1382,-54.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1393.5\" y=\"-68.8\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1405,-54.5 1405,-90.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"1445\" y=\"-68.8\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1485,-54.5 1485,-90.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"1526\" y=\"-68.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140307056976512tanh -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>140307056976512tanh</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"1319\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1319\" y=\"-68.8\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
       "</g>\n",
       "<!-- 140307056976512tanh&#45;&gt;140307056976512 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140307056976512tanh&#45;&gt;140307056976512</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1346.04,-72.5C1353.58,-72.5 1362.3,-72.5 1371.57,-72.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1371.81,-76 1381.81,-72.5 1371.81,-69 1371.81,-76\"/>\n",
       "</g>\n",
       "<!-- 140307056978576 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>140307056978576</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-36.5 200,-36.5 200,-0.5 0,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"16.5\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"33,-0.5 33,-36.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"75.5\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;3.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"118,-0.5 118,-36.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"159\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140307056978576&#45;&gt;140307056979488* -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>140307056978576&#45;&gt;140307056979488*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M169.28,-36.5C179.65,-39.61 190.16,-42.98 200,-46.5 210.28,-50.17 221.28,-54.74 231.11,-59.07\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"229.93,-62.37 240.48,-63.27 232.79,-55.99 229.93,-62.37\"/>\n",
       "</g>\n",
       "<!-- 140307056978624 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>140307056978624</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"716,-27.5 716,-63.5 901,-63.5 901,-27.5 716,-27.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"727.5\" y=\"-41.8\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"739,-27.5 739,-63.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"779\" y=\"-41.8\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"819,-27.5 819,-63.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"860\" y=\"-41.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140307056976704+ -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>140307056976704+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"1008\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1008\" y=\"-68.8\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140307056978624&#45;&gt;140307056976704+ -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>140307056978624&#45;&gt;140307056976704+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M901.02,-58.01C926,-61.43 951.59,-64.93 971.37,-67.63\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"971.05,-71.12 981.43,-69.01 972,-64.18 971.05,-71.12\"/>\n",
       "</g>\n",
       "<!-- 140307056977616 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>140307056977616</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"672,-82.5 672,-118.5 945,-118.5 945,-82.5 672,-82.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"725\" y=\"-96.8\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"778,-82.5 778,-118.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"820.5\" y=\"-96.8\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"863,-82.5 863,-118.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"904\" y=\"-96.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140307056977616&#45;&gt;140307056976704+ -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>140307056977616&#45;&gt;140307056976704+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M936.53,-82.49C949.09,-80.71 960.99,-79.02 971.3,-77.56\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"972.03,-80.99 981.44,-76.12 971.05,-74.06 972.03,-80.99\"/>\n",
       "</g>\n",
       "<!-- 140307056977616+&#45;&gt;140307056977616 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>140307056977616+&#45;&gt;140307056977616</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M636.23,-100.5C643.7,-100.5 652.41,-100.5 661.87,-100.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"661.98,-104 671.98,-100.5 661.98,-97 661.98,-104\"/>\n",
       "</g>\n",
       "<!-- 140307056976608 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>140307056976608</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"328.5,-110.5 328.5,-146.5 543.5,-146.5 543.5,-110.5 328.5,-110.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"355\" y=\"-124.8\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"381.5,-110.5 381.5,-146.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"421.5\" y=\"-124.8\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"461.5,-110.5 461.5,-146.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"502.5\" y=\"-124.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140307056976608&#45;&gt;140307056977616+ -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>140307056976608&#45;&gt;140307056977616+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M543.84,-111.01C554.01,-109.34 563.76,-107.74 572.44,-106.32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"573.17,-109.75 582.47,-104.68 572.04,-102.84 573.17,-109.75\"/>\n",
       "</g>\n",
       "<!-- 140307056976608*&#45;&gt;140307056976608 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>140307056976608*&#45;&gt;140307056976608</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M290.34,-128.5C298.51,-128.5 308.08,-128.5 318.36,-128.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"318.39,-132 328.39,-128.5 318.39,-125 318.39,-132\"/>\n",
       "</g>\n",
       "<!-- 140307056976704 -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>140307056976704</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1071,-54.5 1071,-90.5 1256,-90.5 1256,-54.5 1071,-54.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"1082.5\" y=\"-68.8\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1094,-54.5 1094,-90.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"1134\" y=\"-68.8\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1174,-54.5 1174,-90.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"1215\" y=\"-68.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
       "</g>\n",
       "<!-- 140307056976704&#45;&gt;140307056976512tanh -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>140307056976704&#45;&gt;140307056976512tanh</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1256.01,-72.5C1265.01,-72.5 1273.74,-72.5 1281.66,-72.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1281.91,-76 1291.91,-72.5 1281.91,-69 1281.91,-76\"/>\n",
       "</g>\n",
       "<!-- 140307056976704+&#45;&gt;140307056976704 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>140307056976704+&#45;&gt;140307056976704</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1035.04,-72.5C1042.58,-72.5 1051.3,-72.5 1060.57,-72.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1060.81,-76 1070.81,-72.5 1060.81,-69 1060.81,-76\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f9bc8477bb0>"
      ]
     },
     "execution_count": 244,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "draw_dot(o)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 六、自动化：_backward 与拓扑排序\n",
    "【译者导读】手推一遍之后就能体会：每种运算（加法、乘法、tanh）的「本地求导规则」是固定的，完全可以让代码自己做。做法是给每个节点挂一个 `_backward` 闭包，再按**拓扑排序 (topological sort)** 的逆序依次调用——保证算每个节点的梯度时，它的下游都已经算完。下面这些单元既有自动版 `o.backward()`，也保留了视频里逐节点手动调用、手动赋值的推演过程，方便对照理解。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {},
   "outputs": [],
   "source": [
    "o.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Value(data=6.881373587019543),\n",
       " Value(data=2.0),\n",
       " Value(data=-3.0),\n",
       " Value(data=-6.0),\n",
       " Value(data=0.0),\n",
       " Value(data=1.0),\n",
       " Value(data=0.0),\n",
       " Value(data=-6.0),\n",
       " Value(data=0.8813735870195432),\n",
       " Value(data=0.7071067811865476)]"
      ]
     },
     "execution_count": 235,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topo = []\n",
    "visited = set()\n",
    "def build_topo(v):\n",
    "  if v not in visited:\n",
    "    visited.add(v)\n",
    "    for child in v._prev:\n",
    "      build_topo(child)\n",
    "    topo.append(v)\n",
    "build_topo(o)\n",
    "topo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {},
   "outputs": [],
   "source": [
    "o.grad = 1.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [],
   "source": [
    "o._backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {},
   "outputs": [],
   "source": [
    "n._backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {},
   "outputs": [],
   "source": [
    "b._backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {},
   "outputs": [],
   "source": [
    "x1w1x2w2._backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {},
   "outputs": [],
   "source": [
    "x2w2._backward()\n",
    "x1w1._backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [],
   "source": [
    "x1.grad = w1.data * x1w1.grad\n",
    "w1.grad = x1.data * x1w1.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [],
   "source": [
    "x2.grad = w2.data * x2w2.grad\n",
    "w2.grad = x2.data * x2w2.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {},
   "outputs": [],
   "source": [
    "x1w1.grad = 0.5\n",
    "x2w2.grad = 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [],
   "source": [
    "x1w1x2w2.grad = 0.5\n",
    "b.grad = 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [],
   "source": [
    "n.grad = 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [],
   "source": [
    "o.grad = 1.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4999999999999999"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1 - o.data**2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# o = tanh(n)\n",
    "# do/dn = 1 - o**2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 七、一个经典 bug：同一节点被使用多次\n",
    "【译者导读】如果一个节点在图中被用了不止一次（比如 b = a + a），它会从多条路径收到梯度。直接给 `grad` 赋值会互相覆盖，正确做法是**累加**（`+=`，数学上对应多元链式法则）。下面两个例子专门触发这个 bug——这也是所有自动求导框架把梯度设计成累加、并需要 `zero_grad` 清零的原因。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 258,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 3.0.0 (20220226.1711)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"504pt\" height=\"45pt\"\n",
       " viewBox=\"0.00 0.00 504.00 45.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 41)\">\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-41 500,-41 500,4 -4,4\"/>\n",
       "<!-- 140307056784240 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140307056784240</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-36.5 185,-36.5 185,-0.5 0,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"11.5\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">a</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"23,-0.5 23,-36.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"63\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">data 3.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"103,-0.5 103,-36.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"144\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad 2.0000</text>\n",
       "</g>\n",
       "<!-- 140307056785008+ -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140307056785008+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"248\" cy=\"-18.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"248\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140307056784240&#45;&gt;140307056785008+ -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140307056784240&#45;&gt;140307056785008+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M185.01,-18.5C194.01,-18.5 202.74,-18.5 210.66,-18.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"210.91,-22 220.91,-18.5 210.91,-15 210.91,-22\"/>\n",
       "</g>\n",
       "<!-- 140307056785008 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140307056785008</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"311,-0.5 311,-36.5 496,-36.5 496,-0.5 311,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"322.5\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"334,-0.5 334,-36.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"374\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"414,-0.5 414,-36.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"455\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140307056785008+&#45;&gt;140307056785008 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140307056785008+&#45;&gt;140307056785008</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M275.04,-18.5C282.58,-18.5 291.3,-18.5 300.57,-18.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"300.81,-22 310.81,-18.5 300.81,-15 300.81,-22\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f9b983636d0>"
      ]
     },
     "execution_count": 258,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = Value(3.0, label='a')\n",
    "b = a + a   ; b.label = 'b'\n",
    "b.backward()\n",
    "draw_dot(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 3.0.0 (20220226.1711)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"832pt\" height=\"100pt\"\n",
       " viewBox=\"0.00 0.00 832.00 100.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 96)\">\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-96 828,-96 828,4 -4,4\"/>\n",
       "<!-- 140306525506048 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>140306525506048</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-55.5 0,-91.5 194,-91.5 194,-55.5 0,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"11.5\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">a</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"23,-55.5 23,-91.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"65.5\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;2.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"108,-55.5 108,-91.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"151\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;3.0000</text>\n",
       "</g>\n",
       "<!-- 140307056785968+ -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>140307056785968+</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"257\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"257\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
       "</g>\n",
       "<!-- 140306525506048&#45;&gt;140307056785968+ -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>140306525506048&#45;&gt;140307056785968+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M194.01,-73.5C202.98,-73.5 211.65,-73.5 219.51,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"219.7,-77 229.7,-73.5 219.7,-70 219.7,-77\"/>\n",
       "</g>\n",
       "<!-- 140306525506912* -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>140306525506912*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"257\" cy=\"-18.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"257\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140306525506048&#45;&gt;140306525506912* -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>140306525506048&#45;&gt;140306525506912*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M166.84,-55.49C176.04,-52.69 185.3,-49.67 194,-46.5 204.51,-42.67 215.73,-37.84 225.7,-33.28\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"227.23,-36.43 234.81,-29.02 224.27,-30.09 227.23,-36.43\"/>\n",
       "</g>\n",
       "<!-- 140307056785968 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>140307056785968</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"320.5,-55.5 320.5,-91.5 509.5,-91.5 509.5,-55.5 320.5,-55.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"332\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">e</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"343.5,-55.5 343.5,-91.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"383.5\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"423.5,-55.5 423.5,-91.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"466.5\" y=\"-69.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;6.0000</text>\n",
       "</g>\n",
       "<!-- 140307056783712* -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>140307056783712*</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"573\" cy=\"-45.5\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"573\" y=\"-41.8\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
       "</g>\n",
       "<!-- 140307056785968&#45;&gt;140307056783712* -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>140307056785968&#45;&gt;140307056783712*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M509.9,-56.65C519.42,-54.94 528.62,-53.29 536.89,-51.8\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"537.59,-55.23 546.81,-50.02 536.35,-48.34 537.59,-55.23\"/>\n",
       "</g>\n",
       "<!-- 140307056785968+&#45;&gt;140307056785968 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>140307056785968+&#45;&gt;140307056785968</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M284.11,-73.5C291.84,-73.5 300.8,-73.5 310.35,-73.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"310.48,-77 320.48,-73.5 310.48,-70 310.48,-77\"/>\n",
       "</g>\n",
       "<!-- 140306525506912 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>140306525506912</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"320,-0.5 320,-36.5 510,-36.5 510,-0.5 320,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"331.5\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">d</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"343,-0.5 343,-36.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"385.5\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"428,-0.5 428,-36.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"469\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140306525506912&#45;&gt;140307056783712* -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>140306525506912&#45;&gt;140307056783712*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M510.34,-34.83C519.62,-36.44 528.58,-37.99 536.66,-39.38\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"536.27,-42.87 546.72,-41.12 537.46,-35.97 536.27,-42.87\"/>\n",
       "</g>\n",
       "<!-- 140306525506912*&#45;&gt;140306525506912 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>140306525506912*&#45;&gt;140306525506912</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M284.11,-18.5C291.63,-18.5 300.32,-18.5 309.58,-18.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"309.81,-22 319.81,-18.5 309.81,-15 309.81,-22\"/>\n",
       "</g>\n",
       "<!-- 140307056783712 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>140307056783712</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"636,-27.5 636,-63.5 824,-63.5 824,-27.5 636,-27.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"646.5\" y=\"-41.8\" font-family=\"Times,serif\" font-size=\"14.00\">f</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"657,-27.5 657,-63.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"699.5\" y=\"-41.8\" font-family=\"Times,serif\" font-size=\"14.00\">data &#45;6.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"742,-27.5 742,-63.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"783\" y=\"-41.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
       "</g>\n",
       "<!-- 140307056783712*&#45;&gt;140307056783712 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>140307056783712*&#45;&gt;140307056783712</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M600.29,-45.5C607.78,-45.5 616.42,-45.5 625.6,-45.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"625.76,-49 635.76,-45.5 625.76,-42 625.76,-49\"/>\n",
       "</g>\n",
       "<!-- 140306525506432 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>140306525506432</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"2.5,-0.5 2.5,-36.5 191.5,-36.5 191.5,-0.5 2.5,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"14\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"25.5,-0.5 25.5,-36.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"65.5\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">data 3.0000</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"105.5,-0.5 105.5,-36.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"148.5\" y=\"-14.8\" font-family=\"Times,serif\" font-size=\"14.00\">grad &#45;8.0000</text>\n",
       "</g>\n",
       "<!-- 140306525506432&#45;&gt;140307056785968+ -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>140306525506432&#45;&gt;140307056785968+</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M163.94,-36.53C174.08,-39.65 184.37,-43.01 194,-46.5 204.27,-50.22 215.26,-54.8 225.09,-59.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"223.91,-62.43 234.47,-63.32 226.77,-56.04 223.91,-62.43\"/>\n",
       "</g>\n",
       "<!-- 140306525506432&#45;&gt;140306525506912* -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>140306525506432&#45;&gt;140306525506912*</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M191.74,-18.5C201.55,-18.5 211.04,-18.5 219.59,-18.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"219.87,-22 229.87,-18.5 219.87,-15 219.87,-22\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f9ba89cc880>"
      ]
     },
     "execution_count": 259,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = Value(-2.0, label='a')\n",
    "b = Value(3.0, label='b')\n",
    "d = a * b    ; d.label = 'd'\n",
    "e = a + b    ; e.label = 'e'\n",
    "f = d * e    ; f.label = 'f'\n",
    "\n",
    "f.backward()\n",
    "\n",
    "draw_dot(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}