{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 中文汉化版说明\n", "\n", "本 notebook 是 Andrej Karpathy《Neural Networks: Zero to Hero》课程 **第 1 讲（micrograd，下半场）** 配套 notebook 的中文汉化版。\n", "\n", "- 原仓库：[karpathy/nn-zero-to-hero](https://github.com/karpathy/nn-zero-to-hero)（MIT License，Copyright (c) 2022 Andrej Karpathy）\n", "- 原文件：`lectures/micrograd/micrograd_lecture_second_half_roughly.ipynb`\n", "- 视频课程：[The spelled-out intro to neural networks and backpropagation: building micrograd](https://www.youtube.com/watch?v=VMj-3S1tku0)\n", "- 汉化：极客教育（jikejiao.com），依 MIT 许可翻译改编，完整保留原版权声明（见随附 LICENSE 文件）\n", "- 配套中文导学页：\n", "\n", "**译者说明**：原 notebook 是课程视频的粗略代码记录（文件名中的 \"roughly\" 即此意），原版不含任何文字说明单元。为方便中文读者对照视频学习，我们在关键节点插入了标有【译者导读】的说明单元；**所有代码单元与原版完全一致，未做任何改动**。建议配合视频与导学页一起食用。" ] }, { "cell_type": "code", "execution_count": 382, "metadata": {}, "outputs": [], "source": [ "import math\n", "import random\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 一、回顾：数值导数\n", "【译者导读】前几个单元与上半场相同：定义 f(x) = 3x² − 4x + 5，用 (f(x+h) − f(x)) / h 复习导数的定义，再看多输入函数 d = a·b + c 的偏导数。若你刚做完上半场，可以快速跑过。" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "def f(x):\n", " return 3*x**2 - 4*x + 5" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "20.0" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "f(3.0)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAkRklEQVR4nO3dd3yV5f3/8dcnkwQCIRBmwg5hD4mIimhlSAXRWkXpF8E6UGtrbSutddTafq1YW7+ttQ6cqNRRxOLCURwIAhL2StgbQghkEMi+fn9w7I8qSNbJfcb7+XjwOOc+I/f76CNvLu5z3ddtzjlERCS0RHgdQERE6p/KXUQkBKncRURCkMpdRCQEqdxFREJQlNcBAFq2bOk6derkdQwRkaCybNmyg8655JM9FxDl3qlTJzIzM72OISISVMxsx6me02EZEZEQpHIXEQlBKncRkRCkchcRCUEqdxGREHTacjez58zsgJmtPeGxJDP7yMw2+W6bn/Dcr81ss5llm9lF/gouIiKnVp2R+wvA6K89dicwzzmXBszzbWNmvYCrgd6+9zxuZpH1llZERKrltOXunJsPHPraw5cCM3z3ZwCXnfD4q865UufcNmAzMLh+on5TTmEJv39nPYeLy/y1CxGRoFTbY+6tnXP7AHy3rXyPtwd2nfC63b7HvsHMpphZppll5ubm1ipE/tFynl2wjZcXn3Iev4hIWKrvL1TtJI+d9GogzrnpzrkM51xGcvJJz549rfQ2CVyQnsyMRdspKa+s1c8QEQlFtS33HDNrC+C7PeB7fDeQesLrUoC9tY93elOGdeHgkTLeWL7bn7sREQkqtS33t4DJvvuTgTknPH61mcWaWWcgDfiybhG/3dldWtAvpRnPfL6NyipdMlBEBKo3FfIVYBGQbma7zex6YBow0sw2ASN92zjn1gGvA+uB94FbnXN+PV5iZkwZ1oVtB4v5aH2OP3clIhI0TrsqpHNuwimeGn6K1z8APFCXUDU1uncbUpPimD5/C6P7tGnIXYuIBKSQOEM1KjKCG4Z2YfnOfDK3f33WpohI+AmJcge4MiOFxPhonpq/1esoIiKeC5lyj4+JYtKQjvx7Qw5bco94HUdExFMhU+4Ak87pRExkBM98rtG7iIS3kCr3lk1i+f6gFN5YtocDRSVexxER8UxIlTvAjed1obyqihlfbPc6ioiIZ0Ku3Du3bMyoXq15efFOiksrvI4jIuKJkCt3gCnDulJwrJzXlu46/YtFREJQSJb7oI7NObNTc55dsI2Kyiqv44iINLiQLHc4Pnrfk3+Md9fs8zqKiEiDC9lyH96jFV2TGzN9/lac04JiIhJeQrbcIyKOLyi2bm8hn2866HUcEZEGFbLlDnDZwPa0adqIxz7Z7HUUEZEGFdLlHhsVyZRhXfhy2yGWakExEQkjIV3uAFcPTiWpcQx/1+hdRMJIyJd7fEwU1w/tzKfZuazdU+B1HBGRBhHy5Q4wcUhHEmKjePxTjd5FJDyERbk3i4tm0jkdmbt2P5sPFHkdR0TE78Ki3AGuO7czsVERPPGplgMWkdAXNuXeokksEwZ34F8r97Dr0FGv44iI+FXYlDvAlGFdiDCYrkvxiUiIC6tyb9ssju+fkcJrmbs4UKiLeYhI6Aqrcge4+fyuVFRW8cyCbV5HERHxm7Ar904tG3NJ/3a8vHgH+UfLvI4jIuIXYVfuAD+6oBtHyyp5fuF2r6OIiPhFWJZ7epsERvZqzQtfbOeILsUnIiEoLMsd4NbvdKPgWDkzF+/wOoqISL0L23IfkJrI0G4tefrzbRwrq/Q6johIvQrbcge4bXgaB4+UMnOJRu8iElrCutwHd07i3G4tePKzLRq9i0hICetyB7h9RHcOHinjZR17F5EQEvblfmanJIZ2a8mTn23haJlmzohIaKhTuZvZz8xsnZmtNbNXzKyRmSWZ2Udmtsl327y+wvrLz0amkVdcxkuLNHoXkdBQ63I3s/bAbUCGc64PEAlcDdwJzHPOpQHzfNsBbVDHJM5La8lT87dq9C4iIaGuh2WigDgziwLigb3ApcAM3/MzgMvquI8GcfuI7hwqLuNFjd5FJATUutydc3uAPwE7gX1AgXPuQ6C1c26f7zX7gFYne7+ZTTGzTDPLzM3NrW2MejOoY3OGdU9m+vytFOusVREJcnU5LNOc46P0zkA7oLGZTazu+51z051zGc65jOTk5NrGqFc/G5Gm0buIhIS6HJYZAWxzzuU658qB2cA5QI6ZtQXw3R6oe8yGMbBDcy5IT2b6/C1ac0ZEglpdyn0nMMTM4s3MgOHABuAtYLLvNZOBOXWL2LBuH9Gdw0fLmfHFdq+jiIjUWl2OuS8BZgHLgTW+nzUdmAaMNLNNwEjfdtAYkJrId9KTefrzrRSVlHsdR0SkVuo0W8Y5d59zrodzro9z7hrnXKlzLs85N9w5l+a7PVRfYRvK7SO6k3+0XMfeRSRohf0ZqifTPzWR4T1aMX2+Ru8iEpxU7qfw0xFpFBwr5wVdrUlEgpDK/RT6pSQyomdrpn++VddaFZGgo3L/FlMvSudIaQVPfLbF6ygiIjWicv8W6W0SuGxAe15YuJ2cwhKv44iIVJvK/TR+NqI7Vc7x6LxNXkcREak2lftpdGgRz4TBHXht6S62Hyz2Oo6ISLWo3Kvhxxd2Izoygkc+2uh1FBGRalG5V0OrhEZcN7QTb63ay7q9BV7HERE5LZV7NU0Z1pVmcdH86YNsr6OIiJyWyr2amsVFc8sFXfkkO5cvtwXdigoiEmZU7jUw+exOtG4ayx/fz8I553UcEZFTUrnXQFxMJLcNTyNzx2E+yQ6aZepFJAyp3GtofEYqnVrE88f3s6mq0uhdRAKTyr2GoiMj+PmodLL2F/H26r1exxEROSmVey2M7duWXm2b8ucPN1JWUeV1HBGRb1C510JEhDF1dDo7Dx3ltaU7vY4jIvINKvdauqB7MkO6JPGXf2/SBT1EJOCo3GvJzLj74l7kFZfx+KdaElhEAovKvQ76pjTj8oHteXbBNnYfPup1HBGR/1C519EdF6VjwMNalkBEAojKvY7aJcZx43ldmLNyLyt35XsdR0QEULnXi5sv6ErLJrE88O56LUsgIgFB5V4PmsRG8fOR3Vm6/TAfrNvvdRwREZV7fRmfkUJaqyZMm5ulE5tExHMq93oSFRnBXWN6sj3vKC8t3uF1HBEJcyr3enRB92TOS2vJo/M2kX+0zOs4IhLGVO71yMy46+KeFJaU87ePN3sdR0TCmMq9nvVs25Txg1J5cdF2duQVex1HRMKUyt0PfjGqO9GRETz0fpbXUUQkTKnc/aBV00bcNKwr763Zz9Ltut6qiDQ8lbuf3DisM+2aNeK+Oeuo1BWbRKSB1anczSzRzGaZWZaZbTCzs80sycw+MrNNvtvm9RU2mMTHRHHXmJ6s31fIP77Umu8i0rDqOnL/K/C+c64H0B/YANwJzHPOpQHzfNthaUzftpzdpQV//jCbw8WaGikiDafW5W5mTYFhwLMAzrky51w+cCkww/eyGcBldYsYvMyM347rTVFJBX/6UKtGikjDqcvIvQuQCzxvZivM7Bkzawy0ds7tA/DdtjrZm81sipllmllmbm5uHWIEtvQ2CUw6uyP/+HIna/cUeB1HRMJEXco9CjgDeMI5NxAopgaHYJxz051zGc65jOTk5DrECHy3j+hOUnwM9721TqtGikiDqEu57wZ2O+eW+LZncbzsc8ysLYDv9kDdIga/ZnHR/Gp0D5btOMybK/Z4HUdEwkCty905tx/YZWbpvoeGA+uBt4DJvscmA3PqlDBEXDEohf6piTw4N0sX1BYRv6vrbJmfADPNbDUwAPgDMA0YaWabgJG+7bAXEWHcP643uUWlWndGRPwuqi5vds6tBDJO8tTwuvzcUDUgNZGrMlJ5bsE2xmek0q1VE68jiUiI0hmqDWzq6HTiYiK5/219uSoi/qNyb2Atm8Ty85Hd+XzTQT5Yl+N1HBEJUSp3D1wzpCPprRP4/TvrOVZW6XUcEQlBKncPREVGcP+lvdmTf4xHP97kdRwRCUEqd48M6dKCKwal8PT8rWTtL/Q6joiEGJW7h+6+uCdN46K5a/YaqrQssIjUI5W7h5o3juHui3uyfGe+lgUWCUOvL93Fsh2H/fKzVe4eu/yM9pzbrQUPvZ/FgcISr+OISAPZknuEe+as5bmF2/zy81XuHjMz/veyvpRWVHH/2+u9jiMiDaCqyvHr2WtoFBXBfZf08ss+VO4BoHPLxvzkO914d80+Ps7S3HeRUPfq0l18ue0Q94zpRauERn7Zh8o9QNx0fle6tWrCvf9ax9GyCq/jiIif5BSW8OB7GzinawuuzEjx235U7gEiJiqCBy/vy578Y/zl35r7LhKqfjNnLWWVVfzhe30xM7/tR+UeQM7slMSEwak8u2Ab6/bqqk0ioeb9tfv4YF0OPxvZnU4tG/t1Xyr3AHPn6J40jz8+971Sc99FQkbBsXLunbOO3u2acsPQzn7fn8o9wDSLj+besb1YtbuAlxZt9zqOiNSTaXM3cKi4jIe+34+oSP9Xr8o9AI3r345h3ZN5+INsdh066nUcEamjRVvyeOXLXdwwtDN92jdrkH2q3AOQmfGH7/UB4M7Zq7Xuu0gQKymv5K4319AhKZ7bR3RvsP2q3ANUSvN47h7Ti4Wb85i5REsTiASrR+dtYtvBYh68vC9xMZENtl+VewCbMDiV89Ja8of3NujwjEgQWr+3kKfmb+XKQSmc261lg+5b5R7AzIxp3+9HhBm/nLVaK0eKBJGyiiqmzlpF8/ho7h7Ts8H3r3IPcO0T47h7TE8Wbc1j5pIdXscRkWp67ONNrNtbyAPf60tifEyD71/lHgSuPvP44ZkH52bp8IxIEFi1K5+/f7qFy89oz0W923iSQeUeBE48PDN11iodnhEJYCXllfz89ZW0Sojlvkt6e5ZD5R4k2ifGcc+YnizeeoiXdXhGJGD98f1stuQW8/AV/WkWF+1ZDpV7ELnqzFSGdU/mwfey2JFX7HUcEfmaRVvyeG7hNiad3ZGhaQ07O+brVO5BxMx46Pt9iYowpmr2jEhAOVJawR3/XEWnFvHc+d0eXsdRuQebts3iuHdsL77cdogZWntGJGD87zvr2VdwjD+PH0B8TJTXcVTuwejKjBS+k57MtLlZbMwp8jqOSNj7OCuHV5fu4qbzuzKoY3Ov4wAq96BkZvzxiv40iY3itldWUFJe6XUkkbB1uLiMX72xhh5tErh9RJrXcf5D5R6kkhNiefjKfmTtL+Kh97O8jiMStu6ds5b8o2U8Mn4AsVENt3bM6ajcg9iFPVpz7TmdeH7hdj7NPuB1HJGwM2flHt5ZvY+fDk+jV7umXsf5Lyr3IHfnd3uQ3jqBO/65moNHSr2OIxI2duYd5e4313JGh0RuPr+r13G+oc7lbmaRZrbCzN7xbSeZ2Udmtsl3GxjfLoSoRtGRPDphIIUl5Uz95yqt/S7SAMoqqvjJK8uJMHh0wsAGubJSTdVHop8CG07YvhOY55xLA+b5tsWP0tskcPfFPfkkO5cXF+nsVRF/+/OH2azaXcAfr+hHSvN4r+OcVJ3K3cxSgDHAMyc8fCkww3d/BnBZXfYh1TPp7I5c2KMVD7y3gez9mh4p4i+fZh/gqflbmTikA6P7tPU6zinVdeT+F+CXQNUJj7V2zu0D8N22OtkbzWyKmWWaWWZubm4dY8jx6ZH9aNooWtMjRfzkQGEJv3h9FT3aJHDPmF5ex/lWtS53MxsLHHDOLavN+51z051zGc65jOTk5NrGkBO0bBLLn67sR3ZOEdPmanqkSH2qqnL87PWVFJdV8NgPBtIoOnCmPZ5MXUbu5wLjzGw78CpwoZm9DOSYWVsA363m6DWgC9Jbcd25nXnhi+18nJXjdRyRkPHEZ1tYuDmP+8f1plurBK/jnFaty90592vnXIpzrhNwNfCxc24i8BYw2feyycCcOqeUGvnVd9Pp1bYpP399FbsP6+IeInW1bMchHvloI5f0b8f4jFSv41SLP+bvTANGmtkmYKRvWxpQbFQkT0w8g8oqx49mLqe0QsffRWqr4Gg5t72yknaJjXjge30wM68jVUu9lLtz7lPn3Fjf/Tzn3HDnXJrv9lB97ENqpmOLxvz5yv6s3l3A795e73UckaDknOPO2avJKSzhbxPOoGkj7y6+UVOBN/Ne6s2o3m246fwuzFyykzdX7PY6jkjQeeGL7cxdu5+pF6UzIDXR6zg1onIPcVNHpXNW5yR+PXuN5r+L1MCSrXk88O4GRvRszY3ndfE6To2p3ENcVGQEf/vBQBIaRXPLy8soKin3OpJIwNtfUMKt/1hBh6R4HrmqPxERwXGc/UQq9zDQKqERj00YyI5DR/nlrNVaf0bkW5RWVHLLzGUcLavgqWsGBdVx9hOp3MPEWV1a8KvR6cxdu59nF2zzOo5IwPrd2+tZsTOfP13Zn7TWgT+f/VRU7mHkxvO6cFHv1kybm0Xmdk1iEvm615fuYuaSndx8flcu7hu468ZUh8o9jJgZD1/Zn5Tmcdz6j+XkFmn9d5GvrN6dzz1z1jK0W0vuGNXd6zh1pnIPM00bRfP4/wyi4Fg5N72UqQXGRIC8I6Xc/NIykpvEBuz67DUV/J9AaqxXu6Y8Mn4Ay3fmc9fsNfqCVcJaRWUVP3llBQeLy3hy4iCSGsd4HaleqNzD1MV92/Lzkd2ZvWIPT3621es4Ip754wfZfLEljz98ry99U5p5HafeRHkdQLzzkwu7senAEf74QRZdkxszqncbryOJNKhZy3Yz3XfhjSsGpXgdp15p5B7GzIyHr+hHv/bNuP21lazfW+h1JJEGs2hLHr+evZpzu7Xgvkt6ex2n3qncw1yj6EimT8qgaaNobpixVDNoJCxsyT3CzS8vo2OLxjz+P4OIDoEvUL8u9D6R1Fjrpo14elIGh46WaQaNhLxDxWVc98JSoiKM5689k2ZxwXkG6umo3AWAvinNNINGQl5JeSVTXsxkf0EJT0/OIDUp3utIfqNyl/+4uG9bfjbi+AyaJz7b4nUckXrlnOOXs1aTueMwj4wfwBkdmnsdya80W0b+y23Du7E59wgPf5BNSvN4xvVv53UkkXrxfx9t5K1Ve/nl6HTG9AvupQWqQ+Uu/+WrGTQ5BSX84vWVJMXHMDStpdexROrkjWW7efTjzVyVkcot53f1Ok6D0GEZ+YZG0ZE8PTmDrslNuOmlTFbvzvc6kkitLd6ax52zV3NO1xb8bxBdA7WuVO5yUs3ioplx3WAS42P44fNL2Xaw2OtIIjW2fm8hU17MpGOLxjwxMTSnPJ5K+HxSqbHWTRvx0vWDccA1zy7hQGGJ15FEqm3bwWImPbeExrFRvPDD0J3yeCoqd/lWXZKb8Ny1Z3KouIxJz31JoS7TJ0Fgb/4xJj6zhCoHL11/FinNQ3fK46mo3OW0BqQm8uTEQWw+cIQbZ+gkJwlseUdKmfjsEgqPlfPidYPp1qqJ15E8oXKXahnWPZk/j+/Pkm2H+OmrK6is0klOEngKS8qZ9NyX7Dl8jGevPZM+7UNnlceaUrlLtV06oD33ju3FB+tyuOdfOotVAsuxskpueCGT7P1FPHnNIAZ3TvI6kqc0z11q5Pqhnck7Usrjn24hOjKC+8f1DpupZRK4yiqquGXmMpbuOMTfJgzkO+mtvI7kOZW71NjUi9Ipr6zi6c+3YcBvVfDiocoqx89fX8mn2bk8eHlfxvbTWdWgcpdaMDPuurgnzsEzC7YBKnjxRmWV49ezV/PO6n3cdXEPJgzu4HWkgKFyl1oxM+4e0xMHPLtgG2bGfZf0UsFLg6morGLqrNW8uWIPtw1PY8qw8FhWoLpU7lJrZsY9Y3oCxwseUMFLgyivrOL2V1fy7pp93DGqOz++MM3rSAFH5S518lXBOwfPLdyGGfxmrApe/Ke0opIf/2MFH63P4e6Le3LjsC5eRwpIKnepMzPj3rE9cTieX7gdUMGLf5SUV3Lzy8v4NDuX313am0lnd/I6UsCqdbmbWSrwItAGqAKmO+f+amZJwGtAJ2A7MN45d7juUSWQmRm/GdsLgOcXbse54wUfEaGCl/pxtKyCG1/M5IsteTx4eV99eXoadTmJqQL4hXOuJzAEuNXMegF3AvOcc2nAPN+2hIGvCv6GoZ154Yvt3P7aSsoqqryOJSHgSGkF1z63lEVb8vjzlf1V7NVQ65G7c24fsM93v8jMNgDtgUuBC3wvmwF8CvyqTiklaHw1i6ZFk1geej+Lw0fLeGLiIJrE6gig1E7BsXKuff5LVu8u4NEJAzWPvZrqZfkBM+sEDASWAK19xf/VXwAnPVXMzKaYWaaZZebm5tZHDAkQZsYtF3Tl4Sv68cWWPCZMX8zBI6Vex5IgtCf/GFc++QVr9xTw+P+coWKvgTqXu5k1Ad4AbnfOFVb3fc656c65DOdcRnJycl1jSAC6MiOVpycNYtOBIq544gt25h31OpIEkbV7Crjs7wvZV1DCjB8O5qLebbyOFFTqVO5mFs3xYp/pnJvtezjHzNr6nm8LHKhbRAlmF/ZozT9uHEL+sXIuf+L4CEzkdD7JOsD4pxYRExnBG7ecwznddB3fmqp1udvxeW7PAhucc4+c8NRbwGTf/cnAnNrHk1BwRofmzLr5bGIijaunL+aLzQe9jiQBbOaSHdzwYiadWzbmzR+dQ/fWCV5HCkp1GbmfC1wDXGhmK31/LgamASPNbBMw0rctYa5bqwTe+NE5tEtsxLXPL2XOyj1eR5IAU1XlmDY3i7vfXMuwtJa8ftPZtGrayOtYQasus2UWAKeaxDy8tj9XQlfbZnH886ZzmPJSJj99dSXr9xbyy9E9iNRc+LBXUl7J1FmreXvVXn5wVgd+N643UWF0MWt/0H89aVDN4qN5+YazmHR2R56av5Vrn/+S/KNlXscSDx0qLmPSs1/y9qq93PndHjxwWR8Vez3Qf0FpcNGREfzu0j5Mu7wvS7YeYtxjC8naX+2JVhJClu88zJhHP2fl7nz+NmEgN5/fVctW1BOVu3jm6sEdeGXKEErKK7n88S+Yu2af15GkgTjneHHRdq56ahGREcbsW87hkv6aw16fVO7iqUEdm/P2T4aS3iaBW2Yu508fZFOli2+HtKNlFdz+2kp+M2cd56Ul8+5PzgvrC1n7i8pdPNe6aSNenTKEqzJSeeyTzdzwYiYFR8u9jiV+sPnAES59bCFvr9rL1IvSeWZSBs3io72OFZJU7hIQYqMimfb9vvz+sj7M35jLd/86n0Vb8ryOJfXo3dX7uPSxBeQVl/HidWdx63e6adVQP1K5S8AwM64Z0pE3bjmH2OhIfvDMYh6cu0ErSwa50opKfvf2em79x3LS2yTw7m1DGZqmM079TeUuAad/aiLv3jaUq8/swFOfbeV7jy9k84Eir2NJLazZXcAlf1vAcwu3ce05nXh1ytm0bRbndaywoHKXgBQfE8WDl/dl+jWD2FdQwphHF/DSou04py9bg0FZRRWPfJjNZY8vpOBYOc//8Ex+O643MVGqnIaiRbYloI3q3YYBHRKZ+s/V3DtnHZ9k5/LQ9/uRnBDrdTQ5hfV7C/nFP1exYV8hl5/RnvvG9taXph7QX6MS8FolNOKFH57Jby/pxYLNBxn9l/nMXr5bo/gAU15ZxaPzNjHusQXkFpXy9KQMHhk/QMXuEQuEX5CMjAyXmZnpdQwJAhtzipg6azWrduVzVuckfn9ZH60aGAA25hTxi9dXsWZPAeP6t+P+cb1p3jjG61ghz8yWOecyTvqcyl2CTVWV47XMXUybm0VxaQXXn9eZ2y5Mo7Eu5dfgCo6W85d5G3lp0Q6axkXzwGV9+G7ftl7HChvfVu76bZCgExFhTBjcgVG9WvPQ+1k89dlW3lq5l/su6cVFvdtobZIGUFFZxStLd/HIh9nkHyvn6jM7cMeo7rRoou9CAoVG7hL0Mrcf4p5/rSVrfxHnd0/m/nG96dSysdexQtbCzQf53dvryc4pYkiXJH4ztje92jX1OlZY0mEZCXkVlVXMWLSDRz7MpqyyivEZqfz4wm6aU12Pth8s5oH3NvDR+hxSmsdx98U9Gd1H/1LykspdwkZOYQmPztvE65m7MIwfnNWBH13QVVf0qYP9BSVMn7+VlxfvICrSuPU73bh+aGcaRUd6HS3sqdwl7Ow6dJTHPt7MrOW7iYo4vqzBzRd0paWOCVfb9oPFPDV/C7OW7abKweUD2zP1onT9RRlAVO4StnbkFfPXeZv414o9xEZFMvmcTkwZ1oUkTdM7paz9hTz+yRbeWb2XqMgIrspIZcqwLqQmxXsdTb5G5S5hb0vuEf767028vXov0ZERjO3XlolDOjIwNVHHjH2W7zzM459s5t8bDtA4JpKJZ3fk+qGdaZWgkXqgUrmL+GzKKWLGou28uXwPxWWV9G7XlIlDOnLpgHbEx4TfzOCCo+W8tXovs5btZtWufBLjo7nu3M5MPruTziwNAip3ka85UlrBmyv2MHPxDrL2F5EQG8XlZ7Rn4pCOpIX4Ga+VVY6Fmw/yz2W7+WDdfsoqqujRJoHxGalcdWaqTgYLIip3kVNwzrFsx2FeWryDuWv2U1ZZRb+UZozq1ZpRvduQ1qpJyBy22XawmFnLdjF7+R72FZTQLC6aywa044pBqfRp3zRkPmc4UbmLVEPekVJmLdvN3LX7WbkrH4BOLeIZ1bsNo3q1ZmCH5kQG0ZWDSsor+XLbIT7bmMv8jblsOnCECINh3ZO5clAqI3q1IjZK0xmDmcpdpIZyCkv4aH0OH67PYdGWg5RXOlo2iWF4j9YM6ZpE/5REOrdsHFCjXeccW3KLmb8xl8825rJ4ax6lFVXEREVwVuckzu+ezNh+7WjTTF+QhgqVu0gdFJaU82l2Lh+u289n2bkUlVYA0Cwumv6piQxIacaADon0T0lssLVVKqscO/KK2bCviKz9hWzYV8S6vQXsKygBoEtyY4alJXN+ejJDOrcgLkYj9FCkchepJ5VVjk0Hili5M5+Vu47/2ZhTRJXv16h9YhypSXG0S4yjXbPjt20TG9E+MY62zRqR0Kh6M1COllWQd6SMQ8XH/+QVl3GouJQtB4rJ2l9Idk4RJeXHry0bGWF0admYHm2bMqRLEsPSkjUnPUyo3EX8qLi0gjV7Cli5K591ewvZm3+MffnHyCkqpbLqv3+/YqMiiI6MICrSiIowIiOMqIgI361RWlFFXnHpf4r765Iax9CzbQI92jSlR5sEerZtSrdWTbQUQJjSkr8iftQ4NoohXVowpEuL/3q8orKKA0Wl7Cs4xt78EvbmHyOvuIyKSkdlVRUVVY6KSkdF1f/fjomMIKlxDElNYmjROIakxrEkNfbdbxJDQmxUQB3nl8Clchfxk6jIiOOHZxLjGNTR6zQSbnQNVRGREKRyFxEJQX4rdzMbbWbZZrbZzO70135EROSb/FLuZhYJ/B34LtALmGBmvfyxLxER+SZ/jdwHA5udc1udc2XAq8ClftqXiIh8jb/KvT2w64Tt3b7H/sPMpphZppll5ubm+imGiEh48le5n2wi7n+dzeGcm+6cy3DOZSQnJ/sphohIePJXue8GUk/YTgH2+mlfIiLyNX5ZfsDMooCNwHBgD7AU+IFzbt0pXp8L7Kj3IA2jJXDQ6xAeCNfPDeH72fW5A09H59xJD3345QxV51yFmf0Y+ACIBJ47VbH7Xh+0x2XMLPNUazuEsnD93BC+n12fO7j4bfkB59x7wHv++vkiInJqOkNVRCQEqdzrbrrXATwSrp8bwvez63MHkYBYz11EROqXRu4iIiFI5S4iEoJU7vXEzO4wM2dmLb3O0lDM7GEzyzKz1Wb2ppklep3Jn8J1pVMzSzWzT8xsg5mtM7Ofep2pIZlZpJmtMLN3vM5SEyr3emBmqcBIYKfXWRrYR0Af51w/jp+09muP8/hNmK90WgH8wjnXExgC3BpGnx3gp8AGr0PUlMq9fvwf8Eu+tn5OqHPOfeicq/BtLub4MhOhKmxXOnXO7XPOLffdL+J40bX/9neFBjNLAcYAz3idpaZU7nVkZuOAPc65VV5n8dh1wFyvQ/jRaVc6DQdm1gkYCCzxOEpD+QvHB25VHueoMV0guxrM7N9Am5M8dTdwFzCqYRM1nG/77M65Ob7X3M3xf7rPbMhsDey0K52GOjNrArwB3O6cK/Q6j7+Z2VjggHNumZld4HGcGlO5V4NzbsTJHjezvkBnYJWZwfHDEsvNbLBzbn8DRvSbU332r5jZZGAsMNyF9kkTYb3SqZlFc7zYZzrnZnudp4GcC4wzs4uBRkBTM3vZOTfR41zVopOY6pGZbQcynHOBuoJcvTKz0cAjwPnOuZC+4kpNVzoNJXZ85DIDOOScu93jOJ7wjdzvcM6N9ThKtemYu9TFY0AC8JGZrTSzJ70O5C++L46/Wul0A/B6OBS7z7nANcCFvv/PK32jWQlgGrmLiIQgjdxFREKQyl1EJASp3EVEQpDKXUQkBKncRURCkMpdRCQEqdxFRELQ/wOVnmci5YAbRAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "xs = np.arange(-5, 5, 0.25)\n", "ys = f(xs)\n", "plt.plot(xs, ys)" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "2.999378523327323e-06" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "h = 0.000001\n", "x = 2/3\n", "(f(x + h) - f(x))/h" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "4.0\n" ] } ], "source": [ "# les get more complex\n", "a = 2.0\n", "b = -3.0\n", "c = 10.0\n", "d = a*b + c\n", "print(d)" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "d1 4.0\n", "d2 4.0001\n", "slope 0.9999999999976694\n" ] } ], "source": [ "h = 0.0001\n", "\n", "# inputs\n", "a = 2.0\n", "b = -3.0\n", "c = 10.0\n", "\n", "d1 = a*b + c\n", "c += h\n", "d2 = a*b + c\n", "\n", "print('d1', d1)\n", "print('d2', d2)\n", "print('slope', (d2 - d1)/h)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 二、完整版 Value：更多运算与自动反向传播\n", "【译者导读】这是比上半场更完整的 `Value` 类：除了加法、乘法、tanh，还补上了减法、除法、幂运算、`exp` 等，并处理了 `2 * a` 这类「常数在左边」的情况（`__radd__` / `__rmul__`）。每种运算都自带 `_backward` 闭包，`backward()` 里用拓扑排序统一调度，梯度一律**累加**（`+=`）以正确处理节点复用。这个类就是完整的 micrograd 引擎，总共不过百来行。" ] }, { "cell_type": "code", "execution_count": 328, "metadata": {}, "outputs": [], "source": [ "class Value:\n", " \n", " def __init__(self, data, _children=(), _op='', label=''):\n", " self.data = data\n", " self.grad = 0.0\n", " self._backward = lambda: None\n", " self._prev = set(_children)\n", " self._op = _op\n", " self.label = label\n", "\n", " def __repr__(self):\n", " return f\"Value(data={self.data})\"\n", " \n", " def __add__(self, other):\n", " other = other if isinstance(other, Value) else Value(other)\n", " out = Value(self.data + other.data, (self, other), '+')\n", " \n", " def _backward():\n", " self.grad += 1.0 * out.grad\n", " other.grad += 1.0 * out.grad\n", " out._backward = _backward\n", " \n", " return out\n", "\n", " def __mul__(self, other):\n", " other = other if isinstance(other, Value) else Value(other)\n", " out = Value(self.data * other.data, (self, other), '*')\n", " \n", " def _backward():\n", " self.grad += other.data * out.grad\n", " other.grad += self.data * out.grad\n", " out._backward = _backward\n", " \n", " return out\n", " \n", " def __pow__(self, other):\n", " assert isinstance(other, (int, float)), \"only supporting int/float powers for now\"\n", " out = Value(self.data**other, (self,), f'**{other}')\n", "\n", " def _backward():\n", " self.grad += other * (self.data ** (other - 1)) * out.grad\n", " out._backward = _backward\n", "\n", " return out\n", " \n", " def __rmul__(self, other): # other * self\n", " return self * other\n", "\n", " def __truediv__(self, other): # self / other\n", " return self * other**-1\n", "\n", " def __neg__(self): # -self\n", " return self * -1\n", "\n", " def __sub__(self, other): # self - other\n", " return self + (-other)\n", "\n", " def __radd__(self, other): # other + self\n", " return self + other\n", "\n", " def tanh(self):\n", " x = self.data\n", " t = (math.exp(2*x) - 1)/(math.exp(2*x) + 1)\n", " out = Value(t, (self, ), 'tanh')\n", " \n", " def _backward():\n", " self.grad += (1 - t**2) * out.grad\n", " out._backward = _backward\n", " \n", " return out\n", " \n", " def exp(self):\n", " x = self.data\n", " out = Value(math.exp(x), (self, ), 'exp')\n", " \n", " def _backward():\n", " self.grad += out.data * out.grad # NOTE: in the video I incorrectly used = instead of +=. Fixed here.\n", " out._backward = _backward\n", " \n", " return out\n", " \n", " \n", " def backward(self):\n", " \n", " topo = []\n", " visited = set()\n", " def build_topo(v):\n", " if v not in visited:\n", " visited.add(v)\n", " for child in v._prev:\n", " build_topo(child)\n", " topo.append(v)\n", " build_topo(self)\n", " \n", " self.grad = 1.0\n", " for node in reversed(topo):\n", " node._backward()\n", "\n" ] }, { "cell_type": "code", "execution_count": 139, "metadata": {}, "outputs": [], "source": [ "from graphviz import Digraph\n", "\n", "def trace(root):\n", " # builds a set of all nodes and edges in a graph\n", " nodes, edges = set(), set()\n", " def build(v):\n", " if v not in nodes:\n", " nodes.add(v)\n", " for child in v._prev:\n", " edges.add((child, v))\n", " build(child)\n", " build(root)\n", " return nodes, edges\n", "\n", "def draw_dot(root):\n", " dot = Digraph(format='svg', graph_attr={'rankdir': 'LR'}) # LR = left to right\n", " \n", " nodes, edges = trace(root)\n", " for n in nodes:\n", " uid = str(id(n))\n", " # for any value in the graph, create a rectangular ('record') node for it\n", " dot.node(name = uid, label = \"{ %s | data %.4f | grad %.4f }\" % (n.label, n.data, n.grad), shape='record')\n", " if n._op:\n", " # if this value is a result of some operation, create an op node for it\n", " dot.node(name = uid + n._op, label = n._op)\n", " # and connect this node to it\n", " dot.edge(uid + n._op, uid)\n", "\n", " for n1, n2 in edges:\n", " # connect n1 to the op node of n2\n", " dot.edge(str(id(n1)), str(id(n2)) + n2._op)\n", "\n", " return dot\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 三、单个神经元：tanh 整体用 vs 拆开用\n", "【译者导读】还是那个神经元 o = tanh(x1·w1 + x2·w2 + b)。第二个单元故意不用现成的 `tanh`，而是用 exp 和除法把它拆开来算——结果（前向值和所有梯度）完全一致。这说明：自动求导不在乎你在哪个「粒度」上定义运算，只要每个运算的本地导数写对了，链式法则会照顾好其余一切。" ] }, { "cell_type": "code", "execution_count": 318, "metadata": {}, "outputs": [], "source": [ "# inputs x1,x2\n", "x1 = Value(2.0, label='x1')\n", "x2 = Value(0.0, label='x2')\n", "# weights w1,w2\n", "w1 = Value(-3.0, label='w1')\n", "w2 = Value(1.0, label='w2')\n", "# bias of the neuron\n", "b = Value(6.8813735870195432, label='b')\n", "# x1*w1 + x2*w2 + b\n", "x1w1 = x1*w1; x1w1.label = 'x1*w1'\n", "x2w2 = x2*w2; x2w2.label = 'x2*w2'\n", "x1w1x2w2 = x1w1 + x2w2; x1w1x2w2.label = 'x1*w1 + x2*w2'\n", "n = x1w1x2w2 + b; n.label = 'n'\n", "o = n.tanh(); o.label = 'o'\n", "o.backward()" ] }, { "cell_type": "code", "execution_count": 319, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "140307325553680\n", "\n", "x1*w1\n", "\n", "data -6.0000\n", "\n", "grad 0.5000\n", "\n", "\n", "\n", "140307325551664+\n", "\n", "+\n", "\n", "\n", "\n", "140307325553680->140307325551664+\n", "\n", "\n", "\n", "\n", "\n", "140307325553680*\n", "\n", "*\n", "\n", "\n", "\n", "140307325553680*->140307325553680\n", "\n", "\n", "\n", "\n", "\n", "140306250485280\n", "\n", "w1\n", "\n", "data -3.0000\n", "\n", "grad 1.0000\n", "\n", "\n", "\n", "140306250485280->140307325553680*\n", "\n", "\n", "\n", "\n", "\n", "140307325551664\n", "\n", "x1*w1 + x2*w2\n", "\n", "data -6.0000\n", "\n", "grad 0.5000\n", "\n", "\n", "\n", "140307325551424+\n", "\n", "+\n", "\n", "\n", "\n", "140307325551664->140307325551424+\n", "\n", "\n", "\n", "\n", "\n", "140307325551664+->140307325551664\n", "\n", "\n", "\n", "\n", "\n", "140307325553728\n", "\n", "o\n", "\n", "data 0.7071\n", "\n", "grad 1.0000\n", "\n", "\n", "\n", "140307325553728tanh\n", "\n", "tanh\n", "\n", "\n", "\n", "140307325553728tanh->140307325553728\n", "\n", "\n", "\n", "\n", "\n", "140306786257456\n", "\n", "w2\n", "\n", "data 1.0000\n", "\n", "grad 0.0000\n", "\n", "\n", "\n", "140307325550800*\n", "\n", "*\n", "\n", "\n", "\n", "140306786257456->140307325550800*\n", "\n", "\n", "\n", "\n", "\n", "140307325550800\n", "\n", "x2*w2\n", "\n", "data 0.0000\n", "\n", "grad 0.5000\n", "\n", "\n", "\n", "140307325550800->140307325551664+\n", "\n", "\n", "\n", "\n", "\n", "140307325550800*->140307325550800\n", "\n", "\n", "\n", "\n", "\n", "140306250483984\n", "\n", "b\n", "\n", "data 6.8814\n", "\n", "grad 0.5000\n", "\n", "\n", "\n", "140306250483984->140307325551424+\n", "\n", "\n", "\n", "\n", "\n", "140306786256688\n", "\n", "x1\n", "\n", "data 2.0000\n", "\n", "grad -1.5000\n", "\n", "\n", "\n", "140306786256688->140307325553680*\n", "\n", "\n", "\n", "\n", "\n", "140307325551424\n", "\n", "n\n", "\n", "data 0.8814\n", "\n", "grad 0.5000\n", "\n", "\n", "\n", "140307325551424->140307325553728tanh\n", "\n", "\n", "\n", "\n", "\n", "140307325551424+->140307325551424\n", "\n", "\n", "\n", "\n", "\n", "140306786257264\n", "\n", "x2\n", "\n", "data 0.0000\n", "\n", "grad 0.5000\n", "\n", "\n", "\n", "140306786257264->140307325550800*\n", "\n", "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "execution_count": 319, "metadata": {}, "output_type": "execute_result" } ], "source": [ "draw_dot(o)" ] }, { "cell_type": "code", "execution_count": 320, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "140306525507104\n", "\n", " \n", "\n", "data 4.8284\n", "\n", "grad 0.1464\n", "\n", "\n", "\n", "140307325465216*\n", "\n", "*\n", "\n", "\n", "\n", "140306525507104->140307325465216*\n", "\n", "\n", "\n", "\n", "\n", "140306525507104+\n", "\n", "+\n", "\n", "\n", "\n", "140306525507104+->140306525507104\n", "\n", "\n", "\n", "\n", "\n", "140307325464640\n", "\n", " \n", "\n", "data 0.1464\n", "\n", "grad 4.8284\n", "\n", "\n", "\n", "140307325464640->140307325465216*\n", "\n", "\n", "\n", "\n", "\n", "140307325464640**-1\n", "\n", "**-1\n", "\n", "\n", "\n", "140307325464640**-1->140307325464640\n", "\n", "\n", "\n", "\n", "\n", "140307057181792\n", "\n", "x2*w2\n", "\n", "data 0.0000\n", "\n", "grad 0.5000\n", "\n", "\n", "\n", "140307057184240+\n", "\n", "+\n", "\n", "\n", "\n", "140307057181792->140307057184240+\n", "\n", "\n", "\n", "\n", "\n", "140307057181792*\n", "\n", "*\n", "\n", "\n", "\n", "140307057181792*->140307057181792\n", "\n", "\n", "\n", "\n", "\n", "140307325465216\n", "\n", "o\n", "\n", "data 0.7071\n", "\n", "grad 1.0000\n", "\n", "\n", "\n", "140307325465216*->140307325465216\n", "\n", "\n", "\n", "\n", "\n", "140306786144896\n", "\n", "b\n", "\n", "data 6.8814\n", "\n", "grad 0.5000\n", "\n", "\n", "\n", "140307057183664+\n", "\n", "+\n", "\n", "\n", "\n", "140306786144896->140307057183664+\n", "\n", "\n", "\n", "\n", "\n", "140307325467808\n", "\n", " \n", "\n", "data 1.0000\n", "\n", "grad -0.1036\n", "\n", "\n", "\n", "140307325466992+\n", "\n", "+\n", "\n", "\n", "\n", "140307325467808->140307325466992+\n", "\n", "\n", "\n", "\n", "\n", "140306786145184\n", "\n", "x1\n", "\n", "data 2.0000\n", "\n", "grad -1.5000\n", "\n", "\n", "\n", "140306786146720*\n", "\n", "*\n", "\n", "\n", "\n", "140306786145184->140306786146720*\n", "\n", "\n", "\n", "\n", "\n", "140306786147536\n", "\n", "w1\n", "\n", "data -3.0000\n", "\n", "grad 1.0000\n", "\n", "\n", "\n", "140306786147536->140306786146720*\n", "\n", "\n", "\n", "\n", "\n", "140306525528800\n", "\n", " \n", "\n", "data 5.8284\n", "\n", "grad 0.0429\n", "\n", "\n", "\n", "140306525528800->140306525507104+\n", "\n", "\n", "\n", "\n", "\n", "140306525528800->140307325466992+\n", "\n", "\n", "\n", "\n", "\n", "140306525528800exp\n", "\n", "exp\n", "\n", "\n", "\n", "140306525528800exp->140306525528800\n", "\n", "\n", "\n", "\n", "\n", "140306786146720\n", "\n", "x1*w1\n", "\n", "data -6.0000\n", "\n", "grad 0.5000\n", "\n", "\n", "\n", "140306786146720->140307057184240+\n", "\n", "\n", "\n", "\n", "\n", "140306786146720*->140306786146720\n", "\n", "\n", "\n", "\n", "\n", "140307057180928\n", "\n", " \n", "\n", "data 2.0000\n", "\n", "grad 0.2203\n", "\n", "\n", "\n", "140307057184096*\n", "\n", "*\n", "\n", "\n", "\n", "140307057180928->140307057184096*\n", "\n", "\n", "\n", "\n", "\n", "140306786145712\n", "\n", "x2\n", "\n", "data 0.0000\n", "\n", "grad 0.5000\n", "\n", "\n", "\n", "140306786145712->140307057181792*\n", "\n", "\n", "\n", "\n", "\n", "140307057184096\n", "\n", " \n", "\n", "data 1.7627\n", "\n", "grad 0.2500\n", "\n", "\n", "\n", "140307057184096->140306525528800exp\n", "\n", "\n", "\n", "\n", "\n", "140307057184096*->140307057184096\n", "\n", "\n", "\n", "\n", "\n", "140307325466992\n", "\n", " \n", "\n", "data 6.8284\n", "\n", "grad -0.1036\n", "\n", "\n", "\n", "140307325466992->140307325464640**-1\n", "\n", "\n", "\n", "\n", "\n", "140307325466992+->140307325466992\n", "\n", "\n", "\n", "\n", "\n", "140306525505952\n", "\n", " \n", "\n", "data -1.0000\n", "\n", "grad 0.1464\n", "\n", "\n", "\n", "140306525505952->140306525507104+\n", "\n", "\n", "\n", "\n", "\n", "140307057183664\n", "\n", "n\n", "\n", "data 0.8814\n", "\n", "grad 0.5000\n", "\n", "\n", "\n", "140307057183664->140307057184096*\n", "\n", "\n", "\n", "\n", "\n", "140307057183664+->140307057183664\n", "\n", "\n", "\n", "\n", "\n", "140306786145232\n", "\n", "w2\n", "\n", "data 1.0000\n", "\n", "grad 0.0000\n", "\n", "\n", "\n", "140306786145232->140307057181792*\n", "\n", "\n", "\n", "\n", "\n", "140307057184240\n", "\n", "x1*w1 + x2*w2\n", "\n", "data -6.0000\n", "\n", "grad 0.5000\n", "\n", "\n", "\n", "140307057184240->140307057183664+\n", "\n", "\n", "\n", "\n", "\n", "140307057184240+->140307057184240\n", "\n", "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "execution_count": 320, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# inputs x1,x2\n", "x1 = Value(2.0, label='x1')\n", "x2 = Value(0.0, label='x2')\n", "# weights w1,w2\n", "w1 = Value(-3.0, label='w1')\n", "w2 = Value(1.0, label='w2')\n", "# bias of the neuron\n", "b = Value(6.8813735870195432, label='b')\n", "# x1*w1 + x2*w2 + b\n", "x1w1 = x1*w1; x1w1.label = 'x1*w1'\n", "x2w2 = x2*w2; x2w2.label = 'x2*w2'\n", "x1w1x2w2 = x1w1 + x2w2; x1w1x2w2.label = 'x1*w1 + x2*w2'\n", "n = x1w1x2w2 + b; n.label = 'n'\n", "# ----\n", "e = (2*n).exp()\n", "o = (e - 1) / (e + 1)\n", "# ----\n", "o.label = 'o'\n", "o.backward()\n", "draw_dot(o)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 四、与 PyTorch 对照\n", "【译者导读】用 PyTorch 的张量 (tensor) 把同一个神经元再算一遍：`requires_grad=True` 声明需要梯度，`o.backward()` 触发反向传播——API 和我们刚写的 micrograd 一模一样。区别只在于 PyTorch 的基本单位是张量（标量数组）且运算经过高度优化，但背后的原理与这一百多行代码没有本质区别。" ] }, { "cell_type": "code", "execution_count": 369, "metadata": {}, "outputs": [], "source": [ "import torch" ] }, { "cell_type": "code", "execution_count": 376, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.7071066904050358\n", "---\n", "x2 0.5000001283844369\n", "w2 0.0\n", "x1 -1.5000003851533106\n", "w1 1.0000002567688737\n" ] } ], "source": [ "\n", "x1 = torch.Tensor([2.0]).double() ; x1.requires_grad = True\n", "x2 = torch.Tensor([0.0]).double() ; x2.requires_grad = True\n", "w1 = torch.Tensor([-3.0]).double() ; w1.requires_grad = True\n", "w2 = torch.Tensor([1.0]).double() ; w2.requires_grad = True\n", "b = torch.Tensor([6.8813735870195432]).double() ; b.requires_grad = True\n", "n = x1*w1 + x2*w2 + b\n", "o = torch.tanh(n)\n", "\n", "print(o.data.item())\n", "o.backward()\n", "\n", "print('---')\n", "print('x2', x2.grad.item())\n", "print('w2', w2.grad.item())\n", "print('x1', x1.grad.item())\n", "print('w1', w1.grad.item())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 五、迷你神经网络库：Neuron / Layer / MLP\n", "【译者导读】有了引擎，几十行就能搭出一个神经网络库：`Neuron`（若干权重 w、一个偏置 b、tanh 激活）→ `Layer`（一组并列的 Neuron）→ `MLP`（多层感知机，逐层串联）。每个类都实现 `parameters()`，把散落在各处的可训练参数收集成一个列表，供训练时统一更新——PyTorch 的 `nn.Module.parameters()` 就是同样的设计。" ] }, { "cell_type": "code", "execution_count": 592, "metadata": {}, "outputs": [], "source": [ "\n", "class Neuron:\n", " \n", " def __init__(self, nin):\n", " self.w = [Value(random.uniform(-1,1)) for _ in range(nin)]\n", " self.b = Value(random.uniform(-1,1))\n", " \n", " def __call__(self, x):\n", " # w * x + b\n", " act = sum((wi*xi for wi, xi in zip(self.w, x)), self.b)\n", " out = act.tanh()\n", " return out\n", " \n", " def parameters(self):\n", " return self.w + [self.b]\n", "\n", "class Layer:\n", " \n", " def __init__(self, nin, nout):\n", " self.neurons = [Neuron(nin) for _ in range(nout)]\n", " \n", " def __call__(self, x):\n", " outs = [n(x) for n in self.neurons]\n", " return outs[0] if len(outs) == 1 else outs\n", " \n", " def parameters(self):\n", " return [p for neuron in self.neurons for p in neuron.parameters()]\n", "\n", "class MLP:\n", " \n", " def __init__(self, nin, nouts):\n", " sz = [nin] + nouts\n", " self.layers = [Layer(sz[i], sz[i+1]) for i in range(len(nouts))]\n", " \n", " def __call__(self, x):\n", " for layer in self.layers:\n", " x = layer(x)\n", " return x\n", " \n", " def parameters(self):\n", " return [p for layer in self.layers for p in layer.parameters()]\n" ] }, { "cell_type": "code", "execution_count": 665, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Value(data=0.16578526021381612)" ] }, "execution_count": 665, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x = [2.0, 3.0, -1.0]\n", "n = MLP(3, [4, 4, 1])\n", "n(x)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 六、小数据集与训练循环\n", "【译者导读】最后是完整的训练闭环：4 个样本的二分类玩具数据集，损失函数 (loss function) 用均方误差。每一轮迭代做三件事——① 前向传播算预测和 loss；② **先把所有参数的 grad 清零**，再 `loss.backward()` 反向传播；③ 沿负梯度方向更新参数（学习率 0.1）。看着 loss 一路下降、预测逼近目标，你就亲手训练了第一个神经网络。忘记清零梯度是初学者最常见的 bug，值得在这里多看一眼。" ] }, { "cell_type": "code", "execution_count": 666, "metadata": {}, "outputs": [], "source": [ "xs = [\n", " [2.0, 3.0, -1.0],\n", " [3.0, -1.0, 0.5],\n", " [0.5, 1.0, 1.0],\n", " [1.0, 1.0, -1.0],\n", "]\n", "ys = [1.0, -1.0, -1.0, 1.0] # desired targets" ] }, { "cell_type": "code", "execution_count": 682, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0 0.002056123958292787\n", "1 0.0020404768419831024\n", "2 0.0020250564320649566\n", "3 0.002009857924015696\n", "4 0.001994876646733686\n", "5 0.0019801080579702683\n", "6 0.001965547739947282\n", "7 0.0019511913951512907\n", "8 0.0019370348422964524\n", "9 0.0019230740124479978\n", "10 0.001909304945299319\n", "11 0.0018957237855951486\n", "12 0.0018823267796946328\n", "13 0.0018691102722676993\n", "14 0.0018560707031189828\n", "15 0.0018432046041333716\n", "16 0.0018305085963379896\n", "17 0.0018179793870754363\n", "18 0.0018056137672833098\n", "19 0.0017934086088756394\n" ] } ], "source": [ "\n", "for k in range(20):\n", " \n", " # forward pass\n", " ypred = [n(x) for x in xs]\n", " loss = sum((yout - ygt)**2 for ygt, yout in zip(ys, ypred))\n", " \n", " # backward pass\n", " for p in n.parameters():\n", " p.grad = 0.0\n", " loss.backward()\n", " \n", " # update\n", " for p in n.parameters():\n", " p.data += -0.1 * p.grad\n", " \n", " print(k, loss.data)\n", " " ] }, { "cell_type": "code", "execution_count": 683, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Value(data=0.9817830812439714),\n", " Value(data=-0.9863881624765284),\n", " Value(data=-0.9766534529377958),\n", " Value(data=0.9729591216966093)]" ] }, "execution_count": 683, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ypred" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 4 }