矩阵微分与矩阵导数的关系

标量矩阵的求导，定义为$$\frac{\partial f}{\partial X}=\left[\frac{\partial f}{\partial X_{ij}}\right]$$即$f$逐元素求导排成与$X$相同的矩阵。

一元微积分中的导数与微分的关系$df=f^{\prime }(x)dx$，多元微积分中的梯度(标量对向量的导数)也与微分有联系$$df=\sum _{i=1}^n\frac{\partial f}{\partial x_i}d{x}_i={\left[\frac{\partial f}{\partial \mathbf{x}}\right]}^{T}d\mathbf{x}$$这里第一个等号是全微分公式，第二个等号表达了梯度与微分的联系：

全微分$df$是梯度向量$\frac{\partial f}{\partial \mathbf{x}}(n\times 1)$与微分向量$d\mathbf{x}(n\times 1)$的内积。

受此启发，我们将矩阵导数与微分建立联系：
$$df=\sum _{i=1}^m\sum _{j=1}^n\frac{\partial f}{\partial X_{ij}}d{X}_{ij}=\text{tr}\left({\left[\frac{\partial f}{\partial X}\right]}^{T}dX\right)$$其中$\text{tr}$代表迹（trace）是方阵对角线元素之和，且有性质，对尺寸相同的矩阵$A,B,\text{tr}(A^{T}B)={\sum }_{i,j}{A}_{ij}{B}_{ij}$，即$\text{tr}(A^{T}B)$是矩阵$A,B$的内积。

与梯度相似，这里第一个等号是全微分公式，第二个等号表达了矩阵导数与微分的联系：全微分$df$是导数$\frac{\partial f}{\partial X}(m\times n)$与微分矩阵$dX(m\times n)$的内积。

矩阵微分的运算法则

想遇到较复杂的一元函数如$f=\log (2+\sin x)e^{\sqrt{x}}$，我们是如何求导的呢？通常不是从定义开始求极限，而是先建立了初等函数求导和四则运算、复合等法则，再来运用这些法则。故而，我们来创立常用的矩阵微分的运算法则：

• 加减法：$d(X\pm Y)=dX\pm dY$
• 矩阵乘法： $d(XY)=d(X)Y+XdY$
• 转置：$d(X^T)=(dX{)}^T$
• 迹：$d\text{tr}(X)=\text{tr}(dX)$
• 逆：$d{X}^{-1}=-X^{-1}dX{X}^{-1}$，此式可在$X{X}^{-1}=I$两侧求微分证明。
• 行列式$d|X|=\text{tr}(X^{\#}dX)$，其中$X^{\#}$表示$X$的伴随矩阵，在$X$可逆时又可写作$d|X|=|X|\text{tr}(X^{-1}dX)$。此式可用Laplace展开来证明，详见张贤达《矩阵分析与应用》第279页。
• 逐元素乘法：$d(X\odot Y)=dX\odot Y+X\odot dY$，$\odot$表示尺寸相同的矩阵$X,Y$逐元素相乘。
• 逐元素函数：$d\sigma (X)={\sigma }^{\prime }(X)\odot dX,\sigma (X)=\left[\sigma (X_{ij})\right]$逐元素标量函数运算，${\sigma }^{\prime }(X)=\left[{\sigma }^{\prime }(X_{ij})\right]$是逐元素求导数。例如
  $$X=\left[\begin{array}{cc}{X}_{11}& X_{12}\\ X_{21}& X_{22}\end{array}\right],d\sin (X)=\left[\begin{array}{cc}\cos X_{11}d{X}_{11}& \cos X_{12}d{X}_{12}\\ \cos X_{21}d{X}_{21}& \cos X_{22}d{X}_{22}\end{array}\right]=\cos (X)\odot d(X)$$

利用迹求矩阵的导数

我们试图利用矩阵导数与微分的联系$df=\text{tr}\left[{\left(\frac{\partial f}{\partial X}\right)}^{T}dX\right]$，在求出左侧的微分$df$后，该如何写成右侧的形式并得到导数呢？这需要一些迹技巧(trace trick)：

• 标量套上迹：$a=\text{tr}(a)$
• 转置：$\text{tr}(A^T)=\text{tr}(A)$
• 线性：$\text{tr}(A\pm B)=\text{tr}(A)\pm \text{tr}(B)$
• 矩阵乘法交换：$\text{tr}(AB)=\text{tr}(BA)$，其中$A$与$B^T$尺寸相同。两侧都等于${\sum }_{ij}{A}_{ij}{B}_{ij}$。
• 矩阵乘法/逐元素乘法交换：$\text{tr}(A^T(B\odot C))=\text{tr}((A\odot B{)}^{T}C)$，其中$A,B,C$尺寸相同。两侧都等于${\sum }_{ij}{A}_{ij}{B}_{ij}{C}_{ij}$。$$\begin{array}{rl}{A}^T(B\odot C)& =\left[\begin{array}{cccc}{a}_{11}& a_{21}& \dots & a_{n1}\\ a_{12}& a_{22}& \dots & a_{n2}\\ ⋮& ⋮& \ddots & ⋮\\ a_{1n}& a_{2n}& \dots & a_{nn}\end{array}\right]\left[\begin{array}{cccc}{b}_{11}{c}_{11}& b_{12}{c}_{12}& \dots & b_{1n}{c}_{1n}\\ b_{21}{c}_{21}& b_{22}{c}_{22}& \dots & b_{2n}{c}_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ b_{n1}{c}_{n1}& b_{n2}{c}_{n2}& \dots & b_{nn}{c}_{nn}\end{array}\right]\\ & =\left[\begin{array}{cccc}{\sum }_{j=1}{a}_{j1}{b}_{j1}{c}_{j1}& {\sum }_{j=1}{a}_{j1}{b}_{j2}{c}_{j2}& \dots & {\sum }_{j=1}{a}_{j1}{b}_{jn}{c}_{jn}\\ {\sum }_{j=1}{a}_{j2}{b}_{j1}{c}_{j1}& {\sum }_{j=1}{a}_{j2}{b}_{j2}{c}_{j2}& \dots & {\sum }_{j=1}{a}_{j2}{b}_{jn}{c}_{jn}\\ ⋮& ⋮& \ddots & ⋮\\ {\sum }_{j=1}{a}_{jn}{b}_{j1}{c}_{j1}& {\sum }_{j=1}{a}_{jn}{b}_{j2}{c}_{j2}& \dots & {\sum }_{j=1}{a}_{jn}{b}_{jn}{c}_{jn}\end{array}\right]\end{array}$$
  容易推得$$\text{tr}\left[A^T(B\odot C)\right]=\sum _{i=1}\sum _{j=1}{a}_{ji}{b}_{ji}{c}_{ji}=\text{tr}\left[(A\odot B{)}^{T}C\right]$$
  观察一下可以断言：

若标量函数$f$是矩阵$X$经加减乘法、逆、行列式、逐元素函数等运算构成，则使用相应的运算法则对$f$求微分，再使用迹技巧给$df$套上迹并将其它项交换至$dX$左侧，对照导数与微分的联系$df=\text{tr}\left[{\left(\frac{\partial f}{\partial X}\right)}^{T}dX\right]$，即能得到导数。
特别地，若矩阵退化为向量，对照导数与微分的联系$df={\left(\frac{\partial f}{\partial \mathbf{x}}\right)}^{T}d\mathbf{x}$，即能得到导数。

在建立法则的最后，来谈一谈复合：假设已求得$\frac{\partial f}{\partial Y}$，而$Y$是$X$的函数，如何求$\frac{\partial f}{\partial X}$呢？在微积分中有标量求导的链式法则$\frac{\partial f}{\partial x}=\frac{\partial f}{\partial y}\frac{\partial y}{\partial x}$，但这里我们不能随意沿用标量的链式法则，因为矩阵对矩阵的导数$\frac{\partial Y}{\partial X}$截至目前仍是未定义的。于是我们继续追本溯源，链式法则是从何而来？源头仍然是微分。我们直接从微分入手建立复合法则：先写出[公式]，再将$dY$用$dX$表示出来代入，并使用迹技巧将其他项交换至$dX$左侧，即可得到$\frac{\partial f}{\partial X}$。

一些例子

例1

$f=a^{T}Xb$，求$\frac{\partial f}{\partial X}$。
其中$a$是$m\times 1$列向量，$X$是$m\times n$矩阵，$b$是$n\times 1$向量，$f$是标量。
求微分
$df=d{a}^{T}Xb+a^{T}dXb+a^{T}Xdb=a^{T}dXb$，这里$a,b$为常量，所以$da=0,db=0$
$df=tr(df)=tr(a^{T}dXb)=tr(b{a}^{T}dX)=tr((a{b}^T{)}^{T}dX)$，得$\frac{\partial f}{\partial X}=a{b}^T$

例2

$f=a^T\exp (Xb)$，求$\frac{\partial f}{\partial X}$。其中$a$是$m\times 1$列向量，$X$是$m\times n$矩阵，$b$是$n\times 1$向量，$\exp$表示逐元素求指数，$f$是标量。
求微分
$$\begin{gathered} df=a^T(\exp (Xb)\odot (dXb))=\text{tr}(df)=\text{tr}\left((a\odot \exp (Xb){)}^{T}dXb\right)=\text{tr}\left({\left(a\odot \exp (Xb))b^T\right)}^{T}dX\right) \\ \Rightarrow \frac{\partial f}{\partial X}=(a\odot \exp (Xb))b^T \end{gathered}$$

例3

$f=\text{tr}(Y^{T}MY),Y=\sigma (WX)$，求$\frac{\partial f}{\partial X}$。
其中$W$是$l\times m$列向量，$X$是$m\times n$矩阵，$Y$是$l\times n$向量，$M$是$l\times l$对称矩阵，$\sigma$表示逐元素求指数，$f$是标量。
求微分$df=\text{tr}((dY{)}^{T}MY)+\text{tr}(Y^{T}MdY)=\text{tr}(Y^T{M}^{T}dY+Y^{T}MdY)=\text{tr}(Y^T(M^T+M)dY)$
得导数$\frac{\partial f}{\partial Y}=(M+M^T)Y=2MY$

继续求微分
$$\begin{array}{rl}df& =\text{tr}({\frac{\partial f}{\partial Y}}^{T}dY)=\text{tr}\left({\frac{\partial f}{\partial Y}}^T(1-\sigma (WX))\odot WdX\right)\\ & =\text{tr}\left({\left(\frac{\partial f}{\partial Y}\odot (1-\sigma (WdX))\right)}^{T}WdX\right)=\text{tr}\left({\left(W^T\frac{\partial f}{\partial Y}\odot (1-\sigma (WX))\right)}^{T}dW\right)\\ & \Rightarrow \frac{\partial f}{\partial X}=W^T\frac{\partial f}{\partial Y}\odot (1-\sigma (WX))\end{array}$$

线性回归

$l=\Vert Xw-y{\Vert }^2$，求$w$的最小二乘估计，即求$\frac{\partial l}{\partial w}$的零点。
其中$y$是$m\times 1$列向量，$X$是$m\times n$矩阵，$w$是$n\times 1$列向量，$l$是标量。

这是标量对向量的导数，不过可以把向量看做矩阵的特例。
$l=(Xw-y{)}^T(Xw-y)$

求微分，使用矩阵乘法、转置等法则：
$dl=(Xdw{)}^T(Xw-y)+(Xw-y{)}^T(Xdw)$

由两个列向量满足性子：$u^{T}v=v^{T}u$
$dl=2(Xw-y{)}^T(Xdw)=\text{tr}(dl)=\text{tr}\left({\left(2X^T\left(Xw-y\right)\right)}^{T}dw\right)$
根据标量对向量的微分公式：
$\frac{\partial l}{\partial w}=2X^T\left(Xw-y\right)=0\Rightarrow w=(X^{T}X{)}^{-1}{X}^{T}y$

线性规划的扩展

$l=\frac{1}{N}{\sum }_{i=1}^N\Vert x_{i}w+b-y_i{\Vert }^2$，求$w,b$的最小二乘估计，即求$\frac{\partial l}{\partial w},\frac{\partial l}{\partial b}$的零点。
其中$y_i$是$n\times 1$行向量，$x_i$是$m\times 1$列向量，$w$是$m\times n$矩阵，$l$是标量。
将$a_i=x_{i}w+b$带入得：$l=\frac{1}{N}{\sum }_{i=1}^N\Vert a_i-y_i{\Vert }^2$
求全微分：
$dl=d\left(\frac{1}{N}{\sum }_{i=1}^N(a_i-y_i{)}^T(a_i-y_i)\right)=\frac{1}{N}{\sum }_{i=1}^{N}2(a_i-y_i{)}^{T}d{a}_i$
得到
$\frac{\partial l}{\partial a_i}=\frac{2}{N}(a_i-y_i)$

根据链式求导法则：
$dl={\sum }_{i=1}^N\text{tr}\left({\left(\frac{\partial l}{\partial a_i}\right)}^T(x_{i}dw+db)\right)={\sum }_{i=1}^N\text{tr}\left({\left(\frac{\partial l}{\partial a_i}\right)}^T{x}_{i}dw+{\left(\frac{\partial l}{\partial a_i}\right)}^{T}db\right)$
得$w,b$的偏导
$\frac{\partial l}{\partial b}={\sum }_{i=1}^N\frac{2}{N}(x_{i}w+b-y_i)=0\Rightarrow b={\mu }_y-{\mu }_{x}w$
$$\begin{array}{rl}\frac{\partial l}{\partial w}& =\sum _{i=1}^N{x}_i^T\frac{\partial l}{\partial a_i}\\ & =\frac{2}{N}\sum _{i=1}^N{x}_i^T(x_{i}w+b-y_i)\\ & =\frac{2}{N}\sum _{i=1}^N{x}_i^T(x_{i}w+{\mu }_y-{\mu }_{x}w-y_i)\\ & =\frac{2}{N}\sum _{i=1}^N(x_i-{\mu }_x{)}^T((x_i-{\mu }_x)w+{\mu }_y-y_i)=0\frac{2}{N}\sum _{i=1}^N{\mu }_x^T(x_{i}w+{\mu }_y-{\mu }_{x}w-y_i))=0\\ & \Rightarrow w={\Sigma }_{xx}^{-1}{\Sigma }_{xy},b={\mu }_y-{\mu }_x{\Sigma }_{xx}^{-1}{\Sigma }_{xy}\end{array}$$

多维正态分布

m维随机向量可以写作：
$$p(\mathbf{x})=(2\pi {)}^{-m/2}|\Sigma {|}^{-1/2}{e}^{-1/2(\mathbf{x}-\mu {)}^T{\Sigma }^{-1}(\mathbf{x}-\mu )}$$
极大似然估计$$\begin{array}{rl}l& =\log \prod _{i=1}^{N}p(x_i)\\ & =\log \left[(2\pi {)}^{-mN/2}|\Sigma {|}^{-N/2}{e}^{{\sum }_{i=1}^N-1/2(x_i-\mu {)}^T{\Sigma }^{-1}(x_i-\mu )}\right]\\ & =-\frac{mN}{2}\log (2\pi )-\left[\frac{N}{2}|\Sigma |+\frac{N}{2}\cdot \frac{1}{N}\sum _{i=1}^N(x_i-\mu {)}^T{\Sigma }^{-1}(x-\mu )\right]\\ & =-\frac{mN}{2}\log (2\pi )-\frac{N}{2}\left[|\Sigma |+\cdot \frac{1}{N}\sum _{i=1}^N(x_i-\mu {)}^T{\Sigma }^{-1}(x-\mu )\right]\end{array}$$
其中$x_i$表示$m\times 1$维向量，$\mu$表示$m\times 1$维均值向量。
所以，只需要求下式极值：$$l=|\Sigma |+\frac{1}{N}\sum _{i=1}^N(x_i-\mu {)}^T{\Sigma }^{-1}(x-\mu )$$ 求微分，先看$\log |\Sigma |$
$d\log |\Sigma |=|\Sigma {|}^{-1}d|\Sigma |=|\Sigma {|}^{-1}|\Sigma |\text{tr}({\Sigma }^{-1}d\Sigma )=tr({\Sigma }^{-1}d\Sigma )$
第二项求微分
$$\begin{array}{rl}d\left[\frac{1}{N}\sum _{i=1}^N(x_i-\mu {)}^T{\Sigma }^{-1}(x-\mu )\right]& =-\frac{1}{N}\sum _{i=1}^N(x_i-\mu {)}^T{\Sigma }^{-1}d\Sigma {\Sigma }^{-1}(x_i-\mu )\\ & =-\frac{1}{N}\sum _{i=1}^N\text{tr}({\Sigma }^{-1}(x_i-\mu )(x_i-\mu {)}^T{\Sigma }^{-1}d\Sigma )\\ & =-\text{tr}({\Sigma }^{-1}{\Sigma }_{xx}{\Sigma }^{-1}d\Sigma )\end{array}$$
两项微分相加
$$\begin{array}{rl}dl& =\text{tr}({\Sigma }^{-1}d\Sigma -{\Sigma }^{-1}{\Sigma }_{xx}{\Sigma }^{-1}d\Sigma )=\text{tr}\left(\left({\Sigma }^{-1}-\Sigma {\Sigma }_{xx}{\Sigma }^{-1}\right)d\Sigma \right)\\ & \Rightarrow \frac{\partial l}{\partial \Sigma }={\left({\Sigma }^{-1}-{\Sigma }^{-1}{\Sigma }_{xx}{\Sigma }^{-1}\right)}^T\\ & \Rightarrow \frac{\partial l}{\partial \Sigma }=0\Rightarrow \Sigma ={\Sigma }_{xx}\end{array}$$

多元逻辑回归

$l=-y^T\log \text{softmax}(Wx)$，求$\frac{\partial l}{\partial W}$。
其中$y$是除一个元素为1其它元素为0的$m\times 1$列向量，$W$是$m\times n$矩阵，$x$是$n\times 1$列向量，$l$是标量。
$\log$表示自然对数，$\text{softmax}(a)=\frac{\exp (a)}{1^T\exp (a)}$，其中$\text{exp}$表示逐元素求指数，$1$代表全1向量。
令$a=Wx$，将$\text{softmax}$带入求微分
$$\begin{array}{rl}dl& =d\left[-y^T(a-1\log (1^T\exp (a)))\right]\\ & =d\left[-y^{T}a+\log (1^T\exp (a))\right]\\ & =\frac{1^T\exp (a)\odot da}{1^T\exp (a)}-y^{T}da\\ & =\text{tr}(\frac{1^T\exp (a)\odot da}{1^T\exp (a)}-y^{T}da)\\ & =\text{tr}(\frac{\exp (a{)}^{T}da}{1^T\exp (a)}-y^{T}da)\\ & =\text{tr}\left((\text{softmax(a)}-y{)}^{T}da\right)\\ & \Rightarrow \frac{\partial l}{\partial a}=\text{softmax}(a)-y\end{array}$$
将$a=Wx$带入，求微分
$$\begin{array}{rl}dl& =\text{tr}\left({\frac{\partial l}{\partial a}}^{T}da\right)=\text{tr}\left({\frac{\partial l}{\partial a}}^{T}dWx\right)=\text{tr}\left({\left(\frac{\partial l}{\partial a}{x}^T\right)}^{T}dx\right)\\ & \Rightarrow \frac{\partial l}{\partial w}=\frac{\partial l}{\partial a}{x}^T=\left(\text{softmax}(Wx)-y\right)x^T\end{array}$$

二层神经网络

$l=-y^T\log \text{softmax}(W_2\sigma (W_{1}x))$，求$\frac{\partial l}{\partial W_1},\frac{\partial l}{\partial W_2}$。
其中$y$是除一个元素为$1$外其它元素为$0$的$m\times 1$列向量，$W_2$是$m\times p$矩阵，$W_1$是$p\times n$矩阵，$x$是$n\times 1$列向量，$l$是标量。
$\log$表示自然对数，$\text{softmax}(a)=\frac{\exp (a)}{1^T\exp (a)}$，$\sigma$是逐元素$\text{sigmoid}$函数$\sigma (a)=\frac{1}{1+\exp (-a)}$

令$a_2=W_2\sigma (W_{1}x)$，根据上面的逻辑回归求导结果
$\frac{\partial l}{\partial a_2}=\text{softmax}(a_2)-y$

令$a_1=\sigma (W_{1}x)$继续求微分
$dl=\text{tr}({\frac{\partial l}{\partial a_2}}^{T}d{a}_2)=\text{tr}\left({\frac{\partial l}{\partial a_2}}^{T}d{W}_2{a}_1+{\frac{\partial l}{\partial a_2}}^T{W}_{2}d{a}_1\right)$
得$\frac{\partial l}{\partial W_2}=\frac{\partial l}{\partial a_2}{a}_1^T$

继续求解第二部分$$\begin{array}{rl}\text{tr}({\frac{\partial l}{\partial a_2}}^T{W}_{2}d{a}_1)& =\text{tr}\left({\frac{\partial l}{\partial a_2}}^T{W}_{2}d\sigma (W_{1}x)\right)\\ & =\text{tr}\left({\frac{\partial l}{\partial a_2}}^T{W}_2(1-\sigma (W_{1}x))\odot d{W}_{1}x\right)\\ & =\text{tr}\left({\left(W_2^T\frac{\partial l}{\partial a_2}\right)}^T(1-\sigma (W_{1}x))\odot d{W}_{1}x\right)\\ & =\text{tr}\left({\left(W_2^T\frac{\partial l}{\partial a_2}\odot (1-\sigma (W_{1}x))\right)}^{T}d{W}_{1}x\right)\\ & =\text{tr}\left({\left(W_2^T\frac{\partial l}{\partial a_2}\odot (1-\sigma (W_{1}x))x^T\right)}^{T}d{W}_1\right)\\ & \Rightarrow \frac{\partial l}{\partial W_1}=W_2^T\frac{\partial l}{\partial a_2}\odot (1-\sigma (W_{1}x))x^T\end{array}$$
推广：样本$(x_1,y_1),\dots ,(x_N,y_N)$，$l=-{\sum }_{i=1}^N{y}_i^T\log \text{softmax}(W_2\sigma (W_1{x}_i+b_1)+b_2)$。其中$b_1$是$p\times 1$列向量，$b_2$是$m\times 1$列向量。

解1：定义$a_{1,i}=W_1{x}_i+b_1,a_{2,i}=W_2\sigma (a_1)+b_2$。
则$l=-{\sum }_{i=1}^N{y}^T\log \text{softmax}(a_{2,i})$，同上得$\frac{\partial l}{\partial a_{2,i}}=\text{softmax}(a_{2,i})-y_i$，则
$$\begin{array}{rl}dl& =\sum _{i=1}^N\text{tr}({\frac{\partial l}{\partial a_{2,i}}}^{T}d{a}_{2,i})\\ & =\sum _{i=1}^N\text{tr}\left({\frac{\partial l}{\partial a_{2,i}}}^{T}d{W}_2\sigma (a_{1,i})+{\frac{\partial l}{\partial a_{2,i}}}^T{W}_{2}d\sigma (a_{1,i})+{\frac{\partial l}{\partial a_{2,i}}}^{T}d{b}_2\right)\\ & \Rightarrow \frac{\partial l}{\partial W_2}=\sum _{i=1}^N\frac{\partial l}{\partial a_{2,i}}\sigma (a_{1,i}{)}^T,\frac{\partial l}{\partial b_2}=\sum _{i=1}^N\frac{\partial l}{\partial a_{2,i}}\end{array}$$继续求解$\frac{\partial l}{\partial W_1},\frac{\partial l}{\partial b_1}$
$$\begin{array}{rl}d{l}_2& =\sum _{i=1}^N\text{tr}({\frac{\partial l}{\partial a_{2,i}}}^T{W}_{2}d\sigma (a_{1,i}))\\ & =\sum _{i=1}^N\text{tr}({\frac{\partial l}{\partial a_{2,i}}}^T{W}_2(1-\sigma (a_{1,i}))\odot d{a}_{1,i})\\ & =\sum _{i=1}^N\text{tr}\left({\left(W_2^T\frac{\partial l}{\partial a_{2,i}}\odot (1-\sigma (a_{1,i}))\right)}^{T}d{a}_{1,i}\right)\\ & \Rightarrow \frac{\partial l}{\partial a_{1,i}}=\left(W_2^T\frac{\partial l}{\partial a_{2,i}}\odot (1-\sigma (a_{1,i}))\right)\end{array}$$则有
$$\begin{array}{rl}d{l}_2& =\sum _{i=1}^N\text{tr}\left({\frac{\partial l}{\partial a_{1,i}}}^{T}d{a}_{1,i}\right)\\ & =\sum _{i=1}^N\text{tr}\left({\frac{\partial l}{\partial a_{1,i}}}^{T}d{W}_1{x}_i+{\frac{\partial l}{\partial a_{1,i}}}^{T}d{b}_1\right)\\ & \Rightarrow \frac{\partial l}{\partial W_1}=\sum _{i=1}^N\frac{\partial l}{\partial a_{1,i}}{x}_i^T,\frac{\partial l}{\partial b_2}=\sum _{i=1}^N\frac{\partial l}{\partial a_{1,i}}\end{array}$$
解2：可以用矩阵来表示$N$个样本，以简化形式。定义
$$\begin{gathered} X=[x_1,x_2,\dots ,x_N] \\ {A}_1=[a_{1,1},a_{1,2},\dots ,a_{1,N}]=W_{1}X+b{1}^T \\ {A}_2=[a_{2,1},a_{2,2},\dots ,a_{2,N}]=W_2\sigma (A_1)+b_2{1}^T \\ \sigma (A_1)=[\sigma (a_{1,1}),\sigma (a_{1,2}),\dots ,\sigma (a_{1,N})] \end{gathered}$$

则有$\frac{\partial l}{\partial A_2}=\left[\text{softmax}(a_{2,1})-y_1,\text{softmax}(a_{2,2})-y_2,\dots ,\text{softmax}(a_{2,N})-y_N\right]$
$$\begin{array}{rl}dl& =\text{tr}({\frac{\partial l}{\partial A_2}}^{T}d{A}_2)\\ & =\text{tr}\left({\frac{\partial l}{\partial A_2}}^{T}d{W}_2\sigma (A_1)+{\frac{\partial l}{\partial A_2}}^T{W}_{2}d\sigma (A_1)+{\frac{\partial l}{\partial A_2}}^{T}d{b}_2{1}^T\right)\\ & \Rightarrow \frac{\partial l}{\partial W_2}=\frac{\partial l}{\partial A_2}\sigma (A_1{)}^T,\frac{\partial l}{\partial b_2}=\frac{\partial l}{\partial A_2}\end{array}$$
求解$\frac{\partial l}{\partial A_1}$ $$\begin{array}{rl}d{l}_2& =\text{tr}\left({\frac{\partial l}{\partial A_2}}^T{W}_{2}d\sigma (A_1)\right)\\ & =\text{tr}\left({\frac{\partial l}{\partial A_2}}^T{W}_2\left(1-\sigma (A_1)\right)\odot d{A}_1\right)\\ & =\text{tr}\left({\left(W_2^T\frac{\partial l}{\partial A_2}\odot \left(1-\sigma (A_1)\right)\right)}^{T}d{A}_1\right)\\ & \Rightarrow \frac{\partial l}{\partial A_1}=W_2^T\frac{\partial l}{\partial A_2}\odot \left(1-\sigma (A_1)\right)\end{array}$$
继续求解$\frac{\partial l}{\partial W_1},\frac{\partial l}{\partial b_1}$
$$\begin{array}{rl}d{l}_2& =\text{tr}\left({\frac{\partial l}{\partial A_1}}^{T}d{A}_1\right)=\text{tr}\left({\frac{\partial l}{\partial A_1}}^{T}d{W}_{1}X+{\frac{\partial l}{\partial A_1}}^{T}d{b}_1\right)\\ & \Rightarrow \frac{\partial l}{\partial W_1}=\frac{\partial l}{\partial A_1}{X}^T,\frac{\partial l}{\partial b_1}=\frac{\partial l}{\partial A_1}\end{array}$$

矩阵求导术（上）