In other words, how much is $ \dfrac{\partial E}{\partial w_{jk}}$?

where the error is defined as $ E = \dfrac{(T - L_k)^2}{2} $, where $T$ is the expected/target output

$ L_h = input$

$ L_i = intermediate$

$ L_j = intermediate$

$ L_k = output$

the output of each neuron would be:

$L_{i}=\sigma (\sum w_{hi}L_{h}) = \sigma (net_i)$

$L_{j}=\sigma (\sum w_{ij}L_{i}) = \sigma (net_j)$

$L_{k}=\sigma (\sum w_{jk}L_{j}) = \sigma (net_k)$

The output of the NN given the $weights$ and the $inputs$ is then:

$L_{k}=\sigma(\sum w_{jk}\sigma(\sum w_{ij}\sigma(\sum w_{hi}L_{h})))$

Now, just some derivatives that will come handy later on:

- $ \dfrac{\partial \sigma (x)}{\partial x} = \sigma (x)(1-\sigma (x)) $
- $(\dfrac{\partial net_k}{\partial{ L_j}})=w_jk$

**Let's start from the Output neuron:**

$ \dfrac{\partial E}{\partial w_{jk}} =

\dfrac{\partial E}{\partial L_k}\dfrac{\partial L_k}{\partial w_{jk}}=

(\dfrac{\partial E}{\partial L_k})(\dfrac{\partial \sigma (net_k)}{\partial w_{jk}})=(\dfrac{\partial E}{\partial{ L_k}}) ( \dfrac{\partial{ \sigma (net_k)}}{\partial net_k})(\dfrac{\partial{net_k}}{\partial w_{jk}})=$

- $(\dfrac{\partial E}{\partial{ L_k}})= \dfrac{\partial{(\dfrac{(T - L_k)^2}{2})}}{\partial{L_k}}=(T - L_k)(-1)=(L_k-T)$
- $(\dfrac{\partial{ \sigma (net_k)}}{\partial net_k})=\sigma (net_k)(1-\sigma (net_k)=L_k(1-L_k)$
- $(\dfrac{\partial{net_k}}{\partial w_{jk}})=L_j$

hence:

$\dfrac{\partial E}{\partial w_{jk}} =(L_k - T)\cdot{L_k(1-L_k)}\cdot{L_j}$

**Let's look at the previous layer (layer k):**

$ E = \dfrac{(T - L_k)^2}{2} = \dfrac{(T -\sigma (net_k))^2}{2} = \dfrac{(T -\sigma (\sum w_{jk}L_{j}))^2}{2} $

$ \dfrac{\partial E}{\partial w_{ij}} = \dfrac{\partial E}{\partial{ L_j}} [L_j(1-L_j)][L_i]$

$\dfrac{\partial E}{\partial{ L_j}}=(\dfrac{\partial E}{\partial{ L_k}})(\dfrac{\partial L_k}{\partial{ L_j}})=(\dfrac{\partial E}{\partial{ L_k}})(\dfrac{\partial \sigma (net_k)}{\partial{ L_j}})=(\dfrac{\partial E}{\partial{ L_k}})(\dfrac{\partial \sigma (net_k)}{\partial{ net_k}})(\dfrac{\partial net_k}{\partial{ L_j}})=(L_k - T) \cdot{L_k(1-L_k)} \cdot{w_{jk}}$

hence:

$ \dfrac{\partial E}{\partial w_{ij}} = [(L_k - T) \cdot{L_k(1-L_k)} \cdot{w_{jk}}] \cdot{[L_j(1-L_j)][L_i]}$

From this we see that for any other intermediate layer we have that:

$ \dfrac{\partial E}{\partial w_{xy}} = \dfrac{\partial E}{\partial L_z} \cdot{[L_y(1-L_y)][L_x]}$

(where $x$ and $y$ are from intermediate layers)

**Hence for the layer before $k$, that is Layer $i$:**

$ \dfrac{\partial E}{\partial w_{hi}} = \dfrac{\partial E}{\partial L_j} \cdot{[L_i(1-L_i)][L_h]}$

where the tricky bit is to compute $\dfrac{\partial E}{\partial L_x}, from the above steps we can see the following recurrence rule:

- $\dfrac{\partial E}{\partial L_k} =(L_k-T)$
- $\dfrac{\partial E}{\partial L_j} =(L_k-T) \cdot{L_k(1-L_k)} \cdot{w_{jk}}$
- $\dfrac{\partial E}{\partial L_i} =(L_k-T) \cdot{L_k(1-L_k)} \cdot{w_{jk}} \cdot{L_j(1-L_j)} \cdot{w_{ij}}$

Once we have $\dfrac{\partial E}{\partial w_{xy}}$ we can use this value to increase/decrease the weight by that amount, all modulated by an $\alpha$

$\Delta{w_{xy}}= -\alpha \dfrac{\partial E}{\partial w_{xy}}$

and the new weight $w'$ would be then:

$w'_{xy} = w_{xy} + \Delta{w_{xy}}$

## No comments:

## Post a Comment