Neural Networks

September 27, 2022 All posts

Perceptron

Dot product properties:

\[\begin{aligned} \overrightarrow{A}\cdot \overrightarrow{B} = 0 ~~ \text{if} ~ \measuredangle \overrightarrow{A} \overrightarrow{B} = 90^\circ \\ \overrightarrow{A}\cdot \overrightarrow{B} > 0 ~~ \text{if} ~ \measuredangle \overrightarrow{A} \overrightarrow{B} < 90^\circ \\ \overrightarrow{A}\cdot \overrightarrow{B} < 0 ~~ \text{if} ~ \measuredangle \overrightarrow{A} \overrightarrow{B} > 90^\circ \\ \end{aligned}\]

The idea is that the weight vector $\textbf{w}$ is perpendicular to the decision boundary.

The weight vector is pushed in opposite direction from the input vector every time wrong class is assigned, thus effectively pushing the input vector to the other side of the decision boundary.

Learning rule

\[w' = w + \alpha(t_i - y_i)x_i\]

where $t_i, y_i \in \{0, 1\}$ and \(y_i = \begin{Bmatrix} 1 &~ \text{if}~ x\cdot w > 0 \\ 0 & \end{Bmatrix}\)

Delta Rule

More general learning rule than the Perceptron one, that includes non-linear classifiers.

\[\Delta w_{ij} = \alpha(t_j - y_j)g'(h_j)x_i\]

Since $\frac{\partial E}{\partial w_{ij}} = -(t_j - y_j)g’(h_j)x_i$:

\[w_{ij}' = w_{ij} - \alpha \frac{\partial E}{\partial w_{ij}}\]

Backpropagation

XOR Network

Implementation. Notebook.

RNNs