Neural Networks
All postsPerceptron
Dot product properties:
\[\begin{aligned} \overrightarrow{A}\cdot \overrightarrow{B} = 0 ~~ \text{if} ~ \measuredangle \overrightarrow{A} \overrightarrow{B} = 90^\circ \\ \overrightarrow{A}\cdot \overrightarrow{B} > 0 ~~ \text{if} ~ \measuredangle \overrightarrow{A} \overrightarrow{B} < 90^\circ \\ \overrightarrow{A}\cdot \overrightarrow{B} < 0 ~~ \text{if} ~ \measuredangle \overrightarrow{A} \overrightarrow{B} > 90^\circ \\ \end{aligned}\]The idea is that the weight vector $\textbf{w}$ is perpendicular to the decision boundary.
The weight vector is pushed in opposite direction from the input vector every time wrong class is assigned, thus effectively pushing the input vector to the other side of the decision boundary.
Learning rule
\[w' = w + \alpha(t_i - y_i)x_i\]where $t_i, y_i \in \{0, 1\}$ and \(y_i = \begin{Bmatrix} 1 &~ \text{if}~ x\cdot w > 0 \\ 0 & \end{Bmatrix}\)
Delta Rule
More general learning rule than the Perceptron one, that includes non-linear classifiers.
\[\Delta w_{ij} = \alpha(t_j - y_j)g'(h_j)x_i\]Since $\frac{\partial E}{\partial w_{ij}} = -(t_j - y_j)g’(h_j)x_i$:
\[w_{ij}' = w_{ij} - \alpha \frac{\partial E}{\partial w_{ij}}\]