Support Vector Machines

ELEC320 Lectures 2023-03-20

Optimal Hyperplane

Support vector machines create a hyperplane that linearly separates a set of data. This is called the optimal hyperplane.

The equation of optimal hyperplane takes the form:

$$g(x)={\mathbf{x}}^T+b=0$$

Support Vectors

Support vectors are the closes data-points in the dataset to the optimal hyperplane. They are used to determine the location of the hyperplane that separates the set.

Using the function for the optimal hyperplane we can say:

$$g({\mathbf{x}}_{\text{sv}})=1$$

The euclidean distance of a support vector from the optimal hyperplane can be calculated by:

$$\begin{array}{rl}\text{distance}r& =\frac{g(\mathbf{x})}{\parallel \mathbf{w}{\parallel }_2}\\ & =\frac{1}{\parallel \mathbf{w}{\parallel }_2}\end{array}$$

Non-Separable Patterns

We can either use a slack variable ($\xi$) if the data is close to linearly separable, or we can transform the data to become linear.

Kernels

We can project each patterns ${\mathbf{x}}_i$ to a new feature space of higher dimensionality (compared to the original input space) using a projection function $\varphi$:

$$\varphi ({\mathbf{x}}_i):{\mathbb{R}}^d\to {\mathbb{R}}^b$$

We can use various types of kernels but the general form of the kernel machine is:

$$\begin{array}{rl}K({\mathbf{x}}_i,{\mathbf{x}}_j)& =\varphi ({\mathbf{x}}_i{)}^T\varphi ({\mathbf{x}}_j)\\ & =\sum _{k=1}^b{\varphi }_k({\mathbf{x}}_i){\varphi }_k({\mathbf{x}}_j)\end{array}$$