Concept

In binary classification, where is labelled as 0 for negative and 1 for positive, is parameter column vector, and is data column vector, optimization objective of SVM is the following.

This math takes the form of,

For ,

• is a cost function when . When (not just 0), cost is 0. When , cost linearly increases as gets smaller.
• is a cost function when . When (not just 0), cost is 0. When , cost linearly increases as gets larger.

By in the below image,

In An Introduction to Statistical Learning (ISL), where is labelled as -1 for negative and 1 for positive, the SVM optimization objective is expressed as,

This math takes the form of,

works the same as the cost functions and .

• When , produces 0 if . Loss is when and increase as gets smaller.
• When , produces 0 if . Loss is when and increase as gets larger.

This loss functions is known as hinge loss.

By in the below image,

To see the relationship with logistic regression, the optimization objective of logistic regression that we wanna minimize is the followin. is the number of data. is the number of features.

is the igmoid function used in logistic regression.

• When , we want ( to be close to ), we want small, we want ( is much larger than 0).
• When , we want ( to be close to ), we want large, we want ( is much smaller than 0).

In below sigmoid function visualization,