In this blog post I model a support vector machine in Python. Previously, I modeled and solved the quadratic assignment problem (QAP) in Python using Pyomo (+). I described that the “similarity” of two facilities could be a reason for putting them closer to each other when assigning them to a set of pre-determined positions. Now consider that we have a set of (new) data points (test dataset) to assign them to a set of classes. Undoubtedly, those with similar features should go into the same class. However, this time, the difference is that we want to build a machine intelligent enough to identify those similarities automatically. To detect such similarities it uses support vectors.
In this article, I model the intelligence of such a machine (i.e. a support vector machine) using mathematical relations and optimize its intelligence via Gekko, an optimization interface in Python with large-scale non-linear solvers.
Modeling and solving the inherent optimization problem in Python
Herein, I code the decision problem for creating a support vector machine (SVM):
The decision here is to find values of alpha (support vectors). Note that the optimization model coded here is the dual form of the main optimization problem for SVMs. The dual form of the model does not need to have constraints over observed data. Moreover, it can create non-linear boundaries for classes using kernel tricks. Such characteristics make it more suitable for deriving better boundaries and for datasets with higher dimensions (without a need for dimensionality reduction!), e.g., images, while making it more computationally efficient. Accordingly, an SVM can perform better than complex deep neural networks.
Next, when training the machine is completed, the stored values for alpha and z can be used to make predictions using the following prediction function:
For instance, consider the following dataset:
To make predictions, at first we model and solve the mentioned optimization problem:
The results are as follows:
Then, we implement the prediction function:
Fortunately, the result is 100% accurate 🙂 and the output is as follows:
That’s all! We have successfully built our support vector machine without using extra packages but considering the optimization logic behind it!
Concluding remarks on supper vector machines in Python
In this article, I showed how a classifier, called a support vector machine, can be modeled using an optimization model, then solved to optimality to characterize a prediction function, which can accurately label new observations. In the following articles, I will try to discuss kernels and how to use them effectively.
If this article is going to be used in research or other publishing methods, you can cite it as Tafakkori (2022) (in text) and refer to it as follows: Tafakkori, K. (2022). Creating a support vector machine using Gekko in Python. Supply Chain Data Analytics. url: https://www.supplychaindataanalytics.com/creating-a-support-vector-machine-using-gekko-in-python/
Industrial engineer focusing on leveraging optimization methods and artificial intelligence technologies using multiple programming languages to empower a business to achieve its goals!