Machine learning algorithms
linear regression
Linear regression helps predict continuous outcomes, such as house prices, based on input features like size, income, or number of rooms.
For this linear regression example, we use a real dataset of U.S. housing data available on https://www.kaggle.com/datasets/vedavyasv/usa-housing
The dataset, called USA Housing, contains information about house prices and their related features such as area population, number of rooms, and average income.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
# Load the dataset
houses = pd.read_csv("USA_Housing.csv")
# Display the first 20 rows
print(houses.head(20))
# General information
print(houses.info())Before modeling, it is useful to visualize distributions and correlations to understand relationships between variables. distribution plot
sns.histplot(houses["Price"], bins=30, kde=True)
plt.title("Distribution of House Prices")
plt.show()
correlation heatmap
A heatmap helps identify which features are most correlated with the target variable (`Price`).
corr_matrix = houses.corr()
sns.heatmap(corr_matrix, annot=True, cmap="coolwarm")
plt.title("Correlation Heatmap")
plt.show()
The strongest correlation is typically between Average Area Income and Price, meaning wealthier areas tend to have more expensive houses. preparing the data
preparing the data
Before training the model, we split the dataset into features (X) and target (y).
# Define features (independent variables)
X = houses[["Avg. Area Income", "Avg. Area House Age",
"Avg. Area Number of Rooms", "Avg. Area Number of Bedrooms",
"Area Population"]]
# Define target (dependent variable)
y = houses["Price"]
print(X.head())
print(y.head())
splitting data for training and testing
We divide the dataset into training data (used to build the model) and testing data (used to evaluate it).
from sklearn.model_selection import train_test_split
# Split data: 70% for training, 30% for testing
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42
)
print("Training samples:", X_train.shape[0])
print("Testing samples:", X_test.shape[0])
# Training samples: 3500
# Testing samples: 1500
training the model
We now create and train a Linear Regression model using scikit-learn.
from sklearn.linear_model import LinearRegression
# Create model
model = LinearRegression()
# Train the model
model.fit(X_train, y_train)
# Display coefficients and intercept
print("Intercept:", model.intercept_)
print("Coefficients:", model.coef_)
# Create a DataFrame for better readability
coef_df = pd.DataFrame(model.coef_, X.columns, columns=["Coefficient"])
print(coef_df) Coefficient
Avg. Area Income 21.625799
Avg. Area House Age 165590.392746
Avg. Area Number of Rooms 119827.783390
Avg. Area Number of Bedrooms 2361.095262
Area Population 15.216581Each coefficient represents how much the Price changes with one unit of change in the corresponding feature.
making predictions
We can now predict house prices using the test data.
# Predict on the test set
predictions = model.predict(X_test)
# Compare actual vs predicted
comparison = pd.DataFrame({"Actual": y_test, "Predicted": predictions})
print(comparison.head()) Actual Predicted
1501 1.339096e+06 1.308536e+06
2586 1.251794e+06 1.237123e+06
2653 1.340095e+06 1.243836e+06
1055 1.431508e+06 1.229242e+06
705 1.042374e+06 1.059353e+06
visualizing predictions
We can visualize how close the predictions are to the actual prices. If the points align closely to the diagonal line, the model predictions are accurate.
scatter plot (real vs predicted)
plt.scatter(y_test, predictions)
plt.xlabel("Actual Prices")
plt.ylabel("Predicted Prices")
plt.title("Actual vs Predicted House Prices")
plt.show()
histogram of residuals
Residuals (errors) are the differences between actual and predicted prices. A normal distribution of residuals indicates a good fit.
sns.histplot((y_test - predictions), bins=30, kde=True)
plt.title("Distribution of Residuals")
plt.xlabel("Prediction Error")
plt.show()
evaluating the model
We use three evaluation metrics to measure model performance:
1. MAE (Mean Absolute Error) average of absolute errors.
2. MSE (Mean Squared Error) average of squared errors.
3. RMSE (Root Mean Squared Error) square root of MSE.
from sklearn import metrics
mae = metrics.mean_absolute_error(y_test, predictions)
mse = metrics.mean_squared_error(y_test, predictions)
rmse = np.sqrt(mse)
print("Mean Absolute Error (MAE):", mae)
print("Mean Squared Error (MSE):", mse)
print("Root Mean Squared Error (RMSE):", rmse)The smaller these values are, the better the model performs.
A well-trained linear regression model for this dataset typically achieves a very low error, indicating good predictive accuracy.
logistic regression
Logistic regression is a supervised learning algorithm used for classification problems, where the target variable is categorical, meaning it can take a limited number of discrete values.
It is used to model the probability that an event occurs based on one or more independent variables.
Typical applications include:
- Detecting spam vs. non-spam emails
- Predicting whether a loan applicant will repay or default
- Determining if a patient has a certain disease or not
These are examples of binary classifications, where the result can be either 1 (yes) or 0 (no).
what is a logistic function
The logistic function produces an S-shaped curve that converts any real-valued number into a value between 0 and 1.
If the output is ≥ 0.5, we classify it as 1 (event occurs), otherwise as 0 (event does not occur).
This makes it ideal for probabilistic models in classification tasks.
confusion matrix
A confusion matrix is a table used to evaluate classification performance.
It compares predicted values with actual values and shows:
- True positives (TP): correctly predicted positive cases
- True negatives (TN): correctly predicted negative cases
- False positives (FP): predicted positive but actually negative
- False negatives (FN): predicted negative but actually positive
From the confusion matrix, we can calculate:
- Accuracy = (TP + TN) / Total
- Error rate = (FP + FN) / Total
For example, in a spam detection test with 200 emails:
- TP = 30
- TN = 150
- FP = 5
- FN = 15
Then:
Accuracy = (30 + 150) / 200 = 0.9 → 90%
Error rate = (5 + 15) / 200 = 0.1 → 10%
example of logistic regression with the Titanic dataset
We will now create a logistic regression model using the Titanic dataset from Kaggle.
This dataset contains information about Titanic passengers, including age, class, gender, and whether they survived.
importing required libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
# Load the training dataset
train = pd.read_csv("train.csv")
# Display the first rows
print(train.head())exploring the data
We can check for missing values and visualize them.
# Check for missing values
print(train.isnull().sum())
# Visualize missing data
sns.heatmap(train.isnull(), cbar=False, yticklabels=False)
plt.title("Missing values heatmap")
plt.show()Columns like Age and Cabin often have missing values. Cabin usually has too many nulls and can be removed.
analyzing survival data
We can analyze the number of survivors and non-survivors.
sns.countplot(x="Survived", data=train)
plt.title("Count of survivors vs non-survivors")
plt.show()
# Survival by gender
sns.countplot(x="Survived", hue="Sex", data=train)
plt.title("Survival rate by gender")
plt.show()
# Survival by class
sns.countplot(x="Survived", hue="Pclass", data=train)
plt.title("Survival rate by passenger class")
plt.show()We observe that:
- Most non-survivors were men
- Most survivors were women
- Passengers in first class had a higher survival rate than those in third class
age distribution
We can visualize the distribution of passengers' ages.
sns.histplot(train["Age"].dropna(), bins=30, kde=False)
plt.title("Distribution of passenger ages")
plt.xlabel("Age")
plt.ylabel("Count")
plt.show()Most passengers were between 20 and 40 years old.
cleaning the data
Before training the model, we must clean and prepare the dataset.
# Fill missing ages based on passenger class (Pclass)
def fill_age(cols):
age = cols[0]
pclass = cols[1]
if pd.isnull(age):
if pclass == 1:
return 38
elif pclass == 2:
return 30
else:
return 25
else:
return age
train["Age"] = train[["Age", "Pclass"]].apply(fill_age, axis=1)
# Drop 'Cabin' due to too many missing values
train.drop("Cabin", axis=1, inplace=True)
# Drop irrelevant columns
train.drop(["Name", "Ticket", "PassengerId"], axis=1, inplace=True)encoding categorical variables
We need to convert text columns into numeric values. For example, we can convert gender into 1 (male) and 0 (female).
# Convert 'Sex' to numeric
sex = pd.get_dummies(train["Sex"], drop_first=True)
train = pd.concat([train, sex], axis=1)
train.drop("Sex", axis=1, inplace=True)
# Convert 'Embarked' to numeric
embark = pd.get_dummies(train["Embarked"], drop_first=True)
train = pd.concat([train, embark], axis=1)
train.drop("Embarked", axis=1, inplace=True)
print(train.head())Now the dataset is completely numeric and ready for modeling.
splitting the data
We separate our data into features (X) and target (y), then split it into training and testing sets.
from sklearn.model_selection import train_test_split
X = train.drop("Survived", axis=1)
y = train["Survived"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=45
)
print("Training samples:", X_train.shape[0])
print("Testing samples:", X_test.shape[0])training the model
We create and train a logistic regression model.
from sklearn.linear_model import LogisticRegression
# Initialize and train model
log_model = LogisticRegression(max_iter=1000)
log_model.fit(X_train, y_train)
# Make predictions
predictions = log_model.predict(X_test)
print(predictions[:10])evaluating the model
We can now evaluate the model using various metrics.
from sklearn.metrics import classification_report, confusion_matrix
# Classification report
print(classification_report(y_test, predictions))
# Confusion matrix
cm = confusion_matrix(y_test, predictions)
print(cm)
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
plt.title("Confusion matrix")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()Typical output:
Precision for class 0 (did not survive): 0.88 Precision for class 1 (survived): 0.75
This means the model correctly predicts non-survivors about 88% of the time and survivors about 75% of the time.
K-Nearest Neighbors (KNN)
K-Nearest Neighbors (KNN) is a supervised classification algorithm used to estimate the probability that an element *X* belongs to a class *C*, based on its proximity to other known examples in the dataset. It is simple, intuitive, and works with any number of classes.
The idea is to classify a new element by looking at the K nearest points (neighbors) in the training data and assigning it the class most common among them.
how it works
Imagine we want to determine the class of a green circle based on nearby red triangles and blue squares.
The KNN algorithm:
1. Calculates the distance between the green circle and all other points.
2. Sorts them from closest to farthest.
3. Looks at the K nearest data points.
4. Assigns the green circle to the class that appears most frequently among those neighbors.
For example:
- If K = 3 and two of the closest points are red triangles and one is a blue square, the circle is classified as a red triangle.
- If K = 5 and three closest points are blue squares, then the circle is classified as a blue square.
Choosing the right value for K is crucial, too small can lead to noise sensitivity, too large can oversimplify the classification.
example with vehicle type prediction
We will demonstrate KNN using the Vehicle Silhouettes dataset, which predicts the type of vehicle based on its physical characteristics.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
# Load dataset
data = pd.read_csv("vehicles.csv")
# Display first rows
print(data.head())
# Show the unique vehicle classes
print(data["Vehicle Class"].unique())The dataset contains various vehicle features and a target column called Vehicle Class, which indicates the type of vehicle (e.g., Bus, Opel, Saab, Van).
preparing the data
We define the features (X) and the target variable (y).
# Define features (all columns except 'Vehicle Class')
X = data.drop("Vehicle Class", axis=1)
# Define target
y = data["Vehicle Class"]
print("Features shape:", X.shape)
print("Target shape:", y.shape)splitting data into training and testing
We divide our dataset into two parts: - Training data to train the model - Testing data to evaluate its performance
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=45
)
print("Training samples:", X_train.shape[0])
print("Testing samples:", X_test.shape[0])training the model
We train a KNN classifier with an initial K value of 1.
from sklearn.neighbors import KNeighborsClassifier
# Create model with K=1
knn = KNeighborsClassifier(n_neighbors=1)
# Train model
knn.fit(X_train, y_train)
# Make predictions
predictions = knn.predict(X_test)
# Display sample predictions
print(predictions[:10])
evaluating the model
To measure model performance, we use a confusion matrix and a classification report.
from sklearn.metrics import confusion_matrix, classification_report
# Confusion matrix
cm = confusion_matrix(y_test, predictions)
print("Confusion matrix:")
print(cm)
sns.heatmap(cm, annot=True, cmap="Blues", fmt="d")
plt.title("Confusion matrix for K=1")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()
# Classification report
print("Classification report:")
print(classification_report(y_test, predictions))This report shows precision, recall, and F1-score for each vehicle class.
For example, the model might correctly predict:
- Bus: 82% accuracy
- Opel: 48% accuracy
- Saab: 38% accuracy
- Van: 78% accuracy
These results can improve by finding a better value for K.
optimizing the K value
We can test multiple K values (from 1 to 30) to find the one with the lowest error rate.
error_rate = []
# Loop through K values from 1 to 30
for i in range(1, 31):
knn = KNeighborsClassifier(n_neighbors=i)
knn.fit(X_train, y_train)
pred_i = knn.predict(X_test)
error_rate.append(np.mean(pred_i != y_test))
# Plot error rate vs K
plt.figure(figsize=(10, 5))
plt.plot(range(1, 31), error_rate, color="green", marker="o",
markerfacecolor="red", markersize=8)
plt.title("Error rate vs K value")
plt.xlabel("K")
plt.ylabel("Error rate")
plt.show()The optimal K corresponds to the lowest point on this graph, typically, K = 3 in this dataset.
retraining with optimal K
We now retrain the model with the optimal K = 3 and re-evaluate its performance.
# Train with K=3
knn_optimal = KNeighborsClassifier(n_neighbors=3)
knn_optimal.fit(X_train, y_train)
# Make predictions
predictions_opt = knn_optimal.predict(X_test)
# Evaluate
cm_opt = confusion_matrix(y_test, predictions_opt)
print("Confusion matrix (K=3):")
print(cm_opt)
sns.heatmap(cm_opt, annot=True, cmap="Blues", fmt="d")
plt.title("Confusion matrix for K=3 (optimized model)")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()
# Classification report
print("Classification report for K=3:")
print(classification_report(y_test, predictions_opt))After optimization, the accuracy of all vehicle classes improves, and the error rate decreases significantly.
conclusion
K-Nearest Neighbors (KNN) is a simple yet effective algorithm for classification.
It makes predictions based on proximity between data points, using a distance metric such as Euclidean distance.
The key points to remember:
- It is non-parametric, no assumptions about the data distribution.
- It is instance-based, stores all training data for comparison.
The choice of K greatly affects accuracy:
- Small K → sensitive to noise. - Large K → oversimplifies classification.
By tuning K and testing error rates, we can build a reliable model for classifying objects like vehicles, customers, or any feature-based categories.
k-means algorithm
The K-Means algorithm is an unsupervised learning algorithm used to solve clustering problems.
Its goal is to partition a set of *n* observations into *k* groups, where each observation belongs to the group whose mean value is nearest.
It is one of the most popular methods in data mining and machine learning.
The algorithm works iteratively to:
1. Assign each point to the nearest cluster center (centroid).
2. Recalculate the centroids as the mean of the points in each cluster.
3. Repeat the process until the centroids no longer change significantly.
The K-Means algorithm is a powerful and simple technique for data clustering.
It is widely used for:
- grouping customers by behavior,
- image compression,
- anomaly detection,
- and pattern discovery in large datasets.
However, K-Means works best when clusters are spherical and evenly sized. It can perform poorly when clusters overlap or have different densities.
Below is a visual example showing how K-Means progressively organizes data points into three groups.
generating data
Before applying the K-Means algorithm, we will generate artificial data using the make_blobs function from `sklearn.datasets`. This function creates random clusters of points that we can use to test the algorithm.
# Import necessary modules
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
# Generate 200 sample points with 2 features and 4 cluster centers
data = make_blobs(n_samples=200, n_features=2, centers=4, random_state=42)
# data[0] contains the coordinates, data[1] contains the cluster labels
print("Shape of data:", data[0].shape)
print("Cluster labels:", np.unique(data[1]))We have now created:
- 200 random data points,
- each with two characteristics (X and Y),
- grouped around four centers (clusters 0, 1, 2, and 3).
visualizing the generated data
Let’s create a scatter plot to visualize the points.
plt.scatter(data[0][:, 0], data[0][:, 1], c=data[1], cmap='rainbow')
plt.title("Generated sample data with 4 centers")
plt.xlabel("Feature 1")
plt.ylabel("Feature 2")
plt.show()This plot shows how the 200 points are distributed across four clusters, each represented by a different color.
training the k-means model
Now we’ll apply the K-Means algorithm to group these points automatically.
The algorithm should find approximately the same clusters that `make_blobs` created.
from sklearn.cluster import KMeans
# Create and fit the model
kmeans = KMeans(n_clusters=4, random_state=42)
kmeans.fit(data[0])
# Get the cluster centers and predicted labels
centers = kmeans.cluster_centers_
labels = kmeans.labels_
print("Cluster centers:\n", centers)
print("Assigned groups:\n", labels[:10])In this example:
- `kmeans.fit(data[0])` trains the model.
- `kmeans.cluster_centers_` gives the center of each cluster.
- `kmeans.labels_` contains the cluster assignment for each point.
visualizing the model result
To compare the original clusters and the K-Means results, we can plot both side by side.
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
# Plot K-Means results
ax1.scatter(data[0][:, 0], data[0][:, 1], c=labels, cmap='rainbow')
ax1.scatter(centers[:, 0], centers[:, 1], c='black', marker='x', s=100)
ax1.set_title("K-Means clustering result")
# Plot original data
ax2.scatter(data[0][:, 0], data[0][:, 1], c=data[1], cmap='rainbow')
ax2.set_title("Original clusters (make_blobs)")
plt.show()On the left, you can see the clusters identified by K-Means.
On the right, the original clusters generated by `make_blobs`.
Both should look very similar, showing that the algorithm has correctly identified the structure of the data.
understanding k-means
K-Means minimizes the intra-cluster variance, meaning that points within the same cluster are as close as possible to each other, and as far as possible from points in other clusters.
The algorithm steps are:
1. Randomly select *k* initial centroids.
2. Assign each point to the nearest centroid.
3. Recalculate the centroids as the mean of assigned points.
4. Repeat until centroids stabilize.
choosing k
Choosing the correct number of clusters (*k*) is essential.
A common method is the Elbow Method, which plots the inertia (sum of squared distances of samples to their closest cluster center) versus *k* to find the optimal number.
inertias = []
for k in range(1, 10):
km = KMeans(n_clusters=k, random_state=42)
km.fit(data[0])
inertias.append(km.inertia_)
plt.plot(range(1, 10), inertias, marker='o')
plt.title("Elbow Method for Optimal k")
plt.xlabel("Number of clusters (k)")
plt.ylabel("Inertia")
plt.show()The “elbow” point in this curve indicates the optimal number of clusters.
decision trees
A decision tree is a predictive model used in various fields, from artificial intelligence to economics.
Given a dataset, it constructs logical diagrams that represent and categorize a sequence of conditions leading to a decision or outcome.
Each branch of the tree represents a possible choice, and each leaf represents a result.
For example, a decision tree might determine the discount applied based on:
- the number of units ordered, and
- the shipping region (Spain or Europe).
Depending on these conditions, the algorithm decides which discount applies.
importing libraries and dataset We will use the Wine dataset, available from Kaggle.
This dataset includes various features such as alcohol level, magnesium content, color intensity, and more, each wine belongs to one of three classes (1, 2, or 3).
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
# Load the dataset
wines = pd.read_csv("vinos.csv")
# Display the first rows
print(wines.head())
# Check the unique wine types
print(wines["Wine Type"].unique())
# Count wines per type
print(wines["Wine Type"].value_counts())We can see that:
- There are 59 wines of type 1
- 71 wines of type 2
- 48 wines of type 3
defining features and target
We separate the dataset into:
- X (features): all wine characteristics except the wine type
- y (target): the column indicating the wine type (1, 2, or 3)
# Define features (X) and target (y)
X = wines.drop("Wine Type", axis=1)
y = wines["Wine Type"]
print("Features shape:", X.shape)
print("Target shape:", y.shape)
splitting data for training and testing
We divide the dataset into training and testing sets to train and evaluate the model fairly.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42
)
print("Training samples:", X_train.shape[0])
print("Testing samples:", X_test.shape[0])Now, 70% of the data is used for training and 30% for testing.
training the decision tree
We create and train a Decision Tree Classifier model.
from sklearn.tree import DecisionTreeClassifier
# Initialize the model
tree_model = DecisionTreeClassifier(random_state=42)
# Train (fit) the model
tree_model.fit(X_train, y_train)
# Make predictions on the test data
predictions = tree_model.predict(X_test)
# Display a few predictions
print("Predictions:", predictions[:10])Each prediction corresponds to the predicted wine type for a test sample.
evaluating the model
We evaluate the model’s performance using a classification report and a confusion matrix.
from sklearn.metrics import classification_report, confusion_matrix
# Generate classification report
print("Classification report:")
print(classification_report(y_test, predictions))
# Generate confusion matrix
cm = confusion_matrix(y_test, predictions)
print("Confusion matrix:")
print(cm)
# Visualize confusion matrix
sns.heatmap(cm, annot=True, cmap="Blues", fmt="d")
plt.title("Confusion matrix - Decision Tree Wine Classifier")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()Interpretation:
- The diagonal cells represent correct predictions.
- Off-diagonal cells show misclassifications (false positives/negatives).
In this case, accuracy is typically very high (around 90–100%), showing that the model performs well at predicting the correct wine type.
visualizing the decision tree (optional)
We can visualize the structure of the trained decision tree to understand how it splits data at each step.
from sklearn import tree
plt.figure(figsize=(15, 10))
tree.plot_tree(tree_model, filled=True, fontsize=8, feature_names=X.columns, class_names=["1", "2", "3"])
plt.title("Decision Tree Visualization")
plt.show()Each node represents a feature and a threshold condition (e.g., “Color intensity ≤ 3.5”).
The model branches until reaching a decision (a leaf node) that determines the wine type.
random forest algorithm
The Random Forest algorithm is an ensemble learning method that combines multiple decision trees.
Each tree in the forest votes for a class, and the class with the majority of votes becomes the final prediction.
Random Forests are among the most accurate classification algorithms.
They perform efficiently with large datasets, can handle hundreds of input variables, and help to reduce overfitting by averaging the results of many decision trees.
using the random forest classifier
We’ll now create a classification model using the Random Forest algorithm.
We’ll use the same dataset as in the Decision Tree example, the *wine dataset*, stored in `vinos.csv`.
importing the necessary modules
First, we import the required Python modules for data processing, visualization, and machine learning.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, confusion_matrixloading the dataset
Next, we load the dataset `wines.csv` into a DataFrame.
# Load the dataset
vinos = pd.read_csv("wines.csv")
# Display the first few rows
print(vinos.head())
# Show basic info
print(vinos.info())This dataset contains several chemical characteristics of different wines and a column called wine type, which can have three possible values (1, 2, or 3).
preparing the data
We separate the dataset into
- X → the features (all columns except the last one),
- y → the target variable (the last column, *wine type*).
# Separate features and target variable
X = vinos.drop("Wine Type", axis=1)
y = vinos["Wine Type"]
# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.30, random_state=42
)
print("Training data shape:", X_train.shape)
print("Test data shape:", X_test.shape)Here, 70% of the data is used for training and 30% for testing.
training the random forest model
We now create and train the Random Forest Classifier.
The main parameter is `n_estimators`, which determines the number of trees in the forest.
# Create and train the model
random_forest = RandomForestClassifier(n_estimators=80, random_state=42)
random_forest.fit(X_train, y_train)
# Make predictions on the test set
predictions = random_forest.predict(X_test)
# Show some sample predictions
print("Sample predictions:", predictions[:10])Each tree in the forest makes its own decision, and the final result is the majority vote among them.
evaluating the model
We can now evaluate the model’s accuracy using a classification report and a confusion matrix.
# Generate a classification report
print("Classification Report:")
print(classification_report(y_test, predictions))
# Generate a confusion matrix
print("Confusion Matrix:")
print(confusion_matrix(y_test, predictions))The precision, recall, and f1-score are all very high, meaning the Random Forest model performs with excellent accuracy.
visualizing the confusion matrix
We can visualize the confusion matrix using Seaborn for a clearer view.
# Plot confusion matrix
cm = confusion_matrix(y_test, predictions)
sns.heatmap(cm, annot=True, cmap="Greens", fmt="d")
plt.title("Confusion Matrix - Random Forest Classifier")
plt.xlabel("Predicted labels")
plt.ylabel("True labels")
plt.show()Most of the correct predictions appear along the diagonal, confirming that the model classifies the wines correctly.
advantages of random forest
Random Forest offers several advantages:
- Works well with large datasets.
- Handles both numerical and categorical features.
- Reduces overfitting compared to single decision trees.
- Provides good accuracy even with missing data.
- Can estimate the importance of each feature.
support vector machine (SVM)
The Support Vector Machine (SVM) is a supervised learning algorithm used for classification and regression problems.
Given a training dataset with labeled examples, an SVM constructs a model that can predict the class of new samples.
SVMs represent data points in space and find the best separating boundary between different classes.
This boundary is called a hyperplane, and the support vectors are the data points closest to it.
The algorithm tries to maximize the margin, the distance between the hyperplane and the nearest data points of each class.
In simpler terms, SVMs separate points (for example, blue and green dots) using lines or curves that create the widest possible gap between classes.
We will now see an example of the Support Vector Machine method for classification, using the built-in Iris dataset available in Seaborn.
importing the necessary modules
# Import required libraries
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.metrics import classification_report, confusion_matrixloading the dataset
We will use the Iris dataset, a classic dataset in machine learning containing information about flower measurements.
# Load the dataset from Seaborn
flowers = sns.load_dataset("iris")
# Display the first few rows
print(flowers.head())
# Show the unique flower species
print(flowers["species"].unique())This dataset contains four numerical features:
- Sepal length
- Sepal width
- Petal length
- Petal width
and a target variable (species) that can be one of three types: *setosa*, *versicolor*, or *virginica*.
exploring the data visually
We can visualize how the features relate to each other and how they differ across species.
# Pairplot of all features colored by species
sns.pairplot(flowers, hue="species", palette="husl")
plt.show()This creates multiple scatter plots comparing each pair of features, with points colored according to the flower species.
preparing the data We now prepare the dataset for the SVM model by separating the features (X) and the target variable (y), then splitting them into training and testing sets.
# Separate features and target
X = flowers.drop("species", axis=1)
y = flowers["species"]
# Split into training and test data (70% train, 30% test)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42
)
print("Training data shape:", X_train.shape)
print("Testing data shape:", X_test.shape)training the svm model
We use the SVC (Support Vector Classifier) class from Scikit-learn to train the model.
We’ll use the parameter `gamma="auto"` for simplicity.
# Create and train the SVM model
model = SVC(gamma="auto")
model.fit(X_train, y_train)
# Make predictions
predictions = model.predict(X_test)
# Display sample predictions
print("Predictions:", predictions[:10])
print("Actual:", y_test[:10].values)The SVM model finds the optimal hyperplane that separates the flower species based on their physical measurements.
evaluating the model
We can now measure how accurate the model is using the classification report and confusion matrix.
# Classification report
print("Classification Report:")
print(classification_report(y_test, predictions))
# Confusion matrix
print("Confusion Matrix:")
print(confusion_matrix(y_test, predictions))For the Iris dataset, SVM often achieves 100% accuracy, as it is a simple and well-separated dataset.
visualizing the confusion matrix
We can visualize the confusion matrix to confirm all predictions were correct.
# Visualize confusion matrix
cm = confusion_matrix(y_test, predictions)
sns.heatmap(cm, annot=True, cmap="Blues", fmt="d",
xticklabels=model.classes_,
yticklabels=model.classes_)
plt.title("Confusion Matrix - SVM Classifier")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()If the model has 100% accuracy, all the values will appear on the diagonal, with no off-diagonal errors.
understanding svm
Support Vector Machines can use different kernels (linear, polynomial, radial basis function, etc.) to separate data that is not linearly separable.
They are widely used in:
- image recognition
- text classification
- bioinformatics
- any problem where data needs to be categorized into distinct classes.