Root at 18:41, 29 October 2025

2025-10-29T18:41:18Z

← Older revision		Revision as of 19:41, 29 October 2025
Line 22:		Line 22:
	plt.show()</syntaxhighlight>		plt.show()</syntaxhighlight>
	[[File:Python linear regression04.png\|none\|thumb\|531x531px]]		[[File:Python linear regression04.png\|none\|thumb\|531x531px]]



	'''correlation heatmap'''		'''correlation heatmap'''
Line 54:		Line 52:
	print(X.head())		print(X.head())
	print(y.head())</syntaxhighlight>		print(y.head())</syntaxhighlight>



	'''splitting data for training and testing'''		'''splitting data for training and testing'''
Line 73:		Line 69:
	# Training samples: 3500		# Training samples: 3500
	# Testing samples: 1500</syntaxhighlight>		# Testing samples: 1500</syntaxhighlight>



	'''training the model'''		'''training the model'''
Line 520:		Line 514:
	print(predictions[:10])		print(predictions[:10])
	</syntaxhighlight>		</syntaxhighlight>



	'''evaluating the model'''		'''evaluating the model'''
Line 759:		Line 751:
	Both should look very similar, showing that the algorithm has correctly identified the structure of the data.		Both should look very similar, showing that the algorithm has correctly identified the structure of the data.

	~~'''~~understanding k-means~~'''~~		== understanding k-means ==

	K-Means minimizes the '''intra-cluster variance''', meaning that points within the same cluster are as close as possible to each other, and as far as possible from points in other clusters.		K-Means minimizes the '''intra-cluster variance''', meaning that points within the same cluster are as close as possible to each other, and as far as possible from points in other clusters.

Line 773:		Line 764:
	4. Repeat until centroids stabilize.		4. Repeat until centroids stabilize.

	~~'''~~choosing k~~'''~~		== choosing k ==

	Choosing the correct number of clusters (k) is essential.		Choosing the correct number of clusters (k) is essential.

Line 842:		Line 832:

	- '''48 wines of type 3'''		- '''48 wines of type 3'''



Line 860:		Line 851:
	print("Target shape:", y.shape)		print("Target shape:", y.shape)
	</syntaxhighlight>		</syntaxhighlight>



	'''splitting data for training and testing'''		'''splitting data for training and testing'''
Line 879:		Line 868:

	Now, 70% of the data is used for training and 30% for testing.		Now, 70% of the data is used for training and 30% for testing.



Line 903:		Line 891:

	Each prediction corresponds to the '''predicted wine type''' for a test sample.		Each prediction corresponds to the '''predicted wine type''' for a test sample.
			'''evaluating the model'''

	'''evaluating the model'''		'''evaluating the model'''
Line 927:		Line 916:
	plt.show()		plt.show()
	</syntaxhighlight>		</syntaxhighlight>


	'''Interpretation:'''		'''Interpretation:'''

Root at 18:37, 29 October 2025

2025-10-29T18:37:27Z

Show changes

Root: Created page with " = linear regression = Linear regression helps predict continuous outcomes, such as house prices, based on input features like size, income, or number of rooms. For this linear regression example, we use a real dataset of '''U.S. housing data''' available on https://www.kaggle.com/datasets/vedavyasv/usa-housing The dataset, called '''USA Housing''', contains information about house prices and their related features such as area population, number of rooms, and average..."

2025-10-28T08:47:00Z

Created page with " = linear regression = Linear regression helps predict continuous outcomes, such as house prices, based on input features like size, income, or number of rooms. For this linear regression example, we use a real dataset of '''U.S. housing data''' available on https://www.kaggle.com/datasets/vedavyasv/usa-housing The dataset, called '''USA Housing''', contains information about house prices and their related features such as area population, number of rooms, and average..."

New page

= linear regression =
Linear regression helps predict continuous outcomes, such as house prices, based on input features like size, income, or number of rooms.

For this linear regression example, we use a real dataset of '''U.S. housing data''' available on https://www.kaggle.com/datasets/vedavyasv/usa-housing

The dataset, called '''USA Housing''', contains information about house prices and their related features such as area population, number of rooms, and average income.<syntaxhighlight lang="python">import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load the dataset
houses = pd.read_csv("USA_Housing.csv")

# Display the first 20 rows
print(houses.head(20))
# General information
print(houses.info())</syntaxhighlight>Before modeling, it is useful to visualize distributions and correlations to understand relationships between variables.

'''distribution plot'''<syntaxhighlight lang="python">sns.histplot(houses["Price"], bins=30, kde=True)
plt.title("Distribution of House Prices")
plt.show()</syntaxhighlight>
[[File:Python linear regression04.png|none|thumb|531x531px]]

'''correlation heatmap'''

A '''heatmap''' helps identify which features are most correlated with the target variable (`Price`).

<syntaxhighlight lang="python">
corr_matrix = houses.corr()
sns.heatmap(corr_matrix, annot=True, cmap="coolwarm")
plt.title("Correlation Heatmap")
plt.show()
</syntaxhighlight>
[[File:Python linear regression03.png|none|thumb|522x522px]]

The strongest correlation is typically between '''Average Area Income''' and '''Price''', meaning wealthier areas tend to have more expensive houses.
'''preparing the data'''

'''preparing the data'''

Before training the model, we split the dataset into '''features (X)''' and '''target (y)'''.

<syntaxhighlight lang="python"># Define features (independent variables)
X = houses[["Avg. Area Income", "Avg. Area House Age",
"Avg. Area Number of Rooms", "Avg. Area Number of Bedrooms",
"Area Population"]]

# Define target (dependent variable)
y = houses["Price"]

print(X.head())
print(y.head())</syntaxhighlight>

'''splitting data for training and testing'''

We divide the dataset into '''training data''' (used to build the model) and '''testing data''' (used to evaluate it).

<syntaxhighlight lang="python">from sklearn.model_selection import train_test_split

# Split data: 70% for training, 30% for testing
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42
)

print("Training samples:", X_train.shape[0])
print("Testing samples:", X_test.shape[0])

# Training samples: 3500
# Testing samples: 1500</syntaxhighlight>

'''training the model'''

We now create and train a '''Linear Regression model''' using scikit-learn.

<syntaxhighlight lang="python">from sklearn.linear_model import LinearRegression

# Create model
model = LinearRegression()

# Train the model
model.fit(X_train, y_train)

# Display coefficients and intercept
print("Intercept:", model.intercept_)
print("Coefficients:", model.coef_)

# Create a DataFrame for better readability
coef_df = pd.DataFrame(model.coef_, X.columns, columns=["Coefficient"])
print(coef_df)</syntaxhighlight><syntaxhighlight>
Coefficient
Avg. Area Income 21.625799
Avg. Area House Age 165590.392746
Avg. Area Number of Rooms 119827.783390
Avg. Area Number of Bedrooms 2361.095262
Area Population 15.216581
</syntaxhighlight>Each coefficient represents how much the '''Price''' changes with one unit of change in the corresponding feature.
'''making predictions'''

'''making predictions'''

We can now predict house prices using the test data.

<syntaxhighlight lang="python">
# Predict on the test set
predictions = model.predict(X_test)

# Compare actual vs predicted
comparison = pd.DataFrame({"Actual": y_test, "Predicted": predictions})
print(comparison.head())
</syntaxhighlight><syntaxhighlight>
Actual Predicted
1501 1.339096e+06 1.308536e+06
2586 1.251794e+06 1.237123e+06
2653 1.340095e+06 1.243836e+06
1055 1.431508e+06 1.229242e+06
705 1.042374e+06 1.059353e+06
</syntaxhighlight>

'''visualizing predictions'''

We can visualize how close the predictions are to the actual prices. If the points align closely to the diagonal line, the model predictions are accurate.

'''scatter plot (real vs predicted)'''<syntaxhighlight lang="python">
plt.scatter(y_test, predictions)
plt.xlabel("Actual Prices")
plt.ylabel("Predicted Prices")
plt.title("Actual vs Predicted House Prices")
plt.show()
</syntaxhighlight>
[[File:Python linear regression01.png|none|thumb|500x500px]]

'''histogram of residuals'''

Residuals (errors) are the differences between actual and predicted prices.
A normal distribution of residuals indicates a good fit.

<syntaxhighlight lang="python">
sns.histplot((y_test - predictions), bins=30, kde=True)
plt.title("Distribution of Residuals")
plt.xlabel("Prediction Error")
plt.show()
</syntaxhighlight>
[[File:Python linear regression02.png|none|thumb|495x495px]]'''evaluating the model'''

We use three evaluation metrics to measure model performance:

1. '''MAE (Mean Absolute Error)''' average of absolute errors.

2. '''MSE (Mean Squared Error)''' average of squared errors.

3. '''RMSE (Root Mean Squared Error)''' square root of MSE.

<syntaxhighlight lang="python">
from sklearn import metrics

mae = metrics.mean_absolute_error(y_test, predictions)
mse = metrics.mean_squared_error(y_test, predictions)
rmse = np.sqrt(mse)

print("Mean Absolute Error (MAE):", mae)
print("Mean Squared Error (MSE):", mse)
print("Root Mean Squared Error (RMSE):", rmse)
</syntaxhighlight>

The '''smaller''' these values are, the '''better''' the model performs.

A well-trained linear regression model for this dataset typically achieves a very low error, indicating good predictive accuracy.

= logistic regression =
'''Logistic regression''' is a '''supervised learning algorithm''' used for '''classification problems''', where the target variable is '''categorical,''' meaning it can take a limited number of discrete values.

It is used to model the '''probability''' that an event occurs based on one or more independent variables.

Typical applications include:

- Detecting spam vs. non-spam emails

- Predicting whether a loan applicant will repay or default

- Determining if a patient has a certain disease or not

These are examples of '''binary classifications''', where the result can be either '''1 (yes)''' or '''0 (no)'''.

== what is a logistic function ==
The logistic function produces an '''S-shaped curve''' that converts any real-valued number into a value between 0 and 1.

If the output is ≥ 0.5, we classify it as 1 (event occurs), otherwise as 0 (event does not occur).

This makes it ideal for probabilistic models in classification tasks.

'''confusion matrix'''

A confusion matrix is a table used to evaluate classification performance.

It compares '''predicted values''' with '''actual values''' and shows:

- '''True positives (TP):''' correctly predicted positive cases

- '''True negatives (TN):''' correctly predicted negative cases

- '''False positives (FP):''' predicted positive but actually negative

- '''False negatives (FN):''' predicted negative but actually positive

From the confusion matrix, we can calculate:

- '''Accuracy''' = (TP + TN) / Total

- '''Error rate''' = (FP + FN) / Total

For example, in a spam detection test with 200 emails:

TP = 30

- TN = 150

- FP = 5

- FN = 15

Then:

Accuracy = (30 + 150) / 200 = 0.9 → 90%

Error rate = (5 + 15) / 200 = 0.1 → 10%

== example of logistic regression with the Titanic dataset ==
We will now create a logistic regression model using the '''Titanic dataset''' from [https://www.kaggle.com Kaggle].

This dataset contains information about Titanic passengers, including age, class, gender, and whether they survived.

'''importing required libraries'''
<syntaxhighlight lang="python">
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load the training dataset
train = pd.read_csv("train.csv")

# Display the first rows
print(train.head())
</syntaxhighlight>

'''exploring the data'''

We can check for missing values and visualize them.

<syntaxhighlight lang="python">
# Check for missing values
print(train.isnull().sum())

# Visualize missing data
sns.heatmap(train.isnull(), cbar=False, yticklabels=False)
plt.title("Missing values heatmap")
plt.show()
</syntaxhighlight>

Columns like '''Age''' and '''Cabin''' often have missing values. Cabin usually has too many nulls and can be removed.

'''analyzing survival data'''

We can analyze the number of survivors and non-survivors.

<syntaxhighlight lang="python">
sns.countplot(x="Survived", data=train)
plt.title("Count of survivors vs non-survivors")
plt.show()

# Survival by gender
sns.countplot(x="Survived", hue="Sex", data=train)
plt.title("Survival rate by gender")
plt.show()

# Survival by class
sns.countplot(x="Survived", hue="Pclass", data=train)
plt.title("Survival rate by passenger class")
plt.show()
</syntaxhighlight>

We observe that:

- Most '''non-survivors''' were '''men'''

- Most '''survivors''' were '''women'''

- Passengers in '''first class''' had a higher survival rate than those in third class

'''age distribution'''

We can visualize the distribution of passengers' ages.

<syntaxhighlight lang="python">
sns.histplot(train["Age"].dropna(), bins=30, kde=False)
plt.title("Distribution of passenger ages")
plt.xlabel("Age")
plt.ylabel("Count")
plt.show()
</syntaxhighlight>

Most passengers were between '''20 and 40 years old'''.

'''cleaning the data'''

Before training the model, we must clean and prepare the dataset.

<syntaxhighlight lang="python">
# Fill missing ages based on passenger class (Pclass)
def fill_age(cols):
age = cols[0]
pclass = cols[1]
if pd.isnull(age):
if pclass == 1:
return 38
elif pclass == 2:
return 30
else:
return 25
else:
return age

train["Age"] = train[["Age", "Pclass"]].apply(fill_age, axis=1)

# Drop 'Cabin' due to too many missing values
train.drop("Cabin", axis=1, inplace=True)

# Drop irrelevant columns
train.drop(["Name", "Ticket", "PassengerId"], axis=1, inplace=True)
</syntaxhighlight>

'''encoding categorical variables'''

We need to convert text columns into numeric values.
For example, we can convert gender into 1 (male) and 0 (female).

<syntaxhighlight lang="python">
# Convert 'Sex' to numeric
sex = pd.get_dummies(train["Sex"], drop_first=True)
train = pd.concat([train, sex], axis=1)
train.drop("Sex", axis=1, inplace=True)

# Convert 'Embarked' to numeric
embark = pd.get_dummies(train["Embarked"], drop_first=True)
train = pd.concat([train, embark], axis=1)
train.drop("Embarked", axis=1, inplace=True)

print(train.head())
</syntaxhighlight>

Now the dataset is completely numeric and ready for modeling.

'''splitting the data'''

We separate our data into '''features (X)''' and '''target (y)''', then split it into training and testing sets.

<syntaxhighlight lang="python">
from sklearn.model_selection import train_test_split

X = train.drop("Survived", axis=1)
y = train["Survived"]

X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=45
)

print("Training samples:", X_train.shape[0])
print("Testing samples:", X_test.shape[0])
</syntaxhighlight>
'''training the model'''

'''training the model'''

We create and train a '''logistic regression model'''.

<syntaxhighlight lang="python">
from sklearn.linear_model import LogisticRegression

# Initialize and train model
log_model = LogisticRegression(max_iter=1000)
log_model.fit(X_train, y_train)

# Make predictions
predictions = log_model.predict(X_test)
print(predictions[:10])
</syntaxhighlight>

'''evaluating the model'''

We can now evaluate the model using various metrics.

<syntaxhighlight lang="python">
from sklearn.metrics import classification_report, confusion_matrix

# Classification report
print(classification_report(y_test, predictions))

# Confusion matrix
cm = confusion_matrix(y_test, predictions)
print(cm)

sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
plt.title("Confusion matrix")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()
</syntaxhighlight>

Typical output:

Precision for class 0 (did not survive): 0.88
Precision for class 1 (survived): 0.75

This means the model correctly predicts non-survivors about '''88%''' of the time and survivors about '''75%''' of the time.

= K-Nearest Neighbors (KNN) =
K-Nearest Neighbors (KNN) is a '''supervised classification algorithm''' used to estimate the probability that an element *X* belongs to a class *C*, based on its proximity to other known examples in the dataset. It is simple, intuitive, and works with any number of classes.

The idea is to classify a new element by looking at the '''K nearest points''' (neighbors) in the training data and assigning it the class most common among them.

== how it works ==
Imagine we want to determine the class of a '''green circle''' based on nearby red triangles and blue squares.

The KNN algorithm:

1. Calculates the '''distance''' between the green circle and all other points.

2. Sorts them from '''closest to farthest'''.

3. Looks at the '''K nearest''' data points.

4. Assigns the green circle to the class that appears most frequently among those neighbors.

For example:

- If '''K = 3''' and two of the closest points are red triangles and one is a blue square, the circle is classified as a '''red triangle'''.

- If '''K = 5''' and three closest points are blue squares, then the circle is classified as a '''blue square'''.

Choosing the right value for '''K''' is crucial, too small can lead to noise sensitivity, too large can oversimplify the classification.

== example with vehicle type prediction ==
We will demonstrate KNN using the '''Vehicle Silhouettes''' dataset, which predicts the type of vehicle based on its physical characteristics.

<syntaxhighlight lang="python">
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load dataset
data = pd.read_csv("vehicles.csv")

# Display first rows
print(data.head())

# Show the unique vehicle classes
print(data["Vehicle Class"].unique())
</syntaxhighlight>

The dataset contains various vehicle features and a target column called '''Vehicle Class''', which indicates the type of vehicle (e.g., Bus, Opel, Saab, Van).

'''preparing the data'''

We define the '''features (X)''' and the '''target variable (y)'''.

<syntaxhighlight lang="python">
# Define features (all columns except 'Vehicle Class')
X = data.drop("Vehicle Class", axis=1)

# Define target
y = data["Vehicle Class"]

print("Features shape:", X.shape)
print("Target shape:", y.shape)
</syntaxhighlight>

'''splitting data into training and testing'''

We divide our dataset into two parts:
- '''Training data''' to train the model
- '''Testing data''' to evaluate its performance

<syntaxhighlight lang="python">
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=45
)

print("Training samples:", X_train.shape[0])
print("Testing samples:", X_test.shape[0])
</syntaxhighlight>

'''training the model'''

We train a '''KNN classifier''' with an initial K value of 1.

<syntaxhighlight lang="python">
from sklearn.neighbors import KNeighborsClassifier

# Create model with K=1
knn = KNeighborsClassifier(n_neighbors=1)

# Train model
knn.fit(X_train, y_train)

# Make predictions
predictions = knn.predict(X_test)

# Display sample predictions
print(predictions[:10])
</syntaxhighlight>

'''evaluating the model'''

To measure model performance, we use a '''confusion matrix''' and a '''classification report'''.

<syntaxhighlight lang="python">
from sklearn.metrics import confusion_matrix, classification_report

# Confusion matrix
cm = confusion_matrix(y_test, predictions)
print("Confusion matrix:")
print(cm)

sns.heatmap(cm, annot=True, cmap="Blues", fmt="d")
plt.title("Confusion matrix for K=1")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()

# Classification report
print("Classification report:")
print(classification_report(y_test, predictions))
</syntaxhighlight>

This report shows precision, recall, and F1-score for each vehicle class.

For example, the model might correctly predict:

- '''Bus:''' 82% accuracy

- '''Opel:''' 48% accuracy

- '''Saab:''' 38% accuracy

- '''Van:''' 78% accuracy

These results can improve by finding a better value for '''K'''.

'''optimizing the K value'''

We can test multiple K values (from 1 to 30) to find the one with the lowest error rate.

<syntaxhighlight lang="python">
error_rate = []

# Loop through K values from 1 to 30
for i in range(1, 31):
knn = KNeighborsClassifier(n_neighbors=i)
knn.fit(X_train, y_train)
pred_i = knn.predict(X_test)
error_rate.append(np.mean(pred_i != y_test))

# Plot error rate vs K
plt.figure(figsize=(10, 5))
plt.plot(range(1, 31), error_rate, color="green", marker="o",
markerfacecolor="red", markersize=8)
plt.title("Error rate vs K value")
plt.xlabel("K")
plt.ylabel("Error rate")
plt.show()
</syntaxhighlight>

The '''optimal K''' corresponds to the lowest point on this graph, typically, '''K = 3''' in this dataset.

'''retraining with optimal K'''

We now retrain the model with the optimal '''K = 3''' and re-evaluate its performance.

<syntaxhighlight lang="python">
# Train with K=3
knn_optimal = KNeighborsClassifier(n_neighbors=3)
knn_optimal.fit(X_train, y_train)

# Make predictions
predictions_opt = knn_optimal.predict(X_test)

# Evaluate
cm_opt = confusion_matrix(y_test, predictions_opt)
print("Confusion matrix (K=3):")
print(cm_opt)

sns.heatmap(cm_opt, annot=True, cmap="Blues", fmt="d")
plt.title("Confusion matrix for K=3 (optimized model)")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()

# Classification report
print("Classification report for K=3:")
print(classification_report(y_test, predictions_opt))
</syntaxhighlight>

After optimization, the accuracy of all vehicle classes improves, and the error rate decreases significantly.

== conclusion ==
K-Nearest Neighbors (KNN) is a simple yet effective algorithm for classification.

It makes predictions based on proximity between data points, using a distance metric such as Euclidean distance.

The key points to remember:

- It is '''non-parametric,''' no assumptions about the data distribution.

- It is '''instance-based,''' stores all training data for comparison.

The '''choice of K''' greatly affects accuracy:
- Small K → sensitive to noise.
- Large K → oversimplifies classification.

By tuning '''K''' and testing error rates, we can build a reliable model for classifying objects like vehicles, customers, or any feature-based categories.
[[Category:KC AI technical aspects]]

Machine learning algorithms - Revision history

Root at 18:41, 29 October 2025

Root at 18:37, 29 October 2025