Python modules

What is a module and why use it?

A module in Python is a file that contains reusable code - such as functions, classes, and variables — that can be imported into other scripts. Modules help organize code, avoid repetition, and provide specialized tools for different tasks.

Modules on this document

Numpy

NumPy is a powerful linear algebra library Python module for working with vectors, matrix, and arrays. It is widely used for scientific computing, data analysis, and machine learning because it performs mathematical operations extremely fast.

Panda

Pandas is an open-source library built as an extension of numpy for data manipulation and analysis in Python. It allows fast data cleaning, preparation, and visualization, and supports many input and output data sources.

Seaborn

Seaborn is a powerful Python library for creating statistical data visualizations. It is built on top of Matplotlib and provides a high-level interface to create elegant and informative charts with minimal code.

Plotly and Cufflinks

Plotly is a library for creating interactive visualizations in Python. It allows you to create rich, web-based graphics directly from Jupyter Notebooks or scripts.

Cufflinks connects Plotly with Pandas, making it possible to generate interactive plots directly from dataframes with simple syntax.

Scikit learn

Scikit learn includes pre-built algorithms and tools for performing machine learning tasks.

They save time, ensure reliability, and allow developers to focus on solving data problems instead of writing algorithms from scratch.

---

NumPy Module

NumPy is a powerful linear algebra library Python module for working with vectors, matrix, and arrays. It is widely used for scientific computing, data analysis, and machine learning because it performs mathematical operations extremely fast.

import numpy as np

This line imports the NumPy module and gives it the short alias np, so we can call its functions like `np.array()` or `np.arange()`.

Installing NumPy

pip install numpy

Creating Arrays

To start, import NumPy:

import numpy as np

You can create a 1D array (vector) from a list:

my_list = [1, 2, 3, 4]
array = np.array(my_list)
print(array)
# [1 2 3 4]

You can also create a 2D array (matrix) from a list of lists:

list2 = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
matrix = np.array(list2)
print(matrix)

Array generation functions

NumPy offers several ways to generate arrays automatically.

np.arange()

Creates a range of numbers.

import numpy as np
np.arange(5)         # [0 1 2 3 4]
np.arange(2, 10)     # [2 3 4 5 6 7 8 9]
np.arange(2, 10, 2)  # [2 4 6 8]

np.zeros() and np.ones()

Generate arrays filled with zeros or ones.

np.zeros((2, 4))  # 2 rows, 4 columns of zeros
np.ones((3, 3))   # 3x3 matrix of ones

np.linspace()

Creates evenly spaced numbers between two values.

np.linspace(3, 9, 5)
# [3. 4.5 6. 7.5 9.]

np.eye()

Creates an identity matrix (ones on the diagonal, zeros elsewhere).

np.eye(5)

Random numbers

NumPy’s random module can generate random values for testing or simulations.

np.random.rand()

Generates random numbers between 0 and 1 (uniform distribution).

np.random.rand(2, 3)

np.random.randn()

Generates random numbers from a normal (Gaussian) distribution.

np.random.randn(4, 5)

np.random.randint()

Generates random integers between two values.

np.random.randint(1, 20)       # One random integer
np.random.randint(1, 20, 5)    # Array of 5 random integers

Reshaping arrays

You can change the shape (rows and columns) of an array using reshape(), as long as the total number of elements matches.

arr = np.random.randint(1, 100, 25)
matrix = arr.reshape(5, 5)
print(matrix)

Maximum, minimum and index positions

Use these functions to find key values and their positions in arrays.

arr = np.random.randint(1, 100, 50)
print(arr.max())      # Maximum value
print(arr.argmax())   # Index of maximum
print(arr.min())      # Minimum value
print(arr.argmin())   # Index of minimum

Indexing and slicing

Access individual or multiple elements of arrays using indices.

One-Dimensional Arrays

arr = np.arange(0, 10)
print(arr[0])     # First element
print(arr[-1])    # Last element
print(arr[2:5])   # Slice from index 2 to 4

Make copies safely:

arr2 = arr.copy()
arr2[0] = 100
print(arr)   # Original unchanged

Two-dimensional arrays

Access elements by [row, column].

matrix = np.array([[1,2,3],[4,5,6],[7,8,9]])
print(matrix[0, 0])   # 1
print(matrix[1, 2])   # 6
print(matrix[:, 0])   # All rows, first column
print(matrix[:2, :2]) # Top-left 2x2 submatrix

Conditional selection

You can select elements based on conditions.

arr = np.arange(0, 10)
print(arr[arr > 5])   # Elements greater than 5
print(arr[arr % 2 == 0])  # Even elements

Array operations

You can perform arithmetic directly on arrays; operations are applied element-wise.

arr = np.arange(0, 10)
print(arr + 10)
print(arr * 2)
print(arr - arr)

Between two arrays:

a = np.arange(0, 10)
b = np.arange(5, 15)
print(b - a)

Common mathematical functions

NumPy provides built-in math functions.

arr = np.arange(0, 10)
print(np.sqrt(arr))   # Square root
print(np.max(arr))    # Maximum
print(np.min(arr))    # Minimum

---

pandas module

pandas is an open-source library built as an extension of numpy for data manipulation and analysis in Python.

It allows fast data cleaning, preparation, and visualization, and supports many input and output data sources.

Install it with:

pip install pandas

pandas series

A series in pandas is a one-dimensional labeled array that can hold any type of data (integers, strings, floats, etc.). Each element has a label, known as its index.

import numpy as np
import pandas as pd

labels = ['a', 'b', 'c']
data = [1, 2, 3]

series = pd.Series(data=data, index=labels)
print(series)

a    1
b    2
c    3

You can also create a series from a numpy array:

array = np.arange(5, 8)
labels = ['sale1', 'sale2', 'sale3']
series = pd.Series(array, index=labels)
print(series)

sale1    5
sale2    6
sale3    7

Access values by **label** or **position**:

print(series['sale1'])   # access by label
print(series[0])         # access by position

Series can hold different data types:

data = ['computer', 300, 'printer', 400]
series = pd.Series(data)
print(series)

0    computer
1         300
2     printer
3         400

You can perform operations between series:

s1 = pd.Series([1, 2, 3], index=['a', 'b', 'c'])
s2 = pd.Series([4, 8, 5], index=['a', 'b', 'c'])
sum_series = s1 + s2
print(sum_series)

a     5
b    10
c     8

dataframes

A **dataframe** is a two-dimensional table with rows and columns, similar to an Excel spreadsheet.

import pandas as pd

rows = ['sales1', 'sales2', 'sales3']
cols = ['zoneA', 'zoneB', 'zoneC']
data = [
    [123, 400, 200],
    [422, 234, 120],
    [256, 450, 210]
]

df = pd.DataFrame(data, index=rows, columns=cols)
print(df)

        zoneA  zoneB  zoneC
sales1    123    400    200
sales2    422    234    120
sales3    256    450    210

You can select rows or columns:

print(df.loc['sales1'])          # select one row
print(df.loc[['sales1','sales2']]) # select multiple rows
print(df['zoneA'])               # select one column
print(df[['zoneA','zoneC']])     # select multiple columns

Add a new column:

df['total_sales'] = df['zoneA'] + df['zoneB'] + df['zoneC']

        zoneA  zoneB  zoneC  total_sales
sales1    123    400    200          723
sales2    422    234    120          776
sales3    256    450    210          916

Delete a column permanently:

df.drop('total_sales', axis=1, inplace=True)

Delete a row permanently:

df.drop('sales3', inplace=True)

conditional selection

Conditional selection allows you to filter data that meets specific conditions.

condition = df > 200
print(df[condition])

        zoneA  zoneB  zoneC
sales1    NaN    400    NaN
sales2  422.0    234    NaN
sales3  256.0    450  210.0

Filter by column:

print(df[df['zoneA'] > 200])

        zoneA  zoneB  zoneC
sales2    422    234    120
sales3    256    450    210

Combine multiple conditions:

print(df[(df['zoneA'] > 200) & (df['zoneB'] > 300)])
print(df[(df['zoneA'] > 200) | (df['zoneB'] > 300)])

modifying index names

You can rename the row labels (index) of a dataframe.

df['days'] = ['day1', 'day2', 'day3']
df = df.set_index('days')
print(df)

handling null values

Pandas provides tools to remove or replace missing values (NaN).

import numpy as np
import pandas as pd

data = {'A':[4,5,np.nan], 'B':[6,1,5], 'C':[np.nan,4,np.nan]}
df = pd.DataFrame(data)
print(df)

     A  B    C
0  4.0  6  NaN
1  5.0  1  4.0
2  NaN  5  NaN

Remove rows or columns with nulls:

print(df.dropna())          # remove rows with NaN
print(df.dropna(axis=1))    # remove columns with NaN

     A  B    C
1  5.0  1  4.0

   B
0  6
1  1
2  5

Replace nulls:

df.fillna(value=100)
df.fillna(value=df.mean())

       A  B      C
0    4.0  6  100.0
1    5.0  1    4.0
2  100.0  5  100.0

     A  B    C
0  4.0  6  4.0
1  5.0  1  4.0
2  4.5  5  4.0

grouping data

Grouping allows you to aggregate data by one or more columns.

data = {'day':[1,1,2,2,3],
        'seller':['Antonio','Maria','Jose','Maria','Antonio'],
        'sales':[100,400,200,500,300]}
df = pd.DataFrame(data)

print(df.groupby('day').mean())
print(df.groupby('day').sum())
print(df.groupby('day').describe())

           seller  sales
day                     
1    AntonioMaria    500
2       JoseMaria    700
3         Antonio    300

    sales                                                      
    count   mean         std    min    25%    50%    75%    max
day                                                            
1     2.0  250.0  212.132034  100.0  175.0  250.0  325.0  400.0
2     2.0  350.0  212.132034  200.0  275.0  350.0  425.0  500.0
3     1.0  300.0         NaN  300.0  300.0  300.0  300.0  300.0

combining dataframes

You can concatenate dataframes vertically or horizontally.

df1 = pd.DataFrame({'A':[1,2,3], 'B':[4,5,6], 'C':[7,8,9]})
df2 = pd.DataFrame({'A':[11,12,13], 'B':[14,15,16], 'C':[17,18,19]})

print(pd.concat([df1, df2]))            # vertical
print(pd.concat([df1, df2], axis=1))    # horizontal

    A   B   C
0   1   4   7
1   2   5   8
2   3   6   9
0  11  14  17
1  12  15  18
2  13  16  19

   A  B  C   A   B   C
0  1  4  7  11  14  17
1  2  5  8  12  15  18
2  3  6  9  13  16  19

merging dataframes

Merging joins two dataframes based on a common key.

df1 = pd.DataFrame({'A':[1,2,3],'B':[4,5,6],'key':['C1','C2','C3']})
df2 = pd.DataFrame({'C':[7,8,9],'D':[10,11,12],'key':['C1','C2','C3']})

merged = pd.merge(df1, df2, on='key')
print(merged)

   A  B key  C   D
0  1  4  C1  7  10
1  2  5  C2  8  11
2  3  6  C3  9  12

joining dataframes

Joining combines dataframes by their index labels.

df1 = pd.DataFrame({'A':[1,2,3],'B':[4,5,6]}, index=['i1','i2','i3'])
df2 = pd.DataFrame({'C':[11,12,13],'D':[14,15,16]}, index=['i1','i2','i3'])

joined = df1.join(df2)
print(joined)

    A  B   C   D
i1  1  4  11  14
i2  2  5  12  15
i3  3  6  13  16

dataframe operations

You can perform operations such as counting unique values, applying functions, or sorting.

df = pd.DataFrame({'A':[11,12,10,12],'B':[14,18,16,17]}, index=['i1','i2','i3','i4'])
print(df['A'].unique())
print(df['A'].nunique())
print(df['A'].value_counts())

[11 12 10]

3

A
12    2
11    1
10    1

Apply a function:

def multiply(x): return x * 2
print(df['A'].apply(multiply))
print(df['A'].apply(lambda x: x * 2))

i1    22
i2    24
i3    20
i4    24

i1    22
i2    24
i3    20
i4    24

Sort values:

df.sort_values(by='B')

reading and writing excel files

Pandas can read and write Excel files easily.

import pandas as pd

df = pd.read_excel('example.xlsx')
print(df)

df['E'] = [20,22,23]
df.to_excel('output.xlsx', sheet_name='Sheet1')

reading html tables

You can extract data from HTML tables directly into pandas dataframes.

import pandas as pd

url = 'https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population'
tables = pd.read_html(url)
df = tables[0]
print(df.head(10))

working with sql databases

You can write and read dataframes from SQL databases.

from sqlalchemy import create_engine
import pandas as pd

df = pd.DataFrame({'A':[1,2,3],'B':[4,5,6]})
engine = create_engine('sqlite:///:memory:')
df.to_sql('table', con=engine, index=False)
data = pd.read_sql('table', con=engine)
print(data)

plotting dataframes

Pandas integrates with matplotlib to create visualizations directly from dataframes. Plots generated by Matplotlib (and by extension, Pandas plotting functions) are not automatically displayed in separate windows. plt.show() opens all the figures that have been created and displays them.[5][6] Your script will pause until you close all the plot windows. See Matplotlib module for further reference

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df = pd.DataFrame(np.random.randint(0,200,(50,4)), columns=list('ABCD'))

df['A'].hist()
df.plot.area(alpha=0.3)
df.plot.bar(stacked=True)
df.plot.scatter(x='A', y='B', c='C', cmap='coolwarm')
df.plot.box()
df.plot.hexbin(x='A', y='B', gridsize=10)
df.plot.kde()
plt.show()

matplotlib module

matplotlib is a Python library for creating visualizations and plots from numerical data stored in lists or arrays. It provides full control over every aspect of a chart. from layout and colors to markers and line styles.

It is one of the most widely used libraries for data visualization in Python.

To install it use:

pip install matplotlib

basic example

A simple example of a line plot using matplotlib.

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(2, 9, 10)
y = x ** 2

plt.plot(x, y)
plt.show()

You can add colors and labels:

plt.subplot(1, 2, 1)
plt.plot(x, y, color='green')

plt.subplot(1, 2, 2)
plt.plot(y, x, color='blue')

plt.show()

subplots

Subplots allow displaying multiple plots in the same figure.

This creates two plots side by side (1 row, 2 columns).

To stack them vertically:

plt.subplot(2, 1, 1)
plt.plot(x, y, color='red')

plt.subplot(2, 1, 2)
plt.plot(y, x, color='blue')
plt.show()

object oriented approach

Matplotlib can also be used in an object-oriented way, providing more control.

import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(2, 9, 10)
y = x ** 2

fig = plt.figure()
axes = fig.add_axes([0.1, 0.1, 0.8, 0.8])
axes.plot(x, y, color='green')
plt.show()

The list `[0.1, 0.1, 0.8, 0.8]` defines the left margin, bottom margin, width, and height of the axes as percentages of the figure size.

You can add multiple plots inside the same figure:

fig = plt.figure()

# main plot
ax1 = fig.add_axes([0.1, 0.1, 0.9, 0.9])
ax1.plot(x, y, color='red')

# inset plot
ax2 = fig.add_axes([0.15, 0.5, 0.3, 0.3])
ax2.plot(y, x, color='blue')
plt.show()

multiple plots in a grid

You can create several plots organized in rows and columns. This creates four subplots in a 2x2 grid.

fig, axes = plt.subplots(2, 2)

axes[0, 0].plot(x, y, color='blue')
axes[0, 0].set_title('blue')

axes[0, 1].plot(x, y, color='red')
axes[0, 1].set_title('red')

axes[1, 0].plot(x, y, color='yellow')
axes[1, 0].set_title('yellow')

axes[1, 1].plot(x, y, color='green')
axes[1, 1].set_title('green')

plt.show()

figure size

You can set the overall figure size using the figsize parameter.

fig = plt.figure(figsize=(8, 4))
ax = fig.add_axes([0.1, 0.1, 0.8, 0.8])
ax.plot(x, y, color='red')
plt.show()

The `figsize` values are in inches (width, height). For example, `(8, 4)` creates a rectangular figure 8 inches wide and 4 inches tall.

multiple lines and legends

You can draw multiple lines in a single plot and add a legend.

fig = plt.figure()
ax = fig.add_axes([0.1, 0.1, 0.8, 0.8])

x = np.linspace(0, 10, 20)
ax.plot(x, x**2, label='squared')
ax.plot(x, x**3, label='cubed')

ax.legend(loc=0)
plt.show()

The `loc=0` option automatically finds the best location for the legend.

colors

Matplotlib accepts many color formats — by name, shorthand, or hexadecimal code.

ax.plot(x, y, color='tomato')
ax.plot(x, y, color='#00FF00')  # hex code for green
ax.plot(x, y, color='lime')

You can find all available named colors in the official matplotlib color reference.

---

line styles

You can customize the appearance of plot lines — thickness, transparency, and style.

fig = plt.figure()
ax = fig.add_axes([0, 0, 1, 1])

ax.plot(x, y, color='red', linewidth=5, alpha=0.5, linestyle='--')
plt.show()

- **linewidth** controls thickness - **alpha** controls transparency (0 invisible, 1 fully visible) - **linestyle** can be `'--'`, `':'`, `'-.'`, or `'-'`

---

markers

Markers highlight data points on a plot.

fig = plt.figure()
ax = fig.add_axes([0, 0, 1, 1])

ax.plot(x, y, color='green', marker='o', markersize=5)
plt.show()

You can choose from various marker symbols: `'o'` (circle), `'s'` (square), `'d'` (diamond), `'+'`, `'*'`, `'x'`, and many others.

ax.plot(x, y, color='blue', marker='s', markersize=8)

Seaborn module

Seaborn is a powerful Python library for creating statistical data visualizations. It is built on top of Matplotlib and provides a high-level interface to create elegant and informative charts with minimal code.

installing seaborn

Install it with the following command:

pip install seaborn

loading data

Seaborn comes with several sample datasets. One of the most popular is tips, which contains restaurant bills and tips.

import seaborn as sns
import matplotlib.pyplot as plt

# Load built-in dataset
tips = sns.load_dataset("tips")

# Show the first 10 rows
print(tips.head(10))

   total_bill   tip     sex smoker  day    time  size
0       16.99  1.01  Female     No  Sun  Dinner     2
1       10.34  1.66    Male     No  Sun  Dinner     3
2       21.01  3.50    Male     No  Sun  Dinner     3
3       23.68  3.31    Male     No  Sun  Dinner     2
4       24.59  3.61  Female     No  Sun  Dinner     4
5       25.29  4.71    Male     No  Sun  Dinner     4
6        8.77  2.00    Male     No  Sun  Dinner     2
7       26.88  3.12    Male     No  Sun  Dinner     4
8       15.04  1.96    Male     No  Sun  Dinner     2
9       14.78  3.23    Male     No  Sun  Dinner     2

distribution plots

A distribution plot shows how data values are distributed. We can use `sns.histplot()` (modern replacement for the old `distplot`) to visualize the frequency of numerical values.

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

# Histogram for total bill
sns.histplot(data=tips, x="total_bill", bins=20, kde=True)
plt.title("Distribution of Total Bill")
plt.show()

We can also plot the distribution of tips:

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.histplot(data=tips, x="tip", bins=20, kde=True)
plt.title("Distribution of Tips")
plt.show()

comparing two variables with joinplot

`joinplot()` shows the relationship between two numeric variables, combining scatter plots and histograms.

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.jointplot(data=tips, x="total_bill", y="tip")
plt.show()

You can also use other visual styles such as hexbin or kde:

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.jointplot(data=tips, x="total_bill", y="tip", kind="hex")
sns.jointplot(data=tips, x="total_bill", y="tip", kind="kde")

plt.show()

pairplot

The pairplot function compares all numerical columns in a dataset two by two. It’s useful to see correlations between multiple variables.

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.pairplot(tips)
plt.show()

You can add a categorical variable to color the points:

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.pairplot(tips, hue="sex")
plt.show()

categorical plots

Seaborn makes it easy to plot categorical variables, such as gender or day.

barplot

Shows average values of a numerical variable for each category.

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.barplot(data=tips, x="sex", y="total_bill")
plt.title("Average Total Bill by Gender")
plt.show()

countplot

Counts the number of occurrences of each category.

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.countplot(data=tips, x="sex")
plt.title("Count by Gender")
plt.show()

boxplot

Displays the distribution of values using quartiles.

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.boxplot(data=tips, x="day", y="total_bill", hue="smoker")
plt.title("Total Bill by Day and Smoker Status")
plt.show()

violinplot

Similar to boxplot but shows the full data distribution.

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.violinplot(data=tips, x="day", y="total_bill", hue="smoker", split=True)
plt.title("Total Bill by Day (Smoker vs Non-Smoker)")
plt.show()

stripplot and swarmplot

`stripplot` shows all individual data points; `swarmplot` spreads them out to avoid overlap.

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.stripplot(data=tips, x="day", y="total_bill", hue="sex", dodge=True)
plt.title("Total Bill by Day and Gender (Stripplot)")
plt.show()

sns.swarmplot(data=tips, x="day", y="total_bill", hue="sex", dodge=True)
plt.title("Total Bill by Day and Gender (Swarmplot)")
plt.show()

You can also combine them with other plots:

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.violinplot(data=tips, x="day", y="total_bill", inner=None)
sns.swarmplot(data=tips, x="day", y="total_bill", color="k", alpha=0.6)
plt.title("Combined Violin and Swarm Plot")
plt.show()

heatmaps

A heatmap displays values as colors in a grid. Here we use the `flights` dataset to show the number of passengers by month and year.

import seaborn as sns
import matplotlib.pyplot as plt

flights = sns.load_dataset("flights")

# Convert to matrix form
flights_matrix = flights.pivot_table(index="month", columns="year", values="passengers")

# Plot heatmap
sns.heatmap(flights_matrix, cmap="coolwarm", linecolor="white", linewidths=2)
plt.title("Heatmap of Flights Passengers")
plt.show()

grid plots

Grid plots allow us to create multiple charts at once for comparison.

pairgrid

Provides full control over the type of plots used in a pairwise grid.

import seaborn as sns
import matplotlib.pyplot as plt

iris = sns.load_dataset("iris")

# Create a PairGrid
g = sns.PairGrid(iris)
g.map_upper(sns.scatterplot)
g.map_lower(sns.kdeplot)
g.map_diag(sns.histplot)
plt.show()

facetgrid

Creates a grid of plots based on categorical variables.

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

g = sns.FacetGrid(tips, col="time", row="smoker")
g.map(sns.histplot, "total_bill")
plt.show()

regression plots

Regression plots show relationships between variables and a fitted regression line.

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

# Basic regression
sns.lmplot(data=tips, x="total_bill", y="tip")
plt.title("Regression Plot: Total Bill vs Tip")
plt.show()

# Regression separated by gender
sns.lmplot(data=tips, x="total_bill", y="tip", hue="sex", markers=["o", "x"])
plt.title("Regression by Gender")
plt.show()

You can separate by columns or rows:

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.lmplot(data=tips, x="total_bill", y="tip", col="day", row="sex")
plt.show()

And adjust the aspect and size:

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.lmplot(data=tips, x="total_bill", y="tip", height=6, aspect=1)
plt.show()

styles and colors

Seaborn provides easy ways to change styles, remove borders, and customize colors.

import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.set_style("darkgrid")

sns.countplot(data=tips, x="sex")
plt.title("Count by Gender with Darkgrid Style")
plt.show()

# Remove borders
sns.set_style("white")
sns.despine(left=True, bottom=True)

sns.countplot(data=tips, x="smoker")
plt.title("Smoker Count with White Style")
plt.show()

customizing figure size and font

plt.figure(figsize=(10, 4))
sns.countplot(data=tips, x="day")
sns.set_context("notebook", font_scale=1.5)
plt.title("Count by Day with Larger Font")
plt.show()

changing color palettes

You can use built-in color palettes or create your own.

sns.set_palette("Set2")
sns.countplot(data=tips, x="sex")
plt.title("Using Set2 Palette")
plt.show()

# Custom colors
custom_colors = ["#90ee90", "#1e90ff"]
sns.set_palette(custom_colors)
sns.countplot(data=tips, x="sex")
plt.title("Custom Colors")
plt.show()

---

plotly and cufflinks modules

Plotly is a library for creating interactive visualizations in Python. It allows you to create rich, web-based graphics directly from Jupyter Notebooks or scripts.

Cufflinks connects Plotly with Pandas, making it possible to generate interactive plots directly from dataframes with simple syntax.

This allows you to access Plotly’s interactive plotting capabilities and use them directly with Pandas dataframes.

installing plotly and cufflinks

Install them with:

pip install plotly cufflinks

importing the necessary modules

After installing, you need to import the required libraries and initialize the connection to work in offline mode, which is ideal for Jupyter Notebooks.

import numpy as np
import pandas as pd
import plotly.offline as pyo
import cufflinks as cf
import matplotlib.pyplot as plt

# Enable Plotly in offline mode
pyo.init_notebook_mode(connected=True)

# Connect Cufflinks to offline mode
cf.go_offline()

# This line ensures plots appear within Jupyter Notebook cells
%matplotlib inline

creating a sample dataframe

Let’s create a random dataframe with 100 rows and 4 columns (A, B, C, D) using NumPy and Pandas.

np.random.seed(42)

# Create a DataFrame with 100 rows and 4 columns
df = pd.DataFrame(np.random.randn(100, 4), columns=["A", "B", "C", "D"])

print(df.head())

basic interactive plot

We can visualize the data interactively using iplot(), which produces an interactive Plotly chart.

# Interactive plot using Cufflinks and Plotly
df.iplot(title="Interactive Plot with Plotly and Cufflinks")

You can zoom, pan, hover over points, and toggle data series directly in the graph.

scatter plot

A scatter plot compares two variables visually.

You can zoom into specific regions or isolate columns by clicking the legend.

bar chart

Bar charts show quantities across categories. We can visualize the sum of each column as a bar chart.

box plot

A box plot shows the data distribution, including minimum, maximum, median, and quartiles.

dataframe.iplot(
    kind="box",
    title="Box Plot for All Columns"
)

You can hide or show specific columns interactively by clicking the legend items.

histogram

A histogram shows the distribution of numerical data. You can create one for a specific column or for all columns.

# Histogram for one column
dataframe["A"].iplot(
    kind="hist",
    bins=30,
    title="Histogram of Column A"
)

# Histogram for all columns
dataframe.iplot(
    kind="hist",
    bins=30,
    title="Histogram of All Columns"
)

spread plot

The spread plot shows two variables along with their differences in a mirrored format.

dataframe[["A", "B"]].iplot(
    kind="spread",
    title="Spread Plot Between Columns A and B"
)

This helps compare two time series or numeric columns visually.

bubble chart

A bubble chart uses circle sizes to represent an additional variable.

dataframe.iplot(
    kind="bubble",
    x="A",
    y="B",
    size="C",
    title="Bubble Chart (A vs B, size based on C)",
    xTitle="A values",
    yTitle="B values"
)

3d surface plot

3D plots are useful for visualizing three variables in a single graph. You can rotate, zoom, and explore the plot interactively.

# Create a new DataFrame for 3D plotting
dataframe2 = pd.DataFrame({
    "A": [1, 2, 3, 4],
    "B": [30, 40, 20, 10],
    "C": [12, 16, 18, 15]
})

# 3D surface plot
dataframe2.iplot(
    kind="surface",
    title="3D Surface Plot (A, B, C)"
)

saving and exporting plots

Plotly allows exporting graphs as static images or HTML files.

# Save to an interactive HTML file
pyo.plot(df.iplot(asFigure=True), filename="interactive_plot.html")

# Export a static image (requires kaleido)
import plotly.io as pio
pio.write_image(df.iplot(asFigure=True), "plot.png")

Scikit learn module

Scikit learn includes pre-built algorithms and tools for performing machine learning tasks.

They save time, ensure reliability, and allow developers to focus on solving data problems instead of writing algorithms from scratch.

install it with:

pip install scikit-learn

basic workflow with scikit-learn

The standard steps to build a model with scikit-learn are:

1. Import the necessary modules.

2. Initialize the model with its parameters.

3. Split the data into training and testing sets.

4. Train the model with `.fit()`.

5. Predict new values with `.predict()`.

6. Evaluate model accuracy or error metrics.

Example:

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

# Load dataset
iris = load_iris()
X, y = iris.data, iris.target

# Split into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Create and train model
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

# Make predictions
predictions = model.predict(X_test)

# Evaluate
accuracy = accuracy_score(y_test, predictions)
print("Accuracy:", accuracy)

evaluating models

After training a model, we need to measure how well it performs.

Common metrics include

-Accuracy for classification

- Mean squared error (MSE) for regression

- Confusion matrix for detailed error analysis

Example:

from sklearn.metrics import confusion_matrix, classification_report

print("Confusion Matrix:")
print(confusion_matrix(y_test, predictions))

print("\nClassification Report:")
print(classification_report(y_test, predictions))

linear regression example

Linear regression helps predict continuous outcomes, such as house prices, based on input features like size, income, or number of rooms.

For this linear regression example, we use a real dataset of U.S. housing data available on https://www.kaggle.com/datasets/vedavyasv/usa-housing

The dataset, called USA Housing, contains information about house prices and their related features such as area population, number of rooms, and average income.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load the dataset
houses = pd.read_csv("USA_Housing.csv")

# Display the first 20 rows
print(houses.head(20))
# General information
print(houses.info())

Before modeling, it is useful to visualize distributions and correlations to understand relationships between variables. distribution plot

sns.histplot(houses["Price"], bins=30, kde=True)
plt.title("Distribution of House Prices")
plt.show()

correlation heatmap

A heatmap helps identify which features are most correlated with the target variable (`Price`).

corr_matrix = houses.corr()
sns.heatmap(corr_matrix, annot=True, cmap="coolwarm")
plt.title("Correlation Heatmap")
plt.show()

The strongest correlation is typically between Average Area Income and Price, meaning wealthier areas tend to have more expensive houses.

preparing the data

Before training the model, we split the dataset into features (X) and target (y).

# Define features (independent variables)
X = houses[["Avg. Area Income", "Avg. Area House Age",
            "Avg. Area Number of Rooms", "Avg. Area Number of Bedrooms",
            "Area Population"]]

# Define target (dependent variable)
y = houses["Price"]

print(X.head())
print(y.head())

splitting data for training and testing

We divide the dataset into training data (used to build the model) and testing data (used to evaluate it).

from sklearn.model_selection import train_test_split

# Split data: 70% for training, 30% for testing
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

print("Training samples:", X_train.shape[0])
print("Testing samples:", X_test.shape[0])

# Training samples: 3500
# Testing samples: 1500

training the model

We now create and train a Linear Regression model using scikit-learn.

from sklearn.linear_model import LinearRegression

# Create model
model = LinearRegression()

# Train the model
model.fit(X_train, y_train)

# Display coefficients and intercept
print("Intercept:", model.intercept_)
print("Coefficients:", model.coef_)

# Create a DataFrame for better readability
coef_df = pd.DataFrame(model.coef_, X.columns, columns=["Coefficient"])
print(coef_df)

                                Coefficient
Avg. Area Income                  21.625799
Avg. Area House Age           165590.392746
Avg. Area Number of Rooms     119827.783390
Avg. Area Number of Bedrooms    2361.095262
Area Population                   15.216581

Each coefficient represents how much the Price changes with one unit of change in the corresponding feature.

making predictions

We can now predict house prices using the test data.

# Predict on the test set
predictions = model.predict(X_test)

# Compare actual vs predicted
comparison = pd.DataFrame({"Actual": y_test, "Predicted": predictions})
print(comparison.head())

            Actual     Predicted
1501  1.339096e+06  1.308536e+06
2586  1.251794e+06  1.237123e+06
2653  1.340095e+06  1.243836e+06
1055  1.431508e+06  1.229242e+06
705   1.042374e+06  1.059353e+06

visualizing predictions

We can visualize how close the predictions are to the actual prices. If the points align closely to the diagonal line, the model predictions are accurate.

scatter plot (real vs predicted)

plt.scatter(y_test, predictions)
plt.xlabel("Actual Prices")
plt.ylabel("Predicted Prices")
plt.title("Actual vs Predicted House Prices")
plt.show()

histogram of residuals

Residuals (errors) are the differences between actual and predicted prices. A normal distribution of residuals indicates a good fit.

sns.histplot((y_test - predictions), bins=30, kde=True)
plt.title("Distribution of Residuals")
plt.xlabel("Prediction Error")
plt.show()

evaluating the model

We use three evaluation metrics to measure model performance:

1. MAE (Mean Absolute Error) average of absolute errors.

2. MSE (Mean Squared Error) average of squared errors.

3. RMSE (Root Mean Squared Error) square root of MSE.

from sklearn import metrics

mae = metrics.mean_absolute_error(y_test, predictions)
mse = metrics.mean_squared_error(y_test, predictions)
rmse = np.sqrt(mse)

print("Mean Absolute Error (MAE):", mae)
print("Mean Squared Error (MSE):", mse)
print("Root Mean Squared Error (RMSE):", rmse)

The smaller these values are, the better the model performs.

A well-trained linear regression model for this dataset typically achieves a very low error, indicating good predictive accuracy.

logistic regression

Logistic regression is a supervised learning algorithm used for classification problems, where the target variable is categorical, meaning it can take a limited number of discrete values.

It is used to model the probability that an event occurs based on one or more independent variables.

Typical applications include:

- Detecting spam vs. non-spam emails

- Predicting whether a loan applicant will repay or default

- Determining if a patient has a certain disease or not

These are examples of binary classifications, where the result can be either 1 (yes) or 0 (no).

what is a logistic function

The logistic function produces an S-shaped curve that converts any real-valued number into a value between 0 and 1.

If the output is ≥ 0.5, we classify it as 1 (event occurs), otherwise as 0 (event does not occur).

This makes it ideal for probabilistic models in classification tasks.

confusion matrix

A confusion matrix is a table used to evaluate classification performance.

It compares predicted values with actual values and shows:

- True positives (TP): correctly predicted positive cases

- True negatives (TN): correctly predicted negative cases

- False positives (FP): predicted positive but actually negative

- False negatives (FN): predicted negative but actually positive

From the confusion matrix, we can calculate:

- Accuracy = (TP + TN) / Total

- Error rate = (FP + FN) / Total

For example, in a spam detection test with 200 emails:

TP = 30

- TN = 150

- FP = 5

- FN = 15

Then:

Accuracy = (30 + 150) / 200 = 0.9 → 90%

Error rate = (5 + 15) / 200 = 0.1 → 10%

example: logistic regression with the Titanic dataset

We will now create a logistic regression model using the Titanic dataset from Kaggle.

This dataset contains information about Titanic passengers, including age, class, gender, and whether they survived.

importing required libraries

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load the training dataset
train = pd.read_csv("train.csv")

# Display the first rows
print(train.head())

exploring the data

We can check for missing values and visualize them.

# Check for missing values
print(train.isnull().sum())

# Visualize missing data
sns.heatmap(train.isnull(), cbar=False, yticklabels=False)
plt.title("Missing values heatmap")
plt.show()

Columns like Age and Cabin often have missing values. Cabin usually has too many nulls and can be removed.

analyzing survival data

We can analyze the number of survivors and non-survivors.

sns.countplot(x="Survived", data=train)
plt.title("Count of survivors vs non-survivors")
plt.show()

# Survival by gender
sns.countplot(x="Survived", hue="Sex", data=train)
plt.title("Survival rate by gender")
plt.show()

# Survival by class
sns.countplot(x="Survived", hue="Pclass", data=train)
plt.title("Survival rate by passenger class")
plt.show()

We observe that:

- Most non-survivors were men

- Most survivors were women

- Passengers in first class had a higher survival rate than those in third class

age distribution

We can visualize the distribution of passengers' ages.

sns.histplot(train["Age"].dropna(), bins=30, kde=False)
plt.title("Distribution of passenger ages")
plt.xlabel("Age")
plt.ylabel("Count")
plt.show()

Most passengers were between 20 and 40 years old.

cleaning the data

Before training the model, we must clean and prepare the dataset.

# Fill missing ages based on passenger class (Pclass)
def fill_age(cols):
    age = cols[0]
    pclass = cols[1]
    if pd.isnull(age):
        if pclass == 1:
            return 38
        elif pclass == 2:
            return 30
        else:
            return 25
    else:
        return age

train["Age"] = train[["Age", "Pclass"]].apply(fill_age, axis=1)

# Drop 'Cabin' due to too many missing values
train.drop("Cabin", axis=1, inplace=True)

# Drop irrelevant columns
train.drop(["Name", "Ticket", "PassengerId"], axis=1, inplace=True)

encoding categorical variables

We need to convert text columns into numeric values. For example, we can convert gender into 1 (male) and 0 (female).

# Convert 'Sex' to numeric
sex = pd.get_dummies(train["Sex"], drop_first=True)
train = pd.concat([train, sex], axis=1)
train.drop("Sex", axis=1, inplace=True)

# Convert 'Embarked' to numeric
embark = pd.get_dummies(train["Embarked"], drop_first=True)
train = pd.concat([train, embark], axis=1)
train.drop("Embarked", axis=1, inplace=True)

print(train.head())

Now the dataset is completely numeric and ready for modeling.

splitting the data

We separate our data into features (X) and target (y), then split it into training and testing sets.

from sklearn.model_selection import train_test_split

X = train.drop("Survived", axis=1)
y = train["Survived"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=45
)

print("Training samples:", X_train.shape[0])
print("Testing samples:", X_test.shape[0])

training the model

We create and train a logistic regression model.

from sklearn.linear_model import LogisticRegression

# Initialize and train model
log_model = LogisticRegression(max_iter=1000)
log_model.fit(X_train, y_train)

# Make predictions
predictions = log_model.predict(X_test)
print(predictions[:10])

evaluating the model

We can now evaluate the model using various metrics.

from sklearn.metrics import classification_report, confusion_matrix

# Classification report
print(classification_report(y_test, predictions))

# Confusion matrix
cm = confusion_matrix(y_test, predictions)
print(cm)

sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
plt.title("Confusion matrix")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()

Typical output:

Precision for class 0 (did not survive): 0.88 Precision for class 1 (survived): 0.75

This means the model correctly predicts non-survivors about 88% of the time and survivors about 75% of the time.

k-nearest neighbors (KNN)

K-Nearest Neighbors (KNN) is a supervised classification algorithm used to estimate the probability that an element *X* belongs to a class *C*, based on its proximity to other known examples in the dataset. It is simple, intuitive, and works with any number of classes.

The idea is to classify a new element by looking at the K nearest points (neighbors) in the training data and assigning it the class most common among them.

how it works

Imagine we want to determine the class of a green circle based on nearby red triangles and blue squares.

The KNN algorithm:

1. Calculates the distance between the green circle and all other points.

2. Sorts them from closest to farthest.

3. Looks at the K nearest data points.

4. Assigns the green circle to the class that appears most frequently among those neighbors.

For example:

- If K = 3 and two of the closest points are red triangles and one is a blue square, the circle is classified as a red triangle.

- If K = 5 and three closest points are blue squares, then the circle is classified as a blue square.

Choosing the right value for K is crucial, too small can lead to noise sensitivity, too large can oversimplify the classification.

importing libraries and dataset

We will demonstrate KNN using the Vehicle Silhouettes dataset, which predicts the type of vehicle based on its physical characteristics.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load dataset
data = pd.read_csv("vehicles.csv")

# Display first rows
print(data.head())

# Show the unique vehicle classes
print(data["Vehicle Class"].unique())

The dataset contains various vehicle features and a target column called Vehicle Class, which indicates the type of vehicle (e.g., Bus, Opel, Saab, Van).

preparing the data

We define the features (X) and the target variable (y).

# Define features (all columns except 'Vehicle Class')
X = data.drop("Vehicle Class", axis=1)

# Define target
y = data["Vehicle Class"]

print("Features shape:", X.shape)
print("Target shape:", y.shape)

splitting data into training and testing

We divide our dataset into two parts: - Training data to train the model - Testing data to evaluate its performance

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=45
)

print("Training samples:", X_train.shape[0])
print("Testing samples:", X_test.shape[0])

training the model

We train a KNN classifier with an initial K value of 1.

from sklearn.neighbors import KNeighborsClassifier

# Create model with K=1
knn = KNeighborsClassifier(n_neighbors=1)

# Train model
knn.fit(X_train, y_train)

# Make predictions
predictions = knn.predict(X_test)

# Display sample predictions
print(predictions[:10])

evaluating the model

To measure model performance, we use a confusion matrix and a classification report.

from sklearn.metrics import confusion_matrix, classification_report

# Confusion matrix
cm = confusion_matrix(y_test, predictions)
print("Confusion matrix:")
print(cm)

sns.heatmap(cm, annot=True, cmap="Blues", fmt="d")
plt.title("Confusion matrix for K=1")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()

# Classification report
print("Classification report:")
print(classification_report(y_test, predictions))

This report shows precision, recall, and F1-score for each vehicle class.

For example, the model might correctly predict:

- Bus: 82% accuracy

- Opel: 48% accuracy

- Saab: 38% accuracy

- Van: 78% accuracy

These results can improve by finding a better value for K.

optimizing the K value

We can test multiple K values (from 1 to 30) to find the one with the lowest error rate.

error_rate = []

# Loop through K values from 1 to 30
for i in range(1, 31):
    knn = KNeighborsClassifier(n_neighbors=i)
    knn.fit(X_train, y_train)
    pred_i = knn.predict(X_test)
    error_rate.append(np.mean(pred_i != y_test))

# Plot error rate vs K
plt.figure(figsize=(10, 5))
plt.plot(range(1, 31), error_rate, color="green", marker="o",
         markerfacecolor="red", markersize=8)
plt.title("Error rate vs K value")
plt.xlabel("K")
plt.ylabel("Error rate")
plt.show()

The optimal K corresponds to the lowest point on this graph, typically, K = 3 in this dataset.

retraining with optimal K

We now retrain the model with the optimal K = 3 and re-evaluate its performance.

# Train with K=3
knn_optimal = KNeighborsClassifier(n_neighbors=3)
knn_optimal.fit(X_train, y_train)

# Make predictions
predictions_opt = knn_optimal.predict(X_test)

# Evaluate
cm_opt = confusion_matrix(y_test, predictions_opt)
print("Confusion matrix (K=3):")
print(cm_opt)

sns.heatmap(cm_opt, annot=True, cmap="Blues", fmt="d")
plt.title("Confusion matrix for K=3 (optimized model)")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()

# Classification report
print("Classification report for K=3:")
print(classification_report(y_test, predictions_opt))

After optimization, the accuracy of all vehicle classes improves, and the error rate decreases significantly.

conclusion

K-Nearest Neighbors (KNN) is a simple yet effective algorithm for classification.

It makes predictions based on proximity between data points, using a distance metric such as Euclidean distance.

The key points to remember:

- It is non-parametric, no assumptions about the data distribution.

- It is instance-based, stores all training data for comparison.

The choice of K greatly affects accuracy:

 - Small K → sensitive to noise.  
 - Large K → oversimplifies classification.

By tuning K and testing error rates, we can build a reliable model for classifying objects like vehicles, customers, or any feature-based categories.