Python for Data Analytics: Phase 2 Training¶

Notebook 2.1: Data Analysis using NumPy and Pandas Foundations¶

Python NumPy Pandas Jupyter License: MIT Author

Author: Prakash Ukhalkar
Role: Assistant Professor (MCA) | Researcher in Data Science and Machine Learning

Training Date: Monday, 06th July 2026
Notebook Scope: NumPy arrays for numerical computing and Pandas DataFrames for structured data analysis.

Phase 2 -- Data Analysis using NumPy and Pandas Foundations¶

Learning Objectives

  • Understand and create NumPy arrays with controlled dtype, shape, and layout
  • Apply reshape and slicing to extract and transform array data
  • Build Pandas Series and DataFrames from raw data and CSV files
  • Generate date ranges and attach them as a time index
  • Inspect, describe, and filter DataFrames using built-in Pandas methods

Training Date: Monday, 06th July 2026
Estimated Duration: 60 minutes
Prerequisites: Basic Python syntax, lists, and loops

Part 1 -- NumPy: Numerical Computing with Arrays¶

NumPy (Numerical Python) is the foundation of the entire Python data science stack.
It provides an ndarray object -- a fast, fixed-type multi-dimensional array -- along with hundreds of mathematical operations that work on entire arrays at once without explicit loops.

Why learn NumPy before Pandas?
Pandas DataFrames store their data internally as NumPy arrays. Understanding NumPy gives you a clearer mental model of how Pandas operations work and why they are fast.

1.1 Setup -- Imports¶

Concept and Code Explanation (Before Use)

  • numpy is imported as np -- the universal convention in data science code.
  • pandas is imported as pd -- equally universal.
  • np.random.seed(42) pins the random number generator so every run produces identical output.
  • np.__version__ and pd.__version__ print library versions to confirm the environment.
In [1]:
# ── Core library imports ──────────────────────────────────────────────────────
# numpy  → array creation, dtype control, mathematical operations
# pandas → DataFrames, Series, date ranges, and CSV I/O
import numpy as np
import pandas as pd

# Fix the random seed for reproducible results across all training participants.
np.random.seed(42)

print("NumPy version :", np.__version__)
print("Pandas version:", pd.__version__)
print("Environment ready.")

NumPy version : 2.2.6
Pandas version: 2.3.3
Environment ready.

Result Interpretation: Imports¶

  • Version numbers confirm the packages are installed and active.
  • If a ModuleNotFoundError appears, run pip install numpy pandas and restart the kernel.
  • The seed is set silently -- no output expected beyond the version lines.

1.2 Creating NumPy Arrays¶

Concept and Code Explanation (Before Use)

An ndarray is the core NumPy object. Create one by passing a Python list to np.array().

Key attributes every analyst uses daily:

Attribute What it returns Example output
.dtype Data type of elements float64, int32
.shape Tuple of dimension sizes (3, 4) for 3 rows, 4 columns
.ndim Number of dimensions 1 for 1-D, 2 for 2-D
.size Total number of elements 12 for a 3x4 array

NumPy shortcut constructors:

  • np.zeros(n) -- array of n zeros
  • np.ones(n) -- array of n ones
  • np.arange(start, stop, step) -- evenly spaced integers
  • np.linspace(start, stop, n) -- n evenly spaced floats between two values
In [2]:
# ── 1-D array from a Python list ─────────────────────────────────────────────
temperatures = np.array([72.1, 73.5, 71.8, 74.2, 70.9, 75.3, 72.8])

print("Array values  :", temperatures)
print("dtype         :", temperatures.dtype)
print("shape         :", temperatures.shape)
print("ndim          :", temperatures.ndim)
print("size          :", temperatures.size)
print()

# ── 2-D array: rows = days, columns = sensor readings ────────────────────────
sensor_matrix = np.array([
    [71.2, 72.0, 70.8],
    [73.1, 74.5, 72.9],
    [70.5, 71.3, 69.8],
    [74.0, 75.1, 73.6],
])
print("Sensor matrix (4 days x 3 sensors):")
print(sensor_matrix)
print("shape :", sensor_matrix.shape)
print()

# ── Shortcut constructors ─────────────────────────────────────────────────────
zeros_arr    = np.zeros(5)
ones_arr     = np.ones((2, 3))
range_arr    = np.arange(0, 50, 5)
linspace_arr = np.linspace(20.0, 30.0, 6)

print("zeros   :", zeros_arr)
print("ones    :\n", ones_arr)
print("arange  :", range_arr)
print("linspace:", linspace_arr)

Array values  : [72.1 73.5 71.8 74.2 70.9 75.3 72.8]
dtype         : float64
shape         : (7,)
ndim          : 1
size          : 7

Sensor matrix (4 days x 3 sensors):
[[71.2 72.  70.8]
 [73.1 74.5 72.9]
 [70.5 71.3 69.8]
 [74.  75.1 73.6]]
shape : (4, 3)

zeros   : [0. 0. 0. 0. 0.]
ones    :
 [[1. 1. 1.]
 [1. 1. 1.]]
arange  : [ 0  5 10 15 20 25 30 35 40 45]
linspace: [20. 22. 24. 26. 28. 30.]

Result Interpretation: Creating Arrays¶

  • dtype: float64 confirms NumPy inferred decimal (64-bit float) from the input values.
  • shape: (7,) -- the trailing comma shows this is a 1-D array.
  • shape: (4, 3) -- 4 rows, 3 columns; row index comes first -- memorise this convention.
  • np.linspace is preferred over arange when you want a specific count of evenly spaced floats.

1.3 Reshape and Slicing¶

Concept and Code Explanation (Before Use)

Reshape
array.reshape(rows, cols) rearranges elements into a new shape without copying data.
Rule: the total element count must stay the same.

Slicing
NumPy uses start:stop:step syntax extended to multiple dimensions:

array[row_slice, col_slice]
  • array[1, :] -- entire second row
  • array[:, 2] -- entire third column
  • array[0:2, 1:3] -- sub-matrix
In [ ]:
# ── Reshape: 12 elements -> 3 rows x 4 columns ───────────────────────────────
flat   = np.arange(1, 13)
matrix = flat.reshape(3, 4)

print("Original 1-D array:", flat)
print("Reshaped to (3, 4):")
print(matrix)
print()

# ── Slicing 1-D ───────────────────────────────────────────────────────────────
temps = np.array([22.1, 23.4, 21.8, 24.5, 25.0, 23.1, 22.7, 24.9])

print("All temperatures        :", temps)
print("First 3 readings        :", temps[:3])
print("Last 3 readings         :", temps[-3:])
print("Every other reading     :", temps[::2])
print("Values above 23 degrees :", temps[temps > 23])
print()

# ── Slicing 2-D ───────────────────────────────────────────────────────────────
data = np.array([
    [10, 20, 30, 40],
    [50, 60, 70, 80],
    [90, 100, 110, 120],
])
print("Full matrix:")
print(data)
print("Row 0 (all columns)   :", data[0, :])
print("Column 2 (all rows)   :", data[:, 2])
print("Top-right 2x2 block:\n", data[0:2, 2:4])

Result Interpretation: Reshape and Slicing¶

  • reshape(3, 4) reinterprets the same 12 values in row-major order -- no data lost or duplicated.
  • Boolean masking temps[temps > 23] filters an array in a single expression without a loop.
  • 2-D slicing data[0:2, 2:4] selects a rectangular sub-block -- critical for machine learning feature extraction.
  • Slices return views (not copies). Use .copy() when you want an independent copy.

1.4 NumPy Statistical Operations¶

Concept and Code Explanation (Before Use)

NumPy provides vectorised statistical functions that operate on entire arrays in one call.

Function Description
np.mean(a) Arithmetic mean
np.median(a) Middle value
np.std(a) Standard deviation
np.min(a) / np.max(a) Minimum and maximum
np.percentile(a, q) q-th percentile

On 2-D arrays: axis=0 computes along columns; axis=1 computes along rows.

In [3]:
# ── Simulate 30 daily production readings ────────────────────────────────────
production = np.random.normal(loc=500, scale=20, size=30).round(1)
print("30 production readings:")
print(production)
print()

print(f"Mean           : {np.mean(production):.2f} units")
print(f"Median         : {np.median(production):.2f} units")
print(f"Std deviation  : {np.std(production):.2f} units")
print(f"Min            : {np.min(production):.2f} units")
print(f"Max            : {np.max(production):.2f} units")
print(f"25th percentile: {np.percentile(production, 25):.2f} units")
print(f"75th percentile: {np.percentile(production, 75):.2f} units")
print()

# ── Column-wise stats on a 2-D array ─────────────────────────────────────────
machine_output = np.array([
    [480, 495, 502, 510, 488],
    [510, 520, 515, 508, 512],
    [490, 485, 495, 500, 493],
])
print("Daily average per machine (axis=1):")
print(np.mean(machine_output, axis=1))
print("Machine average per day (axis=0):")
print(np.mean(machine_output, axis=0))

30 production readings:
[509.9 497.2 513.  530.5 495.3 495.3 531.6 515.3 490.6 510.9 490.7 490.7
 504.8 461.7 465.5 488.8 479.7 506.3 481.8 471.8 529.3 495.5 501.4 471.5
 489.1 502.2 477.  507.5 488.  494.2]

Mean           : 496.24 units
Median         : 495.30 units
Std deviation  : 17.70 units
Min            : 461.70 units
Max            : 531.60 units
25th percentile: 488.20 units
75th percentile: 507.20 units

Daily average per machine (axis=1):
[495.  513.  492.6]
Machine average per day (axis=0):
[493.33333333 500.         504.         506.         497.66666667]

Result Interpretation: Statistical Operations¶

  • Mean and median should be close for normally distributed data -- a large gap suggests skewness.
  • axis=1 collapses columns (gives one value per machine); axis=0 collapses rows (gives one value per day).
  • Think of the axis argument as "the axis you collapse".

Part 2 -- Pandas: Structured Data Analysis¶

Pandas is built on top of NumPy. It adds named columns, labelled row indices, and a rich set of tools for reading, cleaning, grouping, and summarising tabular data.

Structure Description Analogy
Series 1-D labelled array A single spreadsheet column
DataFrame 2-D labelled table A full spreadsheet or SQL table

2.1 Pandas Series¶

Concept and Code Explanation (Before Use)

A Series is a one-dimensional array with a labelled index.
Create one by passing a Python list (integer index) or a dictionary (labelled index).

Key operations:

  • .index -- the row labels
  • .values -- underlying NumPy array
  • .mean(), .sum(), .max() -- immediate descriptive stats
  • Boolean indexing works identically to NumPy
In [4]:
# ── Series from a list ────────────────────────────────────────────────────────
defect_counts = pd.Series([3, 7, 2, 9, 4, 6, 1], name="Defects")
print("Defect counts (integer index):")
print(defect_counts)
print()

# ── Series from a dictionary (labelled index) ────────────────────────────────
machine_uptime = pd.Series({
    "M-101": 7.5, "M-102": 8.0, "M-103": 6.8,
    "M-104": 7.9, "M-105": 8.2,
}, name="Uptime_hours")

print("Machine uptime:")
print(machine_uptime)
print()

print(f"Mean uptime   : {machine_uptime.mean():.2f} hrs")
print("Machines >= 8 hrs:")
print(machine_uptime[machine_uptime >= 8.0])

Defect counts (integer index):
0    3
1    7
2    2
3    9
4    4
5    6
6    1
Name: Defects, dtype: int64

Machine uptime:
M-101    7.5
M-102    8.0
M-103    6.8
M-104    7.9
M-105    8.2
Name: Uptime_hours, dtype: float64

Mean uptime   : 7.68 hrs
Machines >= 8 hrs:
M-102    8.0
M-105    8.2
Name: Uptime_hours, dtype: float64

Result Interpretation: Pandas Series¶

  • The left column is the index (labels), the right column is the value.
  • Boolean indexing returns a filtered Series with its original labels intact -- the index always travels with the data.

2.2 Creating a DataFrame¶

Concept and Code Explanation (Before Use)

A DataFrame is a 2-D table with labelled rows and columns.

Method When to use
pd.DataFrame(dict) Build from a Python dictionary of equal-length lists
pd.read_csv(path) Load from a CSV file
pd.DataFrame(np_array, columns=[...]) Wrap a NumPy array

Access a column with df["column_name"].
Access a row by position with df.iloc[n] or by label with df.loc[label].

In [5]:
# ── DataFrame from a dictionary ──────────────────────────────────────────────
production_df = pd.DataFrame({
    "machine_id"     : ["M-101","M-102","M-103","M-104","M-105",
                        "M-101","M-102","M-103","M-104","M-105"],
    "shift"          : ["Morning","Morning","Morning","Morning","Morning",
                        "Night",  "Night",  "Night",  "Night",  "Night"],
    "units_produced" : [480, 510, 490, 520, 475, 495, 505, 487, 515, 470],
    "defects"        : [  3,   2,   5,   1,   4,   6,   2,   4,   0,   7],
})
print("Production DataFrame:")
display(production_df)
print()

# ── Computed column: yield rate ───────────────────────────────────────────────
production_df["yield_pct"] = (
    (production_df["units_produced"] - production_df["defects"])
    / production_df["units_produced"] * 100
).round(2)
print("With yield_pct column:")
display(production_df)

Production DataFrame:
machine_id shift units_produced defects
0 M-101 Morning 480 3
1 M-102 Morning 510 2
2 M-103 Morning 490 5
3 M-104 Morning 520 1
4 M-105 Morning 475 4
5 M-101 Night 495 6
6 M-102 Night 505 2
7 M-103 Night 487 4
8 M-104 Night 515 0
9 M-105 Night 470 7 
With yield_pct column:
machine_id shift units_produced defects yield_pct
0 M-101 Morning 480 3 99.38
1 M-102 Morning 510 2 99.61
2 M-103 Morning 490 5 98.98
3 M-104 Morning 520 1 99.81
4 M-105 Morning 475 4 99.16
5 M-101 Night 495 6 98.79
6 M-102 Night 505 2 99.60
7 M-103 Night 487 4 99.18
8 M-104 Night 515 0 100.00
9 M-105 Night 470 7 98.51

Result Interpretation: Creating a DataFrame¶

  • Pandas displays the DataFrame as a formatted HTML table in Jupyter.
  • The integer index on the left (0-9) is the default row label.
  • The computed yield_pct column is derived in a single vectorised expression -- no loop needed.

2.3 Date Range and Time Index¶

Concept and Code Explanation (Before Use)

pd.date_range() generates a sequence of dates:

pd.date_range(start="YYYY-MM-DD", periods=n, freq="D")
Code Frequency
"D" Calendar day
"B" Business day (Mon-Fri)
"h" Hourly
"W" Weekly

Once a date index is set, filter with df.loc["2026-07"] (entire month).

In [6]:
# ── Generate 20 business days starting 01 July 2026 ─────────────────────────
dates = pd.date_range(start="2026-07-01", periods=20, freq="B")
print("First 5 dates:", dates[:5].tolist())
print("Last  5 dates:", dates[-5:].tolist())
print()

# ── Time-indexed DataFrame ────────────────────────────────────────────────────
np.random.seed(42)
ts_df = pd.DataFrame({
    "date"           : dates,
    "units_produced" : np.random.randint(460, 540, size=20),
    "defects"        : np.random.randint(0, 10, size=20),
})
ts_df["yield_pct"] = (
    (ts_df["units_produced"] - ts_df["defects"])
    / ts_df["units_produced"] * 100
).round(2)
ts_df = ts_df.set_index("date")

print("Time-indexed DataFrame (first 5 rows):")
display(ts_df.head())
print()

# ── Date-based slicing ────────────────────────────────────────────────────────
week1 = ts_df.loc["2026-07-01":"2026-07-05"]
print("Week 1 records:")
display(week1)

First 5 dates: [Timestamp('2026-07-01 00:00:00'), Timestamp('2026-07-02 00:00:00'), Timestamp('2026-07-03 00:00:00'), Timestamp('2026-07-06 00:00:00'), Timestamp('2026-07-07 00:00:00')]
Last  5 dates: [Timestamp('2026-07-22 00:00:00'), Timestamp('2026-07-23 00:00:00'), Timestamp('2026-07-24 00:00:00'), Timestamp('2026-07-27 00:00:00'), Timestamp('2026-07-28 00:00:00')]

Time-indexed DataFrame (first 5 rows):
units_produced defects yield_pct
date
2026-07-01 511 9 98.24
2026-07-02 474 5 98.95
2026-07-03 531 8 98.49
2026-07-06 520 0 100.00
2026-07-07 480 9 98.12 
Week 1 records:
units_produced defects yield_pct
date
2026-07-01 511 9 98.24
2026-07-02 474 5 98.95
2026-07-03 531 8 98.49

Result Interpretation: Date Range and Time Index¶

  • freq="B" skips weekends automatically.
  • Setting the date column as the index unlocks .loc["2026-07"] month-level slicing and .resample("W").mean() weekly resampling.
  • .head() shows the first 5 rows -- use .tail() for the last 5.

2.4 Inspecting a DataFrame¶

Concept and Code Explanation (Before Use)

These methods cover 90% of initial dataset exploration:

Method Purpose
.shape Rows and columns count
.dtypes Data type of each column
.info() Shape + types + non-null counts
.describe() Count, mean, std, min, quartiles, max
.value_counts() Frequency table for a categorical column
.isnull().sum() Missing value count per column
In [7]:
# ── Build a richer DataFrame for inspection ───────────────────────────────────
np.random.seed(42)
n = 50
inspect_df = pd.DataFrame({
    "machine_id"    : np.random.choice(["M-101","M-102","M-103","M-104"], size=n),
    "shift"         : np.random.choice(["Morning","Afternoon","Night"], size=n),
    "units_produced": np.random.randint(450, 550, size=n),
    "defects"       : np.random.randint(0, 12, size=n),
    "temperature_c" : np.random.normal(72, 3, size=n).round(1),
})
inspect_df["yield_pct"] = (
    (inspect_df["units_produced"] - inspect_df["defects"])
    / inspect_df["units_produced"] * 100
).round(2)

# Introduce missing values to simulate real-world data quality issues
inspect_df.loc[[5, 18, 33], "temperature_c"] = None

print("Shape:", inspect_df.shape)
print()
print("Data types:")
print(inspect_df.dtypes)
print()
print("Info summary:")
inspect_df.info()
print()
print("Descriptive statistics:")
display(inspect_df.describe())
print()
print("Missing values per column:")
print(inspect_df.isnull().sum())
print()
print("Machine ID frequency:")
print(inspect_df["machine_id"].value_counts())

Shape: (50, 6)

Data types:
machine_id         object
shift              object
units_produced      int32
defects             int32
temperature_c     float64
yield_pct         float64
dtype: object

Info summary:
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 50 entries, 0 to 49
Data columns (total 6 columns):
 #   Column          Non-Null Count  Dtype  
---  ------          --------------  -----  
 0   machine_id      50 non-null     object 
 1   shift           50 non-null     object 
 2   units_produced  50 non-null     int32  
 3   defects         50 non-null     int32  
 4   temperature_c   47 non-null     float64
 5   yield_pct       50 non-null     float64
dtypes: float64(2), int32(2), object(2)
memory usage: 2.1+ KB

Descriptive statistics:
units_produced defects temperature_c yield_pct
count 50.000000 50.000000 47.000000 50.000000
mean 497.780000 6.360000 72.314894 98.714200
std 29.889791 3.480324 3.199931 0.702738
min 450.000000 0.000000 64.500000 97.610000
25% 475.500000 3.250000 70.350000 98.125000
50% 493.500000 6.000000 72.200000 98.680000
75% 525.750000 9.000000 74.550000 99.320000
max 548.000000 11.000000 78.500000 100.000000 
Missing values per column:
machine_id        0
shift             0
units_produced    0
defects           0
temperature_c     3
yield_pct         0
dtype: int64

Machine ID frequency:
machine_id
M-104    16
M-103    13
M-102    11
M-101    10
Name: count, dtype: int64

Result Interpretation: Inspecting a DataFrame¶

  • .shape returns (50, 6) -- always check this first.
  • .info() shows machine_id and shift are object (string) dtype. Dtype mismatches often indicate parsing errors.
  • isnull().sum() finds 3 missing temperature_c values -- typical in real shop-floor data from sensor dropouts.
  • .describe() reveals min, max, and standard deviation in seconds.

2.5 Filtering and Grouping¶

Concept and Code Explanation (Before Use)

Filtering rows:

df[df["column"] > value]
df[(df["col1"] == "A") & (df["col2"] > 5)]   # AND -- use & not 'and'
df[(df["col1"] == "A") | (df["col1"] == "B")] # OR  -- use | not 'or'

Grouping:

df.groupby("category_col")["numeric_col"].mean()
df.groupby(["col1", "col2"]).agg({"col3": "sum", "col4": "mean"})
In [8]:
# ── Filter: Night shift rows with more than 5 defects ────────────────────────
high_defect_night = inspect_df[
    (inspect_df["shift"] == "Night") & (inspect_df["defects"] > 5)
]
print(f"Night shift rows with >5 defects: {len(high_defect_night)}")
display(high_defect_night.head())
print()

# ── Group by machine ──────────────────────────────────────────────────────────
machine_summary = inspect_df.groupby("machine_id").agg(
    avg_units     = ("units_produced", "mean"),
    total_defects = ("defects", "sum"),
    avg_yield     = ("yield_pct", "mean"),
    record_count  = ("units_produced", "count"),
).round(2)
print("Per-machine summary:")
display(machine_summary)
print()

# ── Group by shift ────────────────────────────────────────────────────────────
shift_summary = inspect_df.groupby("shift").agg(
    avg_units   = ("units_produced", "mean"),
    avg_defects = ("defects", "mean"),
    avg_yield   = ("yield_pct", "mean"),
).round(2)
print("Per-shift summary:")
display(shift_summary)

Night shift rows with >5 defects: 11
machine_id shift units_produced defects temperature_c yield_pct
2 M-101 Night 502 9 77.8 98.21
3 M-103 Night 473 6 71.5 98.73
5 M-104 Night 538 10 NaN 98.14
7 M-101 Night 490 9 70.3 98.16
8 M-103 Night 478 9 65.7 98.12 
Per-machine summary:
avg_units total_defects avg_yield record_count
machine_id
M-101 515.60 89 98.27 10
M-102 480.55 59 98.86 11
M-103 504.31 57 99.12 13
M-104 493.19 113 98.56 16 
Per-shift summary:
avg_units avg_defects avg_yield
shift
Afternoon 500.78 6.04 98.78
Morning 487.50 6.33 98.69
Night 501.40 6.87 98.62

Result Interpretation: Filtering and Grouping¶

  • Compound filters require & and | (bitwise operators), not Python's and / or.
  • groupby().agg() with named aggregations produces clearly named output columns.
  • The summaries are ready to be saved to a database, exported to Excel, or used as chart input.

About This Material¶
Author Prakash Ukhalkar
Role Assistant Professor (MCA) · Researcher in Data Science and Machine Learning
GitHub @prakash-ukhalkar
Repository mfg-python-sql-training
License MIT

Provided for educational use in Python Data Analytics Phase 2 training programmes.
For questions, corrections, or contributions, open an issue or pull request on GitHub.

© 2026 Prakash Ukhalkar · Python for Data Analytics · MIT License