Pandas is a Python library used for data manipulation, analysis, and cleaning. It provides fast and flexible data structures designed to work with tabular data (similar to an Excel spreadsheet or a SQL table)

Dataframes and Series

  • DataFrame: A 2-dimensional labeled data structure with rows and columns (a complete table)
  • Series: A 1-dimensional labeled array capable of holding any data type (like a single column in a spreadsheet).

Before getting started with pandas we need to install it first.

pip install pandas 
import pandas as pd # 1. Create a simple DataFrame data = { 'Name': ['Alice', 'Bob', 'Charlie'], 'Age': [25, 30, 35], 'City': ['New York', 'London', 'Paris'] } df = pd.DataFrame(data) # 2. Load data from an external file # df = pd.read_csv('data.csv')  # 3. View the first 5 rows print(df.head())

Reading files in pandas

CSV Files (.csv): Use pd.read_csv() to import comma-separated values.

df = pd.read_csv('data.csv')

Excel Spreadsheets (.xlsx, .xls): Use pd.read_excel() to load spreadsheets. You can specify a sheet using the sheet_name parameter.

df = pd.read_excel('budget.xlsx', sheet_name='Q1_Report')

JSON Data (.json): Use pd.read_json() to convert JSON strings or files into tabular format.

df = pd.read_json('users.json')

Parquet Files (.parquet): Use pd.read_parquet() for highly optimized, columnar storage files common in big data workflows.

df = pd.read_parquet('large_dataset.parquet')

SQL Database Tables: Use pd.read_sql() to pull data directly into Pandas via a database connection (e.g., utilizing an engine from the SQLAlchemy project)

import sqlalchemy engine = sqlalchemy.create_engine('sqlite:///my_database.db') df = pd.read_sql('SELECT * FROM employees', con=engine)

Exploratory Data Analysis (EDA)

  • Exploratory Data Analysis (EDA) using Pandas is the foundational step in any data science workflow. It allows you to understand the structure, distributions, relationships, and anomalies within a dataset before moving to modeling

.head()

  • first n rows of a DataFrame or Series. It is primarily used to quickly inspect and verify the structure, column names, and data types of a dataset after loading it.
  • Inside the brackets you can put the number of rows you want to inspect. 
df.head() df.head(10)

.tail

  • The df.tail() method in pandas returns the last n rows of a DataFrame or Series. 
df.tail()

.shape

  • df.shape is an attribute that returns a tuple representing the dimensions of a DataFrame or Series. It provides a quick way to see how many rows and columns your dataset contains. 
df1.shape

Output:

(5632, 27)

Structural & Descriptive Summaries

  • Before analyzing trends, verify how data types are registered and look at basic statistical summaries. 
# Check data types, memory usage, and non-null counts df.info()

Output:

# Summary statistics for numerical columns (mean, min, max, quartiles) df.describe()

Output

# Summary statistics for ALL columns (including categorical frequencies) df.describe(include="all")

Output

Exploring Single Variables

  • Examine individual variables to understand their distribution, frequency, and pull out any categorical unique counts 
# For Categorical Data: Count frequencies of unique values df["category_column"].value_counts() df["category_column"].unique() # List unique entries df["category_column"].nunique() # Total number of unique entries  # For Numerical Data: Quick pandas native plotting df["numeric_column"].hist(bins=20) # Check skewness and distribution df["numeric_column"].plot(kind="box") # Identify outliers

Data Cleaning

  • Before cleaning is done, a quick EDA must be done to know the areas to clean and what to clean. Below is a sample code to have a sneek- peek of what to actually clean. 
# Find duplicate rows and drop them print(df.duplicated().sum()) df.drop_duplicates(inplace=True) 
# Find missing values print(df.isnull().sum())

Deal with missing values

  • Missing values in a dataset can either be dropped or imputed depending on the size of the data, importance and number of missing values.
  • Here are some of the ways to deal with missing values: 
df.dropna(subset=["critical_column"], inplace=True) # Drop missing rows 
df["numeric_column"] = df["numeric_column"].fillna( df["numeric_column"].median() ) # Impute with median

Dealing with DataTypes

  • Values may be recorded in a datatype which may not be fit for them. In anycase this occurs, they can be changed to the correct datatypes.

  • Using pd.to_numeric() Best for converting text or mixed-type columns into numbers. Use errors='coerce' to turn unconvertible entries (like random text or symbols) into NaN.

df['price'] = pd.to_numeric(df['price'], errors='coerce')
  • Using pd.to_datetime() Converts date and time strings into proper datetime64 objects 
df['sale_date'] = pd.to_datetime(df['sale_date'], errors='coerce')
  • Dealing with mixed date formats e.g dd/mm/yyyy, mm/dd/yyyy, yyyy/mm/dd 
df1['order_date'] = pd.to_datetime( df1['order_date'], format='mixed', errors='coerce' ) 
df1['payment_method'] = df1['payment_method'].str.strip().str.capitalize()
  • replace(), used when you want to change items inside a column to ensure consistency in values recorded and reliability. 
df1['payment_method'] = df1['payment_method'].replace({ 'Mpesa' : 'M-pesa', 'Visa': 'Card' })
  • If the column has mixed characters and values for example here in unit_price, you can also use replace function to standardize the column and maybe change the datatype if need be. 
df1['unit_price_kes'] = df1['unit_price_kes'].str.strip().str.replace(r'[^0-9]','', regex = True).astype(float)

Creating new columns

  • You can also be required to create a new column maybe from from a condition as below or from a calculation. In such a scenario, here's how to deal with it. 
df1['phone_validity'] = np.where(df1['customer_phone'].astype(str).str.len() == 10, 'valid', 'invalid') 
# Calculate total cost (multiplication) df['total_cost'] = df['price'] * df['quantity']
  • The code above will create a new column called total_cost.

Deleting columns

  • You can delete columns from a DataFrame using .drop() 
# deleting a single column df = df.drop(columns=['B']) 
# deleting multiple columns df = df.drop(columns=['C', 'D'])

Aggregation in pandas

  • Aggregations in Pandas reduce a collection of numerical or categorical values into a single summary value (such as a sum, mean, or count). You can perform these operations on an entire Pandas DataFrame, specific columns, or groups of data using the groupby() 
# Clean, flat DataFrame output with custom column names clean_summary = df.groupby('Department').agg( total_salary=('Salary', 'sum'), average_bonus=('Bonus', 'mean'), employee_count=('Employee', 'count') ).reset_index()

Concatenation and Merging in Pandas

Concatenation (pd.concat)

  • Use pd.concat() when your datasets have the same structure and you want to glue them together along an axis. • axis=0 (Default): Stacks DataFrames vertically (appends rows). • axis=1: Stacks DataFrames horizontally (appends columns) aligned by their index. 
import pandas as pd df1 = pd.DataFrame({'A': [1, 2], 'B': [3, 4]}) df2 = pd.DataFrame({'A': [5, 6], 'B': [7, 8]}) # Stack vertically (Row-wise) vertical_stack = pd.concat([df1, df2], ignore_index=True) # Stack horizontally (Column-wise) horizontal_stack = pd.concat([df1, df2], axis=1)

Merging (pd.merge)

  • Use pd.merge() when you have different datasets that share a common identifier column (like an ID or SKU) and you want to align them side-by-side. 
left = pd.DataFrame({'User_ID': [1, 2, 3], 'Name': ['Alice', 'Bob', 'Charlie']}) right = pd.DataFrame({'User_ID': [1, 2, 4], 'Salary': [50000, 60000, 70000]}) # Merge on a matching column merged_df = pd.merge(left, right, on='User_ID', how='inner')

Saving the final dataframe

  • You can save a Pandas DataFrame to different file formats in Python using specific df.to_*() export methods 
import pandas as pd # Assume 'df' is your existing DataFrame # 1. Save as CSV (excluding the default index) df.to_csv("data.csv", index=False) # 2. Save as Excel df.to_excel("data.xlsx", index=False) # 3. Save as JSON df.to_json("data.json", orient="records") # 4. Save as Parquet (requires pyarrow or fastparquet) df.to_parquet("data.parquet") # 5. Save as Pickle df.to_pickle("data.pkl")

Conclusion

Python pandas is the foundational, industry-standard tool for modern data manipulation, cleaning, and analysis. Built around its core DataFrame and Series structures, it transforms raw, messy data into structured, tabular insights by providing powerful, intuitive functions for merging datasets, handling missing values, and performing complex aggregations like groupby. While it integrates seamlessly with the broader Python ecosystem—including NumPy, Matplotlib, and scikit-learn—pandas does suffer from high memory usage and reduced performance when processing massive, multi-gigabyte datasets. Despite these scaling limitations, it remains an indispensable cornerstone of data science workflows, unmatched for exploratory data analysis and rapid prototyping.