Data Analysis with NumPy, Pandas, and Matplotlib
From raw data to insight: how NumPy, pandas, and Matplotlib work together – vectorized computation, groupby aggregation, and a quick plot in a small pipeline.
A DataFrame is not a spreadsheet and not a database table. It is a collection of typed columns in memory, each of them a NumPy array, all tied together by a shared index. This origin explains almost everything that surprises you later: why looping over individual rows is so expensive, why a selection sometimes returns a view of the original data and sometimes a copy, and why a groupby does in a single expression what would otherwise take a loop plus a cache.
Still, the first reflex when you face a CSV file with a few hundred thousand rows is the Python loop. Open it, walk through it line by line, sum up the values, print the result. That works – for a hundred rows. With a million rows, you end up waiting and wondering why it takes so long.
The loop question is actually the smaller one. The bigger one is decided further up: an analysis is rarely an end in itself. In the end, a decision depends on it. A price gets adjusted, a budget gets shifted, a region gets expanded or scaled back. So the value of an analysis is not measured by how elegantly it computes but by how reliably its result supports a decision. And that is where the real risk lies: not in the runtime, but in whether the final number is right and whether I can produce the same number again tomorrow.
The toolkit of NumPy, pandas, and Matplotlib is the answer to this requirement, not just to the speed problem. Together, the three libraries form a path from raw data to the finished chart, and each takes on a clearly defined job. What matters is that this path becomes repeatable: the same code, the same raw data, the same result, no matter who runs it. In this article, I walk the whole path once: first fast computation with NumPy, then structured tables with pandas, and finally visualization with Matplotlib. At the end, there is a small pipeline that loads, cleans, groups, and plots. I focus less on individual function calls and more on the places where an analysis gains or loses its reliability. If you are still wondering how the data gets into your program in the first place, Reading Data in Python covers the basics.
Why not a Python loop? NumPy and the ndarray#
At the core of NumPy is a single data type: the ndarray. At first glance, it looks like a list, but the difference lies in memory. A Python list holds references to arbitrary objects scattered somewhere in memory. An ndarray is homogeneously typed and stored contiguously – all elements have the same data type and sit right next to each other.
This layout is the prerequisite for what makes NumPy fast: vectorized operations. Instead of iterating over individual elements in Python, the whole computation is handed off to compiled C code. An example makes the difference clear.
import numpy as np
values = np.arange(1_000_000) # 0, 1, 2, ... 999999
result = values * 2 + 1 # vectorized, no Python loop
total = result.mean()
print(result.shape) # (1000000,)
print(result.dtype) # int64
print(total) # 1000000.0
The line values * 2 + 1 computes on all million elements without a single Python for loop running. The expression is not only faster, it is also shorter and reads closer to the mathematical intent. That is the real gain: you describe what should be computed, not how to iterate over the elements.
Besides arithmetic operations, two tools show up in virtually every analysis. One is reductions such as .sum(), .mean(), .std(), or .min() and .max(), which turn an array into a single metric. The other is Boolean masks.
import numpy as np
data = np.array([-3, 7, -1, 4, 9, -8])
mask = data > 0 # array of True/False
positives = data[mask] # keep only positive values
clipped = np.where(data < 0, 0, data) # replace negatives with 0
print(positives) # [7 4 9]
print(clipped) # [0 7 0 4 9 0]
data > 0 does not produce a single True or False but a whole array of truth values. You can then use it to filter (data[mask]) or to replace values case by case with np.where. Again, no loop and no condition in Python code – the logic lives entirely in the vectorized operation.
Closely related to vectorization is broadcasting. It lets you combine arrays of different shapes as long as their dimensions are compatible. Conceptually, a scalar is stretched across the entire shape of the array – that is why values * 2 + 1 works without you having to inflate the 2 and the 1 to a million elements yourself. The same rule applies to two-dimensional data: a row vector with a matching number of columns can be applied to every row of a matrix, for example to subtract the mean from each column. Broadcasting saves you many explicit loops and keeps the code close to mathematical notation. The only important thing is to keep the shapes in your head: if the dimensions don’t fit, NumPy responds with a ValueError instead of silently computing something wrong.
Pandas: tables with labels#
NumPy is good at computing with numbers, but real data has structure. Columns have names, rows have meaning, and numbers are often mixed with text and dates. This is where pandas comes in, and pandas builds directly on NumPy. Underneath pandas tables are NumPy arrays.
The two central structures are Series and DataFrame. A Series is a one-dimensional column with an index. A DataFrame is a two-dimensional table with labeled rows and columns – essentially a collection of Series that share the same index.
import pandas as pd
df = pd.DataFrame({
"region": ["north", "south", "north", "west", "south"],
"amount": [120.0, 80.0, 45.0, 200.0, 95.0],
"channel": ["online", "store", "online", "store", "online"],
})
print(df.shape) # (5, 3)
print(df["amount"].mean()) # 108.0
Reading from a file usually goes through pd.read_csv, which turns a CSV file into a DataFrame. From then on, you keep working with the same table, no matter where it came from.
Selecting with loc and iloc – the classic stumbling block#
As soon as you want to pick out specific rows and columns, two access methods show up that beginners reliably mix up: .loc and .iloc. The difference is actually simple, but you have to learn it consciously once.
.loc is label-based. You access rows and columns by their labels, and – this is the quirk – the endpoint of a range is inclusive. .iloc is position-based. You access by integer position, and the endpoint is exclusive, just like regular Python slicing.
import pandas as pd
df = pd.DataFrame(
{"amount": [120, 80, 45, 200, 95]},
index=["a", "b", "c", "d", "e"],
)
# label-based, endpoint INCLUDED -> rows a, b, c
print(df.loc["a":"c", "amount"])
# position-based, endpoint EXCLUDED -> rows a, b, c (positions 0, 1, 2)
print(df.iloc[0:3, 0])
Both expressions return the same three rows here, but for different reasons: once because "c" is included as a label, once because position 3 is just barely left out. If you mix them up, you get either an off-by-one error or a KeyError. So: labels always via .loc, positions always via .iloc, and never mix the two in one expression.
For filtering, you use Boolean masks – just like in NumPy, except that the condition refers to a column.
import pandas as pd
df = pd.DataFrame({
"region": ["north", "south", "north", "west"],
"amount": [120, 80, 45, 200],
})
big = df[df["amount"] > 100] # keep rows above 100
big_regions = df.loc[df["amount"] > 100, "region"] # filter + pick column
The most common mistake: chained indexing#
Now to a pitfall that comes up regularly in code reviews and leads to one of the most searched warnings in the pandas world: the SettingWithCopyWarning. It is caused by chained indexing – two consecutive accesses in one expression.
# problematic: chained indexing
df[df["amount"] > 100]["tier"] = "high" # may write into a copy!
The tricky part: depending on the situation, df[df["amount"] > 100] returns either a view of the original data or a copy of it – and you cannot reliably predict which. If you then write into the result, the assignment may land in a copy that is immediately thrown away. The original stays unchanged, and that is exactly what pandas warns you about. The correct form uses a single .loc access that addresses the row selection and the column in one step:
# correct: single .loc access
df.loc[df["amount"] > 100, "tier"] = "high"
And if you deliberately want to keep working with an independent copy, say so explicitly with df.copy(). Then it is clear that changes should not touch the original. In my experience, this one rule – selection and assignment in a single .loc – saves you most of these warnings.
The second mistake in the same category is iterating over rows. If you come from classic programming, you instinctively reach for df.iterrows() and go row by row. That is slow and memory-hungry, because pandas has to build a new object for every row, and the whole speed advantage of the vectorized NumPy foundation is lost. Almost always, you can express the same intent with a column operation, with groupby, or with np.where – and it then runs orders of magnitude faster.
Grouping and aggregating with groupby#
Perhaps the most important operation for analysis is groupby. It follows a pattern known as split-apply-combine: the data is split into groups by a key (split), an aggregate function is applied to each group (apply), and the partial results are combined into a new table (combine).
import pandas as pd
df = pd.DataFrame({
"region": ["north", "south", "north", "west", "south"],
"amount": [120.0, 80.0, 45.0, 200.0, 95.0],
})
summary = (df.groupby("region")
.agg(total=("amount", "sum"),
avg=("amount", "mean"))
.reset_index())
print(summary)
# region total avg
# 0 north 165.0 82.5
# 1 south 175.0 87.5
# 2 west 200.0 200.0
Named aggregation, total=("amount", "sum"), is my preferred notation: in a single line, it says which column is aggregated with which function and what the result column should be called. The result is a compact table that you can process further right away. The .reset_index() at the end moves the grouping key from the index back into a regular column – handy if you want to plot right afterward.
Often the data does not live in a single table but is spread out. Sales figures here, region information there. To bring them together, you use merge, which works like database joins:
import pandas as pd
sales = pd.DataFrame({
"region": ["north", "south", "west"],
"amount": [165, 175, 200],
})
managers = pd.DataFrame({
"region": ["north", "south", "east"],
"manager": ["Ada", "Ben", "Cleo"],
})
joined = sales.merge(managers, how="inner", on="region")
print(joined)
# region amount manager
# 0 north 165 Ada
# 1 south 175 Ben
It pays to keep a close eye on the how parameter here. With how="inner", only regions that appear in both tables remain – “west” and “east” drop out. A how="outer" would keep all regions and fill the missing values with NaN. If you pick the wrong mode, you either silently lose rows or end up with unexpected gaps. It is worth checking .shape before and after a merge to see whether the row count matches your expectations.
The pipeline: load, clean, group, plot#
Let’s put the pieces together. A typical analysis always passes through the same stations, and it helps to have this flow in front of you as a picture.
flowchart LR
A[Rohdaten<br/>CSV-Datei] --> B[Laden<br/>pandas.read_csv]
B --> C[Bereinigen<br/>dropna / fillna]
C --> D[Transformieren<br/>groupby / agg]
D --> E[Plotten<br/>matplotlib]Grouping is usually preceded by cleaning. Real data has gaps, and groupby or computations on top of it are sensitive to missing values. Two tools cover most cases: dropna throws away rows with missing values, fillna replaces them with a default value. Which one is right depends on the question – sometimes a gap is a genuine nothing, sometimes a disguised zero.
import pandas as pd
df = pd.read_csv("sales.csv")
# clean: drop rows without an amount
df = df.dropna(subset=["amount"])
# transform: aggregate per region
summary = (df.groupby("region")
.agg(total=("amount", "sum"))
.reset_index())
Data quality and reproducibility as a decision risk#
The step df.dropna(subset=["amount"]) looks like a formality. It isn’t. The moment I throw away missing values instead of filling them, I decide which number ends up in front of the decision-maker. dropna shrinks the population and can make a region disappear that was simply poorly recorded. fillna(0) turns an “unknown” into a hard zero and drags every average down with it. Both are sometimes right and sometimes wrong, but never neutral. The question is always the same: is the gap a genuine nothing or a disguised zero? This decision belongs in the documentation, not made silently.
The second, more expensive risk factor is silent data loss. A merge with the wrong how drops rows without warning, a cleanup step quietly halves the population, and the bar chart still looks plausible. That is exactly the danger: a wrong analysis looks deceptively like a correct one. That is why I build checks directly into the pipeline instead of trusting what I see.
before = len(df)
df = df.dropna(subset=["amount"])
assert len(df) >= before * 0.9, "dropped more than 10% of rows"
assert df["amount"].between(0, 10_000).all(), "amount out of expected range"
Assertions like these cost minutes during development. A decision based on a population that shrank by ten percent costs considerably more, and the error is often noticed only after it has already had its effect. That is the real trade-off: a little friction while writing in exchange for a reliable number when deciding.
The third point is reproducibility. An analysis that I click together in a spreadsheet can no longer be traced after two weeks. Nobody knows which cells were touched by hand. An analysis as code is the opposite: the same code on the same raw data delivers the same result, at any time, for everyone on the team. The raw data stays untouched, every transformation is visible in the script, and the result can be traced back to its source. This turns a one-off number into a process that I can check, version, and defend. That is exactly what lifts an analysis from a nice chart to a basis for decisions.
Matplotlib: turning numbers into a picture#
At the end of the pipeline comes visualization. Matplotlib is the established library for it, and I recommend the object-oriented style from the start: first create a figure and an axis with plt.subplots(), then build everything on that axis. It is a bit more typing than the quick plt.plot() one-liner, but it stays clear as soon as a chart contains more than one line.
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.bar(summary["region"], summary["total"])
ax.set_xlabel("Region")
ax.set_ylabel("Total sales")
ax.set_title("Sales by region")
plt.show()
For the three most common chart types, there is a matching function on the axis. A bar chart with ax.bar(x, height) compares categories – ideal for the groupby result from before. A line chart with ax.plot(x, y) shows trends, such as a metric over several months. And a scatter plot with ax.scatter(x, y) shows the relationship between two variables. For me, labels via set_xlabel, set_ylabel, and set_title are always part of it; a chart without axis labels is only half a statement.
Who does what? The division of roles at a glance#
The division of work in the toolkit is clearly cut, and if you know it, you won’t reach for the wrong library for a problem:
- NumPy forms the numerical foundation: homogeneous arrays, vectorized computation, mathematical operations. Everything above builds on it.
- pandas takes care of structured, labeled tabular data: reading, cleaning, selecting, filtering, grouping, merging – the entire handling of real, messy data.
- Matplotlib handles visualization: the fully prepared numbers become a chart that carries a message.
This order is no accident. It describes exactly the path the data takes: from raw numbers through a structured table to a picture.
The common thread#
The real shift in thinking for data analysis with Python is saying goodbye to the loop. No longer going element by element, but treating whole columns and whole tables as a unit. NumPy does this with vectorized operations on the ndarray, pandas lifts the same idea to labeled tables, and groupby aggregates whole groups at once following the split-apply-combine pattern. But the gain is not just speed. If you know the two beginner traps, chained indexing and iterating over rows, you avoid the most common friction points and write code that stays just as calm at a million rows as at a hundred.
This toolkit has its impact where analysis meets decision. A pipeline made of code shortens the cycle from question to answer: if the raw data changes, the same script runs again, and the new number is ready in minutes instead of half a day of manual work. It reduces friction in the team, because everyone can reproduce the same result instead of deciphering someone else’s spreadsheet. And it raises quality, because checks and documented cleanup decisions make sure a wrong number gets noticed before it supports a decision. The limit is just as clear: these tools work well where the data fits in memory and a batch analysis is enough. With data volumes beyond a single machine or with continuous data streams, a different set of tools begins, and the art is recognizing this limit in time instead of forcing pandas beyond its scope.
The rest is practice: load, clean, group, plot, and go through this cycle until it becomes second nature. The tool only becomes useful when the steps stay reproducible and a second person gets the same number.
Further reading#
- Slide deck on this topic: https://slides.mikebild.dev/introduction-python.en.html
- Repository
introduction-pythonwith runnable examples for NumPy, pandas, and Matplotlib - NumPy – “the absolute basics for beginners”: https://numpy.org/doc/stable/user/absolute_beginners.html
- pandas – Indexing and Selecting Data (loc/iloc, chained indexing): https://pandas.pydata.org/docs/user_guide/indexing.html
- pandas – Group by: split-apply-combine: https://pandas.pydata.org/docs/user_guide/groupby.html
- pandas – Merge, join, concatenate: https://pandas.pydata.org/docs/user_guide/merging.html
- Matplotlib – pyplot API: https://matplotlib.org/stable/api/pyplot_summary.html
Revised and translated from German with the help of AI.