Validating Data Across the Workflow: Part I

This is the first part of a two part series on libraries used for data validation.

We like to believe our data is well-behaved. The API payload looks clean, the database schema seems solid, the DataFrame columns appear exactly as expected. But data has a talent for quietly drifting out of its assumed types or goes missing — an empty field here, a wrong type there, a “harmless” null that turns into a production bug three layers downstream. Python’s type hints help, but they don’t guard your runtime boundaries.

When an LLM is told to make data validation look humorous .. I guess it learned from my hobbies!

That’s where explicit validation enters the picture. In this first part, we explore four libraries — Beanie, Pandera, Cerberus, and Msgspec — each tackling data validation from at a different point in the process: API/input parsing, rule enforcement, persistence safety, and analytical integrity. The goal isn’t to compare these tools, but rather to see which tool can be useful at which spot in the process.

Why Data Validation Is Not Optional

Data rarely fails loudly. It doesn’t knock on your door and announce, “Hello, I am not what you expect.” Instead, it just slips through. Most of the time this is in the form of wrong types, a string where you expected an integer. Or a missing key that only matters three function calls later.

Catching these types of problems in the data early is the best and fastest way to avoiding errors. The longer you let them persist, the more performance expensive fixing them is and if you don't the error messages become less and less traceable.

Typed Python ≠ Validated Runtime Data

Type hints improve readability, tooling, and refactoring safety. They are incredibly valuable.

But they do not enforce correctness at runtime.

If an external JSON provides "user_id": "123", Python will not stop you unless you explicitly validate it. Always remember that Python type hints are basically a promise between developers. Validation is enforcement at runtime.

This distinction matters:

• Static typing improves developer experience.

• Runtime validation improves system reliability.

Modern Python gives us tools for both — and understanding the difference is the first step toward designing a reliable data stack.

Different Philosophies of Validation

Not all validation looks the same. In fact, the libraries we’re looking at fall into three broad categories.

At a high level, validation in Python tends to follow one of these patterns: -

• Typed model validation

• Schema-dict validation

• DataFrame schema enforcement (could be considerd part of the first category)

They solve similar problems — but they approach them very differently.

Typed Model Validation (Beanie, Msgspec)

Typed model validation starts from Python’s type system. You define a structured model using type hints. The library then:

• Parses incoming data

• Validates types and constraints

• Produces a well-defined object

This approach is declarative and strongly integrated with modern Python tooling.

With Msgspec, you define fast, typed structs primarily for parsing and validation at system boundaries (like APIs or other input option). With Beanie, you define document models that are validated and persisted to MongoDB — building on Pydantic-style models.

Conceptually, it looks like this:

You are saying: “This object must look exactly like this.”

The benefit:

• IDE support

• Refactoring safety

• Clear structure

• Often better performance (especially with Msgspec)

The tradeoff:

• Less dynamic

• More rigid structure

• Best suited when your data shape is well known

Typed validation shines at system boundaries — APIs, persistence layers, structured messages.

Schema-Dict Validation (Cerberus)

Schema-dict validation takes a different approach. Instead of defining Python classes, you define validation rules in a dictionary-like schema:

This approach is flexble and dynamic. It is also decoupled from Python's type system.

The benefit are:

• You do not need to provide type hints.

• Schemas can be generated dynamically.

• Useful for configurable systems where types can be changed by the user.

The tradeoff:

• You don't really get assistance from IDEs.

• No structured object returned.

• Slightly more verbose for complex models

Hence, schema-based validation works best when configurations can change dynamically and so you do not want to be stuck with specific Python classes.

DataFrame Schema Enforcement (Pandera)

DataFrame validation is a special case — and an important one, considering the widespread use of pandas DataFrames.

Once your data is in pandas, it becomes tabular. Types become column dtypes. Constraints become column-level or dataset-level checks. Pandera acknowledges that a DataFrame is not “just data.” It has structure:

• Required columns

• Dtypes that should not change

• Value ranges

• Statistical constraints

You define a schema that enforces:

• Column existence

• Data types

• Optional constraints like > 0 or within a range

This is not object validation, even without any of these it is still a completely legal pandas DataFrame. It is dataset validation.

The benefits are quite obvious and so are the uses. If you do not store data in a pandas DataFrame, this is of no use to you. If you do, then this can avoid a lot of problems.

The Core Difference

Typed models define objects.

Schema-dicts define rules.

DataFrame schemas define datasets.

Now that we have established the differences, we can look at how these libraries operate in practice.

Cerberus: Flexible Rule-Based Validation

If typed models feel architectural and rigid, Cerberus feels pragmatic and adaptable. It doesn’t ask you to define classes. It doesn’t rely on type hints. Instead, it focuses purely on rules.

And sometimes, rules are exactly what you need.

Dictionary based rules

At the heart of Cerberus is a simple idea:

'Define your validation logic as a dictionary.'

Instead of writing a Python class, you describe constraints in a schema structure, a dictionary with extra context:

This schema is quite self explanatory

• "type" enforces the expected data type

• "min" enforces numeric constraints (if we have a numeric type)

• "empty" declares whether an empty object is allowed (i.e. empty string, empty list, etc)

• "nullable" declares whether "None" is allowed.

You’re not defining an object. You declare a set of rules that your validator uses to check the data.

Note that in contrast to validation methods we see later. Cerberus does not really follow an all-or-nothing approach. Instead validation sets a flag and you can inspect the error, quite similar to how errors are handled in languages like Golang

We see that validation failed, but we can then inspect what happened

In this case we did everything wrong that one could have done wrong apart from using the wrong types.

Dynamic Validation

Because schemas are dictionaries, they can be generated during runtime and they can be modified during runtime. They can also be composed or merged.

For example, imagine:

• A multi-tenant system with different validation rules per client.

• An admin interface where validation rules are configurable.

• A system where allowed fields change based on feature flags.

These are situations where Cerberus shines, especially when the data can change during runtime. Since the schema is a dictionary, you can change it

i.e. if a certain condition is met, age in your schema is now a required data point.

Most importantly, you can do all of this without any extra data structures. This ability for dynamic validation is what sets Cerberus apart from the other libraries we are looking at and puts it in the realm of "business-rule boundaries", not just checking whether your data is complete but also whether it complies with what you need.

Since we are not using fixed data objects, we should have a quick look at how to validate nested data structures. We give a quick example

We can see here that the nested data simply gets another schema.

In the next section we will look at two different libraries that you would use before and after Cerberus. One for taking in raw data (probably from a JSON supplied from an API) and one to store it in a database.

Msgspec — Typed & Fast

Where Cerberus represents flexibility, Msgspec represents discipline and performance.

It lives at the start of the chain of data — where raw bytes enter — and it checks:

• Can this data be trusted structurally?

• Can we process it fast?

Struct-Based Models

Instead of schema dictionaries, Msgspec uses typed structs, i.e. inherited classes. You define your expected structure explicitly:

On first look, this is really just a typed class, or a class with a bunch of type hints. But under the hood, Msgspec performs highly optimized validation and parsing. You decode a JSON file into this type of structure as follows

The main abilities here are

• Type enforcement

• Field validation

In addition, it comes with high performance and you end up with a structured Python object to use. If the input does not match the structure, decoding fails immediately.

JSON Decoding + Validation in one step: Why bother?

Looking at Cerberus above we could have also just decoded the JSON data into a dictionary and then validate it with Cerberus to get a similar data validation outcome.

But the performance would take quite a hit in this case, which is especially a problem in high data volume environments. So we trade in the flexibility of Cerberus for performance and structured data at the end with Msgspec.

Since this decoding and validation happens at the point you receive the data, most likely from an outside source, like an API, performance can be key here.

Encoding works too

Although not the main context of this post, you can use **msgspec** for encoding of data too.

But especially when talking to some API it might be necessary to also quickly send some data back to access more. In which case both directions can be performed using Msgspec.

Performance but also Rigidity

It should be evident by now what the advantages and disadvantas of Msgspec are. It has high performance and provides structured outputs that can be worked with immediately. As a Python object you can add methods to these objects and thus create a powerful, robust system.

But you suffer in the realm of flexibility. Your data structures cannot be changed easily, you can at best provide a few possibilities. Hence you need to know in which format data arrives way in advance.

So the upshot is that you should use Msgspec if

• You need high performance.

• You have stable and clearly defined data coming in.

• You need a typed object immediately to use in your code.

Beanie — Don't save nonsense

Once data has made it past your input checks and possible business rules, the next step is how to store and retrieve it later. That’s where Beanie comes in.

Schema validated Documents

Beanie validates data using schemas, i.e. types plus constraints and rules, but it is not as flexible as Cerberus since it wants to save data to a database and that uses a fixed known data shape for storage.

In Beanie, your data models are documents. They have a schema, get validated on creation and can be serialized cleanly—more about that in a second. In the above example we can see this in play:

• user_id is not just an integer, it should be ≥ 1

• name cannot be an empty string

• age is optional, but if present should be ≥ 0

You can also clearly see the dependance and inclusion of Pydantic.

The big conceptual shift here is, you are not validating data you received or are currently processing, you are validating data you want to store. If those steps beforehand did their job, you are essentially looking out for errors your own system has brought into the data or you combined data that was validated correctly while independent but needs extra validation when combined.

ODM Layer for MongoDB

Apart from validation, Beanie is an Object-Document Mapper for MongoDB.

This especially means that it transforms between a BSON format (MongoDB's format of choice) and a typed document class, a fully typed class object. Of course querying is done in Python code, instead of shoehorning in some database query syntax.

This node that serializes and deserializes your data is of course the perfect spot to enforce schema rules. You can have a look at what you load/store

In a typical MongoDB setup, it’s very easy to accidentally store inconsistent documents over time—especially when different services or scripts work off the same database.

Using Beanie on all the scripts that access the database can minimize the risks. Basically saying: If I can't validate it, it does not go into the database.

Where Validation Becomes Storage Safety

MongoDB is schema-flexible by design. That flexibility is powerful — but over time it can lead to:

• Mixed field types

• Missing fields in older documents

• Diverging document shapes

Beanie prevents that drift by making your model the single source of truth.

In the example above, if a document cannot be represented as a valid User, it should not live in your users collection.

That’s our third level of validation, after data input, business rules, we now validate that the stored data keeps its integrity. After now storing it locally we can proceed to work with it, if this happens to be inside a DataFrame, we can add another level of validation.

Pandera — DataFrames Need Context Too

By the time data reaches pandas, one would think that it is fine, after all

• It came from your API and passed structural validation.

• You enforced business rules when processing it.

• It was stored safely in your database.

Unfortunately, tabular data has a quiet habit of drifting: A column changes dtype from int to float, a missing value sneaks in, a new column appears unexpectedly, a filter step drops required rows.

These are all scenarios that Pandera is made to deal with. It essentially treats a DataFrame as more than just data entries, but something that comes with additional rules.

Defining a DataFrame Schema

As the above suggests, with Pandera, you define a DataFrame schema describing thigs like: required columns, expected dtypes, and constraints on values.

In code this looks as follows

Again the rules are quite self explanatory. We give the expected types, and we demand that some values are ≥ 0 for the age and betwen 0.0 and 100.0 for the score. Again the syntax is very Pydantic inspired.

Validating a DataFrame

If we now create a DataFrame

we can validate it as follows

If everything matches the schema, validation succeeds and returns the DataFrame. If on the other hand validation fails, for example if we try

Pandera raises a detailed exception describing:

• Which column failed.

• Which rows violated the constraint.

• Which check failed.

Essentially all the details one needs to track down the problem.

Those pesky extra columns

Something that can happen easily in DataFrames is creating extra columns that should not be in the original data. This can be checked by using

Then any unexpected columns will also create an error in validation. This protect you from accidentally adding extra data to your DataFrame.

Decorators are really useful here

You are probably using DataFrames because you are working with that data, aggregating or transforming it. Hence a problem will occur when a method is trying to process the data. For this purpose you can decorate your functions to have validation included

Here the function is given a DataFrame and instead of a standard type hint, we also tell Pandera which schema class to use to validate the DataFrame before use.

It becomes even more powerful if we also add a validation for the output

And nothing hinders us to have two different Validation Schemas for input and output. This secures the function both from a faulty input and products the rest of the pipeline from a faulty output from the function.

DataFrame are fickle objects

The question where Pandera fits in is very clear. If you work a lot with DataFrames to process your data, it is a good idea to use Pandera to make sure that your data has the expected form at any point in the process.

This is especially true if you do any of the following

• Aggregate data → output does not have expected shape

• Add derived columns → you might not want these in your original data

• Join multiple DataFrames → dtype drifting happens here easily

• Perform feature engineering → empty entries can happen

All of these can easily break your desired data format with a simple mistake.

Of course one should not overdo it and validate after every single function call. A good model would be to

• Load data → validate

• Major Transform/Aggregate step → validate

• Export final result → validate (this one is probably not done by Pandera)

Validation Is Layered

At this point, something should be clear:

• Validation is not a single tool.

• It is not a single pattern.

• And it does not live in a single place.

Each library we explored guards a different boundary:

• Msgspec protects the data input — validating and parsing JSON payloads quickly and strictly.

• Cerberus enforces domain and business rules — and can be changed dynamically.

• Beanie ensures persisted documents remain structurally sound over time.

• Pandera protects analytical pipelines — where silent drift is most dangerous.

These are simply different layers. Modern Python systems rarely fail because an algorithm is wrong. They fail because assumptions about data go unchecked.

And those assumptions accumulate as data moves:

Incoming JSON
        ↓
Validated object
        ↓
Business logic
        ↓
Database
        ↓
DataFrame
        ↓
Aggregation / Features

At every stage, the shape, meaning, and guarantees of the data subtly shift. Validation is how you make those shifts explicit.

In Part II we will discuss more about the architectural decisions: When to validate? When does it become over-engineering? what are the performance trade-offs?