Designing a Clean JSON-to-Model Pipeline in Python

In many Python projects, especially those involving APIs, data ingestion, or automation pipelines, a surprisingly large portion of complexity comes from one simple task:

Turning messy JSON into clean Python objects you can trust.

The difficulty isn’t really the parsing itself — it’s structure, consistency, and type safety.

Real-world payloads often contain:

• deeply nested dictionaries

• optional or missing fields

• inconsistent shapes across records

• numbers encoded as strings

• lists of objects that should really be lists of primitives

If you don’t establish a clear boundary early, your entire codebase becomes defensive:

Such an approach does not scale well and it becomes an absolute mess very quickly.

A much cleaner architecture is to split the problem into distinct stages:

1. Select what you need from messy JSON (JMESPath)

2. Reshape it into a predictable structure (glom)

3. Materialize it into typed Python objects (dacite / cattrs)

4. Validate and fail loudly when reality disagrees

This creates a maintainable pipeline where each tool has a clear responsibility. Especially if the overall schema of your JSON data changes you will know very quickly where it changed and you can act accordingly. Note that none of this is technically necessary, you could easily write all of these manually with the same outcome. The presented packages are meant to make it more readable and easier to track down errors if your data schema changes.

Apparently ChatGPT thinks that this post would fit well into Zootopia...

The Problem: JSON Is a Loveable Liar

We will skip how the JSON is obtained for this post. It could come from an API or you directly read it in from a file. For now we assume that it looks as follows

At first glance, this looks fine. But it has many of the problems we identified above. What you want is something like:

• User(id=42, full_name="Me Not you", email="menotyou@example.com", roles=["admin","billing"])

• Order(id="A100", total=19.9, items=[...])

That means we need to transform both shape and types.

Instead of writing a custom parsing logic everywhere, we’ll build a layered pipeline.

“Select” with JMESPath

JMESPath is a query language for JSON. If you’ve used SQL for databases, JMESPath plays a similar role for JSON documents. It allows you to declaratively specify:

• what fields you want

• how to traverse nested objects

• how to project lists

One of its biggest advantages is clarity. You can inspect payloads without writing loops or conditionals.

Simple Example using JMESPath

For example, if you want to pluck a few fields

Running this would give us

{'request_id': 'req_3524', 'user_id': '21', 'first': 'Me', 'last': 'Not you', 'email': 'menotyou@example.com', 'role_names': ['admin', 'billing'], 'order_ids': ['A100', 'A101']}

This is extremely useful when you’re exploring unfamiliar data or debugging upstream changes.

Advanced Example — search for expressions

JMESPath isn’t a full programming language, but it can do useful list operations.

Output here would be

[{'id': 'A100', 'currency': 'AUD', 'item_count': 2}, {'id': 'A101', 'currency': 'AUD', 'item_count': 0}]

As expected, we look for all lists that are named orders and create entries including id, currency, and item_count.

JMESPath excels at selection and projection, but once transformations become more structural — renaming fields, inserting defaults, flattening nested objects — the expressions become harder to maintain. That’s where glom comes in.

“Reshape” with glom

If JMESPath answers “Where is the data?”, then glom answers:

“What structure do I want instead?”

Glom uses specifications (specs) that describe how to transform data from one shape into another. These specs are just Python objects — dictionaries, lists, and helper constructs — which makes them readable and composable.

This is one of the biggest architectural wins: your transformation logic becomes declarative documentation. This looks as follows in our example

Output

{'request_id': 'req_3524', 'user': {'id': '12', 'full_name': 'Me Not you', 'email': 'menotyou@example.com', 'roles': ['admin', 'billing']}, 'orders': [{'id': 'A100', 'currency': 'AUD', 'total': '19.90', 'coupon': None, 'items': [{'sku': 'BUN-111', 'qty': 2, 'unit_price': '4.95'}, {'sku': 'GOB-222', 'qty': 3, 'unit_price': '10.00'}]}, {'id': 'A101', 'currency': 'AUD', 'total': '0', 'coupon': 'WELCOME', 'items': []}]}

First a quick summary what we got

• Nested roles flattened into a list of strings

• Coupon normalized to always exist

• Names combined into a single field

• Structural consistency across orders

Note that we still have plenty of wrong types in this output. Let us quickly look at the three commands we used from glom in this little example

T - The Target

Think of T as the target that you are currently processing. Which means you can access it like a dictionary and concatenate values as we did here.

Coalesce

This returns the first successful result, when we search for the entry coupon.code. If it cannot find any, it will return a default value.

glom(...)

This is the main command that applies the specifications to the input payload and returns the structured data.

Typed Models with dataclasses

Dataclasses give you:

• explicit schemas

• type hints

• immutability (optional)

• IDE support

• predictable behavior

They turn “a dictionary with keys” into a domain object.

We are using frozen dataclasses here to make sure that any created instance is afterwards immutable, this of course depends on your application if you want this or not. We also use typing to give type hints. Note that these are only hints, for example

roles: List[str]

suggests that this should be a list of strings, but it does not enforce this. Two excellent tools:

• dacite — simple, focused, easy to start

• cattrs — flexible, configurable conversion framework

What we now need is a tool to get our data from the dictionary into the dataclass. This is a good point to mention again that you could do this manually, the suggested methods are meant to improve convenience, readability and error finding.

Straightforward dict → dataclass with dacite

Dacite is intentionally minimal. It solves one problem well:

“Given this dictionary, construct this dataclass.”

It works immediately… until types disagree

We could look at an output, but with our specifications in shaped (from the glom example) and types in Envelope (from the dataclasses part) we will end up with an error message anyway, since for example User.id is not provided as an integer.

This is not a bug. The library is telling you:

Your data does not match your schema.

That is exactly what we want at system boundaries. To remedy this we need casting rules

Now int and floats that are required get casted automatically from strings into their respective types.

This can of course still be broken. In the JSON data one could without problem change the quantity entry of order "BUN-111" from "2" to "two". This would create an error message even when using casting.

Conversion Engine with cattrs

cattrs approaches the same problem differently. Instead of “load dict into dataclass,” it provides a conversion engine.

This engine can:

• structure data into classes

• unstructure classes back into dictionaries

• register reusable hooks

• centralize conversion logic

That makes it attractive for larger systems or shared libraries. In a basic translation example this looks as follows

Again, our example ends up with an error message. In its basic form it will not convert types.

Instead of a config that told us to convert to int and float as in the dacite example, we have to register explicit converters. The ones we need are of course fairly simple

Which then produces as output

Envelope(request_id='req_3524', user=User(id=42, full_name='Me Not you', email='menotyou@example.com', roles=['admin', 'billing']), orders=[Order(id='A100', currency='AUD', total=19.9, coupon=None, items=[LineItem(sku='BUN-111', qty=2, unit_price=4.95), LineItem(sku='GOB-222', qty=3, unit_price=10.0)]), Order(id='A101', currency='AUD', total=0.0, coupon='WELCOME', items=[])])

The key architectural idea here is centralization. Instead of casting in multiple places, you define conversions once. In addition you can also define conversion rules with included logic. For example if you know that a quantity will come in the form '10.9k' you can include a conversion that transforms this to '10900' directly.

When to Reach for Which Tool

A helpful mental model:

• JMESPath — exploration and extraction

• glom — structural transformation

• dacite — simple dataclass loading

• cattrs — configurable conversion layer

Or more conceptually:

• JMESPath answers: “Where is the data?”

• glom answers: “What shape should it be?”

• dacite/cattrs answer: “What is it?”

Each layer reduces uncertainty.

The main take aways from this are:

• Keep transformation specs centralized. A glom spec can be read similar to a documentation on how you want to have your data be presented.

• Prefer loud failures. Silent coercion is technical debt with interest. If the failure comes because data does not match it should not convert. You can always make sure to catch these errors and save the problematic payload somewhere for a manual check or a more in depth logic that tries to figure out what is the problem—remember 'two' vs '2'.

• Use JMESPath for debugging. It’s fantastic for inspecting payload changes quickly.

• Write a golden payload test. One or two representative JSON samples catch many regressions and type problems.

The Upshot ... or Morale of the Story

This stack works well not because it uses a particular set of libraries, but because it introduces clear separation of concerns. Extraction, reshaping, conversion, and validation each happen in their own step, which makes the overall system easier to reason about and maintain. Instead of mixing data access logic with business logic, you establish a clean boundary where raw, untrusted JSON is transformed into structured, typed objects.

Once you reach something like Envelope(...), the rest of your application can operate with confidence, without constantly checking for missing keys or incorrect types. That shift — from defensive programming to working with trusted data — is where the real value lies.