Day 9: Data and Definitions Split Apart, So Forms Need No Types of Their Own

One request travels front end, API, business object, repository, database, and every layer has to receive the same record. What most systems put here are types they wrote themselves: an Entity per table, a DTO on each side, and a layer of Mapper configuration in between.

This framework puts two things there: a data container (DataSet) plus a definition running alongside it. The data is in the container, and everything said about the data is in FormSchema. What that split buys is one thing. Eight forms or a thousand, not one extra type appears in between.

This post covers:

1. Why a data container rather than strong typing
2. Writing no DTO means pulling data and definition apart
3. What crosses layers is data plus its description
4. What checks the things the compiler cannot, in two stops
5. These costs are paid for the sake of a thousand forms

Why ERP Needs a Definition-Driven Framework counted an expected delivery date on the order landing in thirteen places, four of which were the Entity, the DTO, the Mapper and the front-end type. Why those four can be empty is what these five sections add up to.

1. Why a data container rather than strong typing

The pain: a user asks for an expected delivery date on the order, with no calculation and nothing linked to it. You open the Entity and add a property, open the DTO and add one, open the Mapper and add a line, open the front-end type and add one more. Four files, one compile, one deployment.

That field does nothing at all in business terms, and it has left four signatures in the type system.

Then multiply by a thousand tables, and by dozens of requests like it a year.

Whether a cross layer container should be strongly typed has had two camps with their own reasons for years, and this is not judging between them. What strong typing buys is clear, namely that the compiler watches the names for you, and that is exactly what gets paid here. This section only answers what kind of thing this framework needs in this position.

The answer is a container that does not yet know how many fields it has at the moment of compilation, for two reasons:

• When the shape is settled: which fields a form has is written in the definition, and the definition is data, read in at run time. At the moment of compilation, the fact of which fields an order has does not exist. It also carries on changing, because that data is there precisely so that consultants, tools and even users can edit it.
• When the type has to exist: an earlier post, An Overview of Definition-Driven Architecture, drew a line between producing code at build time and producing objects at run time. A strongly typed cross layer container means that type has to exist at build time, so every definition change means regenerating, recompiling and redeploying. However cleanly everything upstream derives, one compile time type in the last mile grows the whole delivery path back.

The first post said that going from a definition change to it taking effect involves no compiling, packaging or deploying. Half of whether that sentence holds is staked on this layer.

2. Writing no DTO means pulling data and definition apart

The second pain is less glaring and costs more time.

What a field actually has to state goes well beyond what it is called and what type it is. Is it required? Does it show in the list? How wide? Is it currency, and to how many decimal places? It points at the customer form, so once one is picked which values get written back into which fields? When picking through a lookup window, should only the active ones be listed?

None of that has a place ready for it in a DTO. What a type can express is a property name and a property type, and everything else has to be stacked on as a pile of annotations, read by layers that are not this one. So they scatter:

The cost of scattering is not more files, it is that no one place is authoritative. The same thing is declared separately in several places, any two of them can fall out of line, and when they do nobody reports an error.

The framework pulls the two apart: data goes in the container, and everything said about the data goes in the definition.

Once they are apart, the description no longer has to be squeezed into the type system. An earlier post sorted what a single field can declare into six categories, being structure, behavior, presentation, semantics, relationships and security, and what a property can write down by itself is the first two items of the first one: the field name and the data type.

Relationships among those six need a few more words. That they are declared between forms, pointing at the customer program, came up in The Northwind Case Study: Eight Forms and a Custom Business Object. A type can hold a reference to another type, but that reference says only that it points there. It cannot say which columns the list shows, what a person sees in a lookup window, which fields are queryable, or whether to add conditions of its own. The relationship between forms is wider than the relationship between types, and forcing it into types pushes those extra things somewhere outside the types.

Pulling them apart returns one more thing:

A description on a DTO can only be read by the compiler. A description pulled out of it can be read by anyone.

3. What crosses layers is data plus its description

Writing no DTO does not mean throwing the data into a shapeless bag. The shape is still there. It just is not a type, it is a description traveling with the data.

On the way out, one table carries:

• Its own name
• A field list
• The values of each row

Whoever receives it does not have to know it in advance. Because the description arrives with the data, there is no such thing as a type mismatch, and because every layer receives the same object, there is not one turn A into B step between reading from the database and feeding a front-end control. What is gone is not just those few lines of Mapper configuration, it is every error those lines can make.

A container can hold more than one table, and almost every document in an ERP needs that:

• The order's container holds two tables, one master and one detail, with the detail hanging back onto the master by sys_master_rowid
• Open an order and master and detail come back together; press save and they go out together
• How many tables to send, which one is the master, and which column the detail hangs back by, are all things FormSchema has already said

Every row also carries two more things: whether it is added, modified, deleted or untouched, and what its original values were. That lets the save trip travel with no explanation attached. The user changed two fields on the master, deleted one detail line and added another, and what goes out is the container itself, with the server deciding which statement to issue from each row's state.

The cost was recorded in an earlier post's boundaries of use: every response carries the container's field structure with it. For one order that is worth it. For an event stream at tens of thousands per second it is not.

4. What checks the things the compiler cannot

Once data and definition are apart, the joint between them is the field name, and a field name is a string.

row["amout"] is one letter short and the compiler says nothing, throwing only when that line is actually executed. Worse, failure has more than one shape. A typo in a list column just means one column missing on screen, while the same name going into a query fails outright when the SELECT is assembled. A typo in a lookup window column is silent from start to finish: the column does not appear, and the log is clean too.

The same name written inconsistently across layers is harder still to track down, and some front-end controls bind case sensitively. Column names in the container are therefore always lowercase, lined up with the definition files, the database columns, the expression identifiers and the front-end binding keys. There is one set of names the whole way through, so errors of the casing kind have no room to happen.

The compile time gate cannot be won back, and there is no need to pretend it can. Asking to check at compile time a shape that is decided at run time is a contradiction inside the request itself.

What can be done is to move the moment of discovery forward, to the point where the shape is already settled. Once the definition is written, the shape is settled. Moving it forward has two stops.

The first stop: build time

The framework puts these checks in Bee.Analyzers, shipped inside the definition layer package, in effect as soon as it is referenced, with nothing extra to install. Most of them read not code but the XML of the definition files themselves.

Nineteen rules in four groups:

1. Whether a single definition contradicts itself
2. Whether several definitions line up with each other
3. Whether the code holds to framework conventions
4. Whether the shape of types taking part in transport is right

The two below are actual output. Type ref_customer_name in the order's list columns as ref_customer_nmae:

error BEE1004: FormSchema 'Order' lists 'ref_customer_nmae' in ListFields, but the schema declares no such field. The list layout drops the unknown column, but the query keeps it: building the SELECT throws InvalidOperationException at run time. Fix: correct the name, or declare the field. 

The message has three parts: where it is wrong, how it will fail at run time if left alone, and how to fix it. The middle one is what most has to be there, because how it breaks at run time is exactly what decides how easy the error is to track down.

The second answers a cost from TableSchema and Database Structure Upgrades, a field added to FormSchema and forgotten in TableSchema:

error BEE2006: FormSchema 'Order' declares persisted field 'required_date' on table 'ft_order', but the table schema has no such column. Queries touching that field fail at run time. Fix: add the column to the table schema, or mark the field Type="VirtualField" if it is not persisted. 

Both builds stop on that line of the definition file. Missing one of the three places the first post counted for adding a field now ends as one failed build, rather than a support ticket after go live.

The second stop: integration testing

The first stop catches a definition written wrongly. What it cannot catch is the definitions all being right while what this document produces comes out wrong, and that takes data. The approach:

1. From the definitions plus the existing data, export an Excel import format for each kind of data. Which columns this sheet needs, what type each is, which ones a relationship brings in, and which are computed and need no input, are all already answered in the definitions.
2. Work the test data up in full in Excel, one file per kind. This step is the key one: whoever does it does not have to be a programmer. How many kinds of customer an order should cover, how many discount combinations, which boundary values simply have to be tried, are known best by the people who understand the business, and Excel is what they already use.
3. The test imports them in order, master data first and documents after, which is the order of the real operation. The data can be fed to the API or straight to the business object, and the container both entry points receive has the same shape.
4. Finally take a few key reports, turn the results into Excel, and compare them cell by cell against an Excel file already known to be right.

The first three steps only prove the data goes in. The last one is what asks whether the result is right. Pin both ends once each and the whole stretch in between is under test, without an assertion written for every step of it.

This can be written as a tool because the description was pulled out. It is data a program can read, obtained by asking the API once, so the tool does not have to know any form in advance, and adding a form or changing a field does not move it either. After a release, run the whole batch again and you know whether anything broke.

The route skips the front end on purpose, and the reason is not to save effort. The controls on screen are a set the framework owns, which control a field uses is computed from the definition, and a field of a given type grows the same control on every form and runs the same code.

So the shape of how errors are distributed changes: when some control behavior breaks, it breaks across the board, not on one form.

A point everything converges on is what FormLayout and Screen Generation for Multiple Front Ends called the rendering layer, where a hand written screen gets one screen wrong at a time and the rendering layer gets all of them wrong at once. That was said about risk. In testing it is the other side of the same thing: something that goes wrong everywhere cannot hide, and opening any form at all runs into it. Clicking through screens one at a time is the same code on its Nth execution.

5. These costs are paid for the sake of a thousand forms

A complete packaged ERP routinely has thousands of business tables, heavily interlinked. At that scale a set of types per form is not an inconvenience, it is unmanageable, and the costs Definition-Driven asks for are only worth paying at that scale either.

The first return is that adding a form takes no code: three new files plus three rows of registration and it grows. Seven of the case study's eight forms came from exactly that, with data traveling from the database all the way to the screen through every layer in between and not one line written by the application. The remaining one is the order, which added a few rules the definitions cannot state.

The second return is cost turning from multiplication into addition. Add a field and FormSchema gains a line, TableSchema gains a line, FormLayout gains a line, and then the database has one more column, the CRUD statements one more, the container one more, the field list sent to the front end one more, and the screen one more control. There is not one intermediate type anywhere along it. Four of the thirteen places the first post counted are not easier to edit in this model, they do not exist.

Adding a relationship or a front end is addition too. In the world of types, every extra relationship means deciding how to express it, when it loads and who maintains it. Here it is one more declaration in the definition, and the display columns it brings back are a few more columns in the container. An earlier post said supporting one more front end costs one rendering layer, and that front end does not have to bring types of its own, because the data it receives carries its own description.

The three parts of the split, and that bill

Data and definitions splitting apart, so forms need no types of their own: what that sentence really says is not that a few files are saved. It is the sum of three things:

• The container's shape is decided by the definition, so it does not have to exist at compile time
• The description travels with the data, so no layer in between has to know it again
• Once the description is pulled out, those kinds of statement are no longer readable only by the compiler

Take away any one of the three and the split does not hold.

The split has a cost. The compiler now watches one category less for you, and that bill has not disappeared, it has moved to those two stops to be paid. Errors visible from the definition itself are caught at build time. What cannot be caught there gets a run with data prepared from the same definitions, with the reports compared cell by cell at the end.

Whether a design is honest is not about whether it has a cost, it is about whether it knows where its cost lands.

The next post, Multi-Database Support: CRUD and Structure Upgrades, is about how this data finally gets into a database: how a framework that knows no database driver in advance still produces CRUD statements for every one of them, and can change their structures too.

📚 Part of the thirty-post series ERP Architect's Notes: Designing a Definition-Driven Framework

📘 Original article on HackMD, in Traditional Chinese:
👉 https://hackmd.io/@jeff377/day-09-data-and-schema-split

📢 You are welcome to republish this, please credit the source.
📬 Follow for more engineering notes:
Facebook ｜ HackMD ｜ GitHub ｜ NuGet