Do Androids Dream of Microsoft Excel?

originally published November 9th, 2023

This article compares and contrasts techniques for development data management processes and fully automated business processes. What outcomes do we get when we use tools suited for one to build the other?

Data Modeling

A data modeling approach to software design emphasizes state changes in order to implement use cases.

There are two primary approaches:

  • Data Mapping is the pattern of injecting data into object fields, typically using reflection.
  • Active Record is a pattern in which data is held in an internal data structure and accessed through publicly exposed virtual properties on the object.

Designing software using these ‘data bags’ is the current industry standard. Most education programs, instructional videos, articles, etc instruct beginners in this approach. Most software development frameworks provide tools and education to support this paradigm.

Behavioral Domain Modeling

Behavioral domain modeling is an approach to software design that emphasizes behaviors over state. Domain behaviors are implemented as code constructs which relate to one another similarly to how the concepts in the business’s conceptual model relate.

This is sometimes considered to be the Domain-Driven Design (DDD) approach due to the emphasis that DDD places on domain modeling. However, it’s not exclusive to DDD.

Implementing Business Processes

Manual / Data Management Implementation

Many manual or technologically assisted business processes leverage software but rely on human operators to understand and implement the business processes.

Data management tasks such as those solved with standardized content management systems (CMS) are often utilized by businesses to implement processes. These processes are often understood and implemented by a human which uses the CMS.

An example is discount code management for Black Friday sales. For Black Friday, our business wants to identify items which we have in our warehouse in high quantity and items which we sell with particularly high profit margins. We’ll discount market and discount those products in order for Black Friday.

A human can implement this business process with an ecommerce system by analyzing inventory and sale profit margins before manually adding discounts for these products.

Evolution of Manual Processes

As business intelligence leads to changes in the business process, the implementation of the process must change. For manual or assisted implementations, this means that the humans performing the behaviors must understand the new business processes and the software must be changed only so much as is necessary.

Software modification requirements are likely minimal to none.

In our Black Friday example, a year passes and we are one year wiser. We have some new ideas about how to approach our discounts. We identify certain trends that we want to emphasize in our marketing and discounting behavior.

Once a human is up-to-speed on our goals, they will quite possibly be able to implement the changes without paying programmers to modify the ecommerce software.

Fully Automated Implementation

The primary difference between manual / assisted processes and fully automated processes is that human understanding must be encoded in software. It’s possible for us to identify a handful of business rules, then encode them such that Black Friday discounts are managed entirely without human analysis or human data entry.

First the business must understand precisely how it wishes to approach a problem. Then programmers write the logic into software. The software needs access to all the necessary data (including inventory and product profit margins) and it needs the capability to make changes to product discounts on a schedule.

Once this software is written, it should be able to operate autonomously each and every year.

Evolution of Fully Automated Processes

Our system automatically handles discounts every Black Friday. However, when our business process intent changes, it’s not enough just to teach a human the new process. Fully automated business processes are coded by developers. All the logic that a human might use to implement the process themselves is encoded in the software and that code must change.

Software modification requirements are possibly significant.

Requirements of Software

The requirements of software which supports partial or fully automated business processes are distinctly different.

Data Management

Business processes implemented by a human generally involve one primary software interaction, state manipulation. The human needs to be presented with the system’s state. They then use their knowledge of the process to directly modify the state as needed in order to satisfy the functional requirements.

At its core, the software’s job is to present the human with the ability to make direct state changes. Changes to the business process can largely be made by changing the human’s understanding, often with little or no changes to the software itself.

  • Process Knowledge: in a person’s mind
  • Software Function: present user-interface for direct state changes
  • Process Change: education of the person

The software which assists in the implementation of these processes primarily offer up lists of data and forms for modifying data. The human user-interface is the most prominent and important aspect of this implementation.

Humans often implement business processes by making changes to spreadsheets. Common software development practices use relational spreadsheet-like idioms to implement data management software.

Full Automation

Fully automated business processes must incorporate data access, intelligence, and state changes sufficient to satisfy the functional requirements. This is significantly different from manual or assisted implementations.

  • Process Knowledge: encoded in the software
  • Software Function: access and analyze data, make decisions, trigger state changes
  • Process Change: alter the encoded process in the software

The software which implements these processes contain far fewer lists and forms. Human user-interfaces may not be required at all. Systems might need to integrate with others in order to access the data which will be used as business intelligence and decision-making logic must be encoded.

Summary Comparison

Perhaps the largest distinction between these two approaches is the difference between how process changes are implemented. In one, human knowledge is changed. In the other, software implementation must be changed.

The more automation in the process, the higher the expense of implementing the change. Fully automated processes can bring great efficiencies and empower the business in ways that could not be implemented manually. But it comes with the cost of necessary software updates. These updates can range from being relatively minor to being significant ongoing costs.

Concerns for Software Design

Since the software evolutionary profiles are significantly different, it’s easy to understand that the concerns surrounding the design of the software are also different.

Data Management Software

Data management software assists with only a small fraction of the business process. It may be a pivotal tool, but it generally has the luxury of being a relatively stable tool, requiring changes in order to conform to updated business processes.

Effective data management software provides easy access to the data. A lot of focus is on making the user-interfaces easy to use. Perhaps batch operations are exposed.

Additional data or changing the data structures should be easy for developers to do.

Fully Automated Software

Because automated software implements more of the full business process, the requirements for change tend to be far more numerous and the lead times for these changes are much longer. The addition of additional concerns results in much more code to write and maintain.

The time required to change software is generally much longer than the time it takes for people to adopt new ideas. Consequently, a property of effective automated processes is to support change without prohibitive lead times. Long-term maintenance of the software is almost always 10, 100, 1000 times longer than initial development. Small compromises are made up front in order to reduce long-term lead times and to manage the significantly larger amount of essential complexity in such processes.

Finding one’s business in a situation in which their software is unable to react quickly enough to support the business is no fun; but it’s one in which software developers find themselves in nearly constantly.

Manual Processes May Become Automated

Businesses often begin by implementing processes with people before automating these processes over time. We start with data management solutions built using approaches that emphasize state manipulation. Then as the business evolves and requires more fully automated processes, we fail to identify the fact that the requirements for what makes this software solution effective have changed.

If we do not change our software development approach then we may find ourselves using data model manipulation in scenarios which are better suited for behavioral models.

The Cost of Only Automating the Human

The software industry currently utilizes the data model approach to software design as a one-size-fits-all solution.

  1. Imagine a human interacting with data management software.
  2. Imagine the touch-points a human has, the situations in which they change data.
  3. Leave the data management back-end.
  4. Automate the interactions the human would have with the data. Do androids dream of Microsoft Excel?

This is the approach that has led countless software development teams toward designing software that ages poorly, inevitably requiring a rewrite, which is then followed up with new software written using exactly the same techniques.

Developing software with ‘data bags’ and elevating the process of manipulating state creates expensive to maintain, highly coupled software.

Instead of just automating the human away, we can replace both the human and the spreadsheet paradigm with an approach that better suits the problem.

Building Effective Automated Business Processes

When building automated business processes, the most important consideration is “how can we effectively evolve the system over years with the shortest possible lead times?".

A few important concerns can help illustrate some important design decisions:

  1. how will development need to scale?
    • are there certain parts of the system that will be managed by single teams?
    • are there certain parts of the system we want to scale by adding many teams?
  2. how can me manage the essential complexity of the business processes over time?

Scaling Teams

Some types of software production can be accelerated by putting more people on the job. Some can’t. Even the types of work that could support this kind of scaling can not do so unless the software is designed to allow for that.

In order for multiple teams to exist, boundaries must exist.

Without effective boundaries, teams are like climbers chiseling faces into the side of a mountain cliff. Whoever happens to be hanging on a rope closest to the area that needs work owns that work.

We must make deliberate decisions about how work SHOULD be shared in our systems. This is an ongoing and fluid process. When we decide that work should be shared in a NEW and DIFFERENT way, we must first engineer the system to SUPPORT that kind of work. Otherwise, we run into the very likely possibility that we will lose much time and energy locked in communication and consensus making with other teams.

This is the “distributed monolith" version of team splitting. Yes, you CALL them different teams, but they really are just sorta one team with poor communication lines.

Effective boundary design is one of the most critical considerations to make when designing software because it helps you to achieve your business’s goals for .. not JUST scaling and modifying your software .. but for scaling and modifying your business itself.

Business Understanding Changes

The business has a conceptual map of its problem and solution spaces. Business solutions are derived from the business’s understanding of its problems. As the business understanding changes, so do its solutions.

Because so much of the business process is programmed into our software, it’s easy to understand that large business changes result in large software changes.

Thankfully however, there are ways that we can mitigate the need for disproportionate changes between our business understanding and the software.

Managing Complexity With Domain Modeling

The business thinks about solutions in behaviors, not state changes. Many of us have trained ourselves to think about all of the fields and database tables needed when business people are explaining their requirements. That is not an effective approach.

Instead, when the business is talking about behaviors, we should be thinking in behaviors. Yes data is part of the process. But this is a DISTINCTLY different approach to building software with data models. Data models do not help us encode behaviors, they help us to encode state changes.

Data Model Example

$order = Order::create();

$order->discount_code = value;
$order->other_field = value;
$order->something_id = value;
$order->other_id = value;
$order->timestamp = value;

$order->save();

In the above example, a “model" is created. This model represents a row in the database. Data is assigned to database fields and eventually the row is saved to the database table.

What are we DOING in this example? What is the emphasis? The emphasis is on manipulating a database row.

Domain Model Example

$order = Order::place(
   $discountCode,
   $otherField,
   $somethingId,
   $otherId,
   $timestamp,
);

What are we doing in THIS model? We are calling the behavior place(). We are passing the data required and relevant to this behavior.

The EMPHASIS is on the place behavior.

Domain Models and Automated Processes

Behavioral domain models are ideal for fully automated business processes. They are well-suited to being built to mimic the interactions of important business concepts. When software is built to mimic the concepts of the business, we can obtain that PRECIOUS property of small business changes resulting in small code changes.

This property is worth the effort, but it’s not reliably possible without designing software to match the business’ conceptual model.

Data Models and Automated Processes

Data model objects necessarily leak database state and concerns. That is their purpose and their power. They provide a powerful interface for building data management systems.

However, they are a poor fit for fully automated business processes. They emphasize state change over behavior and as a consequence they struggle to mimic the conceptual relationships of the business’s understandings.

Where are Data Models used most effectively?

Data models can be used to great effect in any system in which emphasizing state changes is beneficial. Common examples include highly generalized tools such as CRMs, eCommerce applications, or other data management systems.

Consider this, Active Record is a pattern that makes it possible / trivial to manipulate software using database schema changes, even without code changes. That can be quite powerful in the right situation.

Active Record is often used in the early days of start-ups where the business needs could change at the drop of a hat. This is where relying on human understanding to change faster can sometimes be a desirable alternative to having to change fully automated processes. This is highly situational.

Where data is basic unit of manipulation, that is where ORM data bags shine.

Tools such as Active Record models CAN be used to create highly complex software. However, it’s extremely difficult and unintuitive to design these systems so that they won’t result in explosively expanding lead times.

Where are Domain Models used most effectively?

Domain models require knowledge of business processes. Real effort is put into how the code is built so that it matches business concepts. It does not always shine when implemented in situations which have few to no real automated business processes or in the early days when the nature of business problems / solutions are not yet understood.

Domain models shine when the business problems and solutions are well-defined. This is one reason why it’s so often paired with the strategic aspects of Domain-Driven Design which seek to combine business understanding and discovery with software design.

Data modeling is the industry standard. Businesses able to leverage domain modeling in fully automated business processes can benefit from quicker times to marker.

Behavioral domain models help to mitigate the complexity of designing software by easing the process of designing boundaries which enable teams to scale in the way that the business intends.

Additional Resources

Theme Selection: