geekyprimitives

The Paradox of core versus support domain

This is not new post. This is something I published over two years ago. My former CTO came down from his ivory tower and asked me to remove this post from Internet. He sent me long email, about me being unprofessional, because people can connect through my LinkedIn profile where I work, and connect my writing with the state of the product. Brilliant. I leave the judgement for you, think for yourself and question authorities, especially fake, usurpatory authorities.
So why I publish this post once again? Even if I don’t like it and I don’t think is even worth reading and I disagree with few points.
Because I promised myself that one day life will move on. Life will surprise me and changes will come.
It is time for a change.
Why it took me so long?
Because for past 4 years I worked with the best team ever. I was waking up every day not to write code, because in this system it was impossible. I was waking up to meet crazy, passionate freaks like me. We were team, we were one. The chemistry between us was unbelievable. And people hated us, which was kind of cool for the twisted personality like me.
We were working from coffee shops (Choco Cafe on Wiślna street simply rocks), we were challenging our limits every day. We were not thinking out of the box, we were out of the fucking box. Box full of rules, regulations and office politics. Writing custom parsers to do code translations, instrumenting code for transaction deadlock detection, horizontal scalability based on custom JDBC driver, parsing tcpdumps full of JGroups shit, cutting, mangling code bases going from 40 minutes (really) to 8 minutes of compile time, migrating technologies like crazy, you name it, we did it. It also included tons of bad decisions, incorrect estimates and broken promises. But we enjoyed it and other people hated us, because we were arrogant dreamers, believing in code, trying to change world, taking proud of their work. People hated us because we had a whole lot of fun, we were smiling, we were doing things we wanted to do. What I have learned is that people don’t like when other people have fun, when they enjoy their work. Sad. My biggest mistake as a leader of this team? I became one of you, Jose, Tomek, Marcin, and second Marcin, Jacek, Damian, Paweł. I was so excited about things we were doing that I didn’t noticed what is going around. My mistake and I would repeat it over and over again. Because your job is not your life, life doesn’t end there. There is meaning behind deadlines, overtime, failed builds, stupid code and pointless discussions about JVM internals. Behind all these stupid things, there are people and memories and endless stream of shots of cherry vodka. Guys this is for you, my, big, personal, thank you. In a year from now I won’t remember this code and projects. But will remember you.
The rest of this post is irrelevant. It is just promise I made to myself.

Systems grow, systems change, systems evolve, systems become “big ball of mud”. I am not going to start a discussion about the forces which turn beautifully designed systems into “big ball of mud”, not this time. It just happens, it comes and goes, just like financial crises, seasons, ice age.

One day, you wake up, and you find yourself in the middle of deconstruction of your “big ball of mud”. Your “mothership”, your flag product, your “darkest matter of the universe”. You may ask youself, what is next? How to do it right? This time.

First of all you should feel respect to all people before you, who grew and developed this “big ball of mud” years ago. I am not cynical, really. It takes courage, knowledge and passion to build such platform, taking into account market pressure, available resources and time.
You have to understand that every system has its limits to growth, and you hit this limit. All knobs are at the maximum, all parameters are pushed to limits. It is time to stop, and think about next leverage point which you can use to extend your platform capabilities.

In the early stages of the process, you still don’t know how you are going to do it. There will be endless discussions about end state, there will be flame wars, there will be forces that will push the system in a direction you don’t want to, but there is one goal we all share.

We all know that at the end we want “bigger, better, faster, cheaper” platform to live with. We also know that given the size and complexity of the platform, this is not one month project. Even with limitless resources, due to amount of dependencies, we are not able to attack parts of the system simultaneously. One more reason why dependencies “suck”, and you should do everything you can to avoid them. So at the moment the most important question is where do we start? Which pieces should go first, to get maximum benefits of deconstruction (some people call it rewrite) in shortest time possible. Of course there is always temptation to start from a module which is a dependency of majority of other subsystems. But I was way too many times in such projects. What is wrong with this approach? You rewrite, what you think is a heart of the system, and what I call “dependency hub”, at the next day your system is “dead”. You have tons of issues to solve, issues in many distant parts of the system. Why? Because you got it wrong. You probably didn’t notice this subtle dependency here and there, you didn’t understand the contracts between dependent parts. And let’s be honest, you are probably not the original author of the code, and yet you have enough arrogance to start from the most complex and important part of the system.

So where to start? How to set priorities right? My answer is somehow inspired by Eric Evans talks (especially Eric Evans on DDD: Strategic Design
, and Strategic Design – Responsibility Traps), and Domain-Driven-Design Distillation: Support, Generic and Core Domain, and some of the concepts from system thinking.

According to Eric, you start by identifying your core, support and generic domain. Where:

• core domain, is the part of the domain most closely associated with the strategy of the company
• support domain, is the part of the domain that indirectly supports the core domain without actually belonging to it.
• A generic domain is one that is universally well-known, without any need for specialization in the core domain.

“Domain-Driven-Design Distillation: Support, Generic and Core Domain” at ZenModeler

How do I differentiate domains?

I always try to look from customer’s perspective. Let’s take example of banking system, for example credit scoring. These systems calculate your credit score. If you don’t pay your bills and credits on time, these systems will score you low. You will not get a credit or bank will expect from you additional insurance. It is its core domain, to provide ability to calculate credit score of a person using bank’s secret algorithms. Every algorithm needs data to do its work, to have business value. The problem with credit score is that data is distributed across different institutions, companies. Telcos, other banks, electricity and internet providers. They all have information bank needs to run their secrect formula.Imagine that somebody in a bank getss these data from other systems manually. They go to simliar systems at telcos and other banks and copy and paste records to feed their own algorithm. They make mistakes, the get data only from couple of places, because it takes time. Algorithm is malnourished or, what’s even worse, feed with wrong data. And bank makes decission based on this.
What if we can integrate with other systems using some agreed machine to machine protocol? And let this integration run in background.
This integration belongs to support domain. This is what makes this system usable.
Generic domain include things like security, support for transactions, and other nice things which cut your platform through layers. But don’t get attached to it. Security can be your core or support domain. When? When it directly supports your company strategy. Like in our case, we run SaaS system, and multi tenancy is significant part of our strategy and every user story. And our security model has to support multi tenancy. In this case I would say that security belongs to support domain.

So what is wrong, where is this paradox from title of this post?

Everyone thinks that core domain is THE most important piece of the puzzle. Simply, because it is called “core”, it has to be important, isn’t it :). We want it perfect, tested, stable, developed by best team we can think about (or we can afford to pay). Core domain has to be rock solid, you don’t build house on unstable foundations.

Is this really true?

From time to time we get involved in the process of getting new customers. We get to fill a long Excel spreadsheets, with tons of questions about our platform capabilities, features and so on. Do these prospect customers ask if we have talent acquisition or talent management in our platform?No. Because they know that this is what we do, and this is what they want to buy from us. Do they ask if we are HRIS, or we have recruitment workflow in our system? No. Because this is our core domain, something we have to have to even exist on this market. They don’t want to buy from us airline ticketing system, or office productivity suite. They want HR system, which will support their HR operations. It is so obvious for them.
So what are they asking for? They ask about our search capabilities, our compliance with industry standards, our reporting modules, integrations with other systems and platforms. They ask about our support domain. Yes brothers and sisters, support domain is what sells the product. This is key to buy/don’t buy decisions. Core domain is an “enabler”, something that makes your system one of the options for potential customers. But your support domain is something which is a key driver in final decision.

I know that you start hearing voices in your head. “Yes, I need to focus on support domain, I want my best people to work on it, otherwise I will not make money”.
Wait. Didn’t I tell you that you need your core domain stable, best quality you can get. If you don’t have rock solid core domain, you can just simply forget about the deal. You won’t be invited to the table with big money on it. Don’t have a good quality support domain? You wont’ win a deal. But you can’t have both, not at the same time. You don’t have enough time, money and talent to have both. You need to sacrifice something, or outsource it, or buy, or use open source.

Is there anything else we can do?

If you think in systems you will recognise that what we have here is situation where two interconnected subsystems (core domain and support domain) compete for limited resource (mythical man months if you will, attention of the board, call it whatever you like). And yet, quality of core domain, has direct impact on quality of support domain, but both need time and effort to be grown. This creates viscous circle. If we focus on support domain, quality and completeness of core will suffer. It will lead to a situation where business logic will be dissolved in a support domain functions, it will be “copy pasted”, hidden between lines of support domain code. We are on a way to “big ball of mud”, again. This is what happens in majority of systems. Lack of clear boundaries between core domain and support domain, unbalanced focus on one of them, creates chaotic architectures, which when they reach their “limits to growth” are really hard and thus expensive to improve.

How to break this vicious circle? One thing proposed by system thinkers is to break dependency between subsystems. It is of course not always possible, but let’s try. Let’s de-construct this system. In majority of systems I have seen, there was a code level dependency, where support domain needed core domain classes to do its work. This is wrong, so wrong. Like in our example of search in HRIS system. We could use EJB/DAO code from core domain to feed search engine. I know it is tempting, you get +10 in Reuse skills. But from systems perspective it is just wrong, you have to remember that it is always better to create dependencies to more stable parts of the system. And EJBs and DAO code can, and will change. Especially in cases when you are going to refactor the code, and this is what happens when you start to get rid of “big ball of mud”. But search engine needs data. Yeah…, right…, data!!!. We can simply feed search engine directly from database. I know it sounds controversial. You don’t have to do it, but be aware of this way out this “circle”. You are not always able to break this type of dependency, but you can always change a place where you attach dependent system. This kind of change can be temporary, for the time of deconstruction. Imagine “layered” architecture where you have JAX-RS/JAX-WS endpoints/beans which talk to stateless EJBs which depend on DAOs, and so on and so on. And imagine search engine using REST services to feed its content. During refactoring you have many dependant possible points of change. Creating dependency between search engine and core domain, at the database layer minimises probability of change in search engine, as new improvements are introduced in core domain. You are not able to remove this dependency but by moving it to more stable component you make it less painful. I would say that you stabilize the system.

Another thing you can do is take a look at time and resources dependency. This dependency can be seen as a problem with limited resources (you know, this myth that day has only 24 hours, and you need to rest and sleep). But in fact, it is rather problem with sequence of things to come in time. You have to have core domain first then you can build you support domain. Of course you can build part of core, add support, back to core, and so on. But there is always this thing that some things cannot happen in parallel. But what if support domain already existed? You would just have to write some “glue” code here and there. My advise, if it is support domain, buy it, use open source or outsource it. In a world of support domain everything is already there, just go and use it.

So next time you will try to de-construct your “big ball of mud”, first draw clean boundaries between core, support and generic domain. Use your domain experts, ask tons of questions. For the first weeks there will write a lots of “throw away code”, because you will not understand the domain. You will have to experiment, draw boundaries, erase pieces of domain and redraw then in another context. And yes, you will code from day “zero”.

You need to start modularization from support domain, and we will make it cheap, and yet at the same time you will please our customers with improvements they will see from the very first day. This “quick win” will help you grow confidence in this change. I hope so. As confidence will grow your ability to change the system, in a ways people have always neglected, will grow too. At the same time you will get better understanding of the core. I hope so.

Will we every reach desired end state of beautiful architecture? No. Because you need to be pragmatic , and you have to understand dynamics of technology and business. You have to understand that with such projects is like with “black body” in physics. It doesn’t exist but it is used in many models to help with thought process, as there are many things we don’t know and understand, and we have to use idealized objects to fill the “white pages” in our understanding of the domain.

This is not “silver bullet”, but rather my take on refactoring and rewrite projects.

Good night!