An unhalted ramble about DTOs

Data Transfer Objects are a great way of moving data through layers or modules in our architectures. But when, how and where can we make using them easier? Are there fancy frameworks?

If you feel a little bit angry (or „triggered“ as they say) just because of this initial statement, then you know what i am getting towards.

I’ll use Kotlin as the most prominent programming language to show examples. If you do not know Kotlin yet, don’t worry. I will keep it simple.

Preamble: How do DTOs actually look like?

This is simple. Imagine an object like a Javascript object. Like this:

{
    "foo": "bar",
    "baz": 1
}Code-Sprache: JSON / JSON mit Kommentaren (json)

A DTO, at it’s core, is just a plain and state full object. In Kotlin it might just look like this:

data class TestDto(
    val foo: String,
    val baz: Number
)Code-Sprache: Kotlin (kotlin)

And, for the Java fans out there, like this:

public class TestDto {
    private final String foo;
    private final Int baz;

    public TestDto(String foo, Int baz) {
        this.foo = foo;
        this.baz = baz;
    }

    public String getFoo() {
        return foo;
    }
    public Int getBaz() {
        return baz;
    }
}Code-Sprache: PHP (php)

If you are not yet a fan of Kotlin: The Kotlin and Java code do the same thing, plus the fields cannot be null in the data class. Plus you can use them interchangeable. I strongly encourage you to take a look at Kotlin!

In those examples the DTOs are immutable. Mostly this is how DTOs are structured, but since they are quite universally used, they might look a little bit different from project to project. Some are mutable, some not, some have other additional functions, but most of them look more like a POJO in Java, or a data class in Kotlin. Same have a name ending on DTO, some end on Event, some on neither. The real important thing is the way they are used. If you have an Object, that only holds information and is passed through your application in whatever fashion, you have a DTO. The only thing that they normally never do is business logic. They are acted upon, but normally never act.

DTOs in the wild

One very prominent use of DTOs is to define APIs in enterprise applications. Let us imagine now, that we have a small little application. This application allows a user to register and get/update all of his information. A DTO for the user might then look like this:

data class UserDto(
    val username: String,
    val email: String,
    val firstName: String?,
    val surname: String?
)Code-Sprache: Kotlin (kotlin)

In contrast to that, the underlying entity (whether really designed via DDD, or just in the database) might look like this:

@Entity
class User(
    @Id
    val id: UUID,
    val username: String,
    val email: Email,
    val firstName: Name?,
    val surname: Name?,
    val password: String,
)Code-Sprache: Kotlin (kotlin)

What can we deduce of that? The DTO might be based on the entity, but it is it’s own structure. The names and types of fields can be different and it can have more/less fields overall. Normally now we might have a mapping like this:

@GetMapping("/users/{id}")
fun getUser(@PathVariable id: String): ResponseEntity<UserDto> {
    return userRepository.findById(id)
           .map { userDtoMapper.map(it) }
           .map { ResponseEntity.ok(it) }
           .orElseGet { ResponseEntity.notFound().build() }
}Code-Sprache: JavaScript (javascript)

In this example, this method returns a UserDto, which is constructed through a mapper. A user or different service can consume this by dispatching such a Request.

REQUEST:
> GET /users/12

RESPONSE:
{
    "username": "example",
    "email": "example@test.com",
    "firstName": "Test",
    "surname": "Ing"
}Code-Sprache: JavaScript (javascript)

Now this has some advantages. For example:

1. Prone Against Domain Changes.
   If the entity changes, for example gains, looses or changes fields, the API can still stay the same. No need for consumers to change immediately.
2. Versioning For Backwards Compatibility.
   We can create a new DTO if we provide a new version of the domain and still provide both structures, old an new.
3. Additive Change Behaviour.
   Providing another endpoint does not mean we have to change the database. We can also map the entities surjective or injective into the DTO, resulting in an asymetric design, whilst benefiting from 1 and 2.

Not only that, but we can also generate the api with something like swagger, either API- or code-first and with that decoupling the code and the api.

So, why this blog post?

This all is great, isn’t it? You can decouple things from one another, resulting in cleaner code and better usability. What i did not tell you about is the DTO-Mapper. How does this look like?

Let us provide a mapper for the UserDto:

class UserDtoMapper {
    fun map(user: User): UserDto {
        return UserDto(
            username = user.username,
            email = user.email.value,
            firstName = user.firstName.value,
            surname = user.surname.value
        )
    }
}Code-Sprache: JavaScript (javascript)

It is straight forward, quite easy, isn’t it? Yes, but now, if we introduce versioning, this scales linear. For each version we need a new DTO and with that a new Mapper. Copy and paste. Can we make the easier? And the answer for that is…. drum roll please………. Yes! For example with MapStruct! Then the mapper is generated for us and we only write something like this:

@Mapper
interface UserDtoMapper {
    @Mapping
    map(user: User): UserDto
}Code-Sprache: CSS (css)

Now we can, depending on our ecosystem, just use this mapper. MapStruct just simply generates this mapper. Great! Now we can just create an interface and put it next to the DTO. Done!

Lets now think about architecture. Alright, yes, yes. I know. Sounds boring and it appears that it has nothing to do with DTOs. But bear with me, it will make sens in a minute.

DTOs in architecture

What we previously discussed was a DTO as an abstraction against external systems. The same principle could in theory also be applied to to decouple modules or whole layers from one another. For that let us construct a simple 3-layer architecture like this:

We have a top-layer called External. It contains all external communication like incoming RestController and outgoing clients (in which we consume other services). Because layered architectures are unidirectional, we have a ports module in the Business Logic layer, which contains interfaces for the classes in outgoing. This way we can call external systems within the business logic. Other than that this layer contains Services, which define operations upon the last layer, Domain.

Okay, currently we have talked about the DTOs for Incoming controllers. But what about the outgoing, aka the consuming clients for other APIs? There we need the DTO for different applications, right?

And Since we could have multiple different implementations for the clients that might even change over time, why not introduce DTOs here to? In the diagram now, the red dots are representing DTO conversion.

So far so good. Now we have all of the above advantages for the DTOs against all external systems, including the „external“ definition of the clients in Outgoing. Great.

Next thing up, the architect might come along and say something along the lines of

„We want to have strict layers in our application„

What does that mean? Strict means that each layer may only have access to the layer beneath it. It is used mostly in bigger monolithic applications before going to the modularisation stage of this monolith, to allow changes in the layers without it resulting in changes of other layers. Decoupling, like we did with the external system. Great, okay. We know what to do! Since the External layer now has no access to the domain and the Business Logic layer has no access to the DTOs in the External layer, we need some sort of „DTO-Layer“ between the two. Practically speaking: this means that the Business Logic layer defines DTO classes with which the External layer can communicate with it. Every method of the services either has to have lots of parameters, or a custom DTO. Meaning our diagram grows like this:

Hmm… This is a lot of DTOs, but it is worth it, right? This means that all of our code is decoupled! We can work on all of that individually! And, at the very least we don’t have to write all the mapping logic, we utilized MapStruct previously and can use it again now!

Clean code. Very clean code indeed! Most if not all developers know what is going on, know how to handle changes and where to put stuff. It is their architecture, their baby. This would be the end and most of the time we see it as such, but calling it this would be a lie.

Back to the future of this code

One, two or even three years after we wrote our application, tested it and deployed it on production several times in a row, there are issues that arise. An issue, a bug or just a new feature request. Most of the original developers are long gone, to the next project, developing their next baby. A new set of developers start working on this structure. What do they find?

They find a lot of code. Very much indeed. Even with Kotlin they find a DTO for nearly every class. Which of the 4 UserDto classes are now the one that contains the information for the frontend? Is it named FrontendUserDto? No, obviously not, we name them based on usecases, not based on the technology. Or did we? Why is there a MessagingUserDto? One by one we dive deep into the code. It is hard to read and to understand without someone who has deeper knowledge of the code.

After 4 hours of deep dive one has to take a break and grab a cup of coffee. At the coffee station you meet another developer and you need to vent.

„Man, the code i am working on in the old project xyz is so hard to understand, i can’t even seem to find the place where i need to change stuff. And even if i do, i have to touch soooo many classes, it is not even funny..„

„Funny.„, you get as a response. „I thought this project was very clean and used code generation to generate the mapping for this“.

Let us dissect the previously constructed examples and put on another hat, not the hat of the active developer, but the hat of the reading/using developer.

Switching the perspective

Let us ask the two most important questions in software architecture to all of the previous decisions:

Why?
and
What is the advantage?

Okay, to be completely honest with you: we already tackled the question why a lot of the time, but i pulled a sneaky on you. Why did we create the DTOs for incoming and for external systems? Sure, we wanted to have an individual and decoupled structure from our domain for external systems and have it on the other end too. Keyword: external. So, why do we have the DTO in the ports? And in the Services? I just assumed the same benefits for this without explaining it. And this is something that you might even see a lot in your projects.

Let us construct an example: We receive a message, which results in us triggering another external system. Just a very simple example. The data flow might then look like this:

This is only the change of the data. And further, this example assumes a relaxed layer architecture, which we did not do previously. So, let us adjust the flow for the DTOs of the Business Logic layer for the strict layering, whilst pushing the question „why“ for this a little bit back. Let us also introduce layers in this diagram, to better see where those data structures are located and going to:

Please note: this is only the way of the data, it is not the call chain. You can now imagine that every transition in this diagram requires its own mapper, its own DTO, its own call in code. It is a lot of code, even if we use the generated mappers of MapStruct. Now let us construct a real world example:

We want a user service, having authorization externally via a KeyCloak server. If a request is dispatched, we first validate this request against the KeyCloak and then answer via this API OK or NOT_OK. And now it is getting convoluted:

I want to again emphasize that this is only the way of the data and the way it is transformed.

This is not such a complex scenario. It is a constructed one, for sure, but still this is not that much off, of a real world example. Normally those examples are even more complex. Just imagine how this would look like, if you were to introduce another external system here, like another microservice you call then. Either through REST or through Messaging, it doesn’t matter.

If you narrow your scope on the application and only look isolated on the KeyCloak part, it makes absolutely sens to use those DTOs, it is decoupled. If you only look at the Ports, then it makes absolutely sense to use those DTOs here to, you can greatly argue for them. If you look at the strict layering, it makes absolutely sense to use those DTOs here too, because what else would you use? But the overview is really convoluted and hard to understand. You have to understand, really understand all of those little parts individually to get the total picture. And this is hard to read, there is no overview, no index, no table of contents. It takes time, a lot of time.

I pushed back the „Why“ for the strict layering and this is because i have strong feelings against it: I have never see a usable example of it. If you are faced with this decision i challenge you to ask the questions „Why“ and „What is the advantage“ yourself. Hopefully my point gets clear when we explore the second question.

Alright, the second question: What is the advantage? Let us pull up the 3 advantages i named for DTOs earlier:

1. Prone Against Domain Changes.
2. Versioning For Backwards Compatibility.
3. Additive Change Behaviour.

The first point is obviously not applicable in the given scenario. Let us change the wording a little bit: „Prone Against Changes Of Depending Data Structures“, meaning that whatever uses any method is not depending on the internal data structure of this method. This is a little a case of „the cat that bit it’s own tail“. We always have to adjust code if we change the code that directly is used by it (duh). And this holds true even with a DTO. In this case we only introduce another layer, meaning the change clashes earlier. In the best case scenario we created additional code to do the same, in the worst case we have to maintain a lot more code.

The second point really makes no sens at all. We change the code constantly inside of our application and it would only introduce a hell of a lot more complexity if we introduced versioning for the same internal code.

The last point is a little bit difficult to understand. This point still holds true, we can change the behaviour without needing to change the using code. However, if we add code internally, we do it, to.. well.. use it. So yes, this is true, but why would you do it?

In an isolated view we can argue for DTOs like we did for external systems, i don’t want to say anything against this. If you have ever touched a big monolith you see this a lot. Developers focus only on modules of this monolith, because it otherwise would require a lot more time to understand everything. In suche a scenario, the code in the same application but different modules become „external“ for us. We communicate through abstracted means, like EventBusses or only through interfaces.

But otherwise i again raise the question „What is the advantage“? We use the code directly that we changed, so why do we introduce such a noise into the code? Why do we hinder the flow of reading in this way?

There are more potential advantages to DTOs, but…

The good, the bad and the…

I get it. I am just a random guy on the internet. But let me pull out some quotes from people more experienced, that you might have heard about previously:

The most misused pattern in the Java Enterprise community is the DTO. DTO was clearly defined as a solution for a distribution problem. DTO was meant to be a coarse-grained data container which efficiently transports data between processes (tiers).

Adam Bien ~ https://www.adam-bien.com/roller/abien/entry/how_to_deal_with_j2ee

What Adam Bien mentions is very interesting: „[…]transports data between processes„, but on it’s own it is a little bit hard to understand this quote without the context. Martin Fowler get a little bit more to the point:

DTOs are called Data Transfer Objects because their whole purpose is to shift data in expensive remote calls. They are part of implementing a coarse grained interface which a remote interface needs for performance. Not just do you not need them in a local context, they are actually harmful both because a coarse-grained API is more difficult to use and because you have to do all the work moving data from your domain or data source layer into the DTOs.

Martin Fowler ~ https://martinfowler.com/bliki/LocalDTO.html

All of those advantages i previously mentioned still hold true, but only if we look at them from the context of external systems, in whatever fashion this is called. Rest, Soap or Messaging. So why do we use them to abstract the code internally? Some times it makes sense. Big chunks of code that are used a lot and are like icebergs (i.e. having a lot of code underneath that is fluctuating a lot) are a good example of where one might use DTOs. A complex module in a very big monolith or such. But other than this? Why do we do this to ourselves? Why do we make the total picture of the application this convoluted?

Sadly there is no right way here, it always depends on the context of the application. But let us have a look at how we could do this more responsible.

Use DTOs responsibly

To do this, i like to put on the hat of the future reader of the code, whilst developing it. The DTO is always build on top of what it should be transformed from. So i love to combine the DTO and the mapping logic into one class, which proved to have a lot of advantages. I would tackle the DTO for the User like this:

data class UserDto(
    val username: String,
    val email: String,
    val firstName: String?,
    val surname: String?
) {
    companion object {
        fun ofEntity(user: User): UserDto {
            return UserDto(
                username = user.username,
                email = user.email,value,
                firstName = user.firstName.value,
                surname = user.surname.value,
            )
        }
    }
}Code-Sprache: Kotlin (kotlin)

If we change the dto, which is a very conscious decision, we directly see what has to be done to adjust the mapping. No way around it. Changing one thing requires us to change the other thing. It hurts to do. The same can be done the other way around, like this:

data class CreateUserDto(
    val username: String,
    val email: String,
    val password: String,
    val firstName: String?,
    val surname: String?
) {
    fun toEntity(): User {
        return User(
            id = UUID.randomUUID(),
            username = username,
            email = Email(email),
            firstName = firstName?.let { FirstName(it) },
            surname = surname?.let{ Surname(it) },
            password = password,
        )
    }
}Code-Sprache: Kotlin (kotlin)

Changing the Entity requires us to change the DTO here. Either the toEntity or fromEntity method. It hurts. Every decision we do while writing this code requires effort. Quite a lot actually. But we don’t want to have the DTO for everything. We want to be wide awake while changing the API for other people. We want to reduce the amount of calling external systems. It hurts, but it is a good pain. It makes us conscious. It makes us awake.

It changes the DTO into it’s own „module“. We can use it easily like this:

val user: User = ....
val userDto = UserDto.ofEntity(user)

It is as easy as using the mapper (without actually requiring the mapper). The DTO is its own little ecosystem. Changes hurt, because we have to dive into the DTO „module“, but otherwise we don’t need to do this. Reading become simple. One class, not two or three, or even more. And if we write a new DTO we have to be wide awake of how it is going to be used. Not just „copy and paste this“. We (hopefully) use this less and less because: Why? Why would i write a DTO and it’s mapping functions if i can simply use the Entity? What is the advantage here?

Conclusion

We always talk about objectively clean code, but this nothing that really exists from my experience. Of course we know that the „Rule Of 30“ is obviously good, except in that one method, because it is easier to understand if it is longer, then we reduce the amount of methods in this class. Even the most objectively good rule has exceptions. Case expressions in object oriented code are obviously always a bad code smell, you can always design your code to not only not need them but also make it easier to read, understand, maintain and so one, but some times you just use it because it makes stuff easier to write and to explain. In this moment it makes sens. And that is okay. It might get refactored later. And this is even true for everything i wrote here, as hard as it may be for me. There are exceptions where it makes sens to use DTOs and that is okay.

But overall the DTOs are not the Egg Laying Wooly Pig that they appear to be. Always remember: the more „clean“ you code is, the harder it might be to understand for someone other then you, even if you know where every little detail goes, if awaken at half past 2 in the morning.

I have fallen into this myself. Creating a DTO for a service, or a method. It is cleaner. Then obviously no logic in this, because it is only for transferal. But is makes it really hard to read. Just try it yourself and challenge to put on another hat.

Data Transfer Objects are a great way of moving data from application to application, having the structure be unified and robust for future changes. There are frameworks that make using them easier. But i personally recommend writing everything yourself and make yourself and your fingers bleed writing those. They are powerful but introduce complexity and noise into your code for everyone else.