My Kafka integration platform was shutdown because it was too good


I have a story that I often tell my clients —and that I'm going to make sure it's on my book too. It happens more often than not. It probably happened or will happen to you as well.

For privacy reasons, I'm omitting some info but the core lesson remains.

Back in the days, a friend and I were hired to build an internal integration platform. That is, a internal platform that would allow the rest of the teams to quickly build and deploy new integrations with third-party products. There were multiple motivations for such initiative:

  • Duplicate Integrations: You could find multiple integrations with the same third-party service. Each of them owned by a different team, with a different UI and UX, and different implementations. Users even had to "connect" the integration multiple times depending on the part of the product. Definitely not acceptable.
  • Security Standards: Even though there was a security team (a good one, actually) their recommendations were not being followed. No blame here. These integrations were done before the security team existed and time was never found to rework them. Sounds familiar? To put a simple example, many secrets were not being encrypted at rest (in the database).
  • Operational Overhead: Integrating with the same third-party service multiple times means, in practice, that there are a bunch of people doing exactly the same things. Customer Support teams get the same complaints over and over, and the company is spending a lot of money (salaries, databases, or compute time) on redundant work.

After a few months of work, we started demoing the first version of our platform. It was amazing. I flew to the US to attend a conference organized by a potential business partner. I managed to create 3 integrations during the 2 days of the conference and demo'ed them to some potential partners in the meeting rooms. It didn't go well, it went amazingly well.

But there was a problem...

Our goal as engineers was to build a robust and secure system that would speed things up when building and maintaining integrations. Our promise? Focus on the business logic and we'll handle the rest. However, about half of the employees in our office were working on just two integrations. With our platform in place, their work would eventually become redundant and —most likely— some of the roles too. One might think this is something that a company would welcome. I mean, not having to let people go but increasing their performance and possibly relocating them in other teams.

The reality was very different. My office was in Europe and this was going against the EMEA leadership goals. They wanted to grow the number of European offices and their headcount. You know, more people, more power for them inside the company. Therefore, a project with the potential of making people redundant or harder to justify new hires didn't seem a good thing for them.

The day after I came back from my trip we were told that our project was "getting frozen". That was the euphemism they used to let us know our project was canceled and shutdown. We would relocate to other teams of our choice. Both my friend and I joined different teams and that became a story.

I use this story all the time to showcase people that company goals don't have to necessarily align with personal goals. Everyone's got their own objectives, whether hidden or not.

If you're trying to introduce an Event-Driven Architecture in your organization, you better start paying attention at the culture and the leadership goals. The real ones, not just the ones on the slides. They're harder to spot but not impossible.

The technology behind an Event-Driven Architecture (EDA) is easy. At least for us engineers. However, your EDA will only be as good as the culture of your organization. It's like an invisible ceiling. Make sure you have a high ceiling before shooting for the moon.

P.S. If you're stuck, need a second pair of eyes on your architecture, or want personalized guidance to accelerate your project, a 1:1 call can provide the clarity and direction you need. Book Your 1:1 Consultation.

1 Question For You

When was the last time you hit the cultural ceiling of your company? I'm having tons of conversations with folks from the industry about it. Your stories help me make sense of the current state of the art. Feel free to reply to this email. I'm on the other side :)

Av. Joaquín Costa, 16, Badajoz, Badajoz 06001
Unsubscribe · Preferences

Fran Méndez

Hey hey! I'm Fran, the creator of the AsyncAPI specification (the industry standard for defining asynchronous APIs). Subscribe to my newsletter —The Weekly Shift— where I share expert advice about building Event-Driven Architecture and share my journey writing my first book, Shift: The Playbook for Event-Driven Architecture Advocacy.

Read more from Fran Méndez
A puzzle

I've spent the last few weeks on over 20 calls with all kinds of companies. From tiny startups to huge enterprises. Every single one of them is wrestling with Event-Driven Architecture, trying to make sense of its promise and its messy reality. And as I listened, patterns started jumping out at me. We're not just talking about tech problems here. We're hitting the deep, human stuff that comes with a big shift like this. One story kept popping up, louder than all the others: culture. Forget...

Writing Shift is proving to be challenging to me. Not in a bad way but in a really really good way. Yesterday, in a conversation with Laïla Bougriâ, I told her that I'm writing the book for me, to learn. Obviously, I'm taking into account the target audience all the time but, even this, is part of the challenge. It's not the first time I've done this though. When I drafted the first AsyncAPI specification, I did it for me, so I could learn the ins and outs of OpenAPI and also Event-Driven...

Addressing a culture problem with a tool is like taking a hammer to your own head. The more powerful the tool, the more painful the blow.

This week, I had an interesting discussion with a client. It's a Civil Engineering corporation that's present in multiple continents. The "company culture" challenge becomes especially important when different cultures all around the globe are mixed toward a same goal. Building a globally-distributed Event-Driven Architecture is no exception. Everywhere I look, people are trying to fix culture problems with tools. Not because they're dumb or stupid, actually, I'd have committed the same...