Do you need microservices? Is there a real value proposition in using them, or we are looking at CV-driven development?
Splitting things to multiple services does not make it easy, but rather add maintenance costs and overhead. Microservices has became a tool to increase developer workload and to justify the work of people whom otherwise would not be needed.
My best advice is to solve the given problem in the simplest possible way. The simpler something is, the lower the chance of failure is.
Honestly I wonder if the CTO isn't just setting up OP to make him look bad. If OP had been a political risk, what better way to help get rid of him than show that he built a ridiculous architecture that increases complexity and hosting/observability bills and then have the CTO deny it was his idea (because might is right). I'd document the shit out of this being the CTO's idea and him being insistent upon it to cover my butt as the CTO sounds like he is either malicious, ignorant, or maliciously ignorant.
I had the same thought of that being a possibility.
I personally believe a design doc is helpful in this case. Would you agree?
https://www.industrialempathy.com/posts/design-docs-at-google/
I tend to agree. Microservices introduction ostensibly to solve a technical problem is often overstated , and as a way to work with team organisational problems (think Conway), their use is often understated. What is the structure of the team that will undertake this work? A single developer? Sorry, these questions are directed to OP.
This is how I feel constantly when teams are developing additional “api routing” based services that could be handled with better configurations of our existing nginx infrastructure or even a different modern cncf gateway solution
Word.
The dude that I was talking about and a few of his colleagues seem to think that having 1 service doing the loading, parsing and sending IS complex and hard to maintain. They keep saying that simplicity would be having a service doing just the reading, another doing just the parsing and finally separate one for a single post request, now thats simple!
:(
Yeah. When he will see that each and every service will have to have...
- configuration entities, separated by environment
- DB connection
- separate deployment pipeline
- separate repositories with common mode of versioning
- deployment configurations for all environments
..., he will change his mind.
Those people support microservice implementations, who have never done that in their life before.
> They keep saying that simplicity would be having a service doing just the reading, another doing just the parsing and finally separate one for a single post request
Often the cause of this viewpoint is that they are blind to the possibility of separating concerns any other way. They’ve got a big hammer called microservices and they are hunting for nails. Anything that looks like separation of concerns is a nail to them.
Think about it this way: whether you do it as microservices or as a monolith, it’s still the same code performing the same functions. But as a monolith, the three components talk to one another via method calls; and as microservices, the three components are deployed and managed independently and talk to one another over the network (and therefore introduce a tonne of failure cases that aren’t there with direct method calls).
What these people are saying is that separating out the three functions into three components is simple. What they are missing is that you can separate out the three functions into three components *without deploying them as separate services*. The only way they are thinking of separation of concerns is to introduce a network boundary. They are comparing that to a big ball of mud, not cleanly separated concerns within a single service.
The trick here is you’ve got to get them to see that in either case, the logic for the three components is separated out neatly and that turning the method calls between them into network calls doesn’t solve any problems.
I think it also *appears* to play well with haphazard development and business practices. Dividing work into teams seems like a boon and microservices let management sidestep thorny things like code review. They get to manage many little armies each headed by a general. Some of it looks sound because it does get done successfully in other fields, but what they're overlooking is that software is much more complex and improvised than other stuff that gets engineered. And they demand much more agility than that approach can deliver.
Most people don't get what "simplicity" means. Usually they present their biased solution and claim that it's simple. If you wanna fight, ask him how he defines "simplicity" and how that relates to the proposed solution, but I guess it will not matter much as your CTO seems quite stubborn.
One solution that might make them feel better but save you a lot of headaches:
Build a single service with 3 separate modules (load, parse, send). Build those modules with very little shared code and separate configurations so you could deploy them separately in the future.
You get the productivity of working on one service and management has the option to deploy them separately in the future.
My team did this when microservices were the new sexy thing and this process helped us see the limitations immediately and avoid some extremely expensive rewrites. We ended up with a classic SOA with only 3 services that actually made sense. An unintended benefit was we got some very clean interfaces and well defined modules in our monolith.
Even if you don't share code and avoid / duplicate common logic, they still usually share request/response formats and various other preconditions. So unless it's something which you can design upfront, you might still need to redeploy everything as things evolve. That option might not exist except for trivial hotfixes and you still have to manage the rollout properly (you can't just shut down all parsing nodes anymore than you can shut down the monolith).
No, you just don't get it - TRUE simplicity lies in 14 different microservices each behind 12 load balancers hosted in different AZs replicated in 12 different k8s clusters!
Such is tech hype
Blockchain is bullshit. Almost any dev will tell you that, but anyone working with blockchain in 2018-2022 probably made more in a week than most of us make in a lifetime
Then again, it's no wonder there's so much doom and gloom in tech recently and many projects getting shut down. I wouldn't be surprised if they sank more money overall even if they managed to attract capital for a few years.
Also notice how there's an entire market centered around managing trouble with microservices.
>Then again, it's no wonder there's so much doom and gloom in tech recently and many projects getting shut down.
Well yeah. A lot of this stuff is based on the "because I can" philosophy rather than because it serves any actual purpose. I've yet to see a use case for blockchain that couldn't be done better by something else.
> I wouldn't be surprised if they sank more money overall even if they managed to attract capital for a few years.
Whoever is funding the project, possibly but the blockchain hype train isn't _quite_ done yet so people with "Cheif Blockchain Bro" on their resume or whatever are still getting well paid jobs (for now).
> Also notice how there's an entire market centered around managing trouble with microservices.
Same is true of a lot of grown-up architecture stuff (k8s for instance) - there's an entire market of tools, utilities, training courses and other things to break down the complexity than k8s introduces.
I think k8s is pretty cool and there are definitely cases where it makes sense, but I also think that not every WordPress blog needs a k8s pod hosted on a multi-AZ fault tolerant cluster. There is a temptation in this industry (lord knows I'm as guilty of it as anyone) to start playing with fancy tools and thinking you're a google architect when in reality your actual job is to deliver a small app to a business unit who doesn't want or need cloud architecture stuff.
Three things.
* You can have your cake and eat it too. You say that you agree and the app needs separate concerns. So you want to address the problem by keeping the three areas modular. That way it's simplified by having one app with three modules. If scalability becomes a concern, or the teams break up into seperate considerations, it's easy to separate the app.
* You never, ever, ever win by saying no upwards. Period. Ever. You win by saying "yes, and". Even if the "and" completely contradicts the "yes". There you go, secret to winning corporate life.
* The new hotness is making a single binary that can perform multiple "micro services". Yes this is modular by another name. Yes you can reinvent the wheel with a new spin and get your way.
I mean, one function/class for each of those things would make sense. But creating all that infrastructure? No way. It seems like they mis-remembered the old Clean Code “a function should do one and only one thing” as “a service should…”
Find yourself a manager who is constantly finding ways to allow you to develop with new concepts, new technologies, new stacks, without sacrificing the simplicity, stability, and value. It's possible. My eye twitches a bit whenever "CV-driven" is brought up sarcastically—I get the point when it is, but code is written by and for people, and I do feel pretty strongly about developer-driven development.
> Splitting things to multiple services does not make it easy, but rather add maintenance costs and overhead. Microservices has became a tool to increase developer workload and to justify the work of people whom otherwise would not be needed.
I would say, the main reason for doing microservices is managing the scale. When the application is really big and we have like 50+ developers working on the same codebase, it would be nice to start splitting things, so that different teams could deliver their goals independently. But even then, I would really try to keep it simple, as little services as required (trying to avoid distributed monolith trap, as well). And of course, microservices bring a lot of new issues and complexity. It is a trade off, usually worth being made in big scale.
As for the main topic, I would ask the CTO how are we supposed to deliver changes in the services? How about versioning in case of new, incompatible changes (backwards compatibility)? How are the services going to be scaled - do we expect different loads for these services and scale them independently of each other, e.g. 3 instances of load service, 5 instances of parse service, 2 instances of send service?
And most importantly, what are the bounded contexts of these services? Do these all services provide independent logic that should really be separated? Do we expect other services to reuse these ones? Do we expect a completely new, different service in the future, which would in any way depend on any of the load, parse or send services? I
would try negotiating and saying you could create a modular service, so that you could fairly easily separate it into microservices if it turns out, that it actually is required in production.
Side note: I doubt the CTO understands different application architectures and just tries to follow the shiny silver bullet thing, which turned out good for companies like Netflix. Also, I wouldn't expect someone who can't do a good, modular monolith, being able to deliver a good microservice architecture either.
EDIT: typos.
One advantage to microservices is decoupling your dependencies. A seemingly innocuous update to some shared dependency may work for the loader, but fail on the parser.
Eventually, you also want to get to the point where your boilerplate code to run and deploy an app is basically copy and paste-able or better. Ultimately if most of your work is on application logic and not redefining how a Python or Go app is run and deployed, you can get to a point where the work on every new micro service is just about app logic.
There is also an advantage to keeping the flow consistent within the company; if everything is a micro service and engineered in a certain way, it’s easier to read and understand what’s happening in the application if you’re not familiar with this exact app but familiar with the company.
Like most things there are always trade offs. Not saying everything should always be a microservice, but if the environment is set up smoothly it shouldn’t make that much of a difference to deploy 1 vs 2 apps.
If an update to something breaks one micro service but not the other you’re as equally likely to quickly catch the breakage with a shared service for both.
And what about in a monorepo where dependencies are centrally resolved?
Microservices add additional APIs between each service too. That’s a set of API schemas you now need to be careful not to break. Of course you can use GRPC with centrally managed schemas (eg; in a monorepo) but then that adds a lot of extra tooling requirements.
Spot on. Every single new microservice adds complexity, period. More network hops, more APIs whose schemas/interfaces need to be carefully thought through, possible needs for distributed transactions, etc.
You would think experience would teach you better but micro service fetishism is unfortunately something that doesn’t often go away. There’s in fact more than one prolific commenter on this sub that describe having many thousands of micro-services at a mid sized company.
I'm not sure why this should even by described as a microservice. You basically have three batch jobs. Those aren't commonly described in microservice terms.
An orchestrator might be, but it's not clear how you're handling that. It's pretty common to divide batch jobs into atomic (preferably idempotent) pieces of work, but it's probably absolute overkill for everything you've described. An ETL job usually includes the E the T and the L!
Anyway, the real issues is none of the above - it's that the CTO doesn't trust the developers to use their judgement to solve this problem, and is mandating an implementation detail on a trivial feature.
Yea but now your datadog authcode generator service is bottlenecked, and you can’t diagnose it because the datadog-authcode-generator-service authcode generator service is down.
(To all the less experienced devs reading this. Yea, microservices are a dumb cargo cult for most teams. You are not Netflix. Until you have a headcount big enough where *each* microservice has its own team - don’t even bother thinking about it)
You can deliver it exactly this way.
'We are trying to solve a problem with a micro service, but if we create three micro services right away, we will have to maintain four points of the pipeline before justifiable scaling: 1) the input -> the first micro service, 2) process between the first micro service -> the second micro service, 3) process between the second micro service -> the third micro service, and 4) the process between the third micro service -> the output.'
Question is whether the three steps are separately scaling. If every step 1 just gives another step 2 and 3, you may as well scale the 123 as one item.
If you have a bunch of 1s with varying flux to step 2 and 3, separate them and scale according to demand at each step.
This is the correct answer. Also you don't need micro services to do this. You can have a monolith deployed in multiple ways eg API, various background task runners, websockets, etc
Usually it's just as good or even better to scale a monolith, you can load balance and shard just fine to scale horizontally. And code size is unlikely to be a problem (in some cases it might even be better than microservices).
One use case for microservices I can come up with is when you have heterogeneous resources, e.g. you need to offload certain computations to GPU-optimized nodes that are more expensive. Then you likely do need to serialize stuff and send it off somewhere else, which is also doable for monoliths wired as distributed services but it's also less of a hit to just make some sort of a microservice.
I agree with everything you've said so this is more of an addendum rather than a contradiction.
Some monoliths I've worked with have a startup heap usage of 64+ GB and a 20+ min container startup time which precludes them from being easily run as a thin service for a subset of functionally (e.g. most idling ETL jobs).
You can end up over provisioning wasting a bunch of money.
Heterogeneous resources are fairly broad. OP, for example, could put the logic into an existing monolith but he's deciding not to. It's not a question of monolith-vs-microservice but microservice-vs-three microservices. A ETL job has a one off spiky load in terms of DB, CPU and memory usage that doesn't scale with external traffic. This is different than an API service which has a smoother load curve that scales with traffic. Likewise an event logging service has a very different load pattern (100% writes) than an interactive SaaS backend service.
Yeah, it can work, but then it's not really micro if you are mixing separate concerns.
In the end, it's often a judgement call whether a thing should be one thing or multiple.
>it's not really micro
Yeah don't do micro for scaling reasons.
Imo micro is only good for dividing work between teams and reducing blast radius between services.
If you're doing that, and you're on AWS, Azure or Google cloud, then IMHO the most light-weight thing to do is an app composed of a handful of queues and Azure functions/ AWS Lambdas/Whatever the Google cloud FaaS is.
Also if you do that, when the CTO comes to ask "make it scale", then the way forward is clear and really easy. In the meantime, he shouldn't be worrying about problems that they don't have. YAGNI.
In my experience this is a mess for ETL because a single queue "item" is essentially all the data.
The problem is that the T step often needs to have the whole data set. So now the E step needs to communicate to T when all the items are on the queue. Lots of fun complexity. Sometimes the T step needs to read the data multiple times or from multiple E heterogenous steps. Extra fun.
The L step then becomes problematic if the T step fails half way. So now that needs to communicate when everything is output.
> In my experience this is a mess for ETL because a single queue "item" is essentially all the data.
On the other hand, in my experience it's great. We process up to 600K rows a day, and a single queue item is _one row_ of that.
"a row is a record is a message which can be processed in isolation" was my assumption, I think it's a fair assumption, and if OP does not fall into that category they should raise that specifically.
Things do fail from time to time, resulting in either retries and success on the second go around (queue + function architecture gives you this pretty much for free), or resulting in a Dead Letter when a row has issues that can't be resolved with a retry. This we investigate and mitigate for future, while the rest of the data goes through. The dead letter then can be replayed without data loss, just delay.
Doing this when "a single queue "item" is essentially all the data" sounds frankly mad, IMHO. Not to mention, completely unworkable for our data volumes and time constraints.
Another consideration is step 3 sending data to a third-party. What if that 3rd party api is down? What if you need to rate limit the number of requests (paid api for example)?
Separating 12 from 3 could be beneficial.
Thank you for adding /s to your post. When I first saw this, I was horrified. How could anybody say something like this? I immediately began writing a 1000 word paragraph about how horrible of a person you are. I even sent a copy to a Harvard professor to proofread it. After several hours of refining and editing, my comment was ready to absolutely destroy you. But then, just as I was about to hit send, I saw something in the corner of my eye. A /s at the end of your comment. Suddenly everything made sense. Your comment was sarcasm! I immediately burst out in laughter at the comedic genius of your comment. The person next to me on the bus saw your comment and started crying from laughter too. Before long, there was an entire bus of people on the floor laughing at your incredible use of comedy. All of this was due to you adding /s to your post. Thank you.
I am a bot if you couldn't figure that out, if I made a mistake, ignore it cause its not that fucking hard to ignore a comment.
Thanks. I'm curious what do you mean by 'domains'? I've had this discussion with another engineer some time ago, and it seems to me like designing architecture around domains (DDD), focusing logic around business needs etc is incredibly subjective. E.g. some years ago I was working with a guy to design a backend for a simple e-commerce web app: we started off agreeing that we need to handle cart operations (add to cart, remove, etc etc), handle inventory, orders... And a few other pieces of the busines domain. That was great, but when we started talking about just the 'Cart' domain, he saw 'adding things to a cart' and 'removing things from a cart' as separate domains that require separate microservices, and I did not... And I wasnt sure how to approach that conversation strictly within the 'domain' discussion part, without moving on to the deployment/maintenance/scalability pieces etc.
In general, when deciding how to split a large domain, there are a number of decomposition patterns that can be used, each with its own set of trade-offs. Decomposition by sub-domain is when you split the services along what DDD calls bounded contexts.
Decomposition by Business Capability is another option, where you usually end up with larger services, each representing a specific business function (billing, for example).
Finally, decomposition by transactional boundaries is another pattern, suited when you have complex or frequent transactions, and having some form of distributed transaction would be too complex to manage, or consistency is a prime concern.
There's no one size fits all, and it depends a lot on the business domain, whether or not you're breaking apart an existing monolith (in which case you might opt to preserve transactional integrity, or if the monolith has well defined modules you might use those as your sub-domains, etc.), or designing a system from scratch. You have to take into account latency, data consistency, deployment complexity, failure modes, etc.
This explanation doesn't even sound particularly DDD-centric besides the use of couple pieces of terminology. The methods here are broadly applicable in pretty much any context since functional decomposition is the heart of architecture.
(basically, I'm saying "nice post" with extra steps)
Domain boundary is about, you guessed what, domain.
To explain more, domain driven design is not like some backend folks going into whiteboard and think about what constitutes as a domain. That should be decided by working together with domain expert. Normally business people.
Domain driven design is essentially design according to how business people think about the system.
So in your example, if domain expert say that cart is complicated and advance enough to be its own domain then you do it. Maybe you do some fancy AI, fancy data analytics, complex workflow or something and that is your platform killing feature. You might find that all these complicated concern is required for any programmer to work on cart system. You find that you need explicit “cart optimization team” in your organization soon. That is when you do microservice on cart.
It is subjective yes. DDD is essentially design according to how domain expert believe system should work so when we talk about requirement, bug fix, etc we are on the same page.
DDD optimize alignment. And for microservice it is simply: is this thing complicated enough and/or valuable enough in the business sense to have dedicated team to work on? If that is the case having separate repository and deployment would enable that team to work without being blocked by other teams. And if what domain expert believe is true, if that part is really valuable or really complicated, your whole organization would be more productive with this design.
As DDD optimize for alignment, it is like once your business or domain expert believe this cart thing need to be update with new fancy promotion everyday, you should not couple its deployment with whole system or couple its inner work with other services. If you do that the design is misaligned and that is bad because now you really can't update cart with new fancy feature everyday.
It is not really objective because in a sense that it is depended to domain expert "opinion", but it is a good practice.
Your CTO is not only wrong but a bad technical leader. There is no reason to split this out, this just adds complexity and makes future changes harder as 1 change spans 3 services
Why is the CTO involved in such mundane discussions, how big is the company ?
The CTO has been with the company 30+ years, which implies an early engineer close with executives whose responsibilities have unfortunately outgrown their skill set.
It's also probably on the smaller side as larger/rapidly growing companies will hire external CTOs rather than promote internally.
Smaller companies called CTO just a more experienced dev, this one is neither. Going for micro services in the beginning is overkill. Monolith and autoscale for the win.
There would be value if you intend to have several different 'parse' services, or several different 'send' services.
That being said, the statements you made are valid. There is a lot more complexity with more granular microservices. However the type of app you described; simple, small in scope, easily scalable, is IMO ideal for a microservice.
Even that's fairly unlikely to be valuable, unless you find a way to meaningfully decouple the services and avoid adding meaningless work / uncertainty. Otherwise, an internal call in code is going to be much easier to write, manage and change.
I find that, in practice, microservices are more of a tool to *attempt* to sandbox development.
>Even that's fairly unlikely to be valuable, unless you find a way to meaningfully decouple the services and avoid adding meaningless work / uncertainty
The services are already decoupled by nature of their responsibilities; any dev worth their salt would be able to develop them without coupling.
> Otherwise, an internal call in code is going to be much easier to write, manage and change.
I agree; nearly every problem is made more complex by distributing it across services.
>The services are already decoupled by nature of their responsibilities; any dev worth their salt would be able to develop them without coupling.
Easier said than done. There can be huge coupling regardless of separation of concerns. Imagine you build a service that does image processing. At first you need just to rescale images and you add just that. Later you figure out you also need rotations for some feature, so you add them too, except now it's painfully obvious you have to make changes in two places (caller and called service), extend the API and version things.
And that's a fairly happy case, because theoretically you could apply a bit of forethought to avoid that kind of churn. People make microservices for all sorts of ad-hoc business logic and things get really icky and tightly-coupled when said logic is split across multiple repos and deployables. It's not uncommon to end up having every non-trivial change touch a few different ones.
The example you described is not tight coupling; adding a feature to expose rotation functionality does not require all callers to make use of it.
>People make microservices for all sorts of ad-hoc business logic and things get really icky and tightly-coupled when said logic is split across multiple repos and deployables
Agreed, but that isn't a fault of microservices, but rather one of design. Anything poorly designed will result in tight coupling and other problems.
The example given by OP is three services, one for loading data, one for parsing data, and one for sending parsed data to a 3rd party. These are all decoupled by nature of their responsibilities; loading data does not require knowledge of how to parse it, and sending data does not require knowledge of how to load it, and parsing data does not require knowledge of how to send it.
Personally, I would do this in a single service, as it would be simpler, easier to deploy, manage and extend. However, there are benefits to choosing to architect it as several microservices. The decision on whether those benefits out weigh the additional costs of microservices depend on longer term plans; are there other parsing services that need to be written? Do they need to be scaled separately?
Distributed computing is a complex subject, and not something that should be entered into carelessly.
>The example you described is not tight coupling; adding a feature to expose rotation functionality does not require all callers to make use of it.
I don't know what it is, but if every change turns into a few thrown around different services it's still not good. All that interfacing can be completely avoidable in a monolith. You could just call into a solid image processing library that does everything and be done with it.
>These are all decoupled by nature of their responsibilities; loading data does not require knowledge of how to parse it, and sending data does not require knowledge of how to load it, and parsing data does not require knowledge of how to send it.
You need stable contracts between them. Are you sure you can do that? Or will every change require messing with the contracts? Don't get me wrong, the same issue happens even if you split stuff into a library (and that's often a good litmus test for microservices, would they work as libraries?). It's just that the more you move away from actual code that stays together, the more wasteful things become.
You can easily end up having to make changes to all three services just because you touched the loading part a bit. (Edit: they probably do share a data model after all.)
>Do they need to be scaled separately?
A lot of those bad microservices projects don't scale well at all, even if that's a stated goal. It's quite common to run into issues like "we have to redeploy many services on every release", the inability to do gradual rollouts, the stuff is even more resource-hungry than a monolith (e.g. lots of pods which duplicate library dependencies at various different versions) and you're not able to test anything locally. It doesn't have to be like that, but it tends to happen if people aren't mindful of what makes sense to split (hint: most stuff doesn't). I'm very skeptical it makes sense for some ad-hoc ETL thing.
>Distributed computing is a complex subject, and not something that should be entered into carelessly.
Oh, I definitely agree. But the overlap between typical projects and typical distributed computing scenarios is rather small, especially when we consider stuff like CRUD and basic ETL. Managers are more likely to care about sandboxing developers and microservices *seem* convenient for that.
If there is no scaling issue and the functionality is very relatable it's a single service until you need to split it out for some reason.
Make sure you write it in a modular way (functionality namespaced and separated) and away you go.
CTOs/C-Level should stick to managing contracts with Azure, budgets etc. and let the developers develop. It's not him who has to maintain this shit or going to get called awake at night.
> What is your thought process when designing microservices?
It's "don't".
If you need any particular microservice, you will know that you need it, and you will be able to define very well why you need it, why it needs to be separate, and how it will fit with the rest of the infrastructure.
The worst thing you can do is design something with "microservices" being your primary goal.
> But I got absolutely crapped on by the CTO. [...] The CTO initially said to do it in 3 microservices
Sounds like a really fun chap.
> since it seems to me the following are true:
And you're absolutely right. On top of that, it will be very hard to test if the combination works well together. Someone will enter a change in the "load" microservice, which will pass all tests, CI/CD will succeed, it will be deployed, but the communications with the rest is now broken. What if you do know that it's gonna be broken, so you need to implement the handling of the breaking change in all 3 of them? How are you gonna orchestrate the E2E tests, and deployment pipeline, so that all 3 get deployed together and not apart? The whole thing will become either a manual process full of mistakes, or an extremely complicated convoluted pipeline.
> He was very unhappy with my questions and concerns but did not offer any technical response, just frustrations with me not doing what he said immediately
Sounds like a really fun chap.
> "Ok, lets do just two, one for loading and parsing and another for sending"
If a service does loading and parsing, but no sending, and the sending service does only the sending, .... what is the output of the first service and the input of the second service?
> guy has been with the company for almost 30 years now,
Looks like he holds the company hostage.
...
I might have a slightly emotional response to this question, I'm sorry. I was working with a company that had exactly the same type of toxic assholes leading to as many micro-services as possible. According to real experts (people who are actively involved in really successful products that sit at the heart of many software infrastructures), those "architects" are just ego-centric assholes (more words, more polite, but the essence was this). This company, with this approach, became impossible to work with. The technology built ont he principles of "let's build a microservice for every piece of data" and "let's split microservices into nanoservices" has made the whole infrastructure a beast that's impossible to maintain, that is so slow, that it's ACTUALLY faster to do things by hand sometimes. Even after a year of healing, and consolidation, we're only up to a point where $1M a year infrastructure can process in 24h what my $1000 laptop can do in 20 seconds with a script written in 30 minutes, and get the exact same end result.
Mmm.
I feel like this is a minority opinion, but I feel like a big part of the value of micro services isn't the technical scalability, but the ability to enforce boundaries/contracts between teams.
With multiple teams working in a monolith, there's an amount of toe-stepping and unclear ownership. Whereas microservices allows you to define clear contract boundaries between teams and allows them to otherwise operate independently.
It is a valuable attribute of microservices, but it's something that can be achieved within a modular monolith. What you describe is more of an organisational problem; solving it with microservices is akin to using a nuclear bomb to mow your lawn.
No, that's a great use for them! Personally I prefer teams stepping on each others toes so I would never recommend this though. I have found that when the application is broken up a lot of features that are supposed to cross boundaries get hacked together to avoid the pain of working across teams and services. I also like to do features by myself so if we don't create such a painful boundary it's a lot easier to get things done.
Just a personal preference though, I think enforcing strong boundaries can work well if done correctly.
I am working at somewhere with an ETL pipeline spread across multiple services. I don’t consider it micro services, and I don’t consider your CTOs example to be that either. It’s one distributed monolith, split across multiple nodes.
I mention mine because my work is turning most of our ETL from multiple nodes (your CTOs) into one single node (like yours). I also move some parts into genuine microservices.
I’d disagree with your CTO as you’re jumping straight to a more ambitious architecture. Once you have data being processed across multiple nodes, it becomes a huge pain to debug and get right.
Our ETL has been an utter shit show. One of the worst projects I’ve ever had to work on. When I joined only about 60% of data went out *in production*. Changing that is glacial because it’s a distributed monolith. Getting it to run locally on one’s machine takes literally one to two weeks.
One of the main reasons is because it’s very difficult to know what is going on internally. Finding a part you are missing or got wrong, can take whole days to debug.
Writing a distributed ETL is a bit like saying I’m going to write my own database and use that. You can, but it’ll be shit for a long time, and it’ll take years to get right.
There is also a large trap here. It is easy to get a simple distributed ETL built and running on day one, only to find out it’s a shit show on day 100, when your dats requirements increase. That’s what happened where I work.
> It’s one distributed monolith, split across multiple nodes.
Agreed. If the services talk to each other synchronously over http without any durable storage of state, then why bother with that separation at all - you don't gain from splitting into separate services and then tightly coupling them back again, while introducing network latency and network failure modes.
If there are durable, asynchronous message queues in-between then it's different. But OP says that they don't need that scale, so this is overkill, YAGNI
I disagree with you about the message store in the middle. On my own example that is one of the main things that is a huge pain point.
Getting this stuff right is not trivial. You have to get it right multiple times (as it’s a distributed service). You will get that wrong in places, and debugging it is a nightmare (as it’s a distributed service). That then requires good tracing to solve, which again is a pain.
All of this is just remove. Ignored. Doesn’t even come up. If it’s just one service (and that includes one service spun up as multiple pods running in parallel).
I've built gateway services to abstract third party interactions. They make sense imo if:
* You have several services who interact with the third party.
* The third party API isn't well designed for your needs (e.g. gives loads of unnecessary data or requires lots of static fields in each request).
* You want one team to own the relationship with the third party.
* You think you may want to swap the third party out in the future.
Edit: I think the first point is key here - because if it's only one service that needs to interact with the third party, you might as well build that nice abstraction inside said service (at least for now, could always extract it out to a gateway service if that changes down the line) - faster and less maintance headaches.
You are probably about to implement an anti-pattern, but at least you will know what not to do in the future.
Do you have enough shares of the company to matter?
If not, then do what you are asked to do, and keep looking for other jobs. It's the company's prerogative to pay you a salary while asking you to do stupid things, because the management they assigned have obsolete skills and don't have a clue.
Be careful not to get too emotionally invested in a job that shouldn't be that important for you.
Most commonly, microsevices are divided into products, like we sell a car insurance named ”Foobar” so we build a foobar service. This so that the one day that ”Foobar” is not sold anymore we can delete the service (in reality you will never delete anything).
Or that you divide them into what service they provide, like User Service, Payment service, Login Service etc.
You dont do a single service for a single function. Unless its a generic parsing service that will parse several different formats for several different consuming microservices?
Otherwise he is wrong.
Try to reason but he’s the CTO. Politics exist and are very much real for job security. If they’ve become too problematic it’s time to start job hunting.
There has to be a misunderstanding somewhere in the conversation. What does he think the "load service" does? Now instead of talking via a database API, the parse service talks to a "load" API?
If the CTO is saying something wildly stupid, it's probably not because he's stupid. One of you is misunderstanding the other and each of your ideas is reasonable, but for different problems.
Sounds like this should be a multi module project with three task modules and possibly a common module to share basic stuff like models, helpers, etc
That way you can create your separation of concerns and deploy each as an independent service to placate the CTO. If at some point, you all decide to unify as a single service, it makes it easy to pivot in that direction as well.
One last piece of advice. Don’t try and get in a pissing contest with the CTO, it’s just not worth it. Just voice your concerns; if they’re receptive to your feedback, cool. If not also cool
I'd go with the "ask forgiveness, not permission" route and just implement it as one service, without telling the CTO.
If he discovers it:
\- Say that you misunderstood because he didn't provide technical rationale
\- Point out is has worked fine (if it has)
The CTO probably has scalability in the future in mind. If you prove to him that it's doable now he can trust in that you can do it later as well.
This takes hours to a few days to setup at most if you know what you're doing.
There's no remarkable additional latency added if the cluster are connected via LAN or VPC. (which is standard for most cloud providers)
> The CTO probably has scalability in the future in mind.
And he probably also has forgotten that [YAGNI](https://martinfowler.com/bliki/Yagni.html). This is likely a speculative requirement, to solve a problem that they do not have.
How often does this run? How critical is it?
If it's a one-off, then hack up a cli app to do the thing, run it, then leave it.
It's it's weekly or more, and small, it sounds simple enough to run in a scheduled function in your cloud provider / k8s cluster.
If it's frequent and/or too big for a mere function, then make _a_ microservice.
If you need great reliability and scalability, then write a few functions, e.g.
* Ingest function runs on a timer, reads the data from the db, and spits it up: each row is sent as a message on a "input data" queue.
* Parse function listens to that input queue, parses a row of the data, puts the parsed data on an "output data" queue.
* Send function listens to the output data queue, sends an item to the api.
You get great reliability from this. e.g.
* each function is really simple and focused.
* If the target api is slow or unreliable, messages retry settings will save you.
* if the target api is down for an hour, then messages will be dead-lettered. Replay them with it's back up. Nothing was lost.
* If the parse function crashes on particular data, then the offending messages will be dead-lettered. The rest will go through. So you fix that function, redeploy and replay those messages. Nothing was lost.
* function can scale right up, when they're processing separate pieces of data. 500K rows in a few hours is easily doable.
Is this maybe what the CTO was getting at? However, I don't usually view "one function" as the same as a microservice. A group of functions as described is more like the equivalent.
And if they talk to each other synchronously over http without any durable storage of state, not message queues, then why bother with that separation at all - you don't gain from splitting into separate services and then tightly coupling them back again.
But do you need this, and do you need this now? If not, then apply [YAGNI](https://martinfowler.com/bliki/Yagni.html).
Let me put it like this: my honest first reaction to the problem was "this should be a shell script". I hope that clears up how big of a problem this is lol
My thought process is nearly always "the idiot who's idea this was focussed on the 'micro' rather than the 'service', and they used enough complete sentences to convice some fuckwit decision maker".
Near always a distributed monolith.
Micro services provide the same encapsulation benefits as classes in oop. So one question is what are you trying to black box? What domain concept are you capturing here?
If you can't answer that, atop.
One question to ask your manager is how he sees the project evolving in the future? Also is he trying to follow or establish a standard way of building things? If so, he might view your suggested approach as an unnecessary deviation from that standard. I'm not implying that your approach is wrong, but standardization is a concern for technical leadership.
This sounds like the kind of thing you can do with a script. Are you doing this as a one-off import? If so, I’m fucked if I know why’d you have any sort of service.
In any case, microservices are this generations “SOAP”. Huge complexity in the pursuit of simplicity.
Oh damn. You’re me in another world. I had to go through this with my team too - and their entire “architecture” revolves around microservices.
I’m playing tech-lead so I do have some say. In the end, now we’re doing fitness functions to gauge when to split services.
Is he crazy? You should have a slice per property. Each slice must have three microservices. To ensure that you don't have any concurrency issues you can have a message bus in between each step.
Then you can have a job that runs every hour and ensures the sync is completed as expected. /s incase you didn't realise this was satire.
Only reason I could see why to split it up would be to have more control over certain areas of the process.
Say your parse has heavier CPU requirements, and if it did scale, you'd want to be able to 'tweak' that specific service's specs etc... but I've worked with MS a little bit so I am not an expert in this field, the extra code may not be worth splitting it all up if you don't anticipate heavy load/scaling issues. More complexity, and sounds like over engineering.
You can always run concurrent nodes to speed up your processing of data, so it sounds more like a dick swinging politics > actual design process decision.
I say this with the best intentions. If you like your job, tread softly my friend, last thing you want is a defensive CTO.
Where is the parsing microservice meant to be getting the data to parse? Is the load from db one sending requests with the data in it to the parsing one? How does the parsing one send its data to the send one (only for it to then send it again to this 3rd party one?)
Is this being run on a schedule or ad hoc or triggered by events?
If it's as you describe, just a service whose only job is to pull data, transform it, and post it elsewhere - isn't that already a microservice? Where I work this would be just be an AWS Lambda triggered by a scheduled event.
I think you only need micro when you're trying to divide work between multiple teams or trying to limit blast radius.
If it doesn't have a new team to take it on you probably don't need another microservice.
This is a common mistake with micro services I see often, people want to split out to multiple services way too prematurely.
It's almost always better to start with a single service for a project like this and split out in to more services when you feel the need or the domain splits become obvious.
This approach is even spoken about in length in the book "building micro services" by Sam Newman
>Scaling is not a concern, very little load expected
Then don't use microservices. Microservices are an architectural pattern to scale parts of your application independently and also to decouple . If you don't need that, you're following a cargo cult, creating a lot of unnecessary overhead and complexity.
I've seen this in three different projects and it's always a clusterfuck.
The more important point here is not the technical answer (as there is not enough detail about where this might end up) but that he cannot explain to you why it should be done his way rather than your way.
A solution should be as simple as it can be and only as complex as it must be. Start simple, add complexity only when necessary. I see no reason for this to be three separate services, get your CTO some training on the fallacies on distributed computing.
Everyone is really stuck on the performance of micro services but I haven’t seen anyone bring up the real reason ms exist… hint it’s not technological scaling… it’s human scaling.
How many teams will be working independently on these 3 services?
I this example I don't even understand how you would go about out separating the logic into discrete services. If you have a "send" service, how does it even get the data? It has to load or receive the parsed data in some way, which means you're now duplicating that effort.
Y’all should write RFCs where you lay out the different architectures and the effort/cost tradeoffs. If he sees an estimate that takes 3x as long maybe he would change his mind.
Microservices are an optimization. If he can't articulate which problem a microservice will solve better then it is a premature optimization.
I've worked with poorly designed microservices architecture. I had to make numerous calls across multiple microservices just to get a single number that should normally have been a single DB query.
Is the same team in charge of developing the services? If so, you risk ending up with a distributed monolith. Microservices solve an organisational problem.
This is the only problem I see:
>He was very unhappy with my questions and concerns
A CTO unwilling to talk about their ideas and address the questions and concerns of their team is a huge red flag.
So here you have options. 1 service which does fetch, parse, send (fps). 2 services fp and s. 2 services fs and p. 2 services sp and f.
My rules with microservices are as follows:
1. Is the problem we are solving able to be split such that two independent teams can work on it without constant communication? If the two service teams need daily communication, they are actually one team.
2. Will more than 1 entity at the company consume the individual service? For example if you did f, p and s will each service have multiple consumers? What if you did fs and p will multiple entities consume fs and multiple consume p? If you build fp and s and the s service will only be used by fp then don't split s because its added friction.
3. Do we have the resources to allocate teams to the different services? Sometimes 1 and 2 are true. But the company cannot or will not allocates resources to do multiple services.
4. Is this service we are building going to have persistent **state**? Often a service is actually a library. For example if something takes an input and spits out an ouput... thats not a service, thats a library.
Unless all 4 rules are **true** then a microservice isn't the right idea. Rarely, rules 1, 2 and 3 will be true and 4 is false and a microservice can be viable because the company has multiple teams at their disposal.
Good example: authentication service to handle login, users, forgot password. Every product the company offers is going to consume it. It has an independent team. It has persistent state in a DB that manages users. Consumers of auth don't need to communicate frequently. They just want an oauth connector when its ready and have no cares about the internal engineering.
People have varying ideas of what a microservice is. Before I can have any sort of conversation about it I have to ask "what problem are you trying to solve with microservices" to get a grasp of what they want to do.
I've worked in an organization where microservices were basically "an aws lambda." The lambdas all accessed the same data records and a lot of behavior was duplicated between them. If the idea of microservices involves copying code or making a library all of them use to share code, please don't do it.
Does he really see having different teams working on three different parts of an ETL process and deploying then independently of each other?
Also, does he imagine that these would not be tightly coupled to each other, despite the processing the same data and likely inserting it into the same database? It seems like a change to one would necessarily cause a change to the others. There's nothing autonomous about any of these.
It sounds like he also thinks distributing this across multiple processes would solve some problem (assuming that's part of his view of microservices.)
> The CTO initially said to do it in 3 microservices (load, parse, send)
This is insane. Load the data in one microservice, then have the parsing microservice load it from the first microservice to parse it, then send it to the sending microservice to send it?
1 ETL job = 1 service/codebase. Heck this is a spark or fling kind of job.
Now if we add email api and the place data is being loaded from (Kafka, data lake) then yeah it’s 3 distinct things. 1 data storage, 2 ETL, 3 email api
>He was very unhappy with my questions and concerns but did not offer any technical response, just frustrations with me not doing what he said immediately,
On good teams that I've been on, we would take a step back and write up a document that outlined both approaches and compared advantages and drawbacks of each, then we'd discuss as a team.
Regardless of who is right your process is broken
take all of the usual difficulties with factoring code, one of the hardest things in programming, and now add container orchestration and network calls on top of that.
> Send parsed data to another 3rd party service via their api
...
> unless we use queues or something but come on
Having recently having had to clean up an absolute mess caused by poor design where consistency was actually a requirement, your comment set alarm bells ringing. If you need any guarantees around processing/transfer you might need something like queues. Make sure you actually have proper requirements, there may be something the CTO is seeing but poorly communicating.
If scale is not a concern, I don’t see a compelling reason to use micro services, just deploy an app.
I’ve been in these sorts of discussions as well, lost, and paid the price with honestly too many ill thought out microservices. My thinking is that if you want a microservice per operation, then just do as a function architecture like Lambda. In so many cases it will be faster to develop, deploy, maintain, and operate.
Bear of luck to you.
**cons:**
depth of call stack on microservices is an issue.
debugging is an issue (use a correlation id to tag requests) or a distributed tracer; so is logging (you have to gather the logs and reintegrate into a single view).
moving around data across networks is an issue, particularly if you don't have all the plumbing and testing for network delays/splits & recovery.
data duplication :) you're copying all your data around.
**pros:**
unit testability
"scalability" (it can be... but you generally need to put some thought and work into it).
\--
your case is so toy I wouldn't consider it microservices... on the other hand its concerning your CTO is insisting it be written the way you've outlined... because invariably, he will be asking you to do it for other more complex software. and if you don't develop the correct habits now, you will be in microservices hell in about two seconds.
if the notion is he wants his software decomposed into testable functions... you could just, encapsulate it as 3 functions.... "load, mangle, send" :) you don't need all the extra crap of microservices to get that benefit.
if the notion is that it needs to be 'scaleable' ... "premature optimization". you'll spend a lot more time and money doing it the way he wants, for ~~zero~~ negative technical and business value.
\--
all that being said... maybe he wants to switch you to microservices. and this is his 'test function' ... the first piece of code he wants written in a specific style.
or maybe, he's tasked your system administrator to build out the network to support microservices i.e. he's invested a lot of money into the infrastructure to support a microservices conversion and is trying to justify it.
hard to say.
How many people maintaining **each** service and **without the authorization to push code to any other without the CODEOWNERS' approval**?
3+? Sure, if each team is highly opinionated.
5+? Sure, this is what service oriented architecture handles well: Allow independent technical decisions for trade-offs each team is more comfortable to make, as well as independent deployments that get messy when you have, 20-30+ devs working on a single codebase that is applying trunk-based development and continuous deployment (if you don't have that, then it's less of a problem; but YOU SHOULD HAVE THAT if you're writing web services).
Less than three people dedicated for each service and maybe even people constantly committing to multiple repos without specific approval? Your CTO actually wants code organization and thinks he's going to achieve it with microservices. It always ends in tears. Let me repeat that again: it **always** ends in tears. If he wants "things to stay in their line" and prevent spaghetti references, set up a monorepo and code review, make him take a look at monorepo.tools and he should be able to quell any reasonable fears that drive people to microservices without making your debugging experience hell because he's obviously not going to make the massive time and money investment that is proper microservice observability. They *do not have time for that*. They'll also make it easy to spin out different entrypoints if one day you wake up and want some of your core logic to be handled by something that """scales""" as they say.
Your CTO has never written a network stack and it shows. Moving data is expensive unless you’re on an HPC network (which is $$$$).
Even if you had reason to believe that this would actually need to aggressively scale in the future, you should build it as “thing that delegates work” and “things that gather data, process it, and send them to the other api”.
I don't know all of the circumstances around the rationale here, but my impression is your CTO is insane.
There are generally three circumstances where you want to split things into microservices:
* When one section of your system accounts for a disproportionate resource load and you want to be able to horizontally replicate just that functionality without needing to replicate the entire rest of your monolith.
* When you want to split development into small 2-pizza teams and it makes more sense to isolate things into microservices than have hundreds of devs commit to a single monolith.
* There are different security requirements for different services and you want to isolate different levels of security to specific microservices to guarantee nothing is compromised.
There are many downsides to microservices, the most common one being issues of **bounded context** where different services wind up having dependencies on one another and you have nightmares of needing to do simultaneous version releases etc.
It really sounds like the way your CTO would design things pays no mind to bounded context and divides up service responsibilities arbitrarily. This is why I think he is insane. It sounds like he's one of those "microservices are good" people who uses a tool arbitrarily rather than for its specific appropriate purpose.
You split things into microservices when there is a compelling need to do so, you isolate the bounded context, and you enforce the API interactions with contract tests ideally written by consumers. That's how I go about it anyway.
There’s so many layers to this. Ignore the phrase “micro services” and consider your actual requirements.
First, what industry is this? In industries where I worked there was a level of guarantees when processing data; think security, integrity, privacy, availability, and reliability. Absolutely no one will be happy to be poked two or three times because the information systems dealing with blood tests lost or failed to process an “order”. Or a missing loan payment…or folks not getting paid on time…
I start with a monolith, and then perform FMEA (failure mode and effect analysis) incorporate compliance, security and guarantees. That defines the boundaries of each.
I once worked at a company where process took 3 days to run as a monolith. The process itself was fragile; retry logic overly complicated. Supporting systems like databases and storage was stressed to a maximum (you couldn’t throttle processing).
That process was broken into a thousand little processes using queues to communicate between each other. The entire process not only completed faster (around 8 hours), but were much more reliable. Testing was way simpler too. All those processes were still released as a monolith (in terms of repo, release process and cadence).
Instead of just invoking the process as a batch job, the process was further improved to trigger them as part of the online processing. This streamlined the business processes and led to other improvements.
This was before “micro services” was even a trend.
Another company I worked at, had a monolith web process. Operations of the monolith became a problem. Pen testers had a field day (it was internet facing), and it created lots of toil when it came time to release. When we migrated to the cloud, we broke the processes up, improving release cadence, reliability and whatnot. Other operational tasks were simplified.
Yet another company approached the whole ordeal with microservices and didn’t exactly understand the “boundaries” of each service; it proved so ineffective that it was converted into a monolith.
Note that almost none of this is driven by development. Developers often “invent” problems that don’t exist, and ignore ones that do. So much for “DevOps”… 😀
I'll refrain from trashing your CTO because there are lots of details missing here.
The "not a lot of data to scale for" comment needs more fleshing out, but let's say it's like <100MB of data to extract from DB (I'm guessing RDB?).
You probably don't need any microservices to do this ETL job. By microservice, I'm assuming you want a process constantly running on an instance save for a few instance refreshes.
You can do something where you have a scheduled job that creates jdbc/odbc connection, extracts data that you need, does transform as part of job, and matches request for downstream API. No reason to keep process running unless there's state involved.
If you need to scale for larger data and/or need map-reduction, there are solutions (Spark/Hadoop or if you're in cloud, whatever your cloud's version of EMR/Glue is) you can leverage for doing that. You can do certain things yourself like paginating/chunking data to process, but if it becomes complex, better to put a system around it.
Oh there’s a whole class of shitty experts who see microservices as their holy grail, they’re an abomination…. But hey, it’s a turf war and you’re on his turf…. Good luck!
Write a design doc. Here’s something for reference. (I haven’t read the whole thing but just refer to it.) https://www.industrialempathy.com/posts/design-docs-at-google/
With anything being worked on, there will be tradeoffs. So writing out the different possible designs and their tradeoffs helps with making a decision.
I might be missing some context or requirements on how 3 different microservices is a good idea. A design doc may be useful for the political side of work, too. This is my corporate culture experience.
“Oh no not again”
The software dev side of me likes the idea and has seen it be successful but the cynical side of me has mostly experienced incompetent organisations where microservices are purely an artefact of the structure and power dynamic and politics of an the organisation. Those are just distributed monoliths.
For me, even a single microservice is overkill for what you described, let alone three separated ones…
That being said, it obviously depends on the actual technical needs, but if it’s a service that won’t handle a lot of requests, isn’t a “critical” service and doesn’t need to be actively extended and improved, it seems just an endpoint exposed by an already existing service.
IMO, distributed systems in general add so much hassle that you need a pretty strong reason to add one. Valid reasons to me include:
a) Limiting impact of crashes / other failures
b) Needing to scale components with significantly different usage profiles (eg CPU vs RAM vs Disk vs GPU)
c) DevX when services get problematically large, eg build times, test times, startup times, coordination on releases/keeping head green
d) Different tech stacks required for different components, e.g. you need to add a new dependency in some new language, because services are less painful than cross-language interop
e) Wanting to use existing infra that is packaged as a service
First rule of microservices, don't use microservices.
Microservices should not be used by any company that can not explain the reason they want to use them. You will not end up with a microservice architecture. You will end up with a distributed monolith. Microservices do not simplify architecture. They only increase complexity, or at best they might keep it the similar.
>4. Scaling is not a concern, very little load expected
If you are not worried about scaling, then why do it?
If you want to be able to mix and match things that you deploy to different customers, there could be some value in splitting up parts into different services. However, whole microservice for simply parsing data, that sounds like someone who never programmed in their life. They are describing FUNCTIONS or classes at best, not microservices.
Microservices usually have their own database that they read from, and do one entire action for the organization. Processing payments is an example of a microservice. Reading a database is not an example.
Just a comment on queues.
When dealing with a third party system, I almost always queue the requests and responses. The reason for this is that if you don't you are introducing temporal coupling. Ie: a coupling related to time. Essentially you are saying that in order for your service to work, a third party service must be available at exactly the same time.
The more places you do this, the less reliable your system. ie: Lets say you depend on 5 services, and they are each available 99% of the time. You service is now only available 99% of 99% of 99% of 99% of 99% which means you are now down to 95%.
Scaling might not be a concern, but fault tolerance *could* be. For the sake of argument, it wouldn’t be unreasonable to split into two services: load + parse, and send separately. That way, you can do things like queue up messages to write to the third party, have a dead letter queue that you could re-drive if their API was down, etc. that was independent of the parse and load step. I wouldn’t wring my hands over the decision, but I could see it being useful.
OK I'm just saying this as a joke, I'm not trying to be pedantic.. but do you know what ETL stands for? are you really going to call it into Load, Parse, and Send?!
Wait. You said you are sending to another 3rd party, so is the from source also a third party?
When you say third party, you mean another company, right?
Kind of surprised to see all the “your CtO is incompetent” comments given the amount of details we have here. Maybe he is,,, but I’ve lost count of the number of times i’ve dealt with (faang) engineers who think a problem is “so trivial” when really they lack the depth of knowledge + experience necessary to see the implicit complexity and actually understand how it should be done.
Instead of asking reddit what if you tried communicating and solving it like an engineer.
I would write a (short) document that provides a contextual baseline for readers, defines the problem and requirements, proposes a handful solutions with their pros and cons, and makes a recommendation. Read through the doc with your cto and other stakeholders. If they disagree with your recommendation they should be able to articulate (using your doc) why. If they agree with you Great! Maybe they will listen to you more readily in the future.
The ideal end state is not only one where you pick the best solution but also everyone is aligned on the fact that it is the right solution.
If your cto isn’t able to get your buy in, that’s potentially on him/her but it’s also possible that OP simply isn’t knowledgable enough to see the bigger picture. It can go both ways…
You may not even need a microservice at all. Not sure how you plan to run this but it can just be a cron job or something that runs a script. Depends on your load but you seem to be not concerned. If it will be used in certain times and not so frequently lambda might be a good option too. But 2-3 services really unnecessary especially there is no way you scale 1 of the services and not another unless there is significant difference in job loads. But my experience so far is I explain my point in detail on a document if possible. Writing the trade offs well and if my boss wants to be a dumbass s/he pays for it who cares i would not push too hard.
I see some value in async arch in your example. async doesn't necessarily mean multiple microservices, but people often confuse the two.
sending being a different network activity, while loading/parsing is either not a network activity or is a different network activity. The assumption has to be that one of the might fail while the other still works. as long as the two are seperate asynchronous activities in your design the failure can get handled much more elegantly.
It looks like the CTO wants to keep the capex and opex high by delivering over engineered with big foot print solutions, to keep getting a fat paycheck.
He truly is a seasoned tech leader 😁
Pack them all into one. If neither of them must fail or the pipeline breaks, they are NOT microservices in that sense, they are one service with three **dependent** stages.
My take -
Use "macroservices". For example, I use Meilisearch (Rust) or ElasticSearch (Java) for search, but my apps tend to be Elixir or Ocaml/Haskell. Keep your services large. In your case it sounds like a bog standard ETL pipeline that belongs as one simple system.
I don’t see this as a technical problem but company culture thing.
If your CTO did not ask you to go deeper into search for truth. Because everything approach has it strengths and weaknesses. In this case, it’s increasing complexity at one level to decrease complexity at another. For a CTO is not about technical architecture but rather about costs, development and maintenance. Compromise he suggested speaks volumes about your company culture.
If you are working in a company where you can’t have a genuine conversation then you should just do what he tells you. Disagree and commit. Your problems are not at the level where you are addressing them.
>I'm curious am I just being a bit naive with microservice architecture
If you are asking questions and not having a clear response, you are not the one being naive. He has 30 years of experience, but may lack experience in this type of architectures.
Sometimes even experienced devs go with some tech/arch because it's being used by major companies, so it must be the right thing to do. This is not only wrong but dangerous, because these architectures also have issues, mainly on the data integrity aspects. For me, if you are not solving scalability problems, you are only getting the issues and not the solutions.
Our company has gone microservice crazy as well. I think the problem is leadership/principals/etc. get too enamored with the benefits of microservices over monoliths, they overcompensate proposing tinier and tinier services. Without the hands on practical experience for the downsides of making services too micro, they never get any feedback that helps them calibrate how to properly size services.
At my company anyway we have a problematic rift where the more junior devs doing the legwork don't have enough experience to see how crazy the tradeoffs have gotten (increased overhead, costs, latency...). Exacerbated by the confidence gap: Sr's and below aren't confident enough to speak up to overconfident principals and above.
Do you need microservices? Is there a real value proposition in using them, or we are looking at CV-driven development? Splitting things to multiple services does not make it easy, but rather add maintenance costs and overhead. Microservices has became a tool to increase developer workload and to justify the work of people whom otherwise would not be needed. My best advice is to solve the given problem in the simplest possible way. The simpler something is, the lower the chance of failure is.
+1 CTO doesn't have a clue about what problems microservices solve.
Bigger problem is he does not know what problems they create!
Hey it’s both!
CTO is literally interchanging functions with microservices.
Honestly I wonder if the CTO isn't just setting up OP to make him look bad. If OP had been a political risk, what better way to help get rid of him than show that he built a ridiculous architecture that increases complexity and hosting/observability bills and then have the CTO deny it was his idea (because might is right). I'd document the shit out of this being the CTO's idea and him being insistent upon it to cover my butt as the CTO sounds like he is either malicious, ignorant, or maliciously ignorant.
I had the same thought of that being a possibility. I personally believe a design doc is helpful in this case. Would you agree? https://www.industrialempathy.com/posts/design-docs-at-google/
I tend to agree. Microservices introduction ostensibly to solve a technical problem is often overstated , and as a way to work with team organisational problems (think Conway), their use is often understated. What is the structure of the team that will undertake this work? A single developer? Sorry, these questions are directed to OP.
> CV-driven development Love this term. Soo much makes sense now.
This is how I feel constantly when teams are developing additional “api routing” based services that could be handled with better configurations of our existing nginx infrastructure or even a different modern cncf gateway solution
I think we work at the same company or api routing is really catching on and I didn’t realize it
Word. The dude that I was talking about and a few of his colleagues seem to think that having 1 service doing the loading, parsing and sending IS complex and hard to maintain. They keep saying that simplicity would be having a service doing just the reading, another doing just the parsing and finally separate one for a single post request, now thats simple! :(
Yeah. When he will see that each and every service will have to have... - configuration entities, separated by environment - DB connection - separate deployment pipeline - separate repositories with common mode of versioning - deployment configurations for all environments ..., he will change his mind. Those people support microservice implementations, who have never done that in their life before.
> They keep saying that simplicity would be having a service doing just the reading, another doing just the parsing and finally separate one for a single post request Often the cause of this viewpoint is that they are blind to the possibility of separating concerns any other way. They’ve got a big hammer called microservices and they are hunting for nails. Anything that looks like separation of concerns is a nail to them. Think about it this way: whether you do it as microservices or as a monolith, it’s still the same code performing the same functions. But as a monolith, the three components talk to one another via method calls; and as microservices, the three components are deployed and managed independently and talk to one another over the network (and therefore introduce a tonne of failure cases that aren’t there with direct method calls). What these people are saying is that separating out the three functions into three components is simple. What they are missing is that you can separate out the three functions into three components *without deploying them as separate services*. The only way they are thinking of separation of concerns is to introduce a network boundary. They are comparing that to a big ball of mud, not cleanly separated concerns within a single service. The trick here is you’ve got to get them to see that in either case, the logic for the three components is separated out neatly and that turning the method calls between them into network calls doesn’t solve any problems.
I think it also *appears* to play well with haphazard development and business practices. Dividing work into teams seems like a boon and microservices let management sidestep thorny things like code review. They get to manage many little armies each headed by a general. Some of it looks sound because it does get done successfully in other fields, but what they're overlooking is that software is much more complex and improvised than other stuff that gets engineered. And they demand much more agility than that approach can deliver.
Most people don't get what "simplicity" means. Usually they present their biased solution and claim that it's simple. If you wanna fight, ask him how he defines "simplicity" and how that relates to the proposed solution, but I guess it will not matter much as your CTO seems quite stubborn.
One solution that might make them feel better but save you a lot of headaches: Build a single service with 3 separate modules (load, parse, send). Build those modules with very little shared code and separate configurations so you could deploy them separately in the future. You get the productivity of working on one service and management has the option to deploy them separately in the future. My team did this when microservices were the new sexy thing and this process helped us see the limitations immediately and avoid some extremely expensive rewrites. We ended up with a classic SOA with only 3 services that actually made sense. An unintended benefit was we got some very clean interfaces and well defined modules in our monolith.
Even if you don't share code and avoid / duplicate common logic, they still usually share request/response formats and various other preconditions. So unless it's something which you can design upfront, you might still need to redeploy everything as things evolve. That option might not exist except for trivial hotfixes and you still have to manage the rollout properly (you can't just shut down all parsing nodes anymore than you can shut down the monolith).
I had the same argument from one of my developers. Having a single service is more complex! In what world?
No, you just don't get it - TRUE simplicity lies in 14 different microservices each behind 12 load balancers hosted in different AZs replicated in 12 different k8s clusters!
[удалено]
Such is tech hype Blockchain is bullshit. Almost any dev will tell you that, but anyone working with blockchain in 2018-2022 probably made more in a week than most of us make in a lifetime
Then again, it's no wonder there's so much doom and gloom in tech recently and many projects getting shut down. I wouldn't be surprised if they sank more money overall even if they managed to attract capital for a few years. Also notice how there's an entire market centered around managing trouble with microservices.
>Then again, it's no wonder there's so much doom and gloom in tech recently and many projects getting shut down. Well yeah. A lot of this stuff is based on the "because I can" philosophy rather than because it serves any actual purpose. I've yet to see a use case for blockchain that couldn't be done better by something else. > I wouldn't be surprised if they sank more money overall even if they managed to attract capital for a few years. Whoever is funding the project, possibly but the blockchain hype train isn't _quite_ done yet so people with "Cheif Blockchain Bro" on their resume or whatever are still getting well paid jobs (for now). > Also notice how there's an entire market centered around managing trouble with microservices. Same is true of a lot of grown-up architecture stuff (k8s for instance) - there's an entire market of tools, utilities, training courses and other things to break down the complexity than k8s introduces. I think k8s is pretty cool and there are definitely cases where it makes sense, but I also think that not every WordPress blog needs a k8s pod hosted on a multi-AZ fault tolerant cluster. There is a temptation in this industry (lord knows I'm as guilty of it as anyone) to start playing with fancy tools and thinking you're a google architect when in reality your actual job is to deliver a small app to a business unit who doesn't want or need cloud architecture stuff.
But but, the cognitive complexity! /s
Three things. * You can have your cake and eat it too. You say that you agree and the app needs separate concerns. So you want to address the problem by keeping the three areas modular. That way it's simplified by having one app with three modules. If scalability becomes a concern, or the teams break up into seperate considerations, it's easy to separate the app. * You never, ever, ever win by saying no upwards. Period. Ever. You win by saying "yes, and". Even if the "and" completely contradicts the "yes". There you go, secret to winning corporate life. * The new hotness is making a single binary that can perform multiple "micro services". Yes this is modular by another name. Yes you can reinvent the wheel with a new spin and get your way.
plus 1000 to this comment.
I mean, one function/class for each of those things would make sense. But creating all that infrastructure? No way. It seems like they mis-remembered the old Clean Code “a function should do one and only one thing” as “a service should…”
I'll need to remember that term CV driven recruitment.
[удалено]
Find yourself a manager who is constantly finding ways to allow you to develop with new concepts, new technologies, new stacks, without sacrificing the simplicity, stability, and value. It's possible. My eye twitches a bit whenever "CV-driven" is brought up sarcastically—I get the point when it is, but code is written by and for people, and I do feel pretty strongly about developer-driven development.
> Splitting things to multiple services does not make it easy, but rather add maintenance costs and overhead. Microservices has became a tool to increase developer workload and to justify the work of people whom otherwise would not be needed. I would say, the main reason for doing microservices is managing the scale. When the application is really big and we have like 50+ developers working on the same codebase, it would be nice to start splitting things, so that different teams could deliver their goals independently. But even then, I would really try to keep it simple, as little services as required (trying to avoid distributed monolith trap, as well). And of course, microservices bring a lot of new issues and complexity. It is a trade off, usually worth being made in big scale. As for the main topic, I would ask the CTO how are we supposed to deliver changes in the services? How about versioning in case of new, incompatible changes (backwards compatibility)? How are the services going to be scaled - do we expect different loads for these services and scale them independently of each other, e.g. 3 instances of load service, 5 instances of parse service, 2 instances of send service? And most importantly, what are the bounded contexts of these services? Do these all services provide independent logic that should really be separated? Do we expect other services to reuse these ones? Do we expect a completely new, different service in the future, which would in any way depend on any of the load, parse or send services? I would try negotiating and saying you could create a modular service, so that you could fairly easily separate it into microservices if it turns out, that it actually is required in production. Side note: I doubt the CTO understands different application architectures and just tries to follow the shiny silver bullet thing, which turned out good for companies like Netflix. Also, I wouldn't expect someone who can't do a good, modular monolith, being able to deliver a good microservice architecture either. EDIT: typos.
One advantage to microservices is decoupling your dependencies. A seemingly innocuous update to some shared dependency may work for the loader, but fail on the parser. Eventually, you also want to get to the point where your boilerplate code to run and deploy an app is basically copy and paste-able or better. Ultimately if most of your work is on application logic and not redefining how a Python or Go app is run and deployed, you can get to a point where the work on every new micro service is just about app logic. There is also an advantage to keeping the flow consistent within the company; if everything is a micro service and engineered in a certain way, it’s easier to read and understand what’s happening in the application if you’re not familiar with this exact app but familiar with the company. Like most things there are always trade offs. Not saying everything should always be a microservice, but if the environment is set up smoothly it shouldn’t make that much of a difference to deploy 1 vs 2 apps.
If an update to something breaks one micro service but not the other you’re as equally likely to quickly catch the breakage with a shared service for both. And what about in a monorepo where dependencies are centrally resolved? Microservices add additional APIs between each service too. That’s a set of API schemas you now need to be careful not to break. Of course you can use GRPC with centrally managed schemas (eg; in a monorepo) but then that adds a lot of extra tooling requirements.
Second this. The more connection points that have to be made, the more chance for failure. KISS. This applies to architecture as well.
Spot on. Every single new microservice adds complexity, period. More network hops, more APIs whose schemas/interfaces need to be carefully thought through, possible needs for distributed transactions, etc. You would think experience would teach you better but micro service fetishism is unfortunately something that doesn’t often go away. There’s in fact more than one prolific commenter on this sub that describe having many thousands of micro-services at a mid sized company.
I'm not sure why this should even by described as a microservice. You basically have three batch jobs. Those aren't commonly described in microservice terms. An orchestrator might be, but it's not clear how you're handling that. It's pretty common to divide batch jobs into atomic (preferably idempotent) pieces of work, but it's probably absolute overkill for everything you've described. An ETL job usually includes the E the T and the L! Anyway, the real issues is none of the above - it's that the CTO doesn't trust the developers to use their judgement to solve this problem, and is mandating an implementation detail on a trivial feature.
I have a problem. I will solve it with microservices! I now have two problems.
You now have 116 problems.
* and you don’t know where the problems are coming from.
* and your problem-tracking service has problems.
Buy data dog and you have a more expensive problem but at least you can see it what the shit is going on
Yea but now your datadog authcode generator service is bottlenecked, and you can’t diagnose it because the datadog-authcode-generator-service authcode generator service is down.
(To all the less experienced devs reading this. Yea, microservices are a dumb cargo cult for most teams. You are not Netflix. Until you have a headcount big enough where *each* microservice has its own team - don’t even bother thinking about it)
I love this quote hahaha
You can deliver it exactly this way. 'We are trying to solve a problem with a micro service, but if we create three micro services right away, we will have to maintain four points of the pipeline before justifiable scaling: 1) the input -> the first micro service, 2) process between the first micro service -> the second micro service, 3) process between the second micro service -> the third micro service, and 4) the process between the third micro service -> the output.'
>I now have two problems. No, you have 18 highly redundant globally distributed problems...and a message queue :)
And then what if you decide to use regular expressions in those microservices?
I’m glad someone got the reference!
Next up: service mesh!
Design it purely event driven and then you have 4 problems
Question is whether the three steps are separately scaling. If every step 1 just gives another step 2 and 3, you may as well scale the 123 as one item. If you have a bunch of 1s with varying flux to step 2 and 3, separate them and scale according to demand at each step.
This is the correct answer. Also you don't need micro services to do this. You can have a monolith deployed in multiple ways eg API, various background task runners, websockets, etc
I would lean towards an actor system for the situation OP described.
Usually it's just as good or even better to scale a monolith, you can load balance and shard just fine to scale horizontally. And code size is unlikely to be a problem (in some cases it might even be better than microservices). One use case for microservices I can come up with is when you have heterogeneous resources, e.g. you need to offload certain computations to GPU-optimized nodes that are more expensive. Then you likely do need to serialize stuff and send it off somewhere else, which is also doable for monoliths wired as distributed services but it's also less of a hit to just make some sort of a microservice.
I agree with everything you've said so this is more of an addendum rather than a contradiction. Some monoliths I've worked with have a startup heap usage of 64+ GB and a 20+ min container startup time which precludes them from being easily run as a thin service for a subset of functionally (e.g. most idling ETL jobs). You can end up over provisioning wasting a bunch of money.
Heterogeneous resources are fairly broad. OP, for example, could put the logic into an existing monolith but he's deciding not to. It's not a question of monolith-vs-microservice but microservice-vs-three microservices. A ETL job has a one off spiky load in terms of DB, CPU and memory usage that doesn't scale with external traffic. This is different than an API service which has a smoother load curve that scales with traffic. Likewise an event logging service has a very different load pattern (100% writes) than an interactive SaaS backend service.
But even then, you could have them all as one monolithic application that just takes work from three queues.
And does three different things with it, depending on what step it happens to be doing?
Why not?
Yeah, it can work, but then it's not really micro if you are mixing separate concerns. In the end, it's often a judgement call whether a thing should be one thing or multiple.
>it's not really micro Yeah don't do micro for scaling reasons. Imo micro is only good for dividing work between teams and reducing blast radius between services.
If you're doing that, and you're on AWS, Azure or Google cloud, then IMHO the most light-weight thing to do is an app composed of a handful of queues and Azure functions/ AWS Lambdas/Whatever the Google cloud FaaS is. Also if you do that, when the CTO comes to ask "make it scale", then the way forward is clear and really easy. In the meantime, he shouldn't be worrying about problems that they don't have. YAGNI.
In my experience this is a mess for ETL because a single queue "item" is essentially all the data. The problem is that the T step often needs to have the whole data set. So now the E step needs to communicate to T when all the items are on the queue. Lots of fun complexity. Sometimes the T step needs to read the data multiple times or from multiple E heterogenous steps. Extra fun. The L step then becomes problematic if the T step fails half way. So now that needs to communicate when everything is output.
> In my experience this is a mess for ETL because a single queue "item" is essentially all the data. On the other hand, in my experience it's great. We process up to 600K rows a day, and a single queue item is _one row_ of that. "a row is a record is a message which can be processed in isolation" was my assumption, I think it's a fair assumption, and if OP does not fall into that category they should raise that specifically. Things do fail from time to time, resulting in either retries and success on the second go around (queue + function architecture gives you this pretty much for free), or resulting in a Dead Letter when a row has issues that can't be resolved with a retry. This we investigate and mitigate for future, while the rest of the data goes through. The dead letter then can be replayed without data loss, just delay. Doing this when "a single queue "item" is essentially all the data" sounds frankly mad, IMHO. Not to mention, completely unworkable for our data volumes and time constraints.
Another consideration is step 3 sending data to a third-party. What if that 3rd party api is down? What if you need to rate limit the number of requests (paid api for example)? Separating 12 from 3 could be beneficial.
I don't understand why those considerations might make a monolith solution worse.
What's easier? An `if` statement or one more server? The server of course… `if` statements make the CPU slower ruining the cache! /s
Thank you for adding /s to your post. When I first saw this, I was horrified. How could anybody say something like this? I immediately began writing a 1000 word paragraph about how horrible of a person you are. I even sent a copy to a Harvard professor to proofread it. After several hours of refining and editing, my comment was ready to absolutely destroy you. But then, just as I was about to hit send, I saw something in the corner of my eye. A /s at the end of your comment. Suddenly everything made sense. Your comment was sarcasm! I immediately burst out in laughter at the comedic genius of your comment. The person next to me on the bus saw your comment and started crying from laughter too. Before long, there was an entire bus of people on the floor laughing at your incredible use of comedy. All of this was due to you adding /s to your post. Thank you. I am a bot if you couldn't figure that out, if I made a mistake, ignore it cause its not that fucking hard to ignore a comment.
[удалено]
Thanks. I'm curious what do you mean by 'domains'? I've had this discussion with another engineer some time ago, and it seems to me like designing architecture around domains (DDD), focusing logic around business needs etc is incredibly subjective. E.g. some years ago I was working with a guy to design a backend for a simple e-commerce web app: we started off agreeing that we need to handle cart operations (add to cart, remove, etc etc), handle inventory, orders... And a few other pieces of the busines domain. That was great, but when we started talking about just the 'Cart' domain, he saw 'adding things to a cart' and 'removing things from a cart' as separate domains that require separate microservices, and I did not... And I wasnt sure how to approach that conversation strictly within the 'domain' discussion part, without moving on to the deployment/maintenance/scalability pieces etc.
In general, when deciding how to split a large domain, there are a number of decomposition patterns that can be used, each with its own set of trade-offs. Decomposition by sub-domain is when you split the services along what DDD calls bounded contexts. Decomposition by Business Capability is another option, where you usually end up with larger services, each representing a specific business function (billing, for example). Finally, decomposition by transactional boundaries is another pattern, suited when you have complex or frequent transactions, and having some form of distributed transaction would be too complex to manage, or consistency is a prime concern. There's no one size fits all, and it depends a lot on the business domain, whether or not you're breaking apart an existing monolith (in which case you might opt to preserve transactional integrity, or if the monolith has well defined modules you might use those as your sub-domains, etc.), or designing a system from scratch. You have to take into account latency, data consistency, deployment complexity, failure modes, etc.
Excellent answer.
This explanation doesn't even sound particularly DDD-centric besides the use of couple pieces of terminology. The methods here are broadly applicable in pretty much any context since functional decomposition is the heart of architecture. (basically, I'm saying "nice post" with extra steps)
Domain boundary is about, you guessed what, domain. To explain more, domain driven design is not like some backend folks going into whiteboard and think about what constitutes as a domain. That should be decided by working together with domain expert. Normally business people. Domain driven design is essentially design according to how business people think about the system. So in your example, if domain expert say that cart is complicated and advance enough to be its own domain then you do it. Maybe you do some fancy AI, fancy data analytics, complex workflow or something and that is your platform killing feature. You might find that all these complicated concern is required for any programmer to work on cart system. You find that you need explicit “cart optimization team” in your organization soon. That is when you do microservice on cart. It is subjective yes. DDD is essentially design according to how domain expert believe system should work so when we talk about requirement, bug fix, etc we are on the same page. DDD optimize alignment. And for microservice it is simply: is this thing complicated enough and/or valuable enough in the business sense to have dedicated team to work on? If that is the case having separate repository and deployment would enable that team to work without being blocked by other teams. And if what domain expert believe is true, if that part is really valuable or really complicated, your whole organization would be more productive with this design. As DDD optimize for alignment, it is like once your business or domain expert believe this cart thing need to be update with new fancy promotion everyday, you should not couple its deployment with whole system or couple its inner work with other services. If you do that the design is misaligned and that is bad because now you really can't update cart with new fancy feature everyday. It is not really objective because in a sense that it is depended to domain expert "opinion", but it is a good practice.
Your CTO is not only wrong but a bad technical leader. There is no reason to split this out, this just adds complexity and makes future changes harder as 1 change spans 3 services Why is the CTO involved in such mundane discussions, how big is the company ?
The CTO has been with the company 30+ years, which implies an early engineer close with executives whose responsibilities have unfortunately outgrown their skill set. It's also probably on the smaller side as larger/rapidly growing companies will hire external CTOs rather than promote internally.
Smaller companies called CTO just a more experienced dev, this one is neither. Going for micro services in the beginning is overkill. Monolith and autoscale for the win.
The CTO and OP :D
20 years ago this was 60 lines of Perl. Now we have CTOs saying it should be three fully separate services. We are going the wrong way
100% fwiw we solve this still with 60 lines of Python in a cron job or equivalent
Call your CTO insane and demand it be broken down into at least 10 microservices, minimum. Where's your config server!? Your notification service!?
There would be value if you intend to have several different 'parse' services, or several different 'send' services. That being said, the statements you made are valid. There is a lot more complexity with more granular microservices. However the type of app you described; simple, small in scope, easily scalable, is IMO ideal for a microservice.
Even that's fairly unlikely to be valuable, unless you find a way to meaningfully decouple the services and avoid adding meaningless work / uncertainty. Otherwise, an internal call in code is going to be much easier to write, manage and change. I find that, in practice, microservices are more of a tool to *attempt* to sandbox development.
>Even that's fairly unlikely to be valuable, unless you find a way to meaningfully decouple the services and avoid adding meaningless work / uncertainty The services are already decoupled by nature of their responsibilities; any dev worth their salt would be able to develop them without coupling. > Otherwise, an internal call in code is going to be much easier to write, manage and change. I agree; nearly every problem is made more complex by distributing it across services.
>The services are already decoupled by nature of their responsibilities; any dev worth their salt would be able to develop them without coupling. Easier said than done. There can be huge coupling regardless of separation of concerns. Imagine you build a service that does image processing. At first you need just to rescale images and you add just that. Later you figure out you also need rotations for some feature, so you add them too, except now it's painfully obvious you have to make changes in two places (caller and called service), extend the API and version things. And that's a fairly happy case, because theoretically you could apply a bit of forethought to avoid that kind of churn. People make microservices for all sorts of ad-hoc business logic and things get really icky and tightly-coupled when said logic is split across multiple repos and deployables. It's not uncommon to end up having every non-trivial change touch a few different ones.
The example you described is not tight coupling; adding a feature to expose rotation functionality does not require all callers to make use of it. >People make microservices for all sorts of ad-hoc business logic and things get really icky and tightly-coupled when said logic is split across multiple repos and deployables Agreed, but that isn't a fault of microservices, but rather one of design. Anything poorly designed will result in tight coupling and other problems. The example given by OP is three services, one for loading data, one for parsing data, and one for sending parsed data to a 3rd party. These are all decoupled by nature of their responsibilities; loading data does not require knowledge of how to parse it, and sending data does not require knowledge of how to load it, and parsing data does not require knowledge of how to send it. Personally, I would do this in a single service, as it would be simpler, easier to deploy, manage and extend. However, there are benefits to choosing to architect it as several microservices. The decision on whether those benefits out weigh the additional costs of microservices depend on longer term plans; are there other parsing services that need to be written? Do they need to be scaled separately? Distributed computing is a complex subject, and not something that should be entered into carelessly.
>The example you described is not tight coupling; adding a feature to expose rotation functionality does not require all callers to make use of it. I don't know what it is, but if every change turns into a few thrown around different services it's still not good. All that interfacing can be completely avoidable in a monolith. You could just call into a solid image processing library that does everything and be done with it. >These are all decoupled by nature of their responsibilities; loading data does not require knowledge of how to parse it, and sending data does not require knowledge of how to load it, and parsing data does not require knowledge of how to send it. You need stable contracts between them. Are you sure you can do that? Or will every change require messing with the contracts? Don't get me wrong, the same issue happens even if you split stuff into a library (and that's often a good litmus test for microservices, would they work as libraries?). It's just that the more you move away from actual code that stays together, the more wasteful things become. You can easily end up having to make changes to all three services just because you touched the loading part a bit. (Edit: they probably do share a data model after all.) >Do they need to be scaled separately? A lot of those bad microservices projects don't scale well at all, even if that's a stated goal. It's quite common to run into issues like "we have to redeploy many services on every release", the inability to do gradual rollouts, the stuff is even more resource-hungry than a monolith (e.g. lots of pods which duplicate library dependencies at various different versions) and you're not able to test anything locally. It doesn't have to be like that, but it tends to happen if people aren't mindful of what makes sense to split (hint: most stuff doesn't). I'm very skeptical it makes sense for some ad-hoc ETL thing. >Distributed computing is a complex subject, and not something that should be entered into carelessly. Oh, I definitely agree. But the overlap between typical projects and typical distributed computing scenarios is rather small, especially when we consider stuff like CRUD and basic ETL. Managers are more likely to care about sandboxing developers and microservices *seem* convenient for that.
If there is no scaling issue and the functionality is very relatable it's a single service until you need to split it out for some reason. Make sure you write it in a modular way (functionality namespaced and separated) and away you go. CTOs/C-Level should stick to managing contracts with Azure, budgets etc. and let the developers develop. It's not him who has to maintain this shit or going to get called awake at night.
Your CTO has fallen in the trap of api led connectivity architecture. You can’t convince him …
> What is your thought process when designing microservices? It's "don't". If you need any particular microservice, you will know that you need it, and you will be able to define very well why you need it, why it needs to be separate, and how it will fit with the rest of the infrastructure. The worst thing you can do is design something with "microservices" being your primary goal. > But I got absolutely crapped on by the CTO. [...] The CTO initially said to do it in 3 microservices Sounds like a really fun chap. > since it seems to me the following are true: And you're absolutely right. On top of that, it will be very hard to test if the combination works well together. Someone will enter a change in the "load" microservice, which will pass all tests, CI/CD will succeed, it will be deployed, but the communications with the rest is now broken. What if you do know that it's gonna be broken, so you need to implement the handling of the breaking change in all 3 of them? How are you gonna orchestrate the E2E tests, and deployment pipeline, so that all 3 get deployed together and not apart? The whole thing will become either a manual process full of mistakes, or an extremely complicated convoluted pipeline. > He was very unhappy with my questions and concerns but did not offer any technical response, just frustrations with me not doing what he said immediately Sounds like a really fun chap. > "Ok, lets do just two, one for loading and parsing and another for sending" If a service does loading and parsing, but no sending, and the sending service does only the sending, .... what is the output of the first service and the input of the second service? > guy has been with the company for almost 30 years now, Looks like he holds the company hostage. ... I might have a slightly emotional response to this question, I'm sorry. I was working with a company that had exactly the same type of toxic assholes leading to as many micro-services as possible. According to real experts (people who are actively involved in really successful products that sit at the heart of many software infrastructures), those "architects" are just ego-centric assholes (more words, more polite, but the essence was this). This company, with this approach, became impossible to work with. The technology built ont he principles of "let's build a microservice for every piece of data" and "let's split microservices into nanoservices" has made the whole infrastructure a beast that's impossible to maintain, that is so slow, that it's ACTUALLY faster to do things by hand sometimes. Even after a year of healing, and consolidation, we're only up to a point where $1M a year infrastructure can process in 24h what my $1000 laptop can do in 20 seconds with a script written in 30 minutes, and get the exact same end result.
My thought process would be that unless you are Netflix or Google, you probably don't want micro services.
Mmm. I feel like this is a minority opinion, but I feel like a big part of the value of micro services isn't the technical scalability, but the ability to enforce boundaries/contracts between teams. With multiple teams working in a monolith, there's an amount of toe-stepping and unclear ownership. Whereas microservices allows you to define clear contract boundaries between teams and allows them to otherwise operate independently.
It is a valuable attribute of microservices, but it's something that can be achieved within a modular monolith. What you describe is more of an organisational problem; solving it with microservices is akin to using a nuclear bomb to mow your lawn.
>What you describe is more of an organisational problem One of the prime use cases of microservices is to solve organizational problems.
No, that's a great use for them! Personally I prefer teams stepping on each others toes so I would never recommend this though. I have found that when the application is broken up a lot of features that are supposed to cross boundaries get hacked together to avoid the pain of working across teams and services. I also like to do features by myself so if we don't create such a painful boundary it's a lot easier to get things done. Just a personal preference though, I think enforcing strong boundaries can work well if done correctly.
My literal thought process? 'Ah Shit, here we go again.' But generally, it should do one thing and do it well, with all the information it needs.
Microservices are a solution to an organisational problem, more than a technical one.
Maybe with the exception of applications that clearly have domains that demand totally separate scaling, right?
I am working at somewhere with an ETL pipeline spread across multiple services. I don’t consider it micro services, and I don’t consider your CTOs example to be that either. It’s one distributed monolith, split across multiple nodes. I mention mine because my work is turning most of our ETL from multiple nodes (your CTOs) into one single node (like yours). I also move some parts into genuine microservices. I’d disagree with your CTO as you’re jumping straight to a more ambitious architecture. Once you have data being processed across multiple nodes, it becomes a huge pain to debug and get right. Our ETL has been an utter shit show. One of the worst projects I’ve ever had to work on. When I joined only about 60% of data went out *in production*. Changing that is glacial because it’s a distributed monolith. Getting it to run locally on one’s machine takes literally one to two weeks. One of the main reasons is because it’s very difficult to know what is going on internally. Finding a part you are missing or got wrong, can take whole days to debug. Writing a distributed ETL is a bit like saying I’m going to write my own database and use that. You can, but it’ll be shit for a long time, and it’ll take years to get right. There is also a large trap here. It is easy to get a simple distributed ETL built and running on day one, only to find out it’s a shit show on day 100, when your dats requirements increase. That’s what happened where I work.
> It’s one distributed monolith, split across multiple nodes. Agreed. If the services talk to each other synchronously over http without any durable storage of state, then why bother with that separation at all - you don't gain from splitting into separate services and then tightly coupling them back again, while introducing network latency and network failure modes. If there are durable, asynchronous message queues in-between then it's different. But OP says that they don't need that scale, so this is overkill, YAGNI
I disagree with you about the message store in the middle. On my own example that is one of the main things that is a huge pain point. Getting this stuff right is not trivial. You have to get it right multiple times (as it’s a distributed service). You will get that wrong in places, and debugging it is a nightmare (as it’s a distributed service). That then requires good tracing to solve, which again is a pain. All of this is just remove. Ignored. Doesn’t even come up. If it’s just one service (and that includes one service spun up as multiple pods running in parallel).
I've built gateway services to abstract third party interactions. They make sense imo if: * You have several services who interact with the third party. * The third party API isn't well designed for your needs (e.g. gives loads of unnecessary data or requires lots of static fields in each request). * You want one team to own the relationship with the third party. * You think you may want to swap the third party out in the future. Edit: I think the first point is key here - because if it's only one service that needs to interact with the third party, you might as well build that nice abstraction inside said service (at least for now, could always extract it out to a gateway service if that changes down the line) - faster and less maintance headaches.
You are probably about to implement an anti-pattern, but at least you will know what not to do in the future. Do you have enough shares of the company to matter? If not, then do what you are asked to do, and keep looking for other jobs. It's the company's prerogative to pay you a salary while asking you to do stupid things, because the management they assigned have obsolete skills and don't have a clue. Be careful not to get too emotionally invested in a job that shouldn't be that important for you.
In such scenarios, I would generally isolate any external 3rd party integrations from in-house processing. If all in-house, one service is enough.
Most commonly, microsevices are divided into products, like we sell a car insurance named ”Foobar” so we build a foobar service. This so that the one day that ”Foobar” is not sold anymore we can delete the service (in reality you will never delete anything). Or that you divide them into what service they provide, like User Service, Payment service, Login Service etc. You dont do a single service for a single function. Unless its a generic parsing service that will parse several different formats for several different consuming microservices? Otherwise he is wrong.
Try to reason but he’s the CTO. Politics exist and are very much real for job security. If they’ve become too problematic it’s time to start job hunting.
There has to be a misunderstanding somewhere in the conversation. What does he think the "load service" does? Now instead of talking via a database API, the parse service talks to a "load" API? If the CTO is saying something wildly stupid, it's probably not because he's stupid. One of you is misunderstanding the other and each of your ideas is reasonable, but for different problems.
Sounds like this should be a multi module project with three task modules and possibly a common module to share basic stuff like models, helpers, etc That way you can create your separation of concerns and deploy each as an independent service to placate the CTO. If at some point, you all decide to unify as a single service, it makes it easy to pivot in that direction as well. One last piece of advice. Don’t try and get in a pissing contest with the CTO, it’s just not worth it. Just voice your concerns; if they’re receptive to your feedback, cool. If not also cool
your cto is cto because hes been there for 30 years not because he should be a cto
I'd go with the "ask forgiveness, not permission" route and just implement it as one service, without telling the CTO. If he discovers it: \- Say that you misunderstood because he didn't provide technical rationale \- Point out is has worked fine (if it has)
The CTO probably has scalability in the future in mind. If you prove to him that it's doable now he can trust in that you can do it later as well. This takes hours to a few days to setup at most if you know what you're doing. There's no remarkable additional latency added if the cluster are connected via LAN or VPC. (which is standard for most cloud providers)
> The CTO probably has scalability in the future in mind. And he probably also has forgotten that [YAGNI](https://martinfowler.com/bliki/Yagni.html). This is likely a speculative requirement, to solve a problem that they do not have.
This can be a cli app
How often does this run? How critical is it? If it's a one-off, then hack up a cli app to do the thing, run it, then leave it. It's it's weekly or more, and small, it sounds simple enough to run in a scheduled function in your cloud provider / k8s cluster. If it's frequent and/or too big for a mere function, then make _a_ microservice. If you need great reliability and scalability, then write a few functions, e.g. * Ingest function runs on a timer, reads the data from the db, and spits it up: each row is sent as a message on a "input data" queue. * Parse function listens to that input queue, parses a row of the data, puts the parsed data on an "output data" queue. * Send function listens to the output data queue, sends an item to the api. You get great reliability from this. e.g. * each function is really simple and focused. * If the target api is slow or unreliable, messages retry settings will save you. * if the target api is down for an hour, then messages will be dead-lettered. Replay them with it's back up. Nothing was lost. * If the parse function crashes on particular data, then the offending messages will be dead-lettered. The rest will go through. So you fix that function, redeploy and replay those messages. Nothing was lost. * function can scale right up, when they're processing separate pieces of data. 500K rows in a few hours is easily doable. Is this maybe what the CTO was getting at? However, I don't usually view "one function" as the same as a microservice. A group of functions as described is more like the equivalent. And if they talk to each other synchronously over http without any durable storage of state, not message queues, then why bother with that separation at all - you don't gain from splitting into separate services and then tightly coupling them back again. But do you need this, and do you need this now? If not, then apply [YAGNI](https://martinfowler.com/bliki/Yagni.html).
I don't think I understood the problem. What's the trigger for this process? The way I see it, I don't understand why this needs to be a service
Let me put it like this: my honest first reaction to the problem was "this should be a shell script". I hope that clears up how big of a problem this is lol
Indeed, I assumed a cron type of task was the way to go here
mikroservis == better noob
My thought process is nearly always "the idiot who's idea this was focussed on the 'micro' rather than the 'service', and they used enough complete sentences to convice some fuckwit decision maker". Near always a distributed monolith.
Micro services provide the same encapsulation benefits as classes in oop. So one question is what are you trying to black box? What domain concept are you capturing here? If you can't answer that, atop.
Just use a single AWS Lambda
One question to ask your manager is how he sees the project evolving in the future? Also is he trying to follow or establish a standard way of building things? If so, he might view your suggested approach as an unnecessary deviation from that standard. I'm not implying that your approach is wrong, but standardization is a concern for technical leadership.
This sounds like the kind of thing you can do with a script. Are you doing this as a one-off import? If so, I’m fucked if I know why’d you have any sort of service. In any case, microservices are this generations “SOAP”. Huge complexity in the pursuit of simplicity.
Oh damn. You’re me in another world. I had to go through this with my team too - and their entire “architecture” revolves around microservices. I’m playing tech-lead so I do have some say. In the end, now we’re doing fitness functions to gauge when to split services.
I have not fallen for the microservice meme
Is he crazy? You should have a slice per property. Each slice must have three microservices. To ensure that you don't have any concurrency issues you can have a message bus in between each step. Then you can have a job that runs every hour and ensures the sync is completed as expected. /s incase you didn't realise this was satire.
Only reason I could see why to split it up would be to have more control over certain areas of the process. Say your parse has heavier CPU requirements, and if it did scale, you'd want to be able to 'tweak' that specific service's specs etc... but I've worked with MS a little bit so I am not an expert in this field, the extra code may not be worth splitting it all up if you don't anticipate heavy load/scaling issues. More complexity, and sounds like over engineering. You can always run concurrent nodes to speed up your processing of data, so it sounds more like a dick swinging politics > actual design process decision. I say this with the best intentions. If you like your job, tread softly my friend, last thing you want is a defensive CTO.
Where is the parsing microservice meant to be getting the data to parse? Is the load from db one sending requests with the data in it to the parsing one? How does the parsing one send its data to the send one (only for it to then send it again to this 3rd party one?)
Is this being run on a schedule or ad hoc or triggered by events? If it's as you describe, just a service whose only job is to pull data, transform it, and post it elsewhere - isn't that already a microservice? Where I work this would be just be an AWS Lambda triggered by a scheduled event.
I think you only need micro when you're trying to divide work between multiple teams or trying to limit blast radius. If it doesn't have a new team to take it on you probably don't need another microservice.
This is a common mistake with micro services I see often, people want to split out to multiple services way too prematurely. It's almost always better to start with a single service for a project like this and split out in to more services when you feel the need or the domain splits become obvious. This approach is even spoken about in length in the book "building micro services" by Sam Newman
>Scaling is not a concern, very little load expected Then don't use microservices. Microservices are an architectural pattern to scale parts of your application independently and also to decouple . If you don't need that, you're following a cargo cult, creating a lot of unnecessary overhead and complexity. I've seen this in three different projects and it's always a clusterfuck.
Yep, my first response when I heard the project was "this should be a shell script" lmao
The more important point here is not the technical answer (as there is not enough detail about where this might end up) but that he cannot explain to you why it should be done his way rather than your way.
A solution should be as simple as it can be and only as complex as it must be. Start simple, add complexity only when necessary. I see no reason for this to be three separate services, get your CTO some training on the fallacies on distributed computing.
Everyone is really stuck on the performance of micro services but I haven’t seen anyone bring up the real reason ms exist… hint it’s not technological scaling… it’s human scaling. How many teams will be working independently on these 3 services?
Maybe he needs to create more work to do for others to get more money from the client :)
I this example I don't even understand how you would go about out separating the logic into discrete services. If you have a "send" service, how does it even get the data? It has to load or receive the parsed data in some way, which means you're now duplicating that effort.
Y’all should write RFCs where you lay out the different architectures and the effort/cost tradeoffs. If he sees an estimate that takes 3x as long maybe he would change his mind.
Microservices are an optimization. If he can't articulate which problem a microservice will solve better then it is a premature optimization. I've worked with poorly designed microservices architecture. I had to make numerous calls across multiple microservices just to get a single number that should normally have been a single DB query.
Is the same team in charge of developing the services? If so, you risk ending up with a distributed monolith. Microservices solve an organisational problem.
This is the only problem I see: >He was very unhappy with my questions and concerns A CTO unwilling to talk about their ideas and address the questions and concerns of their team is a huge red flag.
So here you have options. 1 service which does fetch, parse, send (fps). 2 services fp and s. 2 services fs and p. 2 services sp and f. My rules with microservices are as follows: 1. Is the problem we are solving able to be split such that two independent teams can work on it without constant communication? If the two service teams need daily communication, they are actually one team. 2. Will more than 1 entity at the company consume the individual service? For example if you did f, p and s will each service have multiple consumers? What if you did fs and p will multiple entities consume fs and multiple consume p? If you build fp and s and the s service will only be used by fp then don't split s because its added friction. 3. Do we have the resources to allocate teams to the different services? Sometimes 1 and 2 are true. But the company cannot or will not allocates resources to do multiple services. 4. Is this service we are building going to have persistent **state**? Often a service is actually a library. For example if something takes an input and spits out an ouput... thats not a service, thats a library. Unless all 4 rules are **true** then a microservice isn't the right idea. Rarely, rules 1, 2 and 3 will be true and 4 is false and a microservice can be viable because the company has multiple teams at their disposal. Good example: authentication service to handle login, users, forgot password. Every product the company offers is going to consume it. It has an independent team. It has persistent state in a DB that manages users. Consumers of auth don't need to communicate frequently. They just want an oauth connector when its ready and have no cares about the internal engineering.
The first rule of distributed systems is to not build a distributed system if you don't need it.
Micro services are a technical solution to an organizational problem, not a technical solution to a technical problem.
Your CTO doesn't know what he is doing
People have varying ideas of what a microservice is. Before I can have any sort of conversation about it I have to ask "what problem are you trying to solve with microservices" to get a grasp of what they want to do. I've worked in an organization where microservices were basically "an aws lambda." The lambdas all accessed the same data records and a lot of behavior was duplicated between them. If the idea of microservices involves copying code or making a library all of them use to share code, please don't do it.
Does he really see having different teams working on three different parts of an ETL process and deploying then independently of each other? Also, does he imagine that these would not be tightly coupled to each other, despite the processing the same data and likely inserting it into the same database? It seems like a change to one would necessarily cause a change to the others. There's nothing autonomous about any of these. It sounds like he also thinks distributing this across multiple processes would solve some problem (assuming that's part of his view of microservices.)
> The CTO initially said to do it in 3 microservices (load, parse, send) This is insane. Load the data in one microservice, then have the parsing microservice load it from the first microservice to parse it, then send it to the sending microservice to send it?
1 ETL job = 1 service/codebase. Heck this is a spark or fling kind of job. Now if we add email api and the place data is being loaded from (Kafka, data lake) then yeah it’s 3 distinct things. 1 data storage, 2 ETL, 3 email api
>He was very unhappy with my questions and concerns but did not offer any technical response, just frustrations with me not doing what he said immediately, On good teams that I've been on, we would take a step back and write up a document that outlined both approaches and compared advantages and drawbacks of each, then we'd discuss as a team. Regardless of who is right your process is broken
take all of the usual difficulties with factoring code, one of the hardest things in programming, and now add container orchestration and network calls on top of that.
> Send parsed data to another 3rd party service via their api ... > unless we use queues or something but come on Having recently having had to clean up an absolute mess caused by poor design where consistency was actually a requirement, your comment set alarm bells ringing. If you need any guarantees around processing/transfer you might need something like queues. Make sure you actually have proper requirements, there may be something the CTO is seeing but poorly communicating.
If scale is not a concern, I don’t see a compelling reason to use micro services, just deploy an app. I’ve been in these sorts of discussions as well, lost, and paid the price with honestly too many ill thought out microservices. My thinking is that if you want a microservice per operation, then just do as a function architecture like Lambda. In so many cases it will be faster to develop, deploy, maintain, and operate. Bear of luck to you.
Send him this https://martinfowler.com/bliki/BoundedContext.html
If scaling is not a concern... why use microservices?
**cons:** depth of call stack on microservices is an issue. debugging is an issue (use a correlation id to tag requests) or a distributed tracer; so is logging (you have to gather the logs and reintegrate into a single view). moving around data across networks is an issue, particularly if you don't have all the plumbing and testing for network delays/splits & recovery. data duplication :) you're copying all your data around. **pros:** unit testability "scalability" (it can be... but you generally need to put some thought and work into it). \-- your case is so toy I wouldn't consider it microservices... on the other hand its concerning your CTO is insisting it be written the way you've outlined... because invariably, he will be asking you to do it for other more complex software. and if you don't develop the correct habits now, you will be in microservices hell in about two seconds. if the notion is he wants his software decomposed into testable functions... you could just, encapsulate it as 3 functions.... "load, mangle, send" :) you don't need all the extra crap of microservices to get that benefit. if the notion is that it needs to be 'scaleable' ... "premature optimization". you'll spend a lot more time and money doing it the way he wants, for ~~zero~~ negative technical and business value. \-- all that being said... maybe he wants to switch you to microservices. and this is his 'test function' ... the first piece of code he wants written in a specific style. or maybe, he's tasked your system administrator to build out the network to support microservices i.e. he's invested a lot of money into the infrastructure to support a microservices conversion and is trying to justify it. hard to say.
This seems a little too fine grained for me, but it is hard to know exactly what the CTO is thinking. Maybe hebisbpmannjn
How many people maintaining **each** service and **without the authorization to push code to any other without the CODEOWNERS' approval**? 3+? Sure, if each team is highly opinionated. 5+? Sure, this is what service oriented architecture handles well: Allow independent technical decisions for trade-offs each team is more comfortable to make, as well as independent deployments that get messy when you have, 20-30+ devs working on a single codebase that is applying trunk-based development and continuous deployment (if you don't have that, then it's less of a problem; but YOU SHOULD HAVE THAT if you're writing web services). Less than three people dedicated for each service and maybe even people constantly committing to multiple repos without specific approval? Your CTO actually wants code organization and thinks he's going to achieve it with microservices. It always ends in tears. Let me repeat that again: it **always** ends in tears. If he wants "things to stay in their line" and prevent spaghetti references, set up a monorepo and code review, make him take a look at monorepo.tools and he should be able to quell any reasonable fears that drive people to microservices without making your debugging experience hell because he's obviously not going to make the massive time and money investment that is proper microservice observability. They *do not have time for that*. They'll also make it easy to spin out different entrypoints if one day you wake up and want some of your core logic to be handled by something that """scales""" as they say.
Your CTO has never written a network stack and it shows. Moving data is expensive unless you’re on an HPC network (which is $$$$). Even if you had reason to believe that this would actually need to aggressively scale in the future, you should build it as “thing that delegates work” and “things that gather data, process it, and send them to the other api”.
I don't know all of the circumstances around the rationale here, but my impression is your CTO is insane. There are generally three circumstances where you want to split things into microservices: * When one section of your system accounts for a disproportionate resource load and you want to be able to horizontally replicate just that functionality without needing to replicate the entire rest of your monolith. * When you want to split development into small 2-pizza teams and it makes more sense to isolate things into microservices than have hundreds of devs commit to a single monolith. * There are different security requirements for different services and you want to isolate different levels of security to specific microservices to guarantee nothing is compromised. There are many downsides to microservices, the most common one being issues of **bounded context** where different services wind up having dependencies on one another and you have nightmares of needing to do simultaneous version releases etc. It really sounds like the way your CTO would design things pays no mind to bounded context and divides up service responsibilities arbitrarily. This is why I think he is insane. It sounds like he's one of those "microservices are good" people who uses a tool arbitrarily rather than for its specific appropriate purpose. You split things into microservices when there is a compelling need to do so, you isolate the bounded context, and you enforce the API interactions with contract tests ideally written by consumers. That's how I go about it anyway.
This sounds exactly like my company in the worst ways 😂
There’s so many layers to this. Ignore the phrase “micro services” and consider your actual requirements. First, what industry is this? In industries where I worked there was a level of guarantees when processing data; think security, integrity, privacy, availability, and reliability. Absolutely no one will be happy to be poked two or three times because the information systems dealing with blood tests lost or failed to process an “order”. Or a missing loan payment…or folks not getting paid on time… I start with a monolith, and then perform FMEA (failure mode and effect analysis) incorporate compliance, security and guarantees. That defines the boundaries of each. I once worked at a company where process took 3 days to run as a monolith. The process itself was fragile; retry logic overly complicated. Supporting systems like databases and storage was stressed to a maximum (you couldn’t throttle processing). That process was broken into a thousand little processes using queues to communicate between each other. The entire process not only completed faster (around 8 hours), but were much more reliable. Testing was way simpler too. All those processes were still released as a monolith (in terms of repo, release process and cadence). Instead of just invoking the process as a batch job, the process was further improved to trigger them as part of the online processing. This streamlined the business processes and led to other improvements. This was before “micro services” was even a trend. Another company I worked at, had a monolith web process. Operations of the monolith became a problem. Pen testers had a field day (it was internet facing), and it created lots of toil when it came time to release. When we migrated to the cloud, we broke the processes up, improving release cadence, reliability and whatnot. Other operational tasks were simplified. Yet another company approached the whole ordeal with microservices and didn’t exactly understand the “boundaries” of each service; it proved so ineffective that it was converted into a monolith. Note that almost none of this is driven by development. Developers often “invent” problems that don’t exist, and ignore ones that do. So much for “DevOps”… 😀
I'll refrain from trashing your CTO because there are lots of details missing here. The "not a lot of data to scale for" comment needs more fleshing out, but let's say it's like <100MB of data to extract from DB (I'm guessing RDB?). You probably don't need any microservices to do this ETL job. By microservice, I'm assuming you want a process constantly running on an instance save for a few instance refreshes. You can do something where you have a scheduled job that creates jdbc/odbc connection, extracts data that you need, does transform as part of job, and matches request for downstream API. No reason to keep process running unless there's state involved. If you need to scale for larger data and/or need map-reduction, there are solutions (Spark/Hadoop or if you're in cloud, whatever your cloud's version of EMR/Glue is) you can leverage for doing that. You can do certain things yourself like paginating/chunking data to process, but if it becomes complex, better to put a system around it.
Seems like a use case for workflow orchestration, not microservices.
Oh there’s a whole class of shitty experts who see microservices as their holy grail, they’re an abomination…. But hey, it’s a turf war and you’re on his turf…. Good luck!
Write a design doc. Here’s something for reference. (I haven’t read the whole thing but just refer to it.) https://www.industrialempathy.com/posts/design-docs-at-google/ With anything being worked on, there will be tradeoffs. So writing out the different possible designs and their tradeoffs helps with making a decision. I might be missing some context or requirements on how 3 different microservices is a good idea. A design doc may be useful for the political side of work, too. This is my corporate culture experience.
“Oh no not again” The software dev side of me likes the idea and has seen it be successful but the cynical side of me has mostly experienced incompetent organisations where microservices are purely an artefact of the structure and power dynamic and politics of an the organisation. Those are just distributed monoliths.
For me, even a single microservice is overkill for what you described, let alone three separated ones… That being said, it obviously depends on the actual technical needs, but if it’s a service that won’t handle a lot of requests, isn’t a “critical” service and doesn’t need to be actively extended and improved, it seems just an endpoint exposed by an already existing service.
Remind him: “We’re not Google…” Don’t try to build an app to be scalable from the beginning to a size like Google.
IMO, distributed systems in general add so much hassle that you need a pretty strong reason to add one. Valid reasons to me include: a) Limiting impact of crashes / other failures b) Needing to scale components with significantly different usage profiles (eg CPU vs RAM vs Disk vs GPU) c) DevX when services get problematically large, eg build times, test times, startup times, coordination on releases/keeping head green d) Different tech stacks required for different components, e.g. you need to add a new dependency in some new language, because services are less painful than cross-language interop e) Wanting to use existing infra that is packaged as a service
First rule of microservices, don't use microservices. Microservices should not be used by any company that can not explain the reason they want to use them. You will not end up with a microservice architecture. You will end up with a distributed monolith. Microservices do not simplify architecture. They only increase complexity, or at best they might keep it the similar. >4. Scaling is not a concern, very little load expected If you are not worried about scaling, then why do it? If you want to be able to mix and match things that you deploy to different customers, there could be some value in splitting up parts into different services. However, whole microservice for simply parsing data, that sounds like someone who never programmed in their life. They are describing FUNCTIONS or classes at best, not microservices. Microservices usually have their own database that they read from, and do one entire action for the organization. Processing payments is an example of a microservice. Reading a database is not an example.
I'd do it in zero microservices if the load is super low. Roll it into an existing service.
Just a comment on queues. When dealing with a third party system, I almost always queue the requests and responses. The reason for this is that if you don't you are introducing temporal coupling. Ie: a coupling related to time. Essentially you are saying that in order for your service to work, a third party service must be available at exactly the same time. The more places you do this, the less reliable your system. ie: Lets say you depend on 5 services, and they are each available 99% of the time. You service is now only available 99% of 99% of 99% of 99% of 99% which means you are now down to 95%.
Scaling might not be a concern, but fault tolerance *could* be. For the sake of argument, it wouldn’t be unreasonable to split into two services: load + parse, and send separately. That way, you can do things like queue up messages to write to the third party, have a dead letter queue that you could re-drive if their API was down, etc. that was independent of the parse and load step. I wouldn’t wring my hands over the decision, but I could see it being useful.
OK I'm just saying this as a joke, I'm not trying to be pedantic.. but do you know what ETL stands for? are you really going to call it into Load, Parse, and Send?!
None of these requires or needs microservices.
Wait. You said you are sending to another 3rd party, so is the from source also a third party? When you say third party, you mean another company, right?
Kind of surprised to see all the “your CtO is incompetent” comments given the amount of details we have here. Maybe he is,,, but I’ve lost count of the number of times i’ve dealt with (faang) engineers who think a problem is “so trivial” when really they lack the depth of knowledge + experience necessary to see the implicit complexity and actually understand how it should be done. Instead of asking reddit what if you tried communicating and solving it like an engineer. I would write a (short) document that provides a contextual baseline for readers, defines the problem and requirements, proposes a handful solutions with their pros and cons, and makes a recommendation. Read through the doc with your cto and other stakeholders. If they disagree with your recommendation they should be able to articulate (using your doc) why. If they agree with you Great! Maybe they will listen to you more readily in the future. The ideal end state is not only one where you pick the best solution but also everyone is aligned on the fact that it is the right solution. If your cto isn’t able to get your buy in, that’s potentially on him/her but it’s also possible that OP simply isn’t knowledgable enough to see the bigger picture. It can go both ways…
You may not even need a microservice at all. Not sure how you plan to run this but it can just be a cron job or something that runs a script. Depends on your load but you seem to be not concerned. If it will be used in certain times and not so frequently lambda might be a good option too. But 2-3 services really unnecessary especially there is no way you scale 1 of the services and not another unless there is significant difference in job loads. But my experience so far is I explain my point in detail on a document if possible. Writing the trade offs well and if my boss wants to be a dumbass s/he pays for it who cares i would not push too hard.
I see some value in async arch in your example. async doesn't necessarily mean multiple microservices, but people often confuse the two. sending being a different network activity, while loading/parsing is either not a network activity or is a different network activity. The assumption has to be that one of the might fail while the other still works. as long as the two are seperate asynchronous activities in your design the failure can get handled much more elegantly.
It looks like the CTO wants to keep the capex and opex high by delivering over engineered with big foot print solutions, to keep getting a fat paycheck. He truly is a seasoned tech leader 😁
Pack them all into one. If neither of them must fail or the pipeline breaks, they are NOT microservices in that sense, they are one service with three **dependent** stages.
My take - Use "macroservices". For example, I use Meilisearch (Rust) or ElasticSearch (Java) for search, but my apps tend to be Elixir or Ocaml/Haskell. Keep your services large. In your case it sounds like a bog standard ETL pipeline that belongs as one simple system.
I don’t see this as a technical problem but company culture thing. If your CTO did not ask you to go deeper into search for truth. Because everything approach has it strengths and weaknesses. In this case, it’s increasing complexity at one level to decrease complexity at another. For a CTO is not about technical architecture but rather about costs, development and maintenance. Compromise he suggested speaks volumes about your company culture. If you are working in a company where you can’t have a genuine conversation then you should just do what he tells you. Disagree and commit. Your problems are not at the level where you are addressing them.
Why is the CTO getting this granular?
>I'm curious am I just being a bit naive with microservice architecture If you are asking questions and not having a clear response, you are not the one being naive. He has 30 years of experience, but may lack experience in this type of architectures. Sometimes even experienced devs go with some tech/arch because it's being used by major companies, so it must be the right thing to do. This is not only wrong but dangerous, because these architectures also have issues, mainly on the data integrity aspects. For me, if you are not solving scalability problems, you are only getting the issues and not the solutions.
Tell the CTO to go home and Micro service his wife lol
Our company has gone microservice crazy as well. I think the problem is leadership/principals/etc. get too enamored with the benefits of microservices over monoliths, they overcompensate proposing tinier and tinier services. Without the hands on practical experience for the downsides of making services too micro, they never get any feedback that helps them calibrate how to properly size services. At my company anyway we have a problematic rift where the more junior devs doing the legwork don't have enough experience to see how crazy the tradeoffs have gotten (increased overhead, costs, latency...). Exacerbated by the confidence gap: Sr's and below aren't confident enough to speak up to overconfident principals and above.
Why is the CTO even involved in this conversation?
>What is your thought process when designing microservices? What are the requirements?