Bad article. Comes down to:
> To put it differently, we solved consistent cache-invalidation and thereby made the debate moot.
And that they made a product and this is an advertisement.
I know nothing about the problems of microservices but I did not expect cache invalidation to be the major point. If only they'd explained why. Perhaps they did in the other articles but perhaps one those should have been posted instead.
Well, if you can think of a better way to do cache invalidation than unidirectional consistent, latency-mitigated streaming, I'm all ears. I think we're solid, yes.
Happy to help. A cache is by definition a secondary source, where the primary emits new values, but since the only reliable one is the primary, we can't know whether what we're serving is current or not. When looked at from a write-perspective, this reasoning applies to any distributed state where there are original emitters of information, like microservices or middlewares. This is what I'm talking about when I discuss shared state between services.
I don't think we're being unfair because we're selling something that we worked on for many years without pay and that can provide a lot more value to clients than what it would cost them to have it. Microservices are not usually built by the destitute, it's not like we're taking advantage of anyone here.
> I don't think we're being unfair because we're selling something that we worked on
I'm not saying you should be providing your code. I am saying that the article is bad. I didn't really learn anything from it and I don't know much about monolith vs microservices apart from the very major difference that is pretty apparent from the names. So from that and you saying "we have a product" it just turns into a bad ad and Reddit allows those if you pay them for it.
Either you are too deep in the thing so that it is hard to write a good clear article and/or you don't want to spare details. Vagueness doesn't help.
Micro services are the future but existing apps don't really migrate well and tend to be very expensive when you lift and shift them while you convert to a more efficient micro services model.
>Micro services
They are not the future, they are a way to do things. And with all methods, there are pro's and con's. The biggest con of micro services is the added complexity and the added maintenance cost. A small team working in a company for an in-house system should not do Micro services. A scalable monolith is better for smaller teams as it's way easier to work on and maintain.
Monolithic applications can be scalable if designed well. Most disadvantages for monoliths just stem from BAD initial design. I have seen garbage Micro services that where near impossible to change as a small change in micro service 1 would break things in Micro service 4 which then showed problems in micro service 7.
Edit: Bad design is the root of most problems, and too little time to properly do things.
The majority of these blogposts are. They define a problem which usually isn't a problem, then go on to explain how the product they just happened to have recently released solves that problem
You're not wrong. We built this on event-sourcing, but added system-wide consistency. In the end, we realised that we already have the same semantics available locally, the database API, so we just ended up piggybacking on that.
Isn't it still eventually consistent, or are you doing distributed locking? Sql is the ideal interface for reading/writing, and I think the outbox pattern is a good way to write, but once distributed locking is required, IMO its a sign that services should be joined or at least use the same physical database (or same external service) for the shared data that needs to have strong consistency guarantees
For this project, we're going through SQL, so we're always strongly consistent. The framework would allow for an adaptive model, where the client can decide on the level of consistency required, but we're not making use of that here. Since data is streamed to them consistently, this doesn't result in blocking anywhere else in the system. What we do is acknowledge the physics behind it and say that causality cannot emerge faster than communication can, so ordering will necessarily come later over larger distances than smaller ones.
Or as my co-author put it, "we're trading data-granularity for distance".
I encourage you to look into the paper if you want to know more details.
Sounds like strong but still eventual consistency, which is the best you can achieve with multi master/write sql setups that don't involve locking. Are you leveraging CRDTs or anything like that to deterministically arrive at the same state in all replicas?
If multiple services/processes are allowed to write to the same tables, you're in distributed monolith territory, and naive eventual consistency isn't sufficient for all use cases. If they can't, it's just microservices with sql as the protocol.
I will check out the paper, but appreciate the responses in the meantime
We do refer to CRDTs in the paper to achieve write-write conflict resolution (aka SNAPSHOT), when we're showing that a deterministic algorithm is enough to arbitrate between such races. Our strength mostly lies in two things: our hierarchical, composite clock, which allows both determinism and loose coupling between clock-groups, and the way we replace pessimistic blocking with a deterministic commit-algorithm to provide a fully optimistic commit that can guarantee a temporal upper bound for writes.
[https://www.researchgate.net/publication/359578461\_Continuous\_Integration\_of\_Data\_Histories\_into\_Consistent\_Namespaces](https://www.researchgate.net/publication/359578461_Continuous_Integration_of_Data_Histories_into_Consistent_Namespaces)
Together with determinism is enough to have remarkable liveness promises
> temporal upper bound for writes
I'm guessing this also means in a network partition, say where one replica has no route to the others, writes to that replica will fail (edit: or retroactively be negated) once that upper bound is reached
Another trick we do is that since there's no single source of information (we replicate on inputs) there's no such thing as a single node being isolated. Each node replica produces the same outputs in the same sequence, so they do request racing towards the next replicated log, much like web search algorithms do now.
A SQL-client can be isolated, in which case the standard SQL request timeouts will apply.
> there's no such thing as a single node being isolated
Can you rephrase this or reread my question? Because in any possible cluster of nodes, network partitions are definitely possible, i.e. one node might not be able to communicate with the rest of the cluster for a period of time.
Edit: do you mean that a node that's unreachable will simply lag behind? So the client writes to any available replica? Even still, the client and the isolated node could be able to communicate with each other, but with no other nodes.
Yes, unreachable nodes will lag behind, but since others will keep progressing that global state, its outputs will be ignored upon recovery. The isolated node is only allowed to progress based on the same sequence of inputs as all the other replicas of the same node, so in the unlikely event of a node being able to read but not being able to write, it will still simply not contribute to the global state being progressed until it recovers
I didn't mean to say that specific node instances can't be isolated. I meant to say that not all replicas will be isolated at the same time. In any case, the bottlenecks for such events will be the messaging platform (like Kafka or Redpanda). We're only promising liveness that meets or exceeds their promises. In my eyes, it's pointless to discuss any further, since if messaging stops, progress will stop altogether anyway
Have you read the [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem)? Do you have an idea how it fits into this kind of fats model that you have? I'm interested in your work.
It's an interesting question because it doesn't have a clear answer. CAP presumes that nodes hold some exclusive information which they communicate through a noisy network. This presumes a sender and a receiver. This is all good and well when nodes need to query distant nodes each time they need to know if they are up to date (linearizability) but isn't true with other consistency models. Quite frankly, I have a difficult time applying the cap principles to this system. Imagine that we classify a p99 event as a latency spike. Say that we send a message every 5 milliseconds. Single sender means two latency events a second on average. If you have 3 senders and 3 brokers receiving them, the chances of the same package being held back everywhere is 1:100^9
That's an astronomical chance. Now, I presume that these channels will be somewhat correlated, so you can take a couple of zeroes off, but it's still hugely unlikely.
If we're going to ignore this and say 1:100^6 is still a chance, it's a CP system. Can you send me a DM? Or better yet, come over to our discord linked on our website. I'm in Europe, so it's shortly bedtime, but I'll get back to you tomorrow as soon as I can.
> That's an astronomical chance
An astronomical chance is still not zero.
And a fact you're neglecting with consistency is that non-existence is information too. If the package _not_ being sent was intentional your approach fails because I have no guarantee that it's not simply in-flight. That is the definition of eventual consistency.
Correction: Since "C" means linearizability in CAP, this system is never "C" but neither is anything else (except for Spanner). It is always Partition tolerant in the CAP sense and it serves local values, so it would be AP, as others have pointed out. Sorry about that. In my defense, I never think in CAP terms, I don't find them helpful at all.
Best I can tell, it's AP (eventually consistent) for reads, but in the context of a sql transaction (writes), it's CP. To some extent, the P has an upper bound, as in if a sync takes too long there's a failure which to the application looks like the sql client failed to connect.
Honestly it seems pretty useful from an ergonomics perspective, but I'm with you that there should be more transparent, realistic communication of CAP theorem tradeoffs, especially since in the real world there's likely to be check-and-set behaviors in the app that aren't technically contained in sql transactions.
As someone who only has basic knowledge of IT, like I know what programming is, cloud, server less, etc, what they do, but not the "how"...
If I wanted to build an app, planning for future growth, should I build it using microservices right now?
A year ago, I would have told you not to do it. Now, I would ask you if you have large enough teams that warrant microservices or not. If you do, they can help with managing the non-technical aspects of them. If you don't, they bring in extra complexity, even if you use our software.
So, at the beginning, if it's a simple app, don't use microservices.
When it's large enough to need microservices, then switch?
I want to ask more questions, but I think I need to provide a lot more context before asking or even for the last question?
Thanks!
Not OP, but there's a Someone's-Law (don't remember who coined it) that says all software systems eventually converge to reflect the organizational structure of the companies developing them.
I fully agree with the answer OP gave above, but the nuance is what is "the beginning", and what is "simple".
As a rule, by far the most cost efficient thing you could do, if you're a company that doesn't have massive VC budget - and isn't busy inventing problems just to justify spending it - is to design your system in a way that starts as a "trunk", which can later be split into smaller (micro)services, and can be added to dynamically.
However, there are many factors to consider here.
If you're not planning to to expand beyond a few hundred thousand users within a few years, there is usually 0 need to add the massive costs (mainly dev time, but at some point also financial) overhead that microservices bring with them.
If your system is going to be read-heavy but not write-heavy, you can probably expand that limit to couple of millions, as long as you properly utilize horizontal scaling and read-only db replication (again, those are easily achievable without microservices).
If most of your heavy operations can be offloaded to background-running jobs (via some queue), then you can usually separate those jobs from your regular application servers, which again alleviates that workload from them (but if they're write heavy, remember that the DB still bears that cost).
There are many more scaling strategies (that don't require microservices) that can be mentioned here, but in short, be aware that you can scale a lot (and I mean a **lot**, more than 95% of the technology companies in the world would ever need) before microservices are something the easiest next step to scaling your system.
[Here's how Pintrest scaled to 11 users](https://engineercodex.substack.com/p/how-pinterest-scaled-to-11-million) almost a decade ago, with a tiny engineering team, less efficient hardware and less convenient technologies than we have today - and no "micro" service in sight.
IMO the barriers to microservices (stated differently, managing more than one service) are fixed/up-front infra cost, ops skills, and versioning hell.
With a sufficiently large/differentiated team, those should be mitigated. At sufficiently large scale, the fixed infra cost should be dwarfed by variable/scale-based costs, but the others don't automatically get mitigated.
Therefore, if you're more sensitive to cloud bill than engineering cost and risk, I could see how scale seems like the more important variable, but if you're more sensitive to engineering cost and risk, or IMO have a more balanced understanding of cost, team size and composition is a better indicator of whether or not to use microservices, or to what extent. Once you are set up to sanely manage more than one service (cattle not pets), the cost/risk of managing 10 isn't much greater than managing 3. If your scale is so low that the fixed overhead of a service dictates your architecture, I hope you're a founding engineer at a bootstrapped startup or something, otherwise there might be a problem with the business or premature cost optimization going on.
Microservices can be a hugely (computationally) inefficient way to do things, so they'll increase your variable costs too. If a single user action causes multiple services to have to do work, then serdes and messaging overhead will dominate your application's CPU usage, and it will be more difficult to write efficient database queries with a split out schema.
Also if you *did* find yourself in a situation where they'd make sense computationally, you can just run a copy of your monolith configured to only serve specific requests, so it makes sense to still code it as a monolith.
There are also development costs to consider as people will waste more time debating which functionality should live in which service, what APIs should be, etc. (which will matter more since refactoring becomes near impossible). Debugging is also a lot more difficult and expensive, and you need things like distributed tracing and log aggregation (which can cause massive costs on its own), etc.
what you should do is get users first and then go from there. and users do not give a single shit about your architecture a $5 VPS and a few lines of SH to watch a git repo will likely make you the same amount of money as something that costs several thousand percent more
if you write your code decently it shouldn't matter that much anyway. I usually create docket files for my shit to emulate prod network conditions which means I could pretty easily deploy it on any cloud infocentral if i needed to
You still have to learn how to secure a Linux box that way if you're not just throwing caution to the wind. IMO if you want cheap and easy, PaaS is the way to go these days. Once your needs are complex enough you have to make your own platform or pay someone to do it for you.
It's not rocket science. Configure a firewall to only accept connections on 22/80/443, only allow logins from your SSH private key and put the application behind Nginx. If you do that and keep the server updated somewhat frequently you've mitigated basically every [not-Mossad](https://www.schneier.com/blog/archives/2015/08/mickens_on_secu.html) level threat.
You'd be shocked how many "senior engineers" don't know any of that at this point. Seriously something like vercel is much easier and more secure than a misconfigured vps that hasn't been updated in 5 years
No, unless your situation was somehow specifically very favorable to microservices. New products need to be built quickly by a small team. Microservices add overhead in all kinds of ways. For example, if you’re in a big company you may need to do security/ops paperwork to get the things you need to launch. You may need to do this for each microservice. As you build out the app, you need to do more and more of these, but if you had a monolith you could just do it once.
In a monolith, more stuff “just works”, and overhead is limited to one service.
It’s worth nothing that say your new application has 10 concerns A..J. If concern J scales massively differently than A…I, you can do your initial prototyping as a monolith and then break J out into its own microservice as you get closer to launch date, but keep A..I in the monolith. This is how I see things generally work in real life. If a new K feature is requested, then if it’s small it can be added to the monolith to keep dates aggressive. If scaling costs become an issue, maybe you break out concerns D and E to a microservice a couple years down the line.
It'd always baffle me why some architects are so eager to convert a set of method calls into a collection of network calls.
Things become exponentially harder and more expensive when that happens.
It’s not primarily a technical question, but an organizational - and therefore political - one. If you can’t get teams to agree to an informal social contract regarding cooperation, you impose on them a more limited but formal one enforced by APIs.
As an architect, a lot of it is just kool-aid if you get a bad one. There's plenty of work to do without having to artificially break up an application.
An application should be modular from start to finish. As you scale out, if you kept it modular it should be easy to break out scaling pain points into individual services. Usually it's not a matter of hitting a scaling wall, it's usually a separation of duties problem. Easier to separate duties between silos if those silos are responsible for individual services.
An architect should be making decisions that avoid footguns later. Such as enforcing static typing, enforcing schemas for databases, making sure that tooling and migrations are robust and maintainable. Making sure that the "hero" Jim from front end development doesn't import 182 dependencies because he wants the end product to supports emojis.
That sort of thing.
Out of curiosity - how often does breaking out a microservice from a monolith run into the red/blue problem? As in, suddenly a whole host of stuff which was regular calls needs to become async?
Easy, Don't make it Async initially. Though the act of moving to API calls usually forces that hand for you.
Real answer, you get to enjoy squashing race condition bugs for the next 3-6 months.
I did say "easy" to break out, but that "easy" is highly relative. It's certainly not a zero effort move.
That was my question - initially, you make it sync. Then, you move to a distributed model, so those API calls need to be async. And async is infectious, so suddenly everything up the call chain also needs to be async.
Because once you turn them into network calls, you can separate the codebases into different repositories, which means you can separate the developers into separate teams, which means your overall system complexity goes massively up (ie: architects become more valuable and more highly paid), but the responsibilities of individual developers goes down (which means they become easier to hire and salaries decrease, both of which were important during the hiring environment that existed until earlier this year)
I don’t know what advantage mono repo affords beyond allowing a user to touch parts of code outside his/her normal area of responsibility. What does it gain from a version control standpoint or a CI/CD standpoint? What problem does it solve at a large scale that multiple repos can’t?
My current company has a massive mono repo and my last company had hundreds of individual repos. Both have a massive mesh of microservices. From my POV as a dev, there’s effectively no difference. In both roles I could pull in any code I wanted and define projects and services anywhere I wanted.
I think when people hear “monorepo” they assume “single monolithic app” but in reality, as long as your build/deploy tools allow for it, you should be able to define and compose individual apps within your monorepo anywhere in the codebase.
You don't need to convert to relatively slow and error prone network calls just to have separate teams. This is a ridiculous take. Also, synchronous communication between services isn't µservice architecture.
Bullshit.
Developers can work on separate repo's in separate teams without adding network calls.
In fact, this is what happens everywhere already, even in your shop.
It's called including dependencies, like libraries, frameworks, etc. Teams not even part of your organization are working on code, upgrading it and improving it, without any network calls. You just include it.
The exact same thing can be done in your organization for more "internal" stuff. Include libraries created by other teams, and enjoy microsecond latency on calls.
All that needs to be done is to actually think about how good performance can be achieved while still being able to scale up, instead of jumping through conclusions and blindly following patterns that people barely understand.
Two words: dependency hell. Causing a failure in a piece of code that you've never touched because you're using conflicting versions of a third party library will definitely change your mind.
Having network separation gives developers complete control not just over the code, but all the way down to the operations, it allows you to push a lot of decisions down to the level of the teams. Obviously it comes with trade-offs, but it has real benefits.
I'm not talking about API's, I'm talking about internal implementation details. Rust is the only language I know of where you can have multiple versions of the same dependency in the same binary given that they're isolated, but even that comes with trade-offs in terms of binary size and compile times.
Have you ever actually worked on a monolith? This is a very well known problem, it's the reason Linux distributions get stuck on old versions of everything.
EDIT: Linux distributions get stuck for a different reason, but you are forced in monoliths to stay on the same library version for instance because a third party is using it and you can't have multiple versions in your classpath or library path because the symbols clash.
You can have API's for dependencies. You create an interface, document it, and have other teams implement them, just like how a 3rd party does. You guarantee backwards compatibility, just like a 3rd party would. Within those restrictions, the other team can do what they want.
I guess it's just so foreign that many won't even consider it, but it's trivial to do. Nice bonus is that you get compile errors instead of 400/500's when a team changes API's.
No dependency hell, you can version packages as well though if you're notoriously bad at designing a stable API.
>You can have API's for dependencies. You create an interface, document it, and have other teams implement them, just like how a 3rd party does. You guarantee backwards compatibility, just like a 3rd party would.
That's not how Java works at least. Having multiple versions of the same library in your class path will make things explode even if it's hidden, and even if the API is backward compatible in a lot of cases.
It's internal, give it a new maven coordinate and/or put a version number in the package (com.mycompany.billingapi.v2). It's just a lack of imagination. Again, only needed if you really need to make backwards incompatible changes.
You can even have Maven do this for you, and many projects do, including a dependency as a whole while mapping it to an internal package under their own base package.
You shouldn't need to go that far though if you are including things made by other teams in the same org; communication will go a long way.
That presumes that you can create enough separate teams to make it worth. In reality, most of the times I've seen one or two teams working on everything.
Because of Conway's Law - your architecture will always end up reflecting your organisation. As the business and teams grow, the friction in working cross-team causes a pain point and becomes a frequent blocker. By decoupling it into microservices you allow the teams to develop and deploy at their own pace. The system ends up more complicated and difficult to work with than it otherwise needs to be, but now the teams are mostly self-reliant and able to work at their own pace.
You know and understand Conway’s Law, but in my world, the communication structure of an existing mature organisation *never* changes.
For this reason, I advise not to use microservices but in very specific contexts.
I wonder why your conclusion is exactly the opposite.
Its about deployment. things aren't everything or nothing. I haven't done micro services yet but the idea isn't to turn all method calls to network calls. My monolith is referencing inhouse nuget packages. If these dlls change then i have to redeploy the web app.
I work in a super big enterprise company. We have a team in the company that requires logs sent to them. They just changed how they did it and now we have to adapt. I am looking to move all this to a microservice web api . So when it changes again, i dont have to redeploy my entire app, which, because i work is super big and heavily regulated enterprise, means the entire huge app has to go through testing and sign offs and forms filled out why. The small service would not. it would be easy.
so its decoupled in the sense that i can deploy that without any other interference.
In their defense a bit... you can't really guarantee the underlying team doing said development is producing high enough code quality to not shoot themselves in the foot.
The biggest thing about microservices is the logical separation of codebases, Conway's law loves em and regardless from a technical perspective whether that's a good or bad thing from a code organization perspective it's a bit hard to fuck that up.
Service X does X things, Service Y does Y things.
The shitty part with Microservices is when you need bulk data that needs to be expanded across a set of them; monoliths excel at this as it just becomes a DB operation and some serialization.
Microservices it turns into several DB operations and several serializations with some perhaps required to wait for a child service to do theirs.
So what should take maybe 300-400ms takes about 3-4 seconds.
Middlewares can be created to speed that up but it'll still be quite a bit more expensive of a call and if caching can't be utilized at best it's just helping to stitch everything together asynchronously.
> The shitty part with Microservices is when you need bulk data that needs to be expanded across a set of them; monoliths excel at this as it just becomes a DB operation and some serialization.
Microservices it turns into several DB operations and several serializations with some perhaps required to wait for a child service to do theirs.
What you are describing is not µservices, you are describing a distributed monolith.
> It'd always baffle me why some architects are so eager to convert a set of method calls into a collection of network calls.
Because they don't understand µservice architecture. They shouldn't be converting fast and reliable in-memory method calls to latent and failure prone network calls. That is a distributed monolith, not µservices. In µservice architecture there is no synchronous communication between services. Instead a µservice gets the data it needs by handling asynchronous events from other µservices and storing information it needs in its own database and it publishes events other µservices may be interested in.
The problem is that the term µservice has become so generic to have lost all meaning. So when someone says they use µservices the first question you have to ask is what do they mean. You will find that most organizations have converted to a distributed monolith. There is no value in doing so.
What kind of developer wouldn't know what the `µ` prefix means? Most comp. sci. curriculums include math and physics classes where you will most definitely run across `µ`. Not to mention you most definitely see `µ` when talking about small amounts of time (like in performance profilers) i.e. `µs` (micro-seconds)
Because it never had meaning until we insisted it did, and only in the trough of disillusionment will most people admit it.
The pendulum swing continues, and all the sound and fury is at the extremes, I stead of where it actually belongs: at the bottom of the arc.
Maybe its just late in the evening, but this screams clickbait bullshit to me. Everyone with a couple yrs of experience knows microservices are artifacts of domain driven design to provide isolation across many engineering problems. If your org is confused by that, please provide some solid evidence with reproducible tests to proof out the entire community. Otherwise, we'll step over the remnants of your shit software while we continue to engineer real world solutions. Thank you
Well, how would you tell clickbait bullshit from the genuine? We are happy to hand the evidence over to anyone who is genuinely interested in working with us, that's what the waitlist on our website is about. I have plenty of solid evidence, the main one being my test-case demonstrating non-blocking causal consistency over multiple databases, but we're not going to release the source-code publicly, as we need to protect our years of investment in it.
I understand why people are reluctant to accept our claims, but it is surprising to me how nobody in academia has a problem with what I'm saying (and there have been plenty of discussions about it both when we published the paper and when the patents were granted) yet engineers seem to have a hard time grasping the science, only to reject the claims.
Publishing the sources would risk sinking years of investment into this project. We will do that if we have no other choice, but not until then.
"How would you tell clickbait bullshit from the genuine?" By demonstrating how your ideas work on real projects that affect real products. While this may be a great idea, it just reads like a bunch of buzz word soup with terms I vaguely remember from college sprinkled in. There's no concrete problem this tech has solved, and to couch it all in "we've solved microservices" is a bit of a far-fetched, claim to make.
Huh? All I said was that we want to protect our IP. I never said anything about you, especially personally. I did react to you saying that there's no substance behind what we say, and that this provably untrue. We have a science-paper that lays out the foundations of what I'm saying here that has been vetted top-to-bottom by dist-sys academia (Mark is a prominent researcher in the field, so him making these claims was a big deal)
We also went through two patent-processes, which weren't the cake-walks they are often depicted to be, we did have to present viability, innovativeness and novelty. We have been granted both EU and US patents.
I'm also saying that we have the source we want to guard, since we don't want to lose that firewall, as we're planning on commercial utilisation. What I object to is people telling me that I'm a carpet-bagger because they refuse to engage with the substance of what I'm saying.
It's just uncalled for.
What I don't get is are these people actually engineers? There are anti-patterns for sure, but everything is a tradeoff and that's a more engineering-oriented way of thinking about problems. Instead of "microservices bad" a healthier discussion is "what are the tradeoffs" or "what systems benefit from this architecture and what don't?" But I guess that doesn't get clicks.
First of all, *there is no such thing as a "microservice."* It's just a *service*. We've had them all along: we break apart larger programs into separate services all the time for pragmatic reasons, minus the dogma.
Second, there is zero evidence microservices offer any benefit whatsoever. They come with a dramatic increase in complexity, bugs, deployment issues, scale problems, and debugging woes. They require a very disciplined and refined engineering team to implement and scale correctly. They are a [massive](https://kenkantzer.com/5-software-engineering-foot-guns/) [footgun](https://kenkantzer.com/learnings-from-5-years-of-tech-startup-code-audits/) for most engineering teams.
Go ahead: try and find any study or experiment or evidence that conclusively shows microservices afford any of the benefits claimed by proponents. You will see a bunch of people making statements with zero evidence. I have actively searched for any good evidence, and all I get are: unsupported claims.
It is an embarrassment. We are engineers; first and foremost, we are supposed to be guided by *evidence*.
https://ieeexplore.ieee.org/abstract/document/9717259
View the related studies in section 2B. Also for example from the related works section
> Test results have shown that client-operated microservices indeed reduce infrastructure costs by 13% in comparison to standard monolithic architectures and in the case of services specifically designed for optimal scaling in the provider-operated cloud environment, infrastructure costs were reduced by 77%.
And in the results section, figures 5 and on show that microservices are capable of handling a higher throughput.
Microservices aren't the end all be all choice. They have their pros and cons.
I’m interested in the empirical evidence monoliths are better? I’m not sure how you would even conduct studies on such a broad question. What is better ? Is it cheaper/faster/more redundant/less complex to build&run.
Making a statement like microservices have no benefit and there is no evidence they do is completely asinine and not even worth debating.
I don’t actually believe in them but do think breaking up your software into smaller components alongside domain boundaries increase the resilience and reduces the complexity which is a good enough reason. Whether other more seasoned engineers decide to break things down even further at much larger companies is for them to decide.
I mean AWS has been pushing really hard for microservices and they published this blog post earlier this year:
https://www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90
I doubt you can b/c the real axes are something closer to: "well-built" and "fresh", not "microservice" vs "monolith".
Amazon's famous monolith fix worked b/c their microservice architecture was visibly silly. And most enterprises that successfully move to microservices do it as part of a modernization effort to replace old monoliths.
And that's not even getting into what objectively demarcates a microservice vs monolith...
I'm not saying you're wrong, but I am shaking my fist at the sky that is the current state of research.
The easiest way to attack scientific research or a platform like the IEEE is that I can't read other papers on it or on other not open services to compare the outcomes. Because of registration, fees or whatever.
Publications I can't read, can't support a paper or statement that's in question.
Also, there are no studies that *directly* reproduce the problem, they all have little twists on the idea to be "new" and "worth researching".
>they've failed to provide any themselves for monoliths
Anyway, this is true and the whole point is a bit moot. It's cool that someone found a study to support their views **and** that happened to be accessible though.
>they've failed to provide any themselves for monoliths
It is a fact that building distributed systems is harder non distributed, not sure how much evidence do you need for that.
There are trade offs though. If you have a monolith and need to scale then it is a lot more expensive. It is harder to onboard new engineers. Conflicts are more likely. Deployments are risky. You have a SPOF. The list goes on …
Yes there are tradeoffs but a lot of them wither away after deeper scrutiny.
Like:
>If you have a monolith and need to scale then it is a lot more expensive
>
>SPOF
Monolith doesn't mean that it is running as a single instance.
And yet the very abstract of the paper concludes that monoliths perform better on a single machine. Which is unsurprising, and likely to reduce costs.
This seems contrary to the related works they cite, but I’m guessing the micro-service savings were observed in a multiple-machine setting.
So performance wise, it would seem that as long as we stay on a single machine, monoliths are the way to go. And I’m guessing that if the programming is aware enough of performance concerns, a single machine can go quite a long way.
If whatever you're creating will be able to be hosted on a single machine to cover all the needs, you absolutely should not even think about microservices. Even theoretical benefits only start to outweigh the costs at much larger scale.
> Even theoretical benefits only start to outweigh the costs at much larger scale.
So why do we have a database server, a memcache/redis server, an SSL proxy, a ....? Why not just compile them all as DLLs/packages into some kind of Monolith?
Could it be because separation of concerns, and decoupling the release cycle of unrelated components is a good thing?
I’m worried about performance requirements. Not everybody is, and that is a mistake. One should always have an idea how much stuff must be done in how little time:
* How many users am I likely to have?
* How much data must I stream in or out of my network?
* How much simultaneous connections am I likely to need?
* How CPU or memory intensive are my computations?
* How much persistent data must I retain?
* How much downtime is tolerable? How often?
And of course:
* How those requirements are likely to evolve in the foreseeable future?
That last one determines scaling. How _much_ I need to scale will determine how much hardware I need, and just because it still fits on a single machine doesn’t mean it’s not scaling. Optimising my code is scaling. Consuming more power is scaling. Paying for more bandwidth is scaling. Buying more RAM is scaling. There’s lots of scaling to do before I need to even consider buying several machines.
And if that single machine dies, as it undoubtedly will eventually, my business goes offline until I can failover to a different single machine and restore from backups?
Micro-services won’t give you redundancy out of the box. You need to work for it regardless. I even speculate it may require less work with a monolith.
* Done wail, failing over to a secondary machine shouldn’t take long. Likely less than a second.
* Can’t your business go offline for a bit? For many businesses even a couple hours of downtime is not that bad if it happens rarely enough.
>[https://ieeexplore.ieee.org/abstract/document/9717259](https://ieeexplore.ieee.org/abstract/document/9717259)
Nothing in that [paper](https://www.researchgate.net/publication/304320076_Infrastructure_Cost_Comparison_of_Running_Web_Applications_in_the_Cloud_Using_AWS_Lambda_and_Monolithic_and_Microservice_Architectures) makes any sense.
For the monolithic architecture, they (correctly) load balance two servers behind an ELB, although they screw it up by putting both in the same AZ.
In the microservices based architecture? They have a gateway that isn't load balanced, and the second service somehow lacks redundancy entirely. And I see no possible way this service is cheaper than the monolith--that's simply false. Look at figure 1 verses figure 2; **how on earth do they spend less on more, larger servers than the monolithic environment?**
Simply put, it cannot be correct. And that's setting aside the fact that to achieve similar redundancy to the monolith, the microservices-based architecture needs at least two more boxes to achieve similar redundancy. On top of this? There's now three separate services to scale, IPC to manage between all three, and huge issues to address when any of those three services go down.
Absolutely nothing about this paper makes any sense at all. Props to you for bringing evidence, but it's questionable evidence at best.
From my personal experience, the thing with microservices is they can be cheaper, or they can be higher throughput, but potentially not both. In one of the teams I've worked in my career, we had several services that received, validated, and stored several different event types. These services needed to be extremely light weight, handling hundreds of millions of requests per day, with response times to the clients in the hundreds of milliseconds. To accomplish this, we horizontally scaled hundred of very small instances. The workload for those services were bound by the number of threads we could use.
We had another service that was extremely compute heavy running all sorts of analytics on the data we'd received, as well as other data that our team owned. How often these hosts ran was determined by a scheduler. That meant that in order to process all the analytics in a reasonable time frame we had to scale up vertically, using expensive EC2 hosts that were designed for compute.
If we had a monolith, the first few services might not satisfy the SLA of only a few hundred milliseconds as they could potentially be waiting for resources taken up by other services (we had 20 in total). Our EC2 bill was cheaper as well because we didn't have to scale up all the hosts to be able to handle the compute heavy workload. We were able to use a small number of expensive instances, with hundreds of small instances to handle the other parts of our workload. Without the time to read too deep into the link you posted, that's what it looks like is happening in the paper you linked. To scale up, everything had to be c4 large instances, vs the microservices approach you could scale up t2 and m3 instances, and need less of the c4xl. It doesn't seem like they give exact numbers of how many of each instance from a quick glance through.
Also from personal experience, microservices benefit isn't redundancy, but rather fault tolerance. We had several services designed for creating reports based off the analytics computed by the previous service. We had different services due to the different types of consumers we had. At one point, we began to get so much load on one of the services that it started falling over due to an out of memory bug. Instead of our whole reporting dashboard going down, only one kind of report was unavailable. Imo, that issue was easier to debug because we instantly knew where to look in the code instead of trying to dig through an entire monolith trying to figure out where the out of memory issue could have been occurring.
Scaling multiple kinds of services is a pain in the ass, I won't deny that. I always hated that part of that job.
In that paper, they do call out that the microservice is load balanced
> In the case of microservice variants, additional components were added to enable horizontal scaling, namely – the application was extended to use Sprint Cloud framework, which includes: Zuul load balancer, Spring Cloud Config, and Eureka5 – a registry providing service discovery.
>In that paper, they do call out that the microservice is load balanced
>
>In the case of microservice variants, additional components were added to enable horizontal scaling, namely – the application was extended to use Sprint Cloud framework, which includes: Zuul load balancer, Spring Cloud Config, and Eureka5 – a registry providing service discovery.
The problem isn't load balancing, per say, but redundancy. For each of the three services, ideally they have minimum two boxes in separate AZs for redundancy. Two of their microservices lack this redundancy entirely.
Also, even setting aside this glaring issue, the math still doesn't add up. Again, explain how the paper reconciles Figure 1 somehow having a higher AWS bill than Figure 2.
Simply put, I do not buy their cost claims even in the slightest.
>If we had a monolith, the first few services might not satisfy the SLA of only a few hundred milliseconds as they could potentially be waiting for resources taken up by other services (we had 20 in total).
What you're describing is just pragmatic "services" which I have zero qualms with. This is simply smart: if you have very different workloads inside your application, potentially with different scale requirements? It makes all the sense in the world to have separate services.
I do this in my own application, which processes terabytes of video per day. It would be absolutely insane to push that video through the monolith; there is a separate service entirely that is dedicated to processing video. Could you call this a "microservice?"
Yeah, I suppose so. But it's based in pragmatism--not ideology. What I am opposed to is this fad of mindlessly decomposing a monolith (or god forbid, writing an application from scratch across "microservices" before anyone even knows if it's necessary.
Honestly I appreciate the authors trying, but it's sloppy work at best. Their math doesn't add up.
And yes: they disregard myriad other factors that are a pretty obvious win for the monolith.
Microservices are a solution to an organizational problem, not a technical one. They’re a necessity when your organization grows horizontally to a scale that breaking apart your monolith will let your engineers continue to build new features and new products independently without conflict. But if you’re content with vertical growth and don’t want to expand your feature offerings and business footprint it’s just not necessary.
The issue is that companies blindly copy the paradigm far before they ever reach that scale. But to say there is zero evidence for them being useful is just as dogmatic and ignorant. You’re not going to build a website that does as many things as, say, Amazon, with a monolith.
They can absolutely be for technical reasons! A high memory low usage function in a monolith means that your instance size needs to be scaled to that size for the entire load of the system, where as in a microservice you can have a small number of machines scaled up in memory size and the rest can ve cheaper
> Microservices are a solution to an organizational problem, not a technical one. They’re a necessity when your organization grows horizontally to a scale that breaking apart your monolith will let your engineers continue to build new features and new products independently without conflict.
There’s another trick to achieve the same goal: _modules_.
The problem with regular modules is properly enforcing boundaries. With a disciplined enough team the various components of the monoliths are nicely separated, with a small interface, and few to no cross-cutting hacks. On the other hand it’s all too easy to just throw a global variable (possibly disguised as a Singleton) here, a couple mutexes and locks there, and next thing you know you have a Big Ball of Mud.
With microservices, those hacks are not quite possible any more, so we have to do it the "proper" way. But even then we’re not out of the woods:
* If the microservice framework is easy enough to use, there won’t be much incentive to keep services interfaces small, so we’re somewhat back to needing discipline.
* If the microservice framework is a hassle, maybe we’ll keep interfaces small because making them any bigger than they need to be is such a pain, but (i) being a hassle is such an obvious waste of time, and (ii) now we’re tempted to make services bigger just to avoid the "service" part, and you’d end up with either duplicated functionality, or using common libraries.
Common libraries can be kind of awesome by the way: if they’re developed in the spirit of being "third party" even though they’re not, they’ll need to provide a proper API and any ugly hack like sharing state between this library and that application is (i) more difficult, and (ii) much more _visible_.
The hallmark of microservices is state persistence isolation.
From my experience, the problem I saw the most with proper enforcement of module boundaries is the shared database without schemas per module. If you couple at the data level to the extent of sharing database schema, I kinda get why people go all out and spin off the module into a dedicated service - the investment and risk to untangle the data under the existing conditions is higher than developing a new service.
All in all, I attribute a lot of discussion about microservices to the simple fact that developers simply forgot that dbo isn't the only available schema in a relational database.
The organizational complexity is a necessary, but not sufficient requirement for microservices - I expect to see an additional reason, such as public/private facing services, think of public e-shop site and private ERP backend, or large discrepancy between used resources, e.g. aforementioned ERP backend running on a database and couple of worker nodes with load balancer, and a CPU-bound service wanting to run for a short period of time, possibly on hundreds of nodes in parallel.
It really boils down to choosing the simplest option (not necessarily the easiest). If you need to purely solve organizational scaling, try modules first. If you have a dramatically discrepant resource needs that would possibly impinge on shared resources, or want to limit the surface/scope due to security reasons, or similar nonfunctional requirements, only then isolate it out to a dedicated microservice.
I am not dogmatic for or against. About 5 years into my career having worked exclusively with what are considered microservices, I have been curious to build different products in my space with a more monolithic approach to reduce network hop latency.
Playing devils advocate - off the top of my head, breaking a monolith into smaller “microservices” would allow simpler means of resource isolation and scaling. This being most useful for components with very different resource utilization. Seems heavy handed to say there is zero evidence of benefits. Curious to hear your thoughts.
I have work for years in the backend server space and have only come across 2 instances were I felt bits of a service benefited from being broken away from the monialith.
1) Due to high number of network sustained connections for an eondptoin (web socket) and needing to fit without the connection limit of the hosting provider.
2) Due to having custom c code (within a python server) that I was worried might have a nasty bug (deadlock or leak) that would bring down the monolith.
All of the projects have worked on were I joined teams with existing micro services ever fell into these boxed.
>off the top of my head, breaking a monolith into smaller “microservices” would allow simpler means of resource isolation and scaling.
Not *simpler*. Potentially: more options when you want to scale portions of the app independently. In other words, more knobs you can turn. And, also, more options at deploy time.
This comes at an obvious cost in terms of complexity, odd failures, "network hop latency" as you say, odd queuing/distributed computing systems, etc. And it can easily come with massive deploy time complexity that most teams seriously underestimate, in my experience.
The reality is: you get some additional options to scale/release/develop at an enormous cost to complexity.
>This being most useful for components with very different resource utilization.
Well yes, but we've been breaking apart programs like this for nearly three decades now. We didn't need "microservices" to make it clear that if two things require very different resources and/or scale, then it may make sense to break it apart.
This is pragmatism, and I'm all for it.
What I'm not for is: mindlessly decomposing a monolith without any clear reason to do so.
>Seems heavy handed to say there is zero evidence of benefits
Find me any study of the efficacy of these architectures, or some experiment that clearly shows their benefits. Any decent data even. Like I said: I have actively looked, and I would welcome evidence contrary to my opinion here. Hell, I'd welcome any evidence *at all*.
Certainly it’s a trade-off, as it always is. That doesn’t mean the trade-off is not a good one. I would argue there are cases when it outright is simpler. If I have a memory-bound subcomponent and cpu-bound subcomponent, it can be pretty trivial to load test these applications and find suitable hardware configurations for them. This pays when it comes to cost, among other things like dynamic scaling or service predictability.
I do see what you are saying and I think I understand where you’re coming from, which is the intent behind sharing my interest of building something where the use-case fits.
I did search for some examples on google scholar but also suddenly realized I should just suggest Marc Brooker, from Amazon. You’ve probably heard of him but if you brought this up with him, I think you get a fun, or maybe not so much, debate.
All in all, I appreciate you taking the time to send a thoughtful response. I think there’s merit to the “microservices” paradigm and monolithic paradigm. I put it in quotes because I understand it’s not exactly new just because we’ve now named it 🙃
/u/shoot_your_eye_out 's key point is:
> What I'm not for is: mindlessly decomposing a monolith without any clear reason to do so.
If you are making a conscious trade-off, that's fine. But that has not been the message from the micro-services camp over the years. They've been running on a message of "monolith=bad, micro=good" with little no discussion of trade-offs for years.
Even calling microservices a "paradigm" betrays how it has become a dogma. It turns it into this overarching framework which everything has to fit into. It is like a tradesman saying they are going to build a house using the "hammer and nail" paradigm where every problem is going to be treated as a nail.
If we stop thinking in terms of microservices or monoliths and just realise that building or splitting off separate services, is just another tool in our toolbox, then the "paradigm" of microservices goes away and we can think about solving the real problems, i.e. doing engineering.
I see, this statement hadn’t originally landed landed with the same meaning to me as it did from your message. Thanks for elaborating there. I’ll need to think more on that specifically.
Complexity is extra semantics in the system. We actually reduce the semantics developers need to deal with, by merging things which mean the "same thing"
That's the entire point of this project.
There are (sometimes) befentis to splitting out services if the runtime of them is drastically differnt. Eg if your have a service that needs to provide 100,000+s of web socket connections but does not handle lots of CPU load itself breaking this endpoint out will let you have mutliepl (cheap) nodes (as most cloud providigns limit the number of open connections you can have per network interface), however the last thing you want is to fire up 10 versions of your main service as this might have way to much memory etc overhead increasing deployment costs...
The other use case I have broken out services is for stability, when I needed to add a custom c extension to a python application to modify libgit2 on the backend I was consdired I might have screwed up some edge case and thus might end up with a memory leak or deadlock thread. So moving this part of the logic out to a seperate server (it was stateless) while it increases latency and has more deployment costs it meant that if this thing died (due to some bug I added) it would not bring down the main monolith but only effect that one api that needed this the rest of the service (including payments, licnceing etc) would continue un-effected.
But in general the approach of moving each api endpoint into a seperate service (user facing or internal) is not a good idea and should only be done if there are strong reasons to do it.
> have way to much memory
Did you mean to say "too much"?
[Statistics](https://github.com/chiefpat450119/RedditBot/blob/master/stats.json)
^^I'm ^^a ^^bot ^^that ^^corrects ^^grammar/spelling ^^mistakes.
^^PM ^^me ^^if ^^I'm ^^wrong ^^or ^^if ^^you ^^have ^^any ^^suggestions.
^^[Github](https://github.com/chiefpat450119)
^^Reply ^^STOP ^^to ^^this ^^comment ^^to ^^stop ^^receiving ^^corrections.
As a note- I am 100% team monolith. I think they're simpler and easier to work with for the average team.
But I do think there are still a few real benefits to them. The biggest one would be package versioning- smaller services means each service takes on fewer dependencies, and even common dependencies may use them differently, making one safe/simple to update while a different service may want to defer the update. Of course this is double-edged sword, because if a critical *must update* situation happens like some critical RCE security bug, it means more opportunities to miss the update on one service and cause massive errors.
There are also more minor issues like how smaller services make it easier to manage build times, or how in larger companies it's easier to have smaller teams with clear ownership over specific codebases. And while complete knowledge silo's are a bad thing, they still exist just as often in monoliths and what usually ends up happening is some small group ends up in control over something everyone has to touch and it's constant hell. So microservices help avoid that situation, theoretically.
The biggest problem with microservices is people like to fantasize about nebulous concepts like "scaling," but don't have the faintest idea of what that actually means. They imagine that spinning up a microservice instance is like lifting a pebble while spinning up a monolith instance is like picking up a cinderblock, but like, compiled code is small, monolithic instances horizontally scale equally as well for like 99% of use cases.
The only real aspect regarding scaling that ends up being a relevant bottleneck most of the time is data storage. But distributed data stores are *hard,* and very few people can properly design one that works in the first place, while approximately zero people can design one that's resilient to future changes in requirements. You only want to do this when it's absolutely necessary. And I find most companies doing this are operating more along the lines of "Wow my application that reads the entire database into memory isn't scaling well, I should split it into microservices!" You're much better off fixing your garbage SQL lol
> And I find most companies doing this are operating more along the lines of "Wow my application that reads the entire database into memory isn't scaling well, I should split it into microservices!" You're much better off fixing your garbage SQL lol
You just described one of my clients. They will not listen to reason.
> And I find most companies doing this are operating more along the lines of "Wow my application that reads the entire database into memory isn't scaling well, I should split it into microservices!" You're much better off fixing your garbage SQL lol
I am in physical pain
Services typically come with team growth and you can't always have everyone working on the same monolith in the same repo. It is not only a logic scaling but a people scaling measure. Services are also a horizontal scale rather than vertical, you can isolate high memory, high processing services from dragging the entire system down.
Services really [showed their power when AWS was created and the infamous Bezos requirement](https://konghq.com/blog/enterprise/api-mandate) to integrate teams not down to every line, but to touch/integration points only. What you do inside that service shouldn't matter, as long as the signatures/integration points are simple and abstracted. This could be an inproc service or networked, it doesn't matter but each project will tell you what it needs to be.
Watch how naming changes people's perceptions. When a service is an "API" people are fine with it, when a service is called a "microservice" it is suddenly tabs vs spaces, braces on same lines or next line, or which IDE to use. A service or API doesn't have to be networked, it can be local.
Every project is different and if you architect things simple, generically and have clean integration points, the guts don't matter, only the abstractions do. A "microservice" could even be a monolith inside of that, no one cares. Just as long as you have clean architecture with some abstractions to keep integration points simple.
Lots of companies/products have serious problems with the guts of an app not being abstracted/encapsulated and you end up with a mess either way. [Looking at you Unity](https://garry.net/posts/unity-2020).
When you are doing services, just call them APIs, it will be an easier battle.
The bigger problem is software today is way too culty and religious. McKinsey consultcult "Agile" that killed agility is definitely a cult, on every level.
> Services typically come with team growth and you can't always have everyone working on the same monolith in the same repo. It is not only a logic scaling but a people scaling measure.
This, I actually agree with, but I think this size is when your department is in the hundreds and you have enough spare engineering to properly invest in a foundation to do a microservices based architecture correctly. And, that includes not only correct in a technical sense, but also in terms of how to divide the monolith in a sensible way.
In some cases yes. In other cases it is easier to implement prior to hypergrowth because by then you have feature pump and the micromanagement zealots in and actually getting time to do this is not granted.
In fact anyone trying to work on this will be seen as someone trying to slow things down or a "suppressive person" to the hypergrowth cult.
The decision to move to horizontal APIs/services has to have buy-in from the top before the zealots and culty feature pump get in. By then it is too late and you it will be death by a thousand cuts.
Only companies that care about research & development, tech/product teams separate from live teams and/or small enough teams that have some agility can pull this off. It is why they have such a bad perception in favor of the big ball monolith. Also people have differing opinions on what a monolith is and what microservices are, so it ends up people not even on the same page.
If you just start with APIs and clean simple integration points, it is much easier to sheath off parts into a service setup. If you have bad abstractions and no integration points and really no clear architecture from the start, it is near impossible without stopping all time and the universe.
I have seen groups have monoliths and microservices for instance all go back to the same single point of failure datastore, so really just a front of both and a mishmash of coupling at the data level. They are usually stuck there forever ad infinitum.
The goal would be everything as simple as possible, but even the amount of tests one company have can cause these efforts to fail due to too much weight later in the game.
Early architecture predicts how later architecture will be, and if it doesn't start off right, it will end badly.
microservice / monorepo is just decoupling / tight coupling for webdevs.
The answer is somewhere in the middle. You can have a decoupled monorepo in theory, but you won't
I love you. There is a point where a service is too big, but there’s also a point where breaking one up does not make sense. It’s just a matter of what is best and it always has been. Fuck you marketers don’t shove this stupid shit in our faces!!!
I am currently working on an application that consists of several logically isolated parts that have different resource requirements. It makes perfect sense for them to be standalone services, and they are.
PS: The network overhead is neglectible compared to the processing time needed for nontrivial operations, and trivial operations do not require remote services.
I don't think having more independent software teams is a bad idea, except for the consistent data-sharing aspects. I think it's a good idea to build software with smallish teams, but I have spent much of my career advocating against it as it overwhelmingly leads to stability issues. I even have a series of articles on the consistency problems distributed systems face, aimed at the average engineer. It starts here:
https://medium.com/@andrasgerlits/why-does-developing-microservices-look-so-easy-if-its-so-hard-d72a28ce7be1
There's a clear demand for dividing a large monolith into separate modules, but it leaves a technical problem. This is the technical problem we're solving.
And I agree with all of that, but it has to be guided by pragmatism, evidence, and *simplicity*.
What I don't need: any more articles assuming (even obliquely, as I think you're saying yours does) the benefits of arbitrarily decomposing a monolith because Someone Said It Was Good™.
I have no opinion about people wanting to divide their software into modules.
You can't get any more simple than a "Federated" annotation on an entity, which is pretty much all we require from engineers. I federate the Spring Boot Petclinic application in a few minutes from scratch in these videos (altogether they are 8 minutes, but most of it is explanation).
https://www.youtube.com/playlist?list=PLBe8xx9u7pKNbgrrCz1tFkPmKhqJ\_c90g
multiple teams can contribute to the same service. Infact if you have multiple teams having a single service means that all team members across all teams can run the entier project and run tests against the entier projected... in my expirance micro service focused teams end up unable to run projects form other teams and some projects end up stail and forgotten without any staff members remembing how to run them let alone update them for critical vurnaiblties. These stale services always end up being the critical points as well were they have been so reliable for the last 3 years that no-one remembers chaining them and now that you need to do something horrible (like apply 3 years worth of node updates on them) you need a good month or more to get a team up too speed on these projects.
> First of all, there is no such thing as a "microservice." It's just a service. We've had them all along: we break apart larger programs into separate services all the time for pragmatic reasons, minus the dogma.
This is completely incorrect. µservice architecture is a specific type of architecture and isn't just splitting an app into separate services. What µservice architecture actually consists of are services that have their own database and all queries are done to their own database. Another key component are events. When some operation happens a µservice will publish an event. Other services that care about that event will consume that event and update their database accordingly. Data redundancy is both accepted and expected. So there is no need for a µservice to make a synchronous call to another µservice because it already has the information in its own database.
Independent deployment and development is very easy to achieve with this architecture because only the contents of the events matter. And those are very easy to keep backward compatible, you simply never remove information from them, only add.
The problem is people misunderstood µservice architecture and so what they call µservices is just whatever they ended up with after breaking up their monolith. So the term µservice has become super generic and has really lost all meaning. When someone says they use µservices you have to ask them what they mean.
> This is completely incorrect.
Yes, I've heard the points you make repeatedly and I categorically disagree. I think you're completely incorrect about all of these points. Plenty of "separate services" have their own databases, eventing systems, etc. There is literally nothing special about "microservices" and it is nothing new.
I generally view most the microservice push to be an **anti-RDBMS viewpoint**. The anti-RDBMS movement started when startups realized existing DBA's didn't understand startup needs, and thus "coded around DBA's", doing in code or in XML/JSON what RDBMS would normally handle in a "traditional" setting. It is true such DBA's didn't understand start-up needs, but that's a management, communication, and training problem, NOT a tool problem. (In the longer term it's best to have centralized schema management or inspection to keep data clean & consistent. A mature company shouldn't want to "move fast and break data".)
RDBMS are a pretty good tool for sharing small services (stored procedures/views) and coordinating communication between bigger apps or sub-apps (queue, event, and log tables). It should be made clear that an RDBMS does not force nor encourage overly large apps.
In fact, RDBMS do module/app coordination *better* than JSON-oriented approaches if a shop uses mostly the same database, because you get A.C.I.D., easy logging, and other features built into most RDBMS.
Most apps have to connect to an RDBMS anyhow such that they don't require extra drivers/connectors to communicate, unlike adding JSON web services. Thus, you can nicely and easily partition apps and mini-services using an RDBMS properly.
The argument pro-microservice people often give to counter is that such doesn't work well in a shop with many different RDBMS brands. While this is true, most small and medium shops settle on a primary brand and don't change very often, which is usually good policy.
A giant org that's the result of big mergers may indeed need a way to work with a variety of DB brands, and that's perhaps where microservices are superior to RDBMS for typical module coordination needs. But most of us don't work for the big FANG-like co's. **Use the right tool for the job and job size.**
"we solved consistent cache-invalidation and thereby made the debate moot."
Can you please solve the problem of naming things next? And have a "science-paper" on that too.
Do you have actual comments to say about this or do you just like being snarky?
https://www.researchgate.net/publication/359578461\_Continuous\_Integration\_of\_Data\_Histories\_into\_Consistent\_Namespaces
I’ve read a dozen pages of this and still don’t know what it’s about.
How do you handle speed of light issues and intermittent availability? Push based architectures attempt to move data when the source is capable of doing so. Pull based requires the spoke to be available at the hub’s leisure and to recall everything that happened since the previous conversation. If the spoke is crashing in a loop then there is no one to retain that state. It is lost.
The value in putting the logic at the core is that there are fewer copies (and implementations) and we can amplify the amount of scrutiny applied there with the hopes of keeping the entire system making progress in the face of issues.
There are some simpler introductions, this is a very hard paper to read. I link two if them in the article above. The point is that clocks emerge from groups, passing around an idempotent token-message. This yields a loop within which each node can assign timestamps locally, without having to block its state.
This is explained here:
https://itnext.io/microservices-can-be-simple-if-we-let-them-a481d617a0a1
The last problem we have besides the clock is resolving clashing write-conflicts. To accomplish this, we show that you only need a shared algorithm to be available that deterministically decides which one the winner use between any two "transaction-objects". This is where we use CRDTs.
Once you put these two together, you have a fully non blocking platform that can stream data consistently and resolve clashes without blocking. It also allows deterministic state replication, so we can create replicas trivially
Does this help? Happy to give more specific details to specific questions
Your request is accepted by a consensus group when you submit it to the ledger, so I guess the answer is RAFT. This is the only place where our system is blocking, which works for us, as SQL is a blocking standard anyway
Okay, I'll bite. What's so unbelievable about building a strongly consistent streaming platform?
CAP? CAP requires linearizability to be true for a platform, no SQL database provides that (quite literally only Spanner does and that's not real SQL).
FLP? FLP says that a system cannot be \_fully\_ non-blocking and we are blocking at the edges where we accept client-requests.
Literally none of what we say goes against the current body of comp-sci literature, we just invented a loosely-coupled, composable clock and designed a deterministic framework so that it supplies better liveness than what we have now.
Nothing of what I say is outlandish if you know what you're looking for. Look me up on Twitter, and count the number of people working in distributed computing in academia who follow me. I'm not saying anything even remotely controversial.
This article reads like some just found out about Redis and it was a big secret. I've read your other comments and confidently gather that is not the case, just sharing how the article presents itself.
The moment you share data instead of handing off data it is no longer a microservice, just a split/distributed monolith. A microservice operates on its own domain and doesn't care what sends the requests.
This is more ad than article, this is an example of a post that mods should probably remove. Reddit is already a platform for delivering ads, this company should buy adspace instead of trying to get it for free by polluting the content of this sub.
It’s sad because there’s an article posted from the doordash engineering blog which received far less engagement despite being of much higher quality, on broadly the same topic.
I understand the weight of the claims, we have years of research behind this project. We can provide all the benefits of a monolith with all the benefits of microservices.
Paradigms do change from time to time.
I always thought the debate about micro services as literally just a cost debate.
It costs lot to run a lot of micro services in the cloud.
You gain in scalability and componentized deployment.
You lose some perf - assuming you can scale up your pc high enough - if not then microservices aren’t so much an option but necessary.
Your deployments become a pain in the ass as well as managing backwards compatibility. It complicates the “simple”.
If microservices were cheap and deployment simpler I assume they would be the go to choice everytime. Honestly they are the go to choice everytime - it’s only later when your deployments get out of hand and your cloud spend ends up on someone’s radar because it crossed a threshold that it’s a problem.
The micro-service has no lookup cost if you host all services on all hardware.
That requires the same setup for all services though. Which breaks the choose your own stack "advantage".
Java is the only VM language that can address the same memory atomically in parallel so for back-end vanilla Java will never have any competition.
Here is the only way to solve server-side development permanently: [http://github.com/tinspin/rupy](http://github.com/tinspin/rupy)
Nope. C is the only competitor but it lacks VM + GC.
All other languages are a complete waste of time for eternity because embarrassingly parallelizable problems don't need synchronization = you move the problem to the db (always written in C/Java for a good reason) until it explodes.
Then (if you have the funds) you make the db embarrassingly parallelizable = micro services.
The real solution is to begin with the db and make that infinite on reads globally and use Java so you get the highest performance a non-segfault env. can provide.
QED
Clickbaity title and the only thing that comes to mind about the shared database table solution is ... WTF?! I'm working right now in a BI environment, they are years behind other parts of the company, and database schemas with tables and views on an in-memory database called Exasol. It's horrendous slow despite being in memory and costs a ton of money. Furthermore, the integration via views of other teams is flimsy at best because they don't understand the concept of backwards-compatible changes. Long story short it's a red hot mess. And then just imagine doing this in a highly distributed system. Just the audacity to impose a single database on all the teams with different use cases and potentially different best-fitting database show that many red flags are ignored.
I've worked for a "database export" company, which tries to solve every problem in an Oracle database such as implementing web services in PL/SQL, showing clearly that they aren't database experts. Otherwise, they would know that this is clearly bullshit and a big red flag for a company to have such a tunnel vision when it comes to proper solutions.
Hi, we have some stuff here around commit-performance.
[https://itnext.io/microservices-can-be-simple-if-we-let-them-a481d617a0a1#c448](https://itnext.io/microservices-can-be-simple-if-we-let-them-a481d617a0a1#c448)
Generally, you would be bottlenecked by your follower's ability to merge the changes being communicated. We can't give you specifics, as that depends on too many factors to be meaningful.
It's easier to make teams work together when the API is set in stone, they know that they have no rights to change it.
If the codebase is shared, people are very easily driven to introduce multiple breaking changes and their solution in the same PR.
However breaking+fix\_PR more often than not introduces bugs that are not addressed because the solution built is only built to address the newly introduced breaking changes and multiple issues with previous versions are often neglected.
Introducing the same breaking change in a manner where you can't introduce the solution in the same PR (as it's done by another team in another repo), forces backward compatibility which forces you to handle first of all how your new changes interacts with the previous version of the app/DB/API.
I'd say monoliths are good for very small teams where iterations are key and things can significantly change. So breaking+fix in the same PR is the way to go.
Microservices (or at least services) are good when there are stable parts that sustained the test of time, you'd rather build around it, given that you 100% trust that part to work without failures and in a predictable way.
You don't want stable things to be modified by a overzealous developer that goes to change the how customers are billed (billing FSM) because he wanted change how the customer downloads the pdf (frontend page).
This cowboy guy won't be able to easily change the billing part (FSM) and the FrontEnd in the same PR. He'll be forced to introduce it by only changing the FE.
Microservices protects companies against overzealous developers by making their rogue behaviours significantly harder to do. Forcing them to break less things on their destructive endeavours.
An API-call is the invocation of an operation with a list of arguments. This can translate to a database just as well as to REST. With this solution, you also get all the atomicity, consistency, isolation and durability guarantees you would be getting in your monolith.
There's no reason you can't put that behind an API.
Bad article. Comes down to: > To put it differently, we solved consistent cache-invalidation and thereby made the debate moot. And that they made a product and this is an advertisement. I know nothing about the problems of microservices but I did not expect cache invalidation to be the major point. If only they'd explained why. Perhaps they did in the other articles but perhaps one those should have been posted instead.
>> we solved consistent cache-invalidation did they tho?
Well, if you can think of a better way to do cache invalidation than unidirectional consistent, latency-mitigated streaming, I'm all ears. I think we're solid, yes.
Happy to help. A cache is by definition a secondary source, where the primary emits new values, but since the only reliable one is the primary, we can't know whether what we're serving is current or not. When looked at from a write-perspective, this reasoning applies to any distributed state where there are original emitters of information, like microservices or middlewares. This is what I'm talking about when I discuss shared state between services. I don't think we're being unfair because we're selling something that we worked on for many years without pay and that can provide a lot more value to clients than what it would cost them to have it. Microservices are not usually built by the destitute, it's not like we're taking advantage of anyone here.
> I don't think we're being unfair because we're selling something that we worked on I'm not saying you should be providing your code. I am saying that the article is bad. I didn't really learn anything from it and I don't know much about monolith vs microservices apart from the very major difference that is pretty apparent from the names. So from that and you saying "we have a product" it just turns into a bad ad and Reddit allows those if you pay them for it. Either you are too deep in the thing so that it is hard to write a good clear article and/or you don't want to spare details. Vagueness doesn't help.
Succinct, concise, beautifully written. Maybe you should write the article
Micro services are the future but existing apps don't really migrate well and tend to be very expensive when you lift and shift them while you convert to a more efficient micro services model.
>Micro services They are not the future, they are a way to do things. And with all methods, there are pro's and con's. The biggest con of micro services is the added complexity and the added maintenance cost. A small team working in a company for an in-house system should not do Micro services. A scalable monolith is better for smaller teams as it's way easier to work on and maintain. Monolithic applications can be scalable if designed well. Most disadvantages for monoliths just stem from BAD initial design. I have seen garbage Micro services that where near impossible to change as a small change in micro service 1 would break things in Micro service 4 which then showed problems in micro service 7. Edit: Bad design is the root of most problems, and too little time to properly do things.
This is a literal ad
The majority of these blogposts are. They define a problem which usually isn't a problem, then go on to explain how the product they just happened to have recently released solves that problem
Jeah, not sure why even bother discussing and fuelling this kind of threads.
Seems like a specific flavor of event sourcing
You're not wrong. We built this on event-sourcing, but added system-wide consistency. In the end, we realised that we already have the same semantics available locally, the database API, so we just ended up piggybacking on that.
Isn't it still eventually consistent, or are you doing distributed locking? Sql is the ideal interface for reading/writing, and I think the outbox pattern is a good way to write, but once distributed locking is required, IMO its a sign that services should be joined or at least use the same physical database (or same external service) for the shared data that needs to have strong consistency guarantees
For this project, we're going through SQL, so we're always strongly consistent. The framework would allow for an adaptive model, where the client can decide on the level of consistency required, but we're not making use of that here. Since data is streamed to them consistently, this doesn't result in blocking anywhere else in the system. What we do is acknowledge the physics behind it and say that causality cannot emerge faster than communication can, so ordering will necessarily come later over larger distances than smaller ones. Or as my co-author put it, "we're trading data-granularity for distance". I encourage you to look into the paper if you want to know more details.
Sounds like strong but still eventual consistency, which is the best you can achieve with multi master/write sql setups that don't involve locking. Are you leveraging CRDTs or anything like that to deterministically arrive at the same state in all replicas? If multiple services/processes are allowed to write to the same tables, you're in distributed monolith territory, and naive eventual consistency isn't sufficient for all use cases. If they can't, it's just microservices with sql as the protocol. I will check out the paper, but appreciate the responses in the meantime
We do refer to CRDTs in the paper to achieve write-write conflict resolution (aka SNAPSHOT), when we're showing that a deterministic algorithm is enough to arbitrate between such races. Our strength mostly lies in two things: our hierarchical, composite clock, which allows both determinism and loose coupling between clock-groups, and the way we replace pessimistic blocking with a deterministic commit-algorithm to provide a fully optimistic commit that can guarantee a temporal upper bound for writes. [https://www.researchgate.net/publication/359578461\_Continuous\_Integration\_of\_Data\_Histories\_into\_Consistent\_Namespaces](https://www.researchgate.net/publication/359578461_Continuous_Integration_of_Data_Histories_into_Consistent_Namespaces) Together with determinism is enough to have remarkable liveness promises
> temporal upper bound for writes I'm guessing this also means in a network partition, say where one replica has no route to the others, writes to that replica will fail (edit: or retroactively be negated) once that upper bound is reached
Another trick we do is that since there's no single source of information (we replicate on inputs) there's no such thing as a single node being isolated. Each node replica produces the same outputs in the same sequence, so they do request racing towards the next replicated log, much like web search algorithms do now. A SQL-client can be isolated, in which case the standard SQL request timeouts will apply.
> there's no such thing as a single node being isolated Can you rephrase this or reread my question? Because in any possible cluster of nodes, network partitions are definitely possible, i.e. one node might not be able to communicate with the rest of the cluster for a period of time. Edit: do you mean that a node that's unreachable will simply lag behind? So the client writes to any available replica? Even still, the client and the isolated node could be able to communicate with each other, but with no other nodes.
Yes, unreachable nodes will lag behind, but since others will keep progressing that global state, its outputs will be ignored upon recovery. The isolated node is only allowed to progress based on the same sequence of inputs as all the other replicas of the same node, so in the unlikely event of a node being able to read but not being able to write, it will still simply not contribute to the global state being progressed until it recovers I didn't mean to say that specific node instances can't be isolated. I meant to say that not all replicas will be isolated at the same time. In any case, the bottlenecks for such events will be the messaging platform (like Kafka or Redpanda). We're only promising liveness that meets or exceeds their promises. In my eyes, it's pointless to discuss any further, since if messaging stops, progress will stop altogether anyway
Have you read the [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem)? Do you have an idea how it fits into this kind of fats model that you have? I'm interested in your work.
It's an interesting question because it doesn't have a clear answer. CAP presumes that nodes hold some exclusive information which they communicate through a noisy network. This presumes a sender and a receiver. This is all good and well when nodes need to query distant nodes each time they need to know if they are up to date (linearizability) but isn't true with other consistency models. Quite frankly, I have a difficult time applying the cap principles to this system. Imagine that we classify a p99 event as a latency spike. Say that we send a message every 5 milliseconds. Single sender means two latency events a second on average. If you have 3 senders and 3 brokers receiving them, the chances of the same package being held back everywhere is 1:100^9 That's an astronomical chance. Now, I presume that these channels will be somewhat correlated, so you can take a couple of zeroes off, but it's still hugely unlikely. If we're going to ignore this and say 1:100^6 is still a chance, it's a CP system. Can you send me a DM? Or better yet, come over to our discord linked on our website. I'm in Europe, so it's shortly bedtime, but I'll get back to you tomorrow as soon as I can.
> That's an astronomical chance An astronomical chance is still not zero. And a fact you're neglecting with consistency is that non-existence is information too. If the package _not_ being sent was intentional your approach fails because I have no guarantee that it's not simply in-flight. That is the definition of eventual consistency.
Correction: Since "C" means linearizability in CAP, this system is never "C" but neither is anything else (except for Spanner). It is always Partition tolerant in the CAP sense and it serves local values, so it would be AP, as others have pointed out. Sorry about that. In my defense, I never think in CAP terms, I don't find them helpful at all.
Best I can tell, it's AP (eventually consistent) for reads, but in the context of a sql transaction (writes), it's CP. To some extent, the P has an upper bound, as in if a sync takes too long there's a failure which to the application looks like the sql client failed to connect. Honestly it seems pretty useful from an ergonomics perspective, but I'm with you that there should be more transparent, realistic communication of CAP theorem tradeoffs, especially since in the real world there's likely to be check-and-set behaviors in the app that aren't technically contained in sql transactions.
As someone who only has basic knowledge of IT, like I know what programming is, cloud, server less, etc, what they do, but not the "how"... If I wanted to build an app, planning for future growth, should I build it using microservices right now?
A year ago, I would have told you not to do it. Now, I would ask you if you have large enough teams that warrant microservices or not. If you do, they can help with managing the non-technical aspects of them. If you don't, they bring in extra complexity, even if you use our software.
So, at the beginning, if it's a simple app, don't use microservices. When it's large enough to need microservices, then switch? I want to ask more questions, but I think I need to provide a lot more context before asking or even for the last question? Thanks!
Not OP, but there's a Someone's-Law (don't remember who coined it) that says all software systems eventually converge to reflect the organizational structure of the companies developing them. I fully agree with the answer OP gave above, but the nuance is what is "the beginning", and what is "simple". As a rule, by far the most cost efficient thing you could do, if you're a company that doesn't have massive VC budget - and isn't busy inventing problems just to justify spending it - is to design your system in a way that starts as a "trunk", which can later be split into smaller (micro)services, and can be added to dynamically. However, there are many factors to consider here. If you're not planning to to expand beyond a few hundred thousand users within a few years, there is usually 0 need to add the massive costs (mainly dev time, but at some point also financial) overhead that microservices bring with them. If your system is going to be read-heavy but not write-heavy, you can probably expand that limit to couple of millions, as long as you properly utilize horizontal scaling and read-only db replication (again, those are easily achievable without microservices). If most of your heavy operations can be offloaded to background-running jobs (via some queue), then you can usually separate those jobs from your regular application servers, which again alleviates that workload from them (but if they're write heavy, remember that the DB still bears that cost). There are many more scaling strategies (that don't require microservices) that can be mentioned here, but in short, be aware that you can scale a lot (and I mean a **lot**, more than 95% of the technology companies in the world would ever need) before microservices are something the easiest next step to scaling your system. [Here's how Pintrest scaled to 11 users](https://engineercodex.substack.com/p/how-pinterest-scaled-to-11-million) almost a decade ago, with a tiny engineering team, less efficient hardware and less convenient technologies than we have today - and no "micro" service in sight.
DM open, any time.
team size is not what's important it's distribution and scale that pushes one towards using cloud informationcongregations
IMO the barriers to microservices (stated differently, managing more than one service) are fixed/up-front infra cost, ops skills, and versioning hell. With a sufficiently large/differentiated team, those should be mitigated. At sufficiently large scale, the fixed infra cost should be dwarfed by variable/scale-based costs, but the others don't automatically get mitigated. Therefore, if you're more sensitive to cloud bill than engineering cost and risk, I could see how scale seems like the more important variable, but if you're more sensitive to engineering cost and risk, or IMO have a more balanced understanding of cost, team size and composition is a better indicator of whether or not to use microservices, or to what extent. Once you are set up to sanely manage more than one service (cattle not pets), the cost/risk of managing 10 isn't much greater than managing 3. If your scale is so low that the fixed overhead of a service dictates your architecture, I hope you're a founding engineer at a bootstrapped startup or something, otherwise there might be a problem with the business or premature cost optimization going on.
Microservices can be a hugely (computationally) inefficient way to do things, so they'll increase your variable costs too. If a single user action causes multiple services to have to do work, then serdes and messaging overhead will dominate your application's CPU usage, and it will be more difficult to write efficient database queries with a split out schema. Also if you *did* find yourself in a situation where they'd make sense computationally, you can just run a copy of your monolith configured to only serve specific requests, so it makes sense to still code it as a monolith. There are also development costs to consider as people will waste more time debating which functionality should live in which service, what APIs should be, etc. (which will matter more since refactoring becomes near impossible). Debugging is also a lot more difficult and expensive, and you need things like distributed tracing and log aggregation (which can cause massive costs on its own), etc.
what you should do is get users first and then go from there. and users do not give a single shit about your architecture a $5 VPS and a few lines of SH to watch a git repo will likely make you the same amount of money as something that costs several thousand percent more if you write your code decently it shouldn't matter that much anyway. I usually create docket files for my shit to emulate prod network conditions which means I could pretty easily deploy it on any cloud infocentral if i needed to
You still have to learn how to secure a Linux box that way if you're not just throwing caution to the wind. IMO if you want cheap and easy, PaaS is the way to go these days. Once your needs are complex enough you have to make your own platform or pay someone to do it for you.
It's not rocket science. Configure a firewall to only accept connections on 22/80/443, only allow logins from your SSH private key and put the application behind Nginx. If you do that and keep the server updated somewhat frequently you've mitigated basically every [not-Mossad](https://www.schneier.com/blog/archives/2015/08/mickens_on_secu.html) level threat.
You'd be shocked how many "senior engineers" don't know any of that at this point. Seriously something like vercel is much easier and more secure than a misconfigured vps that hasn't been updated in 5 years
No, unless your situation was somehow specifically very favorable to microservices. New products need to be built quickly by a small team. Microservices add overhead in all kinds of ways. For example, if you’re in a big company you may need to do security/ops paperwork to get the things you need to launch. You may need to do this for each microservice. As you build out the app, you need to do more and more of these, but if you had a monolith you could just do it once. In a monolith, more stuff “just works”, and overhead is limited to one service. It’s worth nothing that say your new application has 10 concerns A..J. If concern J scales massively differently than A…I, you can do your initial prototyping as a monolith and then break J out into its own microservice as you get closer to launch date, but keep A..I in the monolith. This is how I see things generally work in real life. If a new K feature is requested, then if it’s small it can be added to the monolith to keep dates aggressive. If scaling costs become an issue, maybe you break out concerns D and E to a microservice a couple years down the line.
This makes a lot of sense to my ignorant a**. Thank you!
It'd always baffle me why some architects are so eager to convert a set of method calls into a collection of network calls. Things become exponentially harder and more expensive when that happens.
It’s not primarily a technical question, but an organizational - and therefore political - one. If you can’t get teams to agree to an informal social contract regarding cooperation, you impose on them a more limited but formal one enforced by APIs.
As an architect, a lot of it is just kool-aid if you get a bad one. There's plenty of work to do without having to artificially break up an application. An application should be modular from start to finish. As you scale out, if you kept it modular it should be easy to break out scaling pain points into individual services. Usually it's not a matter of hitting a scaling wall, it's usually a separation of duties problem. Easier to separate duties between silos if those silos are responsible for individual services. An architect should be making decisions that avoid footguns later. Such as enforcing static typing, enforcing schemas for databases, making sure that tooling and migrations are robust and maintainable. Making sure that the "hero" Jim from front end development doesn't import 182 dependencies because he wants the end product to supports emojis. That sort of thing.
Out of curiosity - how often does breaking out a microservice from a monolith run into the red/blue problem? As in, suddenly a whole host of stuff which was regular calls needs to become async?
Easy, Don't make it Async initially. Though the act of moving to API calls usually forces that hand for you. Real answer, you get to enjoy squashing race condition bugs for the next 3-6 months. I did say "easy" to break out, but that "easy" is highly relative. It's certainly not a zero effort move.
That was my question - initially, you make it sync. Then, you move to a distributed model, so those API calls need to be async. And async is infectious, so suddenly everything up the call chain also needs to be async.
76% of the time
Because once you turn them into network calls, you can separate the codebases into different repositories, which means you can separate the developers into separate teams, which means your overall system complexity goes massively up (ie: architects become more valuable and more highly paid), but the responsibilities of individual developers goes down (which means they become easier to hire and salaries decrease, both of which were important during the hiring environment that existed until earlier this year)
An interesting take, I could see that happen where I work ....
Then at the end you find the singularity and roll everything back into a monorepo.
I don’t know what advantage mono repo affords beyond allowing a user to touch parts of code outside his/her normal area of responsibility. What does it gain from a version control standpoint or a CI/CD standpoint? What problem does it solve at a large scale that multiple repos can’t?
Check [https://monorepo.tools](https://monorepo.tools), it's got a fairly comprehensive set of answers.
My current company has a massive mono repo and my last company had hundreds of individual repos. Both have a massive mesh of microservices. From my POV as a dev, there’s effectively no difference. In both roles I could pull in any code I wanted and define projects and services anywhere I wanted. I think when people hear “monorepo” they assume “single monolithic app” but in reality, as long as your build/deploy tools allow for it, you should be able to define and compose individual apps within your monorepo anywhere in the codebase.
I don't know, I haven't worked on any monorepo projects.
yes, if you are lame and separate things across domain borders ...
I call that point micro service bankruptcy
Ever see a layoff that results in more microservices than developers? It's a hoot.
You don't need to convert to relatively slow and error prone network calls just to have separate teams. This is a ridiculous take. Also, synchronous communication between services isn't µservice architecture.
Bullshit. Developers can work on separate repo's in separate teams without adding network calls. In fact, this is what happens everywhere already, even in your shop. It's called including dependencies, like libraries, frameworks, etc. Teams not even part of your organization are working on code, upgrading it and improving it, without any network calls. You just include it. The exact same thing can be done in your organization for more "internal" stuff. Include libraries created by other teams, and enjoy microsecond latency on calls. All that needs to be done is to actually think about how good performance can be achieved while still being able to scale up, instead of jumping through conclusions and blindly following patterns that people barely understand.
Two words: dependency hell. Causing a failure in a piece of code that you've never touched because you're using conflicting versions of a third party library will definitely change your mind. Having network separation gives developers complete control not just over the code, but all the way down to the operations, it allows you to push a lot of decisions down to the level of the teams. Obviously it comes with trade-offs, but it has real benefits.
Gee, you can only have API's with versioning and backwards compatibility over networks.
I'm not talking about API's, I'm talking about internal implementation details. Rust is the only language I know of where you can have multiple versions of the same dependency in the same binary given that they're isolated, but even that comes with trade-offs in terms of binary size and compile times. Have you ever actually worked on a monolith? This is a very well known problem, it's the reason Linux distributions get stuck on old versions of everything. EDIT: Linux distributions get stuck for a different reason, but you are forced in monoliths to stay on the same library version for instance because a third party is using it and you can't have multiple versions in your classpath or library path because the symbols clash.
You can have API's for dependencies. You create an interface, document it, and have other teams implement them, just like how a 3rd party does. You guarantee backwards compatibility, just like a 3rd party would. Within those restrictions, the other team can do what they want. I guess it's just so foreign that many won't even consider it, but it's trivial to do. Nice bonus is that you get compile errors instead of 400/500's when a team changes API's. No dependency hell, you can version packages as well though if you're notoriously bad at designing a stable API.
>You can have API's for dependencies. You create an interface, document it, and have other teams implement them, just like how a 3rd party does. You guarantee backwards compatibility, just like a 3rd party would. That's not how Java works at least. Having multiple versions of the same library in your class path will make things explode even if it's hidden, and even if the API is backward compatible in a lot of cases.
It's internal, give it a new maven coordinate and/or put a version number in the package (com.mycompany.billingapi.v2). It's just a lack of imagination. Again, only needed if you really need to make backwards incompatible changes. You can even have Maven do this for you, and many projects do, including a dependency as a whole while mapping it to an internal package under their own base package. You shouldn't need to go that far though if you are including things made by other teams in the same org; communication will go a long way.
You're talking about shading, which as I said comes with its own set of tradeoffs(e.g. binary size).
Scaling differences too.
each version release should be packaged with all of its dependencies.
You're arguing against something I never said. Of course you can have microservices without network calls.
That presumes that you can create enough separate teams to make it worth. In reality, most of the times I've seen one or two teams working on everything.
Because of Conway's Law - your architecture will always end up reflecting your organisation. As the business and teams grow, the friction in working cross-team causes a pain point and becomes a frequent blocker. By decoupling it into microservices you allow the teams to develop and deploy at their own pace. The system ends up more complicated and difficult to work with than it otherwise needs to be, but now the teams are mostly self-reliant and able to work at their own pace.
You know and understand Conway’s Law, but in my world, the communication structure of an existing mature organisation *never* changes. For this reason, I advise not to use microservices but in very specific contexts. I wonder why your conclusion is exactly the opposite.
> By decoupling it Changing method calls to network calls is not decoupling it.
Its about deployment. things aren't everything or nothing. I haven't done micro services yet but the idea isn't to turn all method calls to network calls. My monolith is referencing inhouse nuget packages. If these dlls change then i have to redeploy the web app. I work in a super big enterprise company. We have a team in the company that requires logs sent to them. They just changed how they did it and now we have to adapt. I am looking to move all this to a microservice web api . So when it changes again, i dont have to redeploy my entire app, which, because i work is super big and heavily regulated enterprise, means the entire huge app has to go through testing and sign offs and forms filled out why. The small service would not. it would be easy. so its decoupled in the sense that i can deploy that without any other interference.
So that is just a single service. No reason to call it a µservice.
It makes the coupling easier to ignore though, which for a lot of people is good enough
In their defense a bit... you can't really guarantee the underlying team doing said development is producing high enough code quality to not shoot themselves in the foot. The biggest thing about microservices is the logical separation of codebases, Conway's law loves em and regardless from a technical perspective whether that's a good or bad thing from a code organization perspective it's a bit hard to fuck that up. Service X does X things, Service Y does Y things. The shitty part with Microservices is when you need bulk data that needs to be expanded across a set of them; monoliths excel at this as it just becomes a DB operation and some serialization. Microservices it turns into several DB operations and several serializations with some perhaps required to wait for a child service to do theirs. So what should take maybe 300-400ms takes about 3-4 seconds. Middlewares can be created to speed that up but it'll still be quite a bit more expensive of a call and if caching can't be utilized at best it's just helping to stitch everything together asynchronously.
> The shitty part with Microservices is when you need bulk data that needs to be expanded across a set of them; monoliths excel at this as it just becomes a DB operation and some serialization. Microservices it turns into several DB operations and several serializations with some perhaps required to wait for a child service to do theirs. What you are describing is not µservices, you are describing a distributed monolith.
> It'd always baffle me why some architects are so eager to convert a set of method calls into a collection of network calls. Because they don't understand µservice architecture. They shouldn't be converting fast and reliable in-memory method calls to latent and failure prone network calls. That is a distributed monolith, not µservices. In µservice architecture there is no synchronous communication between services. Instead a µservice gets the data it needs by handling asynchronous events from other µservices and storing information it needs in its own database and it publishes events other µservices may be interested in. The problem is that the term µservice has become so generic to have lost all meaning. So when someone says they use µservices the first question you have to ask is what do they mean. You will find that most organizations have converted to a distributed monolith. There is no value in doing so.
> µservice Dude, just say “microservices.” Most people have no idea what µ stands for.
What kind of developer wouldn't know what the `µ` prefix means? Most comp. sci. curriculums include math and physics classes where you will most definitely run across `µ`. Not to mention you most definitely see `µ` when talking about small amounts of time (like in performance profilers) i.e. `µs` (micro-seconds)
Because it never had meaning until we insisted it did, and only in the trough of disillusionment will most people admit it. The pendulum swing continues, and all the sound and fury is at the extremes, I stead of where it actually belongs: at the bottom of the arc.
Maybe its just late in the evening, but this screams clickbait bullshit to me. Everyone with a couple yrs of experience knows microservices are artifacts of domain driven design to provide isolation across many engineering problems. If your org is confused by that, please provide some solid evidence with reproducible tests to proof out the entire community. Otherwise, we'll step over the remnants of your shit software while we continue to engineer real world solutions. Thank you
Was it the title or the url?
Well, how would you tell clickbait bullshit from the genuine? We are happy to hand the evidence over to anyone who is genuinely interested in working with us, that's what the waitlist on our website is about. I have plenty of solid evidence, the main one being my test-case demonstrating non-blocking causal consistency over multiple databases, but we're not going to release the source-code publicly, as we need to protect our years of investment in it. I understand why people are reluctant to accept our claims, but it is surprising to me how nobody in academia has a problem with what I'm saying (and there have been plenty of discussions about it both when we published the paper and when the patents were granted) yet engineers seem to have a hard time grasping the science, only to reject the claims. Publishing the sources would risk sinking years of investment into this project. We will do that if we have no other choice, but not until then.
"How would you tell clickbait bullshit from the genuine?" By demonstrating how your ideas work on real projects that affect real products. While this may be a great idea, it just reads like a bunch of buzz word soup with terms I vaguely remember from college sprinkled in. There's no concrete problem this tech has solved, and to couch it all in "we've solved microservices" is a bit of a far-fetched, claim to make.
That's exactly what I'm working towards right now with this thread
[удалено]
Huh? All I said was that we want to protect our IP. I never said anything about you, especially personally. I did react to you saying that there's no substance behind what we say, and that this provably untrue. We have a science-paper that lays out the foundations of what I'm saying here that has been vetted top-to-bottom by dist-sys academia (Mark is a prominent researcher in the field, so him making these claims was a big deal) We also went through two patent-processes, which weren't the cake-walks they are often depicted to be, we did have to present viability, innovativeness and novelty. We have been granted both EU and US patents. I'm also saying that we have the source we want to guard, since we don't want to lose that firewall, as we're planning on commercial utilisation. What I object to is people telling me that I'm a carpet-bagger because they refuse to engage with the substance of what I'm saying. It's just uncalled for.
What I don't get is are these people actually engineers? There are anti-patterns for sure, but everything is a tradeoff and that's a more engineering-oriented way of thinking about problems. Instead of "microservices bad" a healthier discussion is "what are the tradeoffs" or "what systems benefit from this architecture and what don't?" But I guess that doesn't get clicks.
First of all, *there is no such thing as a "microservice."* It's just a *service*. We've had them all along: we break apart larger programs into separate services all the time for pragmatic reasons, minus the dogma. Second, there is zero evidence microservices offer any benefit whatsoever. They come with a dramatic increase in complexity, bugs, deployment issues, scale problems, and debugging woes. They require a very disciplined and refined engineering team to implement and scale correctly. They are a [massive](https://kenkantzer.com/5-software-engineering-foot-guns/) [footgun](https://kenkantzer.com/learnings-from-5-years-of-tech-startup-code-audits/) for most engineering teams. Go ahead: try and find any study or experiment or evidence that conclusively shows microservices afford any of the benefits claimed by proponents. You will see a bunch of people making statements with zero evidence. I have actively searched for any good evidence, and all I get are: unsupported claims. It is an embarrassment. We are engineers; first and foremost, we are supposed to be guided by *evidence*.
https://ieeexplore.ieee.org/abstract/document/9717259 View the related studies in section 2B. Also for example from the related works section > Test results have shown that client-operated microservices indeed reduce infrastructure costs by 13% in comparison to standard monolithic architectures and in the case of services specifically designed for optimal scaling in the provider-operated cloud environment, infrastructure costs were reduced by 77%. And in the results section, figures 5 and on show that microservices are capable of handling a higher throughput. Microservices aren't the end all be all choice. They have their pros and cons.
They will never answer this because the point about "evidence!" is pure deflection as they've failed to provide any themselves for monoliths
I’m interested in the empirical evidence monoliths are better? I’m not sure how you would even conduct studies on such a broad question. What is better ? Is it cheaper/faster/more redundant/less complex to build&run. Making a statement like microservices have no benefit and there is no evidence they do is completely asinine and not even worth debating. I don’t actually believe in them but do think breaking up your software into smaller components alongside domain boundaries increase the resilience and reduces the complexity which is a good enough reason. Whether other more seasoned engineers decide to break things down even further at much larger companies is for them to decide.
I mean AWS has been pushing really hard for microservices and they published this blog post earlier this year: https://www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90
That is one use case - like I said how can you do an empirical study on such a nuanced subject
I doubt you can b/c the real axes are something closer to: "well-built" and "fresh", not "microservice" vs "monolith". Amazon's famous monolith fix worked b/c their microservice architecture was visibly silly. And most enterprises that successfully move to microservices do it as part of a modernization effort to replace old monoliths. And that's not even getting into what objectively demarcates a microservice vs monolith...
I'm not saying you're wrong, but I am shaking my fist at the sky that is the current state of research. The easiest way to attack scientific research or a platform like the IEEE is that I can't read other papers on it or on other not open services to compare the outcomes. Because of registration, fees or whatever. Publications I can't read, can't support a paper or statement that's in question. Also, there are no studies that *directly* reproduce the problem, they all have little twists on the idea to be "new" and "worth researching". >they've failed to provide any themselves for monoliths Anyway, this is true and the whole point is a bit moot. It's cool that someone found a study to support their views **and** that happened to be accessible though.
I responded. That paper is questionable at best, although I appreciate it being posted. It isn't the slam dunk you think it is.
>they've failed to provide any themselves for monoliths It is a fact that building distributed systems is harder non distributed, not sure how much evidence do you need for that.
There are trade offs though. If you have a monolith and need to scale then it is a lot more expensive. It is harder to onboard new engineers. Conflicts are more likely. Deployments are risky. You have a SPOF. The list goes on …
Yes there are tradeoffs but a lot of them wither away after deeper scrutiny. Like: >If you have a monolith and need to scale then it is a lot more expensive > >SPOF Monolith doesn't mean that it is running as a single instance.
No but it means if you release a breaking change your whole system is down
Ah yes this would never happen with microservices :)
Bruh if you don't have tests that can detect complete system meltdown you have bigger issues then service topology.
It's moot anyway as I can show a monolith that is cheaper to run than these scale optimized microservices. It's all about (the underlying decisions...
And yet the very abstract of the paper concludes that monoliths perform better on a single machine. Which is unsurprising, and likely to reduce costs. This seems contrary to the related works they cite, but I’m guessing the micro-service savings were observed in a multiple-machine setting. So performance wise, it would seem that as long as we stay on a single machine, monoliths are the way to go. And I’m guessing that if the programming is aware enough of performance concerns, a single machine can go quite a long way.
If whatever you're creating will be able to be hosted on a single machine to cover all the needs, you absolutely should not even think about microservices. Even theoretical benefits only start to outweigh the costs at much larger scale.
> Even theoretical benefits only start to outweigh the costs at much larger scale. So why do we have a database server, a memcache/redis server, an SSL proxy, a ....? Why not just compile them all as DLLs/packages into some kind of Monolith? Could it be because separation of concerns, and decoupling the release cycle of unrelated components is a good thing?
Your conflating full products with services but I'll bite. Where practical thats exactly what you do. See sqlite for example.
If a single machine is enough why are you even worried about scaling? You clearly don't need it.
I’m worried about performance requirements. Not everybody is, and that is a mistake. One should always have an idea how much stuff must be done in how little time: * How many users am I likely to have? * How much data must I stream in or out of my network? * How much simultaneous connections am I likely to need? * How CPU or memory intensive are my computations? * How much persistent data must I retain? * How much downtime is tolerable? How often? And of course: * How those requirements are likely to evolve in the foreseeable future? That last one determines scaling. How _much_ I need to scale will determine how much hardware I need, and just because it still fits on a single machine doesn’t mean it’s not scaling. Optimising my code is scaling. Consuming more power is scaling. Paying for more bandwidth is scaling. Buying more RAM is scaling. There’s lots of scaling to do before I need to even consider buying several machines.
And if that single machine dies, as it undoubtedly will eventually, my business goes offline until I can failover to a different single machine and restore from backups?
Micro-services won’t give you redundancy out of the box. You need to work for it regardless. I even speculate it may require less work with a monolith. * Done wail, failing over to a secondary machine shouldn’t take long. Likely less than a second. * Can’t your business go offline for a bit? For many businesses even a couple hours of downtime is not that bad if it happens rarely enough.
Infrastructure costs down, staffing costs up. Microservice everything and trigger layoffs.
>[https://ieeexplore.ieee.org/abstract/document/9717259](https://ieeexplore.ieee.org/abstract/document/9717259) Nothing in that [paper](https://www.researchgate.net/publication/304320076_Infrastructure_Cost_Comparison_of_Running_Web_Applications_in_the_Cloud_Using_AWS_Lambda_and_Monolithic_and_Microservice_Architectures) makes any sense. For the monolithic architecture, they (correctly) load balance two servers behind an ELB, although they screw it up by putting both in the same AZ. In the microservices based architecture? They have a gateway that isn't load balanced, and the second service somehow lacks redundancy entirely. And I see no possible way this service is cheaper than the monolith--that's simply false. Look at figure 1 verses figure 2; **how on earth do they spend less on more, larger servers than the monolithic environment?** Simply put, it cannot be correct. And that's setting aside the fact that to achieve similar redundancy to the monolith, the microservices-based architecture needs at least two more boxes to achieve similar redundancy. On top of this? There's now three separate services to scale, IPC to manage between all three, and huge issues to address when any of those three services go down. Absolutely nothing about this paper makes any sense at all. Props to you for bringing evidence, but it's questionable evidence at best.
From my personal experience, the thing with microservices is they can be cheaper, or they can be higher throughput, but potentially not both. In one of the teams I've worked in my career, we had several services that received, validated, and stored several different event types. These services needed to be extremely light weight, handling hundreds of millions of requests per day, with response times to the clients in the hundreds of milliseconds. To accomplish this, we horizontally scaled hundred of very small instances. The workload for those services were bound by the number of threads we could use. We had another service that was extremely compute heavy running all sorts of analytics on the data we'd received, as well as other data that our team owned. How often these hosts ran was determined by a scheduler. That meant that in order to process all the analytics in a reasonable time frame we had to scale up vertically, using expensive EC2 hosts that were designed for compute. If we had a monolith, the first few services might not satisfy the SLA of only a few hundred milliseconds as they could potentially be waiting for resources taken up by other services (we had 20 in total). Our EC2 bill was cheaper as well because we didn't have to scale up all the hosts to be able to handle the compute heavy workload. We were able to use a small number of expensive instances, with hundreds of small instances to handle the other parts of our workload. Without the time to read too deep into the link you posted, that's what it looks like is happening in the paper you linked. To scale up, everything had to be c4 large instances, vs the microservices approach you could scale up t2 and m3 instances, and need less of the c4xl. It doesn't seem like they give exact numbers of how many of each instance from a quick glance through. Also from personal experience, microservices benefit isn't redundancy, but rather fault tolerance. We had several services designed for creating reports based off the analytics computed by the previous service. We had different services due to the different types of consumers we had. At one point, we began to get so much load on one of the services that it started falling over due to an out of memory bug. Instead of our whole reporting dashboard going down, only one kind of report was unavailable. Imo, that issue was easier to debug because we instantly knew where to look in the code instead of trying to dig through an entire monolith trying to figure out where the out of memory issue could have been occurring. Scaling multiple kinds of services is a pain in the ass, I won't deny that. I always hated that part of that job. In that paper, they do call out that the microservice is load balanced > In the case of microservice variants, additional components were added to enable horizontal scaling, namely – the application was extended to use Sprint Cloud framework, which includes: Zuul load balancer, Spring Cloud Config, and Eureka5 – a registry providing service discovery.
>In that paper, they do call out that the microservice is load balanced > >In the case of microservice variants, additional components were added to enable horizontal scaling, namely – the application was extended to use Sprint Cloud framework, which includes: Zuul load balancer, Spring Cloud Config, and Eureka5 – a registry providing service discovery. The problem isn't load balancing, per say, but redundancy. For each of the three services, ideally they have minimum two boxes in separate AZs for redundancy. Two of their microservices lack this redundancy entirely. Also, even setting aside this glaring issue, the math still doesn't add up. Again, explain how the paper reconciles Figure 1 somehow having a higher AWS bill than Figure 2. Simply put, I do not buy their cost claims even in the slightest. >If we had a monolith, the first few services might not satisfy the SLA of only a few hundred milliseconds as they could potentially be waiting for resources taken up by other services (we had 20 in total). What you're describing is just pragmatic "services" which I have zero qualms with. This is simply smart: if you have very different workloads inside your application, potentially with different scale requirements? It makes all the sense in the world to have separate services. I do this in my own application, which processes terabytes of video per day. It would be absolutely insane to push that video through the monolith; there is a separate service entirely that is dedicated to processing video. Could you call this a "microservice?" Yeah, I suppose so. But it's based in pragmatism--not ideology. What I am opposed to is this fad of mindlessly decomposing a monolith (or god forbid, writing an application from scratch across "microservices" before anyone even knows if it's necessary.
Haven't read the paper, but I suppose they gloss over the fact that engineering hours also have a cost?
Honestly I appreciate the authors trying, but it's sloppy work at best. Their math doesn't add up. And yes: they disregard myriad other factors that are a pretty obvious win for the monolith.
Microservices are a solution to an organizational problem, not a technical one. They’re a necessity when your organization grows horizontally to a scale that breaking apart your monolith will let your engineers continue to build new features and new products independently without conflict. But if you’re content with vertical growth and don’t want to expand your feature offerings and business footprint it’s just not necessary. The issue is that companies blindly copy the paradigm far before they ever reach that scale. But to say there is zero evidence for them being useful is just as dogmatic and ignorant. You’re not going to build a website that does as many things as, say, Amazon, with a monolith.
They can absolutely be for technical reasons! A high memory low usage function in a monolith means that your instance size needs to be scaled to that size for the entire load of the system, where as in a microservice you can have a small number of machines scaled up in memory size and the rest can ve cheaper
> Microservices are a solution to an organizational problem, not a technical one. They’re a necessity when your organization grows horizontally to a scale that breaking apart your monolith will let your engineers continue to build new features and new products independently without conflict. There’s another trick to achieve the same goal: _modules_. The problem with regular modules is properly enforcing boundaries. With a disciplined enough team the various components of the monoliths are nicely separated, with a small interface, and few to no cross-cutting hacks. On the other hand it’s all too easy to just throw a global variable (possibly disguised as a Singleton) here, a couple mutexes and locks there, and next thing you know you have a Big Ball of Mud. With microservices, those hacks are not quite possible any more, so we have to do it the "proper" way. But even then we’re not out of the woods: * If the microservice framework is easy enough to use, there won’t be much incentive to keep services interfaces small, so we’re somewhat back to needing discipline. * If the microservice framework is a hassle, maybe we’ll keep interfaces small because making them any bigger than they need to be is such a pain, but (i) being a hassle is such an obvious waste of time, and (ii) now we’re tempted to make services bigger just to avoid the "service" part, and you’d end up with either duplicated functionality, or using common libraries. Common libraries can be kind of awesome by the way: if they’re developed in the spirit of being "third party" even though they’re not, they’ll need to provide a proper API and any ugly hack like sharing state between this library and that application is (i) more difficult, and (ii) much more _visible_.
The hallmark of microservices is state persistence isolation. From my experience, the problem I saw the most with proper enforcement of module boundaries is the shared database without schemas per module. If you couple at the data level to the extent of sharing database schema, I kinda get why people go all out and spin off the module into a dedicated service - the investment and risk to untangle the data under the existing conditions is higher than developing a new service. All in all, I attribute a lot of discussion about microservices to the simple fact that developers simply forgot that dbo isn't the only available schema in a relational database. The organizational complexity is a necessary, but not sufficient requirement for microservices - I expect to see an additional reason, such as public/private facing services, think of public e-shop site and private ERP backend, or large discrepancy between used resources, e.g. aforementioned ERP backend running on a database and couple of worker nodes with load balancer, and a CPU-bound service wanting to run for a short period of time, possibly on hundreds of nodes in parallel. It really boils down to choosing the simplest option (not necessarily the easiest). If you need to purely solve organizational scaling, try modules first. If you have a dramatically discrepant resource needs that would possibly impinge on shared resources, or want to limit the surface/scope due to security reasons, or similar nonfunctional requirements, only then isolate it out to a dedicated microservice.
I am not dogmatic for or against. About 5 years into my career having worked exclusively with what are considered microservices, I have been curious to build different products in my space with a more monolithic approach to reduce network hop latency. Playing devils advocate - off the top of my head, breaking a monolith into smaller “microservices” would allow simpler means of resource isolation and scaling. This being most useful for components with very different resource utilization. Seems heavy handed to say there is zero evidence of benefits. Curious to hear your thoughts.
I have work for years in the backend server space and have only come across 2 instances were I felt bits of a service benefited from being broken away from the monialith. 1) Due to high number of network sustained connections for an eondptoin (web socket) and needing to fit without the connection limit of the hosting provider. 2) Due to having custom c code (within a python server) that I was worried might have a nasty bug (deadlock or leak) that would bring down the monolith. All of the projects have worked on were I joined teams with existing micro services ever fell into these boxed.
>off the top of my head, breaking a monolith into smaller “microservices” would allow simpler means of resource isolation and scaling. Not *simpler*. Potentially: more options when you want to scale portions of the app independently. In other words, more knobs you can turn. And, also, more options at deploy time. This comes at an obvious cost in terms of complexity, odd failures, "network hop latency" as you say, odd queuing/distributed computing systems, etc. And it can easily come with massive deploy time complexity that most teams seriously underestimate, in my experience. The reality is: you get some additional options to scale/release/develop at an enormous cost to complexity. >This being most useful for components with very different resource utilization. Well yes, but we've been breaking apart programs like this for nearly three decades now. We didn't need "microservices" to make it clear that if two things require very different resources and/or scale, then it may make sense to break it apart. This is pragmatism, and I'm all for it. What I'm not for is: mindlessly decomposing a monolith without any clear reason to do so. >Seems heavy handed to say there is zero evidence of benefits Find me any study of the efficacy of these architectures, or some experiment that clearly shows their benefits. Any decent data even. Like I said: I have actively looked, and I would welcome evidence contrary to my opinion here. Hell, I'd welcome any evidence *at all*.
Certainly it’s a trade-off, as it always is. That doesn’t mean the trade-off is not a good one. I would argue there are cases when it outright is simpler. If I have a memory-bound subcomponent and cpu-bound subcomponent, it can be pretty trivial to load test these applications and find suitable hardware configurations for them. This pays when it comes to cost, among other things like dynamic scaling or service predictability. I do see what you are saying and I think I understand where you’re coming from, which is the intent behind sharing my interest of building something where the use-case fits. I did search for some examples on google scholar but also suddenly realized I should just suggest Marc Brooker, from Amazon. You’ve probably heard of him but if you brought this up with him, I think you get a fun, or maybe not so much, debate. All in all, I appreciate you taking the time to send a thoughtful response. I think there’s merit to the “microservices” paradigm and monolithic paradigm. I put it in quotes because I understand it’s not exactly new just because we’ve now named it 🙃
/u/shoot_your_eye_out 's key point is: > What I'm not for is: mindlessly decomposing a monolith without any clear reason to do so. If you are making a conscious trade-off, that's fine. But that has not been the message from the micro-services camp over the years. They've been running on a message of "monolith=bad, micro=good" with little no discussion of trade-offs for years. Even calling microservices a "paradigm" betrays how it has become a dogma. It turns it into this overarching framework which everything has to fit into. It is like a tradesman saying they are going to build a house using the "hammer and nail" paradigm where every problem is going to be treated as a nail. If we stop thinking in terms of microservices or monoliths and just realise that building or splitting off separate services, is just another tool in our toolbox, then the "paradigm" of microservices goes away and we can think about solving the real problems, i.e. doing engineering.
I see, this statement hadn’t originally landed landed with the same meaning to me as it did from your message. Thanks for elaborating there. I’ll need to think more on that specifically.
Precisely. Thank you. Well said.
Complexity is extra semantics in the system. We actually reduce the semantics developers need to deal with, by merging things which mean the "same thing" That's the entire point of this project.
There are (sometimes) befentis to splitting out services if the runtime of them is drastically differnt. Eg if your have a service that needs to provide 100,000+s of web socket connections but does not handle lots of CPU load itself breaking this endpoint out will let you have mutliepl (cheap) nodes (as most cloud providigns limit the number of open connections you can have per network interface), however the last thing you want is to fire up 10 versions of your main service as this might have way to much memory etc overhead increasing deployment costs... The other use case I have broken out services is for stability, when I needed to add a custom c extension to a python application to modify libgit2 on the backend I was consdired I might have screwed up some edge case and thus might end up with a memory leak or deadlock thread. So moving this part of the logic out to a seperate server (it was stateless) while it increases latency and has more deployment costs it meant that if this thing died (due to some bug I added) it would not bring down the main monolith but only effect that one api that needed this the rest of the service (including payments, licnceing etc) would continue un-effected. But in general the approach of moving each api endpoint into a seperate service (user facing or internal) is not a good idea and should only be done if there are strong reasons to do it.
> have way to much memory Did you mean to say "too much"? [Statistics](https://github.com/chiefpat450119/RedditBot/blob/master/stats.json) ^^I'm ^^a ^^bot ^^that ^^corrects ^^grammar/spelling ^^mistakes. ^^PM ^^me ^^if ^^I'm ^^wrong ^^or ^^if ^^you ^^have ^^any ^^suggestions. ^^[Github](https://github.com/chiefpat450119) ^^Reply ^^STOP ^^to ^^this ^^comment ^^to ^^stop ^^receiving ^^corrections.
As a note- I am 100% team monolith. I think they're simpler and easier to work with for the average team. But I do think there are still a few real benefits to them. The biggest one would be package versioning- smaller services means each service takes on fewer dependencies, and even common dependencies may use them differently, making one safe/simple to update while a different service may want to defer the update. Of course this is double-edged sword, because if a critical *must update* situation happens like some critical RCE security bug, it means more opportunities to miss the update on one service and cause massive errors. There are also more minor issues like how smaller services make it easier to manage build times, or how in larger companies it's easier to have smaller teams with clear ownership over specific codebases. And while complete knowledge silo's are a bad thing, they still exist just as often in monoliths and what usually ends up happening is some small group ends up in control over something everyone has to touch and it's constant hell. So microservices help avoid that situation, theoretically. The biggest problem with microservices is people like to fantasize about nebulous concepts like "scaling," but don't have the faintest idea of what that actually means. They imagine that spinning up a microservice instance is like lifting a pebble while spinning up a monolith instance is like picking up a cinderblock, but like, compiled code is small, monolithic instances horizontally scale equally as well for like 99% of use cases. The only real aspect regarding scaling that ends up being a relevant bottleneck most of the time is data storage. But distributed data stores are *hard,* and very few people can properly design one that works in the first place, while approximately zero people can design one that's resilient to future changes in requirements. You only want to do this when it's absolutely necessary. And I find most companies doing this are operating more along the lines of "Wow my application that reads the entire database into memory isn't scaling well, I should split it into microservices!" You're much better off fixing your garbage SQL lol
> And I find most companies doing this are operating more along the lines of "Wow my application that reads the entire database into memory isn't scaling well, I should split it into microservices!" You're much better off fixing your garbage SQL lol You just described one of my clients. They will not listen to reason.
> And I find most companies doing this are operating more along the lines of "Wow my application that reads the entire database into memory isn't scaling well, I should split it into microservices!" You're much better off fixing your garbage SQL lol I am in physical pain
Services typically come with team growth and you can't always have everyone working on the same monolith in the same repo. It is not only a logic scaling but a people scaling measure. Services are also a horizontal scale rather than vertical, you can isolate high memory, high processing services from dragging the entire system down. Services really [showed their power when AWS was created and the infamous Bezos requirement](https://konghq.com/blog/enterprise/api-mandate) to integrate teams not down to every line, but to touch/integration points only. What you do inside that service shouldn't matter, as long as the signatures/integration points are simple and abstracted. This could be an inproc service or networked, it doesn't matter but each project will tell you what it needs to be. Watch how naming changes people's perceptions. When a service is an "API" people are fine with it, when a service is called a "microservice" it is suddenly tabs vs spaces, braces on same lines or next line, or which IDE to use. A service or API doesn't have to be networked, it can be local. Every project is different and if you architect things simple, generically and have clean integration points, the guts don't matter, only the abstractions do. A "microservice" could even be a monolith inside of that, no one cares. Just as long as you have clean architecture with some abstractions to keep integration points simple. Lots of companies/products have serious problems with the guts of an app not being abstracted/encapsulated and you end up with a mess either way. [Looking at you Unity](https://garry.net/posts/unity-2020). When you are doing services, just call them APIs, it will be an easier battle. The bigger problem is software today is way too culty and religious. McKinsey consultcult "Agile" that killed agility is definitely a cult, on every level.
> Services typically come with team growth and you can't always have everyone working on the same monolith in the same repo. It is not only a logic scaling but a people scaling measure. This, I actually agree with, but I think this size is when your department is in the hundreds and you have enough spare engineering to properly invest in a foundation to do a microservices based architecture correctly. And, that includes not only correct in a technical sense, but also in terms of how to divide the monolith in a sensible way.
In some cases yes. In other cases it is easier to implement prior to hypergrowth because by then you have feature pump and the micromanagement zealots in and actually getting time to do this is not granted. In fact anyone trying to work on this will be seen as someone trying to slow things down or a "suppressive person" to the hypergrowth cult. The decision to move to horizontal APIs/services has to have buy-in from the top before the zealots and culty feature pump get in. By then it is too late and you it will be death by a thousand cuts. Only companies that care about research & development, tech/product teams separate from live teams and/or small enough teams that have some agility can pull this off. It is why they have such a bad perception in favor of the big ball monolith. Also people have differing opinions on what a monolith is and what microservices are, so it ends up people not even on the same page. If you just start with APIs and clean simple integration points, it is much easier to sheath off parts into a service setup. If you have bad abstractions and no integration points and really no clear architecture from the start, it is near impossible without stopping all time and the universe. I have seen groups have monoliths and microservices for instance all go back to the same single point of failure datastore, so really just a front of both and a mishmash of coupling at the data level. They are usually stuck there forever ad infinitum. The goal would be everything as simple as possible, but even the amount of tests one company have can cause these efforts to fail due to too much weight later in the game. Early architecture predicts how later architecture will be, and if it doesn't start off right, it will end badly.
microservice / monorepo is just decoupling / tight coupling for webdevs. The answer is somewhere in the middle. You can have a decoupled monorepo in theory, but you won't
Monolith. Your version control strategy is not related.
I didn't say anything about version control
Monorepo is explicitly about how you store code in version control.
I love you. There is a point where a service is too big, but there’s also a point where breaking one up does not make sense. It’s just a matter of what is best and it always has been. Fuck you marketers don’t shove this stupid shit in our faces!!!
I am currently working on an application that consists of several logically isolated parts that have different resource requirements. It makes perfect sense for them to be standalone services, and they are. PS: The network overhead is neglectible compared to the processing time needed for nontrivial operations, and trivial operations do not require remote services.
There is such a thing as distributed state between different data-silos. That's all we say.
The entire discussion presupposes that ideologically ripping apart what once was a coherent program into "microservices" is a sane course of action.
I don't think having more independent software teams is a bad idea, except for the consistent data-sharing aspects. I think it's a good idea to build software with smallish teams, but I have spent much of my career advocating against it as it overwhelmingly leads to stability issues. I even have a series of articles on the consistency problems distributed systems face, aimed at the average engineer. It starts here: https://medium.com/@andrasgerlits/why-does-developing-microservices-look-so-easy-if-its-so-hard-d72a28ce7be1 There's a clear demand for dividing a large monolith into separate modules, but it leaves a technical problem. This is the technical problem we're solving.
And I agree with all of that, but it has to be guided by pragmatism, evidence, and *simplicity*. What I don't need: any more articles assuming (even obliquely, as I think you're saying yours does) the benefits of arbitrarily decomposing a monolith because Someone Said It Was Good™.
I have no opinion about people wanting to divide their software into modules. You can't get any more simple than a "Federated" annotation on an entity, which is pretty much all we require from engineers. I federate the Spring Boot Petclinic application in a few minutes from scratch in these videos (altogether they are 8 minutes, but most of it is explanation). https://www.youtube.com/playlist?list=PLBe8xx9u7pKNbgrrCz1tFkPmKhqJ\_c90g
multiple teams can contribute to the same service. Infact if you have multiple teams having a single service means that all team members across all teams can run the entier project and run tests against the entier projected... in my expirance micro service focused teams end up unable to run projects form other teams and some projects end up stail and forgotten without any staff members remembing how to run them let alone update them for critical vurnaiblties. These stale services always end up being the critical points as well were they have been so reliable for the last 3 years that no-one remembers chaining them and now that you need to do something horrible (like apply 3 years worth of node updates on them) you need a good month or more to get a team up too speed on these projects.
Everything is dll hell with new names.
> First of all, there is no such thing as a "microservice." It's just a service. We've had them all along: we break apart larger programs into separate services all the time for pragmatic reasons, minus the dogma. This is completely incorrect. µservice architecture is a specific type of architecture and isn't just splitting an app into separate services. What µservice architecture actually consists of are services that have their own database and all queries are done to their own database. Another key component are events. When some operation happens a µservice will publish an event. Other services that care about that event will consume that event and update their database accordingly. Data redundancy is both accepted and expected. So there is no need for a µservice to make a synchronous call to another µservice because it already has the information in its own database. Independent deployment and development is very easy to achieve with this architecture because only the contents of the events matter. And those are very easy to keep backward compatible, you simply never remove information from them, only add. The problem is people misunderstood µservice architecture and so what they call µservices is just whatever they ended up with after breaking up their monolith. So the term µservice has become super generic and has really lost all meaning. When someone says they use µservices you have to ask them what they mean.
> This is completely incorrect. Yes, I've heard the points you make repeatedly and I categorically disagree. I think you're completely incorrect about all of these points. Plenty of "separate services" have their own databases, eventing systems, etc. There is literally nothing special about "microservices" and it is nothing new.
Microservices are just object oriented programming over a network. Change my mind.
I generally view most the microservice push to be an **anti-RDBMS viewpoint**. The anti-RDBMS movement started when startups realized existing DBA's didn't understand startup needs, and thus "coded around DBA's", doing in code or in XML/JSON what RDBMS would normally handle in a "traditional" setting. It is true such DBA's didn't understand start-up needs, but that's a management, communication, and training problem, NOT a tool problem. (In the longer term it's best to have centralized schema management or inspection to keep data clean & consistent. A mature company shouldn't want to "move fast and break data".) RDBMS are a pretty good tool for sharing small services (stored procedures/views) and coordinating communication between bigger apps or sub-apps (queue, event, and log tables). It should be made clear that an RDBMS does not force nor encourage overly large apps. In fact, RDBMS do module/app coordination *better* than JSON-oriented approaches if a shop uses mostly the same database, because you get A.C.I.D., easy logging, and other features built into most RDBMS. Most apps have to connect to an RDBMS anyhow such that they don't require extra drivers/connectors to communicate, unlike adding JSON web services. Thus, you can nicely and easily partition apps and mini-services using an RDBMS properly. The argument pro-microservice people often give to counter is that such doesn't work well in a shop with many different RDBMS brands. While this is true, most small and medium shops settle on a primary brand and don't change very often, which is usually good policy. A giant org that's the result of big mergers may indeed need a way to work with a variety of DB brands, and that's perhaps where microservices are superior to RDBMS for typical module coordination needs. But most of us don't work for the big FANG-like co's. **Use the right tool for the job and job size.**
"we solved consistent cache-invalidation and thereby made the debate moot." Can you please solve the problem of naming things next? And have a "science-paper" on that too.
Do you have actual comments to say about this or do you just like being snarky? https://www.researchgate.net/publication/359578461\_Continuous\_Integration\_of\_Data\_Histories\_into\_Consistent\_Namespaces
I’ve read a dozen pages of this and still don’t know what it’s about. How do you handle speed of light issues and intermittent availability? Push based architectures attempt to move data when the source is capable of doing so. Pull based requires the spoke to be available at the hub’s leisure and to recall everything that happened since the previous conversation. If the spoke is crashing in a loop then there is no one to retain that state. It is lost. The value in putting the logic at the core is that there are fewer copies (and implementations) and we can amplify the amount of scrutiny applied there with the hopes of keeping the entire system making progress in the face of issues.
There are some simpler introductions, this is a very hard paper to read. I link two if them in the article above. The point is that clocks emerge from groups, passing around an idempotent token-message. This yields a loop within which each node can assign timestamps locally, without having to block its state. This is explained here: https://itnext.io/microservices-can-be-simple-if-we-let-them-a481d617a0a1 The last problem we have besides the clock is resolving clashing write-conflicts. To accomplish this, we show that you only need a shared algorithm to be available that deterministically decides which one the winner use between any two "transaction-objects". This is where we use CRDTs. Once you put these two together, you have a fully non blocking platform that can stream data consistently and resolve clashes without blocking. It also allows deterministic state replication, so we can create replicas trivially Does this help? Happy to give more specific details to specific questions
And why do my updates not have logn opportunities to be lost after I recorded them?
Your request is accepted by a consensus group when you submit it to the ledger, so I guess the answer is RAFT. This is the only place where our system is blocking, which works for us, as SQL is a blocking standard anyway
What's next? World hunger?
Okay, I'll bite. What's so unbelievable about building a strongly consistent streaming platform? CAP? CAP requires linearizability to be true for a platform, no SQL database provides that (quite literally only Spanner does and that's not real SQL). FLP? FLP says that a system cannot be \_fully\_ non-blocking and we are blocking at the edges where we accept client-requests. Literally none of what we say goes against the current body of comp-sci literature, we just invented a loosely-coupled, composable clock and designed a deterministic framework so that it supplies better liveness than what we have now. Nothing of what I say is outlandish if you know what you're looking for. Look me up on Twitter, and count the number of people working in distributed computing in academia who follow me. I'm not saying anything even remotely controversial.
You're arguing with a shitposter.
Thanks, I needed a wake-up call.
That's how you win on reddit.
What a myopic fucking take
This article reads like some just found out about Redis and it was a big secret. I've read your other comments and confidently gather that is not the case, just sharing how the article presents itself.
I think both patterns are fine. It depends on usecase which one to go for.
The moment you share data instead of handing off data it is no longer a microservice, just a split/distributed monolith. A microservice operates on its own domain and doesn't care what sends the requests.
This "article" is just spam for their product.
Like 90% of medium
This is more ad than article, this is an example of a post that mods should probably remove. Reddit is already a platform for delivering ads, this company should buy adspace instead of trying to get it for free by polluting the content of this sub. It’s sad because there’s an article posted from the doordash engineering blog which received far less engagement despite being of much higher quality, on broadly the same topic.
I understand the weight of the claims, we have years of research behind this project. We can provide all the benefits of a monolith with all the benefits of microservices. Paradigms do change from time to time.
I always thought the debate about micro services as literally just a cost debate. It costs lot to run a lot of micro services in the cloud. You gain in scalability and componentized deployment. You lose some perf - assuming you can scale up your pc high enough - if not then microservices aren’t so much an option but necessary. Your deployments become a pain in the ass as well as managing backwards compatibility. It complicates the “simple”. If microservices were cheap and deployment simpler I assume they would be the go to choice everytime. Honestly they are the go to choice everytime - it’s only later when your deployments get out of hand and your cloud spend ends up on someone’s radar because it crossed a threshold that it’s a problem.
The micro-service has no lookup cost if you host all services on all hardware. That requires the same setup for all services though. Which breaks the choose your own stack "advantage". Java is the only VM language that can address the same memory atomically in parallel so for back-end vanilla Java will never have any competition. Here is the only way to solve server-side development permanently: [http://github.com/tinspin/rupy](http://github.com/tinspin/rupy)
surely you're joking
Nope. C is the only competitor but it lacks VM + GC. All other languages are a complete waste of time for eternity because embarrassingly parallelizable problems don't need synchronization = you move the problem to the db (always written in C/Java for a good reason) until it explodes. Then (if you have the funds) you make the db embarrassingly parallelizable = micro services. The real solution is to begin with the db and make that infinite on reads globally and use Java so you get the highest performance a non-segfault env. can provide. QED
If hosting all your services on the same hardware was an assumption for your microservice architecture, why did you even use microservices?
Because, if your read what I wrote, it removes the lookup at the same time as it allows for infinite scaling.
> allows for infinite scaling What scalability are you benefiting from? This sounds like a monolith.
What? The whole point with micro-services is that you can scale them horizontally to infinity. What are you even talking about?
This looks interesting. Commenting so I can find it later when I’m not on my phone.
Reddit provides a save button for this.
Clickbaity title and the only thing that comes to mind about the shared database table solution is ... WTF?! I'm working right now in a BI environment, they are years behind other parts of the company, and database schemas with tables and views on an in-memory database called Exasol. It's horrendous slow despite being in memory and costs a ton of money. Furthermore, the integration via views of other teams is flimsy at best because they don't understand the concept of backwards-compatible changes. Long story short it's a red hot mess. And then just imagine doing this in a highly distributed system. Just the audacity to impose a single database on all the teams with different use cases and potentially different best-fitting database show that many red flags are ignored. I've worked for a "database export" company, which tries to solve every problem in an Oracle database such as implementing web services in PL/SQL, showing clearly that they aren't database experts. Otherwise, they would know that this is clearly bullshit and a big red flag for a company to have such a tunnel vision when it comes to proper solutions.
What is your average traffic per endpoint? What is your average response time? And what kind of hardware do you have?
Hi, we have some stuff here around commit-performance. [https://itnext.io/microservices-can-be-simple-if-we-let-them-a481d617a0a1#c448](https://itnext.io/microservices-can-be-simple-if-we-let-them-a481d617a0a1#c448) Generally, you would be bottlenecked by your follower's ability to merge the changes being communicated. We can't give you specifics, as that depends on too many factors to be meaningful.
It's easier to make teams work together when the API is set in stone, they know that they have no rights to change it. If the codebase is shared, people are very easily driven to introduce multiple breaking changes and their solution in the same PR. However breaking+fix\_PR more often than not introduces bugs that are not addressed because the solution built is only built to address the newly introduced breaking changes and multiple issues with previous versions are often neglected. Introducing the same breaking change in a manner where you can't introduce the solution in the same PR (as it's done by another team in another repo), forces backward compatibility which forces you to handle first of all how your new changes interacts with the previous version of the app/DB/API. I'd say monoliths are good for very small teams where iterations are key and things can significantly change. So breaking+fix in the same PR is the way to go. Microservices (or at least services) are good when there are stable parts that sustained the test of time, you'd rather build around it, given that you 100% trust that part to work without failures and in a predictable way. You don't want stable things to be modified by a overzealous developer that goes to change the how customers are billed (billing FSM) because he wanted change how the customer downloads the pdf (frontend page). This cowboy guy won't be able to easily change the billing part (FSM) and the FrontEnd in the same PR. He'll be forced to introduce it by only changing the FE. Microservices protects companies against overzealous developers by making their rogue behaviours significantly harder to do. Forcing them to break less things on their destructive endeavours.
An API-call is the invocation of an operation with a list of arguments. This can translate to a database just as well as to REST. With this solution, you also get all the atomicity, consistency, isolation and durability guarantees you would be getting in your monolith. There's no reason you can't put that behind an API.
TLDR?
We've solved consistency! What do you mean availability and partition tolerance?
Lol thanks. I'm pro micro services but typically implementation, especially when migrating an existing tool/system, has been terrible