Distributed systems for stream processing : Apache Kafka #apachekafka #freedata

Characteristics of Distributed Systems
This lesson discusses the various cornerstones of distributed systems: reliability, scalability, availability, consistency, and maintainability.

We'll cover the following

Reliability
Fault vs. failure
Scalability
Load
Examples
Measuring performance
Horizontal vs. vertical scaling
Maintainability
Availability
Consistency

Reliability
Reliability is a system’s ability to continue to work correctly in spite of faults. A distributed system is usually made of several smaller sub-components that together work to deliver a service. A reliable system can be banked upon to continue to work without degradation of service if a part of the overall system fails. The reliability concept can be extended to include a system’s ability to continue to perform with the expected functionality, tolerate human errors and unexpected use of the system, maintain performance under high data volume load, and, prevent any unauthorized use or abuse of the system when failures do happen.

Fault vs. failure
The terms “fault” and “failure” are often used interchangeably but mean different things. When a part of a system experiences failure but the system as a whole can still continue to operate reliably, we label the system as fault-tolerant or resilient. The system can tolerate components deviating from the spec but still function correctly. A failure occurs when the system as a whole fails. No system can be made fault-tolerant from all types of possible faults and can always potentially fail as a whole.
Scalability
A system is said to be scalable if it can continue to work correctly as the load on the system increases.
Measuring performance
When the load parameters increase, a scalable system is expected to keep its expected performance. Amazon’s S3 service level agreement (SLA), for example, promises a certain performance benchmark to its users. For instance, the customers are entitled to service credits if S3’s monthly uptime is less than 99.9%.

Horizontal vs. vertical scaling
Once load starts to increase on a system, it may be necessary to re-architect the system to handle the new load. There are generally two ways to scale a system demanding more resources:

Vertical scaling: Add a more expensive or beefier machine than the one on which the current system runs. Consider a MySQL server that runs slowly on the current machine. If we replace it with a new machine with ten times the memory and computing power than the current one, the database server will be able to process a much larger number of queries.

Horizontal scaling: Horizontal scaling refers to distributing load across several smaller machines. There’s a ceiling to vertical scaling as machines can only become so powerful before they run into the limits of physics. This is assuming you don’t hit the budget limits of your company in the first place. Scaling horizontally brings complexity with it, especially for stateful services. A distributed database is not a trivial service to maintain when spread across several machines. However, stateless services are much easier to scale horizontally.

In practice, a hybrid approach is usually selected to address scaling issues which uses more powerful machines that also spread load horizontally. Note that there is no one-size-fits-all architecture for distributed applications operating at scale, as each application can have unique load parameters. Access patterns may be different and so too the SLA’s response time requirements.

Maintainability
One of the cornerstones of good software design is maintainability. There are three major aspects to maintainability
Availability
Availability is another key characteristic of distributed systems. Though availability is quantified mathematically, we can, for our purposes, think of availability as the length of time a system is responsive and functioning within a specified period of time. We’ll say the system is 100% available

CAP theorem
Consistency
In our web server example, if we designed the service to always return the latest data from the database, the service will not always be available if the database is down, but it will still be considered consistent. In distributed systems, there are generally