Site Reliability Engineering

Site Reliability Engineering (SRE) originated at Google when they tasked a team to make Google's already highly reliable services even more reliable. It's a discipline that incorporates aspects of software engineering and applies them to infrastructure and operations problems. The main goal is to create scalable and highly reliable software systems.

Key Principles of SRE

Reliability first

The foremost goal is to ensure that services are reliable and available to users. However, SREs recognize that striving for 100% availability is often not the best use of resources. Instead, they focus on ensuring a service meets its Service Level Objective (SLO), which is a bit less than 100%.

Embrace risk

SREs accept that no system can achieve 100% uptime. They define an acceptable level of risk and ensure that services are available within that threshold.

Automate everything

To scale systems and processes, automation is essential. SREs aim to minimize manual operations work, seeing it as a toil and an error-prone task.

Reduce toil

Toil is repetitive, manual work that doesn't bring lasting value. SREs strive to keep toil below a certain percentage of their total workload.

Measure everything

Measurement and monitoring are vital. SREs focus on metrics, logs, and traces to measure the reliability and performance of services.

SRE Practices

Monitoring and Alerting

To detect and notify of anomalies or outages in services.

Capacity Planning

Estimating how much infrastructure is needed to serve projected future demand.

Change Management

Safely rolling out changes while minimizing the risk of outages.

Emergency Response

Having procedures in place for dealing with incidents and outages when they occur.

Postmortem Analysis

After an incident is resolved, SREs conduct a blameless postmortem to understand why it happened, how it can be prevented in the future, and how to improve the incident response.

Performance Tuning

Optimizing services and infrastructure for speed and responsiveness.

Efficiency and Cost Optimization

Making sure resources, both in terms of infrastructure and manpower, are used efficiently.

Software Practices in SRE

Infrastructure as Code

Managing infrastructure using code and automation tools.

Canary Releases

Rolling out changes to a small subset of users to assess impact before a full release.

Blue-Green Deployments

Using two production environments to ensure zero downtime during deployments.

Chaos Engineering

Intentionally introducing failures to test system resilience and improve it.

Key Concepts in SRE

Service Level Indicators (SLIs)

Metrics or key performance indicators used to measure the reliability and health of services. Examples: request latency, error rate.

SLIs can be broadly categorized based on what they measure:

Availability

How often a system is operational and accessible. It's usually measured as a percentage of uptime in a given period.

Latency

The time it takes to process a request. This can be measured as mean latency, percentile latency (e.g., 95th percentile), etc.

Throughput

The number of requests a system can handle in a given time period.

Error Rate

The percentage of all requests that result in an error.

Saturation

How "full" a service is. This might be measured in terms of system utilization or other capacity metrics.

Durability

Often used for storage systems, it measures the likelihood of data being intact and retrievable over long periods.

Service Level Objectives (SLOs)

Targets set for SLIs. For instance, if an SLI is request latency, an SLO might state that 99% of requests should complete within 200ms.

Components

• Threshold: This is the target value or range for an SLI. For example, an SLO might state that "our API will have a latency of less than 200ms for 99.5% of requests over a 30-day window."
• Evaluation period: This defines the period over which the SLO is measured, e.g., a week, a month, or a year.

Why are SLOs important?

• Focus on What Matters: SLOs help teams concentrate on what's crucial for users. For example, it might not be vital for an internal dashboard to be available 100% of the time, but it could be critical for a payment gateway to be highly available.
• Informed Decision Making: Knowing how close you are to your SLO can guide decisions. If you're well within your error budget, you might decide to push a new feature. If you're close to violating your SLO, you might hold off on that new release.
• Balancing Speed and Reliability: SLOs and error budgets help balance the often competing goals of moving quickly and maintaining reliability. Without them, teams might lean too heavily towards one extreme.
• Stakeholder Communication: SLOs provide a clear, measurable way to communicate with stakeholders, both internal and external, about system performance and reliability.

How to set an SLO

• Understand user expectations: Talk to users and understand their pain points and needs.
• Choose meaningful SLIs: Measure aspects of your service that truly matter to its functionality and user satisfaction.
• Set realistic targets: It's tempting to aim for 100% availability, but that's often neither realistic nor necessary. Understand what level of service is required and aim for that, allowing room for maintenance, unexpected issues, and feature releases.
• Evaluate and Aajust: Regularly review your SLOs to ensure they remain relevant and adapt them as your service or user needs change.

Service Level Agreements (SLAs)

Formalized agreements with users or customers about the level of service they can expect. They often include consequences for not meeting those levels, such as financial reimbursements.

Purpose of an SLA

• Define Expectations: SLAs clearly define what the client can expect from the service provider. This clarity helps avoid misunderstandings.
• Measurement and Monitoring: They offer concrete metrics against which the service can be measured. This is crucial for assessing the service provider's performance.
• Compensation: In cases where the agreed-upon service levels are not met, SLAs can provide a framework for compensations or penalties.
• Legal Protection: In the event of disagreements or disputes, the SLA can serve as a reference or even as evidence.

Common Components of an SLA

• Service Description: A detailed description of the service being provided.
• Performance Metrics: Clear metrics that will be used to measure the performance of the service. Examples include uptime percentages, response times, and resolution times.
• Service Availability: Specifies when the service will be available and might also detail the planned maintenance windows.
• Duties and Responsibilities: Clear demarcation of the responsibilities of both the service provider and the customer.
• Support and Escalation Procedures: Detailed process for reporting issues and how those issues will be escalated through the provider's support chain.
• Confidentiality and Data Protection: Assurances regarding how customer data will be handled and protected, especially relevant with GDPR and other data protection laws.
• Disaster Recovery and Backup: Details on how data will be backed up and how and when it will be restored in the event of a disaster.
• Penalties and Remedies: Specifies the penalties the service provider will incur if the service levels are not met. This could be in the form of credits, refunds, or other compensations.
• Termination Conditions: Under what circumstances the agreement can be terminated by either party.
• Review and Modification Procedures: Specifies how and when the SLA will be reviewed and the procedures to modify it.

Types of SLAs

• Customer-Based SLA: An agreement with an individual customer group, covering all the services they use. For instance, a company might have an SLA with its ISP or cloud provider.
• Service-Based SLA: An agreement for a specific service, irrespective of the customer using it. For example, a cloud provider might have an SLA that applies to all its storage or compute services.
• Multi-Level SLA: This divides the agreement into different levels, each addressing different sets of customers for the same services, in the same SLA.

Monitoring and Enforcement

It's crucial for both parties to continuously monitor the parameters mentioned in the SLA. Many companies use specialized software to automate this process. If an SLA is breached, the agreed-upon remedies or penalties come into play.

SLA vs. OLA

It's worth noting the distinction between SLAs and OLAs (Operational Level Agreements). While SLAs are made with external customers or between departments, OLAs exist internally between different teams in an organization, helping them provide a consistent level of service to each other.

Error Budget

The difference between 100% uptime and the SLO. If a service has an SLO of 99.9% uptime, then it has an error budget of 0.1% downtime. This budget allows for planned outages, deployments, and other activities without breaching the SLO.