Architecting for High Availability and Disaster Recovery

1) Why Architect for High Availability and Disaster Recovery?

High Availability (HA) ensures systems remain operational with minimal downtime, while Disaster Recovery (DR) enables recovery from catastrophic failures. Together, they guarantee business continuity and compliance with SLAs. Key metrics include:

• RTO (Recovery Time Objective): Time to restore operations after a failure.
• RPO (Recovery Point Objective): Acceptable data loss window during recovery.

This guide uses AWS services like Route 53, Auto Scaling, RDS, and S3 to achieve HA/DR without compromising performance or security.

2) Architecture: Multi-AZ and Multi-Region Design

An HA/DR architecture distributes workloads across multiple Availability Zones (AZs) and regions. The client-server model includes load balancers, compute resources, and databases, with failover mechanisms for resilience.

Client
  └─> Route 53 (DNS failover)
      ├─ Application Load Balancer (ALB)
      ├─ Auto Scaling Groups (multi-AZ EC2 instances)
      └─ RDS (multi-AZ, read replicas)

Backend Services
└─> S3 (cross-region replication)
    └─ AWS Backup (snapshots, cross-region backups)
(auth, health checks, failover policies applied)

Rule of thumb: Use multiple AZs for HA within a region; replicate to another region for DR.

3) Key Components for HA/DR

3.1 Application Load Balancer (ALB)

Distributes traffic across multiple AZs, ensuring availability even if one AZ fails.

{
  "LoadBalancerName": "my-alb",
  "Subnets": ["subnet-12345678", "subnet-87654321"],
  "Type": "application",
  "Scheme": "internet-facing"
}

3.2 Auto Scaling Groups

Maintains desired instance count across AZs, replacing unhealthy instances automatically.

{
  "AutoScalingGroupName": "my-asg",
  "MinSize": 2,
  "MaxSize": 4,
  "DesiredCapacity": 2,
  "VPCZoneIdentifier": "subnet-12345678,subnet-87654321",
  "HealthCheckType": "ELB",
  "HealthCheckGracePeriod": 300
}

3.3 Multi-AZ RDS

Provides a standby replica in another AZ for automatic failover.

{
  "DBInstanceIdentifier": "my-rds",
  "Engine": "mysql",
  "MultiAZ": true,
  "AllocatedStorage": 100,
  "DBInstanceClass": "db.t3.medium"
}

3.4 S3 Cross-Region Replication

Replicates data to another region for durability and DR.

{
  "ReplicationConfiguration": {
    "Role": "arn:aws:iam::123456789012:role/s3-replication-role",
    "Rules": [
      {
        "Status": "Enabled",
        "Destination": {
          "Bucket": "arn:aws:s3:::destination-bucket"
        },
        "Filter": { "Prefix": "" }
      }
    ]
  }
}

4) Failover with Route 53

Route 53 enables DNS-based failover using active-active or active-passive patterns.

• Active-Active: Traffic split across multiple regions simultaneously.
• Active-Passive: Secondary region activates during primary failure.

{
  "Type": "AWS::Route53::RecordSet",
  "Properties": {
    "HostedZoneId": "Z123456789",
    "Name": "example.com",
    "Type": "A",
    "SetIdentifier": "Primary",
    "Failover": "PRIMARY",
    "HealthCheckId": "hc-123456",
    "AliasTarget": {
      "HostedZoneId": "Z987654321",
      "DNSName": "alb-primary.us-east-1.elb.amazonaws.com"
    }
  }
}

5) Backup and Recovery Strategies

Automate backups and replication to minimize RPO. Use AWS Backup for snapshots and cross-region replication.

• Snapshots: Regular database and EBS volume snapshots.
• Cross-Region Replication: S3 and RDS replication to another region.
• Automation: Schedule backups with AWS Backup or Lambda scripts.

{
  "BackupPlan": {
    "BackupPlanName": "DailyBackup",
    "Rules": [
      {
        "RuleName": "DailyRDSBackup",
        "TargetBackupVaultName": "MyBackupVault",
        "ScheduleExpression": "cron(0 5 * * ? *)",
        "StartWindowMinutes": 60,
        "CopyActions": [
          {
            "DestinationBackupVaultArn": "arn:aws:backup:us-west-2:123456789012:backup-vault:MyBackupVault",
            "Lifecycle": { "DeleteAfterDays": 90 }
          }
        ]
      }
    ]
  }
}

6) Security and Governance

• Encryption: Use KMS for data at rest; TLS for data in transit.
• Access Control: IAM roles for services; least privilege policies.
• Audit Logging: Enable CloudTrail for API call tracking.

{
  "PolicyName": "RDSPolicy",
  "PolicyDocument": {
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "rds:DescribeDBInstances",
          "rds:CreateDBSnapshot"
        ],
        "Resource": "arn:aws:rds:us-east-1:123456789012:db:my-rds"
      }
    ]
  }
}

7) Monitoring and Observability

Use CloudWatch for monitoring and alarming to detect and respond to issues quickly.

• Metrics: Track latency, error rates, and instance health.
• Alarms: Set thresholds for failover triggers or performance degradation.
• Logs: Centralize logs for troubleshooting and auditing.

{
  "AlarmName": "HighLatencyAlarm",
  "MetricName": "Latency",
  "Namespace": "AWS/ELB",
  "Threshold": 1.0,
  "ComparisonOperator": "GreaterThanThreshold",
  "Period": 300,
  "EvaluationPeriods": 2,
  "Statistic": "Average",
  "AlarmActions": ["arn:aws:sns:us-east-1:123456789012:my-sns-topic"]
}

8) Testing and Validation

Regularly test HA/DR setups to ensure reliability.

• Failover Testing: Simulate AZ or region failures.
• Recovery Drills: Restore from backups and validate RTO/RPO.
• Chaos Engineering: Inject failures to test resilience.

# Example: Lambda function to simulate AZ failure
import boto3
def lambda_handler(event, context):
    ec2 = boto3.client('ec2')
    ec2.terminate_instances(InstanceIds=['i-1234567890abcdef0'])
    return {'statusCode': 200, 'body': 'Instance terminated'}

9) Example: Healthcare Platform HA/DR

A healthcare platform requires HIPAA compliance and near-zero downtime. The architecture includes:

• Multi-AZ RDS with automated backups for patient data.
• S3 with CRR for encrypted medical records.
• Route 53 for DNS failover to a secondary region.
• Auto Scaling Groups with health checks for application servers.

This ensures compliance, meets SLAs, and maintains operations during outages.

10) 30–60–90 Roadmap

Days 0–30:
• Deploy ALB and Auto Scaling across two AZs.
• Configure RDS Multi-AZ and S3 CRR.
• Set up Route 53 health checks and failover.

Days 31–60:
• Automate backups with AWS Backup.
• Implement CloudWatch alarms and logging.
• Test failover with simulated AZ failure.

Days 61–90:
• Conduct full DR drill with cross-region recovery.
• Refine RTO/RPO targets based on test results.
• Document and train team on HA/DR processes.

11) FAQ

Q: How do I balance cost and HA/DR?
A: Start with multi-AZ for HA; add cross-region replication only for critical data to optimize costs.

Q: How often should I test failover?
A: Quarterly for critical systems; monthly for high-risk environments.

Q: Can I use this for on-premises systems?
A: Yes, but you’ll need equivalent load balancing, replication, and failover mechanisms tailored to your infrastructure.