AWS VPC !!!!!!

VPC tidbits 

Resembles our own datacenter like our own on premisis private corporate network.

Has private and public subnets and a scalable architecture. Ability to extend corporate/on-premise network to the cloud as if it was part of our network (using VPN).

Benefits of VPC :

• Launch instances into subnet
• Define custom CIDR(IP address range) inside each subnet.
• Configure route tables between subnets
• Configure internet gateways and attach them to subnets
• Create a layered network of resources.
• Security settings to protect cloud resources.
• Expand network in cloud with VPN/tunnelling.
• Layered Security
• Instance security group
• Network ACL at subnet level

Default VPC : Has an internet gateway attached Each instance has a default private and public IP address(defined on subnet settings).Public IP address are attached to an Elastic Network Interface that has a private IP address associated with it. This is called NATing. i.e. routing from public IP to private IP and eventually to the instance.

VPC peering : Direct network route between one VPC and another. Allows sharing of resources as if they were on same network. 

• VPC peering can happen between other AWS accounts and other VPCs within the same region.
• VPC peering cannot occur between two regions.
• Scenarios -
• Peering two VPCs - Multiple VPCs of a company linked under one private network.
• Peering to a VPC - Multiple VPCs can connect with a central VPC but cannot communicate with each other. The only communication that can occur is between the peered VPC and the primary. Eg. If Third party wanted sharing of resource like a File share etc.

VPC Limits :

• 5 VPC per region
• 5 internet gateways (this is equal to per VPC as we can only have one internet gateway per VPC)
• 50 customer gateways per region.
• 50 VPN connections per region
• 200 route tables per region/ 50 entries per route table
• 5 elastic IP address
• 500 security groups per VPC
• 50 rules per security group (Remember security groups act at VPC level)

All instances launched into the default SG can communicate with each other.

By default each instance has a route to each other instance in the subnet which it is created in.

The difference between Public and Private subnet is having an internet gateway and not having an internet gateway.

In order to connect to the internet an instance must have an internet gateway and also a public IP.

For high availability you should have multiple instances in multiple AZs. And you can use Load Balancer to achieve the high availability.

Unless required, never expose instances to the internet by assigning it with public IP address.

Some takeaways :

• Each subnet must be associated with a route table.
• By default all subnets traffic is allowed to each other available subnet within our VPC which is called local route.
• We cannot modify or delete local route.
• Best practice is to leave the default route and create new routes as needed.
• To enable access to/from internet for instances in a VPC subnet, we must attach internet gateway to VPC and also ensure that the subnet's route table should point to the internet gateway.
• In order to download software and updates to a private instance one can use a NAT instance. 
• NAT instance must be created in a public subnet.
• NAT instance must be part of private subnet's route table.

AWS provides DNS servers for VPCs so each instance has a host name.

VPC Security

Security Groups

• Operate at instance level
• Supports allow rules only
• Is stateful - so return traffic requests are allowed regardless of rules.
• Evaluates all rules before deciding to allow traffic

Network ACL

• Operates at subnet level
• Stateless - so return traffic must be explicitly allowed through an outbound rule.
• Processes rules in number order. So if a traffic is allowed at a higher number rule and denied at a lower number rule then the traffic will be denied.
• Applies at network level - so one deny will block all traffic for all instances.
• Deny all is the default rule for NACL
• Example : in the below snap, the DENY on port 80 wont make any difference because all traffic is allowed at a lower level.

NAT instance - There are NAT specific AMIs present in AWS marketplace to create a NAT instance.
NAT instance must be inside public subnet and it must have a public/Elastic IP associated to it. Ports (HTTP/HTTPS) must be open. On NAT instance the source/destination check must be disabled ; This will allow traffic from our private subnet to the internet via NAT.

Extending VPC to on premises must always be controlled and encrypted. VPN allows one subnet from one geographic location to extend to another geographic location. This means it can extend to on-premisis network too. SO we would connect to all resources internally without need for public/private addresses.
VPG (Virtual Private Gateway) - It acts like a connector on the VPC side of the VPN connection.
VPG is conected to VPC. and the VPN is associated with the customer gateway creating the endpoint. A Customer Gateway acts as a connector on the on-premises side of VPN. This is where you configure the public IP address for the on-premise network.

Both VPG and the Customer Gateway are required to establish a VPN connection.

Only one VPG is attached to a VPC.

Alternate methods for deploying a VPN to AWS - Configure an OpenVPN instance which lives in a public subnet and we can connect to the public IP using the OpenVPN client.
But one must also consider high availability to this instance. What is this instance fails ? A single point of failure ? So High Availability must be applied to this instance.

VPC peering

You cannot do a VPC peering between two VPCs in different region. Both VPCs should reside in same region. VPCs are not AZ specific.

When peering you need to make sure that each VPC should have different CIDR ranges.
If one VPC is connected to second VPC, and second VPC is connected to third VPC then first and third VPC cannot communicate to each other unless explicitly connected.

You can create peering connections from VPC-VPC, VPC-Subnet, VPC-Instance too.

Limitations : You cannot just extend VPC connections from a Cloud network to an on-prem network. You still need a VPN and a VPGateway inorder to connect.
Even you cannot access an S3 end point which is present in second VPC from a first VPC (inspite of having a peering connection)

Amazon RDS

RDS Essentials

RDS is fully managed relational DB in cloud. Does not allow access to underlying OS. Can connect to DB server itself as normal ex. cmd line. Ability to provision/resize hardware on demand. Multi-AZ deployments. Read Replicas for reading efficiently (MySQL/PostgreSQL/Aurora).

Supported DB engines :

• MySQL
• PostgreSQL
• Oracle
• Microsoft SQL Server
• Aurora

RDS Disk space Minimum - 5 GB and Maximum - 3 TB

SSD gives burstable performance. Provisioned IOPS gives the specified performance.

Benefits of running our own RDS instance instead of a Relational DB hosted on an EC2 instance,

• Automatic minor updates
• Automatic backups
• No need to manage OS
• Multi-AZ
• Automatic recovery in case of failure

Automatic AZ fail-over - Multi AZ sync/replication of data to backup instances located in another availability zone in same region, In event of a failure AWS will automatically change the cname for that RDS instance to the standby/backup instance. In following cases the failover takes place,

• Availability zone outage
• Primary DB instance fails
• Instance server type changed
• Manual failover initiated
• Updating software
• Backups taken against standby instance to reduce I/O freezes and slow down if multi-AZ is enabled

RDS Backups : Automated point in time backups are provided by AWS. Eg. MySQL requires InnoDB for reliable backups.

Read Replicas : Replicas of original instance used for Read only

• MySQL/postgreSQL/Aurora support
• Uses native replication on by MySQL/PostgreSQL
• Can be created from other read replicas
• Read replicas are only supported by InnoDB MySQL storage engine.
• Can offload tasks off of production DB
• Can promote a read replica to a primary instance
• MySQL
• Replicate for importing/exporting to RDS
• Can replicate across regions

When to use Read replicas ???

• High non cached DB read traffic
• Running business function such as data warehousing
• Importing/Exporting data in RDS
• Rebuilding indexes
• Can promote Read Replica to primary

RDS integrates with CloudWatch and also many events can be notified.

As compared to EC2, here AWS has access to underlying OS it can manage many metrics like memory, swap usage etc. In case of EC2 AWS doesn't have access to OS, so we have to install the custom scripts for monitoring other factors.

RDS Subnet Groups

In order for high availability an RDS instance must be in different AZs. Here comes the part of Subnet Groups. We create an RDS instance, it automatically creates a standby in different AZ as part of Multi-AZ failover.

RDS Security Groups

RDS instances can share same SGs as that of EC2s or different. Same port rules n source rules apply to RDS SGs.

AWS EC2 stuff !

EC2 Basics

Instance types -

1. T2 - Burstable Performance Instances
2. M3 - Nice Balance
3. C4 - Compute Optimized
4. R2 - Memory Optimized
5. G2 - GPU Optimized
6. I2 - Storage Optimized
7. EBS Optimized

EC2 instance sizes also decide the network throughput capacity limitations. Like a Micro instance cannot have same network throughput as a Medium instance.

Instance Storages -

1. Instance-store volume (ephemeral data) : Data would be erased if instance is stopped. But if instance is rebooted then the data is retained. 
2. EBS backed volumes : Network attached storage. Remain persistant to life of instance. 1 instance can have multiple EBS volumes. 1 EBS volume can be attached to only 1 instance.

EBS have IOPS (input/output operations). IOP is 256KB or smaller. So a 512KB operation would count as 2 IOPS.

We can provision upto 20,000 IOPS on a EBS instance.

EBS types - 

• General Purpose SSD : Commonly used as "root" volume.
• Used on smaller instances.
• 3 IOPS/GiB (burstable performance) - credits can be accrued in this case when the IOPS is not utilized.
• Volume size of 1 GiB to 16 TiB
• Provisioned IOPS : Used for Mission critical applications.
• Large DB workloads.
• Volume size of 4GiB to 16TiB
• Can provision upto 20000 IOPS
• Magnetic :
• Low Storage cost
• Used in cases where performance is not important/ Data infrequently accessed
• Volume size of 1GiB to 1024 GiB

Prewarming of volumes - Sometimes we get already used volumes. AWS runs through erasing protocol when provisioning such volumes for us. That decreases the IOPS performance. At such times we can run some commands like Linux DD which will touch each block on the storage and pre warm it thereby giving us an improved performance.

EB2 Snapshots - Snapshots are incremental in nature. They store only changes since the most recent snapshot thus reducing costs and only paying for the incremental storages. Even if original snapshot is deleted, we have the newer snapshots with us. So all the data is there.

Snapshots are stored on S3 buckets. But we cannot go and directly list them.

Snapshots taken on EBS volumes will degrade the performance for that time.

Remember : When an EC2 instance in stopped, we are not paying for that instance. We are paying only for storage.

EC2 bootstrapping : Writing Bash scripts which will be executed during the instance provisioning.

This script is also called "User-Data/Cloud-init" and can be accessed from within the instance by going to http://169.254.169.254/latest/meta-data or http://169.254.169.254/latest/user-data

EC2-classic instances are not part of VPCs. They are assigned a public IP address and cname. Also receives private IP address but isnt part of VPC. Also the private IP is lost once the EC2-Classic instance is rebooted.

Security Groups are used as firewalls for EC2 instances. An instance can belong to multiple security groups. Security Groups can reference themselves as "source" traffic in firewall rules.

Action Item - Go and read the EC2 service limits on AWS console ! Its Important !!!

We can have 40 total instances out of which 20 is the max running instance limit.
EC2-VPC Elastic IPs has limit of 5 IPs per VPC.
Rules per VPC SG - 50
VPCs per env - 5
Security Groups per VPC - 100

The longer we purchase a reserved instance the lesser we pay per hour.

While creating EC2 instance, you can open SSH port with appropriate source in order to log into it :)

Note : Private IP addresses are persistance when EC2 instances are shutdown. But Public IP addresses are not persistent. They are lost once an instance is shut down.

There are two types of subnets in a VPC - Private and public.

An instance in the private/public subnet needs a public IP address and an Internet gateway attached to the subnet in order to connect to the internet.

Elastic IP addresses are always convenient because if any instance becomes unhealthy or unresponsive or needs to be removed them we can always detach the Elastic IP address and attach it to any other instance. For example you can have two NAT instances and can use one Elastic IP address which can act as a failover to second NAT instance.

Security Groups

We can have upto 500 security groups per VPC.
Each Security Group can have up to 50 rules.
We can assign 5 Security Groups per EC2 neteork interface
An instance can belong to multiple Security Groups.
Remember all Security Groups' rules have an DENY  by default. We cannot create deny rules.
Remember responses to outbound/inbound are stateful i.e. If inbound traffic at port 80 is open then outbound traffic over port 80 will be allowed even if port 80 is not open for outbound.
One can change SG (Security Group) of an instance but isnt same for EC2-Classic.

The "default" SG has all ports open. Instance which are part of default SG can communicate with each other in the default SG. i.e. there is an inbound rule which allows all traffic from the "default" source.

Bastion Host - A system identified as a critical strong point which can be used as a gateway to connect to other instances. One can ssh into private resources without using a VPN. A Bastion host may have additional security/additional software installed for further security tightening.

Monitoring on EC2

Types of Status Checks :

• System Status Check - Loss of Network connectivity, loss of system power, physical host problems. Solution - Stopping and starting an instance will start it on different physical hardware device.
• Instance Status Checks - Corrupted file system, failed system status checks,  Exhausted memory. Generally reboot or rebuilding the AMI solves the issue.

CloudWatch Alarms : By default CloudWatch will automatically monitor metrics that can be monitored on the host level. If we wanna look at memory utilization we have to use a perl script provided by amazon 

• Basic level monitoring - Data is available automatically in 5 minute periods at no charge.
• Detailed level monitoring - Data is available in 1 minute periods.

OS level metrics that require a third party script to be installed,

• Memory utilization, memory used, memory available.
• Disk swap utilization.
• Disk space utilization, disk space used, disk space available.

You can also monitor CPU credits related to T2 micro instances which can acquire credits for underutilized CPU utilization.

EC2 Placement Groups

Placement groups - Cluster of instances within same availability zones. Have low latency 10 Gbps connection between them. Used for instances which require extremely low latency. Physically close they are placed.

If instance in placement group is stopped, AWS is gonna try to place it physically as close as possible to the placement group. Sometimes it can happen that later instances added will result in insufficient capacity error which can be resolved by stopping and starting all instances again.

Troubleshooting - Instances not originally placed into a placement group cannot be moved into placement group. Placement groups cannot be merged together. Placement groups cannot span multiple AZs. Placement groups name must be unique within your own AWS account. Placement groups can be connected. instances must have 10 gigabit network speeds in order to take advantage of placement groups.

Serving traffic to private web servers

An ELB should have at least 2 subnets.

Connection draining in ELB - Before a LB de-registers an unhealthy instance , it will wait for a certain amount of time before draining current connections.

AWS S3

S3 essentials

Static files. Set of name-key pairs. Buckets are grouping of information and have sub name spaces.

Busket names must be Unique across AWS.

S3 provides unlimited storage. 11 nines durability. 99.99% availability (Synced to multiple AZs)

Objects can be as small as 0 Bytes and as Large as 5TB.

Bucket limitations - 100 buckets per account at a time. Bucket ownership cannot be transferred once created. 

S3 security - All buckets/objects are private by default. ACLs for sharing S3 buckets with 2 different acounts. There are different bucket policies too(granting access to anonymous user, restricting access off of IP address, Restricting access based off HTTP referrer). Public content can be downloaded via URL. Signed URLs uses a key to create a unique URL; Can be done with API; time restrictions can be applied too.

S3 also has server side encryption.

S3 can be used to host static websites using route 53. S3 can act as an origin to the CloudFront CDN. Multi part upload facility available. Required for objects 5GB or larger. But suggested use is 100 MB and larger.

An uploaded object is synced across all AZs within that region.

Eventual consistency - If you upload an object and instantly make a read request, then your new object will be available.

S3 uses - Hosting static files, Origin for CloudFront CDN, Hosting static websites. Fileshare for networks, Backing up/Archiving (AWS Storage Gateway)

S3 Event Notification - events can be like RRSObjectLost (for recreation of lost RRS objects)

Events can be sent to SNS, LAmbda, SQS queue.

AWS IAM (Identity and Access Management)

IAM

Can federate with SAML providers auch as active directory for temporary and single sign on access.

MFA can be managed. Also IAM provides pre-built policy templates to users and groups.

Users & Groups

Groups - Assign permission policies to more than one user at a time.

Users - Best policy is to use IAM users. By default an explicit deny always overrides an allow. By default an user who has an explicit deny on all will need to have an allow policy to access something particular. Unless an allowed policy is applied, the user will not have access to anything.

Roles

Role can be created for EC2 instances or other accounts to assume some permissions. Can be temporarily granted too.

Best Practices for New Accounts

All resources in Amazon have a specific resource name. called ARN.

User can belong to multiple groups, multiple policies can be applied to a user. Now remember about the Explicit Deny :)

API Keys and Roles

EC2 instance can assume only 1 IAM role at a time. And the role can be applied only while creating the instance.

IAM Policies

We can create our own policies too. We can have policy versions too. One can also simulate policies.

IAM event logging with CloudTrail

CloudTrail service stores log into S3 bucket. We can also create SNS for each log file delivery.

One can create SNS topic. but after that one needs to create a subscription so that one can send notifications. Like one can create an Email subscription for a topic.

AWS CSA Prep 3

Storage services overview 

S3 (Simple Storage Service) - It is an unlimited object storage (Key-Value). Can be used for static website hosting too (needs coupling with Route 53). Designed for 11 nines (99.999999999%) durability and 99.99 % availability. Charges are based per Gib storage as well as the data sent out of the region. Same region data transfer does not cost. The more storage used, the less it will cost.

S3 objects can be encrypted too and HTTPS end point security can also be applied.

Remember Bucket names should be unique across all AWS. U-N-I-Q-U-E !

RRS - Reduced redundancy storage  : has 99.99 % durability vs "eleven nines" for standard storage type. Can be used for the data which can be reproduced easily. Which can be recreated easily.

Lifecycle policies & object versioning - We can have unlimited versions if enabled on S3. We can archive\backup older versions into Amazon Glacier.

Now what is Amazon Glacier - Its an archival storage. Used for data not frequently accessed. Takes several hours to check in and check out. 0.1/gig per month.

Amazon Storage Gateway - This is the connection between local data center to cloud services like Amazon S3. Has two types,

• Gateway-Cached Volumes - It will create a storage volume & mount as iSCSI device on premises. It will then store the data to Amazon S3 but will cache frequently accessed data on premises. To have faster access time.
• Gateway-Stored Volumes - Stores all data locally and backs up the data as incremental backups on S3.

Amazon Import/Export - Users can physically snail mail large amount of data to Amazon. Amazon will upload it to cloud within one day of receiving the data.

Database services

Amazon RDS - Fully managed DB service from Amazon. No access to underlying OS allowed. Patching etc is all managed by AWS.

DBs supported by RDS -

1. MySQL
2. PostgreSQL
3. Oracle
4. MS SQL server
5. Aurora

Aurora - Forked from MySQL by Amazon. Has 5 times better performance than MySQL with lower cost.

Amazon ElastiCache - In-memory cache for high performance DB queries. Caches results of queries etc. Application needs to be built to work with either Redis/Memcached.

Amazon DyanoDB - NoSQL fully managed service. Fully managed by AWS. Built as fault tolerant so in backend it syncs data in all AZs for the region. Easily integrates with Elastic MapReduce.

Amazon RedShift - Used as petabyte scale data warehousing service.

Analytics

Amazon Elastic MapReduce - Spins EC2 instances with Hadoop clusters. We have access to underlying OS.

App Services

Amazon SWF (Simple Work Flow) - used for work flow executions. Control panel to monitor task work flow. Scalable parallel EC2 processing. Service can be used with on-premises servers too. Guarantees execution of work flow.

Amazon SQS (Simple Queue Service) - Similar to Azure queues. Guarantees delivery of at least 1 message but not guarantees no duplicates. 

Amazon SNS (Simple Notification Service) - Co-ordinates delivery of messages to specific end points. Endpoints can be SQS, Email, SMS, HTTPS, Applicatoin etc.

Deployment Services

Amazon EB (Elastic Beanstalk) - Deploy complete app environment automatically. Support for docker containers.

Amazon CloudFormation - Allows us to code infra and deploy resources based off a pre-build template. Good for disaster recovery. We can even version control our AWS infrastructure.

Management Services

IAM - Identity Access Management - Manage permissions to AWS resources. Resource level\API call permissions can be managed.

CloudTrail - Logs all API calls made to AWS.

CloudWatch - Monitor services like EC2. Provides centralized logging for performance metrics etc. Is heavily used in AutoScaling.

Directory services - Can connect on-premise Microsoft Active Directory with AD connector. Also has ability to setup and operate new directory.

AWS CSA Prep 2

Compute & Networking :

EC2 – Elastic Compute Cloud provides VMs. Servers you call ’em. You can always customize them like processors, memory, volumes, n\w throughput. The pricing models for EC2 are,

• Reserved instances – This option is used when you know that you wanna use those web servers 24*7 and would want those instances up and running given at any point in time.
• On-Demand instance – Spin up when needed and then spin down when not needed.
• Spot-Instances – These are non-production instances which are unreliable as AWS can take ’em back when they feel so. Has to be bidden.

Elastic Load Balancing – AWS’ load balancer to distribute traffic to different instances in different AZs. Used in Auto-scaling and Fault tolerance.

Route 53 – Doman management service. To direct traffic from domain to EC2 instances.

AMI – Amazon machine images. If you don’t know what Machine Images are, go home.

Instance Store-backed instance (Ephemeral storage) – Storage provided by EC2 instances. This is wiped out when the instance is turned off.

EBS backed instance – Block storage attached to EC2 instances. Used to backup snapshots. Can provision IOPS\ optimized EBS instances to help traffic between instance and EBS volume. Min size – 1 Gib Max size – 16 Tib.  EBS volumes cannot be attached to instances in different AZs. Can be attached to only one instance at a time. Point in time snapshots a can be taken.

User responsibilities regarding EC2 instances – Security groups, Firewalls, EBS encryption available only on M3 or larger. Any instance smaller than M3 should not use EBS encryption as its a resource hog. Applying SSL cert to ELB.

AWS responsibilities – DDOS protection, Port scanning is not allowed in our own environment too. Ingress network filtering.

VPC – Isolation of resources. Internal communication allowed between resources. But for inter-VPC communication peering should be done. VPC is free of cost, yay !

There is something called EC2-Classic which is a deprecated service by AWS. They dont belong to VPCs. They have been discontinued from Dec 2013.

Route 53 – DNS hosting solution. Register\Transfer domains. can be used to route http://www.xyz.comto a CloudFront distribution\ELB\EC2\RDS etc Route 53 also is used to manage internal DNS inside VPCs. Can be used for Latency\GEO\Failover routing.

AWS Certified Solutions Architect Prep Part 1

Okay so I will list down all the concepts that you will need to clear AWS CSA examination. I may be brief but I will just jolt down the points one needs to study. Here we go.

Some basic concepts :

AWS has different AZs known as availability zone which are geographically separated. They are data centers which are separated Geo wise. AZs can also help store data in the specific region. Not all AWS services have to be AZ specific like the IAM service.

CloudFront is a CDN by AWS. Caters with fast static file delivery. These files are stored on hundreds to AWS edge locations and can cater as per user affinity.

Scalability is cloud’s ability to expand\contract as per demand. Why would you pay for 10 servers when you are using 10 servers only at a peak time and like 4 servers at non-peak time. Thats where scalability comes into play. Elasticity is kind of same : scale up and scale down as per need as per work load. Now S3 is highly fault tolerant but EC2 is not. There are two types of scaling :

Proactive cycle scaling : scaling up at peak time.

Proactive event scaling : scaling up with anticipated demand.

Auto scaling : based on metrics like CPU\Network utilization.

Fault tolerance is ability of our system to recover\continue operating in event of failures.Fault tolerance can make use of auto-scaling, Route 53, AZ, Multiple regions. When designing fault tolerant operations, your aim should be to design an App which is designed with failures in mind.

There are many AWS services primarily grouped as,

Compute & Networking, Storange and Content Delivery, DB, Analytics, App services, Deployment services, Management services.

My Experiments With Cloud

Hey all, This would be my nth attempt at writing a blog. Most likely I drew inspiration from movies where stupid people like me write blogs to pacify the anxiety of life. Hell this ain't gonna be a life science blog, so without much ado lemme tell you what this blog will be about. It will be about my small experiments with recent technologies. Mind me, I am a software guy. And Cloud\DevOps has got my attention.
Even though I might sound like talking to a third person in my posts, the blog is only for my own personal convenience to remember highlights of the work I do. There might be random posts which you may find lurking around, dont blame me, you are reading at your own risk ;)

So here we go.