AWS Application - Whitepaper

SQS:

Messages oriented api. Useful in loosely coupled architecture ie. decouple the cloud application components.

Pull based not push based .

Easing message management components.

Messages are up to 256 KB, billed in 64k chunk

Messages can be kept in queue from 1 m to 14 days, default 4 days.

Visibility Time Out: Duration for which message will be invisible in SQL after Reader picks up the message. Default visibility timeout is 30 secs. If the Job is processed within that time the messages will be deleted otherwise it will put in back in SQL for other Reader.
-We can create multiple jpb queue for different priority.Queue with higher priority are evaluated first.

An SQS queue can be used to create distributed/decoupled applications.

This may result same message being delivered twice. Max VTO is 12 hours.

Guarantees that messages will be processed at least once.
SQS Dead-Letter Queues:
Delivery Delay: default 0 sec max 15 mins. Messages delivered only after delivery delay.
SQS -ASG: Custom metric based on number of messages/# instances
 When the ReceiveCount for a message exceeds the maxReceiveCount for a queue, Amazon SQS moves the message to a dead-letter queue (with its original message ID).
Access key ID and request signature or X.509 certificate are authentication mechanisms provided to ensure that message stored in Amazon SQS queues are secured against unauthorized access

Web service that gives access to a message queue. A queue is a temporary repository for messages that are awaiting processing.

1.User upload an image to a websites->upload to S3à Trigger a lambda function à store information into SQS-àFleet of EC2 instancesà Poll  messages from SQS and then process it and store it to S3.

2. User book holiday tickets. User puts a query to a webservice with dare details. Ec2 instance get the query and put it to SQS queue.  Fleet of EC2 web service and poll the messages and check with all the airline services. And then send the response back to the service the user initially requested.

1. Standard: Generally FIFO, can have duplicates. 

2. FIFO: Guaranteed fifo behavior, no duplicates.You can add sequencing information if ordering needed as per they arrived. throughput limit 300 msg/se and 3000/s in batch

 - Every message sent to a FIFO queue requires a message group ID. It distributes on basis of groupId. For ordering 100 truck--> 100 GroupID.
 - If you don't need multiple ordered message groups, specify the same message group ID for all your       messages.
 - If your application can send messages with identical message bodies, you can modify your                     application to provide a unique message deduplication ID for each sent message.
- If your application sends messages with unique message bodies, you can enable content-based deduplication.
-For the same combination of event type and prefix(/images) only one S3 event tule can be added. Otherwise we can have many SQS queue , use fan-out.
S3 event SNS Topic- SQS Queue/Lambda
SQS Polling:

Long Polling. Does not return the response immediately if the queue is empty. Useful for cost savings .

Short Polling: Returns immediately even if the poll is empty.

SWS : Simple workflow Service:

-Task oriented api.

Eg: Warehouse, Amazon ordering, where human elements required.

 Execution workflow can last up to 1 year.

Task is assigned only once.
Starter, Decider and workers are three actors.

Keeps track of all the tasks in api.

SWF Actor:An application that can start a workflow, parallell work in the background.. Task are stored up to 1 year.
Task

Decider: control the flow of activity tasks in a workflow execution

Activity Worker:

                                                

SNS:

To send alerts/notification from cloud.
Create Topic.

Instantaneous/ Push notifications:  Multiple transport protocols/ email/by SMS messages/SQS, http end point

Topic: Multiple recipients/ subscriber. One topic can support deliveries through multiple endpoints types.
-In flight encrypytion using http/https.
SNS subscription protocol- http/https/email/email-jspn/AWS SQS/ AWS Lambda.

SNS Vs SQS

SNS : Push

SQS : Poll/Pulls based mostly by Ec2

SNS + SQS Fanout: For sending messages to multiple application.
SNS can not send messages to SQS FIFO queues (AWS limitatation).

Elastic Transcoder:

Media transcoder in the cloud. Converts media file from the original format to other format that will play on smartphones, tablets,PCs etc.

-Using Amazon Elastic Transcoder, Amazon S3, and Amazon CloudFront, you can store, transcode, and deliver your content.

API Gateways:

At a high level door to AWS environment.

Front Door for application to access data, business logic or back end service functionality, Ec2 apps,lambda,DynamoDB, other webapps.

We can throttle requests to any limit.

API Gateway Caching: To cache your end point response till TTL specified.

Same origin policy to prevent cross origin policy. CORS (cross origin resource sharing) is enabled to support serve data from remote source.

CORS is enforced by the client.

Log results to cloud watch.

Kinesis: (Alternative to Kafka)

To send your streaming data to AWS. Makes easy to analyze streaming data, provide ability to  build custom app as per business needs.
Kinesis consumers 'pull data'.

Kinesis Streams: (Real Time ~200ms) Data producers à (stream the data to )Kinesis  (made up of shards), 24 hours to 7 days  à Consumers (Ec2) will analysis the dataà Store data into RDS/non-RDS, S3            
1000 messages or  1 MB/s at 1 write per shard.2 MB/s at read shard.
Records are ordered per shard.
Put Record API ---->Kinesis Shard (based on hash value of message)
Choose a partition key which gives highly distributed to avoid hot shard/partition. all messages will go into only 1 shard.
Shard level odering.Suppose 100 trucks, 5 shards. Within each shards trucks are ordered.
ProvisionThroughputExceeded Exception happnes when sending more data than provisioned (MB/s TP/s).
Encryption in flight using https endpoint. At rest using KMS.
VPC endpoint available for kinesis.

Kinesis Data Firehouse: (Near Real Time ~60s)On the fly, Data has to be analyzed as its comes in, can have lambda function à output to some other place say Es2. It loads data into RedShift/ AWS S3/Elatic Search/ Splunk.
60 secs latency for non-full batches

Serverless data transformation using lamda.

No data persistence. Ingest data into persistence.

Kinetics Analytics: Analyze the data inside Kinesis streams and Firehouse and can analysis on fly and stores the data inside EC clusters/S3à Redshift. It uses SQL.

No data storage.
Auto scaling solution
- Begin with a CloudWatch alarm that monitors Kinesis Data Stream shard metrics. When a custom threshold of the alarm is reached, for example because the number of requests has grown, the alarm is fired. This firing sends a notification to an Application 

- Auto Scaling policy that responds based on the stated preference, scale up or down.
- Amazon Kinesis Scaling Utilities is not free of cost.

Amazon MQ: (Managed Apache MQ)
-Migrating your application using Migrating queue.

Web Indentity Federation:

From Web Id provider, facebook/google etc it gets an authentication code which they can trade for temporary AWS Sec credentials.

Amazon Cognito

Works as a webId provider.

Sign-up/in to your apps.

Work as an identity  broker between your  app and webId provider.

Synchronizes user data from multiple-devices.

Recommended for all mobile apps application AWS services.

Use Case:

Facebookàauthentication token àCognitoàgrant access to AWS env. And then execute lambda funcn, write to DB/S3.

Cognito User Pools: User directory to sign in /up for mobile/web apps.it is more like user id/pwd/registration

Cognito  Identity Pools: temp  AWS credential access to AWS. It’s the actual grant access/Roles/ Authorization access to resources.

AWS CloudFormation (Templating aws resources):

-stack

Scripting your cloud environment. Using json, create templates, deployed resources at scale.

Quick start is a bunch of templates to create complex env quickly.

-Based on template. Asks for stack, credential details, rollback condition.
-Can be used to spin up/replicate resources in another region during disaster recovery.
-Use a simple text file to model and provision, in an automated and secure manner, all the resources needed for your applications across all regions and accounts.

-With change sets, you can preview the changes AWS CloudFormation will make to your stack, and then decide whether to apply those changes.
 -You create a template that describes all the AWS resources that you want (like Amazon EC2 instances or Amazon RDS DB instances), and AWS CloudFormation takes care of provisioning and configuring those resources for you.
 When you use that template to create an AWS CloudFormation stack, AWS CloudFormation provisions the Auto Scaling group, load balancer, and database for you.
-An existing technical design can be trandlated into templates which  cloudfromation will use it later.
-Stack deletion  Retain policy: S3 bucket  retain, DB instance/Cluster: Snapshopt

-Use drift detection to identify stack resources to which configuration changes have been made outside of CloudFormation management.
CloudFormation only determines drift for property values that are explicitly set, either through the stack template or by specifying template parameters. This does not include default values for resource properties. To have CloudFormation track a resource property for purposes of determining drift, explicitly set the property value, even if you are setting it to the default value.

Elastic Beanstalk: A compute services. Application + version + env name

-To manage and deploy applications.

It's a way of quickly deploy and manage application to the cloud. Just upload the code don’t need to worry about infrastructure details.
-Used to create web server environments and worker environments,.

Base Configuration (Pre configuration platform): Php/java/docker/Ruby/Python

We can configure various  capacity provisioning, autoscaling group etc.

Elastic Beanstalk deploys your code to the platform version you selected to create an active application environment.
-Is a PaaS service that helps you to build and manage web applications

  Provides a variety of platforms on which you can build your applications.

 You design your web application to one of the available platforms.

Provides platforms for programming languages (Go, Java, Node.js, PHP, Python, Ruby), application servers (Tomcat, Passenger, Puma),

and Docker containers. Some platforms have multiple concurrently-supported versions.

Instance Metadata:

Used to get information about the instance., such as public ip

Curl 169.254/latest/userdata

169.254/latest/latest-metadata

Local-ipv4

Elastic File System:

Managed elastic NFS file system for use with AWS Cloud services and on-premises resources.

Built to scale on demand to petabytes without disrupting applications,

 growing and shrinking automatically as you add and remove files, eliminating the need

to provision and manage capacity to accommodate growth.
- Provides file level storage.

AWS Storage Gateway:
 - A volume gateway provides cloud-backed storage volumes that you can mount as Internet Small Computer System Interface (iSCSI) devices from your on-premises application servers.
Volume Gateway
Cached volumes – You store your data in Amazon Simple Storage Service (Amazon S3) and retain a copy of frequently accessed data subsets locally

Stored volumes – If you need low-latency access to your entire dataset, first configure your on-premises gateway to store all your data locally. Then asynchronously back up point-in-time snapshots of this data to Amazon S3.

Amazon Athena:
 Is serverless and an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL.
Athena is easy to use. Simply point to your data in Amazon S3, define the schema, and start querying using standard SQL.
- With Athena, there’s no need for complex ETL jobs to prepare your data for analysis. This makes it easy for anyone with SQL skills to quickly analyze large-scale datasets
Amazon Quick Sight:
-Useful for data visualization of logs and all that.
Amazon EMR:
A web service that enables businesses, researchers, data analysts, and developers to easily and cost-effectively process vast amounts of data.
-Utilizes a hosted Hadoop framework running on Amazon EC2 and Amazon S3.
Also support Apache Spark, HBase, Presto and Flink.
Most commonly used for log analysis, financial analysis, or extract, translate and loading (ETL) activities.

Amazon SWS: Simple Workflow Service:
-To state-tracker and task coordimator in the cloud.
- eg. can be used in online ordering. Tasks and Actor.
-Retention period is 1 year,  Task oriented api, cannot have multiple deliveries.

AWS KMS:
 -To create and control the encryption keys used to encrypt your data.
-  The customer master keys that you create in AWS KMS are protected by hardware security modules (HSMs

ECS: (Batch Jobs)
Docker containers are particularly suited for batch job workloads.
You can package your batch processing application into a Docker image so that you can deploy it anywhere, such as in an Amazon ECS task.
You define the IAM role to use in your task definitions, or you can use a taskRoleArn override when running a task manually with the RunTask API operation.

Fargate: ECS cluster does not require  EC2 instances. Just defaine tasks that we want onto the services.

Stateless applications- EC2, Lambda

Managed Blockchain network  :
Resource endpoint: ResourceID.MemberID.NetworkID.managedblockchain.AWSRegion.amazonaws.com:PortNumber

Serverless design:
AWS Lambda, DynamoDB,Cognito, API gateway/S3/ SQS/SNS/Kinesis fireshouse/Aurora serverless/step function/Fargate.
Dcoker is not for AWS Lambda, It is for ECS/Fargate.
AWS Limits: (Per region)
-Memory(128 MB-3008 MB)/Max execution time (15m). Env variables (4 KB), disk capacity 512 MB.concurrency 1000 (can be incresed).
Lambda@Edge:
To change/modify cloudFront Request/Response/User view

API gateaway:
-Edge Optimized (for global cleint)
-Regional (within the same region)
-Has support of cache response as well
-Private - Only be accessed from VPC using interface endpoint. Use resource policy for acceess.
IAM: great for  users/role within aws account.
Seg v4 capabilties: IAM credential is in the header.
Lambda Authorizer (Custom): option to cache result authorizer. handles both authentication and authorizer.
Cognito User Pool: For Authentication. no code is required and it gived authentication token.
Cognito is integrated with STS as well.
CloudFornt Signed Cookies Canned Policy (single file applicable )Custom Policy (can applied to multiple files)
CloudFront geo restriction feature to restrict spcecifc country user.
MicroServices: ECS, AWS Lambda, API gateway.
AWS OPS: Stack, chef/
Elastic Fabric Adapter (EFA) 
you get the application performance of on-premises HPC clusters with the on-demand elasticity and flexibility of the AWS cloud.

Cloudfront: You can use an OAI to restrict access to content in Amazon S3 but not on EC2 or ELB.

Lambda:

To enable your Lambda function to access resources inside your private VPC, you must provide additional VPC-specific configuration information that includes VPC subnet IDs and security group IDs. AWS Lambda uses this information to set up elastic network interfaces (ENIs) that enable your function.