___ # Tags #aws #certifications #cloud #unfinished # Helpful Docs # Notes #### Availability Zones and Regions - Availability zone is a data center, with redundant services like power, networking, and connectivity - Region is a geographical area with 2 or more availability zones within it - Edge locations are used for caching content, usually involves CloudFront which is Amazon's CDN #### In the Cloud, Who Owns What? - The **shared responsibility model** ![[Pasted image 20230501165100.png]] - Ask yourself - "Can I do this myself in the AWS management console?", if yes, then you are likely responsible #### Computer, Storage, DBs, Networking - Compute is the processing power to build whatever you are trying to build - EC2, virtual machines - Lambda (serverless) - Elastic Beanstalk, automate deployments - Storage, disks in the cloud - S3, simple storage service - EBS, elastic block store - EFS, elastic file service - FSx - Storage gateway - Databases, think of it like a spreadsheet - RDS, relational data service - DynamoDB, a non-relational database - Redshift, a DB warehousing technology - Networking - VPCs - Direct Connect - Route 53 - API Gateway - AWS Global Accelerator #### Well-architected Framework - [Link](https://aws.amazon.com/architecture/well-architected/?wa-lens-whitepapers.sort-by=item.additionalFields.sortDate&wa-lens-whitepapers.sort-order=desc&wa-guidance-whitepapers.sort-by=item.additionalFields.sortDate&wa-guidance-whitepapers.sort-order=desc) to the whitepaper - Read this whitepaper before the exam - Operational excellence - Performance efficiency - Security - Cost optimization - Reliability - Sustainability #### Securing the Root Account - Username > My security credentials - Turn on MFA - Create admins group - Create accounts for admins - Add users to the admin group #### Control Actions with IAM Policies - AWS policies are in JSON format - IAM policy docs can be assigned to groups, users, and roles - Normally you don't assign them to a user - IAM does not use region selection, actions happen globally - AWS has a lot of managed policies and it's often preferable to use their managed policies #### Permanent IAM Credentials - Users, someone who exists - Groups, functions like admin or dev - Roles, used internally for AWS, allows something in AWS to access something else in AWS - Best practice to **inherit** permissions from groups - Least privilege, the minimum amount of permissions needed to do the job - When creating groups you can add tags for key-value pairs - Access key ID and secret access keys are not the same as usernames and passwords - One-time-view only, otherwise you have to regenerate them #### s3 Overview - Simple storage service - Object storage - Can upload any file type to s3, place to store static files - Unlimited storage, up to 5TB in size - s3 buckets are basically a folder within s3 - All AWS accounts share s3 namespace - You couldn't use "testbucket" as a name because that was likely taken years ago - When you create a bucket the URL will always be `https://bucketname.s3.REGION.amazonaws.com/key-name` - When uploading a file - your browser will receive an HTTP 200 code - Key, the name of the object you uploaded or an object on the s3 bucket - Value, the actual data itself - Metadata, data about the object such as last-modified, content-type, etc - Version ID, updated versions of the same file - s3 standard is designed for high availability and durability of the data - 11 9's, 99.99999999999% - 99.99% availability - Good for frequent access and most workloads - websites, big data analytics, gaming applications - Server-side encryption can be enabled by default - ACLs can be attached to objects within an s3 buckets - You can enable buckets to allow certain actions such as PUT but not DELETE on an s3 bucket - Read-after-write, anytime an object is overwritten, all READ requests that come after automatically get the latest version of the object #### Securing s3 Buckets - ACLs work on individual objects within the bucket - Bucket policies work on the entire bucket #### Static Sites on s3 - Basic static sites like markdown, html, etc are perfect for s3 - Websites requiring a DB can't be hosted - s3 scales automatically based on load #### Versioning in s3 - Version control with your objects within s3 - All versions of your objects can be stored in the same s3 - Once enabled cannot be disabled, only paused - Supports MFA to delete objects - Can be enabled with lifecycle rules - When you allow all objects in s3 to be public that does not apply to previous versions of your objects - When you delete an object with versioning turned on it will put a delete marker over the top of the object, you can restore your previous versioned objects by deleting the delete marker #### s3 Storage Classes - s3 standard-infrequent access - Rapid access - Pay to access the data, low price per-gb of storage - Use-cases, great for backups, long term storage, disaster recovery - s3 one zone-infrequent access - Data is stored in one AZ - Costs 20% less than s3-ia - Great for non-critical data - s3 intelligent tiering - Moves data to the most cost-effective tier based on access patterns - Glacier options - Cheap storage - Archiving data - Pay for access - Glacier instant retrieval - Long-term data archiving with instant retrieval time - Glacier flexible retrieval - Great for archiving that doesn't require immediate access to the data - Can retrieve large data sets at no cost and can be used for backup or disaster recovery - Takes minutes to 12 hours to access the data - Glacier deep archive - Cheapest storage - Designed for customers with long data retention reqs such as compliance or regulatory bodies - 12-48 hours to retrieve data ![[Pasted image 20230507145426.png]] #### s3 Lifecycle Management - 30d default management for s3 standard - After 30d it goes to s3-ia, then glacier after 90d - Can combine lifecycle management with versioning #### s3 Object Lock and Glacier Lock - s3 WORM - write once, read many - Helps prevent objects from being deleted or modified - Can be used to meet regulatory requirements - s3 Governance mode, users can't overwrite or delete an object version or alter the lock - Some users can still be given permissions to alter retention settings - s3 compliance mode, retention mode/period cannot be changed by anyone, more restrictive than governance mode - Retention period, protects objects for a fixed amount of time, when expired the object can be overwritten or deleted - Legal hold, object locks can allow you to place legal holds on a specific version of an object and prevents that version from being modified but does not associate to a retention period, can be placed and removed freely by users with the right permission - Glacier vault lock, allows for the deployment of compliance controls using the WORM model for s3 glacier vault, once locked the policy can no longer be changed #### Encrypting s3 Objects - Encryption in transit, sending objects between buckets - Uses SSL/TLS - HTTPS - Encryption at rest, server-side encryption - SSE-S3, s3-managed keys, aes 256 - SSE-KMS, AWS key management service-managed keys - SSE-C, customer provided keys - Encryption at rest, client-side encryption - Encrypting the files before they are uploaded to an s3 bucket - Checkbox in a console to enable server-side encryption through the console, you can also do it through a s3 bucket policy - `x-amz-server-side-encryption` is a parameter your browser can include when it initiates a PUT request to activate the encryption, this can be with `aes256`, or with `aws:kms` to indicate the type of encryption - You can use a bucket policy that denies any PUT request that does not include the desired encryption parameter #### Optimizing s3 Performance - Bucketname/folder1/subfolder1 - Prefixes in s3 are just folders - s3 bucket performance can be limited by prefixes - 3500 PUT/COPY/POST/DELETE and 5500 GET/HEAD requests per second, per prefix - The more prefixes you have, the more you can divide up that bandwidth - Spreading the READ requests across prefixes - KMS has limitations on s3 performance - Uploading a file invokes the `GenerateDataKey` call in the KMS API - Downloading a file will call `Decrypt` in the KMS API - Requests per second are region-specific but vary from 5500, 10000, 30000 - Multipart uploads - Files over 100mb it is recommended - Required for files over 5gb - Parallelize uploads by breaking files into parts and reassembling on the s3 bucket - Byte-range fetches - Specifying a byte-range to parallelize the downloads so if part of the download fails only the byte-range that failed will be affected #### s3 Replication for Backing up Data - Replicate objects from one bucket to another - Objects in existing buckets are not replicated automatically - Delete markers are not replicated by default - Versioning needs to be turned on on your source and dest s3 buckets #### EC2 Overview - Elastic compute cloud - Pay for what you use - Pricing options - On-demand, pay by the hour, second depending on instance - Flexibility, testing a website, short-term investment - Reserved Instance, 1 or 3 yrs, 72% discount on hourly charge - Predictable usage, capacity requirements, pay up front - Convertible RIs - Scheduled RIs, launched within a time window, something that is predictable - Spot, purchase unused capacity - Applications that have flexible start and end times, not good for websites - Use-cases, image rendering, genomic sequencing, trading engines - Dedicated, dedicated EC2, the most expensive - Useful for compliance, not for multi-tenant virtualization - Licensing - Can be purchased for on-demand - These can be reserved for greater savings - Calculator.aws can be used to price out your infrastructure with estimates on converting it to AWS #### AWS CLI - AWS Console is the UI - AWS CLI is the command line - You need to run `aws configure` to setup your AWS CLI - `aws s3 ls`, will list out all s3 buckets - `aws s3 mb s3://INSERTNAME`, will make an s3 bucket - Secret access key, only see it once, you have to make a new key in order to regenerate it #### Using Roles - An identity in IAM that has specific permissions, a role is similar to a user in that it has an AWS identity with permission policies - A role is designed to be assumed by anyone who needs to access it, temporary security credentials - Roles can allow cross-account access - Preferred option so you don't have to use hard-coding creds - You can attach and detach roles to existing resources #### Security Groups & Bootstrap Scripts - Virtual firewalls for EC2 instances - 0.0.0.0/0 to let everything in - Bootstrap scripts run when an instance first starts - Paste the script into the User Data part of the EC2 setup - Changes to security groups take effect immediately - Can have any number of security groups assigned to an EC2 #### EC2 Meta and User Data - Metadata is data about the data - You can query metadata about your EC2 instance using `curl http://169.254.169.254/latest/meta-data/local-ipv4` #### Networking with EC2 - ENI, elastic network int, good for daily driver use - Private IPv4, Public IPv4, MAC addr, Many IPv6 addrs, 1 or more security groups - Great for management networks, network and security appliances in your VPC, dual-homed instances, cheap and high availability - EN, enhanced networking, provides high performance - 10Gbps to 100Gbps, elastic network adapter (ena), and intel 82599 virtual function interface - EFA, elastic fabric adapter, good for throughput intensive applications like machine learning - Accelerates high performance computing, OS-bypass is a lot faster with much lower latency #### EC2 Placement Groups - Cluster, groups of instances in an AZ - Apps that need low latency and high network throughput - Spread, placed on distinct underlying hardware - Used for individual instances, for instance you wouldn't want your backups to be on the same hardware as your primary - Partition, each partition group has its own racks - When you have multiple instances that need to be on their own hardware or power sources #### Spot Instances and Spot Fleets - Unused EC2 capacity in AWS with a 90% discount - For flexible applications - You have to specify a max price, instances will run based on this figure - AZ and Regions pricing changes - Spot block, 1-6hrs of denying spot instances from stopping even above price ceiling - Not good for persistent workloads, critical jobs, or DBs - Spot fleet, a collection of spot instances that attempted to maintain a target fleet capacity if the spot instances are interrupted by pricing - Trying to meet capacity within your price constraints #### VMWare in AWS - Hybrid cloud - Cloud migration, from private to AWS - Disaster recovery - Leverage AWS #### AWS Outposts - Extending AWS into your private cloud - Bringing AWS services to your own data center using rack up to 42U - Allows for hybrid cloud, fully managed by AWS, allows for consistency in your hybrid environment - Outputs racks, for datacenters up to 42U rack and scales to 96 racks - Outposts servers, individual servers instead of full racks, useful for smaller operations #### EBS Overview - Elastic block storage - Can be attached to EC2 instances - General purpose, think of it like a virtual disk, nothing special to it - Types of disk - GP2, general purpose, balanced for price/performance - GP3, general purpose, next gen of GP2 - IO1, provisioned IOPS, the most expensive and highest performance - IO2, provisioned IOPS, next gen of IO1, high durability and the fastest - ST1, throughput optimized HDD, old school HDD, lowest-cost, big data, data warehouses, cannot be a boot volume - SC1, cold HDD, lowest cost - Throughput = big data, MB/s for read/write, working with large datasets - IOPS = read/write operations per second, import for quick transactions, low-latency apps #### Volumes and Snapshots - Volumes are virtual hard disks, exist on EBS - Snapshots are stored on s3, a point in time copy of a volume, these are incremental, only changes between snapshots are moved to s3 - Consistent snapshots, stop the instance and then take a snapshot - Encrypted snapshots, any snapshots that are taken of an encrypted EBS volume will be encrypted also - Sharing snapshots, only in the region where they were created unless you copy them to the destination region first - Always know the location of EBS volumes - Resizing can be done on the fly, you will need to resize the filesystem within the OS, do not need to stop the instance - You can switch volume types on the fly also, do not need to stop the instance #### EBS Volumes with Encryption - Encrypted with AES-256, uses AWS KMS customer master keys when creating encrypted volumes and snapshots - Data at rest and at rest are encrypted - Encryption is handled transparently, does not impact latency, root device volumes are encrypted - You can create an AMI from an unencrypted snapshot but copying the snapshot and select encryption, then create an AMI from that encrypted snapshot, then use the AMI to launch encrypted instances from #### EC2 Hibernation - Hibernation saves the contents from memory to the EBS root volume - Will restart from the previous state when restarted from hibernation - Useful for long-running processes - Cannot be hibernated for more than 60 days #### EFS Overview - Network file system - Elastic file system, works with EC2 instances - Only pay for what you use - Expensive, but available and scalable - Content mgmt, wordpress blogs, web servers - Uses NFSv4 protocol - Storage tiers - Standard, frequently access files - IA, not frequently accessed #### FSx Overview - Fully managed Windows file system - Based off of Windows Server - FSx for Lustre - AI, machine learning capabilities ![[Pasted image 20230511162541.png]] #### AMIs, EBS vs Instance Store - AMI, amazon machine image - Region - OS - Arch - Launch perms - Storage - Can be stopped so you don't lose your data - A blueprint for an EC2 image - Instance store volumes - Ephemeral storage, cannot be stopped - if your host fails you lose data - Can reboot store volumes to not lose data #### AWS Backup - Consolidation, used AWS backup to backup AWS services like EC2, EBS, EFS, WFS, AWS Storage Gateway, and more - Organization, use organizations with AWS backup to backup different AWS services across multiple account - Benefits, centralized control, automate backups and define lifecycle policies, better compliance, can enforce backup policies and have encrypted backups #### Databases Overview - RDS, relational database service - Data is organized into tables, RDS is multiple tables - Types of RDS, SQL, Oracle, MySQL, PostgreSQL, MariaDB, Amazon Aurora - OLTP, online transaction processing - Payments, booking, transactional data - Not suitable for large data analysis - OLAP, online analytical processing - Complex queries, analyzing historical data, data analysis - Multi-AZ RDS, creating an exact copy of your prod DB to another AZ - For disaster recovery not HA #### Read Replicas - Read replicas, a read-only replica of your DB - Read replica is for boosting performance - Each read replica has its own DNS endpoint - Doing this is for scaling read performance - Promoting a read replica can turn it into its own database #### Amazon Aurora - Allows for auto scaling, goes from 10gb to 128tb - 2 copies of your data are contained in each AZ, with a minimum of 3 AZ for 6 copies - Available replicas - Aurora replicas, MySQL replicas, PostgreSQL replicas - Amazon Aurora Serverless, auto scales capacity based on your apps needs #### DynamoDB - Non relational NoSQL DB, wide fit for many different data types, mobile, web, gaming, IOT, etc - Stored on SSD, spread across 3 geographically distinct DCs - Eventually consistent reads, usually reached in 1 second, best read performance - Strongly consistent read, returns a result that reflects all writes - DAX, dynamoDB accelerator - Pay per request pricing, great for new product launching - Encryption at rest using KMS - ACID, atomic, consistent, isolated, durable - Need to use DynamoDB transactions to use ACID with DDB - All or nothing transactions across one or more tables within an AWS account or region #### MongoDB-Compatible DBs in Amazon DocumentDB - MongoDB, document database with strong querying and indexing - Amazon DocumentDB, running MongoDB in AWS, scales with your workloads #### Apache Cassandra and Amazon Keyspaces - Cassandra, distributed DB using NoSQL, big data - Amazon Keyspaces, Cassandra DB in AWS, serverless - Big data Cassandra cluster = Keyspaces #### Graph DBs and Amazon Neptune - Graph DB, data stored like sketching on a whiteboard - Amazon Neptune, Graph DB for AWS #### AWS Quantum Ledger (QLDB) - Ledger DB, NoSQL DB, can only add new records to a DB and can't overwrite old records - Blockchain, bitcoin, etc - QLDB, a ledger DB for AWS #### Time-series Data with AWS Timestream - Time-series data, data points over a length of time - Example would be temperature from weather stations around the world, on the hour - Timestream, Time-series DB for AWS #### VPCs and More - Logical data center, consists of internet gateways, routing tables, NACLs, subnets, security groups - One subnet is always in One AZ - AWS reserves the first four IP addresses in each subnet - When you create a VPC it will create a NACL, security group, and a route table - NAT Gateways, redundant inside AZs, automatically given a public IP, not associated with security groups - Having multiple AZs map to one NAT GW will result in multiple AZs going offline, you need to map one NAT GW per AZ - Use when you want private subnets to access the internet without being publicly accessible - Are deployed into the public subnets, not private subnets - Not associated with security groups, you don't need to patch NGs, redundant inside the AZ, also automatically assigned a public IP - Security Groups, stateful traffic, responses initiated by the host are allowed despite outbound rules - Last line of defense before the asset is touched by an incoming connection - Virtual firewalls for ec2 instances - Network ACLs, by default allows all in/out traffic, custom ACLs are deny unless permitted by default, each subnet must be associated with a network ACL, block IPs using ACLs and not security groups - Network ACLs can associate with multiple subnets but a subnet can only have 1 NACL, each new NACL overrides the previous - Rules in a NACL are evaluated in order, separate rules for inbound and outbound traffic, NACLs are stateless - First line of defense - NACLs have rules for inbound and outbound traffic since they are stateless - It's best practice to increment your firewall rules by 100s - You should also have an outbound rule for ephemeral ports such as 1024-65535 - Direct Connect, connects your datacenter to AWS, used for high-throughput workloads - VPC Endpoints, connecting to AWS services without leaving the AWS intranet - Two type of VPC endpoints, interface and gateway - Interface is an ELI with a private IP - Gateway is a virtual device you setup - Supports s3 and DynamoDB - Are virtual devices, have zero bandwidth requirements - Peering, allows one VPC to connect to another, behave as if on the same LAN, can connect to other AWS accounts and other VPCs within the same account, uses hub and spoke configuration, can peer between AWS regions - VPC endpoints are powered by PrivateLink and do not require IG, NAT, VPN, or AWS Direct Connect - PrivateLink, peering VPCs to LOTS of customer VPCs but doesn't require VPC peering however it does require a NLB (network load balancer) on the svc VPC and an ENI on the customer VPC - AWS Transit Gateway, IP multicasting, works with Direct Connect as well as VPNs, use routing tables to limit how VPCs talk - VPN Hub, simplifies network topology by having AWS manage your VPN services #### Route53 and More - Alias, translating your DNS name to a ARN - IPv4 vs IPv6 - 4 billion available IPv4s addrs - 340 undecillion IPv6 addresses - TLD, top-level domains - .gov, .com, .zip, .net - Always choose an alias over a CNAME whenever possible - CNAME, translating a subdomain to another domain, a1.domain.com = 1a.domain.com, like a redirect (sort of) - SOA, start of authority, contains DNS information about the registrant - CNAME, canonical name, used for redirecting from one domain to another - NS Records, name server, - A Records, turn web addresses to IP addresses - Route53 routing policies - Simple, one record with multiple IPs, when going to "site.com" whatever IP the site is resolving to gets returned, if multiple IPs resolve the site a random one will get returned to the user - Weighted, choosing a region to send traffic based on weight, 75% of traffic to us-east-1, 25% traffic to us-west-1 - Latency-based, will send traffic to the lowest latencied region - Failover, traffic will be routed to the region that passes the health check, active/passive setups - Geolocation, traffic will be sent to the region closest to the user requesting resources - Geoproximity, let's traffic to your resources based on the location of your users, can set biases which shrinks the region traffic routed to a resource, must be using route53 traffic flow - MV Answer, similar to failover based on health checks - Health checks, checking the health of DNS records - Failing a check will remove it from route53 until it passes - SNS, simple notification service, can use SNS to alert about failed health checks #### Elastic Load Balancing (ELB) and More - App load balancers, works at layer 7 - Listeners, checks for requests from clients using proto and ports you configure - Rules, how LBs route requests to targets, based on priority, actions, and conditions - Target groups, group routes requests to registered targets using proto and ports you configure - Works only with HTTP and HTTPS - SSL cert lives on the load balancer, terminates the connection and then forwards the traffic onward - Network load balancers, works at layer 4 - When high performance is needed, also when protocols are needed that aren't supported by ALBs, NLBs can decrypt traffic but you need to install a certificate - Classic load balancers, works at layer 4 and 7 - 504 means GW has timed out, app isn't responding within a set time period - Is the DB server offline? - X-forwarded-for will give the IP of your end-users - Sticky sessions, will continue to send users to the same resource, sometimes you will see LBs sending users to a resource that has been removed - Disable sticky sessions to stop this behavior - Deregistration delay and connection draining, keep connections open if an EC2 becomes unhealthy - Disable if you want the LBs to close connections instead #### Monitoring and More - CloudWatch is the main tool for alerting - EC2, on-prem, RDS, Lamda, and CloudTrail can all integrate - To do SQL queries, think CloudWatch Logs Insights - Real time logs means using Kinesis - A monitoring and observability platform - Can collect system metrics - Can collect application metrics - Alarm, create custom alarms using the data - CloudWatch Logs, monitor, store and access log files - Log event, data point with a timestamp - Log stream, collection of log events from the same source - Log group, collection of log streams, an example would be logs from all of your apache web servers - Filter patterns, search queries - Insights, SQL-like querying of log - Alarms, building alarms off of queries - Agent based, have to install it on a host and load a config file - Some things can go through AWS Config, such as anything AWS Standards related - Standard metrics are delivered every 5 minutes, detailed monitoring delivers data every one minute - Grafana, for visualization of container metrics, good for correlation and visuals of IoT as well - Amazon managed grafana can be easily deployed - Workspaces are logical grafana servers allowing for separation of data visualizations and querying - Auto scales the setup and maintenance of all workspaces - Secure, help meet governance and compliance requirements - Use case, container metric visualization, IoT, troubleshooting - Prometheus, open-source data model - Amazon managed prometheus, AWS handles the scaling base don ingestion, storage, and querying of metrics - High availability, replicates data across three AZs - Worked with EKS or self-managed k8s - PromQL, open-source query language - Data is stored for up to 150 days #### Horizontal vs Vertical Scaling - Scaling vertically is older in architecture, building one large box and feeding it more resources over time - Scaling horizontally is running multiple boxes in parallel and splitting up the work among many workers - Three W's of scaling - What, ec2, s3, eks, what resource are we scaling? - Where, what region do we scale to - When, how do we know when to scale? - Launch template, a collection of specific settings to build an ec2 instance so you don't have to manually configure each time - Template vs configuration, prefer templates by default, AWS recommended - Scaling ec2 instance with autoscaling - 1, define your template - 2, networking and purchasing - 3, elb configuration - 4, set scaling policy - 5, notifications, using sns ![[Pasted image 20230601224128.png]] - Autoscaling is vital to creating a highly available application, spread resources out over multiple AZs and utilize load balancers - Reactive scaling, once a resource demand/load is there, measure the scaling resources required, then scale up - Scheduled scaling, when a resource demand/load is predictable you can scale the resources before you need them - Predictive scaling, ML utilized to guess when a demand/load will arrive - Scaling relational dbs, types of scaling - Vertical scaling, more resources - Scaling storage, can be sized up but not down - Read replicas, read-only copies of your data sets - Aurora serverless, offload scaling to AWS - Scaling non-relational DBs - DynamoDB, fully managed by AWS - Provisioned, great for a predictable workload - On-demand, for sporadic workloads #### Decoupling Workflows Overview - Tight coupling, one instance (ec2, let's say) relying on another instance directly - Loose coupling, distributed ec2 to distributed ec2 to handle for multiple failures and making it seamless to a user - SQS, simple queue service, can manage message queueing that enabled decoupling of microservices, or serverless applications - Poll-based messaging, think of it like a postman delivering a message into a mailbox for the user to read whenever they are ready to consume it, asynchronous process - Delivery delay, does what it sounds like - Message size, up to 256KB in any format - Encrypted, encrypted in transit, but not at rest by default - Message retention, default is 4d, max is 14d - Long vs Short, long polling is usually what you want to use - Queue depth, this can be a trigger for autoscaling, for example if the log queue begins backing up - Visibility timeout, when the backend processes a message the message is locked for 30s until the backend releases a "all clear" to delete the message - Dead-letter queue, rollover messages queue to roll messages over to after the 14d timeout, the DLQ also has a 14d timeout - SQS offers best-effort ordering and there could be duplicates, - FIFO, first-in, first-out, guarantees the ordering - SNS, simple notification service, A2A (app to app), A2P (app to person) communications - Push-based messaging, postman just checks the message into your house, you have to be ready to receive it at any time - Subscribers, SQS, SMS, Lambda, Email, HTTPS - Message size, up to 256KB - DLQ - FIFO or Standard as an option - Encryption in transit, can add-on at rest - API gateway, easy for devs to publish maintain, monitor, and secure APIs at scale - Protects APIs by baking in security like WAFs, rate-limiting, ease-of-use - Front door to the application - AWS Batch - Running batch computing workloads on ec2 or ecs/fargate - No installation required, capable of accurately guessing the size/need of compute resources - Job, a unit of work submitted to AWS - Shell scripts, executables, docker images - Job definitions, blueprint for resources - Job queues, lines where jobs wait to be executed - Use fargate as the recommended approach to submit batch jobs - Amazon MQ - Managed message broker, can use a variety of languages, OSs, messaging protos - Supports Apache ActiveMQ and RabbitMQ - Amazon MQ is easy to adopt with existing messaging technologies, you'd use SNS/SQS if you were building from scratch - Several types of messaging brokers - Single-instance, one broker in one AZ, great for dev - HA - RabbitMQ, supports cluster deployments across AZs - ActiveMQ, supports active/standby deployment to maintain uptime - AWS step functions - Serverless orchestration for event-driven task execution - Standard workflows, long-running and auditable executions - Express workflows, high-event-rate executions - Amazon states language, all machines are written in this format - States are elements within your state machine - Integrations, many services integrate with step functions - State types, pass, task, choice, wait, succeed, fail, parallel, map - Amazon AppFlow - Fully managed service for transferring data from 3rd party SaaS vendors and apps - Flows are bi-directional - Good for solutions needing easy and fast transfer #### Big Data Overview - Redshift, not standard and shouldn't take the place of RDS - It's a relational DB, can support up to 16PB of data - Only supports single-AZ deployments - EMR, elastic map resource - A managed fleet of ec2 instances running open-source tooling - Can used reserved instances and spot instances to manage costs - All normal VPC rules apply to ec2 clusters - ETL, extract transform load - Kinesis, streams data in real-time, only service that does it - SQS is not real-time - Amazon Athena, interactive query service for data in s3 using SQL - Think of it as serverless SQL - Amazon Glue, serverless data integration, perform ETL workloads without a fleet of ec2 instances - Think of it as serverless ETL - QuickSight, data visualization service - AWS Data Pipeline, a managed ETL (extract, transform, load) for automating movement and transformation of your data - Data-driven workflows, define your parameters for data transforms, automatically retries when tasks fail, integrates with DynamoDB, RDS, Redshift, and s3 - Pipeline definition - Managed compute, will manage ec2 instances - Tasks runners, poll for tasks and complete them when found - Data nodes, define the locations and types of data that will be input and output - Amazon Managed Streaming for Apache Kafka - Amazon MSK, fully managed service for running data streaming applications that leverage Kafka - Creates, deletes, updates clusters as needed - Works for integrating existing apps, tools, and plugins - Broker nodes, specify amount of broker nodes per AZ - Zookeeper nodes, these are created for you - Producers, consumers, and topics, allows for creation of topics and can produce/consume data - Flexible cluster operations #### Serverless Architecture Overview - Paying for only the code you execute and the computer cost it took to execute it - Lambda, write the code, build the function, deploy - Serverless compute service - Lambdas cannot execute longer than 15m - Needs IAM roles to perform the functions you have designed it to do ![[Pasted image 20230603201312.png]] - Containers - Dockerfile, set of instructions that are used to build a docker image - Image, all the code, dependencies, libraries, and configs that are used to run an application - Registry, stores docker images for distribution - ECS, elastic container service, the preferred method for managing containers/k8s - Fargate, run containers, no servers, no patching, fully managed AWS service for containers - EventBridge, serverless event bus, used to be CloudWatch Events, it generates alerts - It's the glue for serverless architecture, any API call can kick off EventBridge to call other resources - Amazon Aurora Serverless, on-demand or auto scaling DBs, variable traffic or workloads, capacity planning - AWS X-Ray, gain application insights using requests/responses of services, traces downstream response times, tightly integrated with API Gateway and Lambda - AWS AppSync, scalable GraphQL interface, pulls in data from multiple sources like DynamoDB, Lambda #### Security Overview - DDoS - Layer 4, syn flood, draining all open sockets - Amplification, uses UDP packets for common services like NTP, DNS, etc, to send a request, the response is larger than the request and you redirect that to a different source, flooding it with data - Layer 7, GET request flooding - Logging API calls with CloudTrail - CloudTrail records AWS management console API calls - Metadata around API calls, identity, timestamp, source IP, request params, response elements - After-the-fact investigations - Near real-time intrusion detection - Shield, free DDoS protection on ELB, CloudFront, and Route53, protects against syn/udp floods, reflection attacks, and other layer 3/4 attacks also - Shield Advanced costs 3k/mo - AWS WAF, web app firewall, monitoring http/https traffic, operates at layer 7 - Three types of behavior - Allow all requests except the ones you say - Block all requests except the ones you say - Count the requests - GuardDuty, threat detection service with ML/AI, cost after 30d is based on amount of CloudTrail logs and volume of DNS/VPC flow logs - You can use CloudWatch Event to trigger a Lambda function to address a threat - AWS Firewall Mgr, single pane of glass for addressing firewall rules across accounts and applications within AWS organizations - Create WAF rules for application load balancers, API gateways, and Cloudfront distributions - s3 buckets and Macie, pattern matching to discover CI/PII, public buckets, unencrypted buckets - Amazon Inspector, automated security assessment service - Network assessment, checks VPCs for reachable hosts from the internet, no agent is required for this - Host assessment, checks for CVEs in software, host hardening, and security best practices, agent is required - Installed on ec2 - KMS, key management service, integrates with other AWS services to encrypt your data - CMK, customer master key - You control the lifecycle of the CMK, no automatic key rotation for customer managed keys - Three ways to generate, AWS creates it for you, import your own key management infrastructure, have the key material generated and used in an AWS CloudHSM - Can have automatic key rotation - CloudHSM, hardware security module designed specifically for you - SecretsManager, managed service that stores, encrypts, and rotates your DB creds and other secrets - Encryption in transit and at rest using KMS - If rotation is enabled then the secret will be rotated once to test the configuration - Parameter Store, capability within AWS Systems Mgr to provide secure storage for config data management and secrets management - Can store passwords, DB strings, AMI IDs, license codes, can be stored as plain text or encrypted data - No key rotation - s3 presigned URLs, path to your object that requires security creds, bucket name, object key, also the HTTP method to download such as GET and the expiration date/time - Advanced IAM Policy Documents - ARN, amazon resource names, the format looks like this `arn:partition:service:region:account_id:resource` ![[Pasted image 20230604194616.png]] - IAM Policies - JSON document that defines permissions within policies such as identity or resource policies, does not go into effect until attached to a resource, essentially just a list of statements - Permissions Boundaries - Used to delegate administration to other users - Used to prevent priv esc or overly broad permissions - Amazon Cognito - Provides authentication, authorization, and user management - Acts as an identity broker between your app and other web ID providers - Synchronizes users across devices - User pool, user directories for sign-up/sign-in options - Identity pool, allows users to access other AWS services - Amazon Detective - Allows the triage and investigation of security incidents - Detective can be used for threat-hunting with the detail visualizations of all the relationships between resources #### Automation Overview - Automation is faster, better, more secure - CloudFormation - Infrastructure as code - Declarative programming, JSON or YAML formatting - Used for creating immutable architecture - Elastic Beanstalk - IT dept in a can - PaaS, platform as a service - Does not handle serverless architecture - Systems Manager - Suite of tools to manage, view, and control AWS and on-prem resources - Automation Documents are now known as runbooks #### Caching Overview - CloudFront, a CDN, takes content and caches it all across the globe, external facing cache - Defaults to HTTPS - ElastiCache, AWS managed version of Memcached/Redis, internally facing cache architecture, usually for RDS DBs - DAX, dynamo DB accelerator, in-memory cache - Global Accelerator, externally facing networking service, helps with IP caching - Helps mask complex architecture, speeds things up, can create weighted pools #### Machine Learning Overview - Amazon Comprehend, NLP (natural language process), comprehension at scale, opinions during elections as example - Amazon Kendra, allows for intelligent search services powered by ML - Amazon Textract, automatically extracts text, handwriting, from images and documents - Amazon Forecast, send time-series data into AMZ Forecast - Amazon Fraud Detector, AI that detects fraud in your data - Amazon Polly, turns text into lifelike speech - Amazon Transcribe, speech to text automatically - Amazon Translate, one language to another using deep learning/neural networks - Amazon Sagemaker, a way to build and deploy ML learning models in the cloud - Amazon Lex, talking to a chat bot, it's likely Lex, a natural language processing service - Amazon Rekognition, picture processing using deep learning and neural networks, useful for content moderation, face detection and analysis #### Final Prep for the Exam, Random Notes - [Link](https://aws.amazon.com/architecture/?cards-all.sort-by=item.additionalFields.sortDate&cards-all.sort-order=desc&awsf.content-type=*all&awsf.methodology=*all&awsf.tech-category=*all&awsf.industries=*all&awsf.business-category=*all) to AWS Architecture - Spot distractors for the test - Watch out for modifiers - Word-soups are rarely right - Slow down, read every answer - AWS Shield vs Shield Advanced = Advanced has cost protection - AWS Wavelength = embeds compute and storage within 5G - AWS MGN = moving existing network infrastructure to the cloud