AWS Cloud Computing

Exploring AWS Compute Services: Powering Scalable and Flexible Applications

Posted by Vineet Kumar Loyer on Monday, December 30, 2024

AWS Compute Services

AWS offers a range of compute services to meet diverse application needs, from running virtual servers to serverless computing. Below are some of the core services -

  1. Amazon EC2 (Elastic Compute Cloud): Virtual servers to run applications.
  2. AWS Lambda: Serverless compute that runs code in response to events.
  3. Amazon ECS (Elastic Container Service): Container orchestration for Docker containers.
  4. Amazon EKS (Elastic Kubernetes Service): Managed Kubernetes service.
  5. AWS Fargate: Serverless compute for containers.
  6. Amazon Lightsail: Simplified virtual private servers for small-scale applications.

A more wide range can be found on AWS products page (AWS Compute Services)

Category Service Description AWS Service
Instances (virtual machines) Easy-to-use service for deploying and scaling web applications and services AWS Elastic Beanstalk
Secure and resizable compute capacity (virtual servers) in the cloud Amazon EC2
Run fault-tolerant workloads for up to 90% off Amazon EC2 Spot
Auto add or remove compute capacity to meet changes and demand EC2 Autoscaling
Easy-to-use cloud platform that offers you everything you need to build an applicaiton or website Amazon Lightsail
Fully managed batch processing at scale AWS Batch
Containers Highly secure, reliable, and scalable way to run containers Amazon Elastic Contianer Service (ECS)
Run containers on customer-managed infra Amazon ECS Anywhere
Easily store, manage, and deploy container images Amazon Elastic Container Registry (ECR)
Fully managed Kubernetes services Amazon Elastic Kubernetes Services (EKS)
Create and operate Kubernetes clusters on your own infra Amazon EKS Anywhere
Serverless compute for containers AWS Fargate
Build and run containerized applications on a fully managed service AWS App Runner
Serverless Run code without thinking about servers. Pay only for the compute time you consume AWS Lambda
Edge and hybrid Run AWS infrastructure and services on premises for a truly consistent hybrid experience AWS Outposts
Collect and process data in rugged or disconnected edge environments AWS Snow Family
Deliver ultra-low latency application for 5G devices AWS Wavelength
Preferred service for all vSphere workloads to rapidly extend and migrate to the cloud VMware cloud on AWS
Run latency sensitive applications closer to end-users AWS Local Zones
Cost and capacity management Flexible pricing model that provides savings of up to 72% on AWS compute usage AWS Savings Plan
Recommends optimal AWS compute resources for your workloads to reduce costs and improve performance AWS Compute Optimzer
Build and maintain secure Linux or Windows Server images EC2 Image Builder
Automatically distribute incoming application traffic across multiple targets Elastic Load Balancing (ELB)

AWS EC2

Amazon EC2 (Elastic Compute Cloud) is a foundational AWS compute service that provides resizable, scalable virtual servers in the cloud. It enables users to launch and manage instances (virtual machines) tailored to their specific needs, ranging from small-scale workloads to enterprise-grade applications.

1. EC2 Deployment and Fault Tolerance

  • Deployment into One AZ:
    • By default, an EC2 instance is deployed into a single Availability Zone (AZ).
    • If the AZ experiences a failure (e.g., power outage), the instance in that AZ will fail unless fault tolerance mechanisms are set up.
    • To mitigate risks:
      • Use Auto Scaling Groups to span instances across multiple AZs.
      • Configure Elastic Load Balancers (ELBs) to distribute traffic between instances in different AZs.

2. EC2 Pricing

EC2 offers a flexible pricing structure tailored to various workloads and budgets:

  1. On-Demand Instances:
    • Pay for compute capacity by the second (Linux) or hour (Windows).
    • No upfront commitment; suitable for unpredictable workloads.
    • Higher cost compared to other options.
  2. Reserved Instances (RIs):
    • Commit to a 1- or 3-year term for up to 75% cost savings.
    • Offers flexibility to change instance family or region in certain cases ( Convertible RIs ).
  3. Spot Instances:
    • Bid for unused EC2 capacity at a significantly reduced price (up to 90% savings).
    • Instances may be terminated by AWS with two minutes' notice if capacity is required elsewhere.
    • Best for fault-tolerant, batch processing, or interruptible workloads.
  4. Savings Plans:
    • Flexible pricing model offering lower rates for committed usage (measured in dollars/hour).
    • Applies across instance types and regions.
  5. Dedicated Hosts:
    • Physical servers fully dedicated to your workloads.
    • Ideal for regulatory compliance or specific licensing requirements.
  6. Dedicated Instances:
    • Run on hardware dedicated to a single customer.
  7. Free Tier:
    • New users get 750 hours/month of t2.micro or t4g.micro instances for 12 months.

3. Amazon Machine Image (AMI)

  • What is AMI?

    • AMIs are pre-configured templates used to launch EC2 instances.
    • They include:
      • An operating system (e.g., Linux, Windows).
      • Application server or software (optional).
      • Storage volumes and permissions.

  • Types of AMIs:

    1. AWS-provided AMIs: Standard Linux/Windows OS configurations.
    2. Marketplace AMIs: Pre-configured with software (e.g., WordPress, MySQL).
    3. Custom AMIs: Created by users for specific configurations.

  • Use Cases for AMIs:

    • Quickly replicate environments for consistency across multiple instances.

    • Back up and restore instances by creating an AMI from a running instance.

      Fig-1: AWS Server Virtualization / AMIs

      Fig-1: AWS Server Virtualization / AMIs

      Fig-2: AWS AMI Selection during launching EC2 instance

      Fig-2: AWS AMI Selection during launching EC2 instance

4. When to Use EC2 ?

Amazon EC2 is ideal for:

  1. Custom Applications:
    • Applications requiring specific OS configurations, libraries, or dependencies.
    • Examples: Web servers, application servers.
  2. Scalable Applications:
    • Workloads with unpredictable demand where Auto Scaling can adjust capacity.
  3. Batch Processing:
    • Big data jobs or computationally intensive tasks using Spot Instances.
  4. Development and Testing:
    • Isolated environments for testing software before deployment.
  5. Machine Learning and AI:
    • Training ML models on GPU-enabled instances (e.g., p3 , g5 ).
  6. High-Performance Computing (HPC):
    • Applications requiring low-latency networking and high compute power.
Feature EC2 Lambda ECS/EKS Lightsail
Control Full control over OS, instance type, and configurations. Fully serverless; no control over infrastructure. Container orchestration and management. Simplified, beginner-friendly setup.
Scaling Manual or Auto Scaling. Automatic scaling based on events. Containerized scaling. Limited scaling options.
Use Case Complex workloads requiring flexibility. Event-driven or short-lived workloads. Containerized microservices. Small-scale apps, websites, and blogs.
Pricing/Model Pay for instance time; multiple pricing models available. Pay per execution time and request count. Pay per container resources Predictable monthly pricing.

AWS Lambda

AWS Lambda is a serverless compute service that automatically runs your code in response to events and manages the underlying compute resources. With Lambda, you focus solely on your code while AWS handles provisioning, scaling, and maintaining servers, offering unparalleled simplicity and cost-efficiency.

1. Key Features of AWS Lambda

  • Serverless Architecture:
    • No need to provision or manage servers.
    • Automatically scales based on the number of requests.
  • Event-Driven Execution:
    • Executes code in response to events from AWS services (e.g., S3, DynamoDB, API Gateway) or external sources.
  • Cost Efficiency:
    • Pay only for the compute time consumed (measured in milliseconds) and the number of requests.
    • No charges for idle time.
  • Wide Language Support:
    • Supports popular programming languages like Python, Node.js, Java, Go, Ruby, and .NET Core.
    • Use custom runtimes to run additional languages.
  • Flexible Triggers:
    • Integrated with various AWS services, including S3, DynamoDB, Kinesis, SNS, and EventBridge, for event-driven workflows.
  • Built-in High Availability:
    • Automatically replicates across multiple Availability Zones for fault tolerance and high availability.

2. AWS Lambda Pricing

AWS Lambda pricing is straightforward and based on two main factors:

  1. Compute Time:
    • Charged based on the number of milliseconds the code executes.
    • The price depends on the amount of memory allocated (128 MB to 10 GB) and the execution time.
  2. Requests:
    • Charged per request. The first 1 million requests per month are free.
  3. Additional Costs:
    • Any associated costs for data transfer or usage of integrated AWS services (e.g., DynamoDB, S3).

3. AWS Lambda Deployment and Workflow

  • Deployment:
    • Upload the code as a ZIP file or a container image (up to 10 GB).
    • Specify runtime and memory requirements.
    • Configure event sources and permissions.
  • Execution Workflow:
    1. Event triggers Lambda.
    2. Lambda automatically provisions and scales compute capacity.
    3. Code executes, and results are sent to the configured destination (e.g., logs in CloudWatch or responses in API Gateway).

4. When to Use AWS Lambda?

AWS Lambda is ideal for:

  1. Event-Driven Applications:
    • Automatically process events from services like S3 (file uploads) or DynamoDB (data changes).
  2. Real-Time Data Processing:
    • Stream processing using Kinesis or Kafka.
    • Log analysis, metrics generation, and anomaly detection.
  3. Back-End Services for APIs:
    • Integrate with API Gateway to create serverless APIs for web and mobile apps.
  4. Automation Tasks:
    • Perform scheduled tasks using Amazon EventBridge (e.g., nightly reports or maintenance).
  5. IoT Applications:
    • Process and analyze data from IoT devices using AWS IoT Core.
  6. Machine Learning Inference:
    • Deploy trained models for real-time predictions without provisioning dedicated instances.
  7. Cost-Sensitive Applications:
    • Use Lambda for workloads with intermittent traffic or unpredictable usage patterns to avoid idle compute costs.

Amazon ECS (Elastic Container Service)

Amazon ECS (Elastic Container Service) is a fully managed container orchestration service that enables you to deploy, manage, and scale Docker containers. It integrates deeply with the AWS ecosystem, making it a powerful choice for containerized workloads that need to scale efficiently and reliably.

1. Key Features of Amazon ECS

  1. Fully Managed Service:
    • AWS manages the orchestration of your containers, eliminating the need to operate and maintain infrastructure.
    • Automates container deployment, scaling, and load balancing.
  2. Native AWS Integration:
    • Deep integration with AWS services like EC2, Fargate, IAM, CloudWatch, and Elastic Load Balancing.
    • Leverages AWS security features for granular control over container access.
  3. Flexible Compute Options:
    • ECS on EC2: Use EC2 instances as the underlying infrastructure for hosting containers.
    • ECS with AWS Fargate: Serverless compute option for running containers without managing instances.
  4. High Performance:
    • Optimized for high performance and low-latency containerized applications.
    • Supports thousands of containers simultaneously with fine-grained resource control.
  5. Networking Support:
    • Integrated with Amazon VPC for secure and scalable networking.
    • Allows task-level networking and supports Elastic Network Interfaces (ENIs).
  6. Task Definitions and Clusters:
    • Define your containers' configurations, including memory, CPU, and port mappings, in a Task Definition .
    • Use Clusters to group and manage your containerized applications.

2. ECS Deployment Options

  1. EC2 Launch Type:
    • Deploy containers on Amazon EC2 instances.
    • Offers complete control over the instance type, storage, and networking configurations.
  2. Fargate Launch Type:
    • Run containers without provisioning or managing EC2 instances.
    • Ideal for serverless container workloads, reducing operational overhead.
  3. External Instances:
    • Extend ECS to manage on-premises servers or virtual machines using AWS Systems Manager.

3. Amazon ECS Pricing

Amazon ECS pricing depends on the launch type:

  1. EC2 Launch Type:
    • Pay for the underlying EC2 instances, storage, and networking.
    • No additional cost for using ECS; you pay only for the AWS resources you provision.
  2. Fargate Launch Type:
    • Pricing is based on the vCPU and memory resources allocated to containers.
    • Pay only for the resources used during container execution.
  3. Networking Costs:
    • Charges for data transfer and Elastic Load Balancing apply as per standard AWS pricing.

4. When to Use Amazon ECS ?

Amazon ECS is ideal for:

  1. Microservices Architectures:
    • Run and manage containerized microservices with seamless scaling.
  2. Batch Processing:
    • Deploy batch jobs in containers, taking advantage of fault tolerance and efficient resource allocation.
  3. Machine Learning:
    • Use ECS to containerize machine learning workflows for model training and inference.
  4. Hybrid Deployments:
    • Manage container workloads across AWS cloud and on-premises environments.
  5. Serverless Containers:
    • Use the Fargate launch type to eliminate infrastructure management for container workloads.

Amazon EKS (Elastic Kubernetes Services)

Amazon EKS (Elastic Kubernetes Service) is a fully managed Kubernetes service that allows you to deploy, manage, and scale containerized applications using Kubernetes. EKS handles the complexities of setting up and operating Kubernetes clusters, enabling developers to focus on building and deploying applications.

1. Key Features of Amazon EKS

  1. Fully Managed Kubernetes:
    • AWS manages the Kubernetes control plane, ensuring it is secure, highly available, and scalable.
    • Includes automatic patching, upgrades, and scaling of the control plane.
  2. Seamless Integration with AWS Services:
    • Integrates with services like Elastic Load Balancing, IAM, CloudWatch, and AWS Fargate.
    • Leverages AWS networking features like VPC and Security Groups for secure communication.
  3. Flexibility in Compute Options:
    • Run Kubernetes worker nodes on EC2 instances for complete control over infrastructure.
    • Use AWS Fargate for serverless, containerized workloads.
  4. High Availability:
    • The EKS control plane is distributed across multiple Availability Zones (AZs) for fault tolerance.
    • Ensures 99.95% uptime SLA for the Kubernetes API.
  5. Kubernetes Compatibility:
    • Fully compatible with open-source Kubernetes, enabling easy migration of workloads from other Kubernetes environments.
    • Supports Kubernetes-native tooling like kubectl, Helm, and custom CRDs.
  6. Scalability:
    • Automatically scales worker nodes and workloads based on demand using Kubernetes auto-scaling features (HPA, CA, and Karpenter).
  7. Security and Compliance:
    • Tight integration with AWS IAM for role-based access control (RBAC).
    • Encrypts Kubernetes secrets with AWS KMS.

2. EKS Deployment and Architecture

  1. Control Plane:
    • Managed by AWS, including Kubernetes API servers and etcd databases.
    • Automatically scales to handle the load and ensures high availability.
  2. Worker Nodes:
    • Run on EC2 instances or AWS Fargate.
    • Fully customizable based on CPU, memory, and storage requirements.
  3. Networking:
    • Leverages Amazon VPC for secure, isolated communication between worker nodes.
    • Supports Kubernetes-native networking (CNI) for pod-to-pod communication.
  4. Monitoring and Logging:
    • Integrated with Amazon CloudWatch for cluster monitoring.
    • Supports Kubernetes-native monitoring tools like Prometheus and Grafana.

3. Amazon EKS Pricing

EKS pricing consists of:

  1. Control Plane Fee:
    • $0.10 per hour per cluster, regardless of cluster size.
  2. Worker Node Costs:
    • Pay for the EC2 instances or AWS Fargate resources used to run worker nodes.
  3. Networking and Add-Ons:
    • Standard AWS charges apply for networking (e.g., Elastic Load Balancing) and optional add-ons (e.g., AWS App Mesh).
  4. Free Tier:
    • Includes 750 hours of EKS control plane usage per month for the first 12 months.

4. When to Use Amazon EKS ?

Amazon EKS is ideal for:

  1. Microservices Architectures:
    • Orchestrate and scale containerized microservices across multiple worker nodes.
  2. Hybrid and Multi-Cloud Deployments:
    • Leverage Kubernetes to build portable applications that can run on-premises and in the cloud.
  3. High Availability Applications:
    • Deploy fault-tolerant applications across multiple AZs.
  4. Machine Learning and Data Processing:
    • Run distributed machine learning training jobs and data pipelines using Kubernetes.
  5. Custom Kubernetes Workloads:
    • Utilize Kubernetes-native tools and configurations for specialized workloads.

Amazon Fargate

Amazon Fargate is a serverless compute engine designed to run containers without requiring users to manage the underlying infrastructure. It works seamlessly with Amazon ECS and Amazon EKS , enabling organizations to focus on deploying and managing containers rather than maintaining virtual machines.

1. Key Features of Amazon Fargate

  1. Serverless Container Management:
    • Automatically provisions and manages the infrastructure needed to run containers.
    • Eliminates the need to manage EC2 instances or clusters.
  2. Seamless Integration with ECS and EKS:
    • Works natively with both Amazon ECS and Amazon EKS for container orchestration.
    • Supports Kubernetes workloads through Amazon EKS.
  3. Granular Resource Allocation:
    • Specify CPU and memory requirements for each task or pod individually.
    • Pay only for the resources the containers consume.
  4. Secure by Default:
    • Containers are isolated at the task level using AWS security features.
    • Integrates with AWS Identity and Access Management (IAM) for fine-grained access control.
  5. High Scalability:
    • Automatically scales to handle demand without manual intervention.
    • Suitable for burst workloads and unpredictable traffic patterns.
  6. Cost-Efficiency:
    • No charges for idle resources; you pay only for the CPU and memory used during container execution.

2. How Amazon Fargate Works

  1. Integration with Orchestration Services:
    • Users define container workloads in ECS Task Definitions or Kubernetes Pod Specifications .
    • Fargate launches and manages these containers without requiring EC2 instances.
  2. Resource Isolation:
    • Each container task or pod is provisioned with its own CPU, memory, and networking resources, ensuring high security and performance.
  3. Networking and Security:
    • Integrates with Amazon VPC for isolated networking.
    • Supports task-level IAM roles to control permissions for individual workloads.

3. Amazon Fargate Pricing

Pricing for Fargate is based on two key factors:

  1. Compute Resources:
    • Charges for the vCPU and memory allocated to a task or pod, measured per second.
  2. Networking Costs:
    • Standard AWS data transfer charges apply for ingress and egress.
  3. Free Tier:
    • Includes 750 free hours per month for Fargate usage with ECS or EKS for the first 12 months.

4. When to Use Amazon Fargate ?

Amazon Fargate is ideal for:

  1. Serverless Container Workloads:
    • Deploying applications without managing underlying servers or clusters.
  2. Microservices Architectures:
    • Running containerized microservices that scale independently.
  3. Batch Processing:
    • Executing fault-tolerant jobs, such as data processing pipelines or machine learning inference.
  4. Event-Driven Applications:
    • Handling workloads triggered by events (e.g., file uploads, API calls).
  5. Development and Testing:
    • Quickly deploying containerized environments for testing and debugging.
  6. Unpredictable Workloads:
    • Managing workloads with highly variable traffic, ensuring cost-efficiency.

AWS Lightsail

AWS Lightsail is a simplified cloud platform designed for small-scale applications, offering an easy-to-use interface and predictable pricing. It provides virtual private servers (VPS), storage, and networking capabilities, making it an ideal choice for users new to cloud computing or looking for a straightforward solution.

1. Key Features of AWS Lightsail

  1. Simplified Virtual Servers:
    • Provides pre-configured virtual private servers (VPS) for web applications, development environments, or simple workloads.
    • Instances come with a fixed amount of CPU, memory, and storage, making it easy to choose a configuration.
  2. Predictable Pricing:
    • Fixed monthly plans starting as low as $3.50/month.
    • Includes data transfer allowances in the pricing, avoiding unexpected costs.
  3. Pre-Configured Blueprints:
    • Launch instances with pre-installed applications or operating systems, including:
      • WordPress, Joomla, Drupal, and Magento.
      • Linux and Windows OS options.
      • Developer stacks like LAMP, MEAN, or Node.js.
  4. Integrated Networking:
    • Includes a static IP address for each instance.
    • Simplified DNS management through Lightsail’s built-in domain name service.
  5. Storage and Databases:
    • Offers SSD-based block storage for better performance.
    • Managed databases with automatic backups, scaling, and monitoring.
  6. Scalability:
    • Ideal for small workloads but can be integrated with other AWS services (e.g., S3, RDS, or EC2) for scaling beyond Lightsail.
  7. Monitoring and Management:
    • Provides built-in metrics for instance performance monitoring.
    • Easily manage instances, snapshots, and networking via an intuitive console.

2. Lightsail Pricing

AWS Lightsail offers a predictable, all-inclusive pricing model:

  1. Monthly Plans:
    • Pricing is based on the instance plan chosen, with fixed CPU, RAM, storage, and data transfer allowances.
    • Plans range from $3.50/month for 512 MB RAM to $160/month for 32 GB RAM and multiple vCPUs.
  2. Data Transfer:
    • Each plan includes a specified amount of outbound data transfer (e.g., 1 TB for basic plans).
    • Additional data transfer is charged at a flat rate per GB.
  3. Free Tier:
    • Includes 750 hours of Lightsail usage for free during the first month, with 15 GB data transfer included.

3. Lightsail Use Cases

AWS Lightsail is designed for:

  1. Web Hosting:
    • Launch WordPress or other CMS platforms for small websites and blogs.
  2. Simple Applications:
    • Run lightweight web or mobile applications with minimal infrastructure requirements.
  3. Development and Testing:
    • Create isolated environments for coding, testing, and experimentation.
  4. Managed Databases:
    • Deploy simple databases for small-scale applications without complex configuration.
  5. E-Commerce Platforms:
    • Host small e-commerce stores using platforms like Magento or PrestaShop.

4. How AWS Lightsail Works ?

  1. Launch an Instance:
    • Choose a blueprint (e.g., WordPress) or operating system (Linux, Windows).
    • Select a plan based on your compute, memory, and storage needs.
  2. Simplified Networking:
    • Assign a static IP address and manage DNS records using the Lightsail console.
  3. Monitor and Scale:
    • Monitor instance performance and create snapshots for backup.
    • Integrate with other AWS services if additional resources are needed.
CATALOG
    FEATURED TAGS
    aws data-engineering docker