AWS DEA-C01 Data Engineering

Getting Started with AWS

Posted by Vineet Kumar Loyer on Sunday, December 29, 2024

AWS Core Services

As of December 2024, Amazon Web Services (AWS) offers over 200 fully featured services across various categories, including computing, storage, databases, analytics, networking, mobile, developer tools, management tools, IoT, security, and enterprise applications.

It’s important to note that AWS continuously expands its service offerings to meet evolving customer needs, so the exact number of services may change over time.

However, we can categorize the services in 11 core sectors as shown in image below.

Fig-1: AWS service sectors

Fig-1: AWS service sectors

Before getting started with details, let’s touch upon the basics of cloud and AWS.

What’s Cloud?

Cloud provides –

  • On-Demand Self-Service:
    • Users can provision and terminate resources (like servers or storage) whenever they need via a user interface or command-line interface, without needing manual intervention from the service provider.
    • Example: Launching an EC2 instance in AWS within minutes.
  • Broad Network Access:
    • Cloud services are accessible over the internet from any device (e.g., laptops, phones, tablets) using standard protocols and methods.
    • Example: Accessing AWS S3 or Google Drive from a browser or mobile app.
  • Resource Pooling:
    • Resources (e.g., computing power, storage) are pooled and shared among multiple users, leveraging economies of scale to provide cost-effective services.
    • Example: A shared pool of servers dynamically allocated to users based on demand.
  • Rapid Elasticity:
    • The cloud can automatically scale resources up or down in real-time to match system load or demand.
    • Example: Scaling EC2 instances during a traffic spike on an e-commerce site and reducing them during off-peak hours.
  • Measured Service:
    • Resource usage is tracked and billed based on consumption, ensuring users only pay for what they use (e.g., per-hour, per-GB storage).
    • Example: Paying for AWS Lambda based on the number of requests and execution time.

Public Vs Private Vs Multi-cloud and Hybrid-cloud models

1. Public Cloud:

  • Using a single cloud service provider (like AWS, Azure, or Google Cloud) to host and manage resources.
  • Features:
    • Accessible to multiple organizations.
    • Cost-effective due to shared infrastructure.
    • Scalable and flexible with pay-as-you-go pricing.
  • Use Case: Startups or businesses with limited IT budgets and no strict compliance requirements.

2. Private Cloud:

  • A cloud environment dedicated to a single organization, hosted on-premises or on a private third-party platform.
  • Requirements: Must meet these five characteristics of cloud computing:
    1. On-Demand Self-Service
    2. Broad Network Access
    3. Resource Pooling
    4. Rapid Elasticity
    5. Measured Service
  • Features:
    • Higher security and control.
    • Customized to specific business needs.
    • Often more expensive to maintain.
  • Use Case: Organizations with stringent data privacy, regulatory, or performance requirements (e.g., finance, healthcare).

3. Multi-Cloud:

  • Utilizing more than one public cloud provider (e.g., AWS + Azure) within a single deployment to leverage specific strengths of each provider.
  • Features:
    • Avoid vendor lock-in.
    • Improved resilience and redundancy.
    • Optimized performance by using the best cloud for specific tasks.
  • Use Case: Large enterprises aiming to diversify workloads and reduce risks of reliance on one provider.

4. Hybrid Cloud:

  • Integrating private and public cloud infrastructures to work seamlessly within one environment.
    • Note: This is distinct from using public cloud alongside legacy on-premises hardware; a hybrid cloud involves private cloud infrastructure that adheres to cloud computing principles.
  • Features:
    • Flexibility to run sensitive workloads in private clouds while leveraging public clouds for scalability.
    • Centralized management of resources across environments.
  • Use Case: Organizations that need to handle sensitive data (in private cloud) while benefiting from public cloud scalability (e.g., disaster recovery, burst computing).
Feature Public Cloud Private Cloud Multi-Cloud Hybrid-Cloud
Ownership Third-party provider(AWS/Azure/GCP) Single Org. Multiple public providers Combination of private and public clouds
Cost Pay-as-you-go Higher (setup & maintenance) Pay-as-you-go for each provider Variable (private setup + public fees)
Security Shared responsibility High control Shared responsibility High control + scalability
Scalability High Limited by hardware High High

Cloud Service Models

Different cloud service models determine which components are managed by the vendor and which are managed by the customer. Here’s a breakdown:

1. On-Premises

  • The customer manages all components, from hardware and storage to applications and networking, within their own facilities.
  • Features:
    • Most control and flexibility.
    • High IT overhead and cost for maintenance and upgrades.
    • Best suited for organizations with strict compliance or legacy applications.
  • Example: Running your own data center.

2. Data Center Hosting

  • The customer places their own hardware in a third-party vendor’s facility, where the vendor manages the building, cooling, and power.
  • Features:
    • Vendor manages facilities like power, cooling, and security.
    • Customer manages hardware, software, and applications.
    • Reduced physical infrastructure costs but retains operational overhead.
  • Example: Colocating servers in a managed data center.

3. Infrastructure as a Service (IaaS)

  • The vendor manages the physical infrastructure, virtualization, and the operating system. Customers focus on managing applications and data.
  • Features:
    • Pay-per-use pricing for computing resources (e.g., virtual machines, storage).
    • Risk reduction since hardware management is outsourced.
    • Flexible and scalable for dynamic workloads.
  • Examples: AWS EC2, Google Compute Engine, Microsoft Azure VMs.

4. Platform as a Service (PaaS)

  • The vendor provides a complete platform for building, running, and managing applications without worrying about underlying infrastructure.
  • Features:
    • Vendor manages infrastructure, runtime environment, and middleware.
    • Customer manages applications and data.
    • Ideal for developers to focus solely on building and deploying applications.
  • Examples: AWS Elastic Beanstalk, Google App Engine, Microsoft Azure App Services.

5. Software as a Service (SaaS)

  • The customer uses software applications hosted and fully managed by the vendor. All infrastructure and platform elements are abstracted away.
  • Features:
    • Almost zero IT involvement required.
    • Customers pay for software usage as a subscription or per user.
    • Limited control over customization or configurations.
  • Examples: Gmail, Microsoft Office 365, Netflix, Salesforce.
Feature On-Premises Data Center Hosting IaaS PaaS SaaS
Management Fully by customer Customer manages hardware Vendor manages infrastructure Vendor manages platform Vendor manages everything
Flexibility High Medium Medium-High Medium Low
IT Overhead Very High High Medium Low Very Low
Cost Efficiency Low Moderate Pay-as-you-go Pay-as-you-go Subscription based
Use Cases Legacy apps, full control Reduce physical overhead Scalable workloads Application development End-user software needs

AWS Networking overview

AWS networking is designed to ensure secure, scalable, and high-performing communication within and outside the cloud environment. Key concepts include public and private zones, which determine how services interact with the internet and private environments.

1. Public Internet

  • AWS services are connected to the public internet for global access using standard internet protocols. However, AWS itself is not directly on the public internet but operates adjacent to it for secure communication.

2. AWS Public Zone

  • Services like S3 buckets are hosted in the public zone, allowing access from the public internet when configured.
  • Access requires proper permissions; public connectivity doesn’t mean unrestricted access.

3. AWS Private Zone

  • The private zone is isolated from the public internet. Public access is only allowed when explicitly configured. This involves projecting private services into the public zone to allow controlled inbound or outbound connections.

AWS Global Infrastructure

AWS provides a robust global infrastructure designed for scalability, resilience, and low latency. It includes Regions, Availability Zones (AZs), and Edge Locations.

Fig-2: AWS regions

Fig-2: AWS regions

1. Regions

  • A geographic area where AWS deploys a full set of infrastructure.
  • Features:
    • Regions are connected by high-speed networking.
    • Some services, like EC2, are region-specific, while others, like IAM, are global.
    • Advantages of Regions:
      • Geographical Separation: Provides isolated fault domains and disaster recovery options.
      • Geopolitical Separation: Adapts to regional laws and political stability.
      • Location Control: Improves performance by deploying infrastructure closer to users.
  • Examples of Regions:
    • Asia Pacific (Sydney) [Code: ap-southeast-2]
    • North America (Ohio)
    • Europe (London)

2. Edge Locations

  • Localized points of presence designed to cache and deliver content with low latency and high speed to end-users.

  • Features:

    • AWS operates hundreds of edge locations globally, often outside the main regions.
    • Edge locations are integral to content delivery networks (CDNs) like Amazon CloudFront .
    • They serve cached data, reducing the load on origin servers and ensuring faster delivery.
    • Connected to AWS regions through high-speed networks, enabling seamless data retrieval.
    • Improve user experience for applications involving streaming, gaming, and real-time data delivery.
    • Provide services like DNS resolution through Amazon Route 53 and DDoS mitigation through AWS Shield .

3. Availability Zones (AZs)

  • Isolated locations within a region designed for fault tolerance and resilience.

  • Features:

    • AWS provides 2-6 AZs per region.
    • AZs are connected by high-speed, redundant networks.
    • Each AZ includes independent power, cooling, and networking.
    • Distribute workloads across AZs to ensure high availability.
    Fig-3: AWS regions-AZs-Edge Locations

    Fig-3: AWS regions-AZs-Edge Locations

Service Resilience Levels

AWS services are designed with varying levels of resilience:

  1. Globally Resilient:
    • Services like IAM or Route 53 operate globally and replicate data across multiple regions to ensure no single point of failure.
  2. Region Resilient:
    • Services like RDS or S3 are region-specific and replicate data across multiple AZs within a region for high availability.
  3. AZ Resilient:
    • Services like specific EC2 instances run within a single AZ. Redundant equipment ensures minimal impact during hardware failures, but such setups should not be relied upon for critical workloads.

AWS Shared Responsibility Model

When hosting applications in the cloud, AWS follows a shared responsibility model :

  1. AWS’s Responsibility – Security of the Cloud:
    • AWS handles the physical security of data centers, including compute, storage, and networking hardware.
    • They protect the global infrastructure, such as inter-region cables, as well as the hardware and software that run AWS services.
  2. Your Responsibility – Security in the Cloud:
    • You manage and secure your data, ensuring it is protected during storage (at rest) and transfer (in transit).
    • You control access to your resources, including permissions, and handle configurations for the services you use.
CATALOG
    FEATURED TAGS
    aws data-engineering docker