> For the complete documentation index, see [llms.txt](https://ysfang82.gitbook.io/development-notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ysfang82.gitbook.io/development-notes/aws-notes/service/data-engineering/batch.md).

# Batch

**Introduction**

* Run batch jobs **as Docker images**
* Dynamic provisioning of the instances (EC2, Spot Fleet, ECS) in VPC
* Optimal quantity and type based on volume and requirements
* No need to manage clusters, fully severless, pay for EC2 instances
* Use case:
  * batch process of images
  * running thousands of concurrent jobs
* Schedule Batch jobs using CloudWatch Events
* Orchestrate Batch jobs using AWS Step Functions

**Feature**

* Compute Environment
  * Managed compute environment
    * AWS Batch manages the capacity and instance types within the environment. (Doesn't need to configure ASG.)
    * Can choose On-demand / Spot Instances / Spot Fleet
    * Can set a maximum price for Spot Instances
    * Launched within your own VPC
      * If launch within your own private subnet, make sure it has access to the ECS services.
      * Either use a NAT Gateway or using VPC Endpoints for ECS.
  * Unmanaged compute environment
    * You control and manage instance configuration, provisioning and scaling.
* Multi Node Mode (for large scale, high performance computing)
  * 1 main node, many child node. 
  * Leverage multiple EC2 / ECS instances at the same time
  * Doesn't work with Spot Instances
  * Good for tightly coupled workloads
  * Represents a single job, and specific how many nodes to create for the job
  * Works better if your EC2 launch mode is a placement group "cluster"
* Lambda vs Batch
  * Lambda
    * with time limit
    * limited runtimes
    * limited temporary disk space
    * Serverless
  * Batch
    * without time limit
    * Any runtime as long as it's packaged as a Docker Image
    * Rely on EBS / instance store for disk space
    * Relies on EC2 (can be managed by AWS)

**Scenario**

* Batch architecture example

![AWS Batch architecture example](https://3303577320-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M4cDbT2F2VmcAohuhSN%2F-MEWO8VkqHX2Zm_tO3Fr%2F-MEWQiS3_w60Zn75AQap%2FScreen%20Shot%202020-08-12%20at%202.49.14%20PM.png?alt=media\&token=7527ba2f-736b-43d1-a636-b2368da83fd9)