Skip to content Skip to footer
0 items - $0.00 0
Menu

AWS Should Be Easy by ithkuil

0CommentsShare Post
AWS Should Be Easy by ithkuil
News

AWS Should Be Easy by ithkuil

May 23, 2022
0Comments
Share This Article
Share Post
Newsletter

Sed ut perspiciatis unde.

Subscribe

Send to HN

why

aws is amazing, but it’s hard to see the forest for the trees.

aws should have fewer knobs.

aws should be easy to use, and hard to screw up.

aws should be fast, and fun.

aws should have a tldr.

how

deploy scale-to-zero systems as infrastructure sets that contain:

what

a simpler infrastructure as code specification that is easy to use and extend.

there are two interfaces:

the primary entrypoints are:

ensure operations are positive assertions. they assert that some named infrastructure exists, and is configured correctly, creating or updating it if needed.

many other entrypoints exist, and can be explored by type. they fall into two categories:

  • mutate aws state:

    view aws state:

    what about the rest of aws

    compared with the full aws api, systems specified as infrastructure sets:

    readme index

    install

    cli

    go install github.com/nathants/libaws@latest

export PATH=$PATH:$(go env GOPATH)/bin

    go api

    go get github.com/nathants/libaws@latest

    tldr

    define an infrastructure set

    name: test-infraset-${uid}

s3:
  test-bucket-${uid}:
    attr:
      - acl=private

lambda:
  test-lambda-${uid}:
    entrypoint: main.go
    attr:
      - concurrency=0
      - memory=128
      - timeout=60
    policy:
      - AWSLambdaBasicExecutionRole
    trigger:
      - type: s3
        attr:
          - test-bucket-${uid}

    package main

import (
	"context"
	"fmt"
	"github.com/aws/aws-lambda-go/events"
	"github.com/aws/aws-lambda-go/lambda"
)

func handleRequest(_ context.Context, e events.S3Event) (events.APIGatewayProxyResponse, error) {
	for _, record := range e.Records {
		fmt.Println(record.S3.Object.Key)
	}
	return events.APIGatewayProxyResponse{StatusCode: 200}, nil
}

func main() {
	lambda.Start(handleRequest)
}

    ensure the infrastructure set

    trigger the infrastructure set

    update the infrastructure set

    delete the infrastructure set

    usage

    explore the cli

    explore a cli entrypoint
    explore the go api
    )
    |——————————————————————————–|
    |f AcmClient func() *acm.ACM (Function) |
    |f AcmClientExplicit func(accessKeyID string, accessKeySecret string, region stri|
    |f AcmListCertificates func(ctx context.Context) ([]*acm.CertificateSummary, erro|
    |f Api func(ctx context.Context, name string) (*apigatewayv2.Api, error) (Functio|
    |f ApiClient func() *apigatewayv2.ApiGatewayV2 (Function) |
    |f ApiClientExplicit func(accessKeyID string, accessKeySecret string, region stri|
    |f ApiList func(ctx context.Context) ([]*apigatewayv2.Api, error) (Function) |
    |f ApiListDomains func(ctx context.Context) ([]*apigatewayv2.DomainName, error) (|
    |f ApiUrl func(ctx context.Context, name string) (string, error) (Function) |
    |f ApiUrlDomain func(ctx context.Context, name string) (string, error) (Function)|
    |… |
    |——————————————————————————–|
    }”>

    package main

import (
	"github.com/nathants/libaws/lib"
)

func main() {
    lib. (TAB =>)
      |--------------------------------------------------------------------------------|
      |f AcmClient func() *acm.ACM (Function)                                          |
      |f AcmClientExplicit func(accessKeyID string, accessKeySecret string, region stri|
      |f AcmListCertificates func(ctx context.Context) ([]*acm.CertificateSummary, erro|
      |f Api func(ctx context.Context, name string) (*apigatewayv2.Api, error) (Functio|
      |f ApiClient func() *apigatewayv2.ApiGatewayV2 (Function)                        |
      |f ApiClientExplicit func(accessKeyID string, accessKeySecret string, region stri|
      |f ApiList func(ctx context.Context) ([]*apigatewayv2.Api, error) (Function)     |
      |f ApiListDomains func(ctx context.Context) ([]*apigatewayv2.DomainName, error) (|
      |f ApiUrl func(ctx context.Context, name string) (string, error) (Function)      |
      |f ApiUrlDomain func(ctx context.Context, name string) (string, error) (Function)|
      |...                                                                             |
      |--------------------------------------------------------------------------------|
}

    explore simple examples

    explore complex examples

    • s3-ec2:
      • write to s3 in-bucket
      • which triggers lambda
      • which launches ec2 spot
      • which reads from in-bucket, writes to out-bucket, and terminates

    explore external examples

    infrastructure set

    an infrastructure set is defined by yaml or go struct and contains:

    typical usage

    • use infra-ls to view deployed infrastructure sets.

    • use infra-ensure to deploy an infrastructure set.

    • use infra-rm to remove an infrastructure set.

    • use lambdas in the infrastructure set to:

      • respond to triggers.
      • manage fleets of ec2 instances.
      • monitor services.
      • do anything.

    • rapidly iterate by responding to file changes:

      • on changes run: 
        infra-ensure infra.yaml --quick LAMBDA_NAME
      • this updates the lambda code without making other changes.

    design

    • there is no implicit coordination.

      • if you aren’t already serializing your infrastructure mutations, lock around dynamodb.

    • there are only two state locations:

      • aws.
      • your code.

    • aws resources are uniquely identified by name.

      • all aws resources share a private namespace scoped to account/region. use good names.
      • except s3, which shares a public namespace scoped to earth. use better names.

    • mutative operations manipulate aws state.

      • mutative operations are idempotent. if they fail due to a transient error, run them again.
      • mutative operations can --preview. no output means no changes.

    • ensure are mutative operations that create or update infrastructure.

    • rm are mutative operations that delete infrastructure.

    • ls, get, scan, and describe operations are non-mutative.

    • multiple infrastructure sets can be deployed into the same account/region.

    tradeoffs

    • no attempt is made to avoid vendor lock-in.

      • migrating between cloud providers will always be non-trivial.
      • attempting to mitigate future migrations has more cost than benefit in the typical case.

    • ensure operations are positive assertions. they assert that some named infrastructure exists, and is configured correctly, creating or updating it if needed.

      • positive assertions CANNOT remove top level infrastructure, but CAN remove configuration from them.

      • removing a trigger, policy, or allow WILL remove that from the lambda.

      • removing policy, or allow WILL remove that from the instance-profile.

      • removing a security-group WILL remove that from the vpc.

      • removing a rule WILL remove that from the security-group.

      • removing an attr WILL remove that from a sqs, s3, dynamodb, or lambda.

      • removing a keypair, vpc, instance-profile, sqs, s3, dynamodb, or lambda WON’T remove that from the account/region.

        • the operator decides IF and WHEN top level infrastructure should be deleted, then uses an rm operation to do so.

        • as a convenience, infra-rm will remove ALL infrastructure CURRENTLY declared in an infra.yaml.

    infra.yaml

    use an infra.yaml file to declare an infrastructure set. the schema is as follows:

    name: VALUE
lambda:
  VALUE:
    entrypoint: VALUE
    policy:     [VALUE ...]
    allow:      [VALUE ...]
    attr:       [VALUE ...]
    require:    [VALUE ...]
    env:        [VALUE ...]
    include:    [VALUE ...]
    trigger:
      - type: VALUE
        attr: [VALUE ...]
s3:
  VALUE:
    attr: [VALUE ...]
dynamodb:
  VALUE:
    key:  [VALUE ...]
    attr: [VALUE ...]
sqs:
  VALUE:
    attr: [VALUE ...]
vpc:
  VALUE:
    security-group:
      VALUE:
        rule: [VALUE ...]
keypair:
  VALUE:
    pubkey-content: VALUE
instance-profile:
  VALUE:
    allow: [VALUE ...]
    policy: [VALUE ...]

    environment variable substitution

    anywhere in infra.yaml you can substitute environment variables from the caller’s environment:

    • example: 

      s3:
  test-bucket-${uid}:
    attr:
      - versioning=${versioning}

    the following variables are defined during deployment, and are useful in allow declarations:

    • ${API_ID} the id of the apigateway v2 api created by an api trigger.

    • ${WEBSOCKET_ID} the id of the apigateway v2 websocket created by a websocket trigger.

    name

    defines the name of the infrastructure set.

    • schema:

    • example:

    s3

    defines a s3 bucket:

    • the following attributes can be defined:

      • acl=VALUE, values: public | private, default: private
      • versioning=VALUE, values: true | false, default: false
      • metrics=VALUE, values: true | false, default: true
      • cors=VALUE, values: true | false, default: false
      • ttldays=VALUE, values: 0 | n, default: 0

    • schema:

    • example:

      s3:
  test-bucket:
    attr:
      - versioning=true
      - acl=public

    dynamodb

    defines a dynamodb table:

    • specify key as:

      • NAME:ATTR_TYPE:KEY_TYPE

    • the following attributes can be defined:

      • ProvisionedThroughput.ReadCapacityUnits=VALUE, shortcut: read=VALUE, default: 0
      • ProvisionedThroughput.WriteCapacityUnits=VALUE, shortcut: write=VALUE, default: 0
      • StreamSpecification.StreamViewType=VALUE, shortcut: stream=VALUE, default: 0
      • LocalSecondaryIndexes.INTEGER.IndexName=VALUE
      • LocalSecondaryIndexes.INTEGER.Key.INTEGER=NAME:ATTR_TYPE:KEY_TYPE
      • LocalSecondaryIndexes.INTEGER.Projection.ProjectionType=VALUE
      • LocalSecondaryIndexes.INTEGER.Projection.NonKeyAttributes.INTEGER=VALUE
      • GlobalSecondaryIndexes.INTEGER.IndexName=VALUE
      • GlobalSecondaryIndexes.INTEGER.Key.INTEGER=NAME:ATTR_TYPE:KEY_TYPE
      • GlobalSecondaryIndexes.INTEGER.Projection.ProjectionType=VALUE
      • GlobalSecondaryIndexes.INTEGER.Projection.NonKeyAttributes.INTEGER=VALUE
      • GlobalSecondaryIndexes.INTEGER.ProvisionedThroughput.ReadCapacityUnits=VALUE
      • GlobalSecondaryIndexes.INTEGER.ProvisionedThroughput.WriteCapacityUnits=VALUE

    • schema:

      dynamodb:
  VALUE:
    key:
      - NAME:ATTR_TYPE:KEY_TYPE
    attr:
      - VALUE

    • example:

      dynamodb:
  stream-table:
    key:
      - userid:s:hash
      - timestamp:n:range
    attr:
      - stream=keys_only
  auth-table:
    key:
      - id:s:hash
    attr:
      - write=50
      - read=150

    sqs

    defines a sqs queue:

    • the following attributes can be defined:

      • DelaySeconds=VALUE, shortcut: delay=VALUE, default: 0
      • MaximumMessageSize=VALUE, shortcut: size=VALUE, default: 262144
      • MessageRetentionPeriod=VALUE, shortcut: retention=VALUE, default: 345600
      • ReceiveMessageWaitTimeSeconds=VALUE, shortcut: wait=VALUE, default: 0
      • VisibilityTimeout=VALUE, shortcut: timeout=VALUE, default: 30

    • schema:

      sqs:
  VALUE:
    attr:
      - VALUE

    • example:

      sqs:
  test-queue:
    attr:
      - delay=20
      - timeout=300

    keypair

    defines an ec2 keypair.

    • example:

      keypair:
  VALUE:
    pubkey-content: VALUE

    • example:

      keypair:
  test-keypair:
    pubkey-content: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICVp11Z99AySWfbLrMBewZluh7cwLlkjifGH5u22RXor

    vpc

    defines a default-like vpc with an internet gateway and public access.

    • schema:

    • example:

    security group

    defines a security group on a vpc

    • schema:

      vpc:
  VALUE:
    security-group:
      VALUE:
        rule:
          - PROTO:PORT:SOURCE

    • example:

      vpc:
  test-vpc:
    security-group:
      test-sg:
        rule:
          - tcp:22:0.0.0.0/0

    instance profile

    defines an ec2 instance profile.

    • schema:

      instance-profile:
  VALUE:
    allow:
      - SERVICE:ACTION ARN
    policy:
      - VALUE

    • example:

      instance-profile:
  test-profile:
    allow:
      - s3:* *
    policy:
      - AWSLambdaBasicExecutionRole

    lambda

    defines a lambda.

    • schema:

    • example:

    entrypoint

    defines the code of the lambda. it is one of:

    • a python file.

    • a go file.

    • an ecr container uri.

    • schema:

      lambda:
  VALUE:
    entrypoint: VALUE

    • example:

      lambda:
  test-lambda:
    entrypoint: main.go

    attr

    defines lambda attributes. the following can be defined:

    • concurrency defines the reserved concurrent executions, default: 0

    • memory defines lambda ram in megabytes, default: 128

    • timeout defines the lambda timeout in seconds, default: 300

    • logs-ttl-days defines the ttl days for cloudwatch logs, default: 7

    • schema:

      lambda:
  VALUE:
    attr:
      - KEY=VALUE

    • example:

      lambda:
  test-lambda:
    attr:
      - concurrency=100
      - memory=256
      - timeout=60
      - logs-ttl-days=1

    policy

    defines policies on the lambda’s iam role.

    • schema:

      lambda:
  VALUE:
    policy:
      - VALUE

    • example:

      lambda:
  test-lambda:
    policy:
      - AWSLambdaBasicExecutionRole

    allow

    defines allows on the lambda’s iam role.

    • schema:

      lambda:
  VALUE:
    allow:
      - SERVICE:ACTION ARN

    • example:

      lambda:
  test-lambda:
    allow:
      - s3:* *
      - dynamodb:* arn:aws:dynamodb:*:*:table/test-table

    env

    defines environment variables on the lambda:

    • schema:

      lambda:
  VALUE:
    env:
      - KEY=VALUE

    • example:

      lambda:
  test-lambda:
    env:
      - kind=production

    include

    defines extra content to include in the lambda zip:

    • this is ignored when entrypoint is an ecr container uri.

    • schema:

      lambda:
  VALUE:
    include:
      - VALUE

    • example:

      lambda:
  test-lambda:
    include:
      - ./cacerts.crt
      - ../frontend/public/*

    require

    defines dependencies to install with pip in the virtualenv zip.

    • this is ignored unless the entrypoint is a python file.

    • schema:

      lambda:
  VALUE:
    require:
      - VALUE

    • example:

      lambda:
  test-lambda:
    require:
      - fastapi==0.76.0

    trigger

    defines triggers for the lambda:

    • schema:

      lambda:
  VALUE:
    trigger:
      - type: VALUE
        attr:
          - VALUE

    • example:

      lambda:
  test-lambda:
    trigger:
      - type: dynamodb
        attr:
          - test-table

    trigger types

    api

    defines an apigateway v2 http api:

    • add a custom domain with attr: domain=api.example.com

    • add a custom domain and update route53 with attr: dns=api.example.com

      • this domain, or its parent domain, must already exist as a hosted zone in route53.

      • this domain, or its parent domain, must already have an acm certificate with subdomain wildcard.

    • schema:

      lambda:
  VALUE:
    trigger:
      - type: api
        attr:
          - VALUE

    • example:

      lambda:
  test-lambda:
    trigger:
      - type: api
        attr:
          - dns=api.example.com
    websocket

    defines an apigateway v2 websocket api:

    • add a custom domain with attr: domain=ws.example.com

    • add

Read More

Tags:
0Likes
Leave a comment

Leave a comment

In the Shadows of Innovation”

© 2025 HackTech.info. All Rights Reserved.

Sign Up to Our Newsletter

Be the first to know the latest updates

Whoops, you're not connected to Mailchimp. You need to enter a valid Mailchimp API key.