Lambdas

Five TypeScript Lambda packages live under app/packages/lambda/. Each builds via esbuild to a single CJS file at dist/handler.cjs; Terraform's archive_file data source zips that at apply time.

npm run app:build:lambdas        # produces every dist/handler.cjs

The four core Lambdas (interactions, followup, update-dns, watchdog) read AWS region from process.env.AWS_REGION_ (trailing underscore — AWS_REGION is reserved by the Lambda runtime). Terraform sets the variable with that name in every function definition, including efs-seeder's — but efs-seeder never reads it, since it makes no AWS SDK calls (see below).

interactions

Package	@hyveon/lambda-interactions
Trigger	Lambda Function URL (public HTTPS, auth_type = NONE, CORS for https://discord.com) — Discord POSTs every interaction here.
Terraform	terraform/aws/interactions.tf. Output: interactions_invoke_url.
IAM	dynamodb:GetItem on the Discord table, secretsmanager:GetSecretValue on the public-key secret, lambda:InvokeFunction on the followup Lambda.
Env vars	AWS_REGION_, TABLE_NAME, DISCORD_PUBLIC_KEY_SECRET_ARN, FOLLOWUP_LAMBDA_NAME, GAME_NAMES, HOSTED_ZONE_NAME.

Behaviour

1. Signature verify — reads x-signature-ed25519 + x-signature-timestamp headers, fetches the public key from Secrets Manager (5-minute cache via @hyveon/shared/secrets), and verifies with @noble/ed25519 over timestamp + rawBody. Rejects 401 on mismatch; without a valid signature Discord stops routing to the URL.
2. PING (type === 1) → respond { type: 1 } (PONG).
3. Autocomplete (type === 4) → filter GAME_NAMES by the user's partial input, then filter by canRun() against the DynamoDB config, return choices synchronously. No ECS calls — has to fit in Discord's 3-second budget.
4. Application command (type === 2):
   • Confirm the guild is in allowedGuilds.
   • Confirm canRun(cfg, { userId, roleIds, game, action }).
   • Return a deferred ack (type: 5, ephemeral flag 64) immediately.
   • Async-invoke the followup Lambda (InvokeCommand with InvocationType: 'Event') with a FollowupPayload (kind, applicationId, interactionToken, userId, guildId, roleIds, optional game).

If anything above throws, Discord sees either a non-200 response or a silent timeout — the user gets no reply, which is the correct failure mode (replying with an error would require another signed response, which we can't forge).

followup

Package	@hyveon/lambda-followup
Trigger	Async invoke from the interactions Lambda (InvocationType: 'Event'). Not exposed externally.
Terraform	terraform/aws/followup.tf.
IAM	ecs:RunTask / StopTask / ListTasks / DescribeTasks / TagResource, iam:PassRole (task execution role — required for RunTask with Fargate), ec2:DescribeNetworkInterfaces, dynamodb:GetItem / PutItem, secretsmanager:GetSecretValue on the public key (only read for downstream calls in some paths).
Env vars	AWS_REGION_, TABLE_NAME, ECS_CLUSTER, SUBNET_IDS (comma-separated), SECURITY_GROUP_ID, DOMAIN_NAME, GAME_NAMES.

Behaviour

Event is a FollowupPayload:

type FollowupPayload = {
  kind: 'start' | 'stop' | 'status' | 'list'
  applicationId: string
  interactionToken: string
  userId: string
  guildId: string
  roleIds: string[]
  game?: string
}

1. Re-fetch the Discord config (defensive re-check — the interactions Lambda already ran canRun, but config could change between the two invocations).
2. Dispatch by kind:
   • start — runStart(): ecs.runTask with the game's task definition family ({game}-server), public-IP-enabled network config, the Fargate launch type. If successful, call putPending() (PENDING#{taskArn} with 15-min TTL); then PATCH the original interaction with "starting …".
   • stop — find the running task via findRunningTask(), call ecs.stopTask, PATCH "stopping …".
   • status — single-game getStatus() (ListTasks → DescribeTasks → ec2.describeNetworkInterfaces for the public IP), PATCH with the resolved state + hostname/IP.
   • list — status for every game the user has at least status permission for, joined into one ephemeral message.
3. PATCH the Discord webhook at https://discord.com/api/v10/webhooks/{applicationId}/{interactionToken}/messages/@original. Valid for 15 minutes after the original interaction.

Failure modes:

• ECS call fails → error message in the PATCH body.
• CloudWatch ENI lag (task RUNNING but no ENI yet) → getStatus() returns { state: 'error', message: ... }; caller sees it in Discord.
• DynamoDB write fails (for start) → logged, PATCH still happens so user sees "starting"; but update-dns won't later PATCH with the final IP because the pending row doesn't exist.
• Discord PATCH fails (stale token, network) → logged; user's deferred message is not edited.

update-dns

Package	@hyveon/lambda-update-dns
Trigger	EventBridge rule on source: aws.ecs, detail-type: 'ECS Task State Change', lastStatus in ['RUNNING', 'STOPPED'].
Terraform	terraform/aws/route53.tf.
IAM	route53:ChangeResourceRecordSets, route53:ListResourceRecordSets, ecs:DescribeTasks, ec2:DescribeNetworkInterfaces, dynamodb:GetItem / DeleteItem.
Env vars	HOSTED_ZONE_ID, DOMAIN_NAME, GAME_NAMES, DNS_TTL, AWS_REGION_, TABLE_NAME.

Behaviour

Event shape (simplified):

{
  "detail": {
    "lastStatus": "RUNNING | STOPPED",
    "taskArn": "...",
    "clusterArn": "...",
    "group": "family:palworld-server"
  }
}

1. Parse the task family from detail.group, map to a game via FAMILY_TO_GAME. Skip unknown families. Every game — including https = true ones, which terminate TLS in-task via a Caddy sidecar and share the task's public IP — follows the same path below.
2. On RUNNING: resolvePublicIp() — retries up to 5 times with 3-second sleeps to survive ENI attach lag; then upsertDns() writes an A record {game}.{domain} → IP with DNS_TTL.
3. On STOPPED: read the current record, verify its IP, deleteDns().
4. On RUNNING: call notifyDiscordIfPending() — look up PENDING#{taskArn} in DynamoDB, format a final status message (including the resolved public IP), PATCH the original Discord interaction, delete the pending row.

Failure modes:

• IP not available after 5 retries → log warning, skip; the handler returns {status: 'error', reason: 'no_ip'}. No retry is scheduled — the task stays up but unreachable by DNS until another RUNNING/STOPPED state-change event fires for it (e.g. the task is stopped and started again).
• Route 53 call fails → log, continue. There is no retry and no backstop — the watchdog only issues StopTask, it never touches Route 53 — so a stale record persists until another RUNNING/STOPPED event fires for that game (e.g. the next start/stop cycle) or someone deletes it manually.
• Pending row missing (expired / never written / stop flow) → skip the Discord PATCH; no user-visible issue.
• Discord PATCH fails (stale token) → log, continue.

watchdog

Package	@hyveon/lambda-watchdog
Trigger	EventBridge schedule at rate(${watchdog_interval_minutes} minute(s)). No event payload.
Terraform	terraform/aws/watchdog.tf.
IAM	ecs:ListTasks / DescribeTasks / StopTask / TagResource / ListTagsForResource, cloudwatch:GetMetricStatistics.
Env vars	ECS_CLUSTER, GAME_NAMES, IDLE_CHECKS, MIN_PACKETS, CHECK_WINDOW_MINUTES, AWS_REGION_.

Behaviour

1. ListTasks(desiredStatus: RUNNING) across the cluster. Paginates.
2. DescribeTasks on the batch to get attachments and tags.
3. For each task:
   • Resolve the ENI ID from attachments.
   • cloudwatch.GetMetricStatistics → AWS/EC2/NetworkPacketsIn over the last CHECK_WINDOW_MINUTES. If the call fails, assume active (fails-safe for fresh tasks with no metrics yet).
   • If packets < MIN_PACKETS:
     • Increment the idle_checks tag.
     • If the counter reaches IDLE_CHECKS:
       • StopTask with reason Watchdog: idle for {N} minutes. This Lambda does not touch DNS directly — the resulting STOPPED state-change event is what triggers update-dns's record deletion. Deleting the record here first would risk leaving a running task unreachable by DNS if StopTask then failed.
     • Otherwise persist the incremented counter via TagResource.
   • Else (active), if the counter is non-zero, reset it to 0.

Watchdog state lives only in the idle_checks ECS task tag. It's inherently scoped to the task — when the task goes away, so does the state, which is exactly what we want. Do not move it to DDB/SSM.

Failure modes:

• CloudWatch query fails → treated as active (no accidental shutdowns).
• Tagging fails → logged; a task might hang around a cycle longer than intended.
• StopTask fails → logged; next schedule tick retries.

efs-seeder

Package	@hyveon/lambda-efs-seeder
Trigger	aws_lambda_invocation in terraform/aws/efs-seeder.tf, run synchronously during terraform apply. Not exposed externally, not event-driven, and not part of the always-on control flow the other four Lambdas belong to.
Terraform	terraform/aws/efs-seeder.tf. One function per game that declares file_seeds — this Lambda is conditionally created, not fixed like the other four. Games with no file_seeds get no seeder function, no seeder IAM role, and no seeder log group.
IAM	Per-game role: logs:CreateLogGroup/CreateLogStream/PutLogEvents, ec2:CreateNetworkInterface/DescribeNetworkInterfaces/DeleteNetworkInterface (required for Lambda VPC networking), elasticfilesystem:ClientMount/ClientWrite scoped to the shared EFS filesystem.
Env vars	AWS_REGION_ — set by Terraform for consistency with the other Lambdas, but unused: this handler makes no AWS SDK calls (see below).

Behaviour

The Lambda mounts the game's first volume's EFS access point at /mnt/efs via file_system_config (VPC-attached, using the same public subnets and a dedicated efs-seeder security group whose egress is currently unrestricted — all protocols to 0.0.0.0/0, not scoped to NFS; see terraform/aws/efs-seeder.tf) and receives { game, seeds, container_path } as its invocation payload — container_path is volumes[0].container_path.

1. For each FileSeed in seeds: strip the container_path prefix from path and resolve the remainder under /mnt/efs, rejecting anything that resolves outside the mount point (path-traversal guard) or has no file component left after stripping the prefix.
2. Decode content (UTF-8 text) or content_base64 (binary) — exactly one must be set — and validate mode is a 3–4 digit octal string (default "0644").
3. mkdirSync(..., { recursive: true }) then writeFileSync with that mode.
4. Throw on any error — aws_lambda_invocation is a synchronous Terraform resource, so a thrown error surfaces directly as a terraform apply failure rather than an async CloudWatch-only failure.

Unlike the other four Lambdas, this handler imports no @aws-sdk/* package at all — it only touches the filesystem (fs, path), which is why the AWS_REGION_ env var Terraform sets on it is never actually read.

Re-trigger behaviour: the aws_lambda_invocation.efs_seeder resource's triggers block is { seeds_hash = sha256(jsonencode(each.value.file_seeds)) }, so terraform apply only re-invokes the Lambda for a game when that game's file_seeds content actually changes — an apply with unchanged seeds is a no-op for this Lambda. Removed seed entries are not deleted from EFS; clean them up via the FileBrowser task.

Failure modes:

• Path validation error (traversal, missing file component, both/neither of content/content_base64 set, invalid mode) → thrown synchronously, fails the terraform apply.
• EFS mount not ready (mount targets still propagating) → Lambda invocation fails; terraform apply reports the error, retry the apply once mount targets are up (~30 s after aws_efs_mount_target creation).

The /server-start critical path, assembled

User types /server-start palworld
  → Discord POSTs to interactions Function URL
    → interactions verifies + returns type:5 ack + async-invokes followup
      → followup RunTask + put PENDING#{arn} + PATCH @original "starting"
        → ECS reaches RUNNING
          → EventBridge fires update-dns
            → update-dns resolves IP + UPSERT A + get+delete PENDING#{arn}
              + PATCH @original "🟢 running — palworld.example.com"

Every Lambda has its own CloudWatch log group; when a step goes wrong, the group with the latest events is the one that last ran. The interactions Lambda logs the async invoke of followup line; if you see that but no followup logs, check IAM.