AWS Step Functions ingest pipeline

This pipeline signs masters and derivatives at upload time. An object arrives in an uploads bucket, and a Step Functions state machine runs four stages in sequence: extract, derive, sign, register. Signed bytes are written to a signed bucket fronted by CloudFront.

State machine

lib/ingest-stack.ts

import { Stack, type StackProps } from "aws-cdk-lib";
import { Construct } from "constructs";
import * as sfn from "aws-cdk-lib/aws-stepfunctions";
import * as tasks from "aws-cdk-lib/aws-stepfunctions-tasks";
import { NodejsFunction } from "aws-cdk-lib/aws-lambda-nodejs";
import { Bucket } from "aws-cdk-lib/aws-s3";

export class IngestStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    const signed = new Bucket(this, "Signed");

    const extract = new NodejsFunction(this, "extract", { entry: "lambdas/extract.ts" });
    const derive = new NodejsFunction(this, "derive", { entry: "lambdas/derive.ts" });
    const sign = new NodejsFunction(this, "sign", {
      entry: "lambdas/sign.ts",
      environment: {
        SIGNER_URL: "https://api.trylimbo.com",
        SIGNED_BUCKET: signed.bucketName,
      },
    });
    const register = new NodejsFunction(this, "register", { entry: "lambdas/register.ts" });
    signed.grantWrite(sign);

    const chain = new tasks.LambdaInvoke(this, "Extract", { lambdaFunction: extract, outputPath: "$.Payload" })
      .next(new tasks.LambdaInvoke(this, "Derive", { lambdaFunction: derive }))
      .next(new tasks.LambdaInvoke(this, "Sign", { lambdaFunction: sign }))
      .next(new tasks.LambdaInvoke(this, "Register", { lambdaFunction: register }));

    new sfn.StateMachine(this, "Ingest", {
      definitionBody: sfn.DefinitionBody.fromChainable(chain),
    });
  }
}

Configure the Sign Lambda with the signing endpoint (https://api.trylimbo.com) and your integration token. Store the token in Secrets Manager and inject it into the Lambda environment rather than committing it.

Sign Lambda

lambdas/sign.ts

import { S3Client, GetObjectCommand, PutObjectCommand } from "@aws-sdk/client-s3";
import { SignerClient, AssetSdk, embed } from "@limboai/verifox-sdk-issuer";

const s3 = new S3Client({});
const client = await SignerClient.connect(process.env.SIGNER_URL!, {
  headers: { authorization: `Bearer ${process.env.LIMBO_API_KEY}` },
});
const sdk = AssetSdk.create(client);
const SIGNED_BUCKET = process.env.SIGNED_BUCKET!;

export async function handler({ bucket, key, mime }: { bucket: string; key: string; mime: string }) {
  const obj = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
  const bytes = await obj.Body!.transformToByteArray();

  const { sidecar } = await sdk.sign(bytes, key, {
    origin: { sourceType: "digitalCapture" },
  }, { storage: { mode: "embedded" } });

  const signedBytes = embed(bytes, sidecar, mime);
  await s3.send(new PutObjectCommand({
    Bucket: SIGNED_BUCKET,
    Key: key,
    Body: signedBytes,
    ContentType: mime,
  }));

  return { key };
}

SignerClient.connect runs once at module load, so the connection is reused across warm invocations.

Notes

Large-master escape hatch

For masters that exceed Lambda's storage or timeout limits, replace the Sign task with an ECS Fargate container backed by EFS scratch storage. The SDK call remains identical.

Each derivative requires one signing call to the Signer, performed once at ingest. Set reservedConcurrentExecutions based on the Signer throughput allocated to you.