migrate-sdk
Cookbook

Files SDK

Read records from JSON files, copy a folder with the CLI, and save progress using Files SDK.

Files SDK can read your source files, write destination files, and save migration progress. Each role can use its own storage client.

Choose the Source by what you want to migrate:

  • Records inside files: use DocumentSource with a Files SDK Document Fetcher.
  • Whole files: use FilesSDKSource.objects, with one Source Item per file.

Install

Requires Node.js 22.19.0+, Effect 4, and files-sdk 3.1.1+:

pnpm add migrate-sdk @migrate-sdk/files-sdk files-sdk effect @effect/platform-node

The local filesystem adapter needs no cloud credentials or extra provider packages.

Read records from JSON files

Suppose source-storage/input/ contains JSON files like this:

{ "products": [{ "id": "p1" }, { "id": "p2" }] }

This Source reads those files and emits each product as a Source Item:

document-source.ts
import {
  FilesSDKDocumentFetchers,
  type FilesSDKObjectDiscoveryOptions,
} from "@migrate-sdk/files-sdk/source";
import { Schema } from "effect";
import { Files as FilesSDKClient } from "files-sdk";
import { Files } from "files-sdk/effect";
import { fs } from "files-sdk/fs";
import { SourceIdentity } from "migrate-sdk";
import { DocumentParsers, DocumentSource } from "migrate-sdk/sources/document";

export const CatalogDocument = Schema.Struct({
  products: Schema.Array(Schema.Struct({ id: Schema.String })),
});

const client = new FilesSDKClient({
  adapter: fs({ root: "./source-storage" }),
});
const filesLayer = Files.layer(client);

export const selection = {
  storageId: "catalog",
  prefix: "input/",
  pageSize: 20,
  filter: { id: "json-files@v1", test: (file) => file.key.endsWith(".json") },
} satisfies FilesSDKObjectDiscoveryOptions;

export const catalogSource = DocumentSource.make({
  fetcher: FilesSDKDocumentFetchers.texts({
    ...selection,
    filesLayer,
    maxBytes: 10 * 1024 * 1024,
  }),
  parser: DocumentParsers.each(DocumentParsers.json(CatalogDocument)),
  selector: { item: (document) => document.products },
  identity: {
    id: "catalog-product@v1",
    schema: SourceIdentity.key("id", Schema.String),
    key: ({ item }) => item.id,
  },
  lookup: { kind: "scan" },
  version: { kind: "content-hash" },
  batchSize: 100,
});

The Document Fetcher reads file text. DocumentParsers.each runs the JSON parser on each file. The selector picks products, and the Process Pipeline receives each product at sourceItem.item.item.

Product IDs must be unique across the selected files. The content hash detects changes to each product, so unchanged products can be skipped on a later run.

For one known file, use text and one parser:

const fetcher = FilesSDKDocumentFetchers.text({
  key: "catalog.json",
  filesLayer,
});
const parser = DocumentParsers.json(CatalogDocument);

The fetcher returns text; the parser handles JSON.

pageSize sets how many files to list at a time. maxBytes limits the combined file text downloaded for that page. Lower the page size if files are large. Pass the same selection to FilesSDKSource.objects(selection) when you want to migrate those JSON files as complete files instead of selecting products.

Copy a folder with the CLI

The runnable example copies four files, including nested paths and a binary file. From a repository checkout:

cd examples/files-sdk
pnpm install
pnpm run setup
pnpm migrate run folder-copy --plan
pnpm migrate run folder-copy --progress log
pnpm verify

Its folders have separate jobs:

.data/source/               Files to copy
.data/destination/copied/   Copied files, with the same relative paths
.data/state/                Saved migration progress

These are the files used by the CLI. folder-copy.ts defines the migration, storage.ts chooses the folders and Migration Store, and migrate.config.ts registers the definition.

folder-copy.ts
import { FilesSDKDestination } from "@migrate-sdk/files-sdk/destination";
import { FilesSDKSource } from "@migrate-sdk/files-sdk/source";
import { Effect, Schema } from "effect";
import { Files, make } from "files-sdk/effect";
import { MigrationDefinition, Tracking } from "migrate-sdk";
import { destinationClient, sourceClient, store } from "./storage.ts";

const source = FilesSDKSource.objects({
  storageId: "local-source",
  pageSize: 2,
  version: { kind: "sha256" },
}).provide(Files.layer(sourceClient));

const destination = FilesSDKDestination.make({
  storageId: "local-destination",
}).provide(Files.layer(destinationClient));

export const folderCopy = MigrationDefinition.make({
  id: "folder-copy",
  source,
  store,
  tracking: Tracking.record({
    id: "folder-copy@v1",
    schema: Schema.Struct({
      key: Schema.String,
      etag: Schema.NullOr(Schema.String),
    }),
  }),
  process: Effect.fn("folder-copy.process")(function* (sourceItem) {
    const copied = yield* destination.transferFrom({
      source: make(sourceClient),
      object: sourceItem.item,
      key: `copied/${sourceItem.item.key}`,
      write: { kind: "overwrite" },
      verify: "sha256",
    });
    yield* Tracking.setRecord({ key: copied.key, etag: copied.etag ?? null });
  }),
  rollback: Effect.fn("folder-copy.rollback")(function* (state) {
    for (const entry of state.journal?.process.entries ?? []) {
      if (destination.changes.uploaded.is(entry)) {
        const { value: change } =
          yield* destination.changes.uploaded.decode(entry);
        yield* destination.delete({
          key: change.key,
          // This example reserves copied/ exclusively for this migration.
          policy: { kind: "owned-prefix", prefix: "copied/" },
        });
      }
    }
  }),
});

FilesSDKSource.objects supplies each file's key and metadata at sourceItem.item. The Process Pipeline calls transferFrom to copy the file and checks that its SHA-256 hash matches. Each transfer finishes within one Process Pipeline execution. If it fails, a retry starts the transfer again from the beginning.

Rerun or roll back

Run the same command again:

pnpm migrate run folder-copy --progress log
pnpm migrate status folder-copy

The second run reports four unchanged files. The Migration Store keeps that progress in .data/state/ between CLI invocations.

After editing source files, rescan to copy the changed files and skip the rest:

pnpm migrate run folder-copy --rescan --progress log
pnpm verify

Use --update when you want to reprocess every migrated file, even if unchanged.

To delete the files copied by this migration:

pnpm migrate rollback folder-copy --plan
pnpm migrate rollback folder-copy --progress log

Reserve copied/ for this migration. Rollback uses the saved journal to delete files from that prefix. The example allows overwrites, and rollback cannot restore the previous contents of an overwritten file.

Keep one writer per namespace

Files SDK Migration Definition Locks are non-atomic. Two processes can both acquire a lock and overwrite saved state. Run one writer at a time for the folder-copy namespace, including commands that update cancellation state.

The required locking: "advisory" option acknowledges this limit, and the store logs a warning when initialized. Locks do not expire. Stop the original worker before unlocking an abandoned lock.

Use a SQL Migration Store when you need concurrent writers. The Source and Destination can stay the same: replace the store layer in storage.ts. For an existing Effect SQL layer named databaseLayer:

import { SqlMigrationStore } from "migrate-sdk/stores/sql";

export const store = SqlMigrationStore.layerFromClient(databaseLayer);

See Migration stores for SQL setup. The Files SDK package reference covers provider options, file metadata, and rollback with a saved ETag.

On this page