Files SDK
Read records from JSON files, copy a folder with the CLI, and save progress using Files SDK.
Files SDK can read your source files, write destination files, and save migration progress. Each role can use its own storage client.
Choose the Source by what you want to migrate:
- Records inside files: use
DocumentSourcewith a Files SDK Document Fetcher. - Whole files: use
FilesSDKSource.objects, with one Source Item per file.
Install
Requires Node.js 22.19.0+, Effect 4, and files-sdk 3.1.1+:
pnpm add migrate-sdk @migrate-sdk/files-sdk files-sdk effect @effect/platform-nodeThe local filesystem adapter needs no cloud credentials or extra provider packages.
Read records from JSON files
Suppose source-storage/input/ contains JSON files like this:
{ "products": [{ "id": "p1" }, { "id": "p2" }] }This Source reads those files and emits each product as a Source Item:
import {
FilesSDKDocumentFetchers,
type FilesSDKObjectDiscoveryOptions,
} from "@migrate-sdk/files-sdk/source";
import { Schema } from "effect";
import { Files as FilesSDKClient } from "files-sdk";
import { Files } from "files-sdk/effect";
import { fs } from "files-sdk/fs";
import { SourceIdentity } from "migrate-sdk";
import { DocumentParsers, DocumentSource } from "migrate-sdk/sources/document";
export const CatalogDocument = Schema.Struct({
products: Schema.Array(Schema.Struct({ id: Schema.String })),
});
const client = new FilesSDKClient({
adapter: fs({ root: "./source-storage" }),
});
const filesLayer = Files.layer(client);
export const selection = {
storageId: "catalog",
prefix: "input/",
pageSize: 20,
filter: { id: "json-files@v1", test: (file) => file.key.endsWith(".json") },
} satisfies FilesSDKObjectDiscoveryOptions;
export const catalogSource = DocumentSource.make({
fetcher: FilesSDKDocumentFetchers.texts({
...selection,
filesLayer,
maxBytes: 10 * 1024 * 1024,
}),
parser: DocumentParsers.each(DocumentParsers.json(CatalogDocument)),
selector: { item: (document) => document.products },
identity: {
id: "catalog-product@v1",
schema: SourceIdentity.key("id", Schema.String),
key: ({ item }) => item.id,
},
lookup: { kind: "scan" },
version: { kind: "content-hash" },
batchSize: 100,
});
The Document Fetcher reads file text. DocumentParsers.each runs the JSON parser
on each file. The selector picks products, and the Process Pipeline receives
each product at sourceItem.item.item.
Product IDs must be unique across the selected files. The content hash detects changes to each product, so unchanged products can be skipped on a later run.
For one known file, use text and one parser:
const fetcher = FilesSDKDocumentFetchers.text({
key: "catalog.json",
filesLayer,
});
const parser = DocumentParsers.json(CatalogDocument);The fetcher returns text; the parser handles JSON.
pageSize sets how many files to list at a time. maxBytes limits the combined
file text downloaded for that page. Lower the page size if files are large.
Pass the same selection to FilesSDKSource.objects(selection) when you want
to migrate those JSON files as complete files instead of selecting products.
Copy a folder with the CLI
The runnable example copies four files, including nested paths and a binary file. From a repository checkout:
cd examples/files-sdk
pnpm install
pnpm run setup
pnpm migrate run folder-copy --plan
pnpm migrate run folder-copy --progress log
pnpm verifyIts folders have separate jobs:
.data/source/ Files to copy
.data/destination/copied/ Copied files, with the same relative paths
.data/state/ Saved migration progressThese are the files used by the CLI. folder-copy.ts defines the migration,
storage.ts chooses the folders and Migration Store, and migrate.config.ts
registers the definition.
import { FilesSDKDestination } from "@migrate-sdk/files-sdk/destination";
import { FilesSDKSource } from "@migrate-sdk/files-sdk/source";
import { Effect, Schema } from "effect";
import { Files, make } from "files-sdk/effect";
import { MigrationDefinition, Tracking } from "migrate-sdk";
import { destinationClient, sourceClient, store } from "./storage.ts";
const source = FilesSDKSource.objects({
storageId: "local-source",
pageSize: 2,
version: { kind: "sha256" },
}).provide(Files.layer(sourceClient));
const destination = FilesSDKDestination.make({
storageId: "local-destination",
}).provide(Files.layer(destinationClient));
export const folderCopy = MigrationDefinition.make({
id: "folder-copy",
source,
store,
tracking: Tracking.record({
id: "folder-copy@v1",
schema: Schema.Struct({
key: Schema.String,
etag: Schema.NullOr(Schema.String),
}),
}),
process: Effect.fn("folder-copy.process")(function* (sourceItem) {
const copied = yield* destination.transferFrom({
source: make(sourceClient),
object: sourceItem.item,
key: `copied/${sourceItem.item.key}`,
write: { kind: "overwrite" },
verify: "sha256",
});
yield* Tracking.setRecord({ key: copied.key, etag: copied.etag ?? null });
}),
rollback: Effect.fn("folder-copy.rollback")(function* (state) {
for (const entry of state.journal?.process.entries ?? []) {
if (destination.changes.uploaded.is(entry)) {
const { value: change } =
yield* destination.changes.uploaded.decode(entry);
yield* destination.delete({
key: change.key,
// This example reserves copied/ exclusively for this migration.
policy: { kind: "owned-prefix", prefix: "copied/" },
});
}
}
}),
});
FilesSDKSource.objects supplies each file's key and metadata at
sourceItem.item. The Process Pipeline calls transferFrom to copy the file
and checks that its SHA-256 hash matches. Each transfer finishes within one
Process Pipeline execution. If it fails, a retry starts the transfer again from
the beginning.
Rerun or roll back
Run the same command again:
pnpm migrate run folder-copy --progress log
pnpm migrate status folder-copyThe second run reports four unchanged files. The Migration Store keeps that
progress in .data/state/ between CLI invocations.
After editing source files, rescan to copy the changed files and skip the rest:
pnpm migrate run folder-copy --rescan --progress log
pnpm verifyUse --update when you want to reprocess every migrated file, even if unchanged.
To delete the files copied by this migration:
pnpm migrate rollback folder-copy --plan
pnpm migrate rollback folder-copy --progress logReserve copied/ for this migration. Rollback uses the saved journal to delete
files from that prefix. The example allows overwrites, and rollback cannot
restore the previous contents of an overwritten file.
Keep one writer per namespace
Files SDK Migration Definition Locks are non-atomic. Two processes can both
acquire a lock and overwrite saved state. Run one writer at a time for the
folder-copy namespace, including commands that update cancellation state.
The required locking: "advisory" option acknowledges this limit, and the store
logs a warning when initialized. Locks do not expire. Stop the original worker
before unlocking an abandoned lock.
Use a SQL Migration Store when you need concurrent writers. The Source and
Destination can stay the same: replace the store layer in storage.ts. For an
existing Effect SQL layer named databaseLayer:
import { SqlMigrationStore } from "migrate-sdk/stores/sql";
export const store = SqlMigrationStore.layerFromClient(databaseLayer);See Migration stores for SQL setup. The Files SDK package reference covers provider options, file metadata, and rollback with a saved ETag.