Integration · Object storage
Read the newest object under a prefix on a schedule, map it against your template, and write the mapped result back as a timestamped file. SigV4 against S3 and every S3-compatible store — R2, MinIO, Spaces, B2, Scaleway.
How it works
A bucket is where most regulated data already lands: an export job writes a CSV every night and something has to pick it up. The S3 connector lists your prefix, takes the newest object modified since the last successful run, parses it by its own key extension, and hands the grid to the cascade. Point the same connector the other way and the mapped rows are serialized into one file and put back under a timestamped key. One credential, encrypted once, used for both directions.
config.prefix, normalized into the same StoredObject[] shape all three providers return.modified is newer than the last successful sync survive; the newest of those is chosen. Nothing new is not_modified, not a failure.What you get
How · SigV4 is computed in lib/cloudAuth.ts; config.endpoint only swaps the host. Provider errors come back verbatim — an AccessDenied is the actionable thing — but never the signature.
How · The watermark is connector.last_synced_at, overridable per call with ?since=. A failed run does NOT bump it, so the next tick retries the same window rather than skipping it.
How · The default key is export-{date}-{time}.csv under your prefix; destination.format picks csv, json, xlsx or another emit format. batches is 1 — the object lands or it does not.
How · normalizeSecretField() folds every provider spelling — private_key, token, secret_access_key, account_key, service_account_json — into one auth_value field, which is KEK-envelope-encrypted before the row is written. A GET masks it to a 4-character hint. If encryption fails the field is dropped rather than stored in plaintext.
Configuration
Anything you send under a provider’s own spelling for the secret is normalized into auth_value and envelope-encrypted before the row is written — one encrypted field, one place to get right.
| Key | Required | What it is |
|---|---|---|
| bucket | Required | Bucket name. Its presence is what selects the credentialled path over the legacy presigned-URL form. |
| access_key_id | Required | Access key id. Validated at save time. |
| auth_valuesecret | Required | The secret access key. secret_access_key is accepted as an alias and folded into this field before storage. |
| region | Optional | Signing region. Defaults to us-east-1. |
| endpoint | Optional | Custom host for an S3-compatible store — R2, MinIO, Spaces, B2, Scaleway. Omit for AWS. |
| session_tokensecret | Optional | STS session token, for temporary credentials. |
| prefix | Optional | Key prefix that scopes both the read and the write, e.g. incoming/. |
| format | Optional | Override the parser instead of inferring it from the key. |
| url | Optional | Legacy alternative: a presigned https URL. Readable, but a presigned URL cannot be written to — a write returns config_invalid. |
In code
Save the bucket once from the dashboard, then pull it from anywhere with a bearer key. The credential stays in the connector record; the request carries only its id.
/v1/connectorsSave the connector. The secret is encrypted before it reaches Postgres.session/v1/connectors/{id}/testMake a real call and report what was actually proven.session/v1/connectors/{id}/syncPull now. Accepts ?since=<iso> to override the watermark.bearer/v1/gatewayAny input in, this destination populated, a delivery report out.bearer/v1/connectors/{id}/rotate-secretReplace the credential in place; the old one becomes unrecoverable.sessionlayer_distribution is the cost story: a recurring export settles in the deterministic layers, so the metered layer is never called at all.PUT with.{
"kind": "s3",
"name": "Nightly lab export",
"template_id": "lab_results",
"config": {
"bucket": "acme-clinical-exports",
"prefix": "labs/incoming/",
"region": "eu-central-1",
"access_key_id": "AKIA…",
"secret_access_key": "…"
}
}curl -X POST https://api.adaptivmapr.com/v1/connectors/con_9f2c…/sync \
-H "Authorization: Bearer $MAPR_API_KEY"{
"ok": true,
"status": "parsed",
"upload_id": "upl_7d13…",
"row_count": 1840,
"accepted": 1839,
"format": "csv",
"incremental": "incremental",
"layer_distribution": { "statistics": 9, "heuristic": 21, "fuzzy": 4, "semantic": 0, "llm": 0 }
}Limits & failure modes
| Code | When | What to do |
|---|---|---|
400 ssrf_blocked | The configured host resolves to a private, link-local or loopback address. | Every outbound request is DNS-resolved and checked before it is made, on the scheduled path and the on-demand path alike. The reason is returned with the code. |
400 config_invalid | A credential or identifier the provider needs is missing or malformed. | Refused at save time rather than at 3am during a scheduled run. The message names the exact key. |
413 fetch_too_large | The chosen object is larger than 25 MiB. | The parser’s own ceiling — an object is an upload by another route and the same memory limits apply. |
400 config_invalid (write) | A destination write is attempted against a presigned-URL connector. | The message says it plainly: a presigned URL can be read but not written to. Store real credentials to write. |
502 fetch_failed | The provider refused the list or the get. | The provider’s own message is surfaced verbatim; the watermark is not advanced, so the next tick retries the same window. |
400 parse_failed | The object is not readable as the format its key claims. | Set config.format to override extension sniffing, or fix the producer. |
Real. The listing is filtered to objects modified after the watermark and the newest survivor is taken, so a bucket that gains one file a night is read once a night. Nothing new returns status: "not_modified" with row_count: 0 — a success, not an error, and the watermark still advances only on success.
X-PHI and X-Region to phi-cloud so a regulated run lands on an in-region, BAA-eligible model, it costs +20% on the whole charge, and it is locked until the workspace accepts the BAA in Settings → Security & Data. An explicit PHI ask without an acceptance is 403 agreement_required, never a silent downgrade. A standard run keeps the workspace’s region pin — the region decides where compute may run, and the sandbox refuses a region-less run.What it costs
Moving bytes is not a line item. A sync that pulls a file and a destination write that lands the rows are both part of one map, and the map is what the wallet sees. There is no free tier and no subscription — top up from $10, a balance shared across the phi-cloud suite.
| Charge | Rate | Notes |
|---|---|---|
| Every map | $0.001 | A flat per-map fee — a few tokens — charged even when the run was fully deterministic or hit the layout cache and used no AI at all. |
| AI, only when it ran | at cost × 2 | Layer-5 cleanup, any-to-any convert and structural reshape bill the phi-cloud tokens actually consumed. Bring your own model key and it is × 0.5. |
| PHI / enterprise routing | +20% | Multiplies the whole charge, flat fee included — and only when the run genuinely got that routing. Locked until the workspace accepts the BAA in-app. |
Questions
Verified against lib/objectStore.ts · lib/connectorSyncRunner.ts · lib/destinations.ts · app/api/v1/connectors/route.ts
Amazon S3 is a trademark of Amazon.com, Inc. Cloudflare R2, MinIO, DigitalOcean Spaces, Backblaze B2 and Scaleway are trademarks of their respective owners. Named here to describe interoperability only — no affiliation, endorsement or partnership is claimed.
Ready when you are
Start with a $10 prepaid wallet. Every map draws a few tokens; in schema-only mode only headers and a few sample rows ever leave you.