Integration · Warehouse

Streaming inserts, with the rejects counted.

Mapped rows land in a BigQuery table via tabledata.insertAll — in batches of 500, with per-row rejections reported individually instead of hidden behind a 200.

Get an API key Read the docs →

POST /v1/gatewaykind: bigquery

How it works

What actually happens on a sync run

BigQuery’s streaming insert has a trap in it: tabledata.insertAll answers HTTP 200 even when it rejected rows, listing them under insertErrors. A caller that checks only the status code reports a successful import that silently dropped records — for a regulated-data product, the worst possible failure. So written_rows counts only rows NOT named in that array, and the rejections come back individually with the reason BigQuery gave.

Step 1
Mint a scoped token
The service-account key is exchanged for a BigQuery-scoped access token per call.
Step 2
Read the table schema
The destination’s own columns are read so the mapping targets what actually exists.
Step 3
Stream in batches
500 rows per insertAll request, each batch independent.
Step 4
Count what landed
Rows named in insertErrors are subtracted and returned as rejected_rows with their reasons.

What you get

Built for files that keep arriving

The 200 trap

A rejected row is never counted as written

insertAll returns 200 with a per-row error array. AdaptivMapr subtracts those rows from the written count and returns each rejection with its reason.

How · rejected_rows: [{ index, reason }] alongside written_rows. A partial write reads as a partial write, which is the only honest way to report it.

Schema

The dataset already knows its columns

The table’s schema is readable, so a gateway call maps onto the real columns instead of a restated copy that drifts.

How · GET /v1/connectors/{id}/schema returns the column list and a ready-to-use template. Metadata only — it never selects row data.

Auth

One service account, scoped to one dataset

Grant the account write on the dataset it needs. AdaptivMapr exchanges the key for a scoped token per call and stores nothing else.

How · The key file is validated by parseGcpCredentials() at save time, envelope-encrypted at rest, and masked on every read.

Secrets

The credential never travels in a request body

You reference a connector by id. Keys, tokens and service-account JSON are encrypted at rest and read only by the code that makes the call.

How · normalizeSecretField() folds every provider spelling — private_key, token, secret_access_key, account_key, service_account_json — into one auth_value field, which is KEK-envelope-encrypted before the row is written. A GET masks it to a 4-character hint. If encryption fails the field is dropped rather than stored in plaintext.

Configuration

The connector record, field by field

Dataset and table are validated as bare identifiers at save time, so a name that would have to be rewritten before it could be used is refused rather than quietly reshaped.

Key	Required	What it is
service_account_jsonsecret	Required	The service-account key file. Must parse and carry `client_email` and `private_key`. Stored as `auth_value`.
dataset	Required	Dataset id. Must be a bare identifier.
table	Required	Table id. A bare identifier, optionally dotted. Overridable per call with `destination.table`.
project_id	Optional	Project id. Falls back to the `project_id` inside the key file.

In code

Any input, landed in the warehouse.

Hand the gateway a file, a query or a URL and a BigQuery destination. The columns come from the table, the validators run per row, and the report says exactly what landed.

POST/v1/connectorsSave the connector. The secret is encrypted before it reaches Postgres.session
GET/v1/connectors/{id}/schemaRead the target’s own columns — metadata only, never row data.bearer
POST/v1/gatewayAny input in, this destination populated, a delivery report out.bearer
POST/v1/connectors/{id}/rotate-secretReplace the credential in place; the old one becomes unrecoverable.session

written_rows excludes anything BigQuery named in insertErrors. A run that reports 4 998 wrote 4 998.
Rows are streamed in batches of 500 so one bad batch does not take the whole delivery with it.
dry_run validates and reports the batch count without sending.

POST /v1/connectors

{
  "kind": "bigquery",
  "name": "Analytics — orders",
  "config": {
    "project_id": "acme-analytics",
    "dataset": "ops",
    "table": "orders",
    "service_account_json": "{\"client_email\":\"mapr@acme-analytics.iam.gserviceaccount.com\",\"private_key\":\"-----BEGIN PRIVATE KEY-----\\n…\"}"
  }
}

curl

curl https://api.adaptivmapr.com/v1/gateway \
  -H "Authorization: Bearer $MAPR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input": { "url": "https://acme.example.com/exports/orders.csv" },
    "destination": { "connector_id": "con_bb14…" }
  }'

response

{
  "schema_id": "orders_v1",
  "source": "destination",
  "row_count": 5000,
  "destination": {
    "connector_id": "con_bb14…",
    "kind": "bigquery",
    "table": "orders",
    "schema_source": "destination",
    "protocol": "bigquery",
    "written_rows": 4998,
    "batches": 10,
    "failed_batches": [],
    "rejected_rows": [
      { "index": 1204, "reason": "no such field: legacy_sku" },
      { "index": 3781, "reason": "invalid value for TIMESTAMP" }
    ]
  }
}

→ 4 998 of 5 000 rows written · 2 rejected by BigQuery and named · 0 batches failed

Limits & failure modes

What it refuses, and what it tells you

Code	When	What to do
`400 config_invalid`	The key file does not parse, or dataset/table are not bare identifiers.	Checked at save time so an unusable connector is never created.
`401 auth_failed`	The token exchange was refused.	Usually a revoked key or a missing role on the dataset. Google’s message is passed through.
`200 with rejected_rows`	BigQuery accepted the request but rejected individual rows.	The rejects are listed with reasons and subtracted from `written_rows` — never reported as written.
`422 schema_destination_mismatch`	A supplied schema names a column the table does not have.	Caught in pre-flight, before anything is streamed.

Incremental sync

Not applicable: BigQuery is a destination here, not a source. Repeat safety comes from what you send, not from a watermark — the write is a streaming insert, so de-duplication belongs in the query layer or in the input you feed the gateway.

PHI & residency

A warehouse delivery runs under the workspace’s routing policy, not around it. PHI routing is a separate axis from the data mode: it sends X-PHI and X-Region to phi-cloud so a regulated run lands on an in-region, BAA-eligible model, it costs +20% on the whole charge, and it is locked until the workspace accepts the BAA in Settings → Security & Data. An explicit PHI ask without an acceptance is 403 agreement_required, never a silent downgrade. A standard run keeps the workspace’s region pin — the region decides where compute may run, and the sandbox refuses a region-less run.

What it costs

Billed on the same prepaid wallet

Moving bytes is not a line item. A sync that pulls a file and a destination write that lands the rows are both part of one map, and the map is what the wallet sees. There is no free tier and no subscription — top up from $10, a balance shared across the phi-cloud suite.

Charge	Rate	Notes
Every map	$0.001	A flat per-map fee — a few tokens — charged even when the run was fully deterministic or hit the layout cache and used no AI at all.
AI, only when it ran	at cost × 2	Layer-5 cleanup, any-to-any convert and structural reshape bill the phi-cloud tokens actually consumed. Bring your own model key and it is × 0.5.
PHI / enterprise routing	+20%	Multiplies the whole charge, flat fee included — and only when the run genuinely got that routing. Locked until the workspace accepts the BAA in-app.

Questions

Before you wire it up

Because BigQuery does. A failed batch is a request that did not succeed; a rejected row is one that insertAll refused inside a request that returned HTTP 200. Reporting only the status code would let a run that dropped records read as a clean import, which for regulated data is the worst possible failure mode. written_rows counts only rows BigQuery did not name.

No. The table already declares its columns, so the gateway reads them and maps onto those. Supply schema only when you mean a deliberate subset; the request is still pre-flighted against the real table, and naming a column that does not exist fails 422 schema_destination_mismatch before anything is written.

Not as a scheduled connector today — the BigQuery integration is a write destination, and its schema is readable for mapping. To pull data out of a warehouse on a schedule, expose the query over an HTTPS endpoint and use the SQL-over-HTTP source, which supports a {{since}} watermark.

The map does — a flat $0.001 per map, plus AI tokens only if the metered layer actually ran, plus 20% if the run was PHI-routed. The warehouse write itself is not separately metered. There is no free tier; the wallet is prepaid from $10 and shared across the phi-cloud suite.

Snowflake →Databricks →Google Cloud Storage →

Verified against lib/googleProviders.ts · lib/destinations.ts · lib/introspect.ts · app/api/v1/connectors/route.ts

BigQuery is a trademark of Google LLC. Named here to describe interoperability only — no affiliation, endorsement or partnership is claimed.

Ready when you are

Point it at BigQuery. Get your schema back.

Start with a $10 prepaid wallet. Every map draws a few tokens; in schema-only mode only headers and a few sample rows ever leave you.

Get an API key All integrations →

$10 minimum to start · pay only for what you map · PHI under BAA coverage