Documents

Import, export, create, update, and delete documents in a collection. Torque supports three ingestion methods with different throughput and use-case trade-offs.

Ingestion Methods

On a 309,236-document dataset with 67 fields, Torque's TCP binary ingest completes in 4.4 seconds (70K docs/s) — 9.5x faster than Typesense on the same hardware. TQBF file upload handles even larger datasets in a single HTTP POST with no per-document JSON parsing.

Import Documents (HTTP JSONL)

Import documents in JSONL format (one JSON object per line):

POST /collections/products/documents/import
Content-Type: text/plain
X-TYPESENSE-API-KEY: YOUR_API_KEY

{"id": "1", "title": "Running Shoes", "price": 89.99, "category": "Footwear"}
{"id": "2", "title": "Hiking Boots", "price": 149.99, "category": "Footwear"}
{"id": "3", "title": "Laptop Stand", "price": 49.99, "category": "Electronics"}

The response returns one JSON result per line, indicating success or failure for each document:

{"success": true}
{"success": true}
{"success": true}

Import supports three actions via the action query parameter:

POST /collections/products/documents/import?action=upsert

Import Documents (TCP Binary Protocol)

The TCP binary protocol streams documents over a persistent connection in a compact binary encoding, bypassing JSON parsing overhead. It runs on port 8109 by default and supports two ingestion modes:

Batch Mode

Stream a large volume of documents, then commit to trigger a full index rebuild. Best for initial loads and periodic bulk updates where all data arrives at once.

1. Client sends StartIngest (collection name, mode=batch)
2. Client sends one or more DocBatch messages (e.g., 5,000 docs per batch)
3. Server acknowledges each batch with a sequence number
4. Client sends Commit — server rebuilds the full index atomically

Realtime Mode

Instant upserts and deletes without a commit step. Documents are immediately searchable via an overlay that merges with the base index. Best for live sync from databases.

1. Client sends StartIngest (collection name, mode=realtime)
2. Client sends Upsert messages — each document is searchable immediately
3. Periodic compaction consolidates the overlay into the base index

See TCP Protocol for the full wire format specification. Client libraries are available for .NET, Python, Go, Java, and Node.js.

Import Binary File (TQBF)

The TQBF (Torque Binary File) format provides the fastest bulk import throughput. Documents are pre-encoded in the binary wire format with ZLib compression, eliminating JSON parsing and compression overhead at import time.

File Format (TQBF v2)

A TQBF v2 file has a 24-byte header followed by a compressed payload:

The uncompressed-size hint lets the server pre-size its decompression buffer exactly, avoiding doubling-realloc fragmentation on multi-GB imports. All client SDKs and torque-admin produce v2 files.

Upload via HTTP

POST /collections/products/documents/import-binary
Content-Type: application/octet-stream
X-TYPESENSE-API-KEY: YOUR_API_KEY

[binary TQBF file contents]

# Upload with curl
curl -X POST -H "X-TYPESENSE-API-KEY: KEY" \
  "http://localhost:8108/collections/products/documents/import-binary" \
  --data-binary @data.tqbf

Create TQBF Files

TQBF files can be created using the SDK binary writers or torque-admin:

# Export existing collection to TQBF file
torque-admin --api-key KEY export-binary my_collection -o data.tqbf

# Import TQBF file into a collection
torque-admin --api-key KEY import-binary my_collection -f data.tqbf

Each SDK includes a binary writer class for creating TQBF files programmatically:

Export Documents

Export all documents in a collection as JSONL:

GET /collections/products/documents/export
X-TYPESENSE-API-KEY: YOUR_API_KEY

The response streams one JSON document per line.

Create a Document

POST /collections/products/documents
Content-Type: application/json
X-TYPESENSE-API-KEY: YOUR_API_KEY

{
  "id": "4",
  "title": "Wireless Mouse",
  "price": 29.99,
  "category": "Electronics"
}

Get a Document

GET /collections/products/documents/4
X-TYPESENSE-API-KEY: YOUR_API_KEY

Update a Document

Partially update a document by merging fields:

PATCH /collections/products/documents/4
Content-Type: application/json
X-TYPESENSE-API-KEY: YOUR_API_KEY

{"price": 24.99}

Delete a Document

DELETE /collections/products/documents/4
X-TYPESENSE-API-KEY: YOUR_API_KEY

Document IDs

Every document has a string id field that uniquely identifies it within the collection. If you don't provide an id during import, Torque auto-generates one.

Compaction

After many realtime upserts and deletes, the index may accumulate overlay data. Trigger compaction to consolidate:

POST /collections/products/documents/compact
X-TYPESENSE-API-KEY: YOUR_API_KEY

Compaction also runs automatically based on the --compaction-interval-secs and --compaction-memory-threshold-percent settings.

Suspend and Resume

Pause realtime ingestion for a collection (e.g., during maintenance):

# Suspend ingestion
POST /collections/products/suspend
X-TYPESENSE-API-KEY: YOUR_API_KEY

# Resume ingestion
POST /collections/products/resume
X-TYPESENSE-API-KEY: YOUR_API_KEY

torque-admin CLI

The torque-admin binary provides command-line data management. All commands require --api-key or the TORQUE_API_KEY environment variable.

# List collections
torque-admin --api-key KEY collections

# Export collection (schema + JSONL)
torque-admin --api-key KEY export my_collection

# Import collection via TCP binary (batch mode — full rebuild)
torque-admin --api-key KEY import my_collection --batch-size 5000

# Import collection via TCP binary (realtime mode — overlay upsert)
torque-admin --api-key KEY import my_collection --mode realtime

# Export to TQBF binary file (ZLib compressed)
torque-admin --api-key KEY export-binary my_collection -o data.tqbf

# Import from TQBF binary file
torque-admin --api-key KEY import-binary my_collection -f data.tqbf

# Delete documents by ID (realtime collections only)
torque-admin --api-key KEY delete my_collection --ids id1,id2,id3