collab/docs/README.md

Collab — Yjs Collaboration Server

Node.js (CommonJS) real-time collaboration server that maintains Yjs documents in memory, persists updates to Redis, and notifies downstream consumers of dirty documents via Redis Pub/Sub or RabbitMQ.

Quick Start

npm install
npm start

Server listens on PORT (default 1234).

Architecture

server.js
├── src/http/server.js      — plain Node HTTP server (no Express)
│   └── src/http/router.js  — routes: /health, /livez, /readyz, /api/active-rooms, /internal/yjs/*
├── src/ws-gateway.js       — WebSocket upgrade handler, extracts room name from URL path
├── src/ws-utils.js         — Yjs sync/awareness protocol, WSSharedDoc, connection lifecycle
├── src/doc-loader.js       — loads doc: Redis list → gRPC snapshot → gRPC content (cascade)
├── src/persistence.js      — binds Y.Doc 'update' event → Redis list append + dirty tracking
├── src/redis/              — Redis client, room tracking, document buffer lists
├── src/grpc/               — gRPC client to backend (DocumentService, SystemService)
├── src/mq.js               — dirty-doc publisher (Redis Pub/Sub and/or RabbitMQ)
├── src/config.js            — all config from process.env (no .env loader)
├── src/key.js               — Redis key patterns
├── src/bootstrap.js         — listen + init Redis/gRPC
└── src/lifecycle.js         — graceful shutdown (SIGTERM/SIGINT)

Connection Flow

1. Client connects via WebSocket to /{roomName} (e.g. /doc-{docId})
2. ws-gateway.js extracts room name from URL path
3. ws-utils.js loads or creates WSSharedDoc for the room
4. doc-loader.js attempts to load existing state:
   • Redis list buffers → gRPC GetDocumentYjsSnapshot → gRPC GetDocumentContent → empty doc
5. persistence.js binds to Y.Doc update event
6. Updates are appended to Redis list and tracked for dirty-doc notifications

Persistence Model

• Each document update is appended to Redis list: collab:yjs:doc:updates:{docId}
• After DOC_UPDATE_COMPACT_THRESHOLD (default 1000) updates, list is compacted into single state update
• Document is marked dirty in Redis set: collab:yjs:dirty:doc
• After DIRTY_DOC_NOTIFY_THRESHOLD (default 200) updates, dirty-doc event is published via MQ

Dirty-Document Notification

Dirty-doc events notify downstream consumers (snapshot-worker) that a document has been modified.

MQ backend controlled by DIRTY_DOC_MQ environment variable:

• redis — Redis Pub/Sub (default, no consumer groups)
• rabbitmq — RabbitMQ with consumer groups
• both — publish to both

Payload format:

{
  "doc_id": "document-id",
  "ts": 1234567890
}

Topic: DIRTY_DOC_TOPIC (default collab.dirty_docs)

HTTP Routes

Health probes:

• GET /health — overall health status
• GET /livez — liveness probe
• GET /readyz — readiness probe (fails during drain)

API:

• GET /api/active-rooms — list active WebSocket rooms

Internal (used by snapshot-worker):

• GET /internal/yjs/doc-snapshot/{docId}?type={type} — get document snapshot
• GET /internal/yjs/doc/{docId} — get document as binary Yjs update

WebSocket Protocol

Room name is extracted from WebSocket URL path. For /doc-{docId}, the doc ID is {docId}.

Protocol messages:

• Sync messages (Yjs sync protocol)
• Awareness messages (cursor positions, user presence)

When last connection to a room closes, the document is removed from memory.

Proto Codegen

src/gen/ contains gRPC stubs generated from parent repo's proto/yoresee_doc/v1/yoresee_doc.proto.

This directory is gitignored. In production, Dockerfile.prod generates stubs during build. For development, run from parent repo:

bash deploy/script/gen_proto.sh

Environment Variables

Variable	Default	Description
PORT	1234	HTTP listen port
REDIS_HOST	redis	Redis hostname
REDIS_PORT	6379	Redis port
REDIS_PASSWORD	''	Redis password
REDIS_DB	0	Redis DB index
BACKEND_ADDR	backend:9090	Backend gRPC address
DIRTY_DOC_MQ	redis	MQ type: redis, rabbitmq, or both
DIRTY_DOC_TOPIC	collab.dirty_docs	Dirty-doc pub/sub topic
RABBITMQ_URL	''	RabbitMQ connection URL (empty = disabled)
INTERNAL_RPC_KEY	''	gRPC metadata auth key (x-internal-key)
DIRTY_DOC_NOTIFY_THRESHOLD	200	Updates before publishing dirty-doc event
DOC_UPDATES_TTL	259200	TTL for doc update counters (seconds)
DOC_UPDATE_COMPACT_THRESHOLD	1000	Updates before compacting Redis list
GC	true	Yjs garbage collection (false/0 to disable)
BACKEND_SNAPSHOT_TIMEOUT_MS	3000	Timeout for backend snapshot gRPC call

Redis Keys

Defined in src/key.js:

• collab:yjs:doc:updates:{docId} — document update list
• collab:room:doc-{docId} — room metadata
• collab:yjs:dirty:doc — dirty document set

Graceful Shutdown

On SIGTERM or SIGINT:

1. Server stops accepting new connections
2. Readiness probe returns not_ready
3. Existing WebSocket connections are closed
4. MQ connections are closed
5. Redis connections are closed
6. Process exits (10s timeout, then force exit)