update documents
This commit is contained in:
+138
@@ -0,0 +1,138 @@
|
||||
# Collab — Yjs Collaboration Server
|
||||
|
||||
Node.js (CommonJS) real-time collaboration server that maintains Yjs documents in memory, persists updates to Redis, and notifies downstream consumers of dirty documents via Redis Pub/Sub or RabbitMQ.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
npm install
|
||||
npm start
|
||||
```
|
||||
|
||||
Server listens on `PORT` (default `1234`).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
server.js
|
||||
├── src/http/server.js — plain Node HTTP server (no Express)
|
||||
│ └── src/http/router.js — routes: /health, /livez, /readyz, /api/active-rooms, /internal/yjs/*
|
||||
├── src/ws-gateway.js — WebSocket upgrade handler, extracts room name from URL path
|
||||
├── src/ws-utils.js — Yjs sync/awareness protocol, WSSharedDoc, connection lifecycle
|
||||
├── src/doc-loader.js — loads doc: Redis list → gRPC snapshot → gRPC content (cascade)
|
||||
├── src/persistence.js — binds Y.Doc 'update' event → Redis list append + dirty tracking
|
||||
├── src/redis/ — Redis client, room tracking, document buffer lists
|
||||
├── src/grpc/ — gRPC client to backend (DocumentService, SystemService)
|
||||
├── src/mq.js — dirty-doc publisher (Redis Pub/Sub and/or RabbitMQ)
|
||||
├── src/config.js — all config from process.env (no .env loader)
|
||||
├── src/key.js — Redis key patterns
|
||||
├── src/bootstrap.js — listen + init Redis/gRPC
|
||||
└── src/lifecycle.js — graceful shutdown (SIGTERM/SIGINT)
|
||||
```
|
||||
|
||||
## Connection Flow
|
||||
|
||||
1. Client connects via WebSocket to `/{roomName}` (e.g. `/doc-{docId}`)
|
||||
2. `ws-gateway.js` extracts room name from URL path
|
||||
3. `ws-utils.js` loads or creates `WSSharedDoc` for the room
|
||||
4. `doc-loader.js` attempts to load existing state:
|
||||
- Redis list buffers → gRPC `GetDocumentYjsSnapshot` → gRPC `GetDocumentContent` → empty doc
|
||||
5. `persistence.js` binds to Y.Doc `update` event
|
||||
6. Updates are appended to Redis list and tracked for dirty-doc notifications
|
||||
|
||||
## Persistence Model
|
||||
|
||||
- Each document update is appended to Redis list: `collab:yjs:doc:updates:{docId}`
|
||||
- After `DOC_UPDATE_COMPACT_THRESHOLD` (default 1000) updates, list is compacted into single state update
|
||||
- Document is marked dirty in Redis set: `collab:yjs:dirty:doc`
|
||||
- After `DIRTY_DOC_NOTIFY_THRESHOLD` (default 200) updates, dirty-doc event is published via MQ
|
||||
|
||||
## Dirty-Document Notification
|
||||
|
||||
Dirty-doc events notify downstream consumers (snapshot-worker) that a document has been modified.
|
||||
|
||||
MQ backend controlled by `DIRTY_DOC_MQ` environment variable:
|
||||
- `redis` — Redis Pub/Sub (default, no consumer groups)
|
||||
- `rabbitmq` — RabbitMQ with consumer groups
|
||||
- `both` — publish to both
|
||||
|
||||
Payload format:
|
||||
```json
|
||||
{
|
||||
"doc_id": "document-id",
|
||||
"ts": 1234567890
|
||||
}
|
||||
```
|
||||
|
||||
Topic: `DIRTY_DOC_TOPIC` (default `collab.dirty_docs`)
|
||||
|
||||
## HTTP Routes
|
||||
|
||||
Health probes:
|
||||
- `GET /health` — overall health status
|
||||
- `GET /livez` — liveness probe
|
||||
- `GET /readyz` — readiness probe (fails during drain)
|
||||
|
||||
API:
|
||||
- `GET /api/active-rooms` — list active WebSocket rooms
|
||||
|
||||
Internal (used by snapshot-worker):
|
||||
- `GET /internal/yjs/doc-snapshot/{docId}?type={type}` — get document snapshot
|
||||
- `GET /internal/yjs/doc/{docId}` — get document as binary Yjs update
|
||||
|
||||
## WebSocket Protocol
|
||||
|
||||
Room name is extracted from WebSocket URL path. For `/doc-{docId}`, the doc ID is `{docId}`.
|
||||
|
||||
Protocol messages:
|
||||
- Sync messages (Yjs sync protocol)
|
||||
- Awareness messages (cursor positions, user presence)
|
||||
|
||||
When last connection to a room closes, the document is removed from memory.
|
||||
|
||||
## Proto Codegen
|
||||
|
||||
`src/gen/` contains gRPC stubs generated from parent repo's `proto/yoresee_doc/v1/yoresee_doc.proto`.
|
||||
|
||||
This directory is **gitignored**. In production, `Dockerfile.prod` generates stubs during build. For development, run from parent repo:
|
||||
|
||||
```bash
|
||||
bash deploy/script/gen_proto.sh
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `PORT` | `1234` | HTTP listen port |
|
||||
| `REDIS_HOST` | `redis` | Redis hostname |
|
||||
| `REDIS_PORT` | `6379` | Redis port |
|
||||
| `REDIS_PASSWORD` | `''` | Redis password |
|
||||
| `REDIS_DB` | `0` | Redis DB index |
|
||||
| `BACKEND_ADDR` | `backend:9090` | Backend gRPC address |
|
||||
| `DIRTY_DOC_MQ` | `redis` | MQ type: `redis`, `rabbitmq`, or `both` |
|
||||
| `DIRTY_DOC_TOPIC` | `collab.dirty_docs` | Dirty-doc pub/sub topic |
|
||||
| `RABBITMQ_URL` | `''` | RabbitMQ connection URL (empty = disabled) |
|
||||
| `INTERNAL_RPC_KEY` | `''` | gRPC metadata auth key (`x-internal-key`) |
|
||||
| `DIRTY_DOC_NOTIFY_THRESHOLD` | `200` | Updates before publishing dirty-doc event |
|
||||
| `DOC_UPDATES_TTL` | `259200` | TTL for doc update counters (seconds) |
|
||||
| `DOC_UPDATE_COMPACT_THRESHOLD` | `1000` | Updates before compacting Redis list |
|
||||
| `GC` | `true` | Yjs garbage collection (`false`/`0` to disable) |
|
||||
| `BACKEND_SNAPSHOT_TIMEOUT_MS` | `3000` | Timeout for backend snapshot gRPC call |
|
||||
|
||||
## Redis Keys
|
||||
|
||||
Defined in `src/key.js`:
|
||||
- `collab:yjs:doc:updates:{docId}` — document update list
|
||||
- `collab:room:doc-{docId}` — room metadata
|
||||
- `collab:yjs:dirty:doc` — dirty document set
|
||||
|
||||
## Graceful Shutdown
|
||||
|
||||
On `SIGTERM` or `SIGINT`:
|
||||
1. Server stops accepting new connections
|
||||
2. Readiness probe returns `not_ready`
|
||||
3. Existing WebSocket connections are closed
|
||||
4. MQ connections are closed
|
||||
5. Redis connections are closed
|
||||
6. Process exits (10s timeout, then force exit)
|
||||
Reference in New Issue
Block a user