update documents

2026-08-11 14:23:50 +08:00
parent afe7d4f9b2
commit 2ebf42bb9b
3 changed files with 348 additions and 0 deletions
@@ -0,0 +1,72 @@
+# AGENTS.md — collab
+
+Node.js (CommonJS) Yjs collaboration server. Maintains Yjs documents in memory, persists updates to Redis, and notifies downstream consumers of dirty documents via Redis Pub/Sub or RabbitMQ.
+
+## Commands
+
+```bash
+npm start          # node server.js — listens on PORT (default 1234)
+```
+
+No test, lint, typecheck, or build scripts exist. Verify by reading the code.
+
+## Proto Codegen
+
+`src/gen/` contains gRPC stubs generated from `proto/yoresee_doc/v1/yoresee_doc.proto` (parent repo). This directory is **gitignored**.
+
+- Dev: generated automatically by `deploy/script/gen_proto.sh` from the parent repo.
+- Prod: `Dockerfile.prod` generates stubs in a builder stage using `grpc-tools` and `protoc-gen-js`.
+
+The file `src/grpc/client.js` imports from `../gen/yoresee_doc/v1/yoresee_doc_grpc_pb` — code will fail at startup if stubs are missing.
+
+## Architecture
+
+Entry: `server.js` → wires HTTP server, WebSocket gateway, Redis, gRPC, and graceful shutdown.
+
+```
+server.js
+├── src/http/server.js      — plain Node HTTP server (no Express)
+│   └── src/http/router.js  — routes: /health, /livez, /readyz, /api/active-rooms, /internal/yjs/*
+├── src/ws-gateway.js       — WebSocket upgrade handler, extracts room name from URL path
+├── src/ws-utils.js         — Yjs sync/awareness protocol, WSSharedDoc, connection lifecycle
+├── src/doc-loader.js       — loads doc: Redis list → gRPC snapshot → gRPC content (cascade)
+├── src/persistence.js      — binds Y.Doc 'update' event → Redis list append + dirty tracking
+├── src/redis/              — Redis client, room tracking, document buffer lists
+├── src/grpc/               — gRPC client to backend (DocumentService, SystemService)
+├── src/mq.js               — dirty-doc publisher (Redis Pub/Sub and/or RabbitMQ)
+├── src/config.js            — all config from process.env (no .env loader)
+├── src/key.js               — Redis key patterns
+├── src/bootstrap.js         — listen + init Redis/gRPC
+└── src/lifecycle.js         — graceful shutdown (SIGTERM/SIGINT)
+```
+
+## Key Patterns
+
+- **Module style**: CommonJS (`require`/`module.exports`). No ESM.
+- **Config**: plain `process.env` reads in `src/config.js`. No dotenv — env vars must be set externally.
+- **Doc loading cascade**: Redis list buffers → gRPC `GetDocumentYjsSnapshot` → gRPC `GetDocumentContent` → empty doc.
+- **Persistence**: updates appended to Redis list (`collab:yjs:doc:updates:{docId}`). Compacted into single state update every `DOC_UPDATE_COMPACT_THRESHOLD` (default 1000) updates.
+- **Dirty-doc notification**: published after `DIRTY_DOC_NOTIFY_THRESHOLD` (default 200) updates per doc. MQ backend controlled by `DIRTY_DOC_MQ` env var (`redis` | `rabbitmq` | `both`).
+- **Room naming**: WebSocket path is the room name (e.g. `/doc-{docId}`). Room name with `doc-` prefix is stripped to get the doc ID.
+- **Redis keys**: defined in `src/key.js` — `collab:yjs:doc:updates:{docId}`, `collab:room:doc-{docId}`, `collab:yjs:dirty:doc`.
+- **Health probes**: `/health`, `/livez`, `/readyz` — readiness depends on Redis connection and drain state.
+
+## Environment Variables
+
+| Variable | Default | Description |
+|---|---|---|
+| `PORT` | `1234` | HTTP listen port |
+| `REDIS_HOST` | `redis` | Redis hostname |
+| `REDIS_PORT` | `6379` | Redis port |
+| `REDIS_PASSWORD` | `''` | Redis password |
+| `REDIS_DB` | `0` | Redis DB index |
+| `BACKEND_ADDR` | `backend:9090` | Backend gRPC address |
+| `DIRTY_DOC_MQ` | `redis` | MQ type: `redis`, `rabbitmq`, or `both` |
+| `DIRTY_DOC_TOPIC` | `collab.dirty_docs` | Dirty-doc pub/sub topic |
+| `RABBITMQ_URL` | `''` | RabbitMQ connection URL (empty = disabled) |
+| `INTERNAL_RPC_KEY` | `''` | gRPC metadata auth key (`x-internal-key`) |
+| `DIRTY_DOC_NOTIFY_THRESHOLD` | `200` | Updates before publishing dirty-doc event |
+| `DOC_UPDATES_TTL` | `259200` | TTL for doc update counters (seconds) |
+| `DOC_UPDATE_COMPACT_THRESHOLD` | `1000` | Updates before compacting Redis list |
+| `GC` | `true` | Yjs garbage collection (`false`/`0` to disable) |
+| `BACKEND_SNAPSHOT_TIMEOUT_MS` | `3000` | Timeout for backend snapshot gRPC call |
@@ -0,0 +1,138 @@
+# Collab — Yjs Collaboration Server
+
+Node.js (CommonJS) real-time collaboration server that maintains Yjs documents in memory, persists updates to Redis, and notifies downstream consumers of dirty documents via Redis Pub/Sub or RabbitMQ.
+
+## Quick Start
+
+```bash
+npm install
+npm start
+```
+
+Server listens on `PORT` (default `1234`).
+
+## Architecture
+
+```
+server.js
+├── src/http/server.js      — plain Node HTTP server (no Express)
+│   └── src/http/router.js  — routes: /health, /livez, /readyz, /api/active-rooms, /internal/yjs/*
+├── src/ws-gateway.js       — WebSocket upgrade handler, extracts room name from URL path
+├── src/ws-utils.js         — Yjs sync/awareness protocol, WSSharedDoc, connection lifecycle
+├── src/doc-loader.js       — loads doc: Redis list → gRPC snapshot → gRPC content (cascade)
+├── src/persistence.js      — binds Y.Doc 'update' event → Redis list append + dirty tracking
+├── src/redis/              — Redis client, room tracking, document buffer lists
+├── src/grpc/               — gRPC client to backend (DocumentService, SystemService)
+├── src/mq.js               — dirty-doc publisher (Redis Pub/Sub and/or RabbitMQ)
+├── src/config.js            — all config from process.env (no .env loader)
+├── src/key.js               — Redis key patterns
+├── src/bootstrap.js         — listen + init Redis/gRPC
+└── src/lifecycle.js         — graceful shutdown (SIGTERM/SIGINT)
+```
+
+## Connection Flow
+
+1. Client connects via WebSocket to `/{roomName}` (e.g. `/doc-{docId}`)
+2. `ws-gateway.js` extracts room name from URL path
+3. `ws-utils.js` loads or creates `WSSharedDoc` for the room
+4. `doc-loader.js` attempts to load existing state:
+   - Redis list buffers → gRPC `GetDocumentYjsSnapshot` → gRPC `GetDocumentContent` → empty doc
+5. `persistence.js` binds to Y.Doc `update` event
+6. Updates are appended to Redis list and tracked for dirty-doc notifications
+
+## Persistence Model
+
+- Each document update is appended to Redis list: `collab:yjs:doc:updates:{docId}`
+- After `DOC_UPDATE_COMPACT_THRESHOLD` (default 1000) updates, list is compacted into single state update
+- Document is marked dirty in Redis set: `collab:yjs:dirty:doc`
+- After `DIRTY_DOC_NOTIFY_THRESHOLD` (default 200) updates, dirty-doc event is published via MQ
+
+## Dirty-Document Notification
+
+Dirty-doc events notify downstream consumers (snapshot-worker) that a document has been modified.
+
+MQ backend controlled by `DIRTY_DOC_MQ` environment variable:
+- `redis` — Redis Pub/Sub (default, no consumer groups)
+- `rabbitmq` — RabbitMQ with consumer groups
+- `both` — publish to both
+
+Payload format:
+```json
+{
+  "doc_id": "document-id",
+  "ts": 1234567890
+}
+```
+
+Topic: `DIRTY_DOC_TOPIC` (default `collab.dirty_docs`)
+
+## HTTP Routes
+
+Health probes:
+- `GET /health` — overall health status
+- `GET /livez` — liveness probe
+- `GET /readyz` — readiness probe (fails during drain)
+
+API:
+- `GET /api/active-rooms` — list active WebSocket rooms
+
+Internal (used by snapshot-worker):
+- `GET /internal/yjs/doc-snapshot/{docId}?type={type}` — get document snapshot
+- `GET /internal/yjs/doc/{docId}` — get document as binary Yjs update
+
+## WebSocket Protocol
+
+Room name is extracted from WebSocket URL path. For `/doc-{docId}`, the doc ID is `{docId}`.
+
+Protocol messages:
+- Sync messages (Yjs sync protocol)
+- Awareness messages (cursor positions, user presence)
+
+When last connection to a room closes, the document is removed from memory.
+
+## Proto Codegen
+
+`src/gen/` contains gRPC stubs generated from parent repo's `proto/yoresee_doc/v1/yoresee_doc.proto`.
+
+This directory is **gitignored**. In production, `Dockerfile.prod` generates stubs during build. For development, run from parent repo:
+
+```bash
+bash deploy/script/gen_proto.sh
+```
+
+## Environment Variables
+
+| Variable | Default | Description |
+|---|---|---|
+| `PORT` | `1234` | HTTP listen port |
+| `REDIS_HOST` | `redis` | Redis hostname |
+| `REDIS_PORT` | `6379` | Redis port |
+| `REDIS_PASSWORD` | `''` | Redis password |
+| `REDIS_DB` | `0` | Redis DB index |
+| `BACKEND_ADDR` | `backend:9090` | Backend gRPC address |
+| `DIRTY_DOC_MQ` | `redis` | MQ type: `redis`, `rabbitmq`, or `both` |
+| `DIRTY_DOC_TOPIC` | `collab.dirty_docs` | Dirty-doc pub/sub topic |
+| `RABBITMQ_URL` | `''` | RabbitMQ connection URL (empty = disabled) |
+| `INTERNAL_RPC_KEY` | `''` | gRPC metadata auth key (`x-internal-key`) |
+| `DIRTY_DOC_NOTIFY_THRESHOLD` | `200` | Updates before publishing dirty-doc event |
+| `DOC_UPDATES_TTL` | `259200` | TTL for doc update counters (seconds) |
+| `DOC_UPDATE_COMPACT_THRESHOLD` | `1000` | Updates before compacting Redis list |
+| `GC` | `true` | Yjs garbage collection (`false`/`0` to disable) |
+| `BACKEND_SNAPSHOT_TIMEOUT_MS` | `3000` | Timeout for backend snapshot gRPC call |
+
+## Redis Keys
+
+Defined in `src/key.js`:
+- `collab:yjs:doc:updates:{docId}` — document update list
+- `collab:room:doc-{docId}` — room metadata
+- `collab:yjs:dirty:doc` — dirty document set
+
+## Graceful Shutdown
+
+On `SIGTERM` or `SIGINT`:
+1. Server stops accepting new connections
+2. Readiness probe returns `not_ready`
+3. Existing WebSocket connections are closed
+4. MQ connections are closed
+5. Redis connections are closed
+6. Process exits (10s timeout, then force exit)
@@ -0,0 +1,138 @@
+# Collab — Yjs 协作服务器
+
+Node.js (CommonJS) 实时协作服务器，在内存中维护 Yjs 文档，将更新持久化到 Redis，并通过 Redis Pub/Sub 或 RabbitMQ 通知下游消费者脏文档。
+
+## 快速启动
+
+```bash
+npm install
+npm start
+```
+
+服务器监听 `PORT`（默认 `1234`）。
+
+## 架构
+
+```
+server.js
+├── src/http/server.js      — 原生 Node HTTP 服务器（无 Express）
+│   └── src/http/router.js  — 路由：/health, /livez, /readyz, /api/active-rooms, /internal/yjs/*
+├── src/ws-gateway.js       — WebSocket 升级处理器，从 URL 路径提取房间名
+├── src/ws-utils.js         — Yjs 同步/感知协议，WSSharedDoc，连接生命周期
+├── src/doc-loader.js       — 加载文档：Redis 列表 → gRPC 快照 → gRPC 内容（级联）
+├── src/persistence.js      — 绑定 Y.Doc 'update' 事件 → Redis 列表追加 + 脏文档追踪
+├── src/redis/              — Redis 客户端，房间追踪，文档缓冲区列表
+├── src/grpc/               — gRPC 客户端连接后端（DocumentService, SystemService）
+├── src/mq.js               — 脏文档发布者（Redis Pub/Sub 和/或 RabbitMQ）
+├── src/config.js            — 所有配置来自 process.env（无 .env 加载器）
+├── src/key.js               — Redis 键模式
+├── src/bootstrap.js         — 监听 + 初始化 Redis/gRPC
+└── src/lifecycle.js         — 优雅停机（SIGTERM/SIGINT）
+```
+
+## 连接流程
+
+1. 客户端通过 WebSocket 连接到 `/{roomName}`（例如 `/doc-{docId}`）
+2. `ws-gateway.js` 从 URL 路径提取房间名
+3. `ws-utils.js` 为房间加载或创建 `WSSharedDoc`
+4. `doc-loader.js` 尝试加载已有状态：
+   - Redis 列表缓冲区 → gRPC `GetDocumentYjsSnapshot` → gRPC `GetDocumentContent` → 空文档
+5. `persistence.js` 绑定到 Y.Doc 的 `update` 事件
+6. 更新被追加到 Redis 列表并追踪用于脏文档通知
+
+## 持久化模型
+
+- 每次文档更新追加到 Redis 列表：`collab:yjs:doc:updates:{docId}`
+- 达到 `DOC_UPDATE_COMPACT_THRESHOLD`（默认 1000）次更新后，列表压缩为单个状态更新
+- 文档在 Redis 集合中标记为脏：`collab:yjs:dirty:doc`
+- 达到 `DIRTY_DOC_NOTIFY_THRESHOLD`（默认 200）次更新后，通过 MQ 发布脏文档事件
+
+## 脏文档通知
+
+脏文档事件通知下游消费者（snapshot-worker）文档已被修改。
+
+MQ 后端由 `DIRTY_DOC_MQ` 环境变量控制：
+- `redis` — Redis Pub/Sub（默认，无消费者组）
+- `rabbitmq` — RabbitMQ 带消费者组
+- `both` — 同时发布到两者
+
+载荷格式：
+```json
+{
+  "doc_id": "document-id",
+  "ts": 1234567890
+}
+```
+
+主题：`DIRTY_DOC_TOPIC`（默认 `collab.dirty_docs`）
+
+## HTTP 路由
+
+健康探针：
+- `GET /health` — 整体健康状态
+- `GET /livez` — 存活探针
+- `GET /readyz` — 就绪探针（drain 期间失败）
+
+API：
+- `GET /api/active-rooms` — 列出活跃的 WebSocket 房间
+
+内部接口（snapshot-worker 使用）：
+- `GET /internal/yjs/doc-snapshot/{docId}?type={type}` — 获取文档快照
+- `GET /internal/yjs/doc/{docId}` — 获取文档的二进制 Yjs 更新
+
+## WebSocket 协议
+
+房间名从 WebSocket URL 路径提取。对于 `/doc-{docId}`，文档 ID 为 `{docId}`。
+
+协议消息：
+- 同步消息（Yjs 同步协议）
+- 感知消息（光标位置、用户存在状态）
+
+当房间的最后一个连接关闭时，文档从内存中移除。
+
+## Proto 代码生成
+
+`src/gen/` 包含从父仓库的 `proto/yoresee_doc/v1/yoresee_doc.proto` 生成的 gRPC 存根。
+
+此目录已**gitignore**。生产环境中，`Dockerfile.prod` 在构建时生成存根。开发环境从父仓库执行：
+
+```bash
+bash deploy/script/gen_proto.sh
+```
+
+## 环境变量
+
+| 变量 | 默认值 | 描述 |
+|---|---|---|
+| `PORT` | `1234` | HTTP 监听端口 |
+| `REDIS_HOST` | `redis` | Redis 主机名 |
+| `REDIS_PORT` | `6379` | Redis 端口 |
+| `REDIS_PASSWORD` | `''` | Redis 密码 |
+| `REDIS_DB` | `0` | Redis DB 索引 |
+| `BACKEND_ADDR` | `backend:9090` | 后端 gRPC 地址 |
+| `DIRTY_DOC_MQ` | `redis` | MQ 类型：`redis`、`rabbitmq` 或 `both` |
+| `DIRTY_DOC_TOPIC` | `collab.dirty_docs` | 脏文档发布/订阅主题 |
+| `RABBITMQ_URL` | `''` | RabbitMQ 连接 URL（空 = 禁用） |
+| `INTERNAL_RPC_KEY` | `''` | gRPC 元数据认证键（`x-internal-key`） |
+| `DIRTY_DOC_NOTIFY_THRESHOLD` | `200` | 发布脏文档事件前的更新次数 |
+| `DOC_UPDATES_TTL` | `259200` | 文档更新计数器的 TTL（秒） |
+| `DOC_UPDATE_COMPACT_THRESHOLD` | `1000` | 压缩 Redis 列表前的更新次数 |
+| `GC` | `true` | Yjs 垃圾回收（`false`/`0` 禁用） |
+| `BACKEND_SNAPSHOT_TIMEOUT_MS` | `3000` | 后端快照 gRPC 调用超时 |
+
+## Redis 键
+
+在 `src/key.js` 中定义：
+- `collab:yjs:doc:updates:{docId}` — 文档更新列表
+- `collab:room:doc-{docId}` — 房间元数据
+- `collab:yjs:dirty:doc` — 脏文档集合
+
+## 优雅停机
+
+收到 `SIGTERM` 或 `SIGINT` 时：
+1. 服务器停止接受新连接
+2. 就绪探针返回 `not_ready`
+3. 关闭现有 WebSocket 连接
+4. 关闭 MQ 连接
+5. 关闭 Redis 连接
+6. 进程退出（10 秒超时后强制退出）