update docs

2026-08-11 19:59:18 +08:00
parent 3ad559f05e
commit a6ef1da9bc
3 changed files with 1058 additions and 0 deletions
@@ -0,0 +1,483 @@
+# Yoresee Doc Backend
+
+Backend service for the Yoresee Doc collaborative document management platform, built with Go and Connect RPC.
+
+## Overview
+
+Yoresee Doc Backend provides:
+- **Document Management**: Create, edit, version, and organize documents with attachment support
+- **Knowledge Base**: Hierarchical knowledge base organization
+- **User & Access Control**: User groups, organization structure, membership management, invitations
+- **Real-time Collaboration**: Yjs snapshot persistence and conflict resolution
+- **Full-text Search**: Elasticsearch integration for document search
+- **Notifications**: Event-driven notification system via RabbitMQ
+
+## Architecture
+
+### High-Level Structure
+
+```
+┌─────────────────────────────────────────────────────────┐
+│                    API Layer                             │
+│  Connect RPC handlers (HTTP/2 + gRPC-web)               │
+│  - Auth interceptor (JWT)                                │
+│  - CORS support                                          │
+└─────────────────────────────────────────────────────────┘
+                           ↓
+┌─────────────────────────────────────────────────────────┐
+│                 Service Layer                            │
+│  Business logic with package-level singletons            │
+│  - auth_service, document_service, etc.                  │
+└─────────────────────────────────────────────────────────┘
+                           ↓
+┌─────────────────────────────────────────────────────────┐
+│                Repository Layer                          │
+│  Data access with GORM (PostgreSQL)                      │
+│  - Constructed via repository.NewRepositories()          │
+└─────────────────────────────────────────────────────────┘
+                           ↓
+┌─────────────────────────────────────────────────────────┐
+│              Infrastructure Layer                        │
+│  - PostgreSQL (primary data)                             │
+│  - Redis (cache, sessions, locks)                        │
+│  - MinIO (file storage)                                  │
+│  - Elasticsearch (search index)                          │
+│  - RabbitMQ (message queue)                              │
+│  - Consul (service discovery/config)                     │
+└─────────────────────────────────────────────────────────┘
+```
+
+### Project Layout
+
+```
+cmd/
+├── main.go                  # API server entry point
+├── migrate/                 # Database migration (GORM AutoMigrate)
+├── db_init/                 # Seed data (admin user, default groups, templates)
+├── es_init/                 # Elasticsearch index initialization
+├── notification-worker/     # Notification consumer
+├── search-sync-worker/      # Search index sync consumer
+└── snapshot-worker/         # Yjs snapshot persistence consumer
+
+internal/
+├── bootstrap/               # Chain-style initializer (config → infra → services)
+├── config/                  # Viper TOML config loader
+├── transport/
+│   ├── connectserver/       # HTTP mux, RPC handlers, auth interceptor
+│   └── grpcserver/          # Per-service RPC implementations
+├── service/                 # Business logic (domain services)
+│   ├── <domain>_service/    # Service implementations
+│   ├── interface/           # Service interfaces
+│   └── mq_service/          # Message queue abstraction
+├── repository/              # Data access layer
+│   ├── <domain>_repo/       # Repository implementations
+│   └── init.go              # Repository construction
+├── model/                   # GORM model definitions (16 models)
+├── dto/                     # Data transfer objects
+├── mapper/                  # Model ↔ DTO mappers
+├── types/                   # Domain types
+└── status/                  # Typed error codes
+
+pkg/
+├── storage/                 # Infrastructure clients (DB, Redis, MinIO, ES, Consul)
+├── mq/                      # Message queue interface + RabbitMQ impl
+├── cache/                   # Cache utilities
+├── lock/                    # Distributed lock
+├── key/                     # Shared cache keys
+├── errs/                    # Shared errors
+├── constant/                # Shared constants
+└── gen/                     # GENERATED protobuf stubs (gitignored)
+```
+
+## Prerequisites
+
+- **Go 1.24+**
+- **PostgreSQL 15+** with `ltree` extension support
+- **Redis 7+**
+- **MinIO** (or S3-compatible object storage)
+- **RabbitMQ 3+**
+- **Consul 1.16+**
+- **Elasticsearch 8+** (optional, for search functionality)
+
+All infrastructure services can be started via the project's Docker Compose setup.
+
+## Configuration
+
+### Config File
+
+Configuration is loaded from `config.toml` (gitignored), generated from `config.toml.tmpl` by `deploy/script/prepare.sh`.
+
+**Generation:**
+```bash
+# From repo root
+bash deploy/script/prepare.sh
+```
+
+**Format:** TOML with environment variable substitution (`${ENV_VAR}`).
+
+**Environment variables:** Defined in `deploy/.env` (template: `deploy/.env.example`).
+
+**Access:** Via `config.GlobalConfig.*` singleton.
+
+### Key Configuration Sections
+
+```toml
+[server]
+grpc_port = 9090           # Native gRPC port
+grpc_web_port = 8080       # gRPC-web port (browser clients)
+
+[database]
+host = "localhost"
+port = 5432
+user = "root"
+password = "your_password"
+name = "yoresee_doc_db"
+
+[redis]
+host = "localhost"
+port = 6379
+password = "your_redis_password"
+db = 0
+
+[consul]
+enabled = true
+address = "localhost:8500"
+scheme = "http"
+token = "yoresee_doc_root_token"
+prefix = "yoresee_doc"
+
+[minio]
+endpoint = "localhost:9000"
+access_key = "minioadmin"
+secret_key = "minioadmin"
+bucket = "yoresee_doc"
+use_ssl = false
+
+[elasticsearch]
+enabled = true
+addresses = ["http://localhost:9200"]
+username = "elastic"
+password = "your_password"
+index_prefix = "yoresee_doc"
+
+[mq_config.rabbitmq]
+url = "amqp://guest:guest@localhost:5672/"
+
+[backend]
+system_name = "yoresee_doc"
+internal_rpc_key = "yoresee_doc_internal_key"
+
+[backend.jwt]
+secret = "yoresee_doc_jwt_secret_key"
+expire = 86400           # 24 hours
+refresh_expire = 604800  # 7 days
+
+[backend.security]
+password_hash_cost = 12
+max_login_attempts = 5
+login_lock_duration = 30  # minutes
+```
+
+## Getting Started
+
+### 1. Start Infrastructure
+
+From the repository root:
+
+```bash
+# Generate config files
+bash deploy/script/configure.sh    # Interactive setup
+# OR
+cp deploy/.env.example deploy/.env
+bash deploy/script/prepare.sh      # Non-interactive
+
+# Start all services via Docker Compose
+bash deploy/script/start.sh dev up
+```
+
+This starts:
+- PostgreSQL, Redis, Consul, MinIO, RabbitMQ, Elasticsearch
+- Backend API server (with automatic migration + seeding)
+- Background workers (snapshot, notification, search-sync)
+- Frontend, collaboration services, nginx
+
+### 2. Run Backend Standalone (Optional)
+
+If infrastructure is already running externally:
+
+```bash
+cd backend
+
+# Run migration + seeding
+go run ./cmd/migrate
+go run ./cmd/db_init
+go run ./cmd/es_init
+
+# Start API server
+go run ./cmd/main.go
+```
+
+### 3. Verify
+
+Health endpoints:
+- `GET /health` — Liveness check
+- `GET /readyz` — Readiness (checks DB + Redis)
+- `GET /livez` — Liveness alias
+
+## API Structure
+
+### Connect RPC
+
+The backend exposes two HTTP listeners using Connect RPC (not raw gRPC):
+
+| Port | Purpose | Protocol |
+|------|---------|----------|
+| 9090 | Native gRPC | HTTP/2 |
+| 8080 | gRPC-web | HTTP/1.1+ (browser clients) |
+
+Both listeners use h2c (HTTP/2 cleartext).
+
+### Services
+
+The API is organized into the following services:
+
+- **AuthService**: Login, register, profile management
+- **DocumentService**: Document CRUD, versioning, attachments, templates
+- **KnowledgeBaseService**: Knowledge base management
+- **MembershipService**: User groups, org structure, memberships
+- **InvitationService**: Invitation creation and management
+- **NotificationService**: Notification CRUD, read status
+- **CommentService**: Document comments
+- **SettingService**: System settings
+- **SystemService**: Health checks, system info
+
+### Authentication
+
+JWT-based authentication with the following unauthenticated endpoints:
+- `AuthService.Login`
+- `AuthService.Register`
+- `SystemService.Health`
+- `SystemService.SystemInfo`
+
+All other endpoints require a valid JWT in the `Authorization: Bearer <token>` header.
+
+## Background Workers
+
+### Snapshot Worker
+
+**Purpose:** Persist Yjs document snapshots to the database.
+
+**Queue:** Consumes from RabbitMQ topic for dirty document notifications.
+
+**Flow:**
+1. Collab service marks document as dirty after edits
+2. Snapshot worker receives notification
+3. Fetches current Yjs state from collab service
+4. Persists snapshot to `document_yjs_snapshots` table
+
+### Notification Worker
+
+**Purpose:** Process and deliver notifications.
+
+**Queue:** Consumes from RabbitMQ notification topics.
+
+### Search Sync Worker
+
+**Purpose:** Keep Elasticsearch index in sync with database.
+
+**Queue:** Consumes from RabbitMQ for document/KB changes.
+
+**Flow:**
+1. Document/KB created/updated/deleted
+2. Worker receives event
+3. Updates Elasticsearch index accordingly
+
+## Database
+
+### Schema Management
+
+**Migration:** GORM `AutoMigrate` based on model struct definitions.
+
+**Location:** `cmd/migrate/migration.go`
+
+**Process:**
+1. Creates PostgreSQL `ltree` extension (for hierarchical org structure)
+2. Auto-migrates all 16 models
+3. Creates custom indexes (unique constraints on recent items)
+4. Applies manual migrations (e.g., dropping deprecated columns)
+
+**Models:**
+- User
+- Document, DocumentVersion, DocumentYjsSnapshot, DocumentComment
+- KnowledgeBase
+- Template
+- Attachment
+- Invitation, InvitationRecord
+- Notification
+- UserGroupMeta, MembershipRelation
+- OrgNodeMeta
+- RecentDocument, RecentKnowledgeBase, RecentTemplate
+
+### Seeding
+
+**Location:** `cmd/db_init/`
+
+**Process:**
+1. Creates admin user (username: `admin`, password from config)
+2. Creates default permissions
+3. Creates default user groups
+4. Creates default templates
+
+## Development
+
+### Code Structure Patterns
+
+**Service Layer:**
+```go
+// Package-level singleton
+var AuthSvc *AuthService
+
+// Initialization in service.Init()
+func Init(cfg *config.Config, repos *repository.Repositories) error {
+    auth_service.AuthSvc = auth_service.NewAuthService(repos)
+    // ...
+}
+```
+
+**Repository Layer:**
+```go
+// Constructed via Repositories struct
+type Repositories struct {
+    DB                  *gorm.DB
+    Redis               *redis.Client
+    Document            *document_repo.DocumentRepository
+    // ...
+}
+
+func NewRepositories(db *gorm.DB, redis *redis.Client) *Repositories {
+    return &Repositories{
+        Document: document_repo.NewDocumentRepository(db, redis),
+        // ...
+    }
+}
+```
+
+**Error Handling:**
+```go
+// Typed errors with status codes
+return status.GenErrWithCustomMsg(
+    status.StatusServiceInternalError,
+    "operation failed",
+)
+```
+
+### Protobuf Codegen
+
+**Source:** `proto/yoresee_doc/v1/yoresee_doc.proto`
+
+**Generated Go code:** `pkg/gen/` (gitignored)
+
+**Regenerate:**
+```bash
+# From repo root
+bash deploy/script/gen_proto.sh
+```
+
+**Targets:**
+- `backend/pkg/gen` — Go stubs
+- `collab-go/pkg/gen` — Go stubs
+- `frontend/src/gen` — TypeScript ES modules
+- `collab/src/gen` — Node gRPC stubs
+
+### Verification
+
+No linter or test suite configured. Verify changes with:
+
+```bash
+go build ./...
+go vet ./...
+```
+
+## Deployment
+
+### Docker Images
+
+| Dockerfile | Purpose |
+|------------|---------|
+| `Dockerfile` | Dev image (with Delve debugger) |
+| `Dockerfile.prod` | Production API server |
+| `Dockerfile.snapshot.prod` | Production snapshot worker |
+| `Dockerfile.notification.prod` | Production notification worker |
+| `Dockerfile.search-sync.prod` | Production search-sync worker |
+
+### Startup Sequence
+
+Container CMD for production:
+```bash
+./migrate.out && ./db_init.out && ./es_init.out && ./main.out
+```
+
+Workers are separate containers with their own entrypoints.
+
+### Health Checks
+
+Docker Compose uses `wget` to check `/health` endpoint:
+```yaml
+healthcheck:
+  test: ["CMD", "wget", "--spider", "-q", "http://backend:9090/health"]
+  interval: 5s
+  timeout: 3s
+  retries: 20
+  start_period: 30s
+```
+
+## Infrastructure Ports
+
+| Service | Default Port |
+|---------|--------------|
+| Backend gRPC | 9090 |
+| Backend gRPC-web | 8080 |
+| Backend Debug (Delve) | 2345 |
+| PostgreSQL | 5432 |
+| Redis | 6379 |
+| Consul | 8500 |
+| MinIO API | 9000 |
+| MinIO Console | 9001 |
+| RabbitMQ AMQP | 5672 |
+| RabbitMQ Management | 15672 |
+| Elasticsearch | 9200 |
+
+## Key Dependencies
+
+- **Connect RPC** (`connectrpc.com/connect`) — Modern RPC framework
+- **GORM** (`gorm.io/gorm`) — ORM for PostgreSQL
+- **go-redis** (`github.com/redis/go-redis/v9`) — Redis client
+- **MinIO Go** (`github.com/minio/minio-go/v7`) — S3-compatible storage
+- **Viper** (`github.com/spf13/viper`) — Configuration management
+- **Logrus** (`github.com/sirupsen/logrus`) — Structured logging
+- **JWT** (`github.com/golang-jwt/jwt/v5`) — Authentication
+- **amqp091-go** (`github.com/rabbitmq/amqp091-go`) — RabbitMQ client
+- **improbable-eng/grpc-web** — gRPC-web proxy for browser clients
+- **Snowflake** (`github.com/bwmarrin/snowflake`) — Distributed ID generation
+
+## Environment Variables
+
+All environment variables are defined in `deploy/.env` (see `deploy/.env.example` for template).
+
+Key variables:
+- `POSTGRES_*` — Database connection
+- `REDIS_*` — Redis connection
+- `CONSUL_*` — Consul configuration
+- `MINIO_*` — MinIO/S3 configuration
+- `ELASTICSEARCH_*` — Elasticsearch configuration
+- `RABBITMQ_*` — RabbitMQ configuration
+- `BACKEND_*` — Backend service configuration
+- `JWT_SECRET` — JWT signing secret
+
+## License
+
+[Your license here]
+
+## Contributing
+
+[Your contributing guidelines here]