backend/docs/README.md

# Yoresee Doc Backend

Backend service for the Yoresee Doc collaborative document management platform, built with Go and Connect RPC.

## Overview

Yoresee Doc Backend provides:
- **Document Management**: Create, edit, version, and organize documents with attachment support
- **Knowledge Base**: Hierarchical knowledge base organization
- **User & Access Control**: User groups, organization structure, membership management, invitations
- **Real-time Collaboration**: Yjs snapshot persistence and conflict resolution
- **Full-text Search**: Elasticsearch integration for document search
- **Notifications**: Event-driven notification system via RabbitMQ

## Architecture

### High-Level Structure

```
┌─────────────────────────────────────────────────────────┐
│                    API Layer                             │
│  Connect RPC handlers (HTTP/2 + gRPC-web)               │
│  - Auth interceptor (JWT)                                │
│  - CORS support                                          │
└─────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────┐
│                 Service Layer                            │
│  Business logic with package-level singletons            │
│  - auth_service, document_service, etc.                  │
└─────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────┐
│                Repository Layer                          │
│  Data access with GORM (PostgreSQL)                      │
│  - Constructed via repository.NewRepositories()          │
└─────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────┐
│              Infrastructure Layer                        │
│  - PostgreSQL (primary data)                             │
│  - Redis (cache, sessions, locks)                        │
│  - MinIO (file storage)                                  │
│  - Elasticsearch (search index)                          │
│  - RabbitMQ (message queue)                              │
│  - Consul (service discovery/config)                     │
└─────────────────────────────────────────────────────────┘
```

### Project Layout

```
cmd/
├── main.go                  # API server entry point
├── migrate/                 # Database migration (GORM AutoMigrate)
├── db_init/                 # Seed data (admin user, default groups, templates)
├── es_init/                 # Elasticsearch index initialization
├── notification-worker/     # Notification consumer
├── search-sync-worker/      # Search index sync consumer
└── snapshot-worker/         # Yjs snapshot persistence consumer

internal/
├── bootstrap/               # Chain-style initializer (config → infra → services)
├── config/                  # Viper TOML config loader
├── transport/
│   ├── connectserver/       # HTTP mux, RPC handlers, auth interceptor
│   └── grpcserver/          # Per-service RPC implementations
├── service/                 # Business logic (domain services)
│   ├── <domain>_service/    # Service implementations
│   ├── interface/           # Service interfaces
│   └── mq_service/          # Message queue abstraction
├── repository/              # Data access layer
│   ├── <domain>_repo/       # Repository implementations
│   └── init.go              # Repository construction
├── model/                   # GORM model definitions (16 models)
├── dto/                     # Data transfer objects
├── mapper/                  # Model ↔ DTO mappers
├── types/                   # Domain types
└── status/                  # Typed error codes

pkg/
├── storage/                 # Infrastructure clients (DB, Redis, MinIO, ES, Consul)
├── mq/                      # Message queue interface + RabbitMQ impl
├── cache/                   # Cache utilities
├── lock/                    # Distributed lock
├── key/                     # Shared cache keys
├── errs/                    # Shared errors
├── constant/                # Shared constants
└── gen/                     # GENERATED protobuf stubs (gitignored)
```

## Prerequisites

- **Go 1.24+**
- **PostgreSQL 15+** with `ltree` extension support
- **Redis 7+**
- **MinIO** (or S3-compatible object storage)
- **RabbitMQ 3+**
- **Consul 1.16+**
- **Elasticsearch 8+** (optional, for search functionality)

All infrastructure services can be started via the project's Docker Compose setup.

## Configuration

### Config File

Configuration is loaded from `config.toml` (gitignored), generated from `config.toml.tmpl` by `deploy/script/prepare.sh`.

**Generation:**
```bash
# From repo root
bash deploy/script/prepare.sh
```

**Format:** TOML with environment variable substitution (`${ENV_VAR}`).

**Environment variables:** Defined in `deploy/.env` (template: `deploy/.env.example`).

**Access:** Via `config.GlobalConfig.*` singleton.

### Key Configuration Sections

```toml
[server]
grpc_port = 9090           # Native gRPC port
grpc_web_port = 8080       # gRPC-web port (browser clients)

[database]
host = "localhost"
port = 5432
user = "root"
password = "your_password"
name = "yoresee_doc_db"

[redis]
host = "localhost"
port = 6379
password = "your_redis_password"
db = 0

[consul]
enabled = true
address = "localhost:8500"
scheme = "http"
token = "yoresee_doc_root_token"
prefix = "yoresee_doc"

[minio]
endpoint = "localhost:9000"
access_key = "minioadmin"
secret_key = "minioadmin"
bucket = "yoresee_doc"
use_ssl = false

[elasticsearch]
enabled = true
addresses = ["http://localhost:9200"]
username = "elastic"
password = "your_password"
index_prefix = "yoresee_doc"

[mq_config.rabbitmq]
url = "amqp://guest:guest@localhost:5672/"

[backend]
system_name = "yoresee_doc"
internal_rpc_key = "yoresee_doc_internal_key"

[backend.jwt]
secret = "yoresee_doc_jwt_secret_key"
expire = 86400           # 24 hours
refresh_expire = 604800  # 7 days

[backend.security]
password_hash_cost = 12
max_login_attempts = 5
login_lock_duration = 30  # minutes
```

## Getting Started

### 1. Start Infrastructure

From the repository root:

```bash
# Generate config files
bash deploy/script/configure.sh    # Interactive setup
# OR
cp deploy/.env.example deploy/.env
bash deploy/script/prepare.sh      # Non-interactive

# Start all services via Docker Compose
bash deploy/script/start.sh dev up
```

This starts:
- PostgreSQL, Redis, Consul, MinIO, RabbitMQ, Elasticsearch
- Backend API server (with automatic migration + seeding)
- Background workers (snapshot, notification, search-sync)
- Frontend, collaboration services, nginx

### 2. Run Backend Standalone (Optional)

If infrastructure is already running externally:

```bash
cd backend

# Run migration + seeding
go run ./cmd/migrate
go run ./cmd/db_init
go run ./cmd/es_init

# Start API server
go run ./cmd/main.go
```

### 3. Verify

Health endpoints:
- `GET /health` — Liveness check
- `GET /readyz` — Readiness (checks DB + Redis)
- `GET /livez` — Liveness alias

## API Structure

### Connect RPC

The backend exposes two HTTP listeners using Connect RPC (not raw gRPC):

| Port | Purpose | Protocol |
|------|---------|----------|
| 9090 | Native gRPC | HTTP/2 |
| 8080 | gRPC-web | HTTP/1.1+ (browser clients) |

Both listeners use h2c (HTTP/2 cleartext).

### Services

The API is organized into the following services:

- **AuthService**: Login, register, profile management
- **DocumentService**: Document CRUD, versioning, attachments, templates
- **KnowledgeBaseService**: Knowledge base management
- **MembershipService**: User groups, org structure, memberships
- **InvitationService**: Invitation creation and management
- **NotificationService**: Notification CRUD, read status
- **CommentService**: Document comments
- **SettingService**: System settings
- **SystemService**: Health checks, system info

### Authentication

JWT-based authentication with the following unauthenticated endpoints:
- `AuthService.Login`
- `AuthService.Register`
- `SystemService.Health`
- `SystemService.SystemInfo`

All other endpoints require a valid JWT in the `Authorization: Bearer <token>` header.

## Background Workers

### Snapshot Worker

**Purpose:** Persist Yjs document snapshots to the database.

**Queue:** Consumes from RabbitMQ topic for dirty document notifications.

**Flow:**
1. Collab service marks document as dirty after edits
2. Snapshot worker receives notification
3. Fetches current Yjs state from collab service
4. Persists snapshot to `document_yjs_snapshots` table

### Notification Worker

**Purpose:** Process and deliver notifications.

**Queue:** Consumes from RabbitMQ notification topics.

### Search Sync Worker

**Purpose:** Keep Elasticsearch index in sync with database.

**Queue:** Consumes from RabbitMQ for document/KB changes.

**Flow:**
1. Document/KB created/updated/deleted
2. Worker receives event
3. Updates Elasticsearch index accordingly

## Database

### Schema Management

**Migration:** GORM `AutoMigrate` based on model struct definitions.

**Location:** `cmd/migrate/migration.go`

**Process:**
1. Creates PostgreSQL `ltree` extension (for hierarchical org structure)
2. Auto-migrates all 16 models
3. Creates custom indexes (unique constraints on recent items)
4. Applies manual migrations (e.g., dropping deprecated columns)

**Models:**
- User
- Document, DocumentVersion, DocumentYjsSnapshot, DocumentComment
- KnowledgeBase
- Template
- Attachment
- Invitation, InvitationRecord
- Notification
- UserGroupMeta, MembershipRelation
- OrgNodeMeta
- RecentDocument, RecentKnowledgeBase, RecentTemplate

### Seeding

**Location:** `cmd/db_init/`

**Process:**
1. Creates admin user (username: `admin`, password from config)
2. Creates default permissions
3. Creates default user groups
4. Creates default templates

## Development

### Code Structure Patterns

**Service Layer:**
```go
// Package-level singleton
var AuthSvc *AuthService

// Initialization in service.Init()
func Init(cfg *config.Config, repos *repository.Repositories) error {
    auth_service.AuthSvc = auth_service.NewAuthService(repos)
    // ...
}
```

**Repository Layer:**
```go
// Constructed via Repositories struct
type Repositories struct {
    DB                  *gorm.DB
    Redis               *redis.Client
    Document            *document_repo.DocumentRepository
    // ...
}

func NewRepositories(db *gorm.DB, redis *redis.Client) *Repositories {
    return &Repositories{
        Document: document_repo.NewDocumentRepository(db, redis),
        // ...
    }
}
```

**Error Handling:**
```go
// Typed errors with status codes
return status.GenErrWithCustomMsg(
    status.StatusServiceInternalError,
    "operation failed",
)
```

### Protobuf Codegen

**Source:** `proto/yoresee_doc/v1/yoresee_doc.proto`

**Generated Go code:** `pkg/gen/` (gitignored)

**Regenerate:**
```bash
# From repo root
bash deploy/script/gen_proto.sh
```

**Targets:**
- `backend/pkg/gen` — Go stubs
- `collab-go/pkg/gen` — Go stubs
- `frontend/src/gen` — TypeScript ES modules
- `collab/src/gen` — Node gRPC stubs

### Verification

No linter or test suite configured. Verify changes with:

```bash
go build ./...
go vet ./...
```

## Deployment

### Docker Images

| Dockerfile | Purpose |
|------------|---------|
| `Dockerfile` | Dev image (with Delve debugger) |
| `Dockerfile.prod` | Production API server |
| `Dockerfile.snapshot.prod` | Production snapshot worker |
| `Dockerfile.notification.prod` | Production notification worker |
| `Dockerfile.search-sync.prod` | Production search-sync worker |

### Startup Sequence

Container CMD for production:
```bash
./migrate.out && ./db_init.out && ./es_init.out && ./main.out
```

Workers are separate containers with their own entrypoints.

### Health Checks

Docker Compose uses `wget` to check `/health` endpoint:
```yaml
healthcheck:
  test: ["CMD", "wget", "--spider", "-q", "http://backend:9090/health"]
  interval: 5s
  timeout: 3s
  retries: 20
  start_period: 30s
```

## Infrastructure Ports

| Service | Default Port |
|---------|--------------|
| Backend gRPC | 9090 |
| Backend gRPC-web | 8080 |
| Backend Debug (Delve) | 2345 |
| PostgreSQL | 5432 |
| Redis | 6379 |
| Consul | 8500 |
| MinIO API | 9000 |
| MinIO Console | 9001 |
| RabbitMQ AMQP | 5672 |
| RabbitMQ Management | 15672 |
| Elasticsearch | 9200 |

## Key Dependencies

- **Connect RPC** (`connectrpc.com/connect`) — Modern RPC framework
- **GORM** (`gorm.io/gorm`) — ORM for PostgreSQL
- **go-redis** (`github.com/redis/go-redis/v9`) — Redis client
- **MinIO Go** (`github.com/minio/minio-go/v7`) — S3-compatible storage
- **Viper** (`github.com/spf13/viper`) — Configuration management
- **Logrus** (`github.com/sirupsen/logrus`) — Structured logging
- **JWT** (`github.com/golang-jwt/jwt/v5`) — Authentication
- **amqp091-go** (`github.com/rabbitmq/amqp091-go`) — RabbitMQ client
- **improbable-eng/grpc-web** — gRPC-web proxy for browser clients
- **Snowflake** (`github.com/bwmarrin/snowflake`) — Distributed ID generation

## Environment Variables

All environment variables are defined in `deploy/.env` (see `deploy/.env.example` for template).

Key variables:
- `POSTGRES_*` — Database connection
- `REDIS_*` — Redis connection
- `CONSUL_*` — Consul configuration
- `MINIO_*` — MinIO/S3 configuration
- `ELASTICSEARCH_*` — Elasticsearch configuration
- `RABBITMQ_*` — RabbitMQ configuration
- `BACKEND_*` — Backend service configuration
- `JWT_SECRET` — JWT signing secret

## License

[Your license here]

## Contributing

[Your contributing guidelines here]