backend/docs

Yoresee Doc Backend

Backend service for the Yoresee Doc collaborative document management platform, built with Go and Connect RPC.

Overview

Yoresee Doc Backend provides:

• Document Management: Create, edit, version, and organize documents with attachment support
• Knowledge Base: Hierarchical knowledge base organization
• User & Access Control: User groups, organization structure, membership management, invitations
• Real-time Collaboration: Yjs snapshot persistence and conflict resolution
• Full-text Search: Elasticsearch integration for document search
• Notifications: Event-driven notification system via RabbitMQ

Architecture

High-Level Structure

┌─────────────────────────────────────────────────────────┐
│                    API Layer                             │
│  Connect RPC handlers (HTTP/2 + gRPC-web)               │
│  - Auth interceptor (JWT)                                │
│  - CORS support                                          │
└─────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────┐
│                 Service Layer                            │
│  Business logic with package-level singletons            │
│  - auth_service, document_service, etc.                  │
└─────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────┐
│                Repository Layer                          │
│  Data access with GORM (PostgreSQL)                      │
│  - Constructed via repository.NewRepositories()          │
└─────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────┐
│              Infrastructure Layer                        │
│  - PostgreSQL (primary data)                             │
│  - Redis (cache, sessions, locks)                        │
│  - MinIO (file storage)                                  │
│  - Elasticsearch (search index)                          │
│  - RabbitMQ (message queue)                              │
│  - Consul (service discovery/config)                     │
└─────────────────────────────────────────────────────────┘

Project Layout

cmd/
├── main.go                  # API server entry point
├── migrate/                 # Database migration (GORM AutoMigrate)
├── db_init/                 # Seed data (admin user, default groups, templates)
├── es_init/                 # Elasticsearch index initialization
├── notification-worker/     # Notification consumer
├── search-sync-worker/      # Search index sync consumer
└── snapshot-worker/         # Yjs snapshot persistence consumer

internal/
├── bootstrap/               # Chain-style initializer (config → infra → services)
├── config/                  # Viper TOML config loader
├── transport/
│   ├── connectserver/       # HTTP mux, RPC handlers, auth interceptor
│   └── grpcserver/          # Per-service RPC implementations
├── service/                 # Business logic (domain services)
│   ├── <domain>_service/    # Service implementations
│   ├── interface/           # Service interfaces
│   └── mq_service/          # Message queue abstraction
├── repository/              # Data access layer
│   ├── <domain>_repo/       # Repository implementations
│   └── init.go              # Repository construction
├── model/                   # GORM model definitions (16 models)
├── dto/                     # Data transfer objects
├── mapper/                  # Model ↔ DTO mappers
├── types/                   # Domain types
└── status/                  # Typed error codes

pkg/
├── storage/                 # Infrastructure clients (DB, Redis, MinIO, ES, Consul)
├── mq/                      # Message queue interface + RabbitMQ impl
├── cache/                   # Cache utilities
├── lock/                    # Distributed lock
├── key/                     # Shared cache keys
├── errs/                    # Shared errors
├── constant/                # Shared constants
└── gen/                     # GENERATED protobuf stubs (gitignored)

Prerequisites

• Go 1.24+
• PostgreSQL 15+ with ltree extension support
• Redis 7+
• MinIO (or S3-compatible object storage)
• RabbitMQ 3+
• Consul 1.16+
• Elasticsearch 8+ (optional, for search functionality)

All infrastructure services can be started via the project's Docker Compose setup.

Configuration

Config File

Configuration is loaded from config.toml (gitignored), generated from config.toml.tmpl by deploy/script/prepare.sh.

Generation:

# From repo root
bash deploy/script/prepare.sh

Format: TOML with environment variable substitution (${ENV_VAR}).

Environment variables: Defined in deploy/.env (template: deploy/.env.example).

Access: Via config.GlobalConfig.* singleton.

Key Configuration Sections

[server]
grpc_port = 9090           # Native gRPC port
grpc_web_port = 8080       # gRPC-web port (browser clients)

[database]
host = "localhost"
port = 5432
user = "root"
password = "your_password"
name = "yoresee_doc_db"

[redis]
host = "localhost"
port = 6379
password = "your_redis_password"
db = 0

[consul]
enabled = true
address = "localhost:8500"
scheme = "http"
token = "yoresee_doc_root_token"
prefix = "yoresee_doc"

[minio]
endpoint = "localhost:9000"
access_key = "minioadmin"
secret_key = "minioadmin"
bucket = "yoresee_doc"
use_ssl = false

[elasticsearch]
enabled = true
addresses = ["http://localhost:9200"]
username = "elastic"
password = "your_password"
index_prefix = "yoresee_doc"

[mq_config.rabbitmq]
url = "amqp://guest:guest@localhost:5672/"

[backend]
system_name = "yoresee_doc"
internal_rpc_key = "yoresee_doc_internal_key"

[backend.jwt]
secret = "yoresee_doc_jwt_secret_key"
expire = 86400           # 24 hours
refresh_expire = 604800  # 7 days

[backend.security]
password_hash_cost = 12
max_login_attempts = 5
login_lock_duration = 30  # minutes

Getting Started

1. Start Infrastructure

From the repository root:

# Generate config files
bash deploy/script/configure.sh    # Interactive setup
# OR
cp deploy/.env.example deploy/.env
bash deploy/script/prepare.sh      # Non-interactive

# Start all services via Docker Compose
bash deploy/script/start.sh dev up

This starts:

• PostgreSQL, Redis, Consul, MinIO, RabbitMQ, Elasticsearch
• Backend API server (with automatic migration + seeding)
• Background workers (snapshot, notification, search-sync)
• Frontend, collaboration services, nginx

2. Run Backend Standalone (Optional)

If infrastructure is already running externally:

cd backend

# Run migration + seeding
go run ./cmd/migrate
go run ./cmd/db_init
go run ./cmd/es_init

# Start API server
go run ./cmd/main.go

3. Verify

Health endpoints:

• GET /health — Liveness check
• GET /readyz — Readiness (checks DB + Redis)
• GET /livez — Liveness alias

API Structure

Connect RPC

The backend exposes two HTTP listeners using Connect RPC (not raw gRPC):

Port	Purpose	Protocol
9090	Native gRPC	HTTP/2
8080	gRPC-web	HTTP/1.1+ (browser clients)

Both listeners use h2c (HTTP/2 cleartext).

Services

The API is organized into the following services:

• AuthService: Login, register, profile management
• DocumentService: Document CRUD, versioning, attachments, templates
• KnowledgeBaseService: Knowledge base management
• MembershipService: User groups, org structure, memberships
• InvitationService: Invitation creation and management
• NotificationService: Notification CRUD, read status
• CommentService: Document comments
• SettingService: System settings
• SystemService: Health checks, system info

Authentication

JWT-based authentication with the following unauthenticated endpoints:

• AuthService.Login
• AuthService.Register
• SystemService.Health
• SystemService.SystemInfo

All other endpoints require a valid JWT in the Authorization: Bearer <token> header.

Background Workers

Snapshot Worker

Purpose: Persist Yjs document snapshots to the database.

Queue: Consumes from RabbitMQ topic for dirty document notifications.

Flow:

1. Collab service marks document as dirty after edits
2. Snapshot worker receives notification
3. Fetches current Yjs state from collab service
4. Persists snapshot to document_yjs_snapshots table

Notification Worker

Purpose: Process and deliver notifications.

Queue: Consumes from RabbitMQ notification topics.

Search Sync Worker

Purpose: Keep Elasticsearch index in sync with database.

Queue: Consumes from RabbitMQ for document/KB changes.

Flow:

1. Document/KB created/updated/deleted
2. Worker receives event
3. Updates Elasticsearch index accordingly

Database

Schema Management

Migration: GORM AutoMigrate based on model struct definitions.

Location: cmd/migrate/migration.go

Process:

1. Creates PostgreSQL ltree extension (for hierarchical org structure)
2. Auto-migrates all 16 models
3. Creates custom indexes (unique constraints on recent items)
4. Applies manual migrations (e.g., dropping deprecated columns)

Models:

• User
• Document, DocumentVersion, DocumentYjsSnapshot, DocumentComment
• KnowledgeBase
• Template
• Attachment
• Invitation, InvitationRecord
• Notification
• UserGroupMeta, MembershipRelation
• OrgNodeMeta
• RecentDocument, RecentKnowledgeBase, RecentTemplate

Seeding

Location: cmd/db_init/

Process:

1. Creates admin user (username: admin, password from config)
2. Creates default permissions
3. Creates default user groups
4. Creates default templates

Development

Code Structure Patterns

Service Layer:

// Package-level singleton
var AuthSvc *AuthService

// Initialization in service.Init()
func Init(cfg *config.Config, repos *repository.Repositories) error {
    auth_service.AuthSvc = auth_service.NewAuthService(repos)
    // ...
}

Repository Layer:

// Constructed via Repositories struct
type Repositories struct {
    DB                  *gorm.DB
    Redis               *redis.Client
    Document            *document_repo.DocumentRepository
    // ...
}

func NewRepositories(db *gorm.DB, redis *redis.Client) *Repositories {
    return &Repositories{
        Document: document_repo.NewDocumentRepository(db, redis),
        // ...
    }
}

Error Handling:

// Typed errors with status codes
return status.GenErrWithCustomMsg(
    status.StatusServiceInternalError,
    "operation failed",
)

Protobuf Codegen

Source: proto/yoresee_doc/v1/yoresee_doc.proto

Generated Go code: pkg/gen/ (gitignored)

Regenerate:

# From repo root
bash deploy/script/gen_proto.sh

Targets:

• backend/pkg/gen — Go stubs
• collab-go/pkg/gen — Go stubs
• frontend/src/gen — TypeScript ES modules
• collab/src/gen — Node gRPC stubs

Verification

No linter or test suite configured. Verify changes with:

go build ./...
go vet ./...

Deployment

Docker Images

Dockerfile	Purpose
Dockerfile	Dev image (with Delve debugger)
Dockerfile.prod	Production API server
Dockerfile.snapshot.prod	Production snapshot worker
Dockerfile.notification.prod	Production notification worker
Dockerfile.search-sync.prod	Production search-sync worker

Startup Sequence

Container CMD for production:

./migrate.out && ./db_init.out && ./es_init.out && ./main.out

Workers are separate containers with their own entrypoints.

Health Checks

Docker Compose uses wget to check /health endpoint:

healthcheck:
  test: ["CMD", "wget", "--spider", "-q", "http://backend:9090/health"]
  interval: 5s
  timeout: 3s
  retries: 20
  start_period: 30s

Infrastructure Ports

Service	Default Port
Backend gRPC	9090
Backend gRPC-web	8080
Backend Debug (Delve)	2345
PostgreSQL	5432
Redis	6379
Consul	8500
MinIO API	9000
MinIO Console	9001
RabbitMQ AMQP	5672
RabbitMQ Management	15672
Elasticsearch	9200

Key Dependencies

• Connect RPC (connectrpc.com/connect) — Modern RPC framework
• GORM (gorm.io/gorm) — ORM for PostgreSQL
• go-redis (github.com/redis/go-redis/v9) — Redis client
• MinIO Go (github.com/minio/minio-go/v7) — S3-compatible storage
• Viper (github.com/spf13/viper) — Configuration management
• Logrus (github.com/sirupsen/logrus) — Structured logging
• JWT (github.com/golang-jwt/jwt/v5) — Authentication
• amqp091-go (github.com/rabbitmq/amqp091-go) — RabbitMQ client
• improbable-eng/grpc-web — gRPC-web proxy for browser clients
• Snowflake (github.com/bwmarrin/snowflake) — Distributed ID generation

Environment Variables

All environment variables are defined in deploy/.env (see deploy/.env.example for template).

Key variables:

• POSTGRES_* — Database connection
• REDIS_* — Redis connection
• CONSUL_* — Consul configuration
• MINIO_* — MinIO/S3 configuration
• ELASTICSEARCH_* — Elasticsearch configuration
• RABBITMQ_* — RabbitMQ configuration
• BACKEND_* — Backend service configuration
• JWT_SECRET — JWT signing secret

License

[Your license here]

Contributing

[Your contributing guidelines here]