Architecture Overview

TFRS Operator is built on Kubebuilder and manages the full lifecycle of TFRobot multi-tenant AI/robotics services through three levels of Custom Resource Definitions (CRDs).

Resource Model

The Operator defines three CRD layers forming a top-down management structure:

TFRCluster (Cluster-scoped)
  └── TFRTenant (Namespace-scoped)
        └── TFRServer (Namespace-scoped)

CRD	Scope	Responsibility
TFRCluster	Cluster	Manages cluster-level infrastructure: Istio ingress, TLS certificates, Operator installation, image registry credentials
TFRTenant	Namespace	Configures tenant environments: databases, caching, object storage, message queues, monitoring stack
TFRServer	Namespace	Manages individual TFRobot service instances: API workers, robot workers, configuration and secrets

Multi-Tenancy Model

TFRTenant supports two tenant types:

Type	Description	Isolation Level
enterprise	Enterprise tenant with dedicated namespace	Full isolation: independent network policies, storage, and middleware instances
personal	Personal tenant with shared namespace by job type	Logical isolation: differentiated by RobotId, shared middleware

Both types achieve network isolation through Istio PeerAuthentication (STRICT mTLS) and AuthorizationPolicy.

Infrastructure Components

Cluster-Level (Installed by TFRCluster)

Component	Deployment Method	Description
cert-manager	Helm Chart	Automated TLS certificate management
Istio	Helm Chart	Service mesh foundation
Istio Ingress Gateway	Helm Chart	Ingress gateway, supports TKE CLB / NodePort / Docker Desktop
CNPG Operator	YAML Manifest	CloudNativePG PostgreSQL Operator
MinIO Operator	Helm / YAML	MinIO object storage Operator
RabbitMQ Operator	YAML Manifest	RabbitMQ Cluster and Topology Operator
Image Registry Credentials	Secret	Cross-namespace shared imagePullSecret

Tenant-Level (Instances Created by TFRTenant)

Component	Deployment Method	Description
PostgreSQL	CNPG Cluster CR	Primary database, 16-minimal-bookworm
Redis	Deployment + PVC	Cache service (redis-stack-server)
MinIO	MinIO Tenant CR	Object storage
RabbitMQ	RabbitMQ Cluster CR	Message queue
TimescaleDB	CNPG + Extension	Time-series database
Jaeger	Helm Chart	Distributed tracing
Grafana	Helm Chart	Monitoring dashboard
Prometheus	Deployment	Metrics collection
OTel Collector	Deployment	OpenTelemetry trace collection
Vector	Deployment	Log collection
SharedCache PVC	PVC (RWX)	Shared image cache volume

Service-Level (Configured by TFRServer)

Component	Description
API Worker	TFRobot API service
Robot Worker	Robot task Worker
DB Init Job	Database schema initialization
MinIO Init Job	Storage bucket initialization
Grafana SA Job	Grafana service account setup
TOML Config	Confz configuration auto-generation

Deployment Dependency Graph

TFRCluster Component Deployment Order

graph TD
    CM[cert-manager] --> Istio
    Istio --> IG[Istio Ingress]
    IG --> GW[Gateway]
    Istio --> EFC[EnvoyFilter Cookie]
    Istio --> EFL[EnvoyFilter Lua Routing]
    GW --> VS[ClusterVirtualService]
    EFC --> VS
    Istio --> IT[Istio Telemetry]

    MO[MinIO Operator]
    CO[CNPG Operator]
    RO[RabbitMQ Operator]
    IR[ImageRegistry]

cert-manager → Istio → Istio Ingress → Gateway/EnvoyFilter → VirtualService form a serial chain. The three middleware Operators and ImageRegistry can be installed in parallel.

TFRTenant Component Deployment Order

graph TD
    NS[Namespace] --> PG[PostgreSQL]
    NS --> RD[Redis]
    NS --> MN[MinIO]
    NS --> JG[Jaeger]
    NS --> PR[Prometheus]
    NS --> VT[Vector]
    NS --> SC[SharedCache PVC]
    NS --> TF[TFRobotFront]
    NS --> TU[TFRSUtils]

    PG --> TD[TimescaleDB]
    JG --> OT[OTel Collector]
    RO_DEP[RabbitMQ Operator Ready] --> RQ[RabbitMQ]
    NS --> GF[Grafana]

Most components can be deployed in parallel once the Namespace is created. TimescaleDB depends on PostgreSQL, OTel Collector depends on Jaeger, and RabbitMQ depends on the Operator being ready.

Credential Management

TFRS Operator manages credentials at three levels:

Level	Secret Example	Description
TFRCluster	{registry}-pull-secret	Image registry pull credentials, shared across namespaces
TFRTenant	{namespace}-pg-credentials, MinIO root credentials	Middleware access credentials, auto-generated or user-specified
TFRServer	{robot}-secret, {robot}-config	Application-level secrets (API Keys) and configuration

When password fields are not specified, the Operator automatically generates random passwords and stores them in Secrets.

ConfzData Auto-Configuration

TFRServer uses ConfzData (TOML format) to manage application configuration. The Operator automatically injects middleware connection information:

# Auto-generated example (no manual input required)
[postgresql]
host = "{namespace}-pg-rw"
port = 5432
user = "app_user"
dbname = "{namespace}"

[redis]
host = "redis.{namespace}.svc.cluster.local"
port = 6379

[celery]
broker_url = "amqp://user:pass@rabbitmq.{namespace}.svc.cluster.local:5672/"

[cos]
host = "minio.{namespace}.svc.cluster.local"
port = 9000

Users can override any field via spec.confzData. Unspecified fields use Operator-generated dynamic values.

Environment Variable Injection

TFRServer supports environment variable injection through secretData and configData:

Secret-type: OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_GEMINI_API_KEY, etc.

Config-type (ConfigMap): OPENAI_BASE_URL, TFS_PROFILE, TFS_ROBOT__DB_SCHEMA, etc.

Auto-injected (highest priority): TFS_ROBOT__DOC_EXTRACTED_IMAGES_DIR is automatically injected from sharedCache.cacheDir, overriding any ConfigMap value of the same name.