> For the complete documentation index, see [llms.txt](https://docs.enclaive.cloud/enclaive-multi-cloud-platform/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.enclaive.cloud/enclaive-multi-cloud-platform/tenant-auto-creation.md).

# Tenant auto creation

## Tenant Onboarding Automation - Swimlane Flow Diagram

### Issue Reference: enclaive/emcp-be#2279

***

### Swimlane: Full Tenant Provisioning Flow

```mermaid
sequenceDiagram
    autonumber
    actor Admin as Super Admin
    participant UI as admin-ui<br/>(React)
    participant BE as emcp-be<br/>(Orchestrator)
    participant DB as MongoDB
    participant RS as Redis Streams
    participant KC as Keycloak
    participant VLT as Vault
    participant WR as workload-runner<br/>(NestJS)
    participant CF as Cloudflare
    participant GH as GitHub Actions
    participant ARGO as ArgoCD

    %% ──────────────────────────────────────────────
    rect rgb(240, 248, 255)
    Note over Admin,ARGO: PHASE 1 ── Request & Validation
    %% ──────────────────────────────────────────────

    Admin->>UI: Fills "Create Tenant" form<br/>(subdomain, admin email, plan)
    UI->>UI: Client-side validation<br/>- subdomain format (lowercase, no special chars)<br/>- email format<br/>- reserved word check (api, admin, www, mail...)

    UI->>BE: POST /api/admin/tenants<br/>{subdomain, adminEmail, plan, createdBy}

    BE->>BE: Server-side validation<br/>- sanitize input<br/>- verify caller is super-admin

    BE->>DB: Query: does subdomain already exist?
    DB-->>BE: No match found - subdomain available

    BE->>BE: Generate env-specific subdomains<br/>- {tenant}.enclaive.cloud (prod)<br/>- {tenant}.staging.enclaive.cloud<br/>- {tenant}.dev.enclaive.cloud

    BE->>DB: INSERT tenant record<br/>{id, subdomain, status: PROVISIONING,<br/>createdBy: adminId, createdAt: now(),<br/>steps: {keycloak: pending, vault: pending,<br/>dns: pending, ingress: pending,<br/>deployment: pending}}
    DB-->>BE: Tenant doc created (tenantId)

    BE->>DB: INSERT audit log<br/>{action: TENANT_CREATE, actor: adminId,<br/>timestamp, previousState: null, newState: PROVISIONING}

    BE-->>UI: 202 Accepted<br/>{tenantId, status: PROVISIONING}
    UI->>Admin: Show provisioning progress tracker<br/>(step indicators with live updates)
    end

    %% ──────────────────────────────────────────────
    rect rgb(255, 248, 240)
    Note over Admin,ARGO: PHASE 2 ── Keycloak Realm Provisioning
    %% ──────────────────────────────────────────────

    BE->>DB: UPDATE tenant.steps.keycloak = "in_progress"
    BE-->>UI: SSE push: {step: keycloak, status: in_progress}

    BE->>BE: Render realm JSON from Handlebars template<br/>- inject: subdomain, adminEmail, clientURLs<br/>- template includes: theme, token config, scopes

    BE->>KC: POST /admin/realms<br/>Create realm via keycloak-config-cli JAR
    KC-->>BE: 201 Realm "{tenant}" created

    BE->>KC: Configure realm settings<br/>- Login theme: enclaive-theme<br/>- Email provider: SMTP/SendGrid<br/>- Token lifespans (access: 5m, refresh: 30m, SSO: 8h)<br/>- Password policy: 8 chars, mixed case, special
    KC-->>BE: Realm settings applied

    BE->>KC: Create client scopes<br/>- openid, profile, email, roles
    KC-->>BE: Scopes created

    BE->>KC: Register OIDC clients<br/>- console-ui (public, PKCE)<br/>- admin-ui (public, PKCE)<br/>- api (confidential, client_credentials)
    KC-->>BE: Clients registered + client secrets returned

    BE->>KC: POST /admin/realms/{tenant}/users<br/>Create admin user (email, temp password,<br/>requiredActions: [UPDATE_PASSWORD])
    KC-->>BE: Admin user created (userId)

    BE->>KC: Assign roles to admin user<br/>- realm-admin (realm-management client)<br/>- view-users, manage-users, manage-clients
    KC-->>BE: Roles assigned

    BE->>DB: UPDATE tenant<br/>steps.keycloak = "completed"<br/>keycloakRealmId, oidcClientIds, adminUserId
    BE-->>UI: SSE push: {step: keycloak, status: completed}
    UI->>Admin: Keycloak step turns green
    end

    %% ──────────────────────────────────────────────
    rect rgb(240, 255, 240)
    Note over Admin,ARGO: PHASE 3 ── Vault Namespace & Secrets
    %% ──────────────────────────────────────────────

    BE->>DB: UPDATE tenant.steps.vault = "in_progress"
    BE-->>UI: SSE push: {step: vault, status: in_progress}

    BE->>VLT: PUT /v1/sys/namespaces/{tenant}<br/>Create isolated namespace
    VLT-->>BE: Namespace created

    BE->>VLT: POST /v1/{tenant}/sys/mounts/identity<br/>Enable identity engine
    VLT-->>BE: Identity engine mounted

    BE->>VLT: POST /v1/{tenant}/sys/mounts/secret<br/>Enable KV-v2 secrets engine
    VLT-->>BE: KV-v2 engine mounted at secret/

    BE->>VLT: Write secrets batch<br/>- secret/db → MongoDB connection string<br/>- secret/oidc → client secrets from Keycloak<br/>- secret/api-keys → Stripe key, Cloudflare token<br/>- secret/encryption → AES-256 key for data at rest
    VLT-->>BE: All secrets written

    BE->>VLT: PUT /v1/{tenant}/sys/policies/acl/tenant-policy<br/>Create access policy (path-based ACL)<br/>- allow read on secret/*<br/>- deny sys/*
    VLT-->>BE: Policy created

    BE->>VLT: POST /v1/{tenant}/auth/kubernetes/role/{tenant}-role<br/>Create K8s auth role<br/>- bound service account: {tenant}-sa<br/>- bound namespace: emcp-{env}<br/>- policies: tenant-policy<br/>- TTL: 1h
    VLT-->>BE: K8s auth role created

    BE->>VLT: Generate Vault Agent sidecar config<br/>- pod annotation template for injection<br/>- auto-auth via K8s service account
    VLT-->>BE: Agent config stored

    BE->>DB: UPDATE tenant<br/>steps.vault = "completed"<br/>vaultNamespace: "{tenant}"
    BE-->>UI: SSE push: {step: vault, status: completed}
    UI->>Admin: Vault step turns green
    end

    %% ──────────────────────────────────────────────
    rect rgb(255, 255, 235)
    Note over Admin,ARGO: PHASE 4 ── Cloudflare DNS Records (via workload-runner)
    %% ──────────────────────────────────────────────

    BE->>DB: UPDATE tenant.steps.dns = "in_progress"
    BE-->>UI: SSE push: {step: dns, status: in_progress}

    BE->>RS: XADD stream:dns-provisioning<br/>{tenantId, subdomain, environments: [prod, staging, dev],<br/>zoneId, recordType: CNAME, target: ingress-lb.enclaive.cloud}

    RS-->>WR: Consumer picks up DNS task

    WR->>CF: POST /zones/{zoneId}/dns_records<br/>{type: CNAME, name: "{tenant}.enclaive.cloud",<br/>content: "ingress-lb.enclaive.cloud", proxied: true}
    CF-->>WR: Record created (prod)

    WR->>CF: POST /zones/{zoneId}/dns_records<br/>{type: CNAME, name: "{tenant}.staging.enclaive.cloud",<br/>content: "ingress-lb.enclaive.cloud", proxied: true}
    CF-->>WR: Record created (staging)

    WR->>CF: POST /zones/{zoneId}/dns_records<br/>{type: CNAME, name: "{tenant}.dev.enclaive.cloud",<br/>content: "ingress-lb.enclaive.cloud", proxied: true}
    CF-->>WR: Record created (dev)

    %% ── RETRY LOGIC ──
    rect rgb(255, 230, 230)
    Note over WR,CF: RETRY LOGIC (Cloudflare is external ── most failure-prone)
    alt CF returns 429 (rate limit) or 5xx (server error)
        WR->>WR: Retry with exponential backoff<br/>Attempt 1: wait 2s<br/>Attempt 2: wait 4s<br/>Attempt 3: wait 8s
        WR->>CF: Retry failed DNS record creation
        CF-->>WR: Success on retry
    else CF returns 409 (record exists)
        WR->>WR: Idempotent ── skip, treat as success<br/>(record already provisioned from prior attempt)
    else All 3 retries exhausted
        WR->>RS: XADD stream:dns-result<br/>{tenantId, status: FAILED,<br/>error: "CF API unreachable after 3 retries",<br/>failedRecords: [...], succeededRecords: [...]}
        RS-->>BE: DNS failure event
        BE->>DB: UPDATE tenant<br/>steps.dns = "failed", lastError: "..."
        BE-->>UI: SSE push: {step: dns, status: failed, error: "..."}
        UI->>Admin: DNS step turns red<br/>Show "Retry" button
        Admin->>UI: Clicks "Retry DNS"
        UI->>BE: POST /api/admin/tenants/{id}/retry?step=dns
        Note over BE,CF: Restarts from Phase 4<br/>Only retries failed records (idempotent)
    end
    end

    WR->>RS: XADD stream:dns-result<br/>{tenantId, status: SUCCESS,<br/>records: [prod, staging, dev]}
    RS-->>BE: DNS success event

    BE->>DB: UPDATE tenant<br/>steps.dns = "completed"<br/>dnsRecords: [{env, recordId, fqdn}]
    BE-->>UI: SSE push: {step: dns, status: completed}
    UI->>Admin: DNS step turns green
    end

    %% ──────────────────────────────────────────────
    rect rgb(245, 240, 255)
    Note over Admin,ARGO: PHASE 5 ── GitHub Action: Ingress Generation
    %% ──────────────────────────────────────────────

    BE->>DB: UPDATE tenant.steps.ingress = "in_progress"
    BE-->>UI: SSE push: {step: ingress, status: in_progress}

    BE->>GH: POST /repos/enclaive/enclaive-deployment/actions/workflows/tenant-ingress.yml/dispatches<br/>{ref: "main", inputs: {<br/>  subdomain: "{tenant}",<br/>  environments: "prod,staging,dev",<br/>  ingressClass: "nginx",<br/>  tlsSecret: "{tenant}-tls",<br/>  certIssuer: "letsencrypt-prod"<br/>}}

    GH->>GH: Checkout enclaive-deployment
    GH->>GH: Create branch: tenant/{tenant}-ingress

    GH->>GH: Modify Helm values.yaml for each env<br/>- 203-emcp-prod/backend/values.yaml<br/>- 202-emcp-staging/backend/values.yaml<br/>- 201-emcp-dev/backend/values.yaml<br/>Add ingress entries:<br/>  hosts:<br/>    - host: {tenant}.enclaive.cloud<br/>      paths: [/]<br/>    - host: admin.{tenant}.enclaive.cloud<br/>      paths: [/]<br/>    - host: console.{tenant}.enclaive.cloud<br/>      paths: [/]<br/>  tls:<br/>    - secretName: {tenant}-tls<br/>      hosts: [*.{tenant}.enclaive.cloud]

    GH->>GH: Commit + push branch
    GH->>GH: Open PR: "feat: add ingress for tenant {tenant}"

    GH->>GH: Auto-approve workflow triggers<br/>- Validate: only ingress fields changed<br/>- Validate: no unexpected file modifications<br/>- If validation passes: approve + merge

    alt Auto-approve validation fails
        GH-->>BE: Webhook: PR flagged for manual review
        BE->>DB: UPDATE tenant<br/>steps.ingress = "needs_review"
        BE-->>UI: SSE push: {step: ingress, status: needs_review,<br/>message: "PR needs manual approval"}
        UI->>Admin: Show warning + link to PR
    else Validation passes
        GH->>GH: Squash merge PR to main
        GH->>GH: Delete branch tenant/{tenant}-ingress
        GH-->>BE: Webhook: PR merged (merge_commit_sha)
        BE->>DB: UPDATE tenant<br/>steps.ingress = "completed"<br/>ingressPR: {number, mergeCommit}
        BE-->>UI: SSE push: {step: ingress, status: completed}
        UI->>Admin: Ingress step turns green
    end
    end

    %% ──────────────────────────────────────────────
    rect rgb(240, 250, 255)
    Note over Admin,ARGO: PHASE 6 ── ArgoCD Deployment & TLS
    %% ──────────────────────────────────────────────

    BE->>DB: UPDATE tenant.steps.deployment = "in_progress"
    BE-->>UI: SSE push: {step: deployment, status: in_progress}

    ARGO->>ARGO: Detect Git change on main branch<br/>(poll interval or webhook trigger)
    ARGO->>ARGO: Diff: new ingress entries detected
    ARGO->>ARGO: Auto-sync: apply Helm chart changes<br/>to prod/staging/dev namespaces

    ARGO->>ARGO: K8s applies new Ingress resources
    Note over ARGO: cert-manager detects new Ingress<br/>with tls annotation → provisions<br/>Let's Encrypt certificates<br/>(ACME HTTP-01 challenge via Cloudflare)

    ARGO-->>BE: Webhook: sync status = Synced<br/>(or BE polls ArgoCD API for sync status)

    %% ── HEALTH CHECK WITH RETRY ──
    rect rgb(230, 255, 230)
    Note over BE,ARGO: HEALTH CHECK POLLING (with retry + backoff)

    BE->>BE: Wait 30s for cert-manager + DNS propagation

    loop Poll every 15s ── max 20 attempts (5 min timeout)
        BE->>BE: GET https://{tenant}.enclaive.cloud/health
        BE->>BE: GET https://admin.{tenant}.enclaive.cloud/health
        BE->>BE: GET https://console.{tenant}.enclaive.cloud/health

        alt All 3 endpoints return 200
            BE->>BE: Health check PASSED
        else Any endpoint not ready
            BE->>BE: Increment attempt counter<br/>Log: "Attempt {n}/20 - waiting for endpoints"
            BE-->>UI: SSE push: {step: deployment,<br/>status: in_progress, attempt: n,<br/>detail: "Waiting for endpoints..."}
        end
    end

    alt Health check passed within timeout
        BE->>DB: UPDATE tenant<br/>steps.deployment = "completed"
        BE-->>UI: SSE push: {step: deployment, status: completed}
        UI->>Admin: Deployment step turns green
    else Timeout after 5 min
        BE->>DB: UPDATE tenant<br/>steps.deployment = "timeout"<br/>lastError: "Health check timeout after 20 attempts"
        BE-->>UI: SSE push: {step: deployment, status: timeout}
        UI->>Admin: Deployment step shows warning<br/>"Infrastructure deployed but endpoints not responding"<br/>Show "Retry Health Check" button
    end
    end
    end

    %% ──────────────────────────────────────────────
    rect rgb(240, 255, 245)
    Note over Admin,ARGO: PHASE 7 ── Finalization & Activation
    %% ──────────────────────────────────────────────

    BE->>DB: Create payment/billing settings<br/>{tenantId, plan, billingCycle: monthly,<br/>stripeCustomerId: null (created on first payment)}
    DB-->>BE: Billing record created

    BE->>DB: Sync default theme settings<br/>{tenantId, logo: default, primaryColor: "#1976d2",<br/>favicon: default, loginBackground: default}
    DB-->>BE: Theme synced

    BE->>DB: Sync default tenant settings<br/>{tenantId, features: [...defaultFeatures],<br/>limits: {maxUsers: planLimit, maxProjects: planLimit},<br/>notifications: {email: true, slack: false}}
    DB-->>BE: Settings synced

    BE->>RS: XADD stream:tenant-events<br/>{type: TENANT_CREATED, tenantId, subdomain,<br/>timestamp, createdBy: adminId}
    Note over RS: finops-service consumes this<br/>for billing initialization +<br/>welcome email via SendGrid

    BE->>DB: UPDATE tenant<br/>status: ACTIVE<br/>activatedAt: now()

    BE->>DB: INSERT audit log<br/>{action: TENANT_ACTIVATED, actor: system,<br/>timestamp, previousState: PROVISIONING,<br/>newState: ACTIVE, duration: totalMs}

    BE-->>UI: SSE push: {status: ACTIVE,<br/>urls: {<br/>  console: "https://console.{tenant}.enclaive.cloud",<br/>  admin: "https://admin.{tenant}.enclaive.cloud",<br/>  api: "https://{tenant}.enclaive.cloud/api"<br/>},<br/>credentials: {<br/>  adminEmail: "...",<br/>  tempPassword: "sent via email"<br/>}}

    UI->>Admin: All steps green!<br/>Show tenant URLs + "Copy credentials" button
    end
```

***

### State Machine: Tenant Provisioning Lifecycle

```mermaid
stateDiagram-v2
    [*] --> VALIDATING: Admin submits form

    VALIDATING --> PROVISIONING: Validation OK ── record created in DB
    VALIDATING --> REJECTED: Subdomain taken / invalid input / not authorized

    state PROVISIONING {
        [*] --> KEYCLOAK_IN_PROGRESS

        KEYCLOAK_IN_PROGRESS --> KEYCLOAK_COMPLETED: Realm + user + roles + clients done
        KEYCLOAK_IN_PROGRESS --> STEP_FAILED: Keycloak API error

        KEYCLOAK_COMPLETED --> VAULT_IN_PROGRESS

        VAULT_IN_PROGRESS --> VAULT_COMPLETED: Namespace + secrets + policies done
        VAULT_IN_PROGRESS --> STEP_FAILED: Vault API error

        VAULT_COMPLETED --> DNS_IN_PROGRESS

        DNS_IN_PROGRESS --> DNS_COMPLETED: All CNAME records created
        DNS_IN_PROGRESS --> DNS_RETRYING: CF 429/5xx ── exponential backoff
        DNS_RETRYING --> DNS_IN_PROGRESS: Retry attempt
        DNS_RETRYING --> STEP_FAILED: Max retries (3) exhausted
        DNS_IN_PROGRESS --> STEP_FAILED: CF error (non-retryable)

        DNS_COMPLETED --> INGRESS_IN_PROGRESS

        INGRESS_IN_PROGRESS --> INGRESS_COMPLETED: PR auto-approved + merged
        INGRESS_IN_PROGRESS --> INGRESS_NEEDS_REVIEW: PR flagged (unexpected changes)
        INGRESS_NEEDS_REVIEW --> INGRESS_COMPLETED: Manual approval + merge
        INGRESS_IN_PROGRESS --> STEP_FAILED: GitHub API error

        INGRESS_COMPLETED --> DEPLOYMENT_IN_PROGRESS

        DEPLOYMENT_IN_PROGRESS --> HEALTH_CHECKING: ArgoCD sync completed
        DEPLOYMENT_IN_PROGRESS --> STEP_FAILED: ArgoCD sync failed

        HEALTH_CHECKING --> DEPLOYMENT_COMPLETED: All endpoints 200 OK
        HEALTH_CHECKING --> DEPLOYMENT_TIMEOUT: 5 min timeout exceeded
        DEPLOYMENT_TIMEOUT --> HEALTH_CHECKING: Admin clicks retry

        DEPLOYMENT_COMPLETED --> FINALIZING

        FINALIZING --> [*]: Billing + theme + settings + events done
    }

    PROVISIONING --> ACTIVE: All steps completed

    STEP_FAILED --> PROVISIONING: Admin clicks "Retry"<br/>(resumes from failed step)
    STEP_FAILED --> CANCELLED: Admin cancels

    REJECTED --> [*]
    CANCELLED --> [*]
    ACTIVE --> [*]
```

***

### Service Responsibility Matrix

```mermaid
flowchart TB
    subgraph UI["admin-ui"]
        direction TB
        U1["Form + client validation"]
        U2["Progress tracker (SSE)"]
        U3["Retry buttons per step"]
        U4["Success: URLs + credentials"]
    end

    subgraph BE["emcp-be ── The Brain"]
        direction TB
        B1["API endpoint + auth guard"]
        B2["Subdomain validation"]
        B3["Keycloak provisioning service"]
        B4["Vault provisioning service"]
        B5["DNS dispatch (→ Redis Streams)"]
        B6["GitHub API client"]
        B7["Health check poller"]
        B8["Status manager (DB + SSE)"]
        B9["Audit logger"]
        B10["Billing/theme/settings init"]
    end

    subgraph WR["workload-runner"]
        direction TB
        W1["Redis Streams consumer"]
        W2["Cloudflare DNS module"]
        W3["Retry engine (exp. backoff)"]
    end

    subgraph EXT["External Systems"]
        direction TB
        E1["Keycloak 26"]
        E2["Vault (vhsm.vshopping.enclaive.cloud)"]
        E3["Cloudflare API"]
        E4["GitHub Actions (enclaive-deployment)"]
        E5["ArgoCD"]
        E6["cert-manager + Let's Encrypt"]
    end

    U1 --> B1
    B3 --> E1
    B4 --> E2
    B5 --> W1 --> W2 --> E3
    B6 --> E4 --> E5 --> E6
    B7 -.->|polls| E5
    B8 --> U2
```

***

### Retry Strategy Summary

| Phase       | Retry Target   | Trigger                  | Strategy                | Max Attempts | Backoff          | On Exhaustion                    |
| ----------- | -------------- | ------------------------ | ----------------------- | :----------: | ---------------- | -------------------------------- |
| **Phase 2** | Keycloak API   | Connection timeout / 5xx | Simple retry            |       2      | 3s fixed         | STEP\_FAILED ── admin notified   |
| **Phase 3** | Vault API      | Connection timeout / 5xx | Simple retry            |       2      | 3s fixed         | STEP\_FAILED ── admin notified   |
| **Phase 4** | Cloudflare API | 429 rate limit / 5xx     | **Exponential backoff** |     **3**    | **2s → 4s → 8s** | STEP\_FAILED ── admin can retry  |
| **Phase 4** | Cloudflare API | 409 conflict (exists)    | Skip (idempotent)       |       -      | -                | Treat as success                 |
| **Phase 5** | GitHub API     | 403 / 5xx                | Simple retry            |       2      | 5s fixed         | STEP\_FAILED ── manual PR needed |
| **Phase 6** | Health check   | Endpoint not ready       | **Polling**             |    **20**    | **15s interval** | TIMEOUT ── admin can retry       |
| **Any**     | Admin retry    | Button click             | Resume from failed step |   Unlimited  | -                | Admin decides                    |

***

### Missing Steps Added (vs Original List)

| Your Step        | What Was Added                                                                                                            |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Keycloak created | + OIDC client registration, admin user creation, role assignment, theme & email config, token lifespans, password policy  |
| Vault created    | + Identity engine, KV-v2 engine, K8s auth role for pod injection, access policies, agent sidecar config                   |
| Domains created  | + Redis Streams dispatch (BE → WR), per-environment DNS (prod/staging/dev), idempotent duplicate handling                 |
| DNS → Git Action | + Branch creation, Helm values.yaml modification per env, auto-approve validation, squash merge                           |
| Deployment       | + ArgoCD sync detection, cert-manager TLS provisioning, structured health check polling                                   |
| Completed        | + Billing setup, theme sync, settings sync, tenant-events stream (→ finops-service), audit log with actor tracking        |
| *(new)*          | + Client-side validation, subdomain uniqueness check, env prefix generation, SSE progress updates, retry from failed step |