Complete Architecture Guide¶

This page is the practical source of truth for how Studio is constructed. It describes the running Docker Compose architecture, the service boundaries, the request paths, the data stores, and the main background processing flows.

Architecture Summary¶

Studio is a Docker Compose based microservices platform for compliance, audit, evidence, endpoint, cloud posture, risk, reporting, and AI-assisted workflows.

The platform is organized into these layers:

High-Level System Diagram¶

graph TD
    User["User Browser"] --> Kong["Kong API Gateway"]
    Kong --> Frontend["Next.js Frontend"]
    Kong --> Backend["Backend API"]
    Kong --> Viewer["Secure Evidence Viewer"]
    Kong --> Kratos["Ory Kratos"]
    Kong --> OPA["Open Policy Agent"]

    Frontend --> Backend
    Frontend --> Kratos

    Backend --> Postgres["PostgreSQL + pgvector"]
    Backend --> Redis["Redis / BullMQ"]
    Backend --> MinIO["MinIO Evidence Storage"]
    Backend --> Neo4j["Neo4j Graph DB"]
    Backend --> GraphRetriever["Graph Retriever"]
    Backend --> VectorStore["Vector Store"]
    Backend --> FleetService["Fleet Service"]

    AIService["AI Service"] --> Backend
    AIService --> VectorStore
    AIService --> DocumentRAG["Document RAG Service"]
    AIService --> Phoenix["Phoenix AI Observability"]
    AIService --> LLMs["Gemini / Moonshot / configured LLMs"]

    DocumentRAG --> Postgres
    VectorStore --> VectorData["Persistent Vector Data"]
    GraphRetriever --> Neo4j

    AnalysisWorker["Analysis Worker"] --> Redis
    AnalysisWorker --> Postgres
    AnalysisWorker --> Neo4j

    ChunkingWorker["Chunking Worker"] --> Postgres
    ChunkingWorker --> MinIO

    URLWorker["URL Downloader Worker"] --> Postgres
    URLWorker --> MinIO

    FleetService --> Fleet["FleetDM"]
    Fleet --> FleetDB["Fleet MySQL"]
    Fleet --> Redis

    ProwlerAPI["Prowler API"] --> Postgres
    ProwlerWorker["Prowler Worker"] --> Postgres
    ProwlerBeat["Prowler Beat"] --> ProwlerWorker

    Backend --> FluentBit["Fluent Bit"]
    Frontend --> FluentBit
    FluentBit --> Loki["Loki"]
    Prometheus["Prometheus"] --> Backend
    Prometheus --> Kong
    Prometheus --> PostgresExporter["Postgres Exporter"]
    Prometheus --> RedisExporter["Redis Exporter"]
    Grafana["Grafana"] --> Prometheus
    Grafana --> Loki
    Tempo["Tempo"] --> Grafana

Request Flow¶

All normal browser traffic enters through Kong on ports 80 and 443.

sequenceDiagram
    participant U as Browser
    participant K as Kong
    participant O as OPA
    participant F as Frontend
    participant B as Backend
    participant DB as PostgreSQL

    U->>K: HTTPS request
    K->>O: Optional authorization decision
    O-->>K: Allow or deny
    K->>F: Route page request
    F->>B: Internal API call or browser API call through /api
    B->>DB: Read or write business data
    DB-->>B: Result
    B-->>F: JSON response
    F-->>K: Rendered UI or API response
    K-->>U: Response

Public Edge Responsibilities¶

Kong is the public entry point. It handles:

• Routing to frontend, backend, viewer, auth, and service endpoints.
• TLS termination using certificates mounted from certs/.
• Custom OPA authorization plugin.
• JWT support.
• CORS.
• Prometheus metrics.
• Dynamic rate limiting through the custom dynamic-ratelimit plugin.

OPA evaluates policy decisions from Rego policies stored in gateway/opa/policies.

Ory Kratos owns identity flows such as login, registration, account recovery, and session validation. Kratos stores identity data in the kratos PostgreSQL database.

Service Inventory¶

Repository Structure¶

Backend Construction¶

The backend is a Node.js and Express service written in TypeScript. It is the main business API.

Important backend areas:

The backend exposes domain routes for:

• Authentication and token handling.
• Admin metrics, users, frameworks, controls, questionnaires, AI compliance settings, and limits.
• Customer dashboards, project requests, evidence upload, framework controls, and issue reporting.
• Auditor dashboards, evidence requests, observations, analysis, review, control status, meetings, and project activity.
• Manager dashboards, project approval, assignment, auditor performance, customer profiles, and team management.
• Compliance overview, projection, summary, dashboard, and project details.
• Evidence records, evidence vault, annotations, linking, refresh, AI analysis, and batch analysis.
• Findings, comments, resolution, and statistics.
• Endpoint assets, agent downloads, scans, pings, and framework scan processing.
• CASB integrations and sync.
• Learning policies, training, assignments, and review status.
• Chat contacts, conversations, messages, push subscriptions.
• Tasks, tags, notifications, questionnaires, threats, threat intelligence, profile, share links, uploads, health, and system checks.

Frontend Construction¶

The frontend is a Next.js app using the App Router, TypeScript, Tailwind CSS, Radix UI primitives, and feature components.

Main authenticated screens include:

The frontend uses NEXT_PUBLIC_API_URL=/api so browser API traffic goes through Kong. Server-side internal calls use BACKEND_URL=http://studio-backend:4000.

AI Architecture¶

Studio's AI architecture is split into orchestration, retrieval, and storage services.

graph TD
    ChatUI["AI Chat UI"] --> Kong
    Kong --> AIService["AI Service"]
    AIService --> LangGraph["LangGraph Workflows"]
    AIService --> MCP["MCP Server and Tools"]
    MCP --> Backend["Backend APIs"]
    MCP --> FleetService["Fleet Service"]
    AIService --> DocumentRAG["Document RAG"]
    AIService --> VectorStore["Vector Store"]
    AIService --> LLM["Configured LLM Provider"]
    DocumentRAG --> Postgres
    VectorStore --> Embeddings["Embedding Index"]
    AIService --> Phoenix["Phoenix Tracing"]

The AI service is responsible for:

• Chat orchestration.
• LangGraph workflows.
• MCP tool execution.
• Policy generation and refinement.
• Evidence and compliance analysis.
• Context retrieval from backend APIs.
• Long-term memory and project memory workflows.
• LLM provider routing through configured API keys and gateway settings.
• AI telemetry export to Phoenix and OpenTelemetry collectors.

The document RAG service handles semantic retrieval and hybrid search. The vector store persists embeddings. The graph retriever answers graph relationship queries from Neo4j.

Evidence and RAG Flow¶

sequenceDiagram
    participant U as User
    participant F as Frontend
    participant B as Backend
    participant M as MinIO
    participant Q as Redis Queue
    participant C as Chunking Worker
    participant V as Vector Store
    participant G as Neo4j

    U->>F: Upload evidence
    F->>B: Create evidence record
    B->>M: Store object
    B->>Q: Enqueue analysis/chunking job
    C->>M: Read document
    C->>B: Update chunking status
    C->>V: Store embeddings
    B->>G: Link evidence/control relationships
    U->>F: Ask AI or review control
    F->>B: Request context
    B->>V: Similarity search when needed
    B->>G: Relationship search when needed

Evidence metadata lives in PostgreSQL. Evidence files live in MinIO. Evidence-to-control relationships are mirrored into Neo4j for graph search. Semantic chunks live in the vector store for retrieval augmented generation.

Data Architecture¶

Queue and Worker Architecture¶

Redis backs background jobs through BullMQ and related worker processes.

Worker responsibilities:

• analysis-worker: runs backend processors for analysis, graph updates, Prowler polling, deadline checks, AI document processing, and scheduled compliance jobs.
• chunking-worker: extracts chunks from evidence files and prepares them for retrieval.
• url-downloader-worker: downloads external URL evidence and writes files into MinIO.
• prowler-worker: executes Prowler scan workloads.
• prowler-worker-beat: schedules recurring Prowler jobs.

Background workers keep user-facing API calls fast by moving expensive document, AI, graph, and scan operations out of the request path.

Security Architecture¶

Studio applies security at several layers.

Role Model¶

The application uses role-aware routes and dashboards:

• admin: system configuration, users, frameworks, AI compliance, limits.
• manager: project approval, assignment, team oversight.
• auditor: evidence review, observations, analysis, control status.
• customer: evidence upload, project requests, compliance work.
• reviewer: assigned review workflows.

Authorization is enforced through a mix of Kong/OPA decisions, backend authentication middleware, backend role middleware, and service-level ownership checks.

Endpoint and Fleet Architecture¶

Studio integrates FleetDM for endpoint and osquery management.

graph TD
    Endpoint["Endpoint / osquery"] --> Fleet["FleetDM Server"]
    Fleet --> FleetDB["Fleet MySQL"]
    Backend["Backend API"] --> FleetService["Studio Fleet Service"]
    FleetService --> Fleet
    FleetService --> Postgres["Studio PostgreSQL"]
    FleetService --> MinIO["MinIO"]
    Backend --> Findings["Findings and Risk"]

The backend exposes endpoint routes for asset listing, manual assets, agent downloads, install scripts, pings, scans, and framework scans. The Fleet service isolates Fleet-specific integration logic and writes normalized results back to Studio's main data model.

Cloud Posture and CASB Architecture¶

Prowler provides cloud security posture scanning. CASB routes provide SaaS or cloud application integration management.

graph TD
    Admin["Admin/User"] --> Frontend["CASB / Risk UI"]
    Frontend --> Backend["Backend CASB and Risk Routes"]
    Backend --> ProwlerAPI["Prowler API"]
    ProwlerAPI --> ProwlerWorker["Prowler Worker"]
    ProwlerWorker --> Cloud["Cloud Accounts"]
    ProwlerWorker --> ProwlerDB["Prowler PostgreSQL DB"]
    Backend --> Findings["Findings"]
    Findings --> Risk["Risk Dashboard"]

Cloud findings are normalized into findings and risk views so they can be linked to projects, controls, and remediation workflows.

Observability Architecture¶

Studio includes a full observability stack.

Application services expose or forward telemetry using environment variables such as FLUENT_BIT_URL, LOKI_URL, TEMPO_URL, OTEL_SERVICE_NAME, and PHOENIX_COLLECTOR_ENDPOINT.

Deployment Model¶

Local and Compose Deployment¶

Primary local command:

docker-compose up -d --build

Root package scripts wrap common compose operations:

npm run start
npm run dev
npm run start:fast
npm run stop
npm run logs
npm run clean

Exposed Ports¶

Most internal services are intentionally not exposed directly. Access should go through Kong or internal Docker networking.

Environment Configuration¶

The platform is configured through the root .env file.

Important variables include:

Main Functional Flows¶

Login Flow¶

sequenceDiagram
    participant U as User
    participant K as Kong
    participant F as Frontend
    participant R as Kratos
    participant B as Backend

    U->>K: Open login page
    K->>F: Serve frontend
    F->>R: Start Kratos login flow
    R-->>F: Flow data
    U->>F: Submit credentials
    F->>R: Complete login
    R-->>F: Session
    F->>B: Load current user
    B->>R: Validate session when needed
    B-->>F: Local role/profile context

Project and Evidence Flow¶

sequenceDiagram
    participant C as Customer
    participant B as Backend
    participant M as Manager
    participant A as Auditor
    participant S as Storage
    participant W as Worker

    C->>B: Create project request
    M->>B: Approve and assign project
    A->>B: Review framework controls
    C->>B: Upload evidence
    B->>S: Store evidence file
    B->>W: Queue analysis and graph work
    A->>B: Add observation and review
    B-->>C: Updated compliance status

Report Flow¶

graph TD
    Project["Project Data"] --> Report["Report Builder"]
    Controls["Control Status"] --> Report
    Evidence["Linked Evidence"] --> Report
    Findings["Findings and Risk"] --> Report
    AI["AI Summaries"] --> Report
    Report --> Export["PDF / Export / Share"]

Reports combine control status, evidence, findings, risk context, and AI-generated analysis. The user should review AI summaries against source evidence before exporting.

Reliability and Scaling Notes¶

The architecture is designed so high-cost work can scale separately:

• Scale frontend horizontally for UI traffic.
• Scale backend horizontally for API traffic.
• Scale analysis-worker for queue throughput.
• Scale chunking-worker for document ingestion throughput.
• Scale ai-service, document-rag-service, and vector-store for AI workloads.
• Scale Prowler workers independently for cloud scanning.
• Keep PostgreSQL, MinIO, Redis, Neo4j, Fleet MySQL, and vector storage on persistent volumes.

For production deployments, place public access behind Kong, restrict database and observability ports, rotate all secrets, configure real TLS certificates, monitor queues, and back up all persistent volumes.

What To Read Next¶

• Data Flow for detailed data movement.
• Security Model for auth, authorization, and evidence isolation.
• Microservices for service boundary patterns.
• Deployment for environment and release guidance.
• Monitoring for operational dashboards.