Skip to content

AI Gateway Rate Limiting

The Studio platform implements a three-tiered rate limiting architecture at the API Gateway (Kong) level to ensure system stability, manage AI costs, and provide fair resource allocation across different tenants and users.

Architecture Overview

Rate limiting is enforced at three distinct levels for all AI-related endpoints:

  1. Global Level: Enforced across the entire service to prevent infrastructure overload.
  2. Tenant Level: Enforced per company/tenant to ensure that one tenant's usage doesn't impact others.
  3. User Level: Enforced per individual user to prevent accidental or malicious runaway scripts.

Rate Limit Tiers

Tier Scope Identifier Default Limit Purpose
Global Service-wide Service ID 1000 req/min Infrastructure protection
Tenant Per Tenant X-Tenant-Id 100 req/min B2B Resource isolation
User Per User X-User-Id 10 req/min Personal usage safety

[!NOTE] These limits apply specifically to high-cost AI operations such as Chat and Analysis. Standard API endpoints may have different or no limits.

How it Works

When a request is made to an AI endpoint:

  1. Authentication: The gateway verifies the user's JWT.
  2. Identity Extraction: The OPA plugin extracts the userId and tenantId from the JWT claims and injects them as internal headers (X-User-Id, X-Tenant-Id).
  3. Tiered Evaluation:
    • Kong checks the Global limit for the service.
    • Kong checks the Tenant limit based on the X-Tenant-Id header.
    • Kong checks the User limit based on the X-User-Id header.
  4. Enforcement: If any tier's limit is exceeded, the request is rejected immediately.

Error Handling

When a rate limit is exceeded, the gateway returns a structured 429 Too Many Requests response.

Example Response

{
  "error": "User rate limit exceeded",
  "code": "USER_LIMIT_REACHED"
}

Headers

Every rate-limited response includes a Retry-After header indicating how many seconds to wait before retrying.

Future Customization

Currently, rate limits are globally defined for all tenants. Future updates will allow:

  • Variable Tiering: Assigning different limits to different tenants based on their subscription plan.
  • Dynamic Adjustments: Automatically increasing limits during peak audit periods.
  • Dashboard Visibility: Real-time monitoring of rate limit usage per tenant in the Admin UI.