Grafana

When using CEEMS LB to provide access control and loading balancing for TSDB servers, the Prometheus datasource on the Grafana must be configured slightly differently than using a regular native Prometheus server. As discussed in CEEMS LB Configuration, a path parameter corresponding to the cluster must be appended to CEEMS LB server URL.

For instance, if CEEMS API server and CEEMS LB has following configuration:

ceems_api_server:
  data:
    path: /var/lib/ceems
    update_interval: 15m

  admin:
    users:
      - adm1
  
  web:
    url: http://localhost:9020
    requests_limit: 30

clusters:
  - id: slurm-0
    manager: slurm
    updaters:
      - tsdb-0
    cli: 
      <omitted for brevity>

  - id: os-0
    manager: openstack
    updaters:
      - tsdb-1
    web: 
      <omitted for brevity>

updaters:
  - id: tsdb-0
    updater: tsdb
    web:
      url: http://tsdb-0
    extra_config:
      <omitted for brevity>
  
  - id: tsdb-1
    updater: tsdb
    web:
      url: http://tsdb-1
    extra_config:
      <omitted for brevity>

ceems_lb:
  strategy: resource-based
  backends:
    - id: slurm-0
      tsdb_urls: 
        - http://tsdb-0
        - http://tsdb-0-replica

    - id: os-0
      tsdb_urls: 
        - http://tsdb-1
        - http://tsdb-1-replica

it is clear that there are two different clusters, slurm-0 and os-0 and each cluster has its own TSDB server tsdb-0 and tsdb-1, respectively. In Grafana, a Prometheus datasource for each cluster must be configured to present the metrics of each cluster separately. Thus, the following provisioning config can be used to configure datasources of each cluster

datasources:
  - name: SLURM-TSDB
    type: prometheus
    access: proxy
    url: http://ceems-lb:9030
    basicAuth: true
    basicAuthUser: <ceems_lb_basic_auth_user>
    # Notice we are setting custom header X-Ceems-Cluster-Id to `slurm-0`.
    # IT IS IMPORTANT TO HAVE IT
    jsonData:
      httpHeaderName1: X-Ceems-Cluster-Id
    secureJsonData:
      basicAuthPassword: <ceems_lb_basic_auth_password>
      httpHeaderValue1: slurm-0

  - name: OS-TSDB
    type: prometheus
    access: proxy
    url: http://ceems-lb:9030
    basicAuth: true
    basicAuthUser: <ceems_lb_basic_auth_user>
    # Notice we are setting custom header X-Ceems-Cluster-Id to `slurm-0`.
    # IT IS IMPORTANT TO HAVE IT
    jsonData:
      httpHeaderName1: X-Ceems-Cluster-Id
    secureJsonData:
      basicAuthPassword: <ceems_lb_basic_auth_password>
      httpHeaderValue1: os-0

Internally, CEEMS LB will check the header X-Ceems-Cluster-Id and forwards the request to the correct backends group based on the provided cluster ID. This ensures that we can use a single instance of CEEMS LB to load balance across multiple clusters.

IMPORTANT

Even if there is only one cluster and one TSDB instance for that cluster, we need to configure the datasource on Grafana with custom header as explained above if we wish to use CEEMS LB. This is the only way for the CEEMS LB to know which cluster to target.