Fixing reverse proxy 'origin not allowed' from Percona PMM/Grafana in Kubernetes

This can be a tricky one to debug. The 'origin not allowed' only comes from one place in Grafana:

hostnameMatches := origin == netAddr.Host
if netAddr.Host == "" || !trustedOrigin && !hostnameMatches {
	return &errorWithStatus{Underlying: errors.New("origin not allowed"), HTTPStatus: http.StatusForbidden}
}

Normally you'd either modify the header or add the proxy domain to csrf_trusted_origins in /etc/grafana/grafana.ini:

[security]
csrf_trusted_origins = pmm.reverse-proxy-domain.network

Modifying the header isn't always possible, and the grafana.ini cannot be directly modified in the Helm chart.

The fix is to override the setting using a GF_$SECTION_$VALUE environment variable in pmmEnv, so for the above example:

    pmmEnv:
      GF_SECURITY_CSRF_TRUSTED_ORIGINS: pmm.reverse-proxy-domain.network

There are a couple of gotchas though:

• This variable MUST be without protocol. Do not put http:// or https://
• This variable will override grafana.ini, so if it's not working double check your StatefulSet and helmrelease to make sure you don't have competing values.