Nov 25, 2020

Running Percona Monitoring & Management 2 (PMM2) Server behind an Apache HTTP/2 reverse proxy

mod_proxy_http2 is considered experimental at the time of writing. Follow this guide at your own risk

EDIT (10th July 2022):

As of Debian 11 (I'm not sure if this was due to a newer Apache or a newer Nginx) this stopped working again because Apache is sending multiple Host headers to the proxy backend. The fix is to add RequestHeader unset Host and a2enmod headers. The Apache config below has been updated.

Percona provide easy to follow instructions for installing PMM - their open source MySQL and MongoDB monitoring platform - on your own server.

If you follow the above instructions, PMM will be installed in Docker with ports 80 and 443 mapped directly. This works fine, but if you want to run multiple virtual hosts on one IP address using the same ports, add extra authentication, or have PMM respond on a different subdirectory you may wish to place Apache in front and have it reverse proxy to PMM.

This isn't quite as trivial as it sounds and some of the errors I received were either empty, or reporting the wrong error entirely.

I started by binding the docker image to localhost instead of 0.0.0.0 and only exposing port 443:

docker create -v /srv --name pmm-data percona/pmm-server:2 /bin/true
docker run -d -p 127.0.0.1:8443:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2

I opted to leave the default SSL certificates in there as the proxy can be configured to not check them, and the connection was running locally anyway. Additionally, I used LetsEncrypt (certbot) to generate the SSL certificates and having them available inside the container would have required either post-hooks or mounting all of /var/lib/acme to allow symlinks to function.

Next I configured Apache to sit in front and reverse proxy port 443 to the container. This worked to register nodes, but left them "unconfigured":

# pmm-admin config --server-url=https://admin:password@testsite.silvermou.se:443
Checking local pmm-agent status...
pmm-agent is running.
Registering pmm-agent on PMM Server...
Registered.
Configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml updated.
Reloading pmm-agent configuration...
Configuration reloaded.
Checking local pmm-agent status...
pmm-agent is running.

# pmm-admin add mysql --username pmm --password testing
Failed to get PMM Server parameters from local pmm-agent: pmm-agent is running, but not set up.

# pmm-admin config --server-url=https://admin:password@testsite.silvermou.se:443
Checking local pmm-agent status...
pmm-agent is running.
Registering pmm-agent on PMM Server...
Failed to register pmm-agent on PMM Server: Node with name "server_hostname" already exists..

The only hint of the actual error was in the Apache access logs:

x.x.x.x - - [24/Nov/2020:12:24:47 +0100] "POST /v1/management/Node/Register HTTP/1.1" 200 4484 "-" "Go-http-client/1.1"
x.x.x.x - - [24/Nov/2020:12:24:48 +0100] "PRI * HTTP/2.0" 400 3737 "-" "-"

Per the RFC the PRI method:

is never used by an actual client. This method will appear to be used when an HTTP/1.1 server or intermediary attempts to parse an HTTP/2 connection preface.

So it seems that the PMM client requires you to configure a TLS connection to the server, which is possible over HTTP/1.1, but then starts polling the server over HTTP/2. If the server does not respond over HTTP/2 it will not be correctly configured - annoyingly this also means that you cannot remove the client once it has been added.

The solution is to add HTTP/2 support into Apache. If you're running a recent distribution (eg. Debian 9 or later) it should be available via the http2 module. If not, you'll need to either upgrade or recompile Apache.

# a2enmod http2
Enabling module http2.
To activate the new configuration, you need to run:
  systemctl restart apache2

Then you'll need to add a Protocols line in the VirtualHost definition specifying the available HTTP protocols. The order is important (unless specifying ProtocolsHonorOrder Off):

Protocols h2 h2c http/1.1

If you're upgrading from an older version of Debian (or still running mpm_prefork for another reason) Apache will restart correctly but HTTP/2 will not work. You'll need to replace mpm_prefork with mpm_event, and mod_php with php-fpm. Example instructions below, but test before using in production:

root@silvermouse-testrig1 ~ # a2dismod php5
Module php5 disabled.
root@silvermouse-testrig1 ~ # a2dismod mpm_prefork
Module mpm_prefork disabled.
root@silvermouse-testrig1 ~ # a2enmod mpm_event
Considering conflict mpm_worker for mpm_event:
Considering conflict mpm_prefork for mpm_event:
Enabling module mpm_event.
root@silvermouse-testrig1 ~ # a2enmod proxy_fcgi
Considering dependency proxy for proxy_fcgi:
Module proxy already enabled
Enabling module proxy_fcgi.
root@silvermouse-testrig1 ~ # apt-get install php7.3-fpm
root@silvermouse-testrig1 ~ # a2enconf php7.3-fpm
Enabling conf php7.3-fpm.
root@silvermouse-testrig1 ~ # systemctl restart apache2

Testing HTTP/2 with curl, the first line should show HTTP/2:

root@silvermouse-testrig1 ~ # curl -I --http2 https://testsite.silvermou.se
HTTP/2 200
date: Wed, 25 Nov 2020 08:56:37 GMT
server: Apache/2.4.38 (Debian)
last-modified: Fri, 17 Jul 2020 16:40:41 GMT
etag: "29cd-5aaa5d1484fb1"
accept-ranges: bytes
content-length: 10701
vary: Accept-Encoding
content-type: text/html

Next make sure to also proxy the requests to the Docker container over HTTP/2 (this is where it gets experimental) otherwise you'll see the following in the Apache error logs:

[Wed Nov 25 10:00:04.514626 2020] [proxy_http:error] [pid 30594:tid 140482123885840] (103)Software caused connection abort: [client x.x.x.x:49152] AH01095: prefetch request body failed to 127.0.0.1:8443 (127.0.0.1) from x.x.x.x ()

To proxy requests over HTTP/2 you need to ProxyPass to h2://127.0.0.1:8443/ instead of to https://127.0.0.1:8443, and enable mod_proxy_http2:

# a2enmod proxy_http2
# systemctl restart apache2

The clients should now connect correctly:

client:~# pmm-admin config --server-url=https://pmm:pmm@testsite.silvermou.se:443 --force
Checking local pmm-agent status...
pmm-agent is running.
Registering pmm-agent on PMM Server...
Registered.
Configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml updated.
Reloading pmm-agent configuration...
Configuration reloaded.
Checking local pmm-agent status...
pmm-agent is running.
client:~# pmm-admin add mysql --username pmm --password testing
MySQL Service added.
Service ID  : /service_id/9975a27b-3e4a-4cc3-bd40-bd134292dfcd
Service name: server_hostname

The full Apache config used in this example is as follows:

<VirtualHost *:443>
        Protocols h2 h2c http/1.1
        ServerName testsite.silvermou.se
        SSLProxyEngine On
     	ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
        ProxyRequests On
        ProxyPreserveHost On
        SSLProxyCheckPeerCN off
        SSLProxyCheckPeerName off
        SSLProxyVerify none
        RequestHeader unset Host
        
        ProxyPass / h2://127.0.0.1:8443/
        ProxyPassReverse / https://127.0.0.1:8443/
        
        SSLCertificateFile /etc/letsencrypt/live/testsite.silvermou.se/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/testsite.silvermou.se/privkey.pem
</VirtualHost>