Archive

Fixing reverse proxy 'origin not allowed' from Percona PMM/Grafana in Kubernetes

This can be a tricky one to debug. The 'origin not allowed' only comes from one place in Grafana: hostnameMatches := origin == netAddr.Host if netAddr.Host == "" || !trustedOrigin && !hostnameMatches { return &errorWithStatus{Underlying: errors.New("origin not allowed"), HTTPStatus: http.StatusForbidden} } Normally you&

How to GeoIP block certain countries in Nginx with MaxMind

I spent too long thinking that ModSecurity was necessary for this, but you can do it with just the ngx_http_geoip2_module Install nginx along with the module: apt-get install nginx libnginx-mod-http-geoip2 Download the free GeoLite mmdb database from MaxMind into /etc/nginx/GeoLite2-Country.mmdb (you can and probably

Why ProFTPD isn't loading your config (for example PassivePorts)

Recently I had an issue where I migrated ProFTPD from one server to another, but the PassivePorts was being ignored (as was AllowOverwrite). Connections were timing out and tcpdump showed the client was connecting to the wrong port range for passive mode. Nothing in the logs, but dumping the config

Using ProxySQL caching to reduce MySQL query latency

In a previous post I discussed an issue where 1ms of increased latency to a MySQL server was causing a problem due to thousands of queries. Generally, there are only two ways to handle this problem: 1. Bring the servers closer together (according to the speed of light); 2. Reduce

Be aware of latency in distributed systems

I want to begin by saying that this specific example is very niche problem, but real. A client moved their LAMP application to a new hosting provider and noticed that simple operations were taking several seconds longer, yet profiled queries were quicker if anything. I setup a test database server

Setting up multiple MySQL slaves on one server using Docker Compose

This article is going to go through the steps to setup multiple MySQL slaves on one server using Docker Compose. These slaves will be used only for backup purposes, and MySQL slaves work by pulling the binary logs from the master, so no ports need to be exposed to the

How to install Wordpress with SSL on a new Ubuntu server

There are probably thousands of guides on this, but I was looking for the basic steps and couldn't find anything I would be happy sending someone, so here's my take on it. Aside from the operating system itself, three parts are needed to serve a Wordpress

How to find Azure IP ranges automatically and add them to IPTables

The IP ranges used by Azure Public Cloud are updated weekly, and the URL to download the JSON file containing them changes too. I found a couple of scripts online to automate this but they mostly relied on updating the URL manually every week, which I did not want to

Is the Web becoming a walled garden?

A walled garden, in the context of the Internet and freedom of data, is an environment where access to data within the ‘garden’ is controlled and can only be accessed via approved apps or websites. There are many well-known examples of these, such as app stores, social media platforms, and

The state of the Internet as of January 2022

In a previous post, I wrote about working with The Internet Society to rewrite an IPv6 crawler. In this post, I wanted to share some of the results I found interesting from the most recent crawl of the top million websites (27th January 2022). Website Records This crawl found 1,

Monitoring the state of IPv6 deployment with The Internet Society

Last year I was approached by The Internet Society to help with an IPv6 crawler they'd had running since 2010. They're keen for other people to run their own version of this crawler, so if you're interested please see instructions at the bottom of

LetsEncrypt (60) SSL certificate problem: certificate has expired

If you're using LetsEncrypt for SSL certificates and have been paying attention, you'll know that one of their root CA certificates (DST Root CA X3) expired on 30/09/2021 and that older devices (really old devices) will have stopped working. What might not be so

Dynamic DNS using a HTTP API Gateway and Lambda with Terraform and aws_apigatewayv2_api

Serverless is one of those technologies that sounds really simple, but can be quite tricky to actually implement. There are so many different parts and sometimes the only error you'll see if you get something wrong is either a 404 or a ValidationException. I also found that a

Manually generating Virtualmin configuration for Apache

A client came to me with what was initially a Virtualmin backup problem: Copying Apache aliases .. .. failed to find target virtual website! The virtual host existed within /etc/httpd/conf/httpd.conf but there were errors: # apachectl configtest httpd: Syntax error on line 394 of /etc/httpd/conf/httpd.conf:

Package varnishapi was not found in the pkg-config search path (libvmod-curl)

Quick post about this error you might experience if you're trying to compile a Varnish module from source. root@varnish001:libvmod-curl# ./autogen.sh Package varnishapi was not found in the pkg-config search path. Perhaps you should add the directory containing `varnishapi.pc' to the PKG_CONFIG_PATH

Varnish 6: Expression has type STRING, expected STEVEDORE

This might be a really simple error to solve, but I couldn't find anything in the documentation about how to use a STEVEDORE instead - in fact some of the documentation examples or ServerFault provided by Varnish (eg. [1]  [2]) used strings instead of stevedores, so I'

Running Percona Monitoring & Management 2 (PMM2) Server behind an Apache HTTP/2 reverse proxy

mod_proxy_http2 is considered experimental at the time of writing. Follow this guide at your own risk EDIT (10th July 2022): As of Debian 11 (I'm not sure if this was due to a newer Apache or a newer Nginx) this stopped working again because Apache is

Case Study: FLG Business Technology

In late 2019 I was approached by FLG Business Technology to submit a proposal to refresh their existing infrastructure. FLG is a CRM product focused on workflow to automate and control processes. They are a UK-based business founded over 11 years ago. Existing Setup In brief, they had a number

Fixing miscellaneous Webmin SilverStripe errors after a restore

This is mostly a reference post in case I ever have to go through this again. If it helps you too - well I've been there, I feel your pain, hopefully your pain doesn't last as long and you find some answers here. * PHP memory_limit

How to log SQL syntax errors in MySQL with tcpdump

Today I needed to capture a list of erroring MySQL queries on a server without introducing client-side code changes or a man in the middle proxy. The slow_query_log will only log successful queries. The general_log will log everything if raw_log is set to ON, but will

Splitting bold text from an Excel column without using Excel

Warning - this post contains almost everything you shouldn't do. Parsing XML with regular expressions, interfacing with binary files and expecting that they won't change, using code outside of Excel instead of using any of the in-built functions, making assumptions about data formats, and using Perl

Sending emails through Postfix timing out when using saslauthd

I was recently debugging an issue where Postfix users were able to receive emails, but emails in their outbox were timing out and refusing to send. Restarting Postfix or saslauthd would briefly alleviate the issue. The mail log was full of PAM authentication errors, and even testing authentication with testsaslauthd

InnoDB ERROR 1846 (0A000) when creating an index in MySQL

I recently had an issue where I was trying to hot-create an index on an InnoDB table running on MySQL 5.6.47, and received the following error: The MySQL documentation says: Tables created before MySQL 5.6 that include temporal columns (DATE, DATETIME or TIMESTAMP) and have not been

How to resolve "No chain/target/match by that name" when using iptables-multiport and Fail2Ban

This issue isn't specifically related to Fail2Ban but is a common way for it to present itself. The Fail2Ban log output will show something similar to: 2020-06-22 16:18:09,356 fail2ban.actions [17988]: ERROR Failed to execute ban jail 'sasl' action 'iptables-multiport' info

Associate a static IP address for Fargate outbound traffic using Terraform

Sometimes, when migrating an application into Fargate, you might find that it has outbound connections to legacy services which require IP whitelisting. As the IP address with each Fargate task is ephemeral, whitelisting its IP address every time it changes is unsustainable. One solution is to provision a NAT Gateway

Dynamically protect wp-login.php across multiple domains with htpasswd

Some pages, such as wp-login.php, are often the target of brute force attempts. Each failed login attempt requires a database search so even if all of your users have a strong password, constant brute forcing can still have a detrimental effect on the server. One workaround is to add

A Case Study on De-duplication

In early 2019, a customer approached us about an issue with the file storage aspect of their CMS application. Their Infrastructure Their application ran CMS for several of their clients, part of which handled the generation of invoices, contracts, and emails. Each application server expected the files to be stored

How to fix ERROR 2003 (HY000): Can't connect to MySQL server on 'node' (13)

If you get error 13 trying to connect to an IPv6 MySQL server for example: ERROR 2003 (HY000): Can’t connect to MySQL server on ‘node2’ (13) It could be a Permission Denied error trying to open the socket: # telnet node2 3306 Trying 2a01:4f8:161:xxxx::2b72… telnet: connect

Why should Transparent Huge Pages be disabled for database servers?

Linux treats physical pages as the basic unit of memory, splitting the available memory into page-sized ‘chunks’, and, no matter how little memory an application actually needs to write, it will be allocated at least one page (for example, 4096 bytes). A page is always contiguous, meaning that all of