Jason Sultana

Follow this space for writings (and ramblings) about interesting things related to software development.

NGINX and Kestrel: HTTP Connection Upgrade mayhem

12 Dec 2020 » linux, dotnet, htaccess, others

G’day guys!

Today, I present you an epic tale of two servers: Nginx and Kestrel. Except that it’s not so much a tale, as a fable. Except that it’s not so much a fable, as a nightmarish horror story that will haunt you for the rest of your days. Well, not really - but it’s still pretty scary until you get to the resolution.

Come on, it can’t be that bad!

I hear ya. Nginx? Awesome! Kestrel? Awesome! What can go wrong? I’ll tell you what can bloody go wrong. Let’s start with some background.

I had recently(ish) moved from one web-hosting provider (WebFaction) to another (OpalStack), since the former was being acquired and the replacement service no longer supported dotnet. I migrated the application and data, did some basic tests and everything was looking fine. Until I deployed a new version and tried to create a new record. The result?

Microsoft.AspNetCore.Server.Kestrel) Connection id ""0HM4PMSK2S9NV"" bad request data: ""Requests with 'Connection: Upgrade' cannot have content in the request body.""

What in Gandalf’s name is that!?

That’s what I said!

Anyway, this wasn’t specific to the new feature I deployed. Turns out I hadn’t tried saving anything when I did my tests post-migration, and just discovered that there was an issue saving data now. I know, I know - but I guesss that’s why I’m not in QA/QE. Well, after a little googling, I found this, this and this.

It turns out that the front-facing nginx server hosted and maintained by my webhost is proactively (aggressively?) adding a Connection: Upgrade HTTP header when proxying requests to apps that they host. Kestrel, on the other hand, disallows requests that contain a Connection: Upgrade HTTP header and also a request body. i.e - all HTTP POST and HTTP PUT requests.

I reached out to my webhost, and while they were sympathetic to my cause (or at least the support guy was), it wasn’t something that they were able to change in the foreseeable future, nor was I able to provide my own nginx configuration to rectify it just for my own application. So I was, as some might say, in between a rock and a hard place. Or as we say in Australia, up sh*t creek.

Eesh. New web host?

That did look likely, but I wanted to try to avoid it. Firstly, it is a bit of a pain to move to YAWH (Yet Another Web Host). Secondly, the payment plan on OpalStack is incredibly simple and incredibly reasonable - even cheaper than my previous host. While AWS, Azure or even DigitalOcean would be technologically better, I can guarantee that my client wouldn’t be happy with the extra costs, which are mainly related to the managed database. Finally, OpalStack does actually have a very nice, simple product and great support as well, so I didn’t want to leave them if I didn’t have to.

I get the feeling that you didn’t have to?

You’re damn right! The main problem is that pesky nginx server adding the Connection: Upgrade http header. If I had a way to remove the header before it hits kestrel, then everything would work fine. So I tried adding an extra Apache proxy application in between the front-facing nxinx server and my kestrel dotnet app, which simply proxies all traffic to a designated host (the url of the app) to the port that the app is listening on. The result? Works like a charm! Sure, there is a tiny bit of extra complexity due to the proxy app, but it’s flexible enough to be used for multiple sites, environments, etc. So it’s not like I’d need to create a new proxy per application. And yes, there would be a tiny bit of extra latency on the server as well, since the request needs to be proxied, but for my purposes, the effect is unnoticable.

Why did you go with Apache instead of nginx?

For two reasons. Firstly, I’m more familiar with Apache and htaccess config than nginx. Secondly (and more importantly), an Apache server would be managed by the webhost, meaning that I could just apply some htaccess config and be done with it. With nginx, I’d need to download, build, install and maintain the nginx instance myself, introducing more complexity and another point of failure for the system. i.e - if the custom nginx server stopped running, the app would stop working. I don’t have to worry about the same thing with Apache, since it’s managed by the webhost.

Apache to the rescue!

RewriteEngine On
RewriteCond %{HTTP_HOST} ^api.your-account.webhost.com$ [NC]
RewriteRule ^(.*)$ http://localhost:1234%{REQUEST_URI} [P,NC,L,QSA]

It’s very simple, actually. For any request that comes into the Apache reverse proxy, if the host matches api.your-account.webhost.com (obviously change this to suit your situation), it’ll proxy the request to whatever app is running on the same server under port 1234, along with whatever trailing URI was in the request. Since Connection: Upgrade is a hop-by-hop header, it won’t be included in the proxied request. If it were, we could include a little extra htaccess magic to remove it, but it’s even simpler that we don’t have to.

Anyway, hopefully you don’t run into this problem, but if you do (and are lucky enough to have read this), at least now you have an alternative to switching to a more expensive cloud provider. Apache to the rescue!

Well, that’s it from me for now. Catch ya!