Sending services through Tailscale

I tried out Tailscale’s experimental in-process networking support. It did what I wanted: I was able to serve HTTPS to my Tailscale network from an unprivileged process. I also wrote a small program, tsproxy, to provide the same for any server; this let me “install” code-server on a Chromebook, but with the compute power of another host.

This post walks through what I was trying to do, and what I learned along the way.

TLS over Tailscale

I’ve been using Tailscale for a while now, mostly to SSH between my laptop and my development machine. I wanted to also use it to run a reading list server, so the admin interface (add/edit/publish entries) is never visible outside “my devices”.

A key portion of flow I want for a reading list is “share-to” support: open a link and send it to “read later”. This has a bunch of requirements stemming from the notion of progressive web apps; most importantly for connectivity purposes, it requires that the site is served via HTTPS. ¹

Luckily, Tailscale has recently (fall 2021) launched a feature that lets servers in your Tailscale network get valid TLS certificates from Let’s Encrypt, at hostnames like <host>.<network>.ts.net. I was able to pretty quickly walk through the example, reach my server in the browser (via HTTPS!), and “install” it.

Great success!

In which cceckman is picky

…but I wasn’t satisfied.

My main concern was around automation and permissions. At first glance, the servetls example expects to have access to the tailscaled daemon in order to acquire a TLS certificate; that’s not easily possible on my system.² There are workarounds - regenerating certificates with a cron, chmod, something something user namespace - but I didn’t want to maintain even more mechanism.

The other problem is identity.³ In the above configuration, the installed app was found at

https://<hostname>.<tailnet>.ts.net:<five-digit port>

The name doesn’t show up often, so it’s not the worst name. But that presentation doesn’t match my plans for this app: I’m not going to be running this service on my desktop in the longer term, and I’m not huge on saving / hard-coding a port.

On a more abstract level, I’m used to thinking of DNS as naming “what you want” rather than “how you get there”. If “what you want” is a particular host, sure, use the hostname. But my mental model for this server is that it’s a service, not a daemon- it’s not logically tied to the host.

What I’d really like to do is put just this server on my Tailscale network- without having the client know the particular host / port incantation, and without the server having host-level credentials.

Experiments with tsnet

Of course, that functionality is already on the roadmap; Tailscale provides experimental (!) code for offering a service directly to the network. The tsnet package provides a net.Listener with a configurable hostname, and tailscale.GetCredentials knows how to do-the-right-thing for such a connection.

The net result is that the server can be reached at the much-prettier address:

https://<servername>.<tailnet>.ts.net

without any additional host-level plumbing: no installing Tailscale, tweaking permissions or ownership, certificate crons, etc.

I started by working through the example. As expected for experimental code, I had to do a little finagling to get it to work:

• The tsnet package won’t do anything unless the user acknowledges that it’s experimental code (by setting an environment variable). This is A Good Thing- it forces the user to set their expectations to “ready for breakage”.

  I’m intentionally going to leave that environment variable out of this post. Copy-pasting won’t work- if you want to try out this experimental feature, you’ll also need to accept the risk.

• The first time I ran the program, I had to do some tweaks to authenticate.

  A server or host connecting to Tailscale needs to authenticate to the control server; once it’s known to the control server it can use locally-stored credentials to authenticate. The human-facing tailscale CLI or gui will automatically prompt for login; but by default, it seems that a tsnet-using program will just stall out waiting for login.

  It seems that TS_LOGIN=1 will cause tsnet to approximate the CLI’s “go to a link” registration flow; with that environment variable, within the slew of logs, I found a link to https://login.tailscale.com that completed the authentication flow for the program. But this wasn’t super friendly- I had to watch through a somewhat verbose log stream to get the URL.

  I had better success registering an auth key for Tailscale and providing it to the process via the TS_AUTHKEY environment variable (e.g. TS_AUTHKEY=tskey-asdf-jlksemicolon). For testing, I used the “ephemeral” key typ, so the server automatically disappeared from my network when the processe exited. For “real” use, a one-off key seems like the way to go; after the initial enrollment, a persistent saved directory will keep long-term credentials.

  For the future of this feature (hi Tailscale team!), it would be nice to have more of this accessible programmatically. At least, I think I’d prefer an error rather than stall if there’s no authentication path available; a more complicated flow where tsnet returns a login URL might facilitate friendlier flows.

After working through the example, I ported my own server to tsnet; you can see the results in commit faa374.

Expanding the network: tsproxy

I did most of this development remotely: my eyes and hands at my laptop, but files and servers on another machine. Over the last couple years I’ve experimented with various ways to accomplish this, with varied results;⁴ so far, I’ve settled on using code-server as an “installed” web app. (“Installed” rather than “in-browser” allows keybindings to behave like a local program - very useful for an IDE.)

I’d had a script to make this work with Tailscale’s creds feature, but it was rather clunky, using sudo tailscale creds and moves/chowns to get the keys to code-server.⁵ Again, my problem is getting just this one server onto my network - is there a way to do that without diving into code-server’s source?

“Yes”, it turns out. I didn’t golf, but a simple proxy took me a little more than 100 lines of Golang code. An example invocation, after the initial authentication:

nohup code-server 2>&1 --socket ~/.code-server.sock &
tsproxy --from code-server --to ~/.code-server.sock

That gets me https://code-server.<my-network>.ts.net as an install-friendly URL.

Please be wary of using this program; it was hacked, not engineered. (No one has reviewed the code; it has no tests or continuous builds; etc etc.) But if you’re interested, the code is at https://github.com/cceckman/tsproxy; and it does seem to work for this use case.

Ur doin it wrong (?)

I’m sure there are other ways to do what tsproxy does. As I was writing this post, Tailscale’s December newsletter came out, pointing to this tool for running Tailscale in Docker. More generally, if I understand correctly, it’s possible to run multiple tailscaleds in different network namespaces - providing even stronger isolation around the proxied server’s connectivity.

But…well, I understand net.Listener and a two-flag program much better than I understand network namespaces. tsproxy gets me a useful result without much work; and for now, that’s good enough for me. If anyone has other suggestions, feel free to reach out - I am always interested in learning new things!

Comments? Ideas?s See cceckman.com for my contact info.