Migrating From Ngrok To Cloudflare Tunnels

Max Cantor

A blog of thoughts on management, tech, aviation, and fitness.

Introduction

I’ve been using a paid ngrok premium account for ages for web application development and overall its worked quite well. My usual stack is:

  • A datastore running off of Hasura.
  • AWS Lambda functions orchestrated by Serverless.com and running locally using the serverless-offline plugin.
  • One or more ReactJS apps created by create-react-app
  • Infrastructure orchestrated by Pulumi or Terraform. *This isn’t really relevant to this blog post, however, which is about local development`

Overall, I’ve got great skeleton code setup which allows me to spin up new projects incredibly quickly with a mature, scalable setup.

Regarding ngrok, there have been some nagging concerns about the company’s relative opacity, but to be honest, they we somewhat immaterial to me as nothing going over ngrok was particularly sensitive and the dollar cost is manageable.

However, I recently started using ngrok for iOS development where a SwiftUI app would have to connect to a Hasura service proxied by ngrok. Unfortunately, ngrok’s SSL certificate isn’t compliant with Apple’s SSL requirements which I believe was causing intermmitent SSL certificate validation errors to arise from my Swift app.

The Hunt For Alternatives

There are quite a few alternatives to ngrok and after figuring out which ones either no longer supported, dead, broken, or scary for some other reason, I settled on Cloudflare Tunnels. They’re free and from an exceptionally trustworthy company, not to mention a company who’s main business is making the internet faster so I was relatively confident that they would be performant.

How I Used Ngrok

My skeleton project includes a base ngrok configuration:

authtoken: ${NGROK_AUTH_TOKEN}
remote_management: null
tunnels:
  app:
    proto: http
    bind_tls: true
    addr: ${REACT_APP_PORT}
    subdomain: ${SUBDOMAIN_APP}
  lambdas:
    proto: http
    bind_tls: true
    addr: ${LAMBDA_LOCAL_PORT}
    subdomain: ${SUBDOMAIN_LAMBDAS}
  console:
    proto: http
    addr: ${HASURA_CONSOLE_PORT}
    bind_tls: true
    subdomain: ${SUBDOMAIN_HASURA_CONSOLE}
  hasura:
    proto: http
    addr: ${HASURA_EXPOSED_PORT}
    bind_tls: true
    subdomain: ${SUBDOMAIN_HASURA}

And a make command to populate those variables and run ngrok:

ngrok:
	export NGROK=.ngrok-generated.yml ;  envsubst < ngrok-base.yml > $${NGROK} ; \
	ngrok start -config $${NGROK} --all

This has the nice property that it is trivially easy to add new services. For example, just adding this to the tunnels section as long as all of the variables are set in my .env.develompent files

adminapp:
  proto: http
  bind_tls: true
  addr: ${REACT_APP_ADMIN_APP_PORT}
  subdomain: ${SUBDOMAIN_ADMIN_APP}

How I thought I’d Use Cloudflare Tunnels

With this post on “many services, onecloudflared”, this is how I imagined my cloudflare configuration would look:

tunnel: ${CLOUDFLARE_TUNNEL_UUID}
credentials-file: ${CLOUDFLARE_TUNNEL_CREDS}
ingress:
  - hostname: ${SUBDOMAIN_LAMBDAS}
    service: https://localhost:${LAMBDA_LOCAL_PORT}
  - hostname: ${SUBDOMAIN_HASURA_CONSOLE}
    service: https://localhost:${HASURA_CONSOLE_PORT}
  - hostname: ${SUBDOMAIN_HASURA}
    service: https://localhost:${HASURA_EXPOSED_PORT}
  - hostname: ${SUBDOMAIN_APP_ADMIN}
    service: https://localhost:${REACT_APP_ADMIN_PORT}

Before running this, I needed to create a tunnel with these commands:

brew install cloudflare/cloudflare/cloudflared ;
cloudflared login ;
cloudflared tunnel create development

The create command will generate a yaml file with a token into it, copy it ox

I kind of assumed Cloudflare would operate like ngrok and create subdomains on some shared tunnel subdomain. Right to trusty make:

CLOUDFLARE_TUNNEL_CONF_GENERATED=.norepo.cloudflare-tunnel-${ENV}.yaml
cloudflare_tunnel: cloudflare_tunnel_cfg
	cloudflared tunnel --config ${CLOUDFLARE_TUNNEL_CONF_GENERATED} run

First Gotcha

I imagined that cloudflare would simply create these subdomains on some shared tld so after running the run command I searched all over the internet and all I could find was this somewhat mysterious cfargotunnel.com tld.

According to this blog post, the subdomain will simply have the CLOUDFLARE_TUNNEL_UUID from above prepended to it.

Also, I noticed that the hostnames in the examples were all fully qualified, unlike in ngrok. Ok, let me try that.

Second Gotcha

tunnel: ${CLOUDFLARE_TUNNEL_UUID}
credentials-file: ${CLOUDFLARE_TUNNEL_CREDS}
ingress:
  - hostname: ${SUBDOMAIN_LAMBDAS}.<my-uuid>.cfargotunnel.com
    service: https://localhost:${LAMBDA_LOCAL_PORT}
  - hostname: ${SUBDOMAIN_HASURA_CONSOLE}.<my-uuid>.cfargotunnel.com
    service: https://localhost:${HASURA_CONSOLE_PORT}
  - hostname: ${SUBDOMAIN_HASURA}.<my-uuid>.cfargotunnel.com
    service: https://localhost:${HASURA_EXPOSED_PORT}
  - hostname: ${SUBDOMAIN_APP_ADMIN}.<my-uuid>.cfargotunnel.com
    service: https://localhost:${REACT_APP_ADMIN_PORT}

Oops, need a catch all 404 ingress route. Easy enough to fix:

tunnel: ${CLOUDFLARE_TUNNEL_UUID}
credentials-file: ${CLOUDFLARE_TUNNEL_CREDS}
ingress:
  - hostname: ${SUBDOMAIN_LAMBDAS}.<my-uuid>.cfargotunnel.com
    service: https://localhost:${LAMBDA_LOCAL_PORT}
  - hostname: ${SUBDOMAIN_HASURA_CONSOLE}.<my-uuid>.cfargotunnel.com
    service: https://localhost:${HASURA_CONSOLE_PORT}
  - hostname: ${SUBDOMAIN_HASURA}.<my-uuid>.cfargotunnel.com
    service: https://localhost:${HASURA_EXPOSED_PORT}
  - hostname: ${SUBDOMAIN_APP_ADMIN}.<my-uuid>.cfargotunnel.com
    service: https://localhost:${REACT_APP_ADMIN_PORT}
  - service: http_status:404

Hmm.. still didn’t work. lambdas-development.<my_uuid>.cfargotunnel.com didn’t resolve to anything. At this point, I broke out the enviroment into a separate domain level so: lambdas-development.<my_uuid>.cfargotunnel.com would become: lambdas.development.<my_uuid>.cfargotunnel.com

This will be important later.

Maybe I need my own tld?

Third Gotcha

Ok, so I went and registered a quick and dirty .com domain on Cloudflare. Now my config file looked like:

tunnel: ${CLOUDFLARE_TUNNEL_UUID}
credentials-file: ${CLOUDFLARE_TUNNEL_CREDS}
ingress:
  - hostname: ${SUBDOMAIN_LAMBDAS}.${ENV}.<my-uuid>.mydomain.com
    service: https://localhost:${LAMBDA_LOCAL_PORT}
  - hostname: ${SUBDOMAIN_HASURA_CONSOLE}.${ENV}.<my-uuid>.mydomain.com
    service: https://localhost:${HASURA_CONSOLE_PORT}
  - hostname: ${SUBDOMAIN_HASURA}.${ENV}.<my-uuid>.mydomain.com
    service: https://localhost:${HASURA_EXPOSED_PORT}
  - hostname: ${SUBDOMAIN_APP_ADMIN}.${ENV}.<my-uuid>.mydomain.com
    service: https://localhost:${REACT_APP_ADMIN_PORT}
  - service: http_status:404

Still nothing.. I ran the run command and those domains still turned up nothing.

Turns out that even with config files, you need to manually tell Cloudflare to create CNAMEs for your domains

so I just ran:

cloudflared tunnel route dns development lambdas.development.mydomain.com

Still nothing.

Fourth Gotcha

Scratching my head for a bit. Even with these CNAMEs setup, I couldn’t successfully send requests to my lambda handlers.

At this point, I don’t remember if the DNS was still broken or I was just getting the ERR_SSL_VERSION_OR_CIPHER_MISMATCH errors.

So, the combination of the next two fixes eventually resolved the problem, but I don’t remember if adding every route was 100% necessary to get a single route working.

The Promised Land

In case Cloudflare simply needed all routes defined, I ran the route command for all of my ingress routes:

cloudflared tunnel route dns development hasura.development.mydomain.com && \
cloudflared tunnel route dns development hasura-console.development.mydomain.com && \
cloudflared tunnel route dns development app.development.mydomain.com

Eventually I was able successfully do domain lookups on lambda.mydomain.com but still could not send requests as I was getting ERR_SSL_VERSION_OR_CIPHER_MISMATCH errors. Being new to Cloudflare, I struggled on this one. I experimented with different versions of TLS, turning off TLS 1.3, etc. People who know OpenSSL better than I probably could have solved this in a heartbeat.

The problem was including the environment as a separate domain heirarchy. With wildcard certificates, *.mydomain.com will match lambda.mydomain.com or lambda-development.mydomain.com but not lambda.development.mydomain.com. So, one last change to my config file (note the change from .${ENV} to -${ENV}):

I don’t believe that this limitation on multi level wildcard SSL certs is in any way a Cloudflare issue, just a limitation of the SSL spec.

tunnel: ${CLOUDFLARE_TUNNEL_UUID}
credentials-file: ${CLOUDFLARE_TUNNEL_CREDS}
ingress:
  - hostname: ${SUBDOMAIN_LAMBDAS}-${ENV}.<my-uuid>.mydomain.com
    service: https://localhost:${LAMBDA_LOCAL_PORT}
  - hostname: ${SUBDOMAIN_HASURA_CONSOLE}-${ENV}.<my-uuid>.mydomain.com
    service: https://localhost:${HASURA_CONSOLE_PORT}
  - hostname: ${SUBDOMAIN_HASURA}-${ENV}.<my-uuid>.mydomain.com
    service: https://localhost:${HASURA_EXPOSED_PORT}
  - hostname: ${SUBDOMAIN_APP_ADMIN}-${ENV}.<my-uuid>.mydomain.com
    service: https://localhost:${REACT_APP_ADMIN_PORT}
  - service: http_status:404

And

cloudflared tunnel route dns development lambdas-development.mydomain.com && \
cloudflared tunnel route dns development hasura-development.mydomain.com && \
cloudflared tunnel route dns development hasura-console-development.mydomain.com && \
cloudflared tunnel route dns development app-development.mydomain.com

Success!