homelab-docs/TRAEFIK.md

Traefik Reverse Proxy

Documentation for Traefik reverse proxy setup, SSL certificates, and deploying new public services.

Overview

There are TWO separate Traefik instances handling different services. Understanding which one to use is critical.

Instance	Location	IP	Purpose	Managed By
Traefik-Primary	CT 202	10.10.10.250	General services	Manual config files
Traefik-Saltbox	VM 101 (Docker)	10.10.10.100	Saltbox services only	Saltbox Ansible

---

⚠️ CRITICAL RULE: Which Traefik to Use

When Adding ANY New Service:

✅ USE Traefik-Primary (CT 202 @ 10.10.10.250) - For ALL new services ❌ DO NOT touch Traefik-Saltbox - Unless you're modifying Saltbox itself

Why This Matters:

• Traefik-Saltbox has complex Saltbox-managed configs (Ansible-generated)
• Messing with it breaks Plex, Sonarr, Radarr, and all media services
• Each Traefik has its own Let's Encrypt certificates
• Mixing them causes certificate conflicts and routing issues

---

Traefik-Primary (CT 202) - For New Services

Configuration

Location: Container 202 on PVE (10.10.10.250) Config Directory: /etc/traefik/ Main Config: /etc/traefik/traefik.yaml Dynamic Configs: /etc/traefik/conf.d/*.yaml

Access Traefik Config

# From Mac Mini:
ssh pve 'pct exec 202 -- cat /etc/traefik/traefik.yaml'
ssh pve 'pct exec 202 -- ls /etc/traefik/conf.d/'

# Edit a service config:
ssh pve 'pct exec 202 -- vi /etc/traefik/conf.d/myservice.yaml'

# View logs:
ssh pve 'pct exec 202 -- tail -f /var/log/traefik/traefik.log'

Services Using Traefik-Primary

Service	Domain	Backend
Excalidraw	excalidraw.htsn.io	10.10.10.206:8080 (docker-host)
FindShyt	findshyt.htsn.io	10.10.10.205 (CT 205)
Gitea	git.htsn.io	10.10.10.220:3000
Home Assistant	homeassistant.htsn.io	10.10.10.110
LM Dev	lmdev.htsn.io	10.10.10.111
Pi-hole	pihole.htsn.io	10.10.10.200
TrueNAS	truenas.htsn.io	10.10.10.200
Proxmox	pve.htsn.io	10.10.10.120
Copyparty	copyparty.htsn.io	10.10.10.201
AI Trade	aitrade.htsn.io	(trading server)
Pulse	pulse.htsn.io	10.10.10.206:7655 (monitoring)
Happy	happy.htsn.io	10.10.10.206:3002 (Happy Coder relay)

---

Traefik-Saltbox (VM 101) - DO NOT MODIFY

Configuration

Location: /opt/traefik/ inside Saltbox VM Managed By: Saltbox Ansible playbooks (automatic) Docker Mount: /opt/traefik → /etc/traefik in container

Services Using Traefik-Saltbox

• Plex (plex.htsn.io)
• Sonarr, Radarr, Lidarr
• SABnzbd, NZBGet, qBittorrent
• Overseerr, Tautulli, Organizr
• Jackett, NZBHydra2
• Authelia (SSO authentication)
• All other Saltbox-managed containers

View Saltbox Traefik (Read-Only)

# View config (don't edit!)
ssh pve 'qm guest exec 101 -- bash -c "docker exec traefik cat /etc/traefik/traefik.yml"'

# View logs
ssh saltbox 'docker logs -f traefik'

⚠️ WARNING: Editing Saltbox Traefik configs manually will be overwritten by Ansible and may break media services.

---

Adding a New Public Service - Complete Workflow

Follow these steps to deploy a new service and make it accessible at servicename.htsn.io.

Step 0: Deploy Your Service

First, deploy your service on the appropriate host.

Option A: Docker on docker-host (10.10.10.206)

ssh hutson@10.10.10.206
sudo mkdir -p /opt/myservice
cat > /opt/myservice/docker-compose.yml << 'EOF'
version: "3.8"
services:
  myservice:
    image: myimage:latest
    ports:
      - "8080:80"
    restart: unless-stopped
EOF
cd /opt/myservice && sudo docker-compose up -d

Option B: New LXC Container on PVE

ssh pve 'pct create CTID local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
  --hostname myservice --memory 2048 --cores 2 \
  --net0 name=eth0,bridge=vmbr0,ip=10.10.10.XXX/24,gw=10.10.10.1 \
  --rootfs local-zfs:8 --unprivileged 1 --start 1'

Option C: New VM on PVE

ssh pve 'qm create VMID --name myservice --memory 2048 --cores 2 \
  --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-pci'

Step 1: Create Traefik Config File

Use this template for new services on Traefik-Primary (CT 202):

Basic Template

# /etc/traefik/conf.d/myservice.yaml
http:
  routers:
    # HTTPS router
    myservice-secure:
      entryPoints:
        - websecure
      rule: "Host(`myservice.htsn.io`)"
      service: myservice
      tls:
        certResolver: cloudflare  # Use 'cloudflare' for proxied domains, 'letsencrypt' for DNS-only
      priority: 50

    # HTTP → HTTPS redirect
    myservice-redirect:
      entryPoints:
        - web
      rule: "Host(`myservice.htsn.io`)"
      middlewares:
        - myservice-https-redirect
      service: myservice
      priority: 50

  services:
    myservice:
      loadBalancer:
        servers:
          - url: "http://10.10.10.XXX:PORT"

  middlewares:
    myservice-https-redirect:
      redirectScheme:
        scheme: https
        permanent: true

Deploy the Config

# Create file on CT 202
ssh pve 'pct exec 202 -- bash -c "cat > /etc/traefik/conf.d/myservice.yaml << '\''EOF'\''
<paste config here>
EOF"'

# Traefik auto-reloads (watches conf.d directory)
# Check logs:
ssh pve 'pct exec 202 -- tail -f /var/log/traefik/traefik.log'

Step 2: Add Cloudflare DNS Entry

Cloudflare Credentials

Field	Value
Email	cloudflare@htsn.io
API Key	849ebefd163d2ccdec25e49b3e1b3fe2cdadc
Zone ID (htsn.io)	c0f5a80448c608af35d39aa820a5f3af
Public IP	70.237.94.174

Method 1: Manual (Cloudflare Dashboard)

1. Go to https://dash.cloudflare.com/
2. Select htsn.io domain
3. DNS → Add Record
4. Type: A, Name: myservice, IPv4: 70.237.94.174, Proxied: ☑️

Method 2: Automated (CLI)

Save this as ~/bin/add-cloudflare-dns.sh:

#!/bin/bash
# Add DNS record to Cloudflare for htsn.io

SUBDOMAIN="$1"
CF_EMAIL="cloudflare@htsn.io"
CF_API_KEY="849ebefd163d2ccdec25e49b3e1b3fe2cdadc"
ZONE_ID="c0f5a80448c608af35d39aa820a5f3af"
PUBLIC_IP="70.237.94.174"

if [ -z "$SUBDOMAIN" ]; then
  echo "Usage: $0 <subdomain>"
  echo "Example: $0 myservice  # Creates myservice.htsn.io"
  exit 1
fi

curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
  -H "X-Auth-Email: $CF_EMAIL" \
  -H "X-Auth-Key: $CF_API_KEY" \
  -H "Content-Type: application/json" \
  --data "{
    \"type\":\"A\",
    \"name\":\"$SUBDOMAIN\",
    \"content\":\"$PUBLIC_IP\",
    \"ttl\":1,
    \"proxied\":true
  }" | jq .

Usage:

chmod +x ~/bin/add-cloudflare-dns.sh
~/bin/add-cloudflare-dns.sh myservice  # Creates myservice.htsn.io

Step 3: Testing

# Check if DNS resolves
dig myservice.htsn.io

# Should return: 70.237.94.174 (or Cloudflare IPs if proxied)

# Test HTTP redirect
curl -I http://myservice.htsn.io

# Expected: 301 redirect to https://

# Test HTTPS
curl -I https://myservice.htsn.io

# Expected: 200 OK

# Check Traefik dashboard (if enabled)
# http://10.10.10.250:8080/dashboard/

Step 4: Update Documentation

After deploying, update:

1. IP-ASSIGNMENTS.md - Add to Services & Reverse Proxy Mapping table
2. This file (TRAEFIK.md) - Add to "Services Using Traefik-Primary" list
3. CLAUDE.md - Update quick reference if needed

---

SSL Certificates

Traefik has two certificate resolvers configured:

Resolver	Use When	Challenge Type	Notes
letsencrypt	Cloudflare DNS-only (gray cloud ☁️)	HTTP-01	Requires port 80 reachable
cloudflare	Cloudflare Proxied (orange cloud 🟠)	DNS-01	Works with Cloudflare proxy

⚠️ Important: HTTP Challenge vs DNS Challenge

If Cloudflare proxy is enabled (orange cloud), HTTP challenge FAILS because Cloudflare redirects HTTP→HTTPS before the challenge reaches your server.

Solution: Use cloudflare resolver (DNS-01 challenge) instead.

Certificate Resolver Configuration

Cloudflare API credentials are configured in /etc/systemd/system/traefik.service:

Environment="CF_API_EMAIL=cloudflare@htsn.io"
Environment="CF_API_KEY=849ebefd163d2ccdec25e49b3e1b3fe2cdadc"

Certificate Storage

Resolver	Storage File
HTTP challenge (letsencrypt)	/etc/traefik/acme.json
DNS challenge (cloudflare)	/etc/traefik/acme-cf.json

Permissions: Must be 600 (read/write owner only)

# Check permissions
ssh pve 'pct exec 202 -- ls -la /etc/traefik/acme*.json'

# Fix if needed
ssh pve 'pct exec 202 -- chmod 600 /etc/traefik/acme.json'
ssh pve 'pct exec 202 -- chmod 600 /etc/traefik/acme-cf.json'

Certificate Renewal

• Automatic via Traefik
• Checks every 24 hours
• Renews 30 days before expiry
• No manual intervention needed

Troubleshooting Certificates

Certificate Fails to Issue

# Check Traefik logs
ssh pve 'pct exec 202 -- tail -f /var/log/traefik/traefik.log | grep -i error'

# Verify Cloudflare API access
curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
  -H "X-Auth-Email: cloudflare@htsn.io" \
  -H "X-Auth-Key: 849ebefd163d2ccdec25e49b3e1b3fe2cdadc"

# Check acme.json permissions
ssh pve 'pct exec 202 -- ls -la /etc/traefik/acme*.json'

Force Certificate Renewal

# Delete certificate (Traefik will re-request)
ssh pve 'pct exec 202 -- rm /etc/traefik/acme-cf.json'
ssh pve 'pct exec 202 -- touch /etc/traefik/acme-cf.json'
ssh pve 'pct exec 202 -- chmod 600 /etc/traefik/acme-cf.json'
ssh pve 'pct exec 202 -- systemctl restart traefik'

# Watch logs
ssh pve 'pct exec 202 -- tail -f /var/log/traefik/traefik.log'

---

Quick Deployment - One-Liner

For fast deployment, use this all-in-one command:

# === DEPLOY SERVICE (example: myservice on docker-host port 8080) ===

# 1. Create Traefik config
ssh pve 'pct exec 202 -- bash -c "cat > /etc/traefik/conf.d/myservice.yaml << EOF
http:
  routers:
    myservice-secure:
      entryPoints: [websecure]
      rule: Host(\\\`myservice.htsn.io\\\`)
      service: myservice
      tls: {certResolver: cloudflare}
  services:
    myservice:
      loadBalancer:
        servers:
          - url: http://10.10.10.206:8080
EOF"'

# 2. Add Cloudflare DNS
curl -s -X POST "https://api.cloudflare.com/client/v4/zones/c0f5a80448c608af35d39aa820a5f3af/dns_records" \
  -H "X-Auth-Email: cloudflare@htsn.io" \
  -H "X-Auth-Key: 849ebefd163d2ccdec25e49b3e1b3fe2cdadc" \
  -H "Content-Type: application/json" \
  --data '{"type":"A","name":"myservice","content":"70.237.94.174","proxied":true}'

# 3. Test (wait a few seconds for DNS propagation)
curl -I https://myservice.htsn.io

---

Docker Service with Traefik Labels (Alternative)

If deploying a service via Docker on docker-host (VM 206), you can use Traefik labels instead of config files.

Requirements:

• Traefik must have access to Docker socket
• Service must be on same Docker network as Traefik

Example docker-compose.yml:

version: "3.8"

services:
  myservice:
    image: myimage:latest
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.myservice.rule=Host(`myservice.htsn.io`)"
      - "traefik.http.routers.myservice.entrypoints=websecure"
      - "traefik.http.routers.myservice.tls.certresolver=letsencrypt"
      - "traefik.http.services.myservice.loadbalancer.server.port=8080"
    networks:
      - traefik

networks:
  traefik:
    external: true

Note: This method is NOT currently used on Traefik-Primary (CT 202), as it doesn't have Docker socket access. Config files are preferred.

---

Cloudflare API Reference

API Credentials

Field	Value
Email	cloudflare@htsn.io
API Key	849ebefd163d2ccdec25e49b3e1b3fe2cdadc
Zone ID	c0f5a80448c608af35d39aa820a5f3af

Common API Operations

Set credentials:

CF_EMAIL="cloudflare@htsn.io"
CF_API_KEY="849ebefd163d2ccdec25e49b3e1b3fe2cdadc"
ZONE_ID="c0f5a80448c608af35d39aa820a5f3af"

List all DNS records:

curl -X GET "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
  -H "X-Auth-Email: $CF_EMAIL" \
  -H "X-Auth-Key: $CF_API_KEY" | jq

Add A record:

curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
  -H "X-Auth-Email: $CF_EMAIL" \
  -H "X-Auth-Key: $CF_API_KEY" \
  -H "Content-Type: application/json" \
  --data '{
    "type":"A",
    "name":"subdomain",
    "content":"70.237.94.174",
    "proxied":true
  }'

Delete record:

curl -X DELETE "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID" \
  -H "X-Auth-Email: $CF_EMAIL" \
  -H "X-Auth-Key: $CF_API_KEY"

Update record (toggle proxy):

curl -X PATCH "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID" \
  -H "X-Auth-Email: $CF_EMAIL" \
  -H "X-Auth-Key: $CF_API_KEY" \
  -H "Content-Type: application/json" \
  --data '{"proxied":false}'

---

Troubleshooting

Service Not Accessible

# 1. Check if DNS resolves
dig myservice.htsn.io

# 2. Check if backend is reachable
curl -I http://10.10.10.XXX:PORT

# 3. Check Traefik logs
ssh pve 'pct exec 202 -- tail -f /var/log/traefik/traefik.log'

# 4. Check Traefik config is valid
ssh pve 'pct exec 202 -- cat /etc/traefik/conf.d/myservice.yaml'

# 5. Restart Traefik (if needed)
ssh pve 'pct exec 202 -- systemctl restart traefik'

Certificate Issues

# Check certificate status in acme.json
ssh pve 'pct exec 202 -- cat /etc/traefik/acme-cf.json | jq'

# Check certificate expiry
echo | openssl s_client -servername myservice.htsn.io -connect myservice.htsn.io:443 2>/dev/null | openssl x509 -noout -dates

502 Bad Gateway

Cause: Backend service is down or unreachable

# Check if backend is running
ssh backend-host 'systemctl status myservice'

# Check if port is open
nc -zv 10.10.10.XXX PORT

# Check firewall
ssh backend-host 'iptables -L -n | grep PORT'

404 Not Found

Cause: Traefik can't match the request to a router

# Check router rule matches domain
ssh pve 'pct exec 202 -- cat /etc/traefik/conf.d/myservice.yaml | grep rule'

# Should be: rule: "Host(`myservice.htsn.io`)"

# Check DNS is pointing to correct IP
dig myservice.htsn.io

# Restart Traefik to reload config
ssh pve 'pct exec 202 -- systemctl restart traefik'

---

Advanced Configuration Examples

WebSocket Support

For services that use WebSockets (like Home Assistant):

http:
  routers:
    myservice-secure:
      entryPoints:
        - websecure
      rule: "Host(`myservice.htsn.io`)"
      service: myservice
      tls:
        certResolver: cloudflare

  services:
    myservice:
      loadBalancer:
        servers:
          - url: "http://10.10.10.XXX:PORT"
        # No special config needed - WebSockets work by default in Traefik v2+

Custom Headers

Add custom headers (e.g., security headers):

http:
  routers:
    myservice-secure:
      middlewares:
        - myservice-headers

  middlewares:
    myservice-headers:
      headers:
        customResponseHeaders:
          X-Frame-Options: "DENY"
          X-Content-Type-Options: "nosniff"
          Referrer-Policy: "strict-origin-when-cross-origin"

Basic Authentication

Protect a service with basic auth:

http:
  routers:
    myservice-secure:
      middlewares:
        - myservice-auth

  middlewares:
    myservice-auth:
      basicAuth:
        users:
          - "user:$apr1$..." # Generate with: htpasswd -nb user password

---

Maintenance

Monthly Checks

# Check Traefik status
ssh pve 'pct exec 202 -- systemctl status traefik'

# Review logs for errors
ssh pve 'pct exec 202 -- grep -i error /var/log/traefik/traefik.log | tail -20'

# Check certificate expiry dates
ssh pve 'pct exec 202 -- cat /etc/traefik/acme-cf.json | jq ".cloudflare.Certificates[] | {domain: .domain.main, expiry: .certificate}"'

# Verify all services responding
for domain in plex.htsn.io git.htsn.io truenas.htsn.io; do
  echo "Testing $domain..."
  curl -sI https://$domain | head -1
done

Backup Traefik Config

# Backup all configs
ssh pve 'pct exec 202 -- tar czf /tmp/traefik-backup-$(date +%Y%m%d).tar.gz /etc/traefik'

# Copy to safe location
scp "pve:/var/lib/lxc/202/rootfs/tmp/traefik-backup-*.tar.gz" ~/Backups/traefik/

---

Related Documentation

• IP-ASSIGNMENTS.md - Service IP addresses
• CLOUDFLARE.md - Cloudflare DNS management (coming soon)
• SERVICES.md - Complete service inventory (coming soon)

---

Last Updated: 2025-12-22