homelab-docs/TRAEFIK.md

# Traefik Reverse Proxy

Documentation for Traefik reverse proxy setup, SSL certificates, and deploying new public services.

## Overview

There are **TWO separate Traefik instances** handling different services. Understanding which one to use is critical.

| Instance | Location | IP | Purpose | Managed By |
|----------|----------|-----|---------|------------|
| **Traefik-Primary** | CT 202 | **10.10.10.250** | General services | Manual config files |
| **Traefik-Saltbox** | VM 101 (Docker) | **10.10.10.100** | Saltbox services only | Saltbox Ansible |

---

## ⚠️ CRITICAL RULE: Which Traefik to Use

### When Adding ANY New Service:

✅ **USE Traefik-Primary (CT 202 @ 10.10.10.250)** - For ALL new services
❌ **DO NOT touch Traefik-Saltbox** - Unless you're modifying Saltbox itself

### Why This Matters:

- **Traefik-Saltbox** has complex Saltbox-managed configs (Ansible-generated)
- Messing with it breaks Plex, Sonarr, Radarr, and all media services
- Each Traefik has its own Let's Encrypt certificates
- Mixing them causes certificate conflicts and routing issues

---

## Traefik-Primary (CT 202) - For New Services

### Configuration

**Location**: Container 202 on PVE (10.10.10.250)
**Config Directory**: `/etc/traefik/`
**Main Config**: `/etc/traefik/traefik.yaml`
**Dynamic Configs**: `/etc/traefik/conf.d/*.yaml`

### Access Traefik Config

```bash
# From Mac Mini:
ssh pve 'pct exec 202 -- cat /etc/traefik/traefik.yaml'
ssh pve 'pct exec 202 -- ls /etc/traefik/conf.d/'

# Edit a service config:
ssh pve 'pct exec 202 -- vi /etc/traefik/conf.d/myservice.yaml'

# View logs:
ssh pve 'pct exec 202 -- tail -f /var/log/traefik/traefik.log'
```

### Services Using Traefik-Primary

| Service | Domain | Backend |
|---------|--------|---------|
| Excalidraw | excalidraw.htsn.io | 10.10.10.206:8080 (docker-host) |
| FindShyt | findshyt.htsn.io | 10.10.10.205 (CT 205) |
| Gitea | git.htsn.io | 10.10.10.220:3000 |
| Home Assistant | homeassistant.htsn.io | 10.10.10.110 |
| LM Dev | lmdev.htsn.io | 10.10.10.111 |
| MetaMCP | metamcp.htsn.io | 10.10.10.207:12008 (docker-host2) |
| Pi-hole | pihole.htsn.io | 10.10.10.200 |
| TrueNAS | truenas.htsn.io | 10.10.10.200 |
| Proxmox | pve.htsn.io | 10.10.10.120 |
| Copyparty | copyparty.htsn.io | 10.10.10.201 |
| AI Trade | aitrade.htsn.io | (trading server) |
| Pulse | pulse.htsn.io | 10.10.10.206:7655 (monitoring) |
| Happy | happy.htsn.io | 10.10.10.206:3002 (Happy Coder relay) |

---

## Traefik-Saltbox (VM 101) - DO NOT MODIFY

### Configuration

**Location**: `/opt/traefik/` inside Saltbox VM
**Managed By**: Saltbox Ansible playbooks (automatic)
**Docker Mount**: `/opt/traefik` → `/etc/traefik` in container

### Services Using Traefik-Saltbox

- Plex (plex.htsn.io)
- Sonarr, Radarr, Lidarr
- SABnzbd, NZBGet, qBittorrent
- Overseerr, Tautulli, Organizr
- Jackett, NZBHydra2
- Authelia (SSO authentication)
- All other Saltbox-managed containers

### View Saltbox Traefik (Read-Only)

```bash
# View config (don't edit!)
ssh pve 'qm guest exec 101 -- bash -c "docker exec traefik cat /etc/traefik/traefik.yml"'

# View logs
ssh saltbox 'docker logs -f traefik'
```

**⚠️ WARNING**: Editing Saltbox Traefik configs manually will be overwritten by Ansible and may break media services.

---

## Adding a New Public Service - Complete Workflow

Follow these steps to deploy a new service and make it accessible at `servicename.htsn.io`.

### Step 0: Deploy Your Service

First, deploy your service on the appropriate host.

#### Option A: Docker on docker-host (10.10.10.206)

```bash
ssh hutson@10.10.10.206
sudo mkdir -p /opt/myservice
cat > /opt/myservice/docker-compose.yml << 'EOF'
version: "3.8"
services:
  myservice:
    image: myimage:latest
    ports:
      - "8080:80"
    restart: unless-stopped
EOF
cd /opt/myservice && sudo docker-compose up -d
```

#### Option B: New LXC Container on PVE

```bash
ssh pve 'pct create CTID local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
  --hostname myservice --memory 2048 --cores 2 \
  --net0 name=eth0,bridge=vmbr0,ip=10.10.10.XXX/24,gw=10.10.10.1 \
  --rootfs local-zfs:8 --unprivileged 1 --start 1'
```

#### Option C: New VM on PVE

```bash
ssh pve 'qm create VMID --name myservice --memory 2048 --cores 2 \
  --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-pci'
```

### Step 1: Create Traefik Config File

Use this template for new services on **Traefik-Primary (CT 202)**:

#### Basic Template

```yaml
# /etc/traefik/conf.d/myservice.yaml
http:
  routers:
    # HTTPS router
    myservice-secure:
      entryPoints:
        - websecure
      rule: "Host(`myservice.htsn.io`)"
      service: myservice
      tls:
        certResolver: cloudflare  # Use 'cloudflare' for proxied domains, 'letsencrypt' for DNS-only
      priority: 50

    # HTTP → HTTPS redirect
    myservice-redirect:
      entryPoints:
        - web
      rule: "Host(`myservice.htsn.io`)"
      middlewares:
        - myservice-https-redirect
      service: myservice
      priority: 50

  services:
    myservice:
      loadBalancer:
        servers:
          - url: "http://10.10.10.XXX:PORT"

  middlewares:
    myservice-https-redirect:
      redirectScheme:
        scheme: https
        permanent: true
```

#### Deploy the Config

```bash
# Create file on CT 202
ssh pve 'pct exec 202 -- bash -c "cat > /etc/traefik/conf.d/myservice.yaml << '\''EOF'\''
<paste config here>
EOF"'

# Traefik auto-reloads (watches conf.d directory)
# Check logs:
ssh pve 'pct exec 202 -- tail -f /var/log/traefik/traefik.log'
```

### Step 2: Add Cloudflare DNS Entry

#### Cloudflare Credentials

| Field | Value |
|-------|-------|
| Email | cloudflare@htsn.io |
| API Key | 849ebefd163d2ccdec25e49b3e1b3fe2cdadc |
| Zone ID (htsn.io) | c0f5a80448c608af35d39aa820a5f3af |
| Public IP | 70.237.94.174 |

#### Method 1: Manual (Cloudflare Dashboard)

1. Go to https://dash.cloudflare.com/
2. Select `htsn.io` domain
3. DNS → Add Record
4. Type: `A`, Name: `myservice`, IPv4: `70.237.94.174`, Proxied: ☑️

#### Method 2: Automated (CLI)

Save this as `~/bin/add-cloudflare-dns.sh`:

```bash
#!/bin/bash
# Add DNS record to Cloudflare for htsn.io

SUBDOMAIN="$1"
CF_EMAIL="cloudflare@htsn.io"
CF_API_KEY="849ebefd163d2ccdec25e49b3e1b3fe2cdadc"
ZONE_ID="c0f5a80448c608af35d39aa820a5f3af"
PUBLIC_IP="70.237.94.174"

if [ -z "$SUBDOMAIN" ]; then
  echo "Usage: $0 <subdomain>"
  echo "Example: $0 myservice  # Creates myservice.htsn.io"
  exit 1
fi

curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
  -H "X-Auth-Email: $CF_EMAIL" \
  -H "X-Auth-Key: $CF_API_KEY" \
  -H "Content-Type: application/json" \
  --data "{
    \"type\":\"A\",
    \"name\":\"$SUBDOMAIN\",
    \"content\":\"$PUBLIC_IP\",
    \"ttl\":1,
    \"proxied\":true
  }" | jq .
```

**Usage**:
```bash
chmod +x ~/bin/add-cloudflare-dns.sh
~/bin/add-cloudflare-dns.sh myservice  # Creates myservice.htsn.io
```

### Step 3: Testing

```bash
# Check if DNS resolves
dig myservice.htsn.io

# Should return: 70.237.94.174 (or Cloudflare IPs if proxied)

# Test HTTP redirect
curl -I http://myservice.htsn.io

# Expected: 301 redirect to https://

# Test HTTPS
curl -I https://myservice.htsn.io

# Expected: 200 OK

# Check Traefik dashboard (if enabled)
# http://10.10.10.250:8080/dashboard/
```

### Step 4: Update Documentation

After deploying, update:

1. **IP-ASSIGNMENTS.md** - Add to Services & Reverse Proxy Mapping table
2. **This file (TRAEFIK.md)** - Add to "Services Using Traefik-Primary" list
3. **CLAUDE.md** - Update quick reference if needed

---

## SSL Certificates

Traefik has **two certificate resolvers** configured:

| Resolver | Use When | Challenge Type | Notes |
|----------|----------|----------------|-------|
| `letsencrypt` | Cloudflare DNS-only (gray cloud ☁️) | HTTP-01 | Requires port 80 reachable |
| `cloudflare` | Cloudflare Proxied (orange cloud 🟠) | DNS-01 | Works with Cloudflare proxy |

### ⚠️ Important: HTTP Challenge vs DNS Challenge

**If Cloudflare proxy is enabled** (orange cloud), HTTP challenge **FAILS** because Cloudflare redirects HTTP→HTTPS before the challenge reaches your server.

**Solution**: Use `cloudflare` resolver (DNS-01 challenge) instead.

### Certificate Resolver Configuration

**Cloudflare API credentials** are configured in `/etc/systemd/system/traefik.service`:

```ini
Environment="CF_API_EMAIL=cloudflare@htsn.io"
Environment="CF_API_KEY=849ebefd163d2ccdec25e49b3e1b3fe2cdadc"
```

### Certificate Storage

| Resolver | Storage File |
|----------|--------------|
| HTTP challenge (`letsencrypt`) | `/etc/traefik/acme.json` |
| DNS challenge (`cloudflare`) | `/etc/traefik/acme-cf.json` |

**Permissions**: Must be `600` (read/write owner only)

```bash
# Check permissions
ssh pve 'pct exec 202 -- ls -la /etc/traefik/acme*.json'

# Fix if needed
ssh pve 'pct exec 202 -- chmod 600 /etc/traefik/acme.json'
ssh pve 'pct exec 202 -- chmod 600 /etc/traefik/acme-cf.json'
```

### Certificate Renewal

- **Automatic** via Traefik
- Checks every 24 hours
- Renews 30 days before expiry
- No manual intervention needed

### Troubleshooting Certificates

#### Certificate Fails to Issue

```bash
# Check Traefik logs
ssh pve 'pct exec 202 -- tail -f /var/log/traefik/traefik.log | grep -i error'

# Verify Cloudflare API access
curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
  -H "X-Auth-Email: cloudflare@htsn.io" \
  -H "X-Auth-Key: 849ebefd163d2ccdec25e49b3e1b3fe2cdadc"

# Check acme.json permissions
ssh pve 'pct exec 202 -- ls -la /etc/traefik/acme*.json'
```

#### Force Certificate Renewal

```bash
# Delete certificate (Traefik will re-request)
ssh pve 'pct exec 202 -- rm /etc/traefik/acme-cf.json'
ssh pve 'pct exec 202 -- touch /etc/traefik/acme-cf.json'
ssh pve 'pct exec 202 -- chmod 600 /etc/traefik/acme-cf.json'
ssh pve 'pct exec 202 -- systemctl restart traefik'

# Watch logs
ssh pve 'pct exec 202 -- tail -f /var/log/traefik/traefik.log'
```

---

## Quick Deployment - One-Liner

For fast deployment, use this all-in-one command:

```bash
# === DEPLOY SERVICE (example: myservice on docker-host port 8080) ===

# 1. Create Traefik config
ssh pve 'pct exec 202 -- bash -c "cat > /etc/traefik/conf.d/myservice.yaml << EOF
http:
  routers:
    myservice-secure:
      entryPoints: [websecure]
      rule: Host(\\\`myservice.htsn.io\\\`)
      service: myservice
      tls: {certResolver: cloudflare}
  services:
    myservice:
      loadBalancer:
        servers:
          - url: http://10.10.10.206:8080
EOF"'

# 2. Add Cloudflare DNS
curl -s -X POST "https://api.cloudflare.com/client/v4/zones/c0f5a80448c608af35d39aa820a5f3af/dns_records" \
  -H "X-Auth-Email: cloudflare@htsn.io" \
  -H "X-Auth-Key: 849ebefd163d2ccdec25e49b3e1b3fe2cdadc" \
  -H "Content-Type: application/json" \
  --data '{"type":"A","name":"myservice","content":"70.237.94.174","proxied":true}'

# 3. Test (wait a few seconds for DNS propagation)
curl -I https://myservice.htsn.io
```

---

## Docker Service with Traefik Labels (Alternative)

If deploying a service via Docker on `docker-host` (VM 206), you can use Traefik labels instead of config files.

**Requirements**:
- Traefik must have access to Docker socket
- Service must be on same Docker network as Traefik

**Example docker-compose.yml**:

```yaml
version: "3.8"

services:
  myservice:
    image: myimage:latest
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.myservice.rule=Host(`myservice.htsn.io`)"
      - "traefik.http.routers.myservice.entrypoints=websecure"
      - "traefik.http.routers.myservice.tls.certresolver=letsencrypt"
      - "traefik.http.services.myservice.loadbalancer.server.port=8080"
    networks:
      - traefik

networks:
  traefik:
    external: true
```

**Note**: This method is NOT currently used on Traefik-Primary (CT 202), as it doesn't have Docker socket access. Config files are preferred.

---

## Cloudflare API Reference

### API Credentials

| Field | Value |
|-------|-------|
| Email | cloudflare@htsn.io |
| API Key | 849ebefd163d2ccdec25e49b3e1b3fe2cdadc |
| Zone ID | c0f5a80448c608af35d39aa820a5f3af |

### Common API Operations

Set credentials:
```bash
CF_EMAIL="cloudflare@htsn.io"
CF_API_KEY="849ebefd163d2ccdec25e49b3e1b3fe2cdadc"
ZONE_ID="c0f5a80448c608af35d39aa820a5f3af"
```

**List all DNS records**:
```bash
curl -X GET "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
  -H "X-Auth-Email: $CF_EMAIL" \
  -H "X-Auth-Key: $CF_API_KEY" | jq
```

**Add A record**:
```bash
curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
  -H "X-Auth-Email: $CF_EMAIL" \
  -H "X-Auth-Key: $CF_API_KEY" \
  -H "Content-Type: application/json" \
  --data '{
    "type":"A",
    "name":"subdomain",
    "content":"70.237.94.174",
    "proxied":true
  }'
```

**Delete record**:
```bash
curl -X DELETE "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID" \
  -H "X-Auth-Email: $CF_EMAIL" \
  -H "X-Auth-Key: $CF_API_KEY"
```

**Update record** (toggle proxy):
```bash
curl -X PATCH "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID" \
  -H "X-Auth-Email: $CF_EMAIL" \
  -H "X-Auth-Key: $CF_API_KEY" \
  -H "Content-Type: application/json" \
  --data '{"proxied":false}'
```

---

## Troubleshooting

### Service Not Accessible

```bash
# 1. Check if DNS resolves
dig myservice.htsn.io

# 2. Check if backend is reachable
curl -I http://10.10.10.XXX:PORT

# 3. Check Traefik logs
ssh pve 'pct exec 202 -- tail -f /var/log/traefik/traefik.log'

# 4. Check Traefik config is valid
ssh pve 'pct exec 202 -- cat /etc/traefik/conf.d/myservice.yaml'

# 5. Restart Traefik (if needed)
ssh pve 'pct exec 202 -- systemctl restart traefik'
```

### Certificate Issues

```bash
# Check certificate status in acme.json
ssh pve 'pct exec 202 -- cat /etc/traefik/acme-cf.json | jq'

# Check certificate expiry
echo | openssl s_client -servername myservice.htsn.io -connect myservice.htsn.io:443 2>/dev/null | openssl x509 -noout -dates
```

### 502 Bad Gateway

**Cause**: Backend service is down or unreachable

```bash
# Check if backend is running
ssh backend-host 'systemctl status myservice'

# Check if port is open
nc -zv 10.10.10.XXX PORT

# Check firewall
ssh backend-host 'iptables -L -n | grep PORT'
```

### 404 Not Found

**Cause**: Traefik can't match the request to a router

```bash
# Check router rule matches domain
ssh pve 'pct exec 202 -- cat /etc/traefik/conf.d/myservice.yaml | grep rule'

# Should be: rule: "Host(`myservice.htsn.io`)"

# Check DNS is pointing to correct IP
dig myservice.htsn.io

# Restart Traefik to reload config
ssh pve 'pct exec 202 -- systemctl restart traefik'
```

---

## Advanced Configuration Examples

### WebSocket Support

For services that use WebSockets (like Home Assistant):

```yaml
http:
  routers:
    myservice-secure:
      entryPoints:
        - websecure
      rule: "Host(`myservice.htsn.io`)"
      service: myservice
      tls:
        certResolver: cloudflare

  services:
    myservice:
      loadBalancer:
        servers:
          - url: "http://10.10.10.XXX:PORT"
        # No special config needed - WebSockets work by default in Traefik v2+
```

### Custom Headers

Add custom headers (e.g., security headers):

```yaml
http:
  routers:
    myservice-secure:
      middlewares:
        - myservice-headers

  middlewares:
    myservice-headers:
      headers:
        customResponseHeaders:
          X-Frame-Options: "DENY"
          X-Content-Type-Options: "nosniff"
          Referrer-Policy: "strict-origin-when-cross-origin"
```

### Basic Authentication

Protect a service with basic auth:

```yaml
http:
  routers:
    myservice-secure:
      middlewares:
        - myservice-auth

  middlewares:
    myservice-auth:
      basicAuth:
        users:
          - "user:$apr1$..." # Generate with: htpasswd -nb user password
```

---

## Maintenance

### Monthly Checks

```bash
# Check Traefik status
ssh pve 'pct exec 202 -- systemctl status traefik'

# Review logs for errors
ssh pve 'pct exec 202 -- grep -i error /var/log/traefik/traefik.log | tail -20'

# Check certificate expiry dates
ssh pve 'pct exec 202 -- cat /etc/traefik/acme-cf.json | jq ".cloudflare.Certificates[] | {domain: .domain.main, expiry: .certificate}"'

# Verify all services responding
for domain in plex.htsn.io git.htsn.io truenas.htsn.io; do
  echo "Testing $domain..."
  curl -sI https://$domain | head -1
done
```

### Backup Traefik Config

```bash
# Backup all configs
ssh pve 'pct exec 202 -- tar czf /tmp/traefik-backup-$(date +%Y%m%d).tar.gz /etc/traefik'

# Copy to safe location
scp "pve:/var/lib/lxc/202/rootfs/tmp/traefik-backup-*.tar.gz" ~/Backups/traefik/
```

---

## Related Documentation

- [IP-ASSIGNMENTS.md](IP-ASSIGNMENTS.md) - Service IP addresses
- [CLOUDFLARE.md](#) - Cloudflare DNS management (coming soon)
- [SERVICES.md](#) - Complete service inventory (coming soon)

---

**Last Updated**: 2025-12-22