Complete Phase 2 documentation: Add HARDWARE, SERVICES, MONITORING, MAINTENANCE

Phase 2 documentation implementation: - Created HARDWARE.md: Complete hardware inventory (servers, GPUs, storage, network cards) - Created SERVICES.md: Service inventory with URLs, credentials, health checks (25+ services) - Created MONITORING.md: Health monitoring recommendations, alert setup, implementation plan - Created MAINTENANCE.md: Regular procedures, update schedules, testing checklists - Updated README.md: Added all Phase 2 documentation links - Updated CLAUDE.md: Cleaned up to quick reference only (1340→377 lines) All detailed content now in specialized documentation files with cross-references. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-23 00:34:21 -05:00
parent 23e9df68c9
commit 56b82df497
14 changed files with 6328 additions and 1036 deletions
--- a/TRAEFIK.md
+++ b/TRAEFIK.md
@@ -0,0 +1,672 @@
+# Traefik Reverse Proxy
+
+Documentation for Traefik reverse proxy setup, SSL certificates, and deploying new public services.
+
+## Overview
+
+There are **TWO separate Traefik instances** handling different services. Understanding which one to use is critical.
+
+| Instance | Location | IP | Purpose | Managed By |
+|----------|----------|-----|---------|------------|
+| **Traefik-Primary** | CT 202 | **10.10.10.250** | General services | Manual config files |
+| **Traefik-Saltbox** | VM 101 (Docker) | **10.10.10.100** | Saltbox services only | Saltbox Ansible |
+
+---
+
+## ⚠️ CRITICAL RULE: Which Traefik to Use
+
+### When Adding ANY New Service:
+
+✅ **USE Traefik-Primary (CT 202 @ 10.10.10.250)** - For ALL new services
+❌ **DO NOT touch Traefik-Saltbox** - Unless you're modifying Saltbox itself
+
+### Why This Matters:
+
+- **Traefik-Saltbox** has complex Saltbox-managed configs (Ansible-generated)
+- Messing with it breaks Plex, Sonarr, Radarr, and all media services
+- Each Traefik has its own Let's Encrypt certificates
+- Mixing them causes certificate conflicts and routing issues
+
+---
+
+## Traefik-Primary (CT 202) - For New Services
+
+### Configuration
+
+**Location**: Container 202 on PVE (10.10.10.250)
+**Config Directory**: `/etc/traefik/`
+**Main Config**: `/etc/traefik/traefik.yaml`
+**Dynamic Configs**: `/etc/traefik/conf.d/*.yaml`
+
+### Access Traefik Config
+
+```bash
+# From Mac Mini:
+ssh pve 'pct exec 202 -- cat /etc/traefik/traefik.yaml'
+ssh pve 'pct exec 202 -- ls /etc/traefik/conf.d/'
+
+# Edit a service config:
+ssh pve 'pct exec 202 -- vi /etc/traefik/conf.d/myservice.yaml'
+
+# View logs:
+ssh pve 'pct exec 202 -- tail -f /var/log/traefik/traefik.log'
+```
+
+### Services Using Traefik-Primary
+
+| Service | Domain | Backend |
+|---------|--------|---------|
+| Excalidraw | excalidraw.htsn.io | 10.10.10.206:8080 (docker-host) |
+| FindShyt | findshyt.htsn.io | 10.10.10.205 (CT 205) |
+| Gitea | git.htsn.io | 10.10.10.220:3000 |
+| Home Assistant | homeassistant.htsn.io | 10.10.10.110 |
+| LM Dev | lmdev.htsn.io | 10.10.10.111 |
+| Pi-hole | pihole.htsn.io | 10.10.10.200 |
+| TrueNAS | truenas.htsn.io | 10.10.10.200 |
+| Proxmox | pve.htsn.io | 10.10.10.120 |
+| Copyparty | copyparty.htsn.io | 10.10.10.201 |
+| AI Trade | aitrade.htsn.io | (trading server) |
+| Pulse | pulse.htsn.io | 10.10.10.206:7655 (monitoring) |
+| Happy | happy.htsn.io | 10.10.10.206:3002 (Happy Coder relay) |
+
+---
+
+## Traefik-Saltbox (VM 101) - DO NOT MODIFY
+
+### Configuration
+
+**Location**: `/opt/traefik/` inside Saltbox VM
+**Managed By**: Saltbox Ansible playbooks (automatic)
+**Docker Mount**: `/opt/traefik` → `/etc/traefik` in container
+
+### Services Using Traefik-Saltbox
+
+- Plex (plex.htsn.io)
+- Sonarr, Radarr, Lidarr
+- SABnzbd, NZBGet, qBittorrent
+- Overseerr, Tautulli, Organizr
+- Jackett, NZBHydra2
+- Authelia (SSO authentication)
+- All other Saltbox-managed containers
+
+### View Saltbox Traefik (Read-Only)
+
+```bash
+# View config (don't edit!)
+ssh pve 'qm guest exec 101 -- bash -c "docker exec traefik cat /etc/traefik/traefik.yml"'
+
+# View logs
+ssh saltbox 'docker logs -f traefik'
+```
+
+**⚠️ WARNING**: Editing Saltbox Traefik configs manually will be overwritten by Ansible and may break media services.
+
+---
+
+## Adding a New Public Service - Complete Workflow
+
+Follow these steps to deploy a new service and make it accessible at `servicename.htsn.io`.
+
+### Step 0: Deploy Your Service
+
+First, deploy your service on the appropriate host.
+
+#### Option A: Docker on docker-host (10.10.10.206)
+
+```bash
+ssh hutson@10.10.10.206
+sudo mkdir -p /opt/myservice
+cat > /opt/myservice/docker-compose.yml << 'EOF'
+version: "3.8"
+services:
+  myservice:
+    image: myimage:latest
+    ports:
+      - "8080:80"
+    restart: unless-stopped
+EOF
+cd /opt/myservice && sudo docker-compose up -d
+```
+
+#### Option B: New LXC Container on PVE
+
+```bash
+ssh pve 'pct create CTID local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
+  --hostname myservice --memory 2048 --cores 2 \
+  --net0 name=eth0,bridge=vmbr0,ip=10.10.10.XXX/24,gw=10.10.10.1 \
+  --rootfs local-zfs:8 --unprivileged 1 --start 1'
+```
+
+#### Option C: New VM on PVE
+
+```bash
+ssh pve 'qm create VMID --name myservice --memory 2048 --cores 2 \
+  --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-pci'
+```
+
+### Step 1: Create Traefik Config File
+
+Use this template for new services on **Traefik-Primary (CT 202)**:
+
+#### Basic Template
+
+```yaml
+# /etc/traefik/conf.d/myservice.yaml
+http:
+  routers:
+    # HTTPS router
+    myservice-secure:
+      entryPoints:
+        - websecure
+      rule: "Host(`myservice.htsn.io`)"
+      service: myservice
+      tls:
+        certResolver: cloudflare  # Use 'cloudflare' for proxied domains, 'letsencrypt' for DNS-only
+      priority: 50
+
+    # HTTP → HTTPS redirect
+    myservice-redirect:
+      entryPoints:
+        - web
+      rule: "Host(`myservice.htsn.io`)"
+      middlewares:
+        - myservice-https-redirect
+      service: myservice
+      priority: 50
+
+  services:
+    myservice:
+      loadBalancer:
+        servers:
+          - url: "http://10.10.10.XXX:PORT"
+
+  middlewares:
+    myservice-https-redirect:
+      redirectScheme:
+        scheme: https
+        permanent: true
+```
+
+#### Deploy the Config
+
+```bash
+# Create file on CT 202
+ssh pve 'pct exec 202 -- bash -c "cat > /etc/traefik/conf.d/myservice.yaml << '\''EOF'\''
+<paste config here>
+EOF"'
+
+# Traefik auto-reloads (watches conf.d directory)
+# Check logs:
+ssh pve 'pct exec 202 -- tail -f /var/log/traefik/traefik.log'
+```
+
+### Step 2: Add Cloudflare DNS Entry
+
+#### Cloudflare Credentials
+
+| Field | Value |
+|-------|-------|
+| Email | cloudflare@htsn.io |
+| API Key | 849ebefd163d2ccdec25e49b3e1b3fe2cdadc |
+| Zone ID (htsn.io) | c0f5a80448c608af35d39aa820a5f3af |
+| Public IP | 70.237.94.174 |
+
+#### Method 1: Manual (Cloudflare Dashboard)
+
+1. Go to https://dash.cloudflare.com/
+2. Select `htsn.io` domain
+3. DNS → Add Record
+4. Type: `A`, Name: `myservice`, IPv4: `70.237.94.174`, Proxied: ☑️
+
+#### Method 2: Automated (CLI)
+
+Save this as `~/bin/add-cloudflare-dns.sh`:
+
+```bash
+#!/bin/bash
+# Add DNS record to Cloudflare for htsn.io
+
+SUBDOMAIN="$1"
+CF_EMAIL="cloudflare@htsn.io"
+CF_API_KEY="849ebefd163d2ccdec25e49b3e1b3fe2cdadc"
+ZONE_ID="c0f5a80448c608af35d39aa820a5f3af"
+PUBLIC_IP="70.237.94.174"
+
+if [ -z "$SUBDOMAIN" ]; then
+  echo "Usage: $0 <subdomain>"
+  echo "Example: $0 myservice  # Creates myservice.htsn.io"
+  exit 1
+fi
+
+curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
+  -H "X-Auth-Email: $CF_EMAIL" \
+  -H "X-Auth-Key: $CF_API_KEY" \
+  -H "Content-Type: application/json" \
+  --data "{
+    \"type\":\"A\",
+    \"name\":\"$SUBDOMAIN\",
+    \"content\":\"$PUBLIC_IP\",
+    \"ttl\":1,
+    \"proxied\":true
+  }" | jq .
+```
+
+**Usage**:
+```bash
+chmod +x ~/bin/add-cloudflare-dns.sh
+~/bin/add-cloudflare-dns.sh myservice  # Creates myservice.htsn.io
+```
+
+### Step 3: Testing
+
+```bash
+# Check if DNS resolves
+dig myservice.htsn.io
+
+# Should return: 70.237.94.174 (or Cloudflare IPs if proxied)
+
+# Test HTTP redirect
+curl -I http://myservice.htsn.io
+
+# Expected: 301 redirect to https://
+
+# Test HTTPS
+curl -I https://myservice.htsn.io
+
+# Expected: 200 OK
+
+# Check Traefik dashboard (if enabled)
+# http://10.10.10.250:8080/dashboard/
+```
+
+### Step 4: Update Documentation
+
+After deploying, update:
+
+1. **IP-ASSIGNMENTS.md** - Add to Services & Reverse Proxy Mapping table
+2. **This file (TRAEFIK.md)** - Add to "Services Using Traefik-Primary" list
+3. **CLAUDE.md** - Update quick reference if needed
+
+---
+
+## SSL Certificates
+
+Traefik has **two certificate resolvers** configured:
+
+| Resolver | Use When | Challenge Type | Notes |
+|----------|----------|----------------|-------|
+| `letsencrypt` | Cloudflare DNS-only (gray cloud ☁️) | HTTP-01 | Requires port 80 reachable |
+| `cloudflare` | Cloudflare Proxied (orange cloud 🟠) | DNS-01 | Works with Cloudflare proxy |
+
+### ⚠️ Important: HTTP Challenge vs DNS Challenge
+
+**If Cloudflare proxy is enabled** (orange cloud), HTTP challenge **FAILS** because Cloudflare redirects HTTP→HTTPS before the challenge reaches your server.
+
+**Solution**: Use `cloudflare` resolver (DNS-01 challenge) instead.
+
+### Certificate Resolver Configuration
+
+**Cloudflare API credentials** are configured in `/etc/systemd/system/traefik.service`:
+
+```ini
+Environment="CF_API_EMAIL=cloudflare@htsn.io"
+Environment="CF_API_KEY=849ebefd163d2ccdec25e49b3e1b3fe2cdadc"
+```
+
+### Certificate Storage
+
+| Resolver | Storage File |
+|----------|--------------|
+| HTTP challenge (`letsencrypt`) | `/etc/traefik/acme.json` |
+| DNS challenge (`cloudflare`) | `/etc/traefik/acme-cf.json` |
+
+**Permissions**: Must be `600` (read/write owner only)
+
+```bash
+# Check permissions
+ssh pve 'pct exec 202 -- ls -la /etc/traefik/acme*.json'
+
+# Fix if needed
+ssh pve 'pct exec 202 -- chmod 600 /etc/traefik/acme.json'
+ssh pve 'pct exec 202 -- chmod 600 /etc/traefik/acme-cf.json'
+```
+
+### Certificate Renewal
+
+- **Automatic** via Traefik
+- Checks every 24 hours
+- Renews 30 days before expiry
+- No manual intervention needed
+
+### Troubleshooting Certificates
+
+#### Certificate Fails to Issue
+
+```bash
+# Check Traefik logs
+ssh pve 'pct exec 202 -- tail -f /var/log/traefik/traefik.log | grep -i error'
+
+# Verify Cloudflare API access
+curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
+  -H "X-Auth-Email: cloudflare@htsn.io" \
+  -H "X-Auth-Key: 849ebefd163d2ccdec25e49b3e1b3fe2cdadc"
+
+# Check acme.json permissions
+ssh pve 'pct exec 202 -- ls -la /etc/traefik/acme*.json'
+```
+
+#### Force Certificate Renewal
+
+```bash
+# Delete certificate (Traefik will re-request)
+ssh pve 'pct exec 202 -- rm /etc/traefik/acme-cf.json'
+ssh pve 'pct exec 202 -- touch /etc/traefik/acme-cf.json'
+ssh pve 'pct exec 202 -- chmod 600 /etc/traefik/acme-cf.json'
+ssh pve 'pct exec 202 -- systemctl restart traefik'
+
+# Watch logs
+ssh pve 'pct exec 202 -- tail -f /var/log/traefik/traefik.log'
+```
+
+---
+
+## Quick Deployment - One-Liner
+
+For fast deployment, use this all-in-one command:
+
+```bash
+# === DEPLOY SERVICE (example: myservice on docker-host port 8080) ===
+
+# 1. Create Traefik config
+ssh pve 'pct exec 202 -- bash -c "cat > /etc/traefik/conf.d/myservice.yaml << EOF
+http:
+  routers:
+    myservice-secure:
+      entryPoints: [websecure]
+      rule: Host(\\\`myservice.htsn.io\\\`)
+      service: myservice
+      tls: {certResolver: cloudflare}
+  services:
+    myservice:
+      loadBalancer:
+        servers:
+          - url: http://10.10.10.206:8080
+EOF"'
+
+# 2. Add Cloudflare DNS
+curl -s -X POST "https://api.cloudflare.com/client/v4/zones/c0f5a80448c608af35d39aa820a5f3af/dns_records" \
+  -H "X-Auth-Email: cloudflare@htsn.io" \
+  -H "X-Auth-Key: 849ebefd163d2ccdec25e49b3e1b3fe2cdadc" \
+  -H "Content-Type: application/json" \
+  --data '{"type":"A","name":"myservice","content":"70.237.94.174","proxied":true}'
+
+# 3. Test (wait a few seconds for DNS propagation)
+curl -I https://myservice.htsn.io
+```
+
+---
+
+## Docker Service with Traefik Labels (Alternative)
+
+If deploying a service via Docker on `docker-host` (VM 206), you can use Traefik labels instead of config files.
+
+**Requirements**:
+- Traefik must have access to Docker socket
+- Service must be on same Docker network as Traefik
+
+**Example docker-compose.yml**:
+
+```yaml
+version: "3.8"
+
+services:
+  myservice:
+    image: myimage:latest
+    labels:
+      - "traefik.enable=true"
+      - "traefik.http.routers.myservice.rule=Host(`myservice.htsn.io`)"
+      - "traefik.http.routers.myservice.entrypoints=websecure"
+      - "traefik.http.routers.myservice.tls.certresolver=letsencrypt"
+      - "traefik.http.services.myservice.loadbalancer.server.port=8080"
+    networks:
+      - traefik
+
+networks:
+  traefik:
+    external: true
+```
+
+**Note**: This method is NOT currently used on Traefik-Primary (CT 202), as it doesn't have Docker socket access. Config files are preferred.
+
+---
+
+## Cloudflare API Reference
+
+### API Credentials
+
+| Field | Value |
+|-------|-------|
+| Email | cloudflare@htsn.io |
+| API Key | 849ebefd163d2ccdec25e49b3e1b3fe2cdadc |
+| Zone ID | c0f5a80448c608af35d39aa820a5f3af |
+
+### Common API Operations
+
+Set credentials:
+```bash
+CF_EMAIL="cloudflare@htsn.io"
+CF_API_KEY="849ebefd163d2ccdec25e49b3e1b3fe2cdadc"
+ZONE_ID="c0f5a80448c608af35d39aa820a5f3af"
+```
+
+**List all DNS records**:
+```bash
+curl -X GET "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
+  -H "X-Auth-Email: $CF_EMAIL" \
+  -H "X-Auth-Key: $CF_API_KEY" | jq
+```
+
+**Add A record**:
+```bash
+curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
+  -H "X-Auth-Email: $CF_EMAIL" \
+  -H "X-Auth-Key: $CF_API_KEY" \
+  -H "Content-Type: application/json" \
+  --data '{
+    "type":"A",
+    "name":"subdomain",
+    "content":"70.237.94.174",
+    "proxied":true
+  }'
+```
+
+**Delete record**:
+```bash
+curl -X DELETE "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID" \
+  -H "X-Auth-Email: $CF_EMAIL" \
+  -H "X-Auth-Key: $CF_API_KEY"
+```
+
+**Update record** (toggle proxy):
+```bash
+curl -X PATCH "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID" \
+  -H "X-Auth-Email: $CF_EMAIL" \
+  -H "X-Auth-Key: $CF_API_KEY" \
+  -H "Content-Type: application/json" \
+  --data '{"proxied":false}'
+```
+
+---
+
+## Troubleshooting
+
+### Service Not Accessible
+
+```bash
+# 1. Check if DNS resolves
+dig myservice.htsn.io
+
+# 2. Check if backend is reachable
+curl -I http://10.10.10.XXX:PORT
+
+# 3. Check Traefik logs
+ssh pve 'pct exec 202 -- tail -f /var/log/traefik/traefik.log'
+
+# 4. Check Traefik config is valid
+ssh pve 'pct exec 202 -- cat /etc/traefik/conf.d/myservice.yaml'
+
+# 5. Restart Traefik (if needed)
+ssh pve 'pct exec 202 -- systemctl restart traefik'
+```
+
+### Certificate Issues
+
+```bash
+# Check certificate status in acme.json
+ssh pve 'pct exec 202 -- cat /etc/traefik/acme-cf.json | jq'
+
+# Check certificate expiry
+echo | openssl s_client -servername myservice.htsn.io -connect myservice.htsn.io:443 2>/dev/null | openssl x509 -noout -dates
+```
+
+### 502 Bad Gateway
+
+**Cause**: Backend service is down or unreachable
+
+```bash
+# Check if backend is running
+ssh backend-host 'systemctl status myservice'
+
+# Check if port is open
+nc -zv 10.10.10.XXX PORT
+
+# Check firewall
+ssh backend-host 'iptables -L -n | grep PORT'
+```
+
+### 404 Not Found
+
+**Cause**: Traefik can't match the request to a router
+
+```bash
+# Check router rule matches domain
+ssh pve 'pct exec 202 -- cat /etc/traefik/conf.d/myservice.yaml | grep rule'
+
+# Should be: rule: "Host(`myservice.htsn.io`)"
+
+# Check DNS is pointing to correct IP
+dig myservice.htsn.io
+
+# Restart Traefik to reload config
+ssh pve 'pct exec 202 -- systemctl restart traefik'
+```
+
+---
+
+## Advanced Configuration Examples
+
+### WebSocket Support
+
+For services that use WebSockets (like Home Assistant):
+
+```yaml
+http:
+  routers:
+    myservice-secure:
+      entryPoints:
+        - websecure
+      rule: "Host(`myservice.htsn.io`)"
+      service: myservice
+      tls:
+        certResolver: cloudflare
+
+  services:
+    myservice:
+      loadBalancer:
+        servers:
+          - url: "http://10.10.10.XXX:PORT"
+        # No special config needed - WebSockets work by default in Traefik v2+
+```
+
+### Custom Headers
+
+Add custom headers (e.g., security headers):
+
+```yaml
+http:
+  routers:
+    myservice-secure:
+      middlewares:
+        - myservice-headers
+
+  middlewares:
+    myservice-headers:
+      headers:
+        customResponseHeaders:
+          X-Frame-Options: "DENY"
+          X-Content-Type-Options: "nosniff"
+          Referrer-Policy: "strict-origin-when-cross-origin"
+```
+
+### Basic Authentication
+
+Protect a service with basic auth:
+
+```yaml
+http:
+  routers:
+    myservice-secure:
+      middlewares:
+        - myservice-auth
+
+  middlewares:
+    myservice-auth:
+      basicAuth:
+        users:
+          - "user:$apr1$..." # Generate with: htpasswd -nb user password
+```
+
+---
+
+## Maintenance
+
+### Monthly Checks
+
+```bash
+# Check Traefik status
+ssh pve 'pct exec 202 -- systemctl status traefik'
+
+# Review logs for errors
+ssh pve 'pct exec 202 -- grep -i error /var/log/traefik/traefik.log | tail -20'
+
+# Check certificate expiry dates
+ssh pve 'pct exec 202 -- cat /etc/traefik/acme-cf.json | jq ".cloudflare.Certificates[] | {domain: .domain.main, expiry: .certificate}"'
+
+# Verify all services responding
+for domain in plex.htsn.io git.htsn.io truenas.htsn.io; do
+  echo "Testing $domain..."
+  curl -sI https://$domain | head -1
+done
+```
+
+### Backup Traefik Config
+
+```bash
+# Backup all configs
+ssh pve 'pct exec 202 -- tar czf /tmp/traefik-backup-$(date +%Y%m%d).tar.gz /etc/traefik'
+
+# Copy to safe location
+scp "pve:/var/lib/lxc/202/rootfs/tmp/traefik-backup-*.tar.gz" ~/Backups/traefik/
+```
+
+---
+
+## Related Documentation
+
+- [IP-ASSIGNMENTS.md](IP-ASSIGNMENTS.md) - Service IP addresses
+- [CLOUDFLARE.md](#) - Cloudflare DNS management (coming soon)
+- [SERVICES.md](#) - Complete service inventory (coming soon)
+
+---
+
+**Last Updated**: 2025-12-22