homelab-docs/SSH-ACCESS.md

# SSH Access

Documentation for SSH access to all homelab systems, including key authentication, password authentication for special cases, and QEMU guest agent usage.

## Overview

Most systems use **SSH key authentication** with the `~/.ssh/homelab` key. A few special cases require **password authentication** (router, Windows PC) due to platform limitations.

**SSH Password**: `GrilledCh33s3#` (for systems without key auth)

---

## SSH Key Authentication (Primary Method)

### SSH Key Configuration

SSH keys are configured in `~/.ssh/config` on both Mac Mini and MacBook.

**Key file**: `~/.ssh/homelab` (Ed25519 key)

**Key deployed to**: All Proxmox hosts, VMs, and LXCs (13 total hosts)

### Host Aliases

Use these convenient aliases instead of IP addresses:

| Host Alias | IP | User | Type | Notes |
|------------|-----|------|------|-------|
| `pve` | 10.10.10.120 | root | Proxmox | Primary server |
| `pve2` | 10.10.10.102 | root | Proxmox | Secondary server |
| `truenas` | 10.10.10.200 | root | VM | NAS/storage |
| `saltbox` | 10.10.10.100 | hutson | VM | Media automation |
| `lmdev1` | 10.10.10.111 | hutson | VM | AI/LLM development |
| `docker-host` | 10.10.10.206 | hutson | VM | Docker services |
| `fs-dev` | 10.10.10.5 | hutson | VM | Development |
| `copyparty` | 10.10.10.201 | hutson | VM | File sharing |
| `gitea-vm` | 10.10.10.220 | hutson | VM | Git server |
| `trading-vm` | 10.10.10.221 | hutson | VM | AI trading platform |
| `pihole` | 10.10.10.10 | root | LXC | DNS/Ad blocking |
| `traefik` | 10.10.10.250 | root | LXC | Reverse proxy |
| `findshyt` | 10.10.10.8 | root | LXC | Custom app |

### Usage Examples

```bash
# List VMs on PVE
ssh pve 'qm list'

# Check ZFS pool on TrueNAS
ssh truenas 'zpool status vault'

# List Docker containers on Saltbox
ssh saltbox 'docker ps'

# Check Pi-hole status
ssh pihole 'pihole status'

# View Traefik config
ssh pve 'pct exec 202 -- cat /etc/traefik/traefik.yaml'
```

### SSH Config File

**Location**: `~/.ssh/config`

**Example entries**:

```sshconfig
# Proxmox Servers
Host pve
    HostName 10.10.10.120
    User root
    IdentityFile ~/.ssh/homelab

Host pve2
    HostName 10.10.10.102
    User root
    IdentityFile ~/.ssh/homelab
    # Post-quantum KEX causes MTU issues - use classic
    KexAlgorithms curve25519-sha256

# VMs
Host truenas
    HostName 10.10.10.200
    User root
    IdentityFile ~/.ssh/homelab

Host saltbox
    HostName 10.10.10.100
    User hutson
    IdentityFile ~/.ssh/homelab

Host lmdev1
    HostName 10.10.10.111
    User hutson
    IdentityFile ~/.ssh/homelab

Host docker-host
    HostName 10.10.10.206
    User hutson
    IdentityFile ~/.ssh/homelab

Host fs-dev
    HostName 10.10.10.5
    User hutson
    IdentityFile ~/.ssh/homelab

Host copyparty
    HostName 10.10.10.201
    User hutson
    IdentityFile ~/.ssh/homelab

Host gitea-vm
    HostName 10.10.10.220
    User hutson
    IdentityFile ~/.ssh/homelab

Host trading-vm
    HostName 10.10.10.221
    User hutson
    IdentityFile ~/.ssh/homelab

# LXC Containers
Host pihole
    HostName 10.10.10.10
    User root
    IdentityFile ~/.ssh/homelab

Host traefik
    HostName 10.10.10.250
    User root
    IdentityFile ~/.ssh/homelab

Host findshyt
    HostName 10.10.10.8
    User root
    IdentityFile ~/.ssh/homelab
```

---

## Password Authentication (Special Cases)

Some systems don't support SSH key auth or have other limitations.

### UniFi Router (10.10.10.1)

**Issue**: Uses `keyboard-interactive` auth method, incompatible with `sshpass`
**Solution**: Use `expect` to automate password entry

**Commands**:

```bash
# Run command on router
expect -c 'spawn ssh root@10.10.10.1 "hostname"; expect "Password:"; send "GrilledCh33s3#\r"; expect eof'

# Get ARP table (all device IPs)
expect -c 'spawn ssh root@10.10.10.1 "cat /proc/net/arp"; expect "Password:"; send "GrilledCh33s3#\r"; expect eof'

# Check Tailscale status
expect -c 'spawn ssh root@10.10.10.1 "tailscale status"; expect "Password:"; send "GrilledCh33s3#\r"; expect eof'
```

**Why not key auth?**: UniFi router firmware doesn't persist SSH keys across reboots.

### Windows PC (10.10.10.150)

**OS**: Windows with OpenSSH server
**User**: `claude`
**Password**: `GrilledCh33s3#`
**Shell**: PowerShell (not bash)

**Commands**:

```bash
# Run PowerShell command
sshpass -p 'GrilledCh33s3#' ssh claude@10.10.10.150 'Get-Process | Select -First 5'

# Check Syncthing status
sshpass -p 'GrilledCh33s3#' ssh claude@10.10.10.150 'Get-Process -Name syncthing -ErrorAction SilentlyContinue'

# Restart Syncthing
sshpass -p 'GrilledCh33s3#' ssh claude@10.10.10.150 'Stop-Process -Name syncthing -Force; Start-ScheduledTask -TaskName "Syncthing"'
```

**⚠️ Important**: Use `;` (semicolon) to chain PowerShell commands, NOT `&&` (bash syntax).

**Why not key auth?**: Could be configured, but password auth works and is simpler for Windows.

---

## QEMU Guest Agent

Most VMs have the QEMU guest agent installed, allowing command execution without SSH.

### VMs with QEMU Agent

| VMID | VM Name | Use Case |
|------|---------|----------|
| 100 | truenas | Execute commands, check ZFS |
| 101 | saltbox | Execute commands, Docker mgmt |
| 105 | fs-dev | Execute commands |
| 111 | lmdev1 | Execute commands |
| 201 | copyparty | Execute commands |
| 206 | docker-host | Execute commands |
| 300 | gitea-vm | Execute commands |
| 301 | trading-vm | Execute commands |

### VM WITHOUT QEMU Agent

**VMID 110 (homeassistant)**: No QEMU agent installed
- Access via web UI only
- Or install SSH server manually if needed

### Usage Examples

**Basic syntax**:
```bash
ssh pve 'qm guest exec VMID -- bash -c "COMMAND"'
```

**Examples**:

```bash
# Check ZFS pool on TrueNAS (without SSH)
ssh pve 'qm guest exec 100 -- bash -c "zpool status vault"'

# Get VM IP addresses
ssh pve 'qm guest exec 100 -- bash -c "ip addr"'

# Check Docker containers on Saltbox
ssh pve 'qm guest exec 101 -- bash -c "docker ps"'

# Run multi-line command
ssh pve 'qm guest exec 100 -- bash -c "df -h; free -h; uptime"'
```

**When to use QEMU agent vs SSH**:
- ✅ Use **SSH** for interactive sessions, file editing, complex tasks
- ✅ Use **QEMU agent** for one-off commands, when SSH is down, or VM has no network
- ⚠️ QEMU agent is slower for multiple commands (use SSH instead)

---

## Troubleshooting SSH Issues

### Connection Refused

```bash
# Check if SSH service is running
ssh pve 'systemctl status sshd'

# Check if port 22 is open
nc -zv 10.10.10.XXX 22

# Check firewall
ssh pve 'iptables -L -n | grep 22'
```

### Permission Denied (Public Key)

```bash
# Verify key file exists
ls -la ~/.ssh/homelab

# Check key permissions (should be 600)
chmod 600 ~/.ssh/homelab

# Test SSH key auth verbosely
ssh -vvv -i ~/.ssh/homelab root@10.10.10.120

# Check authorized_keys on remote (via QEMU agent if SSH broken)
ssh pve 'qm guest exec VMID -- bash -c "cat ~/.ssh/authorized_keys"'
```

### Slow SSH Connection (PVE2 Issue)

**Problem**: SSH to PVE2 hangs for 30+ seconds before connecting
**Cause**: MTU mismatch (vmbr0=9000, nic1=1500) causing post-quantum KEX packet fragmentation
**Fix**: Use classic KEX algorithm instead

**In `~/.ssh/config`**:
```sshconfig
Host pve2
    HostName 10.10.10.102
    User root
    IdentityFile ~/.ssh/homelab
    KexAlgorithms curve25519-sha256  # Avoid mlkem768x25519-sha256
```

**Permanent fix**: Set `nic1` MTU to 9000 in `/etc/network/interfaces` on PVE2

---

## Adding SSH Keys to New Systems

### Linux (VMs/LXCs)

```bash
# Copy public key to new host
ssh-copy-id -i ~/.ssh/homelab user@hostname

# Or manually:
ssh user@hostname 'mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys' < ~/.ssh/homelab.pub
ssh user@hostname 'chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys'
```

### LXC Containers (Root User)

```bash
# Via pct exec from Proxmox host
ssh pve 'pct exec CTID -- bash -c "mkdir -p /root/.ssh"'
ssh pve 'pct exec CTID -- bash -c "echo \"$(cat ~/.ssh/homelab.pub)\" >> /root/.ssh/authorized_keys"'
ssh pve 'pct exec CTID -- bash -c "chmod 700 /root/.ssh && chmod 600 /root/.ssh/authorized_keys"'

# Also enable PermitRootLogin in sshd_config
ssh pve 'pct exec CTID -- bash -c "sed -i \"s/^#*PermitRootLogin.*/PermitRootLogin prohibit-password/\" /etc/ssh/sshd_config"'
ssh pve 'pct exec CTID -- bash -c "systemctl restart sshd"'
```

### VMs (via QEMU Agent)

```bash
# Add key via QEMU agent (if SSH not working)
ssh pve 'qm guest exec VMID -- bash -c "mkdir -p ~/.ssh"'
ssh pve 'qm guest exec VMID -- bash -c "echo \"$(cat ~/.ssh/homelab.pub)\" >> ~/.ssh/authorized_keys"'
ssh pve 'qm guest exec VMID -- bash -c "chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys"'
```

---

## SSH Key Management

### Rotate SSH Keys (Future)

When rotating SSH keys:

1. Generate new key pair:
   ```bash
   ssh-keygen -t ed25519 -f ~/.ssh/homelab-new -C "homelab-new"
   ```

2. Deploy new key to all hosts (keep old key for now):
   ```bash
   for host in pve pve2 truenas saltbox lmdev1 docker-host fs-dev copyparty gitea-vm trading-vm pihole traefik findshyt; do
     ssh-copy-id -i ~/.ssh/homelab-new $host
   done
   ```

3. Update `~/.ssh/config` to use new key:
   ```sshconfig
   IdentityFile ~/.ssh/homelab-new
   ```

4. Test all connections:
   ```bash
   for host in pve pve2 truenas saltbox lmdev1 docker-host fs-dev copyparty gitea-vm trading-vm pihole traefik findshyt; do
     echo "Testing $host..."
     ssh $host 'hostname'
   done
   ```

5. Remove old key from all hosts once confirmed working

---

## Quick Reference

### Common SSH Operations

```bash
# Execute command on remote host
ssh host 'command'

# Execute multiple commands
ssh host 'command1 && command2'

# Copy file to remote
scp file host:/path/

# Copy file from remote
scp host:/path/file ./

# Execute command on Proxmox VM (via QEMU agent)
ssh pve 'qm guest exec VMID -- bash -c "command"'

# Execute command on LXC
ssh pve 'pct exec CTID -- command'

# Interactive shell
ssh host

# SSH with X11 forwarding
ssh -X host
```

### Troubleshooting Commands

```bash
# Test SSH with verbose output
ssh -vvv host

# Check SSH service status (remote)
ssh host 'systemctl status sshd'

# Check SSH config (local)
ssh -G host

# Test port connectivity
nc -zv hostname 22
```

---

## Security Best Practices

### Current Security Posture

✅ **Good**:
- SSH keys used instead of passwords (where possible)
- Keys use Ed25519 (modern, secure algorithm)
- Root login disabled on VMs (use sudo instead)
- SSH keys have proper permissions (600)

⚠️ **Could Improve**:
- [ ] Disable password authentication on all hosts (force key-only)
- [ ] Use SSH certificate authority instead of individual keys
- [ ] Set up SSH bastion host (jump server)
- [ ] Enable 2FA for SSH (via PAM + Google Authenticator)
- [ ] Implement SSH key rotation policy (annually)

### Hardening SSH (Future)

For additional security, consider:

```sshconfig
# /etc/ssh/sshd_config (on remote hosts)
PermitRootLogin prohibit-password  # No root password login
PasswordAuthentication no          # Disable password auth entirely
PubkeyAuthentication yes           # Only allow key auth
AuthorizedKeysFile .ssh/authorized_keys
MaxAuthTries 3                     # Limit auth attempts
MaxSessions 10                     # Limit concurrent sessions
ClientAliveInterval 300            # Timeout idle sessions
ClientAliveCountMax 2              # Drop after 2 keepalives
```

**Apply after editing**:
```bash
systemctl restart sshd
```

---

## Related Documentation

- [VMS.md](VMS.md) - Complete VM/CT inventory
- [NETWORK.md](NETWORK.md) - Network configuration
- [IP-ASSIGNMENTS.md](IP-ASSIGNMENTS.md) - IP addresses for all hosts
- [SECURITY.md](#) - Security policies (coming soon)

---

**Last Updated**: 2025-12-22