Complete Phase 2 documentation: Add HARDWARE, SERVICES, MONITORING, MAINTENANCE
Phase 2 documentation implementation: - Created HARDWARE.md: Complete hardware inventory (servers, GPUs, storage, network cards) - Created SERVICES.md: Service inventory with URLs, credentials, health checks (25+ services) - Created MONITORING.md: Health monitoring recommendations, alert setup, implementation plan - Created MAINTENANCE.md: Regular procedures, update schedules, testing checklists - Updated README.md: Added all Phase 2 documentation links - Updated CLAUDE.md: Cleaned up to quick reference only (1340→377 lines) All detailed content now in specialized documentation files with cross-references. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Hutson
464
SSH-ACCESS.md
Normal file
464
SSH-ACCESS.md
Normal file
@@ -0,0 +1,464 @@
|
||||
# SSH Access
|
||||
|
||||
Documentation for SSH access to all homelab systems, including key authentication, password authentication for special cases, and QEMU guest agent usage.
|
||||
|
||||
## Overview
|
||||
|
||||
Most systems use **SSH key authentication** with the `~/.ssh/homelab` key. A few special cases require **password authentication** (router, Windows PC) due to platform limitations.
|
||||
|
||||
**SSH Password**: `GrilledCh33s3#` (for systems without key auth)
|
||||
|
||||
---
|
||||
|
||||
## SSH Key Authentication (Primary Method)
|
||||
|
||||
### SSH Key Configuration
|
||||
|
||||
SSH keys are configured in `~/.ssh/config` on both Mac Mini and MacBook.
|
||||
|
||||
**Key file**: `~/.ssh/homelab` (Ed25519 key)
|
||||
|
||||
**Key deployed to**: All Proxmox hosts, VMs, and LXCs (13 total hosts)
|
||||
|
||||
### Host Aliases
|
||||
|
||||
Use these convenient aliases instead of IP addresses:
|
||||
|
||||
| Host Alias | IP | User | Type | Notes |
|
||||
|------------|-----|------|------|-------|
|
||||
| `pve` | 10.10.10.120 | root | Proxmox | Primary server |
|
||||
| `pve2` | 10.10.10.102 | root | Proxmox | Secondary server |
|
||||
| `truenas` | 10.10.10.200 | root | VM | NAS/storage |
|
||||
| `saltbox` | 10.10.10.100 | hutson | VM | Media automation |
|
||||
| `lmdev1` | 10.10.10.111 | hutson | VM | AI/LLM development |
|
||||
| `docker-host` | 10.10.10.206 | hutson | VM | Docker services |
|
||||
| `fs-dev` | 10.10.10.5 | hutson | VM | Development |
|
||||
| `copyparty` | 10.10.10.201 | hutson | VM | File sharing |
|
||||
| `gitea-vm` | 10.10.10.220 | hutson | VM | Git server |
|
||||
| `trading-vm` | 10.10.10.221 | hutson | VM | AI trading platform |
|
||||
| `pihole` | 10.10.10.10 | root | LXC | DNS/Ad blocking |
|
||||
| `traefik` | 10.10.10.250 | root | LXC | Reverse proxy |
|
||||
| `findshyt` | 10.10.10.8 | root | LXC | Custom app |
|
||||
|
||||
### Usage Examples
|
||||
|
||||
```bash
|
||||
# List VMs on PVE
|
||||
ssh pve 'qm list'
|
||||
|
||||
# Check ZFS pool on TrueNAS
|
||||
ssh truenas 'zpool status vault'
|
||||
|
||||
# List Docker containers on Saltbox
|
||||
ssh saltbox 'docker ps'
|
||||
|
||||
# Check Pi-hole status
|
||||
ssh pihole 'pihole status'
|
||||
|
||||
# View Traefik config
|
||||
ssh pve 'pct exec 202 -- cat /etc/traefik/traefik.yaml'
|
||||
```
|
||||
|
||||
### SSH Config File
|
||||
|
||||
**Location**: `~/.ssh/config`
|
||||
|
||||
**Example entries**:
|
||||
|
||||
```sshconfig
|
||||
# Proxmox Servers
|
||||
Host pve
|
||||
HostName 10.10.10.120
|
||||
User root
|
||||
IdentityFile ~/.ssh/homelab
|
||||
|
||||
Host pve2
|
||||
HostName 10.10.10.102
|
||||
User root
|
||||
IdentityFile ~/.ssh/homelab
|
||||
# Post-quantum KEX causes MTU issues - use classic
|
||||
KexAlgorithms curve25519-sha256
|
||||
|
||||
# VMs
|
||||
Host truenas
|
||||
HostName 10.10.10.200
|
||||
User root
|
||||
IdentityFile ~/.ssh/homelab
|
||||
|
||||
Host saltbox
|
||||
HostName 10.10.10.100
|
||||
User hutson
|
||||
IdentityFile ~/.ssh/homelab
|
||||
|
||||
Host lmdev1
|
||||
HostName 10.10.10.111
|
||||
User hutson
|
||||
IdentityFile ~/.ssh/homelab
|
||||
|
||||
Host docker-host
|
||||
HostName 10.10.10.206
|
||||
User hutson
|
||||
IdentityFile ~/.ssh/homelab
|
||||
|
||||
Host fs-dev
|
||||
HostName 10.10.10.5
|
||||
User hutson
|
||||
IdentityFile ~/.ssh/homelab
|
||||
|
||||
Host copyparty
|
||||
HostName 10.10.10.201
|
||||
User hutson
|
||||
IdentityFile ~/.ssh/homelab
|
||||
|
||||
Host gitea-vm
|
||||
HostName 10.10.10.220
|
||||
User hutson
|
||||
IdentityFile ~/.ssh/homelab
|
||||
|
||||
Host trading-vm
|
||||
HostName 10.10.10.221
|
||||
User hutson
|
||||
IdentityFile ~/.ssh/homelab
|
||||
|
||||
# LXC Containers
|
||||
Host pihole
|
||||
HostName 10.10.10.10
|
||||
User root
|
||||
IdentityFile ~/.ssh/homelab
|
||||
|
||||
Host traefik
|
||||
HostName 10.10.10.250
|
||||
User root
|
||||
IdentityFile ~/.ssh/homelab
|
||||
|
||||
Host findshyt
|
||||
HostName 10.10.10.8
|
||||
User root
|
||||
IdentityFile ~/.ssh/homelab
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Password Authentication (Special Cases)
|
||||
|
||||
Some systems don't support SSH key auth or have other limitations.
|
||||
|
||||
### UniFi Router (10.10.10.1)
|
||||
|
||||
**Issue**: Uses `keyboard-interactive` auth method, incompatible with `sshpass`
|
||||
**Solution**: Use `expect` to automate password entry
|
||||
|
||||
**Commands**:
|
||||
|
||||
```bash
|
||||
# Run command on router
|
||||
expect -c 'spawn ssh root@10.10.10.1 "hostname"; expect "Password:"; send "GrilledCh33s3#\r"; expect eof'
|
||||
|
||||
# Get ARP table (all device IPs)
|
||||
expect -c 'spawn ssh root@10.10.10.1 "cat /proc/net/arp"; expect "Password:"; send "GrilledCh33s3#\r"; expect eof'
|
||||
|
||||
# Check Tailscale status
|
||||
expect -c 'spawn ssh root@10.10.10.1 "tailscale status"; expect "Password:"; send "GrilledCh33s3#\r"; expect eof'
|
||||
```
|
||||
|
||||
**Why not key auth?**: UniFi router firmware doesn't persist SSH keys across reboots.
|
||||
|
||||
### Windows PC (10.10.10.150)
|
||||
|
||||
**OS**: Windows with OpenSSH server
|
||||
**User**: `claude`
|
||||
**Password**: `GrilledCh33s3#`
|
||||
**Shell**: PowerShell (not bash)
|
||||
|
||||
**Commands**:
|
||||
|
||||
```bash
|
||||
# Run PowerShell command
|
||||
sshpass -p 'GrilledCh33s3#' ssh claude@10.10.10.150 'Get-Process | Select -First 5'
|
||||
|
||||
# Check Syncthing status
|
||||
sshpass -p 'GrilledCh33s3#' ssh claude@10.10.10.150 'Get-Process -Name syncthing -ErrorAction SilentlyContinue'
|
||||
|
||||
# Restart Syncthing
|
||||
sshpass -p 'GrilledCh33s3#' ssh claude@10.10.10.150 'Stop-Process -Name syncthing -Force; Start-ScheduledTask -TaskName "Syncthing"'
|
||||
```
|
||||
|
||||
**⚠️ Important**: Use `;` (semicolon) to chain PowerShell commands, NOT `&&` (bash syntax).
|
||||
|
||||
**Why not key auth?**: Could be configured, but password auth works and is simpler for Windows.
|
||||
|
||||
---
|
||||
|
||||
## QEMU Guest Agent
|
||||
|
||||
Most VMs have the QEMU guest agent installed, allowing command execution without SSH.
|
||||
|
||||
### VMs with QEMU Agent
|
||||
|
||||
| VMID | VM Name | Use Case |
|
||||
|------|---------|----------|
|
||||
| 100 | truenas | Execute commands, check ZFS |
|
||||
| 101 | saltbox | Execute commands, Docker mgmt |
|
||||
| 105 | fs-dev | Execute commands |
|
||||
| 111 | lmdev1 | Execute commands |
|
||||
| 201 | copyparty | Execute commands |
|
||||
| 206 | docker-host | Execute commands |
|
||||
| 300 | gitea-vm | Execute commands |
|
||||
| 301 | trading-vm | Execute commands |
|
||||
|
||||
### VM WITHOUT QEMU Agent
|
||||
|
||||
**VMID 110 (homeassistant)**: No QEMU agent installed
|
||||
- Access via web UI only
|
||||
- Or install SSH server manually if needed
|
||||
|
||||
### Usage Examples
|
||||
|
||||
**Basic syntax**:
|
||||
```bash
|
||||
ssh pve 'qm guest exec VMID -- bash -c "COMMAND"'
|
||||
```
|
||||
|
||||
**Examples**:
|
||||
|
||||
```bash
|
||||
# Check ZFS pool on TrueNAS (without SSH)
|
||||
ssh pve 'qm guest exec 100 -- bash -c "zpool status vault"'
|
||||
|
||||
# Get VM IP addresses
|
||||
ssh pve 'qm guest exec 100 -- bash -c "ip addr"'
|
||||
|
||||
# Check Docker containers on Saltbox
|
||||
ssh pve 'qm guest exec 101 -- bash -c "docker ps"'
|
||||
|
||||
# Run multi-line command
|
||||
ssh pve 'qm guest exec 100 -- bash -c "df -h; free -h; uptime"'
|
||||
```
|
||||
|
||||
**When to use QEMU agent vs SSH**:
|
||||
- ✅ Use **SSH** for interactive sessions, file editing, complex tasks
|
||||
- ✅ Use **QEMU agent** for one-off commands, when SSH is down, or VM has no network
|
||||
- ⚠️ QEMU agent is slower for multiple commands (use SSH instead)
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting SSH Issues
|
||||
|
||||
### Connection Refused
|
||||
|
||||
```bash
|
||||
# Check if SSH service is running
|
||||
ssh pve 'systemctl status sshd'
|
||||
|
||||
# Check if port 22 is open
|
||||
nc -zv 10.10.10.XXX 22
|
||||
|
||||
# Check firewall
|
||||
ssh pve 'iptables -L -n | grep 22'
|
||||
```
|
||||
|
||||
### Permission Denied (Public Key)
|
||||
|
||||
```bash
|
||||
# Verify key file exists
|
||||
ls -la ~/.ssh/homelab
|
||||
|
||||
# Check key permissions (should be 600)
|
||||
chmod 600 ~/.ssh/homelab
|
||||
|
||||
# Test SSH key auth verbosely
|
||||
ssh -vvv -i ~/.ssh/homelab root@10.10.10.120
|
||||
|
||||
# Check authorized_keys on remote (via QEMU agent if SSH broken)
|
||||
ssh pve 'qm guest exec VMID -- bash -c "cat ~/.ssh/authorized_keys"'
|
||||
```
|
||||
|
||||
### Slow SSH Connection (PVE2 Issue)
|
||||
|
||||
**Problem**: SSH to PVE2 hangs for 30+ seconds before connecting
|
||||
**Cause**: MTU mismatch (vmbr0=9000, nic1=1500) causing post-quantum KEX packet fragmentation
|
||||
**Fix**: Use classic KEX algorithm instead
|
||||
|
||||
**In `~/.ssh/config`**:
|
||||
```sshconfig
|
||||
Host pve2
|
||||
HostName 10.10.10.102
|
||||
User root
|
||||
IdentityFile ~/.ssh/homelab
|
||||
KexAlgorithms curve25519-sha256 # Avoid mlkem768x25519-sha256
|
||||
```
|
||||
|
||||
**Permanent fix**: Set `nic1` MTU to 9000 in `/etc/network/interfaces` on PVE2
|
||||
|
||||
---
|
||||
|
||||
## Adding SSH Keys to New Systems
|
||||
|
||||
### Linux (VMs/LXCs)
|
||||
|
||||
```bash
|
||||
# Copy public key to new host
|
||||
ssh-copy-id -i ~/.ssh/homelab user@hostname
|
||||
|
||||
# Or manually:
|
||||
ssh user@hostname 'mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys' < ~/.ssh/homelab.pub
|
||||
ssh user@hostname 'chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys'
|
||||
```
|
||||
|
||||
### LXC Containers (Root User)
|
||||
|
||||
```bash
|
||||
# Via pct exec from Proxmox host
|
||||
ssh pve 'pct exec CTID -- bash -c "mkdir -p /root/.ssh"'
|
||||
ssh pve 'pct exec CTID -- bash -c "echo \"$(cat ~/.ssh/homelab.pub)\" >> /root/.ssh/authorized_keys"'
|
||||
ssh pve 'pct exec CTID -- bash -c "chmod 700 /root/.ssh && chmod 600 /root/.ssh/authorized_keys"'
|
||||
|
||||
# Also enable PermitRootLogin in sshd_config
|
||||
ssh pve 'pct exec CTID -- bash -c "sed -i \"s/^#*PermitRootLogin.*/PermitRootLogin prohibit-password/\" /etc/ssh/sshd_config"'
|
||||
ssh pve 'pct exec CTID -- bash -c "systemctl restart sshd"'
|
||||
```
|
||||
|
||||
### VMs (via QEMU Agent)
|
||||
|
||||
```bash
|
||||
# Add key via QEMU agent (if SSH not working)
|
||||
ssh pve 'qm guest exec VMID -- bash -c "mkdir -p ~/.ssh"'
|
||||
ssh pve 'qm guest exec VMID -- bash -c "echo \"$(cat ~/.ssh/homelab.pub)\" >> ~/.ssh/authorized_keys"'
|
||||
ssh pve 'qm guest exec VMID -- bash -c "chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys"'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SSH Key Management
|
||||
|
||||
### Rotate SSH Keys (Future)
|
||||
|
||||
When rotating SSH keys:
|
||||
|
||||
1. Generate new key pair:
|
||||
```bash
|
||||
ssh-keygen -t ed25519 -f ~/.ssh/homelab-new -C "homelab-new"
|
||||
```
|
||||
|
||||
2. Deploy new key to all hosts (keep old key for now):
|
||||
```bash
|
||||
for host in pve pve2 truenas saltbox lmdev1 docker-host fs-dev copyparty gitea-vm trading-vm pihole traefik findshyt; do
|
||||
ssh-copy-id -i ~/.ssh/homelab-new $host
|
||||
done
|
||||
```
|
||||
|
||||
3. Update `~/.ssh/config` to use new key:
|
||||
```sshconfig
|
||||
IdentityFile ~/.ssh/homelab-new
|
||||
```
|
||||
|
||||
4. Test all connections:
|
||||
```bash
|
||||
for host in pve pve2 truenas saltbox lmdev1 docker-host fs-dev copyparty gitea-vm trading-vm pihole traefik findshyt; do
|
||||
echo "Testing $host..."
|
||||
ssh $host 'hostname'
|
||||
done
|
||||
```
|
||||
|
||||
5. Remove old key from all hosts once confirmed working
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Common SSH Operations
|
||||
|
||||
```bash
|
||||
# Execute command on remote host
|
||||
ssh host 'command'
|
||||
|
||||
# Execute multiple commands
|
||||
ssh host 'command1 && command2'
|
||||
|
||||
# Copy file to remote
|
||||
scp file host:/path/
|
||||
|
||||
# Copy file from remote
|
||||
scp host:/path/file ./
|
||||
|
||||
# Execute command on Proxmox VM (via QEMU agent)
|
||||
ssh pve 'qm guest exec VMID -- bash -c "command"'
|
||||
|
||||
# Execute command on LXC
|
||||
ssh pve 'pct exec CTID -- command'
|
||||
|
||||
# Interactive shell
|
||||
ssh host
|
||||
|
||||
# SSH with X11 forwarding
|
||||
ssh -X host
|
||||
```
|
||||
|
||||
### Troubleshooting Commands
|
||||
|
||||
```bash
|
||||
# Test SSH with verbose output
|
||||
ssh -vvv host
|
||||
|
||||
# Check SSH service status (remote)
|
||||
ssh host 'systemctl status sshd'
|
||||
|
||||
# Check SSH config (local)
|
||||
ssh -G host
|
||||
|
||||
# Test port connectivity
|
||||
nc -zv hostname 22
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### Current Security Posture
|
||||
|
||||
✅ **Good**:
|
||||
- SSH keys used instead of passwords (where possible)
|
||||
- Keys use Ed25519 (modern, secure algorithm)
|
||||
- Root login disabled on VMs (use sudo instead)
|
||||
- SSH keys have proper permissions (600)
|
||||
|
||||
⚠️ **Could Improve**:
|
||||
- [ ] Disable password authentication on all hosts (force key-only)
|
||||
- [ ] Use SSH certificate authority instead of individual keys
|
||||
- [ ] Set up SSH bastion host (jump server)
|
||||
- [ ] Enable 2FA for SSH (via PAM + Google Authenticator)
|
||||
- [ ] Implement SSH key rotation policy (annually)
|
||||
|
||||
### Hardening SSH (Future)
|
||||
|
||||
For additional security, consider:
|
||||
|
||||
```sshconfig
|
||||
# /etc/ssh/sshd_config (on remote hosts)
|
||||
PermitRootLogin prohibit-password # No root password login
|
||||
PasswordAuthentication no # Disable password auth entirely
|
||||
PubkeyAuthentication yes # Only allow key auth
|
||||
AuthorizedKeysFile .ssh/authorized_keys
|
||||
MaxAuthTries 3 # Limit auth attempts
|
||||
MaxSessions 10 # Limit concurrent sessions
|
||||
ClientAliveInterval 300 # Timeout idle sessions
|
||||
ClientAliveCountMax 2 # Drop after 2 keepalives
|
||||
```
|
||||
|
||||
**Apply after editing**:
|
||||
```bash
|
||||
systemctl restart sshd
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [VMS.md](VMS.md) - Complete VM/CT inventory
|
||||
- [NETWORK.md](NETWORK.md) - Network configuration
|
||||
- [IP-ASSIGNMENTS.md](IP-ASSIGNMENTS.md) - IP addresses for all hosts
|
||||
- [SECURITY.md](#) - Security policies (coming soon)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-22
|
||||
Reference in New Issue
Block a user