name: spectrocloud-appliance-mode
description: Build and deploy Spectro Cloud Palette Edge clusters using Appliance Mode. Covers CanvOS builds, provider images, ISOs, content bundles, 2-node deployments, and CI/CD integration patterns.

Spectro Cloud Appliance Mode

Appliance Mode creates immutable edge deployments using Kairos-based images built with CanvOS.

When to Use

Key Concepts

• CanvOS: Earthly-based build system for edge artifacts
• Provider Image: Container image with OS + specific K8s version
• Installer ISO: Bootable image with Palette agent + user-data
• Content Bundle: Pre-cached images for offline deployments

Prerequisites

• Linux AMD64 build machine (4+ CPU, 8GB+ RAM, 150GB+ storage)
• Git, Docker, Earthly

# Install Earthly
curl -fsSL https://releases.earthly.dev/earthly-linux-amd64 -o earthly
sudo mv earthly /usr/local/bin/ && sudo chmod +x /usr/local/bin/earthly
earthly bootstrap

Build Process

Ask the user:
1. Registry for provider images? (ttl.sh for testing, or specify)
2. K8s distribution? (k3s, rke2, kubeadm, kubeadm-fips)
3. 2-node deployment? (requires K3s + TWO_NODE=true)
4. Need content bundles for offline?

Step 0: Determine K8s Version

Query CanvOS for supported versions before building. Default to n-1 minor for stability.

# Get latest CanvOS tag
CANVOS_TAG=$(gh api repos/spectrocloud/CanvOS/tags --jq '.[0].name')
echo "Latest CanvOS: $CANVOS_TAG"

# Fetch supported K8s versions
curl -s "https://raw.githubusercontent.com/spectrocloud/CanvOS/$CANVOS_TAG/k8s_version.json" > k8s_versions.json

# Get n-1 minor for chosen distribution (default: k3s)
DISTRO="k3s"  # or: rke2, kubeadm, kubeadm-fips
K8S_VERSION=$(jq -r --arg d "$DISTRO" '.[$d] |
  sort_by(split(".") | map(tonumber)) | reverse |
  group_by(split(".")[0:2] | join(".")) |
  sort_by(.[0] | split(".") | map(tonumber)) | reverse | .[1][0] // .[0][0]' k8s_versions.json)
echo "Recommended $DISTRO version: $K8S_VERSION"

# Show all available for reference
jq -r --arg d "$DISTRO" '.[$d] | sort_by(split(".") | map(tonumber)) | reverse | .[0:5] | join(", ")' k8s_versions.json

Use this K8S_VERSION in Step 1's .arg file.

Step 1: Clone and Configure

# Clone fresh or update existing repo
if [ -d "CanvOS" ]; then
  cd CanvOS && git fetch --tags && git pull
else
  git clone https://github.com/spectrocloud/CanvOS.git && cd CanvOS
fi
LATEST_TAG=$(git describe --tags --abbrev=0)
echo "Using CanvOS $LATEST_TAG"
git checkout "$LATEST_TAG"

# Use K8S_VERSION from Step 0, or set manually
cat << EOF > .arg
OS_DISTRIBUTION=ubuntu
OS_VERSION=24.04
K8S_DISTRIBUTION=${DISTRO:-k3s}
K8S_VERSION=${K8S_VERSION:-1.32.9}
IMAGE_REGISTRY=ttl.sh
IMAGE_REPO=my-edge-images
CUSTOM_TAG=demo
ARCH=amd64
FIPS_ENABLED=false
TWO_NODE=false
EOF

Step 2: Create user-data

Bridge networking is recommended for VM-based edge deployments (allows containers to get IPs on host network).

cat << 'EOF' > user-data
#cloud-config
install:
  reboot: true

users:
  - name: kairos
    passwd: kairos
    groups: [sudo, admin]
    sudo: ALL=(ALL) NOPASSWD:ALL

stages:
  initramfs:
    - name: "Setup bridge networking"
      files:
        - path: /etc/systemd/network/20-dhcp.network
          content: |
            [Match]
            Name=en*
            [Network]
            Bridge=br0
            LinkLocalAddressing=no
          permissions: 0644
          owner: 0
          group: 0
        - path: /etc/systemd/network/bridge0.netdev
          content: |
            [NetDev]
            Name=br0
            Kind=bridge
          permissions: 0644
          owner: 0
          group: 0
        - path: /etc/systemd/network/bridge0.network
          content: |
            [Match]
            Name=br0
            [Network]
            DHCP=yes
          permissions: 0644
          owner: 0
          group: 0

stylus:
  site:
    paletteEndpoint: api.spectrocloud.com
    edgeHostToken: <registration-token>
    projectName: <project-name>
EOF

For proxy, static IP, bonds: See references/user-data-examples.yaml

Step 3: Build

docker login <registry>  # Skip for ttl.sh
earthly +iso  # Build ISO first - start imaging nodes while provider images push
earthly --push +build-provider-images
# Output: build/palette-edge-installer.iso

Step 4: Version the ISO

Critical: The default ISO name has no version info. Rename it before uploading:

# Create versioned ISO name with date and K8s version
ISO_NAME="palette-edge-$(grep K8S_VERSION .arg | cut -d= -f2)-$(date +%Y%m%d-%H%M).iso"
mv build/palette-edge-installer.iso "build/$ISO_NAME"
echo "Built: $ISO_NAME"

Before imaging any VM, verify you're using the correct ISO:

# Check ISO build date (should match your recent build)
ls -la build/*.iso

# If uploading to Proxmox, include version in the storage name
# e.g., "palette-edge-k3s-1.30.5-20241222-1430.iso"

2-Node Deployment

2-node HA uses Postgres + Kine backend. Appliance-mode only.

Requirements

• K3s only (not kubeadm/rke2)
• Ubuntu 22.04 recommended
• Cannot expand to 3+ nodes later

2-Node .arg

cat << 'EOF' > .arg
OS_DISTRIBUTION=ubuntu
OS_VERSION=22.04
K8S_DISTRIBUTION=k3s
K8S_VERSION=1.33.3
IMAGE_REGISTRY=ttl.sh
IMAGE_REPO=my-edge-2node
CUSTOM_TAG=two-node-demo
ARCH=amd64
TWO_NODE=true
EOF

Build & Deploy

earthly +iso  # Build ISO first - start imaging nodes while provider images push
earthly --push +provider-image

When creating cluster: toggle "Two-Node Mode", select exactly 2 edge hosts.

2-Node Failover Behavior

• One node is leader (writes), other is follower
• Follower detects leader failure and self-promotes (~30s failover)
• On recovery, nodes compare timestamps; most recent becomes leader

Content Bundles

Pre-cache images for offline deployments:

# Install Palette CLI
curl -LO https://github.com/spectrocloud/palette-cli/releases/latest/download/palette-linux-amd64
chmod +x palette-linux-amd64 && sudo mv palette-linux-amd64 /usr/local/bin/palette

# Build bundle
palette content build --arch amd64 --project-id <uid> \
  --cluster-profile-ids <id1>,<id2> --output ./content-bundle.tar.zst

# Include in ISO
mkdir -p content && cp content-bundle.tar.zst content/
earthly +iso

Deploy

Pre-Deploy Checklist

• [ ] ISO name includes version/date (not generic palette-edge-installer.iso)
• [ ] Provider image tag matches ISO build (check .arg used)
• [ ] Old ISOs cleaned up from hypervisor storage
• [ ] Old edge hosts deleted from Palette (prevents duplicate ID errors)

Proxmox Upload

# Upload versioned ISO to Proxmox
ISO_PATH="/path/to/palette-edge-k3s-1.30.5-20241222.iso"
scp "$ISO_PATH" [email protected]:/var/lib/vz/template/iso/

# List ISOs on Proxmox to verify and clean up old ones
ssh [email protected] "ls -la /var/lib/vz/template/iso/palette-*.iso"

# Remove stale ISOs (keep only latest)
ssh [email protected] "rm /var/lib/vz/template/iso/palette-edge-OLD*.iso"

VM Creation

Sizing considerations:
| Use Case | Disk Size | Notes |
|----------|-----------|-------|
| Basic edge | 100GB | ~2.5GB free after OS for storage pools |
| With Piraeus file pool | 200GB+ | File pools use root partition space |
| With separate data disk | 100GB + data disk | Attach second disk for storage pool |

Boot order is critical:

# Proxmox boot order format:
boot: order=scsi0;ide2;net0

• Disk first (scsi0), then CD-ROM (ide2)
• Empty disk falls through to CD on first boot
• After install, boots from disk (avoids reinstall loop)
• Wrong order = reinstall loop or hang

VM creation steps:
1. Create VM: 4+ CPU, 8GB+ RAM, disk sized for use case
2. Attach the versioned ISO (verify name before attaching)
3. Set boot order: order=scsi0;ide2;net0
4. Boot - installation is automatic
5. Verify in Palette: Clusters > Edge Hosts > Registered

Automated Builds via SSH

For environments with a dedicated build machine, automate the entire process:

BUILD_HOST="[email protected]"  # Or your build machine

# Clone, configure, and build in one session
ssh $BUILD_HOST << 'ENDSSH'
cd ~
if [ -d "CanvOS" ]; then
  cd CanvOS && git fetch --tags && git pull
else
  git clone https://github.com/spectrocloud/CanvOS.git && cd CanvOS
fi
LATEST_TAG=$(git describe --tags --abbrev=0)
echo "Using CanvOS $LATEST_TAG"
git checkout "$LATEST_TAG"

cat << 'ARGFILE' > .arg
OS_DISTRIBUTION=ubuntu
OS_VERSION=22.04
K8S_DISTRIBUTION=k3s
K8S_VERSION=1.33.3
IMAGE_REGISTRY=ttl.sh
IMAGE_REPO=my-edge
CUSTOM_TAG=2node
ARCH=amd64
TWO_NODE=true
ARGFILE

cat << 'USERDATA' > user-data
#cloud-config
install:
  reboot: true
users:
  - name: kairos
    passwd: kairos
    groups: [sudo, admin]
    sudo: ALL=(ALL) NOPASSWD:ALL
stages:
  initramfs:
    - name: "Setup bridge networking"
      files:
        - path: /etc/systemd/network/20-dhcp.network
          content: |
            [Match]
            Name=en*
            [Network]
            Bridge=br0
            LinkLocalAddressing=no
        - path: /etc/systemd/network/bridge0.netdev
          content: |
            [NetDev]
            Name=br0
            Kind=bridge
        - path: /etc/systemd/network/bridge0.network
          content: |
            [Match]
            Name=br0
            [Network]
            DHCP=yes
stylus:
  site:
    paletteEndpoint: api.spectrocloud.com
    edgeHostToken: <TOKEN>
    projectName: <PROJECT>
USERDATA

earthly +iso  # Build ISO first - start imaging nodes while provider images push
earthly --push +provider-image
ENDSSH

# Copy ISO back with version name
scp $BUILD_HOST:~/CanvOS/build/palette-edge-installer.iso \
  "./palette-edge-k3s-1.33.3-$(date +%Y%m%d).iso"

CI/CD Integration

See references/cicd-workflow.md for GitHub Actions and GitLab CI examples.

Troubleshooting

User-Data and /oem Directory

All #cloud-config files in /oem are merged to generate /run/stylus/userdata. If you modify files in /oem, regenerate userdata:

# On the edge node - regenerates /run/stylus/userdata from /oem files
kairos-agent notify agent.bootstrap

Use cases:
- Debugging user-data issues: check /run/stylus/userdata for merged result
- Adding config post-install: place files in /oem/*.yaml, then regenerate
- Troubleshooting registration: verify stylus config in /run/stylus/userdata

Fixing Install Loop (Boot Order)

Symptom: VMs keep reinstalling from ISO instead of booting from installed disk. Cluster stays in "Provisioning" with no nodes appearing for 30+ minutes.

Cause: Boot order is ide2;scsi0 (CD-ROM first) instead of scsi0;ide2 (disk first).

Fix:

# 1. Change boot order via Proxmox API
curl -k -X PUT "https://<PROXMOX>:8006/api2/json/nodes/proxmox/qemu/<VMID>/config" \
  -H "Cookie: PVEAuthCookie=${TICKET}" \
  -H "CSRFPreventionToken: ${CSRF}" \
  --data-urlencode "boot=order=scsi0;ide2;net0"

# 2. MUST power off completely (not reboot/reset!)
curl -k -X POST ".../qemu/<VMID>/status/stop" ...

# 3. Wait for VM to stop, then power on
curl -k -X POST ".../qemu/<VMID>/status/start" ...

CRITICAL: Reboot and reset do NOT reliably apply boot order changes. You MUST do a full power off then power on.

BYOOS Pack Version

Always use the latest BYOOS pack version when creating cluster profiles. As of Jan 2025, this is 2.1.0.

Query to confirm latest:

curl -s "https://api.spectrocloud.com/v1/packs?filters=metadata.name=edge-native-byoi&limit=50" \
  -H "ApiKey: $API_KEY" | jq -r '[.items[] | select(.spec.registryUid == "5eecc89d0b150045ae661cef")] |
  sort_by(.spec.version | split(".") | map(tonumber)) | reverse | .[0].spec.version'

Registry note: BYOOS exists in two registries. Use Public Repo (type=spectro) for Terraform/API:
- Public Repo UID: 5eecc89d0b150045ae661cef
- Type in Terraform: type = "spectro" (NOT "oci")

Provider Image Tag

Don't assume the tag format - CanvOS generates tags based on .arg values but includes Kairos version, not OS version:
- Expected: k3s-1.33.5-ubuntu-22.04-2node
- Actual: k3s-1.33.5-v4.8.1-2node (Kairos version)

Always check the actual tag after build:

# Check local images after build
docker images | grep $IMAGE_REPO

# Check registry (for Docker Hub)
curl -s "https://hub.docker.com/v2/repositories/$IMAGE_REPO/tags" | jq '.results[].name'

# Check ttl.sh (use crane or skopeo)
crane ls ttl.sh/$IMAGE_REPO

CRITICAL: Use the EXACT tag from the build output in:
1. BYOOS pack options.system.uri in the cluster profile
2. Ensure K8s pack version matches what was built into the image

Quick Reference

Additional Resources

• references/networking-examples.yaml - Bridge and bond configurations
• references/user-data-examples.yaml - Advanced user-data options
• references/cicd-workflow.md - CI/CD pipeline examples

Links

• EdgeForge Workflow
• Two-Node Architecture
• CanvOS GitHub
• Edge Config Examples