name: spectrocloud-troubleshooting
description: Debug Spectro Cloud Palette clusters using event streams, log bundles, and edge host logs. Tracks known errors - both actionable and noise.

Spectro Cloud Troubleshooting

Debug cluster provisioning, upgrades, and operational issues using Palette's event streams and logs.

Key Concepts

• Event Stream: Live cluster events showing orchestration progress and errors
• Cluster Conditions: Milestones like "Creating Infrastructure", "Adding Control Plane Node"
• Log Bundle: Downloadable archive with Spectro logs, system logs, and manifests
• Reconciliation: Palette retries failed operations - intermittent errors may resolve automatically

Cluster Event Stream (API)

Events stream live and show orchestration progress. Watch these during provisioning.

Get Cluster Events

# Get recent events for a cluster
curl -s "https://api.spectrocloud.com/v1/spectroclusters/$CLUSTER_UID/events?limit=50" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" | \
  jq '[.items[] | {time: .metadata.creationTimestamp, type: .involvedObject.kind,
      reason: .reason, message: .message}]'

Get Cluster Status & Conditions

# Overview with conditions and upgrade history
curl -s "https://api.spectrocloud.com/v1/dashboard/spectroclusters/$CLUSTER_UID/overview" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" | \
  jq '{state: .status.state, conditions: .status.conditions, upgrades: .status.upgrades}'

Watch Events (Poll)

# Poll events every 10 seconds during provisioning
while true; do
  echo "=== $(date) ==="
  curl -s "https://api.spectrocloud.com/v1/spectroclusters/$CLUSTER_UID/events?limit=10" \
    -H "ApiKey: $PALETTE_API_KEY" \
    -H "ProjectUid: $PROJECT_UID" | \
    jq -r '.items[] | "\(.metadata.creationTimestamp) [\(.reason)] \(.message)"' | head -10
  sleep 10
done

Log Bundle Download

Download comprehensive logs for deep debugging or support requests.

Download via API

# Download log bundle (returns zip file)
curl -s "https://api.spectrocloud.com/v1/spectroclusters/$CLUSTER_UID/features/logFetcher/logs" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" \
  -o spectro_logs.zip

# Extract and examine
unzip spectro_logs.zip -d cluster-logs/
ls cluster-logs/

Log Bundle Contents

Edge Host Logs (SSH)

For edge deployments, SSH to the host for live debugging.

# SSH to edge host (default creds: kairos/kairos)
ssh kairos@<edge-host-ip>

# Watch Palette agent logs (most useful)
sudo journalctl -u spectro-stylus-agent.service -f

# Check all Spectro services
sudo systemctl list-units 'spectro*'

# Key log files
cat /var/log/stylus-upgrade.log
cat /var/log/kube-init.log
cat /var/log/kube-join.log

# Check kubelet if K8s is initializing
sudo journalctl -u kubelet -f

Cluster State Reference

Known Errors: Noise vs. Actionable

Palette's reconciliation pattern means some errors are transient and resolve automatically.

Normal/Transient (Usually Safe to Ignore)

These often appear during provisioning but resolve:

Actionable Errors (Investigate)

These indicate real problems:

2-Node Specific Errors

Debugging Workflow

1. Check Cluster State

curl -s "https://api.spectrocloud.com/v1/spectroclusters/$CLUSTER_UID" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" | \
  jq '{name: .metadata.name, state: .status.state, health: .status.clusterHealth}'

2. Check Active Condition

curl -s "https://api.spectrocloud.com/v1/dashboard/spectroclusters/$CLUSTER_UID/overview" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" | \
  jq '.status.conditions[] | select(.status != "True") | {type, message, reason}'

3. Recent Events

curl -s "https://api.spectrocloud.com/v1/spectroclusters/$CLUSTER_UID/events?limit=20" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" | \
  jq -r '.items[] | "\(.metadata.creationTimestamp) [\(.reason)] \(.message)"'

4. SSH to Edge Host (if applicable)

ssh kairos@<edge-host-ip>
sudo journalctl -u spectro-stylus-agent.service -n 100 --no-pager

5. Download Full Logs (for support)

curl -s "https://api.spectrocloud.com/v1/spectroclusters/$CLUSTER_UID/features/logFetcher/logs" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" \
  -o spectro_logs_$(date +%Y%m%d_%H%M).zip

Edge Host Registration Issues

# List edge hosts and their state
curl -s "https://api.spectrocloud.com/v1/edgehosts" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" | \
  jq '[.items[] | {name: .metadata.name, uid: .metadata.uid, state: .status.state,
      health: .status.health.state, cluster: .status.clusterUid}]'

# Check specific edge host
curl -s "https://api.spectrocloud.com/v1/edgehosts/$EDGEHOST_UID" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" | \
  jq '{name: .metadata.name, state: .status.state, health: .status.health,
      lastHeartbeat: .status.lastHeartbeat}'

Pack Deployment Issues

# Check pack status on cluster
curl -s "https://api.spectrocloud.com/v1/spectroclusters/$CLUSTER_UID/packs/status" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" | \
  jq '[.items[] | {name: .name, status: .status, message: .message}]'

Adding New Learnings

This skill is a living document. When you encounter new errors:

1. Determine if it's noise (transient, resolved by retry) or actionable
2. Add to the appropriate table above
3. Include the error pattern, cause, and recommended action
4. Commit with descriptive message

Quick Reference

Links

• Nodes and Clusters Troubleshooting
• Edge Troubleshooting
• Common Issues