Connect Two Sparks

This guide connects spark-0296 (node 1) and spark-efd0 (node 2) with a direct 200 GbE link. It follows NVIDIA's Connect Two Sparks playbook, with the exact commands used for this pair.

Check the cable connection

• Connect a QSFP cable between the two DGX Spark systems. Use the same physical port on each device — mismatched ports cause issues with NCCL tests.
• Each physical port exposes two logical interfaces. In this setup the interfaces that come up are enp1s0f1np1 / enP2p1s0f1np1.
• After connecting, run ibdev2netdev on a node to see which logical interfaces are Up. Those are the ones you configure in the next step.

ibdev2netdev
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)

If none of the interfaces show Up, reseat the QSFP cable and reboot, then check again.

Assign static IPs

The original playbook offers two options: a netplan file (survives reboot) or raw ip addr add commands (lost on reboot). This setup uses netplan.

On node 1:

# Create the netplan configuration file
sudo tee /etc/netplan/40-cx7.yaml > /dev/null <<EOF
network:
  version: 2
  ethernets:
    enp1s0f1np1:
      addresses:
        - 192.168.100.10/24
      dhcp4: no
    enP2p1s0f1np1:
      addresses:
        - 192.168.101.10/24
      dhcp4: no
EOF

# Set appropriate permissions
sudo chmod 600 /etc/netplan/40-cx7.yaml

# Apply the configuration
sudo netplan apply

On node 2:

# Create the netplan configuration file
sudo tee /etc/netplan/40-cx7.yaml > /dev/null <<EOF
network:
  version: 2
  ethernets:
    enp1s0f1np1:
      addresses:
        - 192.168.100.11/24
      dhcp4: no
    enP2p1s0f1np1:
      addresses:
        - 192.168.101.11/24
      dhcp4: no
EOF

# Set appropriate permissions
sudo chmod 600 /etc/netplan/40-cx7.yaml

# Apply the configuration
sudo netplan apply

These are the addresses of the direct QSFP link. The nodes also have their own management IPs — in this setup spark-0296 is 192.168.100.1 and spark-efd0 is 192.168.100.2. Don't let that trip you up: pinging 192.168.100.1 can succeed over the management path even when the QSFP link is down (this exact confusion showed up in the interconnect test).

Set up passwordless SSH

Download the discovery script from the playbook repo and run it on one of the nodes:

wget https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/connect-two-sparks/assets/discover-sparks
bash discover-sparks

The downloaded file is named discover-sparks (no .sh). The first run prompts for your password on each node, then sets up bidirectional passwordless SSH. If it errors, fall back to manual ssh-copy-id as described in the original playbook.

Verify multi-node communication

From node 1:

ssh [email protected]
Welcome to NVIDIA DGX Spark Version 7.5.0 (GNU/Linux 6.17.0-1026-nvidia aarch64)

The link is working once this lands in a shell without a password prompt. Next step for most workloads: interconnect test.

Rollback and removal

Run on both nodes. Use the first block if you configured with netplan (Option 1), the second if you used raw ip addr commands (Option 2).

# Rollback network configuration (Option 1: netplan) on both nodes.
sudo rm /etc/netplan/40-cx7.yaml
sudo netplan apply

# Rollback network configuration (Option 2: command line) on both nodes.
# Node 1:
sudo ip addr del 192.168.100.10/24 dev enp1s0f1np1  # Adjust to the interface names you configured above.
sudo ip addr del 192.168.101.10/24 dev enP2p1s0f1np1

# Node 2:
sudo ip addr del 192.168.100.11/24 dev enp1s0f1np1
sudo ip addr del 192.168.101.11/24 dev enP2p1s0f1np1

Resources

• Two-node clustering on DGX Spark
• sglang-dgx-spark cluster startup script
• Connecting 4 Sparks through a switch
• DGX Spark documentation