Spark Interconnect Test
iperf3 validation of the direct 200 GbE link between spark-0296 and spark-efd0, including issues found and how they were fixed.
Validation report for the link set up in Connect Two Sparks. Two nodes (spark-0296 = 192.168.100.1, spark-efd0 = 192.168.100.2) were interconnected over their high-speed NICs and validated with iperf3. After resolving SSH config and cable/carrier issues, the link is up and passing traffic cleanly.
Result: link is functional.
Issues found and fixed
| # | Issue | Symptom | Fix |
|---|---|---|---|
| 1 | Malformed ~/.ssh/config | unsupported option "yesHost" on every SSH command | Line 5 had IdentitiesOnly yes and Host * merged onto one line. Split into two separate lines. |
| 2 | No carrier on enp1s0f1np1 | ethtool reported Link detected: no (No cable); TX carrier errors present | Reseated/verified QSFP cable |
| 3 | Ambiguous ping test | ping 192.168.100.1 succeeded even while the high-speed interface had no carrier | Traffic was going over a different (management) path, not the intended NIC — confirmed via ip route get |
| 4 | Stale iperf3 server | Server left running from a prior session | sudo pkill iperf3 before restarting cleanly |
Test method
Server (node 2 / spark-efd0, 192.168.100.2):
sudo pkill iperf3
iperf3 -sClient (node 1 / spark-0296, 192.168.100.1):
iperf3 -c 192.168.100.2Default single-stream TCP test, 10-second duration.
Results
| Metric | Value |
|---|---|
| Duration | 10.01 sec |
| Data transferred | 47.8 GBytes |
| Average bitrate | 41.0 Gbits/sec |
| Retransmits | 0 |
| Congestion window | Stable ~4.16 MBytes throughout |
| Direction tested | Node 1 → Node 2 (single direction) |
Per-second bitrate ranged between ~38–45 Gbits/sec, no drops or spikes — consistent and clean.
Server-side log confirms an exact match with the client (47.8 GBytes / 41.0 Gbits/sec, sender and receiver figures agree), so there's no packet loss or reporting discrepancy between the two nodes.
Reading the number: is 41 Gbit/s expected?
This was a single-stream, single-core TCP test, which is often CPU/thread-bound rather than link-bound — on very fast NICs (100 GbE/200 GbE-class links can bottleneck on a single core well before the wire is saturated). To see the speed the NIC believes it's running at:
ethtool enp1s0f1np1 | grep -i speedIf the negotiated speed is significantly higher than 41 Gbit/s, the number reflects single-stream CPU/software limits, not the physical link ceiling.
Follow-up tests
The initial run only covered one direction with a single stream. The full validation battery:
-
Confirm negotiated link speed — on both nodes:
ethtool enp1s0f1np1 | grep -i speed -
Parallel-stream throughput (rules out the single-core bottleneck):
iperf3 -c 192.168.100.2 -P 8 -
Reverse direction (initial run only tested node 1 → node 2):
iperf3 -c 192.168.100.2 -R -
Bidirectional simultaneous:
iperf3 -c 192.168.100.2 --bidir -
MTU / jumbo frames — mismatched MTU can silently cap throughput:
ip link show enp1s0f1np1 | grep mtu -
Passwordless SSH in both directions (re-verify after the config fix):
ssh [email protected] hostname # from node 1 ssh [email protected] hostname # from node 2
Status
- SSH config fixed
- Cable/carrier issue resolved
- Single-stream throughput confirmed, 0 retransmits
- Negotiated link speed confirmed via
ethtool - Parallel/bidirectional throughput tested
- MTU verified
- Passwordless SSH re-confirmed both directions