SSH across clusters

Let’s say you want to create a Flux instance in an allocation on a cluster (e.g., let’s say out first cluster is “noodle”) 🍜️ and then connect to it via ssh from another cluster (let’s say our second cluster is called “quartz”). This is possible with the right setup of your ~/.ssh/config.

Create a Flux Instance

First, let’s create the allocation on the first cluster. We typically want to ask for an allocation, and run flux start via our job manager. Here we might be on a login node:

# Slurm specific
$ salloc -N4 --exclusive
$ srun -N4 -n4 --pty --mpibind=off flux start

And then we get our allocation! You might adapt this command to be more specific to your resource manager. E.g., Slurm uses srun. After you run flux start, you are inside of a Flux instance on your allocation! Let’s run a simple job on our allocation. This first example will ask to see the hostnames of your nodes:

noodle:~$ flux run -N 4 hostname

You can sanity check the resources you have within the instance by then running:

noodle:~$ flux resource list
      free      4      160        0 noodle[220,221,222,223]
 allocated      0        0        0
      down      0        0        0

And you can echo $FLUX_URI to see the path of the socket that you will also need later:

noodle:~$ echo $FLUX_URI

We now have defined a goal for success - getting this listing working by running a command from a different cluster node.

Connect to the Instance

Next, let’s ssh into another cluster. Take the hostname where your instance is running, and create a proxy jump in your ~/.ssh/config:

Host noodle
    HostName noodle

Host noodle220
    hostname noodle220
    ProxyJump noodle


This ~/.ssh/config needs to be written on the cluster system where you are going to connect from. In many cases, the shared filesystem could map your home across clusters so you can see the file in multiple places.

You’ll first need to tell Flux to use ssh for the proxy command:

quartz:~$ export FLUX_SSH=ssh

Next, from this same location, try using flux proxy to connect to your Flux Instance! Target the URI that you found before, local:///var/tmp/flux-MLmxy2/local-0, and add the hostname noodle220 to the address:

quartz:~$ flux proxy ssh://noodle220/var/tmp/flux-MLmxy2/local-0

If you have trouble - use the force!

quartz:~$ flux proxy --force ssh://noodle220/var/tmp/flux-MLmxy2/local-0

You should then be able to run the same resource list:

quartz:~$ flux resource list
      free      4      160        0 noodle[220,221,222,223]
 allocated      0        0        0
      down      0        0        0

Next, try submitting a job! You should be able to see that you are running on the first cluster, but from the second.

quartz:~$ flux run hostname

If you are still connected to the first, you should also be able to query the jobs. E.g., here we submit a sleep from the second connected cluster:

quartz:~$ flux submit sleep 60

And then see it from either cluster node!

$ flux jobs
f22hdyb35 fluxuser sleep      R       1      1     1.842s

And that’s it! With this strategy, it should be easy to interact with Flux instances from two resources where ssh is supported. If you have any questions, please let us know.

Last update: Apr 24, 2024