Cluster · Gefion HPC

P1 Gefion HPC

The P1 Gefion HPC is a sandbox providing on-demand access to H100 GPUs on Gefion for iterative development, testing, and small-scale pilots.

Its purpose is to get a workload ready to run at scale. You develop and mature it here, on the same hardware generation you will eventually run on, so that when you do scale up the work is already proven. The step to full 8-GPU production nodes is a separate matter and needs its own funding, for example a Novo Nordisk Foundation (NNF) compute voucher or another allocation. The sandbox is what puts you in a position to apply for one with something that already works.

Getting Access

Talk to the compute coordinator

You are welcome to go straight to the sign-up form below. If you would like a hand, the workload survey tells the P1 compute coordinator about your needs, expectations, and workload. For the sandbox it also helps us plan the step up to full production nodes if your project scales.

Before accessing the P1 Gefion sandbox, you must be a P1 affiliate. Sign in to your P1 profile (also reachable via Update Profile in the header), update your details, and submit a request to become an affiliate member. A P1 staff member will review and approve.

Once your affiliation is approved, fill in the sandbox sign-up form. A DCAI team member will then contact you to get you set up with a DCAI username and initial-setup-password.

P1 Gefion Sandbox Sign-up Form · Requires an approved P1 affiliation.

New tab

Access Exception

PhD P1 affiliates and above can skip this section. If you are a P1 member currently enrolled in an MSc, a research assistant (RA) role, or similar, you are not eligible for access without a written approval of exception from a responsible person (a P1 co-lead or faculty member) who can vouch for your request. Fill in the form and have the responsible person send a confirmation to compute-governance-p1@aicentre.dk so they can audit the request. The exception applies across all P1 clusters.

P1 HPC Access Exception Form · Requires sign-off from a P1 co-lead or faculty member.

New tab

Who's eligible?

Requirements: PhD or higher (exceptions may apply), valid Danish university email, and registered P1 affiliation.

Getting Started

How to connect

Gefion uses Teleport for secure SSH access. Install the Teleport CLI client (tsh) and authenticate:

bash

tsh login --proxy secureremote.dcai.dk --user <username>

Then add the following to your ~/.ssh/config:

~/.ssh/config

Host gefion
  Port 3022
  User <username>
  HostName slurm-login-01.hpc.ite.dcai.dk
  ProxyCommand tsh proxy ssh --cluster=secureremote.dcai.dk --proxy=secureremote.dcai.dk:443 %r@%h:%p
  CertificateFile ~/.tsh/keys/secureremote.dcai.dk/<username>-ssh/secureremote.dcai.dk-cert.pub
  ConnectTimeout 30
  ControlMaster no
  ControlPersist no
  IdentityFile ~/.tsh/keys/secureremote.dcai.dk/<username>
  UserKnownHostsFile ~/.tsh/known_hosts

Connect with:

bash

ssh gefion

Visual Studio Code

With the SSH config in place you can use the VS Code Remote SSH extension to connect to gefion via the command palette (Ctrl+Shift+P, then "Remote-SSH: Connect to Host").

How to transfer data

Use rsync for source code and small datasets, and rclone for large datasets. Both work over the Teleport-backed SSH access and are more ergonomic than a plain SFTP session. If rsync / rclone hit issues (they wrap OpenSSH and don't tolerate every intermediate config), fall back to an sftp client directly.

Large transfers go through the SFTP endpoint xfer.dcai.dk, which requires your public IP to be whitelisted. Contact DCAI support and share the output of curl ifconfig.io to request access.

~/.ssh/config

Host gefion-xfer
  Port 2234
  User <username>
  HostName xfer.dcai.dk

~/.config/rclone/rclone.conf

[gefion]
disable_hashcheck = true
shell_type = unix
ssh = ssh gefion-xfer
type = sftp

Code from git

For source code, cloning on the login node is usually simpler than pushing from your machine. It keeps the cluster copy tracked, so later updates are a git pull rather than another rsync.

bash

ssh gefion
git clone https://github.com/your-org/your-repo.git

An HTTPS remote is the path of least resistance and works with a personal access token for private repositories. An SSH remote needs its own key generated on the cluster and added to your git host, separate from the Teleport credentials you use to log in:

bash

ssh-keygen -t ed25519 -C "gefion"     # then add ~/.ssh/id_ed25519.pub to your git host
git clone git@github.com:your-org/your-repo.git

The P1 partition sits on Gefion's open system, so the login node reaches public git hosts normally. Keep code in git and move only datasets and checkpoints with the rsync and rclone recipes below.

Inbound (push to Gefion)

Small data: rsync over the existing gefion SSH host:

bash

rsync -avz ./src gefion:~/project/src

Large datasets: rclone via the SFTP endpoint:

bash

rclone sync ./datasets gefion:to_gefion/dest --dry-run   # preview changes
rclone sync ./datasets gefion:to_gefion/dest             # sync for real
rclone ls gefion:to_gefion/dest                          # list files

Outbound (pull from Gefion)

Same tools, in reverse; run from your local machine pulling from gefion:...:

bash

rsync -avz gefion:~/project/results ./results

bash

rclone sync gefion:from_gefion/results ./results

Between clusters

For large inter-cluster transfers, the recommended bridge is via the DTU transfer node. All P1 clusters and most Danish university systems sit on forskningsnettet, so going cluster to cluster is much faster than round-tripping via your local machine over the public internet.

From inside Gefion, SFTP into transfer.gbar.dtu.dk, then SSH into the DTU P1 cluster from there and move the data further. Use the sftp client directly for this hop, since rsync and rclone wrap OpenSSH and currently don't traverse this path cleanly.

bash

# from inside Gefion
sftp <dtu-username>@transfer.gbar.dtu.dk
# then SSH into your DTU account and pull from /tmp or your transfer landing dir

SFTP fallback

If rsync / rclone aren't working, connect with the SFTP client directly:

bash

sftp gefion-xfer

Tip: Using the DTU cluster as a jump host

If your IP cannot be whitelisted on xfer.dcai.dk but you do have DTU access, you can use the DTU cluster as a proxy. Add these entries:

~/.ssh/config

Host dtu-transfer
  ForwardAgent yes
  User <dtu-username>
  HostName transfer.gbar.dtu.dk
  ControlMaster no
  ControlPersist no

Host gefion-jump
  Port 2234
  User <gefion-username>
  HostName xfer.dcai.dk
  ProxyJump dtu-transfer

~/.config/rclone/rclone.conf

[gefion-jump]
disable_hashcheck = true
shell_type = unix
ssh = ssh gefion-jump
type = sftp

Then use gefion-jump: instead of gefion: in your rclone commands.

Software environment

Software comes from environment modules rather than being installed system-wide. module spider searches the whole tree, including modules that stay hidden until a dependency is loaded:

bash

module avail            # what is loadable right now
module spider CUDA      # search everything, including hidden modules
module load CUDA/12.8
module list
module purge            # start clean

Load the same modules inside your job script as you did when testing interactively. A login shell and a batch job do not share an environment.

For Python, uv is the quickest route to a reproducible environment and is what the batch example below uses:

bash

uv venv
uv pip install -r requirements.txt
uv run train.py

Containers are the more reproducible option when your stack is awkward to install. Apptainer runs them without root, and --nv exposes the node's GPUs inside the container:

bash

apptainer exec --nv ~/images/pytorch.sif python train.py

How to start an interactive job

Request an interactive session with GPUs via srun:

bash

ssh gefion
srun --pty -N 1 -n 1 --gpus=2 --mem=64GB --time=1:00:00 /bin/bash

Which will open an interactive session with 2 H100 GPUs and 64 GB of RAM. Size --mem for the data you are loading rather than the GPUs: the nodes carry 2 TB, and asking for a few GB is the usual reason a job dies partway through with an out-of-memory error.

How to submit a batch job

train.sh

#!/bin/bash

#SBATCH --account=<account-nb>
#SBATCH --job-name=testing-train-script
#SBATCH --nodes=1
#SBATCH --gpus=2
#SBATCH --mem=20GB
#SBATCH --time=1:00:00
#SBATCH --error=slurm_files/error/%x_%j.err
#SBATCH --output=slurm_files/output/%x_%j.out

cd $SLURM_SUBMIT_DIR
module load CUDA/12.8
nvidia-smi
uv run train.py

Create the output directories before submitting, since SLURM fails the job at start if it cannot write the log files:

bash

mkdir -p slurm_files/output slurm_files/error
sbatch train.sh              # submit above script 1 time
sbatch --array=0-7 train.sh  # submit above script 8 times

Monitoring and cancelling jobs

bash

squeue --me                                            # your jobs and their state
squeue --me -o "%.10i %.9P %.20j %.2t %.10M %R"        # %R gives the pending reason
sinfo                                                  # node and partition availability
scancel <jobid>                                        # cancel
sacct -j <jobid> --format=JobID,State,Elapsed,MaxRSS   # after it finishes

A job sitting in PD is normal. The reason in %R is usually Resources (the GPUs or memory you asked for are busy) or Priority (other jobs are ahead of you). sacct after the fact is the fastest way to see whether a job was killed for exceeding its memory or time request.

Policy & Support

Usage Rules

Default per-person limit: 2 concurrent jobs with 2 GPUs each, so 4 GPUs in use at once
16 H100 GPUs in the sandbox in total, shared across everyone using it
300 TB shared storage, with a 500 GB default quota and a 25 GB home directory
User accounts are individual and non-transferable
Accounts can be associated with one or more Gefion projects

Limits may be adjusted on an ongoing basis to meet the needs of P1 students and faculty.

Storage, quotas, and backups

Your home directory holds 25 GB, so datasets and checkpoints belong in your project space against the 500 GB default quota. Check what you are using before a large transfer:

bash

du -sh ~                    # your home directory, 25 GB
df -h .                     # space left on the filesystem you are standing in

Remember that data arriving through the SFTP endpoint lands on the transfer staging area rather than in your home directory, so it counts twice until you clean up the staged copy.

Treat cluster storage as working space rather than an archive. Keep code in git and keep a second copy of anything you cannot regenerate, either on institutional storage or on your own machine. Before a project winds down, talk to the compute coordinator about what needs to be moved off.

Temporary GPU Increase

Per-user GPU limits keep the cluster fair when demand is high. When resources are underutilized, you can request a temporary increase of your GPU allowance. Increases are granted on a best-effort basis and roll back when demand picks up again.

Temporary GPU Allowance Increase · Tell us which cluster, how many GPUs, and for how long.

New tab

Who to contact

Technical Support: Broken or missing packages, scheduler issues
Policy Support: Priority access, queueing, access exceptions
General Questions: Open chat with the P1 compute community
Compute Coordinator: Onboarding, workload guidance, general questions. Always happy to help; Helped with your work? Acknowledge ORCID 0000-0002-2740-1651

Hardware Specification

Nodes: 2× NVIDIA DGX H100 (8 GPUs each)
GPUs: 16× H100 SXM, 640 GB GPU memory, around 32 petaFLOPS FP8
CPU per node: 2× Intel Xeon Platinum 8480C (56 cores each)
RAM per node: 2 TB
Interconnect: NVLink at 900 GB/s GPU to GPU, InfiniBand up to 400 Gbps
Shared storage: 300 TB, 500 GB default quota
Home directory: 25 GB
Operating system: NVIDIA DGX OS (Ubuntu based)
Scheduler: Slurm