Cluster · Gefion HPC

P1 Gefion HPC

The P1 Gefion HPC is a sandbox providing on-demand access to H100 GPUs on Gefion for iterative development, testing, and small-scale pilots.

Its purpose is to get a workload ready to run at scale. You develop and mature it here, on the same hardware generation you will eventually run on, so that when you do scale up the work is already proven. The step to full 8-GPU production nodes is a separate matter and needs its own funding, for example a Novo Nordisk Foundation (NNF) compute voucher or another allocation. The sandbox is what puts you in a position to apply for one with something that already works.

Getting Access

Who's eligible?

Requirements: PhD or higher (exceptions may apply), valid Danish university email, and registered P1 affiliation.

Do I need to talk to someone first?

You are welcome to go straight to the sign-up form below. If you would like a hand, the workload survey tells the P1 compute coordinator about your needs, expectations, and workload. For the sandbox it also helps us plan the step up to full production nodes if your project scales.

Before accessing the P1 Gefion sandbox, you must be a P1 affiliate. Sign in to your P1 profile (also reachable via Update Profile in the header), update your details, and submit a request to become an affiliate member. A P1 staff member will review and approve.

Once your affiliation is approved, fill in the sandbox sign-up form. A DCAI team member will then contact you to get you set up with a DCAI username and initial-setup-password.

P1 Gefion Sandbox Sign-up Form · Requires an approved P1 affiliation.

New tab

What if I don't qualify?

PhD P1 affiliates and above can skip this section. If you are a P1 member currently enrolled in an MSc, a research assistant (RA) role, or similar, you are not eligible for access without a written approval of exception from a responsible person (a P1 co-lead or faculty member) who can vouch for your request. Fill in the form and have the responsible person send a confirmation to compute-governance-p1@aicentre.dk so they can audit the request. The exception applies across all P1 clusters.

P1 HPC Access Exception Form · Requires sign-off from a P1 co-lead or faculty member.

New tab

Getting Started

How do I connect?

Gefion uses Teleport for secure SSH access. Install the Teleport CLI client (tsh) and authenticate:

bash

tsh login --proxy secureremote.dcai.dk --user <username>

Then add the following to your ~/.ssh/config:

~/.ssh/config

Host gefion
  Port 3022
  User <username>
  HostName slurm-login-01.hpc.ite.dcai.dk
  ProxyCommand tsh proxy ssh --cluster=secureremote.dcai.dk --proxy=secureremote.dcai.dk:443 %r@%h:%p
  CertificateFile ~/.tsh/keys/secureremote.dcai.dk/<username>-ssh/secureremote.dcai.dk-cert.pub
  ConnectTimeout 30
  ControlMaster no
  ControlPersist no
  IdentityFile ~/.tsh/keys/secureremote.dcai.dk/<username>
  UserKnownHostsFile ~/.tsh/known_hosts

Connect with:

bash

ssh gefion

Visual Studio Code

With the SSH config in place you can use the VS Code Remote SSH extension to connect to gefion via the command palette (Ctrl+Shift+P, then "Remote-SSH: Connect to Host").

How do I transfer data?

Use rsync for source code and small datasets, and rclone for large datasets. Both work over the Teleport-backed SSH access and are more ergonomic than a plain SFTP session. If rsync / rclone hit issues (they wrap OpenSSH and don't tolerate every intermediate config), fall back to an sftp client directly.

Large transfers go through the SFTP endpoint xfer.dcai.dk, which requires your public IP to be whitelisted. Contact DCAI support and share the output of curl ifconfig.io to request access.

~/.ssh/config

Host gefion-xfer
  Port 2234
  User <username>
  HostName xfer.dcai.dk

~/.config/rclone/rclone.conf

[gefion]
disable_hashcheck = true
shell_type = unix
ssh = ssh gefion-xfer
type = sftp

Code from git

For source code, cloning on the login node is usually simpler than pushing from your machine. It keeps the cluster copy tracked, so later updates are a git pull rather than another rsync.

bash

ssh gefion
git clone https://github.com/your-org/your-repo.git

An HTTPS remote is the path of least resistance and works with a personal access token for private repositories. An SSH remote needs its own key generated on the cluster and added to your git host, separate from the Teleport credentials you use to log in:

bash

ssh-keygen -t ed25519 -C "gefion"     # then add ~/.ssh/id_ed25519.pub to your git host
git clone git@github.com:your-org/your-repo.git

The P1 partition sits on Gefion's open system, so the login node reaches public git hosts normally. Keep code in git and move only datasets and checkpoints with the rsync and rclone recipes below.

Inbound (push to Gefion)

Small data: rsync over the existing gefion SSH host:

bash

rsync -avz ./src gefion:~/project/src

Large datasets: rclone via the SFTP endpoint:

bash

rclone sync ./datasets gefion:to_gefion/dest --dry-run   # preview changes
rclone sync ./datasets gefion:to_gefion/dest             # sync for real
rclone ls gefion:to_gefion/dest                          # list files

Outbound (pull from Gefion)

Same tools, in reverse; run from your local machine pulling from gefion:...:

bash

rsync -avz gefion:~/project/results ./results

bash

rclone sync gefion:from_gefion/results ./results

Between clusters

For large inter-cluster transfers, the recommended bridge is via the DTU transfer node. All P1 clusters and most Danish university systems sit on forskningsnettet, so going cluster to cluster is much faster than round-tripping via your local machine over the public internet.

From inside Gefion, SFTP into transfer.gbar.dtu.dk, then SSH into the DTU P1 cluster from there and move the data further. Use the sftp client directly for this hop, since rsync and rclone wrap OpenSSH and currently don't traverse this path cleanly.

bash

# from inside Gefion
sftp <dtu-username>@transfer.gbar.dtu.dk
# then SSH into your DTU account and pull from /tmp or your transfer landing dir

SFTP fallback

If rsync / rclone aren't working, connect with the SFTP client directly:

bash

sftp gefion-xfer

Tip: Using the DTU cluster as a jump host

If your IP cannot be whitelisted on xfer.dcai.dk but you do have DTU access, you can use the DTU cluster as a proxy. Add these entries:

~/.ssh/config

Host dtu-transfer
  ForwardAgent yes
  User <dtu-username>
  HostName transfer.gbar.dtu.dk
  ControlMaster no
  ControlPersist no

Host gefion-jump
  Port 2234
  User <gefion-username>
  HostName xfer.dcai.dk
  ProxyJump dtu-transfer

~/.config/rclone/rclone.conf

[gefion-jump]
disable_hashcheck = true
shell_type = unix
ssh = ssh gefion-jump
type = sftp

Then use gefion-jump: instead of gefion: in your rclone commands.

How do I start an interactive job?

Request an interactive session with GPUs via srun:

bash

ssh gefion
srun --pty -N 1 -n 1 --gpus=2 --mem=64GB --time=1:00:00 /bin/bash

Which will open an interactive session with 2 H100 GPUs and 64 GB of RAM. Size --mem for the data you are loading rather than the GPUs: the nodes carry 2 TB, and asking for a few GB is the usual reason a job dies partway through with an out-of-memory error.

How do I submit a batch job?

train.sh

#!/bin/bash

#SBATCH --account=<account-nb>
#SBATCH --job-name=testing-train-script
#SBATCH --nodes=1
#SBATCH --gpus=2
#SBATCH --mem=20GB
#SBATCH --time=1:00:00
#SBATCH --error=slurm_files/error/%x_%j.err
#SBATCH --output=slurm_files/output/%x_%j.out

cd $SLURM_SUBMIT_DIR
module load CUDA/12.8
nvidia-smi
uv run train.py

Create the output directories before submitting, since SLURM fails the job at start if it cannot write the log files:

bash

mkdir -p slurm_files/output slurm_files/error
sbatch train.sh              # submit above script 1 time
sbatch --array=0-7 train.sh  # submit above script 8 times

How do I monitor jobs?

bash

squeue --me                                            # your jobs and their state
squeue --me -o "%.10i %.9P %.20j %.2t %.10M %R"        # %R gives the pending reason
sinfo                                                  # node and partition availability
scancel <jobid>                                        # cancel
sacct -j <jobid> --format=JobID,State,Elapsed,MaxRSS   # after it finishes

A job sitting in PD is normal. The reason in %R is usually Resources (the GPUs or memory you asked for are busy) or Priority (other jobs are ahead of you). sacct after the fact is the fastest way to see whether a job was killed for exceeding its memory or time request.

Policy & Support

What are the limits?

Compute

Max wall time: Not published; ask the compute coordinator before planning a long run
GPUs per job: 2
GPUs at once, per person: 4, as 2 concurrent jobs of 2 GPUs, more while GPUs sit idle

Storage

Home directory: 25 GB
Project storage: 500 GB default
Shared pool: 300 TB

16 H100 GPUs in the sandbox in total, shared across everyone using it
User accounts are individual and non-transferable
Accounts can be associated with one or more Gefion projects

Limits may be adjusted on an ongoing basis to meet the needs of P1 students and faculty.

How much storage do I get?

Your home directory holds 25 GB, so datasets and checkpoints belong in your project space against the 500 GB default quota. Check what you are using before a large transfer:

bash

du -sh ~                    # your home directory, 25 GB
df -h .                     # space left on the filesystem you are standing in

Remember that data arriving through the SFTP endpoint lands on the transfer staging area rather than in your home directory, so it counts twice until you clean up the staged copy.

Treat cluster storage as working space rather than an archive. Keep code in git and keep a second copy of anything you cannot regenerate, either on institutional storage or on your own machine. Before a project winds down, talk to the compute coordinator about what needs to be moved off.

What if I need more GPUs?

You can allocate beyond the default limit of 2 concurrent jobs at a maximum of 2 GPUs per job when GPUs are idle on the two NVIDIA DGX H100 nodes in our group iu_0096. Submit the larger job directly; no request is needed. Check what is free with sinfo first.

Where a run has to finish by a date, or cannot checkpoint at all, ask for it to be planned instead.

Default per-user limits keep the cluster fair when demand is high. Use this form when a run needs more than the default and needs to be protected: a multi-day training, a deadline, or a workload that cannot checkpoint. Tell us the size and the window, and the compute coordinator arranges it with the cluster operator.

Planned GPU Allocation · Tell us which cluster, how many GPUs, for how long, and by when.

New tab

Who do I contact?

Technical Support: Broken or missing packages, scheduler issues
Policy Support: Priority access, queueing, access exceptions
General Questions: Open chat with the P1 compute community
Compute Coordinator: Onboarding, workload guidance, general questions. Always happy to help; Helped with your work? Acknowledge ORCID 0000-0002-2740-1651

Hardware Specification

Nodes: 2× NVIDIA DGX H100 (8 GPUs each)
GPUs: 16× H100 SXM, 640 GB GPU memory, around 32 petaFLOPS FP8
CPU per node: 2× Intel Xeon Platinum 8480C (56 cores each)
RAM per node: 2 TB
Interconnect: NVLink at 900 GB/s GPU to GPU, InfiniBand up to 400 Gbps
Shared storage: 300 TB, 500 GB default quota
Home directory: 25 GB
Operating system: NVIDIA DGX OS (Ubuntu based)
Scheduler: Slurm

P1 Gefion HPC

Getting Access

Who's eligible?

Do I need to talk to someone first?

How do I sign up?

What if I don't qualify?

Getting Started

How do I connect?

Visual Studio Code

How do I transfer data?

Code from git

Inbound (push to Gefion)

Outbound (pull from Gefion)

Between clusters

SFTP fallback

How do I start an interactive job?

How do I submit a batch job?

How do I monitor jobs?

Policy & Support

What are the limits?

How much storage do I get?

What if I need more GPUs?

Who do I contact?

Hardware Specification