Cluster · DTU HPC

P1 DTU HPC

The P1 DTU HPC is hosted at DTU and provides high-performance computing resources for P1 members (PhD and above). It is particularly suitable for medium to large-scale machine learning experiments and research projects.

Getting Access

Talk to the compute coordinator

You are welcome to go straight to the sign-up forms below. If you would like a hand, the workload survey tells the P1 compute coordinator about your needs, expectations, and workload. We use it to point you to the right resource and to help you get the most out of it.

Two forms get you onto the cluster: start with P1 affiliation, then request a DTU account.

Before accessing the P1 DTU HPC, you must be a P1 affiliate. Sign in to your P1 profile (also reachable via Update Profile in the header), update your details, and submit a request to become an affiliate member. A P1 staff member will review and approve.

Once your affiliation is approved, fill out the DTU account request form. Only signups using an official university email address are accepted. The form is processed by Henning Christiansen (head of DTU's compute centre) and you will receive account details via email once created.

DTU Account Signup Form · Use your official university email address.

New tab

Access Exception

PhD P1 affiliates and above can skip this section. If you are a P1 member currently enrolled in an MSc, a research assistant (RA) role, or similar, you are not eligible for access without a written approval of exception from a responsible person (a P1 co-lead or faculty member) who can vouch for your request. Fill in the form and have the responsible person send a confirmation to compute-governance-p1@aicentre.dk so they can audit the request. The exception applies across all P1 clusters.

P1 HPC Access Exception Form · Requires sign-off from a P1 co-lead or faculty member.

New tab

Who's eligible?

Requirements: PhD or higher (exceptions may apply), valid Danish university email, and registered P1 affiliation.

Getting Started

How to connect

The cluster is reachable at login9.hpc.dtu.dk via SSH. From outside DTU's network, first connect through Cisco AnyConnect to vpn.dtu.dk. See DTU IT's VPN guide for setup details, including MFA.

For persistent access, generate a key and copy the public half to the cluster:

bash

ssh-keygen -t ed25519 -f ~/.ssh/keyname

bash

ssh-copy-id -i ~/.ssh/keyname.pub username@login9.hpc.dtu.dk

Then connect:

bash

ssh -i ~/.ssh/keyname username@login9.hpc.dtu.dk

How to transfer data

Use rsync for source code and small datasets, and rclone for large datasets. Both work over the standard SSH connection and are more ergonomic than a plain SFTP session. If rsync / rclone aren't available, an sftp client is the lowest-common-denominator fallback.

Project data lives under the shared project storage at /dtu/p1/. Home directories are capped at 30 GB; anything larger should land in your project's directory under /dtu/p1/. For very large transfers, DTU also exposes dedicated transfer nodes including transfer.gbar.dtu.dk.

Code from git

For source code, cloning on the login node is usually simpler than pushing from your machine. It keeps the cluster copy tracked, so later updates are a git pull rather than another rsync.

bash

ssh username@login9.hpc.dtu.dk
cd /dtu/p1/username
git clone https://github.com/your-org/your-repo.git

An HTTPS remote is the path of least resistance and works with a personal access token for private repositories. An SSH remote needs its own key generated on the cluster and added to your git host. That key is separate from the one you use to log in to DTU:

bash

ssh-keygen -t ed25519 -C "dtu-hpc"     # then add ~/.ssh/id_ed25519.pub to your git host
git clone git@github.com:your-org/your-repo.git

Keep code in git and data in /dtu/p1/. Use the rsync and rclone recipes below for the datasets and checkpoints that do not belong in a repository.

Inbound (push to DTU)

Send datasets and checkpoints to your project directory, not to your home directory, which only has 30 GB:

bash

rsync -avz ./datasets username@login9.hpc.dtu.dk:/dtu/p1/username/datasets

Outbound (pull from DTU)

bash

rsync -avz username@login9.hpc.dtu.dk:/dtu/p1/username/results ./results

Between clusters

All P1 clusters and most Danish university systems sit on forskningsnettet, so a cluster-to-cluster hop is much faster than round-tripping through your own machine. For large inter-cluster transfers, DTU's transfer.gbar.dtu.dk SFTP endpoint is the canonical staging point: SSH from it into other P1 systems and forward data from there.

SFTP fallback

When rsync / rclone aren't usable (managed lab machines, network maintenance, etc.), connect with the SFTP client directly:

bash

sftp username@transfer.gbar.dtu.dk

Software environment

Software comes from environment modules rather than being installed system-wide, so nothing is loaded until you ask for it:

bash

module avail            # everything on offer
module load cuda/12.4
module list             # what is loaded right now
module purge            # start clean

Load the same modules inside your job script as you did when testing interactively. A login shell and a batch job do not share an environment.

For Python, build a virtual environment in your project directory rather than your home directory, so package installs do not eat the 30 GB home quota:

bash

python3 -m venv /dtu/p1/username/venvs/myproject
source /dtu/p1/username/venvs/myproject/bin/activate
pip install -r requirements.txt

Containers are the more reproducible option when your stack is awkward to install. Apptainer runs them without root, and --nv exposes the node's GPUs inside the container:

bash

apptainer exec --nv /dtu/p1/username/images/pytorch.sif python train.py

How to start an interactive job

Interactive sessions run on the p1i queue, intended for package installation, test runs, and short debugging sessions:

bash

bsub -q p1i -gpu "num=1:mode=exclusive_process" -W 1:00 -Is /bin/bash

Refer to the DTU LSF guide for queue and resource flags.

How to submit a batch job

Batch jobs go to the p1 queue:

train.sh

#!/bin/bash
#BSUB -J train
#BSUB -q p1
#BSUB -gpu "num=2:mode=exclusive_process"
#BSUB -W 24:00
#BSUB -R "rusage[mem=64GB]"
#BSUB -o logs/%J.out
#BSUB -e logs/%J.err

module load cuda/12.4
python train.py

Create the log directory before submitting, since LSF fails the job at start if it cannot write the output files:

bash

mkdir -p logs
bsub < train.sh

Monitoring and cancelling jobs

bash

bjobs                   # your jobs and their state
bjobs -l <jobid>        # full detail, including why a job is still pending
bpeek <jobid>           # tail the output of a running job
bqueues p1 p1i          # how busy the queues are
bkill <jobid>           # cancel

A job sitting in PEND is normal. bjobs -l states the reason, which is usually that the requested GPUs, memory, or wall time are not free yet. Asking for less tends to start sooner.

Policy & Support

Usage Rules

Maximum wall time: 72 hours
Maximum GPUs per job: 2 (one node)
Maximum GPUs in use at once per person: about half the cluster, so roughly 8 of the 16 H100s
Storage: 500 GB per project under /dtu/p1/, plus a 30 GB home directory

If your project requires more storage, contact the governance group at compute-governance-p1@aicentre.dk to discuss your needs.

Storage, quotas, and backups

Your 500 GB project allowance is a slice of the cluster's 60 TiB shared pool. Check what you are using before a large transfer:

bash

du -sh /dtu/p1/username     # project storage
du -sh ~                    # home, capped at 30 GB

Treat cluster storage as working space rather than an archive. Keep code in git and keep a second copy of anything you cannot regenerate, either on institutional storage or on your own machine. Before a project winds down, talk to the compute coordinator about what needs to be moved off.

Temporary GPU Increase

Per-user GPU limits keep the cluster fair when demand is high. When resources are underutilized, you can request a temporary increase of your GPU allowance. Increases are granted on a best-effort basis and roll back when demand picks up again.

Temporary GPU Allowance Increase · Tell us which cluster, how many GPUs, and for how long.

New tab

Who to contact

Technical Support: Broken or missing packages, scheduler issues
Policy Support: Priority access, queueing, access exceptions
General Questions: Open chat with the P1 compute community
Compute Coordinator: Onboarding, workload guidance, general questions. Always happy to help; Helped with your work? Acknowledge ORCID 0000-0002-2740-1651

Hardware Specification

Nodes: 8× Lenovo ThinkSystem SR665 V3 (7 batch, 1 interactive)
CPU per node: 2× AMD EPYC 9354 (32 cores each)
RAM per node: 768 GB
GPUs: 16× NVIDIA H100 PCIe (80 GB), 2 per node
Shared storage: 60 TiB at /dtu/p1, 500 GB default project quota
Home directory: 30 GB
Storage fabric: NFSv4.2 over 100 Gbit EDR InfiniBand with RDMA, 11× Micron 7450 NVMe (RAID 5)
Operating system: AlmaLinux
Scheduler: IBM Spectrum LSF
Queues: p1 (batch, 7 nodes) / p1i (interactive, 1 node)

P1 DTU HPC

Getting Access

Talk to the compute coordinator

Sign-up Form

Access Exception

Who's eligible?

Getting Started

How to connect

How to transfer data

Code from git

Inbound (push to DTU)

Outbound (pull from DTU)

Between clusters

SFTP fallback

Software environment

How to start an interactive job

How to submit a batch job

Monitoring and cancelling jobs

Policy & Support

Usage Rules

Storage, quotas, and backups

Temporary GPU Increase

Who to contact

Hardware Specification