P1 DTU HPC

The P1 DTU HPC is hosted at DTU and provides high-performance computing resources for P1 members (PhD and above). It is particularly suitable for medium to large-scale machine learning experiments and research projects.

Getting Access

Who's eligible?

Requirements: PhD or higher (exceptions may apply), valid Danish university email, and registered P1 affiliation.

Do I need to talk to someone first?

You are welcome to go straight to the sign-up forms below. If you would like a hand, the workload survey tells the P1 compute coordinator about your needs, expectations, and workload. We use it to point you to the right resource and to help you get the most out of it.

How do I sign up?

Two forms get you onto the cluster: start with P1 affiliation, then request a DTU account.

Before accessing the P1 DTU HPC, you must be a P1 affiliate. Sign in to your P1 profile (also reachable via Update Profile in the header), update your details, and submit a request to become an affiliate member. A P1 staff member will review and approve.

Once your affiliation is approved, fill out the DTU account request form. Only signups using an official university email address are accepted. The form is processed by Henning Christiansen (head of DTU's compute centre) and you will receive account details via email once created.

What if I don't qualify?

PhD P1 affiliates and above can skip this section. If you are a P1 member currently enrolled in an MSc, a research assistant (RA) role, or similar, you are not eligible for access without a written approval of exception from a responsible person (a P1 co-lead or faculty member) who can vouch for your request. Fill in the form and have the responsible person send a confirmation to compute-governance-p1@aicentre.dk so they can audit the request. The exception applies across all P1 clusters.

Getting Started

How do I connect?

The cluster is reachable at login9.hpc.dtu.dk via SSH. From outside DTU's network, first connect through Cisco AnyConnect to vpn.dtu.dk. See DTU IT's VPN guide for setup details, including MFA.

For persistent access, generate a key and copy the public half to the cluster:

ssh-keygen -t ed25519 -f ~/.ssh/keyname

ssh-copy-id -i ~/.ssh/keyname.pub username@login9.hpc.dtu.dk

Then connect:

ssh -i ~/.ssh/keyname username@login9.hpc.dtu.dk

How do I transfer data?

Use rsync for source code and small datasets, and rclone for large datasets. Both work over the standard SSH connection and are more ergonomic than a plain SFTP session. If rsync / rclone aren't available, an sftp client is the lowest-common-denominator fallback.

Project data lives under the shared project storage at /dtu/p1/. Home directories are capped at 30 GB; anything larger should land in your project's directory under /dtu/p1/. For very large transfers, DTU also exposes dedicated transfer nodes including transfer.gbar.dtu.dk.

Code from git

For source code, cloning on the login node is usually simpler than pushing from your machine. It keeps the cluster copy tracked, so later updates are a git pull rather than another rsync.

ssh username@login9.hpc.dtu.dk
cd /dtu/p1/username
git clone https://github.com/your-org/your-repo.git

An HTTPS remote is the path of least resistance and works with a personal access token for private repositories. An SSH remote needs its own key generated on the cluster and added to your git host. That key is separate from the one you use to log in to DTU:

ssh-keygen -t ed25519 -C "dtu-hpc"     # then add ~/.ssh/id_ed25519.pub to your git host
git clone git@github.com:your-org/your-repo.git

Keep code in git and data in /dtu/p1/. Use the rsync and rclone recipes below for the datasets and checkpoints that do not belong in a repository.

Inbound (push to DTU)

Send datasets and checkpoints to your project directory, not to your home directory, which only has 30 GB:

rsync -avz ./datasets username@login9.hpc.dtu.dk:/dtu/p1/username/datasets

Outbound (pull from DTU)

rsync -avz username@login9.hpc.dtu.dk:/dtu/p1/username/results ./results

Between clusters

All P1 clusters and most Danish university systems sit on forskningsnettet, so a cluster-to-cluster hop is much faster than round-tripping through your own machine. For large inter-cluster transfers, DTU's transfer.gbar.dtu.dk SFTP endpoint is the canonical staging point: SSH from it into other P1 systems and forward data from there.

SFTP fallback

When rsync / rclone aren't usable (managed lab machines, network maintenance, etc.), connect with the SFTP client directly:

sftp username@transfer.gbar.dtu.dk

How do I start an interactive job?

Interactive sessions run on the p1i queue, intended for package installation, test runs, and short debugging sessions:

bsub -q p1i -gpu "num=1:mode=exclusive_process" -W 1:00 -Is /bin/bash

Refer to the DTU LSF guide for queue and resource flags.

How do I submit a batch job?

Batch jobs go to the p1 queue:

#!/bin/bash
#BSUB -J train
#BSUB -q p1
#BSUB -gpu "num=2:mode=exclusive_process"
#BSUB -W 24:00
#BSUB -R "rusage[mem=64GB]"
#BSUB -o logs/%J.out
#BSUB -e logs/%J.err

module load cuda/12.4
python train.py

Create the log directory before submitting, since LSF fails the job at start if it cannot write the output files:

mkdir -p logs
bsub < train.sh

How do I monitor jobs?

bjobs                   # your jobs and their state
bjobs -l <jobid>        # full detail, including why a job is still pending
bpeek <jobid>           # tail the output of a running job
bqueues p1 p1i          # how busy the queues are
bkill <jobid>           # cancel

A job sitting in PEND is normal. bjobs -l states the reason, which is usually that the requested GPUs, memory, or wall time are not free yet. Asking for less tends to start sooner.

Policy & Support

What are the limits?

If your project requires more storage, contact the governance group at compute-governance-p1@aicentre.dk to discuss your needs.

How much storage do I get?

Your 500 GB project allowance is a slice of the cluster's 60 TiB shared pool. Check what you are using before a large transfer:

du -sh /dtu/p1/username     # project storage
du -sh ~                    # home, capped at 30 GB

Treat cluster storage as working space rather than an archive. Keep code in git and keep a second copy of anything you cannot regenerate, either on institutional storage or on your own machine. Before a project winds down, talk to the compute coordinator about what needs to be moved off.

What if I need more GPUs?

Default per-user limits keep the cluster fair when demand is high. Use this form when a run needs more than the default and needs to be protected: a multi-day training, a deadline, or a workload that cannot checkpoint. Tell us the size and the window, and the compute coordinator arranges it with the cluster operator.

If your workload checkpoints cleanly and you only want GPUs that are sitting idle, check whether your cluster lets you submit past the default limit directly. That path is faster and needs no request, at the cost of the job being cancellable when demand picks up.

Who do I contact?

Technical Support

Broken or missing packages, scheduler issues

DTU HPC Support Team

Policy Support

Priority access, queueing, access exceptions

compute-governance-p1@aicentre.dk

General Questions

Open chat with the P1 compute community

#compute #ask-compute

Compute Coordinator

Onboarding, workload guidance, general questions. Always happy to help

Helped with your work? Acknowledge ORCID 0000-0002-2740-1651

workload survey book a call bstja@dtu.dk

Hardware Specification

Nodes

8× Lenovo ThinkSystem SR665 V3 (7 batch, 1 interactive)

CPU per node

2× AMD EPYC 9354 (32 cores each)

RAM per node

768 GB

GPUs

16× NVIDIA H100 PCIe (80 GB), 2 per node

Shared storage

60 TiB at /dtu/p1, 500 GB default project quota

Home directory

30 GB

Storage fabric

NFSv4.2 over 100 Gbit EDR InfiniBand with RDMA, 11× Micron 7450 NVMe (RAID 5)

Operating system

AlmaLinux

Scheduler

IBM Spectrum LSF

Queues

p1 (batch, 7 nodes) / p1i (interactive, 1 node)