# P1 DTU HPC

> **Provider documentation.** This page is an overview to help you onboard and get started. The technical and advanced documentation is maintained by DTU DCC, who host and operate the system.
>
> DTU DCC documentation: https://www.hpc.dtu.dk/?page_id=5028

The P1 DTU HPC is hosted at DTU and provides high-performance computing resources for P1 members
(PhD and above). It is particularly suitable for medium to large-scale machine learning
experiments and research projects.

## Getting Access

### Talk to the compute coordinator

You are welcome to go straight to the sign-up forms below. If you would like a hand, the
[workload survey](/survey/) tells the P1 compute coordinator about your needs, expectations, and
workload. We use it to point you to the right resource and to help you get the most out of it.

### Sign-up Form

Two forms get you onto the cluster: start with P1 affiliation, then request a DTU account.

Before accessing the P1 DTU HPC, you must be a P1 affiliate. Sign in to
[your P1 profile](https://www.aicentre.dk/p1adb) (also reachable via **Update Profile** in the header), update your
details, and submit a request to become an affiliate member. A P1 staff member will review and
approve.

Once your affiliation is approved, fill out the DTU account request form. Only signups using an
official university email address are accepted. The form is processed by Henning Christiansen
(head of DTU's compute centre) and you will receive account details via email once created.

**Form:** [DTU Account Signup Form](https://forms.cloud.microsoft/e/DG5qCfs6Wm) Use your official university email address.

### Access Exception

**Form:** [Access exception request](https://docs.google.com/forms/d/e/1FAIpQLSciaOm-CYwl48LGGZC7qlmCSPU7kfEitjGy4kvZXAXEPbo_eA/viewform) For MSc students, research assistants, and similar roles who need cluster access with written approval from a responsible P1 co-lead or faculty member.

### Who's eligible?

**Requirements:** PhD or higher (exceptions may apply), valid Danish university email, and
registered P1 affiliation.

## Getting Started

### How to connect

The cluster is reachable at `login9.hpc.dtu.dk` via SSH. From outside DTU's network, first connect
through Cisco AnyConnect to `vpn.dtu.dk`. See
[DTU IT's VPN guide](https://www.hpc.dtu.dk/?page_id=2501) for setup details, including MFA.

For persistent access, generate a key and copy the public half to the cluster:

```bash
ssh-keygen -t ed25519 -f ~/.ssh/keyname
```

```bash
ssh-copy-id -i ~/.ssh/keyname.pub username@login9.hpc.dtu.dk
```

Then connect:

```bash
ssh -i ~/.ssh/keyname username@login9.hpc.dtu.dk
```

### How to transfer data

Use `rsync` for source code and small datasets, and [rclone](https://rclone.org/) for large
datasets. Both work over the standard SSH connection and are more ergonomic than a plain SFTP
session. If `rsync` / `rclone` aren't available, an `sftp` client is the
lowest-common-denominator fallback.

Project data lives under the shared project storage at `/dtu/p1/`. Home directories are capped at
30 GB; anything larger should land in your project's directory under `/dtu/p1/`. For very large
transfers, DTU also exposes
[dedicated transfer nodes](https://www.hpc.dtu.dk/?page_id=2671) including
`transfer.gbar.dtu.dk`.

#### Code from git

For source code, cloning on the login node is usually simpler than pushing from your machine. It
keeps the cluster copy tracked, so later updates are a `git pull` rather than another `rsync`.

```bash
ssh username@login9.hpc.dtu.dk
cd /dtu/p1/username
git clone https://github.com/your-org/your-repo.git
```

An HTTPS remote is the path of least resistance and works with a personal access token for
private repositories. An SSH remote needs its own key generated on the cluster and added to your
git host. That key is separate from the one you use to log in to DTU:

```bash
ssh-keygen -t ed25519 -C "dtu-hpc"     # then add ~/.ssh/id_ed25519.pub to your git host
git clone git@github.com:your-org/your-repo.git
```

Keep code in git and data in `/dtu/p1/`. Use the `rsync` and `rclone` recipes below for the
datasets and checkpoints that do not belong in a repository.

#### Inbound (push to DTU)

Send datasets and checkpoints to your project directory, not to your home directory, which only
has 30 GB:

```bash
rsync -avz ./datasets username@login9.hpc.dtu.dk:/dtu/p1/username/datasets
```

#### Outbound (pull from DTU)

```bash
rsync -avz username@login9.hpc.dtu.dk:/dtu/p1/username/results ./results
```

#### Between clusters

All P1 clusters and most Danish university systems sit on
[forskningsnettet](/ecosystem/#the-danish-research-network), so a cluster-to-cluster hop is much
faster than round-tripping through your own machine. For large inter-cluster transfers, DTU's
`transfer.gbar.dtu.dk` SFTP endpoint is the canonical staging point: SSH from it into other P1
systems and forward data from there.

#### SFTP fallback

When `rsync` / `rclone` aren't usable (managed lab machines, network maintenance, etc.), connect
with the SFTP client directly:

```bash
sftp username@transfer.gbar.dtu.dk
```

### Software environment

Software comes from environment modules rather than being installed system-wide, so nothing is
loaded until you ask for it:

```bash
module avail            # everything on offer
module load cuda/12.4
module list             # what is loaded right now
module purge            # start clean
```

Load the same modules inside your job script as you did when testing interactively. A login shell
and a batch job do not share an environment.

For Python, build a virtual environment in your project directory rather than your home directory,
so package installs do not eat the 30 GB home quota:

```bash
python3 -m venv /dtu/p1/username/venvs/myproject
source /dtu/p1/username/venvs/myproject/bin/activate
pip install -r requirements.txt
```

Containers are the more reproducible option when your stack is awkward to install. Apptainer runs
them without root, and `--nv` exposes the node's GPUs inside the container:

```bash
apptainer exec --nv /dtu/p1/username/images/pytorch.sif python train.py
```

### How to start an interactive job

Interactive sessions run on the `p1i` queue, intended for package installation, test runs, and
short debugging sessions:

```bash
bsub -q p1i -gpu "num=1:mode=exclusive_process" -W 1:00 -Is /bin/bash
```

Refer to the [DTU LSF guide](https://www.hpc.dtu.dk/?page_id=2698) for queue and resource flags.

### How to submit a batch job

Batch jobs go to the `p1` queue:

```bash file=train.sh
#!/bin/bash
#BSUB -J train
#BSUB -q p1
#BSUB -gpu "num=2:mode=exclusive_process"
#BSUB -W 24:00
#BSUB -R "rusage[mem=64GB]"
#BSUB -o logs/%J.out
#BSUB -e logs/%J.err

module load cuda/12.4
python train.py
```

Create the log directory before submitting, since LSF fails the job at start if it cannot write
the output files:

```bash
mkdir -p logs
bsub < train.sh
```

### Monitoring and cancelling jobs

```bash
bjobs                   # your jobs and their state
bjobs -l <jobid>        # full detail, including why a job is still pending
bpeek <jobid>           # tail the output of a running job
bqueues p1 p1i          # how busy the queues are
bkill <jobid>           # cancel
```

A job sitting in `PEND` is normal. `bjobs -l` states the reason, which is usually that the
requested GPUs, memory, or wall time are not free yet. Asking for less tends to start sooner.

## Policy & Support

### Usage Rules

- Maximum wall time: 72 hours
- Maximum GPUs per job: 2 (one node)
- Maximum GPUs in use at once per person: about half the cluster, so roughly 8 of the 16 H100s
- Storage: 500 GB per project under `/dtu/p1/`, plus a 30 GB home directory

If your project requires more storage, contact the governance group at
[compute-governance-p1@aicentre.dk](mailto:compute-governance-p1@aicentre.dk) to discuss your
needs.

### Storage, quotas, and backups

Your 500 GB project allowance is a slice of the cluster's 60 TiB shared pool. Check what you are
using before a large transfer:

```bash
du -sh /dtu/p1/username     # project storage
du -sh ~                    # home, capped at 30 GB
```

Treat cluster storage as working space rather than an archive. Keep code in git and keep a second
copy of anything you cannot regenerate, either on institutional storage or on your own machine.
Before a project winds down, talk to the compute coordinator about what needs to be moved off.

> **GDPR Compliance**
>
> The P1 DTU HPC is only intended for non-GDPR data: public datasets, open benchmarks, etc. Your
> home directory is not readable by other users, but the cluster as a whole is not an approved
> environment for processing personal data, and no file-level measure on your side changes that.
> If your project involves personal data, talk to the compute coordinator before you start.

### Temporary GPU Increase

**Form:** [Temporary GPU allowance increase](https://forms.cloud.microsoft/e/yN8Vxi7KTc) Request a temporary raise of your GPU limit while a cluster is underutilized.

**Citing this cluster:** if the P1 DTU HPC contributed to your work, please cite the DTU Computing Center resources DOI, [10.48714/DTU.HPC.0001](https://doi.org/10.48714/DTU.HPC.0001). That covers help from the DTU DCC team or the P1 compute coordinator just as much as time spent running jobs.

### Who to contact

- DTU HPC Support Team: support@hpc.dtu.dk
- P1 compute coordinator: bstja@dtu.dk
- Helped with your work? Acknowledge the coordinator's ORCID: https://orcid.org/0000-0002-2740-1651

## Hardware Specification

| | |
| --- | --- |
| Nodes | 8× Lenovo ThinkSystem SR665 V3 (7 batch, 1 interactive) |
| CPU per node | 2× AMD EPYC 9354 (32 cores each) |
| RAM per node | 768 GB |
| GPUs | 16× NVIDIA H100 PCIe (80 GB), 2 per node |
| Shared storage | 60 TiB at /dtu/p1, 500 GB default project quota |
| Home directory | 30 GB |
| Storage fabric | NFSv4.2 over 100 Gbit EDR InfiniBand with RDMA, 11× Micron 7450 NVMe (RAID 5) |
| Operating system | AlmaLinux |
| Scheduler | IBM Spectrum LSF |
| Queues | p1 (batch, 7 nodes) / p1i (interactive, 1 node) |