P1 NGC HPC

The P1 NGC HPC is hosted at the National Genome Centre and is designed for secure data processing with GDPR compliance. Projects are isolated from one another through VMware vSphere virtualization, and the facility carries petabyte-scale storage, so it suits sensitive-data work with large datasets. For details on the underlying infrastructure, see NGC's supercomputer page.

Getting Access

Talk to the compute coordinator

You are welcome to go straight to the sign-up steps below. If you would like a hand, the workload survey tells the P1 compute coordinator about your needs, expectations, and workload. For NGC it is worth doing early, since GDPR projects need their data-processing paperwork lined up.

Sign-up Form

Before accessing the P1 NGC HPC, you must be a P1 affiliate. Sign in to your P1 profile (also reachable via Update Profile in the header), update your details, and submit a request to become an affiliate member. A P1 staff member will review and approve.

Once your affiliation is approved, three steps get you an account.

1. Obtain the user application form from NGC. They issue it themselves, so it is not something we can hand out. Their how to apply for access page explains how to get it.
2. Complete and sign it, then email it to the compute coordinator.
3. Fill in the access request below, so the coordinator knows the paperwork is on its way and can raise the request with NGC.

You will be added to the NGC Slack channel once you gain access.

Access Exception

PhD P1 affiliates and above can skip this section. If you are a P1 member currently enrolled in an MSc, a research assistant (RA) role, or similar, you are not eligible for access without a written approval of exception from a responsible person (a P1 co-lead or faculty member) who can vouch for your request. Fill in the form and have the responsible person send a confirmation to compute-governance-p1@aicentre.dk so they can audit the request. The exception applies across all P1 clusters.

Who's eligible?

Requirements: PhD or higher (exceptions may apply), valid Danish university email, and registered P1 affiliation.

Each project needs to bring a record (a signed Data Processing Agreement) that explicitly mentions NGC as a data processor and that the data is allowed to be stored there. If the project poses a high risk to individuals whose personal data is being processed, a Data Protection Impact Assessment (DPIA) is also required.

Getting Started

How to connect

The P1 NGC HPC is an air-gapped system requiring multi-factor authentication and a remote desktop client. Once registered, you'll receive specific access instructions including an Omnissa Remote Desktop client.

Once connected with the remote desktop client, access the login node using ssh -X <your-username>@login.

Useful hosts:

https://console.cld076.vmc/status   # Internal status page
cld076-0004.cld076.vmc              # Internal SFTP
sftp.spc.ngc.dk                     # External Ingress/Egress (SFTP)

How to transfer data

The P1 NGC HPC is air-gapped, so rsync and rclone from outside aren't available end-to-end: every transfer goes through approved SFTP staging under admin supervision. Request access to /data/upload from your NGC admin as your project's data gateway, then add an SSH entry to your ~/.ssh/config:

Host ngc
    HostName sftp.spc.ngc.dk
    Port 6433
    User <your-username>_sftp
    HostKeyAlgorithms +ssh-rsa

Code from git

The same air gap applies to git: the cluster cannot reach GitHub, GitLab, or any other public forge directly. An internal GitHub proxy covers some public repositories, so try a plain clone first and fall back to a bundle when the proxy does not have what you need.

A bundle is a single file holding the full history, which means the cluster copy stays a real git repository rather than a flattened snapshot. Private repositories always take this path.

On your own machine:

git bundle create repo.bundle --all
scp repo.bundle ngc:/data/upload/

Then on the cluster, clone from the staged bundle:

scp <your-username>_sftp@cld076-0004.cld076.vmc:/data/upload/repo.bundle ~/
git clone repo.bundle your-repo

To ship later changes, create a bundle covering only the new commits and pull it in:

git bundle create update.bundle <last-transferred-commit>..HEAD    # on your machine
git pull ~/update.bundle HEAD                                      # on the cluster

If a bundle is more machinery than you need, the Small files recipe below copies a plain working tree without any history.

Inbound (push to NGC)

From outside the cluster, upload to the staging area:

scp ~/datasets/ISLES-2022.zip ngc:/data/upload/

Then inside the cluster, copy from staging into your home directory:

scp <your-username>_sftp@cld076-0004.cld076.vmc:/data/upload/ISLES-2022.zip ~/datasets/

Outbound (pull from NGC)

Outbound transfers are the same path in reverse: you stage files in /data/upload under admin supervision, then fetch them from outside via the SFTP endpoint. Coordinate with your NGC admin before placing outbound files, since releasing data from the air-gapped environment requires their approval.

scp ngc:/data/upload/results.tar ~/results.tar

Between clusters

Direct inter-cluster transfers aren't supported because of the air-gap. Move data via your local research workstation: pull from NGC's staging area, then push to the destination cluster (e.g. transfer.gbar.dtu.dk on the DTU side). The external SFTP endpoint sftp.spc.ngc.dk is reachable over forskningsnettet, so using a workstation that is also on the research network gives you high-bandwidth ingress and egress.

Small files (code, configs)

For source code and similar small files you have a faster path inside the cluster: mount a host directory through the Omnissa Remote Desktop client, which an NGC admin has to enable for you. Public repositories available through the internal GitHub proxy can also be cloned directly, as described under Code from git.

Software environment

The air gap shapes everything here: pip install, conda, and apt cannot reach the public internet, so an environment that builds itself on first run will fail. Plan on one of three routes.

Environment modules cover what is already installed:

module avail
module load <module>
module list
module purge

Otherwise, bring the dependencies in the same way you bring data. Resolve them on a machine with network access, then stage the result through /data/upload:

pip download -r requirements.txt -d wheels/     # on your own machine
tar czf wheels.tar.gz wheels/
scp wheels.tar.gz ngc:/data/upload/

pip install --no-index --find-links=wheels/ -r requirements.txt   # on the cluster

For anything with awkward system libraries, build a container image outside and stage the image file instead. That also makes the environment reproducible, which matters more here than on the open clusters, since you cannot simply reinstall later.

How to start an interactive job

Quick interactive session:

iqsub

Interactive session for a specific group with explicit resources. Substitute your own project group and account, which you receive when your access is approved:

qsub -I -X -W group_list=<group> -A <account> -l nodes=1:ppn=4,mem=20gb,walltime=02:00:00

How to submit a batch job

#!/bin/bash
#PBS -N train
#PBS -W group_list=<group>
#PBS -A <account>
#PBS -l nodes=1:ppn=4,mem=20gb,walltime=24:00:00
#PBS -o logs/
#PBS -e logs/

cd $PBS_O_WORKDIR
python train.py

PBS reads the #PBS lines before any shell runs, so a variable such as $PBS_JOBID is not expanded there. Naming a directory instead lets PBS write <jobname>.o<jobid> and <jobname>.e<jobid> into it.

Create the log directory before submitting, since the job fails at start if it does not exist:

mkdir -p logs
qsub train.sh

Refer to NGC's internal documentation (linked from the status page) for the full list of scheduler flags and worked examples.

Monitoring and cancelling jobs

qstat -u $USER          # your jobs and their state
qstat -f <jobid>        # full detail; the comment field says why a job is still queued
qstat -Q                # queue overview
qdel <jobid>            # cancel

A job sitting in Q is normal. The comment in qstat -f states what it is waiting for, usually that the requested nodes, memory, or wall time are not free yet.

Policy & Support

Usage Rules

• GDPR-compliant infrastructure. Only use this cluster for projects that need it
• Each project must carry a signed Data Processing Agreement naming NGC as data processor
• High-risk projects additionally require a Data Protection Impact Assessment (DPIA)
• Per-project storage quotas are set during onboarding, from the roughly 2 PB available

Storage, quotas, and backups

Your quota is set when your project is onboarded. Check what you are using before staging a large dataset:

du -sh ~                    # your home directory
df -h .                     # space left on the filesystem you are standing in

The staging area at /data/upload is a gateway, not storage. Clear your files from it once they have landed, so the next transfer is not competing with them.

Retention is governed by your project's Data Processing Agreement rather than by convenience: personal data may not simply be left in place once the project ends. Agree the deletion or archiving steps with your NGC admin and the compute coordinator well before that date.

Temporary GPU Increase

Per-user GPU limits keep the cluster fair when demand is high. When resources are underutilized, you can request a temporary increase of your GPU allowance. Increases are granted on a best-effort basis and roll back when demand picks up again.

Who to contact

Technical Support

Broken or missing packages, scheduler issues

NGC HPC Support Team

Policy Support

Priority access, queueing, access exceptions

compute-governance-p1@aicentre.dk

General Questions

Open chat with the P1 compute community

#compute #ask-compute

Compute Coordinator

Onboarding, workload guidance, general questions. Always happy to help

Helped with your work? Acknowledge ORCID 0000-0002-2740-1651

workload survey bstja@dtu.dk

Hardware Specification

Nodes

8 compute (Lenovo VX665) + 3 storage (VX650)

CPU per compute node

2× AMD EPYC 9354 (32 cores each)

RAM per compute node

768 GB

GPUs

14× NVIDIA H100 PCIe (80 GB)

Storage

About 2 PB total (245 TB SSD, 368 TB HDD, 1.44 PB cold)

Topology

Air-gapped, per-project isolation through VMware vSphere

Compliance

GDPR-compliant infrastructure

Operating system

Rocky Linux

Scheduler

TORQUE (PBS)