# P1 NGC HPC > **Provider documentation.** This page is an overview to help you onboard and get started. The technical and advanced documentation is maintained by NGC, who host and operate the system. > > NGC has not published technical documentation yet. > > Until it is published, the internal documentation linked from the status page inside the environment is the reference, and the compute coordinator can help with anything it does not cover. The P1 NGC HPC is hosted at the National Genome Centre and is designed for secure data processing with GDPR compliance. Projects are isolated from one another through VMware vSphere virtualization, and the facility carries petabyte-scale storage, so it suits sensitive-data work with large datasets. For details on the underlying infrastructure, see [NGC's supercomputer page](https://www.ngc.dk/infrastrukturen-for-personlig-medicin/supercomputer). ## Getting Access ### Talk to the compute coordinator You are welcome to go straight to the sign-up steps below. If you would like a hand, the [workload survey](/survey/) tells the P1 compute coordinator about your needs, expectations, and workload. For NGC it is worth doing early, since GDPR projects need their data-processing paperwork lined up. ### Sign-up Form Before accessing the P1 NGC HPC, you must be a P1 affiliate. Sign in to [your P1 profile](https://www.aicentre.dk/p1adb) (also reachable via **Update Profile** in the header), update your details, and submit a request to become an affiliate member. A P1 staff member will review and approve. Once your affiliation is approved, three steps get you an account. 1. Obtain the user application form from NGC. They issue it themselves, so it is not something we can hand out. Their [how to apply for access](https://www.eng.ngc.dk/research-and-international-collaboration/dngc-research-services/how-to-apply-for-access) page explains how to get it. 2. Complete and sign it, then email it to the [compute coordinator](mailto:bstja@dtu.dk). 3. Fill in the access request below, so the coordinator knows the paperwork is on its way and can raise the request with NGC. You will be added to the NGC Slack channel once you gain access. **Form:** [P1 NGC HPC Access Request](https://forms.cloud.microsoft/e/gtfrV7ap4C) Requires an approved P1 affiliation and a Data Processing Agreement naming NGC. ### Access Exception **Form:** [Access exception request](https://docs.google.com/forms/d/e/1FAIpQLSciaOm-CYwl48LGGZC7qlmCSPU7kfEitjGy4kvZXAXEPbo_eA/viewform) For MSc students, research assistants, and similar roles who need cluster access with written approval from a responsible P1 co-lead or faculty member. ### Who's eligible? **Requirements:** PhD or higher (exceptions may apply), valid Danish university email, and registered P1 affiliation. Each project needs to bring a record (a signed Data Processing Agreement) that explicitly mentions NGC as a data processor and that the data is allowed to be stored there. If the project poses a high risk to individuals whose personal data is being processed, a Data Protection Impact Assessment (DPIA) is also required. ## Getting Started ### How to connect The P1 NGC HPC is an air-gapped system requiring multi-factor authentication and a remote desktop client. Once registered, you'll receive specific access instructions including an Omnissa Remote Desktop client. Once connected with the remote desktop client, access the login node using `ssh -X @login`. Useful hosts: ```text https://console.cld076.vmc/status # Internal status page cld076-0004.cld076.vmc # Internal SFTP sftp.spc.ngc.dk # External Ingress/Egress (SFTP) ``` ### How to transfer data The P1 NGC HPC is air-gapped, so `rsync` and `rclone` from outside aren't available end-to-end: every transfer goes through approved SFTP staging under admin supervision. Request access to `/data/upload` from your NGC admin as your project's data gateway, then add an SSH entry to your `~/.ssh/config`: ```text file=~/.ssh/config Host ngc HostName sftp.spc.ngc.dk Port 6433 User _sftp HostKeyAlgorithms +ssh-rsa ``` #### Code from git The same air gap applies to git: the cluster cannot reach GitHub, GitLab, or any other public forge directly. An internal GitHub proxy covers some public repositories, so try a plain clone first and fall back to a bundle when the proxy does not have what you need. A bundle is a single file holding the full history, which means the cluster copy stays a real git repository rather than a flattened snapshot. Private repositories always take this path. On your own machine: ```bash git bundle create repo.bundle --all scp repo.bundle ngc:/data/upload/ ``` Then on the cluster, clone from the staged bundle: ```bash scp _sftp@cld076-0004.cld076.vmc:/data/upload/repo.bundle ~/ git clone repo.bundle your-repo ``` To ship later changes, create a bundle covering only the new commits and pull it in: ```bash git bundle create update.bundle ..HEAD # on your machine git pull ~/update.bundle HEAD # on the cluster ``` If a bundle is more machinery than you need, the [Small files](#small-files-code-configs) recipe below copies a plain working tree without any history. #### Inbound (push to NGC) From outside the cluster, upload to the staging area: ```bash scp ~/datasets/ISLES-2022.zip ngc:/data/upload/ ``` Then inside the cluster, copy from staging into your home directory: ```bash scp _sftp@cld076-0004.cld076.vmc:/data/upload/ISLES-2022.zip ~/datasets/ ``` #### Outbound (pull from NGC) Outbound transfers are the same path in reverse: you stage files in `/data/upload` under admin supervision, then fetch them from outside via the SFTP endpoint. Coordinate with your NGC admin before placing outbound files, since releasing data from the air-gapped environment requires their approval. ```bash scp ngc:/data/upload/results.tar ~/results.tar ``` #### Between clusters Direct inter-cluster transfers aren't supported because of the air-gap. Move data via your local research workstation: pull from NGC's staging area, then push to the destination cluster (e.g. `transfer.gbar.dtu.dk` on the DTU side). The external SFTP endpoint `sftp.spc.ngc.dk` is reachable over [forskningsnettet](/ecosystem/#the-danish-research-network), so using a workstation that is also on the research network gives you high-bandwidth ingress and egress. #### Small files (code, configs) For source code and similar small files you have a faster path inside the cluster: mount a host directory through the Omnissa Remote Desktop client, which an NGC admin has to enable for you. Public repositories available through the internal GitHub proxy can also be cloned directly, as described under [Code from git](#code-from-git). ### Software environment The air gap shapes everything here: `pip install`, `conda`, and `apt` cannot reach the public internet, so an environment that builds itself on first run will fail. Plan on one of three routes. Environment modules cover what is already installed: ```bash module avail module load module list module purge ``` Otherwise, bring the dependencies in the same way you bring data. Resolve them on a machine with network access, then stage the result through `/data/upload`: ```bash pip download -r requirements.txt -d wheels/ # on your own machine tar czf wheels.tar.gz wheels/ scp wheels.tar.gz ngc:/data/upload/ ``` ```bash pip install --no-index --find-links=wheels/ -r requirements.txt # on the cluster ``` For anything with awkward system libraries, build a container image outside and stage the image file instead. That also makes the environment reproducible, which matters more here than on the open clusters, since you cannot simply reinstall later. ### How to start an interactive job Quick interactive session: ```bash iqsub ``` Interactive session for a specific group with explicit resources. Substitute your own project group and account, which you receive when your access is approved: ```bash qsub -I -X -W group_list= -A -l nodes=1:ppn=4,mem=20gb,walltime=02:00:00 ``` ### How to submit a batch job ```bash file=train.sh #!/bin/bash #PBS -N train #PBS -W group_list= #PBS -A #PBS -l nodes=1:ppn=4,mem=20gb,walltime=24:00:00 #PBS -o logs/ #PBS -e logs/ cd $PBS_O_WORKDIR python train.py ``` PBS reads the `#PBS` lines before any shell runs, so a variable such as `$PBS_JOBID` is not expanded there. Naming a directory instead lets PBS write `.o` and `.e` into it. Create the log directory before submitting, since the job fails at start if it does not exist: ```bash mkdir -p logs qsub train.sh ``` Refer to NGC's internal documentation (linked from the status page) for the full list of scheduler flags and worked examples. ### Monitoring and cancelling jobs ```bash qstat -u $USER # your jobs and their state qstat -f # full detail; the comment field says why a job is still queued qstat -Q # queue overview qdel # cancel ``` A job sitting in `Q` is normal. The comment in `qstat -f` states what it is waiting for, usually that the requested nodes, memory, or wall time are not free yet. ## Policy & Support ### Usage Rules - GDPR-compliant infrastructure. Only use this cluster for projects that need it - Each project must carry a signed Data Processing Agreement naming NGC as data processor - High-risk projects additionally require a Data Protection Impact Assessment (DPIA) - Per-project storage quotas are set during onboarding, from the roughly 2 PB available ### Storage, quotas, and backups Your quota is set when your project is onboarded. Check what you are using before staging a large dataset: ```bash du -sh ~ # your home directory df -h . # space left on the filesystem you are standing in ``` The staging area at `/data/upload` is a gateway, not storage. Clear your files from it once they have landed, so the next transfer is not competing with them. Retention is governed by your project's Data Processing Agreement rather than by convenience: personal data may not simply be left in place once the project ends. Agree the deletion or archiving steps with your NGC admin and the compute coordinator well before that date. ### Temporary GPU Increase **Form:** [Temporary GPU allowance increase](https://forms.cloud.microsoft/e/yN8Vxi7KTc) Request a temporary raise of your GPU limit while a cluster is underutilized. ### Who to contact - NGC HPC Support Team: support@ngc.dk - P1 compute coordinator: bstja@dtu.dk - Helped with your work? Acknowledge the coordinator's ORCID: https://orcid.org/0000-0002-2740-1651 ## Hardware Specification | | | | --- | --- | | Nodes | 8 compute (Lenovo VX665) + 3 storage (VX650) | | CPU per compute node | 2× AMD EPYC 9354 (32 cores each) | | RAM per compute node | 768 GB | | GPUs | 14× NVIDIA H100 PCIe (80 GB) | | Storage | About 2 PB total (245 TB SSD, 368 TB HDD, 1.44 PB cold) | | Topology | Air-gapped, per-project isolation through VMware vSphere | | Compliance | GDPR-compliant infrastructure | | Operating system | Rocky Linux | | Scheduler | TORQUE (PBS) |