# P1 NGC HPC

> **Provider documentation.** This page is an overview to help you onboard and get started. The technical and advanced documentation is maintained by NGC, who host and operate the system.
>
> NGC has not published technical documentation yet.
>
> Until it is published, the internal documentation linked from the status page inside the environment is the reference, and the compute coordinator can help with anything it does not cover.

The P1 NGC HPC is hosted at the National Genome Centre, for GDPR-compliant work on sensitive data
and large datasets. VMware vSphere isolates projects from one another, and storage is
petabyte-scale. [NGC's supercomputer page](https://www.ngc.dk/infrastrukturen-for-personlig-medicin/supercomputer)
describes the infrastructure.

## Getting Access

### Who's eligible?

**Requirements:** PhD or higher (exceptions may apply), valid Danish university email, and
registered P1 affiliation.

Each project must bring a signed Data Processing Agreement as its record. It must name NGC as a
data processor and allow the data to be stored there. A project that poses a high risk to the people
whose personal data it processes also needs a Data Protection Impact Assessment (DPIA).

### Do I need to talk to someone first?

No. The optional [workload survey](/survey/) tells the P1 compute coordinator about your
workload. Fill it in early for NGC, since GDPR projects need their data-processing paperwork lined
up.

### How do I sign up?

1. Request P1 affiliation in [your P1 profile](https://www.aicentre.dk/p1adb) (**Update Profile** in the header). P1
   staff approve it.
2. Get the user application form from NGC. Only NGC issues it; their
   [how to apply for access](https://www.eng.ngc.dk/research-and-international-collaboration/dngc-research-services/how-to-apply-for-access) page explains how.
3. Complete and sign it, then email it to the [compute coordinator](mailto:bstja@dtu.dk).
4. Fill in the access request below. The coordinator then knows the paperwork is coming and
   raises the request with NGC.

You join the NGC Slack channel once you have access.

**Form:** [P1 NGC HPC Access Request](https://forms.cloud.microsoft/e/gtfrV7ap4C) Requires an approved P1 affiliation and a Data Processing Agreement naming NGC.

### What if I don't qualify?

**Form:** [Access exception request](https://forms.gle/jSp12oBU2BcM6GJR7) For MSc students, research assistants, and similar roles who need cluster access with written approval from a responsible P1 co-lead or faculty member.

## Connecting

The P1 NGC HPC is air-gapped. Access needs multi-factor authentication and the Omnissa Remote
Desktop client, and you get instructions once registered. From the remote desktop,
`ssh -X <your-username>@login` reaches the login node.

Useful hosts:

```text
https://console.cld076.vmc/status   # Internal status page
cld076-0004.cld076.vmc              # Internal SFTP
sftp.spc.ngc.dk                     # External Ingress/Egress (SFTP)
```

## Using the Cluster

The cluster runs PBS, and `iqsub` opens a quick interactive session. Jobs name your project's
group and account, which you get when access is approved. Scheduler flags are in NGC's internal
documentation, linked from the status page.

The cluster has no internet access, so `git clone` from a public forge, `pip install` and `conda`
do not work. Code, data, packages and results pass through the SFTP staging area at `/data/upload`
under admin supervision, and anything leaving needs your NGC admin's approval. Ask your NGC admin
for access to `/data/upload` as your project's data gateway.

The gallery shows how to move [code, data and dependencies](/gallery/xfr-dat/) through the staging
area. See [every example that runs on P1 NGC HPC](/gallery/?provider=ngc).

## Policy & Support

### What are the limits?

**Compute**

| Limit | Value |
| --- | --- |
| Max wall time | Not published; ask your NGC admin or the compute coordinator |
| GPUs per job | Not published; ask your NGC admin or the compute coordinator |
| GPUs at once, per person | Not published; ask your NGC admin or the compute coordinator |

**Storage**

| Limit | Value |
| --- | --- |
| Home directory | Set per project at onboarding |
| Project storage | Set per project at onboarding |
| Shared pool | About 2 PB (245 TB SSD, 368 TB HDD, 1.44 PB cold) |

- Only use this GDPR-compliant cluster for projects that need it
- Each project needs a signed Data Processing Agreement naming NGC as data processor
- High-risk projects also need a DPIA

### How much storage do I get?

Your quota is set when your project is onboarded. The staging area at `/data/upload` is a gateway
only. Clear your files from it once they have landed, so the next transfer does not compete with
them.

Your Data Processing Agreement governs retention, and personal data may not stay once the project
ends. Agree deletion or archiving with your NGC admin and the compute coordinator well before
then.

### What if I need more GPUs?

**Form:** [Planned GPU allocation](https://forms.cloud.microsoft/e/yN8Vxi7KTc) Request a raised GPU limit for a run that also needs protecting from cancellation: a multi-day training, a deadline, or a workload that cannot checkpoint.

### Who do I contact?

- NGC HPC Support Team: support@ngc.dk
- P1 compute coordinator: bstja@dtu.dk
- Book a consultation with the coordinator: https://book.starostka.io/benjamin/compute-needs
- Helped with your work? Acknowledge the coordinator's ORCID: https://orcid.org/0000-0002-2740-1651

## Hardware Specification

| | |
| --- | --- |
| Nodes | 8 compute (Lenovo VX665) + 3 storage (VX650) |
| CPU per compute node | 2× AMD EPYC 9354 (32 cores each) |
| RAM per compute node | 768 GB |
| GPUs | 14× NVIDIA H100 PCIe (80 GB) |
| Storage | About 2 PB total (245 TB SSD, 368 TB HDD, 1.44 PB cold) |
| Topology | Air-gapped, per-project isolation through VMware vSphere |
| Compliance | GDPR-compliant infrastructure |
| Operating system | Rocky Linux |
| Scheduler | TORQUE (PBS) |