# P1 Gefion HPC

> **Provider documentation.** This page is an overview to help you onboard and get started. The technical and advanced documentation is maintained by DCAI, who host and operate the system.
>
> DCAI documentation: https://doc-open-ai.secureremote.dcai.dk
>
> Requires signing in to DCAI before the link resolves.

The P1 Gefion HPC is a sandbox with on-demand H100 GPUs on [Gefion](https://dcai.dk/gefion). Use
it to develop, test and pilot a workload on the hardware generation it will scale up on. Full 8-GPU
production nodes need separate funding, such as a Novo Nordisk Foundation (NNF) compute voucher or
another allocation. A workload that already runs here is what you apply with.

## Getting Access

### Who's eligible?

**Requirements:** PhD or higher (exceptions may apply), valid Danish university email, and
registered P1 affiliation.

### Do I need to talk to someone first?

No. The optional [workload survey](/survey/) tells the P1 compute coordinator about your
workload, and helps us plan production nodes if your project scales.

### How do I sign up?

1. Request P1 affiliation in [your P1 profile](https://www.aicentre.dk/p1adb) (**Update Profile** in the header). P1
   staff approve it.
2. Fill in the sandbox sign-up form below. A DCAI team member then contacts you with a DCAI
   username and initial-setup-password.

**Form:** [P1 Gefion Sandbox Sign-up Form](https://forms.cloud.microsoft/e/f0gbc83GQU) Requires an approved P1 affiliation.

### What if I don't qualify?

**Form:** [Access exception request](https://forms.gle/jSp12oBU2BcM6GJR7) For MSc students, research assistants, and similar roles who need cluster access with written approval from a responsible P1 co-lead or faculty member.

## Connecting

```bash
aicentre check -p gefion --connect
```

`aicentre check` shows what your machine needs for Gefion: the Host block for `~/.ssh/config`
and when to sign in to DCAI again. Then it tests the login. DCAI's documentation has the
details.

## Using the Cluster

The sandbox runs Slurm. Submit GPU work as batch jobs instead of holding an interactive session
open.

- Every job names the Slurm account it is charged to. P1's default account is `iu_0096`
  (`--account=iu_0096`). A project with its own account uses that one instead.
- Nodes have 2 TB of memory, so size `--mem` for the data a job loads, not its GPUs.
- The P1 partition is on Gefion's open system. The login node reaches public git hosts and package
  indexes.
- Large transfers use the SFTP endpoint `xfer.dcai.dk`. To get your public IP whitelisted, send
  [DCAI support](mailto:support@dcai.dk) the output of `curl ifconfig.io`.

From the gallery:

- [VS Code on the login node](/gallery/rem-lgn/)
- [moving code and data](/gallery/xfr-dat/) on and off the cluster

See [every example that runs on P1 Gefion HPC](/gallery/?provider=gefion).

## Policy & Support

### What are the limits?

**Compute**

| Limit | Value |
| --- | --- |
| Max wall time | Not published; ask the compute coordinator before planning a long run |
| GPUs per job | 2 |
| GPUs at once, per person | 4, as 2 concurrent jobs of 2 GPUs, more while GPUs sit idle |

**Storage**

| Limit | Value |
| --- | --- |
| Home directory | 25 GB |
| Project storage | 500 GB default |
| Shared pool | 300 TB |

- 16 H100 GPUs in total, shared by all sandbox users
- User accounts are individual and non-transferable
- An account can belong to one or more Gefion projects

Limits may change to meet the needs of P1 students and faculty.

> **GDPR Compliance**
>
> The P1 Gefion HPC is GDPR-compliant, so projects with personal data can run here. Have the
> data-processing paperwork in place, and agree the details with DCAI and the compute coordinator,
> before you upload any.

### How much storage do I get?

Home directories hold 25 GB, so keep datasets and checkpoints in your project space (500 GB
default). Data sent to the SFTP endpoint lands in a staging area first and counts twice until you
delete the staged copy.

Use cluster storage as working space, not an archive. Keep code in git and a second copy of
anything you cannot regenerate on institutional storage or your own machine. Before a project
ends, ask the compute coordinator what to move off.

### What if I need more GPUs?

The default limit is 2 concurrent jobs of at most 2 GPUs each. When GPUs are idle on P1's two
NVIDIA DGX H100 nodes (account `iu_0096`), you can go past it. Check what is free with `sinfo` and
submit the larger job. No request is needed.

> **caution**
>
> Jobs beyond the default limits can be cancelled when demand picks up. You are responsible for
> checkpointing, so a cancelled job resumes instead of starting over.

**Form:** [Planned GPU allocation](https://forms.cloud.microsoft/e/yN8Vxi7KTc) Request a raised GPU limit for a run that also needs protecting from cancellation: a multi-day training, a deadline, or a workload that cannot checkpoint.

### Who do I contact?

- DCAI Support Team: support@dcai.dk
- P1 compute coordinator: bstja@dtu.dk
- Book a consultation with the coordinator: https://book.starostka.io/benjamin/compute-needs
- Helped with your work? Acknowledge the coordinator's ORCID: https://orcid.org/0000-0002-2740-1651

## Hardware Specification

| | |
| --- | --- |
| Nodes | 2× NVIDIA DGX H100 (8 GPUs each) |
| GPUs | 16× H100 SXM, 640 GB GPU memory, around 32 petaFLOPS FP8 |
| CPU per node | 2× Intel Xeon Platinum 8480C (56 cores each) |
| RAM per node | 2 TB |
| Interconnect | NVLink at 900 GB/s GPU to GPU, InfiniBand up to 400 Gbps |
| Shared storage | 300 TB, 500 GB default quota |
| Home directory | 25 GB |
| Operating system | NVIDIA DGX OS (Ubuntu based) |
| Scheduler | Slurm |