Hands-On with Gadi

Overview

This workshop provides an introduction to basic command-line operations and guidelines for working efficiently on Gadi, the high-performance computing (HPC) machine at the National Computational Infrastructure (NCI).

Learning Outcomes

By the end of this course, participants will be able to:

1. Understand Gadi’s Architecture: Identify the key components of the Gadi supercomputer and combine the appropriate file systems (home, scratch, /g/data, JOBFS) for different stages of their project.

2. Operate in an HPC Environment: Confidently use the Australian Research Environment (ARE) to access Gadi via web-based interfaces, including Jupyter Notebooks and Virtual Desktops.

3. Use the Command Line Interface: Perform essential remote file management and text editing using Linux commands tailored for a high-performance computing context.

4. Manage Computational Jobs: Write and submit batch scripts to the job scheduler, and differentiate between interactive and batch modes to optimize resource usage.

5. Utilise Specialised Services: Gain familiarity with the range of services available on Gadi, including CPU and GPU systems, interactive ARE sessions, and Jupyter Notebook environments for scientific computing and data analysis.

6. Apply HPC Best Practices: Understand how job queues, scheduling policies, and resource limits influence workload execution and resource utilisation in an HPC environment.

Prerequisites

1. Access to a terminal environment.

2. Access to a web browser.

3. Familiarity with the Linux command-line interface is recommended; however, basic concepts will be covered during the workshop.

Suggested Workshop Agenda

Topics	Duration
Introduction to HPC, NCI and Gadi	20 minutes
Logging in to Gadi	10 minutes
Navigating Gadi File System	30 minutes
Modules	10 minutes
Using Python on Gadi	10 minutes
Requesting Resources	30 minutes
Run Jupyter Notebooks on Gadi	20 minutes
Virtual Desktop in ARE	10 minutes
Total	140 minutes

Contents

Before the Workshop

• Access and Setup

Tutorials

• 1. Introduction to HPC, NCI and Gadi
  • 1.1. Why HPC?
  • 1.2. What is HPC?
  • 1.3. Differences from Desktop Computing
  • 1.4. Gadi
• 2. Logging in to Gadi
  • 2.1. By Australian Research Environment (ARE)
  • 2.2. Practice: Login to ARE and open a terminal
  • 2.3. By Terminal
• 3. Navigating Gadi File Systems
  • 3.1. Gadi File Systems
  • 3.2. Understanding your location (pwd)
  • 3.3. Listing files (ls)
  • 3.4. Practice: Find your bearings
  • 3.5. Changing directory (cd)
  • 3.6. Practice: Explore shared locations
  • 3.7. Managing files and directories
  • 3.8. Viewing and editing text
  • 3.9. Practice: Create, inspect, tidy up
  • 3.10. Copying files to and from Gadi (scp)
  • 3.11. Practice: Prepare the code for your first job on Gadi
• 4. Modules
  • 4.1. Module Commands
  • 4.2. Practice: Load the openmpi module
• 5. Using Python on Gadi
  • 5.1. Create a Python Virtual Environment on Gadi
• 6. Gadi Compute Job Basics
  • 6.1. Planning Jobs
  • 6.2. Running Batch Jobs
  • 6.3. Practice: Write a PBS job script
  • 6.4. Submit and Monitor Jobs
  • 6.5. Practice: Submit and Monitor Your First Job
  • 6.6. Interactive Jobs
• 7. Run Jupyter Notebooks on Gadi
  • 7.1. Cluster Access
  • 7.2. Launch JupyterLab Session on ARE
  • 7.3. ARE FAQs
  • 7.4. JupyterLab Usage
  • 7.5. JupyterLab FAQs
• 8. Virtual Desktop in ARE
  • 8.1. Requesting GPUs

References

• Reference