Oxford RSE

Debug Python on HPC Compute Nodes

October 29, 2025 by Alex Allmont4 minutes

Overview

When working on a high-performance computing (HPC) cluster, debugging code using an IDE such as VSCode can be complicated by network restrictions on compute nodes. This article demonstrates a workaround of starting an interactive session and launching code via debugpy in a ‘wait for client’ state on an open port. VSCode can then be connected to the debugger through the port rather than SSH.

HPC Login and Compute Nodes

An HPC cluster consists of nodes with different responsibilities. For example, the Biomedical Research Computing Facility (BMRC) is divided into login nodes and compute nodes, where a small number of login nodes are online for users to SSH into to configure and launch tasks, and then there are many more compute nodes which are not online on where the actual code is run. A Slurm job scheduler is used to launch compute node tasks, specifying hardware requirements such as number of CPUs, GPUs, and memory requirements.

Debugger Workaround

Despite the BMRC documentation, some internet access is available on their compute nodes as one can pip install on them, but connectivity is limited in such a way that a debugger cannot be connected out of the box as if running in VSCode on your own computer. In the case of BMRC, it is complicated by two-factor authentication and not being able to SSH into a compute node after it has started. Other HPC clusters will have their own configuration nuances.

Rather than using sbatch to start the Slurm job, srun is used below for an interactive session allowing you to modify the port settings whilst connected. If you have more complex sbatch launch requirements, these can be passed as srun command-line arguments.

Required VSCode extensions

  • Remote Explorer Extension ms-vscode.remote-explorer
  • Python Debugger ms-python.debugpy

Setup

  1. In VSCode, use Remote Explorer to connect to the login node using normal credentials.

  2. Open the remote folder where your code is located.

  3. Create a .vscode/launch.json if it does not already exist and add the following (python) remote debug configuration for attaching to a running Python debugger over a port:

    {
    "configurations": [
        {
            "name": "(python) remote debug",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "HOSTNAME_TODO",
                "port": 5678
            }
        }
    ]
}

  4. For the shell operations below, you can use a regular remote terminal or a VSCode integrated terminal (shortcut Ctrl+`).

  5. Ensure you have Python installed and a venv set up, e.g.:

    • module load Python (or whichever Python version you need)
    • python -m venv venv

  6. Source the venv and ensure that debugpy is pip-installed.

    • source .venv/bin/activate
    • pip install debugpy

  7. Start an interactive session on a compute node, e.g.:

    • srun -p short --pty bash
    • When this command completes you will be in an interactive session shell. The CLI commands in the steps hereafter are run in this interactive shell. Note that the VSCode GUI is still running through the original login node.

  8. Inside the interactive session, get the hostname of the compute node.

    • hostname -s
    • For this example, say this returns compa000

  9. In .vscode/launch.json above, change the HOSTNAME_TODO text with the host name of the compute node, e.g.

                "connect": {
                "host": "compa000",
                "port": 5678
            },

  10. In VSCode, set breakpoints on the code you want to interrogate.

  11. In the terminal, run Python with the debugpy module starting the session in a ‘wait for client’ state so it does not do anything until we attach:

    • python -m debugpy --listen 0.0.0.0:5678 --wait-for-client myfile.py
    • Replace myfile.py with your own script and Python flags.

  12. Open the debug section in the VSCode side bar, choose the (python) remote debug configuration created earlier, and click on the run arrow.

    • You will see the terminal pick up the debug connection and it will run to the first breakpoint you have set. You can now use the VSCode debug UI and debug terminal to interrogate.

  13. The debugpy command stops in the terminal when the code completes. If you need to stop the run early, hit Ctrl-C to stop the session and return to the terminal. If you hit problems restarting a debug session, open the VSCode PORTS panel and close any forwarded ports.

Repeated runs

To debug again, re-run the python -m debugpy... command above and click the run arrow in VSCode or press F5.

When you start a new interactive session, remember to repeat steps 8 and 9 above, i.e. hostname -s then update "host" in launch.json.

Resources

python debugging vscode