High-Performance Computing Engineer Job Description

A High-Performance Computing Engineer designs, builds and runs the shared clusters that researchers and model developers submit batch jobs to: racks of Linux compute nodes tied together by a low-latency fabric, fed by a parallel file system and governed by a job scheduler. Users see a login node, a queue and a scratch directory. The engineer owns everything behind them, and treats the whole machine as a service that many research groups depend on at once.

The job breaks into layers, and a senior engineer works across all of them.

Scheduling and allocation. Slurm, PBS Pro or OpenPBS, and IBM Spectrum LSF decide which job runs where and when. The engineer writes the partition layout, fair-share weights, quality-of-service tiers, GPU and memory limits, preemption rules and accounting that determine how a grant-funded lab or a model training team gets its share of the machine. When a queue backs up the week before a conference deadline, the first message lands with this person.

Storage. Parallel file systems such as Lustre, IBM Storage Scale (formerly GPFS), BeeGFS and DAOS hold scratch and project data. The engineer sizes metadata and object storage servers, sets striping defaults, enforces quotas and purge policies, and hunts down the job that is flooding the metadata servers with small-file operations.

Interconnect. InfiniBand, HPE Slingshot and RoCE Ethernet fabrics carry MPI and collective-communication traffic between nodes. Engineers manage subnet managers, routing and topology, firmware, and link-error counters, then run OSU micro-benchmarks or NCCL tests to prove the fabric is healthy after maintenance.

Software environment. Researchers expect compilers, MPI libraries such as Open MPI and MPICH, math libraries and domain codes on demand. Engineers build them with Spack or EasyBuild, expose them through Lmod modules, and support Apptainer containers so a workflow that runs on a workstation also runs across many nodes. Web portals such as Open OnDemand and data-transfer services such as Globus sit in the same stack.

Operations. Node images go out through Warewulf, xCAT or a vendor cluster manager, with configuration held in Ansible. Monitoring watches node health, power and cooling. Engineers roll kernel and firmware updates, swap failed memory and GPUs with vendor field engineers, and keep security patches current without killing week-long jobs.

Procurement and acceptance. When a center buys a system, HPC engineers help write the technical requirements, then run HPL, HPCG, IOR and application benchmarks during acceptance testing to decide whether the delivered machine meets the contract.

Penn State's job profile for its Research Computing Systems Engineer captures the blend: design and deploy HPC systems, configure clusters and cloud resources, monitor performance, implement security protocols, support researchers directly and work with vendors on hardware and software issues.

The role is not GPU kernel programming. A kernel specialist tunes the inside of one application; an HPC engineer keeps a multi-user system fast, fair and available for every application on it. It also differs from running a container fleet for a single product: the HPC engineer's world is batch scheduling, MPI, parallel storage and allocation policy shared by many projects with competing deadlines.

Day to day, the engineer works with research software engineers, research computing facilitators who onboard new users, faculty and lab staff, facilities teams that manage power and liquid cooling, the institution's security office, and hardware and storage vendors.

As a rough guide, HPC engineers arrive from one of two directions: Linux systems administration, or computational research that ran on a cluster. The usual starting credential is a bachelor's degree in computer science, computer engineering, physics, or another computational field, though demonstrated cluster experience can stand in for it. A graduate degree can help for roles that mix systems work with application performance consulting.

A 2018 draft career guide for research computing and data professionals from the Campus Research Computing Consortium set the baseline for systems-facing roles as "a bachelor's degree or relevant experience in domain science and related areas," and said technicians with broad systems engineering experience, sometimes from the military, are also acceptable. It listed basic Linux system administration, command-line work and scripting as the starting experience, and named scheduler configuration, monitoring and alerting, account management and security posture among the job elements.

Employer job ladders show how that experience is staged. Penn State's Research Computing Systems Engineer profile asks for a bachelor's degree with 3+ years of relevant experience at its Advanced Professional level and 8+ years at its Principal Professional level, and accepts an equivalent combination of education and experience.

Skills that carry the job:

• Linux internals: boot, kernel modules, systemd, cgroups, networking and file systems, plus the debugging habits to read a kernel log under pressure.
• Scheduler administration in Slurm or PBS, including accounting databases, fair-share math and GPU resource definitions.
• Parallel storage: Lustre or IBM Storage Scale at minimum, with an understanding of striping, metadata load and client tuning.
• High-speed networking: InfiniBand fabric management, RDMA, and the basics of RoCE on lossless Ethernet.
• Automation: Bash, Python, Ansible and Git, so changes are repeatable across many nodes.
• Software builds: compilers, MPI implementations, Spack or EasyBuild recipes, and container runtimes such as Apptainer.
• Communication: translating a frustrated researcher's description of a slow job into a fix, and writing documentation users actually read.

Credentials. No license is required to work on a cluster. Linux certifications can document the base layer: the Red Hat Certified System Administrator (RHCSA) from Red Hat and the Linux Foundation Certified System Administrator (LFCS) from The Linux Foundation both test the administration skills the job assumes. Scheduler, storage and fabric vendors run their own training courses, which become useful once a candidate is responsible for a specific product.

Getting in. As a rough guide, people move into a cluster engineering role after 3 to 5 years of Linux systems work or after graduate research on HPC systems. Student operator jobs at a campus research computing center, internships at national laboratories, and student cluster competitions at supercomputing conferences are on-ramps for those starting earlier.

One signal for this specialty is public money going into new machines. In October 2025, the Department of Energy announced two systems at Oak Ridge National Laboratory. ORNL said the Lux AI cluster would be deployed in 2026 and that Discovery would be delivered in 2028. Discovery is based on the HPE Cray Supercomputing GX5000 and is augmented by an HPE Cray Supercomputing Storage Systems K3000, a storage system built on DAOS, the distributed asynchronous object storage software. A day later, Argonne National Laboratory said its Solstice system would feature 100,000 NVIDIA Blackwell GPUs and would be the largest AI supercomputer in the DOE lab complex, with a second system, Equinox, featuring 10,000 NVIDIA Blackwell GPUs.

Deployments like these are the work this role exists to do: site preparation with facilities staff, integration of storage and fabric, scheduler and allocation design, acceptance benchmarking, and years of operations afterward. They also show where the skill set is heading. DOE described these machines as AI supercomputers, so engineers who can run GPU-dense partitions, object storage such as DAOS, and AI training jobs alongside traditional simulation codes match what the labs are buying.

For the broader occupation, O*NET projects 5% to 6% growth from 2025 to 2035 for computer systems engineers and architects, which it rates faster than average, with about 27,000 openings a year on average. That projection covers a group much wider than HPC, so read it as context rather than a forecast for cluster engineers specifically.

Pay inside universities is public in some places. As one example, Penn State's salary structure puts its Advanced Professional Research Computing Systems Engineer in grade 13, with a range of $91,488 to $137,280, and its Principal Professional level in grade 15, from $115,860 to $173,844. That is one university's table, not a national range.

How AI is changing the work. The near-term change is new kinds of users. At the SC25 conference workshops in November 2025, Oak Ridge researchers presented an architecture for AI agents that run autonomous cross-facility experiments at ORNL's HPC and manufacturing user facilities, combining a natural language interface powered by a large language model with a multiagent framework for decision making. If agents like these submit jobs and move data on researchers' behalf, HPC engineers inherit questions about credentials, quotas, audit trails and how much of the queue a piece of software may claim.

Career paths. From a senior engineer seat, next steps include HPC architect, who leads system design and procurement; operations or center manager, who owns staffing, budgets and service levels; and solutions roles at system, storage and networking vendors. Engineers can also move into AI infrastructure teams at companies that train large models, where scheduler, fabric and storage experience carries over.

Skills worth building now: GPU partition management and accounting, direct liquid cooling operations, object and burst-buffer storage, Kubernetes alongside a batch scheduler, and the security controls that controlled research data requires.

Dear Hiring Committee,

I am applying for the High-Performance Computing Engineer position with your research computing center. For the past four years I have been one of two systems engineers on a mid-sized university cluster that serves chemistry, genomics, climate modeling and a growing group of machine learning labs, and I would like to bring that experience to a larger shared facility.

My job has covered the full stack. I maintain our Slurm configuration, and last year I rebuilt our fair-share and quality-of-service policy after two labs with large grants were starving smaller groups out of the GPU partition. The new policy uses separate accounts for course allocations and preemptible scavenger jobs, and queue complaints to our help desk dropped noticeably in the following semester.

On the storage side, I led the migration of our Lustre scratch file system to new object storage servers during a planned maintenance window, with a rollback plan that we fortunately never needed. I also wrote the purge and quota reports that department chairs now receive each month.

I build our software stack with Spack and publish it through Lmod, and I added Apptainer support so genomics users could bring the containers their pipelines already use. When an MPI job behaves strangely, I start with the OSU micro-benchmarks and the fabric error counters before blaming the code, and I have learned to explain those results to researchers in plain terms.

Your posting mentions a new GPU system arriving next year. I helped write acceptance tests for our last expansion, including HPL and IOR runs and a set of user application benchmarks, and I would welcome the chance to do that work at a larger scale.

Thank you for your consideration. I would be glad to walk through any of these projects in more detail.

Sincerely, Jordan Alvarez