CUDA Engineer

CUDA Engineers are the specialists who extract maximum performance from NVIDIA GPU hardware — writing, profiling, and tuning the low-level code that makes large-scale AI training and inference economically viable. When a model takes three days to train instead of one, or when an inference endpoint costs twice what it should at a given throughput, a CUDA Engineer is the person who finds out why and fixes it.

The work starts with architecture. A100 and H100 GPUs have specific memory hierarchies — registers, L1/shared memory, L2 cache, HBM2e/HBM3 — and writing fast code means understanding how each level behaves under realistic access patterns. A matrix multiplication kernel that ignores shared memory tiling will run at a fraction of the roofline performance no matter how clever the algorithm is. CUDA Engineers spend substantial time studying profiling traces in Nsight Compute — occupancy percentages, warp stall reasons, L1/L2 hit rates — and translating what they see into concrete code changes.

For AI workloads specifically, the focus areas are attention mechanisms, matrix multiplications using Tensor Cores, normalization layers, and the fused operator patterns that eliminate unnecessary memory round-trips. Flash Attention, for example, is a pure CUDA optimization insight — reordering the attention computation to keep activations in SRAM rather than writing them to HBM — and it delivered 3–4× speedups in training throughput that no algorithmic change could replicate. CUDA Engineers are the people who build things like Flash Attention and maintain them across new GPU generations.

Multi-GPU parallelism adds another dimension. Training a large language model across hundreds or thousands of GPUs requires NCCL collective operations (AllReduce for gradient synchronization, AllGather for tensor parallel layers) that need to be tuned for the topology of the underlying interconnect — NVLink within a node, InfiniBand across nodes. A poorly tuned AllReduce can stall compute and leave expensive GPU time idle.

On the inference side, the priorities shift toward throughput at minimum latency: INT8 and FP8 quantization, continuous batching, KV-cache management, and speculative decoding. Each of these involves CUDA-level implementation that determines whether a serving system hits its SLA at production load.

The job requires close collaboration with ML researchers and framework teams. When a researcher proposes a new attention variant or a novel activation function, someone has to decide whether the naive PyTorch implementation is fast enough or whether it warrants a custom kernel — and if a custom kernel is warranted, write and validate it. CUDA Engineers often own that decision and do that work.

Education:

• Bachelor's degree in computer science, electrical engineering, or applied mathematics with strong systems coursework
• Master's or PhD common at NVIDIA, AI labs, and research-adjacent roles; not required at infrastructure-focused companies
• Self-taught candidates with demonstrable kernel optimization results do get hired, but the bar on portfolio evidence is high

Core technical skills:

• CUDA C/C++: kernel launch configuration, shared memory management, warp-level primitives (__shfl_sync, __ballot_sync), cooperative groups
• GPU memory hierarchy: registers, L1/shared, L2, HBM — bandwidth vs. latency tradeoffs at each level
• Tensor Core programming: mma PTX instructions, WMMA API, CUTLASS template library for structured matrix operations
• Profiling tools: Nsight Compute (kernel-level metrics), Nsight Systems (system-level traces), ncu command-line profiling
• NCCL and CUDA-aware MPI for multi-GPU collective operations
• PyTorch internals: custom C++ extensions, ATen operator registration, torch.compile and TorchInductor interaction points

Supporting skills:

• Triton for rapid kernel prototyping and comparison benchmarks
• Python proficiency for test harnesses, benchmark automation, and ML framework integration
• CMake and CUDA build toolchain for library development
• Roofline model analysis — knowing whether a kernel is compute-bound or memory-bandwidth-bound before optimizing
• Linear algebra fundamentals: matrix decompositions, numerical stability considerations for mixed-precision arithmetic

What employers actually look for:

The interview process at NVIDIA, Meta AI, and similar companies involves writing CUDA kernels from scratch under time pressure — a matrix transpose with a specific tile size, an attention forward pass with masking, a reduction with warp shuffle instructions. Candidates are expected to know the arithmetic: how many clock cycles a global memory load takes on H100, what thread block size maximizes occupancy for a given register count, why a particular kernel shows high lgfm stall reasons in the profiler.

Papers and conference presentations on GPU optimization (CUTLASS, FlashAttention, DeepSpeed Triton kernels) are relevant to reading lists for this role. CUDA engineers who have contributed to open-source GPU libraries — or who have published performance analyses — tend to move through hiring loops faster.

Certifications:

• No formal certification pathway dominates. NVIDIA Deep Learning Institute courses provide baseline GPU programming credential, but employers weight demonstrated results over certificates.

The market for CUDA Engineers has expanded dramatically since 2022 and shows no sign of compressing. Every foundation model generation demands more GPU compute — GPT-4 was trained on roughly 25,000 A100s, and subsequent frontier models have consumed substantially more. The economics force organizations to extract maximum performance from the hardware they have, which means the people who can write and optimize GPU kernels are in persistent short supply relative to demand.

The supply side has not kept up. CUDA programming requires a combination of skills — hardware architecture intuition, systems programming discipline, numerical methods knowledge — that is rare and takes years to develop. Universities are producing more ML engineers than ever, but most ML curricula stop at the PyTorch API level. The gap between framework users and kernel writers is wide, and closing it takes real investment in learning the hardware.

Where demand is coming from:

Hyperscalers (Google, Meta, Amazon, Microsoft) are building and operating GPU clusters at unprecedented scale. Each of those clusters requires infrastructure software — kernel libraries, collective communication stacks, compiler backends — maintained by CUDA engineers. The infrastructure teams at these companies are several hundred people each and still growing.

AI labs (OpenAI, Anthropic, Mistral, xAI) compete on training efficiency. A 10% speedup in training throughput at OpenAI's scale translates to millions of dollars in saved compute cost per training run. That math makes CUDA engineers among the highest-leverage technical hires an AI lab can make.

Startups building AI inference infrastructure — Groq, Cerebras, Fireworks AI, Baseten — compete partly on kernel quality. Their go-to-market argument is often latency or throughput superiority, which requires CUDA engineers or their equivalent working on custom inference kernels.

The compiler question:

The rise of torch.compile, XLA, and MLIR-based compilers has automated kernel fusion and some memory layout optimization that engineers previously did by hand. This is real, and it has raised the floor of GPU performance that framework users can achieve without writing any CUDA. It has not, however, replaced hand-written kernels at the frontier — attention variants, speculative decoding, sparse operations, and custom quantization routines still require engineers who understand the hardware. The likely trajectory is that compilers handle the routine 80%, leaving CUDA engineers focused on the high-leverage 20% that determines competitive differentiation.

Compensation trajectory:

Total compensation for senior CUDA engineers at top-tier AI companies already reaches $400K–$600K when equity is included. As frontier AI spending continues to scale, the competition for engineers who can move the performance needle is intensifying rather than easing. BLS does not track this specialty separately, but industry surveys consistently show GPU software engineering among the highest-compensated software disciplines as of 2025–2026.

Dear Hiring Manager,

I'm applying for the CUDA Engineer position at [Company]. I've spent four years working on GPU kernel optimization for deep learning infrastructure, most recently on the model efficiency team at [Company], where I own the custom attention kernel library used across our production training stack.

The work I'm most proud of is a fused multi-head attention implementation for our 70B parameter model that replaced the baseline PyTorch SDPA path. Using Nsight Compute traces, I identified that the naive implementation was spending 60% of kernel time on HBM reads for intermediate attention scores that were immediately discarded after softmax. I implemented an online softmax pass with shared memory tiling — similar in spirit to Flash Attention v2 but adapted to our specific head dimension and sequence length distribution — and brought the attention step from 38% of total forward pass time down to 14%. That single kernel change reduced our training cost on H100s by roughly 9% end-to-end.

I'm also comfortable working up the stack. I've registered several of these kernels as PyTorch custom ops using the ATen dispatcher, written the Python benchmark harnesses that run in our CI pipeline, and walked ML researchers through the constraints (sequence length divisibility, dtype requirements) that determine when the fast path fires versus the fallback. Making the kernel usable matters as much as making it fast.

I'm drawn to [Company] because of the work your infrastructure team has published on speculative decoding and INT8 quantization — both are areas where I have hands-on implementation experience and would want to go deeper. I'd welcome the chance to discuss what you're working on.

[Your Name]