Sgr A*

HomeContactBlog
HPC Infrastructure at UTA: A Summer in Monitoring thumbnail

HPC Infrastructure at UTA: A Summer in Monitoring

Six weeks at the University of Texas at Arlington, working on the infrastructure behind the ATLAS experiment at CERN. Here's what I actually got to build, debug and deploy.

Maksym's avatarMaksym
November 16, 2026

This summer, I spent six weeks at the University of Texas at Arlington (UTA), working with the SouthWest Tier-2 (SWT2) Computing Center.

SWT2 is part of the Worldwide LHC Computing Grid (WLCG), providing computing and storage resources for the ATLAS experiment at CERN. In practice, that means operating a distributed HPC environment where reliability, resource management and monitoring are not exactly optional.

My internship was focused on infrastructure monitoring and automation. I worked with Linux, SLURM, Prometheus, Grafana, Zabbix, Python, Bash and Go, contributing to several projects across the monitoring stack.

It was a pretty interesting change of perspective from application development. Instead of building something that a user interacts with directly, most of my work was about making sure the infrastructure underneath everything was observable, healthy and doing what it was supposed to do.

And, as it turns out, there's quite a lot to talk about.

Getting familiar with the infrastructure

Before touching anything, I had to understand how the existing environment was put together.

The monitoring stack primarily relied on Prometheus for metrics collection and Grafana for visualization. SLURM handled workload scheduling across the compute nodes, while other components provided hardware metrics and operational information.

One of my first tasks was deploying Prometheus Node Exporter on the test cluster and configuring Prometheus to scrape the metrics.

There was a small automation problem worth addressing: maintaining a list of compute nodes manually is not particularly attractive when the cluster itself already knows which nodes exist.

I wrote a Python utility that queried SLURM and generated Prometheus target configurations from the available node information. This removed the need to maintain those target lists by hand and established a more convenient starting point for the monitoring work that followed.

Nothing revolutionary, but a good example of the kind of engineering I ended up doing throughout the internship: taking an existing workflow and removing unnecessary manual steps.

Thermal monitoring and SLURM health checks

One of the more interesting projects involved monitoring CPU temperatures across compute nodes.

HPC workloads can keep processors under sustained load for extended periods. That makes thermal monitoring useful not only for performance analysis, but also for detecting hardware conditions that could affect node reliability.

The first part was relatively straightforward: expose CPU temperature metrics and build Grafana dashboards to visualize them.

The more interesting part was making that information actionable.

A dashboard can tell an administrator that a processor is getting too hot. But what if the infrastructure could detect that condition automatically and report the affected node to the scheduler?

That's where SLURM's existing Node Health Check (NHC) framework came into play.

I developed a custom thermal health check that reads processor temperature information and compares it against a predefined threshold. When the temperature exceeds that threshold, the check can report the node through the existing health-check mechanism.

To test the monitoring under realistic conditions, I also created a dedicated SLURM workload that generated sustained CPU load on a selected compute node. This allowed me to observe temperature changes and validate the monitoring without putting the rest of the cluster under unnecessary stress.

One important implementation decision was to integrate with the existing NHC workflow rather than modifying the framework itself. Keeping the solution compatible with the infrastructure already in place made it easier to maintain and deploy.

The result was integrated into the production monitoring environment and progressively deployed across the production ATLAS cluster.

This was one of my favourite projects because it connected several layers of the infrastructure: hardware telemetry, monitoring, custom scripting, SLURM and automated node health management.

Making Grafana actually useful

I also spent a considerable amount of time working on Grafana dashboards.

The goal wasn't simply to add more graphs. A monitoring dashboard is only useful if the information it presents is easy to interpret and helps administrators understand what is happening.

I extended existing visualizations to include CPU temperatures, utilization, memory usage, disk activity and network traffic. I also worked on improving PromQL queries, cleaning up unnecessary aggregations, using metric labels more effectively and adding data links between related dashboards.

There was another issue I got to investigate: Prometheus occasionally stopped receiving metrics from Node Exporter when compute nodes were under sustained CPU load.

This was particularly interesting because the exporter itself was running on the machine it was supposed to monitor. Under heavy load, the process could struggle to get enough CPU time to collect and expose metrics reliably.

I experimented with systemd service configuration, including increasing the scheduling priority of Node Exporter. During stress testing, this improved metric collection reliability and reduced temporary gaps in the monitoring data.

It's a nice reminder that monitoring systems have their own resource requirements. Collecting metrics is not free, and the monitoring agents need to remain functional even when the machines they're observing are busy doing actual work.

Exploring Zabbix alongside Prometheus

During the second half of the internship, I worked with Zabbix.

The existing environment relied heavily on Prometheus and Grafana, so the idea was to explore how Zabbix could complement the monitoring stack and provide additional visualization and alerting capabilities.

I started by experimenting with custom monitoring items, triggers and dashboards in the test environment.

One of the tasks involved extending the default Linux monitoring template with CPU temperature metrics. Instead of creating an entirely separate dashboard, I duplicated the existing template and added the required items.

I also implemented preprocessing using JavaScript to convert temperature values from Celsius to Fahrenheit. This allowed both measurement systems to be displayed depending on the administrator's preferences.

Alongside the configuration work, I documented the process of modifying the default Zabbix dashboard so that the changes could be reproduced or extended later.

This project was less about writing large amounts of code and more about understanding how an existing monitoring platform can be adapted to the operational requirements of a particular environment.

Adding service availability checks with Blackbox Exporter

Most of the monitoring work so far had focused on the machines themselves: CPU, memory, temperature and other hardware metrics.

But a healthy machine doesn't necessarily mean that the services running on it are reachable.

I worked on extending the monitoring setup with Prometheus Blackbox Exporter, which allows Prometheus to actively probe network services instead of relying exclusively on metrics exposed by the machines.

I deployed the exporter in the development environment, configured it as a systemd service, defined probe modules and integrated it into the existing Prometheus configuration.

For validation, I configured TCP probes against selected services and created Grafana visualizations for the resulting metrics.

I also simplified the target configuration during development. An initial file-based discovery setup was replaced with a static configuration that was more appropriate for the relatively small set of services being monitored.

The result was another layer of observability: instead of only asking whether a machine was reporting metrics, administrators could also check whether selected services were actually reachable.

Fixing the SLURM Exporter in Go

Probably the most interesting software development task I worked on was contributing to the Prometheus SLURM Exporter.

For context, SLURM is a workload manager commonly used in HPC environments. It schedules jobs across compute nodes and manages the resources allocated to those jobs.

The Prometheus SLURM Exporter exposes information from SLURM as metrics that can be collected and visualized through Prometheus and Grafana.

During routine monitoring, I noticed an issue with how the exporter calculated allocated CPU resources for partitions. The metric being used did not accurately represent the actual number of CPU cores allocated to running jobs.

This meant that the monitoring data could give an inaccurate picture of resource allocation.

I started by digging through the existing Go codebase to understand how the exporter collected and processed SLURM information. Rather than trying to patch the calculation blindly, I wanted to understand the existing collection workflow and where the discrepancy came from.

The solution was to implement a new collection method that queried SLURM directly and aggregated allocated CPU core information on a per-partition basis.

func AllocatedCoresData() []byte {
		cmd := exec.Command("squeue", "-h", "-t", "RUNNING", "-O", "Partition,NumCPUs")
		stdout, err := cmd.StdoutPipe()
		if err != nil {
			log.Fatal(err)
		}
		if err := cmd.Start(); err != nil {
			log.Fatal(err)
		}
		out, _ := ioutil.ReadAll(stdout)
		if err := cmd.Wait(); err != nil {
			log.Fatal(err)
		}
		return out
}

// get list of allocated cores from the function that uses squeue command
allocatedCoreLines := strings.Split(string(AllocatedCoresData()), "\n")
for _, line := range allocatedCoreLines {
		fields := strings.Fields(line)

		// if the field contains 2 elements, it has partition and number of cores
		if len(fields) == 2 {
			partition := fields[0]
			cores, err := strconv.ParseFloat(fields[1], 64)

			if err != nil {
				log.Printf(err.Error())
				continue
			}

			// add cores if partition exists
			if _, exists := partitions[partition]; exists {
				partitions[partition].allocated += cores
			}
		}
}

After implementing the change, I compiled and tested the modified exporter, then deployed it in the production ATLAS monitoring environment.

The updated metric collection was successfully integrated into the production monitoring workflow.

I also submitted an upstream GitHub Issue describing the problem and a Pull Request containing the proposed improvement.

This was a particularly rewarding task because it went beyond configuring monitoring tools. I had to understand an existing open-source project, navigate its codebase, implement a change in Go, validate it against a real workload and follow the contribution process.

It's one thing to write code that works on your machine. It's another to make a change to a component that is already part of a production infrastructure.

A bit of actual infrastructure work

Not everything happened in a terminal.

I also participated in daily team meetings, where we discussed ongoing tasks, infrastructure status and operational issues. It was a useful way to understand how a research computing team coordinates its work and how technical decisions are made.

I also had the opportunity to participate in hardware-related activities inside the data center, including replacing components and working around network and power cabling.

That was a nice change of pace from staring at dashboards and source code. It also gave me a better understanding of the physical infrastructure behind the systems I was monitoring.

What I learned

Looking back, the biggest takeaway wasn't a particular technology or tool.

It was getting to work with an existing production environment where changes actually matter.

A lot of the work involved understanding systems before changing them, figuring out how different components interact, and making improvements without breaking workflows that were already in use.

I got practical experience with Linux system administration, SLURM, Prometheus, Grafana, Zabbix, automation and Go development. More importantly, I got to see how these technologies fit together in a real HPC environment.

Some of the solutions I developed were deployed into production, and others were prepared for upstream contribution. That made the experience especially valuable.

It also reinforced my interest in systems engineering and infrastructure. There's something particularly satisfying about working on the less visible parts of computing: the monitoring, automation and infrastructure that allow everything else to function.

Six weeks went by surprisingly quickly. I came back to Switzerland with a lot of new experience, a much better understanding of HPC operations, and quite a few things to explore further.

And, of course, a few thousand photos of Texas and Florida (after the internship was finished I went there for a couple days).


Thanks to Dr. Kaushik De, Andrey Zarochentsev, Zachary Booth and the entire SWT2 team for the opportunity, guidance and support throughout the internship.

Any questions?

Contact meSee other posts