Archive

Hi!

Last week we talked about ways to say no to incoming requests; besides a flat-out no, I mentioned the possibility of involving another team, or supporting the research group in doing the work themselves. RCT community member Adam DeConinck sent in another suggestion:

Occasionally, I’ve seen requests come in for work that the research computing team isn’t going to do; but which is also not something that another provider exists to do, and which the customer can’t do themselves even with training. When that happens, the “support” I’ve been able to offer is to join the customer in advocating for change. For example, aggregate similar requests and find commonalities; then organize the customers involved and advocate to upper management that they provide resources for this work to be done. That’s often not terribly satisfying, and I’ll admit that my success rate in doing this isn’t very high. (Non-zero, though!) But it’s often the only way I’ve found to create new classes of service within an existing organization.

This is a great suggestion, taking the long term view - and by lending your voice to the research groups’ in advocating for support for different kinds of work, it turns the situation into “your team and the research group vs the problem” rather than “your team disagreeing with the research group”.

Hi!

A question came in from a reader - I’ve talked before about how to not do things, and in my recent talk I said “do whatever researchers ask for” isn’t a reasonable strategy. We have finite resources, and we owe it to our communities and to science to focus those resources on the highest-impact things we can be doing. But what does that actually look like when you’ve been asked to do something well out of scope? Below’s a lightly edited version of what I sent back:

It’s important to pay attention to user needs, and thank them for their requests - if nothing else it’s useful data! But we can’t fall into the trap of trying to do everything a researcher asks for (or even every thing many researchers are asking for!). People want a lot of things, and our time and resources are finite. It’s perfectly ok for people to ask us for things, and it’s perfectly ok for us to say no.

Here’s a partial list of examples where it wouldn’t make sense for a team to provide a service that’s asked for:

Hi!

I want to write one more thing about strategy this week before going back into the basics of managing our research computing teams.

It turns out I’m still a bit discouraged by one section of the recent (very good!) PEARC21 report on strategic planning for RCD groups. That part described all too clearly the confusion and helplessness many team leaders of teams like ours feel when asked to do some positioning and strategy work. Like so many things, we’re thrust into responsibility for these kind of activities without ever being taught anything about them. It’s intimidating. But it shouldn’t be, and doesn’t have to be, that way.

The thing is, finding a path forward for a team or organization is a research and problem-solving exercise.

Hi!

I write a lot here about challenges our teams face, but I don’t think I mention often enough the huge advantage our background in science is in our work (or in similar work in other environments).

Whether we were trained in science or just picked up that mindset by immersion in the ecosystem, the skills we have are skills that (say) tech companies pay real money to train their staff in. The advanced collaboration and influencing skills we have to develop in scientific collaborations is a huge advantage all the way up the management or technical leadership ranks, for instance. Being willing to think in terms of experiments when making changes (like Beinbold’s article, below), or having a “growth mindset” (current jargon for believing that people can develop new skills(!!)), or being able to synthesize information - these fundamental skills, skills we don’t even bother mentioning in our environment, are in high demand.

It’s not just people skills, either - I see an article about staff engineer interview skills that include being able to estimate well, something every quantitative scientist does without thinking about it, or another article this week about how tech companies are having to relearn basic performance things like array-of-structs vs struct-of-arrays, which many of us have had to learn optimizing scientific code.

I heard a number of answers back about what sorts of service request processes people were using:

• A lot of software development teams, of course, took bug reports and small feature enhancement request via GitHub/GitLab/Jira etc ticketing systems
• Several groups are using Znuny, a fork of OTRS
• Someone juggling multiple projects used a timetracker to make sure reasonable amounts of time were being spent on various external projects
• A technical leader of an internal platform team said that it’s mostly “’support channel in Slack’ With complex issues moving to email” (but they had succeeded in at least getting the requests into the channel rather than DMs!)

One data science team had a longer answer that I want to quote here:

For a relatively small team of a few staff and about 10 students taking about 250 consultation requests a year one on one help on data science topics, not help tickets for a service), we use […] – basically a form + spreadsheet (google forms/sheet would work too). We have columns to track the status (incoming request, in progress, completed, etc.) and who has responded/been assigned, along with the info about the request. We don’t put other significant updates on the consults in this sheet – just for tracking status. We make sure our team brings the status of everything up to date at team meetings. Requests ping a slack channel when they come in, and we determine who will respond. So we can make sure that someone took each consult, even if the sheet doesn’t get updated immediately (it should be, but it doesn’t always happen).

For longer projects, we have only 6-12 or so per year – those go on a separate sheet for recordkeeping/reporting. All updates are communicated via team meetings or 1:1 meetings.

Before we get started, please answer if you can a question from a reader: What do you use to get and keep track requests for your work from researchers? Is there a ticket-tracker type piece of software that works for you, or do you use something else? Does it fit your workflow - What do you like, not like, and wish for?

I’ll summarize results next week - it’s an important topic!

One of the great things about this community is the different kinds of research computing and data teams I get to talk with, “here” or outside the inbox. Sometimes in these conversations it’s pretty clear that some teams, or the team and their funder, or a team and I, are talking a bit past each other. And that’s usually because they or we are (currently) operating with very different mental models.

Hi!

I write a lot here on the challenges that research computing and data teams face, and the challenges that their managers and leaders face in particular. And that’s because I want them to have fewer challenges! I hope that by highlighting the challenges they at least don’t face them unawares, and that as they grow their skills they bring along the next generation of managers and leaders under clearer, less stressfull, and more suppported environments than we found ourselves in.

Which is all well and good, but it can get a little gloomy in here sometimes. I don’t think I spend enough time highlighting successes.

Do you have management or leadership wins you want to share, no matter how small? Just with me, or with the community (anonymously or otherwise)? Let me know - just hit reply and send me an email at jonathan@resarchcomputingteams.org. I think it would be good for everyone here to see what some of those wins look like.

Hi, all!

I want to write a little bit more in the coming year about strategy for leaders of research computing and data teams: setting priorities; deciding what and what not to do; and deciding what success looks like. It’s an important topic - and yet as with so many things in our line of work, no one teaches us how to do it. It’s also much more ambiguous in our discipline than it might have been a researcher, where you can keep score pretty simply with papers and grants.

Those of us who have gone through University strategic planning processes come out even more confused, because these “Strategic Plan” documents aren’t a great introduction to strategy at the best of times, and they’re particularly odd ducks in the context of a University.

There was a great article by Alex Usher and team at Higher Education Strategy Associates last week. I want to use it to set the stage for what a strategy is, what a good overall framework does for teams or orgs, why Strategic Plans for universities are so constrained, and what that means for us. Usher has written quite a bit about strategic plans in the past, some of it quite cutting, and all of it insightful - I recommend the articles, and the blog in general, if this topic interests you.

Hi!

I wanted to write more this week about the consequences of research computing teams as vendors, and what that means for strategy - I’ll do that next week.

But there was big news in HPC world in this past week - as you almost certainly already know, the Frontier super at ORNL is now the world’s first Exascale system as measured by the HPL benchmark.

This is an enormous technical (and project and product management!) accomplishment, the capstone of work that’s been ongoing since 2006 or so. Colleagues at AMD (which, disclaimer, has long been a worthy competitor of my current employer), HPE, and ORNL deserve all the praise they’ll rightly get. My sympathies to other colleagues at Argonne whose star-crossed Aurora system, facing a series of delays, didn’t make it across the line in time.

Hi!

My new job, working for a large company that explicitly sells stuff, has indeed been eye-opening — but not in the ways I expected. Mainly I’m confronted with clarity about some of my old jobs, and peer groups I’ve worked with.

I’ve always been sort of puzzled by the decisions made and priorities chosen by some research computing and data teams. Those teams seemed more… insular, somehow, than others. I saw it less often in contract research software development teams, or library research data management teams, or bioinformatics core facilities. I saw it more often in research systems teams — not all, of course, and not only.

Meeting with some of these old peer groups, wearing a vendor hat, and watching them interact with us, it becomes a lot clearer.

Hi!

I took part in a hackathon this week, working directly with researchers on improving their code for the first time in — gosh, maybe eight years? Nine?

It’s funny what sticks in the mind and what doesn’t. I kept coding on and off over the past decade, so that was fine, but all the glue stuff that I didn’t think to stay in practice on like “how do you unload all modules?” or “why doesn’t this shell script that sets environment variables work when I run it?” (yeah, that one’s kind of embarrassing) took some time to remember.

There were more data analysis projects, and infinitely more AI/ML projects, then there would have been a decade ago when it would have been 90% simulation codes. Technology has changed over the past decade — there are some new profilers, and tooling, systems are faster, and hey Power systems aren’t big endian any more! But I was mainly struck by how little change there had been in the basic challenges. Maybe the percentage of research groups with extensive in-house research computing and data skills has gone up a bit, but mostly the research groups we saw were very capable, deeply knowledgeable in their domain, with significantly less expertise on the computing, software, and data management side. They needed, and knew they needed, collaborators to work with to achieve their goals. And our responsibility was to do a little bit of hands-on-keyboard work, but mainly to give them the skills and resources they needed so they could continue to advance even after the engagement; to give guidance as trusted experts. And that expertise ends up straddling at least two of software, systems, and data - those aren’t useful in isolation.

Hi!

The (159-item!) intern process checklist from last week generated a few interesting discussions, and those plus Chris Dwan’s interesting talk discussed below return to a common theme here in the newsletter - of projects versus processes, or learning versus production.

In #91 I alluded to the fact that what distinguishes a hobbyist from a professional is repeatability. The hobbyist is still developing their skill, trying to make their one-off thing come out with high quality. The professional is no longer struggling to make one thing well; they’re trying to ensure that things reliably, repeatedly come out with the same high quality.

The hobbyist is thinking about the mean quality, and trying to get it to trend upward; the professional is concerned with the variance, and reducing outliers. (Mean matters, too, but for a professional, a high mean quality is table stakes). There’s a progression from focussing on skill to focussing on reliability, from craft to process. One of Dwan’s slides says:

A reader wrote in to ask:

We have a new summer intern starting in a few weeks who I will be directly supervising. Although I have been the technical/project lead on various projects throughout my career, this is my first time having a direct report (even though it will be temporary!). I was wondering if you had any “go to” or favorite literature for managing in a technical setting.

Congratulations to them, and this is a great question. It’s such a great question that I’m disappointed to discover that I didn’t have handy resources to immediately share with them, and I’d love to hear what your recommendations are.

In research computing, probably the two most common ways people find themselves managing is to be promoted to lead their existing team, or the more gradual approach which is to start with one or more interns/co-ops/grad students/trainees/etc.

Hi!

I wanted to talk a little bit more about Research Computing and Data (RCD), how it connects to other work, and how that informs how I think about when RCD work is and isn’t a research output.

First, I want to make a claim that most readers here would likely agree with, but may not yet be universally accepted elsewhere: RCD is a discipline in and of itself. It is a body of knowledge and practice, with full time practitioners, conferences, knowledge creation, sharing, and — increasingly — degree programs.

In fact, I’d make the argument that RCD been one of the most productive and influential academic disciplines of the last several decades. And an academic discipline it is - I don’t think anyone would doubt that the development of the FFT, or finite element methods, or new methods for processing next generation sequencing data or supporting digital humanities work, are real and significant research and intellectual accomplishments.

Hi!

My discussion last week about when research computing and data work is and isn’t a research output in and of itself produced some replies, as the topic often does, that ranged from intrigued to puzzled to grumpy. I’ll write some more next week on this (I even have a diagram sketched out), so people can have some really specific things to be intrigued or puzzled or grumpy with, but I first want to talk about the multi-dimensional aspect of research computing and data efforts.

Here’s the thing - all research computing and data work is advanced research computing.

It might not feel like that. Reading about the cool cutting edge tech things that are being done in industry, or comparing to the huge HPC systems at the DOE labs or at PRACE, it’s easy to feel a bit in a backwater. (I’ve written about this in the past).

Hi!

I’ve long tried to describe the sudden disorientation of losing immediate feedback that comes with being a first-time manager. Recently I heard the term “managerial sensory deprivation”, and I think that’s a useful phrase. When you’re hands-on doing the work of reearch computing and data, you get immediate, almost tactile feedback with what you’re doing, whether it’s from the systems or from the researchers. As a manager, any action you take has only indirect results that could take days, weeks, or months to really play out. And by that time, it’s hard to connect it unambigiously to something you’ve done.

Your work senses are getting no signal back, or weak signals seemingly unconnected to what you’re currently doing. You’re feeling a kind of sensory deprivation. And wikipedia helpfully reminds us that “extended or forced sensory deprivation can result in extreme anxiety, hallucinations, bizarre thoughts, temporary senselessness, and depression”.

This has come to mind lately being a new part of a big organization and, frankly, having no idea what I’m doing yet. I’m accomplishing stuff, but is it the right stuff? Is it having any kind of impact? If yes, is that impact positive? How and when will I know? Having moved organizations (and sectors!) I’ve again lost my bearings and my feedback mechanism. It’s apparently not an unusual situation; I heard the same thing from a reader who I talked with earlier this week, who’s also in a new job. (By the way, I love hearing from RCT readers - never hesitate to email, or even book a quick chat.)

“Contact us for pricing”.

Say we’re looking at a potential new tool, for us or the team as a whole. When we want to learn about whether it’s feasible as a solution to our needs, we see “Contact us for pricing”. Or occasionally, even for information about the product itself, “Contact us for details.”

“Fantastic!”, we think. “After a few emails and a call or two, not only will we get the information we want, maybe we’ll make a new friend!”

We of course do not think this, because no human being has ever thought this. More likely, we reasonably view it as an imposition on our time, an artificial barrier to getting the information we want. Most of the time we likely bounce, never looking at the page again. Sometimes, we may reluctantly agree to the process, and trepidatiously fill out the form. Perhaps we use a burner or obscured email address, perhaps we don’t. Either way we kind of resent it.

Hi!

Well, it finally happened. The time came for something I’ve been dreading since taking a job with a vendor; something which held me back from taking such a job earlier.

I had my first meetings with former colleagues from the vendor side of the table.

And… it was fine! It wasn’t weird at all.

Hi!

It’s been pointed out I have my view of the role of specialization in research computing teams seems a bit contradictory. I am very pro-specialization, but anti-siloing. Fair point! Let’s dig more into that.

No research computing and data team can do everything. It’s not possible. Pretending otherwise leads to unhappy team members and unsatisfied researchers. And as I like to point out, there’s already tonnes of computing things our teams don’t do for researchers. We don’t help them with their printers, or lay out designs of posters for conferences. So my advice is wherever possible — and it’s more possible than you think — is to add a few things to that list. To niche down into a specialization.

The most meaningful specialization is of the form of “we help [community] with [problem]”. One very effective way to do that is where community is a particular discipline (defined broadly). “We help wet lab molecular biologists with their bioinformatics analyses”. “We help social sciences researchers with their GIS software development needs”. “We help researchers with long term data retention needs affordably and reliably store and share their data”. “We help health scientists write and deploy desktop and mobile data collection applications”. “We help researchers new to quantitative methods with data science”. “We help researchers with web applications deploy, operate and maintain their application stacks”.

Hi!

We now have 200 members of our community; people who care about research computing and data teams, their potential, and the importance of leading them professionally.

So, welcome new readers! Some resources that might be useful:

• The Rands Leadership Slack (a community of 20,000 managers and leaders in or around tech) and our new and so far small (16 people) #research-computing-and-data channel there
• My “Help, I’m a Research Software Manager!” talk & slides from 2020 which covers in 10 minutes the basic approach I take in the newsletter and elsewhere.

Hi!

My current job, like most of my previous jobs, include a lot of people who have worked for a long time in research — either as researchers themselves or in other roles, with Ph.D.s, or without.

I sometimes write about how working in research teaches us behaviours that are unhelpful when we get to management. That’s certainly worth calling out, but it’s only part of the story. Some of the behaviours and habits of mind we learn are extremely helpful, and we don’t even notice it.

In business management writing, you’ll see lots of discussion of “growth mindset” and “comfort with uncertainty”. These are real gaps they see in new hires and especially new managers. And yet it sounds like meaningless fluff to us, because we can’t even imagine it.

Hi!

I’ve been thinking of jobs a lot lately, of course, for my own reasons and just through the effort of curating the research computing teams leadership job board. (The board has gotten a number of community submissions lately - keep them coming! I just check them for spam before accepting them).

A year and a half ago I posted my observations on the first 500 jobs posted to the board - we’re getting close to 1,500 now, and it’s worth taking a look to see what if anything has changed in research computing team leadership jobs.

There are some trends that have continued since the posting. The jobs in industry are growing vastly beyond what I would have imagined possible when I started in research computing in the 1990s. Rather than technical computing being a niche, it’s utterly mainstream now. There are a lot of jobs out there, and I don’t even bother posting generic “data science manager” jobs unless they’re connected to some real complex research questions - which happens a lot, whether it’s fraud detection or improving financial modelling or whether it’s supporting biomedical research. Some really fun-looking jobs that would probably feel a lot like working at a research computing centre keep coming up at consultancies – go visit a client and help them with their data science/data engineering/etc needs. There’s also a growing number of data science/engineering jobs at Universities that fall under the Provost/VP Operations rather than the VPR’s side of the house - Institutional Research, looking at (say) student success in support of the teaching mission.

Hi!

I won’t lie; when I started offboarding myself from my current job a few weeks ago I was a little anxious. I was pretty sure that team members would step up and learn quickly. It felt relatively unlikely that we’d discover gaps that could threaten near-term milestones. One worries nonetheless.

Instead, I’m really impressed with how well everything has gone. If anything the team has come together even closer, rallying to fill such holes as my departing creates. There’ll certainly need to be another hire - the team is losing a person’s worth of effort, after all. But the big picture work, both technical and stakeholder facing, is in extremely capable hands. I’ve felt increasingly superfluous this past week. That’s the best possible outcome.

But that also means that I’ve been surrounded all month with examples of things I could (should!) have been delegating and documenting earlier. Activities I thought only I could perform are in fact somehow getting performed. Things I could have usefully have documented ages ago, are only now finally getting written up. It’s not a disaster, nobody died of it. And yet there were real missed opportunities for team member growth and for me to have spent more time focussing on strategic goals.

Hi!

A slightly short newsletter today, as I finally claw my way back onto the Friday schedule.

One community update from last week - there was a fair amount of interest in using the Rands Leadership Slack as a pre-existing place with a large (>20,000!) leadership and management community as a place to discuss topics of interest. There is now a #research-computing-and-data channel, with sixteen members so far - some newsletter readers, but also some who had already been on rands and quickly joined once the channel was created. If that sounds potentially of interest to you, we’d love to see you there!

Otherwise this week I’ve mainly been continuing to offboarding myself from my current job. The coming week is my last (and I think the final 1:1s are going to be tougher than I was anticipating). Most of the knowledge transfer has now been done — although to wrap it up, next week I’m giving a demo of a key piece of technology to the team. Manager does a demo! What could possibly go wrong? This week there’s been a lot of 1:1:1s, with a team member and either I and my replacement, or the replacement and our boss, about transfer of responsibilities or “what’s next” conversations.

Hi, everyone:

Several readers responded about their version of the challenges I described last week - most teams are still working remote; some had reduced-occupancy office space available for those who wanted to come in, although it sounds like during the Omicron wave that was sparsely taken up where allowed at all. And those who mentioned it agreed uniformly on how hard it was to hire these days.

I think in some places we need to come up with our own solutions to these problems that meet the challenges our teams face (small, relatively slowly-growing teams with wide scope and demands that can change month-to-month and weird funding), and in other places there are opportunities to learn from other kinds of groups. This newsletter is a great medium for sharing resources, but it’s a bit of a slow way to have a back-and-forth on topics of timely interest! I’ve mentioned it before and included it on lists of resources, but the Rands Leadership Slack is a great community for technical managers and leaders, largely from tech and startups; if that’s something that interests you, dear reader, let me know you’ve joined or are considering joining and we can create a research computing and data channel for discussion of our particular needs.

In the meantime, news on my front. I’ve been thinking about people leaving jobs a lot lately, because we just found out one of my team members is leaving, leaving a big hole in the team - but also I’ll be changing jobs, too. (More about this in a bit). So we’ve been going through two off-boarding processes at the same time, one (mine) much lengthier than the other, involving two somewhat different groups of people, and requiring knowledge transfer at different levels of granularity, which is an interesting perspective into the situation.

I’ve had great conversations with research computing and data team managers this week. And while the the discussions were great, they weren’t all sunshine and light. RCT leads are facing some common struggles right now. We’re all wrestling with two big challenges: industry hiring like mad; and the likely end of a pandemic wave.

Industry’s sustained hiring spree means we all have to fight to hire and retain team members. Anyone with strong computational skills and a LinkedIn profile has to fend off requests for conversations about attractive positions. To hire and retain we need strong career ladders, which is something we can work on together. We need to find ways to build on the strengths of our teams and our work places, which is something where we can benefit from can sharing ideas.

And as another pandemic wave looks to subside, we face the opportunity and challenge of more in-office work. As was pretty clear even in 2020, “everything resets back to 2019” isn’t going to happen for most of our teams. On one hand, pure remote is out of reach for many. It may violate institutional or even funding requirements; and it doesn’t take full advantage of the real strength of being embedded in a research ecosystem. On the other hand, being expected to commute in every day like clockwork is going to rankle many team members, and the opportunity to hire from a broader geographic catchment area is too valuable for a lot of teams to forgo entirely.

So many of us are wrestling with what hybrid work arrangements will look like. Hybrid is going to be much harder than in-person or 100%-remote teams, each of which has pretty clear expectations. There’s mostly just one set of best practices for working on-site, and one set of emerging practices for 100% remote. Hybrid will be different. A whole spectrum of arrangements are “hybrid”; there’ll end up being a few clusters of good practices depending on what teams decide to aim for. Finding and disseminate what works for our very client-driven, interdisciplinary work and teams will take a lot of knowledge-sharing

Hi!

As you can see from this issue again being late (but less late!), here at RCT HQ I’m still getting into the rhythm of the new year. I hope you and your team are already firing on all cylinders.

There’s several articles this issue on decision making, priorities, and strategy. I keep coming back to this, because it’s important.

For research computing teams to succeed, we need priorities and a strategy. Without those, we can’t even tell if we’re succeeding, or on our way towards success.

Hi everyone - Happy New Year!

Here at RCT HQ, we’re slowly getting back into the swing of things after a long and relaxing holiday break - I hope it was relaxing and refreshing for you, your close ones, and your teams as well.

Since last we spoke, the big RCT-relevant discussions online have been about the aftermath of the Log4j fiasco, which was so big and had such large impact that in the US, the Federal Trade Commission as well as the national security apparatus have gotten involved.

Leaving the security aspects aside, the fundamental issue of scandalously under-resourcing the maintenance of underpinning open-source software is one which has been brought to the forefront, with even policy makers now well aware of the problem. Useful software will get used, often in unpredictable ways - that link takes you to a Google security blog post describing how 8% of Maven Central packages use log4j, but only 1/5 of that is of direct usage, the rest being indirect use by a dependency of the package, or a dependency-of-a-dependency-of-a-dependency. But there’s no robust mechanism for making sure the foundational work that underpins the flourishing ecosystem is supported. That, coupled with the high-profile bricking of faker.js and colors.js, have made the questions of what the broader community owes to foundational software maintenance, and what (if anything!) foundational software maintainers owe to the community, a central topic.

Hi!

There seems to be something in the air - in the past week I’ve had or heard a number of conversations around leadership and managmement.

Leadership vs management is a fun conversation to have in the same way that “is a hotdog a sandwich” is a fun conversation; it can clarify your own thinking but is unlikely to change too many minds.

Still, I think the basic outlines are clear and well agreed on. Management is a specific set of skills and activities particularly useful in particular jobs — people management especially, but also product or project management. It’s a woefully under-appreciated set of skills and behaviours, especially in academia-adjacent areas, and that means that you you can get significantly better at it than most peers pretty easily. By caring enough about it to subscribe to a newsletter, gentle reader, you’re already comfortably in the top 50%.

Hi!

It’s a busy time of year, as we try to get things closed out before the holidays. For a lot of us, outlines of opportunities and connections for the next year are starting to come into view; if that’s the case for you, I hope you like what you are seeing!

For us, the familar challenges of negotiating multi-institutional (and multi-sectoral!) collaborations are starting to yield to steady effort, and the exciting work we want to do will be beginning in the new year. It’s an exciting time, but means the newsletter is late again this week.

Next issue will be the last one for 2021; that will give us all a little bit of a breather to rest and recharge, and start again fresh in 2022.

Hi!

If you follow HPC twitter at all, you will have seen a heartfelt thread by a well-known research software developer, one who was a key contributor to the Singularity project among others, lamenting the frankly appalling state of developer productivity in HPC - both in what tools exist, and support for them (and other tools for developers) at academic centres. A lot of people chimed into the discussion, including one of the leading developers of the PetSC project, embedded software developers, some key people at big computing centres, all agreeing that there was a problem, but typically zooming in on one or another particular technical or procedural issue and not coming to any conclusion.

I think the issue is a lot bigger than HPC software development - it comes up in too many contexts to be about specific technical issues of running CI/CD pipelines on fixed infrastructure. The only people to identify the correct underlying issue, in my opinion, were people from the private sector, such as Brendan Bouffler from AWS:

Too much reliance on ‘free’ labour - postgrads and postdocs, who, invariably, decide that burning their time being mechanical turks for their ‘superiors’ just sucks, so they come and work for us. And since we pay $, we’re not gonna waste them on things software can do.

Hi!

A reader from a U15 research institution [Canadian research-intensive universities; think Russell Group in the UK, or R1 universities in the US] writes in to describe expanding their research computing team with unconventional roles:

First, a grant advisor, specifically to assist PIs in writing sane tech inclusions in their grants. You may have reviewed grant proposals where the medical science, particle physics, quantum chemistry, etc. is very clear, and then the explanation of the computational aspects and the equipment justification sounds like Dilbert’s boss wrote it. That is precisely what this position is intended to improve, but also sitting on internal panels that judge Innovation Fund proposal maturities before they’re allowed to apply, etc. Second, a Communities of Interest Coordinator, who will foster and support research communities of like-minded graduate students, PDFs, etc. around research fields making use of computation—bioinformatics, AI, digital humanities—or around digital research tools—R, Julia, MATLAB, Gaussian, etc. By supporting communities of interest, these groups can become shared knowledge hubs, where newbies can find guidance or “the ropes” and experienced but stuck researchers can find inspiration or “an ear” that might enable them to unstick. Both positions have been filled internally and start in December. More traditional ARC job descriptions are being written up now as part of a further expansion.

I love this! It’s a long-standing tenet of this newsletter that research computing is much more than just technology. It’s teams, it’s communities, it’s product management - it’s people. Connecting researchers and their more directly to the computation, software, and data resources that can advance their work — whether that means in grant writing or capacity building within a practitioner community — is very much part of the our broad remit in research computing and data.

Hi, everyone:

It’s been a good week here at RCT world headquarters.

First, our team finally published our paper describing our v1 platform at a high level - a mere 29 months after creating the first version’s Google Doc. The effort tied together years of not just software development and technical architecture but stakeholder engagement, privacy considerations, team building, and domain knowledge. Several co-authors were software developers who had never been on a paper before, were pretty new to the whole process, and hadn’t necessarily appreciated the “full stack” of the effort. It was fun to help them be part of the process not just of writing a paper but of creating a piece of the scientific record of humanity. Knowing they’ll be able to walk into many University libraries all over the world, for decades, and find a copy of it in the stacks, with their name on it, with authorship and citation records kept basically in perpetuity, is pretty cool.

Secondly, on a personal note, I spent some time at an arm HPC hackathon, which was both exciting (new tech! With many different systems to play with!) and surprising (Oracle’s cloud seems… pretty ok?). But more importantly it was really rewarding to see that after a probably eight year hiatus from day-to-day performance tuning of HPC codes, some of the names and tools may have changed, but a basic understanding of the tradeoffs at play and the techniques used to balance between those tradeoffs translate unscathed and can be put to use immediately. These are fundamental skills.

Hi, research computing team managers and leaders:

In our team there’s been a lot of passages lately - a paper for our original work is (finally!) coming out, as our new version is (finally!) coming together; we’re gearing up for a new batch of co-ops as our current co-ops are starting to document and getting ready to present their work; a project manager is joining the team for the first time now that the effort has reached a size and scope that it needs one (well, it needed one a year ago, but here we are).

These passages - and especially the influx of new people, new tasks, new scope - are really important for a team’s well being. Stasis isn’t stable; systems, including systems of people, are either growing or stagnating.

In academia sometimes it’s far too easy for groups to become very comfortable with “the way we do things”, and set in their ways. As Boulanger points out in the first article in the roundup, that can quickly lead to problems not being addressed - or even really noticed any more - and eventually people both within the team and “clients” of the team starting to drift away. In fact, I was talking to a colleague this week about one group’s services becoming ossified to the point where consumers of those services started moving to those of a different and newer group - the first group didn’t take feedback or feature requests seriously, and now there’s a real chance it will simply be disbanded (or, maybe worse, left go on indefinitely with less and less actual purpose).

Hi!

At least one other research group has also taken to providing some interview questions ahead of time. In response to the discussion in last issue, Titus Brown wrote in on twitter:

Hi!

So, two final aspects of our recent hiring situation that I haven’t had room to mention earlier: we’re giving out interview questions ahead of time, and interacting with peer teams is still hard in this hybrid/remote world.

The first of the two is likely more surprising and/or controversial. We’ve changed a bit how we’re hiring, including sending out some key interview questions ahead of time. This was initiated by a team member, interviewing co-op students. I was initially pretty skeptical, until I attended the resulting interviews; the discussions were so much better, and went so much deeper, that I wanted to keep trying it. Our initial attempts with interns went well enough that we’ve hired our first contract staff member using this approach too (and sent a pretty detailed ).

Hi!

I wanted to give you an update on the co-op hiring mini-fiasco from last week.

Even with the chaos, we still managed to get 3 of our top 4 candidates, because we still have a great team, a sensible set of interview questions, we contacted candidates (just!) before the interview, and followed up afterwards.

Let me tell you about a mini-fiasco this week that was entirely my own doing.

In our team, we routinely hire students for semester-long co-op positions. It happens three times a year - I think we’ve taken part 12 times over the past five years. It generally works out pretty well, for us and the co-op student.

The process is pretty uneventful generally. Our tireless administrative staff, without whom the place would fall apart, lets me know that it’s time again; we post our usual job ad; we interview some students and submit a ranked list. We’ve lately been pretty good at having projects ready for them on day one.

We had a couple more potential student supervisors with projects this semester, which is good. In the past year we’ve been upping our game at hiring full time staff, and part of that is better job ads; so we wrote a much better job ad for the co-op position this year and that resulted in fewer candidates but who were overall better matches for the team. Win-win!

Happy October!

There’s a couple of articles in the roundup this week on professional development and career paths for those in research computing & data - and one of them even emphasizes professional development for managers.

We have a long way to go, but there’s widespread recognition that our profession needs to grow, and slowly growing recognition about the resources that will take. Even, finally, for us managers or team leads.

The weird truth is that the research world greatly undervalues training - it’s just expected that everyone will learn on their own. But from what resources, for research computing and data management? And when are we supposed to take the time to learn? We still need some help.

Hi - I hope your week has gone well!

I’ve been prompted by a few things to think about activity versus accomplishing lately.

Falling into a trap of doing stuff, and so feeling busy and industrious, but not actually getting anything meaningful accomplished is certainly a trap that individual contributors can get ensnared in. We’ve all worked with someone (“or been someone”, he said sheepishly) prone to bike shedding or yak-shaving or some other flavour of spinning around in a rabbit hole without making forward progress. (Whenever I have to explain yak-shaving to someone, is the youtube video I point them to). It can like work to the IC, but to an external observer it’s pretty clear they’re stuck - they were tasked with doing the thing and they are very visibly not taking the direct route towards doing the thing.

Hi!

We’ve talked here in several issues about the “great resignation” that many companies are seeing as the pandemic starts to wane a bit. I think this is playing out differently in our world; there are definitely people leaving, or making new work arrangements, but it’s less en masse. Rather than having trouble with people leaving, instead, it seems to be more of an (even bigger) problem bringing people on.

On the there have been academic positions open since late winter, some that have been re-posted an uncomfortable number of times. That’s for manager positions; I’m also hearing from a lot of people that hiring ICs is getting almost impossibly difficult.

Research computing and data is increasingly a single discipline, a single area of practice. It’s a complex one, too; one which includes the support of first tentative steps which themselves may be research activities, all the way through to production operations of routine infrastructure.

You can read more, or go straight on to the roundup.

There was a on twitter (it happens) about software as a facility, as a tool for enabling research rather than merely as a research output itself.

Quick: what’s your team’s specialty?

Your team’s specialty is its reputation for what it’s good at. Not what you think your team is good at; what matters is what specific thing your stakeholders (funders, clients, institutional decision makers) think your specialty is. What they recommend you for to peers, what they recommend funding you for to decision makers.

In the post-pandemic world, researchers are used to getting their support remotely from anywhere. To compete, your team will need well-defined specialties; and “HPC” or “research software development” isn’t a specialty.

Hi, everyone:

I hope that as the start of the academic year approaches in the northern hemisphere, those of you who are going back to campus are comfortable doing so, and supported by administration.

Thanks to community members volunteering, we have a small core of 4-5 people who will start helping with resources for the community making suggestions to guide the newsletter. If you have suggestions, or want to take part, just hit reply or email me.

Thanks, everyone, for your responses last week.

I’ve been thinking a lot about strategy in other contexts lately - some of you will have noticed that I’ve been back on my nonsense on twitter, about the importance of having a focus. The very insightful comments and suggestions you sent last week about how we can help more research computing teams were very on point, and I think combine to a feasible strategy.

Because within this newsletter community we’ve built together, we have a number of strengths:

Hi, all:

So the last newsletter resonated quite a bit (welcome to new members of the Research Computing Teams community)! It was a bit of a cri de coeur about research computing’s inferiority complex that comes from unfairly comparing ourselves to both research and tech industry computing, and - to my mind - explains why we don’t advocate as well for ourselves as we could, support each other as well as we could, and why too often we don’t hold ourselves to high enough standards. It’s easy enough to see some of the results; poorly supported and run teams, not enough of institutional backing, demoralized team members, people leaving or checking out.

Research computing and data, as a community, has an inferiority complex. And it’s a problem.

This week I saw yet another press release - I won’t link to it, there’s no point in calling out this University or team in particular, it’s not like they’re sole offenders here - about how their new HPC cluster would “transform research”. Look — no it won’t. What would that even mean? Transform it into what? A biscuit? Rugby?

The infuriating thing here is that by all accounts, this was a good and useful procurement. Not only did it increase capacity (even if not to the “capacity knows no bounds” limit specified in the press release) but also the range of capabilities, with heterogenous nodes and a more flexible scheduler. That mean the existing researchers will have access to more resources, and it looks like it’ll be easier to provide support to a wider range of researchers, including those whose needs weren’t being met before.

Hi there:

We’re going into a long weekend here in Toronto - the second-to-last one of the summer - and it’s very much needed. We have a number of pretty ambitious efforts we’re working on, and it’s been a long year already. I hope that you and your team are taking care of yourself, and that you in particular as manager or team lead are taking some time off. There’s an article below on the importance - both for you and your team - of you taking some time off, to recharge yourself and give your team the opportunity to step up.

Also, there was some interest in the AWS ARM HPC hackathon that was in the roundup last week. I know that a number of readers are, like me, in the genomics space right now. Let me know if you think you or your team might be interested in participating in a similar week-long hackathon for ARM specifically around genomics; as always, just hit “reply” if you get this in your email, or email jonathan@researchcomputingteams.org if you want to talk about that or about anything that comes up in the newsletter.

And on to the roundup:

Hi, all:

Success as a manger is defined by a lot of hard work and tough conversations that will pay off over very long timescales. It’s way less immediately gratifying than deploying a new feature or making the CI/CD dashboard lights all green again.

Success or possible success for me this week week: I think I managed to convince some stakeholders to not make a bad and limiting data-related decision which would have limited the scientific effort of a four-year effort; and I’m coaching some team members to take on increasing planning and coordination responsibilities, with an eye towards gauging their interest and current ability towards being leads themselves, over a process which will likely take months. Small steps, tough conversations (there’ll be a lot of feedback conversations about effectiveness in the new responsibilities - many giving positive feedback, but not all), long term payoff.

Hi, everyone:

I hope you’re doing well.

I’ve neglected the “managing your own career” section lately, which I’m going to try to fix; we spend a lot of time here talking about helping our team members develop their skills, which is good and we certainly have an important role to play there, but we have to look after our own careers as well. Luckily in the past week several very relevant articles have crossed my browser, and so I present for you this week an attempt to bring that back into balance a bit.

Are there particular things you’re doing to track your own career progress, or get ready for future next steps? Are their particular gaps you’re not sure how to address or questions about how to progress? Please feel free to email me at jonathan@researchcomputingteams.org or just hit reply, and I’ll answer as best as I can and with your permission ask the newsletter readership to chime in, too.