Archive

This week I did something a bit different: I interviewed Matthew Smith, ARC Research Systems Manager at the University of British Columbia.

We had a great discussion, which touched on the self-doubt that comes with first becoming a manager of experts, working with other teams to have bigger impact than your team can manage by itself, transitioning to remote work as a new manager during the pandemic, the growing need for research platforms, the importance of campus-wide communities of practice, and the coaching mindset to leading a team.

My favourite quote came from the end:

Just love what you do, love the people that you support. I think all of us in research computing are in a unique position. It’s quite the niche that we’re involved in here, and I think we have a rare opportunity to contribute to what I certainly see as one of the key parts of society, which is research, new learnings, new innovation, and being able to support that is a rare opportunity.

Hi!

I keep hearing from leaders and members of teams where retention is a big problem. It starts small, with a single person leaving, and then a steady cascade of other people follow.

Worse still, there’s a worrying asymmetry between what I hear from the team members at these institutions, and the leaders and managers.

The managers seem convinced that industry’s salaries can’t be competed with, so there’s nothing that can be done.

Teams are having a hard time hiring - or even getting people to apply for their jobs. Whenever groups of research computing and data managers and leaders get together, this is a topic that comes up - I’ve heard it as a common refrain in the past year especially.

Unfortunately, the old approach of academic hiring - post a templated job description on a website and wait for the candidates to find the job and send in an application - doesn’t work any more. It doesn’t work anywhere particularly well, but it falls especially flat in our industry. You see this everywhere across academia; even what used to be highly competitive postdoc positions just aren’t being applied for.

We have to be more active in searching for job candidates today. That means a lot more work, but it’s upfront work which will make the hiring and onboarding earlier.

(We also have to be more active and put more work into retaining team members - I’ll talk about that next week)

Issue #140 was controversial, but not for the reasons that I thought it might be. In the last two weeks I’ve had now had several people flat-out deny this claim:

There are core facilities all over your institution which get much of their revenue from fee-for-service quite successfully.

saying, essentially, that such a tawdry arrangement would never be tolerated at their institution. In each case, googling “[Institution] core facilities” returned lists of teams that nonetheless did exactly that.

I’m not at all surprised that we still have so much to learn about other technical research support teams even within our own institutions. I’d like to change things, though.

In #139, we talked about some general purpose first steps when taking on a new responsibility. Let’s talk about the other side of that, now — how to make sure someone we’re bringing on to tackle some new challenges can be as productive as possible.

Back in #135, we talked about beginning with the end in mind: having clear goals for the person taking on a new role, what it should like in the day to day. When scoping out a new responsibility, it’s important to define what success looks like after they’re fully onboarded. That onboarding timeframe may vary: say after 3-4 months for an individual contributor, maybe after 6 months or even longer for someone with significant leadership duties. That doesn’t mean they won’t continue to gro after that! But by that point the goal is that they’re functioning competently in the role. Our job, then, is to help them get there as quickly as possible, for the sake of the team and so that they can start feeling settled in and successful. That means designing an onboarding process.

There are four kinds of information to gather to design the onboarding process:

• Make a list of all the knowledge skills and behaviours the person will have to learn to be at that successful end-state
• Make a list of the people that the person should meet and talk to to understand the inputs to and impact of the work they’re taking on
• Take an inventory of the resources (people, but also training materials or funding for training) you have available to help bring the person up to speed
• Identify meaningful self-contained projects, increasingly non-trivial, that they could work on that would give them concrete places to apply to what they’re learning, and that could be milestones to being an onboarded, independent team member.

This week I got a question about time-and-materials billing. So I want to talk a little bit about how consultants and other expert services firms increasingly see fees-for-service, and how we should see them to be of the most value to our community. I suspect this topic is going to be even more controversial than specialization is good and we should do more of it (#114), or making the case that we are vendors too (#123), or how too much of the framing around strategy discussions is self-important puffery (#130). But charging fees is an important topic if we want to sustain our teams and the research they support.

(By the way, the question came from a call from a member in our community, so a reminder that I’m happy to jump on a 30 minute advice or coaching call with anyone in our community who wants one - just pick a time).

Read on, or go straight to the roundup!

First, I want to emphasize that charging fees for offerings is not only ok, it’s good. I’d like to see more if not most of our teams have it as part of their revenue mix; how much of a part will depend on how they’re funded. We and our institutions talk about sustainability models quite a bit, but too often charging money for goods and services seems to be the sustainability model that daren’t speak its name. Which is odd! People are quite happy to exchange money for goods and services they value. There are core facilities all over your institution which get much of their revenue from fee-for-service quite successfully.

I got asked a question building on the last post about taking on a new project, and the current top unanswered poll.ly question is “I’m starting a new job! What should I do to ramp up quickly?”

Let me answer a slight generalization of those questions: say you have taken on an existing responsibility. It might be being promoted to manager, it might be a new job, it might being put in charge of some effort that already has some history. (And in our institutions, every effort already has some history. There are no completely de novo efforts in our line of work). What should you do?

(Read on, or go straight on to the roundup).

It’s mathematically possible, of course, that someone in our roles taking on a new responsibility has clear guidance from their boss, and is receiving routine and helpful support and coaching on on their efforts, so that they feel like they’re in good hands. Yes, yes, stop laughing, I said it was possible.

I got a question from a reader this week which I’m surprised to see that I haven’t addressed before, and that I’m sharing with permission:

I have several years experience [as an IC in research computing and data team]. With some changes coming up here in the next year or two there will be at least one manager job opening up. I’ve been reading your newsletter for six months, and it’s been really helpful! Do you have any suggestions for what I can do to be ready as a good candidate for the job?

Below is the advice I shared, and I’d welcome any other suggestions you have. Is there anything else this reader should be doing or thinking about? Let me know and I’ll pass it along, to them and (with your permission) to other readers!

Congratulations! Our community needs more new managers with enthusiasm for doing the job well. And I’d say that you paying enough attention to your team’s broader context to see this coming and be thinking about the consequences is already a good sign.

I get a lot of questions about both hiring and communicating expectations/performance management. I’m trying to compile and clean up some of what I’ve written on those topics (e.g. #65,66,67,68) in an ebook like I put together for one-on-ones at the beginning of the pandemic.

There’s two big topics left on the topic of communicating expectations; they both touch on the current top un-answered question on the pol.ly poll:

How can I get better at receiving feedback from my team?

A lot of what we see about expectations management and feedback is about managers providing feedback to individual team members. It’s an important topic! A hesitation to give routine feedback (frequent, mostly positive, always rooted in behaviour and impact) is the biggest gap I see among new managers of research computing teams. This is an vital “beginning as a manager” 101-level skill. It’s also one scandalously overlooked and under-appreciated in most of our institutions.

A housekeeping item before we get started -

I’ve had the searchable archives for back issues for a while, but there’s a limit to what a simple text search can do. I’ve wanted to have previous articles and write-ups browsable by category for some time now. There’s two time-consuming parts to that - going back and curating the best-of evergreen articles (right now mostly from the roundup), and then figuring out how to make them available.

I’ve done a first pass at the first piece, and now have a VERY crude initial attempt at this - so far what I think it’s taught me is that I’ll need some sort of hierarchy to the tagging of articles, which I need to think a bit about. What would be useful to you in a categorized “best-of” list? What would you like to see? Drop me a note - hit reply or email me at jonathan@researchcomputingteams.org.

Long time reader Adam DeConinck wrote in answering a question about Firecracker, a light-weight VM which is used for hosting tools like (as in last issue) CI/CD pipelines:

I love Firecracker, but so far there’s no support for PCIe pass through (eg GPUs, NICs, etc). Which I find really unfortunate […] because research computing in general is making heavy use of such devices, wherever they get them. I’d love to see a micro-VM approach in HPC at some point. Or really anything that can provide a better multi-tenancy story for shared research computing facilities in an increasingly security-conscious world. And as some Kubernetes devs have reminded me lately, the kernel (and containerization) are not a security barrier. […] Selfishly, I just want a VM platform that’s as easy to use in HPC as Singularity or Enroot.

This was new information to me - thanks!

I’ve been catching up on some reader feedback now that I’m back in the swing of newsletter things. Thank you all so much for your emails! I’ve learned a lot from our exchanges, and often shifted my position on things (or at the very least corrected how I talk about those things).

I especially appreciate pushback. Some areas I’ve really gotten pushback on in the past couple of weeks are what I’ve written about about utilities vs professional service firms (#127), or strategy and strategic planning (#125, #130). Discussion about those topics has helped me realize that I’ve never really been explicit about the audience I’m prioritizing for this newsletter.

In the past I’ve worked with VPRs and those that report to them, people putting together strategic plans for national communities, and those reporting to CIOs. People in those roles are doing important work, and deserve more support than they get!

But the most urgent need in our community, and the priority for me in this newsletter, isn’t supporting those that already regularly meet with CIOs or VPRs. It’s helping the first-level managers of individual small teams, and those who aspire to have more say in leadership in of those groups. Managers and leads and aspirants who were never given any training, are still given little to no direction or support, and yet find themselves accountable for a key research support function. People trained in academia’s years-long timescales, who are now having to figure out on their own how to run a team that has people depending on them weekly for firm deliverables. Leads who are figuring out how to do research project management when the research problem is still fuzzy. People managers still new to hiring, managing, and figuring out how to structure and strategize for a service organization. People now responsible for research software development groups, or data science service teams, or computing ops teams, or informatics-heavy core facilities, or even compute-and-data heavy spinouts from academic research groups.

Hi - I hope you’re enjoying your August!

One of the things I’d like to do now that I’m back from a couple of weeks off from the newsletter, is start being systematic about continually speaking with new research computing and data team managers, leads, PMs, etc - I’d like to talk to fifty or so and make sure I know the problems they have faced, are facing, and their successes. I’ll write up the aggregated, anonymized results results, and circulate it to participants first. I have five or six questions I’d like to ask them (and maybe you?) over a 20-30 minute zoom chat:

1. Tell me about how you started as a manager or lead, and the challenges you had at the beginning that you could have used help with.
2. Tell me about your team now - what they do, how they work, and who you report to.
3. What are some successes you’ve had with this team as a manager or lead?
4. What are some challenges you face right now, and what materials and resources are you using to help you navigate them?
5. What kinds of resources would help people in your position?
6. Who else (in particular or in general) would be good for me to talk to?

If that’s something you’d be up for, or you know someone else who’d be game, let me know - hit reply or email me at jonathan@researchcomputingteams.org!

Hi!

Last week we talked about ways to say no to incoming requests; besides a flat-out no, I mentioned the possibility of involving another team, or supporting the research group in doing the work themselves. RCT community member Adam DeConinck sent in another suggestion:

Occasionally, I’ve seen requests come in for work that the research computing team isn’t going to do; but which is also not something that another provider exists to do, and which the customer can’t do themselves even with training. When that happens, the “support” I’ve been able to offer is to join the customer in advocating for change. For example, aggregate similar requests and find commonalities; then organize the customers involved and advocate to upper management that they provide resources for this work to be done. That’s often not terribly satisfying, and I’ll admit that my success rate in doing this isn’t very high. (Non-zero, though!) But it’s often the only way I’ve found to create new classes of service within an existing organization.

This is a great suggestion, taking the long term view - and by lending your voice to the research groups’ in advocating for support for different kinds of work, it turns the situation into “your team and the research group vs the problem” rather than “your team disagreeing with the research group”.

Hi!

A question came in from a reader - I’ve talked before about how to not do things, and in my recent talk I said “do whatever researchers ask for” isn’t a reasonable strategy. We have finite resources, and we owe it to our communities and to science to focus those resources on the highest-impact things we can be doing. But what does that actually look like when you’ve been asked to do something well out of scope? Below’s a lightly edited version of what I sent back:

It’s important to pay attention to user needs, and thank them for their requests - if nothing else it’s useful data! But we can’t fall into the trap of trying to do everything a researcher asks for (or even every thing many researchers are asking for!). People want a lot of things, and our time and resources are finite. It’s perfectly ok for people to ask us for things, and it’s perfectly ok for us to say no.

Here’s a partial list of examples where it wouldn’t make sense for a team to provide a service that’s asked for:

Hi!

I want to write one more thing about strategy this week before going back into the basics of managing our research computing teams.

It turns out I’m still a bit discouraged by one section of the recent (very good!) PEARC21 report on strategic planning for RCD groups. That part described all too clearly the confusion and helplessness many team leaders of teams like ours feel when asked to do some positioning and strategy work. Like so many things, we’re thrust into responsibility for these kind of activities without ever being taught anything about them. It’s intimidating. But it shouldn’t be, and doesn’t have to be, that way.

The thing is, finding a path forward for a team or organization is a research and problem-solving exercise.

Hi!

I write a lot here about challenges our teams face, but I don’t think I mention often enough the huge advantage our background in science is in our work (or in similar work in other environments).

Whether we were trained in science or just picked up that mindset by immersion in the ecosystem, the skills we have are skills that (say) tech companies pay real money to train their staff in. The advanced collaboration and influencing skills we have to develop in scientific collaborations is a huge advantage all the way up the management or technical leadership ranks, for instance. Being willing to think in terms of experiments when making changes (like Beinbold’s article, below), or having a “growth mindset” (current jargon for believing that people can develop new skills(!!)), or being able to synthesize information - these fundamental skills, skills we don’t even bother mentioning in our environment, are in high demand.

It’s not just people skills, either - I see an article about staff engineer interview skills that include being able to estimate well, something every quantitative scientist does without thinking about it, or another article this week about how tech companies are having to relearn basic performance things like array-of-structs vs struct-of-arrays, which many of us have had to learn optimizing scientific code.

I heard a number of answers back about what sorts of service request processes people were using:

• A lot of software development teams, of course, took bug reports and small feature enhancement request via GitHub/GitLab/Jira etc ticketing systems
• Several groups are using Znuny, a fork of OTRS
• Someone juggling multiple projects used a timetracker to make sure reasonable amounts of time were being spent on various external projects
• A technical leader of an internal platform team said that it’s mostly “’support channel in Slack’ With complex issues moving to email” (but they had succeeded in at least getting the requests into the channel rather than DMs!)

One data science team had a longer answer that I want to quote here:

For a relatively small team of a few staff and about 10 students taking about 250 consultation requests a year one on one help on data science topics, not help tickets for a service), we use […] – basically a form + spreadsheet (google forms/sheet would work too). We have columns to track the status (incoming request, in progress, completed, etc.) and who has responded/been assigned, along with the info about the request. We don’t put other significant updates on the consults in this sheet – just for tracking status. We make sure our team brings the status of everything up to date at team meetings. Requests ping a slack channel when they come in, and we determine who will respond. So we can make sure that someone took each consult, even if the sheet doesn’t get updated immediately (it should be, but it doesn’t always happen).

For longer projects, we have only 6-12 or so per year – those go on a separate sheet for recordkeeping/reporting. All updates are communicated via team meetings or 1:1 meetings.

Before we get started, please answer if you can a question from a reader: What do you use to get and keep track requests for your work from researchers? Is there a ticket-tracker type piece of software that works for you, or do you use something else? Does it fit your workflow - What do you like, not like, and wish for?

I’ll summarize results next week - it’s an important topic!

One of the great things about this community is the different kinds of research computing and data teams I get to talk with, “here” or outside the inbox. Sometimes in these conversations it’s pretty clear that some teams, or the team and their funder, or a team and I, are talking a bit past each other. And that’s usually because they or we are (currently) operating with very different mental models.

Hi!

I write a lot here on the challenges that research computing and data teams face, and the challenges that their managers and leaders face in particular. And that’s because I want them to have fewer challenges! I hope that by highlighting the challenges they at least don’t face them unawares, and that as they grow their skills they bring along the next generation of managers and leaders under clearer, less stressfull, and more suppported environments than we found ourselves in.

Which is all well and good, but it can get a little gloomy in here sometimes. I don’t think I spend enough time highlighting successes.

Do you have management or leadership wins you want to share, no matter how small? Just with me, or with the community (anonymously or otherwise)? Let me know - just hit reply and send me an email at jonathan@resarchcomputingteams.org. I think it would be good for everyone here to see what some of those wins look like.

Hi, all!

I want to write a little bit more in the coming year about strategy for leaders of research computing and data teams: setting priorities; deciding what and what not to do; and deciding what success looks like. It’s an important topic - and yet as with so many things in our line of work, no one teaches us how to do it. It’s also much more ambiguous in our discipline than it might have been a researcher, where you can keep score pretty simply with papers and grants.

Those of us who have gone through University strategic planning processes come out even more confused, because these “Strategic Plan” documents aren’t a great introduction to strategy at the best of times, and they’re particularly odd ducks in the context of a University.

There was a great article by Alex Usher and team at Higher Education Strategy Associates last week. I want to use it to set the stage for what a strategy is, what a good overall framework does for teams or orgs, why Strategic Plans for universities are so constrained, and what that means for us. Usher has written quite a bit about strategic plans in the past, some of it quite cutting, and all of it insightful - I recommend the articles, and the blog in general, if this topic interests you.

Hi!

I wanted to write more this week about the consequences of research computing teams as vendors, and what that means for strategy - I’ll do that next week.

But there was big news in HPC world in this past week - as you almost certainly already know, the Frontier super at ORNL is now the world’s first Exascale system as measured by the HPL benchmark.

This is an enormous technical (and project and product management!) accomplishment, the capstone of work that’s been ongoing since 2006 or so. Colleagues at AMD (which, disclaimer, has long been a worthy competitor of my current employer), HPE, and ORNL deserve all the praise they’ll rightly get. My sympathies to other colleagues at Argonne whose star-crossed Aurora system, facing a series of delays, didn’t make it across the line in time.

Hi!

My new job, working for a large company that explicitly sells stuff, has indeed been eye-opening — but not in the ways I expected. Mainly I’m confronted with clarity about some of my old jobs, and peer groups I’ve worked with.

I’ve always been sort of puzzled by the decisions made and priorities chosen by some research computing and data teams. Those teams seemed more… insular, somehow, than others. I saw it less often in contract research software development teams, or library research data management teams, or bioinformatics core facilities. I saw it more often in research systems teams — not all, of course, and not only.

Meeting with some of these old peer groups, wearing a vendor hat, and watching them interact with us, it becomes a lot clearer.

Hi!

I took part in a hackathon this week, working directly with researchers on improving their code for the first time in — gosh, maybe eight years? Nine?

It’s funny what sticks in the mind and what doesn’t. I kept coding on and off over the past decade, so that was fine, but all the glue stuff that I didn’t think to stay in practice on like “how do you unload all modules?” or “why doesn’t this shell script that sets environment variables work when I run it?” (yeah, that one’s kind of embarrassing) took some time to remember.

There were more data analysis projects, and infinitely more AI/ML projects, then there would have been a decade ago when it would have been 90% simulation codes. Technology has changed over the past decade — there are some new profilers, and tooling, systems are faster, and hey Power systems aren’t big endian any more! But I was mainly struck by how little change there had been in the basic challenges. Maybe the percentage of research groups with extensive in-house research computing and data skills has gone up a bit, but mostly the research groups we saw were very capable, deeply knowledgeable in their domain, with significantly less expertise on the computing, software, and data management side. They needed, and knew they needed, collaborators to work with to achieve their goals. And our responsibility was to do a little bit of hands-on-keyboard work, but mainly to give them the skills and resources they needed so they could continue to advance even after the engagement; to give guidance as trusted experts. And that expertise ends up straddling at least two of software, systems, and data - those aren’t useful in isolation.

Hi!

The (159-item!) intern process checklist from last week generated a few interesting discussions, and those plus Chris Dwan’s interesting talk discussed below return to a common theme here in the newsletter - of projects versus processes, or learning versus production.

In #91 I alluded to the fact that what distinguishes a hobbyist from a professional is repeatability. The hobbyist is still developing their skill, trying to make their one-off thing come out with high quality. The professional is no longer struggling to make one thing well; they’re trying to ensure that things reliably, repeatedly come out with the same high quality.

The hobbyist is thinking about the mean quality, and trying to get it to trend upward; the professional is concerned with the variance, and reducing outliers. (Mean matters, too, but for a professional, a high mean quality is table stakes). There’s a progression from focussing on skill to focussing on reliability, from craft to process. One of Dwan’s slides says:

A reader wrote in to ask:

We have a new summer intern starting in a few weeks who I will be directly supervising. Although I have been the technical/project lead on various projects throughout my career, this is my first time having a direct report (even though it will be temporary!). I was wondering if you had any “go to” or favorite literature for managing in a technical setting.

Congratulations to them, and this is a great question. It’s such a great question that I’m disappointed to discover that I didn’t have handy resources to immediately share with them, and I’d love to hear what your recommendations are.

In research computing, probably the two most common ways people find themselves managing is to be promoted to lead their existing team, or the more gradual approach which is to start with one or more interns/co-ops/grad students/trainees/etc.

Hi!

I wanted to talk a little bit more about Research Computing and Data (RCD), how it connects to other work, and how that informs how I think about when RCD work is and isn’t a research output.

First, I want to make a claim that most readers here would likely agree with, but may not yet be universally accepted elsewhere: RCD is a discipline in and of itself. It is a body of knowledge and practice, with full time practitioners, conferences, knowledge creation, sharing, and — increasingly — degree programs.

In fact, I’d make the argument that RCD been one of the most productive and influential academic disciplines of the last several decades. And an academic discipline it is - I don’t think anyone would doubt that the development of the FFT, or finite element methods, or new methods for processing next generation sequencing data or supporting digital humanities work, are real and significant research and intellectual accomplishments.

Hi!

My discussion last week about when research computing and data work is and isn’t a research output in and of itself produced some replies, as the topic often does, that ranged from intrigued to puzzled to grumpy. I’ll write some more next week on this (I even have a diagram sketched out), so people can have some really specific things to be intrigued or puzzled or grumpy with, but I first want to talk about the multi-dimensional aspect of research computing and data efforts.

Here’s the thing - all research computing and data work is advanced research computing.

It might not feel like that. Reading about the cool cutting edge tech things that are being done in industry, or comparing to the huge HPC systems at the DOE labs or at PRACE, it’s easy to feel a bit in a backwater. (I’ve written about this in the past).

Hi!

I’ve long tried to describe the sudden disorientation of losing immediate feedback that comes with being a first-time manager. Recently I heard the term “managerial sensory deprivation”, and I think that’s a useful phrase. When you’re hands-on doing the work of reearch computing and data, you get immediate, almost tactile feedback with what you’re doing, whether it’s from the systems or from the researchers. As a manager, any action you take has only indirect results that could take days, weeks, or months to really play out. And by that time, it’s hard to connect it unambigiously to something you’ve done.

Your work senses are getting no signal back, or weak signals seemingly unconnected to what you’re currently doing. You’re feeling a kind of sensory deprivation. And wikipedia helpfully reminds us that “extended or forced sensory deprivation can result in extreme anxiety, hallucinations, bizarre thoughts, temporary senselessness, and depression”.

This has come to mind lately being a new part of a big organization and, frankly, having no idea what I’m doing yet. I’m accomplishing stuff, but is it the right stuff? Is it having any kind of impact? If yes, is that impact positive? How and when will I know? Having moved organizations (and sectors!) I’ve again lost my bearings and my feedback mechanism. It’s apparently not an unusual situation; I heard the same thing from a reader who I talked with earlier this week, who’s also in a new job. (By the way, I love hearing from RCT readers - never hesitate to email, or even book a quick chat.)

“Contact us for pricing”.

Say we’re looking at a potential new tool, for us or the team as a whole. When we want to learn about whether it’s feasible as a solution to our needs, we see “Contact us for pricing”. Or occasionally, even for information about the product itself, “Contact us for details.”

“Fantastic!”, we think. “After a few emails and a call or two, not only will we get the information we want, maybe we’ll make a new friend!”

We of course do not think this, because no human being has ever thought this. More likely, we reasonably view it as an imposition on our time, an artificial barrier to getting the information we want. Most of the time we likely bounce, never looking at the page again. Sometimes, we may reluctantly agree to the process, and trepidatiously fill out the form. Perhaps we use a burner or obscured email address, perhaps we don’t. Either way we kind of resent it.

Hi!

Well, it finally happened. The time came for something I’ve been dreading since taking a job with a vendor; something which held me back from taking such a job earlier.

I had my first meetings with former colleagues from the vendor side of the table.

And… it was fine! It wasn’t weird at all.

Hi!

It’s been pointed out I have my view of the role of specialization in research computing teams seems a bit contradictory. I am very pro-specialization, but anti-siloing. Fair point! Let’s dig more into that.

No research computing and data team can do everything. It’s not possible. Pretending otherwise leads to unhappy team members and unsatisfied researchers. And as I like to point out, there’s already tonnes of computing things our teams don’t do for researchers. We don’t help them with their printers, or lay out designs of posters for conferences. So my advice is wherever possible — and it’s more possible than you think — is to add a few things to that list. To niche down into a specialization.

The most meaningful specialization is of the form of “we help [community] with [problem]”. One very effective way to do that is where community is a particular discipline (defined broadly). “We help wet lab molecular biologists with their bioinformatics analyses”. “We help social sciences researchers with their GIS software development needs”. “We help researchers with long term data retention needs affordably and reliably store and share their data”. “We help health scientists write and deploy desktop and mobile data collection applications”. “We help researchers new to quantitative methods with data science”. “We help researchers with web applications deploy, operate and maintain their application stacks”.

Hi!

We now have 200 members of our community; people who care about research computing and data teams, their potential, and the importance of leading them professionally.

So, welcome new readers! Some resources that might be useful:

• The Rands Leadership Slack (a community of 20,000 managers and leaders in or around tech) and our new and so far small (16 people) #research-computing-and-data channel there
• My “Help, I’m a Research Software Manager!” talk & slides from 2020 which covers in 10 minutes the basic approach I take in the newsletter and elsewhere.

Hi!

My current job, like most of my previous jobs, include a lot of people who have worked for a long time in research — either as researchers themselves or in other roles, with Ph.D.s, or without.

I sometimes write about how working in research teaches us behaviours that are unhelpful when we get to management. That’s certainly worth calling out, but it’s only part of the story. Some of the behaviours and habits of mind we learn are extremely helpful, and we don’t even notice it.

In business management writing, you’ll see lots of discussion of “growth mindset” and “comfort with uncertainty”. These are real gaps they see in new hires and especially new managers. And yet it sounds like meaningless fluff to us, because we can’t even imagine it.

Hi!

I’ve been thinking of jobs a lot lately, of course, for my own reasons and just through the effort of curating the research computing teams leadership job board. (The board has gotten a number of community submissions lately - keep them coming! I just check them for spam before accepting them).

A year and a half ago I posted my observations on the first 500 jobs posted to the board - we’re getting close to 1,500 now, and it’s worth taking a look to see what if anything has changed in research computing team leadership jobs.

There are some trends that have continued since the posting. The jobs in industry are growing vastly beyond what I would have imagined possible when I started in research computing in the 1990s. Rather than technical computing being a niche, it’s utterly mainstream now. There are a lot of jobs out there, and I don’t even bother posting generic “data science manager” jobs unless they’re connected to some real complex research questions - which happens a lot, whether it’s fraud detection or improving financial modelling or whether it’s supporting biomedical research. Some really fun-looking jobs that would probably feel a lot like working at a research computing centre keep coming up at consultancies – go visit a client and help them with their data science/data engineering/etc needs. There’s also a growing number of data science/engineering jobs at Universities that fall under the Provost/VP Operations rather than the VPR’s side of the house - Institutional Research, looking at (say) student success in support of the teaching mission.

Hi!

I won’t lie; when I started offboarding myself from my current job a few weeks ago I was a little anxious. I was pretty sure that team members would step up and learn quickly. It felt relatively unlikely that we’d discover gaps that could threaten near-term milestones. One worries nonetheless.

Instead, I’m really impressed with how well everything has gone. If anything the team has come together even closer, rallying to fill such holes as my departing creates. There’ll certainly need to be another hire - the team is losing a person’s worth of effort, after all. But the big picture work, both technical and stakeholder facing, is in extremely capable hands. I’ve felt increasingly superfluous this past week. That’s the best possible outcome.

But that also means that I’ve been surrounded all month with examples of things I could (should!) have been delegating and documenting earlier. Activities I thought only I could perform are in fact somehow getting performed. Things I could have usefully have documented ages ago, are only now finally getting written up. It’s not a disaster, nobody died of it. And yet there were real missed opportunities for team member growth and for me to have spent more time focussing on strategic goals.

Hi!

A slightly short newsletter today, as I finally claw my way back onto the Friday schedule.

One community update from last week - there was a fair amount of interest in using the Rands Leadership Slack as a pre-existing place with a large (>20,000!) leadership and management community as a place to discuss topics of interest. There is now a #research-computing-and-data channel, with sixteen members so far - some newsletter readers, but also some who had already been on rands and quickly joined once the channel was created. If that sounds potentially of interest to you, we’d love to see you there!

Otherwise this week I’ve mainly been continuing to offboarding myself from my current job. The coming week is my last (and I think the final 1:1s are going to be tougher than I was anticipating). Most of the knowledge transfer has now been done — although to wrap it up, next week I’m giving a demo of a key piece of technology to the team. Manager does a demo! What could possibly go wrong? This week there’s been a lot of 1:1:1s, with a team member and either I and my replacement, or the replacement and our boss, about transfer of responsibilities or “what’s next” conversations.

Hi, everyone:

Several readers responded about their version of the challenges I described last week - most teams are still working remote; some had reduced-occupancy office space available for those who wanted to come in, although it sounds like during the Omicron wave that was sparsely taken up where allowed at all. And those who mentioned it agreed uniformly on how hard it was to hire these days.

I think in some places we need to come up with our own solutions to these problems that meet the challenges our teams face (small, relatively slowly-growing teams with wide scope and demands that can change month-to-month and weird funding), and in other places there are opportunities to learn from other kinds of groups. This newsletter is a great medium for sharing resources, but it’s a bit of a slow way to have a back-and-forth on topics of timely interest! I’ve mentioned it before and included it on lists of resources, but the Rands Leadership Slack is a great community for technical managers and leaders, largely from tech and startups; if that’s something that interests you, dear reader, let me know you’ve joined or are considering joining and we can create a research computing and data channel for discussion of our particular needs.

In the meantime, news on my front. I’ve been thinking about people leaving jobs a lot lately, because we just found out one of my team members is leaving, leaving a big hole in the team - but also I’ll be changing jobs, too. (More about this in a bit). So we’ve been going through two off-boarding processes at the same time, one (mine) much lengthier than the other, involving two somewhat different groups of people, and requiring knowledge transfer at different levels of granularity, which is an interesting perspective into the situation.

I’ve had great conversations with research computing and data team managers this week. And while the the discussions were great, they weren’t all sunshine and light. RCT leads are facing some common struggles right now. We’re all wrestling with two big challenges: industry hiring like mad; and the likely end of a pandemic wave.

Industry’s sustained hiring spree means we all have to fight to hire and retain team members. Anyone with strong computational skills and a LinkedIn profile has to fend off requests for conversations about attractive positions. To hire and retain we need strong career ladders, which is something we can work on together. We need to find ways to build on the strengths of our teams and our work places, which is something where we can benefit from can sharing ideas.

And as another pandemic wave looks to subside, we face the opportunity and challenge of more in-office work. As was pretty clear even in 2020, “everything resets back to 2019” isn’t going to happen for most of our teams. On one hand, pure remote is out of reach for many. It may violate institutional or even funding requirements; and it doesn’t take full advantage of the real strength of being embedded in a research ecosystem. On the other hand, being expected to commute in every day like clockwork is going to rankle many team members, and the opportunity to hire from a broader geographic catchment area is too valuable for a lot of teams to forgo entirely.

So many of us are wrestling with what hybrid work arrangements will look like. Hybrid is going to be much harder than in-person or 100%-remote teams, each of which has pretty clear expectations. There’s mostly just one set of best practices for working on-site, and one set of emerging practices for 100% remote. Hybrid will be different. A whole spectrum of arrangements are “hybrid”; there’ll end up being a few clusters of good practices depending on what teams decide to aim for. Finding and disseminate what works for our very client-driven, interdisciplinary work and teams will take a lot of knowledge-sharing

Hi!

As you can see from this issue again being late (but less late!), here at RCT HQ I’m still getting into the rhythm of the new year. I hope you and your team are already firing on all cylinders.

There’s several articles this issue on decision making, priorities, and strategy. I keep coming back to this, because it’s important.

For research computing teams to succeed, we need priorities and a strategy. Without those, we can’t even tell if we’re succeeding, or on our way towards success.

Hi everyone - Happy New Year!

Here at RCT HQ, we’re slowly getting back into the swing of things after a long and relaxing holiday break - I hope it was relaxing and refreshing for you, your close ones, and your teams as well.

Since last we spoke, the big RCT-relevant discussions online have been about the aftermath of the Log4j fiasco, which was so big and had such large impact that in the US, the Federal Trade Commission as well as the national security apparatus have gotten involved.

Leaving the security aspects aside, the fundamental issue of scandalously under-resourcing the maintenance of underpinning open-source software is one which has been brought to the forefront, with even policy makers now well aware of the problem. Useful software will get used, often in unpredictable ways - that link takes you to a Google security blog post describing how 8% of Maven Central packages use log4j, but only 1/5 of that is of direct usage, the rest being indirect use by a dependency of the package, or a dependency-of-a-dependency-of-a-dependency. But there’s no robust mechanism for making sure the foundational work that underpins the flourishing ecosystem is supported. That, coupled with the high-profile bricking of faker.js and colors.js, have made the questions of what the broader community owes to foundational software maintenance, and what (if anything!) foundational software maintainers owe to the community, a central topic.

Hi!

There seems to be something in the air - in the past week I’ve had or heard a number of conversations around leadership and managmement.

Leadership vs management is a fun conversation to have in the same way that “is a hotdog a sandwich” is a fun conversation; it can clarify your own thinking but is unlikely to change too many minds.

Still, I think the basic outlines are clear and well agreed on. Management is a specific set of skills and activities particularly useful in particular jobs — people management especially, but also product or project management. It’s a woefully under-appreciated set of skills and behaviours, especially in academia-adjacent areas, and that means that you you can get significantly better at it than most peers pretty easily. By caring enough about it to subscribe to a newsletter, gentle reader, you’re already comfortably in the top 50%.

Hi!

It’s a busy time of year, as we try to get things closed out before the holidays. For a lot of us, outlines of opportunities and connections for the next year are starting to come into view; if that’s the case for you, I hope you like what you are seeing!

For us, the familar challenges of negotiating multi-institutional (and multi-sectoral!) collaborations are starting to yield to steady effort, and the exciting work we want to do will be beginning in the new year. It’s an exciting time, but means the newsletter is late again this week.

Next issue will be the last one for 2021; that will give us all a little bit of a breather to rest and recharge, and start again fresh in 2022.

Hi!

If you follow HPC twitter at all, you will have seen a heartfelt thread by a well-known research software developer, one who was a key contributor to the Singularity project among others, lamenting the frankly appalling state of developer productivity in HPC - both in what tools exist, and support for them (and other tools for developers) at academic centres. A lot of people chimed into the discussion, including one of the leading developers of the PetSC project, embedded software developers, some key people at big computing centres, all agreeing that there was a problem, but typically zooming in on one or another particular technical or procedural issue and not coming to any conclusion.

I think the issue is a lot bigger than HPC software development - it comes up in too many contexts to be about specific technical issues of running CI/CD pipelines on fixed infrastructure. The only people to identify the correct underlying issue, in my opinion, were people from the private sector, such as Brendan Bouffler from AWS:

Too much reliance on ‘free’ labour - postgrads and postdocs, who, invariably, decide that burning their time being mechanical turks for their ‘superiors’ just sucks, so they come and work for us. And since we pay $, we’re not gonna waste them on things software can do.

Hi!

A reader from a U15 research institution [Canadian research-intensive universities; think Russell Group in the UK, or R1 universities in the US] writes in to describe expanding their research computing team with unconventional roles:

First, a grant advisor, specifically to assist PIs in writing sane tech inclusions in their grants. You may have reviewed grant proposals where the medical science, particle physics, quantum chemistry, etc. is very clear, and then the explanation of the computational aspects and the equipment justification sounds like Dilbert’s boss wrote it. That is precisely what this position is intended to improve, but also sitting on internal panels that judge Innovation Fund proposal maturities before they’re allowed to apply, etc. Second, a Communities of Interest Coordinator, who will foster and support research communities of like-minded graduate students, PDFs, etc. around research fields making use of computation—bioinformatics, AI, digital humanities—or around digital research tools—R, Julia, MATLAB, Gaussian, etc. By supporting communities of interest, these groups can become shared knowledge hubs, where newbies can find guidance or “the ropes” and experienced but stuck researchers can find inspiration or “an ear” that might enable them to unstick. Both positions have been filled internally and start in December. More traditional ARC job descriptions are being written up now as part of a further expansion.

I love this! It’s a long-standing tenet of this newsletter that research computing is much more than just technology. It’s teams, it’s communities, it’s product management - it’s people. Connecting researchers and their more directly to the computation, software, and data resources that can advance their work — whether that means in grant writing or capacity building within a practitioner community — is very much part of the our broad remit in research computing and data.

Hi, everyone:

It’s been a good week here at RCT world headquarters.

First, our team finally published our paper describing our v1 platform at a high level - a mere 29 months after creating the first version’s Google Doc. The effort tied together years of not just software development and technical architecture but stakeholder engagement, privacy considerations, team building, and domain knowledge. Several co-authors were software developers who had never been on a paper before, were pretty new to the whole process, and hadn’t necessarily appreciated the “full stack” of the effort. It was fun to help them be part of the process not just of writing a paper but of creating a piece of the scientific record of humanity. Knowing they’ll be able to walk into many University libraries all over the world, for decades, and find a copy of it in the stacks, with their name on it, with authorship and citation records kept basically in perpetuity, is pretty cool.

Secondly, on a personal note, I spent some time at an arm HPC hackathon, which was both exciting (new tech! With many different systems to play with!) and surprising (Oracle’s cloud seems… pretty ok?). But more importantly it was really rewarding to see that after a probably eight year hiatus from day-to-day performance tuning of HPC codes, some of the names and tools may have changed, but a basic understanding of the tradeoffs at play and the techniques used to balance between those tradeoffs translate unscathed and can be put to use immediately. These are fundamental skills.

Hi, research computing team managers and leaders:

In our team there’s been a lot of passages lately - a paper for our original work is (finally!) coming out, as our new version is (finally!) coming together; we’re gearing up for a new batch of co-ops as our current co-ops are starting to document and getting ready to present their work; a project manager is joining the team for the first time now that the effort has reached a size and scope that it needs one (well, it needed one a year ago, but here we are).

These passages - and especially the influx of new people, new tasks, new scope - are really important for a team’s well being. Stasis isn’t stable; systems, including systems of people, are either growing or stagnating.

In academia sometimes it’s far too easy for groups to become very comfortable with “the way we do things”, and set in their ways. As Boulanger points out in the first article in the roundup, that can quickly lead to problems not being addressed - or even really noticed any more - and eventually people both within the team and “clients” of the team starting to drift away. In fact, I was talking to a colleague this week about one group’s services becoming ossified to the point where consumers of those services started moving to those of a different and newer group - the first group didn’t take feedback or feature requests seriously, and now there’s a real chance it will simply be disbanded (or, maybe worse, left go on indefinitely with less and less actual purpose).

Hi!

At least one other research group has also taken to providing some interview questions ahead of time. In response to the discussion in last issue, Titus Brown wrote in on twitter: