Archive

Hi, everyone!

Sorry for missing the last couple of weeks - after a particularly busy time, we ended up making some last-minute plans to enjoy the holiday weekend last weekend here in Canada and I’m just now catching up.

If one of the last couple of weekends was a long weekend for you, I hope you enjoyed it!

Why I Don’t Like “Strategy”, Part I: No Such Single Thing

After over three years writing the newsletter and talking with RCD managers more frequently, the one thing I’ve changed my mind most about has been strategy and strategic planning.

It’s not that I now believe thinking about the big picture is bad - quite the opposite! I just think the dialog around “strategy” and “strategic planning” in our communities is is both badly confusing, and a distraction from what we actually need to be doing. (For what it’s worth I mostly blame University/Research Institute strategic planning processes for this, along with a number of really unhelpful books and online materials).

The problem with it being confusing is that overworked new or front-line managers don’t know what they should be doing except that they’re assured that this nebulous “strategy” thing maters. Since they don’t really know what to do about it, and none of the materials on the topic have any kind of internal consistency, they just sort of find other group’s strategy documents, copy the format, put in some words relevant to their organization, and hope for the best. All the while they’re stressed about it, and have a sinking feeling that they didn’t do “strategy” right.

Last week we described the sad story of our fictional colleague Daniel Miller. Miller, who runs a Research Data Science team at an equally fictional university, is faced with the sudden departure in two weeks of his most experienced team member, Olivia Johnson, who has been acting as an unofficial team lead. Miller was not aware that Johnson was unhappy or looking for other opportunities. He now needs to urgently address several concerns, such as replacing Johnson, managing ongoing and upcoming projects, training events, and team members, and prioritizing Johnson's tasks in the meantime, and has asked us to advise him.

I got a number of people writing in with their advice, but people were hesitant to share their thoughts to the readership! Fair enough - this is the first one. So I’ll give my recommendations, in the form of an email sent to our colleague, and highlight some things I saw in common from those that replied.

Miller has a number of urgent and important priorities to juggle. So let’s tackle these in the order that we recommend handling them. I’ll start the email to him, shout out if you see something I should change or add:

Here’s the first in what will be a recurring series of case study/tabletop exercise: the team member a manager most relies on announces they’re leaving.

In this series, I’m going to propose situations similar to some I’ve seen (heavily reworked and changed), and ask you for your input: What would you do/have you done? What other things should be considered? And, after the situation plays out, what would you suggest the manager or leader do in the future to make things better?

Daniel Miller runs a central Research Data Science team at the University of West Northshire (all names and places entirely fictional, situation only slightly less so). After a long while of having trouble getting potential clients to have even heard about them, much less engage and bring their problems, they forged collaborations with the stats help desk and the HPC centre to mutually refer promising projects, and now things are getting pleasantly busy. The team hadn’t had to hire past its initial cohort from a couple of years ago, but is now considering growing with at least some student workers. Currently, Miller spends most of his time building and maintaining relationships with stakeholders, hoping to secure additional funding.

I’m going to close out this discussion of focussing on the ends (outcomes and impact) and not the means, and measuring what matters, with a point that I would hesitate to make in a public talk.

One of the nice things about writing a weekly 3000+ word newsletter on research computing and data teams is the selection effect. With you, reader, I can raise some uncomfortable topics. The fact that you take a significant chunk of time out of your week to read and think about managing and leading research computing and data teams well, that you agree such a thing is important, that it’s worth considering our role more deeply, already means that the RCT newsletter community here isn’t full of typical teams.

So here’s the thing. We’re in funny sorts of roles.

The people we report to in our institutions or regions mostly have a pretty fuzzy understanding of what impact our team has now, much less what impact it could have.

The Logic of Collaboration For Nonprofits Like Us

In this third issue in the “measure what matters” series, I want to continue on the topic of planning and evaluating programs based on impact, not activity.

Issue #162 covered the idea with Kirkpatrick evaluation model for training. Last week (#163) we talked about logic models, which are simple but powerful tools that help organize our thoughts, discussions, planning, and evaluation in terms of outcomes and how we get there, not just on the activities. They help clarify assumptions, align resources and actions with desired results, and (like the Kirkpatrick model) give a framework in which to track and demonstrate your progress and achievements.

Maybe most importantly, they can be used to usefully redirect discussions with those stakeholders or decision makers who otherwise have an unhelpful “underpants gnomes” mental model of our work (for example, Phase 1: Run a computer! Phase 2: ??? Phase 3: Science!)

Last week (#162) I spent a lot of time on training evaluation and the Kirkpatrick model, but the underlying idea has much wider application.

We chose training interventions — or other services or products for our community — because, consciously or unconsciously, we have some mental model for how this work will impact research and scholarship in our organization. There’s typically some pretty simple mechanism that we’ve hypothesized. That mechanism is observable, even if qualitatively, and from those observations we can improve the intervention’s impact.

Tools Like Logic Models Describe How Our Activities Drive Research Impact

The Kirkpatrick model can be thought of as evaluating along different stages of some logic model:

I’ve had some great conversations following up from the surveys article (#159) - thanks! You’ll likely remember that my argument was that while surveys can, used carefully, be a useful instrument, there are some common failure modes in our line of work:

• Surveys are too often a way to avoid talking to people
• It’s too easy to ask questions in a survey without putting in the effort to decide what really matters, and what decisions will be make based on the results. So the results don’t mean anything and nothing changes, but it still feels like something’s been done.

Training came up a lot as an example in the conversations. That’s a great application area, because training has a well-developed practice around evaluation. I’d like to use that as a baseline way to talk in the next issue about measuring what matters, and outcomes and impact versus inputs.

So let’s talk about the Kirkpatrick Evaluation Model for training interventions:

I feel strongly about the RCT job board — operationally, one of the things that defines a profession is some common places to look for new career opportunities. It indicates that there is widespread recognition of and demand for the skills and expertise of the community. It demonstrates that the community has grown and diversified enough to offer a variety of career paths and options for its members, and that the community supports their professional development and mobility. It’s important for the specialized and important community of practice we’re trying to build here together, so please keep submitting those jobs!

But another terrific use of the job board is just as a source of data, tracking the needs of the institutions who require our services. In maintaining the job board over the last three-ish years, I’ve already seen some changes. And going through the postings recently, there are some pretty clear shifts.

Some won’t be that surprising to… well, to anyone. There are a tonne of quantum computing lead or manger jobs, something that barely even registered as an option three years ago. There are a lot of jobs for very applied AI teams, whereas when I started they were dominated by exploratory research groups. (And when I started this, “machine learning” was declining as a term used, in favour of “data science”; “machine learning” has already caught back up.) The demand for bioinformatics manager roles, which skyrocketed during the early stages of the pandemic, has somehow stayed steady.

Academic job postings are still staying up forever before getting filled, but I notice amongst those jobs a richer and more challenging set of roles - increasingly, campus-wide jobs, often with multiple of our disciplines (software, data management, data science, computing platforms), and explicitly designed to interact with other teams within the same institution.

Research Computing and Data managers typically have a lot more responsibility for given level of seniority than (say) equivalent levels in a large company. We’re charged with not only supervising work and overall direction and professional development of the team (which in tech companies is already split into two roles, a tech lead and a people manager) but we’re also typically given a lot of autonomy about setting direction and strategy.

On top of that, we are in this line of work because we want to make things better for research. We see how we could have so much more impact, if only we had more money, or if only Department X was on board, or…. We genuinely want things to be better, and have already taken on so much responsibility, so we feel the duty to fix that, too.

So when I talk to managers, especially new managers, I hear this sort of question a lot: “How do I get my senior leadership to give us more money?” or “How do I convince Dept X to get on board with what we’re trying to do?” It's completely understandable, given the situation they find themselves placed in. But I find it discouraging, because it’s the wrong question.

We can’t “get” people to do anything.

How many surveys have you skipped over or deleted in emails or web pages in the last month?

Whenever I suggest we talk to people more often, the question of surveys always comes up.

Surveys are a limited but useful tool for collecting data from our researcher clients, but work best in a very specific use cases. They should definitely be one tool in our toolbox, but they get overused, get used lazily, and get used for purposes where other methods would be a better fit.

As in so many areas of our work, when we’re considering gathering data from our client community, the key question is to ask ourselevs is: “what problem am I trying to solve?”

In #153 I mentioned Arm’s new hybrid training delivery model, where they have synchronous meetings for kickoff, Q&A sessions, and closing; but the bulk of the material is pre-recorded and students go through it on their own pace asynchronously. This is a good example of the kind of leverage I was talking about last issue, where the team is aiming to have a similar training impact with a lot less staff time by relying on things they’ve already created (labs, assignments, recordings of previous lectures) and something that already exists (the cohort of students).

A reader wrote in suggesting they were experimenting the same idea:

[…]So I have some comments on the ARM training delivery. In the previous semester, we conducted 2 rounds of our workshop series […] This semester we are more short staffed so only did them once each and only on zoom. All participants said they preferred zoom delivery rather than the nice room […] But we also have recordings on our YouTube channel for asynchronous delivery. We are contemplating just promoting the videos and offering our Office Hours if anyone would like to discuss the content, especially the hands on exercises.

This could be really useful, especially if there’s some kid of forcing function for students to keep going through the material together! I’ll be excited to see how this works. Are there other teams trying similar things? Let me know - just hit reply or email me at jonathan@researchcomputingteams.org.

When I talk to team managers and leads, individually and in groups, I often say something along the lines of our mission, as RCD leads, is to advance science as much as possible given our priorities and finite resources. I’ve yet to receive any push back on that; everyone agrees.

But it’s really only the best teams that I’ve seen that manage themselves as if that really is their goal.

If our mission is to have as much impact on science as possible - and I believe it is - then that mission has to shape strategy and decision making throughout the team. It has consequences for what work we take on, how we do it, and how much we value learning and replicatability.

Doing the Right Things Is More Important Than Maximizing Productivity

Hi!

A couple of reader replies from the last issue - here’s long-time reader Scott Delinger, CEO of Canadian regional academic HPC consortium Prairies DRI (digital research infrastructure), responding to an article about remote work:

Our situation in Canadian ARC [Advanced Research Computing] is slightly unusual, in that we’ve ALWAYS had remote work, even when staff were sitting in the same room prior to COVID. So the main difference is the effort required around group social aspects, not work tasks.

This is exactly right - Canadian ARC is an extreme example, but in our line of work, distributed collaborations are pretty common. That experience gives us a huge advantage when it comes to remote work more generally; we can make use of that by applying what we’ve learned there (around collaborating in a more document-based way, asynchronous discussions, etc) more consistently in other contexts. Some things are still hard remotely - social cohesion, mentoring juniors - but we can build on our existing experience. A lot of other communities didn’t benefit from having had that starting point.

Hi, everyone!

As you can see, I’m slowly getting back onto the regular newsletter schedule after coming back from my belated holidays trip.

Thanks for you comments on last week’s newsletter about a refactoring. I had several people say that they like having things in one place, although I worry that there’s a bit of a survivor bias to that.

Still, it should be taken seriously - so I’ll tweak my plans. Let me tell you what I’ll be trying over the next couple of months.

Hi everyone -

A quick request before we get started.

I’m thinking of refactoring the newsletter a bit, pulling some things out into a separate email. It’d be an experiment that would play out over a couple of months. I have some ideas for how that would work (I’ve even played with something as a test), but I’d like to hear from you. These emails are pretty long! What would best serve you if I was planning to rearrange things a bit? Let me know - hit reply, or email me at jonathan@researchcomputingteams.org.

Hi!

Last week I talked about the purpose of a meeting, and how for recurring catch-all meetings like team meetings, each agenda item might have its purpose.

This week I want to talk about how to make sure how people are engaged during the meetings. To keep people engages, there needs to be regular, valuable, opportunities to contribute and interact.

As people of science who have become team managers or leads, we understand the importance of considering our audience's needs when giving a talk. We tune the topics to local interests, we adjust the amount of background information to be appropriate to level of familiarity we expect most of the audience to have, we make sure there's time for Q&A so that there's interactivity. We learn this early on, whether it's explicitly taught to us or not, because (at least in person) we get very rapid feedback when a talk isn't landing - people start looking confused or bored, tune out, and start doing work or messing around on their phones.

Hi!

Sorry for today’s issue being late - I’ve just gotten back from my first trip since Feb 2020, a long-delayed trip to visit family. Travel was… fine? Other than the usual “travel to the east coast of Canada during winter” logistical problems caused by weather. We were extremely careful, and it seems like we made it back without getting sick, which is particularly important for our household.

In other news, thanks to everyone who's already offered some feedback on a quick writeup on feedback. I've incorporated some of those changes, and now have a draft I'm happy to share (even though it will continue to need updating); if you'd like to see it it's available on the newsletter webpage here. If that's something you'd like to try out and work through, let me know - just drop me an email.

Happy 2023, everyone! I hope you enjoyed the holidays and that the new year has been good to you so far.

This will be a short newsletter while we all get back into the swing of things.

First, I want to share a success story from a reader who wrote in to share with the community:

[…] I used the feedback discussion from December to write a short but detailed email to each team member at the end of December. I told them what impressed me most and related it to how they helped the team overall. The emails were VERY well received and told me how they were feeling, so the feedback to them also was feedback to me. […] Thanks for the idea

Your End-Of-Year Review

You did great this year.

I wanted to take some time to personally recognize the work you've put in during 2022. You have been a terrific support for your team and the research your team enables, and I appreciate all of your efforts.

It hasn't been easy, but you worked hard over the last twelve months. You've done well by your team members, and by the research your team supports. Enjoy that and celebrate that over the holidays. You deserve it.

I got a couple of emails about last weeks issue, asking about both communicating expectations and team processes. The most important, the most fundamental of these two are expectations. In a team without a common understanding of expectation of each other, no process in the world will matter; they’re just so many words on paper.

With team members, peers, or colleagues, we share whether our expectations are met or not via feedback.

Feedback’s vital to calibrate and communicate shared expectations. Feedback is how we acknowledge that people have done a good or great job, meeting or exceeding expectations; and it’s how we give people nudges or start a conversation when they haven’t met expectations.

The fact is we all want feedback, from our managers, our community, our peers. We’re ambitious, driven people, and we want to know where the bar set is so we can clear it; we want recognition when we do clear it; and when we miss, we want to know so we can improve. Working in an environment where you’re not getting feedback means knowing there’s a bar there, somewhere, but not being able to see or feel it, and so having any idea whether your jumps are high enough. It’s disorienting and unsettling and unsatisfying.

I was part of a discussion recently about a disengaged team member, and the topic of process vs expectations came up.

I write and talk a lot about expectations and processes. I’m a big believer in both! I think they should both be documented, discussed, and violation of either should result in some discussion.

But they’re not the same, and they’re not substitutes.

I think we’ve all worked in (or heard about) places where there was some restrictive process in place, because “one time, this guy Frank - you weren’t here, you wouldn’t know him - well, he…” did something. This kind of process reaction to an expectation violation is stifling, and it gives process a bad name, because everyone knows it’s a cop-out. The unsuitability of this approach came up in the discussion, and everyone knew the failure mode being described, because it’s common enough to be a trope.

I spent much of this week helping teach a small part of EMBL-EBI’s excellent Managing a Bioinformatics Core Facility course (Here’s the 2021 public materials). It was my first time participating, and I was really impressed. There were a large number of new, motivated, engaged core facility leads; the workshop was very interactive, with multiple exercises every day, lots of group discussion and sharing of experience, and a lot of material covered.

The audience being leaders of core facilities, much more time was spent on cost recovery models than would be common in research computing and data teams (there was even one brief section which touched on hourly vs fixed-price billing, which was fantastic to see - readers will know where I stand on this, e.g. #140). There was also a terrific series of exercises on service design, which also isn’t common to discuss in our circles but really should be.

But it was fascinating to see how similar the fundamental issues were. Many of the questions which came up would be very familiar to you, reader — hiring and retention, motivating staff, working with recalcitrant researchers, dealing with benign neglect from leadership, demonstrating impact, improving execution, prioritizing efforts, juggling the needs of a diverse and growing research community.

I think it’s pretty clear that in the RCD community we’re currently about five years behind our core facility colleagues on issues of building a community of practice and professional development, certainly for managers and leaders. Honestly, this is great news! We were much further behind a decade ago. The tireless and thankless work of groups like CaRCC and CASC have been pushing us ahead in terms of building a community of practice around research computing systems in particular, and in building an IC career ladder framework; and of course the RSE movement has been great for building similar things at least at the IC level for software.

One of the things I wrestle with a bit with the newsletter is trying to cover the range of topics that research computing and data managers need to think about without being overwhelming.

But we have genuinely challenging jobs, that span a range of concerns. I gave a presentation about this earlier in the year, but I haven’t quite talked about it the same way here.

We have three main areas of concern which we have to make sure we’re paying attention to - the people in our team (individually and collectively), the work output/products our team produces, and the processes by which we operate. All of those need careful tending to, both for the future (nurturing growth and development), performing regular maintenance care, and dealing with occasional urgent matters. And they’re all essential; people firing on all cylinders but held back by incoherent processes and working on the wrong outputs aren’t going to have the impact they could.

This week I did something a bit different: I interviewed Matthew Smith, ARC Research Systems Manager at the University of British Columbia.

We had a great discussion, which touched on the self-doubt that comes with first becoming a manager of experts, working with other teams to have bigger impact than your team can manage by itself, transitioning to remote work as a new manager during the pandemic, the growing need for research platforms, the importance of campus-wide communities of practice, and the coaching mindset to leading a team.

My favourite quote came from the end:

Just love what you do, love the people that you support. I think all of us in research computing are in a unique position. It’s quite the niche that we’re involved in here, and I think we have a rare opportunity to contribute to what I certainly see as one of the key parts of society, which is research, new learnings, new innovation, and being able to support that is a rare opportunity.

Hi!

I keep hearing from leaders and members of teams where retention is a big problem. It starts small, with a single person leaving, and then a steady cascade of other people follow.

Worse still, there’s a worrying asymmetry between what I hear from the team members at these institutions, and the leaders and managers.

The managers seem convinced that industry’s salaries can’t be competed with, so there’s nothing that can be done.

Teams are having a hard time hiring - or even getting people to apply for their jobs. Whenever groups of research computing and data managers and leaders get together, this is a topic that comes up - I’ve heard it as a common refrain in the past year especially.

Unfortunately, the old approach of academic hiring - post a templated job description on a website and wait for the candidates to find the job and send in an application - doesn’t work any more. It doesn’t work anywhere particularly well, but it falls especially flat in our industry. You see this everywhere across academia; even what used to be highly competitive postdoc positions just aren’t being applied for.

We have to be more active in searching for job candidates today. That means a lot more work, but it’s upfront work which will make the hiring and onboarding earlier.

(We also have to be more active and put more work into retaining team members - I’ll talk about that next week)

Issue #140 was controversial, but not for the reasons that I thought it might be. In the last two weeks I’ve had now had several people flat-out deny this claim:

There are core facilities all over your institution which get much of their revenue from fee-for-service quite successfully.

saying, essentially, that such a tawdry arrangement would never be tolerated at their institution. In each case, googling “[Institution] core facilities” returned lists of teams that nonetheless did exactly that.

I’m not at all surprised that we still have so much to learn about other technical research support teams even within our own institutions. I’d like to change things, though.

In #139, we talked about some general purpose first steps when taking on a new responsibility. Let’s talk about the other side of that, now — how to make sure someone we’re bringing on to tackle some new challenges can be as productive as possible.

Back in #135, we talked about beginning with the end in mind: having clear goals for the person taking on a new role, what it should like in the day to day. When scoping out a new responsibility, it’s important to define what success looks like after they’re fully onboarded. That onboarding timeframe may vary: say after 3-4 months for an individual contributor, maybe after 6 months or even longer for someone with significant leadership duties. That doesn’t mean they won’t continue to gro after that! But by that point the goal is that they’re functioning competently in the role. Our job, then, is to help them get there as quickly as possible, for the sake of the team and so that they can start feeling settled in and successful. That means designing an onboarding process.

There are four kinds of information to gather to design the onboarding process:

• Make a list of all the knowledge skills and behaviours the person will have to learn to be at that successful end-state
• Make a list of the people that the person should meet and talk to to understand the inputs to and impact of the work they’re taking on
• Take an inventory of the resources (people, but also training materials or funding for training) you have available to help bring the person up to speed
• Identify meaningful self-contained projects, increasingly non-trivial, that they could work on that would give them concrete places to apply to what they’re learning, and that could be milestones to being an onboarded, independent team member.

This week I got a question about time-and-materials billing. So I want to talk a little bit about how consultants and other expert services firms increasingly see fees-for-service, and how we should see them to be of the most value to our community. I suspect this topic is going to be even more controversial than specialization is good and we should do more of it (#114), or making the case that we are vendors too (#123), or how too much of the framing around strategy discussions is self-important puffery (#130). But charging fees is an important topic if we want to sustain our teams and the research they support.

(By the way, the question came from a call from a member in our community, so a reminder that I’m happy to jump on a 30 minute advice or coaching call with anyone in our community who wants one - just pick a time).

Read on, or go straight to the roundup!

First, I want to emphasize that charging fees for offerings is not only ok, it’s good. I’d like to see more if not most of our teams have it as part of their revenue mix; how much of a part will depend on how they’re funded. We and our institutions talk about sustainability models quite a bit, but too often charging money for goods and services seems to be the sustainability model that daren’t speak its name. Which is odd! People are quite happy to exchange money for goods and services they value. There are core facilities all over your institution which get much of their revenue from fee-for-service quite successfully.

I got asked a question building on the last post about taking on a new project, and the current top unanswered poll.ly question is “I'm starting a new job! What should I do to ramp up quickly?”

Let me answer a slight generalization of those questions: say you have taken on an existing responsibility. It might be being promoted to manager, it might be a new job, it might being put in charge of some effort that already has some history. (And in our institutions, every effort already has some history. There are no completely de novo efforts in our line of work). What should you do?

(Read on, or go straight on to the roundup).

It’s mathematically possible, of course, that someone in our roles taking on a new responsibility has clear guidance from their boss, and is receiving routine and helpful support and coaching on on their efforts, so that they feel like they’re in good hands. Yes, yes, stop laughing, I said it was possible.

I got a question from a reader this week which I’m surprised to see that I haven’t addressed before, and that I’m sharing with permission:

I have several years experience [as an IC in research computing and data team]. With some changes coming up here in the next year or two there will be at least one manager job opening up. I’ve been reading your newsletter for six months, and it’s been really helpful! Do you have any suggestions for what I can do to be ready as a good candidate for the job?

Below is the advice I shared, and I’d welcome any other suggestions you have. Is there anything else this reader should be doing or thinking about? Let me know and I’ll pass it along, to them and (with your permission) to other readers!

Congratulations! Our community needs more new managers with enthusiasm for doing the job well. And I’d say that you paying enough attention to your team’s broader context to see this coming and be thinking about the consequences is already a good sign.

I get a lot of questions about both hiring and communicating expectations/performance management. I’m trying to compile and clean up some of what I’ve written on those topics (e.g. #65,66,67,68) in an ebook like I put together for one-on-ones at the beginning of the pandemic.

There's two big topics left on the topic of communicating expectations; they both touch on the current top un-answered question on the pol.ly poll:

How can I get better at receiving feedback from my team?

A lot of what we see about expectations management and feedback is about managers providing feedback to individual team members. It’s an important topic! A hesitation to give routine feedback (frequent, mostly positive, always rooted in behaviour and impact) is the biggest gap I see among new managers of research computing teams. This is an vital “beginning as a manager” 101-level skill. It's also one scandalously overlooked and under-appreciated in most of our institutions.

A housekeeping item before we get started -

I’ve had the searchable archives for back issues for a while, but there’s a limit to what a simple text search can do. I’ve wanted to have previous articles and write-ups browsable by category for some time now. There’s two time-consuming parts to that - going back and curating the best-of evergreen articles (right now mostly from the roundup), and then figuring out how to make them available.

I’ve done a first pass at the first piece, and now have a VERY crude initial attempt at this - so far what I think it’s taught me is that I’ll need some sort of hierarchy to the tagging of articles, which I need to think a bit about. What would be useful to you in a categorized “best-of” list? What would you like to see? Drop me a note - hit reply or email me at jonathan@researchcomputingteams.org.

Long time reader Adam DeConinck wrote in answering a question about Firecracker, a light-weight VM which is used for hosting tools like (as in last issue) CI/CD pipelines:

I love Firecracker, but so far there’s no support for PCIe pass through (eg GPUs, NICs, etc). Which I find really unfortunate […] because research computing in general is making heavy use of such devices, wherever they get them. I’d love to see a micro-VM approach in HPC at some point. Or really anything that can provide a better multi-tenancy story for shared research computing facilities in an increasingly security-conscious world. And as some Kubernetes devs have reminded me lately, the kernel (and containerization) are not a security barrier. […] Selfishly, I just want a VM platform that's as easy to use in HPC as Singularity or Enroot.

This was new information to me - thanks!

I’ve been catching up on some reader feedback now that I’m back in the swing of newsletter things. Thank you all so much for your emails! I’ve learned a lot from our exchanges, and often shifted my position on things (or at the very least corrected how I talk about those things).

I especially appreciate pushback. Some areas I’ve really gotten pushback on in the past couple of weeks are what I’ve written about about utilities vs professional service firms (#127), or strategy and strategic planning (#125, #130). Discussion about those topics has helped me realize that I’ve never really been explicit about the audience I’m prioritizing for this newsletter.

In the past I’ve worked with VPRs and those that report to them, people putting together strategic plans for national communities, and those reporting to CIOs. People in those roles are doing important work, and deserve more support than they get!

But the most urgent need in our community, and the priority for me in this newsletter, isn’t supporting those that already regularly meet with CIOs or VPRs. It’s helping the first-level managers of individual small teams, and those who aspire to have more say in leadership in of those groups. Managers and leads and aspirants who were never given any training, are still given little to no direction or support, and yet find themselves accountable for a key research support function. People trained in academia's years-long timescales, who are now having to figure out on their own how to run a team that has people depending on them weekly for firm deliverables. Leads who are figuring out how to do research project management when the research problem is still fuzzy. People managers still new to hiring, managing, and figuring out how to structure and strategize for a service organization. People now responsible for research software development groups, or data science service teams, or computing ops teams, or informatics-heavy core facilities, or even compute-and-data heavy spinouts from academic research groups.

Hi - I hope you’re enjoying your August!

One of the things I’d like to do now that I’m back from a couple of weeks off from the newsletter, is start being systematic about continually speaking with new research computing and data team managers, leads, PMs, etc - I’d like to talk to fifty or so and make sure I know the problems they have faced, are facing, and their successes. I’ll write up the aggregated, anonymized results results, and circulate it to participants first. I have five or six questions I’d like to ask them (and maybe you?) over a 20-30 minute zoom chat:

1. Tell me about how you started as a manager or lead, and the challenges you had at the beginning that you could have used help with.
2. Tell me about your team now - what they do, how they work, and who you report to.
3. What are some successes you’ve had with this team as a manager or lead?
4. What are some challenges you face right now, and what materials and resources are you using to help you navigate them?
5. What kinds of resources would help people in your position?
6. Who else (in particular or in general) would be good for me to talk to?

If that’s something you’d be up for, or you know someone else who’d be game, let me know - hit reply or email me at jonathan@researchcomputingteams.org!

Hi!

Last week we talked about ways to say no to incoming requests; besides a flat-out no, I mentioned the possibility of involving another team, or supporting the research group in doing the work themselves. RCT community member Adam DeConinck sent in another suggestion:

Occasionally, I’ve seen requests come in for work that the research computing team isn’t going to do; but which is also not something that another provider exists to do, and which the customer can’t do themselves even with training. When that happens, the “support” I’ve been able to offer is to join the customer in advocating for change. For example, aggregate similar requests and find commonalities; then organize the customers involved and advocate to upper management that they provide resources for this work to be done. That’s often not terribly satisfying, and I’ll admit that my success rate in doing this isn’t very high. (Non-zero, though!) But it’s often the only way I’ve found to create new classes of service within an existing organization.

This is a great suggestion, taking the long term view - and by lending your voice to the research groups’ in advocating for support for different kinds of work, it turns the situation into “your team and the research group vs the problem” rather than “your team disagreeing with the research group”.

Hi!

A question came in from a reader - I’ve talked before about how to not do things, and in my recent talk I said “do whatever researchers ask for” isn’t a reasonable strategy. We have finite resources, and we owe it to our communities and to science to focus those resources on the highest-impact things we can be doing. But what does that actually look like when you've been asked to do something well out of scope? Below’s a lightly edited version of what I sent back:

It’s important to pay attention to user needs, and thank them for their requests - if nothing else it’s useful data! But we can’t fall into the trap of trying to do everything a researcher asks for (or even every thing many researchers are asking for!). People want a lot of things, and our time and resources are finite. It’s perfectly ok for people to ask us for things, and it’s perfectly ok for us to say no.

Here’s a partial list of examples where it wouldn’t make sense for a team to provide a service that’s asked for:

Hi!

I want to write one more thing about strategy this week before going back into the basics of managing our research computing teams.

It turns out I’m still a bit discouraged by one section of the recent (very good!) PEARC21 report on strategic planning for RCD groups. That part described all too clearly the confusion and helplessness many team leaders of teams like ours feel when asked to do some positioning and strategy work. Like so many things, we’re thrust into responsibility for these kind of activities without ever being taught anything about them. It’s intimidating. But it shouldn’t be, and doesn’t have to be, that way.

The thing is, finding a path forward for a team or organization is a research and problem-solving exercise.

Hi!

I write a lot here about challenges our teams face, but I don’t think I mention often enough the huge advantage our background in science is in our work (or in similar work in other environments).

Whether we were trained in science or just picked up that mindset by immersion in the ecosystem, the skills we have are skills that (say) tech companies pay real money to train their staff in. The advanced collaboration and influencing skills we have to develop in scientific collaborations is a huge advantage all the way up the management or technical leadership ranks, for instance. Being willing to think in terms of experiments when making changes (like Beinbold’s article, below), or having a “growth mindset” (current jargon for believing that people can develop new skills(!!)), or being able to synthesize information - these fundamental skills, skills we don’t even bother mentioning in our environment, are in high demand.

It’s not just people skills, either - I see an article about staff engineer interview skills that include being able to estimate well, something every quantitative scientist does without thinking about it, or another article this week about how tech companies are having to relearn basic performance things like array-of-structs vs struct-of-arrays, which many of us have had to learn optimizing scientific code.

I heard a number of answers back about what sorts of service request processes people were using:

• A lot of software development teams, of course, took bug reports and small feature enhancement request via GitHub/GitLab/Jira etc ticketing systems
• Several groups are using Znuny, a fork of OTRS
• Someone juggling multiple projects used a timetracker to make sure reasonable amounts of time were being spent on various external projects
• A technical leader of an internal platform team said that it’s mostly “’support channel in Slack’ With complex issues moving to email” (but they had succeeded in at least getting the requests into the channel rather than DMs!)

One data science team had a longer answer that I want to quote here:

For a relatively small team of a few staff and about 10 students taking about 250 consultation requests a year one on one help on data science topics, not help tickets for a service), we use […] – basically a form + spreadsheet (google forms/sheet would work too). We have columns to track the status (incoming request, in progress, completed, etc.) and who has responded/been assigned, along with the info about the request. We don’t put other significant updates on the consults in this sheet – just for tracking status. We make sure our team brings the status of everything up to date at team meetings. Requests ping a slack channel when they come in, and we determine who will respond. So we can make sure that someone took each consult, even if the sheet doesn’t get updated immediately (it should be, but it doesn’t always happen).

For longer projects, we have only 6-12 or so per year – those go on a separate sheet for recordkeeping/reporting. All updates are communicated via team meetings or 1:1 meetings.

Before we get started, please answer if you can a question from a reader: What do you use to get and keep track requests for your work from researchers? Is there a ticket-tracker type piece of software that works for you, or do you use something else? Does it fit your workflow - What do you like, not like, and wish for?

I’ll summarize results next week - it’s an important topic!

One of the great things about this community is the different kinds of research computing and data teams I get to talk with, “here” or outside the inbox. Sometimes in these conversations it’s pretty clear that some teams, or the team and their funder, or a team and I, are talking a bit past each other. And that’s usually because they or we are (currently) operating with very different mental models.

Hi!

I write a lot here on the challenges that research computing and data teams face, and the challenges that their managers and leaders face in particular. And that’s because I want them to have fewer challenges! I hope that by highlighting the challenges they at least don’t face them unawares, and that as they grow their skills they bring along the next generation of managers and leaders under clearer, less stressfull, and more suppported environments than we found ourselves in.

Which is all well and good, but it can get a little gloomy in here sometimes. I don’t think I spend enough time highlighting successes.

Do you have management or leadership wins you want to share, no matter how small? Just with me, or with the community (anonymously or otherwise)? Let me know - just hit reply and send me an email at jonathan@resarchcomputingteams.org. I think it would be good for everyone here to see what some of those wins look like.

Hi, all!

I want to write a little bit more in the coming year about strategy for leaders of research computing and data teams: setting priorities; deciding what and what not to do; and deciding what success looks like. It’s an important topic - and yet as with so many things in our line of work, no one teaches us how to do it. It’s also much more ambiguous in our discipline than it might have been a researcher, where you can keep score pretty simply with papers and grants.

Those of us who have gone through University strategic planning processes come out even more confused, because these “Strategic Plan” documents aren’t a great introduction to strategy at the best of times, and they’re particularly odd ducks in the context of a University.

There was a great article by Alex Usher and team at Higher Education Strategy Associates last week. I want to use it to set the stage for what a strategy is, what a good overall framework does for teams or orgs, why Strategic Plans for universities are so constrained, and what that means for us. Usher has written quite a bit about strategic plans in the past, some of it quite cutting, and all of it insightful - I recommend the articles, and the blog in general, if this topic interests you.

Hi!

I wanted to write more this week about the consequences of research computing teams as vendors, and what that means for strategy - I’ll do that next week.

But there was big news in HPC world in this past week - as you almost certainly already know, the Frontier super at ORNL is now the world’s first Exascale system as measured by the HPL benchmark.

This is an enormous technical (and project and product management!) accomplishment, the capstone of work that’s been ongoing since 2006 or so. Colleagues at AMD (which, disclaimer, has long been a worthy competitor of my current employer), HPE, and ORNL deserve all the praise they’ll rightly get. My sympathies to other colleagues at Argonne whose star-crossed Aurora system, facing a series of delays, didn’t make it across the line in time.

Hi!

My new job, working for a large company that explicitly sells stuff, has indeed been eye-opening — but not in the ways I expected. Mainly I’m confronted with clarity about some of my old jobs, and peer groups I’ve worked with.

I’ve always been sort of puzzled by the decisions made and priorities chosen by some research computing and data teams. Those teams seemed more… insular, somehow, than others. I saw it less often in contract research software development teams, or library research data management teams, or bioinformatics core facilities. I saw it more often in research systems teams — not all, of course, and not only.

Meeting with some of these old peer groups, wearing a vendor hat, and watching them interact with us, it becomes a lot clearer.

Hi!

I took part in a hackathon this week, working directly with researchers on improving their code for the first time in — gosh, maybe eight years? Nine?

It’s funny what sticks in the mind and what doesn’t. I kept coding on and off over the past decade, so that was fine, but all the glue stuff that I didn’t think to stay in practice on like “how do you unload all modules?” or “why doesn’t this shell script that sets environment variables work when I run it?” (yeah, that one’s kind of embarrassing) took some time to remember.

There were more data analysis projects, and infinitely more AI/ML projects, then there would have been a decade ago when it would have been 90% simulation codes. Technology has changed over the past decade — there are some new profilers, and tooling, systems are faster, and hey Power systems aren’t big endian any more! But I was mainly struck by how little change there had been in the basic challenges. Maybe the percentage of research groups with extensive in-house research computing and data skills has gone up a bit, but mostly the research groups we saw were very capable, deeply knowledgeable in their domain, with significantly less expertise on the computing, software, and data management side. They needed, and knew they needed, collaborators to work with to achieve their goals. And our responsibility was to do a little bit of hands-on-keyboard work, but mainly to give them the skills and resources they needed so they could continue to advance even after the engagement; to give guidance as trusted experts. And that expertise ends up straddling at least two of software, systems, and data - those aren’t useful in isolation.

Hi!

The (159-item!) intern process checklist from last week generated a few interesting discussions, and those plus Chris Dwan’s interesting talk discussed below return to a common theme here in the newsletter - of projects versus processes, or learning versus production.

In #91 I alluded to the fact that what distinguishes a hobbyist from a professional is repeatability. The hobbyist is still developing their skill, trying to make their one-off thing come out with high quality. The professional is no longer struggling to make one thing well; they’re trying to ensure that things reliably, repeatedly come out with the same high quality.

The hobbyist is thinking about the mean quality, and trying to get it to trend upward; the professional is concerned with the variance, and reducing outliers. (Mean matters, too, but for a professional, a high mean quality is table stakes). There’s a progression from focussing on skill to focussing on reliability, from craft to process. One of Dwan’s slides says:

A reader wrote in to ask:

We have a new summer intern starting in a few weeks who I will be directly supervising. Although I have been the technical/project lead on various projects throughout my career, this is my first time having a direct report (even though it will be temporary!). I was wondering if you had any “go to” or favorite literature for managing in a technical setting.

Congratulations to them, and this is a great question. It’s such a great question that I’m disappointed to discover that I didn’t have handy resources to immediately share with them, and I’d love to hear what your recommendations are.

In research computing, probably the two most common ways people find themselves managing is to be promoted to lead their existing team, or the more gradual approach which is to start with one or more interns/co-ops/grad students/trainees/etc.