RCT #167 - Case Discussion: Advising The Peer Whose Key Team Member Is Leaving. Plus: Getting retrospectives right; Productized Services; Mojo; LLNL Sees Fortran’s Risks; Attack of the micro LLMs

Dealing with the sudden departure of an experienced team member and managing priorities and stakeholders during the transition.

Last week we described the sad story of our fictional colleague Daniel Miller. Miller, who runs a Research Data Science team at an equally fictional university, is faced with the sudden departure in two weeks of his most experienced team member, Olivia Johnson, who has been acting as an unofficial team lead. Miller was not aware that Johnson was unhappy or looking for other opportunities. He now needs to urgently address several concerns, such as replacing Johnson, managing ongoing and upcoming projects, training events, and team members, and prioritizing Johnson's tasks in the meantime, and has asked us to advise him.

I got a number of people writing in with their advice, but people were hesitant to share their thoughts to the readership! Fair enough - this is the first one. So I’ll give my recommendations, in the form of an email sent to our colleague, and highlight some things I saw in common from those that replied.

Miller has a number of urgent and important priorities to juggle. So let’s tackle these in the order that we recommend handling them. I’ll start the email to him, shout out if you see something I should change or add:

Hi, David:

Ugh, that’s a challenging situation! I’m sorry you’re going through this. Let me break this down into some immediate steps, next steps, and then some recommendations for the future.

Immediate steps: Johnson, HR, Coworkers, Stakeholders

Johnson

There’s nothing really to say to Olivia about her new job other than “Congratulations! Obviously I’m very disappointed that you’re leaving us, but this sounds like a terrific role, and you must be very excited. Tell me about it!”.

There’s no point in being upset with Johnson for leaving - it’s good and healthy for people to move on and seek new responsibilities. Ideally we’d be offering them growth opportunities within our institution, but either way they won’t be on our teams forever.

There’s equally no point in trying to make a “diving save” to keep Johnson in this role. By the time an offer is signed, that ship has sailed. If she had come expressing some flexibility about start dates, we could advocate for as much time remaining in this position as possible to ease the transition. You could certainly try gently raising that possibility now, but honestly I wouldn’t expect much.

I’d recommend asking her what she’s thinking in terms of an offboarding plan. It sounds like she’s put a lot of thought into this, and she’s always been very diligent and clear-sighted about what needs to be done, so my guess is she’ll have a pretty good plan of action.

My recommendations to you for priorities for her off-boarding are pretty simple - the only real priority is getting as much of her knowledge shared as possible. That means:

• The state of the team - who’s happy, who’s not, who’s got what skills, who’s growing where, who’s working on what
• The state of the projects - current status, and if there are any credentials etc. that need to be handed over
• The state of the external stakeholders she’s been working with
• How various activities get done
• What her day to day activities are, and if she sees any of the team being able to step up to take any of them on
• Team priorities as she sees them

Some of this knowledge-sharing be done in the form of documentation; other can be done be shared through demonstrating, talking through, or having someone sit in on something.

Documenting mundane things like “day-to-day activities” really matters - it helps make sure there aren’t any things she does that get forgotten. Lots of tasks don’t seem that important, right up to the point where they stop getting done. Right now, no task is too small to get written down somewhere.

To make time for this brain dumping I’d recommend you ask her to cancel anything on her calendar that doesn’t absolutely have to be done, or can’t be double-purposed into supporting this knowledge sharing. For instance, if she’s meeting one-on-one with team members, that can be turned into “one-on-one-on-ones” (#49) with you participating and turning them into hand-overs. If there’s some work she’s doing to finish a project, see if someone can pair with her to see how something is done. If she’s meeting with a stakeholder to hand over a completed project, that’s a great opportunity for someone to learn how to handle that activity.

Then talk about sharing the information with the team. Make sure you agree on some simple wording about when she’s leaving, to where (if she's ok with that being shared), and some immediate next steps. Commit to sharing the information together.

Schedule a meeting with her for later in the day, and let both of you catch your breath. You’ll be meeting her a lot in the next two weeks. But first there’s two urgent things to do, then two important things to do.

Your calendar

Urgent thing one - as with Olivia, I’m going to strongly suggest you cancel or postpone everything on your calendar that can be cancelled or postponed. You’re going to be busy handling all aspects of this. Very little on your calendar right now is as far to the top right of the “urgent + important” quadrant of the ol’ Eisenhower matrix as dealing with this is.

HR

Your next urgent item is a call to HR to start the administrative offboarding process. While you’re on that call, give them the heads up that you intend to hire, and ask for any updates on the hiring process, get them to re-send you links to the forms and processes, etc. It sounds like you were going to be hiring shortly anyway, so let’s get a jump on that. It’s going to take a little while to figure out exactly what you’ll be hiring for, but do whatever you can to get that machinery warmed up. Send off the appropriate emails to finance, etc. Doing these sorts of mechanical tasks will help clear your head for the important stuff.

The Team

Ok, so this is the first of the important things, and the most important. The rest of the team is going to be (collectively and individually) worried. People hate uncertainty about their jobs. How is this going to affect them? How are you going to react to Johnson leaving? Are they going to be overloaded? Are they going to continue to get the mentorship they used to get from her?

You don’t know the answers to most of that yet, but it’s important to communicate to them that you’re on it, continuity of their work and work situation is important to you, and you’ll all figure it out together. Steady hand on the tiller sort of thing. But that message will only really start to take once they see you follow through with it.

So figure out when you’re going to tell them all at once. If your team meeting is in the next day or (maybe) two, you and Olivia can share the news and next steps together there. If it’s not, you probably don’t want to wait for the team meeting. Try to bring everyone together as quickly as possible - “Olivia has some exciting news for us” - and share the news and the immediate next steps you’ve worked out. Then announce you’ll be meeting with everyone one-on-one (because you will) to check up on them, address any concerns, and find out what they need. Make a list of times for people to sign up for those one-on-ones, and make them recurring times. (We’ll talk more about that in a bit).

Answer what questions you can, and don’t be afraid to say “I don’t know yet” and “Great question - we’ll have to work that out together” a lot. Pay attention to reactions and any questions that are being raised. I know you’re going to be very distracted through all this, but the team’s going through this too, and keeping them happy and not-worried through this is your most important job in the coming weeks.

The order of operations is important here. Public, then one-on-ones. If there’s someone Olivia is grooming to be her replacement, you can loop them in earlier and swear them to secrecy; but otherwise, far better if everyone finds out at once. Otherwise you have rumours going around, which are bad at the best of times and horrible in times of uncertainty.

The Stakeholders

A lot of projects are going to get delayed. You worked hard to build that pipeline of projects, and handling them and the research groups and other stakeholders through this period is going to be your job two. (This probably goes without saying, but projects and clients are job two because if the team starts falling apart, the projects are and clients are irrelevant).

There’s no point in trying to guess right now which projects will get finished on time, and which won’t - you’ll get some wrong. False negatives are way worse here than false positives. For now, assume they all will be delayed. Get ready to contact the clients one by one (you can use a mostly standard email, but send them individually, don’t just send out a mass blast) and tell them that you’re very excited about Johnson’s new opportunity, you’re working out how this will effect each project now, but you wanted to give them the heads up right away, and will let them know once you know more. Once the team knows, start sending these out. If there’s some that you’re wondering if you should phone instead, phone them. Or discuss in person.

Some people will be angry, others will be surprisingly generous and tell you pre-emptively that it’s ok if their projects are delayed, and most will be various degrees of understanding in-between. It’s not like they are unfamiliar with (or unworried about) this kind of scenario on their own teams. Thank them all profusely for their input and understanding, regardless of whether or not you feel particularly thankful about their responses. You’ve given them some bad news, and not all of them will accept it with as much grace as you have.

You’ve built a pretty good working relationship with the HPC centre and stats help desk - let your peers there know, too. You can probably continue to do intake on some projects, but don't give any expected dates until things are sorted. See if there’s anyone on those team that could help you out in the meantime. You might be surprised - they know full well that at any given point in the future they might be asking you for the same favour.

And I know this isn’t a welcome suggestion, but think carefully about that training your team was going to give. I know you were hoping to use it to accomplish a few goals, but would it really be so bad if it was put off another semester? Wouldn’t it be better to not have it hanging over your head right now? I’d really lean towards postponing it.

Next Steps

Ok, so now everyone’s been told, and we can start moving forward.

For the coming two weeks, meet weekly with the team members, and meet frequently with Olivia. There’s a few things I’d suggest:

Don’t Become a Hero Data Scientist

It’s going to be super tempting to throw yourself into IC work during the transition. It’s something you can easily do, be another pair of hands; there’s a case to be made for it; it lets you postpone thinking about all the hard team stuff.

Don’t do it.

For sure, you might have some particular skills or knowledge that would be useful. Sharing some of that in moderation might be useful. If no one else knows those things but Olivia and she’s busy handing off other responsibilities, making yourself available to pair with another interested team member to show them how it’s done, in the spirit of promoting knowledge sharing during the offboarding, might be super useful. But the team is going through a lot right now, and the clients are uncertain too; an hour you spend making things better for them is an hour much better spent than plugging away at a notebook.

One on ones with the team members

For instance, this stuff is important. Your main mission during these meetings is to listen. Ask them what they want to talk about, and dig into those topics. There’ll be a lot about their work, the switch over, and their needs. Take notes assiduously, and follow up on anything you can help them with.

Identify team members who are ready and willing to take on some of Olivia’s tasks. Double-check possibilities you identify with Olivia, make sure you both agree (she’s been working with them a lot more closely recently than you have).

Identify any concerns, and reassure them wherever you can. Keep an eye out for team members who might be at risk of leaving - some may have really enjoyed working with Olivia. You can’t stop them from leaving but you can increase the odds of their staying by making sure they have what they need within the team.

Ask the team for suggestions

Before the next team meeting, let everyone know you’ll be asking them for suggestions about how things can be done differently for the next while to adjust to this situation. This will help the team members feel a bit more control over the transition, it will give you some good ideas, and it will use this challenge as an opportunity to find potential improvements. Find a few suggestions you can implement right away, and come back to the others later. Keep having these discussions routinely.

Build on the offboarding documentation

There’s going to be a game of musical chairs matching tasks to people once Olivia’s gone. People will have new task mixes, and so knowledge-sharing among the remaining team members will also be crucial. Use the work Olivia’s already putting into documenting things to create a knowledge base for the team. Encourage people to share information there; make it the first place to go to look things up. If people share things on slack, encourage them to copy it into the documentation. This is going to be important going forward.

Keep the clients up to date

As the projects start moving forward again, make sure clients are kept abreast of the state of their projects. Check in with them regularly, make sure they are getting the information they need. Ask them, too, what could be better from their point of view, what would be valuable to them.

Get a sense from Olivia what could be done better

You’ll be meeting with her regularly making sure handover is going well. A departing team member may not feel like they can be candid about the things that eventually nudged them to leaving. They might not be 100% sure in their own head what ended up being dispositive. But you can ask them for advice about how to bring a new team member in, about how to set up a new team lead for success. Those questions, safely distanced from her own experience, will be much easier for her to frankly answer.

Hash out some ideas with her about what sort of skills you should be looking for as you hire. Ideally you’ll have a job description ready to go during this period.

By the way, a nice and generous thing you can do for Olivia is write a first draft of a letter of recommendation and run it by her, getting feedback on what she thinks she did that was most valuable in this job. It may come in handy for her (and you) in the future for her job search - and it’ll be easier for you to write the reference now when everything’s still fresh. Offer to post it as a recommendation for her on LinkedIn, which may save future possible hiring managers even contacting you in the first place.

Identify what skills will be needed in a new hire

Talking to Olivia, HR, and finance, and the team as a whole, start figuring out what to hire for. This will depend on a lot of things - what you think future projects will look like, who is able to step up into what part of Olivia’s old role, and financial concerns of course. Find ways to get the remaining team members involved in the role design and hiring process (e.g. #135, #136, #141). Remember, you’re hiring for a role not a laundry list of skills. Figure out what the role will be, and about what success in the new role looks like, and how you would onboard them first. Everything else shakes out from there.

Future steps

After that, David, the crisis atmosphere should have dissipated. You’re well on your way to having a hiring plan, the team is in a good situation, and project flow is returning to normal. As a bonus of this adventure, you have a lot more documentation of how team work gets done (from Johnson’s off boarding), a lot of knowledge has been shared, and you have pretty close insights into how the team members are doing.

But this was a lot of work and stress. Let’s see if we can avoid going through this again.

There’s a few things that happened here that we’d like to reduce the likelihood of recurring - not to zero, but we can reduce the probability of the following:

• Someone felt they had to leave the team and institution to advance in their skills and responsibility
• We were surprised by the above
• We didn’t have a good handle on what they were doing
• We didn’t have a good handle on what headspace the team members were in, or how work was going for them
• We didn’t have a good sense of who had what skills, or who could take up which new responsibilities/opportunities.

You can build on what you’ve already been doing through this to make things better in the future.

• Team members having regular weekly one-on-ones which start with things they want to talk about are an important management tool generally, and are very good (if not infallible) early warning detectors for people you’re at risk of losing. Whether you continue having them, or the new team lead does when they start (and you having on-on-ones with them and the occasional skip one-on-ones with everyone else), they’re very important.
• What one-on-ones are to individual team members, retrospectives are for teams (#137) - you can make sure concerns about how work is going at a team level get expressed, and by acting on them you can make the team happier and more productive. This can be periodic, or after each project gets done. This can be an extension of the work you’ve already put into finding out how the team’s work is going and what can be changed to make things better.
• Quarterly work/career goal setting and review sessions (#68) are really good tools for making sure people are working on their career goals within your team, and excellent ways of finding out who is ready to take on new tasks. So if someone does leave (or work expands), you have some information about who could take over which tasks.
• The team can build on the new documentation of knowledge and processes that came from Olivia’s offboarding - keep that current and live, and document other processes (#148) along the way.

This has been hard work, but you can use that work to keep improving things.

Good luck; you can do this. Don’t hesitate to reach out again if you have any questions or new problems arise.

All the best,

Jonathan

That was pretty long! The responses from other readers largely included points covered above. People agreed pretty much unanimously on telling the team members immediately, guiding them through the process. There were a few different approaches described on how to handle the projects - I took a simpler approach in my writeup, but the right thing to do would really depend on the state of the project work and the relationship with stakeholders. YMMV.

What do you think? Are there more problems our fictional David might encounter along the way? Anything different I should have told him? How did you feel about this case discussion - is it something you'd like to continue seeing? Are there situations you'd like to see discussed? Let me know - hit reply or email me at jonathan@researchcomputingteams.org.

And now, on to a (short!) roundup!

Managing Individuals and Teams

Over in Manager, Ph.D. this week I wrote about 1000 words in praise of management jargon.

In the roundup:

• Why Candour is Hard
• Hybrid work supports team culture, but is hard on managers

Technical Leadership

5 Steps to get your Sprint Retrospective right - David Pereira

Pereira has some recommendations to get the most value out of retrospectives (here for software sprints, but could be any retrospectives), for us and the team:

1. Icebreaker : Get people out of the everyday routine into big picture and thinking about the team.
2. Actions ### Review### : The whole purpose is the actions we follow up on! Pereira suggests spending a good long time on reviewing actions taken after the last retrospective and how they’ve gone.
3. Inspection : Vary the approach you’re using to have people consider what has gone well and where changes might be beneficial.
4. Action : Separate out the “how did things go” and “what should we do about it” stages. Brainstorm on a (small, maximally useful) number of actions to take to improve things
5. Tune Out : Vary the wrap-up

Product Management and Working with Research Communities

The Productized Service Manifesto - Greg Isenberg

Isenberg talks in general about what I tried to discuss in #157 about leverage (and in Mitchell’s article in #165 about balancing customization and standardization) - the development of productized services.

“Refactoring” one-off service engagements with researchers into one or more standard “productized” offerings, with checklist and SOPs, lets us advance science further, faster by letting us get really efficient at delivering these new core offerings. It doesn’t have to mean that there’s no customization, or that there’s no other one-off bespoke engagements; but it does mean we can take advantage of some efficiencies of scale that otherwise wouldn’t be available to us.

Research Software Development

Mojo may be the biggest programming language advance in decades - Fast.ai

Hyperbole in the title aside, there have been a number of “make python faster” attempts in the past that have had only modest success. Numba is really good at what it does, and has probably caught on the furthest, but unless your code is basically Fortran written in Python there’s not much it can do. Jax is really promising for certain kinds of things.

Mojo is a new fast Python++ put together by someone with deep LLVM experience, and someone who led the development of Swift. I don’t know if it will fare much better, but in the AI community people are modestly curious. I like this article because it gives some historical context that I think is useful.

LANL Report – Looming Fortran Talent Scarcity is Threatening - John Russell, HPCWire

There’s been a widely discussed and short report out of LANL about Fortran’s continued suitability for DOE codes - in particular the risk of having developers who are willing to code in Fortran, and having first-class software support.

The DOE is adapting to a world where they they no longer define top-end computing, or are even particularly large players compared to large cloud or internet companies. That means:

• They can’t expect that working on legacy codes in languages not used much elsewhere will be continue to be appealing as a career path
• They can’t just command indefinite investment in compilers
• They might need to be a lot more nimble in how they develop and evolve software in the future to take advantage of technology trends determined (and thus funded) elsewhere.

This report highlights these risks - that compilers might fall behind (and lack C++-style cross-hardware compatibility frameworks like Kokkos), and that finding excellent staff will be hard.

Emerging Technologies and Practices

Leaked Google document: “We Have No Moat, And Neither Does OpenAI” - Simon Willison

(Disclaimer - I currently work for NVIDIA, which obviously has a huge interest in growth of AI.)

There’s been an explosion of LLMs in recent weeks (like this brand new one MPT-7B) that are much smaller than GPT-3, and tuned much less expensively, that are nonetheless performing quite well. Not nearly as well as GPT-4 yet, but shockingly good.

This is unambiguously good for us in the research world - it means that researchers (and research support groups) don’t have to have OpenAI or Facebook AI sized infrastructures to do good and meaningful work, and it likely will mean a very large number of specialty LLMs.

WIllison writes on a leaked internal Google document suggesting that Google thinks this is a threat; that the “competitive moat” they have around lots of experts and vast scale compute resources doesn’t actually protect them (or their competitor, OpenAI) from facing heavy competition in the LLM market. What’s more, the huge and enthusiastic community means that some of these more open models can benefit from absurdly fast iteration by researchers, developers, and even just hobbyists, moving collectively much faster than any one company could.

Obviously, this assessment could turn out to be simply mistaken - things are moving far too quickly to say with certainty how things will play out. But certainly the last two months have provided some really exciting evidence that there will be a lot of cool things that can be done at much smaller than Google scales.

People seem quite excited by mrsk, a kind of ruby-based imperative docker compose for standing up application stacks. I sort of refuse to believe that we all have to internally master k8s to deploy simple applications, and yet docker-compose is clearly inadequate, so I’m curious to see how this plays out. Anyone have any experience with it?

Random

A nice description of WebGPU (think OpenGL in the browser) that puts it in a surprisingly comprehensive historical context. Long read [Like you should talk, Jonathan: ed] but 100% worth reading if high performance visualization in the browser is at all of interest.

Using ChatGPT to generate mermaid.js diagrams (which render in GitHub).

Good walkthrough of perfect hashing for strings, which is a very useful technique for building very succinct relatively static indices.

I’m deeply skeptical of “rewrite it in rust” (or Python, or go, or julia, or anything really), but sudo and su should probably be memory safe, yeah.

Github action to set up a matrix of fortran compilers for testing your fortran code.

That’s it…

And that’s it for another week. Let me know what you thought, or if you have anything you’d like to share about the newsletter or management. Just email me or reply to this newsletter if you get it in your inbox.

Have a great weekend, and good luck in the coming week with your research computing team,

Jonathan

About This Newsletter

Research computing - the intertwined streams of software development, systems, data management and analysis - is much more than technology. It’s teams, it’s communities, it’s product management - it’s people. It’s also one of the most important ways we can be supporting science, scholarship, and R&D today.

So research computing teams are too important to research to be managed poorly. But no one teaches us how to be effective managers and leaders in academia. We have an advantage, though - working in research collaborations have taught us the advanced management skills, but not the basics.

This newsletter focusses on providing new and experienced research computing and data managers the tools they need to be good managers without the stress, and to help their teams achieve great results and grow their careers.