Research Computing Teams #139, 1 Oct 2022
I got asked a question building on the last post about taking on a new project, and the current top unanswered poll.ly question is “I'm starting a new job! What should I do to ramp up quickly?”
Let me answer a slight generalization of those questions: say you have taken on an existing responsibility. It might be being promoted to manager, it might be a new job, it might being put in charge of some effort that already has some history. (And in our institutions, every effort already has some history. There are no completely de novo efforts in our line of work). What should you do?
(Read on, or go straight on to the roundup).
It’s mathematically possible, of course, that someone in our roles taking on a new responsibility has clear guidance from their boss, and is receiving routine and helpful support and coaching on on their efforts, so that they feel like they’re in good hands. Yes, yes, stop laughing, I said it was possible.
For the rest of us, figuring things out on our own, there are two common failure modes: freezing, or doing too much too soon. It’s important to fight both of those instincts.
My guidance in these situations is to keep things moving steadily, gather all the data you can, report back to everyone, and only then - likely three or so months later - start building a library of small quick wins that, hopefully, solve multiple problems.
The big picture here, the guide star to follow, is that someone taking on a new work responsibility for things that will affect people has two and only two initial tasks. Task one is to learn what the heck is going on; task two is to build trust and respect with the people their efforts will effect. You can't execute on your responsibilities unless you see them and the consequences of any actions clearly; and you can’t get anything significant and lasting done without the trust and at least grudging support of most of those you’ll be affecting. Everything else is tied for a dim and distant third.
First: Do no harm
So, congratulations! You’ve been dropped into some pre-existing human system, and are charged with executing on some new responsibilities.
Human systems are complex. Changes to complex systems always have multiple consequences. And the thing about complex systems is that unintended consequences are basically always bad. Until you know enough about the humans you’re dealing with and the related systems they’re part of, not only can you not guess ahead of time what the consequences of any changes will be, you won’t even know how to detect them when they do happen.
So your first job, while you’re learning things and building trust, is to keep the lights on, keep expected activities happening, and not make any changes yet. Learn what the steady state looks like first.
You might think (as I certainly did in an earlier role) that if you already have experience with this effort or this team — say you’ve just been promoted from IC to leadership — that you have the background knowledge necessary to skip this and the next step. I was, and you would be, very much mistaken. Indeed, I did and you would know just enough to be dangerous. There’s still a bunch of important context you’d be missing. Steady on.
There may be something that is obviously, dangerously on fire and leaving it alone for a while isn’t an option. My guidance here will be painful to hear for those of us with expertise and a desire to fix things. And yet. Unless you’ve been directed otherwise, do the smallest, most self-contained, most hacky thing that can possibly be done to keep the flames down to a containable smoulder for a while, make a note of what’s happening, and move on. You don’t know enough to fix this properly yet. You just don’t. Give the people around you some credit. If the fix was so obvious and easily done that someone just walking in could fix it, the thing wouldn’t be on fire right now.
Second: Listen, Listen, Listen
Your second job is to gather all the data you possibly can.
(I mean people data. We’re all experts here of one sort or another, so I’m not going to advise you to learn the relevant domain knowledge for this responsibility - we take that as a given, and frankly have probably already started doing that because we enjoy that part. I’m focussing on the messier, trickier, people stuff.)
If you’re part of a group of people, and there’s no practice of one-on-ones yet, start one-on-ones.
And start having one on one conversations with others. Create a list of key stakeholders, of teams you’ll be interacting with, of people your actions will affect, of people whose actions will affect you. Start working your way down that list, having one-on-one conversations with them.
In both cases, ask them questions about how the responsibility you’re now charged with has gone in the past, what’s important about it to them, what they worry about, what they think is under control, what kinds of interactions they’ve found valuable - keep them talking and take copious notes. Write down exact quotes of anything they say that illustrates a point particularly well.
Use all your active listening skills here - listen more, speak less; paraphrase what they’ve said for validation and clarification; ask followup questions; sympathize; don’t offer your own thoughts.
The task here is to gather all the data you possibly can. Fight against the temptation to interpret it right away – you can’t interpret a individual noisy data points. The work here is to gather lots of data points so that trend lines later emerge.
Remember that if a person says they experience a problem, they are absolutely correct about their experience. But anything else they offer (an assessment of the problem’s priority broadly for the community, a diagnosis of the underlying issue, or a proposal for a solution) is just their own interpretation filtered through their own perspective and likely lacks other important context, context you’re still collecting. If a number of people experience a problem in the same general area, there is obviously a very real problem somewhere around there that will eventually bear thinking about, but it might not be what they think it is. Collect the data impassively, gather it without interpreting it.
Third: Report Back
Report-back will be in a few different phases.
For external people you’re not talking weekly with, within a day or at most two send them a thank-you. Include with the thank you distilled and coherent notes from your meeting, ask them for feedback to correct or expand on anything in the notes, and invite them to follow up by email or another meeting if there’s anything else they’d like to cover.
This follow-up is absolutely entry-level professionalism stuff, but I’m continuously surprised by how uncommon a practice it is in our profession. Yes, it takes work. In my experience, distilling notes from a meeting into something other people could read and learn from takes roughly as long as the meeting did. But remember, the goals here are to learn what’s going on, and build trust. Sending lucid notes after your meeting and requesting feedback means organizing what you learned while the meeting is still fresh in mind, getting that validated by the person you learned it from, and communicating to them both that you took what they said seriously and that you follow through on conversations with them. That is what makes the meeting worthwhile (for both of you) in the first place!
If you’re working with a team, periodically report back distilled, anonymized versions of what you’re hearing from stakeholders. Making use of the direct quotes you jotted down is an extremely effective way of helping the stakeholders communicate a point. Once you’ve heard from all the team members, you can also summarize what you’ve heard from the team members back in an anonymized way. (Be careful with direct quotes in that context - people might recognize each other’s words). Don’t let the team react too strongly (positively or negatively) to what you’re reporting back, and don’t start making changes yet; the context here is that this is what people are saying, stuff to be aware of, and things to think about. Make it clear that this is part of an ongoing process, and as it winds up you’ll collectively decide what’s next.
When you’ve heard from all of the external voices, distill everything you’ve heard — still without interpretation or next steps — into an easy to read document. A slide deck works well for many, but it could be anything. As before, make generous use of (anonymized) direct quotes. Run it by your team if appropriate, and then send it individually out to everyone external you’ve spoken to, and then offer to schedule a time to walk through it with them and collect their feedback.
This has likely taken months. Ninety calendar days is a commonly cited length of time, enough to learn the lay of the land, and for people to start thinking “one of us” rather than “that new one who just started”. You now have a firm foundation to build from.
Fourth: Start building small multi-purpose changes
The first three steps are all about learning what’s going on and building trust. You build trust with people by listening to what they have to say, being seen to take it seriously, and following through on things. Those are all things you’ve started doing. Now you have enough background knowledge and trust built up to start getting some things done.
In the distillation of data you’ve done, some patterns have begun to emerge, and some likely priority areas taking shape. You’re already starting to see where systems interact, which people are affected by what area, and where the effects of small changes would have ripple effects.
What the next activities will look like will depend a lot on the nature of your new responsibilities. But in all cases, the goal is to identify some small wins. You’re still learning how systems interact, and still learning how to work with new people, so it’s baby steps first.
Find small changes that could be made that address more than one problem area, or more than one opportunity. You don’t want to start by making surface-level tweaks. Also, remember the law of unintended consequences! It’s likely that any change will cause its own downstream issues, and you don’t want to just shuffle problems from one area to the other. The goal is small changes that have noticeable effects on a few issues that a number of people have suggested.
Validate your proposed priority and approach with (at least) the affected people, and start making changes. Keep talking to people, and pay very close attention to how things are going. Once one change has been successfully made, and the consequences understood, report that around and begin with the second. Keep your lines of communication open, and start building lines of communication directly between stakeholders. Start building momentum.
Making things happen with a new responsibility is hard, but the key is to start with the foundation of knowledge and trust from other people. Put that work in, steadily and consistently, and big things slowly become possible. We too often overestimate what we can do in a week and underestimate what we can do in a year.
Some resources I really like for this topic (and if you’re familiar with them you will have seen their influence on what I’ve written above) include:
- The First 90 Days - Michael D. Watkins; mainly aimed at senior leadership, but much of the discussion here applies
- 90 Day New Job Plan Podcast Series - Manager-Tools; I particularly like the bits about taking on the role of “the voice of the customer” (and yes, you have customers)
- Hack Your Bureaucracy - Marina Nitze and Nick Sinai; just working my way through this now, but I think you’ll hear more about this book from me. It’s a very practical guide to getting change done in complex human systems. (And our human systems are more complex than most.)
I’d like to emphasize that even if you’ve had a responsibility for a while, it’s never really a bad time to go through this process. You can always find some excuse — a new academic or financial year, an upcoming project kickoff or wind-down, a new hire or departure — to collect a bunch of data, have conversations with stakeholders, and start making some new plans and changes.
Any thoughts? Anything I’ve missed that you’d add? Let me know - just hit reply or email me at jonathan@researchcomputingteams.org.
And with that, on to the roundup!
Managing Teams
Setting Direction: Create a Strategy - Chase Seibert
Siebert describes the basics of strategy, using 1997 Apple as an example and leaning heavily on Richard Rumelt’s “Good Strategy Bad Strategy” book:
He also selects a quote that is both correct and weirdly appreciated in our line of work:
Good strategy requires leaders who are willing and able to say no to a wide variety of actions and interests. Strategy is at least as much about what an organization does not do as it is about what it does.
Everything is tradeoffs. Any strategy is necessarily some form of specialization along some dimension. “We do everything” is not, and never can be, a strategy.
Rumelt’s two books are really quite thought provoking. (They’re more filled with anecdotes than I care for, but others find that an extremely helpful part of the book, so.) If you want to read a book on business strategy, one of his are the ones I’d recommend. But our context, where we (unlike Apple) can’t just pivot to something completely different, is a more constrained than business strategy writing generally allows for. La Piana’s book, “The Nonprofit Strategy Revolution,” is starting to grow on me for an RCD team situation.
The Coach and the Fixer - Michael Lopp
Lopp talks here about the tension between being the coach/listener, and being the fixer/one who takes action. One or the other of those is likely more comfortable for us, depending on the situation. The problem is, we as managers and leaders need to be able to call upon both as needed.
Technical Leadership
Why are process gates the hellish spawn of evil you should avoid at all costs? - Jade Rubick
When something fails, the perfectly understandable first reaction is to impose more oversight on whatever that process was, and to require some sort of quality control between the creation of the next thing and its release.
This approach is seductive, because from a management or lead point of view it feels like we’re Doing Something. And, you know, there are some times when these kinds of gates are needed. If you’re going to be updating medical devices, you'd best make absolutely sure the new versions work properly.
But as Rubick points out, these stage gates multiply, slow everything down, erode the sense of ownership of team members (since it’s pretty clear they’re not trusted to do their job). Our jobs as managers and leaders isn’t strict oversight, it’s to build a team and processes by which good outcomes happen without our intervention.
As in Silverio’s article in #134 says, the key is to make sure any bit of process function has a clear purpose, and to constantly review them make sure they’re needed and that they’re the best way of achieving their aims.
Managing Your Own Career
Being an Introverted Leader - Matheus Tait, Thoughtworks
We’ve talked about this on and off since way back in #4, but it’s worth repeating. As a manager or lead at various times you’ll need to talk and to listen, to be assertive and to let others take the lead, to be detail-oriented and to see the big picture, to be people focussed and to be task focussed. There is no one whose default behaviours covers all of what’s needed, because different situations call for different and even conflicting behaviours.
Here Tait talks about his own struggle with being an introverted leader, and his first three suggestions could be just as easily applied to any personality trait:
- It is not a problem to be solved
- Capitalize on your strengths
- Recharge moments
Product Management and Working with Research Communities
Community Smells -- The Sources of Social Debt: A Systematic Literature Review - Eduardo Cabllero-Espinosa, Jeffrey C. Carver, Kimberly Stowers, arXiv:2209.10671
This is an interesting paper where authors review the literature for issues with communities in charge of stewarding some technical work (here mostly software, but it could just as easily be a data resource or something else). They distill thirty or so identified issues down into four main causes (lack of context, poor communication, the wrong composition of teams, poor coordination) along with some lesser ones, and suggest a list of community and project management approaches that can improve things.
Research Software Development
Software engineering practices - Simon Willison
Inspired by a twitter thread, Willison writes down some good practices that he’s found useful in software development teams.
The list is good, and I think it’s worth highlighting that the recommendations are all of the form “make it easy to do the right thing” (documentation alongside code, infrastructure for DB migrations, templates for new work) and/or automation (code formatting checks, test data generation, new dev environments, preview environments).
Ultimately, these support-the-team’s-progress sorts of practices matter much more than particular technical choices that we tend to bikeshed.
Rethinking Software Variants - Axel Hubel
Hubel lists some of the problems with the developer (and user) experience with software that has compile-time options to control functionality (e.g. turning on parallel support). Suggestions for improving things fall into two categories:
- Options should extend functionality only, not disable or switch between funcitonality
- Use run-time and compile time functionality together
In the HPC world, where we assume everyone will be compiling our code, I think we let ourselves get a little lazy this way. Being more disciplined about how we enable code extensions while maintaining API stability (particularly for libraries) definitely matters! People will (hopefully) be building on our code and recompiling it, and maintaining ease of developer experience matters.
But it’s also increasingly straightforward to build in multiple functionalities at compile time, shipping binaries and handling functionality changes on different platforms at run time. (Think of all the e.g. math libraries that do just this). Not every package needs or is worth that level of more sophisticated development tooling, but the more we want our packages to be used, the easier we might want it to be for someone to download a binary that ‘just works’.
Research Data Management and Analysis
Data Management Planning for an Eight-Institution, Multi-Year Research Project - Kristin A. Briney, Abigail Goben, Kyle M.L. Jones, Int. J. Dig. Curation 17:1
This is a nice writeup of the hard data management work that usually goes unheralded and undescribed in big collaborations. Not only does that mean credit isn’t given, but we miss out on the opportunity to learn what others have done. Unsurprisingly, data management plans can and should be more than just compliance tick-the-box exercises. Quoting from the conclusion:
[..] First, the team used DMPs as living documents that were regularly refined to meet the project’s and researchers’ needs; if something in the DMP wasn’t working, it was changed. Second, the project leveraged multiple DMPs; each project phase had distinct data collection and analysis methods, making individual DMPs beneficial. Finally, the project’s DMPs were vital to coordinating work done by multiple people in different locations; the large and distributed project required consistent data handling and the DMPs facilitated this. The project’s four DMPs augmented the funding application, streamlined data collection, documented analysis workflows, and made data handling uniform, allowing researchers to focus on research instead of being mired in the data problems that can easily arise in a multi-site, multi-year research project
Research Computing Systems
We’re going to see more and more laws and regulations - and even just community expectations - about prompt and full disclosures of breaches in our systems, and I think a lot of our community isn’t ready for that yet.
Emerging Technologies and Practices
Handling WebAuthn over remote SSH connections - Matthew Garret
How Passwordless Works - Alan Parra, Teleport
I strongly suspect our community is going to see more and more use of WebAuthn, and that means we’re going to have to figure out how it interoperates with other login mechanisms. Garret an initial exploration of trying to get the two to work together, and Parra has a nice overview of how WebAuthn works.
Random
A pastebin that you can netcat to from the command line - termbin.
You can play old video games I grew up with like asteroids, centipede, breakout, snake, and missile command in the web browser now, of course - including on the American Association of Retired Persons website. Just a click or two away from the “staying mentally sharp” page. sigh.
Grep pdfs with pdfgrep.
The old numerical-methods hand and number-theory dilettante in me really likes this: reals, a Python package for arbitrary precision arithmetic that only prints digits that are accurate, based on interval arithmetic and continued fractions.
In other cool python package news, a seemingly quite good tool for making modernization recommendations for Python code bases, refurb. The things you can do with static code analysis tools these days continues to impress.
Launch a docker container image as a (firecracker) VM with ignite.
Building a self-destructing USB drive.
Challenges with UNIX signals - the 15 year challenge to fix an old GDB user experience issue when signals are caught and the dangers of using signals in production when they’re not.
That’s it…
And that’s it for another week. Let me know what you thought, or if you have anything you’d like to share about the newsletter or management. Just email me or reply to this newsletter if you get it in your inbox.
Have a great weekend, and good luck in the coming week with your research computing team,
Jonathan
About This Newsletter
Research computing - the intertwined streams of software development, systems, data management and analysis - is much more than technology. It’s teams, it’s communities, it’s product management - it’s people. It’s also one of the most important ways we can be supporting science, scholarship, and R&D today.
So research computing teams are too important to research to be managed poorly. But no one teaches us how to be effective managers and leaders in academia. We have an advantage, though - working in research collaborations have taught us the advanced management skills, but not the basics.
This newsletter focusses on providing new and experienced research computing and data managers the tools they need to be good managers without the stress, and to help their teams achieve great results and grow their careers.
Jobs Leading Research Computing Teams
This week’s new-listing highlights are below in the email edition; the full listing of 207 jobs is, as ever, available on the job board.
Director Advanced Data and Storage Management (HPC) - Princeton University, Princeton NJ
The Director of Advanced Data and Storage Management reports to the Associate CIO for Research Computing and manages the group responsible for the vision, design and support of data storage and management for advancing innovative research at Princeton University. This role will provide leadership to the implementation and support of the TigerData service, a comprehensive set of data storage and management tools and services that provides storage capacity, reliability, functionality, and performance to campus. To successfully implement TigerData, this role will closely partner with the Director of Research Data and Open Scholarship in the Princeton University Library.
Research Manager-Data Science - Inficon, Syracuse NY USA
INFICON's Data Science Research group in Syracuse NY conducts research to analyze data from sensors and to identify events of critical importance to our customers. The group develops algorithms to process faint signals buried in noisy spectral data to help build advanced semiconductors, predict failures, and protect people's lives from toxic chemicals.
Associate Director, Bioinformatics Software Engineering - Quest Diagnostics, Remote USA
The Associate Director, Software Engineering Bioinformatics will support and lead programs in technology and software engineering that will help Quest become the global leader in clinical software technology. This role is partnering with the Global Bioinformatics Leadership team and serves as technology thought partner for Quest’s Advanced Diagnostics (ADx) franchises and operations, defining the vision, strategy and roadmap for software used for our ADx operations. Required is an experienced and transformational leader, with a strong track record in leading technology and software teams and utilizing agile methodologies
Regional Director, Health Information And Informatics - Eastern Health, St John's NL CA
The Regional Director for Health Information Services & Informatics Department (HIS&I) is a member of the senior management team and is accountable for providing leadership, vision and strategic direction to the HIS&I department which has a regional and/or provincial scope and reach in the areas of: continued development of electronic health records, health information management, privacy, confidentiality and security of health information, supporting the access to and utilization of health information, management of health records in multiple media, support in analysis of clinical information, education in the content and utilization of clinical information, registration and admission, support for research.
Lead Bioinformatician - Research Services - Genomics England, London UK
Our Lead Bioinformatician – Research Services is an experienced bioinformatician with strong competencies in genomic data analysis and research support. Through proactive collaborations and agile practices, the Lead position brings technical leadership and people management capacity within the chapter and across different squads/teams
Lead Data Architect, Analytics - St Jude Children's Research Hospital, Memphis TN USA
The Lead Data Architect is key to supporting a wide range of operational data needs, including setting the standards for data architecture, data ingestion, data modeling, security and governance. You will excel at this job through collaborating with customers to truly understand their data needs, putting yourself in their shoes, thinking a few steps ahead of them in building a high-quality data environment that supports quick delivery of analytics to them. You will also help mature the overall data and analytics technology platform and service delivery strategy that aligns with St. Jude priorities and promote an institution-wide data-driven culture
Senior IT Manager, Koch Institute for Integrative Cancer Research - MIT, Boston MA USA
Koch Institute for Integrative Cancer Research, to direct and manage the infrastructure, support, maintenance, and operations of IT systems, hardware and software acquisition and renewal, research computing, information security, collaboration platforms, and support. Will build and manage a diverse team including systems administrators, endpoint support engineers, database administrators, and technology consultants; and support a wide range of technology and services while providing proactive and effective technical strategy and planning to a diverse customer base.
Director, Research Computing - Lehigh University, Bethlehem PA USA
Provide leadership and strategic direction for the Library and Technology Services (LTS) Research Team. Partner with LTS, University leadership, Faculty and Staff in the development of grants, plans and initiatives for the expansion and sustainability of research computing at the University. Direct and advise on the development and support of research systems and applications that deliver on the prioritized requirements of researchers and University partners. Assist faculty, staff, students, and other researchers in the use of research and research computing systems and provide technical expertise relating to scientific software, hardware, and related utilities and equipment.
Head of Research Computing and Support Services - University of Oxford, Oxford UK
Reporting to the Director of Infrastructure Services, the Head of Research Computing and Support Services is the head of a significant delivery group supporting high performance and high throughput compute platforms and delivering research support services to staff and students across all four academic divisions of the University. You will play a leading role in the management of IT Services with key responsibilities in the development of the University’s long-term IT strategic plan for research support.
Director of Imaging and Informatics - Children's Hopsital of Philadelphia, Philadelphia PA USA
In this newly created role, the Director of Imaging and Informatics will be responsible for providing leadership and oversight for the key areas of Imaging and Informatics at CHOP. The primary users of the imaging services are the pediatric radiology faculty along with clinicians from a variety of divisions who rely on tools like the VNA, PACS as well as the support team to provide timely care to the pediatric population. The successful candidate will work closely with executive leadership, clinical informatics, information services, radiologists, researchers, educators, and other departments at CHOP to lead departmental imaging and informatics efforts, provide expert imaging consultative advice and facilitate service excellence. The successful candidate will have experience leading change in complex healthcare environments and proficiency in imaging and informatics.
Research Software Engineer Team Lead - University of York, York UK
Be part of this exciting team where you will have the opportunity to engage with academics across the university facilitating both research and teaching. As part of the University of York Research IT you lead the Research Software Engineering (RSE) team. You will also provide research IT support to students and staff throughout the university with time to participate in grand funded research projects.
Engineering Manager, Learning R&D - Duolingo, Pittsburgh PA USA
The Learning R&D Area works on projects that aim to improve how well we teach languages on Duolingo. This area is responsible for the core learning experience, from optimizing and personalizing the bite-sized lessons that are iconic to Duolingo, to investing in newer features like short stories to improve reading comprehension, ways to learn new scripts and alphabets, and giving learners pronunciation feedback. You will play a significant role in making sure Duolingo delivers on its promise to teach you a language both a fun and effective way!
Manager R&D, Ansys Cloud - Ansys, Waterloo ON or Montreal QC or Vancouver BC CA
The manager R&D joins the SPBU to lead the software development of a cloud offer expanding the platform capabilities of Ansys applications. This role is responsible for planning, resource assignments, and progress tracking of all activities performed by the team members of his/her team. You will be responsible for line management and personal development of your team members, as well as the facilitation of the team's work.