Research Computing Teams #147, 26 Nov 2022
I spent much of this week helping teach a small part of EMBL-EBI’s excellent Managing a Bioinformatics Core Facility course (Here’s the 2021 public materials). It was my first time participating, and I was really impressed. There were a large number of new, motivated, engaged core facility leads; the workshop was very interactive, with multiple exercises every day, lots of group discussion and sharing of experience, and a lot of material covered.
The audience being leaders of core facilities, much more time was spent on cost recovery models than would be common in research computing and data teams (there was even one brief section which touched on hourly vs fixed-price billing, which was fantastic to see - readers will know where I stand on this, e.g. #140). There was also a terrific series of exercises on service design, which also isn’t common to discuss in our circles but really should be.
But it was fascinating to see how similar the fundamental issues were. Many of the questions which came up would be very familiar to you, reader — hiring and retention, motivating staff, working with recalcitrant researchers, dealing with benign neglect from leadership, demonstrating impact, improving execution, prioritizing efforts, juggling the needs of a diverse and growing research community.
I think it’s pretty clear that in the RCD community we’re currently about five years behind our core facility colleagues on issues of building a community of practice and professional development, certainly for managers and leaders. Honestly, this is great news! We were much further behind a decade ago. The tireless and thankless work of groups like CaRCC and CASC have been pushing us ahead in terms of building a community of practice around research computing systems in particular, and in building an IC career ladder framework; and of course the RSE movement has been great for building similar things at least at the IC level for software.
I think we need to do more, in particular across RCD silos, but the fact that there’s been as much progress as there has been is remarkable and a testament to a lot of unsung work by a large number of people.
Still, it’s interesting to think about what we should be teaching new RCD managers. I have my usual 10-minute “Help, I’m a manager!” talks, but what would we cover in a half-week course?
My own ideas, pictured above, wouldn’t be surprising - it’s the topics covered here, plus funding (which is a bit too jurisdiction-specific to cover in this newsletter). People, projects, products, funding, strategy. Maybe also a session on communications/marketing/positioning.
What do you think? What would have helped you when you started, or what do you see newer colleagues struggling with? Are there other topics that would have to be covered? Things that could be cut that are covered well elsewhere? Let me know - hit reply or email jonathan@researchcomputingteams.org.
One last note on the EMBL-EBI course - the short section I covered was a crash course into the 20% of project management that gets you 80% of the benefit. I had what I figure are nuggets of hard-won wisdom gold in there, but what really captured the attention of attendees was routinely having retrospectives after projects, and continually improving. (Core facilities execute on a lot of projects, with each service provided being a project).
It’s pretty easy to underestimate the power of continual incremental improvement. These new leaders saw what I think more experienced and jaded leaders can sometimes be more hard-pressed to see: things can always fairly easily get a little bit better, and sustaining those small improvements over time compounds into something remarkable.
The lovely thing about retrospectives is that once you start taking action on suggestions (whether from the team or the researcher client) there’s a virtuous cycle. People are more willing to make suggestions when they see that suggestions actually matter; it builds trust and confidence, in addition to the improvements it brings. It boosts engagement because people feel more ownership over something they have influence over. People feel respected and more effective.
And internal retrospectives help the team decide collectively how they want to work. Issues with processes or conventions inside the team will naturally come up, and a good facilitator can help the team trial new processes.
Retrospectives are an under appreciated tool, and they don’t have to be a project management thing; they’re something that can happen regularly in any team.
Finally, just a small housekeeping item: we’ll have newsletter issues Dec 3, Dec 10, and Dec 17 as usual, and then RCT will settle in for a long winter’s nap. We’ll start up again Jan 14.
And with that, on to the roundup!
Managing Teams
Comments: Consider working on genomics - Hacker News
Consider Working On Genomics - Clay McLeod
I’ve been writing a bit on recruiting and hiring lately, and I’ve mentioned before that there’s a bit of a sense of helplessness in some RCD team leaders; they can’t compete with salary, so why even try attracting staff from outside of academia?
I’m not sure if this post is going to make people feel more or less helpless, but it will at least point out that salary is not the only or even main thing that matters to most of our candidates.
I don’t generally link to Hacker News - frankly, the comment section can be pretty toxic. If you aren’t familiar, the contributors here are by and large tech workers (software, systems, data scientists…), largely but certainly not exclusively from industry. “The Orange Website”s failure mode is that it can be very tech-bro-y.
But the discussion here is engaged and on-topic and fascinating. It’s in response to McLeod’s article, urging software developers to consider spending some time working in companies and research institutes working on genomics. McLeod points out, correctly, that there’s lots of great work to be done and not enough people doing it.
The hacker news thread is likely eye-opening if you haven’t heard these discussions from this audience before. It’s great to get a chance to “listen in” to a community having this conversation.
The comments from people who have tried working in research before moving to industry are scathing. Most poignantly, there’s a “more-in-sorrow-than-anger” feel to much of the discussion, coming from people who would have very much been willing to take up a career (or at least part of one) supporting research.
Salary certainly comes up, but it’s usually mentioned as an aggravating factor, something along the lines of “there’s no way I’m going to put up with X when they’re paying me so little”.
Coming up much more often is:
- Lack of respect for staff from researchers
- Lack of autonomy
- Poor tooling
- Job security
- Poor state of code/infrastructure, with little appetite to improve things
- Isolation from the research work: The largest centres silo production development from research in a way that likely makes things more efficient but makes the job much less interesting
We can flip these around and cultivate working environments that are attractive:
- Close interactions with the real research work
- Autonomy within their domain
- Providing decent tooling
- Respecting staff and their work output
- Respecting their opinions when they say something needs to be improved
How to… make people happy - Ethan Mollick
Notes of Appreciation Can Boost Individual and Team Morale - Whitney Johnson and Amy Humble, HBR
I’ve mentioned research before (#64, #112) that there’s basically no plausible amount of gratitude we can express, to team members or broader community members, which is too much. Saying thank you in person, or in a short written note, takes approximately zero effort and yet is extremely impactful.
In celebration of US Thanksgiving, Mollick (a prof at Wharton) summarizes a paper on written expressions of gratitude:
…[T]his paper shows we undervalue showing gratitude. We think it will be awkward, we think people know we are grateful, we think it won’t matter much. All of that is wrong. People who were asked to write letters of gratitude to other people overestimated the awkwardness of the experience, and underestimated the impact on the recipient’s mood and happiness.
There are other papers summarized, as well:
- One points out that we also think complimenting people will be unappreciated and awkward, and it isn’t.
- Maybe particularly appropriately for our line of work, another paper demonstrates that even if we can’t completely solve someone’s problem, people appreciate partial help almost as much as full help.
- And a final paper covers spontaneously contacting someone just to say “hi” and catching up. Again, we overestimate how awkward and unappreciated it would be, where in fact it is generally appreciated, whether it’s someone we know well (strong tie) or less (weak tie).
Similarly, Johnson and Humble describe work they do at their leadership retreats, with everyone charged with writing notes of appreciation. They describe the results:
- They help the recipients see their strengths
- They focus the sender and the receiver’s attention on what’s working
- They signal to people th atthey matter
Product Management and Working with Research Communities
How to redesign a scientific website in three simple steps with limited budget and time - Nikiforos Karamanis
One of the sessions in the core facility manager training was a very effective set of exercises on service design. The exercises themselves were enough to serve as gentle nudges to better thinking of interactions with the service from the users point of view.
The instructor shared a blog post he wrote using a stripped-down method of the same approach to resigning a scientific website, with (basically) no budget and a very modest investment in time:
- Write a content model for the site
- Identify audiences, audience goals, and our goals
- Identify where people would be coming from to visit the site, what they’d want, and what they’d do afterwards
- Observe or interview a small number of target users, develop a persona
- Explore alternative designs using pen and paper based on the content model
- Develop “higher fidelity” sketches of a subset of designs
- Get feedback on the designs and iterate
The post is worth reading. It’s so easy to get caught up in how things are and our own internal goals. Going through this process confronts us with what the site visitor/user’s goals, and how things could be.
University Finances 2021-22 - Alex Usher, Higher Education Strategy Associates
It’s always worth paying attention to trends in our institutional finances. While our kinds of organizations typically aren’t buffeted as wildly by economic forces as others are, they aren’t impervious to them, either.
This article focusses on Canadian universities, but the pattern (if not the exact timing) is playing out elsewhere and in other kinds of R&D environments, too:
Income from government fell by 9% in real terms in fiscal 2022 – partly the result of the withdrawal of emergency COVID funding and partly the result of inflation. […] In any event: universities didn’t do too badly as a group in 2021-22 but there are good reasons to think that 2022-23 will be substantially worse as income sources lag inflation.
The broad strokes were foreseeable (and foreseen) some time ago. Certainly, experienced nonprofit leaders saw this coming. When sharp downturns happen, emergency funding (from government and/or donors) is followed by retrenchment, and that retrenchment basically always happens well before the downturn is completely over. Today’s inflation and geopolitical uncertainty amplify the impact of this back-swing of the pendulum.
As institutional budgets tighten, teams that can not clearly communicate their value to core institutional missions in ways that decision makers care about are going to have a bad time. Large regional centres are probably going to be ok, and few-person teams are probably going to be too small to attract much attention. Medium-sized generalist teams in public institutions are going to have large enough budget lines to catch the eye, without necessarily having a ready-made compelling narrative supporting their work.
If you’re in that situation, you probably have multiple decision makers that affect your budget. It isn’t too late to start talking with them, to make sure they can be armed with justifications they individually find compelling for your team’s budget and impact, should tighter budgets come. For some, that compelling justification will be quantitative, like grant numbers. For others it will be qualitative evidence of you enabling research projects they personally prioritized, maybe with a few images and pull quotes from the researchers involved. Others may value something else entirely (Contribution to the education mission? Workforce development? Community outreach?) It’s only through conversation that you’ll know what matters to them.
The goal is to have one or more decision makers actively championing your work, and for the rest to be at least lukewarmly supportive enough not to advocate cutting your budget.
Does that make sense? Do your decision makers have other things they care about? Are there areas you’re concerned about? Hit reply or email me at jonathan@researchcomputingteams.org to share your thoughts or to ask me questions.
Emerging Technologies and Practices
AI4Science report - CSIRO
Australia’s CSIRO has a nice report covering the increasing role AI methods broadly are playing in the sciences. There are some nice plots by field, and some (Australian) highlights.
Section 6 covers what will have to be done to continue to support this, with implications for research organizations: increasing software, hardware, and open access resources; better data; education & training; better workforce diversity; and likely upcoming ethics and regulatory issues.
This report could be a useful starting point for advocacy with your own decision makers, as our teams start wrestling with the increasing AI needs and workloads in our research communities.
Random
Oh My GitHub - work with GitHub (including creating PRs) from within Emacs, if you’re into that. I won’t judge. (I lied, I’m 100% judging).
You like having nice clean git history? How about having your commit hashes listed as “00000000”, “0000001”, “0000002”? (Ah, I miss SVN). Now you can, with extremely linear git history.
Meta is open-sourcing their new git-compatible source control client, sapling. Interesting to see what can be done without sacrificing existing git tooling.
RSS meets Usenet: rssnix, collecting RSS/Atom/JSON feeds to the local filesystem, with a browser.
Enjoyed geoguessr? Codeguessr shows you code snippets from one of the top 200 most starred GitHub repo and asks you to guess the project.
Another datapoint that “modern”, “innovative” workspaces (open concept, beanbags, etc) don’t promote creativity or collaboration.
Pretty good argument about why FOSS so often has bad UI/UX - the very distributed decision making that comes with successful FOSS projects are at odds with the holistic, coherent design process that UI/UX benefits from.
Nice to see projects like Flux that try to extend Rust’s weirdly narrow definition of code safety, to include focus on correctness with Ada-like pre-post conditions and refinement types.
With SSDs and much higher-speed interconnects, old assumptions like “I/O is slow” isn’t true for local disk streaming access and hasn’t been for years. (This IMHO is the biggest difference between HPC with parallel POSIX filesystems and workstation computations - not MPI, not scaling concerns).
Good quick introduction of of Kubernetes, RDMA and OpenStack using ROCE and SR-IOV.
I’m unreasonably excited about advent of code starting soon.
That’s it…
And that’s it for another week. Let me know what you thought, or if you have anything you’d like to share about the newsletter or management. Just email me or reply to this newsletter if you get it in your inbox.
Have a great weekend, and good luck in the coming week with your research computing team,
Jonathan
About This Newsletter
Research computing - the intertwined streams of software development, systems, data management and analysis - is much more than technology. It’s teams, it’s communities, it’s product management - it’s people. It’s also one of the most important ways we can be supporting science, scholarship, and R&D today.
So research computing teams are too important to research to be managed poorly. But no one teaches us how to be effective managers and leaders in academia. We have an advantage, though - working in research collaborations have taught us the advanced management skills, but not the basics.
This newsletter focusses on providing new and experienced research computing and data managers the tools they need to be good managers without the stress, and to help their teams achieve great results and grow their careers.
Jobs Leading Research Computing Teams
This week’s new-listing highlights are below in the email edition; the full listing of 192 jobs is, as ever, available on the job board.
Assistant Director, Center for Statistics and Machine Learning - Princeton University, Princeton NJ USA
The Center for Statistics and Machine Learning (CSML) was established in 2014 as Princeton University’s hub for education and research activities in statistics, machine learning, and the data sciences. CSML seeks a professional experienced in the research environment to manage the operations and strategic growth of the Center. The assistant director represents the Center and the Director, serving as a liaison both internally and externally. The assistant director reports to the Director, with a secondary reporting relationship to the Senior Manager for Academic Administration in the Office of Human Resources.
Technical Project Lead, HPC Initiatives - Oak Ridge National Laboratory, Oak Ridge TN USA
The HPC Custers Group in the National Center of Computational Sciences Division (NCCS) at Oak Ridge National Laboratory (ORNL) is seeking a highly motivated, customer-centric, and operationally minded individual to serve as Technical Project Lead to provide support to HPC initiatives supporting collaborative research projects with external partners. You will work closely with the line management of NCCS to manage the milestones of the research, development, and production projects within the group to support division efforts. You will manage the scope, schedule, and cost of the various projects within the constraints of the available resources and the operational environment. You will have a direct interface with the project/program sponsor and department oversight personnel, responding to requests for information, reporting on progress, leading review meetings, and generating regular sponsor reports. You will champion new initiatives and lead strategic planning activities to enhance and grow the HPC portfolio.
Principal Researcher - Machine Learning - Microsoft Research - Microsoft, Montreal QC CA
We seek outstanding applicants for Principal Researcher positions at Microsoft Research Montreal. Applicants must have the passion and ability to craft and pursue an independent research program to further our understanding of deep learning methods. A strong long-term research program in a relevant field demonstrated by publications in top-tier conferences/journals. A demonstrated ability to shape a research vision for a team or the community, provide technical leadership, and deliver on that vision.
Lead, Software Engineering, Decision Management Platform - Mastercard, Vancouver BC CA
Do you want to be part of a team which helps prevent fraud on every mastercard transaction in this world? The Decision Management program enables intelligent decision based products through streaming analytics with the ability to govern these decisions and manage their outcomes with business agility. This program leverages business rules & AI engines, a streaming big data cluster, an in memory data grids, APIs, & UIs to deliver real time decisions at global scale
Research Infrastructure Team Lead, Data Safe Haven - Morgan Hunt (recruiter), Dundee UK
The organisation has run a Safe Haven environment and a secure infrastructure for research and clinical data collection for over a decade. Safe Havens are used across the UK for statistical investigation of sensitive data. The Organisation are researching and implementing a new cloud-based infrastructure which enhances this functionality to be scalable, support big data such as genomic and imaging data, allow programming within the environment and support AI and Machine Learning. The role holder will provide expertise and be part of the team implementing this next generation Safe Haven environment and secure research computing platform. They will have the opportunity to develop it in line with future requirements, while also contributing to the smooth running of the infrastructure operations of a 50-person research group.
Engineering Manager – Data-Driven Algorithms and Research Section - Aerospace Corporation, Various or Remote USA
The Data Science and Artificial Intelligence Department (DSAID) seeks a creative and enthusiastic Engineering Manager to lead a diverse team of engineers, data scientists, and programmers with a passion for researching, prototyping, understanding, and building AI and data enabled tools across the space enterprise. We are a growing, innovative, and collaborative department that makes meaningful contributions across National Security Space (AF, NRO, etc.), and Civil and Commercial customers (NASA, MDA, DHS, commercial space, autonomous vehicles, etc.).
Research Computing and Data Platforms Manager - NIWA - New Zealand eScience Infrastructure, Wellington NZ
The Research Computing and Data Platforms team is responsible for ensuring that NIWA and NeSI’s research computing infrastructure, platforms, and associated services are operating reliably, efficiently, and are effective at meeting the needs of end-users and stakeholders. The Research Computing and Data Platforms Manager has operational management responsibilities for the team and HPCF, and contributes to the development of the current research computing and data systems, informing and implementing the policies under which they operate, supporting future developments and procurement, and developing and delivering the services that underpin NIWA research, forecasting operations, and commercial services.
Chief of Staff, San Diego Supercomputer Center - University of California San Diego, San Diego CA USA
This position serves as Chief of Staff and as a key strategist at the San Diego Supercomputer Center (SDSC). Both are responsible for management and leadership of the center in alignment with the SDSC Director and UCSD campus leadership. Along with the Director and Deputy Director for HPC and Industry, provides overall strategic and managerial leadership and direction. Is a Technical leader with a high degree of knowledge in multiple areas such as HPC, data science, and Cyberinfrastructure. Develops long and short term plans for the center, including innovative initiatives, setting priorities, establishing center-wide budgets with CAO, and being responsible for efforts to achieve SDSC’s overall goals. Represent organization and leadership at national and international meetings, conferences and committees.
Research Data Architect, RENCI - University of North Carolina at Chapel Hill, Chapel Hill NC USA
Renaissance Computing Institute (RENCI) is looking for a Research Data Architect to support and promote data interoperability, contribute to the development of informatics tools and the analysis of complex data sets. This position primarily supports biomedical and environmental research projects. The Research Data Architect will work with colleagues with expertise in data analytics, advanced computing including cloud computing systems, software engineering and domain expertise in the biomedical and environmental sciences.. The Research Data Architect will work in interdisciplinary teams, promoting innovation and collaboration.
Neurosciences Data Architect, Wu Tsai Neurosciences Institute - Stanford, Stanford CA USA
This position will be situated in the Wu Tsai Neurosciences institute, reporting to and supported by Research IT & Innovation, overseen by a faculty lead who is an expert in research data and computation. The Neural Data Architect will work with research teams affiliated with the Wu Tsai Neurosciences Institute to develop efficient data management best practices and workflows.
Lead Data Manager - Cytel, Waltham MA USA
Cytel is the largest provider of statistical software and advanced analytics for clinical trial design and execution. This is a Lead Data Manager level position with technical leadership and hands-on components. The Lead Data Manager (LDM) has a keen attention for detail and is responsible for overseeing the start-up and execution of several trials with a CRO or in-house to ensure data quality and integrity. LDMs will independently lead multiple, high volume and extremely complex studies within a development program.
Software Development Manager - Canopy Biosciences, St Louis MO USA or Leipzig or Hannover DE
At Canopy Biosciences, a Bruker Company, we develop high-end microscopy for advanced spatial profiling research in biology and medicine.The Canopy Biosciences group is looking for a talented Software Development Manager to join our software team and be an integral player in designing and developing our world leading multiplex spatial profiling technology. You will be responsible for managing and leading a team of data scientists and engineers on the full spectrum of software development from hardware interfaces and instrument control to software application design for interacting with the user. You will design, program, analyze, debug, and modify software enhancements for new tools based on spatial profiling technology.
Chief Cloud Solutions Architect - Pacific Northwest National Laboratory, Richland WA USA
We’re hiring a Chief Cloud Architect to help our sponsors advance their mission in ways never thought possible using cloud computing, while also collaborating with our partners to develop technical expertise and innovation. This position will lead proof-of-concept pilots, technical leadership workshops, and strategic implementation projects. It will be a strategic consultant to our research divisions and their engagements with government sponsors with focus on critical customer solutions ranging from web applications, enterprise applications, HPC, big data analytics, AI/ML integration, and disaster recovery.
Director, Operations - Netherlands eScience Center, Amsterdam NL
The Director Operations is responsible for all financial and legal matters at the eScience Center, administrative processes around our project portfolio, housing and office IT, and human resource management. The candidate steers a small team of financial and human resource advisors and is part of the management team of the eScience Center. The candidate makes sure the internal processes run smoothly and has the final responsibility on the administrative and human resources matters. Together with other members of the management team strategic directions are developed and the implementation is overseen. We are looking for an enthusiastic team player who will contribute to a growing and flexible organization with many stakeholders. Our candidate should have proven experience with managing financial matters and human resources and managing a team. Experience with scientific research and digital technologies is an advantage.
Assistant Managing Director - High Performance Computing - Texas Tech, Lubbock TX USA
Texas Tech University’s High Performance Computing Center (HPCC) promotes and supports research and teaching by integrating leading-edge high performance computing and data processing resources for faculty, staff, and students. Assists the Managing Director with the management, directing and daily operations of all High Performance Computing Center (HPCC) services and resources. Responsible for management and direction of supervised staff. Supervise daily resource operations for the HPCC.
IT Manager - International Computer Science Institute, Berkeley CA USA
We are seeking to hire an IT Manager to manage and oversee ICSI’s operational IT and scientific cyberinfrastructure and related partnerships. We are looking for an individual who can balance technology proficiency with project management expertise, and will establish a solid reputation inside the organization as a partner and trusted advisor. The ideal candidate will be comfortable with cloud computing and virtualization technologies and have experience engaging Managed Service Providers and IT outsourcing services. In addition, they will have the ability to influence and communication skills necessary to function as a change agent.
Head of Software - IQM, Espoo FI
We are looking for a visionary Head of Software to take the responsibility of leading IQM’s existing cross-location software teams and making sure we will succeed in our goal to achieve quantum advantage. You would be taking care of the current and future software workstreams that include quantum computer and control software and infrastructure, R&D Data and Analytics, and Agile. The important part of the role is the vision of how to build the Software organization and attract top talent to join the talented team.
Technical Product Manager - Horizon Quantum Computing, Singapore SG
We are seeking an experienced Technical Product Manager to help us build software development tools to support the emerging field of quantum computing. You will develop and execute on the strategy and technical roadmap for our product, and work closely with our science, engineering, and design teams to ensure the successful integration of our product with partners and users. You should have a strong technical or science background, product sense, design thinking, strategic problem solving, and communication skills.
Product Manager, Quantum Applications - IonQ, Seattle WA USA
We are looking for a highly capable and independent Product Manager to help define the future of applications in this new quantum era. As a member of the product organization, you’ll be part of a cross-functional team whose mission is to lead IonQ on its journey to build the world’s best quantum computing solutions to solve the world’s most complex problems.. Quantum computers and quantum applications rely on cutting edge technologies across diverse engineering disciplines, including Machine Learning, Artificial Intelligence, data analysis, and a hybrid execution model that relies on both quantum computers and classical computing resources such as High Performance Computing. Having interest and experience in these fields will serve you well at IonQ, but we don’t expect you to be an expert in any of them.