Thanks, everyone, for your responses last week.
I’ve been thinking a lot about strategy in other contexts lately - some of you will have noticed that I’ve been back on my nonsense on twitter, about the importance of having a focus. The very insightful comments and suggestions you sent last week about how we can help more research computing teams were very on point, and I think combine to a feasible strategy.
Because within this newsletter community we’ve built together, we have a number of strengths:
And the wide underserved community of reseach computing and data team managers, technical leads, and those thinking of becoming one, could really use our help.
For areas to focus on, readers mentioned the lack of resources available for new managers, especially those put in the hard position of managing yesterday’s peers; lack of ongoing mentorship; and near total absence of people talking about management in a way that sounds relatable to research computing professionals.
There was wide willingness to contribute:
And some suggestions for things we could contribute together that would be valuable:
So if we have a goal of helping other research computing teams and their managers by providing material and mentorship relevant to them, with priorities around building resources that others can easily use and the tools for others to reach out to local communities, I think the next step is to start making a plan of action.
I’ll be speaking with some people in the coming weeks trying to put together some plans. If this sounds like somethign that would be of interest to you, as an observer or a active participant, or can think of other ways this community could help other research computing teams, jmail me - hit reply or email firstname.lastname@example.org - or arrange a fifteen minute call with me. Either way, I’d love to hear from you.
And now, the roundup!
4 Practical Steps For Strategic Planning As A Leader - Anthony Boyd
One hard things for new leaders to really come to terms with is that they typically have a lot more freedom in what their team does and how their team does it than might be comfortable.
Defining a strategy for how to do whatever your team is charged with doing is a big responsibility. And it’s a lot scarier than staying focused on the day-to-day of routine work. As a result, I see lots of teams or organizations with no discernible strategy whatsoever, muddling along with whatever tasks come their way - or, maybe worse, following through with zealous dedication on some idea that three years ago someone said seemed like a good idea.
So I keep an eye out for resources on defining strategies. Lots of them are way too “big” - focused on enormous organizations. Or they’re hyper-focused on things like SWOT Analyses or Business Model Canvases or Wardley Maps - which are all, you know, great, but they’re just devices to nudge thoughts and discussion into directions that might be fruitful. A 2x2 matrix or canvas or map isn’t a strategy. A concrete set of goals and priorities, informed by the context of the team and the environment in which it operates, that‘s a strategy, that’s something that can help guide the routine day-to-day decisions you and team members make, and serve as a nudging guardrail to make sure you and your team are moving in the right direction.
In this article, Boyd describes a very down-to-earth process for defining and following through on a strategy, that he learned and developed during his time as a union leader. It’s simple and pedestrian and it’s all that strategy is - routinely spending some time thinking of the big picture and its context, so that the day-to-day work is steered properly.
Boyd’s steps (followed by a course correction step) are:
This can be done by yourself or with the team, and involving your manager; or you can get feedback from your manager before striking off in a direction.
For brainstorming, whether by yourself or with others, the steps laid out in another article I saw this week, Shopify’s Brainstorming Session Template, can help - in particular the iterative approach of brainstorm, assess, brainstorm …. distill. And yet another article I saw, more for large organizations, about running a strategy offsite for a leadership team, is worth skimming if only to see that it’s the same as Boyd’s steps, but done in a group. One line I like from that last article - “Choose clarity over certainty”.
How to break out of the thread of doom - Tanya Reilly, LeadDev
5 situations when synchronous communication is a must - Hiba Amin, Hypercontext
We’re all spending a lot more time in written communication than we were before, and there are huge advantages! But there are some common failure modes, including having interminable conversations that don’t actually result in some conclusion. Reilly has three hints for winding up those discussions:
Speaking of that last point, Amin talks about some situations where you shouldn’t even try to communicate asynchronously if you can avoid it - they’re all areas where building relationships is the goal, or higher-bandwidth-than-just-text communications are necessary:
How to Plan Your Ideal Hybrid Work Schedule (So You Can Live Your Best Life) - Regina Borsellino
We’ve talked a lot about the challenges of arranging hybrid work for your team members, but you’re a team member too. What will work best for you?
Borsellino’s article goes into much more depth than most I’ve seen, walking you through 37 questions to help you make some decisions, under categories of:
There’s too much to summarize here, but if you haven’t yet started working through your work plans and they might be hybrid, this is a good starting point.
Because of an immune condition, I’m going to be way on the late side of coming back into the office. I like the idea of coming in 1-2 days a week, or maybe 3 days a week but just the afternoons (say) - and spending the rest of time either working from home, or some coffee shops in the neighbourhood. With our team distributed across the country, I can meet my managerial obligations that way comfortably, and it will give me a nice blend of productivity and peace of mind. The exact schedule, well, we’ll have to experiment a bit and see. What are your current plans - and what would your ideal plans be?
Growing a community around an open source research software effort to the point that there are external maintainers is a sign of huge success - but it makes things way more complicated. It’s a pain to be the sole maintainer, but at least there’s clarity in decision making.
Here Colannino describes the “Minimum Viable Governance” (MVG) set of template documents for bootstrapping a real open source governance framework. Some areas - trademark, and antitrust policy(!) - matter less for research computing, but decision making, project charter, code of conduct, and guidelines for contributing are all very relevant. This is a good starting point as is, or for thinking through the issues.
Intel C/C++ compilers complete adoption of LLVM - James R Reinders, Intel Software Blog
I didn’t even know this was in the works - the new Intel C/C++ compilers in the oneAPI suite are now based on LLVM. They’ve changed the name, (icx) so that there’s no confusion and you can have the classic and new compilers installed easily. This should mean that a lot of tooling becomes available, and development moves more quickly - and it looks like compilation time has significantly improved, and performance on some benchmarks are also better. Unfortunately it doesn’t sound like intel contributes all their LLVM work upstream.
Fortran is already in progress.
Speeding up atan2f by 50x - Francesco Mazzo
The standard glibc atan2f is well optimized, but the authors didn’t need the full of accuracy of atan2 for his application, didn’t need special cases like Inf handled, and did need large batches of operands calculated on at once. Mazzo’s article goes from defining the approximation he used (including how tuning parameters to minimize maximum error), then vectorization, minimizing type conversion, avoiding NaN-handling (he just wants them to propagate), taking advantage of fused multiply-adds, and finally unrolling to manually vectorize.
How to write fast Rust Code - Renato Athaydes
Rust is gaining some research software development adherents for new projects that otherwise might be written in something like C++. Here Athaydes walks us through some gotchas that will be familiar to those of us who have had to write performant code in the past, but with some extra twists due to Rust’s approach to memory:
It’s Time to Retire the CSV - Alex Rasmussen
Despite this ubiquity and ease of access, CSV is a wretched way to exchange data. The CSV format itself is notoriously inconsistent, with myriad competing and mutually-exclusive formats that often coexist within a single dataset (or, if you’re particularly unlucky, a single file). Exporting a dataset as a CSV robs it of a wealth of metadata that is very hard for the reader to reconstruct accurately, and many programs’ naïve CSV parsers ignore the metadata reconstruction problem entirely as a result. In practice, CSV’s human-readability is more of a liability than an asset.
Hear, hear! CSV - and, I’d argue, other human-readable text formats - were a costly mistake. Rasmussen itemizes some of the problems - absent metadata, no typing, poor precision - I’d add surprisingly slow to generate (serializing floating point to text is a shockingly time consuming operation!). Rasmussen suggests Avro, parquet, arrow, HDF5, sqlite, even XML - almost anything else, really.
CSVs are especially bad because they’re used to exchange arbitrary data, but I’d continue the point and say that in research computing we use text-based file formats, instead of carefully defining APIs and allowing a number of back-end storage representations - way too much. (I’m looking at you, bioinformatics, with your “tab-separated files are a way of life” nonchalance).
AMD Infinity Hub - AMD
Something NVIDIA’s always done really well is to have a place to go to get information on and distributions of CUDA-powered applications. AMD is belatedly catching on; new to me is this page for AMD-powered tools with ROCm, their software stack (including OpenCL and HIP for programming, and libraries based on top of that.). It includes information and links to container images that it looks like AMD maintains.
Intel Finally Gets Chiplet Religion With Server Chips - Timothy Prickett Morgan, The Next Platform With AMX, Intel Adds AI/ML Sparkle to Saphire Rapids - Nicole Hemsoth, The Next Platform Intel sees CXL as rack-level disaggregator with Optane connectivity - Chris Mellor, Blocks and Files
In the first two articles, Morgan and Hemsoth give new architecture announcement from Intel their usual well-researched coverage.
Morgan overs the big picture, the architectural changes coming for Xeon CPUs (“Sapphire Rapids”) and HPC GPUs (“Ponte Vecchio”). Sapphire Rapids will be based on chiplet “P (performance)-cores” with cores, accelerators, memory controllers, cache, I/O resources, etc., and Embedded Multi-die Interconnect Bridge (EMIB) links between them tying the whole die together, including access to high-bandwidth memory.
Hemsoth digs into “AMX”, Advanced Matrix Extensions, aimed specifically for AI/ML but which will be useful for more traditional technical computing applications as well. As with ARM’s Scalable Matrix Extension (SME) announcement (#84), this will support small tiles of matrices, but it seems at a quick read to be more fully featured, including for instance a matrix tile add-multiply (C+=A*C). Hemsoth says:
The best way for now to think of AMX is that it’s a matrix math overlay for the AVX-512 vector math units.
In the third article, Mellor summarizes Intel’s presentation at IEEE Hot Interconnects, covering their off-chip plans, involving Compute Express Link (CXL). From Mellor, “CXL is based on the coming PCIe Gen-5.0 bus standard to interconnect processors and fast, low-latency peripheral devices.” As with the origins of Infiniband, the plan is to extend on-node fast access to data and peripherals off-node, providing (so the plan goes) a cluster-wide memory and (with persistent memory like Octane) storage tier. A slide from the deck outlining the plan (courtesy the Blocks and Files post) is below.
Is GitHub Actions Suitable for Running Benchmarks? - Jamie Rodríguez-Guerra, Quantsight Labs
As Betteridge’s Law would suggest, the answer to the headline question is no, but it’s closer than I would have expected.
Rodríguez-Guerra walks us through trying GitHub Actions CI workflows as benchmarking, for a large number of replications of scikit-image benchmarks. The results are much less all-over-the-place than I would have expected - the standard deviation is 5% - but unless the kinds of per-commit changes you make are routinely expected to cause performance changes >>5% that’s probably just too noisy.
So this probably isn’t good news for benchmarking (although in fairness it might be useful in early stages of optimizing code when you really are going for substantial improvements) but it does suggest a pretty decent reliability in terms of run times for large Actions, which is interesting to know about.
Workshop on Integrating High-Performance and Quantum Computing - 18-22 Oct, Virtual, Abstracts due 10 Sept
Held as part of IEEE Quantum IEEE Quantum Week, this workshop inviets “high-quality, abstract submissions on all topics related to integrating QC into the existing binary computing eco-System, with a particular focus on combining HPC and QC. “ Topics include everything from data centres to programming models:
Webinar: Get OpenMP Tasks to Do Your Work - 5pm UTC 22 Sept
A one-hour introductory webinar covering OpenMP tasks.
DNANexus and Baylor College of Medicine’s human genome sequencing centre is hosting a 4-day hackathon to build pipelines to analyze large datasets in a cloud infrastructure.
Cloud Learning and Skills Sessions (CLASS) Fall 2021 Advanced Cohort - 12 Oct - 27 Jan, $4,000 - $5350
This is interesting - training aimed squarely at our team members:
Cloud Learning and Skills Sessions is a program that provides research computing and data professionals (those who support researchers) the necessary training to effectively leverage cloud platforms for research workflows. A combination of vendor-neutral guidance across cloud providers, and training on the tools and technologies supported by public cloud providers, allows a broad range of research use-cases to more effectively use these important resources. Participants join the CLASS community of practice where they can share information and lessons learned.
It’s not cheap but it looks like it covers a wide range of cloud technologies, deploying real applications on them, and having someone review your architecture.
Want the code in your IDE to look kind of like (to my eye) the default TeX serifed fonts, computer modern roman? Try New Heterodox Mono, I guess.
Make your GitHub actions (many of which run in Ubuntu containers) faster by turning off initramfs and man db updates, amongst other things.
With GitHub no longer using passwords from the command line, now might be a good time to use security keys. I’ve been using a Yubikey 5 for a while and it works fine.
Twitter thread describing a number of use cases of the linux perf command, with examples.
Cryptography is hard. Here’s a number of common gotchas.
Getting started with tmux.
As longtime readers know, I think constraint solvers, theorem provers, and a host of math-y tools are woefully underused in research computing. Here’s an absurdly easy to use wrapper for z3 in python.
A good quick overview on B-trees and a few of the optimizations necessary to make them actually perform well for database-type applications.
And that’s it for another week. Let me know what you thought, or if you have anything you’d like to share about the newsletter or management. Just email me or reply to this newsletter if you get it in your inbox.
Have a great weekend, and good luck in the coming week with your research computing team,
About This Newsletter
Research computing - the intertwined streams of software development, systems, data management and analysis - is much more than technology. It’s teams, it’s communities, it’s product management - it’s people. It’s also one of the most important ways we can be supporting science, scholarship, and R&D today.
So research computing teams are too important to research to be managed poorly. But no one teaches us how to be effective managers and leaders in academia. We have an advantage, though - working in research collaborations have taught us the advanced management skills, but not the basics.
This newsletter focusses on providing new and experienced research computing and data managers the tools they need to be good managers without the stress, and to help their teams achieve great results and grow their careers.
This week’s new-listing highlights are below; the full listing of 165 jobs is, as ever, available on the job board.
High Performance and Research Computing Manager (Research Fellow) - University of Hertfordshire, Hertfordshire UK
To support and develop research and commercial activities in the area of high-performance computing within the School and the wider university, including support for grant applications and design of research projects; To contribute actively to research interests of the School, including personal research activity that makes use of high-performance computing; To support the work of internal and external collaborators by developing a broad understanding of research goals and computing needs across different disciplines; To contribute to business planning and encourage commercialisation and spin-off projects where appropriate; To develop a strategy based on research needs for the future development of cluster hardware and software.
Project Lead/Principle[sic] Software Engineer - Beth Israel Deaconess Medical Center, Boston MA USA
This position reports to the Director of Academic and Research Computing (ARC). ARC supports the research community with custom, web applications supporting pre and post award grant processing, Human Subject Research, IACUC, IBC, Conflict of Interest, space, training, and lab web publishing. This position , provides technical, project and application management of the development team within the department. As a member of the senior support staff, provides integrated business support to Research and Academic Affairs and the Research Business. Compiles project specifications from the Project Manager and interacts with the Data Architect to determine the appropriate data design for the revised or new application and produces the technical application development specifications. Manages and supports Academic Computing applications. This job does not have any direct reports.
Infrastructure Technology (HPC) Manager - Memorial Sloan-Kettering Cancer Center, New York NY USA
Do you have a background in High Performance Computing and want to help enable and advance groundbreaking computational research in the frontline of the fight to cure cancer? We are seeking HPC Manger to join our Advanced Research Computing team! As a People Leader you will engage and support the academic and research scientists in research projects potentially involving multiple external organizations. Our group is committed to building collaborative environments in which the best software engineering practices are valued, and to sharing and applying cross-disciplinary computational techniques in new and emerging areas. In this position, you will manage and maintain HPC resources, including systems administration; coordinate and direct integration and growth of these resources; guide short- and long-term planning of HPC computing at MSKCC; keep abreast of emerging technologies that may benefit HPC computing research and applications at MSKCC.
Senior ARC Software Developer Team Lead - University of Victoria, Victoria BC CA
The senior advanced research computing software developer team lead plays a major role by providing: human resource and technical leadership to a team of developers, leadership for major research software projects, and expert-level technical leadership in the analysis, design, development, maintenance, and support of research software project and platform development at the University of Victoria. The Senior ARC Software Developer Team Lead will be responsible for leading developers and co-op students and also help with hiring, mentoring and management of grant funded developers and co-ops for researchers and research groups across campus. This position will be responsible for creating/adopting standards and best practices in research software development, by standardizing on development best practices, tools sets, frameworks, and documentation.
Senior or Lead HPC Engineer - University of Illinois at Chicago, Chicago IL USA
ACER seeks a qualified Senior or Lead HPC Engineer to join our impactful team and advance scientific discovery. This position will provide leadership, expertise, and support to build and manage world-class HPC and AI platforms. This role will collaborate with others on the team to develop novel solutions to meet UIC’s research needs either via on-prem resources or cloud environments. Works closely with the Director of ACER to build strong relationships with our users and communicate their needs effectively. Hybrid and flexible work schedules available.
Software Engineer (Infrastructure) - RStudio, remote
RStudio is seeking a Software Engineer focused on infrastructure to join the Platform Team. The team works together on common engineering services, cloud deployments, and developer team productivity. Drive technical decisions for the team’s infrastructure projects. The projects include marketplace images (for AWS, Azure, and GCP), Dockerfiles, Helm charts, and supporting infrastructure for projects. You will receive help from many people on this, but we want you to lead what we do here.
Software Engineering Manager, ML Platforms - Flatiron Health, New York City NY USA
We’re looking for a Software Engineering Manager on our ML Platforms team who is passionate about helping us accomplish our mission to improve lives by learning from the experience of every cancer patient. Our team of ML Platform software engineers own and operate a robust machine learning platform that combines the latest machine learning industry practices with engineering excellence. In this role, you will lead talented software engineers to build platforms that satisfy the ever-growing appetite for machine learning within Flatiron Health. The goal of this team is to allow ML engineers to focus on what they are good at - building machine learning models and safely accelerating the process of bringing a model to production.
Senior Project Manager - BenchSci, Toronto ON CA
We are currently seeking a Senior Project Manager to join our growing Engineering team! Reporting directly to the SVP Engineering, you will drive the end-to-end delivery of OKRs and projects. As part of the job, you will work closely with a number of stakeholders across the organization to make sure projects are successfully delivered to our customers. You’ll have multiple projects on the go at any given time, so you should be able to easily handle competing priorities.
Principal Data Scientist - Signal Processing - Biogen, Cambridge MA USA
We have an exciting new opportunity for a Real World Evidence (RWE) Principal Data Scientist to join our team in Cambridge, MA. You’ll be part of the Machine Learning and Artificial Intelligence (MLAI) team in Personalized Health Research, Analytics and Solutions(PHRAS). In this role, you’ll be responsible for providing MLAI support and guidance on digital data collected from sensors/digital devices/cell phones and may serve as the responsible RWE digital data analysis expert on several RWE projects in different indications.
Senior Research Scientist, Simulation and Technology - NVIDIA, Toronto ON CA
NVIDIA is searching for a world-class Research Scientist to join our growing research team. The ideal candidate will be conducting cutting-edge research at the intersection of Machine Learning, Computer Vision and Computer Graphics, and working alongside top experts in these fields. With incredible resources in AI, graphics and robotics, you will be able to impact, contribute and advance these exciting domains. Topics include but are not limited to AI for simulation, 3D Deep Learning, DL for animation, content generation, transfer learning, domain adaptation, computer vision, and medical imaging. With its unique open culture, NVIDIA is one of the best industry labs to do AI research.
Manager, Biostatistics (Non-Clinical) - Pfizer, Pearl River NY USA
Pfizer’s Clinical Vaccine Research & Development team is looking for a motivated and experienced statistician with a solid background in nonclinical statistics, strong consulting skills and an interest in collaborating with research scientists in support of vaccine assay development. The colleague will be interacting one on one with research scientists to obtain project related data/information and must be able to interpret and effectively interpret statistical results from research projects.
Applied Research Manager, Computer Vision - NVIDIA, Santa Clara CA USA
NVIDIA is searching for outstanding applied research manager to spearhead our Computer Vision efforts. Computer Vision is a core component of NVIDIAs platforms from Drive AV to Isaac autonomous Robotics to Clara for Healthcare to Maxine streaming and broadcast and beyond. The ideal candidate is an established thought leader in academic or industry with deep experience in vision, human understanding, publishing and presenting at industry and academic conferences, and hands-on implementation. This role requires high-level knowledge of multiple areas within production workflow and tools, Deep Learning, network optimization, and vision systems.
Senior Manager - Data and Analytics - KPMG, Bristol UK
The role is within the UK Infrastructure Government and Healthcare (IGH) Data Analytics, based in London. We provide a wide range of data analytics, modelling, data modelling services to our clients in a diverse range of sectors such as health, transport, policing, local and central government. We are looking for a Senior Manager to help lead our rapidly-growing team with strong technical expertise in data visualisation, analytics, and data modelling. The individual should have a proven track record in leading data analytics programmes and projects with exceptional communication and presentation skills.
Director, Software & Application Innovation Lab - Boston University, Boston MA USA
Based in the Hariri Institute for Computing, the Software & Application Innovation Lab (SAIL) is a collaborative partner for Boston University faculty and researchers that engages in a variety of activities to encourage, enable, support, and execute research efforts involving technical, data-driven, computational, and software engineering aspects or components. The Director oversees the overall operations of SAIL, while working closely with the Institute’s Director to integrate SAIL capacity appropriately throughout the spectrum of research projects and programs at the Institute with a software engineering component. The Director will shape and implement long-term plans for SAIL with regard to prioritizing existing opportunities, managing growth, and developing new opportunities for collaboration and partnership. Ultimately responsible for the success of SAIL, the individual in this role will serve as a key bridge between SAIL and Institute leadership and researchers and larger computing efforts throughout the University.
Senior Research Software Engineer in Computational Fluid Dynamics - Imperial College of London, London UK
As a Senior Research Software Engineer within the Department of Aeronautics, you will play a key role in supporting the development of scientific code, liaising and working closely with local researchers to understand their requirements and develop software outputs. You will contribute to enhancing the sustainability and maintainability of the Department’s Computational Fluid Dynamics ecosystem consisting of a variety of flow solvers developed across different research teams (PyFR, Nektar++, Xcompact3d). This will include promoting research software best practices and providing training for less experienced researchers and engineers. You will also be involved in promoting the Department’s work, sharing your knowledge inside and outside the Department, and participating in activities related to software engineering within the RSE community at Imperial and beyond.
Technical Project Manager - Corriculo (recruiter), Oxford UK
Joining a leading Oxfordshire based scientific R&D company, as the new Technical Project Manager you will take ownership of my client’s exciting new applications, fundamental in the name of scientific research. Managing multiple projects concurrently, as the Technical Project Manager you will be responsible for delivering products outlined by each of my client’s customers, and will be ultimately responsible for delivering these vital products, on time, on cost and on quality.
Qualitative Research Project Manager, Patient Centered Solutions - IQVIA, Durham NC USA
As part of the Study Execution Team (SET), the Qualitative Research Project Manager will be responsible for the execution of all operational aspects of the data collection phase of Patient Centered Solutions (PCS) studies which include a primary research phase. You will be an integral member of study teams and will be involved in proactive project planning and take ownership of the management of study fielding. The abilities to multi-task, “think fast on your feet”, creatively problem-solve, and provide accurate information, logistical solutions and support and quality results are keys to success.