I’m back and refreshed from my mini-break from the newsletter; there’s a lot of optimism in the air here as COVID-19 numbers continue to drop and projects are moving forwards.
I don’t have a lot else to add right now, so let’s go straight to the roundup!
A couple good articles on one-on-ones from a a couple of weeks ago.
In the first, Kim offers five pieces of advice around vents. In particular, vents aren’t about problem solving, they’re about letting something out - and that something has been festering for a while. So Kim’s advice is:
In particular sometimes just letting something out is enough - but sometimes it’s raising something that needs to be addressed in future discussions.
In Longworth’s article, she talks about her formula for one-on-ones:
The last article is an old one but includes elements of both Kim and Longworth’s articles. In particular, Lopp considers a one-on-one that is just “an update” to be a bit of a miss, and shares his own approach for dealing with those - bringing up three particular points, doing a mini performance review on some specific area, or taking the time to get input on some fire he‘s fighting. For the vent or the disaster Lopp gives similar advice to Kim - and suggests that one purpose of one-on-ones is to provide an environment where some of those minor dissatisfactions can be surfaced before they become vents or disasters.
Robert Half Research Points To Strong Job Optimism Among U.S. Workers - Robert Half Press Release
A recruiting company (so take with a grain of salt) did an online survey
The online survey was developed by Robert Half and conducted by an independent research firm from March 26 to April 15, 2021. It includes responses from more than 2,800 workers 18 years of age or older at companies in 28 major U.S. cities with 20 or more employees.
and they found that 32% were planning to start a job search in the next few months. There’s a lot of natural attrition that’s not happened over the last year and a half due to people clinging to familiarity in a time of uncertainty, and now as things open a bit up people are tired and looking for a change. Salary and not enough job advancement were tied for top reason why people were considering looking for new work.
The survey also said that and 47% wanted fully remote work.
There’s an interesting discussion of effective messaging for candidates for those doing active recruiting - on LinkedIn in this case, but it could just as well be over email or somewhere else. The advice is pretty straightforward but easy to forget - talk to them like an individual human - with some concrete suggestions about when and how to send the message.
Are other groups in research computing doing active recruiting, or soliciting recommendations? I don’t hear of a lot of groups doing it.
Always be quitting - Julio Merino
If we knew we were quitting (or just going on a long vacation) in two months, what would we be doing differently at work? Probably documenting a lot more, making sure people were coming to meetings so that they could take our place when we weren’t there, training up people to be able to take over parts of our role for us.
But those activities are key and routine parts of being an effective manager or technical leader. Merino commends this approach to us - to be imagining that we were leaving, and being in the mindset of making ourselves less necessary. It’s a vital approach to scaling up our team and making room to take on new challenges, even in the same role.
In particular, Merino suggests -
These are great points to be consistently working on, whether you’re a manager, an individual contributor with a technical leadership role, or even just intending to take on either of those roles.
How to move a 400-person hackathon online - Juri Chomé
Even post pandemic, a lot of people are going to be more willing to do virtual events than they were in 2019, which opens up a lot of collaborative opportunities. Here Chomé walks through how they moved ZuriHac, a Haskell hackathon held in June ever year, to purely online. They talked about the tools they used (Discord for chat, with one use of Zoom, Streamyard + Youtube for streaming presentations, repl.it for some live coding, and an existing registration system) and how they actually ran things. With ~50 projects, the remote collaboration within projects wasn’t hard, but coordinating all of the activities across projects and the hackathon-wide activities took a lot of doing.
This looks to be a promising development for the development of research computing and data as a profession. Since #21 I’ve been talking about the CaRCC’s Research Computing and Data Capabilities Model, a still-underused tool for thinking about the variety of services needed for providing RCD support to researchers in academic institutions. Some of the same group is going to turn their mind to RCD staff training and career pathways. From the press release:
Among the goals of the pilot program is to develop resources for staff training and workforce development, including leading practices for recruitment, onboarding, advancing diversity, equity and inclusion, professional development, and proven models for student internship and training programs. […] In addition to training and workforce development, the Resource and Career Center pilot will create and share a model of career arcs for RCD professionals to explain career options and help existing RCD professionals explore professional development and advancement opportunities.
Glue: the Dark Matter of Software - Marcel Weiher
Weiher hypothesizes that one reason that code sizes of systems are exploding even as individual reusable components are plentiful is that we don’t have any real sensible way to deal with glue code. There are DSLs for algorithms but nothing similar for the plumbing code that makes the output of one component useful as the input for another; and what’s more, the number of those connections is quadratic in the number of components.
Clever vs Insightful Code - Hillel Wayne
Wayne distinguishes between “clever” code, which exploits arcane knowledge of the programming language/os/library, with “insightful” code, which exploits knowledge of the problem. Both can make code brittle - changes may make the cleverness/insight irrelevant - but documenting the insight you’re exploiting is a good and useful thing, and puts the focus where it should be, on the problem.
I tried out VSCode Remote Repositories this past week (including for the newsletter website) and I have to say it was really slick. Fire up vscode, open a GitHub repo, and browse code or even create and develop on a branch right there. If you want to run tests etc you’ll want to use code spaces, but for code browsing or simple development this is really slick.
(Technical) Infosec Core Competencies - Jan Schaumann
Linux Privilege Escalation: Exploiting Capabilities - Stefano Lanaro
Kerckhoffs’s Law for Security Engineers - Devdatta Akhawe
How to Handle Secrets on the Command Line - Carl Tashian
Good security is good operations.
Schaumann goes through a very approachable what he views as core competencies for information security practitioners, and crucially most of them are not deep and profound understanding of obscure corners of the kernel or cryptographic implementations, but being able to read and learn from relevant news sources and being a careful and deliberate system administrator:
You may also notice that a lot of this overlaps with a general understanding of… well, computering on the internet, with operations and system administration concepts. This is no coincidence. Good ops is good security.
Relatedly, Lanaro walks us through exploiting linux capabilities - not by plunging deep into the linux kernel, but by treating it correctly as a finer-grained but similar issue to exploting suid executables, looking for files that have capabilities they shouldn’t have and using them.
Akhawe talks about Kerchoff’s law - no “security via obscurity”, rely on no secrets other than the key - and why it matters; you can change the key easily. Except, unless you actually are changing the key from time to time, you don’t actually know for sure that you can change the key easily. Build in key rotation from the beginning to make sure that a breach of the key can be readily mitigated.
And finally, Tashian goes through the steps necessary to make sure secrets provided on the command line to aren’t available to anyone who happens to “ps” at the right time.
All of these have a similar theme - good security is good operations, and being excellent at security doesn’t mean wizard-level knowledge of hardware and kernels but by broad knowledge of a lot of tools, and using them wisely.
CPUs with bugs aren’t new - some of us will remember the Pentium FDIV bug - but with increasing complexity (and smaller components) hyperscalers are starting to see bugs on individual cores of individual machines that are extremely difficult to detect much less debug.
Rosenthal provides a nice summary of two recent papers, by Facebook (Dixit et al.) and Google (Hochschild et al.) Both tell wonderful debugging stories - Facebook discovering files were being dropped by a pipeline and it all came down to a specific mathematical operation being repeatably mis-performed on core 59 on a particular box - they nailed it down to a 60-line assembly reproducer. Google found silent data corruption caused by a “few mercurial cores per several thousand machines” that included (from Rosenthal’s summary):
Migrate Your Workloads with the Graviton Challenge! - Steve Roberts, AWS News Blog
In #74 we mentioned an AWS-sponsored hackathon for the porting codes for the graviton2s; for those who are interested in the work but are too busy to block off time for a hackathon, AWS is also supporting a graviton challenge, with all valid entries getting a $500 AWS credit, and prizes for best adoptions. Between 22 June and 31 Aug, perform eight steps over any four, non-consecutive, days, and record a short video about your results and you’re in.
A tonne of calls for submissions for events to be held as part of SC21 in November in St Louis, MO:
EMCLPKDD Workshop on Automating Data Science (ADS2021) - 17 Sept, Virtual, 6 page papers due 23 June
Re-envisioning Extreme-Scale I/O for Emerging Hybrid HPC Workloads (REX-IO ’21) - 7 Sept, Virtual, 6 page papers due 2 July
SeptembRSE, the Fifth Conference of Research Software Engineers - 6-30 Sept Virtual, Submissions (Talks, posters, walkthroughs, workshops, panels, discussions) due 2 July
Covering anything of interest to research software development.
18th Annual IFIP International Conference on Network and Parallel Computing (IFIP NPC) - 3-5 Nov 2021, Paris, 12-page papers due 11 July
A wide conference covering topics on parallel and distributed applications and algorithms, architectures and systems, and distributed software environments and tools.
OpenMPCon - Virtual event 3 Sept, Submissions (presentations, posters, tutorials, BoF, panel discussions) due 5 July
Topics of interest include
ACM HPDC 2021 - The 30th International Symposium on High-Performance Parallel and Distributed Computing, 21-25 June Online, $50-$100
Little short notice for this one, sorry, but next week there’s a series of talks on a pretty wide range of parallel and distributed computing topics - program is here.
Nordic-RSE online unconference 2021 - 29-30 June, 13h-16h CET, Free(?) registration
Not limited to those in the nordics (or even those who consider themselves RSEs). Helpfully, the conference provides shorter and longer teasers for the event (note to organizers! This stuff makes it much easier for people like me to include your event in roundups!)
Are you developing software or tools that are driven by research/engineering in either academia or industry? Need to network, share knowledge and experiences with your peers? Maybe you have heard of something called research software engineers and you would like to know more? Nordic-RSE invites everyone interested in such topics to join our online unconference (lightweight get-together) on June 29 and 30, where we let the participants shape the agenda (“birds of a feather”). To kick off the event, we also have four invited speakers, including:
Kristoffer Carlsson (JuliaComputing) “Julia for research software engineers”, Athanasia Monika Mowinckel (University of Oslo) “Developing and distributing in-house R-packages”, Shahnawaz Ahmed (Wallenberg Centre for Quantum Technology) “Keep your code alive - lessons from the QuTiP project”.
In further “cryptocurrencies making everything worse” news, Dockerhub is now cancelling autobuilds for free accounts. I’m sort of dreading GitHub Actions going the same way. We’ve signed up for an open source docker hub teams account but the line is so long we’ve been waiting to hear back for weeks.
A simple handmade CP/M or DOS linker for teaching about how linkers (and the old .COM files) work.
Modularize your bashrc with a .config/bashrc.d directory.
A tutorial on memoizing slow bash script operations that are used repeatedly in a long script - using bash4 coprocesses.
Bash quoting is subtle, and then with ssh commands another layer of quoting and parsing is added. Here’s a walkthrough of quoting bash commands called over ssh.
VSCode now lets you directly work with remote github repos without downloading them; I tried them myself this week. This along side Codespaces is probably going to change my development workflow quite a bit.
A walk through on using extremely cheap google compute platform resources to run an app - the example is an Elixir app and there’s some stuff in there that’s elixir specific to optimize for the constrained resources but most of it is pretty general.
There’ll be a twitter chat on the extreme-scale scientific software stack on June 21st - interesting to see the attempt to use different media for communication and outreach.
“Modulinos” - files that behave as a library when imported, or a script when executed - in bash.
asciiflow for drawing ascii diagrams, and exporting them (including prepending them with typical comment characters for use in code).
And that’s it for another week. Let me know what you thought, or if you have anything you’d like to share about the newsletter or management. Just email me or reply to this newsletter if you get it in your inbox.
Have a great weekend, and good luck in the coming week with your research computing team,
About This Newsletter
Research computing - the intertwined streams of software development, systems, data management and analysis - is much more than technology. It’s teams, it’s communities, it’s product management - it’s people. It’s also one of the most important ways we can be supporting science, scholarship, and R&D today.
So research computing teams are too important to research to be managed poorly. But no one teaches us how to be effective managers and leaders in academia. We have an advantage, though - working in research collaborations have taught us the advanced management skills, but not the basics.
This newsletter focusses on providing new and experienced research computing and data managers the tools they need to be good managers without the stress, and to help their teams achieve great results and grow their careers.
This week’s new-listing highlights are below; the full listing of 172 jobs is, as ever, available on the job board.
Software Engineering Manager - Broad Institute, Cambridge MA USA
Drive technical product development and provide leadership in an innovative and fast-paced environment. Be responsible for the overall execution of product development and the success of your teams. Collaborate with Product Management to plan and deliver new features and projects. Provide strategic oversight for the teams and foster a sense of empowerment and collective ownership within the teams. Form good working relationships across other teams and departments to help break down barriers to collaboration. Provide coaching, mentoring, career development, and support to the teams on both technical and non-technical issues. Uphold a culture that ensures psychological safety, learning, inclusion, and transparency. Ensure operational excellence, including production incident response and mitigation.
Head of Bioinformatics (Fixed Term) - Wellcome Trust/Cancer Research UK Gurdon Institute, Hinxton UK
You will be integral to shaping how Bioinformatics research develops and grows within the Institute. You will collaborate in cutting-edge research, contribute to strategy, develop and implement analysis procedures, and manage a group of computational biologists. You will lead the core team to provide advice on experimental design, develop and implement analysis workflows, provide project-specific computational and statistical analyses, and deliver training. This will primarily involve the analysis of genomics datasets from high-throughput sequencing, including analyses of RNA, protein binding and accessibility, epigenomic marks, single cell studies, and genome sequence alterations. You will also have opportunities to lead research projects, collaborate with research groups, participate in grants, and supervise staff and students.
Research Systems Project Manager - McMaster University, Hamilton ON CA
We are seeking candidates for a Research Systems Project Manager to play a critical role in the administrative management of complex technical projects that support McMaster’s research enterprise and research community. Reporting to both Research & High-Performance Computing Support (RHPCS) and the McMaster University Library, you will help implement projects related to McMaster’s research support infrastructure, which includes research information and facility management systems (McMaster Experts, MacLIMS), library systems, funding application administration systems, and various administrative databases. We are looking for someone whose thoroughness, diligence, and attention to detail allows them to coordinate and implement various projects by managing concurrent tasks, setting priorities, and working to deadlines.
Data Scientist, Scientific Data Steward Director - Parexel, WFH UK
If you’re looking for an opportunity to use your medical informatics expertise and leverage RWE to help move healthcare forward, please consider applying to the Data Scientist/Scientific Data Steward Director role. The Data Scientist/Scientific Data Steward Director ensures that the scientific or real-world data used in a given solution is fit-for-purpose and is used in a manner compliant with the data use agreement and applicable law. Consultation is provided to users of the data in order to ensure that use is optimized.
Senior Manager (Head of Technical Development) - Wellcome Sanger Institute, Hinxton UK
As Head of Technical Development you will understand what scientific challenges are being faced and will be responsible to manage the teams development of solutions that address them using evidence led approaches to identify opportunities and justify resources. You will lead a dedicated team of technical and scientific experts with a remit to scale and operationalise our high throughout sequencing solutions. Delivering the portfolio by working across organisational boundaries, to build and harmonise professional relationships across the organisation for the benefit of our scientific delivery. To keep pace with scientific changes you will ensure that you have the right structures, resources, skills and processes in place to deliver requirements.
Project Manager in Health Research Data Capture and Management - Research Institute, McGill University Health Centre, Montreal QC CA
The new Data Capture and Management Platform (DCMP) is offering researchers resources (expertise, technical support, software and methodological tools) to facilitate data-related operations, from the protocol conceptualization phase to data access and usage for analysis. Under the scientific supervision of the DCMP Scientific Director and the administrative supervision of the CORE Manager, the DCMP Project Manager will coordinate operations of the platform team, provide support to users in the development of data collection tools (e.g. questionnaires), and will coordinate the implementation and use of data capture and storage environments, and the management and quality control of research data.
Director, Research & High Performance Compute Support - McMaster University, Hamilton ON
Reporting to the Vice President, Research, the Director will be engaged in the support and direction of research computing services at McMaster in their capacity as Director. RHPCS partners with McMaster’s AVP and CTO office, University Technology Services (UTS), Computer Services Unit (CSU) in the Faculty of Health Science, the Privacy Office and all other Faculty and campus IT personnel to provide services for all researchers at McMaster. Therefore, it is essential that the Director have familiarity with networking technologies, virtualization technologies, and high-performance computing.
Machine learning / High Performance Compute Roadmap Program Manager - AMD, Markham ON CA
As roadmap program manager in AMD’s machine learning software engineering team, you will drive planning of a project portfolio of leading-edge technology in high performance GPU-accelerated compute and machine learning for the Radeon Open Compute software stack. You will learn about how the power of open-source software can be applied to solve real-world problems. You will interact with product management, customers, software and hardware engineering teams, quality assurance and operations in a new and growing team
Research Operations Manager - Computational Social Science Lab - Penn State, Philadelphia PA USA
The Computational Social Science Lab is a new organization for the research initiatives of Duncan Watts, a PIK Professor affiliated with Wharton, SEAS, and Annenberg. The Lab’s goal is to bridge among the three aforementioned schools research, teaching, and practice in the rapidly emerging field of computational social science (CSS), and to establish Penn as a global leader and standard setter in CSS.The Research Operations Manager is responsible for day-to-day operations across complex projects conducted with collaborators both internal and external to Penn: researchers and data engineers, students and part-time workers, data partners. In addition, the Research Operations Manager B will contribute to the Lab’s long-term success by participating in strategic planning, fundraising, and training.
Senior Manager, Data Science - Bell Canada, Toronto ON CA
The Senior Manager Data Science will join a newly created Data Science team, part of IMS BI group, to help drive insight and business value using advanced analytics and machine learning. The Senior Manager will report into Director, Data Science and play a key role in IMS BI’s Business Intelligence and Campaign transformation journey.
Program Manager, Yale Centre for Research Computing - Yale University, New Haven CT USA
Reporting to the Director of Projects and Programs, Yale Center for Research Computing (YCRC), this position is responsible for the day-to-day management of the skills-based training program at YCRC and other programs of the Center. The successful candidate will work with training presenters (internal and external) to schedule sessions for each semester, communicate the availability of training sessions to the community, coordinate registration and ensure the successful completion of each session. After each training session, metrics are gathered and reported. Subject matter for additional training sessions will be identified in consultation with YCRC Leadership.
Director of Data Engineering - Exscientia, Oxford UK
Exscientia is committed to discovering medicines in the fastest and most effective manner. We achieve this by applying the latest research in Artificial intelligence (AI) to transform drug discovery. Exscientia are the first and only company to have AI designed molecules in clinical trials. We are currently seeking a highly-motivated Director of Data Engineering with a talent for developing high performing teams delivering complex systems within the drug discovery industry.
Director of Research Advanced Computing Services - University of Oregon, Eugene OR USA
Reporting to the Associate CIO for Technology Infrastructure, the Director of Research Advanced Computing Services (RACS) is responsible for providing administrative leadership, fiscal leadership, and oversight for RACS, including the High Performance Computing (HPC) Core Facility. In addition to overseeing HPC, the Director is responsible for building out mid-tier services that position RACS as a full-service destination for research information technology services. The Director is also responsible for aligning the RACS workforce with university objectives for diversity, equity, and inclusion, as well as increasing diversity among faculty customer base by lowering barriers to entry to HPC and performing outreach to a broad range of faculty from multiple academic disciplines across campus.
Senior Manager of Energy Analytics - Power TakeOff, Denver CO USA
The ideal candidate has a knack for identifying areas where energy analytics can provide value to end customers. The right person for the job will apply their knowledge of data analysis, mechanical systems, and energy efficiency to lead a group in solving real-world problems faced by our company and finding opportunities for improvement across multiple projects, teams and business units - ultimately contributing to increased energy savings of business customers. The senior manager of energy analytics will work closely with leaders across product, sales, and customer engagement to support and implement high-quality data analytics projects. They will know when simple methods (Ex. linear regression) are better for the business and when more complex methods (Ex. CNNs) are likely to drive more value for the business.
Technical Program Manager, Quantum AI Education - Google, Goleta or Los Angeles CA USA
We are seeking a program manager to join our team and lead multiple education work streams that are critical to Google’s Quantum AI mission. Our team needs resident expertise to help advise and steer our efforts toward productive ends. Working closely with pedagogical experts and educational researchers, we hope you can help transform how people understand and use quantum computers.