Summary of discussions concerning ICT White Paper held by Physics Computing Committee on 8th June 2020 ====================================================================================================== General notes -------------- 1. Clearly, there are parts of ICT that would benefit from reform and there are some positive aspects to the suggested model such as, budgetary responsibility being devolved to the product management, restructuring some of the higher management that would be beneficial. However, examples of problematic projects tend to be on the administrative/managerial side of the college (SIMP, Banner and ICIS were mentioned) rather than on the core departmental activities of teaching and research. The statement both in the green paper and the white paper that there is a lack of trust between the customers and ICT may or may not be true of the administrative parts of college but is certainly not true of the Physics Department. On the contrary, physics find ICT to be both helpful and very responsive to our teaching and research requests. 2. It appears that the writers of the white paper have not grasped some of the essentially diversity of needs of the activities of a university compared to a business. The statement that ICT supports "anything and everything" is often a reflection of the reality needed for research. A world class scientific instrument does not stop being world class just because the OS on which its drivers were written is no longer supported. This is true of individual instruments in the lab and of wider international collaborations. Space missions exemplify this with the recently ended Cassini probe being a prime example. This was a two decade space flight that, towards the end of its life, relied on computers running out of date operating systems. Without the excellent support of ICT, valuable data would have been lost to humanity. There are many other examples and, in general, collaborators in "Big Science" projects are constrained to run in the environment dictated by the collaboration, in practice this can mean an environment dictated by ESA, ESO, CERN or any other large project/telescope. This cannot be dictated by ICT for their convenience. ICT have recognised this and responded well to the needs of the research communities. If ICT are to stop this support it will inevitably result in a shadow ICT funded by scraps from research grants (explicitly something ICT want to avoid - for good reason). 3. The white paper claims to be based on extensive consultation with staff in departments (explicitly interviews with HoDs) across college and senior ICT members. To our knowledge nobody from within physics has been consulted - certainly not the HoD or the Physics-ICT computing contact. We find this unusual as Physics is one of the largest users of computing within Imperial College. Senior members of ICT also tell us that their consultation was rather perfunctory (however that is a matter internal to ICT). 4. It was generally felt that attempting such a major reform during a period of lockdown when all people and systems are stressed is most politely described as foolhardy. 5. The workings of the entire department are dependent on the support staff provided by ICT and we have great concerns as to how this support network will function with such a drop in the headcount of support staff. 6. The speed at which the reforms are being driven through means that they are bound to cause significant disruption at pretty much the very worst possible time. Specific matters ----------------- The core business of the Physics Department is teaching and research. Members of the Physics Department have grave concerns as to what this reorganisation will mean for both. Teaching --------- 1. We are about to enter a multi mode teaching environment. With some students being remote and others (at least partially) in college. ICT support staff and especially the Digital Partner roles are key to us being able to develop these environments successfully. There is a great deal of knowledge of how the teaching is performed in the different department distilled into the support staff who interact with them. This is extremely valuable as the methods for teaching each subject vary greatly because of the differing nature of the subjects. To be deliberately losing the support staff with this knowledge at this time is at best foolhardy and might be considered to be insane. The existing support staff have already proven their worth through things like the implementation of remote exams. Here a solution specific to physics was required and worked out between physics academics and ICT support staff. 2. Failure to deliver good multi mode teaching carries a huge reputational risk and yet we are removing the very people who have the understanding to be able to deliver this just at the time that they would be most essential. This is simply the wrong time to be contemplating this level of restructuring from a teaching point of view. This is bound to cause very significant disruption at a time when stability and sustained development is needed. 3. The teaching machines in physics have a complex set of software installed that is required for teaching various courses and modules. This needs support and maintenance. Again we are concerned how the reduction in support staff will affect this. 4. With there being potentially large numbers of remote students with a variety of remote access problems we are concerned the reduced number of support staff will not be able cope with the expected large number of queries. 5. The AV support team say that with their numbers reduced they will not be able to provide the expected level of support. Often support from this team is needed very urgently as there is a lecture theatre full of students not being educated as there is a temporary AV problem in need of repair. Research --------- Over recent years the Research Computing Service (RCS) has provided a coherent set of services including: - High Performance Computing (HPC) - The Research Data Service (RDS) - storage compliant with the needs of the research communities and their their funding agencies - Research Software Engineers (RSEs) to develop code required by research groups. - Training for those needing to use HPC resources for their research. RCS has successfully weaned people out of the shadow ICT into using mainstream services. This has been a significant success. Previously groups were reliant on some fraction of a postdoc's time to run the computing for that group as well as write the code needed for the analysis/simulation. These postdocs were generally not computing experts and so a great many inefficiencies were introduced. When the project end and the postdoc has moved on there was no continuity of support for the data. RCS have made great strides into fixing all these problems. RCS has also provided a sufficiently large HPC platform for the users to develop simulations either to get the scientific results that they need or to be able to show that their code scales sufficiently well to be run on nation and international resources such as ARCHER or PRACE. For all these reasons RCS has been viewed as a jewel in the crown of ICT. It is also an area where the product model works well and the RCS team were expecting to have minor internal restructuring to conform to the product model but to otherwise remain as a single coherent entity. However, under the current plan they are to be completely disbanded with their functions being absorbed into a general pools associated with research products. This will result loss of expertise and the inevitable re-emergence of a shadow ICT. While this will save money within ICT it will cost very much more in the individual research groups and result in poor research outcomes. The RCS Manager is currently unable to promise resources beyond the end of July 2020 - this is not an acceptable state of affairs. Networking is important to some research groups in physics. With HEP being the largest single user of network bandwidth in the college (and one of the largest on the whole UK academic network). ICT networking have been extremely supportive to all our research needs. For example Imperial was the first academic institution to have a 100Gb/s connection and because of the strength of the networking group was amongst the first to be able to provide research area specific VRFs - a model that JISC is now planning to roll out across a wide number of subject areas. With the networking team being halved in size it is extremely doubtful that Imperial could even aspire to maintain this current leading role. We hope that these notes are useful. ======================================================================================================================================================================================== Appendix of individual e-mail responses ======================================== ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ From Robert Kingham: Thanks for convening the special committee meeting. Sorry if I lengthened the meeting by going back time and time again to my concerns about the process by which this ICT restructuring is being undertaken. Factoring that out, your notes chime with what I thought we all discussed and our collective concerns. Concerning point 2 of General Notes (of your report/minutes), concerning the "ICT supports anything & everything. ICT is expected to be ‘jack of all trades’, … ” in §2a on pg 5, I would point out that Academics are expected to be jack of all trades! I am struggling to think of another job description that requires such a wide gamut of technical, education, project management, leadership, pastoral, advisory role, on top of constant innovation and the need for ultra-specialisation. Surely if the team leading ICT restructuring have failed to grasp this, they have just - how do I say this nicely - revealed that they are not ready to undertake such a critical project. Their vision is not aligned with the actual business being restructured. This is my view, which may be a little stronger/harsher than what you expressed in the notes. My critique of the Introduction of the White Section: (crystallised since our meeting) It paints a completely negative picture of ICT at Imperial. If I had seen a list of bad and good points, I would personally feel that the rest of the document were more credible. The bottom line is surely that Imperial has maintained its ranking as one of the top universities in the UK and indeed the world for decades. Surely this would not be possible if ICT were as poor as they are purported to be? Is there evidence that we are sliding down the rankings due to ICT provision? Table 1 is possibly very misleading. Imperial College (London) is a university specialising in Science, Technology and Medicine, all disciplines which arguably require more ICT provision per beneficiary than an institution also covering the arts & humanities, law, etc. Again, have the ICT restructuring team (spectacularly) failed to grasp the business they are trying to turn around? Best wishes, Robert |————————————————————————————> Dr Robert J. Kingham Plasma Physics Group Imperial College London rj.kingham@imperial.ac.uk Professional Web Page I work flexibly and aspire to a sensible work-life balance. I’m sending this email now because it suits me, but I don’t expect you to read or respond outside of your own working hours. During the COVID-19 pandemic it may take me longer than usual to reply to emails, due to my care & education responsibilities of my school-age children. <————————————————————————————| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- From Jarvist Moore Frost (taking input from others) Dear Physics Computing Committee, I think as physicists we have a lot of case studies where our unusual needs have been met by bespoke local ICT support. I think maybe providing a dossier of evidence of this might be a useful and concrete thing we can do. My comments from the EXSS perspective: * Research Computer Services disappear as a separate entity, and seemed to be lumped in without any management in the 'portfolio function'. The provision of high performance computing at Imperial has been a real success story over the last 15 years. We have an enormous number of people in EXSS doing computational materials research, even within groups that are mainly experimental, and which would not individually be able to support this work with either hardware or expertise. This has led to some extremely high quality science, and has only been possible due to the support for training by RCS / ICT. * The multiple negative references in the white paper to the cost of supporting obsolete hardware / software from 'shadow ICT'. A lot of the EXSS experimental rigs run custom hardware & drivers often associated with obsolete versions of Windows, so this may cause issues. We might find that suddenly no help is offered for any technical issues with these old machines, and are no longer allowed to connect them to the intranet. * Something similar might be the case for the CNC machines in the workshop. * Generally we have very varied and heterogenous requirements in EXSS, and have definitely benefited from bespoke solutions and local expertise. The CMTH/EXSS 'cluster' group operations manager (Carolyn Dale) is concerned that: I am in complete shock about this and at the start of the lockdown at our first admin teams meeting from home we were asked by Luke if we had to time to complete an ICT survey about the service we receive which after what they have done now seems pointless but from my own admin team view and I made them aware of the redundancies on Friday as I had spoken to Martin Morris we are very concerned about the level of service we would receive as we often have to ask for one to one help to assist new students/visitors and problems with our own computers and group laptops so that message comes from the 3 of use in the cluster office Prof Nelson added after I distributed the above to the EXSS PIs: * EXSS has a relatively large number of students and RAs dong computational research alongside experimental work. Since as an experimental group EXSS does not have and could not support its own computational support staff, the existence of the HPC and the RCS service have been absolutely critical in allowing us to work. * EXSS staff and students have developed software that they have published open source or otherwise want to make available for wider use. The services provided by ICT have been essential in allowing that to happen. At least one such piece of software is currently forming part of a physics department REF case study. * the old MRes in plastic electronics and the new MRes that we are opening this October have a significant computational teaching part where we use networked computers and software to teach concepts in the physics and chemistry of materials. We are stepping up the computational teaching part this year due to the pandemic. The support of ICT has been critical in allowing the computational teaching to be done (I can give specific examples of help given by CT staff at the last minute), since this was not a part of standard UG or PG teaching programmes. The computational teaching is considered a relatively innovative part of our MRes programme. * I have an ERC grant largely for computational work. The support of the HPC and RCS was essential to demonstrate that the work could be done within EXSS. Others (like you?) are likely to have similar examples ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- From Dave Clements > > From: "Clements, Dave L" > Date: 8 June 2020 at 09:00:34 BST > To: "Mortlock, Daniel J" > Subject: Re: [astro-staff] proposed ICT reorganisation > >  Some thought on parts of the document: > > ICT supports anything & everything - this is a result of Imperial being a complex research environment where Imperial & ICT cannot define what researchers have to use. External bodies, such as ESA, CERN, ESO, define what systems we work with and, if we are to use them as is essential fir our research, we must have support for them. If ICT cannot provide that support, then this will have to be done at cost to the grant line, which might not always be available. The alternative is to have what research we do, what facilities we use, defined by what ICT will support, and that is not acceptable for a research led university. > > Organic responsive growth is better than top down plans for a service organisation, which is what ICT is. > > With the way College is top down managed by managers that often ignore problems raised by those lower down the hierarchy, it is no wonder that a culture of fear has developed in ICT. This is a management issue that goes right to the top. A restructure of ICT will not solve this. > > Risk not listed - loss of expertise and capability as the best who can find jobs elsewhere will take redundancy and leave, while those who can’t will stay. > > I don’t see that a product based approach can be customer centred. That would be a customer centred approach. > > 'deliver efficiencies and improve service quality' - you can’t do these two at the same time. This is a nonsense goal put into management speak. > > Other comnents: > > To be planning a huge shakeup of ICT in the middle of a pandemic lockdown which has made us much more dependent of ICT infrastructure and reliability is madness. We are expected to deliver much of our teaching remotely next year. To do that we need support staff solving our problems not being distracted by reorganisations, regrading and worried about their jobs. At the very least this needs to be delayed until a new normal has been established. > > The timing of this reorganization, and the way it has ignored the usual consultation process, is very concerning. It is as if management want to slip this through while we are distracted. This kind of action is why there is an atmosphere of fear and distrust in ICT and much of the rest of the College - if management is playing these games with ICT, what will they mess with next? > > This whole plan strikes me as a macho management push by the new CIO to justify his existence and to show off to more senior management. It has been rushed through in just a few months while ICT is labouring, really rather well, under the huge extra demands of COVID19. Responses to focus groups etc. during this time will not reflect the usual state of business so decisions should not be based on them. > > We need to return to stability, get the huge demands of online teaching, exams, remote working etc.out of the way before any of this can be considered. > > The risks of failure that these changes bring to online teaching & all the rest are too great to make these changes in such a rushed way at such a critical time. > > The new CIO should be censured for proposing this at this time and in a way that has bypassed the usual processes. > > Dave > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- From Rich Bantges Some specifics examples from SPAT where ICT have performed critical roles, and still are: (not a comprehensive sweep by any means, but certainly shows that ICT is NOT failing in all aspects): Rich ----------------------------------------------------------------------------------------------------------------------------------------- [1] Here is an example of how the RSE team helped solve a unique problem for the Physics Department's Cassini team - ensuring the continuity of provision of unique data to the community. The text below was submitted to the "Presidents Award for Research Support" (I don't know the outcome). Statement of support Please include a statement of support from an individual not involved in the nominee's direct line of management Dr Richard Bantges, Department of Physics: The scientific elements of the MAGDA data preservation project precluded employing a contract web development team but the expertise of the Research Computing Service was a perfect fit. The RSE team quickly understood the complexities of the problem at hand, rapidly prototyping web pages and visualisations for us to review. Without the help of Chris Cave-Ayland and Mark Woodbridge in particular, we would have struggled to meet our commitment to maintain a web-based quick-look visualisation tool for accessing and distributing Cassini mission data for the benefit of the global research community. Further to our expectations the team showed an attention to detail and a pro-active approach that was not only effective but very reassuring. They provide an invaluable and unique service to the research community at Imperial. [Richard is the Scientific Project Manager for MAGDA, a separate project to which the RSE team have recently contributed. This work was commissioned by the Head of the Department of Physics and sponsored by the Vice-Provost for Research and Enterprise via the College's Strategic Development Fund] ----------------------------------------------------------------------------------------------------------------------------------------- [2] Another example, of how ICT has been successful was the support from ICT for the Cassini end of mission phase (Apr-Sep 2017). This required: 1. Set up of a dedicated Cassini Virtual Team - consisting of support members from the OS Team, Networks and Data Centre 2. Provision by the OS Team of a series of customised virtual machines to act as backup systems for Windows Server 2000 systems that could not be upgraded 3. Specialist UNIX support to help support SOLARIS 10 (SPARC) systems 4. Advanced network support to ensure seamless dedicated VPN links between Imperial College London and NASA's JPL 5. Liaisons between the College's Estates division and Networks to provide multiple electrical power sources for computing and network infrastructure resilience Certain aspects of the above were called upon in the final stages of the mission, and ICT helped ensure that the Imperial College Cassini team were able to monitor data from its magnetometer onboard the Cassini spacecraft (built in the Magnetometer lab in Huxley 6M) thus ensuring the highest quality science data were obtained at end of mission. ------------------------------------------------------------------------------------------------------------------------------------------- [3] Currently the space mission, Solar Orbiter, with another of SPAT's magnetometers on board, is relying on RCS hosted VMs to provide near realtime data analysis and monitoring. This solution was chosen for resilience and the level of service offered by the RCS. ------------------------------------------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ From Robert Kingham Hi Dave & All, Just a few quick examples of how ICT affect my research & teaching. I believe that my colleagues in PLAS have similar opinions about the research: I believe that the current ICT setup has proved to be effective & resilient through its rapid and often bespoke response in keeping teaching and research working through the current pandemic, especially in the first critical weeks of lockdown. Research – The Research Computing Service at Imperial is the envy of my colleagues employed at other universities, both in the UK and internationally. The RCS has provided a first class service at all levels; HPC hardware, provision of training, expert advice about HPC code engineering, etc. It provides my research team (and the research teams of other colleagues) the basis to go on and secure access to national (e.g. ARCHER) and international (e.g. PRACE) HPC. It enables us to pump-prime and optimise our codes, which are pre-requisites for tier-1 and tier-0 HPC access. It also allows us to perform modest production runs that mean our high-fidelity production runs on tier-1 and tier-0 HPC etc., can be obtained with less resource (or alternatively, we get more quantity and/or quality of publishable results for the fixed resource that we can competitively obtain) . All my 12 PhD students since 2004 used/use it. All my 4 postdocs have used it. Their training and research output would not have been possible without it. An example of the value that RCS has added: From this set, 4 hold faculty positions (or equivalent in national labs). The research teams of (at least) 7 out of 10 academics in PLAS use Imperial RCS (i.e cx1) in their research. The 4 non-experimental academics (like me) use it extensively. Most of the experimental teams perform HPC (e.g. 2D3V particle-in-cell, or 3D magnetohydrodynamic) support simulations. You can’t publish experimental results in PRL or other high-impact journals in our field without this these days! Teaching – ICT has provided a very stable, consistent and useful platform for delivering UG teaching. It could be better. It could be a lot worse. But it has a track record of being fit enough for purpose, through the 220+ students Physics graduates each year. I run the 2nd year UG computing module in my department. (I have had the pleasure(?) of doing this for ~10 years since 2005. I have also lectured the 3rd year Computational Physics course for 3 years. The whole cohort takes 2nd year computing. Passing it is a requirement for UG progression. ICT currently understands that the best solution for the needs of the people like me, tasked with delivering the actual teaching (i.e. academics) is not necessarily the best solution from a purely "computing technology" point of view. Every time the provided software is changed, I need to invest a considerable amount of time to effectively debug the course materials. This is time taken away from improving the academic quality of the material, time taken away from personal interaction with students, time taken away from writing grant proposals and running my research team, and a quite simply a loss of "space" to be truly innovative in research and teaching. I believe that the current ICT staff and management understands this. The ICT restructuring, led by someone with no prior university operational experience, is happening so fast, that I fear that it cannot possible capture such requirements. (Happy to be proved wrong, of course!) Sorry it is a bit wordy. “Research” is probably the most evidenced example of ICT’s worth. “Teaching” probably best exemplifies my serious concerns about the whole ICT restructuring process : Speed & inappropriate timing. Lack of understanding of the end-users’ needs and priorities (or at least that of a typical “Academic”). #1 priority is to deliver teaching for 2020/21 and have ICT stability (known environment, sufficient ICT bodies in place, personal connections to get things done; heck … to have a functioning ICT at all!) to deliver what is looking to be a challenging 2021/22 teaching wise. #2 priority is to have a functioning HPC service. Best wishes, Robert