Physics-Departmental-Computing

Download

physics-departmental-computing@imperial.ac.uk

April 2021

2 participants
2 discussions

Fwd: Status update on filestore and RCS communications
by David Colling 29 Apr '21

29 Apr '21

Please circulate to anybody who might find this useful... -------- Forwarded Message -------- Subject: Status update on filestore and RCS communications Date: Tue, 27 Apr 2021 19:23:49 +0000 From: Clifford, Simon J <s.clifford(a)imperial.ac.uk> Reply-To: Mclean, Andrew <andrew.mclean(a)imperial.ac.uk> To: Mclean, Andrew <andrew.mclean(a)imperial.ac.uk> Hello all, This email has two parts. The first is about the ongoing issues with the Research Data Service (RDS), the second is about our email communications. The RDS is what provides around 11 petabytes of data storage space to the HPC cluster's nodes and logins. It is also accessible as a shared drive. For over a year now it has been suffering an assortment of problems which I will now try to explain. In 2019 we implemented a new parallel storage filesystem, called GPFS, designed to scale to our workloads. It immediately exposed problems with some of the infrastructure switches of the cx2 nodes, which caused system instability. These switches should have worked, but didn't. It is not possible to replace, repair or work around them. As a temporary measure a simpler filesystem, NFS, was put in place across the entire cluster. NFS does not perform well at the scale of our systems, but it is in some ways less sensitive to network instability. Most of the issues currently being experienced on the cluster now are due to NFS struggling to keep up. The best solution is to work towards enabling GPFS on the new cx3 nodes leaving older nodes on NFS. The reduced load on NFS should make its lack of scalability irrelevant. Work has been ongoing to implement this; a major blocker is that the new infrastructure has been put into our network using only IPv6 addresses and this causes conflicts with our existing mixed IPv4 / IPv6 kit. A "big bang" implementation of the network redesign would necessitate an outage of several weeks so we have instead been using an incremental strategy in order to keep the system available as much as possible. This has introduced some unforeseen issues which you will have noticed over the last few weeks. ICT Networks, IBM, and others are assisting in resolving these. We anticipate the migration of cx3 to GPFS should be complete before the end of May. Please be reassured that none of these issues will affect the safety of your data. And while they are very frustrating, for most running jobs the 'hangs' are not terminal -- the job will just wait until the storage is available again. Regarding the RCS's communications: it has become apparent, through some feedback that we are not communicating enough to our users. We apologise for this. We are still quite understaffed, and when a crisis appears our instincts are to fix it as soon as possible. Spending time notifying our users beyond a brief line on the status page (https://api.rcs.imperial.ac.uk/service-status) is perceived as time not spent addressing the problem. However, this mailing list is almost unused, apart from reminders of service outages. We intend to use it more on matters that will still be strictly relevant to the cluster and RDS, addressing service issues and software installs as well as maintenance. We will be guided by your feedback on this.

1 0

IMPORTANT: Vulnerability Scanning of Servers
by Bantges, Richard J 20 Apr '21

20 Apr '21

Dear Physics Dept. Computing Reps., If you have network attached storage (NAS) devices or processing servers attached to the College network that are either hosted in one of the College's Data Centres (DCs) or on one of the College's campuses, please see the email below. Please note, the intention is to check for security vulnerabilities only, and whilst compliance with this request is currently on a good-will basis, there are clear indications that those who do not allow their servers to be scanned for vulnerabilities may find their servers are banned from the College network in future. I would strongly advise you, or those in your research group responsible for managing your servers to be proactive and get in touch with Elton. This is a college-wide exercise instigated by the senior most members of the College as there is great concern over the security of the servers they (ICT) have little visibility of. In addition to the security vulnerability scans, ICT has plans to roll out a remote management application ("Device42") that may reduce the day-to-day monitoring type activities of servers on the network (e.g. looking for failed hardware, corrupt or full filesystems, etc.). This will be optional, but appears to be a positive step towards ICT providing more support for research group computing for those of us that have our own servers for whatever reason. More details on that once the roll out and benefits / implications become clear. I would be grateful if this applies to your research Group (i.e. you have NAS devices, etc.) that when you reply to Elton's email below that you CC me on your reply so that I am aware of your participation/cooperation. Any questions about the security scanning above, I would suggest directing these to Elton in the first instance. Many thanks, Rich From: Sheffield, Elton <e.sheffield(a)imperial.ac.uk> Sent: 13 April 2021 11:34 To: Sheffield, Elton <e.sheffield(a)imperial.ac.uk> Cc: Lax, Andy J <a.lax(a)imperial.ac.uk> Subject: Vulnerability Scanning of Servers Hi As ICT presented in the "Data centre Hosting/ICT Strategy & support" meeting on the 11th of March, as part of ICT's commitment to providing world-class services we've been looking at the footprint in our Data Centres and how we can best support the wider organisation. To that end, we are initially looking to vulnerability scanning all servers within the data centre to ensure our environment is secure. The service uses agentless technology, so we would only require a login to your servers with basic permissions (for some functionality there can be optional elevated permission), the overheads of the scanning are very low as the scans only run daily. By scanning your servers we will be able to identify and report on any security vulnerabilities, and the reports will include details of the vulnerabilities as well as guidance regarding how you can address the vulnerabilities, this will allow you to act and rectify any security risks before they are exploited. This scanning will also highlight any missing OS or application patches so they can be applied promptly (for those interested this can also help support Cyber Essentials Plus or the DSP Toolkit submissions). The following link will give you an overview of the NESUS scanning platform, as well as information about the permissions required. https://docs.tenable.com/nessus/Content/NessusCredentialedChecks.htm https://www.tenable.com/products/nessus/nessus-professional-b For each department, we'll create a separate set of SSH keys for access to Linux servers, and for any Windows servers we'll create separate AD accounts (if you don't use the central AD service, we will need you to create a user for us), to allow us to set up these items could you confirm the OS types you are running, and also if you have any virtualisation layers? Regards Elton IT Security Incident and Event Management SME

1 0