February 2015 - Physics-Departmental-Computing

Re: [Physics-Departmental-Computing] UG batch computing facilities ?
by David Colling 24 Feb '15

24 Feb '15

Thanks Robert, Yes, everybody is really bogged under (I know that I am at the moment ;-) ). I think that condor can provide a High Throughput engine. There was also an ill-defined feeling at the meeting that Physics should provide some storage for things projects like this. Although nobody had a good feel for how much this should be. I am going to mail Kenny (who was aware that this might be a possibility) suggesting that we buy something modest but easily expandable if it becomes popular. My guess, and it is no more than that, is that it will take a couple of years to really be used by anybody, but that if we don't put something there it will never be used. Best, david On 24/02/15 10:59, Robert Kingham wrote: > Hi David, > > I've had little response from my colleagues, which is odd because I am sure that > a good few of the projects listed this year would need them. I guess people are > bogged down at the moment. > > From the list on > http://www3.imperial.ac.uk/physicsuglabs/msciprojects/projectproposals#plas > > > > ... I would say that the following would benefit from such a resource > > Kingham-"Simulations of exotic electron transport phenomena in > inertial-confinement fusion" > Coppins-"Computer simulation of dusty plasma crystals" > Smith-"Modelling Vacuum Optical Levitation Traps" > Chittenden-"Modelling diagnostic data from Inertial Confinement Fusion Experiments" > Mangles-"Target design and characterisation for a compact high-repetition-rate > laser wakefield accelerator" > > For my project I would need very roughly (per student) > > - Computational load: 20x 24 hours runs = 480 CPU hours > - Storage: 20x 5GB = 100 GB(output rather than big data set > to read in) > - Software: Linux environment, Matlab > > Best wishes, > > Robert > > > On 16 Feb 2015, at 18:08, David Colling wrote: > >> Hi All, >> >> >> ################################ >> # Action >> ################################# >> Please could you try to find the computational needs (or even just >> "would like"s) of projects that your group would put forward. >> >> These include: >> >> - Computational load (how many CPUs for how long ... yes I know that >> this is hard) >> >> - Storage: How much data would you like the students to be able to >> access from their compute nodes? >> >> - Software (Matlab, mathematica, root,...) >> >> - unusual (non-windows) environments >> >> >> So Adrian, Mike and I met this afternoon to discuss this and there are a >> number of different solutions for different scales of problems. I think >> that we can stitch these together into a very good solution for a wide >> variety of projects. >> >> So what we discussed were: >> >> - The condor pool is still there however the interface is through a web >> page. I am going to try this out over the next few days and talk to ICT >> about how we could best use it for different projects in physics. >> >> - Storage. With the possibility of people proposing "Big Data" style >> projects we thought that there might be a case for buying some physics >> specific storage for such projects and making it visible from the condor >> nodes. We discussed scales of 50 to 100 TB sort of scale. P{roject >> supervisors would be able to give access to their students only if they >> so wished. >> >> - Non- windows environments. There are redhat VMs installed on the >> machines down stairs, but for anything more specific Imperial has a >> contract with MS azure and it we can set up projects requiring specific >> environments in VMs machines on this cloud. I will be given access over >> the next couple of days and will have a look around. >> >> - If projects require MPI computing we can create accounts on the >> college HPC for them. >> >> Clearly some of these things are (almost) free and others are not. So it >> is important to find out what we need and what we would use if we had it. >> >> >> best, >> david >> >> On 14/02/15 18:10, David Colling wrote: >>> Hi Julia, >>> >>> (adding the computing list and Kenny to aid communications). >>> >>> Yes, this is something that has come from time to time since I have been >>> involved in the computing committee. At one stage we almost bought a >>> dedicated (small) set of servers for this purpose but then decided not >>> to for some (good) reason that I cannot remember. Looking at the >>> machines downstairs I can see that they are actually running condor >>> (which makes them a useful resource). However I think that this is not >>> enough by itself - obviously no documentation is a problem but there is >>> more than that. I can see situations where access to considerables >>> amount of data might be useful etc or to specific environments with >>> specifically licensed software. I can see a number of ways of dealing >>> with this (although they may require a modest outlay - hence Kenny's >>> inclusion ;-) ). >>> >>> I think that the best approach might be to organise a meeting where we >>> can decide this once and for and provide a good working environment. I >>> will set up a doodle poll (although probably not until Monday). >>> >>> Adrian, perhaps you and could meet up earlier to discuss the options? >>> >>> Best, >>> david >>> >>> >>> >>> On 13/02/15 14:39, Julia Sedgbeer wrote: >>>> Hi Dave, Ray, >>>> >>>> I'm emailing you in your capacity as members of the >>>> departmental computing ctte. >>>> >>>> Tim has asked about computing facilities for projects >>>> (see email below). This comes up every year; some >>>> projects use research group facilities, some depend >>>> on students having their own laptops. Is there a >>>> way to make better use of the computing power of the >>>> teaching cluster for projects, e.g. a batch system >>>> (assuming that there currently isn't one) or similar? >>>> >>>> Cheers, >>>> Julia >>>> >>>> >>>> ---------- Forwarded message ---------- >>>> Date: Fri, 13 Feb 2015 08:23:28 +0000 >>>> From: "Evans, Tim S" <t.evans(a)imperial.ac.uk <mailto:t.evans@imperial.ac.uk>> >>>> To: "Sedgbeer, Julia K" <j.sedgbeer(a)imperial.ac.uk >>>> <mailto:j.sedgbeer@imperial.ac.uk>> >>>> Subject: RE: MSci Proposals >>> Dear Julia (cc'ing the departmental list to help communications), >>> >>> >>> >>> >>> >>> >>>> Dear Julia, >>>> >>>> Do you know if there is any specific dept/college support for student >>>> project computing needs? Or who to ask about this? Some of my Erasmus >>>> Bsc students may need some extra power. There was a system I made a >>>> note of but that doesn't seem to exist (see below). >>>> >>>> Thanks >>>> >>>> Tim >>>> >>>> Old system:- >>>> Every year students ask about how they can use cluster computers >>>> for long running experiments/projects and we've been working to >>>> provide a solution to this. Condor is an open source package >>>> which is designed to queue up jobs and submit them to computers >>>> when they're not busy; it's been used on Linux by DoC for some >>>> time but we now have it available for Windows users. Any >>>> student who wants to run code which they've compiled or a >>>> .m file from Matlab can submit a job via Condor using the >>>> website https://icwincondor1.cc.ic.ac.uk/condor.A couple of >>>> key points are that the job must run non-interactively and >>>> must not try to access the user's H: drive (or any other >>>> specific location - the code should assume that all files >>>> are in the current directory) >>> >>> >>> _______________________________________________ >>> Physics-Departmental-Computing mailing list >>> Physics-Departmental-Computing(a)imperial.ac.uk >>> <mailto:Physics-Departmental-Computing@imperial.ac.uk> >>> https://mailman.ic.ac.uk/mailman/listinfo/physics-departmental-computing >>> >> >> >> _______________________________________________ >> Physics-Departmental-Computing mailing list >> Physics-Departmental-Computing(a)imperial.ac.uk >> <mailto:Physics-Departmental-Computing@imperial.ac.uk> >> https://mailman.ic.ac.uk/mailman/listinfo/physics-departmental-computing > > > -- > ============================================================================= > *_Dr Robert J. Kingham_* > Plasma Physics Group, Room 724 Blackett Laboratory, > Imperial College London, London SW7 2AZ, UK > Tel:+44 (0)207-5947637 > E-mail: rj.kingham(a)imperial.ac.uk <mailto:rj.kingham@imperial.ac.uk> > Web: Professional Web Page <http://www3.imperial.ac.uk/people/rj.kingham> > ============================================================================= > > > > > >

1 0

Physics Departmental computing ..
by David Colling 24 Feb '15

24 Feb '15

Dear Lucy, At our last Physics Departmental Computing meeting Adrian suggested that you be added to our mailing list. So I thought that I would mail you asking whether or not you would before do so. It is pretty low volume. There was also a direct question that came up that you might be able to help with. The question was is there a location where academics could edit their own web pages? Somewhere less formulaic than the professional webpages . These was felt to be a need for such a space and I think that some other departments have this. Best, david

1 0

meeting tomorrow
by David Colling 22 Feb '15

22 Feb '15

Dear all, Tomorrow's meeting will start at 12:00 in room 532. Feel free to bring your lunch along. The items that I have for the agenda are: - resources for students projects. - Oracle linux - Web pages for academics - Insurance for accidental damage/loss of laptops and other computing equipment. After this we will have the usual: - status and plans from ICT - any news from groups - AOB If you would like me to add anything extra to the agenda then please email me before the meeting. Best, david

1 0

Last chance to doodle poll
by David Colling 19 Feb '15

19 Feb '15

Dear All, It is your last chance to fill in the doodle poll for the next exciting physics departmental computing meeting. I want to close this poll sometime tomorrow morning. Admittedly at the last minute I thought that it might be an idea to invite Julia and Kenny as discussing compute and storage needs for MSci/MSc projects will be one of the topics on the agenda and so they would be welcome if they can/want to come along. The poll is at: http://doodle.com/h97zmwcrf4k9w59nytnin5r3/admin#table and currently the favourite is next Monday (sorry Jony). If we go for Monday a start soon after 12:00 would be useful for me ass I have aa 'phone conference at 1pm (but for which I can be late). Best, david

1 1

Re: [Physics-Departmental-Computing] UG batch computing facilities ?
by David Colling 16 Feb '15

16 Feb '15

Hi Julia, (adding the computing list and Kenny to aid communications). Yes, this is something that has come from time to time since I have been involved in the computing committee. At one stage we almost bought a dedicated (small) set of servers for this purpose but then decided not to for some (good) reason that I cannot remember. Looking at the machines downstairs I can see that they are actually running condor (which makes them a useful resource). However I think that this is not enough by itself - obviously no documentation is a problem but there is more than that. I can see situations where access to considerables amount of data might be useful etc or to specific environments with specifically licensed software. I can see a number of ways of dealing with this (although they may require a modest outlay - hence Kenny's inclusion ;-) ). I think that the best approach might be to organise a meeting where we can decide this once and for and provide a good working environment. I will set up a doodle poll (although probably not until Monday). Adrian, perhaps you and could meet up earlier to discuss the options? Best, david On 13/02/15 14:39, Julia Sedgbeer wrote: > Hi Dave, Ray, > > I'm emailing you in your capacity as members of the > departmental computing ctte. > > Tim has asked about computing facilities for projects > (see email below). This comes up every year; some > projects use research group facilities, some depend > on students having their own laptops. Is there a > way to make better use of the computing power of the > teaching cluster for projects, e.g. a batch system > (assuming that there currently isn't one) or similar? > > Cheers, > Julia > > > ---------- Forwarded message ---------- > Date: Fri, 13 Feb 2015 08:23:28 +0000 > From: "Evans, Tim S" <t.evans(a)imperial.ac.uk> > To: "Sedgbeer, Julia K" <j.sedgbeer(a)imperial.ac.uk> > Subject: RE: MSci Proposals Dear Julia (cc'ing the departmental list to help communications), > Dear Julia, > > Do you know if there is any specific dept/college support for student > project computing needs? Or who to ask about this? Some of my Erasmus > Bsc students may need some extra power. There was a system I made a > note of but that doesn't seem to exist (see below). > > Thanks > > Tim > > Old system:- > Every year students ask about how they can use cluster computers > for long running experiments/projects and we've been working to > provide a solution to this. Condor is an open source package > which is designed to queue up jobs and submit them to computers > when they're not busy; it's been used on Linux by DoC for some > time but we now have it available for Windows users. Any > student who wants to run code which they've compiled or a > .m file from Matlab can submit a job via Condor using the > website https://icwincondor1.cc.ic.ac.uk/condor.A couple of > key points are that the job must run non-interactively and > must not try to access the user's H: drive (or any other > specific location - the code should assume that all files > are in the current directory)

1 1

Doodle poll for meeting
by David Colling 16 Feb '15

16 Feb '15

Dear All, I think that we have a number of things to discuss and so I have set up a doodle poll for this. OK I suspect that tomorrow is really unlikely unless everybody replies instantly - which is unlikely given that it is half term. Anyway fill it in as much as you can as quickly as you are able. Best, david

1 2

REMINDER: Redhat to Oracle - quick survey
by Bantges, Richard J 12 Feb '15

12 Feb '15

Dear All, I've had a couple of responses (thank you), but it would be helpful to have a wider representation of the thoughts of research groups in the Physics Department. There are only 4 quick questions, so if others could respond that would be appreciated. David - when will be the next Dept. Computing meeting roughly? Thanks in advance, Rich From: Bantges, Richard J Sent: 10 February 2015 15:19 To: physics-departmental-computing Subject: Redhat to Oracle - quick survey Dear Physics Dept. Computing Reps, Just in case you were unaware, a few months ago the College (someone at ICT) decided it was necessary to move from Redhat to Oracle Linux (to enable the "Blue Box" to legally use Linux - or at least that's my understanding of the reason for changing). SPAT has a large (20+) number of servers which run Redhat Linux, which are no longer supported by ICT. ICT have admitted they do not have the resources to determine whether there are any issues with migrating from Redhat to Oracle for SPAT, only that "they should be pretty much the same". I'm looking to arrange a meeting with Mike Russell to discuss amongst a few things, how such a decision was made without any (or at least I'm not aware of any) discussion with the research groups at Imperial. However, before I do so, I would like to know the following (a brief response either way would be appreciated): 1. Were you or your group consulted regarding this change? 2. Does your group rely on Redhat Linux for its core computational ability? 3. If the answer to (2) is yes - have you any experience of using the new Oracle Linux and are there any noticeable impacts asscociated with migrating from Redhat to Oracle Linux? (e.g. compatibility between existing Redhat Enterprise Server and Oracle Linux in a mixed environment, etc.). 4. Again, if the answer to (2) is yes - will your group be purchasing (or have you already purchased) a Redhat Support contract (I'm aware that at least one Group in the College has done so)? David - if there's room in everyone's schedule and it's deemed more appropriate to discuss this at a Dept. Computing Meeting then I'm happy to wait to potentially discuss this then.. Otherwise, I'd appreciate a brief response so that I can better understand whether this is an annoyance for just SPAT or the wider Physics Department. Thanks, Rich ------------------------------------------- Dr Richard Bantges Scientific Project Manager Space & Atmospheric Physics Imperial College London LONDON SW7 2AZ Tel. +44 207 594 7664 Fax. +44 207 594 7772 -------------------------------------------

1 0

Redhat to Oracle - quick survey
by Bantges, Richard J 10 Feb '15

10 Feb '15

Dear Physics Dept. Computing Reps, Just in case you were unaware, a few months ago the College (someone at ICT) decided it was necessary to move from Redhat to Oracle Linux (to enable the "Blue Box" to legally use Linux - or at least that's my understanding of the reason for changing). SPAT has a large (20+) number of servers which run Redhat Linux, which are no longer supported by ICT. ICT have admitted they do not have the resources to determine whether there are any issues with migrating from Redhat to Oracle for SPAT, only that "they should be pretty much the same". I'm looking to arrange a meeting with Mike Russell to discuss amongst a few things, how such a decision was made without any (or at least I'm not aware of any) discussion with the research groups at Imperial. However, before I do so, I would like to know the following (a brief response either way would be appreciated): 1. Were you or your group consulted regarding this change? 2. Does your group rely on Redhat Linux for its core computational ability? 3. If the answer to (2) is yes - have you any experience of using the new Oracle Linux and are there any noticeable impacts asscociated with migrating from Redhat to Oracle Linux? (e.g. compatibility between existing Redhat Enterprise Server and Oracle Linux in a mixed environment, etc.). 4. Again, if the answer to (2) is yes - will your group be purchasing (or have you already purchased) a Redhat Support contract (I'm aware that at least one Group in the College has done so)? David - if there's room in everyone's schedule and it's deemed more appropriate to discuss this at a Dept. Computing Meeting then I'm happy to wait to potentially discuss this then.. Otherwise, I'd appreciate a brief response so that I can better understand whether this is an annoyance for just SPAT or the wider Physics Department. Thanks, Rich ------------------------------------------- Dr Richard Bantges Scientific Project Manager Space & Atmospheric Physics Imperial College London LONDON SW7 2AZ Tel. +44 207 594 7664 Fax. +44 207 594 7772 -------------------------------------------

2 1