Thanks Robert, Yes, everybody is really bogged under (I know that I am at the moment ;-) ). I think that condor can provide a High Throughput engine. There was also an ill-defined feeling at the meeting that Physics should provide some storage for things projects like this. Although nobody had a good feel for how much this should be. I am going to mail Kenny (who was aware that this might be a possibility) suggesting that we buy something modest but easily expandable if it becomes popular. My guess, and it is no more than that, is that it will take a couple of years to really be used by anybody, but that if we don't put something there it will never be used. Best, david On 24/02/15 10:59, Robert Kingham wrote:
Hi David,
I've had little response from my colleagues, which is odd because I am sure that a good few of the projects listed this year would need them. I guess people are bogged down at the moment.
From the list on http://www3.imperial.ac.uk/physicsuglabs/msciprojects/projectproposals#plas
... I would say that the following would benefit from such a resource
Kingham-"Simulations of exotic electron transport phenomena in inertial-confinement fusion" Coppins-"Computer simulation of dusty plasma crystals" Smith-"Modelling Vacuum Optical Levitation Traps" Chittenden-"Modelling diagnostic data from Inertial Confinement Fusion Experiments" Mangles-"Target design and characterisation for a compact high-repetition-rate laser wakefield accelerator"
For my project I would need very roughly (per student)
- Computational load: 20x 24 hours runs = 480 CPU hours - Storage: 20x 5GB = 100 GB(output rather than big data set to read in) - Software: Linux environment, Matlab
Best wishes,
Robert
On 16 Feb 2015, at 18:08, David Colling wrote:
Hi All,
################################ # Action ################################# Please could you try to find the computational needs (or even just "would like"s) of projects that your group would put forward.
These include:
- Computational load (how many CPUs for how long ... yes I know that this is hard)
- Storage: How much data would you like the students to be able to access from their compute nodes?
- Software (Matlab, mathematica, root,...)
- unusual (non-windows) environments
So Adrian, Mike and I met this afternoon to discuss this and there are a number of different solutions for different scales of problems. I think that we can stitch these together into a very good solution for a wide variety of projects.
So what we discussed were:
- The condor pool is still there however the interface is through a web page. I am going to try this out over the next few days and talk to ICT about how we could best use it for different projects in physics.
- Storage. With the possibility of people proposing "Big Data" style projects we thought that there might be a case for buying some physics specific storage for such projects and making it visible from the condor nodes. We discussed scales of 50 to 100 TB sort of scale. P{roject supervisors would be able to give access to their students only if they so wished.
- Non- windows environments. There are redhat VMs installed on the machines down stairs, but for anything more specific Imperial has a contract with MS azure and it we can set up projects requiring specific environments in VMs machines on this cloud. I will be given access over the next couple of days and will have a look around.
- If projects require MPI computing we can create accounts on the college HPC for them.
Clearly some of these things are (almost) free and others are not. So it is important to find out what we need and what we would use if we had it.
best, david
On 14/02/15 18:10, David Colling wrote:
Hi Julia,
(adding the computing list and Kenny to aid communications).
Yes, this is something that has come from time to time since I have been involved in the computing committee. At one stage we almost bought a dedicated (small) set of servers for this purpose but then decided not to for some (good) reason that I cannot remember. Looking at the machines downstairs I can see that they are actually running condor (which makes them a useful resource). However I think that this is not enough by itself - obviously no documentation is a problem but there is more than that. I can see situations where access to considerables amount of data might be useful etc or to specific environments with specifically licensed software. I can see a number of ways of dealing with this (although they may require a modest outlay - hence Kenny's inclusion ;-) ).
I think that the best approach might be to organise a meeting where we can decide this once and for and provide a good working environment. I will set up a doodle poll (although probably not until Monday).
Adrian, perhaps you and could meet up earlier to discuss the options?
Best, david
On 13/02/15 14:39, Julia Sedgbeer wrote:
Hi Dave, Ray,
I'm emailing you in your capacity as members of the departmental computing ctte.
Tim has asked about computing facilities for projects (see email below). This comes up every year; some projects use research group facilities, some depend on students having their own laptops. Is there a way to make better use of the computing power of the teaching cluster for projects, e.g. a batch system (assuming that there currently isn't one) or similar?
Cheers, Julia
---------- Forwarded message ---------- Date: Fri, 13 Feb 2015 08:23:28 +0000 From: "Evans, Tim S" <t.evans@imperial.ac.uk <mailto:t.evans@imperial.ac.uk>> To: "Sedgbeer, Julia K" <j.sedgbeer@imperial.ac.uk <mailto:j.sedgbeer@imperial.ac.uk>> Subject: RE: MSci Proposals Dear Julia (cc'ing the departmental list to help communications),
Dear Julia,
Do you know if there is any specific dept/college support for student project computing needs? Or who to ask about this? Some of my Erasmus Bsc students may need some extra power. There was a system I made a note of but that doesn't seem to exist (see below).
Thanks
Tim
Old system:- Every year students ask about how they can use cluster computers for long running experiments/projects and we've been working to provide a solution to this. Condor is an open source package which is designed to queue up jobs and submit them to computers when they're not busy; it's been used on Linux by DoC for some time but we now have it available for Windows users. Any student who wants to run code which they've compiled or a .m file from Matlab can submit a job via Condor using the website https://icwincondor1.cc.ic.ac.uk/condor.A couple of key points are that the job must run non-interactively and must not try to access the user's H: drive (or any other specific location - the code should assume that all files are in the current directory)
_______________________________________________ Physics-Departmental-Computing mailing list Physics-Departmental-Computing@imperial.ac.uk <mailto:Physics-Departmental-Computing@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/physics-departmental-computing
_______________________________________________ Physics-Departmental-Computing mailing list Physics-Departmental-Computing@imperial.ac.uk <mailto:Physics-Departmental-Computing@imperial.ac.uk> https://mailman.ic.ac.uk/mailman/listinfo/physics-departmental-computing
-- ============================================================================= *_Dr Robert J. Kingham_* Plasma Physics Group, Room 724 Blackett Laboratory, Imperial College London, London SW7 2AZ, UK Tel:+44 (0)207-5947637 E-mail: rj.kingham@imperial.ac.uk <mailto:rj.kingham@imperial.ac.uk> Web: Professional Web Page <http://www3.imperial.ac.uk/people/rj.kingham> =============================================================================