Meet our latest supercomputer Ruby, an SGI UV3000 named after Australia’s pioneering radio astronomer Ruby Payne-Scott.
The international scientific research community is one of the leading users of high performance computing (HPC), and being one of the most diverse scientific research organisations in the world, we’ve got a whole heap of researchers looking for that bit of extra grunt to help drive their research to new heights.
Luckily for them, we’ve got them (very well) covered with our range of high performance computers.
Didn’t these things used to take up a whole building?
We have a long and proud history of building and maintaining high performance computers at CSIRO, dating all the way back to 1949 when we built Australia’s very first digital computer, CSIRAC.
But times have been a-changin since then, and what used to take up an entire room (CSIRAC wasn’t big enough to take up a whole building) now fits into racks and pods in custom designed temperature controlled rooms right next to other computing hardware.
Meet Ruby, our newest super (computing) star
We recently welcomed a new machine into the fold. Her name is Ruby, and she’s an SGI UV3000 named after Australia’s pioneering radio astronomer Ruby Payne-Scott.
Ruby is the latest in a line of shared-memory supercomputers we’ve been using from SGI since 2003, when we introduced an SGI Altix 3700 system named Cherax.
She’s packing 640 cores and eight terabytes of memory at her heart, but her real trick is using SGI’s NUMAlink6 technology to connect all those cores together to appear as one ‘giant PC’.
Ruby is based on a NUMA (non uniform memory architecture) design, which means that she can share her memory across multiple processors. This allows all the processing cores access to the memory like it’s their own, and to see what is being stored in the total amount of memory, allowing each process/core to share information, instead of having its own segregated memory in a traditional distributed memory architecture.
This makes her perfect for tackling data intensive tasks requiring large amounts of memory that benefit from the fast interconnection between the memory and processors.
Our users are anything but average
Our Information Management Technology (IMT) team work closely with researchers from an extremely broad range of disciplines to make sure their work is delivering its maximum impact by fully harnessing the capabilities our HPC resources can offer.
By planning things like the type, format and amount of data they’re producing, our IMT team help our researchers determine how to prepare and manage their data and computing needs during and after the life of the project, while making it as easy as possible to work with research collaborators across organisational boundaries.
Researchers at CSIRO are using Ruby for a range of science and engineering such as bioinformatics, genome assembly, climate modelling, nanoscience and imaging and image processing.
It’s like doing a jigsaw puzzle without the picture on the box
Denis Bauer is a scientist in our Health & Biosecurity unit who specialises in biomedical informatics and smart analytics tools.
She uses Ruby (and before that, Ruby’s predecessors Cherax) for analysing next generation sequencing data to look at specific chunks of the human genome.
It’s highly data and memory intensive work, with each genome of up to three billion base pairs being read on average around 100 times.
Her work is helping identify areas in the human genome that cause disease so we can continue to find out more about them. Some areas of focus so far are renal disease, colorectal cancer and Motor Neurone Disease (MND, also known as ALS or Amyotrophic lateral sclerosis).
Denis is using next generation sequencing technologies that have only been available for the last five years, to be able to quickly sequence individual people. The extra computing power that systems like Ruby provide is the backbone to making this new reality cheaper and more accessible than ever before.
Ruby has been especially useful for Denis’s genome sequencing work, enabling her to load an entire genome into memory at once.
“This type of work benefits from having hundreds of cores able to process the work simultaneously, but as well as the processing power, it needs unprecedented amounts of memory to run.” Denis said.
“While the jobs can be run in parallel, they can’t be run in isolation, as they are all interlinked. This means that all the data needs to be loaded into memory so the computer can see all the information to be able to piece the genomic jigsaw puzzle back together. It’s like doing a jigsaw without the picture on the box.”
And that’s not all folks
As well as looking after number crunching for our researchers, Ruby also looks after our Data Migration Facility (DMF).
Our DMF uses Hierarchical Storage Management (HSM) to make a whole range of storage devices of different speeds, sizes and costs act like one big drive, to be able to be used in the most efficient manner possible.
We’ve been operating our DMF on various machines since 1991, which makes it the longest continually operating DMF deployment in the world. Since it was set up it’s expanded by five orders of magnitude to now store more than 14 petabytes of information. To put that in perspective, that’s enough to store every episode of Game of Thrones (in HD) 325,581 times!
With about 1200 registered researchers using Ruby to access and crunch their data, our researchers will continue to push the limits of possibility.
23rd September 2017 at 12:50 am
Perhaps a mobile app could be made for the public’s contribution like Vodafone has done with Dreamlab for cancer? Especially for MND / ALS. But I’d put my PC to work to contribute also if given both options.
23rd September 2017 at 12:48 am
Damn, wouldn’t it be more efficient to let the public use their computer power also? MND is the worst disease possibly known to man and one if not the most not understood and hard to diagnose.
11th April 2016 at 4:08 pm
Are these computers based at the NCI at the ANU?
13th April 2016 at 12:15 pm
CSIRO is a major collaborator in the NCI along with several other government research agencies.
However, Ruby, along with our other HPC Bragg and Pearcey, is housed in a purpose built datacentre POD at a secure facility managed by our partner Canberra Data Centres.
CSIRO Social Media
6th April 2016 at 3:29 pm
Is the Computer using Main Frame or some other type of Operating System? Are you hooked up to the Volunteer Computer System, Like BOINC Australia., so we can all be involved?
8th April 2016 at 3:57 pm
Ruby is running a standard variant of Linux, and user’s jobs are submitted and managed through a batch system.
Ruby is not being used for volunteer work like BOINC, although similar to BOINC, CSIRO harnesses the spare computing power of our own fleet of thousands of desktop PCs internally for research related work.