Open Source Spotlight - Yabi: Bringing drag-and-drop to supercomputers

27.04.2012

"Imagine a workflow, for example, of 10 different tools that a user might pull together to conduct analysis on some data. The first tool is selecting a file from a computer that is connected to a remote sensor somewhere in outback Australia, then the next tool is running a file format process that may end up being computationally intensive. The third item is a tool that is on a supercomputer local to the researcher.

"The next tool is remotely accessing another supercomputer, then the fourth or fifth tool is actually utilising the cloud resources and the results are stored in the cloud somewhere. There is no lock in so you get to pick and choose where you run the tools or the administrator does that for the users. You do not have to log in to Yabi in order to get access to your data, so you can choose where the results are stored."

Part of the philosophy driving Yabi is not just simplifying workflow management for end users, but for system administrators too. "You can imagine one scientist comes and says 'please install 10 tools for me into this Yabi environment' but then if 50 scientists come and they all have different tools the last thing you want is the bottle neck to be at the system administration level. We can abstract away the complexities for the user but we also want to abstract away the complexities for the sysadmin."

Yabi was open sourced in July last year. "We were mindful not to open source it too early, because there is a level of support that is required for the international community and we were making sure that the system was in a state that would be able to supported by our group," Professor Bellgard says. It has already been deployed at QFAB the Queensland Facility for Advanced Bioinformatics; and the Murdoch team is "in conversation" with scientists at UTS in Sydney as well as at other universities and CSIRO.