The original purpose of GridPilot was to make it easy for researchers to run, preserve, rerun, share computations on the emerging national and international grid infrastructures. The word infrastructure is used in plural here – indicating our failure to create the grid. Thus GridPilot was born with a pluggable backend support.
Eventually, I created my own take at a grid infrastructure product – GridFactory – and of course implemented a GridPilot plugin to support it.
In previous posts, a whole range of sample applications have been described – controlled with GridPilot and running on a variety of backends – NorduGrid, WLCG, GridFactory, EC2. In this post, I’ll summarize and point to a few highlights.
You import apps directly from GridPilot (“File” ? “Import application(s)”). By default you’ll then get a browser allowing you to choose an app from the central GridPilot app repository.
This repository of example applications is provided to help you get started easily with real computations. The idea is that you find an app that matches your particular use case a closely as possible, rename and edit it to suit your needs and then export it back to the repository for the benefit of others (to export you first have to switch to “advanced mode” in your preferences).
Featured use cases
CERN data processing
This example demonstrates the use of GridPilot in data processing in high energy physics (HEP). It makes extensive use of some HEP-specific technologies, that are incapsulated in GridPilot in the form of plugins: the ATLAS DB plugin and the NG and GLite computing system plugins.
More » See also here » and here »
This example is special in that it does not depend on any preinstalled software package (runtime environment), but includes a precompiled binary. This binary will of course only for certain run on the system it was compiled on. We compiled on Debian Sarge and Scientific Linux 5 and run on all back-ends: a local virtual machine, GridFactory without virtualization and with both Qemu and VirtualBox virtualization, EC2, NorduGrid and WLCG (gLite). More »
Given the popularity of the iPhone, an interesting use of a batch system is conversion of movie files from the AVI to the MPG4 format. All you have to do is import the application “avi_to_mp4″, click “Run” and then select one of the back-ends you’ve enabled. You’ll then be asked to populate the input dataset with AVI files. If you just keep the defaults, you’ll then be converting 5 movies in the public domain from AVI to MP4 format. You can of course also choose other movies of your own and/or edit the simple script that does the actual transcoding with ffmpeg. We’ll look at doing this conversion on 3 different back-end systems: virtual machine(s) on your own desktop, virtual machines on EC2 and a GridFactory cluster running on Amazon’s EC2 cloud. More »
To gauge the performance of both GridFactory and virtualization layers in a high-CPU/low-throughput setting, we chose the standard ray-tracing program POV-Ray and a standard benchmarking image, shipped with the program. More » See also here »