Parallelizing Insurance Processing Application using Condor
Published Date:
Apr 29, 2011Abstract
This white paper brings out an approach to increase the utilization of available hardware to process long running Insurance Applications using work load management software – Condor. This results in reducing processing time almost linearly, depending on the number of nodes/ cores in the cluster.
Insurance processing typically includes long running applications. Systematic parallel execution of such programs in a cluster using work-load management software – Condor can provide significant speed up. This can be used to gain business benefits in production / development environment. Similar technique can be employed for huge code builds and automated testing.
Excerpts from the Paper
Insurance Applications typically consists of several batch programs executed daily, weekly, monthly and quarterly. The batch programs – java applications, execute for several hours depending upon the data volumes. Thus, there is a need to reduce the execution time, more so, on daily batches.
Solution approach is to identify the parallel execution paths within the batch program and schedule them to run on the different nodes/ cores in parallel. The implementation involves setting up the cluster, installation of Condor software and application specific dependencies including the database access. Condor is a work load management system, which provides job queuing, scheduling, resource monitoring and management. User has to transform the parallel execution paths to Condor jobs and submit as job files to Condor. Condor schedules the jobs in parallel, taking care of inter-job dependencies.
