Lab Exercise: Condor DAGMan

Note: Replace the username and jobID keywords in the following command prompts with the proper ones.

Part 1: Standard DAG Submission and Execution with DAGMan

1. All of the work we will be during during this lab exercise will be done on the login node of the Purdue Condor cluster.

1.      b. In TeraGrid, you need to specify the Project Number to be able to submit jobs to the Condor. To do that, create a file called .tg_default_project in your home directory, and enter the keyword TG-CCR090020 in this file.

2. Create a directory “DAGMan” in your Home directory, and switch to that directory

3. Copy example files in Condor distribution to your directory

4. Explore each file in your directory.

This is the .dag file, which basically represents a simple diamond-shaped DAG. Each job has associated Condor submit file with it: A.submit,…

$ cat B.submit Universe   = standard Executable = half.condor input      = A.out output     = B.out log        = diamond.log Queue

$ cat C.submit Universe   = standard Executable = half.condor input      = A.out output     = C.out log        = diamond.log Queue

$ cat D.submit Universe   = standard Executable = sum.condor arguments  = B.out C.out output     = D.out error      = D.err log        = diamond.log Queue

            The following are the 4 Condor submit files:

            $ cat A.submit

Universe   = standard

Executable = random.condor

output     = A.out

log        = diamond.log

Queue

Notice that all the jobs use the “standard” universe to get the fault-tolerance support of Condor. Also, notice the data dependencies between jobs. For example, output generated by A.submit (A.out) is being used as input parameter in B.submit and C.submit.

random.condor, half.condor, and sum.condor executables used in these submit files are 3 simple C programs. random.condor generates a random number, half.condor divides an input number by 2, and sum.condor calculates the summation of two input numbers.

Finally, the Makefile looks like this:

Basically, this Makefile generates the executables from the C source files. Notice that compilation is done via the condor_compile linkage to benefit from Condor libraries.

5. Issue the “make” command to compile the C programs.

This command will produce the random.condor, half.condor, and sum.condor executables.

6. Submit the “.dag” file to Condor

7.      Now, watch the Condor queue.

You should be seeing the DAGMan run itself as a job. Also, DAGMan will release the execution of jobs in the DAG based on the dependencies among them.

8. When the DAG finishes executing, explore the files generated by Condor. Especially:

 diamond.dag.condor.sub is automatically generated by the Condor, and serves as a Condor submit file itself. Remember that Condor DAGMan itself runs as a Condor job itself. Content of the file will look something like this:

Notice that it uses the “scheduler” universe, and the executable refers to the location for DAGMan.

Part 2: Fault-Tolerance in DAGMan with Rescue DAGs

9.      To observe the fault-tolerance behavior of DAGMan and demonstrate the usage of Rescue DAGs, we first need to mess with one of the job submit files (C.submit in this case). Before that, if you want to have a clean execution environment, remove the data products generated in the first part (*.out, *.log, *.err, and diamond.dag.* files).

So, modify the C.submit file as follows:

Notice that, we have changed the Executable name in the submit file. In this case, normally, Condor will not be able to find h.condor executable, and the job C.submit should fail.

10.  Submit the DAG again:

11.  Now, watch the Condor queue:

If everything goes as expected, first random.condor (job A) should execute successfully. Then, DAGMan will try to submit half.condor (job B) and h.condor (job C) to the Condor queue at the same time. Job B should execute successfully, whereas job C will not be able to execute, since the executable h.condor is unknown to the Condor.

12.  At this point, DAGMan will not be able to proceed any further, so it will create a Rescue DAG file, and exit. Content of the Rescue file would look something like this:

As you see, DAGMan marked the job A and job B as DONE, so the next time this rescue file is submitted for execution these jobs will not need to be executed again. As long as the data products generated by job A and job B are available for the usage of latter jobs, there is no need to re-run job A and job B.

13.  As you can also see from the Rescue file, the failure in the DAG has occurred due to an error in job C. So, at this point, we will fix the C.submit file to its original version, and submit the Rescue DAG to DAGMan.

So, first, fix the C.submit:

14.  Now, submit the Rescue DAG:

15.  Now, watch the Condor queue:

You should see that DAGMan first submits job C (half.condor), and after it finishes execution DAGMan will submit job D (sum.condor) to the Condor queue. After job D finishes execution, the whole DAG finishes execution and DAGMan exits.

Now, you can check the output files and log files and explore the results…