GCT 6.0 GRAM5: User’s Guide

Condor is a high-throughput job scheduler from the University of Wisconsin. It provides a facility called Condor-G to run jobs via GRAM. See the Condor documentation, especially the section on Grid Universe, which describes how to write Condor Classified Ads to run jobs using GRAM services. The gt5 Grid type provides the best performance for using GRAM with Condor-G.

The Swift system from the University of Chicago is a data-oriented coarse grained scripting language that supports dataset typing and mapping, dataset iteration, conditional branching, and procedural composition. The SwiftScript language can be used to create workflows that are executed on various services, including GRAM. See the Swift User’s Guide for information about using Swift.

The GridWay Metascheduler enables large-scale, reliable and efficient sharing of computing resources: clusters, supercomputers, stand-alone servers. It supports different LRM systems (PBS, SGE, LSF, Condor) within a single organization or scattered across several administrative domains. The GridWay manual describes how to use GridWay.

In addition to the tools above, you can write your own GRAM clients, using the public APIs described in the GRAM5 Developer’s Guide. The client APIs there can be used to write custom applications that interact with GRAM services in C/C++ or Java.

XSEDE provides a number of domain-specific science gateways and portals, which provide interfaces to various computation and data resources, including some managed by GRAM.

These include CGD’s Atmospheric Modeling & Predictability Section from NCAR, the UltraScan LIMS Portal at the Bioinformatics Core Facility at the University of Texas Health Science Center at San Antonio, the Social Informatics Data (SID) Grid at the University of Chicago, and Southern California Earthquake Center headquarted at the University of Southern California.

GRAM uses the Grid Security Infrastructure for its security implementation, based on X.509 certificates and the TLSv1 protocol to authenticate user identities with GRAM services. Before using GRAM, you must first obtain a security credential. This is typically done by requesting a certificate from a site-specific CA, or by using a portal to obtain a temporary credential. In typical use, GRAM uses a proxy certificate which is a short-term credential digitally signed by a private key. Please read the Basic procedure for using GSI C to learn more about how to obtain and use a GSI credential before continuing this guide.

Before interacting with a GRAM service, you must know its contact address. GRAM uses a very flexible URL-like syntax to contain information about the service’s hostname, TCP port number, service name, and security identity. In the basic case, you will only need to use the service’s hostname to contact the service. However, if the service is configured to run on a non-standard port, or with a custom service name, or credential which doesn’t match its hostname, you will need to use one of the longer forms.

A fully-qualified resource name looks something like grid.example.org:2120/jobmanager-sge:/C=US/O=Example/OU=Grid/CN=host/grid.example.org. Breaking this down, the resource name includes:

Any component of the resource name may be omitted, except for the host name, and defaults will be used. The field separator : must be retained when skipping between name components.

The tools globus-job-run and globus-job-submit an both be used to submit jobs to GRAM resources. The difference is that globus-job-run will wait until the job terminates before exiting and prints job standard output and stderr after the job completes, while globus-job-submit will submit the job and then exit immediately, printing the job contact to its standard output stream. The job can be then polled for status with the globus-job-status command, its output can be fetched with the globus-job-get-output and cleaned up with the globus-job-clean command.

The globus-job-run provides a simple blocking command-line interface to the GRAM service. The globus-job-run program submits a job to a GRAM resource and waits for the job to terminate. After the job terminates, the output and error streams of the job are sent to the output and error streams of globus-job-run. Note that truly interactive jobs are not supported with GRAM.

The globus-job-run program has command-line options to control most aspects of jobs run by GRAM. However, certain behaviors must be specified by definition of an RSL string containing various job attributes. A more detailed description about the RSL language is included on the section on running jobs with globusrun below.

The following examples show some of the common command-line options to globus-job-run. Full globus-job-run documentation is available in the GRAM5 public interface guide.

A related tool to globus-job-run is globus-job-submit. This command submits a job to a GRAM5 service then exits without waiting for the job to terminate. Other tools (globus-job-cancel, globus-job-clean, and globus-job-get-output) allow futher interaction with the job.

The globus-job-submit program has most of the same command-line options as globus-job-run. When run, instead of displaying the output and error streams of the job, it prints the job contact, which is used with the other globus-job tools to interact with the job.

This example shows how to check that an RSL document contains a syntactically correct job description. Note that this mode does not do semantic validation of the RSL, so an RSL document that passes this test may not work when submitted to a GRAM5 service.

This example shows how to check that a globus-gatekeeper is running at a particular contact and that the client and service have mutually-trusted credentials.

This example shows how to determine what software version of GRAM5 is deployed at a particular service contact.

This example shows how to submit interactive job with globusrun. When the -s is used, the output of the job command is returned to the client and displayed as if the command ran locally. This is similar to the behavior of the globus-job-run program described above.

This example shows how to submit, monitor, and cancel a batch job using globusrun. This method is useful for the case where the job may run for a long time, the job may be queued for a long time, or when there are network reliability issues between the client and service.

The following example shows how to refresh the credential used by a job manager and a job.

When the Job Manager’s credential is about to expire, it sends a message to all clients registered for GLOBUS_GRAM_PROTOCOL_JOB_STATE_FAILED notifications that the job manager is terminating and that the job will continue to run without the job manager.

Any client which receives such a message can (if necessary) generate a new proxy as described above and then submit a restart request to start a job manager with a new credential. This job manager will resume monitoring the jobs which were started prior to proxy expiration.

In this example, the globusrun displays an error message when the job manager’s proxy is about to expire. The user creates a new proxy and resumes monitoring the job with globusrun.

In addition to the standard output and error stream output done by globusrun, GRAM5 can do basic file management tasks to stage files to the GRAM5 service node before submitting a job and to stage files from the GRAM5 service node to a file service after the job completes.

GRAM5 file staging supports four URL schemes: ftp, gsiftp, http, and https. Note, that for the https scheme, GRAM expects the file server to be running with the same identity as the client.

General file staging is controlled by three RSL attributes: file_stage_in, file_stage_in_shared, and file_stage_out. In addition, the files named by the RSL attributes executable, stdin may be staged in and the files named by the RSL attributes stdout and stderr may be staged out.

The file_stage_in_shared RSL attribute instructs GRAM to store a local copy of the resource named by the URL in the GASS cache. This is useful if multiple concurrent jobs will be accessing one or more common files. The GASS cache will manage a reference count for files in the cache and remove them when all jobs that refer to them complete.

The following example shows how to stage a few files from a GridFTP server to the GRAM node. It uses the rsl_substitution mechanism to define a subsitution variable to reduce the amount of redundancy in the job description.

The next example uses the file_stage_in_shared RSL attribute to stage a file into the cache. The file is transferred from the client using the GASS https server embedded in the globusrun program when the -s option is used.

The final staging example uses the file_stage_out RSL attribute to transfer a file from the GRAM service to an FTP server using anonymous FTP

GRAM5 supports creating a per-job scratch directory which can be used as a place to store files that will be automatically removed by GRAM when the job completes. It also supports an explicit list of files to remove when the job completes.

This example shows how to stage files into a scratch directory. It again uses the embedded GASS https server, stages to the GRAM service, then runs /bin/ls in the temporary directory. After the job completes, the contents of $(SCRATCH_DIRECTORY) and the directory itself are removed.

This example shows how to explicitly remove a file that was created by the job.

The globusrun command supports a two-phase commit protocol to ensure that the client knows the contact of the job which has been created so that it can be monitored or canceled in the case of a client or service error. The two-phase commit affects both job submission and termination.

The two-phase protocol is enabled by using the two_phase RSL attribute, as in the next example. When this is enabled, job submission will fail with the error GLOBUS_GRAM_PROTOCOL_ERROR_WAITING_FOR_COMMIT. The client must respond to this signal with either the GLOBUS_GRAM_PROTOCOL_JOB_SIGNAL_COMMIT_REQUEST or GLOBUS_GRAM_PROTOCOL_JOB_SIGNAL_COMMIT_EXTEND signals to either commit the job to execution or delay the commit timeout. One of these signals must be sent prior to the two phase commit timeout, or the job will be discarded by the GRAM service.

A two phase protocol is also used at job termination if the save_state RSL attribute is used along with the two_phase attribute. When the job manager sends a callback with the job state set to GLOBUS_GRAM_PROTOCOL_JOB_STATE_DONE or GLOBUS_GRAM_PROTOCOL_JOB_STATE_DONE it will wait to clean up the job until the two phase commit occurs. The client must reply with the GLOBUS_GRAM_PROTOCOL_JOB_SIGNAL_COMMIT_END signal to cause the job to be cleaned. Otherwise, the job will be unloaded from memory until a client restarts the job and sends the signal.

If a job manager or client exits before a job has completed, the job will continue to run. The client can reconnect to a job manager and receive job state notifications and output using the restart RSL attribute.

To submit a job that runs a java program, the client must ensure that the job can find the Java interpreter and its classes. This example sets the default PATH and CLASSPATH environment variables and uses the shell to locate the path to the java program.

GRAM requires a client certificate and private key in order authenticate with the GRAM service. If these are not available, the GRAM client will fail. In typical use, a user will create a temporary proxy certificate either derived from their identity certificate issued by some certificate authority, or from a service such as myproxy. If a GRAM client command returns any error containing the string GSS Major Status you’ve hit a credential problem. Look at the Troubleshooting Section of the GSI manual for details about how to diagnose and correct these errors. The tool with the -p command-line option is especially helpful for diagnosing some of these types of problems.

There are a few things which can go wrong when trying to contact a GRAM service. These have slightly different error types which can help diagnose which problem is occurring.

One problem GRAM users sometimes encounter is that it looks like jobs submitted to GRAM are not making any progress, even though the local resource manager thinks they’ve run. There are a couple of reasons why this might occur: GRAM is not getting the information it needs from the local resource manager or the GRAM client is not getting the information it needs. We’ll cover diagnosing and handling the latter case in this document, as the other is an system administrator issue.

The way globus-job-run and globusrun determine that jobs have completed is via GRAM job state callbacks. These are messages sent by the GRAM service to the client node indicating that something significant has happened in the lifecycle of the job. If for some reason the GRAM service can not get those messages to the client, the client will not be able to detect job state changes.

In order to determine if this is the case, submit a job using globus-job-submit, and then use the globus-job-status command to see if the job state changes. If it does not, then consult the GRAM administrator---there might be some problem with the installation. If it does, then for some reason the callbacks are not happening. This might be firewall issues or host naming issues.

The GRAM client sends a "callback contact" to the GRAM service when it submits a job, in order that it can receive notifications. This contact is a reference to a https server embedded in the GRAM client which only handles GRAM state callbacks. As with all web servers, it has a URL which defines how to contact it, which in this case consists of the client host name and the service port number. If the host name that is used is not resolvable (such as a for a laptop with a dynamic address), then the GRAM service will not be able to contact it. If that’s the case, you can set the GLOBUS_HOSTNAME environment variable to the IP address that your client can be reached at, and then submit your jobs. This will cause GRAM to publish that address instead of what it thinks the client’s host name is.

Another way that the GRAM service would be unable to send job state updates to a client would be if there’s a firewall between the service and the client. If that’s the case, you might need to set the GLOBUS_TCP_PORT_RANGE environment variable to a comma-separated list of numbers which represent a range of minimum and maximum TCP port numbers to listen on. You might have to contact your site administrator to determine what TCP ports are allowed. If there are none, you can still use globus-job-submit and globus-job-status to track your job’s state changes, or use another tool like those mentioned in the section about client tools.

The GRAM service has a log file which contains information about the job as it is processed. These logs are located by default in /var/log/globus/gram_$USERNAME.log. There are some different logging levels available, as . There are some different logging levels available, as described in the GRAM Adminstrator’s Guide. These can be controlled on a per-job basis by adding the loglevel RSL attribute to your job description. The default is to log only FATAL and ERROR messages, but other levels can sometimes help understand what is going on.

Sometimes, bugs creep into the LRM adapter scripts. When that occurs, the GRAM job will usually fail with an error like this:

If this occurs, you may have to work with a GRAM administrator to help debug this problem. One helpful thing you can do when reporting it is to save the GRAM internal script data so that it can be used outside of the GRAM service to see what the low-level error looks like. To do this, add the RSL fragment (savejobdescription = yes) to your job request. This will cause GRAM to leave a file called something like $HOME/gram_[0-9]*.pl in your home directory. You can use this with the internal tool in your home directory. You can use this with the internal tool /usr/share/globus/globus-job-manager-script.pl to try to submit the job to the LRM without using the GRAM service. The command line to try to submit the job to the LRM without using the GRAM service. The command line /usr/share/globus/globus-job-manager-script.pl -m will attempt to submit the job to the LRM. It will show all the information the LRM script sends to the GRAM service, which might include some perl-language error or badly formatted output from the script (which must only output lines which begin with GRAM_SCRIPT_.

In some extreme cases, the savejobdescription option will not generate a file. If that’s the case, pass /dev/null as the argument to the as the argument to the -f command-line option. The problem is likely a perl syntax error which will be reached before the job description is loaded.

If all else fails, please send information about your problem to gram-user@globus.org. You’ll have to subscribe to a list before you can send an e-mail to it. See here for general e-mail lists and information on how to subscribe to a list and here for GRAM specific lists. Depending on the problem, you may be requested to file a bug report to the globus project’s Issue Tracker.

globusrun - Execute and manage jobs via GRAM

globusrun [-help ] [-usage ] [-version ] [-versions ]

The globusrun program for submits and manages jobs run on a local or remote job host. The jobs are controlled by the globus-job-manager program which interfaces with a local resource manager that schedules and executes the job.

The globusrun program can be run in a number of different modes chosen by command-line options.

When -help, -usage, -version, or -versions command-line options are used, globusrun will print out diagnostic information and then exit.

When the -p or -parse command-line option is present, globusrun will verify the syntax of the RSL specification and then terminate. If the syntax is valid, globusrun will print out the string "RSL Parsed Successfully…​" and exit with a zero exit code; otherwise, it will print an error message and terminate with a non-zero exit code.

When the -a or -authenticate-only command-line option is present, globusrun will verify that the service named by RESOURCE_CONTACT exists and the client’s credentials are granted permission to access that service. If authentication is successful, globusrun will display the string "GRAM Authentication test successful" and exit with a zero exit code; otherwise it will print an explanation of the problem and will with a non-zero exit code.

When the -j or -jobmanager-version command-line option is present, globusrun will attempt to determine the software version that the service named by RESOURCE_CONTACT is running. If successful, it will display both the Toolkit version and the Job Manager package version and exit with a zero exit code; otherwise, it will print an explanation of the problem and exit with a non-zero exit code.

When the -k or -kill command-line option is present, globusrun will attempt to terminate the job named by JOB_ID. If successful, globusrun will exit with zero; otherwise it will display an explanation of the problem and exit with a non-zero exit code.

When the -y or -refresh-proxy command-line option is present, globusrun will attempt to delegate a new X.509 proxy to the job manager which is managing the job named by JOB_ID. If successful, globusrun will exit with zero; otherwise it will display an explanation of the problem and exit with a non-zero exit code. This behavior can be modified by the -full-proxy or -D command-line options to enable full proxy delegation. The default is limited proxy delegation.

When the -status command-line option is present, globusrun will attempt to determine the current state of the job. If successful, the state will be printed to standard output and globusrun will exit with a zero exit code; otherwise, a description of the error will be displayed and it will exit with a non-zero exit code.

Otherwise, globusrun will submit the job to a GRAM service. By default, globusrun waits until the job has terminated or failed before exiting, displaying information about job state changes and at exit time, the job exit code if it is provided by the GRAM service.

The globusrun program can also function as a GASS file server to allow the globus-job-manager program to stage files to and from the machine on which globusrun is executed to the GRAM service node. This behavior is controlled by the -s, -o, and -w command-line options.

Jobs submitted by globusrun can be monitored interactively or detached. To have globusrun detach from the GRAM service after submitting the job, use the -b or -F command-line options.

The full set of options to globusrun consist of:

-f RSL_FILENAME, -file RSL_FILENAME:: Read job specification from the file named by RSL_FILENAME.

If the following variables affect the execution of globusrun

The globusrun program assumes any failure to contact the job means the job has terminated. In fact, this may be due to the globus-job-manager program exiting after all jobs it is managing have reached the DONE or FAILED states. In order to reliably detect job termination, the two_phase RSL attribute should be used.

globus-job-submit(1), globus-job-run(1), globus-job-clean(1), globus-job-get-output(1), globus-job-cancel(1)

globus-job-cancel - Cancel a GRAM batch job

globus-job-cancel -f | -force -q | -quiet JOBID

The globus-job-cancel program cancels the job named by JOBID. Any cached files associated with the job will remain until globus-job-clean is executed for the job.

By default, globus-job-cancel prompts the user prior to canceling the job. This behavior can be overridden by specifying the -f or -force command-line options.

The full set of options to globus-job-cancel are:

If the following variables affect the execution of globus-job-cancel.

globus-job-clean - Cancel and clean up a GRAM batch job

globus-job-clean -r RESOURCE | -resource RESOURCE

-f | -force -q | -quiet JOBID

The globus-job-clean program cancels the job named by JOBID if it is still running, and then removes any cached files on the GRAM service node related to that job. In order to do the file clean up, it submits a job which removes the cache files. By default this cleanup job is submitted to the default GRAM resource running on the same host as the job. This behavior can be controlled by specifying a resource manager contact string as the parameter to the -r or -resource option.

By default, globus-job-clean prompts the user prior to canceling the job. This behavior can be overridden by specifying the -f or -force command-line options.

The full set of options to globus-job-clean are:

If the following variables affect the execution of globus-job-clean.

globus-job-get-output - Retrieve the output and error streams from a GRAM job

globus-job-get-output -r RESOURCE | -resource RESOURCE

-out | -err -t LINES | -tail LINES -follow LINES | -f LINES JOBID

The globus-job-get-output program retrieves the output and error streams of the job named by JOBID. By default, globus-job-get-output will retrieve all output and error data from the job and display them to its own output and error streams. Other behavior can be controlled by using command-line options. The data retrieval is implemented by submitting another job which simply displays the contents of the first job’s output and error streams. By default this retrieval job is submitted to the default GRAM resource running on the same host as the job. This behavior can be controlled by specifying a particular resource manager contact string as the RESOURCE parameter to the -r or -resource option.

The full set of options to globus-job-get-output are:

If the following variables affect the execution of globus-job-get-output.

globus-job-run - Execute a job using GRAM

globus-job-run [-dumprsl ] [-dryrun ] [-verify ]

[-file ARGUMENT_FILE]

SERVICE_CONTACT

-np PROCESSES | -count PROCESSES

-m MAX_TIME | -maxtime MAX_TIME

-p PROJECT | -project PROJECT

-q QUEUE | -queue QUEUE

-d DIRECTORY | -directory DIRECTORY [-env NAME'VALUE']

[-stdin -l | -s STDIN_FILE] [-stdout -l | -s STDOUT_FILE] [-stderr -l | -s STDERR_FILE]

[-x RSL_CLAUSE]

-l | -s EXECUTABLE [ARGUMENT …​]

The globus-job-run program constructs a job description from its command-line options and then submits the job to the GRAM service running at SERVICE_CONTACT. The executable and arguments to the executable are provided on the command-line after all other options. Note that the -dumprsl, -dryrun, -verify, and -file command-line options must occur before the first non-option argument, the SERVICE_CONTACT.

The globus-job-run provides similar functionality to globusrun in that it allows interactive start-up of GRAM jobs. However, unlike globusrun, it uses command-line parameters to define the job instead of RSL expressions.

The full set of options to globus-job-run are:

If the following variables affect the execution of globus-job-run.

globusrun(1), globus-job-submit(1), globus-job-clean(1), globus-job-get-output(1), globus-job-cancel(1)

globus-job-status - Check the status of a GRAM5 job

globus-job-status JOBID

The globus-job-status program checks the status of a GRAM job by sending a status request to the job manager contact for that job specifed by the JOBID parameter. If successful, it will print the job status to standard output. The states supported by globus-job-status are:

The full set of options to globus-job-status are:

If the following variables affect the execution of globus-job-status.

The globus-job-status program can not distinguish between the case of the job manager terminating for any reason and the job being in the DONE state.

globusrun(1)

globus-job-submit - Submit a batch job using GRAM

globus-job-submit [-dumprsl ] [-dryrun ] [-verify ]

[-file ARGUMENT_FILE]

SERVICE_CONTACT

-np PROCESSES | -count PROCESSES

-m MAX_TIME | -maxtime MAX_TIME

-p PROJECT | -project PROJECT

-q QUEUE | -queue QUEUE

-d DIRECTORY | -directory DIRECTORY [-env NAME'VALUE']

[-stdin -l | -s STDIN_FILE] [-stdout -l | -s STDOUT_FILE] [-stderr -l | -s STDERR_FILE]

[-x RSL_CLAUSE]

-l | -s EXECUTABLE [ARGUMENT …​]

The globus-job-submit program constructs a job description from its command-line options and then submits the job to the GRAM service running at SERVICE_CONTACT. The executable and arguments to the executable are provided on the command-line after all other options. Note that the -dumprsl, -dryrun, -verify, and -file command-line options must occur before the first non-option argument, the SERVICE_CONTACT.

The globus-job-submit provides similar functionality to globusrun in that it allows batch submission of GRAM jobs. However, unlike globusrun, it uses command-line parameters to define the job instead of RSL expressions.

To retrieve the output and error streams of the job, use the program globus-job-get-output. To reclaim resources used by the job by deleting cached files and job state, use the program globus-job-clean. To cancel a batch job submitted by globus-job-submit, use the program globus-job-cancel.

The full set of options to globus-job-submit are:

If the following variables affect the execution of globus-job-submit.

globusrun(1), globus-job-run(1), globus-job-clean(1), globus-job-get-output(1), globus-job-cancel(1)