GCT 6.0 GridFTP : System Administrator’s Guide

GridFTP is built and installed as part of a default GCT 6.0 installation. For basic installation instructions, see Installing GCT 6.0. No extra installation steps are required for this component.

If you wish to install GridFTP without installing the rest of the Grid Community Toolkit, follow the instructions in the above link but just use the group install target globus-gridftp with Linux native packages and make target gridftp (make gridftp) with source installer.

[only a static GridFTP server]

If you wish to build and install a statically linked set of GridFTP binaries from the source installer, follow the instructions on GCT installation from the source installer in the link above but use the following for configure and make:

In GCT 6.0 the GridFTP server and the client globus-url-copy are non-threaded by default. To enable threading, set the environment variable GLOBUS_THREAD_MODEL=pthread. On the server, threading can be enabled using command-line option -threads 1 as well.

[overview]

The configuration interface for GridFTP is the admin tool, , which can be used with a configuration file and/or run-time options.

The configuration file for the GridFTP server is read from the following locations, in the given order. Only the first file found will be loaded:

Options are one per line, with the format:

If the value contains spaces, they should be enclosed in double-quotes ("). Flags or boolean options should only have a value of 0 or 1. Blank lines and lines beginning with # are ignored.

For example:

For complete command documentation including all options, see .

This page includes information about general configuration of the GridFTP server. Security options are discussed here, and more advanced configuration is described here.

The following describes a typical GridFTP configuration of the front end (control channel) and back end (data channels). For other alternatives that provide greater levels of security, see Advanced Configuration.

By default, the data channel and control channel are separate socket connections within the same process. The client sends a command and waits to finish before issuing the next command. This is good for a single host, traditional-type user. If you have a single host and you want an ultra-reliable and light weight file transfer service, this is a good choice. This configuration is also good for testing purposes.

If the GridFTP server is behind a firewall:

If the GridFTP client is behind a firewall:

Additional information on Grid Community Toolkit Firewall Requirements is available here.

There are many security options in GridFTP ranging from no security to higher security via GSI .

The following is a quick guide to running the server and using the client:

Look through the list of options for globus-gridftp-server:

Start the server in anonymous mode (discussed more fully here):

where:

Run a two party transfer with client:

Run 3rd party transfer:

Experiment with -dbg, and -vb options for debugging and checking the performance of your setup:

where:

Ctrl-c - Kill the server.

To enable file sharing using Globus Sharing, you have to add the Globus Sharing CA certificates to your trusted certificates directory (/etc/grid-security/certificates) and use -sharing-dn option in the server as follows:

and use -sharing-rp option to restrict the file paths allowed for sharing:

A very important option for a system administrator to set is the number of simultaneous GridFTP transfers allowed. In other words, the number of clients that are allowed to connect to the server at the same time.

GridFTP is designed to be a high performance, on-demand data transfer service. Quite a bit of system resources (mainly memory) are allocated to each client connection and this is with the assumption that the session will consume even more system resources (CPU, net/disk bandwidth) when performing a high speed data transfer. For this reason, the system administrator must evaluate the resource their host machine has to offer and set a reasonable limit to the number of client connects allowed at one time.

When determining the instance concurrency level, there are two major factors to consider: system memory helps determine the upper limit of the instance range and available I/O bandwidth helps determine the lower limit.

The globus-gridftp-server sits on top of various file systems. Each file system has its own ideal access patterns and I/O buffer sizes. To provide the user with some means of control, we offer the option:

This number indicates the size of the read requests posted to the disk.

We recommend that you update the kernel TCP buffer settings and TCP congestion control algorithm as recommended here for achieving maximum performance.

On systems that have multiple network interfaces, the system admin likely wants to associate data transfers with the fastest possible NIC available. This can be done in the GridFTP server by using the option:

UDT is bundled with GCT starting with Globus v4.2, so downloading UDT separately is no longer needed.

The GridFTP server can be separated into front end and data node processes. This is the architecture used to achieve a striped server, but it can also be exploited to achieve a higher level of security.

Running the server as root is often desirable because it allows the server to fork and setuid on a child process related to an authenticated user. This allows the server to leverage the operating system’s file system permissions and other security devices. However, it is not at all desirable to have a root-running process listening on a port open to the world. If an attacker were to compromise the process, they could obtain root-level access to the machine.

To overcome this security risk, the GridFTP server can be run in a front end/back end manner. The front end can be run as any user, say user globus, that has very limited access to the machine. The front end is the process open to the outside world. If it is compromised, an attacker has only gained access to that limited account. The back end is run as root, but configured to only allow connections from the front end.

This does, however, require that a copy of the host cert and host key be owned by the non-privileged user. If you use this configuration, the non-privileged user should not have write permission to executables, configuration files, etc. This provides greater security and also allows for proxying and load balancing. Many backend data movers can be behind a single point of client contact. Each client is assigned a different backend in a round robin fashion.

To start the front end, run:

To start the back end, run:

The GridFTP server supports separate front end (client control connection) and back end (data node) processes. In addition, a single front end process may connect to multiple back end data nodes.

When multiple back end data nodes are available, the server is said to be in a striped configuration, or simply, is a striped server. In this mode, transfers are divided over all available data nodes, thus allowing the combined bandwidth of all data nodes to be used.

This is recommended for improved performance of large (1GB+) file transfers. This can also be useful if you want to use full data encryption and need to tether together many hosts to handle the processing load.

The ability to use inetd or daemon execution modes applies to both front end servers and data nodes, and the same certificate and user requirements apply.

To start the front end, run:

To start the data-node, run:

The -p <port> option used on the data-node is the port that will be used for IPC connections. This is the port that you will register with the front end server.

For example:

The client would only connect to the front end at machineA:5000, for example, using globus-url-copy with the -stripe option:

Where machineX may be another striped server or a standard GridFTP server.

Furthermore striped servers and split process can be combined. You can have an 8 node cluster that only uses 2 nodes at a time in a striped server configuration and load balances across the rest of the nodes. TODO: any other details here?

GridFTP server can be configured to restrict access to specific path using the command-line option -chroot-path <path> This path must contain a valid certificate structure, /etc/passwd, and /etc/groups. A helper script globus-gridftp-server-setup-chroot can help create a suitable directory structure.

GFork is a service like inetd that listens on a TCP port and runs a configurable executable in a child process whenever a connection is made. GFork also creates bi-directional pipes between the child processes and the master service. These pipes are used for interprocess communication between the child process executables and a master process plugin. More information on GFork can be found here.

For information on enabling bottleneck detection via Netlogger, see the Gridftp-netlogger page on the CEDPS website.

To enable multicasting, you must whitelist the gridftp_multicast driver with the -fs-whitelist file,gridftp_multicast option:

The above command whitelists both the file driver and the gridftp_multicast driver. [NOTE]

For information about using multicasting, click here.

GridFTP Where There Is FTP (GWTFTP) is an intermediate program that acts as a proxy between existing FTP clients and GridFTP servers. Users can connect to GWFTP with their favorite standard FTP client, and GWFTP will then connect to a GridFTP server on the client’s behalf. To clients, GWFTP looks much like an FTP proxy server. When wishing to contact a GridFTP server, FTP clients instead contact GWTFTP.

Clients tell GWFTP their ultimate destination via the FTP USER command. Instead of entering their username, client users send the following:

This command tells GWTFTP the GridFTP endpoint with which the client wants to communicate. For example:

As of Globus 4.2.0, GridFTP server provides system administration logs in 2 different formats. The CEDPS best practices compliant format is a new format provided by GridFTP server available in Globus 6.0 and GCT. For more details on the CEDPS Logging format, see http://cedps.net/index.php/LoggingBestPractices.

Verify that you can establish a control channel connection and that the server has started successfully by telnetting to the port on which the server is running:

If you see anything other than a 220 banner such as the one above, the server has not started correctly.

Verify that there are no configuration files being unexpectedly loaded from /etc/grid-security/gridftp.conf or $GLOBUS_LOCATION/etc/gridftp.conf. If those files exist, and you did not intend for them to be used, rename them to .save, or specify -c none on the command line and try again.

If you can log into the machine where the server is, try running the server from the command line with only the -s option:

The server will print the port it is listening on:

Now try and telnet to that port. If you still do not get the banner listed above, something is preventing the socket connection. Check firewalls, tcp-wrapper, etc.

If you now get a correct banner, add -p 2811 (you will have to disable (x)inetd on port 2811 if you are using them or you will get port already in use):

Now telnet to port 2811. If this does not work, something is blocking port 2811. Check firewalls, tcp-wrapper, etc.

If this works correctly then re-enable your normal server, but remove all options but -i, -s, or -S.

Now telnet to port 2811. If this does not work, something is wrong with your service configuration. Check /etc/services and (x)inetd config, have (x)inetd restarted, etc.

If this works, begin adding options back one at a time, verifying that you can telnet to the server after each option is added. Continue this till you find the problem or get all the options you want.

At this point, you can establish a control connection. Now try running globus-url-copy.

Once you’ve verified that you can establish a control connection, try to make a transfer using globus-url-copy.

If you are doing a client/server transfer (one of your URLs has file: in it) then try: in it) then try:

This will run until you control-c the transfer. If that works, reverse the direction:

Again, this will run until you control-c the transfer.

If you are doing a third party transfer, run this command:

Again, this will run until you control-c the transfer.

If the above transfers work, try your transfer again. If it fails, you likely have some sort of file permissions problem, typo in a file name, etc.

If the server has started correctly, and your problem is with a security failure or gridmap lookup failure, verify that you have security configured properly here.

If the server is running and your client successfully authenticates but has a problem at some other time during the session, please ask for help on gt-user@globus.org. When you send mail or submit bugs, please always include as much of the following information as possible:

If you run GridFTP servers via Xinetd and notice high latency for connections and/or transfers, check if /etc/xinetd.conf or the gsiftp service configuration inside or the gsiftp service configuration inside /etc/xinetd.d is set to log USERID as follows: is set to log USERID as follows:

Such a configuration tells Xinetd to log the remote user using the method defined in RFC 1413, which causes an ident client to attempt to query the machine that the connection is coming from before the service will run. Even when this succeeds, the response can’t be trusted, and more often than not it is rejected or simply dropped (which results in the longest delays) by the remote firewall.

Latency can be reduced by making sure Xinetd does not log the USERID.

The following GridFTP-specific usage statistics are sent in a UDP packet at the end of each transfer, in addition to the standard header information described in the Usage Stats section.

We have made a concerted effort to collect only data that is not too intrusive or private and yet still provides us with information that will help improve and gauge the usage of the GridFTP server. Nevertheless, if you wish to disable this feature for GridFTP only, use the -disable-usage-stats option of globus-gridftp-server. Note that you can disable transmission of usage statistics globally for all C components by setting "GLOBUS_USAGE_OPTOUT=1" in your environment.

Also, please see our policy statement on the collection of usage statistics.

The Storage Resource Broker Data Storage Interface (SRB-DSI) is an extension to the GridFTP server that allows it to interact with SRB. Plugging this extension into a GridFTP server allows the GridFTP server to access a SRB resource and serve it to any GridFTP client as though it were a filesystem.

The above image shows the architecture of the system. There are 4 major components:

You need the following items to use the SRB-DSI:

Instructions for building GCT and SRB are well documented in the above links. The following sections describe one way of building these two packages. However, if any questions or errors are discovered, the reader should look to the above links for solutions.

Before you can run the GridFTP server with the SRB-DSI, you need to set up some files.

Once you have the configuration files in place, you can run the server.

For more information on setting these values and running the GridFTP server see Configuring GridFTP.

Most users can run with:

See the README file for more information.

GFork is a user-configurable super-server daemon very similar to xinetd in that it listens on a TCP port. When clients connect to a port, it runs an administrator-defined program which services that client connection, just as xinetd does.

An unfortunate drawback to xinetd is that there is no way to maintain or share long-term information. Every time a client connects, a new process is created; and every time that client disconnects, the process is destroyed. All of the information regarding the specific interactions with a given client is lost with these transient processes. A further disadvantage is that there is no way for these service instances to share service-specific information with each other while they are running.

There are times when it is useful for a service to maintain long-term service-specific state, or for a service to share state across client connections. GFork is designed to address this situation. GFork runs a long term master program (that is user-defined) and forms communication links via UNIX pipes between this process and all client connection child processes. This allows long-term state to be maintained in memory and allows for communication between all nodes.

Associated with a GFork instance is a master process. When GFork starts, it runs a user-defined master program and opens up bi-directional pipes to it. The master program runs for the lifetime of the GFork daemon. The master is free to do whatever it wants; it is a user-defined program. Some master programs listen on alternative TCP connections to have state remotely injected. Others monitor system resources, such as memory, in order to best share resources. As clients connect to the TCP listener, child processes are forked which then service the client connection. Bi-directional pipes are opened up to the child processes as well. These pipes allow for communication between the master program and all child processes. The master program and the child programs have their own protocol for information exchange over these links. GFork is just a framework for safely and quickly creating these links.

The creation of GFork was motivated by the Globus GridFTP server. GridFTP can be run as a striped server where there is a frontend and several backends. The backends run in tandem to transfer files faster by tying together many NICs. The frontend is the contact point for the client where transfer requests are made. When the frontend is run out of inetd, the list of possible backends must be statically configured. Unfortunately, backends tend to come and go. Sometimes backends fail, and sometimes backends are added to a pool. We needed a way to have a [fixme good synopsis: dynamic pool of backends for use in live transfers]. To accomplish this we created GFork.

A major difference between GFork configuration and xinetd is that GFork only runs one service per instance, where xinetd runs many services per instance all associated with many different ports. GFork takes a single configuration file and handles a single service. If there is demand, GFork will be enhanced to handle many services in the way that xinetd does.

Running the globus-gridftp-server under GFork is almost identical to running it under xinetd. First, you need a configuration file:

That portion is identical to xinetd. In fact, an existing xinetd configuration file should work.

When running GridFTP out of GFork, the server should be run with a master program. The master program provides enhanced functionality such as dynamic backend registration for striped servers, managed system memory pools and internal data monitoring for both striped and non-striped servers.

To run with a master program, the following two lines are needed in the config file.

These last two options relate to the master program and work in the same way that server and server_args do. The first line tells GFork what master program to use (for the GridFTP server, we use gfs-gfork-master). The second line provides options to the master program.

The full list of master options are as follows (this is to date only, run the program with --help for newer options):

The following is an example GFork configuration file:

Once you have a configuration file, run GFork with:

As mentioned in Configuring GridFTP for Cluster-to-Cluster (or Striped) data movement, GridFTP offers a powerful enhancement called striped servers. In this mode a GridFTP server is set up with a single frontend and one or more backends. All of the backends work in concert to transfer a single file and thereby achieve high throughput rates. Here we describe how to configure one frontend and multiple backends for use as a striped server with GFork.

Another feature of the GridFTP GFork plugin is memory usage limiting. Under extreme client loads, it is possible that GridFTP servers require more memory than the system has available. Due to a common kernel memory allocation scheme known as optimistic provisioning, this situation can lead to a full consumption of memory resources and thus trigger the out of memory handler. The OOM handler will kill processes in a difficult-to-predict way in order to free up memory. This will leave the system in an unpredicatable and unstable state; obviously, this is a situation that we want to avoid.

To control this situation, the GridFTP GFork plugin has a memory limiting option. This will attempt to limit memory usage to a given value or to the maximum amount of RAM in the system. Most of the memory is given to the first few connections, but when the plugin detects that it is overloaded, each session is limited to half the available memory.

To enable this feature, one of two options must be passed to the master program via the master_args in the config file:

Another important option should be provided in the GFork config file: instance. When a client connects to GFork, a GridFTP server instance is executed. This instance requires a certain amount of RAM. If connections are coming in too fast, this can act as a DOS attack. Limiting the number of allowed simultaneous connections will help the memory management algorithm do its job. This limit is set with:

We recommend a value of 100 or RAM/2M, whichever is smaller.

The following is an example of a GFork configuration file with memory limiting enabled: