Grid Virtual File System (GVFS) ============================== GVFS version 0.3.1 by Renato J. Figueiredo, Ming Zhao ACIS Lab / Electrical and Computer Engineering University of Florida Contents =========== * Introduction * Requirements * Changes * Installation * Setup * Logging * Support Introduction ============== Grid Virtual File System (GVFS), originally named PVFS, is a virtualized distributed file system for providing high-performance data access in grid environments and seamless integration with unmodified applications. It leverages existing NFS (Network File System) support in operating systems, and uses user-level proxies to authenticate and forward RPC (Remote Procedure Call) requests between the native NFS client and server, and map user identities between different domains. Requirements =============== GVFS is written in C, based on the user-level NFSD code by Mark A. Shand etc. To use it, the following libraries are requied: NSL, Mathematics. Changes ==================== * Changes from v0.2 Added support for write-back caching; Added caching support for NFS V3; Added invalidation-polling based cache consistency. * Changes from v0.2.1: Added support for session key based authentication. * Changes from v0.1: Added support for client-side read-only disk caching (for NFSv2 based GVFS only). A simple weak cache consistency model is used: the GVFS client periodically flushes its cached attributes, so any requests to cached data afterwards will be revalidated from the server. Installation ============== * Configure GVFS: Use "./configure" Use "./configure --help" to see the available switches and options. * Build GVFS: Use "make" to build the binaries, including: gvfs.mountd: The GVFS proxy mount daemon; gvfs.nfsd: The GVFS proxy nfs daemon. * Install GVFS: Use "make install" to deploy the binaries to the location specified during the configuration. * Rebuild GVFS : Use "make clean" to remove the generated binaries. Use "make rpcclean" to remove the stubs generated by rpcgen (optional). Build GVFS again. * Uninstall GVFS: Use "make uninstall" to remove the installed binaries. Setup ======== * Install GVFS: Deploy the GVFS proxy daemons to the desired location on the file system server (and file system client). The examples below assumes the path to the binaries is /gvfs. * Start GVFS: 1. On the file system server, change the configuration of the NFS service to export the desired partitions to localhost. For example, on a Linux NFS server, add the following line to /etc/exports file to make "/netcare" available to the GVFS: /netcare localhost.localdomain(rw) Note: If you plan to run proxies without root/setuid privileges, they will communicate with the kernel server via user-level ports. The NFS kernel server does not accept requests from user-level ports by default; the "insecure" flag needs to be added to the exports entry in this case, as shown below. In this scenario, it is advisable that login from regular user accounts to the file server be disabled. /netcare localhost.localdomain(rw,insecure) 2. On the file system server, prepare the configuration files used by the GVFS proxy daemons, including: a. exports_mountd: It specifies the exported partitions for the proxy mount daemon. For example, add the following line to make "/netcare/user1" available to the file system client "clnt.acis.ufl.edu": /netcare/user1 clnt.acis.ufl.edu b. exports_nfsd: It specifies the exported partitions with the desired read/write permissions and user identity mappings for the proxy nfs daemon. For example, add the following line to make "/netcare/user1" available to the file system client "clnt.acis.ufl.edu" with both read and write permissions, and the static user identity mapping specified in the configuration file "/gvfs/map": /netcare/user1 clnt.acis.ufl.edu(rw,map_static=/gvfs/map) Note: 1. Similarly to NFS exports file, a GVFS exports file can have more than one entires, where each entry represents a file system partition that is exported through GVFS. 2. In GVFS exports files, both name and IP can be used to specify a client which a file system partition is exported to. IP/netmask pairs can also be used to export a partition to a group of clients, e.g.: /netcare/user1 10.5.144.0/255.255.255.0 c. map: It specifies the identity (uid/gid) mappings between the file user on the file system client and the file owner on the file system server. For example, add the following lines to map between the user "shadow1" (uid: 1000, gid: 1000) on the client and the user "user1" on the server (uid: 100, gid: 100): uid 1000 100 gid 1000 100 The map file can store multiple entires to reflect the mappings between more than one pair of identities. 3. Start the proxy mount daemon on the file system server by "/gvfs/gvfs.mountd". Use "/gvfs/gvfs.mountd -h" to see the available switches and options. For example, the following command starts the daemon with exports specified in "/gvfs/exports_mountd", mount version 3, port number 10002, and an UDP connection to the kernel mount daemon at port 33333 on localhost: /gvfs/gvfs.mountd -f /gvfs/exports_mountd -V 3 -P 10002 -O 33333 -t udp -H localhost 4. Start the proxy nfs daemon on the file system server by "/gvfs/gvfs.nfsd". Use "/gvfs/gvfs.nfsd -h" to see the available switches and options. For example, the following command starts the daemon with exports specified in "/gvfs/exports_nfsd", NFS version 3, port number 10001, and a UDP connection to the kernel nfs service at port 2049 on localhost: /gvfs/gvfs.nfsd -f /gvfs/exports_nfsd -V 3 -t udp -P 10001 -O 2049 -H localhost Steps 5 to 7 are optional. You can skip them and go to step 8 directly. 5. On the file system client, prepare the configuration files used by the client-side GVFS proxy daemons, including: a. exports_mountd: Exports the desired partitions to localhost. For example: /netcare/user1 localhost.localdomain b. exports_nfsd: Exports the desired partitions to localhost and specifies the read/write permissions and user identity mappings. For example: /netcare/user1 localhost.localdomain(rw,map_static=/gvfs/map) c. map: Because the user identity mapping is already set up on the server- side proxy, here the mapping is to self. For example: uid 1000 1000 gid 1000 1000 6. Start the proxy mount daemon on the file system client by "/gvfs/gvfs.mountd". For example, the following command starts the daemon with a UDP connection to the proxy mount daemon spawned in step 3 on the file system server "vfs.acis.ufl.edu": /gvfs/gvfs.mountd -f gvfs/exports_mountd -V 3 -t udp -P 10002 -O 10002 -H vfs.acis.ufl.edu 7. Start the proxy nfs daemon on the file system client by "/gvfs/gvfs.nfsd". For example, the following command starts the daemon with a UDP connection to the proxy nfs daemon spawned in step 4 on the file system server "vfs.acis.ufl.edu", and client-side attributes and data caches stored in "/cache/session1": /gvfs/gvfs.nfsd -f /gvfs/exports_nfsd -V 3 -t udp -P 10001 -O 10001 -H vfs.acis.ufl.edu -A -D -c /cache -i session1 Note: 1. With this setup, the client-side proxy do not cache data or attributes modifications, but it cannot guarantee the consistency with the server if the data are shared with other clients. The proxy can be configured to synchronize with the server periodically (see setup 10) or be forced to do the synchronization with a signal (SIGALRM). 2. Write-back caching can also be enabled by starting the client-side proxy with an additional -W flag. But this feature is still experimental, and may cause data loss if the client fails. 8. On the file system client, mount the GVFS through the proxy(s). For example, the following command establish the GVFS through the proxy daemons started in step 3 and 4, and mount "/netcare/user1" from the server to "/shadow1" on the client: mount -t nfs -o port=10001,mountport=10002,udp,rsize=8192,wsize=8192, vers=2,mountvers=2,soft vfs.acis.ufl.edu:/netcare/user1 /shadow1 If step 5 to 7 are also taken, then use the following command to establish the GVFS: mount -t nfs -o port=10001,mountport=10002,udp,rsize=8192,wsize=8192, \ vers=2,mountvers=2,soft localhost:/netcare/user1 /shadow 9. Enable session key based inter-proxy authentication: A session key is an arbitrary 128bit string used by the client-side proxy to authenticate itself to the server-side proxy. The session key based authentication can be enabled as follows: a. Generate a 128bit session key at the client side; b. Put the key in the server-side proxy's key list; c. Start the server-side proxy (both the nfsd and mountd) with -m option to have session key authentication enabled: d. Start the client-side proxy (both the nfsd and mountd) )with -s option to have session key authentication enabled; e. Mount the GVFS as usual. Note: Both the client's session key and the server's key list should be in binary mode. The current implementation does not allow the session key to contain an all-zero byte. (See included sk.tgz for examples.) For example, assume the server-side proxy's key list is stored in file /gvfs/sk_all, and the client-side proxy's key is stored in file /gvfs/sk1. Then, use the following commands to start the proxies: Server-side: /gvfs/gvfs.nfsd -f /gvfs/exports_nfsd -V 3 -t udp -P 10001 -O 2049 -H localhost -m /gvfs/sk_all Client-side: /gvfs/gvfs.nfsd -f /gvfs/exports_nfsd -V 3 -t udp -P 10001 -O 10001 -H vfs.acis.ufl.edu -A -D -c /cache -i 1000 -s /gvfs/sk1 10. Enable invalidation-polling based cache consistency: This allows for a relaxed consistency among data-sharing clients, and is more efficient than native NFS' consistency mechanism. Its basic idea is to let the server-side proxy keep track of modifications made by clients, and let the client-side proxies poll cache invalidations periodically. To enable this consistency protocol, you need to set up the GVFS sessions between the clients and the server as described in the above steps. Let all the client-side proxies connect to the same server-side proxy, where each client-side proxy uses a unique session key to identify itself. The server- side proxy should be started with "-I server" flag, and a client-side proxy should be started with "-I client" flag. In addition, the -T flag can be used to set the polling period for a client-side proxy. For example, use the following commands to start the proxies: Server: /gvfs/gvfs.nfsd -f /gvfs/exports_nfsd -V 3 -t udp -P 10001 -O 2049 -H localhost -m /gvfs/sk_all -I server Client1: /gvfs/gvfs.nfsd -f /gvfs/exports_nfsd -V 3 -t udp -P 10001 -O 10001 -H vfs.acis.ufl.edu -A -D -c /cache -i session1 -s /gvfs/sk1 -T 10 Client2: /gvfs/gvfs.nfsd -f /gvfs/exports_nfsd -V 3 -t udp -P 10001 -O 10001 -H vfs.acis.ufl.edu -A -D -c /cache -i session2 -s /gvfs/sk2 -T 20 After this setup, the two client-side proxies will poll the cache invalidations from the server-side proxy every 10 and 20 seconds. * Destroy GVFS: Use "umount" to unmount the GVFS partition and then kill the proxy daemons. If caching is enabled, the cached data will be automatically flushed before the client-side proxy exits. If write-back is used, it will also submit delayed writes to the server. So please wait for the write-back to finish before you kill the server-side proxy. Logging ========== If syslog is available on the system, logging generated by proxy daemons will be sent to system logs. Otherwise, it will end up in files in /var/tmp. The proxy damons can also be started on foregroud using the "-F" option, and then more debugging information can be generated with the "-d" option. Support ========== Web site: http://www.acis.ufl.edu/~ming/gvfs Email: ming@acis.ufl.edu