Thursday, February 12, 2009

How to configure nfsv4 with kerberos in RHEL?

Below is a small howto which explain how to set this up in a simple environment. Below setup involves three systems.

1 - A kerberos server.
2 - An NFS server.
3 - An NFS client.

All the above three systems are running RHEL-5.3. I am explaining what all need to be done on each server and showing an example to prove that 16 group limitation not affected with nfsv4 while using kerberos authentication. I am not including LDAP/NIS in this setup since that is optional.

Pre-requisites

- There should be a dns server in your network and an FQDN for the above three systems. All FQDNs should resolve forward and revers from all systems appropriately. This is a requirement for kerberos to work.

- Time on all the above three machines would be synced to the same NTP server and should have the same time on all the three machines. This is a requirement for kerberos to work properly.

How to setup a Kerberos server?

Below is what all I did to setup the keberos server. If you already have a kerberos server in your environment like windows AD or Solaris, you can use that system and no need to setup a new one. Please refer respective docs for more details on how to configure those kerberos servers. I am setting up RHEL5 as the kerberos using MIT kerberos.

1.1 - Install krb5-server.

# yum install krb5-server

1.2 - Choose a realm of your choice. It's generally recommended to use the dns domain name in UPPER case as the kerberos realm. My DNS domain name is pnq.redhat.com and I have selected PNQ.REDHAT.COM as the kerberos realm.

1.3 - Run "authconfig-tui". In the "Authentication" section, select "Use Kerberos" -> Next -> Type in the realm name, IP of the KDC and admin server. IP of the KDC and admin server is the ip of the same machine. Click "ok".

1.4 - Edit /var/kerberos/krb5kdc/kdc.conf and change the realm name in [realms] section to match your realm name. Eg,

[realms]
PNQ.REDHAT.COM = {
...............
...............

1.5 - Create the database using the kdb5_util utility from a shell prompt:

# /usr/kerberos/sbin/kdb5_util create -s

Please type a password on the prompt and re-enter it on the verification prompt.

1.6 - Edit the /var/kerberos/krb5kdc/kadm5.acl file and give */admin all privileges on the database. Eg,

*/admin@PNQ.REDHAT.COM *

1.7 - Run "kadmin.local" and add a principle named "root/admin" using the "addprinc" command. Set a password that you wish.

# kadmin.local
Authenticating as principal root/admin@PNQ.REDHAT.COM with password.
kadmin.local: addprinc root/admin
WARNING: no policy specified for root/admin@PNQ.REDHAT.COM; defaulting to no policy
Enter password for principal "root/admin@PNQ.REDHAT.COM":
Re-enter password for principal "root/admin@PNQ.REDHAT.COM":
Principal "root/admin@PNQ.REDHAT.COM" created.

1.8 - Start Kerberos using the following commands:

/sbin/service krb5kdc start
/sbin/service kadmin start
/sbin/service krb524 start

1.9 - Run the "kadmin" command and enter the root/admin password that you entered in 1.7 on the prompt and add a new user named "nfsuser" with a password.

# kadmin
Authenticating as principal root/admin@PNQ.REDHAT.COM with password.
Password for root/admin@PNQ.REDHAT.COM:
kadmin: addprinc nfsuser
WARNING: no policy specified for nfsuser@PNQ.REDHAT.COM; defaulting to no policy
Enter password for principal "nfsuser@PNQ.REDHAT.COM":
Re-enter password for principal "nfsuser@PNQ.REDHAT.COM":
Principal "nfsuser@PNQ.REDHAT.COM" created.

Nothing more to do on the kerberos server at this time. We would do the rest from nfsserver and client after logging into the kerberos server via “kadmin” from there whenever appropriate.

How to configure the NFS server?

2.1 - Run "authconfig-tui". In the "Authentication" section, select "Use Kerberos" -> Next -> Type in the realm name, IP of the KDC and admin server that we setup earlier. Click "ok".

2.2 - Add a user named "nfsuser" with uid 2000 without setting up any password for that user.

# useradd -u 2000 nfsuser

2.3 - Create 20 groups named "group1 - group20" with gid 3001-3020 on the system.

# for i in `seq 1 9`; do groupadd -g 300$i group$i;done

# for i in `seq 10 20`; do groupadd -g 30$i group$i;done

2.4 - Add "nfsuser" member of all these 20 groups.

# usermod -G group1,group2,group3,group4,group5,group6,group7,group8,group9,group10,group11,group12,group13,group14,group15,group16,group17,group18,group19,group20 nfsuser

2.5 - Create a directory to share using nfs.

# mkdir /nfs

Create 20 directories and each are writable by one group.

# for i in `seq 1 20`; do mkdir /nfs/group$i ;chgrp group$i /nfs/group$i; chmod g+w /nfs/group$i; done

2.6 - Edit /etc/sysconfig/nfs and uncomment the below line.

SECURE_NFS="yes"

2.7 - Edit /etc/exports and share /nfs as below.

/nfs gss/krb5p(rw,sync,fsid=0)

2.8 - Run "kadmin" and create a service principle for nfs server and extract it to a keytab file in the nfsserver. The format of the service principle is "nfs/FQDN of nfs server". Use ktadd to extract it to a keytab file.

# kadmin
Authenticating as principal root/admin@PNQ.REDHAT.COM with password.
Password for root/admin@PNQ.REDHAT.COM:
kadmin: addprinc -randkey nfs/dhcp7-135.pnq.redhat.com
WARNING: no policy specified for nfs/dhcp7-135.pnq.redhat.com@PNQ.REDHAT.COM; defaulting to no policy
Principal "nfs/dhcp7-135.pnq.redhat.com@PNQ.REDHAT.COM" created.
kadmin: ktadd -e des-cbc-crc:normal nfs/dhcp7-135.pnq.redhat.com
Entry for principal nfs/dhcp7-135.pnq.redhat.com with kvno 7, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab.
kadmin: quit

Note: "-e des-cbc-crc:normal or -e des-cbc-crc:md5" to the above ktadd command is required only if you are willing to use RHEL4 or RHEL5.2 and below as the clients. If clients are only RHEL5.3 and other Unix systems (like Solaris), that option can be skipped.

2.9 - Start nfs server service and rpcidmapd.

# service nfs start
# service rpcidmapd restart

How to configure the NFS client?

3.1 - Repeat step 2.1

3.2 - Repeat step 2.2

3.3 - Repeat step 2.3

3.4 - Repeat step 2.4

3.5 - Edit /etc/sysconfig/nfs and uncomment the below line.

SECURE_NFS="yes"

3.6 - Run "kadmin" and create a service principle for nfs client and extract it to a keytab file in the nfsserver. The format of the service principle is "nfs/FQDN of nfs client". Use ktadd to extract it to a keytab file.

# kadmin
Authenticating as principal root/admin@PNQ.REDHAT.COM with password.
Password for root/admin@PNQ.REDHAT.COM:
kadmin: addprinc -randkey nfs/dhcp7-143.pnq.redhat.com
WARNING: no policy specified for nfs/dhcp7-143.pnq.redhat.com@PNQ.REDHAT.COM; defaulting to no policy
Principal "nfs/dhcp7-143.pnq.redhat.com@PNQ.REDHAT.COM" created.
kadmin: ktadd nfs/dhcp7-143.pnq.redhat.com
Entry for principal nfs/dhcp7-143.pnq.redhat.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal nfs/dhcp7-143.pnq.redhat.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab.
kadmin: quit

3.7 - Start "rpcgssd" and "rpcidmapd".

# service rpcgssd start
# service rpcidmapd restart

3.8 - Mount the share as below.

# mount -t nfs4 fqdn of the server:/ -o sec=krb5p

Eg,

# mount -t nfs4 dhcp7-135.pnq.redhat.com:/ /mnt -o sec=krb5p

This would successfully mount the share on the client.

3.9 - Switch to user "nfsuser".

# su - nfsuser

- Run "df -h /mnt" and try to cd /mnt. This would fail because "nfsuser" requires a ticket to access the mount point. The "df" command wouldn't show the size of the filesystem, but it would show a "-" at this point.

# df -h /mnt

dhcp7-135.pnq.redhat.com:/
- - - - /mnt

- Get a ticket for "nfsuser".

# kinit

Enter the kerberos password for "nfsuser."

# df -h /mnt

dhcp7-135.pnq.redhat.com:/
7.2G 2.0G 4.9G 29% /mnt

- "nfsuser" is member of "group1 - group20". Direcotry /mnt/group1 is writable by "group1" (first group) whereas /mnt/group20 is writable by "group20" (20th group). "nfsuser" would be able to write to both these directories on the client. Verify it by writing something to that directory as "nfsuser".

Note: If you are seeing any problems, please re-confirm that you have met the pre-requisites successfully without any problems.

It's highly recommended to use LDAP/NIS to store the user and group information and configure both nfs server and client to do name lookup on the LDAP/NIS server rather than configuring them on earch client's local files. How to configure LDAP/NIS is outside the scope of this write up.

What are the advantages of NFV4 over NFCv3?

1 – NFSv4 with kerberos supports authentication. A big disadvantage of nfsv3 was that root user can “su – ”, get the remote user's home directory automounted and delete/modify his files. This is a big security risk in bigger enterprises if they have 1000s of systems. In the above example, this problem is solved. If root on a system do “su - ” and can get his home directory automouted, he can't delete or modify the files without getting a ticket from the kerberos server. To get a ticket, he must pass 's password.

2 – NFSv4 with kerberos supports encryption. While using krb5p, every communication between client and server is sent over the wire after it was encrypted which was not supported by NFSv3.

3 – 16 group limitation is raised. If you don't know more about this, please refer Eisler's nfs blog here

In the above example, I have proved that there is no such limitation in nfsv4.

There are more......................

8 comments:

Unknown said...

This tutorial is a huge help. Thanks for writing it out in one page. It helped me a lot.

Weiyi said...

Thanks for this tutorial. How do you overcome the kerberos ticket expiration problem? Without kerberos ticket auto-renew, a daemon can not be left running with NFS v4 access...

Sadique Puthen said...

Weiyi,

Not sure how much useful the below is going to be,

- Extract the a user's password to keytab. I am taking kerberos4 as an example username.

# kadmin
Authenticating as principal root/admin@PNQ.REDHAT.COM with password.
Password for root/admin@PNQ.REDHAT.COM:
kadmin: ktadd kerberos4
Entry for principal kerberos4 with kvno 2, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal kerberos4 with kvno 2, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal kerberos4 with kvno 2, encryption type DES with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal kerberos4 with kvno 2, encryption type DES cbc mode with RSA-MD5 added to keytab WRFILE:/etc/krb5.keytab.
kadmin:

# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
2 kerberos4@PNQ.REDHAT.COM
2 kerberos4@PNQ.REDHAT.COM
2 kerberos4@PNQ.REDHAT.COM
2 kerberos4@PNQ.REDHAT.COM

- Then export the nfs service principle and the above ticket to a file.

# kinit -k -c /tmp/krb5cc_uid -S nfs/fqdn@REALM kerberos4@REALM

Eg /tmp/service in my system is as below,

[root@dhcp7-136 ~]# klist /tmp/krb5cc_uid
Ticket cache: FILE:/tmp/krb5cc_uid
Default principal: kerberos4@PNQ.REDHAT.COM

Valid starting Expires Service principal
04/18/09 15:21:50 04/19/09 15:21:50 nfs/dhcp6-224.pnq.redhat.com@PNQ.REDHAT.COM
renew until 04/18/09 15:21:50

- Pass the below in an environment variable to the *deamon* before starting that, may be as below. Or start the service with the privileges of that username.

export KRB5CCNAME=/tmp/krb5cc_uid

- Setup a cron job which should run 5 minutes befrore the ticket expires. ie, Once in a day to run the above command. ie,

# kinit -k -c /tmp/krb5cc_uid -S nfs/fqdn@REALM kerberos4@REALM

I dont' kjnow how well this would work for kerberos, but I am using a configuration like this in /etc/init.d/nscd so that it would authenticate via nss_ldap agianst AD using the tickets. Good Luck!

Unknown said...

Thanks for the tutorial, but I do have a simple question and I've yet to find a good solution to it.

In a secure setup with NFS4 and kerberos user root can't 'su -' to a user and thus cannot access it's home directory. This is all good and well but what would the solution be to export a filesystem with applications (say for example Sun Grid Engine or Platform LSF) that need to be run as root? In a default setup, root is squashed away if kerberos security is used in the mount mount options.

How can you selectively permit the (system) root account of a single (or several) system(s) to a NFS4 export with krb5 security enabled? I've read much of the CITI documentation but I've yet to find a good howto / faq on it that has actually been implemented. (I believe Fedora Core with the latest nfs-utils should be able to use umich_ldap in idmapd.conf but documentation and examples are still lacking)

Any ideas?

Regards,

Jeroen

Sadique Puthen said...

I am shooting in the dark since I don't know how Sun Grid Engine or Platform LSF works and what are their requirements!

For root to access an nfs export secured via kerberos is a valid credentials' cache. If the root can do "kinit user" and get a ticket using use password, he can access the nfs share.

If the application can understand a variable from where to read the kerberos cache, you can create the cache and pass the patch to the cache as the value of the variable. See my last comment!

PRAJITH said...

Hi,

I have bee trying to configure a nfs server with ldap and kerbros, I am almost completed it. But while mounting the filesystem in a kerborized nfs client the users and groups always nobody and nogroup, so I can't write anything in the nfs filesystem. I am sure the idmapd is working fine. Could you please help on this.

Sadique Puthen said...

Can both the nfsserver and client see the user with the same uid/gid (getent passwd user) and the files are owned by the uid/guid on the nfs server?

Unknown said...

Nice blog. Thanks for sharing the information.