I’ve been slowly upgrading our Xserves to Mac OS X 10.5 (Leopard), and up until this week it’s been a snap. This week, however, I upgraded a server that had two things go wrong afterwards. It took a long time to figure out what needed to be done, so I thought I’d prime the search engine pump in case anyone else ran into the same issues.

Remember that some of these may be some quirk of the configuration of our server.

BSD configuration file

We use the BSD flat file option in the Directory Utility to handle authentication for this server. In Tiger and earlier, the BSD flat file option looked in /etc/master.passwd for user information. After upgrading to Leopard, no one in the master.passwd file could get in; according to dscacheutil the users didn’t exist. It was as if the server wasn’t paying any attention to the password file.

The Apple web site’s 10.5 BSD Configuration Files list still contains /etc/master.passwd as the file for storing “User names, passwords, IDs, primary group IDs, and so forth” but a chance mention on a forum that someone was using /etc/passwd led me to move /etc/master.passwd to /etc/passwd; the users immediately appeared in dscacheutil and were able to log in.

January 23: sometime in the past month it has started paying attention to /etc/master.passwd as well as /etc/passwd. If an account was in both files, both passwords worked.

setuid root script wrappers

I use a setuid wrapper program to run a Python script as root on our server. Mac OS X doesn’t let scripts run as setuid root (a good thing). The recommended solution for scripts that need to run as setuid root is to wrap them in a C program: mark the C program as setuid root, and then have the C program call the script.

After the upgrade, this script would not run as root. I modified it and the wrapper to report who they were each running as, and neither the wrapper nor the script were running as root. This was a little harder to figure out because there are two changes to the way setuid applications work under Leopard, and I was hitting both of them.

The first clue came when by chance I tried running the application on the local volume rather than over the NFS-mounted volume. The wrapper suddenly began reporting that it was running as root. Further tests showed that, after the upgrade, setuid root files on the NFS-mounted volume would not run as root. Most likely this means that the set-uid option is turned off by default for NFS mounts under Leopard, which is a good thing. Rather than disable that option, I moved the wrapper script to local storage.

That fixed the wrapper, but the Python script was still unable to os.chroot or os.setuid, and it continued to report that it was not running as root. The second change is that setuid wrappers no longer automatically pass their setuid status to scripts run from the system() function. The wrapper now needs to call the setuid() function—using “setuid(0)”—before using the “system()” call to run the desired script.