NOTE Note that this is for a stand alone dedicated fileserver. The new default fileserver, fossil, is described in [setting up fossil], in most cases it is recommended that you use fossil. INSTALLING A CURRENT PLAN 9 4TH EDITION FILE SERVER Installing a Plan 9 file server can be a bewildering experience for a new user. This document is intended to take some of the guesswork out of it. Also read [Upgrading to fourth edition] for some important fourth edition details. When installing a Plan 9 network, the file server should be one of the first machines set up, immediately after a standalone CPU/authentication server. The installation is fairly easy once one knows the steps to follow and understands what is happening. This information is cribbed from many other sources, including several manual pages, installation instructions from the 2nd edition (found online), the paper by Ken Thompson on the Plan 9 File Server, notes from the 9fans@cse.psu.edu archives, and reading the code in /sys/src/fs. Folks who are curious about the file server are encouraged to read any available reference material. In particular, read the manual pages fs(4), fs(8), and fsconfig(8). Also, the paper ``The Plan 9 File Server,'' by Ken Thompson (available in Volume 2 of the manual, or via ``page /sys/doc/fs/fs.ps'') is illuminating. Further, the ``Installing the Plan 9 Distribution'' document from the second edition can fill in a few of the holes, and can be found at [ftp://ftp.dbsystems.com/pub/plan9/install.html]. This wiki provides post-setup installation that can be handy. See the main [plan 9 wiki] page. Finally, the 9fans archives, available on the web at [http://lists.cse.psu.edu/archives/9fans/] have quite a bit of collected wisdom in them, and are well worth a read when you run into a snag. Broadly outlined, the steps for file server installation are as follows: * creating a plan9.ini(8) file describing the file server hardware * creating a boot floppy with a file server kernel * setting network parameters and initializing the filesystem on disk * setting authentication information in nvram or on the floppy * populating the server with data and user information * rebooting to come up in ``normal'' mode To begin, pick a machine on the network that is going to be the Plan 9 file server. This machine will run a specialized kernel, and will be dedicated to serving files. It cannot, for instance, double as a CPU or authentication server. In normal usage at Bell Labs, magnetic disk on the file server is only used as a cache for the contents of a WORM device. The smaller WORM jukeboxes are no longer expensive (see ebay.com for SCSI MMC jukeboxes such as the HP 80EX), and most people can afford one. There is an option to use all or part of a magnetic disk as a ``pseudo worm,'' which still provides one with the ability to use the file server's builtin backup capabilities (described in the file server paper by Ken Thompson). This document will only cover this configuration. ( When the fs documentation says "WORM", it generally applies to any MO cartridge system, be it using WORM or MO Rewritables -- kuroneko ) (I found 80EXs for $300 on ebay 9 Nov 2002 and also brand new 80EXs [here | http://www.codemicro.com/store/prod_results.cfm?mode=5&srchparm=C1100J#ABA] for $6,007.75 but none in the uk sadly ) Note that the file server supports a different set of devices than the ``normal'' kernel. SCSI or IDE disks can be used to hold the file system. There are significantly fewer drivers available for a file server. Just 23 .c files in the /sys/src/fs/pc source code directory should serve as a warning. Relatively few SCSI controllers are supported. The only ones still being manufactured are those from LSI Logic (containing SCSI chips with model numbers of 53C8xx or 53C1010). Finally, the restrictions on VGA controllers for the normal kernel are rather more lenient for the file server. This is because the file server doesn't attempt to take advantage of the display for anything other than a very basic (CGA mode) console. ( Its relatively important to note, the NCR/SYM/LSI driver only supports SCSI cards that implements the SCRIPTS interface - using older cards, such as the NCR 53c810 (non-A) and 825 will result in crashes/panics -- kuroneko ) Once a machine has been chosen, create a plan9.ini file that describes it. In particular, specify all entries, even those that would otherwise automatically be found (especially the SCSI and IDE controllers; mine wasn't automatically detected, though reading plan9.ini(8) led me to believe that it would be). Also, you might need to explicitly set the nvr= option in plan9.ini, as the syntax for specifying the NVRAM location is different between the file server and normal kernels. However, no ``monitor'' entry is required; if one is present, it's ignored. Typical entries might look like this, among others: ! ether0=type=i82557 ! scsi0=type=ncr53c8xx ! nvr=fd!0!plan9.nvr ! bootfile=fd0!dos!9pcfs (Note that the bootfile entry can use the newer style syntax <device>!<partition name>!<file> as opposed to the older <device>!<partition number>!<file>. This is because the bootfile entry is interpreted by 9load, whereas nvr is used by the file server kernel). Once you have a complete and accurate plan9.ini file, the next step is to build a file server kernel and put it, along with the plan9.ini file, onto a bootable floppy. NOTE: if you get a linker error that says "key_setup: etherga620reset: not defined", make sure to "replica/pull" for the latest sources or get the latest cd image. The mkfile has been fixed. An earlier version of the mkfile was missing the etherga620 dependency. To create a file server kernel for a PC, starting with a standalone CPU and authentication server, fetch the current Plan 9 distribution if you aren't sure that yours is current, and install it or upgrade to it. Then cd to /sys/src/fs/emelie, edit 9pcfs.c (notably FIXEDSIZE and localconfinit()) and dat.h to set RBUFSIZE (the block size), and run mk. (Alternately, make a copy of the emelie directory and work in that.) The result should be that a new kernel is built and placed into a file called ``9pcfs.'' See /sys/src/fs/README and Thompson's file server paper for more information. In a nutshell, 16KB blocks will be faster, but less space-efficient than 4KB blocks. Now insert a blank, formatted floppy into the CPU/authentication server and run ! pc/bootfloppy /dev/fd0disk /path/to/fs/plan9.ini assuming that you're using the first floppy disk in the system and that you've done a ``bind -a '#f' /dev'' at some point in the past. Once done, mount the floppy somewhere and copy 9pcfs to it. ( Actually, you can shortcut the whole mess of mounting the floppy to copy 9pcfs by using: ! pc/bootfloppy /dev/fd0disk /path/to/fs/plan9.ini /path/to/fs/9pcfs which will build the floppy, and copy over the kernel in one hit. -- kuroneko ) To give an example of the whole process, I start from the console of the standalone CPU/auth server: ! cd /usr/cross/sys ! ed plan9.ini ! ... ! cd /sys/src/fs/plan9pc ! mk ! (insert floppy disk into drive) ! bind -a '#f' /dev ! pc/bootfloppy /dev/fd0disk /usr/cross/sys/plan9.ini ! a: ! cp 9pcfs /n/a: ! unmount /n/a: The result is a bootable floppy disk with a Plan 9 file server kernel image on it, set as the default kernel to load at boot time. Now, eject the floppy and physically insert it into the floppy drive on the file server machine, and boot it. After a few minutes and a few error messages, the system should realize that the NVRAM checksum is incorrect (which is good because pc/bootfloppy initializes the NVRAM file on the floppy with a block's worth of zeros) and present you with a ``config: '' prompt. You must now configure the basic system parameters, including networking and the filesystem. This is largely covered in the fsconfig(8) manual page, and I won't go into many details about it. Basically, you define the service name by which the fileserver identifies itself using the ``service'' command, the IP configuration using the ``ip,'' ``ipmask,'' and ``ipgw'' commands, the IP address of the authentication server via the ``ipauth'' command, the place to store configuration information via the ``config'' command, filesystem parameters via the ``filsys'' command, the dump device via the ``dump'' command, the instruction to initialize the filesystems via the ``ream'' command, and that's it. You'll probably want to store your list of configuration commands somewhere, in case you ever need to repeat them, though the console command ``printconf'' can print all the permanent state (excluding commandds such ``ream''). Note that you *must* call at least one filsystem ``main.'' Note also that when in config mode, the commands you type don't actually do anything except record actions to be taken once you leave config mode. To get out and actually do what you've specified, type ``end''. If you make a mistake, you can safely reboot the machine and start over before typing ``end''. Finally, the ``config'' command steals a block from a device and stashes configuration information in it. However, if that block is at the very beginning of the device (and remember that a device can be a partition in this context), then it will be safe since the filesystem doesn't use the first block. As an example, here are the commands I typed to configure my file server, which uses an IDE disk as a ``pseudo-worm,'' and a SCSI disk as the cache: ! service lance ! ip 192.168.2.12 ! ipgw 192.168.2.1 ! ipmask 255.255.255.0 ! ipauth 192.168.2.11 ! config w0 ! filsys main cw0fh0 ! filsys dump o ! ream main ! end The file server will now initialize the disks and bring itself up. If the system dies, it's probably due to an incorrect specification of hardware either in plan9.ini or during configuration. In particular, the system will panic if it can't find the disks. Make sure that your configuration is correct and try again. If you have some extra mag disk, you may also want to create an ``other'' filsys on the file server, to store things without consuming WORM space. I use this for archives that I've copied from elsewhere on the web, and thus could recreate if necessary, but need to store locally, at least temporarily. Once up, set up the parameters that the system will use for authentication by typing the ``passwd'' command. This should ask for a password, authentication ID, and authentication domain. These should all be the same as what you used when setting up the standalone CPU/authentication server. Executing this command will also write the configuration information to NVRAM (which is usually a file on the floppy). At this point, the new fileserver will be in a special administrative mode that allows connected clients to bypass normal permission checking, and will also allow clients to modify the ownership of files (which flat out isn't possible when the system is running in ``normal'' mode). Before users can connect, however, the system will authenticate them via the authentication server, and must know what users exist in order to do so. We now have a chicken and egg problem; the system wants to authenticate users, but doesn't have an /adm/users for it to be able to know what users exist to try and authenticate. However, we can't yet mount the new file server to add an /adm/users. Luckily, we can type a command, ``users default'' to create a basic set of users sufficient for installing the system onto the new machine. Type this now: ! users default Now go back to the standalone machine CPU/auth server, mount the new file server, and load the distribution onto it. Note that when you installed the standalone CPU/auth server, you downloaded the distribution to your local disk, and it is in the directory /dist. One uses wrap/inst with the -ovr (set ownership, verbose, root to install to as next argument) to load the distribution onto the new file server. Note that if your CPU/auth server boots using the key of a user who does not exist by default (say ``tom'') you will need to create that user via the ``newuser'' command before loading the distribution: ! newuser tom (Even if the file server cannot yet write to /adm/users, newuser appears to create a new user in its internal table. I had to edit /adm/users later to bring things back into sync. [plus@cosym.net]) (What I Did - I've got a SCSI disk: I took my partition suggestion from fsconfig(8): ! config: service kremvax ! config: config p(w0)50.1 ! config: filsys main cp(w0)50.25f[p(w0)0.25p(w0)25.25p(w0)75.25] ! config: filsys dump o ! config: ream main config: ip 192.168.1.9 ! config: ipgw 192.168.1.1 ! config: ipauth 192.168.1.109 ! config: ipmask 255.255.255.0 ! config: end ! kremvax: create /adm -1 -1 775 d ! kremvax: create /adm/users -1 -1 664 ! kremvax: create /usr 10000 10000 755 d ! kremvax: halt (reboot) ! config: allow ! config: end then with drawterm ! cpu% 9fs kremvax ! cpu% acme /n/kremvax/adm/users then copied over the default kfs version ! kremvax: adduser matt ! kremvax: halt (reboot) and now I've got a writeable by only matt /usr/matt dir on the file server.) THOSE PARENTHESISED INSTRUCTIONS ARE WRONG. See [Upgrading to fourth edition]. Assuming you use the same setup I did earlier, type the following at the standalone CPU/auth server console: ! srv il!lance ! mount -c /srv/il!lance /n/lance ! cd /dist ! wrap/inst -ovr /n/lance plan9.9gz ! #(use wrap/inst for any other packages, e.g., tex) I've found that this doesn't take too long on a modern machine with a fast Ethernet and fast disks. My file server is a 550MHz Pentium III with 256MB RAM, an SYM53C875 SCSI controller, a 9GB SCSI disk holding ``main,'' and an Intel 82557 fast Ethernet controller. Installing the base distribution from a dual processor 733MHz Pentium III with 512MB RAM, a 3com 3c905 Ethernet controller and IDE disks over a Baystack 450-24T switch takes about 15 minutes or so. (It also only took 15 minutes for another 9'er with a 333MHz Pentium II CPU server and a 233MHz Pentium MMX fileserver, each with 128MB RAM, over fast Ethernet. Stuffing a 100Mbps pipe doesn't take much grunt.) Anyway, once the distribution and any additional software you're interested in is installed, you customize a few files on the file server before you reboot. In particular, rc/bin/cpurc, rc/bin/termrc, and lib/ndb/local are good candidates for customization: ! cd /n/lance/rc/bin ! ed termrc ! #... ! ed cpurc ! #... ! cd /n/lance/lib/ndb ! ed local ! #... ! #(edit any other files you feel are pertinent) ! unmount /n/lance See the ``Getting Started with Plan 9'' document for more information. Once the system is installed and administrative files have been customized, you add new users via the ``newuser'' command on the file server console. See fs(8) for more information on the particulars of user and group management. Note that when you create a user, you should also create its mailbox using the ``create'' command. An example of creating a user might be: ! newuser cross ! newuser sys +cross ! create /mail/box/cross cross upas 775 d ! create /mail/box/cross/mbox cross upas 662 al The first command creates a user called cross, and adds him to /adm/users. Next, it adds cross to the group sys. Finally, it creates cross's mailbox (see also the -c option of mail(1) ). See fs(8) for more information. Now you're basically done. Either continue to play with the file server while in the administrative mode where you can bypass access permissions, or reboot it immediately via the ``halt'' command to enable normal access checking: ! halt The system should boot off of the floppy and come up into ``normal'' mode, which permission bypassing disallowed. Congratulations, your file server is now fully set up! Should you need to re-enable bypassing of access checks for whatever reason at some point in the future, you can do so by re-entering config mode when the machine boots and typing the ``allow'' command followed by ``end''. The system will prompt you with ``type a key within the next 5 seconds to enter configuration mode''; this is how you re-enter config mode. If, now that you have a file server, you want to make your CPU/auth server diskless, i.e., make it boot from the file server, you can do so by using a boot floppy with a plan9.ini and a kernel for the CPU server. (You can of course use your hard disk to keep both files there). Here is the plan9.ini I use to boot my diskless CPU/auth server: ! bootfile=sdC0!9fat!9pcdisk ! bootargs=il -g 212.128.4.1 ether /net/ether0 212.128.4.205 255.255.255.0 ! fs=212.128.4.206 ! auth=212.128.4.205 As you can see, you can use bootargs to configure your gateway, ip address, and ip mask; fs for the ip address of your file server; and auth for the ip address of your auth server. In the example, the last IP address is the one for the CPU/auth server using the plan9.ini shown. mike@plan9os.org: After I got my auth server booting diskless, auth/changeuser wouldn't work because bootes didn't have write access to /adm/keys* or /adm/netkeys*. I had to delete these files and (they were of zero size anyway) and recreate them on the file server console like so: ! remove /adm/keys ! remove /adm/keys.who ! remove /adm/netkeys ! remove /adm/netkeys.who ! create /adm/keys bootes adm 660 ! create /adm/keys.who bootes adm 660 ! create /adm/netkeys bootes adm 660 ! create /adm/netkeys.who bootes adm 660 I'm not sure if this is good policy, but it worked.