Plan 9 was designed as a distributed system. After you install the distribution from the cd, you have a self-sufficient one machine system, a standalone terminal. We will consider this as "Level 0" - how do you proceed from here to a network of Plan 9 machines and provide Plan 9 services to other clients? Note that this guide does not imply a strict dependence on the previous level, it is entirely possible to setup a Plan 9 DHCP/PXE boot server (described in level 7) without performing all the steps described in previous levels. This is a rough guide which proceeds in the order of increasing number of machines used and increasing elaboration and customization of configuration.

LEVEL 1: UPGRADING THE INSTALL TO A CPU SERVER

The traditional first step is following the Configuring a Standalone CPU Server wiki page. The transformation from terminal to cpu server provides many additional capabilities:

• Remote access via cpu(1)
• Multiple simultaneous users if desired
• Exportfs(4) and other services such as ftpd(8)
• Authentication with authsrv(6) for securing network services
• Integration with other operating systems via Drawterm

Even if you are only planning on making use of a single Plan 9 system, it is highly recommended to configure it as a cpu server. You probably use other operating systems as well as Plan 9, and configuring your Plan 9 system as a cpu server will make it vastly more useful to you by enabling it to share resources easily with non-Plan 9 systems.

It should also be noted that the division between terminals and cpus is mostly a matter of conventional behavior. It is possible to configure all services to run on a machine using the terminal kernel, but there is no particular advantage to this. The cpu kernel is easy to compile and can run a local gui and be used as a combined terminal/cpu server.

LEVEL 2: CPU AND DRAWTERM CONNECTIONS BETWEEN MULTIPLE MACHINES

The core functionality of a cpu server is to provide cpu(1) service to Plan 9 machines and Drawterm clients from other operating systems. The basic use of cpu(1) is similar to ssh in unix or remote desktop in Windows. You connect to the remote machine, and have full use of its resources from your terminal. However, Plan 9 cpu(1) also makes the resources of the terminal available to the cpu at /mnt/term. This sharing of namespace is actually the method of control also, because the cpu accesses the terminal's devices (screen, keyboard, mouse) by binding them over its own /dev. If you are a new Plan 9 user, you are not expected to understand this.

Drawterm from other operating systems behaves the same way. When you drawterm to a Plan 9 cpu server, your local files will be available at /mnt/term. This means you can freely copy files between Plan 9 and your other os without the use of any additional protocols. In other words, when working with drawterm, your environment is actually a composite of your local os and the Plan 9 system - technically it is a three node grid, because the Drawterm program acts as an ultra-minimal independent Plan 9 terminal system, connecting your host os to the Plan 9 cpu server.

For many home users, this style of small grid matches their needs. A single Plan 9 cpu/file/auth server functions both as its own terminal, and provides drawterm access to integrate with other operating systems. Some users like to add another light terminal only Plan 9 system as well. Recently (2013) Raspberry Pi's have become popular for this purpose. Another option with surprising benefits is using virtual machines for a cpu server. Because of Plan 9's network transparency, it can export all of its services and its normal working environment through the network.

LEVEL 3: A SEPARATE FILE SERVER AND TCP BOOT CPUS

A standard full-sized Plan 9 installation makes use of a separate file server which is used to tcp boot at least one cpu server and possibly additional cpus and terminals. A tcp booted system loads its kernel and then attaches to a root file server from the network. Some of the strengths of the Plan 9 design become more apparent when multiple machines share the same root fs.

• Sharing data is automatic when the root fs is shared
• Configuration and administration for each machine is located in one place
• Hardware use is optimized by assigning systems to the best-suited role (cpu/file/term)
• New machines require almost no configuration to join the grid

If you have already configured a standalone cpu server, it can also act as a file server if you instruct its fossil(4) to listen on the standard port 564 for 9p file service. Choosing "tcp" at the "root is from" option during bootup allows you to select a network file server. You can use plan9.ini(8) to create a menu with options for local vs tcp booting. A single file server can boot any number of cpu servers and terminals.

The first time you work at a terminal and make use of a cpu server when both terminal and cpu are sharing a root fs from a network file server is usually an "AHA!" moment for understanding the full Plan 9 design. In Plan 9 "everything is a file" and the 9p protocol makes all filesystems network transparent, and applications and system services such as the sam(1) editor and the plumber(4) message passing service are designed with distributed architecture in mind. A terminal and cpu both sharing a root fs and controlled by the user at the terminal can provide a unified namespace in which you can easily forget exactly what software is running on which physical machine.

LEVEL 4: SHARING /SRV /PROC AND /DEV BETWEEN MULTIPLE CPUS

The use of synthetic (non-disk) filesystems to provide system services and the network transparent 9p protocol allow Plan 9 to create tightly coupled grids of machines. This is the point at which Plan 9 surpasses traditional unix - in traditional unix, many resources are not available through the file abstraction, nfs does not provide access to synthetic file systems, and multiple interfaces and abstractions such as sockets and TTYs must be managed in addition to the nfs protocol.

In Plan 9 a grid of cpus simply import(4) resources from other machines, and processes will automatically make use of those resources if they appear at the right path in the namespace(4). The most important file trees for sharing between cpus are /srv, /proc, and some files from /dev and /mnt. In general all 9p services running on a machine will post a file descriptor in /srv, so sharing /srv allows machines to make new attaches to services on remote machines exactly as if they were local. The /proc filesystem provides management and information about running processes so import of /proc allows remote machines to control each other's processes. If machines need to make use of each other's input and output devices (the cpu(1) command does this) access is possible via import of /dev. Cpus can run local processes on the display of remote machines by attaching to the remote rio(4) fileserver and then binding in the correct /mnt and /dev files.

LEVEL 5: SEPARATE ARCHIVAL DATA SERVERS AND MULTIPLE FILE SERVERS

This guide has been referring to "the file server" and not making a distinction between systems backed by venti(8) and those without. It is possible and recommended to use venti(8) even for a small single machine setup. As grids become larger or the size of data grows, it is useful to make the venti server separate from the file server. Multiple fossil(4) servers can all use the same venti(8) server. Because of data deduplication multiple independent root filesystems may often be stored with only a slight increase in storage capacity used.

Administering a grid should include a system for tracking the rootscores of the daily fossil snapshots and backing up the venti arenas. A venti-backed fossil by default takes one archival snapshot per day and the reference to this snapshot is contained in a single vac: score. See vac(1). Because fossil(4) is really just a temporary buffer for venti(8) data blocks and a means of working with them as a writable fs, fossils can be almost instantaneously reset to use a different rootscore using flfmt -v.

To keep a full grid of machines backed-up, all that is necessary is to keep a backup of the venti(8) arenas partitions and a record of the fossil(4) rootscores of each machine. The rootscores can be recovered from the raw partition data, but it is more convenient to track them independently for faster and easier recovery. The simplest and best system for keeping a working backup is keeping a second active venti server and using venti/wrarena to progressively backup between them. This makes your backup available on demand simply by formatting a new fossil using a saved rootscore and setting the backup venti as the target. If the data blocks have all been replicated, the same rootscores will be available in both. See venti-backup(8).

LEVEL 6: MULTI-OS GRIDS USING U9FS, 9PFUSE, INFERNO, PLAN9PORT, 9VX

The 9p protocol was created for Plan 9 but is now supported by software and libraries in many other operating systems. It is possible to provide 9p access to both files and execution resources on non Plan 9 systems. For instance, Inferno speaks 9p (also called "styx" within Inferno) and can run commands on the host operating system with its "os" command. Thus an instance of Inferno running on Windows can bring those resources into the namespace of a grid. Another example is using plan9port, the unix version of the rc shell, and 9pfuse to import a hubfs from a Plan 9 machine and attach a local shell to the hubs. This provides persistent unix rc access as a mountable fs to grid nodes.

Please see connecting to other OSes and connecting from other OSes.

LEVEL 7: STANDALONE AUTH SERVER, PLAN 9 DHCP, PXE BOOT, DNS SERVICE, /NET IMPORTS, /NET.ALT

Plan 9 provides mechanisms to manage system roles and bootup from a DHCP/PXE boot server. At the size of grid used by an institution such as Bell Labs itself or a university research department, it is useful to separate system roles as much as possible and automate their assignment by having Plan 9 function to assign system roles and ips to Plan 9 machines via pxe boot and Plan 9 specific dhcp fields. This requires configuring ndb(8) to know the ethernet addresses of client machines and which kernel to serve them and a well-controlled local network.

One of the most common specialized roles in a mid-to large sized grid is the standalone auth-only server. Because auth is so important and may be a single point of failure of a grid, as well as for security reasons, it is often a good idea to make the auth server an independent standalone box which runs nothing at all except auth services and is hardened and secured as much as possible against failure and presents the minimal attack surface. In an institution with semi-trusted users such as a university, the auth server should be in a physically separate and secure location from user terminals.

A grid of this size will probably also have use for DNS service. For personal users on home networks, variables such as the authentication domain are often set to arbitrary strings. For larger grids which will probably connect to public networks at some nodes, the ndb(8) and authsrv(6) configuration will usually be coordinated with the publicly assigned domain names. It is also at this point (public/private interface) where machines may be connected to multiple networks using /net.alt (simply a conventional place for binding another network interface or connection) and may make use of one of Plan 9's most famous applications of network transparency - the import(4) of /net from another machine. If a process replaces the local /net with a remote /net, it will transparently use the remote /net for outgoing and incoming connections.

LEVEL 8: RESEARCH AND COMPUTATION GRIDS WITH CUSTOM CONTROL LAYERS AND UNIQUE CAPABILITIES

In its role as a research operating system, the capabilities of Plan 9 as a distributed system are often extended by specific projects for specific purposes or to match specific hardware. Plan 9 does not include any built-in capability for things like task dispatch and load balancing between nodes. The Plan 9 approach is to provide the cleanest possible set of supporting abstractions for the creation of whatever type of high-level clustering you wish to create. Some examples of research grids with custom software and capabilities:

• The main Bell Labs Murray Hill installation includes additional software and extensive location specific configuration
• The Laboratorio de Sistemas and project leader Francisco J Ballesteros (Nemo) created Plan B and the Octopus, derived from Plan 9 and Inferno respectively.
• The XCPU project is clustering software for Plan 9 and other operating systems created at the Los Alamos National Laboratory
• The Plan 9 on IBM Blue Gene project utilizes special purpose tools to let Plan 9 control the architecture of the IBM Blue Gene L.
• The ANTS Advanced Namespace ToolS are a 9gridchan project for creating failure-tolerant grids and persistent LAN/WAN user environments.

These are examples of projects which are built on 9p and the Plan 9 design and customize or extend the operating system for additional clustering, task management, and specific purposes. The flexibility of Plan 9 is one of its great virtues. Most Plan 9 users customize their setups to a greater or lesser extent with their own scripts or changes to the default configuration. Even if you aren't aspiring to build a 20-node "Manta Ray" swarm to challenge Nemo's Octopus, studying these larger custom systems may help you find useful customizations for your own system, and the Plan 9 modular design means that some of the software tools used by these projects are independently useful.

LEVEL 9: POWER SET GRIDS?

Because existing grids already have the label "9grid" it is theorized the as-yet unreached Ninth Level of Gridding corresponds to the power set operation. The previously described eight levels of gridding already encompass an infinite Continuum of possibilities. To surpass the existing level requires finding a way to construct the power set of this already infinite set. Level Nine grids are therefore transfinite, not merely infinite, and it is an open question if current physical reality could accommodate such structures. At the present moment, research indicates such hypothetical Level Nine grids would need to post mountable file descriptors from quasars and pulsars and store information by entropic dissipation through black hole event horizons. See astro(7) and scat(7).