= Design Overview = The NIS plugin module's aim is to serve up data from the directory server using the NIS protocols. It does this by doing what any gateway would do: it queries the directory server for entries which would correspond to the contents of maps, reads the contents of various attributes from those entries, and uses that data to synthesize entries for maps which it serves to clients. In broad strokes, one design might look like this: ┌──────────┐ NIS ┌───────────┐ LDAP ┌────────────────────┐ │ Client │─────────│ Gateway │──────────│ Directory Server │ └──────────┘ └───────────┘ └────────────────────┘ The links in this diagram represent network traffic. The client uses the NIS protocol to communicate with the gateway, and the gateway uses the LDAP protocol to communicate with the directory server. This implementation requires that the gateway be robust against variations in directory server availability, be flexible enough to use any of a number of methods of authenticating to the directory server, and may additionally require the presence of specific extensions on the server in order to be able to be even reasonably certain of consistency with the directory's contents. In order to sidestep these requirements, and the complexity they add to an implementation, we decided to implement the gateway as a plugin. As a plugin, the gateway starts and stops with the directory server, it does not need to authenticate as a normal client would, and it can be expected to work with a server which can use it. Taking just the gateway and directory server portions of the above diagram, and breaking them down further, we can come to this: ┌──────────────┐ ┌─────────┐ ┌────────────────────────────┐ │ NIS Protocol │───│ Mapping │───│ Directory Server Back Ends │ └──────────────┘ └─────────┘ └────────────────────────────┘ The links in this diagram are now API calls. We've relegated the work of reading a query (parsed from the NIS client by the NIS Protocol handler), converting that query to a directory server search operation, and marshalling the results of that search into a format suitable for transmission as a NIS response, all to the Mapping module. The directory server back ends are exposed by SLAPI, of course. This approach does have its problems, though. NIS, as a protocol, requires that the server be able to supply a few bits of information which can't readily (or shouldn't) be retrieved this way, information which is not normally kept in a directory server. NIS requires that a server be able to report a revision number for a map, which is used as an indicator of the time when the map was last modified. A slave server can use this information to poll for changes in map contents on the master, possibly beginning a full map enumeration to read those new contents in order to serve its clients. A directory server, if it stores revision information at all, stores it on a per-entry basis. So when a gateway designed as we diagrammed above is asked for this information, it has at least these options: a) always use the current time - This causes frequent map updates on clients when they don't need them, and completely unnecessary network traffic. b) always use the same value - This keeps clients from ever noticing that a map has changed. c) return the latest revision of any of the results which formed the contents of the map - This could severely load a directory server if the information needs to be generated by reading the last-modified timestamps from many directory server entries. NIS also requires that a server be able to answer whether or not it services a specified domain, and which maps it serves for a domain that it serves. While the mapping module could search the directory's configuration space whenever it is asked these questions, the first question is asked regularly by each running copy of ypbind, which could also bog servers down (though admittedly, less than the previous case). If we break the mapping portion up further, we can introduce a map cache. In this module we can maintain a cache of the NIS server's data set, taking care to construct it at startup-time, updating it as the contents of the directory server change, and always serving clients using data from the cache. ┌──────────────┐ ┌───────────┐ ┌──────────────┐ ┌──────┐ │ NIS Protocol │──│ Map Cache │──│ Map Back End │──│ Data │ └──────────────┘ └───────────┘ └──────────────┘ └──────┘ Which takes us to the current design. The NIS protocol handler reads data from the map cache, and the map back end uses SLAPI to obtain data which is used to populate the map cache at startup-time, as well as to watch for changes in the directory's contents which would need to be reflected in the map cache. = Components = == Protocol Handler == This NIS protocol handler module takes the opportunity to set up listening sockets (listening on the port specified in the plugin configuration entry's "nsslapd-pluginarg0" attribute, or an unused port if none is specified) and register with the local portmapper at module initialization time. The plugin then starts a listening thread to service its clients. The plugin listens for datagram queries from clients, processing them as they come in, as well as answering connections from clients. Because connected clients may not always transmit an entire request at once, and because the server may find itself unable to transmit an entire response at once, it buffers traffic for connected clients, multiplexing the work it does for all of its clients from inside of the single thread. The actual protocol datagram parsing is performed by libnsl, which is provided as a part of the C library. Datagram responses which exceed the "nis-max-dgram-size" threshold (by default, 1024 bytes, or 1 kilobyte) are simply dropped. Response records larger than "nis-max-value-size" (by default, 256 kilobytes) are also ignored, even for connected clients. Client access is limited by the local tcp_wrappers configuration on the directory server, with a tcp_wrappers service name as dictated by the "nis-tcp-wrappers-name" attribute (by default, "nis-plugin") in the plugin's configuration. Client requests are also limited based on a client's address using "securenet"-style settings in the module's configuration entry's "nis-securenet" attribute. If no values are specified, access is allowed to all clients. Client requests are further classed as "secure" or not, based on the query's originating port. This information is used elsewhere for additional access control. == NIS Layer == The NIS layer processes complete requests, whether they come in from connected or datagram clients, fetches the requested information from the map cache, and uses callbacks provided by the protocol handler to respond. Before doing so, if a value is being retrieved from a map, it checks if the map's contents are restricted to "secure" clients. If they are, but the client is not a "secure" client, the NIS layer will respond as if no data were present in the map. == Map Cache == The map cache keeps a dynamically-constructed set of maps in memory, grouped by domain, and for each map maintains information regarding the last time its contents were modified (to answer client requests for a map's order) and whether or not the map's contents should be restricted to "secure" clients. The map cache can quickly answer whether or not a domain is being served by checking whether or not any maps are defined for it. The definitions of which maps are served for which domains is configurable via internal APIs -- the map cache itself has no forehand knowledge of domain names, map names, or formats, as it merely models data in the way that a conventional NIS server might. Forcing queries to use the cache provides a couple of benefits over an alternate approach of performing an LDAP query for each NIS query: * While the directory server is generally only case-preserving, the NIS server can be case-sensitive, which is preferred by NIS clients and a requirement for some customers. * Because the query used is never used to construct an LDAP filter or query, we don't have to worry about escaping text to avoid string injection attacks. == Back End == The backend interface module sets up, populates, and maintains the map cache. At startup time, it configures the map cache with the list of domains and maps, and populates the maps with initial data. Using postoperation plugin hooks, the backend interface also notes when entries are added, modified, renamed (modrdn'd), or deleted from the directory server. It uses this information to create or destroy maps in the map cache, and to add, remove, or update entries in the map cache's maps, thereby ensuring that the map cache always reflects the current contents of the directory server. The backend interface reads the configuration it should use for the map cache from its configuration area in the directory server. Beneath the plugin's entry, the backend checks for entries with these attributes: * nis-domain * nis-map * nis-secure * nis-base * nis-filter * nis-key-format * nis-keys-format * nis-value-format The backend then instructs the map cache to prepare to hold a map in the given domain (or domains) with the given map name (or names), and then performs a subtree search under the specified base (or bases, if there's more than one "nis-base" value) for entries which match the provided filter. Each entry found is then "added" to the map, using the format specifiers stored in "nis-key-format" to construct the keys for the entry in the map, with the corresponding value in the map being constructed using the format specifier given as the "nis-value-format". The map is marked as a "secure" map according to the "nis-secure" attribute, if so set. For each "nis-key-format" value, exactly one entry will be created in a NIS map. For each "nis-keys-format" value, any number of entries will be created in a NIS map. Should one of the directory server entries which was used to construct one or more NIS map entries be modified or removed, the corresponding entries in every applicable NIS map are updated or removed. Likewise, if an entry is added to the directory server which would correspond to an entry in a NIS map, entries are created in the corresponding NIS maps. == Formatting Data for NIS == The "nis-key-format" and "nis-value-format" specifiers resemble an RPM format specifier, and can include the values of multiple attributes in any part of the specifier. The backend composes the string using the attribute values stored in the directory server entry, using the format specifier as a guide. In this way, the NIS map's contents can be constructed to almost any specification, and can make use of data stored using any schema. An example specification for the value for a user's entry could look something like this: %{uid}:%{userPassword:-*}:%{uidNumber}:%{gidNumber}:%{gecos:-%{cn:-}}:%{homeDirectory}:%{loginShell:-/bin/sh} The syntax borrows from RPM's syntax, which in turn borrows from shell syntax, to allow the specification of alternate values to be used when the directory server entry doesn't include a "userPassword" or "gecos" attribute. Additional operators include "#", "##", "%", "%%", "/", "//", which operate in ways similar to their shell counterparts (with one notable exception being that patterns for the "/" operator can not currently be anchored to the beginning or end of the string). As a safety precaution, any reference to an attribute value which does not also specify an alternate value will cause the directory server entry to be ignored if the referenced attribute has no value defined for that entry, or contains multiple values. In the above example, the entry would be ignored if the "uid", "uidNumber", "gidNumber", or "homeDirectory" attributes of the entry did not each contain exactly one value. The syntax further defines "functions" which can be used to concatenate lists of multiple values into a single result, for example for groups: %{cn}:%{userPassword:-*}:%{gidNumber}:%list{",","memberUid"}) This filter takes advantage of a built-in "list" function, which processes zero or more values of the "memberUid" attribute and concatenates them together with a "," separator, to generate the list of group members. The "nis-filter", "nis-key-format", and "nis-value-format" settings have sensible defaults for the maps which we expect to be commonly used -- this is important because it's easy to subtly construct malformed result specifiers which could trigger undefined behavior on clients -- for example by leaving the user's numeric UID empty in a passwd entry, which may be treated as "0" by inattentive clients. The format specifier includes function-like invocations to allow the backend to be instructed to chase references to other entries, for example to handle flattening of nested groups or netgroups. A function-like invocation expects a comma-separated list of double-quoted arguments. and any arguments which contain a double-quote need to escape the double-quote using a '\' character -- this character itself also needs to be escaped whenever it appears. === Implemented Functions === * echo() - Reproduces what it is given. - Example: %echo("foo","bar") -> foobar * match(ATTRIBUTE,PATTERN[,DEFAULT]) - Selects the value of ATTRIBUTE which matches the globbing pattern PATTERN. If zero or two or more values match, and a DEFAULT was specified, the DEFAULT is produced, otherwise an error occurs. - Example (examining "cn=group"): dn: cn=group member: bob member: dave %match("member","b*") -> bob %match("member","d*") -> dave %match("member","e*") FAILS %match("member","e*","jim") -> jim * regmatch(ATTRIBUTE,PATTERN[,DEFAULT]) - Selects the value of ATTRIBUTE which matches the extended regular expression PATTERN. If zero or two or more values match, and a DEFAULT was specified, the DEFAULT is produced, otherwise an error occurs. - Example (examining "cn=group"): dn: cn=group member: bob member: dave %regmatch("member","^b.*") -> bob %regmatch("member","^d.*") -> dave %regmatch("member","e") -> dave %regmatch("member","^e") FAILS %regmatch("member","^e.*","jim") -> jim * regsub(ATTRIBUTE,PATTERN,TEMPLATE[,DEFAULT]) - Selects the value of ATTRIBUTE which matches the extended regular expression PATTERN and uses TEMPLATE to construct the output. If zero or two or more values match, and a DEFAULT was specified, the DEFAULT is produced, otherwise an error occurs. The template is used to construct a result using the n'th substring from the matched value by using the sequence "%n" in the template. - Example (examining "cn=group"): dn: cn=group member: bob member: dave %regsub("member","o","%0") -> bob %regsub("member","o","%1") -> %regsub("member","^o","%0") FAILS %regsub("member","^d(.).*","%1") -> a %regsub("member","^(.*)e","t%1y") -> tdavy %regsub("member","^e","%1") FAILS %regsub("member","^e.*","%1","jim") -> jim * list(SEPARATOR,ATTRIBUTE[,...]) - Creates a separated list of the values of the named attributes for the current entry. If an attribute has multiple or no values, they are either included or skipped. - Example (examining "cn=group"): dn: cn=group member: bob member: dave %list(",","member") -> bob,dave * deref(SEPARATOR,THISATTRIBUTE,THATATTRIBUTE) - Creates a separated list of the values of THATATTRIBUTE for directory entries named by this entry's THISATTRIBUTE. - Example (examining "cn=group"): dn: cn=group member: uid=bob member: uid=pete - dn: uid=bob uid: bob - dn: uid=pete uid: pete %deref(",","member","uid") -> bob,pete * referred(SEPARATOR,MAP,THATATTRIBUTE,THATOTHERATTRIBUTE) - Creates a separated list of the values of THATOTHERATTRIBUTE for directory entries which have entries in the named MAP and which also have this entry's name as a value for THATATTRIBUTE. - Example (examining "cn=group"): dn: cn=group - dn: uid=bob uid: bob memberOf: cn=group - dn: uid=pete uid: pete memberOf: cn=group %referred(",","memberof","uid") -> bob,pete * merge(SEPARATOR,EXPRESSION[,...]) - Evaluates and then creates a list using multiple expressions. - Example (examining "cn=group"): dn: cn=group membername: jim member: uid=bob member: uid=pete - dn: uid=bob uid: bob - dn: uid=pete uid: pete %merge(",","%list(\",\",\"membername\")","%deref(\",\",\"member\",\"uid\")") -> jim,bob,pete [== Publisher == The module additionally supports re-publishing the contents of the NIS maps through a read-only virtual backend. The published results are "flattened" to resemble the original RFC2307 schema so that naive clients which expect a single directory server entry to contain all of the desired information about a user or group to make use of data which might be stored in a more natural (LDAP-wise) representation elsewhere.]