illumos: manual page: overlay.7

OVERLAY(7) Standards, Environments, and Macros OVERLAY(7)

NAME

overlay - Overlay Devices

DESCRIPTION

Overlay devices are a GLDv3 device that allows users to create overlay
networks that can be used to form the basis of network virtualization and
software defined networking. Overlay networks allow a single physical
network, often called an underlay network, to provide the means for
creating multiple logical, isolated, and discrete layer two and layer three
networks on top of it.

Overlay devices are administered through dladm(8). Overlay devices
themselves cannot be plumbed up with IP, vnd, or any other protocol.
Instead, like an etherstub, they allow for VNICs to be created on top of
them. Like an etherstub, an overlay device acts as a local switch;
however, when it encounters a non-local destination address, it instead
looks up where it should send the packet, encapsulates it, and sends it out
another interface in the system.

A single overlay device encapsulates the logic to answer two different, but
related, questions:

1. How should a packet be transformed and put on the wire?
2. Where should a transformed packet be sent?

Each of these questions is answered by a plugin. The first question is
answered by what's called an encapsulation plugin. The second question is
answered by what's called a search plugin. Packets are encapsulated and
decapsulated using the encapsulation plugin by the kernel. The search
plugins are all user land plugins that are consumed by the varpd service
whose FMRI is svc:/network/varpd:default. This separation allows for the
kernel to be responsible for the data path, while having the search plugins
in userland allows the system to provide a much more expressive interface.

Overlay Types

Overlay devices come in two different flavors, one where all packets are
always sent to a single address, the other, where the destination of a
packet varies based on the target MAC address of the packet. This
information is maintained in a target table, which is independent and
unique to each overlay device. We call the plugins that send traffic to a
single location, for example a single unicast or multicast IP address, a
point to point overlay and the overlay devices that can send traffic to
different locations based on the MAC address of that packet a dynamic
overlay. The plugin type is determined based on the type of the search
plugin. These are all fully listed in the section Plugins and their
Properties.

Overlay Destination

Both encapsulation and search plugins define the kinds of destinations that
they know how to support. An encapsulation plugin always has a single
destination type that's determined based on how the encapsulation is
defined. A search plugin, on the other hand, can support multiple
combinations of destinations. A search plugin must support the destination
type of the encapsulation device. The destination may require any of the
following three pieces of information, depending on the encapsulation
plugin:

MAC Address
An Ethernet MAC address is required to determine the destination.

IP Address
An IP address is required. Both IPv4 and IPv6 addresses are
supported.

Port
An IP protocol level (TCP, UDP, SCTP, etc.) port is required.

The list of destination types that are supported by both the search and
encapsulation plugins is listed in the section Plugins and their
Properties.

varpd
The varpd service, mentioned above, is responsible for providing the
virtual ARP daemon. Its responsibility is conceptually similar to ARP. It
runs all instances of search plugins in the system and is responsible for
answering the kernel's ARP-like questions for where packets should be sent.

The varpd service, svc:/network/varpd:default, must be enabled for overlay
devices to function. If it is disabled while there are active devices,
then most overlay devices will not function correctly and likely will end
up dropping traffic.

PLUGINS AND PROPERTIES

Properties fall into three categories in the system:
1. Generic properties all overlay devices have
2. Properties specific to the encapsulation plugin
3. Properties specific to the search plugin

Each property in the system has the following attributes, which mirror the
traditional dladm(8) link properties:

Name
The name of a property is namespaced by its module and always
structured and referred to as as module/property. This allows for
both an encapsulation and search plugin to have a property with the
same name. Properties that are valid for all overlay devices and
not specific to a module do not generally use a module prefix.

For example, the property vxlan/listen_ip is associated with the
vxlan encapsulation module.

Type
Each property in the system has a type. dladm(8) takes care of
converting between the internal representation and a value, but the
type influences the acceptable input range. The types are:

INT A signed integer that is up to eight bytes long (int64_t).

UINT An unsigned integer that is up to eight bytes long
(uint64_t).

IP Either an IPv4 or IPv6 address in traditional string form.
For example, 192.168.128.23 or 2001:470:8af4::1:1. IPv4
addresses may also be encoded as IPv4-mapped IPv6
addresses.

STRING A string of ASCII or UTF-8 encoded characters terminated
with a NUL byte. The maximum string length, including the
terminator, is currently 256 bytes.

Permissions
Each property has permissions associated with it, which indicate
whether the system considers them read-only properties or read-
write properties. A read-only property can never be updated once
the device is created. This generally includes things like the
overlay's encapsulation module.

Required
This property indicates whether the property is required for the
given plugin. If it is not specified during a call to dladm
create-overlay, then the overlay cannot be successfully created.
Properties which have a default will use that value if one is not
specified rather than cause the overlay creation to fail.

Current Value
The current value of a property, if the property has a value set.
Required properties always have a value set.

Default Value
The default value is an optional part of a given property. If a
property does define a default value, then it will be used when an
overlay is created and no other value is given.

Value ranges
Value ranges are an optional part of a given property. They
indicate a range or set of values that are valid and may be set for
a property. A property may not declare such a range as it may be
impractical or unknown. For example, most properties based on IP
addresses will not declare a range.

The following sections describe both the modules and the properties that
exist for each module, noting their name, type, permissions, whether or not
they are required, and if there is a default value. In addition, the
effects of each property will be described.

Encapsulation Plugins

vxlan The vxlan module is a UDP based encapsulation method. It takes a
frame that would be put on the wire, wraps it up in a VXLAN header
and places it in a UDP packet that gets sent out on the underlying
network. For more details about the specific format of the VXLAN
header, see vxlan(4P).

The vxlan module requires both an IP address and port to address
it. It has a 24-bit virtual network ID space, allowing for virtual
network identifiers that range from 0 - 16777215.

The vxlan module has the following properties:

vxlan/listen_ip
Type: IP | Permissions: Read/Write | Required

The vxlan/listen_ip property determines the IP address that
the system will accept VXLAN encapsulated packets on for
this overlay.

vxlan/listen_port
Type: UINT | Permissions: Read/Write | Required
Default Value: 4789 | Range: 0 - 65535

The vxlan/listen_port property determines the UDP port that
the system will listen on for VXLAN traffic for this
overlay. The default value is 4789, the IANA assigned port
for VXLAN.

The vxlan/listen_ip and vxlan/listen_port properties determine how
the system will accept VXLAN encapsulated packets for this
interface. It does not determine the interface that packets will
be sent out over. Multiple overlays that all use VXLAN can share
the same IP and port combination, as the virtual network identifier
can be used to tell the different overlays apart.

Search Plugins

Because search plugins may support multiple destinations, they may have
more properties listed than necessarily show up for a given overlay. For
example, the direct plugin supports destinations that are identified by
both an IP address and a port, or just an IP address. In cases where the
device is created over an overlay that only uses an IP address for its
destination, then it will not have the direct/dest_port property.

direct The direct plugin is a point to point module that can be used to
create an overlay that forwards all non-local traffic to a single
destination. It supports destinations that are a combination of an
IP Address and a port.

The direct plugin has the following properties:

direct/dest_ip
Type: IP | Permissions: Read/Write | Required

The direct/dest_ip property indicates the IP address that
all traffic will be sent out. Traffic will be sent out the
corresponding interface based on traditional IP routing
rules and the configuration of the networking stack of the
global zone.

direct/dest_port
Type: UINT | Permissions: Read/Write | Required
Default Value: - | Range: 0 - 65535

The direct/dest_port property indicates the TCP or UDP port
that all traffic will be directed to.

files The files plugin implements a dynamic plugin that specifies where
traffic should be sent based on a file. It is a glorified version
of /etc/ethers. The dynamic plugin does not support broadcast or
multicast traffic, but it has support for proxy ARP, NDP, and
DHCPv4. For the full details of the file format, see
overlay_files(5).

The files plugin has the following property:

files/config
Type: String | Permissions: Read/Write | Required

The files/config property specifies an absolute path to a
file to read. The file is a JSON file that is formatted
according to overlay_files(5).

General Properties

Each overlay has the following properties which are used to give additional
information about the system. None of these properties may be specified as
part of a dladm create-overlay, instead they come from other arguments or
from internal parts of the system.

encap
String | Permissions: Read Only

The encap property contains the name of the encapsulation module
that's in use.

mtu
UINT | Permissions: Read/Write
Default Value: 1400 | Range: 576 - 9000

The mtu property describes the maximum transmission unit of the
overlay. The default value is 1400 bytes, which ensures that in a
traditional deployment with an MTU of 1500 bytes, the overhead that
is added from encapsulation is all accounted for. It is the
administrator's responsibility to ensure that the device's MTU and
the encapsulation overhead does not exceed that of the interfaces
that the encapsulated traffic will be sent out of.

To modify the mtu property, use dladm set-linkprop.

search
String | Permissions: Read Only

The search property contains the name of the search plugin that's
in use.

varpd/id
String | Permissions: Read Only

The varpd/id property indicates the identifier which the varpd
service uses for this overlay.

vnetid
UINT | Permissions: Read/Write

The vnetid property has the virtual network identifier that belongs
to this overlay. The valid range for the virtual network
identifier depends on the encapsulation engine.

FMA INTEGRATION

Overlay devices are wired into FMA, the illumos fault management
architecture, and generates error reports depending on the search plugin in
use. Due to limitations in FMA today, when a single overlay enters a
degraded state, meaning that it cannot properly perform look ups or another
error occurred, then it degrades the overall overlay pseudo-device driver.

For more fine-grained information about which overlay is actually in a
degraded state, one should run dladm show-overlay -f. In addition, for
each overlay in a degraded state a more useful diagnostic message is
provided which describes the reason that caused this overlay to enter into
a degraded state.

The overlay driver is self-healing. If the problem corrects itself on its
own, it will clear the fault on the corresponding device.