Architecture

hyveOS is distributed along several system components. The core component is the hyved daemon, managed by systemd.

hyveOS Architecture

SDKs and CLI interfaces

The SDKs and CLI are mostly wrapper logic around rpc calls to the hyved daemon via grpc. See the protobuf definition for interface reference.

hyved

The hyved daemon is starting at boot time and is the long running task in the background where the magic happens.

It’s main activity is taking requests from the SDKs and CLI and then acting on them. To do that it runs the libp2p stack that is responsible for most of the behaviours exposed to the user via the SDKs and CLIs.

libp2p

libp2p is a modular network stack that is used to build peer-to-peer applications. It has wide adoption and is the back bone for most of the big Web3 projects out there including:

In hyveOS we use the Rust implementation of libp2p called rust-libp2p.

For a deeper dive into libp2p see the official documentation. The following is a high level overview of the compoments we use from libp2p as well as our custom sauce.

PeerID

Throught the documentation you will see references to PeerIDs. A PeerID is a unique identifier for a peer in the network. It is derived from the public key of the peer (via hashing), which is checked during the initial contact with a peer ensuring a certain degree of authenticity.

Kademlia

Kademlia is an implementation of a distributed hash table (DHT) that is used to store and retrieve data in a decentralized network. It allows to find data in the network while only contacting O(log n) nodes. It works by traversing the network like a tree while minimizing the distance to the target key. For a deeper dive into Kademlia see the official documentation. The ability of Kademlia to find data in a decentralized networks makes it also the ideal protocol for peer discovery and routing.

Kademlia has been around for decades and was the technology that enabled BitTorrent to become the first widely adopted peer-to-peer network.

Wiithin hyveOS we use Kademlia for the decentralized key-value store as well as the discovery mechanisms via the provider abstractions. It’s also used internally to power the hyveOS filetransfer system.

GossipSub

GossipSub is a protocol that is used to do efficient pub-sub beyond direct connections between publishers and subscribers. It works by maintaining a network graph of subscriptions and then forwarding messages to relevant neighbouring peer. This allows that even a small publisher in a large network can reach a large audience without having to maintain direct connections to all of them. For a deeper dive into GossipSub see the official documentation.

We fine tune GossipSub in hyveOS to encourage forwards to direct neighbours (in the context of the mesh topology), which allows our network to continue to function in congested environments or with topologies that contain a lot of bridges.

Request Response

Request Response is a protocol that is used to send a request to a peer and then wait for a response. It is used in hyveOS for the similiar named functionality in the SDKs and CLI.

QUIC

At a lower level hyveOS uses the QUIC protocol for transport (how the actual bits are pushed over the network) via libp2p. QUIC is a protocol that features built-in encryption, multiplexing, congestion control and efficient connection establishment. It makes it ideal for peer-to-peer networks where the network conditions can be unpredictable.

QUIC is already used by all FANG companies (Facebook, Amazon, Netflix, Google) and is the basis for the new HTTP/3 protocol.

Mesh Networking

hyveOS is built on top of a decentral wireless ad-hoc mesh network. This means that the network is self-organizing and self-healing. It is designed to be robust and resilient to network partitions and failures.

Mesh routing (batman-adv)

To realize this mesh network on conventional hardware we use batman-adv as the mesh routing protocol. It was desgined and developed by the Freifunk community and is widely used in community networks around the world with deployments that have several thousands of nodes¹.

batman-adv is a layer 2 mesh routing protocol that is designed to work on top of WiFi, although it can be used with other layer 2 technologies like Ethernet or (Bluetooth-)PANs as well. Unlike other mesh routing protocols it runs directly inside the Linux kernel which makes it very efficient and performant.

WiFi

hyveOS is mostly designed to run on/with WiFi hardware. To work with batman-adv and subsiquently hyveOS the WiFi hardware needs to support the IBSS² mode which is also known as ad-hoc mode.

Future

In the future we would like to support other wireless mesh networks like LoRa via meshstatic, Bluetooth via Bluetooth Mesh and IEEE 802.15.4 via Thread as well as native WiFi mesh via 802.11s.

Interoperation with libp2p

We use the unique insights we gain from the mesh network to optimize the libp2p stack.

We don’t relay on bootstrapping nodes to find peers, but instead contact neighbours discovered through the mesh network directly and gather routing information from them to find more peers.

We also optimze GossipSub and Kademlia to work well in the mesh network topology.

Custom Protocols

File Transfer

hyveOS has a custom file transfer protocol that is built on top of libp2p. It is used to transfer files between peers in the network.

To find a file via the file transfer protocol a peer queries the Kademlia DHT for the file CID (a unique identifier for the file) and then transfers the file from the peer.

Application

hyveOS has a custom deployment protocol is built on top of File Transfer and Docker. It is used to deploy applications to the network.

After packaging the application with Docker into an image, the image is transferred to the network via the file transfer protocol and then deployed on peers via Docker.

Footnotes

Freifunk Münsterland Grafana shows at the time of writing 3,318 active nodes in the network. ↩
IBSS stands for Independent Basic Service Set and is the technical term for the ad-hoc mode in WiFi networks. It allows devices to connect to each other without the need for a central access point. ↩