vCluster: How Is It Used and a Comparison to NKE

What Is vCluster?

A virtual cluster is a fully working Kubernetes cluster that runs on top of another Kubernetes cluster. It is focused solely on core components, comprising the API server, controller manager and a flexible storage backend. By default, vCluster utilises Embedded SQLite for storage, thereby ensuring simplicity and efficiency. However, users have the freedom to opt for alternative options such as PostgreSQL, MySQL, MariaDB or etcd, depending on their specific requirements. Additionally, vCluster offers the flexibility of including a Scheduler.

While it defaults to utilising k3s as the virtual Kubernetes cluster, it remains agnostic to specific distributions. This means that vCluster is compatible with other certified Kubernetes ‘flavours’ such as Vanilla k8s or k0s. This flexibility allows users to choose the distribution that best suits their needs, without sacrificing the simplicity and efficiency provided by the vCluster’s minimalist architecture.

By default, a vCluster operates as a single pod, housing two essential containers: the Control Plane and Syncer. The Control Plane orchestrates the core Kubernetes components. The Syncer synchronises data between the vCluster and the host cluster. Running on top of another Kubernetes cluster, known as the host cluster, the vCluster leverages the underlying infrastructure while providing a distinct and isolated environment for containerised workloads. It is simply deployed as a regular StatefulSet within a designated Namespace in the host cluster. Notably, everything created within a vCluster resides either within the vCluster itself or within the host Namespace, facilitating clear delineation of boundaries.

Minimalistic Architecture

Synced Resources

When Pods are initiated within the virtual cluster, they undergo a transformation before being launched within the virtual cluster’s designated Namespace in the host cluster. This process involves rewriting the Pod specifications to align with the host cluster’s Namespace conventions, resulting in longer Pod names. For example, the Pod nginx-76d6c9b8c-57xk5 created in the Namespace nginx within the vCluster my-vcluster, becomes nginx-76d6c9b8c-57xk5-x-nginx-x-my-vcluster in the host cluster. Similarly, Services and Endpoints associated with the virtual cluster are adapted and deployed within the corresponding Namespace in the host cluster.

Furthermore, the coordination between virtual and host clusters extends to sharing service cluster IPs. This shared infrastructure simplifies networking configurations and enhances interoperability.

As vCluster’s Syncer diligently synchronises Pods with the host cluster for scheduling, users can seamlessly utilise the host cluster’s storage classes to create PVCs and mount PersistentVolumes. By default, vCluster enables access to the host’s storage classes without the need for duplication within the vCluster environment, thus improving storage management and enhancing flexibility.

Certain resources undergo synchronisation, while others remain confined to the virtual cluster. ConfigMaps and Secrets, when mounted to Pods, seamlessly traverse to the host cluster. However, all other Configmaps or Secrets will purely stay in the virtual cluster to ensure isolation and encapsulation. Notably, Deployments, StatefulSets, Custom Resource Definitions (CRDs), ServiceAccounts and similar resources do not undergo synchronisation and maintain autonomy. This selective synchronisation strategy ensures efficient resource utilisation, while keeping the virtual and host clusters separate. The full list of resources that can be synced or mirrored by vCluster (currently) can be found in the documentation.

vCluster Design Principles

At the heart of vCluster’s architecture lies a set of carefully crafted design principles, each aimed at delivering an efficient Kubernetes experience. First and foremost, vCluster prides itself on being as lightweight as possible. This is achieved by encapsulating the entire vCluster within a single Pod and leveraging k3s as the control plane. This minimalist approach ensures that using vCluster imposes minimal overhead on the underlying infrastructure while maximizing resource utilisation.

Whether within a vCluster or nested vClusters, workloads operate with the same computing power, and offer the same access to persistent storage and network performance as those running directly on the host cluster.

Moreover, vCluster reduces the load on the host cluster’s Kubernetes API server by keeping high-level resources confined within the vCluster. By employing a separate API server and data store dedicated to the vCluster, and synchronising only essential low-level resources with the underlying cluster, a vCluster minimises API server requests, thus optimising performance and scalability.

Finally, vCluster prioritises flexible provisioning and cleanup processes, offering a variety of deployment options such as vCluster CLI, Helm or kubectl (see Deploy vCluster for more info). By adopting a simple StatefulSet + Service deployment model, vCluster integrates seamlessly with existing Kubernetes tools, simplifying all processes for users in a variety of environments.

Interesting Facts

vCluster presents some intriguing capabilities that set it apart from traditional cluster management solutions. One notable feature is the ability to run a vCluster inside another vCluster, a practice known as vCluster nesting. This allows for the creation of hierarchical Kubernetes environments, offering unprecedented flexibility and scalability. With vCluster nesting, users can deploy and manage multiple layers of Kubernetes clusters within a single infrastructure, facilitating complex deployment scenarios and enabling innovative use cases. I recommend starting your exploration of this topic by reading the article Running Nested clusters in Kubernetes using vCluster.

Another interesting detail is how vCluster lists resources: at the moment, it sorts them by age. Why? k3s uses Kine (etcdshim that translates etcd API to, e.g. SQLite). Kine [is] not sorting the keys in the same way as etcd, but I don’t think that’s guaranteed anywhere (see GitHub thread Results of kubectl commands not sorted). This is true for now, but some work on delivering The Implicit Kubernetes-ETCD Contract is ongoing (more info: On the Hunt for Etcd Data Inconsistencies – Marek Siarkowicz, Google) and perhaps in the future we will see some cool changes in this area. Nevertheless, for now, the proposal for displaying resources is as follows: objects listed in response must be in lexical order by their name. And in my opinion, it would be worth following this rule starting… now. 😉

Tips

When navigating a vCluster environment, a few tips can optimise performance. Firstly, there is the option of using a separate vCluster Scheduler. While this feature is available, it’s advisable to disable it, especially if you rely on autoscaling. Although there are trade-offs involved, maintaining autoscaling functionality typically outweighs the benefits of a separate Scheduler. For more insights, refer to the documentation on Virtual Schedulers.

Furthermore, certain vCluster resources may be shared with your ‘normal’ infrastructure, emphasising the need for strict access controls and security measures. It is crucial to ensure that sensitive data or resources are inaccessible from within the vCluster environment, mitigating the risk of unauthorised access or data breaches.

Benefits

vCluster grants users full administrative access, empowering them to deploy Custom Resource Definitions (CRDs) or create Namespaces. This level of control facilitates custom configurations and tuning to accommodate a variety of deployment scenarios.

Moreover, due to lightweight virtual clusters, which share the resources of the underlying host cluster, vCluster can save considerable costs.

Comparing vCluster with alternatives such as KinD, K3d, and Minikube reveals its superiority in terms of speed and efficiency, particularly in local development environments. This is explored further in the article How Virtual Kubernetes Clusters Can Speed Up Your Local Development, highlighting vCluster’s potential to accelerate development cycles and streamline Kubernetes workflows. Overall, vCluster offers a comprehensive solution for Kubernetes orchestration, combining performance, efficiency and security to meet the diverse needs of modern deployment environments.

Use Cases

vCluster presents a versatile solution with a wide range of use cases.

One prominent use case is their application in development environments, both for local and remote scenarios. By providing lightweight and isolated clusters, vCluster facilitates rapid iteration and experimentation, empowering developers to build and test applications with ease.

Moreover, vCluster excels in end-to-end testing (E2E) scenarios, offering a good environment for validating application behaviour and functionality across various deployment configurations.

Additionally, vCluster prove invaluable in testing against multiple versions of Kubernetes, allowing users to assess compatibility and performance across different releases. This versatility enables businesses to future-proof their applications and infrastructure by ensuring compatibility with upcoming Kubernetes updates.

Furthermore, vClusters serve as an indispensable tool for training, workshops and demo clusters. Their lightweight nature and ease of provisioning make them an ideal platform for showcasing Kubernetes concepts and conducting hands-on training sessions.

vClusters address the challenges of multitenancy in Kubernetes environments, offering a great alternative to traditional namespace-based isolation. As highlighted in the article Why Namespaces aren’t Good Enough, vCluster provides robust isolation mechanisms that overcome the limitations of Namespaces, ensuring secure and efficient multitenant deployments.

Additionally, vClusters facilitate the isolation of certain resources that reside globally within the cluster, as discussed in Why use Virtual Kubernetes Clusters? This enables users to effectively manage and secure resources that cannot be isolated using traditional namespace-based approaches, thus enhancing the overall security of Kubernetes deployments.

vCluster vs NKE

vCluster offers an alternative to NKE-backed Kubernetes clusters, catering to different use cases, while also exhibiting some limitations. Utilising vCluster, Kubernetes clusters can be provided at a more affordable price point, though with some trade-offs regarding availability guarantees and managed add-ons. More details (and a detailed comparison) can be found in our documentation on Kubernetes Cluster backed by vcluster.