Applicable versions

All Longhorn versions.

Purpose

The purpose of this article is to help Longhorn users understand how Longhorn typically acts in SELinux-enabled systems and provide them with basic commands to run to verify normal operation. It is a …

Applicable versions

All Longhorn versions.

Symptoms

Attempts to attach an RWX volume are unsuccessful, and the workload using the volume is unable to start. The logs contain the following messages:

Oct 11 07:42:23 dev-worker-1 k3s[1294]: Mounting …

Applicable versions

• Longhorn version >= v1.4.0
• Node kernel version <= v4.12

Symptoms

When running filesystem trim (either by Longhorn UI or manually by fstrim command on the host), it hits the error similar to:

unable to trim filesystem for …

Applicable versions

Confirmed with:

• K3s v1.24.8+k3s1 and RKE2 v1.24.8+rke2r1
• Longhorn v1.3.3
• SUSE Linux Enterprise Server 15 SP5

Likely possible with:

• Any Kubernetes distribution running Flannel CNI < v0.20.2
• Any Longhorn version
• A potential …

Applicable versions

All Longhorn versions, but some features are introduced in v1.4.0 or v1.5.0

Volumes consume much more space than expected

Due to the fact that Longhorn volumes can hold historic data as snapshots, the volume actual size can be …

Applicable versions

Confirmed in:

• Longhorn v1.3.2 - v1.3.3
• Longhorn v1.4.0 - v1.4.2
• Longhorn v1.5.0

Potentially mitigated in:

• Longhorn v1.4.3
• Longhorn v1.5.1

Complete fix planned in:

• Longhorn v1.4.x
• Longhorn v1.5.x
• Longhorn v1.6.0

Symptoms …

Applicable versions

All Longhorn versions.

Kubernetes versions before v1.28. A backported PR to v1.27 is awaiting merging.

Symptoms

In the event of a worker node failure, while hosting active Pods, the Pods are gracefully evicted as the node …

Applicable versions

All Longhorn versions.

Likely environment

Commonly seen in Fedora downstream distributions (e.g. Fedora, RHEL, Rocky, CentOS, etc.) when container-selinux is updated beyond version 2.189.0. This can happen unexpectedly and can …

Applicable versions

All Longhorn versions.

Symptoms

All volumes stuck in Attach/Detach loop. By using dmesg on storage nodes you can see errors like the following:

[Sat Dec 10 18:52:01 2022] audit: type=1400 audit(1670698321.515:7214): avc:  denied …

Applicable versions

All Longhorn versions.

Symptoms

PersistentVolumeClaim is stuck in the Pending state when restoring Longhorn with Velero with Velero CSI Plugin version before v0.4.0.

Reason

For Longhorn versions using …

Applicable versions

• All Longhorn v1.1.x versions
• Longhorn v1.2.x that are < v1.2.6
• Longhorn v1.3.x that are < v1.3.2

Symptoms

After a replica rebuilding, filesystem inside Longhorn volume is corrupted and cannot be mounted automatically. The …

Applicable versions

Longhorn v1.2.x and v1.3.x versions

Scenario

In the past, you had a Rancher v2.5.x instance running. This Rancher instance was managing a few downstream clusters. In one of the downstream clusters, you installed Longhorn using the …

Applicable versions

All Longhorn versions.

Prerequisite

Longhorn supports only NFS versions 4.0, 4.1, and 4.2. Users should ensure that both their backup server and client support one of the supported versions.

How to determine which NFS version is …

Applicable versions

All Longhorn versions.

Symptoms

User has a pod that uses a PVC with volumeMode: Block provisioned by Longhorn CSI driver. After an unexpected crash of the Longhorn volume (due to network, CPU pressure, hardware problem, …

Applicable versions

All Longhorn versions.

Background

Let’s say you have a cluster of 5 nodes (node-1, node-2, …, node-5). You have some fast disks on node-1, node-2, and node-3 so you want Longhorn to use storage on those nodes only. …

Applicable versions

Longhorn >= v0.8.0.

Symptoms

Some old instance manager pods are still running after upgrade.

Details

This behavior is an expected behavior rather than a bug. In the following paragraphs, we will explain why.

Let us first take a …

Applicable versions

All Longhorn versions.

Symptoms

Volume cannot be cleaned up after the node of the workload pod is down and recovered.

Solution

The root cause is a race condition in the pod cleanup process of Kubernetes. It is fixed since …

Applicable versions

All Longhorn versions.

Symptoms

The longhorn-driver-deployer or longhorn-csi-plugin or longhorn-ui Pods unable to access the longhorn manager backend http://longhorn-backend:9500/v1.

Reason

The CoreDNS of the Kubernetes cluster is …

Applicable versions

Longhorn >= v1.1.2.

Symptoms

Not able to investigate the longhorn-manager performance bottlenecks from the external state of the longhorn processes.

Solution

To invesigate the longhorn-manager performance bottlenecks, the …

Applicable versions

All Longhorn versions.

Symptoms

The pod using a longhorn volume with an ext4 filesystem stays in container Creating with errors in the log.

  Warning  FailedMount             30s (x7 over 63s)  kubelet …

Applicable versions

All Longhorn versions.

Symptoms

When the Kubernetes cluster is using a non-standard Kubelet directory, longhorn-csi-plugin is unable to start.

ip-172-30-0-73:/home/ec2-user # kubectl -n longhorn-system get pod
NAME …

Applicable versions

Longhorn version < v1.3.0.

Symptoms

• When upgrading Longhorn system via helm or Rancher App, the modified Longhorn default settings doesn’t persist.

• When modifying the default settings via kubectl -n longhorn-system …

Applicable versions

All Longhorn versions with Traefik 2.x as ingress controller.

Note that, the CORS problem would not happen with Traefik 1.x as ingress controller.

Symptoms

The Longhorn GUI frontend API requests sometimes failed to reach …

Applicable versions

All Longhorn versions.

Symptoms

Not able to create the support bundle with a web browser.

Solution

1. Expose Longhorn backend service. Below is an example using NodePort, you can also export via load balancer if one is set up. …

Applicable versions

Longhorn versions = v1.1.0

Symptoms

When Pod mounts with RWX volume, the Pod share mount directory and all of the ownership of its recurring contents are shown as nobody, but in the share-manager is shown as root. …

Applicable versions

All Longhorn versions.

Symptoms

The pod with the volume is not starting and encounters error messages in longhorn-csi-plugin:

time="2020-04-16T08:49:27Z" level=info msg="GRPC request: …

Applicable versions

Existing Longhorn versions < v1.1.0 upgrade to Longhorn >= v1.1.0.

Symptoms

After upgrading Longhorn to version >= v1.1.0, the following is encountered:

Ingress messages: …

Applicable versions

All Longhorn versions.

Symptoms

When starting a workload pod that uses Longhorn volumes, the Longhorn UI shows that the Longhorn volumes are attached quickly, but it takes a long time for the volumes to finish mounting and for the …

Applicable versions

All Longhorn versions.

Symptoms

When an application writes data to existing files or creates files in the mount point of a Longhorn volume, the following message is shown:

/ # cd data
/data # echo test > test
sh: can't …

Applicable versions

All Longhorn versions.

Symptoms

When creating a Pod with Longhorn Volume as PVC, the Pod cannot start.

When checking for error message using kubectl describe <pod>, the following message is shown:

Warning  FailedAttachVolume …