# LXD 6.10.0 release notes This is a [LTS release](https://canonical.com/lxd/docs/latest/reference/releases-snap/index.html.md#ref-releases-lts) and is recommended for production use. This is the first LTS release for the 6 series. It consolidates new features, storage and networking advancements, disaster recovery capabilities, security hardening, and bug fixes from across the 6 series. #### NOTE Release notes content These release notes cover updates in the [core LXD repository](https://github.com/canonical/lxd) and the [LXD snap package](https://snapcraft.io/lxd). For a tour of [LXD UI](https://github.com/canonical/lxd-ui) updates, please see the release announcement in [our Discourse forum](https://discourse.ubuntu.com/t/lxd-6-10-0-lts-has-been-released/89062). ## Highlights This section highlights new and improved features in this release. ### Image registries Image registries provide a unified view of image sources available to a LXD cluster. Clients now specify an image registry when requesting the LXD daemon to download an image. Projects can restrict image downloads to specific authorized image registries using the new `restricted.registries` project configuration key. This allows an administrator to specify the image sources that tenants can access. Image registries support using another LXD cluster as an image source by referencing a [cluster link](https://canonical.com/lxd/docs/latest/explanation/clusters/index.html.md#exp-cluster-links). Simple Streams image servers are also supported. Client-side image remotes remain supported, but clients should specify an image registry rather than a URL and protocol when requesting an image download. A comprehensive compatibility layer was added to support older LXD clients with the transition. - Documentation: [Image registries](https://canonical.com/lxd/docs/latest/reference/image_registries/index.html.md#ref-image-registries) and [How to manage image registries](https://canonical.com/lxd/docs/latest/howto/image_registries/index.html.md#howto-image-registries) - API extension: [image_registries](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-image-registries) ### Persistent changed block tracking for VM block volumes #### WARNING Changed block tracking is a feature preview and **must not** be enabled in a production environment. See [Configure feature previews](https://canonical.com/lxd/docs/latest/howto/snap/index.html.md#howto-snap-configure-feature-previews) for more details. Virtual machine block volumes now support persistent changed block tracking (CBT). LXD leverages QEMU dirty bitmaps and NBD export interfaces to track disk blocks modified between snapshots. External backup tools and migration pipelines can query and download only the altered blocks rather than whole disk images, significantly reducing backup windows, bandwidth usage, and storage overhead. - Documentation: [How to track changed blocks on virtual machine volumes](https://canonical.com/lxd/docs/latest/howto/storage_block_tracking/index.html.md#howto-storage-block-tracking) - API extension: [storage_volume_block_tracking](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-storage-volume-block-tracking) ### Ceph RBD mirroring for project volume replication Disaster recovery replicators now support asynchronous Ceph RBD storage mirroring. When replicating a project’s instances and exclusively attached volumes using Ceph RBD storage pools, LXD coordinates with Ceph’s native volume mirroring capabilities instead of performing userspace block transfers. Standby clusters receive mirrored image updates natively, and LXD manages all volume promotion, demotion, and leader failover operations. - Documentation: [Replicators](https://canonical.com/lxd/docs/latest/explanation/disaster_recovery/index.html.md#exp-replicators) - API extension: [storage_ceph_replicator](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-storage-ceph-replicator) ### Custom volume support in replicators Disaster recovery replication now supports custom storage volumes with the introduction of the `all-exclusive` disk volumes mode for instance migration. When migrating or replicating instances, any custom storage volumes attached exclusively to the instance, along with their volume snapshots, are automatically transferred in the same operation. Volumes attached to multiple instances or shared across the project are not replicated. - Documentation: [Replicators](https://canonical.com/lxd/docs/latest/explanation/disaster_recovery/index.html.md#exp-replicators) - API extension: [replicator_custom_volumes](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-replicator-custom-volumes) ### Disaster recovery and replicator metrics Dedicated Prometheus metrics and recovery point objective (RPO) tracking have been added for disaster recovery replicators. LXD now exposes gauges for `lxd_replicators`, `lxd_replicator_last_run_status`, `lxd_replicator_last_success_timestamp`, and `lxd_replicator_last_success_oldest_snapshot_timestamp`. These metrics provide real-time visibility into replication health and the age of the oldest replicated snapshot. - Documentation: [Provided metrics](https://canonical.com/lxd/docs/latest/reference/provided_metrics/index.html.md#provided-metrics) - API extension: [metrics_replicators](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-metrics-replicators) ### Unidirectional cluster links The cluster links API now supports unidirectional relationships. A local cluster can establish an authenticated link to a remote cluster by consuming an authorization trust token issued by the remote cluster. The remote cluster validates and authenticates incoming requests from the local cluster while preventing outbound connections back to the local cluster, accommodating network topologies with strict firewall or one-way ingress rules. - Documentation: [Cluster links](https://canonical.com/lxd/docs/latest/explanation/clusters/index.html.md#exp-cluster-links) - API extension: [cluster_links_unidirectional](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-cluster-links-unidirectional) ### Public cluster links Cluster links can now connect to public or read-only remote clusters without client certificate authentication. Public cluster links verify the remote endpoint using TLS certificate fingerprint pinning during a two-phase creation workflow. This enables access to public services and image registries without creating reciprocal identities or exchanging mutual credentials. - Documentation: [Cluster links](https://canonical.com/lxd/docs/latest/explanation/clusters/index.html.md#exp-cluster-links) - API extension: [cluster_links_public](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-cluster-links-public) ### OVN load balancer pool health checks Health checking support has been introduced for OVN load balancer backend pools. Administrators can configure active health checks using the [`healthcheck`](https://canonical.com/lxd/docs/latest/howto/network_load_balancers/index.html.md#network-load-balancer-pool-properties:healthcheck) settings on the pool, specifying interval, timeout, and consecutive success/failure thresholds. The health state of backend instances can be inspected via the new load balancer pool state API. - Documentation: [How to configure network load balancers](https://canonical.com/lxd/docs/latest/howto/network_load_balancers/index.html.md#network-load-balancers) - API extension: [network_load_balancer_pool_health_checks](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-network-load-balancer-pool-health-checks) ### Pure Storage FlashArray Fibre Channel support The Pure Storage FlashArray storage driver has expanded enterprise SAN connectivity options by adding Fibre Channel transport support. Storage pools can now be configured using SCSI over Fibre Channel (`pure.mode=scsi/fc`) or NVMe over Fibre Channel (`pure.mode=nvme/fc`), complete with automatic initiator WWPN discovery and volume registration. - Documentation: [Pure Storage - pure](https://canonical.com/lxd/docs/latest/reference/storage_pure/index.html.md#storage-pure) - API extensions: [storage_driver_pure_scsifc](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-storage-driver-pure-scsifc) and [storage_driver_pure_nvmefc](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-storage-driver-pure-nvmefc) ### Dell PowerStore NVMe support The Dell PowerStore storage driver now supports NVMe over TCP (`nvme/tcp`) and NVMe over Fibre Channel (`nvme/fc`) transport modes. NVMe/TCP is now configured as the default connectivity mode when creating new PowerStore storage pools without an explicit `powerstore.mode` setting. - Documentation: [Dell PowerStore - powerstore](https://canonical.com/lxd/docs/latest/reference/storage_powerstore/index.html.md#storage-powerstore) - API extension: [storage_driver_powerstore_nvme](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-storage-driver-powerstore-nvme) ### Identity effective groups and state endpoint A new identity state API endpoint and `recursion=2` listing mode have been introduced to provide visibility into effective group memberships. The endpoint computes and returns the `effective_groups` for an identity, reporting the union of direct authorization groups and mapped OIDC identity provider groups. - Documentation: [Remote API authentication](https://canonical.com/lxd/docs/latest/authentication/index.html.md#authentication) - API extension: [access_management_identity_effective_groups](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-access-management-identity-effective-groups) ### Credential expiry reporting The `Identity` API struct now includes an `expires_at` field across all identity listing and inspection endpoints. This exposes the expiration date of client TLS certificates and issued bearer tokens, enabling proactive rotation and monitoring of client credentials before expiry. - Documentation: [Remote API authentication](https://canonical.com/lxd/docs/latest/authentication/index.html.md#authentication) - API extension: [access_management_expiry](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-access-management-expiry) ### Pending states for bearer identities LXD introduces pending identity types (`Client token bearer (pending)`, `DevLXD token bearer (pending)`, and `Initial UI token bearer (pending)`). Bearer identities are created in a pending state until a valid token is issued, and automatically revert to the pending type if their token is revoked, providing clear lifecycle tracking for bearer access. - Documentation: [Remote API authentication](https://canonical.com/lxd/docs/latest/authentication/index.html.md#authentication) - API extension: [access_management_bearer_pending](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-access-management-bearer-pending) ### Durable operations A new durable operation class has been introduced for mission-critical daemon workflows. Durable operations persist their state in the cluster database. If the cluster member executing the operation fails or goes offline, the operation is automatically resumed on the DQLite raft leader once heartbeat timeouts elapse. - Documentation: [REST API](https://canonical.com/lxd/docs/latest/rest-api/index.html.md#rest-api) - API extension: [durable_operations](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-durable-operations) ### Dedicated server state endpoint A dedicated `GET /1.0/state` endpoint has been added to retrieve server status. This API endpoint is only available when LXD is in standalone mode. When clustered, the `GET /1.0/cluster/members/{name}/state` endpoint should be called. This allows clients and monitoring tools to quickly inspect system and storage pool state. - Documentation: [REST API](https://canonical.com/lxd/docs/latest/rest-api/index.html.md#rest-api) - API extension: [server_state](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-server-state) ### CDI passthrough for NVIDIA MIG GPUs GPU passthrough for `gputype=mig` devices has transitioned to the Container Device Interface (CDI) instead of legacy `nvidia.runtime` mechanisms. Administrators can supply CDI identifiers directly in the device’s `id` property or continue specifying `mig.uuid` or `mig.gi`/`mig.ci` pairs, which LXD resolves automatically to CDI devices via NVML. - Documentation: [Type: gpu](https://canonical.com/lxd/docs/latest/reference/devices_gpu/index.html.md#devices-gpu) - API extension: [gpu_mig_cdi](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-gpu-mig-cdi) ### Operation child count and child listing The `Operation` struct now includes a `child_count` field, allowing callers to determine whether an operation has spawned child operations without issuing recursive queries. In addition, the `lxc operation` command now includes a `list-children` subcommand. - Documentation: [REST API](https://canonical.com/lxd/docs/latest/rest-api/index.html.md#rest-api) - API extension: [operation_child_count](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-operation-child-count) ### Operation wait status code reporting The operation wait endpoints now return the failed operation state and error status code upon failure. This ensures callers and DevLXD clients receive immediate and accurate error diagnostics when long-running background tasks terminate with an error. - Documentation: [REST API](https://canonical.com/lxd/docs/latest/rest-api/index.html.md#rest-api) - API extension: [operation_wait_status_code](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-operation-wait-status-code) ### VM volatile max vCPUs tracking Virtual machines now track maximum configured vCPUs in volatile configuration keys. This preserves CPU hotplug boundaries across instance life cycles and enables live migration of virtual machines between cluster members with differing CPU core counts. - Documentation: [Instances](https://canonical.com/lxd/docs/latest/instances/index.html.md#instances) - API extension: [vm_volatile_maxcpus](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-vm-volatile-maxcpus) ### Loki configuration readiness check LXD now optionally skips Loki API endpoint readiness checks before attempting to transmit log streams. This improves compatibility with OpenTelemetry (OTLP) and Canonical Observability Stack (COS) deployments. - Documentation: [Provided metrics](https://canonical.com/lxd/docs/latest/reference/provided_metrics/index.html.md#provided-metrics) - API extension: [loki_config_api_check_ready](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-loki-config-api-check-ready) ### Optional project replica mode The `replica_mode` attribute is now omitted for projects that do not participate in disaster recovery replication topologies. A new `lxc project clear-replica` command has also been added to clear replica state and promote/demote guard rails from projects. - Documentation: [Projects](https://canonical.com/lxd/docs/latest/projects/index.html.md#projects) - API extension: [project_replica_mode_optional](https://canonical.com/lxd/docs/latest/api-extensions/index.html.md#extension-project-replica-mode-optional) ## Bug fixes The following bug fixes are included in this release. - [Fix infinite loop in network error log writer](https://github.com/canonical/lxd/pull/18577) - [Fix physical network parent sharing across VLANs in clustered mode](https://github.com/canonical/lxd/pull/18608) - [Improve duplicate cluster link error message](https://github.com/canonical/lxd/pull/18626) - [Enforce project restrictions during instance move](https://github.com/canonical/lxd/pull/18605) - [Fix failed file operation due to deleted forkfile as a result of race condition](https://github.com/canonical/lxd/pull/18654) - [Fix bcache device handling in system resources](https://github.com/canonical/lxd/pull/18695) - [Fix unprotected concurrent write to operation metadata](https://github.com/canonical/lxd/pull/18694) - [Acquire lock before reading read-only operation flag in lxd-agent](https://github.com/canonical/lxd/pull/18698) - [Fix snapshot table layout in lxc info when no volume snapshots exist](https://github.com/canonical/lxd/pull/18723) - [Fix booting Windows in BIOS boot mode](https://github.com/canonical/lxd/pull/18732) - [Make powerstore.mode optional to prevent per-target error](https://github.com/canonical/lxd/pull/18738) - [Fix SRIOV VF used count and total disk allocated in lxc info](https://github.com/canonical/lxd/pull/18742) - [Prevent hang on VM filesystem attach with idmap set](https://github.com/canonical/lxd/pull/18733) - [Fix Ceph cluster_name handling for QEMU disks](https://github.com/canonical/lxd/pull/18743) - [Properly validate disk device propagation option](https://github.com/canonical/lxd/pull/18744) - [Globally enforce that caller can view requested project](https://github.com/canonical/lxd/pull/18440) - [Detect download cancellation with errors.Is in simplestreams client](https://github.com/canonical/lxd/pull/18779) - [Fetch network state when filtering instances by IP address in lxc list](https://github.com/canonical/lxd/pull/18765) - [Limit image download to trusted size](https://github.com/canonical/lxd/pull/18778) - [Use numeric IPv6 zone for NDP neighbour probe](https://github.com/canonical/lxd/pull/18796) - [Include server operations when listing operations in default project](https://github.com/canonical/lxd/pull/18806) - [Prevent busy ZFS dataset from blocking rebuild](https://github.com/canonical/lxd/pull/18807) - [Enhance subprocess handling to prevent PID reuse issues](https://github.com/canonical/lxd/pull/18872) - [Fix STARTTLS peek blocking the accept loop in endpoint listeners](https://github.com/canonical/lxd/pull/18819) - [Fix double-close in idmap shifting](https://github.com/canonical/lxd/pull/18927) - [Fix LXCFS clean up logic and avoid generating new certificates on running instances](https://github.com/canonical/lxd/pull/18894) - [Sanitize server-provided image file names in client](https://github.com/canonical/lxd/pull/18940) - [Fix file descriptor cleanup race condition in unit tests](https://github.com/canonical/lxd/pull/18969) - [Fix Accept cancellation race in endpoint listeners](https://github.com/canonical/lxd/pull/18904) - [Allow volume security.shifted and directory shift when raw.idmap is set on VM](https://github.com/canonical/lxd/pull/18918) - [Preserve protected configuration keys during replicator refresh](https://github.com/canonical/lxd/pull/18938) - [Restrict image reuse to cached images with identical image source](https://github.com/canonical/lxd/pull/18987) - [Fix stacked tmpfs mounts for shared mounts and migrate legacy devlxd path](https://github.com/canonical/lxd/pull/18854) - [Run cluster link address refresh on standalone LXD servers](https://github.com/canonical/lxd/pull/19008) - [Use member address for cluster link bootstrap](https://github.com/canonical/lxd/pull/19018) - [Apply source file mode on target when pulling files via CLI](https://github.com/canonical/lxd/pull/19068) - [Keep target volatile configuration keys on migration refresh](https://github.com/canonical/lxd/pull/19082) - [Fix VM live migration between cluster members with different CPU counts](https://github.com/canonical/lxd/pull/18880) - [Refuse to delete or rename a cluster link referenced by a project replica.cluster](https://github.com/canonical/lxd/pull/19112) - [Fix handling of cluster joins during heartbeat rounds](https://github.com/canonical/lxd/pull/19153) - [Rename VM filesystem volume during snapshot rename on LVM](https://github.com/canonical/lxd/pull/19154) - [Distinguish unassigned LUN from LUN 0 in Pure Storage driver](https://github.com/canonical/lxd/pull/19067) - [Fix image update and btrfs cleanup](https://github.com/canonical/lxd/pull/19170) - [Enforce viewership when granting authorization entitlements](https://github.com/canonical/lxd/pull/18934) - [Accept null properties field in image upload tokens](https://github.com/canonical/lxd/pull/19185) - [Wait for snapshot device to disappear on unmount in ZFS](https://github.com/canonical/lxd/pull/19172) - [Prevent refused replicator refresh from leaving source configuration on target](https://github.com/canonical/lxd/pull/19130) - [Promote and demote mirrored project volumes on failover](https://github.com/canonical/lxd/pull/18941) - [Guard standby project storage on mirrored Ceph pool](https://github.com/canonical/lxd/pull/18929) - [Avoid session key races in HPE Alletra storage client](https://github.com/canonical/lxd/pull/19181) - [Preserve all volume fields in storage driver Clone](https://github.com/canonical/lxd/pull/19180) - [Associate advertised BGP prefixes with their owning peer session](https://github.com/canonical/lxd/pull/19187) - [Keep cross-node operation websockets alive](https://github.com/canonical/lxd/pull/19041) - [Use stored bytes for Ceph storage pool usage calculation](https://github.com/canonical/lxd/pull/19207) - [Make cluster member restore conflict with ongoing evacuations](https://github.com/canonical/lxd/pull/19178) - [Fix backup handling and corruption safeguards](https://github.com/canonical/lxd/pull/19175) - [Increase and pool migration and websocket transfer buffers](https://github.com/canonical/lxd/pull/19208) ## Backwards-incompatible changes These changes are not compatible with older versions of LXD or its clients. ### NVIDIA legacy instance-level configuration removed The legacy instance-level NVIDIA configuration options (`nvidia.runtime`, `nvidia.driver.capabilities`, `nvidia.require.cuda`, and `nvidia.require.driver`) have been removed. In addition, the `nvidia_runtime` and `nvidia_runtime_config` API extensions have been removed. Existing instances, instance snapshots, and profiles will have these legacy options cleaned up automatically on daemon upgrade. NVIDIA GPU passthrough should now be configured using the Container Device Interface (CDI) passthrough. - Documentation: [Type: gpu](https://canonical.com/lxd/docs/latest/reference/devices_gpu/index.html.md#devices-gpu) ### Cluster healing functionality removed The cluster healing feature and its server configuration setting `cluster.healing_threshold` have been removed. In addition, the `cluster_healing` API extension has been removed. Automatic evacuation and eviction of unresponsive cluster members has been discontinued to avoid accidental quorum loss during transient network partitions. ## Known issues This section covers known temporary limitations and integration regressions in this release. ### CDI GPU passthrough failure on Ubuntu Core 26 Users attempting to pass through GPUs to containers on Ubuntu Core 26 environments using the `gpu-2604` interface (provided by the `mesa-2604` snap) will encounter a container startup failure: ```default Error: Failed starting device "gpu0": Failed generating CDI spec: Failed determining NVIDIA driver root path: Failed running: /snap/lxd//gpu-2604/bin/gpu-2604-provider-wrapper printenv NVIDIA_DRIVER_ROOT: exit status 1 ``` This is caused by an upstream architectural mismatch on the Core 26 track between the `pc-kernel` snap and the `mesa-2604` graphics provider snap. There is currently no native LXD configuration workaround. We are working with our partners to resolve the issue. ### Ceph userspace tooling incompatibility LXD bundles Ceph userspace tooling to manage `ceph`, `cephfs`, and `cephobject` storage pools, volumes, and buckets. The version of this tooling is incompatible with the most recent Ceph OSD release. New deployments should use a version of MicroCeph that packages a Ceph version before 20.2.4 (tentacle) or 19.2.6 (squid). Existing deployments that upgrade MicroCeph to a version that contains the newer Ceph must not rotate keys until the newer key type is compatible with the client shipped by LXD. ## Updated minimum Go version If you are building LXD from source instead of using a package manager, the minimum version of Go required to build LXD is now 1.27.1. ## Snap packaging changes - Shipped `qemu-nbd` and `qemu-storage-daemon` binaries to support persistent changed block tracking. - Shipped `virtiofsd` on all architectures with virtual machine support (enabling non-amd64 architectures, except armhf). - QEMU bumped to version 10.2.1+ds-1ubuntu3.2. - EDK2 firmware bumped to 2025.11-3ubuntu7.2, shipping both Secure Boot and non-Secure Boot firmware variants. - Reintroduced the KVM vAPIC option ROM for 32-bit BIOS guests in QEMU. - Bumped ZFS to versions 2.2.11, 2.3.9, and 2.4.4 across respective branches. - Bumped NVIDIA container toolkit and libnvidia-container to v1.20.1. - Bumped LXC and LXCFS to v7.0.0 LTS. - Enabled building LXD UI and documentation on riscv64. ## Change log View the [complete list of all changes in this release](https://github.com/canonical/lxd/compare/lxd-6.9...lxd-6.10.0). ## Downloads The source tarballs and binary clients can be found on our [download page](https://github.com/canonical/lxd/releases/tag/lxd-6.10.0). Binary packages are also available for: - **Linux:** `snap install lxd --channel=6/stable` - **macOS client:** `brew install lxc` - **Windows client:** `choco install lxc`