Skip to content

Upgrading Proxmox 8 to 9

There are two ways to upgrade a Proxmox VE 8.x system to Proxmox VE 9.x:

  • A new installation on new hardware (restoring VMs from the backup)
  • An in-place upgrade via apt (step-by-step)

We are going to look at in-place upgrades which are carried out via CLI using apt and ceph commands.

Prerequisites

  • Upgraded to the latest version of Proxmox VE 8.4 on all nodes.
  • Reliable access to the node. It's recommended to have access over a host independent channel like IKVM/IPMI or physical access.

Warning

If only SSH is available we recommend testing the upgrade on an identical, but non-production machine first. It is also highly recommended to use a terminal multiplexer (for example, tmux or screen) to ensure the upgrade can continue even if the SSH connection gets interrupted.

  • A healthy Proxmox cluster
  • Valid and tested backup of all VMs and CTs (in case something goes wrong)
  • At least 5 GB free disk space on the root mount point, ideally more than 10 GB.

Upgrade Ceph Reef to Squid (Upgrade Time: 15 minutes)

If using Hyper-converged Ceph: upgrade any Ceph cluster to Ceph 19.2 Squid before you start the Proxmox VE upgrade to 9.0. If you are not using Ceph, you can skip directly to Proxmox VE Upgrade.

This section explains how to upgrade Ceph from Reef (18.2+) to Squid (19.2+) on Proxmox VE 8.

Info

Note: All commands starting with ceph need to be run only once. It doesn't matter on which node in the Ceph cluster.

Info

This upgrade requires no downtime and can be performed while guests are running on the nodes.

Preparation on each Ceph cluster node

Change the current Ceph repositories from Reef to Squid.

Bash
sed -i 's/reef/squid/' /etc/apt/sources.list.d/ceph.list

Set the 'noout' flag

Set the noout flag for the duration of the upgrade (optional, but recommended):

Bash
ceph osd set noout

Upgrade on each Ceph cluster node

Upgrade all your nodes with the following commands or by installing the latest updates via the GUI. It will upgrade the Ceph on your node to Squid.

Bash
apt update
apt full-upgrade

Restart the monitor daemon

After upgrading all cluster nodes, you have to restart the monitor on each node where a monitor runs.

Bash
systemctl restart ceph-mon.target
Do so one node at a time. Wait after each restart and periodically check the status of the cluster:
Bash
ceph -s
It should be showing:

Command Output

HEALTH_WARN
noout flag(s) set

Once all monitors are up, verify that the monitor upgrade is complete. Look for the Squid string in the mon map. The command

Bash
ceph mon dump | grep min_mon_release
should report

Command Output

min_mon_release 19 (squid)

If it does not, this implies that one or more monitors haven’t been upgraded and restarted, and/or that the quorum doesn't include all monitors.

Restart the manager daemons on all nodes

If the managers did not automatically restart with the monitors, restart them now on all nodes

Bash
systemctl restart ceph-mgr.target
Verify that the ceph-mgr daemons are running by checking ceph -s
Bash
ceph -s

Command Output
Text Only
...
    health: HEALTH_WARN
            noout flag(s) set

services:
    mon: 3 daemons, quorum nv1,nv2,nv3 (age 2m)
    mgr: nv2(active, since 22s), standbys: nv1, nv3
    mds: 1/1 daemons up, 2 standby
    osd: 18 osds: 18 up (since 4d), 18 in (since 4d)
        flags noout
...

Restart the OSD daemon on all nodes

Restart all OSDs. Only restart OSDs on one node at a time to avoid loss of data redundancy. To restart all OSDs on a node, run the following command:

Bash
systemctl restart ceph-osd.target
Wait after each restart and periodically check the status of the cluster:
Bash
ceph status
It should be showing:

Command Output

HEALTH_WARN
noout flag(s) set

Once all OSDs are running with the latest versions, the following warning can appear:

Command Output

all OSDs are running squid or later but require_osd_release < squid

Disallow pre-Squid OSDs and enable all new Squid-only functionality

Before raising the minimum required OSD version, you should ensure all OSDs got upgraded successfully and report running a Ceph 19.2 version.

Bash
ceph osd require-osd-release squid

Upgrade all CephFS MDS daemons

For each CephFS file system, (you can list the file systems with ceph fs ls)

  1. The number of active MDS should go down to the number of file systems you have. Alternatively, check in the CephFS panel in the GUI that each Ceph filesystem has only one active MDS.
  2. Take all standby MDS daemons offline on the appropriate hosts with:
    Bash
    systemctl stop ceph-mds.target
  3. Confirm that only one MDS is online and is on rank 0 for your FS:
    Bash
    ceph status
  4. Upgrade the last remaining MDS daemon by restarting the daemon:
    Bash
    systemctl restart ceph-mds.target
  5. Restart all standby MDS daemons that were taken offline:
    Bash
    systemctl start ceph-mds.target
  6. This should upgrade all Metadata Servers

Unset the 'noout' flag

Once the upgrade process is finished, don't forget to unset the noout flag.

Bash
ceph osd unset noout

Proxmox VE Upgrade (Upgrade Time: 60 minutes)

The following actions need to be carried out from the command line of each Proxmox VE node in your cluster

Info

Perform the actions via console or ssh; preferably via console to avoid interrupted ssh connections. Do not carry out the upgrade when connected via the virtual console offered by the GUI; as this will get interrupted during the upgrade. If only SSH is available consider using a terminal multiplexer (such as tmux or screen) to avoid issues if the SSH connection gets interrupted.

Remember to ensure that a valid backup of all VMs and CTs has been created before proceeding.

Set the 'noout' flag

Set the noout flag for the duration of the upgrade (optional, but recommended):

Bash
ceph osd set noout

Move important Virtual Machines and Containers

If any VMs and CTs need to keep running for the duration of the upgrade, migrate them away from the node that is being upgraded.

Migration compatibility rules to keep in mind when planning your cluster upgrade:

  • A migration of a VM or CT from an older version of Proxmox VE to a newer version will always work.
  • A migration from a newer Proxmox VE version to an older version may work, but is generally not supported.

Note

In a partially-upgraded cluster, minor inconsistencies may occur if you are logged into the GUI on a node that is already upgraded to Proxmox VE 9, and perform actions (such as guest migrations) on a node that is still running Proxmox VE 8. If you see any errors or warnings while migrating guests away from a node that is still running Proxmox VE 8, log into the GUI on that node and retry the migration.

Test Proxmox 8 to 9 Upgrade

A small checklist program named pve8to9 is included in the latest Proxmox VE 8.4 packages. The program will provide hints and warnings about potential issues before, during and after the upgrade process. You can call it by executing:

Bash
pve8to9 --full
Make sure to run the full checks at least once before the upgrade.

This script only checks and reports things. By default, no changes to the system are made and thus, none of the issues will be automatically fixed. You should keep in mind that Proxmox VE can be heavily customized, so the script may not recognize all the possible problems with a particular setup!

It is recommended to re-run the script after each attempt to fix an issue. This ensures that the actions taken actually fixed the respective warning.

Fixing any warnings

For example here is a sample warning message where intel-microcode is not being found:

Command Output

Text Only
WARN: The matching CPU microcode package 'intel-microcode' could not be found! Consider installing it to receive the latest security and bug fixes for your CPU.
        Ensure you enable the 'non-free-firmware' component in the apt sources and run:
        apt install intel-microcode

= SUMMARY =

TOTAL:    54
PASSED:   48
SKIPPED:  5
WARNINGS: 1
FAILURES: 0

ATTENTION: Please check the output for detailed information!

This can be fixed by enabling non-free-firmware in /etc/apt/sources.list and installing intel-microcode

Bash
sed -i 's/main contrib/main contrib non-free-firmware/g' /etc/apt/sources.list
apt update && apt install -y intel-microcode

Command Output

Text Only
= SUMMARY =

TOTAL:    54
PASSED:   49
SKIPPED:  5
WARNINGS: 0
FAILURES: 0

Update the configured APT repositories

First, make sure that the system is using the latest Proxmox VE 8.4 packages:

Bash
apt update
apt dist-upgrade -y
pveversion

The last command should report at least 8.4.1 or newer.

Update Debian Base Repositories to Trixie

Update all Debian and Proxmox VE repository entries to Trixie.

Bash
sed -i 's/bookworm/trixie/g' /etc/apt/sources.list
sed -i 's/bookworm/trixie/g' /etc/apt/sources.list.d/pve-enterprise.list

Add the Proxmox VE 9 Package Repository

If you are using the enterprise repository, you can add the Proxmox VE 9 enterprise repository and run the following command to create the related pve-enterprise.sources file:

Bash
cat > /etc/apt/sources.list.d/pve-enterprise.sources << EOF
Types: deb
URIs: https://enterprise.proxmox.com/debian/pve
Suites: trixie
Components: pve-enterprise
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

Only if not using Proxmox Enterprise License and using no subscription
Text Only
cat > /etc/apt/sources.list.d/proxmox.sources << EOF
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

Update the Ceph Package Repository

Note

At this point a hyper-converged Ceph cluster installed directly in Proxmox VE must run Ceph 19.2 Squid, otherwise you need to upgrade Ceph first before upgrading to Proxmox VE 9 on Debian 13 Trixie!

Bash
cat > /etc/apt/sources.list.d/ceph.sources << EOF
Types: deb
URIs: https://enterprise.proxmox.com/debian/ceph-squid
Suites: trixie
Components: enterprise
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF
Only if not using Proxmox Enterprise License and using no subscription
Text Only
cat > /etc/apt/sources.list.d/ceph.sources << EOF
Types: deb
URIs: http://download.proxmox.com/debian/ceph-squid
Suites: trixie
Components: no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

Refresh Package Index

Update the repositories' package index and verify that no error is reported:

Bash
apt update

Upgrade the system to Debian Trixie and Proxmox VE 9.0

Note that the time required for finishing this step heavily depends on the system's performance, especially the root filesystem's IOPS and bandwidth. A slow spinner can take up to 60 minutes or more, while for a high-performance server with SSD storage, the dist-upgrade can be finished in under 5 minutes.

Start with this step, to get the initial set of upgraded packages:

Bash
apt dist-upgrade -y

During the above step, you will be asked to approve changes to configuration files and some service restarts, where the default config has been updated by their respective package.

You may also be shown the output of apt-listchanges, you can simply exit there by pressing "q". If you get prompted for your default keyboard selection, simply use the arrow keys to navigate to the one applicable in your case and hit enter.

For questions about service restarts (like Restart services during package upgrades without asking?) use the default if unsure, as the reboot after the upgrade will restart all services cleanly anyway.

Here are some common configuration changes where not to make the default selection:

  • /etc/lvm/lvm.conf -> Changes relevant for Proxmox VE will be updated, and a newer config version might be useful. If you did not make extra changes yourself and are unsure it's suggested to choose "Yes" (install the package maintainer's version) here.
  • Restarting libc6:amd64 service -> "Yes"

And finally reboot the node

Bash
systemctl reboot

Unset the 'noout' flag

Once the upgrade process is finished, don't forget to unset the noout flag.

Bash
ceph osd unset noout