Regression #13895
open
pfSense Plus 23.01 can fail to boot on Hyper-V
0%
Description
After installing pfSense+ 23.01 with ZFS on a HyperV Gen2 VM in Windows 11, pfSense does not boot. The console shows it stops after printing the EFI framebuffer information. See:
https://forum.netgate.com/topic/177220/
- Works: pfSense-CE-2.7.0-DEVELOPMENT-amd64-20230123-0600.iso
- Works: pfSense-plus-22.05-RELEASE-amd64.iso
- Works: FreeBSD-14.0-CURRENT-amd64-20221209-b1ef176eb528-259626-disc1.iso
- Broken (UFS works): pfSense-plus-23.01-RC-amd64-20230123-0600.iso
- Create a new VM with 4GB of Dynamic Memory, default switch, 8GB drive, and select an ISO for the boot image.
- Edit the VM and uncheck Hardware / Security / Secure Boot // Enable Secure Boot.
- Install using default options (ZFS).
- Edit the VM and select Hardware / SCSI Controller / DVD Drive // Media:
None. - Connect to the VM console and start the VM.
Files
Updated by Steve Wheeler about 2 months ago
VMs in Azure are not affected:
System Hyper-V Virtual Machine Netgate Device ID: XXXXXXXXXXXXXXXXXXXXX BIOS Vendor: Microsoft Corporation Version: Hyper-V UEFI Release v4.1 Release Date: Mon May 9 2022 Version 23.01-RC (amd64) built on Mon Jan 23 06:05:21 UTC 2023 FreeBSD 14.0-CURRENT
Updated by Jim Pingle about 2 months ago
I can't reproduce this at all. 23.01 clean ZFS installs and upgrades boot fine here. Windows 10 22H2 and Windows 11 Pro 22H2.
I can warm boot, cold boot, etc. It comes up perfectly every time.
Setup is configured following the recipe at https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-hyper-v.html
There must be something else in the environment contributing to the problem.
Updated by Marcos M about 2 months ago
- Description updated (diff)
Tested 22.05 as well and that worked - updated description. It may be related to https://redmine.pfsense.org/issues/12707
I tried it again using UFS instead of ZFS and it booted correctly.
Updated by Lev Prokofev about 2 months ago
Don't see any issues with pfSense-plus-23.01-BETA-amd64-20230106-0600 on Win11pro 22H2 nor ZFS or UFS. Hyper-V is all default out of the box.
Updated by Marcos M about 2 months ago
- Description updated (diff)
Changing the loader menu option 5. Cons: did not help. However, removing the DVD drive from the VM allows it to boot while connected to the VM console.
Updated by Marcos M 19 days ago
- File vm.png vm.png added
- Subject changed from pfSense Plus 23.01 does not boot on HyperV from Windows Server 2022 or Windows 11 to pfSense Plus 23.01 can fail to boot on Hyper-V
This can be replicated on Azure as well. Deploy a new pfSense+ 23.01 Gen 2 VM on Azure with a size which allows adding more than two NICs (e.g. Standard D8as v4). Booting works correctly when adding the second NIC, however the VM fails to boot after adding a third NIC. See attached image. Removing the third NIC allows it to boot correctly again.
Updated by Chris Poillion 11 days ago
- File screenshot.png screenshot.png added
Here is a screenshot of the memmap command on an affect VM. This machine is on Windows Server 2022. The Pfsense VM is 2GB ram, 40GB ZFS, 2 NICS.
I have a Pfsense VM running 2.6, 8GB Ram, 40GB ZFS and 2 Nics that I tried to update to 23.01 but it also would not boot on 23.01. The VM that the memmap screenshot is from was a fresh install using the 2.6 iso, so same upgrade path.
Updated by Leon Dang 10 days ago
Thanks Chris.
While this doesn't solve your immediate situation, it looks like repro is possible with Windows Server 2022. We haven't exactly root caused why, but there seems to be a problem in 23.01 involving the loader's request to the Hyper-V UEFI firmware to allocate pages, which it uses to put the jump-to-kernel code and page-tables right before switching over to the kernel. If the allocation returns pages that sit right before the LoaderData segment, they get corrupted (probably soon after entering the kernel) and the system either hangs or reboots.