Regression #13895
closed
Early boot hangs on Hyper-V with Gen2 VMs
Added by Marcos M over 1 year ago.
Updated 12 months ago.
Category:
Operating System
Affected Plus Version:
23.01
Description
After installing pfSense+ 23.01 with ZFS on a HyperV Gen2 VM in Windows 11, pfSense does not boot. The console shows it stops after printing the EFI framebuffer information. See:
https://forum.netgate.com/topic/177220/
Tested:
- Works: pfSense-CE-2.7.0-DEVELOPMENT-amd64-20230123-0600.iso
- Works: pfSense-plus-22.05-RELEASE-amd64.iso
- Works: FreeBSD-14.0-CURRENT-amd64-20221209-b1ef176eb528-259626-disc1.iso
- Broken (UFS works): pfSense-plus-23.01-RC-amd64-20230123-0600.iso
Steps:
- Create a new VM with 4GB of Dynamic Memory, default switch, 8GB drive, and select an ISO for the boot image.
- Edit the VM and uncheck Hardware / Security / Secure Boot // Enable Secure Boot.
- Install using default options (ZFS).
- Edit the VM and select Hardware / SCSI Controller / DVD Drive // Media:
None.
- Connect to the VM console and start the VM.
Files
- Description updated (diff)
VMs in Azure are not affected:
System Hyper-V Virtual Machine
Netgate Device ID: XXXXXXXXXXXXXXXXXXXXX
BIOS Vendor: Microsoft Corporation
Version: Hyper-V UEFI Release v4.1
Release Date: Mon May 9 2022
Version 23.01-RC (amd64)
built on Mon Jan 23 06:05:21 UTC 2023
FreeBSD 14.0-CURRENT
I can't reproduce this at all. 23.01 clean ZFS installs and upgrades boot fine here. Windows 10 22H2 and Windows 11 Pro 22H2.
I can warm boot, cold boot, etc. It comes up perfectly every time.
Setup is configured following the recipe at https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-hyper-v.html
There must be something else in the environment contributing to the problem.
- Description updated (diff)
- Description updated (diff)
Don't see any issues with pfSense-plus-23.01-BETA-amd64-20230106-0600 on Win11pro 22H2 nor ZFS or UFS. Hyper-V is all default out of the box.
- Description updated (diff)
Changing the loader menu option 5. Cons: did not help. However, removing the DVD drive from the VM allows it to boot while connected to the VM console.
- Description updated (diff)
- File vm.png vm.png added
- Subject changed from pfSense Plus 23.01 does not boot on HyperV from Windows Server 2022 or Windows 11 to pfSense Plus 23.01 can fail to boot on Hyper-V
This can be replicated on Azure as well. Deploy a new pfSense+ 23.01 Gen 2 VM on Azure with a size which allows adding more than two NICs (e.g. Standard D8as v4). Booting works correctly when adding the second NIC, however the VM fails to boot after adding a third NIC. See attached image. Removing the third NIC allows it to boot correctly again.
I can't repro it in Win-11 Hyper-V. 4GB, ZFS, 3 NICs, ISO still attached.
In the loader prompt (option 3), can you type "memmap"? I am seeing that in my setup, the conventional memory segment before LoaderData is rather big.
Here is a screenshot of the memmap command on an affect VM. This machine is on Windows Server 2022. The Pfsense VM is 2GB ram, 40GB ZFS, 2 NICS.
I have a Pfsense VM running 2.6, 8GB Ram, 40GB ZFS and 2 Nics that I tried to update to 23.01 but it also would not boot on 23.01. The VM that the memmap screenshot is from was a fresh install using the 2.6 iso, so same upgrade path.
Thanks Chris.
While this doesn't solve your immediate situation, it looks like repro is possible with Windows Server 2022. We haven't exactly root caused why, but there seems to be a problem in 23.01 involving the loader's request to the Hyper-V UEFI firmware to allocate pages, which it uses to put the jump-to-kernel code and page-tables right before switching over to the kernel. If the allocation returns pages that sit right before the LoaderData segment, they get corrupted (probably soon after entering the kernel) and the system either hangs or reboots.
- Subject changed from pfSense Plus 23.01 can fail to boot on Hyper-V to Early boot hangs on Hyper-V
- Status changed from New to Feedback
- Assignee set to Leon Dang
- Target version set to 23.05
- Affected Plus Version set to 23.01
- Has duplicate Bug #13875: Boot after upgrade form 22.05 to 23.01 hangs added
- Subject changed from Early boot hangs on Hyper-V to Early boot hangs on Hyper-V with Gen2 VMs
I never could replicate the original hang here, but my Hyper-V test VM (Gen2, ZFS) upgraded fine from 23.01 to 23.05 so at least that didn't regress.
- Status changed from Feedback to Resolved
23.05-RC works correctly with Azure and Windows 11 Hyper-V in all mentioned test cases. If any issues are found after 23.05 is released, a new issue report should be opened.
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by 
Updated by 
Updated by 
Updated by 