Regression #13895
closed
Early boot hangs on Hyper-V with Gen2 VMs
0%
Description
After installing pfSense+ 23.01 with ZFS on a HyperV Gen2 VM in Windows 11, pfSense does not boot. The console shows it stops after printing the EFI framebuffer information. See:
https://forum.netgate.com/topic/177220/
- Works: pfSense-CE-2.7.0-DEVELOPMENT-amd64-20230123-0600.iso
- Works: pfSense-plus-22.05-RELEASE-amd64.iso
- Works: FreeBSD-14.0-CURRENT-amd64-20221209-b1ef176eb528-259626-disc1.iso
- Broken (UFS works): pfSense-plus-23.01-RC-amd64-20230123-0600.iso
- Create a new VM with 4GB of Dynamic Memory, default switch, 8GB drive, and select an ISO for the boot image.
- Edit the VM and uncheck Hardware / Security / Secure Boot // Enable Secure Boot.
- Install using default options (ZFS).
- Edit the VM and select Hardware / SCSI Controller / DVD Drive // Media:
None. - Connect to the VM console and start the VM.
Files
Related issues
Updated by Steve Wheeler almost 2 years ago
VMs in Azure are not affected:
System Hyper-V Virtual Machine Netgate Device ID: XXXXXXXXXXXXXXXXXXXXX BIOS Vendor: Microsoft Corporation Version: Hyper-V UEFI Release v4.1 Release Date: Mon May 9 2022 Version 23.01-RC (amd64) built on Mon Jan 23 06:05:21 UTC 2023 FreeBSD 14.0-CURRENT
Updated by Jim Pingle almost 2 years ago
I can't reproduce this at all. 23.01 clean ZFS installs and upgrades boot fine here. Windows 10 22H2 and Windows 11 Pro 22H2.
I can warm boot, cold boot, etc. It comes up perfectly every time.
Setup is configured following the recipe at https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-hyper-v.html
There must be something else in the environment contributing to the problem.
Updated by Marcos M almost 2 years ago
- Description updated (diff)
Tested 22.05 as well and that worked - updated description. It may be related to https://redmine.pfsense.org/issues/12707
I tried it again using UFS instead of ZFS and it booted correctly.
Updated by Lev Prokofev almost 2 years ago
Don't see any issues with pfSense-plus-23.01-BETA-amd64-20230106-0600 on Win11pro 22H2 nor ZFS or UFS. Hyper-V is all default out of the box.
Updated by Marcos M almost 2 years ago
- Description updated (diff)
Changing the loader menu option 5. Cons: did not help. However, removing the DVD drive from the VM allows it to boot while connected to the VM console.
Updated by Marcos M almost 2 years ago
- File vm.png vm.png added
- Subject changed from pfSense Plus 23.01 does not boot on HyperV from Windows Server 2022 or Windows 11 to pfSense Plus 23.01 can fail to boot on Hyper-V
This can be replicated on Azure as well. Deploy a new pfSense+ 23.01 Gen 2 VM on Azure with a size which allows adding more than two NICs (e.g. Standard D8as v4). Booting works correctly when adding the second NIC, however the VM fails to boot after adding a third NIC. See attached image. Removing the third NIC allows it to boot correctly again.
Updated by Leon Dang almost 2 years ago
I can't repro it in Win-11 Hyper-V. 4GB, ZFS, 3 NICs, ISO still attached.
In the loader prompt (option 3), can you type "memmap"? I am seeing that in my setup, the conventional memory segment before LoaderData is rather big.
Updated by Chris Poillion almost 2 years ago
- File screenshot.png screenshot.png added
Here is a screenshot of the memmap command on an affect VM. This machine is on Windows Server 2022. The Pfsense VM is 2GB ram, 40GB ZFS, 2 NICS.
I have a Pfsense VM running 2.6, 8GB Ram, 40GB ZFS and 2 Nics that I tried to update to 23.01 but it also would not boot on 23.01. The VM that the memmap screenshot is from was a fresh install using the 2.6 iso, so same upgrade path.
Updated by Leon Dang almost 2 years ago
Thanks Chris.
While this doesn't solve your immediate situation, it looks like repro is possible with Windows Server 2022. We haven't exactly root caused why, but there seems to be a problem in 23.01 involving the loader's request to the Hyper-V UEFI firmware to allocate pages, which it uses to put the jump-to-kernel code and page-tables right before switching over to the kernel. If the allocation returns pages that sit right before the LoaderData segment, they get corrupted (probably soon after entering the kernel) and the system either hangs or reboots.
Updated by Marcos M over 1 year ago
For reference:
Early boot hang fixes which resolve serial console issues in Azure/Hyper-V
https://github.com/pfsense/FreeBSD-src/commit/927358dd98cb902160093e0dc0bac002d6b43858
https://github.com/pfsense/FreeBSD-src/commit/8ea7fa16d9fe5acb7d42a223efbfa23627aa5e0c
https://github.com/pfsense/FreeBSD-src/commit/bffe765d8070428e5099c29c47d24b3a666814f1
Multi-NIC issue mentioned in #note-9
https://gitlab.netgate.com/pfSense/FreeBSD-src/-/commit/8b70783c4bc1647a626081551f905ab06b11164a
Updated by Marcos M over 1 year ago
- Subject changed from pfSense Plus 23.01 can fail to boot on Hyper-V to Early boot hangs on Hyper-V
- Status changed from New to Feedback
Updated by Marcos M over 1 year ago
- Assignee set to Leon Dang
- Target version set to 23.05
- Affected Plus Version set to 23.01
Updated by Marcos M over 1 year ago
- Has duplicate Bug #13875: Boot after upgrade form 22.05 to 23.01 hangs added
Updated by Marcos M over 1 year ago
- Subject changed from Early boot hangs on Hyper-V to Early boot hangs on Hyper-V with Gen2 VMs
Updated by Jim Pingle over 1 year ago
I never could replicate the original hang here, but my Hyper-V test VM (Gen2, ZFS) upgraded fine from 23.01 to 23.05 so at least that didn't regress.
Updated by Marcos M over 1 year ago
- Status changed from Feedback to Resolved
23.05-RC works correctly with Azure and Windows 11 Hyper-V in all mentioned test cases. If any issues are found after 23.05 is released, a new issue report should be opened.