Project

General

Profile

Actions

Regression #13895

closed

Early boot hangs on Hyper-V with Gen2 VMs

Added by Marcos M about 1 year ago. Updated 11 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Operating System
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Release Notes:
Default
Affected Plus Version:
23.01
Affected Architecture:

Description

After installing pfSense+ 23.01 with ZFS on a HyperV Gen2 VM in Windows 11, pfSense does not boot. The console shows it stops after printing the EFI framebuffer information. See:
https://forum.netgate.com/topic/177220/

Tested:
  • Works: pfSense-CE-2.7.0-DEVELOPMENT-amd64-20230123-0600.iso
  • Works: pfSense-plus-22.05-RELEASE-amd64.iso
  • Works: FreeBSD-14.0-CURRENT-amd64-20221209-b1ef176eb528-259626-disc1.iso
  • Broken (UFS works): pfSense-plus-23.01-RC-amd64-20230123-0600.iso
Steps:
  1. Create a new VM with 4GB of Dynamic Memory, default switch, 8GB drive, and select an ISO for the boot image.
  2. Edit the VM and uncheck Hardware / Security / Secure Boot // Enable Secure Boot.
  3. Install using default options (ZFS).
  4. Edit the VM and select Hardware / SCSI Controller / DVD Drive // Media: None.
  5. Connect to the VM console and start the VM.

Files

vm.png (16.4 KB) vm.png Marcos M, 03/05/2023 03:43 PM
screenshot.png (57.8 KB) screenshot.png memmap command Chris Poillion, 03/13/2023 08:46 PM

Related issues

Has duplicate Bug #13875: Boot after upgrade form 22.05 to 23.01 hangsDuplicate

Actions
Actions #1

Updated by Marcos M about 1 year ago

  • Description updated (diff)
Actions #2

Updated by Steve Wheeler about 1 year ago

VMs in Azure are not affected:

System     Hyper-V Virtual Machine
Netgate Device ID: XXXXXXXXXXXXXXXXXXXXX    
BIOS     Vendor: Microsoft Corporation
Version: Hyper-V UEFI Release v4.1
Release Date: Mon May 9 2022
Version     23.01-RC (amd64)
built on Mon Jan 23 06:05:21 UTC 2023
FreeBSD 14.0-CURRENT

Actions #3

Updated by Jim Pingle about 1 year ago

I can't reproduce this at all. 23.01 clean ZFS installs and upgrades boot fine here. Windows 10 22H2 and Windows 11 Pro 22H2.

I can warm boot, cold boot, etc. It comes up perfectly every time.

Setup is configured following the recipe at https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-hyper-v.html

There must be something else in the environment contributing to the problem.

Actions #4

Updated by Marcos M about 1 year ago

  • Description updated (diff)

Tested 22.05 as well and that worked - updated description. It may be related to https://redmine.pfsense.org/issues/12707

I tried it again using UFS instead of ZFS and it booted correctly.

Actions #5

Updated by Marcos M about 1 year ago

  • Description updated (diff)
Actions #6

Updated by Lev Prokofev about 1 year ago

Don't see any issues with pfSense-plus-23.01-BETA-amd64-20230106-0600 on Win11pro 22H2 nor ZFS or UFS. Hyper-V is all default out of the box.

Actions #7

Updated by Marcos M about 1 year ago

  • Description updated (diff)

Changing the loader menu option 5. Cons: did not help. However, removing the DVD drive from the VM allows it to boot while connected to the VM console.

Actions #8

Updated by Marcos M about 1 year ago

  • Description updated (diff)
Actions #9

Updated by Marcos M about 1 year ago

  • File vm.png vm.png added
  • Subject changed from pfSense Plus 23.01 does not boot on HyperV from Windows Server 2022 or Windows 11 to pfSense Plus 23.01 can fail to boot on Hyper-V

This can be replicated on Azure as well. Deploy a new pfSense+ 23.01 Gen 2 VM on Azure with a size which allows adding more than two NICs (e.g. Standard D8as v4). Booting works correctly when adding the second NIC, however the VM fails to boot after adding a third NIC. See attached image. Removing the third NIC allows it to boot correctly again.

Actions #10

Updated by Leon Dang about 1 year ago

I can't repro it in Win-11 Hyper-V. 4GB, ZFS, 3 NICs, ISO still attached.

In the loader prompt (option 3), can you type "memmap"? I am seeing that in my setup, the conventional memory segment before LoaderData is rather big.

Actions #11

Updated by Chris Poillion about 1 year ago

Here is a screenshot of the memmap command on an affect VM. This machine is on Windows Server 2022. The Pfsense VM is 2GB ram, 40GB ZFS, 2 NICS.

memmap command

I have a Pfsense VM running 2.6, 8GB Ram, 40GB ZFS and 2 Nics that I tried to update to 23.01 but it also would not boot on 23.01. The VM that the memmap screenshot is from was a fresh install using the 2.6 iso, so same upgrade path.

Actions #12

Updated by Leon Dang about 1 year ago

Thanks Chris.

While this doesn't solve your immediate situation, it looks like repro is possible with Windows Server 2022. We haven't exactly root caused why, but there seems to be a problem in 23.01 involving the loader's request to the Hyper-V UEFI firmware to allocate pages, which it uses to put the jump-to-kernel code and page-tables right before switching over to the kernel. If the allocation returns pages that sit right before the LoaderData segment, they get corrupted (probably soon after entering the kernel) and the system either hangs or reboots.

Actions #14

Updated by Marcos M almost 1 year ago

  • Subject changed from pfSense Plus 23.01 can fail to boot on Hyper-V to Early boot hangs on Hyper-V
  • Status changed from New to Feedback
Actions #15

Updated by Marcos M almost 1 year ago

  • Assignee set to Leon Dang
  • Target version set to 23.05
  • Affected Plus Version set to 23.01
Actions #16

Updated by Marcos M almost 1 year ago

  • Has duplicate Bug #13875: Boot after upgrade form 22.05 to 23.01 hangs added
Actions #17

Updated by Marcos M 12 months ago

  • Subject changed from Early boot hangs on Hyper-V to Early boot hangs on Hyper-V with Gen2 VMs
Actions #18

Updated by Jim Pingle 12 months ago

I never could replicate the original hang here, but my Hyper-V test VM (Gen2, ZFS) upgraded fine from 23.01 to 23.05 so at least that didn't regress.

Actions #19

Updated by Marcos M 11 months ago

  • Status changed from Feedback to Resolved

23.05-RC works correctly with Azure and Windows 11 Hyper-V in all mentioned test cases. If any issues are found after 23.05 is released, a new issue report should be opened.

Actions

Also available in: Atom PDF