Project

General

Profile

Actions

Regression #13895

open

pfSense Plus 23.01 can fail to boot on Hyper-V

Added by Marcos M about 2 months ago. Updated 10 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Operating System
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Release Notes:
Default
Affected Plus Version:
Affected Architecture:

Description

After installing pfSense+ 23.01 with ZFS on a HyperV Gen2 VM in Windows 11, pfSense does not boot. The console shows it stops after printing the EFI framebuffer information. See:
https://forum.netgate.com/topic/177220/

Tested:
  • Works: pfSense-CE-2.7.0-DEVELOPMENT-amd64-20230123-0600.iso
  • Works: pfSense-plus-22.05-RELEASE-amd64.iso
  • Works: FreeBSD-14.0-CURRENT-amd64-20221209-b1ef176eb528-259626-disc1.iso
  • Broken (UFS works): pfSense-plus-23.01-RC-amd64-20230123-0600.iso
Steps:
  1. Create a new VM with 4GB of Dynamic Memory, default switch, 8GB drive, and select an ISO for the boot image.
  2. Edit the VM and uncheck Hardware / Security / Secure Boot // Enable Secure Boot.
  3. Install using default options (ZFS).
  4. Edit the VM and select Hardware / SCSI Controller / DVD Drive // Media: None.
  5. Connect to the VM console and start the VM.

Files

vm.png (16.4 KB) vm.png Marcos M, 03/05/2023 03:43 PM
screenshot.png (57.8 KB) screenshot.png memmap command Chris Poillion, 03/13/2023 08:46 PM
Actions #1

Updated by Marcos M about 2 months ago

  • Description updated (diff)
Actions #2

Updated by Steve Wheeler about 2 months ago

VMs in Azure are not affected:

System     Hyper-V Virtual Machine
Netgate Device ID: XXXXXXXXXXXXXXXXXXXXX    
BIOS     Vendor: Microsoft Corporation
Version: Hyper-V UEFI Release v4.1
Release Date: Mon May 9 2022
Version     23.01-RC (amd64)
built on Mon Jan 23 06:05:21 UTC 2023
FreeBSD 14.0-CURRENT

Actions #3

Updated by Jim Pingle about 2 months ago

I can't reproduce this at all. 23.01 clean ZFS installs and upgrades boot fine here. Windows 10 22H2 and Windows 11 Pro 22H2.

I can warm boot, cold boot, etc. It comes up perfectly every time.

Setup is configured following the recipe at https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-hyper-v.html

There must be something else in the environment contributing to the problem.

Actions #4

Updated by Marcos M about 2 months ago

  • Description updated (diff)

Tested 22.05 as well and that worked - updated description. It may be related to https://redmine.pfsense.org/issues/12707

I tried it again using UFS instead of ZFS and it booted correctly.

Actions #5

Updated by Marcos M about 2 months ago

  • Description updated (diff)
Actions #6

Updated by Lev Prokofev about 2 months ago

Don't see any issues with pfSense-plus-23.01-BETA-amd64-20230106-0600 on Win11pro 22H2 nor ZFS or UFS. Hyper-V is all default out of the box.

Actions #7

Updated by Marcos M about 2 months ago

  • Description updated (diff)

Changing the loader menu option 5. Cons: did not help. However, removing the DVD drive from the VM allows it to boot while connected to the VM console.

Actions #8

Updated by Marcos M about 2 months ago

  • Description updated (diff)
Actions #9

Updated by Marcos M 19 days ago

  • File vm.png vm.png added
  • Subject changed from pfSense Plus 23.01 does not boot on HyperV from Windows Server 2022 or Windows 11 to pfSense Plus 23.01 can fail to boot on Hyper-V

This can be replicated on Azure as well. Deploy a new pfSense+ 23.01 Gen 2 VM on Azure with a size which allows adding more than two NICs (e.g. Standard D8as v4). Booting works correctly when adding the second NIC, however the VM fails to boot after adding a third NIC. See attached image. Removing the third NIC allows it to boot correctly again.

Actions #10

Updated by Leon Dang 16 days ago

I can't repro it in Win-11 Hyper-V. 4GB, ZFS, 3 NICs, ISO still attached.

In the loader prompt (option 3), can you type "memmap"? I am seeing that in my setup, the conventional memory segment before LoaderData is rather big.

Actions #11

Updated by Chris Poillion 11 days ago

Here is a screenshot of the memmap command on an affect VM. This machine is on Windows Server 2022. The Pfsense VM is 2GB ram, 40GB ZFS, 2 NICS.

memmap command

I have a Pfsense VM running 2.6, 8GB Ram, 40GB ZFS and 2 Nics that I tried to update to 23.01 but it also would not boot on 23.01. The VM that the memmap screenshot is from was a fresh install using the 2.6 iso, so same upgrade path.

Actions #12

Updated by Leon Dang 10 days ago

Thanks Chris.

While this doesn't solve your immediate situation, it looks like repro is possible with Windows Server 2022. We haven't exactly root caused why, but there seems to be a problem in 23.01 involving the loader's request to the Hyper-V UEFI firmware to allocate pages, which it uses to put the jump-to-kernel code and page-tables right before switching over to the kernel. If the allocation returns pages that sit right before the LoaderData segment, they get corrupted (probably soon after entering the kernel) and the system either hangs or reboots.

Actions

Also available in: Atom PDF