reply-to on IPv4+6 rules breaks v6
The auto-added reply-to on WAN rules in combination with IPv4+v6 rules breaks v6 connectivity, as the v4 IP is included as the reply-to address. Work around is just disabling the reply-to, at least where that's feasible. PF seems to ignore any v6 matching the rule if a v4 reply-to is specified, as it's not just being improperly routed, it's being logged as blocked.
There isn't a clean easy answer here. Splitting the rule, where it requires reply-to or route-to, into two separate rules in rules.debug is probably the best solution.