pfSense

Also need to add the following lines:
interface: 0.0.0.0@853
interface: ::0@853

The interface address being set to match normal dns settings.

I did some more research on this. A simple checkbox that adds the following lines to the config should do the trick:
#EDIT - needed to accept tcp 853
interface-automatic: no
#/EDIT
ssl-service-key: "/var/etc/cert.key"
ssl-service-pem: "/var/etc/cert.crt"
ssl-port: 853
interface: 0.0.0.0@853
interface: ::0@853

As far as a use case goes I can only imagine this being useful over an open WIFI AP. Android has added the option to enable DNS over TLS to the AOSP here:
https://android-review.googlesource.com/#/c/platform/frameworks/base/+/499911/4/services/core/java/com/android/server/NetworkManagementService.java

This should mean that a connected android device will utilize the service if available at some point down the line. The rest of DNS over TLS implementation will happen in the unbound development.

Edit: I was able to get this to work. info posted below.

Forum link:
https://forum.pfsense.org/index.php?topic=139471.0

I'd like to request that this FR be closed. When I created it I did so because I didn't think it was possible to do through the Custom Options due to the certificate. Since this is stored on the drive and has a static path this really should be done through the Custom Options for the foreseeable future. Particularly while implementation is still being worked on.

If I find a way to get it working I'll post the custom options to the forum for anyone who wants to try it.

Also it would need significantly more logic here than you've shown thus far. For instance, you can't always assume that the GUI is set to HTTPS, it would need a dedicated ca/certificate selection. Plus, you would have to have additional binding options. You could assume the same binding options as the main unbound service, but you can't always bind it to any/all unless that's what the user chose to do.

No harm in keeping it open without a target though, so it can remain as-is.

I was able to get this to work using the following:

#this prevents port 853 tcp from working. Not sure why? Turning off
interface-automatic: no

interface: 0.0.0.0@853
interface: ::0@853
ssl-port: 853
ssl-service-pem: "/var/etc/cert.crt"
ssl-service-key: "/var/etc/cert.key"

As you noted, that cert is only present if https is enabled. I could add the above to the web config and unbound.inc file to be used, and work with the interface selection, but the cert drop down selector (and writing to disk) is beyond my ability.

See also: #8415 and #8388

Tested on on 2.4.4.a.20180406.1258, works as expected.

This feature seems not to be documented here:

https://docs.netgate.com/pfsense/en/latest/services/dns/resolver.html

The strings "TLS" and "DoT" are both absent.