"URL Table (IPs)" and "URL (IPs)" do not work when text file is hosted on a fresh install of pfSense
I ran into this problem on a fresh DVD install of pfSense. An automated upgrade did not experience this problem. On this page:
"URL (IPs)" does not work at all, and it gives this error:
The following input errors were detected: You must provide a valid URL. Could not fetch usable data from 'https://192.168.1.1/some_file.txt'.
"URL Table (IPs)" is accepted by pfSense, but it does not load the actual IP's, and they will not show up in mouse hover on firewall rules page, and they will not actually function either.
If I remember correctly, when I tried to load the backup config XML file from the upgraded machine to the fresh-install machine, it appeared to accept the settings, but failed to actually function. Both machines are the same models, and they have identical hardware.
Print an error when the given URL cannot be fetched instead of just ignoring it.
Fix an use of an uninitialized variable.
#3 Updated by badon _ over 5 years ago
It's a list of IP addresses, one IP on each line. I just tested it in a new install of 2.1.5, and it works fine there too. The only one that never works is a new install of 2.2.2. I can browse to the URL:
And I see the list of IP addresses in my web browser, as expected. This works in all versions of pfSense, and I've tried this every time without fail. I'm 100% certain the problem is NOT the text file. I've even tested various different encodings, like ANSI, UTF-8 without BOM, etc, and the results are exactly the same in every test. Everything works except a new install of 2.2.2.
Is there something else I can look at to provide you with more information to find the cause of this bug? Any logs, or something that might provide more info? I have been testing this bug off and on for a months now, and every variation of the test ends up with the same results. New install fails, everything else works. I don't have a lot of hardware to test on, and my last spare is now in service, which leaves me no extra hardware to test on, so I decided it's time to report the bug, since this is as far as I've been able to get with it.
#4 Updated by badon _ over 5 years ago
Let me point out that the "URL Table (IPs)" version of this test does not produce any error messages. Therefore, if it SHOULD be producing an error message, that's another problem. Either way, the fact that the "URL (IPs)" version of this test produces a different result is a good clue that something is wrong with pfSense. If the text file were borked, both the "URL (IPs)" and "URL Table (IPs)" versions of this test should produce the same or similar results, but they do not. Furthermore, since pfSense functions correctly in most test circumstances (older versions and auto-update versions), that's another clue that the problem is likely isolated to new installs of 2.2.2.
Why only new installs of 2.2.2 is affected, I don't know. I would have to do more testing to figure that out, but I don't have any more spare hardware to try it and see what happens. One test I would like to do, but can't at the moment, is allow an auto-update to proceed on 2.1.5 to see if I can get the auto-update version to reproduce the bug somehow. Or, alternatively, fiddle with a lot of new installs until I can get it to function without triggering the bug (I'm not so clear on how to that in this case, though).
Maybe you have some ideas of how I might be able to do further tests to help you isolate the problem? I don't have any spare hardware, but maybe I can try setting up pfSense in a virtual machine or something weird like that. If I had a good idea of what direction I should go in testing that would be productive in helping you, then maybe I could put more effort in finding some extra hardware to test on. Could it be a quirk with my hardware? This problem doesn't strike me as a hardware issue, which is why I suggest it might be possible to reproduce the bug in VM's. Correct me if I'm wrong about that.
#7 Updated by badon _ over 5 years ago
I tested an auto-upgrade again before doing the test you suggested. The auto-upgrade sort of failed somehow because the machine never rebooted, and it just dropped to a login prompt that didn't respond to login attempts. The Web GUI did show the upgraded version as 2.2.2, so that's something I guess. Although, the test wasn't completely successful, I was able to check if the aliases still worked, and they did, but I can't be sure it was 2.2.2 aliases, or still 2.1.5, since the reboot didn't occur and the upgrade probably didn't complete. I think pfSense is caching the result of the URL contents from before the upgrade, and that's why a new install doesn't work, but an upgrade does.
Chris Buechler wrote:
Guessing it's because we enable certificate validation by default in 2.2.x there, and the default self-signed cert will fail that check. If you switch your GUI to HTTP as a test (System>Advanced), and put in the URL as HTTP, does it work?
No, it did not work, not at first. It was in the "URL Table (IPs)" version of the configuration, which failed to function after I changed the update frequency to 1 day and then saved it to try to get it reload the URL (for some reason pfSense keeps changing the frequency to 7 days). When I switched it to "URL (IPs)", it worked! When I switched it back to "URL Table (IPs)", it continued to work (cached maybe?). So, it looks like this bug in aliases is triggered by Web GUI HTTPS self-signed certificate configuration.
I switched back to HTTPS, and then tested the URL with 127.0.0.1 instead of 192.168.1.1, with the same bug results. I then tested everything again with 192.168.1.1, and the aliases work again, just like they did with the auto-upgrade scenario. Not sure if any of that is relevant, but noted here since I did the work.
It looks like this bug has been isolated! Thanks for the excellent suggestion to test HTTP versus HTTPS, that was the missing tip I needed to finish this.
#8 Updated by Chris Buechler over 5 years ago
the upgrade issue you noted is fixed for 2.2.3, release coming next week. Upgrading to the latest snapshot from snapshots.pfsense.org and re-testing might be useful. I don't recall anything along those lines that's changed though, and the root issue is the cert validation.
Could you provide the contents of your some_file.txt?
#9 Updated by badon _ over 5 years ago
I can't share the IP addresses because they are Tor bridges, which must be kept secret in order to be useful. Does this bug need to be isolated further? If so, do you think the text file might have some role? I seem to remember testing another file that just had the Google Public DNS IP addresses in it: 220.127.116.11 and 18.104.22.168. I don't remember seeing anything different happen with that file, which might be one of the reasons I'm fairly certain the file content isn't important to trigger the bug. I'm guessing you're just trying to reproduce the bug on your end, and you just need a file? I attached one to this message with the Google DNS IP's.
#12 Updated by badon _ about 5 years ago
I have discovered a workaround for this problem. I'm not sure if it's a feature or a bug, so I made a separate report for a feature enhancement to make a form field that is easier to manually enter a lot of data in a quick copy-and-paste operation:
#16 Updated by Marcelo Matos about 4 years ago
In URL you must to use same hostname on self-signed certificate. Change 192.168.1.1 by this hostname. Remember to resolve the hostname to 192.168.1.1 on pfsense.
In Pfsense configuration, verify if option "Verify HTTPS certificates when downloading alias URLs" is unchecked (System / Advanced / Firewall & NAT).