Since the release of Firefox 48 on 2016-08-02, Mozilla Firefox will not run any add-on that has not been signed and verified by Mozilla. This means, for example, that a user cannot modify an add-on and run it for personal use without first submitting it to Mozilla's approval. There is no option to override this behavior in vanilla Firefox, but it can be disabled in the Firefox Developer Edition.
Does anything in this situation violate the Free Software Definition or Open Source Definition?
The Free Software Definition (from the FSF) is:
A program is free software if the program's users have the four essential freedoms:
The freedom to run the program as you wish, for any purpose (freedom 0).
The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
The freedom to redistribute copies so you can help your neighbor (freedom 2).
The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.
As you can see there isn't anything about plugins. Indeed, most programs won't even allow any plugin. If you want to change the software, you can, but you need to go back from the sources. In a sense, this will be the same with Firefox. Nothing will prevent you to modify the source and remove this restriction and run any plugin. So Firefox is still free.
Now I must say that @olliebulle's reasoning is appealing. You could consider that the Firefox released binary is, as a platform, and in some specific sense, nonfree. But the very fact that you can duplicate the platform very easily because it is software whereas you cannot when it is hardware is why tivoization is only a problem in the latter case.
This looks very much like "tivoization" to me, except that instead of being the firmware of a piece of hardware (e.g. Tivo) that requires a signature, it is a software (Firefox).
From FSF:
[...] many products containing computers check signatures on their executable programs to block users from installing different executables; only one privileged company can make executables that can run in the device or can access its full capabilities. We call these devices “tyrants”, and the practice is called “tivoization” after the product (Tivo) where we first saw it. Even if the executable is made from free source code, the users cannot run modified versions of it, so the executable is nonfree.
Also,
The criteria for open source do not recognize this issue; they are concerned solely with the licensing of the source code. Thus, these unmodifiable executables, when made from source code such as Linux that is open source and free, are open source but not free. Many Android products contain nonfree tivoized executables of Linux.
So the issue you mention could still make the piece of software Open Source but not free.
I came across a Firefox add-on that I wanted to modify, and after some digging, the answer appears to be yes. But my results seem counterintuitive, and I would like other people to verify them, so here is the documentation for building Firefox add-ons. I found it easy enough to get up and running but would be happy to help anyone through the process.
On Firefox 49.0.2, Xubuntu 16.10, I was able to modify an add-on and install it in Firefox without signing it, even with xpinstall.signatures.required set to true in about:config. I did have to manually install one of its dependencies with NPM. If that doesn't work for you, you may have to create a free developer account with Mozilla, generate signing keys from that account, and sign your add-on manually. I was able to do this as a "man off the street" figuratively and get a signed add-on. Still, the fact that you have to go through Mozilla to get the signing key might make the software non-free.
Here is the add-on that I modified (GitHub). If someone could modify, compile, and install the add-on to verify that this isn't just some anomaly on my system that'd be great.
Zimm i48 has since verified that the user can change the setting to require signing, even though the developers said it would be gone entirely in later Firefox versions. So no problem at all.
xpinstall.signatures.required in about:config? If yes, it looks even less problematic. It's just a safe-guard for basic users but any advanced user can remove the safe-guard, even without recompiling Firefox-the-platform.
xpinstall.signatures.required in about:config? I can't say whether you're a Firefox user, of course, but if you could take the time to install it and confirm that it wasn't just a quirk because I disabled that setting before add-on signing was required that would solve the mystery.
Commented
Nov 4, 2016 at 3:02