No, you can choose which app to give access to. If you wish to use the Sporran you can disable the Polkadot.JS popup:
1. Go to Settings in the Polkadot.JS App (top right, cog icon)
2. Scroll to "Manage Website Access"
3. Scroll to stakeboard.kilt.io
4. Change access from "allowed" to "denied"