The issue was addressed with improved checks. This issue is fixed in visionOS 1.2, macOS Sonoma 14.5, Safari 17.5. A website's permission dialog may persist after navigation away from the site.
https://support.apple.com/kb/HT214108
https://support.apple.com/kb/HT214106
https://support.apple.com/kb/HT214103
https://support.apple.com/en-us/HT214108
https://support.apple.com/en-us/HT214106