The setting you mention uses the function "addCachedChallengeCheckerResults()" in the script, and reading that I'm pretty sure this is for the checker icon close to the profile-pic in logs on challenge caches, not for the icon in the challenge checker to the right of the cache page.
IIRC this could be related to you cookie-settings in the browser you're using, you need to allow third party cookies or something like that.