Pop-up rule change 2016-12-14 13:48:27 UTC, Kristen Hohenstein
Hello dear testers! We’re changing our rules regarding pop-ups.
In the past, we told you to report any pop-up a test author did not mention in the instructions, including pop-ups not produced from the customer website itself (e.g. pop-ups related to browser updates, network issues, browsers saving your password, etc).
We originally decided on this solution with the intention to disable all pop-ups not produced from a customer’s website itself. Unfortunately, this has been very hard to keep up. Every browser update presents new pop-ups we need to disable and we often don’t learn about them until after customer results have been affected. This results in testers being forced to fail an otherwise passable test due to a random popup the customer likely doesn’t care about. This is not useful for our customers, is incredibly frustrating for our testers, and not a good use of developer time.
Moving forward, you only need to report a pop-up if it is a pop-up produced by the customer’s website that the test author didn’t warn you about. Do NOT report pop-ups that are related to browsers, networks, hardware etc. Here are a couple examples of the distinction we’re drawing:
- If you log into a customer’s website and you get a pop up asking if you want Chrome to save your password, do NOT report this.
- If you are testing for Best Company Ever and encounter a pop-up asking, “Do you want to subscribe to weekly email updates from Best Company Ever?”, but the test author never mentioned encountering this pop-up in the instructions, you need to report it.
I hope this solution makes testing with Rainforest a better experience for both our testers and our customers. Do NOT stress about nitpicking over this in the future — discernment and common sense should hopefully be enough to determine whether a pop-up is directly produced by a customer’s website or by something else, like a browser setting or network update. If we see this solution isn’t working and there’s still a lot of confusion, we will readjust as needed. Moving forward, please only submit support tickets or create forum topics about this if it is a pop-up that’s truly confusing. I’ll be updating the forum and confusing scenarios wiki page with this information, and will also try to make tutorial videos with examples to supplement this announcement in the future. It's important to note that you still need to report error messages you receive. So if you encounter an error message in pop-up format, you need to report this.
I also want to emphasize a point that’s not a rule change, but something I’ve said before: If a test author instructs you at the beginning of the test to close any pop-ups you might encounter and continue testing, this counts as them mentioning any pop-up you might encounter. Do NOT report pop-ups you encounter after a test author says this.
Thank you for hanging in there with us and understanding that we’re always trying to improve our processes even if we don’t get it right the first time. We sincerely appreciate everything you do for us. Happy testing ❤