Compatibility/Mobile Testing Surveys

Mobile Testing Surveys

SoftVision will be testing the top mobile sites—in Firefox for Android—in a list of prioritized locales, and reporting bugs as they find them.

Test Plan

In general, we want to test a site in the most useful ways it can be tested, beyond looking at the home page, i.e., play and pause video, create an account and log in, add and remove an item to/from a cart, open a menu and select items, upvote a post, etc.

It's hard to know a priori what this will end up looking like. The plan is to start exploratory testing with the US locale and make notes of what is and isn't useful (or possible) to test. From that we'll create a test plan of the remaining locales based on the types of sites and their related common interactions.

As part of this testing, we should also document obstacles. For example, do we need to create paid accounts to test some of the sites? Do we need to purchase items, or is it enough to take a transaction all the way until payment is requested?

(Note: if we need to get money for this type of testing, we can request it.)

Wiki Test Plan

Bug Reporting Guidelines

All sites should be tested in Firefox for Android Nightly. If an issue doesn't reproduce in Chrome for Android, AND reproduces in Firefox for Android Nightly, a bug should be filed on webcompat.com with steps to reproduce and a screenshot (except for NSFW content; see below).

Bug identifier

Each bug report should have a lowercase bug identifier to help us link back to all bug reports filed as part of this project, and by country (via a search query). It should look like this:

sv; country: us

That is, the string "sv;" followed by a space, followed by the string "country:" followed by a space, followed by the 2 letter ISO 3166-1 alpha-2 code that corresponds to the top site list.

See here for a reference on country codes: https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2#Officially_assigned_code_elements

NSFW Content

Some sites tested may be adult or pornographic in nature. If bugs are encountered in these types of sites, screenshots should either not be included in the bug reports, or have the Not-Safe-For-Work portions blurred out before uploading.

Locales

We will start with the following locales, more to come after. Each locale should link to a query to bring up reported bugs, a link to the tested URLs, and have a summary once it has been tested.

Getting top sites

Top Sites can be obtained from Alexa: http://www.alexa.com/topsites/countries

The following snippet of code can be run in the browser console on a result page to make it easy to copy and paste into the wiki.

(Note: https://aws.amazon.com/alexa-top-sites/ also exists, but subscribing doesn't seem to be working for me at the moment)

$$('div.tr > div > p > a:nth-child(1)').map(el => {
  return `#  ${el.textContent} \n`;
}).join('')

US

Some of these won't be useful to test (t.co, for example). And others will have additional popular subdomains worth testing. Alexa doesn't provide that info, unfortunately.

1. Google.com
2. Youtube.com
3. Facebook.com
4. Amazon.com
5. Yahoo.com
6. Wikipedia.org
7. Reddit.com
8. Twitter.com
9. Ebay.com
10. Linkedin.com
11. Netflix.com
12. Diply.com
13. Instagram.com
14. Live.com
15. Craigslist.org
16. Ntd.tv
17. Bing.com
18. Imgur.com
19. Cnn.com
20. Office.com
21. Pinterest.com
22. Tumblr.com
23. Microsoftonline.com
24. T.co
25. Chase.com
26. Nytimes.com
27. Blogspot.com
28. Paypal.com
29. Imdb.com
30. Livejasmin.com
31. Pornhub.com
32. Wordpress.com
33. Espn.com
34. Apple.com
35. Walmart.com
36. Msn.com
37. Wikia.com
38. Breitbart.com
39. Salesforce.com
40. Bankofamerica.com
41. Weather.com
42. Wellsfargo.com
43. Washingtonpost.com
44. Microsoft.com
45. Zillow.com
46. Huffingtonpost.com
47. Twitch.tv
48. Instructure.com
49. Dropbox.com
50. Intuit.com
51. Foxnews.com
52. Googleusercontent.com
53. Stackoverflow.com
54. Yelp.com
55. Github.com
56. Pandora.com
57. Adobe.com
58. Conservativetribune.com
59. Soundcloud.com
60. Indeed.com
61. Etsy.com
62. Aol.com
63. Force.com
64. Gfycat.com
65. Amazonaws.com
66. Spotify.com
67. Vice.com
68. Xfinity.com
69. Onclkds.com
70. Buzzfeed.com
71. Godaddy.com
72. Txxx.com
73. Quora.com
74. Weebly.com
75. Patch.com
76. Bestbuy.com
77. Thesaurus.com
78. Capitalone.com
79. Baidu.com
80. Blackboard.com
81. Thepiratebay.org
82. Bbc.com
83. Go.com
84. Tripadvisor.com
85. Roblox.com
86. Dingit.tv
87. Forbes.com
88. Homedepot.com
89. Stackexchange.com
90. Vimeo.com
91. Deviantart.com
92. Target.com
93. Rumble.com
94. Usps.com
95. 123movies.is
96. Irs.gov
97. Cnet.com
98. Citi.com
99. Dailymail.co.uk
100. Businessinsider.com

France

1. Google.fr
2. Youtube.com
3. Google.com
4. Facebook.com
5. Wikipedia.org
6. Amazon.fr
7. Leboncoin.fr
8. Yahoo.com
9. Live.com
10. Orange.fr
11. Twitter.com
12. Livejasmin.com
13. Free.fr
14. Linkedin.com
15. Lemonde.fr
16. Instagram.com
17. Reddit.com
18. Lefigaro.fr
19. Ebay.fr
20. Cdiscount.com
21. T.co
22. Zone-telechargement.ws
23. Jeuxvideo.com
24. Labanquepostale.fr
25. Allocine.fr
26. Msn.com
27. Commentcamarche.net
28. Pole-emploi.fr
29. Sfr.fr
30. Pornhub.com
31. Vk.com
32. Lequipe.fr
33. Twitch.tv
34. Xhamster.com
35. Pinterest.com
36. Voirfilms.co
37. Programme-tv.net
38. Credit-agricole.fr
39. 20minutes.fr
40. Github.com
41. Netflix.com
42. Aliexpress.com
43. Caf.fr
44. Wordpress.com
45. Francetvinfo.fr
46. Bongacams.com
47. Torrent9.biz
48. Tumblr.com
49. Stackoverflow.com
50. Linternaute.com