Extras-testing/QA checklist - Revision history

imported_>sim6: /* Blockers */

2012-05-28T10:53:42Z

Blockers

← Older revision		Revision as of 10:53, 28 May 2012
Line 61:		Line 61:
	# Lack of bug reporting database		# Lack of bug reporting database
	#: It needs to be [[Packaging#Bugtracker location\|identified in the http://maemo.org/packages/ page]].<br> http://bugs.maemo.org/ is the preferred option.		#: It needs to be [[Packaging#Bugtracker location\|identified in the http://maemo.org/packages/ page]].<br> http://bugs.maemo.org/ is the preferred option.
	#: A missing bugtracker is NOT a blocker as of 01.09.2010, but use of bugtracking facilities IS strongly recommended.		#: '''A missing bugtracker is NOT a blocker''' as of 01.09.2010, but use of bugtracking facilities IS strongly recommended.
	#:''Developers: If the bugtracker field is not specified, the address is defaulted to be that of the package itself on maemo.org: <nowiki>http://maemo.org/packages/view/<package>(/<version>)</nowiki> Resolutions for bugtracking are found at [[Packaging#Bugtracker_location]]''		#:''Developers: If the bugtracker field is not specified, the address is defaulted to be that of the package itself on maemo.org: <nowiki>http://maemo.org/packages/view/<package>(/<version>)</nowiki> Resolutions for bugtracking are found at [[Packaging#Bugtracker_location]]''
	# Licensing and legal issues		# Licensing and legal issues

imported_>dneary: Small formatting change

2010-12-14T17:19:56Z

Small formatting change

← Older revision		Revision as of 17:19, 14 December 2010
Line 73:		Line 73:
	# Optification		# Optification
	#: '''''Test using the ''df'' command:'''''		#: '''''Test using the ''df'' command:'''''
	#: In Xterminal, run		#: In Xterminal, run <code>df -h \| grep -e rootfs -e mmcblk0p2</code> before and after you install the package being tested, and see how much the usage of each partition changes. If more than, say, 300 kb (0.3 MB) are added to usage of rootfs, it's too much and the package needs to be optified.
	#:<code>df -h \| grep -e rootfs -e mmcblk0p2</code>
	#: before and after you install the package being tested, and see how much the usage of each partition changes. If more than, say, 300 kb (0.3 MB) are added to usage of rootfs, it's too much and the package needs to be optified.
	#: '''''Test using the ''dpkg'' command:'''''		#: '''''Test using the ''dpkg'' command:'''''
	#: After installation of the package being tested, run in Xterminal		#: After installation of the package being tested, run in Xterminal <code>dpkg -L packagename</code> It will show all the files installed by the package. Inspect the result to confirm that some of the files are placed in the /opt or MyDocs directories.
	#:<code>dpkg -L packagename</code>
	#: It will show all the files installed by the package. Inspect the result to confirm that some of the files are placed in the /opt or MyDocs directories.
	#: Please add those packages which fail optification to the list at [[Opt_Problem/Non-Optified_packages]].		#: Please add those packages which fail optification to the list at [[Opt_Problem/Non-Optified_packages]].
	#: ''Developers: An application is correctly "optified" if all files created during the normal operation of the application are created and stored in /opt or MyDocs, since these directories are not stored on the root filesystem. If the application or its dependencies ignore the recommendation to [[Documentation/Maemo 5 Developer Guide/Packaging, Deploying and Distributing/Installing under opt and MyDocs\|use the eMMC to install as much files as possible]], filling the root partition with 500kb or more, this can be considered a blocker. Also see [[Opt Problem]].''		#: ''Developers: An application is correctly "optified" if all files created during the normal operation of the application are created and stored in /opt or MyDocs, since these directories are not stored on the root filesystem. If the application or its dependencies ignore the recommendation to [[Documentation/Maemo 5 Developer Guide/Packaging, Deploying and Distributing/Installing under opt and MyDocs\|use the eMMC to install as much files as possible]], filling the root partition with 500kb or more, this can be considered a blocker. Also see [[Opt Problem]].''

imported_>rotoflex: /* Blockers */

2010-12-14T17:18:17Z

Blockers

← Older revision		Revision as of 17:18, 14 December 2010
Line 72:		Line 72:
	#:Core features advertised through UI or product page don't work or are missing.		#:Core features advertised through UI or product page don't work or are missing.
	# Optification		# Optification
	#: * Test using the ''df'' command:		#: '''''Test using the ''df'' command:'''''
	#: In Xterminal, run		#: In Xterminal, run
	#:<code>df -h \| grep -e rootfs -e mmcblk0p2</code>		#:<code>df -h \| grep -e rootfs -e mmcblk0p2</code>
	#: before and after you install the package being tested, and see how much the usage of each partition changes. If more than, say, 300 kb (0.3 MB) are added to usage of rootfs, it's too much and the package needs to be optified.		#: before and after you install the package being tested, and see how much the usage of each partition changes. If more than, say, 300 kb (0.3 MB) are added to usage of rootfs, it's too much and the package needs to be optified.
	#: * Test using the ''dpkg'' command:		#: '''''Test using the ''dpkg'' command:'''''
	#: After installation of the package being tested, run in Xterminal		#: After installation of the package being tested, run in Xterminal
	#:<code>dpkg -L packagename</code>		#:<code>dpkg -L packagename</code>

imported_>rotoflex: /* Blockers */

2010-12-14T17:09:08Z

Blockers

← Older revision		Revision as of 17:09, 14 December 2010
Line 72:		Line 72:
	#:Core features advertised through UI or product page don't work or are missing.		#:Core features advertised through UI or product page don't work or are missing.
	# Optification		# Optification
	#: Test using the ''df'' command:		#: * Test using the ''df'' command:
	#: In Xterminal, run		#: In Xterminal, run
	#:<code>df -h \| grep -e rootfs -e mmcblk0p2</code>		#:<code>df -h \| grep -e rootfs -e mmcblk0p2</code>
	#: before and after you install the package being tested, and see how much the usage of each partition changes. If more than, say, 300 kb (0.3 MB) are added to usage of rootfs, it's too much and the package needs to be optified.		#: before and after you install the package being tested, and see how much the usage of each partition changes. If more than, say, 300 kb (0.3 MB) are added to usage of rootfs, it's too much and the package needs to be optified.
	#: Test using the ''dpkg'' command:		#: * Test using the ''dpkg'' command:
	#: After installation of the package being tested, run in Xterminal		#: After installation of the package being tested, run in Xterminal
	#:<code>dpkg -L packagename</code>		#:<code>dpkg -L packagename</code>

imported_>rotoflex: /* Blockers */

2010-12-14T17:05:21Z

Blockers

← Older revision		Revision as of 17:05, 14 December 2010
Line 68:		Line 68:
	#: To avoid any doubt, the content must not depict explicit sexual activity; depict or endorse acts that cause or are intended to cause excessive pain or suffering; promote or endorse the misuse of alcohol, tobacco, illegal drugs or other addictive substances; promote intolerance or discrimination based on racial, political, ethnic, religious, gender differences or sexuality; promote invasion of rights of privacy; promote gambling or promote illegal activity.		#: To avoid any doubt, the content must not depict explicit sexual activity; depict or endorse acts that cause or are intended to cause excessive pain or suffering; promote or endorse the misuse of alcohol, tobacco, illegal drugs or other addictive substances; promote intolerance or discrimination based on racial, political, ethnic, religious, gender differences or sexuality; promote invasion of rights of privacy; promote gambling or promote illegal activity.
	# Working features, broken functionality or reproducible crashes		# Working features, broken functionality or reproducible crashes
	#: There is some functionality that does not work or produces crash. Report the problem in the application's bug reporting database. There you can provide a list of steps that lead to the problem.		#: There is some functionality that does not work or produces a crash. Report the problem in the application's bug reporting database. There you can provide a list of steps that lead to the problem.
	# Missing announced features		# Missing announced features
	#:Core features advertised through UI or product page don't work or are missing.		#:Core features advertised through UI or product page don't work or are missing.
	# Optification		# Optification
	#: An application is correctly "optified" if all files created during the normal operation of the application are created and stored in /opt or MyDocs, since these directories are not stored on the root filesystem.		#: Test using the ''df'' command:
	#: If the application or its dependencies ignore the recommendation to [[Documentation/Maemo 5 Developer Guide/Packaging, Deploying and Distributing/Installing under opt and MyDocs\|use the eMMC to install as much files as possible]], filling the root partition with 500kb or more, this can be considered a blocker.		#: In Xterminal, run
	~~#: <code>dpkg -L packagename</code> shows all files installed by a package.~~ Also see [[Opt Problem]].		#:<code>df -h \| grep -e rootfs -e mmcblk0p2</code>
	~~#: Feel free to add those packages to [[Opt_Problem/Non-Optified_packages]].~~		#: before and after you install the package being tested, and see how much the usage of each partition changes. If more than, say, 300 kb (0.3 MB) are added to usage of rootfs, it's too much and the package needs to be optified.
			#: Test using the ''dpkg'' command:
			#: After installation of the package being tested, run in Xterminal
			#:<code>dpkg -L packagename</code>
			#: It will show all the files installed by the package. Inspect the result to confirm that some of the files are placed in the /opt or MyDocs directories.
			#: Please add those packages which fail optification to the list at [[Opt_Problem/Non-Optified_packages]].
			#: ''Developers: An application is correctly "optified" if all files created during the normal operation of the application are created and stored in /opt or MyDocs, since these directories are not stored on the root filesystem. If the application or its dependencies ignore the recommendation to [[Documentation/Maemo 5 Developer Guide/Packaging, Deploying and Distributing/Installing under opt and MyDocs\|use the eMMC to install as much files as possible]], filling the root partition with 500kb or more, this can be considered a blocker. Also see [[Opt Problem]].''
	# System performance compromised		# System performance compromised
	#: Running the application visibly affects the performance and responsiveness of the system, either through specific actions or leaving the application open/running during a long period of time.		#: Running the application visibly affects the performance and responsiveness of the system, either through specific actions or leaving the application open/running during a long period of time.

imported_>dneary: /* Blockers */ Switch from sections to a numbered list - makes the TOC simpler & fulfills the requirement to number criteria.

2010-12-14T13:32:06Z

Blockers: Switch from sections to a numbered list - makes the TOC simpler & fulfills the requirement to number criteria.

← Older revision		Revision as of 13:32, 14 December 2010
Line 62:		Line 62:
	#: It needs to be [[Packaging#Bugtracker location\|identified in the http://maemo.org/packages/ page]].<br> http://bugs.maemo.org/ is the preferred option.		#: It needs to be [[Packaging#Bugtracker location\|identified in the http://maemo.org/packages/ page]].<br> http://bugs.maemo.org/ is the preferred option.
	#: A missing bugtracker is NOT a blocker as of 01.09.2010, but use of bugtracking facilities IS strongly recommended.		#: A missing bugtracker is NOT a blocker as of 01.09.2010, but use of bugtracking facilities IS strongly recommended.
	#:''Developers: If the bugtracker field is not specified, the address is defaulted to be that of the package itself on maemo.org: ''<nowiki>http://maemo.org/packages/view/<package>(/<version>)</nowiki~~>''<br~~> Resolutions for bugtracking are found at [[/Packaging#Bugtracker_location]]''		#:''Developers: If the bugtracker field is not specified, the address is defaulted to be that of the package itself on maemo.org: <nowiki>http://maemo.org/packages/view/<package>(/<version>)</nowiki> Resolutions for bugtracking are found at [[Packaging#Bugtracker_location]]''
	# Licensing and legal issues		# Licensing and legal issues
	#: Evident licensing problems or copyright violation. The open source licenses need to be compliant. It needs to be clear that the product is not officially supported by Nokia, Maemo or other commercial entities and trademarks.		#: Evident licensing problems or copyright violation. The open source licenses need to be compliant. It needs to be clear that the product is not officially supported by Nokia, Maemo or other commercial entities and trademarks.

imported_>dneary: /* Blockers */

2010-12-14T13:30:43Z

Blockers

← Older revision		Revision as of 13:30, 14 December 2010
Line 59:		Line 59:
	A "blocker" is any item not met in the QA checklist. If any one of the criteria in the QA checklist is not met, the application cannot go into Extras. A Critical bug at http://bugs.maemo.org/ against the application is required to justify your thumbs down.		A "blocker" is any item not met in the QA checklist. If any one of the criteria in the QA checklist is not met, the application cannot go into Extras. A Critical bug at http://bugs.maemo.org/ against the application is required to justify your thumbs down.

	~~===== 1.~~ Lack of bug reporting database ~~=====~~		# Lack of bug reporting database
			#: It needs to be [[Packaging#Bugtracker location\|identified in the http://maemo.org/packages/ page]].<br> http://bugs.maemo.org/ is the preferred option.
	It needs to be [[Packaging#Bugtracker location\|identified in the http://maemo.org/packages/ page]].<br>		#: A missing bugtracker is NOT a blocker as of 01.09.2010, but use of bugtracking facilities IS strongly recommended.
	http://bugs.maemo.org/ is the preferred option.		#:''Developers: If the bugtracker field is not specified, the address is defaulted to be that of the package itself on maemo.org: ''<nowiki>http://maemo.org/packages/view/<package>(/<version>)</nowiki>''<br> Resolutions for bugtracking are found at [[/Packaging#Bugtracker_location]]''
			# Licensing and legal issues
	A missing bugtracker is NOT a blocker as of 01.09.2010, but use of bugtracking facilities IS strongly recommended.		#: Evident licensing problems or copyright violation. The open source licenses need to be compliant. It needs to be clear that the product is not officially supported by Nokia, Maemo or other commercial entities and trademarks.
			# Illegal or dubious content
	{\|		#: To avoid any doubt, the content must not depict explicit sexual activity; depict or endorse acts that cause or are intended to cause excessive pain or suffering; promote or endorse the misuse of alcohol, tobacco, illegal drugs or other addictive substances; promote intolerance or discrimination based on racial, political, ethnic, religious, gender differences or sexuality; promote invasion of rights of privacy; promote gambling or promote illegal activity.
	\|''Developers:		# Working features, broken functionality or reproducible crashes
	If the bugtracker field is not specified, the address is defaulted to be that of the package itself on maemo.org: ''<nowiki>http://maemo.org/packages/view/<package>(/<version>)</nowiki>''<br>		#: There is some functionality that does not work or produces crash. Report the problem in the application's bug reporting database. There you can provide a list of steps that lead to the problem.
	Resolutions for bugtracking are found at ~~http://wiki.maemo.org~~/Packaging#Bugtracker_location''		# Missing announced features
	\|}		#:Core features advertised through UI or product page don't work or are missing.
			# Optification
	~~===== 2.~~ Licensing and legal issues ~~=====~~		#: An application is correctly "optified" if all files created during the normal operation of the application are created and stored in /opt or MyDocs, since these directories are not stored on the root filesystem.
			#: If the application or its dependencies ignore the recommendation to [[Documentation/Maemo 5 Developer Guide/Packaging, Deploying and Distributing/Installing under opt and MyDocs\|use the eMMC to install as much files as possible]], filling the root partition with 500kb or more, this can be considered a blocker.
	Evident licensing problems or copyright violation. The open source licenses need to be compliant. It needs to be clear that the product is not officially supported by Nokia, Maemo or other commercial entities and trademarks.		#: <code>dpkg -L packagename</code> shows all files installed by a package. Also see [[Opt Problem]].
			#: Feel free to add those packages to [[Opt_Problem/Non-Optified_packages]].
	~~===== 3.~~ Illegal or dubious content ~~=====~~		# System performance compromised
			#: Running the application visibly affects the performance and responsiveness of the system, either through specific actions or leaving the application open/running during a long period of time.
	To avoid any doubt, the content must not depict explicit sexual activity; depict or endorse acts that cause or are intended to cause excessive pain or suffering; promote or endorse the misuse of alcohol, tobacco, illegal drugs or other addictive substances; promote intolerance or discrimination based on racial, political, ethnic, religious, gender differences or sexuality; promote invasion of rights of privacy; promote gambling or promote illegal activity.		# Power management issues
			#: Battery life is diminished beyond acceptable standards by having the application running in the background or just through normal use. The use case of a full day without charging should not be compromised.
	~~===== 4.~~ Working features, broken functionality or reproducible crashes ~~=====~~		#: If specific applications are explicitly using a lot of power to boost speed or performance (for instance games, Wi-Fi heavy apps) then they must warn the user explicitly with a first boot banner.
			# Security risks
	There is some functionality that does not work or produces crash. Report the problem in the application's bug reporting database. There you can provide a list of steps that lead to the problem.		#: The main security risks are financial damage, access to private data and harm to device components. If you find such risk in an application then you need to [http://maemo.org/community/security/ report] it and the app can't be uploaded to Extras until a deeper analysis has been done with favourable results.

	~~===== 5.~~ Missing announced features ~~=====~~

	Core features advertised through UI or product page don't work or are missing.

	~~===== 6.~~ Optification ~~=====~~

	An application is correctly "optified" if all files created during the normal operation of the application are created and stored in /opt or MyDocs, since these directories are not stored on the root filesystem.

	If the application or its dependencies ignore the recommendation to [[Documentation/Maemo 5 Developer Guide/Packaging, Deploying and Distributing/Installing under opt and MyDocs\|use the eMMC to install as much files as possible]], filling the root partition with 500kb or more, this can be considered a blocker.

	<code>dpkg -L packagename</code> shows all files installed by a package. Also see [[Opt Problem]].

	Feel free to add those packages to [[Opt_Problem/Non-Optified_packages]].

	~~===== 7.~~ System performance compromised ~~=====~~

	Running the application visibly affects the performance and responsiveness of the system, either through specific actions or leaving the application open/running during a long period of time.

	~~===== 8.~~ Power management issues ~~=====~~

	Battery life is diminished beyond acceptable standards by having the application running in the background or just through normal use. The use case of a full day without charging should not be compromised.

	If specific applications are explicitly using a lot of power to boost speed or performance (for instance games, Wi-Fi heavy apps) then they must warn the user explicitly with a first boot banner.

	~~===== 9.~~ Security risks ~~=====~~

	The main security risks are financial damage, access to private data and harm to device components. If you find such risk in an application then you need to [http://maemo.org/community/security/ report] it and the app can't be uploaded to Extras until a deeper analysis has been done with favourable results.

	====Non-blockers====		====Non-blockers====

imported_>rotoflex: /* 1. Lack of bug reporting database */

2010-12-14T13:07:33Z

1. Lack of bug reporting database

← Older revision		Revision as of 13:07, 14 December 2010
Line 61:		Line 61:
	===== 1. Lack of bug reporting database =====		===== 1. Lack of bug reporting database =====

	It needs to be [[Packaging#Bugtracker location\|identified in the http://maemo.org/packages/ page]]. http://bugs.maemo.org/ is the preferred option.		It needs to be [[Packaging#Bugtracker location\|identified in the http://maemo.org/packages/ page]].<br>
			http://bugs.maemo.org/ is the preferred option.

	~~Developers: filling the ''XSBC-Bugtracker'' field in your packages addresses this problem. Please see the [[Tracking bugs in Extras]] wiki page for the range~~ of ~~valid bugtrackers you can use. You can [[Bugs:Adding_Extra_products\|get your bugtracker at bugs~~.~~maemo~~.~~org]]~~, use ~~an external bugtracker (such as a project [[Getting started with Maemo_Garage#Trackers and tasks\|bugtracker at garage.maemo.org]]), or a talk~~.~~maemo.org thread. An e-mail address for small projects (wallpapers, for example) can be used with the following field in the <code>debian/control</code> file of the package:~~		A missing bugtracker is NOT a blocker as of 01.09.2010, but use of bugtracking facilities IS strongly recommended.

	~~XSBC-Bugtracker: mailto~~:~~yourname@example.com~~		{\|
			\|''Developers:
	~~In case~~ the bugtracker field is not specified, the address is defaulted to be that of the package itself on maemo.org: http://maemo.org/packages/view/<package>(/<version>) ~~A missing bugtracker is NOT a blocker as of 01~~.~~09.2010, but use of bugtracking facilities IS strongly recommended~~.		If the bugtracker field is not specified, the address is defaulted to be that of the package itself on maemo.org: ''<nowiki>http://maemo.org/packages/view/<package>(/<version>)</nowiki>''<br>
			Resolutions for bugtracking are found at http://wiki.maemo.org/Packaging#Bugtracker_location''
			\|}

	===== 2. Licensing and legal issues =====		===== 2. Licensing and legal issues =====

imported_>rotoflex: /* QA Improvements */

2010-12-14T12:19:03Z

QA Improvements

← Older revision		Revision as of 12:19, 14 December 2010
Line 7:		Line 7:
	==QA Improvements==		==QA Improvements==

	There is a ongoing QA improvements discussion in talk.maemo.org (see [http://talk.maemo.org/showthread.php?t=41179 here] and [http://talk.maemo.org/showthread.php?t=33317 here]) and the [http://lists.maemo.org/pipermail/maemo-community/2010-January/003834.html mailing list]. One task is to [http://talk.maemo.org/showthread.php?t=42055 establish a testing squad].		There is a ongoing QA (Quality Assurance) improvements discussion in talk.maemo.org (see [http://talk.maemo.org/showthread.php?t=41179 here] and [http://talk.maemo.org/showthread.php?t=33317 here]) and the [http://lists.maemo.org/pipermail/maemo-community/2010-January/003834.html mailing list]. One task is to [http://talk.maemo.org/showthread.php?t=42055 establish a testing squad].
	The results can be found at this wiki page: [[Extras-testing/QA_Checklist/QA_Improvements\|QA improvements]]		The results can be found at this wiki page: [[Extras-testing/QA_Checklist/QA_Improvements\|QA improvements]]

imported_>rotoflex: /* QA Improvements */

2010-12-14T12:09:14Z

QA Improvements

← Older revision		Revision as of 12:09, 14 December 2010
Line 108:		Line 108:

	The main security risks are financial damage, access to private data and harm to device components. If you find such risk in an application then you need to [http://maemo.org/community/security/ report] it and the app can't be uploaded to Extras until a deeper analysis has been done with favourable results.		The main security risks are financial damage, access to private data and harm to device components. If you find such risk in an application then you need to [http://maemo.org/community/security/ report] it and the app can't be uploaded to Extras until a deeper analysis has been done with favourable results.

			====Non-blockers====

			The extras-testing QA process should not be used to delay the release of new applications and updates with minor/normal bugs or enhancement requests. These can be filed as bugs just as usual.

			* Developers are encouraged to adopt the Maemo 5 UI guidelines and optimize their application to finger use. However, lack of Maemo 5 UI optimization is not a reason to block an application in its way to Extras unless it is unusable even with stylus.
			* Maemo 5 offers stock icons covering most regular use cases, but developers can use the icons they prefer as long as they respect copyrights. Broken icons can be a major bug stopping a release to Extras but discussions about beauty/ugliness of a UI are out of scope in the QA process.

	====Warnings====		====Warnings====
Line 117:		Line 124:

	Testers with good technical knowledge are invited to look deeper in the [http://maemo.org/maemo_release_documentation/maemo4.1.x/node16.html Maemo Quality considerations] (((waiting for Fremantle update))) in order to find weak points in applications waiting to be promoted.		Testers with good technical knowledge are invited to look deeper in the [http://maemo.org/maemo_release_documentation/maemo4.1.x/node16.html Maemo Quality considerations] (((waiting for Fremantle update))) in order to find weak points in applications waiting to be promoted.

	~~====Non-blockers====~~

	~~The extras-testing QA process should not be used to delay the release of new applications and updates with minor/normal bugs or enhancement requests. These can be filed as bugs just as usual.~~

	* Developers are encouraged to adopt the Maemo 5 UI guidelines and optimize their application to finger use. However, lack of Maemo 5 UI optimization is not a reason to block an application in its way to Extras unless it is unusable even with stylus.
	* Maemo 5 offers stock icons covering most regular use cases, but developers can use the icons they prefer as long as they respect copyrights. Broken icons can be a major bug stopping a release to Extras but discussions about beauty/ugliness of a UI are out of scope in the QA process.

	===Command Line applications===		===Command Line applications===
Line 130:		Line 130:

	* The application uses the [[Extras-testing/Command_line_applications \| CLI icon]] in the application manager.		* The application uses the [[Extras-testing/Command_line_applications \| CLI icon]] in the application manager.
	* The application [[Extras-testing/Command_line_applications#Description\|description]] clearly says that the application only runs from the command line.		* The application [[Extras-testing/Command_line_applications#Description\|description]] clearly says that the application only runs from the command line.

	==Testing software==		==Testing software==