jlaska

Adam and I have jumped into a higher gear when it comes to mapping out test days for Fedora 13. Given the large number of features in Fedora 11 and 12, I was expecting a reprieve for Fedora 13.  While it's not the largest release in terms of the features, it's not the smallest either.

• Fedora 13 - 31 features
• Fedora 12 - 42 features
• Fedora 11 - 52 features
• Fedora 10 - 28 features

Unfortunately, there are only so many events the QA team can scope out for a release.  We need community input.  Not only for planning events, but also during execution.  A successful test day requires prep time, and a presence during the event to help triage issues and remove obstacles.  If either is lacking, the event is guaranteed to be forgotten.

So, if there is a feature or focus area you'd like to see test day coverage for during Fedora 13 ... please let the QA team know.  Proposing a test day is pretty straight forward, for details check out the wiki.  Also, take a look at current test day calendar for a list of scheduled events.

For convenience, you can propose a test day using the form below (FAS login required):

Not too much trouble traveling to FUDCon Toronto. The flight out of Raleigh experienced some mechanical issues. Will Woods and I were transferred to Cleveland for a long layover. That gave us plenty of time to catch up on AutoQA. Since I like lists ... some might say too much ... so here it goes.

Day#1 - Barcamp

• boot.kernel.org (or killing CD ISOs w/ the network) -- see log -- Interesting talk on removing the need for media-based installs. Basically, you download a small image and store it on a USB key, tftpserver or optical media (so not really getting rid of the CD's). When booted, that smaller image boots off the network and offers installation of just about anything (Fedora, CentOS, Ubuntu, Gentoo, etc...). Mike McGrath was keen on implementing this within Fedora infrastructure to provide another easy method for users to get involved and help test.
• Fedora QA: What we do, and how you can help -- see log -- Great talk by Adam on what the Fedora QA team does, and how to get involved. Some participants were surprised that the blocker bug process was so open. That's certainly a strength with our current process in that it doesn't require l33t permissions to engage and add value. A big 'hooray!' went out for Kevin Fenzi .. who helped make it even easier to get involved by providing non-destructive nightly live images of rawhide.
• AMQP Qpid -- session not active -- I was hoping to catch this session. However, it seems most of the discussion took place during the previous amqp talk. Instead, I headed over to ...
• Designing the Future of Free Software Operating System User Experiences - GNOME Shell / 3.0 -- see log -- Jon McCann and Colin Walters demonstrated the rationale behind the GNOME shell and gave a quick demonstration (see screencasts) along with screenshots. They're focus on the user experience is interesting.
• AutoQA Beaker Automated Testing -- see log -- Bill Peck and Will Woods discussed recent improvements and the rationale behind the different solutions. Beaker has come a long way since it first jumped into the fedorahosted project scene. Will talked about the initial design goals for the AutoQA project, as well as some upcoming milestones. Bill discussed how beaker is being used now and testing possibilities for Fedora. This is a big area of growth for the QA team. AutoQA was designed for testing that does not require complicated test environments (either dedicated test systems or virtual guests). As our testing needs mature, beaker will likely be involved in helping transition to more complex test infrastructure (multiple test systems, system provisioning, etc...).

Day#2 - Hackfest

• Fedora release criteria discussion with John Poelstra, Adam Williamson, Bill Nottingham and Tim Burke. Our initial thought was the discussion would simply put a bow on the existing pages and wrap-up with the feedback provided leading up to FUDCon (wiki and mailing lists). However, many hours later, we arrived at something that we can all agree with and build on for future releases. For the most part, the release criteria simply restate what we've always done each Fedora release. This time, Adam suggested a clever revamp of the requirements as to not leave too much of a burden on QA for defining policy using our test plans. The current result is something the QA team can execute on for Fedora 13, has the thumbs up from key stake-holders in Desktop, Release Engineering and Development, and leaves room for future growth. I'm pretty jazzed with the results ... https://fedoraproject.org/wiki/Fedora_Release_Criteria.

Day#3 - Hackfest

• Discussion around AutoQA Use Cases with John Poelstra I've been struggling to make sense of the use cases. My intent was that by completing the use cases, we could highlight gaps in AutoQA documentation or process.. What John suggested was keeping them short, sweet and task focused. I cleaned up the initial 'Getting started' cases accordingly. Next step, catch up with AutoQA developer and rpmguard creator, Kamil Paral, to help flesh out more details.
• Spoke with the installer team on their plan for build-time unit tests. David Cantrell and team are working on building unit tests to validate anaconda dependencies at build-time. These are dependencies that normally manifest as install-time failures. This includes libraries such as udev and dbus, binaries like iscsiadm and mdadm, directory locations (/etc/sysconfig/network-scripts) and config files/formats that it depends on. Following a model similar to 'is rawhide broken?', a new AutoQA milestone will likely come from this. The milestone will include a post-git-push watcher to initiate tests whenever anaconda changes are pushed into anaconda.git and another turbogears front end for display and review of test results.
• Happy recipient of guidance from Toshio Kuratomi and Mike McGrath on packaging and deploying autoqa. The previous package was for EL5, after moving to F12 several bugs surfaced related to config files and the reliance on python-cherrypy2 (not the latest cherrypy). With those issues out of the way, next stop ... package review process.
• Discussion with Adam Miller on the possibility of using dogtail to automate tests inside a running live image. Adam, Will (and possibly others) discussed how to automate GUI testing on a live image. Some of the challenges involved getting the test code and dependencies installed on the live image ... without of course changing the environment under test. One clever idea I heard was to use an overlay partition. The other challenge is our general lack of knowledge around dogtail testing. Dogtail has been around for some time, but it's unclear in what state upstream is and how robust of a system it is for GUI testing. Hopefully Adam is able to shed some light on that untapped resource.
• Finalizing the Fedora 13 QA schedule with John Poelstra and Denise Dumas (with input from Jesse Keating). I'm happy with the current QA schedule that John has asked for feedback on. It continues the F-12 trend of having 3 test events leading up to each major milestone (Alpha, Beta, Final). Each milestone will be preceded by the following test runs ... 1) pre-$MILESTONE rawhide acceptance test run, 2) test-compose test run, and 3) release candidate test run. Additionally, the Alpha and Beta will have additional planned rawhide acceptance test runs. I'm confident that the QA team can
• Discussed techniques for using AutoQA to automate the installation test matrix. At different times during the event, Adam, Will and I outlined different techniques for integrating AutoQA and the install test matrix that Liam maintains. I'll catch up with Liam and Hurry to bounce some more ideas around. However, I expect we'll need to get a small group of folks together to collaboratively move things forward so that we have something in place for Fedora 13.

All told, I'm pretty happy with FUDCon Toronto. With one exception, I completed about everything on my todo list. I had hoped to leave FUDCon with a deployed instance of autoqa. However, time was not my friend in that regard. The details of package review are still ahead. Anyone interested in reviewing?

With Fedora 12 on it's way to mirrors and to a Tuesday general availability, I've been watching our internal test instance of israwhidebroken.com. On Saturday, November 14th, Fedora rawhide and Fedora 12 bid each other farewell.

Friday, November 13th, 2009		Saturday, November 14th, 2009
	→

After a quick analysis, the new failures seem to be the result of:

• A possible bug in the repo sanity verification test on ppc (see autoqa-results test output)
• i386 and x86_64 installs are failing while downloading the file images/install.img
  

                        ┌────────────┤ Error ├────────────┐                       
                        │                                 │                       
                        │ Unable to retrieve              │                       
                        │ http://download.fedoraproject.o │                       
                        │ rg/pub/fedora/linux/development │                       
                        │ /x86_64/os//images/install.img. │                       
                        │                                 │                       
                        │             ┌────┐              │                       
                        │             │ OK │              │                       
                        │             └────┘              │                       
                        │error reading header: cpio: read failed -│Success                
                        │                                 │                       
                        └─────────────────────────────────┘                       
  
  

  Further digging shows there may be a networking problem related to setting up the default route. See 537870 - Failure to download images/install.img - error reading header: cpio: read failed

The existing ppc failures observed prior to the change in rawhide are due to our current test harness. Libvirt --accelerated installs are not supported on ppc64 at this time, so the installation test is failing as expected in that environment.

Sure, it's not only the answer to the ultimate question of life, the universe, and everything, it's also the next milestone for the QA team on the fedora quality schedule. So, while beta testing and triage of reported issues continues, fixes to blocker bugs (and non-critical path components) are arriving daily. In case you don't regularly follow fedora-test-list, Liam Li sent out a request for testing today. If you'd like to get involved, here's a few areas in need of extra eyes.

Test the latest installer

With a spare laptop or workstation (or just a virtual guest), you can provide feedback on how the installer is doing by sharing results on https://fedoraproject.org/wiki/Test_Results:Fedora_12_Pre-RC_Install. The installer continues to improve since the beta. Take a look at recent anaconda changes for a change that may impact your system(s).



Test a bug fix

Help a maintainer by verifying a recent bug fix.  A good place to start is by choosing a MODIFIED bug from the F12Blocker list.  The list of MODIFIED bugs will be growing as issues are triaged and patches come in.

This Thursday, October 22, 2009 #fedora-test-day will be host to another test day focused on Power Management improvements in Fedora 12. This round features improvements to the tuned service along with a merge of another performance monitoring tool ktune. Testing is requested against several environments, including X, laptops and single-user mode.

When we first started the Test Day format in Fedora, there was hope that eventually this would provide a forum for test scripts and automation. If you participated in the previous Power Management event, you may notice a few improvements along these lines. Phil Knirsch, Marcela Maslanova, Petr Lautrbach and Jiri Skala have combined forces (think transformers) and are offering a test day package available for download that contains several scripts to aid testing and packaging test results. According to the wiki, once you follow the instructions to install the required package, running a test should be as simple as running the command: testday-run-testname. Following each test, package up the results with the testday-pack-results command, and upload the resulting .tar.gz file.

I'm looking forward to joining and hope others have time to share their power management results. As always, check out the test day wiki for more information.

"Really, a RAID test day? Thanks sounds boring." Well, if you don't like baby seals, or don't care about disk redundancy, than you might not be interested.

But the good news about a Fedora Test Day is that it offers an opportunity to test drive the upcoming release. Whether testing the focus of the day, a previous test day topic, or just a use case/environment you are familiar with, any feedback is good feedback. To get started ...

1. Download an image - Thanks to Kevin Fenzi, live images of Rawhide are built daily and available for download. If a live image doesn't strike your fancy, download a rawhide boot.iso instead.
2. Prepare your media - There are several different ways you can make use of the live image, including burning to a CD/DVD, writing to a USB stick, or booting under a virtual guest. I'm lazy, so unless my tests require physical hardware ... I always go virtual.
3. Tell us what works - Join IRC and tell us what works and what could be improved. File a bug.

Additionally, if you're interesting in improving the install experience for RAID users, please do stop by. Members of the installer development team are at the ready and in need of your feedback. If you have a system with BIOS RAID or hardware RAID, or just a frequent user of software RAID, your feedback is important.



See you on #fedora-test-day on irc.freenode.net this Thursday, October 8, 2009.

In case you missed the announcement on fedora-test-list ...

This week's Fedora Test Day will focus on exercising the Fedora 12 mkinitrd/nash replacement ... drumroll ... Dracut. If you find yourself wondering, "what the heck is that?" The following snippet from the fedoraproject wiki best describes the change.

With Dracut, the initramfs has one purpose in life - getting the rootfs mounted so that we can transition to the real rootfs. This is all driven by device availability. Therefore, instead of scripts hard-coded to do various things, we depend on udev to create device nodes for us and then when we have the rootfs's device node, we mount and carry on.

It is worth noting that dracut will impact every Fedora 12 user (and potentially users of other distros). As a result, there are several ways to get involved with testing, ranging from basic to more advanced:

1. Download the live image and boot on your system(s)


2. Update your existing Fedora 11 (or Fedora 12 Alpha) system(s) to use dracut, and reboot


3. Install Fedora 12/Rawhide targeting specific root partitioning schemes (iSCSI, dmraid, RAID+lvm+luks, etc...)

I invite you to join #fedora-test-day this Thursday, August 27 2009 to help test dracut. At the very least, providing feedback on the live image will be much appreciated. Additional test feedback per the wiki
guidelines is also encouraged.

Stay tuned for more details at https://fedoraproject.org/wiki/Test_Day:2009-08-27_Dracut.

If you follow fedora-test-list, you may have seen recent discussion around improving the Category:Debugging wiki pages. Several contributors have already created really helpful pages intended to guide testers who uncover defects and improve the quality of new bug reports. Great examples include:

• https://fedoraproject.org/wiki/Xorg/Debugging - thanks François
• https://fedoraproject.org/wiki/Dracut/Debugging - thanks Jóhann

I thought this might be a good opportunity to get involved with the QA team. If you're interested in helping improve the quality of bug reports, please read on ...

Top reported components

First, Christopher Beland highlighted several high defect components that are missing Debugging guides. This includes components like yum, rpm, openoffice.org and evolution. If you're interested in a challenge, helping others self-diagnose failures will go a long way, and I'm sure the maintainers will be pleased.

Your favorite application

Alternatively, if there is an application you use and know really well?  If you've had to spend time learning the in's and out's of this application, I bet others will need to do the same thing.  I encourage you to share your knowledge and create a new debugging page.

Getting started

Once you've selected the application ... use the steps below as a guide:

1. Check if a debugging guide exists for your application already  (look here)
2. Login to the wiki and create new page ... https://fedoraproject.org/wiki/Bug_info_myfavoritepackage
3. Is there an upstream debugging guide? If so, add a link ...
4. Is there a list of common issues to check against first?
5. Should any files or output logs be attached to the bug report? For example, if reporting a bug against pulseaudio, you would need to attach: pulseaudio -vvvvv
6. Be sure to include your page in the Debugging category by including the text: [[Category:Debugging]].
7. Finally, let us know what you've created ... join a QA meeting or send a message to fedora-test-list.

That wasn't too bad was it?  Thanks for helping! 

The release of Fedora 11 is just around the corner.  Hopefully by now you've taken a milestone (Alpha, Beta or Preview) for a test drive, maybe participated in a Test Day, contributed feedback to fedora-test-list@redhat.com, or filed a few bugs.  

At this stage of the release we are most interested in identifying the high and urgent severity defects.  If unsure, take a look at the current draft definitions of defect priority and severity.  These are issues when you look back, you might say, "geez, that was embarrasing" or my favorite, "does anyone test this?"  With 52 new features and roughly 7000 bugs filed against rawhide since January 2009, there will likely be quite a few of these issues.   But there is still time left to find them, smooth out the rough spots and document any remaining pain points.

Here's how you can help ...

For the adventurous, including folks with spare systems or hardware virtualization support ... 

1. Give rawhide an install - the installer backend has gone through quite an overhaul.  There remain several conditions where installations can be rough, including when handling previous disk layouts.
2. Find a match? - let us know what issues you encounter and what worked for you by updating the wiki.

Only have one system or a laptop for testing?

1. Grab a recent live image (64 bit) - find instructions for using the live images here.
2. Do what you normally do - listen to some ogg's, watch a few videos on youtube, chat with friends
3. Re-run a F11 test day - test days focus on upcoming Fedora features and often include a batch of test cases available for execution.  It's easy to get started using a live image or an installed system.  Select a test day and run the test cases provided.
4. Test drive a specific feature - each feature comes with several test guidelines.  Pick something that's near and dear to you and take it for a spin.

Have an issue? 

1. Examine the common F11 bugs - This page is intended capture the most frequently reported defects/issues observed in Fedora 11.  Adam Williamson recently added more content and some clean-up to make future contributions easier.  If you experience a problem, check there first.  Don't see your issue on the page?
2. Post to fedora-test-list@redhat.com - Share your findings.  If something doesn't look right, or behave as expected ... tell us.
3. Instant gratification - Share your experiences with fellow users live on #fedora.

Happy testing!

Adam Williamson said it best, anaconda storage testing is a tough thing to get folks excited about. It doesn't rotate on a cube, slowly spew solar flares or read your fingerprint ... but it sets the stage for everything that comes after you first boot up a Fedora 11 install. Naturally, getting things right is important ... and failing during installation is a quick way to ward off potential contributors.

It's an understatement to say the weeks leading up to the test day were very busy for the anaconda development team.  To reduce the disruption to rawhide users, the storage rewrite changes were self-contained in a rather large updates.img.   Which is something considering "all of anaconda's code for creating and configuring disk partitions, LVM, mdraid, dmraid, multipath, iSCSI, LUKS devices, and filesystems is being rewritten." [1]

The day began by emphasizing the raw in rawhide.  Mirrors were still updating which caused confusion while testing patches and raised the question ... what rawhide am I testing?  What seemed to help here was:

1. Patience - the mirrors eventually had the latest and greatest content ... but there's no GREEN light that tells you a mirror is up to date
2. .treeinfo - this file in the base directory of any installation tree contains a SHA256 hash for every file included in the tree.  Comparing it's contents with the main download site can be helpful.

With that behind us the rubber met the road.  The focus for the day was excercising the most common use cases to reduce the number of obvious warts when the changes hit rawhide.  This includes, but not limited to, most every operation you can perform from the disk partitioning step seen below.



At the time of this post, testing that fell in the category of "things not expected to work" included:

• dmraid
• partitioning via kickstart
• discovery of existing mdraid arrays
• user interaction through anaconda screens (clicking Next/Back)
• hardware raid controllers that have nonstandard device paths, like cciss (untested, not expected to work)
• resizing of devices/filesystems
• bootloader setup on multi-boot systems (untested)
• ppc?

The event was very busy and only possible thanks to the hard work of folks like Dave Lehman, Chris Lumens, and Joel Granados.  Throughout the day they were able to traige, patch, review and integrate hotfixes into an updates.img for testers.  Also many thanks to testers for coming armed with interesting storage environments (j-rod and refried). 

While not everything was filed in bugzilla, the list of formerly recorded issues found includes:

488722	Properly align encrypted LV - LUKS device
488738	KeyError: 'LVM2_VG_UUID'
488743	ImportError: No module named partRequests
488747	NameError: global name 'iutil' is not defined
488772	[StorageRewrite] Traceback during kickstart install
488800	[StorageRewrite] DeviceException: Could not find device for path /dev/sdb
488807	[StorageRewrite] PartitionException: no type specified
488812	[StorageRewrite] TypeError: 'NoDevice' object is not iterable
488859	[StorageRewrite] TypeError: argument 2 must be string, not None
488860	[StorageRewrite] Adding iSCSI device fails - TypeError: 'Storage' object is not iterable
488861	[StorageRewrite] LVMError: pvremove failed for /dev/sda1 - custom partitioning with 1 local and 1 iSCSI disk fails

At the end of the test day, several things were very clear:

1. We need more interesting storage environments to throw at anaconda (SAN's, FC, multiple disks, dmraid, mdraid, iSCSI)
2. We need to provide another collaborative forum for testing #1
3. I heart virt-manager and kvm

Stay tuned to the F11 Test Day schedule for information on Anaconda Storage Test Day part#2.  In the meantime, please feel free to take Rawhide for a spin on a spare laptop/workstation/VM.