Pegasos ODW Project
Toolchain regression testsin category Operating Systems
proposed by tbm on 1st June 2006 (accepted on 30th September 2004)
Project Summary
SummaryThe aim of my project was to perform regular tests of the toolchain for PowerPC. Tools are often only tested on mainstream architectures (in particular x86) and problems on other platforms are only discovered later. By doing regular tests from CVS/SVN and providing feedback to developers, such issues can be identified and removed sooner. As part of this project, various tools which allow the recompilation of the Debian archive were improved. During the recompilation of the whole Debian archive (with about 6000 packages that require compilation) many problems were identified, some of which were specific to PowerPC. Problems found include bugs both in the compiler as well as in individual packages.
Background
While interpreted languages are gaining popularity, the majority of packages in Debian (about 60% of over 10,000 source packages) need compilation using standard tools, such as GCC and binutils. These tools are very architecture-specific and can therefore contain bugs which only affect a specific architecture. While the PowerPC platform is gaining popularity, many developers still only have access to x86 (or, nowadays, x86_64) hardware. Rigorous tests of the toolchain on PowerPC allows the identification of many bugs and those can be fixed before a new version is released if feedback to developers is provided early enough.
The idea of this project was to use Debian, with about 6000 packages, as a regression test suite for GCC and binutils. In this process, a chroot environment is setup in which packages are compiled using unreleased versions of the toolchain. In order to carry out this work, a number of preliminary work was needed. Therefore, quite some time was spent on getting experience with and submitting bugs and feature requests about sbuild and schroot. sbuild is the tool used by Debian to compile Debian packages from source. schroot is a fairly new program which makes it easy to setup a chroot environment in which packages can be compiled and tested.
After a proper environment was created and some simple scripts written to handle the recompilation of the whole Debian archive, the process of rebuilding all packages was started. One problem was that the CPU and RAM of the Pegasos2 were quite limited, so rebuilding the whole archive (with about 6000 packages) took a long time. The problem here is that both the toolchain and Debian change: if building the whole archive takes 2 weeks, by the time you are done there have been many changes in the toolchain already and you\'re using a version that is two weeks old. Also, and probably more important, Debian changes on a daily basis, which makes it much harder to track failures and to see why a package didn\'t build.
Nonetheless, this project led to many interesting results. In particular, I compiled the whole Debian archive with GCC 4.1, first on AMD64 and MIPS and later on PowerPC. This was widely announced because many GCC 4.1 specific bugs in packages were found. A number of compiler bugs were also found in this exercise (see the appendix for a sample of bugs).
Problems
There were a number of problems or issues, some of which still need better solutions:
- Finding failures in the compiler (GCC) is easy because they are obvious (the compiler stops, usually with an error saying \"internal compiler error\"). Unfortunately, findings errors in binutils is often much harder since bogus can lead to obscure errors where you don\'t expect them.
- Most failures are caused by generic build failures which have nothing to do with the new version of the compiler or binutils, or with PowerPC. It takes a lot of time to weed out those errors and to find those which are actually interesting. A better framework for keeping track of failures is needed.
Future work
I have submitted an application for the OSW program since the faster machine would allow me to do more frequent tests of newer compilers and other tools. If I receive an OSW, I intend to use the Pegasos2 for Debian and debian-installer development; otherwise I will continue to use the Pegasos2 for toolchain tests. As mentioned above, one of the main problems (since it\'s the most time-consuming tasks) is to keep track of failures (in particular in order not to waste time with known failures). Roland Stigge has written DARTS, a tool to keep track of regressions, and I intend to take a look at it.
Appendix: list of reported bugs
PowerPC specific bugs reported to Debian:
- #362410: fp-compiler: broken on powerpc: /usr/bin/ppcppc segfaults
- #362320: liboil0.2: doesn\'t build on powerpc: PIC register \'30\' clobbered in \'asm\'
- #366965: vlc: doesn\'t build on powerpc: invalid parameter combination for AltiVec intrinsic
- #366753: ppmd: segfaults on powerpc when built with -O2 and gcc 4.1+
- #362307: gcc-4.1: ICE on powerpc: error in extract_insn, at recog.c:2084
Bugs reported against sbuild, a tool for building Debian binary packages from sources:
- #369707: ugly perl messages when there\'s no apt-cache
- #369718: ugly error when $log_dir doesn\'t exist
- #355072: uses apt lock file in main system
- #363351: checkchroot gives \"uninitialized value\" warning
- #355040: \"Toolchain package versions\" only shown when there are build-depends
- #356785: doesn\'t like *-security? \"..doesn\'t appear to be a chroot\"
- #355035: updatechroot: calls chrapt without setting $PATH
- #355037: updatechroot foo assumes chroot is in /foo
- #355038: updatechroot doesn\'t cope with file:/ deb sources lines in the chroot
- #355043: log_command() is too verbous, should be a debug message
- #355802: no error message when sbuild cannot find chroot
- #358668: says debootstrap when it means _c_debootstrap
- #358717: file output buffered in interactive mode?
- #360058: please don\'t expect ref-* to be sorted
- #355034: please don\'t hardcore ~/build
Bugs reported against schroot, a tool that makes it easy to use different chroot environments:
- #362205: FTBFS with GCC 4.2: no matching function for call
- #361108: first part of README.Debian isn\'t Debian specific
- #361113: \"no such chroot\" error isn\'t very helpful
- #366608: please don\'t complain if alias name equals chroot name
- #363475: please change to $HOME if $PWD doesn\'t exist in chroot
Test reports
Project Blog Entries
