ETD is now configured to be tested & monitored in Hudson: http://waterhouse.library.emory.edu:8080/hudson/job/ETD/
It took a bit of work to get PHP SimpleTest results into a format that Hudson can process and give detailed, granular results like it does for the Django apps or anything else that natively produces jUnit-style testsuite xml, but now that I’ve figured it out for ETD it should be pretty easy to set up for any other existing PHP code-bases using simpletest for unit testing.
Here’s what I’m using for the build command in the ETD Hudson configuration:
cd trunk/tests/suites
rm -rf test-results
mkdir test-results
php xml_suite.php > test-results/suite.xml
../simpletest/local/clean_simpletest_xml.php test-results/suite.xml
xsltproc ../simpletest/local/simpletest_to_junit.xsl test-results/suite.xml > test-results/TEST-suite.xmlThen, in the post-build options I enable Publish JUnit test result report with this path:
trunk/tests/suites/test-results/TEST-*.xml
A few notes on the what/why/how.#
Simpletest has an xml reporter that can be extended, but it’s rather limited– it outputs the xml as the tests run, so you don’t have information on total pass/failure until the end. I extended the default XmlReporter to make one minor modification– timing each test method as the test runs. To use this, include simpletest/xmltime.php and use XmlTimeReporter to run the report you wish to pass off to Hudson. For ETD, I created a new test suite script that includes all of the other unit test groups and uses the XmlTimeReporter.
I also discovered that if there are any errors, warnings, etc. before the test actually starts, those warnings are displayed before the xml output– making the resulting file invalid xml. I wrote a very minimal php script to clean up anything before the xml prolog and put it into a system-err tag. Then I wrote an xslt to convert the xml generated by simpletest XmlReporter (with my test duration addition) into the testsuite xml format used by jUnit and understood by Hudson. I couldn’t find any actual documentation of this format, just examples, so the output is an approximation and may miss some things, but Hudson is now correctly picking up failed tests and displaying test results for those failures, which I’m happy with (for now at least).
The additions to simpletest that make this conversion possible are now in our local simpletest subversion repository, so any project that is using simpletest should have access to them.
Update January 29, 2010: for anyone who’s interested but doesn’t have access to our subversion, here’s the content of the relevant two files.
xmltime.php#
<?php
/**
* extend default XmlReporter to record & report time to run each test method
*/
class XmlTimeReporter extends XmlReporter {
var $pre;
function paintMethodStart($test_name) {
$this->pre = microtime();
parent::paintMethodStart($test_name);
}
function paintMethodEnd($test_name) {
$post = microtime();
if ($this->pre != null) {
$duration = $post - $this->pre;
// how can post time be less than pre? assuming zero if this happens..
if ($post < $this->pre) $duration = 0;
print $this->_getIndent(1);
print "<time>$duration</time>\n";
}
parent::paintMethodEnd($test_name);
$this->pre = null;
}
}
?>
simpletest_to_junit.xsl:#
<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
version="1.0">
<!-- convert xml output generated by simpletest xml into junit xml format -->
<xsl:output method="xml"/>
<xsl:template match="/">
<xsl:apply-templates select="run/group"/>
</xsl:template>
<xsl:template match="run/group">
<testsuite>
<xsl:attribute name="errors"><xsl:value-of select="count(.//exception)"/></xsl:attribute>
<xsl:attribute name="failures"><xsl:value-of select="count(.//fail)"/></xsl:attribute>
<xsl:attribute name="tests"><xsl:value-of select="count(.//test)"/></xsl:attribute>
<xsl:attribute name="name"><xsl:value-of select="name"/></xsl:attribute>
<xsl:attribute name="time"><xsl:value-of select="sum(//time)"/></xsl:attribute>
<xsl:apply-templates select=".//case/test"/>
<xsl:copy-of select="//system-err"/>
</testsuite>
</xsl:template>
<xsl:template match="case/test">
<testcase>
<xsl:attribute name="classname"><xsl:value-of select="../name"/></xsl:attribute>
<xsl:attribute name="name"><xsl:value-of select="name"/></xsl:attribute>
<xsl:attribute name="time"><xsl:value-of select="time"/></xsl:attribute>
<xsl:apply-templates select="fail"/>
<xsl:apply-templates select="exception"/>
</testcase>
</xsl:template>
<xsl:template match="fail">
<failure><xsl:attribute name="message"><xsl:value-of select="."/></xsl:attribute>
<!-- content is for stacktrace; not available / broken out by simpletest -->
</failure>
</xsl:template>
<xsl:template match="exception">
<!-- assuming same format as fail -->
<error><xsl:attribute name="message"><xsl:value-of select="."/></xsl:attribute>
<!-- content is for stacktrace; not available / broken out by simpletest -->
</error>
</xsl:template>
</xsl:stylesheet>