Bits in Motion

Quickie

2008-09-19T02:01:00.001-07:00

Sorry for the long silence. I've been super busy lately.

Anyways... I'll tantalize you with this: Gallio v3.0.4 is almost ready to ship. We're just cleaning up some loose ends and putting on the finishing touches.

I am really looking forward to making the final announcement. There's lots of cool (and useful) new stuff on the way!

How to Find Me

2008-08-22T14:30:00.001-07:00

I live in the East Bay and work in San Francisco. I'm always keen on meeting smart and interesting people. If you're local and would like to hang out, shoot me an email.

Upcoming Conferences

November 17th-21st: QCon

On Thursday I'll be giving a talk about Gallio and MbUnit at QCon in the ALT.Net track. I'm planning to talk about the architecture of Gallio and demonstrate a few ways that it can be extended to build new test tools. I will also discuss how MbUnit integrates with Gallio and how other test frameworks can do the same.

November 8th-9th: Silicon Valley Code Camp

I'm planning to attend Silicon Valley Code Camp, a free conference in the Bay Area. I've also signed up to give a talk similar to the one at QCon. Looks like a great opportunity to meet more local talent.

Community Events

I sing Tenor with San Francisco City Chorus, a local amateur choir. It's a lot of fun and we're always looking for new members.

My wife and I also sing with the Mount Diablo Unitarian Universalist Church. I'm still not sure about this whole "church" thing, but it's a nice community of ~~hippies~~ genuine, empathetic, forward-thinking, individualists.

I used to do English Country dancing with the Merrie Pryanksters at the California Renaissance Faires and do Irish dancing at the Starry Plough Pub in Berkeley. Lately my participation has been superceded by other activities but I still hang out.

Occasionally you'll find me hanging out with other Canadian members of the Digital Moose Lounge at a hockey game or Canada Day picnic.

Diffs in Assertion Failures

2008-08-21T06:25:00.001-07:00

Just one of many features I've been working on lately.

More to come... now it's wayyy past bedtime.

Spot the Case-Change

Something More Complex

I'm considering using strike-through to give added emphasis to changed or deleted text from the expected value. What do you think?

Technorati Tags: gallio,mbunit

Getting Stuff Done

2008-07-15T16:12:00.001-07:00

Liz and I have to move next month. The landlady simply wants to move back into her house so she will not renew the lease.

Thus begins the process of dismantling The Cave and going somewhere else. But where? We have searched all over for homes to buy or to rent over the past two months and have not found anything acceptable yet.

A small part of me would like nothing more than for someone else (perhaps my employer) to take care of these details so I can move on with the process of getting stuff done. Foolishness! Alas.

Attribution and the Golden Rule

2008-07-09T16:44:00.001-07:00

Gallio uses the Apache License, Version 2.0.

Today I discovered a breach of that license wherein an author of another Open Source project copied code from Gallio and removed the copyright notices and attribution.

I have contacted the author and hope to resolve the matter amicably.

Giving Credit Where it is Due

You may have noticed that I take pains to acknowledge the work of others when I borrow their ideas. For example, I credit xUnit.Net for their idea of abstracting the reflection layer (among other things). That simple idea has shaped a fair bit of the Gallio test object model so I'm grateful to have encountered it.

On an even grander scale, there are many others whom I credit for the development of unit testing and the idiomatic form it has taken in .Net. MbUnit borrows heavily from NUnit, JUnit and other frameworks, not to mention the work of Kent Beck and other founders.

Professional Courtesy

Recently I have noticed others borrowing ideas from Gallio and MbUnit. I think it's great! Imitation is the sincerest form of flattery. But I do feel something is missing.

Open Source software does not get built in a vacuum. There are many people involved in the conception, development, promotion and distribution of software. We depend crucially upon the goodwill of others and so it behooves us to act with humility and respect for one another and our respective projects.

The more widely we disseminate the good ideas of others, the more positive impact they will have.

Thanks.

Technorati Tags: gallio,mbunit,values

Announcing Gallio Alpha 3 Update 3

2008-06-17T01:57:00.001-07:00

Here comes the next update to Gallio and MbUnit v3.

This is an interim release with a few new features and updates.

Download here: (link)

New and Noteworthy

ReSharper v4.0 Final Integration

This update now supports the final release of ReSharper v4.0.

Enjoy!

MbUnit [Factory] Attribute

Suppose we've got a function like this:

To test it, we'll need some different kinds of objects.

So let's use a factory:

That's quite simple, assuming we only want to use single objects for each run.

But we can do better. We can also return arrays of objects:

However, this doesn't test the case where the value was null. In that case, the formatter function is supposed to throw an ArgumentNullException.

We can simulate the behavior of the [ExpectedException] attribute by returning additional metadata along with the data item. Of course, we can include other metadata too, like test descriptions, and other data.

MbUnit [Repeat] and [ThreadedRepeat] Attributes

Sometimes when testing features with non-deterministic behavior, it's useful to be able to run a test repeatedly.

Use the [Repeat] and [ThreadedRepeat] attributes in MbUnit to run tests sequentially or concurrently.

Since the attribute is a test decorator, it may be used with most any kind of test, including data-driven tests. It can also be used on fixtures, if you like.

Each run appears as a separate item in the report. They are considered dynamic test steps that are nested within the test.

CSV Files with Metadata

Several people have requested the ability to include additional metadata in rows that are specified by CSV files using the [CsvData] attribute in MbUnit.

Now you can.

Just enclose the name of the column that contains metadata with '[' and ']' and that column will be exposed as a metadata item whose key is the name of the column.

So given this test:

And this data file:

Item, Price, [ConsumedBy]
Apples, 1.00, Worm
Bananas, 1.50, Monkey
Cookies, 2.00, "Cookie Monster"

We get this output in the report:

Notice that the metadata appears in the report as well as the name of the resource and the line number where the data came from!

Technorati Tags: gallio,mbunit

Gallio and ReSharper 4.0

2008-06-10T01:46:00.001-07:00

A few people have asked when Gallio will support the final release of ReSharper 4.0.

Unfortunately there were some last-minute breaking changes between ReSharper 4.0 Beta and the Final release. So the ReSharper plugin in Gallio Alpha 3 Update 2 will not work anymore.

No matter... we plan to release another update of Gallio later this week.

Technorati Tags: gallio,mbunit

Announcing Gallio Alpha 3 Update 2

2008-05-29T01:08:00.001-07:00

Today we are releasing an update to Gallio Alpha 3 with a few new features.

Download here: (link)

New and Noteworthy

ReSharper v4.0 Beta Integration

If you haven't already heard, ReSharper v4.0 Beta is out! (link)

And now we have support for it...

Gallio Test Report Action in ReSharper

ReSharper's unit test explorer is a great tool but it does not currently support all of the features needed for full native integration with Gallio.

For example, there is no way for us to display embedded images -- such as screenshots -- as part of the test output. There are also some limitations as to how we can present the results of data-driven tests -- like Row-based tests -- in ReSharper.

No matter. We have now added a custom action to display the full Gallio Test Report.

We have also made a few other small improvements to the output.

Unfortunately stack traces still are not clickable and there does not seem to be much we can do about it just now due to ReSharper limitations.

Pex Integration for MbUnit

Microsoft Pex has finally be released! (link)

We already had support for an earlier prerelease version of Pex but we have now upgraded it to the latest edition.

The basic idea is that you can write parameterized tests using MbUnit v3 syntax and run Pex against them to generate test cases.

Step 1: Add a reference to MbUnit.Pex.

Step 2: Add the MbUnit Pex package attribute to your test assembly.

Step 3: Give Pex something to chew on.

Step 4: Pex It!

Step 5: Examine the generated tests.

There you go!

MbUnit v3 Contract Verifiers

I've added a couple of test fixtures to help verify common contracts such as equality and correct exception implementation.

Currently you use these verifiers by creating subclasses. However, I'm tempted to transform the verifiers into mixins in the long run.

EqualityContractVerifier

This checks Equals, GetHashCode(), IEquatable<T>.Equals(), operator== and operator!=. You just need to provide the verifier with suitable distinct instances of your type.

ExceptionContractVerifier

This checks that an exception is serializable and has the standard constructors.

Upcoming Stuff

There's a bunch of new stuff on the way in Alpha 4 as we march onwards to feature-completeness.

Here's a little list to whet your appetite:

First partial draft of the Gallio Book.
Constraints framework.
Improved assertions.
.Net 3.5 features.
Workspace model for managing test project resources like historical test reports.
Metadata in CSV files.
[Factory] attribute for easier implementation of custom data sources.
New plug-in model and performance optimizations (might slip to Alpha 5).

I'm aware that there are a number of pending bugs and feature requests for the standard Gallio GUI (Icarus). If you're interested in giving Graham a hand with this please let me know!

Ditto for the Visual Studio Team System integration.

Technorati Tags: mbunit,gallio

Help wanted: Release and Promotion Manager for Gallio and MbUnit

2008-05-27T16:30:00.001-07:00

I'm looking for a volunteer to help release and promote new editions of Gallio and MbUnit.

If this sounds like something that would interest you, please let me know.

Thanks!

Release and Promotion Manager

Responsibilities:

Draft the release criteria.
Ensure the release criteria have been met.
Publish the release notes and release files.
Keep the website updated with news and useful resources.
Promote Gallio and MbUnit online and in the local community.
Produce articles, tutorials, podcasts, and other educational or promotional materials.
Monitor reactions from users and use it to steer subsequent releases.

Salary: Cookies, chocolate, beer or other treat of your choice.

Technorati Tags: gallio,mbunit

MSBuild v3.5 ToolsVersion Attribute Gotchas

2008-05-19T17:56:00.001-07:00

I just spent an hour debugging a project trying to use the new Properties and AdditionalProperties metadata in MSBuild v3.5. (MSDN)

This special metadata is used to pass properties to MSBuild projects that are invoked using the MSBuild task.

Example

a.proj

<Project DefaultTargets="Test" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">
<ItemGroup>
    <Project Include="b.proj">
      <Properties>Foo=123</Properties>
    </Project>
</ItemGroup>

<Target Name="Test">
    <MSBuild Projects="@(Project)" />
</Target>
</Project>

b.proj

<Project DefaultTargets="Test" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">
<Target Name="Test">
<Message Importance="High" Text="Foo=$(Foo)" />
</Target>
</Project>

Then when you run %SystemRoot%\Microsoft.NET\Framework\v3.5\MSBuild.exe a.proj, it should print out: Foo=123. Woot!

We have successfully passed some properties to a recursive invocation of MSBuild.

ToolsVersion is Essential

Now what happens if you omit ToolsVersion="3.5".

Well, it you omit it from b.proj then the script will still work.

But it you omit it from a.proj MSBuild will run in a v2.0 compatibility mode. So the value of the Foo property will NOT be passed into b.proj. Not too surprising but it took me a little while to track it down.

This just goes to show how important the ToolsVersion attribute is. Be careful!

Announcing Gallio v3.0 Alpha 3 Update 1

2008-05-16T13:25:00.001-07:00

In my previous post, I mentioned that MbUnit v3's TestFixture attribute is now optional. I have also fixed a couple of minor bugs.

Download Alpha 3 Update 1: GallioBundle-3.0.0.251-Setup.exe

Technorati Tags: gallio,mbunit

MbUnit TestFixture Attribute is Now Optional

2008-05-15T13:57:00.001-07:00

Historically, MbUnit has used a custom attribute called TestFixture to indicate that a class contains tests.

This isn't the only such attribute. Other custom attributes, such as AssemblyFixture in MbUnit v3, denote various special kinds of fixtures. However, TestFixture is the standard and most frequently used kind of fixture.

When James Newkirk and Brad Wilson designed xUnit.Net they omitted several features that they felt were redundant, confusing, or harmful. TestFixture, long a staple of unit testing frameworks in .Net was one such piece of syntax that was axed: it was simply redundant. A class is obviously a test fixture if it is not abstract and it contains suitably annotated test methods.

In a previous post I asked whether people felt that MbUnit should follow suit with xUnit.Net and either drop TestFixture altogether or make it optional. The general sentiment seems to have fallen in the latter category.

So effective now, in MbUnit v3, TestFixture is optional. I will publish an update to Gallio Alpha 3 with this enhancement shortly.

Examples

1. Simple fixture in C#.

public class SimpleFixture
{
    [Test]
    public void SimpleTest()
    {
        Assert.AreEqual(4, 2 + 2);
    }
}

2. Simple fixture in F#.

#light
using MbUnit.Framework

[<Test>]
let SimpleTest() = Assert.AreEqual(4, 2 + 2)

3. Data-driven fixture in C# using generic type parameters.

[Row(typeof(List<int>), typeof(int))]
[Row(typeof(Dictionary<string, int>), typeof(KeyValuePair<string, int>))]
public class DataDrivenFixture<T, S> where T : ICollection<S>, new()
{
    [Test]
    public void CountIsIncrementedWhenOneItemIsAdded()
    {
        T collection = new T();
        collection.Add(default(S));
        Assert.AreEqual(1, collection.Count);
    }
}

4. A different data-driven fixture in C# using a constructor.

public class HomestarRunnerProductTest
{
    private string productName;
    private Product product;

    [Row("Fluffy Puff Marshmallows")]
    [Row("Cheat Command O's")]
    public Fixture(string productName)
    {
        this.productName= productName;
    }

    [SetUp]
    public void BuyProduct()
    {
        product = Store.BuyProduct(productName);
    }

    [TearDown]
    public void ReturnProduct()
    {
        Store.ReturnProduct(product);
    }

    [Test]
    public void ShouldComeInAShinyBox()
    {
        Assert.IsTrue(product.HasShinyBox);
    }

    [Test]
    public void ShouldTasteSweet()
    {
        Assert.AreEqual(Flavor.Sweet, product.Flavor);
    }
}

5. Another data-driven fixture in C# using fields.

public class DataDrivenFixture
{
    [Column(OS.Windows, OS.Mac, OS.Linux)]
    public OS os;

    [Column(100, 1000, 10000)]
    public int dataSetSize;

    public void TestConfiguration()
    {
        // ... do something with os and dataSetSize
    }
}

6. A fixture represented as a static class with static test methods in C#.

public static class StatelessFixture
{
    [Test]
    public static void StatelessTest()
    {
        Assert.AreEqual(4, 2+ 2);
    }
}

7. A data-driven fixture in F# using a constructor. (hypothetical)

I was going to demo something similar using F# but I couldn't get it to compile an attribute with object[] parameters.

I tried a bunch of variations but the only thing I could get to compile properly was a Row with an empty array. Other attempts variously spit out errors like "error FS0191: invalid custom attribute value (not a constant): Format`5.ctor.1@[obj, obj, obj, obj, unit]`"2". "

Looks like there are bugs in the F# compiler as it does not seem to support array creation or type expressions as valid attribute values.

#light
using MbUnit.Framework

[<Row( [| 123, "123" |] )>]             <--- does not compile in F# 1.9.4.
type when_string_formatter_receives_an_integer = { value : int, formattedResult : string } with
    [<Test>]
    member f.it_should_format_the_value_as_a_string() = Assert.AreEqual(f.formattedResult, StringFormatter.Format(f.value));
end

Final Remarks

The above examples only just scratch the surface of the supported variations. Really, you can pick and choose the style you wish to use in your tests as you like.

P.S. I also fixed the bug that prevented Jim Burger from using MbUnit effectively with F#. (more details)

Technorati Tags: gallio,mbunit

Announcing Gallio v3.0 Alpha 3

2008-05-13T02:49:00.001-07:00

Today we have released Gallio v3.0 Alpha 3, including a preview of MbUnit v3.

The installer is available here: GallioBundle-3.0.0.247-Setup.exe (temporary link)

It will also be available from the Gallio website shortly: www.gallio.org.

There has been quite a lot of progress in this release including new features, bug fixes, and various internal changes to make it easier for runner and framework authors to build new cool stuff on top of Gallio.

Here is a quick summary of these changes.

New and Notable

New Features

1. Annotations

Annotations are a means for test frameworks to communicate errors, warnings and informational messages to the user in association with particular code elements.

MbUnit v3 uses annotations to inform the user about improperly used attributes and to highlight points of interest such as ignored or pending tests.

Annotations are displayed as messages in test reports.

They manifest as error/warning stripes in R#.

Graham Hay has also added an Annotation view in the Icarus GUI. Double-clicking on the annotation displays the source code.

2. Report Enhancements

HTML reports now include navigation stripes to provide a quick overview of the contents of the report and to navigate to failed, inconclusive and skipped tests.

When JavaScript is enabled, test results are automatically collapsed for ease of reading. They are automatically expanded during navigation.

HTML reports can grow to become quite large when there are many tests. Accordingly, we have added a "Condensed" version of this report that omits passing tests. This is particularly useful in combination with CruiseControl.Net.

3. Icarus Test Execution Log

Graham Hay has added a new Execution Log view to Icarus. You can view test results here while tests execute. If you periodically refresh the view then you can watch test execution in near real-time including output from long-running tests that are still in flight.

4. Test Assembly Configuration Files

Gallio now provides all of the machinery required to support test assembly configuration files. All frameworks and adapters can benefit from this support, including MbUnit v3. Just create a configuration file named: "MyTestAssembly.dll.config" and put it next to your test assembly (assuming it is called MyTestAssembly).

5. MSTest Adapter

Julian Hidalgo implemented an adapter for MSTest. This feature enables all Gallio runners to run MSTest tests. One of the interesting consequences of this design is that R# v3.1 is now capable of viewing and running MSTest tests via Gallio.

6. Visual Studio Team System Integration (Experimental)

Gallio now includes an experimental ITip provider for integration with Visual Studio Team System 2008.

To use it, be sure to select the feature in the Gallio Installer. It is currently excluded from installation by default because of its experimental nature.

Then just load up your favorite test project in Visual Studio 2008 and run a Build. The tests should now appear in the Visual Studio Test View. You can then run them from there. It should work with all supported test frameworks.

As far as I know, we are the first Open Source testing project to provide this kind of integration.

If would like to contribute to the development of this feature please let me know!

7. Test Runner Extensions

Many new features can be added to Gallio by implementing plug-ins. However, some users have requested the ability to extend test runners to change how test results are reported.

Gallio Alpha 2 included an ITestMonitor interface that roughly served this purposes by making it possible to inject additional behavior into the test runner's event handlers. In Gallio Alpha 3, we have introduced the ITestRunnerExtension interface and provided a mechanism for loading extensions into the runners by means of an additional argument.

Implementing a test runner extension is straightforward:

Create a new assembly for your extension.
Add a reference to Gallio.dll.
Create a subclass of TestRunnerExtension.
In the Initialize method, register event handlers on the Events object provided to the extension.
Build your assembly.
Copy the file wherever you like.

Using a test runner extension is similar to registering an MSBuild logger.

If you are using Gallio.Echo.exe (the command-line test runner), add an argument like: "/extension:MyFancyExtension,MyFancyExtension.dll".
If you are using the Gallio NAnt or MSBuild tasks, the process is similar. Refer to the documentation of these tasks for details.
There is a built-in extension for debugging. Add "/extension:DebugExtension,Gallio" or an equivalent argument to enable it. Be sure to also set the message verbosity to Debug.

The plan is to open up more of the platform to lightweight extensions of this sort.

8. Parameterized Test Enhancements

MbUnit v3 now fully supports generic test fixtures and test methods and various other parameterizations.

A test can now be parameterized in any of the following ways:

Adding a method parameter to a test method.
Adding a generic method parameter to a test method.
Adding a constructor parameter to a test fixture.
Adding a field to a test fixture.
Adding a writable property to a test fixture.
Adding a generic type parameter to a test fixture.

Correspondingly data-source attributes can be specified on any of the following code elements:

Test methods.
Test method parameters.
Test method generic parameters.
Test fixture types.
Test fixture constructors.
Test fixture constructor parameters.
Test fixture fields.
Test fixture properties.
Test fixture generic parameters.

So there are many choices, but each choice is quite simple.

Here are a few interesting combinations.

A basic "Row"-test.

Note: The [RowTest] attribute is not needed anymore and has been eliminated.

[TestFixture]
public class Fixture
{
    [Test]
    [Row(3, 4, 5, true)]
    [Row(1, 2, 3, false)]
    public void Test(int a, int b, int c, bool expectedResult)
    {
        Assert.AreEqual(expectedResult, Math.IsPythagoreanTriple(a, b, c));
    }
}

A generic "Type"-test.

Note: The [TypeFixture] attribute is not needed anymore and has been eliminated.

[TestFixture]
[Row(typeof(List<int>))]
[Row(typeof(LinkedList<int>))]
public class Fixture<T> where T : IList<int>, new()
{
    [Test]
    public void Test()
    {
        T list = new T();
        list.Add(123);
        Assert.AreEqual(1, list.Count);
    }
}

A combinatorial test.

Note: The [CombinatorialTest] attribute is not needed anymore and has been eliminated.

[Test]
public void Test([Column(0, 1)] int a, [Column("a", "b")] string b, [Column(false, true)] bool c)
{
// Test something with the specified combination.
// Will be executed 8 times total.
}

A pairwise test.

Only tests all combinations of pairs of values. Can yield a significant reduction in test cases compared to ordinary combinatorial tests while still obtaining great coverage. See also: www.pairwise.org.

[Test, PairwiseJoin]
public void Test([Column(0, 1)] int a, [Column("a", "b")] string b, [Column(false, true)] bool c)
{
// Test something with the specified combination.
// Will be executed only 4 times instead of 8.
}

A CSV data source.

Just one of many external data sources to be implemented.

[Test]
[CsvData(ResourcePath="CsvDataTest.csv", HasHeader=true)]
public void ImplicitlyScopedResourceWithHeader(decimal price, string item)
{
Log.WriteLine("{0}: {1}", item, price);
}

The data itself should be stored in an embedded resource called "CsvDataTest.csv". We just create a file with that name in the same folder as the test class and then set its Build Action property to Embedded Resource in Visual Studio.

Item, Price
Apples, 1.00
Bananas, 1.50
Cookies, 2.00

Automatic data type conversions.

Can automatically convert strings into DateTimes and doubles into decimals for example.

Architecture Improvements

There were a number of internal architecture improvements in Gallio Alpha 3. Here's a quick summary.

1. Unified Hosting Apparatus

In Gallio Alpha 2 there was some duplication between the responsibilities of the IHost and ITestDomain components. These abstractions have been refactored. As a result, ITestDomain was replaced by ITestDriver to handle the specific needs of setting up a testing environment.

The IHost abstraction provides a mechanism for Gallio to run code remotely such as in an isolated AppDomain or another process.

2. Pattern Test Framework Refactoring

There was an important piece missing from the data binding story in Alpha 2 that prevented certain kinds of parameterized tests from being expressed. This has been resolved by refactoring of the Pattern Test Framework (from which MbUnit v3 is derived) and introducing the concept of a test data context. IPattern is significantly simpler now.

This work also prompted the introduction of the Annotations mechanism when it became clear that we needed a much more direct mechanism for reporting semantic errors in a test to the user.

3. Reflection Policy Enhancements

Introduced the concept of a typed ConstantValue for performing reflection over unevaluated attribute values. For example, when populating tests for the ReSharper add-in we may encounter attribute values that are of type Type. It's quite possible for those types to not be loaded in the runtime (since we're performing reflection against an in-memory code model). So instead we can obtain a ConstantValue that represents the Type without actually loading it.

Gallio Alpha 3 also includes a Cecil-based reflection policy for scanning disk-based assemblies without loading them into the runtime. The Visual Studio Team System add-in uses this policy to perform reflection without locking assemblies or introducing memory leaks.

4. Test Runner Redesign

The ITestRunner interface has been completely redesigned to make it easier to use and more versatile. For one, the reporting functionality has now been integrated somewhat more tightly into the runner itself. An ITestRunnerExtension can register new behaviors on a test runner using the ITestRunnerEvents interface.

5. Aggregate Test Drivers

Supporting test assembly configuration files is not as easy as it sounds. For one, it requires us to create a separate AppDomain for each test assembly. Then we have to somehow merge the test model and results.

Gallio Alpha 3 introduces an aggregate test driver to help with this. However, the current design is still somewhat in flux. I'd appreciate some feedback on how we can improve and simplify the test framework integration story overall.

Bugs Fixed

Here are a few of the bugs from Alpha 2 that have been fixed:

Icarus now unloads assemblies promptly to avoid locking them. You can also enable shadow-copy mode for improved efficiency.
MbUnit v2 version compatibility problems fixed. Should work with any version of MbUnit v2.4+.
Hung Gallio.Host.exe processes are no more.
xUnit.Net adapter has been upgraded to v1.0 RTM.
Installer PATH variable corruption.
Castle dependencies have been internalized to avoid version mismatches.
Default thread apartment state is STA and can be overridden with the [ApartmentState] attribute.
[Explicit] tests are now silent and do not display any output in reports unless explicitly selected, not even a message to say they are being skipped. If you want noisy output, use [Ignore], [Pending] or [Annotation] instead.

Technorati Tags: gallio,mbunit

Openings at Ingenio

2008-05-12T23:16:00.001-07:00

We're hiring! Want to work with me?

To learn more or to apply, send an email with your résumé to jobs@ingenio.com.

Also check out this link for more opportunities at Ingenio.

Platform Toolsmith

Do you love building tools? Do you get a kick out of building integrated systems that just work?

Ingenio is looking for someone to help build a robust and highly integrated platform for software development, deployment and testing. Our system consists of a mixture of web, telephony and data platforms so it takes a lot of ingenuity to assemble these pieces and to verify that they work together correctly to provide a compelling user experience.

We need an experienced toolsmith who understands the challenge of release engineering, system monitoring, infrastructure build-out and software architecture to help us keep our coders coding, our testers testing, our users using, and our systems running smoothly.

Sample projects:

Build new plug-ins for our distributed system monitoring application.
Develop a new deployment tool for coordinating updates to our server farm.
Implement a SIP phone emulator for testing telephony applications.
Plan SOA and ESB middleware deployment projects.
Work on Open Source projects such as Gallio, MbUnit, Archimedes and WatiN.

What we will look for in you:

You love to program!
You like to tinker and take things apart... and put them back together better than they were before.
You have great communication skills.
You know at least one object-oriented programming language such as C#, Java, Python, Ruby or C++.
You have experience with release engineering and deployments.
You have a 4 year degree in computer science or engineering.

Location: San Francisco, CA. Downtown with easy access to BART and Muni.

Test Automation Guru (Lead)

Do you love testing? Do you love working with testers, teaching them new tricks, designing systems and building tools?

Ingenio is looking for someone to lead our test automation effort.

Our system consists of a mixture of web, telephony and data platforms so it takes a lot of ingenuity to test that these pieces work together correctly to provide a compelling user experience. There are many challenges involved in the construction of controlled test environments that can obtain reliable results and find bugs. We need an experienced lead who thoroughly groks testing and can help us take test automation to the next level.

Sample projects:

Design a virtual automation test environment.
Build test tools and monitoring infrastructure.
Implement model-based and automated exploratory testing approaches.
Plan comprehensive automation test projects.
Mentor junior members of the test automation team.
Hire more smart people to work with.
Work on Open Source projects such as Gallio, MbUnit, Archimedes and WatiN.

What we will look for in you:

You have experience leading, teaching and motivating others to excellence.
You have a strong programming background.
You have great communication skills.
You know at least one object-oriented programming language such as C#, Java, Python, Ruby or C++.
You are familiar with at least one xUnit testing tool such as MbUnit, NUnit or JUnit.
You grok test automation.

Location: San Francisco, CA. Downtown with easy access to BART and Muni.

Test Automation Wizard

Do you love programming and testing? Do you enjoy writing high-quality code using the latest tools? Do you hate bugs? Do you like solving challenging system integration problems?

Ingenio is looking for test automation engineers to develop a comprehensive and maintainable test suite for our products.

Our system consists of a mixture of web, telephony and data platforms so it takes a lot of ingenuity to test that these pieces work together correctly to provide a compelling user experience. We need talented tester/developers and developer/testers to decide what to test, to write tests and to adopt novel testing strategies.

Sample projects:

Write automated tests for our system using C#.
Implement model-based and automated exploratory testing approaches.
Build infrastructure for deploying, running and monitoring tests.
Plan comprehensive automation test projects.
Work on Open Source projects such as Gallio, MbUnit, Archimedes and WatiN.

What we will look for in you:

You love to program!
You are smart, passionate, eager to learn and get things done.
You have great communication skills.
You know at least one object-oriented programming language such as C#, Java, Python, Ruby or C++.
You are familiar with at least one xUnit testing tool such as MbUnit, NUnit or JUnit.
You have prior experience writing automated tests.

Location: San Francisco, CA. Downtown with easy access to BART and Muni.

Test Automation Apprentice (Intern)

Do you enjoy programming? Are you curious about test automation? Do you want to get a head start in the industry?

Ingenio is looking for smart students to bring in fresh ideas, challenge the established order and remind us all to have fun.

Our system consists of a mixture of web, telephony and data platforms so it takes a lot of ingenuity to test that these pieces work together correctly to provide a compelling user experience. As part of that effort, we officially sponsor the development of MbUnit, an Open Source testing framework. We are constantly searching for ways to improve our processes.

We hope to learn as much from you as we promise to teach you.

Sample projects:

Create something new and fantastic that we haven't thought of yet in this list...
Develop a tool to automatically spider our web sites using C# and find bugs.
Write automated tests for our system using C#.
Build infrastructure for deploying, running and monitoring tests.
Create test reports.
Work on Open Source projects such as Gallio, MbUnit, Archimedes and WatiN.

What we will look for in you:

You love to program!
You are smart, passionate, eager to learn and get things done.
You have great communication skills.
You know at least one object-oriented programming language such as C#, Java, Python, Ruby or C++.
You are currently enrolled or have recently graduated from an accredited university.

Location: San Francisco, CA. Downtown with easy access to BART and Muni.

Technorati Tags: jobs

Google Code Download WTF

2008-05-12T00:00:00.001-07:00

I was getting ready to publish Gallio Alpha 3 tonight but we're out of space on Google Code.

Ok, no problem... I'll just delete some stuff.

Google Code had other ideas:

Uhh... But I don't need this version anymore: I've archived it myself and I need the free space right now!

Meanwhile my request for more quota is still pending...

(I'm just glad I never wrote a script to upload nighty builds.)

Another Programmer Bites the Dust

2008-05-04T22:22:00.001-07:00

I've been struggling with Carpal Tunnel Syndrome for 6 or 8 months now.

I don't quite remember when it started. I've had bouts of muscle tension on and off throughout my life. After all, I've pretty much spent every other waking hour in front of a computer for the last 18 years or so (total of 22 years programming but during my early years I did not have a computer at home). But recently I started noticing different symptoms.

Numbness and tingling are now commonplace occurrences.
It does not go away after a night's rest.
It does not go away after a day or two of not using the computer.
It has invaded other aspects of my life including cooking, recreation, sleep and sex.
I find myself searching for ways to work differently and to avoid sustained typing or mousing.
I find myself wishing for an eye-tracker and decent voice recognition to help out around the edges.
It's made me cry...

I have already made many adjustments to my ergonomics and home life style.

I use a keyboard tray tilted at a negative angle.
I use a Microsoft Natural Wireless Ergonomic Keyboard and Mouse combo.
I wear elbow braces during the day.
I wear write braces during the night.
I try to sit properly.
My wife opens jars for me.

But it's still getting worse.

Today I installed Workrave to help remind me to take breaks, do my exercises and quit after working a long day.

*sigh*

Flip Side

I am now taking more active leadership roles. I am a Lead Software Engineer at the office and the MbUnit / Gallio Lead on the side.

There's just so much cool stuff to build out there! And I know all sorts of smart people that I can set to the task of building it. :-)

A Good Test Finds Bugs

2008-05-02T03:11:00.001-07:00

This evening I found myself thinking about what my favourite test in Gallio must be.

It could be the formatter tests that are simple and to the point and leverage many of the good traits of the MbUnit Row-test, including generic method parameters.

It could be the MbUnit integration tests that self-host Gallio within itself to run tests and carefully analyze their output.

Nahh...

My favourite Gallio test is the abstract reflection policy exhaustive test suite! It's not pure. It's not pretty. It's not tightly focused. It's not short. It's not precise. It's not self contained (it uses native .Net reflection as an oracle). But boy does it find bugs! All sorts of obscure deviations from expected behaviour in each of the 4 current reflection policy implementations.

It's an awful test but it does what a test should do: it finds bugs...

Technorati Tags: gallio,mbunit

Death By a Thousand Cuts

2008-04-29T04:17:00.001-07:00

It feels like I'll never get this release out.

I'm currently facing an interesting flavour of DLL hell (tastes like bananas) resulting from copy-local assemblies that get loaded into the same application (Visual Studio) in different load contexts. Basically, ReSharper and Visual Studio Team Test both want to load Gallio.dll but they do it in different ways from different locations. This results in some rather curious behavior because two identical copies of the assembly will be loaded but they are nevertheless considered different from one another.

It would be nice if we could just tell ReSharper and Visual Studio Team Test to load my add-in assemblies from the Gallio installation path, but neither of them actually support that behavior. They both require their add-ins to be copied into very specific folders.

I can think of two possible fixes right now:

Add some hacks to install a custom assembly resolver just in time at all entry points into the add-ins so that we can control more precisely how Gallio.dll gets loaded. That way we can avoid having multiple copies of Gallio.dll.
Put Gallio.dll into the GAC.

I previously used option #1 for the Gallio ReSharper add-in. It was a real pain because I had to carefully instrument all entry points into the add-in and avoid referencing any Gallio types until the assembly resolver hook was installed. I ripped this brittle code out a couple of weeks ago since I figured just copying Gallio.dll locally would be good enough. (guess again!)

So option #2 really seems best. Keep in mind that the copy of Gallio in the GAC should only be used for external tools integration. I don't anticipate test assemblies or the basic standalone runner actually referring to anything in the GAC so Gallio should still be usable in-place without requiring any installation steps (except for these add-ins of course).

However, it's 4am now so I'm going to sleep on it for a little while. It may be that using the GAC will cause more headaches such as problems with the NAnt or MSBuild tasks that might also involve a little non-standard assembly resolver-foo of their own.

Next problem: Throw out NSIS (finally!) and rewrite the installer using WiX...

Technorati Tags: gallio

The C# Null-Coalescing Operator, PEVerify and ILMerge

2008-04-28T14:12:00.001-07:00

It seems I errantly blamed ILMerge for those PEVerify errors. At worst it may be a C# compiler optimization gone awry or it may be that PEVerify is not smart enough.

The Source

Consider this code:

public IList<ITestCommand> Children{ get { return children ?? (IList<ITestCommand>)EmptyArray<ITestCommand>.Instance; } }

The left-hand side of the null-coalescing operator has type: List<ITestCommand>.

The right-hand side of the null-coalescing operator has type: ITestCommand[] before the cast.

The result has type: IList<ITestCommand>.

So we need a little silent cast in there to make the types match up on either side. Really, it's just providing a compatible upper bound to which both sides can be converted. At runtime, this operation is a no-op. Technically it's something that the compiler itself could have inferred if it were smart enough.

The Unoptimized IL

Let's take a quick look at the IL that the C# compiler generates for a standard unoptimized build.

Notice that the castclass operator appears in the listing. There is also quite a bit of jiggery-pokery involving a compiler generated temporary local variable.

.method public hidebysig specialname newslot virtual final instance class [mscorlib]System.Collections.Generic.IList`1<class Gallio.Model.Execution.ITestCommand> get_Children() cil managed
{
    .maxstack 2
    .locals init (
        [0] class [mscorlib]System.Collections.Generic.IList`1<class Gallio.Model.Execution.ITestCommand> CS$1$0000)
    L_0000: nop 
    L_0001: ldarg.0 
    L_0002: ldfld class [mscorlib]System.Collections.Generic.List`1<class Gallio.Model.Execution.ITestCommand> Gallio.Model.Execution.ManagedTestCommand::children
    L_0007: dup 
    L_0008: brtrue.s L_0015
    L_000a: pop 
    L_000b: ldsfld !0[] Gallio.Collections.EmptyArray`1<class Gallio.Model.Execution.ITestCommand>::Instance
    L_0010: castclass [mscorlib]System.Collections.Generic.IList`1<class Gallio.Model.Execution.ITestCommand>
    L_0015: stloc.0 
    L_0016: br.s L_0018
    L_0018: ldloc.0 
    L_0019: ret 
}

The Optimized IL

When generating optimized code, the C# compiler realizes that the castclass is unnecessary. So it drops it. It also drops the unnecessary load/stores to a local variable and we end up with the following code.

.method public hidebysig specialname newslot virtual final instance class [mscorlib]System.Collections.Generic.IList`1<class Gallio.Model.Execution.ITestCommand> get_Children() cil managed
{
    .maxstack 8
    L_0000: ldarg.0 
    L_0001: ldfld class [mscorlib]System.Collections.Generic.List`1<class Gallio.Model.Execution.ITestCommand> Gallio.Model.Execution.ManagedTestCommand::children
    L_0006: dup 
    L_0007: brtrue.s L_000f
    L_0009: pop 
    L_000a: ldsfld !0[] Gallio.Collections.EmptyArray`1<class Gallio.Model.Execution.ITestCommand>::Instance
    L_000f: ret 
}

PEVerify Has a Conniption

Without the castclass in the optimized code, the IL is now missing a crucial piece of information: what is the type of the value of the top of the stack when we reach the ret instruction?

In one case, we may have duplicated the value produced by the ldfld instruction so the top has type List<ITestCommand>.

In the other case, we have discarded the original value and instead loaded an array from somewhere else using ldsfld so the top has type ITestCommand[].

Uh oh!

[IL]: Error: [C:\Source\MbUnit\v3\src\Gallio\Gallio\bin\Gallio.dll : Gallio.Model.Execution.ManagedTestCommand::get_Children][offset 0x0000000F][found ref 'System.Collections.IEnumerable'][expected ref 'System.Collections.Generic.IList`1[Gallio.Model.Execution.ITestCommand]'] Unexpected type on the stack.

Without the castclass PEVerify has to do some rather more complicated work to figure out whether the types are compatible. Apparently, it fails. But it does get far enough to realize that one of the values can be converted to IEnumerable.

Is This a Problem?

I don't know. Apparently we have been running this code in Gallio release builds for many months now and not seen any InvalidProgramException errors. So even though PEVerify is confused, the CLR at least does not choke on the code.

In any case, it's definitely not ILMerge's fault. (Sorry Mike!)

Technorati Tags: low-level

ILMerge redux

2008-04-28T02:28:00.001-07:00

In my previous post, I mentioned a problem I was having with ILMerge and how I fixed it.

It turns out that code security attributes were not the only problem. ILMerge is no longer complaining, but it is generating a bad output assembly just the same. This is preventing the C# compiler from linking to the merged assembly. Most immediately it manifested as XmlSerializers not working.

PEVerify shows a long stream of errors:

C:\Source\MbUnit\v3>"c:\Program Files\Microsoft.NET\SDK\v2.0 64bit\Bin\PEVerify.exe" build\target\bin\Gallio.dll

Microsoft (R) .NET Framework PE Verifier. Version 2.0.50727.42
Copyright (c) Microsoft Corporation. All rights reserved.

[MD]: Error: TypeDef is marked Nested but has no enclosing type. [token:0x020004
40]
[MD]: Warning: MemberRef has a duplicate, token=0x0a0004f3. [token:0x0A0001AB]
[MD]: Warning: MemberRef has a duplicate, token=0x0a0004f4. [token:0x0A0001AC]

...

*sigh*

More Good Fortune?

I upgraded ILMerge from 2.8.0202 to 2.8.0422 and now it seems to be working a little better.

Now the TypeDef error is gone but we're still getting MemberRef warnings. Makes me a bit nervous but at least the C# compiler is happy enough linking to the merged assembly and the XmlSerializer problem seems to be resolved.

However, there was nothing in the change history to suggest that any bug were fixed.

Or Maybe Not?

Turns out I missed these errors:

[IL]: Error: [C:\Source\MbUnit\v3\build\target\bin\Gallio.dll : Gallio.Model.Execution.ManagedTestCommand::get_Children][offset 0x00000012][found ref 'System.Collections.IEnumerable'][expected ref 'System.Collections.Generic.IList`1[Gallio.Model.Execution.ITestCommand]'] Unexpected type on the stack.
[IL]: Error: [C:\Source\MbUnit\v3\build\target\bin\Gallio.dll : Gallio.Model.Execution.ManagedTestCommand::get_Dependencies][offset 0x00000012][found ref 'System.Collections.IEnumerable'][expected ref 'System.Collections.Generic.IList`1[Gallio.Model.Execution.ITestCommand]'] Unexpected type on the stack.
2 Errors Verifying build\target\bin\Gallio.dll

Just great. Well, I guess Gallio Alpha 3 will be delayed a little bit further while I cook up a workaround or something.

Technorati Tags: low-level

Don't use self-referential Code Access Security attributes.

2008-04-28T01:27:00.001-07:00

I encountered an interesting little problem today after completing a small refactoring of Gallio. ILMerge broke!

An exception occurred during merging:
ILMerge.Merge: The assembly 'Gallio' was not merged in correctly. It is still listed as an external reference in the target assembly.
at ILMerging.ILMerge.Merge()
at ILMerging.ILMerge.Main(String[] args)

True enough, the merged assembly contained a self-reference.

The proximate cause seems to be the fact that the Gallio assembly defined a code access security attribute that it itself used. Ordinarily these attributes should be defined in other assemblies. Indeed, the compiler will enforce the constraint for any assembly-level permission requests. I assume ILMerge is doing a little magic here to try to enforce similar properties.

Ahh well. I eliminated the custom attribute and all is well.

Leap of Faith

Julian suggested I blog about this issue because of the manner in which I resolved it.

Internal dialogue:

ILMerge is breaking because I am using it on a different assembly than I was before. So there's something essentially wrong about this assembly or with ILMerge.
Google turns up this error message but only in circumstances involving the ASP.Net merge tool and user error. I don't think that's the case here.
The documentation doesn't say anything about this.
Renaming files around and messing with the command-line arguments doesn't change anything.
Panic!

Time passes...

Hmm. When skimming the ILMerge change history to find the entry about to the addition of this error message didn't I see something about fixing security attributes? I wonder what that was about.
Gallio uses a custom security attribute.
Security attributes are subject to these interesting cross-assembly reference constraints when used to declare minimum permission requests...
Oh!

Now I have no idea why the mention of security attributes in the change history caught my attention, but it did.

You might call it lateral thinking, but I'll call it sheer luck!

Technorati Tags: low-level

Gallio integration with Visual Studio Team System

2008-04-24T19:26:00.001-07:00

One of the questions I've often received about Gallio has been about whether it would integrate with MSTest and Visual Studio Team System.

Gallio running MSTest tests in Icarus

Julian did the work of writing a Gallio adapter plugin for MSTest tests. Gallio explores MSTest tests using reflection and runs them by shelling out to MSTest.exe and parsing the test result XML file.

Here's a screenshot of the Gallio Icarus GUI running some MSTest tests.

Gallio running MSTest tests in ReSharper

Of course, you can use any Gallio test runner to run MSTest tests. That means you automatically get ReSharper integration too! (And in all of the other runners too.)

MSTest running Gallio tests in Visual Studio

Another common request has been to provide integration so that Visual Studio Team Test can run Gallio tests in the IDE and using the MSTest.exe command-line.

This particular request was echoed to me a couple of times at ALT.Net from people who, for good or ill, have been instructed by management to use the Team System test runner instead of TestDriven.Net or ReSharper. So I decided to give it a shot.

Gallio Alpha 3 now includes an experimental package for Visual Studio to do just that. (It also works on the command-line).

Technorati Tags: gallio,mbunit