[ODFPlugtest] plugfest test file results

[Jos van den Oever]

On Wednesday, November 03, 2010 23:11:41 pm robert_weir at us.ibm.com wrote:
> plugtest-bounces at opendocsociety.org wrote on 11/03/2010 02:06:52 PM:
> > That's also why Louis suggested to ask around if, for instance, students
> > at universities could generate test documents and/or help us out with
> > testing scenarios in different implementations (many devs are very busy
> > with reviewing the ODF 1.2 spec and/or writing code)
> 
> What if we "crowd-sourced" this?
> 
> In other words, we define a large set of test documents, for example, by
> features used, editors used, etc.  This large list is our "wanted list".
> The upfront work is defining this list.  Maybe do it based on the table of
> contents of the ODF standard, or similar feature list.  Aim for each
> sample to be easy to create, no more than 5-10 minutes.
> 
> Then publish this list on a web site and invite contributions.  Require a
> liberal CC licence on contributions.  Maybe track the wanted list in a
> database so we only list "wants" that are not already covered.  Maybe
> automatically rename the files according to a naming scheme so we can
> easily associate files with the features tested.  Maybe have a
> verification stage, which could also be crowd-sourced, to verify that the
> uploaded file matches the described scenario.
Sounds like a good plan, I think the verification step is important.

Some requirements I see for this proposal for obtaining more test documents:
 - (web) interface to upload documents
 - "limbo" where documents go until they are verified
 - automatic verification with possibility of manual override by TC members
 - query that retrieves all verified documents
 - selection criteria that link test files to features in the specification

Most of these things would fit nicely in OfficeShots.

The hard part is to verify that a document renders as it should render. Having 
a button in OfficeShots that allows people to say: "this render is what I would 
expect" would be nice, along with a way of discussing on the particular 
render.

Cheers,
Jos

-- 
Jos van den Oever, software architect
+49 391 25 19 15 53
http://kogmbh.com/legal/