6. Writing a Test Suite
1. Introduction
This document explains how to use the current CSS test suites, provides a
description of the officially adopted CSS test suite format, and instructs
how to write a test suite in this format.
Sample text and hyperlinks from test suites are
presented in this document with a thin grey border like this: test suite text. test suite
hyperlink. Sample hyperlinks from test suites are not underlined to avoid
misrepresenting them as actual hyperlinks in this document.
2. History
The CSS working group published the first version
of the CSS1 Test
Suite on the 11th of May 1998. This test suite introduced much of the
format and structure of current CSS test suites. On 24th October 2001, the
CSS working group published the CSS Mobile Profile
1.0 Candidate Recommendation and decided that the mobile profile also
required a test suite.
Since the mobile profile referred to features already defined in previous
CSS specifications, there was a strong desire to reuse previous tests rather
than write new tests. During a CSS working group teleconference it was
suggested that the tests in the test suite should be separated
into test pages and navigation pages, specifically using the
<OBJECT> element in order to facilitate reuse of the tests
themselves.
Based on that discussion, a
member of the CSS Working Group took the initiative to attempt a test suite
experiment - a version of the CSS1 Test Suite with its test pages
separated into navigation pages and test pages. The TNG Test Suite Experiment was demonstrated
at the August 2002 CSS Working Group ftf meeting.
As noted in the minutes
of the meeting, numerous implementations were able to use the new test suite
format without difficulty, including a cell phone browser. The TNG format was
subsequently adopted as the official primary test suite format for test
suites produced by the CSS working group. The CSS1 Test Suite was updated to
use the new format, and the CSS Mobile
Profile 1.0 Test Suite was produced in the new format. In addition, the
Selectors
Test Suite now includes several TNG
versions for each of the document languages tested.
3. Using the test suites
As of this writing there are at least three test suites which use the
official CSS test suite format:
Most of these are directly written in the TNG format (CSS1, CSS Mobile
Profile 1.0), and thus their cover pages are consistent and recognizable. The
Selectors Test Suite is written in a meta-format which is then used to
generate several variants, including the TNG format, for several document
languages. For such test suites, you have to first click on a link or two to
select the cover page for the variant of the document language that you want
to test.
3.1. Cover page
The cover page of a test suite contains a table of contents, structured
parallel to the specification being tested. Often a test suite contains a -- Prologue which
documents among other things requirements or recommendations for a specific
system configuration, or suggestions for setting your system preferences in
such a way as to minimize their interference with the tests themselves.
3.2. Getting started with the
tests
Get started with running the tests in a test suite by clicking on the
first numbered link in the table of contents (skipping over the -- Prologue), or by
clicking the get started
here hyperlink that many test suites provide on their cover page.
3.3. Next and Previous
On every following page, there will be a [Next] hyperlink that will take you linearly
through each section, subsection and test of the test suite, finally bringing
you back to the cover page at which point you have completed the test suite.
On any of these pages you can always backtrack this linear order by following
the [Previous]
hyperlinks.
3.4. Test content
On every page that contains a test, you should see the test content
immediately below the abovementioned hyperlinks. Instead of the test itself,
you see a Test
hyperlink, then your browser doesn't properly support the
<OBJECT> element. No matter, simply click the Test hyperlink to
navigate to the test itself. After viewing the test, use the Back
functionality in your web browser to return to the navigation page for that
test and proceed to the next page by clicking the [Next] hyperlink.
The test content is self-describing - that is, the test content contains
a textual description of what the page should look like according to the
test, e.g. This sentence should be
green.
4. Test Suite
Principles
The current test suite format is the evolutionary culmination of the work
of many contributors over several years. Along the way several principles
were identified that aided in the design and advancement of the format.
4.1. Valid tests
This should go without saying, yet the existence
of
numerous
invalid
3rd
party
test
suites
(even some
hosted
at
W3C!) which
do
not
pass
the
simplest
of
validators underscores the importance of emphasizing this principle. In
essence, all files in the test suite, whether navigational or tests
themselves must at a minimum pass the W3C
Validator. In addition, if the relevant selectors, @-rules, properties
and values have been added to the W3C CSS Validator, then the all test suite
files should also pass the W3C
CSS Validator.
Simplicity is always a good design principle. In the case of a test suite,
keeping the format simple helps minimize unintended effects from the format
itself. In addition, a simple format also makes it easier to author the test
and navigation pages.
4.3. Ease of authoring
There are very few easy to access and easy to run test suites at W3C. The
ratio of RECs to Test Suites is large and only getting larger, when it should
be shrinking until it approaches 1. Others have noted that the task of
authoring test suites often demands the allocation of significant time and
resources. To reduce the effective cost of producing test suites, and
therefore hopefully increase the number of test suites that are produced, it
is only logical to adopt of the principle of ease of authoring.
Not wanting to depend on any particular tools, or any particular platform,
or device for that matter, we interpret the principle of ease of authoring to
imply that the test suite format must be easily hand authorable,
that is, easily authorable with only the use of a simple text editor.
4.4. Something rather than
nothing
Another way of looking at the problem of a dearth of W3C test suites, is
to ask which is the bigger problem - the lack of fully automated W3C test
suites which are presented in numerous variants for every conceivable
document language, or the lack of W3C test suites period? Is a simple test
suite better than none at all? The experience of the CSS working group, with
the CSS1 Test Suite, and the many compliant and interoperable implementations
that followed, demonstrated fairly clearly that a simple test suite is far
far better than none at all.
4.5. Sooner rather than later
Corollary: a simple test suite within a few months is more useful, and
thus preferred to, a more complex, more fully featured test suite a year or
more from now.
4.6. Test the technology
being tested, not other technologies
When writing a test suite for a particular technology (e.g. CSS), it is
often necessary to utilize other technologies (e.g. HTML, DOM etc.) in the
construction of that test suite. Test suite authors must be careful while
testing one technology, avoid testing other technologies. Those other
technologies should be authored to be maximally compatible, in a way that is
most forgiving to the various user agents which will be using the test suite.
Specifically, when writing CSS test suite should not try to test:
- XML handling
- XHTML handling
- file extension handling (other than none or ".css")
- mime type handling (other than "text/css")
- HTML/XHTML/XML sniffing/binding capabilities
- DOM handling
Consequently, CSS test suite files should:
- use the file extensions .htm for HTML and .xht for XHTML files
- use the mime types "text/html" for HTML and "application/xhtml+xml" for
XHTML files
- use the HTML4.01 document type for HTML files
- use the XHTML 1.1 document type for XHTML files
- omit '<?xml version="1.0" encoding="utf-8"?>' for XHTML files since
it is not required by XML nor XHTML, and it confuses some UAs
- use <link rel="stylesheet"> instead of <?xml-stylesheet> in
XHTML documents, as all XHTML documents imply HTML semantics therefore this
does not add any additional requirements to both HTML user agents, and XHTML
aware XML user agents, whereas only XML user agents can be expected to
properly handle <?xml-stylesheet>.
4.7. Atomic tests
A test suite should contain at least one atomic (meaning, testing a single
feature at a time) test per feature or area.
4.8. Feature coverage
A test suite should contain at least one basic test for each explicit
feature of a technology. In the case of CSS, there should be at least one
test for each rule, selector, property and value type. Note that some value
types, such as <length>, may take a number of different units. For such
value types, there should be at least one test for each unit, but that does
not mean that every such unit must be tested on every property.
In addition, a number of additional composite, devious, evil, and failure
(error checking) tests should be written for any issue that is likely to be
mis-implemented. In writing these tests, experience with existing
implementations is a great help. As implementations progress, new areas
worthy of being tested will come to light, and the test suite should be
updated regularly to track these developments.
4.9. Self describing
results
Test cases should describe in prose, in the test cases themselves, what is
expected. Each test case should at a minimum describe correct behavior. Error
handling tests should at a minimum describe incorrect, or possibly unexpected
behavior.
4.10. Incremental difficulty
Test suites should begin with simpler, easier test cases, and proceed to
more complex, more challenging tests. This makes sense both from a test case
authoring perspective, since simpler cases are easier to author, and from an
technology implementer's perspective, since simpler cases are easier to
implement.
4.11. Non-requirements
Finally, as with any set of principles and requirements, there are some
non-requirements. These non-requirements are exactly that - they are features
not required of a test suite. They may be desirable, or even included - yet
they are not necessary to produce a test suite.
- automation - the ability to automatically run a test suite.
- use of a meta-format for generating multiple variants
4.12. Additional principles
A separate document has been written to give
guidelines and techniques to use when authoring CSS test cases.
The SVG Test Manual documents additional
Principles Applicable to Test Suite Content . Future test suites authors
should consult these principles as well.
5.1. Test suite pages
The test suite format consists of several different kinds of pages. At the
core are the test pages themselves that exercise the features of the
technology. Each test page is referenced by a navigation page that also
references the previous / next navigation pages, and its section page. The
test suite starts with a cover page, which links directly to each of the
section pages. This document describes these pages from the inside out.
5.2. Test page format
The test page format is documented in a separate document, which also includes a template.
5.3. Navigation page
format
Each test page is contained within a navigation page. The navigation page
starts with a heading documenting the test suite, followed by the numerical
section of the specification being tested which is also used for the page
<TITLE>, e.g. CSS1 Test Suite: 1.1 Containment in
HTML. A horizontal rule separates the heading from the following links:
[Previous]
[Next] [Section] [Contents]
[Specification]
The Previous and
Next hyperlinks link
to the previous and next navigation pages respectively. Previous of the first
navigation page of a section links to that section page. Next of the last
navigation page of a section links to the next section. And Next of the last
navigation page in the suite links to the cover page.
The Section
hyperlink links to the section page for all the navigation pages for this
section of the specification being tested. The Contents hyperlink links to the cover page of
the test suite. The Specification hyperlink links to the specific
section of the specification being tested.
The navigation page transcludes the test page with an
<OBJECT> element, which references the test file using the
data attribute and explicitly sets the type of the
expected data to "text/html". In order to encourage a user agent
to use the entire width of the document, and the remainder of the document,
both the width and height attributes are set to
100%.
The <OBJECT> element was chosen because it is present
in all "current" versions of HTML, from HTML 4.0 Transitional, to XHTML
Basic, to XHTML 1.1. This permits user agents which support any one of these
languages to easily utilize the navigational pages. In addition, the
<OBJECT> element provides an excellent immediate fall back
mechanism for user agents that either happen to not support
<OBJECT>, or that have difficulties transcluding HTML
content.
Inside the <OBJECT> element is a simple Test hyperlink which
directly links to the test file. Thus user agents that do not recognize the
<OBJECT> element and therefore ignore its markup, show the
simple hyperlink instead, thereby providing access to the test file.
The navigation pages use valid HTML 4.0 Transitional,
which is the most compatible version of HTML for present day user agents. In
the future, XHTML 1.0 Transitional which follows the XHTML 1.0 Appendix C
guidelines may be more compatible across more user agents. XHTML Basic, again
following the XHTML 1.0 Appendix C guidelines, may also be more compatible.
Since the navigation pages are separated from the test pages, this could
easily change in the future without having to change the test pages.
Similarly, the test pages could be updated to use XHTML Basic for example
without having to update the navigational pages.
Lastly, the navigation pages are named "sec" followed by the section
number of the section being tested (without periods, e.g. 1.1 becomes 11)
with the extension ".htm". E.g.
"sec11.htm". Note that the navigation file has no test content whatsoever.
This allows the test file to be updated to fix problems in tests without
having to update navigation files.
5.4. Section page
format
For each chapter in the specification, there is a corresponding section
page which hyperlinks to the navigation pages of all the tests for that
chapter. The section page format very much resembles the navigation page
format.
The section page contains a flat list of hyperlinks to the specific
navigation pages, in specification order, and labeled with the respective
section and subsection (and potentially subsubsection) of the specification.
E.g.
1.1 Containment in
HTML
The list is demarcated on both top and bottom by horizontal rules, and a
link bar with the following links:
[Previous]
[Next] [Contents] [Specification]
The Previous
hyperlink links to the last navigation page of the previous section. The
Previous of the first section (typically a -- Prologue section) simply links to the test
suite cover page. The Next hyperlink links to the first navigation
page of this section.
The Contents
hyperlink links to the cover page, and the Specification hyperlink links to the
respective section in the specification.
Similar to the navigation pages, the section pages are also titled with a
heading documenting the name of the test suite, followed by the numerical
section of the specification. This heading is also used for the page
<TITLE>, e.g. CSS1 Test Suite: 1.0 Basic
Concepts. The section pages are also written in valid HTML 4.0
Transitional, and named "sec" followed by the top level section number with a
trailing "0" and the extension ".htm". E.g.
"sec10.htm".
5.5. Cover page format
A test suite's cover page begins with the W3C logo and a heading stating
the name of the test suite which is also used for the page
<TITLE>. E.g.
CSS1 Test Suite
A short paragraph of text introduces the test suite, and documents its
purpose, usage, and any caveats. Another paragraph minimally explains the
structure of the test suite itself. Any additional test suite specific
documentation should be placed in the -- Prologue. A subheading labels the Table of
Contents which immediately follows the introductory text.
The table of contents is a simple list of numbered hyperlinks ( e.g.: 1.0 Basic Concepts ) to
the section pages. Following the numbered section pages for the tests, there
may be one or more lettered pages for test suite appendices, such as A. Acknowledgments, B. Test Suite Errata, and
C. Related Resources.
A horizontal rule separates the Table of Contents from the cover page
footer. The footer contains a mailto hyperlink for test suite feedback, the
proper validation stamps (HTML, CSS), and recognition of the organizations
responsible for the production of the test suite, along with their respective
logos.
6. Writing a Test Suite
The authoring of a test suite for a particular specification begins with
the identification of testable assertions in the specification grouped by
section and subsection of the specification.
Try to make use of existing test pages. Some specifications, such as
profiles, do not contain new features of their own. New versions of
specifications often build upon features of older specifications. In both
cases, it should be possible, and it is desirable, to reuse existing test
cases.
Start by creating a new directory for your test suite, and copying the
necessary support files (listed below). All test suite files for your test
suite will go into this one directory.
Using the templates listed below, author navigation pages for each feature
and their respective testable assertions. If possible, author navigation
pages which refer to tests from previous test suites. Use a relative URL in
the data attribute of the <OBJECT> tag, and
href of the anchor <A>, to refer to test
pages from previous test suites. Replace the heading and document
<TITLE> with the name of the test suite, the subsection
number and title. Fix the link bar hyperlinks (Previous, Next, Section,
Specification) to point to the appropriate files.
For navigation pages which require new tests, write new test pages, also
starting with the template listed below. Be sure to update the document
<TITLE> to be the same as the corresponding navigation
page.
After completing all the navigation (and necessary test) pages for a
section in the specification, write the section page (again, there is a
template which can be used as a starting point). Once the section pages have
been completed, construct the cover page and related support pages (Prologue,
Acknowledgments, Test Suite Errata (hopefully fairly empty to begin with),
and Related Resources). Templates aren't included for for the related support
pages, but they can be easily written starting from the respective pages in
the CSS1 Test
Suite.
Congratulations, you've authored a test suite.
6.1. Support Files
6.2. Templates
These templates are derived from the latest version of the CSS1 Test
Suite, with a small bit of cleanup, so there may be minor inconsequential
differences.
6.2.1. Navigation
page
The template for a navigation page is shown below, and is also available
as a separate file: secnntemplate.htm
<!DOCTYPE HTML PUBLIC "-/W3C/DTD HTML 4.0 Transitional/EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<HTML><HEAD><TITLE>CSSn Test Suite: n.n Subsection Title </TITLE>
<META http-equiv="Content-Style-Type" content="text/css">
<LINK rel="stylesheet" type="text/css" media="screen" href="static.css"></HEAD>
<BODY>
<DIV class="navigation">
<H2>CSSn Test Suite: n.n Subsection Title</H2>
<HR>
[<A HREF="secnn.htm">Previous</A>] [<A HREF="secnn.htm">Next</A>] [<A HREF="secnn.htm">Section</A>] [<A HREF="index.html">Contents</A>] [<A HREF="/cats-d8c4vu/www.w3.org/TR/">Specification</A>]<BR>
</DIV>
<OBJECT height="100%" width="100%" border="0" type="text/html" data="testnn.htm"><A class="navigation" href="testnn.htm" target="testwindow">Test</A></OBJECT>
</BODY></HTML>
6.2.2. Section page
The template for a section page is shown below, and is also available as a
separate file: secn0template.htm
<!DOCTYPE HTML PUBLIC "-/W3C/DTD HTML 4.0 Transitional/EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<HTML><HEAD><TITLE>CSSn Test Suite: n.0 Section Title</TITLE>
<META http-equiv="Content-Style-Type" content="text/css">
<LINK rel="stylesheet" type="text/css" media="screen" href="static.css">
</HEAD>
<BODY class="navigation">
<H2>CSSn Test Suite: n.0 Section Title</H2>
<HR>
[<A HREF="secnn.htm">Previous</A>] [<A HREF="secnn.htm">Next</A>] [<A HREF="index.html">Contents</A>] [<A HREF="/cats-d8c4vu/www.w3.org/TR/">Specification</A>]<BR>
<HR>
<P>
In this section:
</P>
<UL>
<LI>n.1 <TT><A HREF="secn1.htm">Subsection 1</A></TT></LI>
<LI>n.2 <TT><A HREF="secn2.htm">Subsection 2</A></TT></LI>
<LI>n.3 <TT><A HREF="secn3.htm">Subsection 3</A></TT></LI>
<LI>n.4 <TT><A HREF="secn4.htm">Subsection 4</A></TT></LI>
<LI>n.5 <TT><A HREF="secn5.htm">Subsection 5</A></TT></LI>
<LI>n.6 <TT><A HREF="secn6.htm">Subsection 6</A></TT></LI>
<LI>n.7 <TT><A HREF="secn7.htm">Subsection 7</A></TT></LI>
</UL>
<HR>
[<A HREF="secnn.htm">Previous</A>] [<A HREF="secnn.htm">Next</A>] [<A HREF="index.html">Contents</A>] [<A HREF="/cats-d8c4vu/www.w3.org/TR/">Specification</A>]<BR>
</BODY></HTML>
6.2.3. Cover page
The template for the cover page is shown below, and is also available as a
separate file: indextemplate.htm
<!DOCTYPE HTML PUBLIC "-/W3C/DTD HTML 4.0 Transitional/EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<HTML><HEAD><TITLE>CSSn Test Suite</TITLE>
<META http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<META http-equiv="Content-Style-Type" content="text/css">
<LINK REL="stylesheet" TYPE="text/css" MEDIA="screen" HREF="static.css">
<LINK rel="alternate stylesheet" type="text/css" href="W3C-rsrc-test.css" media="screen" title="Experimental W3C Styles">
<STYLE TYPE="text/css">
H1 {border: solid 1px black; border-left: 3px solid black; background-color: #FFFFFF;}
H4 {border-bottom: 1px solid black; margin-left: 2em; margin-right: 5em; font-variant: small-caps; margin-bottom: 0.5em;}
UL {list-style-type: none;}
.validated { text-align:left; font-size: smaller }
</STYLE>
</HEAD>
<BODY CLASS="navigation">
<H1><A HREF="http://www.w3.org/"><IMG BORDER="0" SRC="/cats-d8c4vu/www.w3.org/Icons/WWW/w3c_home" ALT="W3C" WIDTH="72" HEIGHT="48" align="middle"></A>
CSSn Test Suite</H1>
<P>
The CSSn Test Suite is provided as a way for vendors and page authors to test
their browser's conformance to the CSSn specification.
</P>
<P>
For each feature, there is at least one page which tests the feature
in various ways, using HTML 4.0 markup. The test pages are broken out into
a number of sections which reflect the structure of the CSSn specification itself,
in addition to a Prologue and a few appendicies.
</P>
<H4>Table of Contents</H4>
<UL>
<LI style="font-weight: bold;">-- <A HREF="sec00.htm">Prologue</A></LI>
</UL>
<UL>
<LI>1.0 <A HREF="sec10.htm">Section 1</A></LI>
<LI>2.0 <A HREF="sec20.htm">Section 2</A></LI>
<LI>3.0 <A HREF="sec30.htm">Section 3</A></LI>
<LI>4.0 <A HREF="sec40.htm">Section 4</A></LI>
<LI>5.0 <A HREF="sec50.htm">Section 5</A></LI>
<LI>6.0 <A HREF="sec60.htm">Section 6</A></LI>
<LI>7.0 <A HREF="sec70.htm">Section 7</A></LI>
</UL>
<UL style="margin-top: 0;">
<LI> A. <A HREF="tsack.html">Acknowledgments</A></LI>
<LI> B. <A HREF="tserr.html">Test Suite Errata</A></LI>
<LI> C. <A HREF="tsref.html">Related Resources</A></LI>
</UL>
<HR>
<P>
Feedback regarding the CSSn Test Suite should be sent to
<TT><A HREF="mailto:[email protected]">[email protected]</A></TT>.
</P>
<P CLASS="validated">
<A HREF="/cats-d8c4vu/www.w3.org/Icons/valid-html40" ALT="Valid HTML 4.0!"></A>
<A HREF="/cats-d8c4vu/jigsaw.w3.org/css-validator/images/vcss" ALT="Valid CSS!"></A>
</P>
<TABLE cellpadding="5" cellspacing="0" border="0" style="border: solid 1px black;">
<TR bgcolor="white" valign="middle">
<TD style="width:50%; border: none 0px;">
<SMALL>The CSS1 Test Suite is a joint effort of the <A HREF="http://www.w3.org/">World Wide Web Consortium</A>,
and the <A HREF="http://www.example.com/">(your company/organization name here)</A>.</SMALL>
</TD>
<TD align="center" style="width:50%; border: none 0px; white-space: nowrap;">
<A HREF="http://www.w3.org/"><IMG SRC="W3C.gif" border="0" alt=" W3C " height="32" width="57"></A>
<A HREF="http://www.example.com/"><IMG SRC="logo.gif" border="0" alt="(your logo here)" ></A>
</TD>
</TR>
</TABLE></BODY></HTML>
Follow Lee on X/Twitter - Father, Husband, Serial builder creating AI, crypto, games & web tools. We are friends :) AI Will Come To Life!
Check out: eBank.nz (Art Generator) |
Netwrck.com (AI Tools) |
Text-Generator.io (AI API) |
BitBank.nz (Crypto AI) |
ReadingTime (Kids Reading) |
RewordGame |
BigMultiplayerChess |
WebFiddle |
How.nz |
Helix AI Assistant
