Add RDoc

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@5073 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

34 files changed, 12296 insertions, 0 deletions

diff --git a/lib/rdoc/README b/lib/rdoc/README
new file mode 100644
index 0000000000..39ce8bb75b
--- /dev/null
+++ b/lib/rdoc/README
@@ -0,0 +1,445 @@
+= RDOC - Ruby Documentation System
+
+This package contains Rdoc and SimpleMarkup. Rdoc is an application
+that produces documentation for one or more Ruby source files. We work
+similarly to JavaDoc, parsing the source, and extracting the
+definition for classes, modules, and methods (along with includes and
+requires).  We associate with these optional documentation contained
+in the immediately preceding comment block, and then render the result
+using a pluggable output formatter. (Currently, HTML is the only
+supported format. Markup is a library that converts plain text into
+various output formats. The Markup library is used to interpret the
+comment blocks that Rdoc uses to document methods, classes, and so on.
+
+This library contains two packages, rdoc itself and a text markup
+library, 'markup'. 
+
+== Roadmap
+
+* If you want to use Rdoc to create documentation for your Ruby source
+  files, read on.
+* If you want to include extensions written in C, see rdoc/parsers/parse_c.rb.
+* For information on the various markups available in comment
+  blocks, see markup/simple_markup.rb.
+* If you want to drive Rdoc programatically, see RDoc::RDoc.
+* If you want to use the library to format text blocks into HTML,
+  have a look at SM::SimpleMarkup.
+* If you want to try writing your own HTML output template, see
+  RDoc::Page.
+
+== Summary
+
+Once installed, you can create documentation using the 'rdoc' command
+(the command is 'rdoc.bat' under Windows)
+
+  % rdoc [options]  [names...]
+
+Type "rdoc --help" for an up-to-date option summary.
+
+A typical use might be to generate documentation for a package of Ruby
+source (such as rdoc itself). 
+
+  % rdoc
+
+This command generates documentation for all the Ruby and C source
+files in and below the current directory. These will be stored in a
+documentation tree starting in the subdirectory 'doc'.
+
+You can make this slightly more useful for your readers by having the
+index page contain the documentation for the primary file. In our
+case, we could type
+
+  % rdoc --main rdoc/rdoc.rb
+
+You'll find information on the various formatting tricks you can use
+in comment blocks in the documentation this generates.
+
+RDoc uses file extensions to determine how to process each file. File
+names ending <tt>.rb</tt> and <tt>.rbw</tt> are assumed to be Ruby
+source. Files ending <tt>.c</tt> are parsed as C files. All other
+files are assumed to contain just SimpleMarkup-style markup (with or
+without leading '#' comment markers). If directory names are passed to
+RDoc, they are scanned recursively for C and Ruby source files only.
+
+== Credits
+
+* The Ruby parser in rdoc/parse.rb is based heavily on the outstanding
+  work of Keiju ISHITSUKA of Nippon Rational Inc, who produced the Ruby
+  parser for irb and the rtags package.
+
+* Code to diagram classes and modules was written by Sergey A Yanovitsky
+  (Jah) of Enticla. 
+
+* Charset patch from MoonWolf.
+
+* Rich Kilmer wrote the kilmer.rb output template.
+
+* Dan Brickley led the design of the RDF format.
+
+== License
+
+RDoc is Copyright (c) 2001-2003 Dave Thomas, The Pragmatic Programmers.  It
+is free software, and may be redistributed under the terms specified
+in the README file of the Ruby distribution.
+
+
+----
+
+= Usage
+
+RDoc is invoked from the command line using:
+
+   % rdoc <options> [name...]
+
+Files are parsed, and the information they contain collected, before
+any output is produced. This allows cross references between all files
+to be resolved. If a name is a directory, it is traversed. If no
+names are specified, all Ruby files in the current directory (and
+subdirectories) are processed.
+
+Options are:
+
+[<tt>--accessor</tt> <i>name[,name...]</i>]
+    specifies the name(s) of additional methods that should be treated
+    as if they were <tt>attr_</tt><i>xxx</i> methods. Specifying
+    "--accessor db_opt" means lines such as
+
+         db_opt :name, :age
+  
+    will get parsed and displayed in the documentation. Each name may have an
+    optional "=flagtext" appended, in which case the given flagtext will appear
+    where (for example) the 'rw' appears for attr_accessor.
+
+[<tt>--all</tt>]
+    include protected and private methods in the output (by default
+    only public methods are included)
+
+[<tt>--charset</tt> _charset_]
+    Set the character set for the generated HTML.
+
+[<tt>--diagram</tt>]
+    include diagrams showing modules and classes.  This is currently
+    an experimental feature, and may not be supported by all output
+    templates. You need dot V1.8.6 or later to use the --diagram
+    option correctly (http://www.research.att.com/sw/tools/graphviz/).
+
+[<tt>--exclude</tt> <i>pattern</i>]
+    exclude files and directories matching this pattern from processing
+
+[<tt>--extension</tt> <i>new=old</i>]
+    treat files ending <i>.new</i> as if they ended
+    <i>.old</i>. Saying '--extension cgi=rb' causes RDoc to treat .cgi
+    files as Ruby source.
+
+[<tt>fileboxes</tt>]
+    Classes are put in boxes which represents files, where these
+    classes reside. Classes shared between more than one file are
+    shown with list of files that sharing them.  Silently discarded if
+    --diagram is not given Experimental.
+
+[<tt>--fmt</tt> _fmt_]
+    generate output in a particular format.
+
+[<tt>--help</tt>]
+    generate a usage summary.
+
+[<tt>--help-output</tt>]
+    explain the various output options.
+
+[<tt>--image-format</tt> <i>gif/png/jpg/jpeg</i>]
+    sets output image format for diagrams. Can be png, gif, jpeg,
+    jpg. If this option is omitted, png is used. Requires --diagram.
+
+[<tt>--include</tt> <i>dir,...</i>]
+    specify one or more directories to be searched when satisfying
+    :+include+: directives. Multiple <tt>--include</tt> options may be
+    given. The directory containing the file currently being processed
+    is always searched.
+
+[<tt>--inline-source</tt>]
+    By default, the source code of methods is shown in a popup. With
+    this option, it's displayed inline.
+
+[<tt>line-numbers</tt>]
+    include line numbers in the source code
+
+[<tt>--main</tt> _name_]
+    set the class, module, or file to appear on the index page
+
+[<tt>--one-file</tt>]
+    place all the output into a single file
+
+[<tt>--op</tt> _dir_]
+    set the output directory to _dir_ (the default is the directory
+    "doc")
+
+[<tt>--op-name</tt> _name_]
+    set the name of the output. Has no effect for HTML.
+    "doc")
+
+[<tt>--opname</tt> _name_]
+    set the output name (has no effect for HTML).
+
+[<tt>--promiscuous</tt>]
+    If a module or class is defined in more than one source file, and
+    you click on a particular file's name in the top navigation pane,
+    RDoc will normally only show you the inner classes and modules of
+    that class that are defined in the particular file. Using this
+    option makes it show all classes and modules defined in the class,
+    regardless of the file they were defined in.
+
+[<tt>--quiet</tt>]
+    do not display progress messages
+
+[<tt>--show-hash</tt>]
+    A name of the form #name in a comment is a possible hyperlink to
+    an instance method name. When displayed, the '#' is removed unless
+    this option is specified
+
+[<tt>--style</tt> <i>stylesheet url</i>]
+    specifies the URL of an external stylesheet to use (rather than
+    generating one of our own)
+
+[<tt>tab-width</tt> _n_]
+    set the width of tab characters (default 8)
+
+[<tt>--template</tt> <i>name</i>]
+    specify an alternate template to use when generating output (the
+    default is 'standard'). This template should be in a directory
+    accessible via $: as rdoc/generators/xxxx_template, where 'xxxx'
+    depends on the output formatter.
+
+[<tt>--version</tt>]
+   display  RDoc's version
+
+[<tt>--webcvs</tt> _url_]
+    Specify a URL for linking to a web frontend to CVS. If the URL
+    contains a '\%s', the name of the current file will be
+    substituted; if the URL doesn't contain a '\%s', the filename will
+    be appended to it.
+
+= Example
+
+A typical small Ruby program commented using RDoc might be as follows. You
+can see the formatted result in EXAMPLE.rb and Anagram.
+
+      :include: EXAMPLE.rb
+
+= Markup
+
+Comment blocks can be written fairly naturally. 
+
+Paragraphs are lines that share the left margin. Text indented past
+this margin are formatted verbatim.
+
+1. Lists are typed as indented paragraphs with:
+   * a '*' or '-' (for bullet lists)
+   * a digit followed by a period for 
+     numbered lists
+   * an upper or lower case letter followed
+     by a period for alpha lists.
+
+   For example, the input that produced the above paragraph looked like
+       1. Lists are typed as indented 
+          paragraphs with:
+          * a '*' or '-' (for bullet lists)
+          * a digit followed by a period for 
+            numbered lists
+          * an upper or lower case letter followed
+            by a period for alpha lists.
+
+2. Labeled lists (sometimes called description
+   lists) are typed using square brackets for the label.
+      [cat]   small domestic animal
+      [+cat+] command to copy standard input
+
+3. Labeled lists may also be produced by putting a double colon
+   after the label. This sets the result in tabular form, so the
+   descriptions all line up. This was used to create the 'author'
+   block at the bottom of this description.
+      cat::   small domestic animal
+      +cat+:: command to copy standard input
+
+   For both kinds of labeled lists, if the body text starts on the same
+   line as the label, then the start of that text determines the block
+   indent for the rest of the body. The text may also start on the line
+   following the label, indented from the start of the label. This is
+   often preferable if the label is long. Both the following are
+   valid labeled list entries:
+
+      <tt>--output</tt> <i>name [, name]</i>::
+          specify the name of one or more output files. If multiple
+          files are present, the first is used as the index.
+
+      <tt>--quiet:</tt>:: do not output the names, sizes, byte counts,
+                          index areas, or bit ratios of units as
+                          they are processed.
+
+4. Headings are entered using equals signs
+
+      = Level One Heading
+      == Level Two Heading
+   and so on
+
+5. Rules (horizontal lines) are entered using three or
+   more hyphens.
+
+6. Non-verbatim text can be marked up:
+
+   _italic_::     \_word_ or \<em>text</em>
+   *bold*::       \*word* or \<b>text</b>
+   +typewriter+:: \+word+ or \<tt>text</tt>
+
+   The first form only works around 'words', where a word is a
+   sequence of upper and lower case letters and underscores. Putting a
+   backslash before inline markup stops it being interpreted, which is
+   how I created the table above:
+
+     _italic_::     \_word_ or \<em>text</em>
+     *bold*::       \*word* or \<b>text</b>
+     +typewriter+:: \+word+ or \<tt>text</tt>
+
+7. Names of classes, source files, and any method names
+   containing an underscore or preceded by a hash
+   character are automatically hyperlinked from
+   comment text to their description. 
+
+8. Hyperlinks to the web starting http:, mailto:, ftp:, or www. are
+   recognized. An HTTP url that references an external image file is
+   converted into an inline <IMG..>.  Hyperlinks starting 'link:' are
+   assumed to refer to local files whose path is relative to the --op
+   directory.
+
+   Hyperlinks can also be of the form <tt>label</tt>[url], in which
+   case the label is used in the displayed text, and <tt>url</tt> is
+   used as the target.
+       
+9. Method parameter lists are extracted and displayed with
+   the method description. If a method calls +yield+, then
+   the parameters passed to yield will also be displayed:
+
+      def fred
+        ...
+        yield line, address
+
+   This will get documented as
+
+      fred() { |line, address| ... }
+
+   You can override this using a comment containing 
+   ':yields: ...' immediately after the method definition
+
+      def fred      # :yields: index, position
+        ...
+        yield line, address
+
+   which will get documented as
+
+       fred() { |index, position| ... }
+
+
+10. ':yields:' is an example of a documentation modifier. These appear
+    immediately after the start of the document element they are modifying.
+    Other modifiers include
+
+    [<tt>:nodoc:</tt><i>[all]</i>]
+         don't include this element in the documentation.  For classes
+         and modules, the methods, aliases, constants, and attributes
+         directly within the affected class or module will also be
+         omitted.  By default, though, modules and classes within that
+         class of module _will_ be documented. This is turned off by
+         adding the +all+ modifier.
+
+              module SM  #:nodoc:
+                class Input
+                end
+              end
+              module Markup #:nodoc: all
+                class Output
+                end
+              end
+
+         In the above code, only class <tt>SM::Input</tt> will be
+         documented.
+
+    [<tt>:doc:</tt>]
+         force a method or attribute to be documented even if it
+         wouldn't otherwise be. Useful if, for example, you want to
+         include documentation of a particular private method.
+
+    [<tt>:notnew:</tt>]
+         only applicable to the +initialize+ instance method. Normally
+         RDoc assumes that the documentation and parameters for
+         #initialize are actually for the ::new method, and so fakes
+         out a ::new for the class. THe :notnew: modifier stops
+         this. Remember that #initialize is protected, so you won't
+         see the documentation unless you use the -a command line
+         option.
+
+
+11. RDoc stops processing comments if it finds a comment
+    line containing '<tt>#--</tt>'. This can be used to 
+    separate external from internal comments, or 
+    to stop a comment being associated with a method, 
+    class, or module. Commenting can be turned back on with
+    a line that starts '<tt>#++</tt>'.
+
+        # Extract the age and calculate the
+        # date-of-birth.
+        #--
+        # FIXME: fails if the birthday falls on
+        # February 29th
+        #++
+        # The DOB is returned as a Time object.
+
+        def get_dob(person)
+           ...
+
+12. Comment blocks can contain other directives:
+
+    [<tt>:include:</tt><i>filename</i>] 
+         include the contents of the named file at this point. The
+         file will be searched for in the directories listed by
+         the <tt>--include</tt> option, or in the current
+         directory by default.  The contents of the file will be
+         shifted to have the same indentation as the ':' at the
+         start of the :include: directive.
+
+    [<tt>:title:</tt><i>text</i>]
+         Sets the title for the document. Equivalent to the --title command
+         line parameter. (The command line parameter overrides any :title:
+         directive in the source).
+
+    [<tt>:enddoc:</tt>]
+         Document nothing further at the current level.
+
+    [<tt>:main:</tt><i>name</i>]
+         Equivalent to the --main command line parameter.
+
+    [<tt>:stopdoc: / :startdoc:</tt>]
+         Stop and start adding new documentation elements to the
+         current container. For example, if a class has a number of
+         constants that you don't want to document, put a
+         <tt>:stopdoc:</tt> before the first, and a
+         <tt>:startdoc:</tt> after the last. If you don't specifiy a
+         <tt>:startdoc:</tt> by the end of the container, disables
+         documentation for the entire class or module.
+
+
+---
+
+See also markup/simple_markup.rb.
+
+= Other stuff
+
+Author::   Dave Thomas <dave@pragmaticprogrammer.com>
+Requires:: Ruby 1.8.1 or later
+License::  Copyright (c) 2001-2003 Dave Thomas.
+           Released under the same license as Ruby.
+
+== Warranty
+
+This software is provided "as is" and without any express or
+implied warranties, including, without limitation, the implied
+warranties of merchantibility and fitness for a particular
+purpose.
diff --git a/lib/rdoc/code_objects.rb b/lib/rdoc/code_objects.rb
new file mode 100644
index 0000000000..7fdbbde027
--- /dev/null
+++ b/lib/rdoc/code_objects.rb
@@ -0,0 +1,653 @@
+# We represent the various high-level code constructs that appear
+# in Ruby programs: classes, modules, methods, and so on.
+
+require 'rdoc/tokenstream'
+
+module RDoc
+
+
+  # We contain the common stuff for contexts (which are containers)
+  # and other elements (methods, attributes and so on)
+  #
+  class CodeObject
+
+    attr_accessor :parent
+
+    # We are the model of the code, but we know that at some point
+    # we will be worked on by viewers. By implementing the Viewable
+    # protocol, viewers can associated themselves with these objects.
+
+    attr_accessor :viewer
+
+    # are we done documenting (ie, did we come across a :enddoc:)?
+
+    attr_accessor :done_documenting
+
+
+    # do we document ourselves?
+
+    attr_reader :document_self
+
+    def document_self=(val)
+      @document_self = val
+      if !val
+	remove_methods_etc
+      end
+    end
+
+    # set and cleared by :startdoc: and :enddoc:, this is used to toggle
+    # the capturing of documentation
+    def start_doc
+      @document_self = true
+      @document_children = true
+    end
+
+    def stop_doc
+      @document_self = false
+      @document_children = false
+    end
+
+    # do we document ourselves and our children
+
+    attr_reader :document_children
+
+    def document_children=(val)
+      @document_children = val
+      if !val
+	remove_classes_and_modules
+      end
+    end
+
+    # Do we _force_ documentation, even is we wouldn't normally show the entity
+    attr_accessor :force_documentation
+
+    # Default callbacks to nothing, but this is overridden for classes
+    # and modules
+    def remove_classes_and_modules
+    end
+
+    def remove_methods_etc
+    end
+
+    def initialize
+      @document_self = true
+      @document_children = true
+      @force_documentation = false
+      @done_documenting = false
+    end
+
+    # Access the code object's comment
+    attr_reader :comment
+
+    # Update the comment, but don't overwrite a real comment
+    # with an empty one
+    def comment=(comment)
+      @comment = comment unless comment.empty?
+    end
+
+    # There's a wee trick we pull. Comment blocks can have directives that
+    # override the stuff we extract during the parse. So, we have a special
+    # class method, attr_overridable, that lets code objects list
+    # those directives. Wehn a comment is assigned, we then extract
+    # out any matching directives and update our object
+
+    def CodeObject.attr_overridable(name, *aliases)
+      @overridables ||= {}
+
+      attr_accessor name
+
+      aliases.unshift name
+      aliases.each do |directive_name|
+        @overridables[directive_name.to_s] = name
+      end
+    end
+
+  end
+
+  # A Context is something that can hold modules, classes, methods, 
+  # attributes, aliases, requires, and includes. Classes, modules, and
+  # files are all Contexts.
+
+  class Context < CodeObject
+    attr_reader   :name, :method_list, :attributes, :aliases, :constants
+    attr_reader   :requires, :includes, :in_files, :visibility
+    
+
+    def initialize
+      super()
+
+      @in_files    = []
+
+      @name    ||= "unknown"
+      @comment ||= ""
+      @parent  = nil
+      @visibility = :public
+
+      initialize_methods_etc
+      initialize_classes_and_modules
+    end
+
+    # map the class hash to an array externally
+    def classes
+      @classes.values
+    end
+
+    # map the module hash to an array externally
+    def modules
+      @modules.values
+    end
+
+    # Change the default visibility for new methods
+    def ongoing_visibility=(vis)
+      @visibility = vis
+    end
+
+    # Given an array +methods+ of method names, set the
+    # visibility of the corresponding AnyMethod object
+
+    def set_visibility_for(methods, vis, singleton=false)
+      @method_list.each_with_index do |m,i|
+        if methods.include?(m.name) && m.singleton == singleton
+          m.visibility = vis
+        end
+      end
+    end
+
+    # Record the file that we happen to find it in
+    def record_location(toplevel)
+      @in_files << toplevel unless @in_files.include?(toplevel)
+    end
+
+    # Return true if at least part of this thing was defined in +file+
+    def defined_in?(file)
+      @in_files.include?(file)
+    end
+
+    def add_class(class_type, name, superclass)
+      add_class_or_module(@classes, class_type, name, superclass)
+    end
+
+    def add_module(class_type, name)
+      add_class_or_module(@modules, class_type, name, nil)
+    end
+
+    def add_method(a_method)
+      puts "Adding #@visibility method #{a_method.name} to #@name" if $DEBUG
+      a_method.visibility = @visibility
+      add_to(@method_list, a_method)
+    end
+
+    def add_attribute(an_attribute)
+      add_to(@attributes, an_attribute)
+    end
+
+    def add_alias(an_alias)
+      meth = find_method_named(an_alias.old_name)
+      if meth
+        new_meth = AnyMethod.new(an_alias.text, an_alias.new_name)
+        new_meth.is_alias_for = meth
+        new_meth.singleton    = meth.singleton
+        new_meth.params       = meth.params
+        new_meth.comment = "Alias for \##{meth.name}"
+        meth.add_alias(new_meth)
+        add_method(new_meth)
+      else
+        add_to(@aliases, an_alias)
+      end
+    end
+
+    def add_include(an_include)
+      add_to(@includes, an_include)
+    end
+
+    def add_constant(const)
+      add_to(@constants, const)
+    end
+
+    # Requires always get added to the top-level (file) context
+    def add_require(a_require)
+      if self.kind_of? TopLevel
+        add_to(@requires, a_require)
+      else
+        parent.add_require(a_require)
+      end
+    end
+
+    def add_class_or_module(collection, class_type, name, superclass=nil)
+      cls = collection[name]
+      if cls
+        puts "Reusing class/module #{name}" if $DEBUG
+      else
+        cls = class_type.new(name, superclass)
+        puts "Adding class/module #{name} to #@name" if $DEBUG
+        collection[name] = cls
+        cls.parent = self
+      end
+      cls
+    end
+
+    def add_to(array, thing)
+      array <<  thing if @document_self
+      thing.parent = self
+    end
+
+    # If a class's documentation is turned off after we've started
+    # collecting methods etc., we need to remove the ones
+    # we have
+
+    def remove_methods_etc
+      initialize_methods_etc
+    end
+
+    def initialize_methods_etc
+      @method_list = []
+      @attributes  = []
+      @aliases     = []
+      @requires    = []
+      @includes    = []
+      @constants   = []
+    end
+
+    # and remove classes and modules when we see a :nodoc: all
+    def remove_classes_and_modules
+      initialize_classes_and_modules
+    end
+
+    def initialize_classes_and_modules
+      @classes     = {}
+      @modules     = {}
+    end
+
+    # Find a named module
+    def find_module_named(name)
+      return self if self.name == name
+      res = @modules[name] || @classes[name]
+      return res if res
+      parent && parent.find_module_named(name)
+    end
+
+    # Iterate over all the classes and modules in
+    # this object
+
+    def each_classmodule
+      @modules.each_value {|m| yield m}
+      @classes.each_value {|c| yield c}
+    end
+
+    def each_method
+      @method_list.each {|m| yield m}
+    end
+
+    def each_attribute 
+      @attributes.each {|a| yield a}
+    end
+
+    def each_constant
+      @constants.each {|c| yield c}
+    end
+
+    # Return the toplevel that owns us
+
+    def toplevel
+      return @toplevel if defined? @toplevel
+      @toplevel = self
+      @toplevel = @toplevel.parent until TopLevel === @toplevel
+      @toplevel
+    end
+
+    # allow us to sort modules by name
+    def <=>(other)
+      name <=> other.name
+    end
+
+    # Look up the given symbol. If method is non-nil, then
+    # we assume the symbol references a module that
+    # contains that method
+    def find_symbol(symbol, method=nil)
+      result = nil
+      case symbol
+      when /^::(.*)/
+        result = toplevel.find_symbol(symbol)
+      when /::/
+        modules = symbol.split(/::/)
+        unless modules.empty?
+          module_name = modules.shift
+          result = find_module_named(module_name)
+          if result
+            modules.each do |module_name|
+              result = result.find_module_named(module_name)
+            end
+          end
+        end
+      else
+        result = find_local_symbol(symbol)
+        if result.nil?
+          if symbol =~ /^[A-Z]/
+            result = parent
+            while result && result.name != symbol
+              result = result.parent
+            end
+          end
+        end
+      end
+      if result && method
+        result = result.find_local_symbol(method)
+      end
+      result
+    end
+           
+    def find_local_symbol(symbol)
+      res = find_method_named(symbol) ||
+            find_constant_named(symbol) ||
+            find_attribute_named(symbol) ||
+            find_module_named(symbol) 
+    end
+
+    private
+
+    # Find a named method, or return nil
+    def find_method_named(name)
+      @method_list.find {|meth| meth.name == name}
+    end
+
+    # Find a named constant, or return nil
+    def find_constant_named(name)
+      @constants.find {|m| m.name == name}
+    end
+
+    # Find a named attribute, or return nil
+    def find_attribute_named(name)
+      @attributes.find {|m| m.name == name}
+    end
+    
+  end
+
+
+  # A TopLevel context is a source file
+
+  class TopLevel < Context
+    attr_accessor :file_stat
+    attr_accessor :file_relative_name
+    attr_accessor :file_absolute_name
+    attr_accessor :diagram
+    
+    @@all_classes = {}
+    @@all_modules = {}
+
+    def TopLevel::reset
+      @@all_classes = {}
+      @@all_modules = {}
+    end
+
+    def initialize(file_name)
+      super()
+      @name = "TopLevel"
+      @file_relative_name = file_name
+      @file_absolute_name = file_name
+      @file_stat          = File.stat(file_name)
+      @diagram            = nil
+    end
+
+    def full_name
+      nil
+    end
+
+    # Adding a class or module to a TopLevel is special, as we only
+    # want one copy of a particular top-level class. For example,
+    # if both file A and file B implement class C, we only want one
+    # ClassModule object for C. This code arranges to share
+    # classes and modules between files.
+
+    def add_class_or_module(collection, class_type, name, superclass)
+      cls = collection[name]
+      if cls
+        puts "Reusing class/module #{name}" if $DEBUG
+      else
+        if class_type == NormalModule
+          all = @@all_modules
+        else
+          all = @@all_classes
+        end
+        cls = all[name]
+        if !cls
+          cls = class_type.new(name, superclass)
+          all[name] = cls
+        end
+        puts "Adding class/module #{name} to #@name" if $DEBUG
+        collection[name] = cls
+        cls.parent = self
+      end
+      cls
+    end
+
+    def TopLevel.all_classes_and_modules
+      @@all_classes.values + @@all_modules.values
+    end
+
+    def TopLevel.find_class_named(name)
+     @@all_classes.each_value do |c|
+        res = c.find_class_named(name) 
+        return res if res
+      end
+      nil
+    end
+
+    def find_local_symbol(symbol)
+      find_class_or_module_named(symbol) || super
+    end
+
+    def find_class_or_module_named(symbol)
+      @@all_classes.each_value {|c| return c if c.name == symbol}
+      @@all_modules.each_value {|m| return m if m.name == symbol}
+      nil
+    end
+
+  end
+
+  # ClassModule is the base class for objects representing either a
+  # class or a module.
+
+  class ClassModule < Context
+
+    attr_reader   :superclass
+    attr_accessor :diagram
+
+    def initialize(name, superclass = nil)
+      @name       = name
+      @diagram    = nil
+      @superclass = superclass
+      @comment    = ""
+      super()
+    end
+
+    # Return the fully qualified name of this class or module
+    def full_name
+      if @parent && @parent.full_name
+        @parent.full_name + "::" + @name
+      else
+        @name
+      end
+    end
+
+    def http_url(prefix)
+      path = full_name.split("::")
+      File.join(prefix, *path) + ".html"
+    end
+
+    # Return +true+ if this object represents a module
+    def is_module?
+      false
+    end
+
+    # to_s is simply for debugging
+    def to_s
+      res = self.class.name + ": " + @name 
+      res << @comment.to_s
+      res << super
+      res
+    end
+
+    def find_class_named(name)
+      return self if full_name == name
+      @classes.each_value {|c| return c if c.find_class_named(name) }
+      nil
+    end
+  end
+
+  # Anonymous classes
+  class AnonClass < ClassModule
+  end
+
+  # Normal classes
+  class NormalClass < ClassModule
+  end
+
+  # Singleton classes
+  class SingleClass < ClassModule
+  end
+
+  # Module
+  class NormalModule < ClassModule
+    def is_module?
+      true
+    end
+  end
+
+
+  # AnyMethod is the base class for objects representing methods
+
+  class AnyMethod < CodeObject
+    attr_accessor :name
+    attr_accessor :visibility
+    attr_accessor :block_params
+    attr_accessor :dont_rename_initialize
+    attr_accessor :singleton
+    attr_reader   :aliases           # list of other names for this method
+    attr_accessor :is_alias_for      # or a method we're aliasing
+
+    attr_overridable :params, :param, :parameters, :parameter
+
+    include TokenStream
+
+    def initialize(text, name)
+      super()
+      @text = text
+      @name = name
+      @token_stream  = nil
+      @visibility    = :public
+      @dont_rename_initialize = false
+      @block_params  = nil
+      @aliases       = []
+      @is_alias_for  = nil
+      @comment = ""
+    end
+
+    def <=>(other)
+      @name <=> other.name
+    end
+
+    def to_s
+      res = self.class.name + ": " + @name + " (" + @text + ")\n"
+      res << @comment.to_s
+      res
+    end
+
+    def param_seq
+      p = params.gsub(/\s*\#.*/, '')
+      p = p.tr("\n", " ").squeeze(" ")
+      p = "(" + p + ")" unless p[0] == ?(
+
+      if (block = block_params)
+        block.gsub!(/\s*\#.*/, '')
+        block = block.tr("\n", " ").squeeze(" ")
+        if block[0] == ?(
+          block.sub!(/^\(/, '').sub!(/\)/, '')
+        end
+        p << " {|#{block}| ...}"
+      end
+      p
+    end
+
+    def add_alias(method)
+      @aliases << method
+    end
+  end
+
+
+  # Represent an alias, which is an old_name/ new_name pair associated
+  # with a particular context
+  class Alias < CodeObject
+    attr_accessor :text, :old_name, :new_name, :comment
+    
+    def initialize(text, old_name, new_name, comment)
+      super()
+      @text = text
+      @old_name = old_name
+      @new_name = new_name
+      self.comment = comment
+    end
+
+    def to_s
+      "alias: #{self.old_name} ->  #{self.new_name}\n#{self.comment}"
+    end
+  end
+
+  # Represent a constant
+  class Constant < CodeObject
+    attr_accessor :name, :value
+
+    def initialize(name, value, comment)
+      super()
+      @name = name
+      @value = value
+      self.comment = comment
+    end
+  end
+
+  # Represent attributes
+  class Attr < CodeObject
+    attr_accessor :text, :name, :rw
+
+    def initialize(text, name, rw, comment)
+      super()
+      @text = text
+      @name = name
+      @rw = rw
+      self.comment = comment
+    end
+
+    def to_s
+      "attr: #{self.name} #{self.rw}\n#{self.comment}"
+    end
+
+    def <=>(other)
+      self.name <=> other.name
+    end
+  end
+
+  # a required file
+
+  class Require < CodeObject
+    attr_accessor :name
+
+    def initialize(name, comment)
+      super()
+      @name = name.gsub(/'|"/, "") #'
+      self.comment = comment
+    end
+
+  end
+
+  # an included module
+  class Include < CodeObject
+    attr_accessor :name
+
+    def initialize(name, comment)
+      super()
+      @name = name
+      self.comment = comment
+    end
+
+  end
+
+end
diff --git a/lib/rdoc/diagram.rb b/lib/rdoc/diagram.rb
new file mode 100644
index 0000000000..3c7df03417
--- /dev/null
+++ b/lib/rdoc/diagram.rb
@@ -0,0 +1,333 @@
+# A wonderful hack by to draw package diagrams using the dot package.
+# Originally written by  Jah, team Enticla.
+#
+# You must have the V1.7 or later in your path
+# http://www.research.att.com/sw/tools/graphviz/
+
+require "dot/dot"
+require 'rdoc/options'
+
+module RDoc
+
+  # Draw a set of diagrams representing the modules and classes in the
+  # system. We draw one diagram for each file, and one for each toplevel
+  # class or module. This means there will be overlap. However, it also
+  # means that you'll get better context for objects.
+  #
+  # To use, simply
+  #
+  #   d = Diagram.new(info)   # pass in collection of top level infos
+  #   d.draw
+  #
+  # The results will be written to the +dot+ subdirectory. The process
+  # also sets the +diagram+ attribute in each object it graphs to
+  # the name of the file containing the image. This can be used
+  # by output generators to insert images.
+
+  class Diagram
+
+    FONT = "Arial"
+
+    DOT_PATH = "dot"
+
+    # Pass in the set of top level objects. The method also creates
+    # the subdirectory to hold the images
+
+    def initialize(info, options)
+      @info = info
+      @options = options
+      @counter = 0
+      File.makedirs(DOT_PATH)
+    end
+
+    # Draw the diagrams. We traverse the files, drawing a diagram for
+    # each. We also traverse each top-level class and module in that
+    # file drawing a diagram for these too. 
+
+    def draw
+      unless @options.quiet
+        $stderr.print "Diagrams: "
+        $stderr.flush
+      end
+
+      @info.each_with_index do |i, file_count|
+        @done_modules = {}
+        @local_names = find_names(i)
+        @global_names = []
+        @global_graph = graph = DOT::DOTDigraph.new('name' => 'TopLevel',
+                                    'label' => i.file_absolute_name,
+                                    'fontname' => FONT,
+                                    'fontsize' => '8',
+                                    'bgcolor'  => 'lightcyan1',
+                                    'compound' => 'true')
+        
+        # it's a little hack %) i'm too lazy to create a separate class
+        # for default node
+        graph << DOT::DOTNode.new('name' => 'node',
+                                  'fontname' => FONT,
+                                  'color' => 'black',
+                                  'fontsize' => 8)
+        
+        i.modules.each do |mod|
+          draw_module(mod, graph, true, i.file_relative_name)
+        end
+        add_classes(i, graph, i.file_relative_name)
+
+        i.diagram = convert_to_png("f_#{file_count}", graph, i.name)
+        
+        # now go through and document each top level class and
+        # module independently
+        i.modules.each_with_index do |mod, count|
+          @done_modules = {}
+          @local_names = find_names(mod)
+          @global_names = []
+
+          @global_graph = graph = DOT::DOTDigraph.new('name' => 'TopLevel',
+                                      'label' => i.full_name,
+                                      'fontname' => FONT,
+                                      'fontsize' => '8',
+                                      'bgcolor'  => 'lightcyan1',
+                                      'compound' => 'true')
+
+          graph << DOT::DOTNode.new('name' => 'node',
+                                    'fontname' => FONT,
+                                    'color' => 'black',
+                                    'fontsize' => 8)
+          draw_module(mod, graph, true)
+          mod.diagram = convert_to_png("m_#{file_count}_#{count}", 
+                                       graph, 
+                                       "Module: #{mod.name}")
+        end
+      end
+      $stderr.puts unless @options.quiet
+    end
+
+    #######
+    private
+    #######
+
+    def find_names(mod)
+      return [mod.full_name] + mod.classes.collect{|cl| cl.full_name} +
+        mod.modules.collect{|m| find_names(m)}.flatten
+    end
+
+    def find_full_name(name, mod)
+      full_name = name.dup
+      return full_name if @local_names.include?(full_name)
+      mod_path = mod.full_name.split('::')[0..-2]
+      unless mod_path.nil?
+        until mod_path.empty?
+          full_name = mod_path.pop + '::' + full_name
+          return full_name if @local_names.include?(full_name)
+        end
+      end
+      return name
+    end
+
+    def draw_module(mod, graph, toplevel = false, file = nil)
+      return if  @done_modules[mod.full_name] and not toplevel
+
+      @counter += 1
+      url = mod.http_url("classes")
+      m = DOT::DOTSubgraph.new('name' => "cluster_#{mod.full_name.gsub( /:/,'_' )}",
+                               'label' => mod.name,
+                               'fontname' => FONT,
+                               'color' => 'blue', 
+                               'style' => 'filled', 
+                               'URL'   => %{"#{url}"},
+                               'fillcolor' => toplevel ? 'palegreen1' : 'palegreen3')
+      
+      @done_modules[mod.full_name] = m
+      add_classes(mod, m, file)
+      graph << m
+
+      unless mod.includes.empty?
+        mod.includes.each do |m|
+          m_full_name = find_full_name(m.name, mod)
+          if @local_names.include?(m_full_name)
+            @global_graph << DOT::DOTEdge.new('from' => "#{m_full_name.gsub( /:/,'_' )}",
+                                      'to' => "#{mod.full_name.gsub( /:/,'_' )}",
+                                      'ltail' => "cluster_#{m_full_name.gsub( /:/,'_' )}",
+                                      'lhead' => "cluster_#{mod.full_name.gsub( /:/,'_' )}")
+          else
+            unless @global_names.include?(m_full_name)
+              path = m_full_name.split("::")
+              url = File.join('classes', *path) + ".html"
+              @global_graph << DOT::DOTNode.new('name' => "#{m_full_name.gsub( /:/,'_' )}",
+                                        'shape' => 'box',
+                                        'label' => "#{m_full_name}",
+                                        'URL'   => %{"#{url}"})
+              @global_names << m_full_name
+            end
+            @global_graph << DOT::DOTEdge.new('from' => "#{m_full_name.gsub( /:/,'_' )}",
+                                      'to' => "#{mod.full_name.gsub( /:/,'_' )}",
+                                      'lhead' => "cluster_#{mod.full_name.gsub( /:/,'_' )}")
+          end
+        end
+      end
+    end
+
+    def add_classes(container, graph, file = nil )
+
+      use_fileboxes = Options.instance.fileboxes
+
+      files = {}
+
+      # create dummy node (needed if empty and for module includes)
+      if container.full_name
+        graph << DOT::DOTNode.new('name'     => "#{container.full_name.gsub( /:/,'_' )}",
+                                  'label'    => "",
+                                  'width'  => (container.classes.empty? and 
+                                               container.modules.empty?) ? 
+                                  '0.75' : '0.01',
+                                  'height' => '0.01',
+                                  'shape' => 'plaintext')
+      end
+      container.classes.each_with_index do |cl, cl_index|
+        last_file = cl.in_files[-1].file_relative_name
+
+        if use_fileboxes && !files.include?(last_file)
+          @counter += 1
+          files[last_file] =
+            DOT::DOTSubgraph.new('name'     => "cluster_#{@counter}",
+                                 'label'    => "#{last_file}",
+                                 'fontname' => FONT,
+                                 'color'=>
+                                 last_file == file ? 'red' : 'black')
+        end
+
+        next if cl.name == 'Object' || cl.name[0,2] == "<<"
+
+        url = cl.http_url("classes")
+        
+        label = cl.name.dup
+        if use_fileboxes && cl.in_files.length > 1
+          label <<  '\n[' + 
+                        cl.in_files.collect {|i|
+                             i.file_relative_name 
+                        }.sort.join( '\n' ) +
+                    ']'
+        end 
+                
+        attrs = {
+          'name' => "#{cl.full_name.gsub( /:/, '_' )}",
+          'fontcolor' => 'black',
+          'style'=>'filled',
+          'color'=>'palegoldenrod',
+          'label' => label,
+          'shape' => 'ellipse',
+          'URL'   => %{"#{url}"}
+        }
+
+        c = DOT::DOTNode.new(attrs)
+        
+        if use_fileboxes
+          files[last_file].push c 
+        else
+          graph << c
+        end
+      end
+      
+      if use_fileboxes
+        files.each_value do |val|
+          graph << val
+        end
+      end
+      
+      unless container.classes.empty?
+        container.classes.each_with_index do |cl, cl_index|
+          cl.includes.each do |m|
+            m_full_name = find_full_name(m.name, cl)
+            if @local_names.include?(m_full_name)
+              @global_graph << DOT::DOTEdge.new('from' => "#{m_full_name.gsub( /:/,'_' )}",
+                                      'to' => "#{cl.full_name.gsub( /:/,'_' )}",
+                                      'ltail' => "cluster_#{m_full_name.gsub( /:/,'_' )}")
+            else
+              unless @global_names.include?(m_full_name)
+                path = m_full_name.split("::")
+                url = File.join('classes', *path) + ".html"
+                @global_graph << DOT::DOTNode.new('name' => "#{m_full_name.gsub( /:/,'_' )}",
+                                          'shape' => 'box',
+                                          'label' => "#{m_full_name}",
+                                          'URL'   => %{"#{url}"})
+                @global_names << m_full_name
+              end
+              @global_graph << DOT::DOTEdge.new('from' => "#{m_full_name.gsub( /:/,'_' )}",
+                                      'to' => "#{cl.full_name.gsub( /:/, '_')}")
+            end
+          end
+
+          sclass = cl.superclass
+          next if sclass.nil? || sclass == 'Object'
+          sclass_full_name = find_full_name(sclass,cl)
+          unless @local_names.include?(sclass_full_name) or @global_names.include?(sclass_full_name)
+            path = sclass_full_name.split("::")
+            url = File.join('classes', *path) + ".html"
+            @global_graph << DOT::DOTNode.new(
+                       'name' => "#{sclass_full_name.gsub( /:/, '_' )}",
+                       'label' => sclass_full_name,
+                       'URL'   => %{"#{url}"})
+            @global_names << sclass_full_name
+          end
+          @global_graph << DOT::DOTEdge.new('from' => "#{sclass_full_name.gsub( /:/,'_' )}",
+                                    'to' => "#{cl.full_name.gsub( /:/, '_')}")
+        end
+      end
+
+      container.modules.each do |submod|
+        draw_module(submod, graph)
+      end
+      
+    end
+
+    def convert_to_png(file_base, graph, name)
+      op_type = Options.instance.image_format
+      dotfile = File.join(DOT_PATH, file_base)
+      src = dotfile + ".dot"
+      dot = dotfile + "." + op_type
+
+      unless @options.quiet
+        $stderr.print "."
+        $stderr.flush
+      end
+
+      File.open(src, 'w+' ) do |f|
+        f << graph.to_s << "\n"
+      end
+      
+      system "dot -T#{op_type} #{src} -o #{dot}"
+
+      # Now construct the imagemap wrapper around
+      # that png
+
+      return wrap_in_image_map(src, dot, name)
+    end
+
+    # Extract the client-side image map from dot, and use it
+    # to generate the imagemap proper. Return the whole
+    # <map>..<img> combination, suitable for inclusion on
+    # the page
+
+    def wrap_in_image_map(src, dot, name)
+      res = %{<map name="map">\n}
+      dot_map = `dot -Tismap #{src}`
+      dot_map.each do |area|
+        unless area =~ /^rectangle \((\d+),(\d+)\) \((\d+),(\d+)\) ([\/\w.]+)\s*(.*)/
+          $stderr.puts "Unexpected output from dot:\n#{area}"
+          return nil
+        end
+        
+        blx = $1; bly = $2
+        trx = $3; try = $4
+        url = $5; area_name = $6
+        res <<  %{  <area shape="RECT" coords="#{blx},#{try},#{trx},#{bly}" }
+        res <<  %{     href="#{url}" alt="#{area_name}">\n}
+      end
+      res << "</map>\n"
+#      map_file = src.sub(/.dot/, '.map')
+#      system("dot -Timap #{src} -o #{map_file}")
+      res << %{<img src="#{dot}" usemap="#map" border=0 alt="#{name}">}
+      return res
+    end
+  end
+end
diff --git a/lib/rdoc/generators/chm_generator.rb b/lib/rdoc/generators/chm_generator.rb
new file mode 100644
index 0000000000..1c46769cc2
--- /dev/null
+++ b/lib/rdoc/generators/chm_generator.rb
@@ -0,0 +1,112 @@
+require 'rdoc/generators/html_generator'
+
+module Generators
+
+  class CHMGenerator < HTMLGenerator
+
+    HHC_PATH = "c:\\Program Files\\HTML Help Workshop\\hhc.exe"
+
+    # Standard generator factory
+    def CHMGenerator.for(options)
+      CHMGenerator.new(options)
+    end
+
+    
+    def initialize(*args)
+      super
+      @op_name = @options.op_name || "rdoc"
+      check_for_html_help_workshop
+    end
+
+    def check_for_html_help_workshop
+      stat = File.stat(HHC_PATH)
+    rescue
+      $stderr <<
+	"\n.chm output generation requires that Microsoft's Html Help\n" <<
+	"Workshop is installed. RDoc looks for it in:\n\n    " <<
+	HHC_PATH <<
+	"\n\nYou can download a copy for free from:\n\n" <<
+	"    http://msdn.microsoft.com/library/default.asp?" <<
+	"url=/library/en-us/htmlhelp/html/hwMicrosoftHTMLHelpDownloads.asp\n\n"
+      
+      exit 99
+    end
+
+    # Generate the html as normal, then wrap it
+    # in a help project
+    def generate(info)
+      super
+      @project_name = @op_name + ".hhp"
+      create_help_project
+    end
+
+    # The project contains the project file, a table of contents
+    # and an index
+    def create_help_project
+      create_project_file
+      create_contents_and_index
+      compile_project
+    end
+
+    # The project file links together all the various
+    # files that go to make up the help.
+
+    def create_project_file
+      template = TemplatePage.new(RDoc::Page::HPP_FILE)
+      values = { "title" => @options.title, "opname" => @op_name }
+      files = []
+      @files.each do |f|
+	files << { "html_file_name" => f.path }
+      end
+
+      values['all_html_files'] = files
+      
+      File.open(@project_name, "w") do |f|
+        template.write_html_on(f, values)
+      end
+    end
+
+    # The contents is a list of all files and modules.
+    # For each we include  as sub-entries the list
+    # of methods they contain. As we build the contents
+    # we also build an index file
+
+    def create_contents_and_index
+      contents = []
+      index    = []
+
+      (@files+@classes).sort.each do |entry|
+	content_entry = { "c_name" => entry.name, "ref" => entry.path }
+	index << { "name" => entry.name, "aref" => entry.path }
+
+	internals = []
+
+	methods = entry.build_method_summary_list(entry.path)
+
+	content_entry["methods"] = methods unless methods.empty?
+        contents << content_entry
+	index.concat methods
+      end
+
+      values = { "contents" => contents }
+      template = TemplatePage.new(RDoc::Page::CONTENTS)
+      File.open("contents.hhc", "w") do |f|
+	template.write_html_on(f, values)
+      end
+
+      values = { "index" => index }
+      template = TemplatePage.new(RDoc::Page::CHM_INDEX)
+      File.open("index.hhk", "w") do |f|
+	template.write_html_on(f, values)
+      end      
+    end
+
+    # Invoke the windows help compiler to compiler the project
+    def compile_project
+      system("\"#{HHC_PATH}\" #@project_name")
+    end
+
+  end
+
+
+end
diff --git a/lib/rdoc/generators/html_generator.rb b/lib/rdoc/generators/html_generator.rb
new file mode 100644
index 0000000000..36e2f351da
--- /dev/null
+++ b/lib/rdoc/generators/html_generator.rb
@@ -0,0 +1,1370 @@
+# We're responsible for generating all the HTML files
+# from the object tree defined in code_objects.rb. We
+# generate:
+#
+# [files]   an html file for each input file given. These
+#           input files appear as objects of class
+#           TopLevel
+#
+# [classes] an html file for each class or module encountered.
+#           These classes are not grouped by file: if a file
+#           contains four classes, we'll generate an html
+#           file for the file itself, and four html files 
+#           for the individual classes. 
+#
+# [indices] we generate three indices for files, classes,
+#           and methods. These are displayed in a browser
+#           like window with three index panes across the
+#           top and the selected description below
+#
+# Method descriptions appear in whatever entity (file, class,
+# or module) that contains them.
+#
+# We generate files in a structure below a specified subdirectory,
+# normally +doc+.
+#
+#  opdir
+#     |
+#     |___ files
+#     |       |__  per file summaries
+#     |
+#     |___ classes
+#             |__ per class/module descriptions
+#
+# HTML is generated using the Template class.
+#
+
+require 'ftools'
+
+require 'rdoc/options'
+require 'rdoc/template'
+require 'rdoc/markup/simple_markup'
+require 'rdoc/markup/simple_markup/to_html'
+require 'cgi'
+
+module Generators
+
+  # Name of sub-direcories that hold file and class/module descriptions
+
+  FILE_DIR  = "files"
+  CLASS_DIR = "classes"
+  CSS_NAME  = "rdoc-style.css"
+  
+
+  ##
+  # Build a hash of all items that can be cross-referenced.
+  # This is used when we output required and included names: 
+  # if the names appear in this hash, we can generate
+  # an html cross reference to the appropriate description.
+  # We also use this when parsing comment blocks: any decorated 
+  # words matching an entry in this list are hyperlinked.
+
+  class AllReferences
+    @@refs = {}
+
+    def AllReferences::reset
+      @@refs = {}
+    end
+
+    def AllReferences.add(name, html_class)
+      @@refs[name] = html_class
+    end
+
+    def AllReferences.[](name)
+      @@refs[name]
+    end
+  end
+
+
+  ##
+  # Subclass of the SM::ToHtml class that supports looking
+  # up words in the AllReferences list. Those that are
+  # found (like AllReferences in this comment) will
+  # be hyperlinked
+
+  class HyperlinkHtml < SM::ToHtml
+    # We need to record the html path of our caller so we can generate
+    # correct relative paths for any hyperlinks that we find
+    def initialize(from_path, context)
+      super()
+      @from_path = from_path
+
+      @parent_name = context.parent_name
+      @parent_name += "::" if @parent_name
+      @context = context
+    end
+
+    # We're invoked when any text matches the CROSSREF pattern
+    # (defined in MarkUp). If we fine the corresponding reference,
+    # generate a hyperlink. If the name we're looking for contains
+    # no punctuation, we look for it up the module/class chain. For
+    # example, HyperlinkHtml is found, even without the Generators::
+    # prefix, because we look for it in module Generators first.
+
+    def handle_special_CROSSREF(special)
+      name = special.text
+      if name[0,1] == '#'
+        lookup = name[1..-1]
+        name = lookup unless Options.instance.show_hash
+      else
+        lookup = name
+      end
+
+      if /([A-Z].*)[.\#](.*)/ =~ lookup
+        container = $1
+        method = $2
+        ref = @context.find_symbol(container, method)
+      else
+        ref = @context.find_symbol(lookup)
+      end
+
+      if ref and ref.document_self
+        "<a href=\"#{ref.as_href(@from_path)}\">#{name}</a>"
+      else
+        name
+      end
+    end
+
+    # And we're invoked with a potential external hyperlink mailto:
+    # just gets inserted. http: links are checked to see if they
+    # reference an image. If so, that image gets inserted using an
+    # <img> tag. Otherwise a conventional <a href> is used.  We also
+    # support a special type of hyperlink, link:, which is a reference
+    # to a local file whose path is relative to the --op directory.
+
+    def handle_special_HYPERLINK(special)
+      url = special.text
+      if url =~ /([A-Za-z]+):(.*)/
+        type = $1
+        path = $2
+      else
+        type = "http"
+        path = url
+        url  = "http://#{url}"
+      end
+
+      if type == "link"
+        if path[0,1] == '#'     # is this meaningful?
+          url = path
+        else
+          url = HTMLGenerator.gen_url(@from_path, path)
+        end
+      end
+
+      if (type == "http" || type == "link") && 
+          url =~ /\.(gif|png|jpg|jpeg|bmp)$/
+
+        "<img src=\"#{url}\">"
+      else
+        "<a href=\"#{url}\">#{url.sub(%r{^\w+:/*}, '')}</a>"
+      end
+    end
+
+    # HEre's a hypedlink where the label is different to the URL
+    #  <label>[url]
+    #
+    
+    def handle_special_TIDYLINK(special)
+      text = special.text
+      unless text =~ /(\S+)\[(.*?)\]/
+        return text
+      end
+      label = $1
+      url   = $2
+      
+      unless url =~ /\w+?:/
+        url = "http://#{url}"
+      end
+      
+      "<a href=\"#{url}\">#{label}</a>"
+    end
+
+  end
+
+
+  
+  #####################################################################
+  #
+  # Handle common markup tasks for the various Html classes
+  #
+
+  module MarkUp
+
+    # Convert a string in markup format into HTML. We keep a cached
+    # SimpleMarkup object lying around after the first time we're
+    # called per object.
+
+    def markup(str, remove_para=false)
+      return '' unless str
+      unless defined? @markup
+        @markup = SM::SimpleMarkup.new
+
+        # class names, variable names, file names, or instance variables
+        @markup.add_special(/(
+                               \b([A-Z]\w*(::\w+)*[.\#]\w+)  #    A::B.meth
+                             | \b([A-Z]\w+(::\w+)*)       #    A::B..
+                             | \#\w+[!?=]?                #    #meth_name 
+                             | \b\w+([_\/\.]+\w+)+[!?=]?  #    meth_name
+                             )/x, 
+                            :CROSSREF)
+
+        # external hyperlinks
+        @markup.add_special(/((link:|http:|mailto:|ftp:|www\.)\S+\w)/, :HYPERLINK)
+
+        # and links of the form  <text>[<url>]
+        @markup.add_special(/\b(\S+?\[\S+?\.\S+?\])/, :TIDYLINK)
+
+      end
+      unless defined? @html_formatter
+        @html_formatter = HyperlinkHtml.new(self.path, self)
+      end
+
+      # Convert leading comment markers to spaces, but only
+      # if all non-blank lines have them
+
+      if str =~ /^(?>\s*)[^\#]/
+        content = str
+      else
+        content = str.gsub(/^\s*(#+)/)  { $1.tr('#',' ') }
+      end
+
+      res = @markup.convert(content, @html_formatter)
+      if remove_para
+        res.sub!(/^<p>/, '')
+        res.sub!(/<\/p>$/, '')
+      end
+      res
+    end
+
+    # Qualify a stylesheet URL; if if +css_name+ does not begin with '/' or
+    # 'http[s]://', prepend a prefix relative to +path+. Otherwise, return it
+    # unmodified.
+
+    def style_url(path, css_name=nil)
+#      $stderr.puts "style_url( #{path.inspect}, #{css_name.inspect} )"
+      css_name ||= CSS_NAME
+      if %r{^(https?:/)?/} =~ css_name
+        return css_name
+      else
+        return HTMLGenerator.gen_url(path, css_name)
+      end
+    end
+
+    # Build a webcvs URL with the given 'url' argument. URLs with a '%s' in them
+    # get the file's path sprintfed into them; otherwise they're just catenated
+    # together.
+
+    def cvs_url(url, full_path)
+      if /%s/ =~ url
+        return sprintf( url, full_path )
+      else
+        return url + full_path
+      end
+    end
+  end
+
+
+  #####################################################################
+  #
+  # A Context is built by the parser to represent a container: contexts
+  # hold classes, modules, methods, require lists and include lists.
+  # ClassModule and TopLevel are the context objects we process here
+  # 
+  class ContextUser
+
+    include MarkUp
+
+    attr_reader :context
+    
+    def initialize(context, options)
+      @context = context
+      @options = options
+    end
+      
+    # convenience method to build a hyperlink
+    def href(link, cls, name)
+      %{<a href="#{link}" class="#{cls}">#{name}</a>} #"
+    end
+
+    # return a reference to outselves to be used as an href=
+    # the form depends on whether we're all in one file
+    # or in multiple files
+
+    def as_href(from_path)
+      if @options.all_one_file
+        "#" + path
+      else
+        HTMLGenerator.gen_url(from_path, path)
+      end
+    end
+
+    # Create a list of HtmlMethod objects for each method
+    # in the corresponding context object. If the @options.show_all
+    # variable is set (corresponding to the <tt>--all</tt> option,
+    # we include all methods, otherwise just the public ones.
+
+    def collect_methods
+      list = @context.method_list
+      unless @options.show_all
+        list = list.find_all {|m| m.visibility == :public || m.force_documentation }
+      end
+      @methods = list.collect {|m| HtmlMethod.new(m, self, @options) }
+    end
+
+    # Build a summary list of all the methods in this context
+    def build_method_summary_list(path_prefix="")
+      collect_methods unless @methods
+      meths = @methods.sort
+      res = []
+      meths.each do |meth|
+	res << {
+          "name" => CGI.escapeHTML(meth.name),
+          "aref" => "#{path_prefix}\##{meth.aref}" 
+        }
+      end
+      res
+    end
+
+
+    # Build a list of aliases fo which we couldn't find a corresponding method
+    def build_alias_summary_list
+      @context.aliases.map do |al|
+        {
+          'old_name' => al.old_name,
+          'new_name' => al.new_name,
+        }
+      end
+    end
+    
+    # Build a list of constants
+    def build_constants_summary_list
+      @context.constants.map do |co|
+        res = {
+          'name'  => co.name,
+          'value' => co.value
+        }
+        res['desc'] = markup(co.comment, true) if co.comment && !co.comment.empty?
+        res
+      end
+    end
+    
+    def build_requires_list(context)
+      potentially_referenced_list(context.requires) {|fn| [fn + ".rb"] }
+    end
+
+    def build_include_list(context)
+      potentially_referenced_list(context.includes)
+    end
+
+    # Build a list from an array of <i>Htmlxxx</i> items. Look up each
+    # in the AllReferences hash: if we find a corresponding entry,
+    # we generate a hyperlink to it, otherwise just output the name.
+    # However, some names potentially need massaging. For example,
+    # you may require a Ruby file without the .rb extension,
+    # but the file names we know about may have it. To deal with
+    # this, we pass in a block which performs the massaging,
+    # returning an array of alternative names to match
+
+    def potentially_referenced_list(array)
+      res = []
+      array.each do |i|
+        ref = AllReferences[i.name] 
+        if !ref && block_given?
+          possibles = yield(i.name)
+          while !ref and !possibles.empty?
+            ref = AllReferences[possibles.shift]
+          end
+        end
+        h_name = CGI.escapeHTML(i.name)
+        if ref and ref.document_self
+          path = url(ref.path)
+          res << { "name" => h_name, "aref" => path }
+        else
+          res << { "name" => h_name }
+        end
+      end
+      res
+    end
+
+    # Build an array of arrays of method details. The outer array has up
+    # to six entries, public, private, and protected for both class
+    # methods, the other for instance methods. The inner arrays contain
+    # a hash for each method
+
+    def build_method_detail_list
+      outer = []
+
+      for singleton in [true, false]
+        for vis in [ :public, :protected, :private ] 
+          res = []
+          @methods.each do |m|
+            if m.document_self and m.visibility == vis and m.singleton == singleton
+              row = {}
+              row["name"]        = CGI.escapeHTML(m.name)
+              desc = m.description.strip
+              row["m_desc"]      = desc unless desc.empty?
+              row["params"]      = m.params
+              row["aref"]        = m.aref
+              row["visibility"]  = m.visibility.to_s
+
+              unless m.aliases.empty?
+                row["aka"] = m.aliases.map do |other| 
+                  {
+                    'name' => other.name,
+                    'aref'  => other.viewer.as_href(path)
+                  } 
+                end
+              end
+
+              if @options.inline_source
+                code = m.source_code
+                row["sourcecode"] = code if code
+              else
+                code = m.src_url
+                if code
+                  row["codeurl"] = code
+                  row["imgurl"]  = m.img_url
+                end
+              end
+              res << row
+            end
+          end
+          if res.size > 0 
+            outer << {
+              "type"    => vis.to_s.capitalize,
+              "category"    => singleton ? "Class" : "Instance",
+              "methods" => res
+            }
+          end
+        end
+      end
+      outer
+    end
+
+    # Build the structured list of classes and modules contained
+    # in this context. 
+
+    def build_class_list(level, from, infile=nil)
+      res = ""
+      prefix = "&nbsp;&nbsp;::" * level;
+
+      from.modules.sort.each do |mod|
+        next if infile && !mod.defined_in?(infile)
+        if mod.document_self
+          res << 
+            prefix <<
+            "Module " <<
+            href(url(mod.viewer.path), "link", mod.full_name) <<
+            "<br />\n" <<
+            build_class_list(level + 1, mod, infile)
+        end
+      end
+
+      from.classes.sort.each do |cls|
+        next if infile && !cls.defined_in?(infile)
+        if cls.document_self
+          res      <<
+            prefix << 
+            "Class " <<
+            href(url(cls.viewer.path), "link", cls.full_name) <<
+            "<br />\n" <<
+            build_class_list(level + 1, cls, infile)
+        end
+      end
+
+      res
+    end
+    
+    def url(target)
+      HTMLGenerator.gen_url(path, target)
+    end
+
+    def aref_to(target)
+      if @options.all_one_file
+        "#" + target
+      else
+        url(target)
+      end
+    end
+
+    def document_self
+      @context.document_self
+    end
+
+    def diagram_reference(diagram)
+      res = diagram.gsub(/((?:src|href)=")(.*?)"/) {
+        $1 + url($2) + '"'
+      }
+      res
+    end
+
+
+    # Find a symbol in ourselves or our parent
+    def find_symbol(symbol, method=nil)
+      res = @context.find_symbol(symbol, method)
+      if res
+        res = res.viewer
+      end
+      res
+    end
+  end
+
+  #####################################################################
+  #
+  # Wrap a ClassModule context
+
+  class HtmlClass < ContextUser
+
+    attr_reader :path
+
+    def initialize(context, html_file, prefix, options)
+      super(context, options)
+
+      @html_file = html_file
+      @is_module = context.is_module?
+      @values    = {}
+
+      context.viewer = self
+
+      if options.all_one_file
+        @path = context.full_name
+      else
+        @path = http_url(context.full_name, prefix)
+      end
+
+      collect_methods
+
+      AllReferences.add(name, self)
+    end
+
+    # return the relative file name to store this class in,
+    # which is also its url
+    def http_url(full_name, prefix)
+      path = full_name.dup
+      if path['<<']
+        path.gsub!(/<<\s*(\w*)/) { "from-#$1" }
+      end
+      File.join(prefix, path.split("::")) + ".html"
+    end
+
+
+    def name
+      @context.full_name
+    end
+
+    def parent_name
+      @context.parent.full_name
+    end
+
+    def index_name
+      name
+    end
+
+    def write_on(f)
+      value_hash
+      template = TemplatePage.new(RDoc::Page::BODY,
+                                      RDoc::Page::CLASS_PAGE,
+                                      RDoc::Page::METHOD_LIST)
+      template.write_html_on(f, @values)
+    end
+
+    def value_hash
+      class_attribute_values
+
+      @values["charset"] = @options.charset
+      @values["style_url"] = style_url(path, @options.css)
+
+      d = markup(@context.comment)
+      @values["description"] = d unless d.empty?
+
+      ml = build_method_summary_list
+      @values["methods"] = ml unless ml.empty?
+
+      al = build_alias_summary_list
+      @values["aliases"] = al unless al.empty?
+
+      co = build_constants_summary_list
+      @values["constants"] = co unless co.empty?
+
+      il = build_include_list(@context)
+      @values["includes"] = il unless il.empty?
+
+      al = build_attribute_list
+      @values["attributes"] = al unless al.empty?
+      
+      cl = build_class_list(0, @context)
+      @values["classlist"] = cl unless cl.empty?
+
+      mdl = build_method_detail_list
+      @values["method_list"] = mdl unless mdl.empty?
+      @values
+    end
+
+    def build_attribute_list
+      atts = @context.attributes.sort
+      res = []
+      atts.each do |att|
+        res << {
+          "name"   => CGI.escapeHTML(att.name), 
+          "rw"     => att.rw, 
+          "a_desc" => markup(att.comment, true)
+        }
+      end
+      res
+    end
+
+    def class_attribute_values
+      h_name = CGI.escapeHTML(name)
+
+      @values["classmod"]  = @is_module ? "Module" : "Class"
+      @values["title"]     = "#{@values['classmod']}: #{h_name}"
+
+      c = @context
+      c = c.parent while c and !c.diagram
+      if c && c.diagram
+        @values["diagram"] = diagram_reference(c.diagram)
+      end
+
+      @values["full_name"] = h_name
+
+      parent_class = @context.superclass
+
+      if parent_class
+	@values["parent"] = CGI.escapeHTML(parent_class)
+
+	if parent_name
+	  lookup = parent_name + "::" + parent_class
+	else
+	  lookup = parent_class
+	end
+
+	parent_url = AllReferences[lookup] || AllReferences[parent_class]
+
+	if parent_url and parent_url.document_self
+	  @values["par_url"] = aref_to(parent_url.path)
+	end
+      end
+
+      files = []
+      @context.in_files.each do |f|
+        res = {}
+        full_path = CGI.escapeHTML(f.file_absolute_name)
+
+        res["full_path"]     = full_path
+        res["full_path_url"] = aref_to(f.viewer.path) if f.document_self
+
+        if @options.webcvs
+          res["cvsurl"] = cvs_url( @options.webcvs, full_path )
+        end
+
+        files << res
+      end
+
+      @values['infiles'] = files
+
+    end
+
+    def <=>(other)
+      self.name <=> other.name
+    end
+
+  end
+
+  #####################################################################
+  #
+  # Handles the mapping of a file's information to HTML. In reality,
+  # a file corresponds to a +TopLevel+ object, containing modules,
+  # classes, and top-level methods. In theory it _could_ contain
+  # attributes and aliases, but we ignore these for now.
+
+  class HtmlFile < ContextUser
+
+    attr_reader :path
+    attr_reader :name
+
+    def initialize(context, options, file_dir)
+      super(context, options)
+
+      @values = {}
+
+      if options.all_one_file
+        @path = filename_to_label
+      else
+        @path = http_url(file_dir)
+      end
+
+      @name = @context.file_relative_name
+
+      collect_methods
+      AllReferences.add(name, self)
+      context.viewer = self
+    end
+
+    def http_url(file_dir)
+      File.join(file_dir, @context.file_relative_name.tr('.', '_')) +
+        ".html"
+    end
+
+    def filename_to_label
+      @context.file_relative_name.gsub(/%|\/|\?|\#/) {|s| '%' + ("%x" % s[0]) }
+    end
+
+    def index_name
+      name
+    end
+
+    def parent_name
+      nil
+    end
+
+    def value_hash
+      file_attribute_values
+
+      @values["charset"]   = @options.charset
+      @values["href"]      = path
+      @values["style_url"] = style_url(path, @options.css)
+
+      if @context.comment
+        d = markup(@context.comment)
+        @values["description"] = d if d.size > 0
+      end
+
+      ml = build_method_summary_list
+      @values["methods"] = ml unless ml.empty?
+
+      il = build_include_list(@context)
+      @values["includes"] = il unless il.empty?
+
+      rl = build_requires_list(@context)
+      @values["requires"] = rl unless rl.empty?
+
+      co = build_constants_summary_list
+      @values["constants"] = co unless co.empty?
+
+      if @options.promiscuous
+        file_context = nil
+      else
+        file_context = @context
+      end
+
+      cl = build_class_list(0, @context, file_context)
+      @values["classlist"] = cl unless cl.empty?
+
+      mdl = build_method_detail_list
+      @values["method_list"] = mdl unless mdl.empty?
+      @values
+    end
+    
+    def write_on(f)
+      value_hash
+      template = TemplatePage.new(RDoc::Page::BODY,
+                                  RDoc::Page::FILE_PAGE,
+                                  RDoc::Page::METHOD_LIST)
+      template.write_html_on(f, @values)
+    end
+
+    def file_attribute_values
+      full_path = @context.file_absolute_name
+      short_name = File.basename(full_path)
+      
+      @values["title"] = CGI.escapeHTML("File: #{short_name}")
+
+      if @context.diagram
+        @values["diagram"] = diagram_reference(@context.diagram)
+      end
+
+      @values["short_name"]   = CGI.escapeHTML(short_name)
+      @values["full_path"]    = CGI.escapeHTML(full_path)
+      @values["dtm_modified"] = @context.file_stat.mtime.to_s
+
+      if @options.webcvs
+        @values["cvsurl"] = cvs_url( @options.webcvs, @values["full_path"] )
+      end
+    end
+
+    def <=>(other)
+      self.name <=> other.name
+    end
+  end
+
+  #####################################################################
+
+  class HtmlMethod
+    include MarkUp
+
+    attr_reader :context
+    attr_reader :src_url
+    attr_reader :img_url
+    attr_reader :source_code
+
+    @@seq = "M000000"
+
+    @@all_methods = []
+
+    def HtmlMethod::reset
+      @@all_methods = []
+    end
+
+    def initialize(context, html_class, options)
+      @context    = context
+      @html_class = html_class
+      @options    = options
+      @@seq       = @@seq.succ
+      @seq        = @@seq
+      @@all_methods << self
+
+      context.viewer = self
+
+      if (ts = @context.token_stream)
+        @source_code = markup_code(ts)
+        unless @options.inline_source
+          @src_url = create_source_code_file(@source_code)
+          @img_url = HTMLGenerator.gen_url(path, 'source.png')
+        end
+      end
+
+      AllReferences.add(name, self)
+    end
+    
+    # return a reference to outselves to be used as an href=
+    # the form depends on whether we're all in one file
+    # or in multiple files
+
+    def as_href(from_path)
+      if @options.all_one_file
+        "#" + path
+      else
+        HTMLGenerator.gen_url(from_path, path)
+      end
+    end
+
+    def name
+      @context.name
+    end
+
+    def index_name
+      "#{@context.name} (#{@html_class.name})"
+    end
+
+    def parent_name
+      if @context.parent.parent
+        @context.parent.parent.full_name
+      else
+        nil
+      end
+    end
+
+    def aref
+      @seq
+    end
+
+    def path
+      if @options.all_one_file
+	aref
+      else
+	@html_class.path + "#" + aref
+      end
+    end
+
+    def description
+      markup(@context.comment)
+    end
+
+    def visibility
+      @context.visibility
+    end
+
+    def singleton
+      @context.singleton
+    end
+
+    def params
+      p = @context.params.gsub(/\s*\#.*/, '')
+      p = p.tr("\n", " ").squeeze(" ")
+      p = "(" + p + ")" unless p[0] == ?(
+
+      if (block = @context.block_params)
+        block.gsub!(/\s*\#.*/, '')
+        block = block.tr("\n", " ").squeeze(" ")
+        if block[0] == ?(
+          block.sub!(/^\(/, '').sub!(/\)/, '')
+        end
+        p << " {|#{block.strip}| ...}"
+      end
+
+      CGI.escapeHTML(p)
+    end
+    
+    def create_source_code_file(code_body)
+      meth_path = @html_class.path.sub(/\.html$/, '.src')
+      File.makedirs(meth_path)
+      file_path = File.join(meth_path, @seq) + ".html"
+
+      template = TemplatePage.new(RDoc::Page::SRC_PAGE)
+      File.open(file_path, "w") do |f|
+        values = {
+          'title'     => CGI.escapeHTML(index_name),
+          'code'      => code_body,
+          'style_url' => style_url(path, @options.css),
+          'charset'   => @options.charset
+        }
+        template.write_html_on(f, values)
+      end
+      HTMLGenerator.gen_url(path, file_path)
+    end
+
+    def HtmlMethod.all_methods
+      @@all_methods
+    end
+
+    def <=>(other)
+      @context <=> other.context
+    end
+
+    ##
+    # Given a sequence of source tokens, mark up the source code
+    # to make it look purty.
+
+
+    def markup_code(tokens)
+      src = ""
+      tokens.each do |t|
+        next unless t
+        #    p t.class
+#        style = STYLE_MAP[t.class]
+        style = case t
+                when RubyToken::TkCONSTANT then "ruby-constant"
+                when RubyToken::TkKW       then "ruby-keyword kw"
+                when RubyToken::TkIVAR     then "ruby-ivar"
+                when RubyToken::TkOp       then "ruby-operator"
+                when RubyToken::TkId       then "ruby-identifier"
+                when RubyToken::TkNode     then "ruby-node"
+                when RubyToken::TkCOMMENT  then "ruby-comment cmt"
+                when RubyToken::TkREGEXP   then "ruby-regexp re"
+                when RubyToken::TkSTRING   then "ruby-value str"
+                when RubyToken::TkVal      then "ruby-value"
+                else
+                    nil
+                end
+
+        text = CGI.escapeHTML(t.text)
+#         case t
+#         when RubyToken::TkKW
+#           style = "kw"
+#         when RubyToken::TkCOMMENT
+#           style = "cmt"
+#         when RubyToken::TkSTRING
+#           style = "str"
+#         when RubyToken::TkREGEXP
+#           style = "re"
+#         end
+        if style
+          src << "<span class=\"#{style}\">#{text}</span>"
+        else
+          src << text
+        end
+      end
+
+      add_line_numbers(src) if Options.instance.include_line_numbers
+      src
+    end
+
+    # we rely on the fact that the first line of a source code
+    # listing has 
+    #    # File xxxxx, line dddd
+
+    def add_line_numbers(src)
+      if src =~ /\A.*, line (\d+)/
+        first = $1.to_i - 1
+        last  = first + src.count("\n")
+        size = last.to_s.length
+        real_fmt = "%#{size}d: "
+        fmt = " " * (size+2)
+        src.gsub!(/^/) do
+          res = sprintf(fmt, first) 
+          first += 1
+          fmt = real_fmt
+          res
+        end
+      end
+    end
+
+    def document_self
+      @context.document_self
+    end
+
+    def aliases
+      @context.aliases
+    end
+
+    def find_symbol(symbol, method=nil)
+      res = @context.parent.find_symbol(symbol, method)
+      if res
+        res = res.viewer
+      end
+      res
+    end
+  end
+
+  #####################################################################
+
+  class HTMLGenerator
+
+    include MarkUp
+
+    ##
+    # convert a target url to one that is relative to a given
+    # path
+    
+    def HTMLGenerator.gen_url(path, target)
+      from          = File.dirname(path)
+      to, to_file   = File.split(target)
+      
+      from = from.split("/")
+      to   = to.split("/")
+      
+      while from.size > 0 and to.size > 0 and from[0] == to[0]
+        from.shift
+        to.shift
+      end
+      
+      from.fill("..")
+      from.concat(to)
+      from << to_file
+      File.join(*from)
+    end
+
+    # Generators may need to return specific subclasses depending
+    # on the options they are passed. Because of this
+    # we create them using a factory
+
+    def HTMLGenerator.for(options)
+      AllReferences::reset
+      HtmlMethod::reset
+
+      if options.all_one_file
+        HTMLGeneratorInOne.new(options)
+      else
+        HTMLGenerator.new(options)
+      end
+    end
+
+    class <<self
+      protected :new
+    end
+
+    # Set up a new HTML generator. Basically all we do here is load
+    # up the correct output temlate
+
+    def initialize(options) #:not-new:
+      @options    = options
+      load_html_template
+    end
+
+
+    ##
+    # Build the initial indices and output objects
+    # based on an array of TopLevel objects containing
+    # the extracted information. 
+
+    def generate(toplevels)
+      @toplevels  = toplevels
+      @files      = []
+      @classes    = []
+
+      write_style_sheet
+      gen_sub_directories()
+      build_indices
+      generate_html
+    end
+
+    private
+
+    ##
+    # Load up the HTML template specified in the options
+    #
+    def load_html_template
+      template = File.join("rdoc/generators/template",
+                           @options.generator.key, @options.template)
+      require template
+      extend RDoc::Page
+    rescue LoadError
+      $stderr.puts "Could not find HTML template '#{template}'"
+      exit 99
+    end
+
+    ##
+    # Write out the style sheet used by the main frames
+    #
+    
+    def write_style_sheet
+      template = TemplatePage.new(RDoc::Page::STYLE)
+      unless @options.css
+        File.open(CSS_NAME, "w") do |f|
+          values = { "fonts" => RDoc::Page::FONTS }
+          template.write_html_on(f, values)
+        end
+      end
+    end
+
+    ##
+    # See the comments at the top for a description of the
+    # directory structure
+
+    def gen_sub_directories
+      File.makedirs(FILE_DIR, CLASS_DIR)
+    rescue 
+      $stderr.puts $!.message
+      exit 1
+    end
+
+    ##
+    # Generate:
+    #
+    # * a list of HtmlFile objects for each TopLevel object.
+    # * a list of HtmlClass objects for each first level
+    #   class or module in the TopLevel objects
+    # * a complete list of all hyperlinkable terms (file,
+    #   class, module, and method names)
+
+    def build_indices
+
+      @toplevels.each do |toplevel|
+        @files << HtmlFile.new(toplevel, @options, FILE_DIR)
+      end
+
+      RDoc::TopLevel.all_classes_and_modules.each do |cls|
+        build_class_list(cls, @files[0], CLASS_DIR)
+      end
+    end
+
+    def build_class_list(from, html_file, class_dir)
+      @classes << HtmlClass.new(from, html_file, class_dir, @options)
+      from.each_classmodule do |mod|
+        build_class_list(mod, html_file, class_dir)
+      end
+    end
+
+    ##
+    # Generate all the HTML
+    #
+    def generate_html
+      # the individual descriptions for files and classes
+      gen_into(@files)
+      gen_into(@classes)
+      # and the index files
+      gen_file_index
+      gen_class_index
+      gen_method_index
+      gen_main_index
+      
+      # this method is defined in the template file
+      write_extra_pages if defined? write_extra_pages
+    end
+
+    def gen_into(list)
+      list.each do |item|
+        if item.document_self
+          op_file = item.path
+          File.makedirs(File.dirname(op_file))
+          File.open(op_file, "w") { |file| item.write_on(file) }
+        end
+      end
+
+    end
+
+    def gen_file_index
+      gen_an_index(@files, 'Files', 
+                   RDoc::Page::FILE_INDEX, 
+                   "fr_file_index.html")
+    end
+
+    def gen_class_index
+      gen_an_index(@classes, 'Classes',
+                   RDoc::Page::CLASS_INDEX,
+                   "fr_class_index.html")
+    end
+
+    def gen_method_index
+      gen_an_index(HtmlMethod.all_methods, 'Methods', 
+                   RDoc::Page::METHOD_INDEX,
+                   "fr_method_index.html")
+    end
+
+    
+    def gen_an_index(collection, title, template, filename)
+      template = TemplatePage.new(RDoc::Page::FR_INDEX_BODY, template)
+      res = []
+      collection.sort.each do |f|
+        if f.document_self
+          res << { "href" => f.path, "name" => f.index_name }
+        end
+      end
+
+      values = {
+        "entries"    => res,
+        'list_title' => CGI.escapeHTML(title),
+        'index_url'  => main_url,
+        'charset'    => @options.charset,
+        'style_url'  => style_url('', @options.css),
+      }
+
+      File.open(filename, "w") do |f|
+        template.write_html_on(f, values)
+      end
+    end
+
+    # The main index page is mostly a template frameset, but includes
+    # the initial page. If the <tt>--main</tt> option was given,
+    # we use this as our main page, otherwise we use the
+    # first file specified on the command line.
+
+    def gen_main_index
+      template = TemplatePage.new(RDoc::Page::INDEX)
+      File.open("index.html", "w") do |f|
+        values = {
+          "initial_page" => main_url,
+          'title'        => CGI.escapeHTML(@options.title),
+          'charset'      => @options.charset
+        }
+        template.write_html_on(f, values)
+      end
+    end
+
+    # return the url of the main page
+    def main_url
+      main_page = @options.main_page
+      ref = nil
+      if main_page
+        ref = AllReferences[main_page]
+        if ref
+          ref = ref.path
+        else
+          $stderr.puts "Could not find main page #{main_page}"
+        end
+      end
+
+      ref = @files[0].path unless ref
+
+      ref
+    end
+
+
+  end
+
+
+  ######################################################################
+
+
+  class HTMLGeneratorInOne < HTMLGenerator
+
+    def initialize(*args)
+      super
+    end
+
+    ##
+    # Build the initial indices and output objects
+    # based on an array of TopLevel objects containing
+    # the extracted information. 
+
+    def generate(info)
+      @toplevels  = info
+      @files      = []
+      @classes    = []
+      @hyperlinks = {}
+
+      build_indices
+      generate_xml
+    end
+
+
+    ##
+    # Generate:
+    #
+    # * a list of HtmlFile objects for each TopLevel object.
+    # * a list of HtmlClass objects for each first level
+    #   class or module in the TopLevel objects
+    # * a complete list of all hyperlinkable terms (file,
+    #   class, module, and method names)
+
+    def build_indices
+
+      @toplevels.each do |toplevel|
+        @files << HtmlFile.new(toplevel, @options, FILE_DIR)
+      end
+
+      RDoc::TopLevel.all_classes_and_modules.each do |cls|
+        build_class_list(cls, @files[0], CLASS_DIR)
+      end
+    end
+
+    def build_class_list(from, html_file, class_dir)
+      @classes << HtmlClass.new(from, html_file, class_dir, @options)
+      from.each_classmodule do |mod|
+        build_class_list(mod, html_file, class_dir)
+      end
+    end
+
+    ##
+    # Generate all the HTML. For the one-file case, we generate
+    # all the information in to one big hash
+    #
+    def generate_xml
+      values = { 
+        'charset' => @options.charset,
+        'files'   => gen_into(@files),
+        'classes' => gen_into(@classes),
+        'title'        => CGI.escapeHTML(@options.title),
+      }
+      
+      # this method is defined in the template file
+      write_extra_pages if defined? write_extra_pages
+
+      template = TemplatePage.new(RDoc::Page::ONE_PAGE)
+
+      if @options.op_name
+        opfile = File.open(@options.op_name, "w")
+      else
+        opfile = $stdout
+      end
+      template.write_html_on(opfile, values)
+    end
+
+    def gen_into(list)
+      res = []
+      list.each do |item|
+        res << item.value_hash
+      end
+      res
+    end
+
+    def gen_file_index
+      gen_an_index(@files, 'Files')
+    end
+
+    def gen_class_index
+      gen_an_index(@classes, 'Classes')
+    end
+
+    def gen_method_index
+      gen_an_index(HtmlMethod.all_methods, 'Methods')
+    end
+
+    
+    def gen_an_index(collection, title)
+      res = []
+      collection.sort.each do |f|
+        if f.document_self
+          res << { "href" => f.path, "name" => f.index_name }
+        end
+      end
+
+      return {
+        "entries" => res,
+        'list_title' => title,
+        'index_url'  => main_url,
+      }
+    end
+
+  end
+end
diff --git a/lib/rdoc/generators/template/chm/chm.rb b/lib/rdoc/generators/template/chm/chm.rb
new file mode 100644
index 0000000000..6df83a7773
--- /dev/null
+++ b/lib/rdoc/generators/template/chm/chm.rb
@@ -0,0 +1,86 @@
+module RDoc
+module Page
+
+require "rdoc/generators/template/html/html"
+
+# This is a nasty little hack, but hhc doesn't support the <?xml
+# tag, so...
+
+BODY.sub!(/<\?xml.*\?>/, '')
+
+HPP_FILE = %{
+[OPTIONS]
+Auto Index = Yes
+Compatibility=1.1 or later
+Compiled file=%opname%.chm
+Contents file=contents.hhc
+Full-text search=Yes
+Index file=index.hhk
+Language=0x409 English(United States)
+Title=%title%
+
+[FILES]
+START:all_html_files
+%html_file_name%
+END:all_html_files
+}
+
+CONTENTS = %{
+<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
+<HTML>
+<HEAD>
+<meta name="GENERATOR" content="Microsoft&reg; HTML Help Workshop 4.1">
+<!-- Sitemap 1.0 -->
+</HEAD><BODY>
+<OBJECT type="text/site properties">
+	<param name="Foreground" value="0x80">
+	<param name="Window Styles" value="0x800025">
+	<param name="ImageType" value="Folder">
+</OBJECT>
+<UL>
+START:contents
+	<LI> <OBJECT type="text/sitemap">
+		<param name="Name" value="%c_name%">
+		<param name="Local" value="%ref%">
+		</OBJECT>
+IF:methods
+<ul>
+START:methods
+	<LI> <OBJECT type="text/sitemap">
+		<param name="Name" value="%name%">
+		<param name="Local" value="%aref%">
+		</OBJECT>
+END:methods
+</ul>
+ENDIF:methods
+        </LI>
+END:contents
+</UL>
+</BODY></HTML>
+}
+
+
+CHM_INDEX  = %{
+<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
+<HTML>
+<HEAD>
+<meta name="GENERATOR" content="Microsoft&reg; HTML Help Workshop 4.1">
+<!-- Sitemap 1.0 -->
+</HEAD><BODY>
+<OBJECT type="text/site properties">
+	<param name="Foreground" value="0x80">
+	<param name="Window Styles" value="0x800025">
+	<param name="ImageType" value="Folder">
+</OBJECT>
+<UL>
+START:index
+	<LI> <OBJECT type="text/sitemap">
+		<param name="Name" value="%name%">
+		<param name="Local" value="%aref%">
+		</OBJECT>
+END:index
+</UL>
+</BODY></HTML>
+}
+end
+end
diff --git a/lib/rdoc/generators/template/html/css2.rb b/lib/rdoc/generators/template/html/css2.rb
new file mode 100644
index 0000000000..0109ae3e7f
--- /dev/null
+++ b/lib/rdoc/generators/template/html/css2.rb
@@ -0,0 +1,631 @@
+#
+# = CSS2 RDoc HTML template
+#
+# This is a template for RDoc that uses XHTML 1.0 Transitional and dictates a
+# bit more of the appearance of the output to cascading stylesheets than the
+# default. It was designed for clean inline code display, and uses DHTMl to
+# toggle the visbility of each method's source with each click on the '[source]'
+# link.
+#
+# == Authors
+#
+# * Michael Granger <ged@FaerieMUD.org>
+#
+# Copyright (c) 2002, 2003 The FaerieMUD Consortium. Some rights reserved.
+#
+# This work is licensed under the Creative Commons Attribution License. To view
+# a copy of this license, visit http://creativecommons.org/licenses/by/1.0/ or
+# send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California
+# 94305, USA.
+#
+
+module RDoc
+	module Page
+
+		FONTS = "Verdana,Arial,Helvetica,sans-serif"
+
+STYLE = %{
+body {
+    margin: 0;
+    padding: 0;
+}
+
+h1,h2,h3,h4 { margin: 0; color: #efefef; background: transparent; }
+h1 { font-size: 120%; }
+h2,h3,h4 { margin-top: 1em; }
+
+a { background: #eef; color: #039; text-decoration: none; }
+a:hover { background: #039; color: #eef; }
+
+/* Override the base stylesheet's Anchor inside a table cell */
+td > a {
+	background: transparent;
+	color: #039;
+	text-decoration: none;
+}
+
+/* === Structural elements =================================== */
+
+div#index {
+    margin: 0;
+    padding: 0;
+    font-size: 0.9em;
+}
+
+div#index a {
+    margin-left: 0.7em;
+}
+
+div#classHeader {
+    width: auto;
+    background: #039;
+    color: white;
+    padding: 0.5em 1.5em 0.5em 1.5em;
+    margin: 0;
+    border-bottom: 3px solid #006;
+}
+
+div#classHeader a {
+    background: inherit;
+    color: white;
+}
+
+div#classHeader td {
+    background: inherit;
+    color: white;
+}
+
+div#fileHeader {
+    width: auto;
+    background: #039;
+    color: white;
+    padding: 0.5em 1.5em 0.5em 1.5em;
+    margin: 0;
+    border-bottom: 3px solid #006;
+}
+
+div#fileHeader a {
+    background: inherit;
+    color: white;
+}
+
+div#fileHeader td {
+    background: inherit;
+    color: white;
+}
+
+div#bodyContent {
+    padding: 0 1.5em 0 1.5em;
+}
+
+div#description {
+    padding: 0.5em 1.5em;
+    background: #efefef;
+    border: 1px dotted #999;
+}
+
+div#description h1,h2,h3,h4,h5,h6 {
+    color: black;
+    background: transparent;
+}
+
+div#validator-badges {
+    text-align: center;
+}
+div#validator-badges img { border: 0; }
+
+div#copyright {
+    color: #333;
+    background: #efefef;
+    font: 0.75em sans-serif;
+    margin-top: 5em;
+    margin-bottom: 0;
+    padding: 0.5em 2em;
+}
+
+
+/* === Classes =================================== */
+
+table.header-table {
+    color: white;
+    font-size: small;
+}
+
+.type-note {
+    font-size: small;
+    color: #DEDEDE;
+}
+
+.section-bar {
+    background: #eee;
+    color: #333;
+    padding: 3px;
+    border: 1px solid #999;
+}
+
+.top-aligned-row { vertical-align: vertical-align: top }
+
+/* --- Context section classes ----------------------- */
+
+.context-row { }
+.context-item-name { font-family: monospace; font-weight: bold; color: black; }
+.context-item-value { font-size: x-small; color: #448; }
+.context-item-desc { background: #efefef; }
+
+/* --- Method classes -------------------------- */
+.method-detail {
+    background: #EFEFEF;
+    padding: 0;
+    margin-top: 0.5em;
+    margin-bottom: 0.5em;
+    border: 1px dotted #DDD;
+}
+.method-heading {
+	color: black;
+	background: #AAA;
+	border-bottom: 1px solid #666;
+	padding: 0.2em 0.5em 0 0.5em;
+}
+.method-signature { color: black; background: inherit; }
+.method-name { font-weight: bold; }
+.method-args { font-style: italic; }
+.method-description { padding: 0 0.5em 0 0.5em; }
+
+/* --- Source code sections -------------------- */
+
+a.source-toggle { font-size: 90%; }
+div.method-source-code {
+    background: #262626;
+    color: #ffdead;
+	margin: 1em;
+    padding: 0.5em;
+    border: 1px dashed #999;
+    overflow: hidden;
+}
+
+div.method-source-code pre { color: #ffdead; overflow: hidden; }
+
+/* --- Ruby keyword styles --------------------- */
+/* (requires a hacked html_generator.rb to add more class-types) */
+.ruby-constant	{ color: #7fffd4; background: transparent; }
+.ruby-keyword	{ color: #00ffff; background: transparent; }
+.ruby-ivar		{ color: #eedd82; background: transparent; }
+.ruby-operator	{ color: #00ffee; background: transparent; }
+.ruby-identifier { color: #ffdead; background: transparent; }
+.ruby-node		{ color: #ffa07a; background: transparent; }
+.ruby-comment	{ color: #b22222; font-weight: bold; background: transparent; }
+.ruby-regexp	{ color: #ffa07a; background: transparent; }
+.ruby-value		{ color: #7fffd4; background: transparent; }
+}
+
+
+#####################################################################
+###	H E A D E R   T E M P L A T E  
+#####################################################################
+
+XHTML_PREAMBLE = %{<?xml version="1.0" encoding="%charset%"?>
+<!DOCTYPE html 
+     PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+     "DTD/xhtml1-transitional.dtd">
+}
+
+HEADER = XHTML_PREAMBLE + %{
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+	<title>%title%</title>
+	<meta http-equiv="Content-Type" content="text/html; charset=%charset%" />
+	<meta http-equiv="Content-Script-Type" content="text/javascript" />
+	<link rel="stylesheet" href="%style_url%" type="text/css" media="screen" />
+	<script type="text/javascript">
+	// <![CDATA[
+
+	function popupCode( url ) {
+		window.open(url, "Code", "resizable=yes,scrollbars=yes,toolbar=no,status=no,height=150,width=400")
+	}
+
+	function toggleCode( id ) {
+		if ( document.getElementById )
+			elem = document.getElementById( id );
+		else if ( document.all )
+			elem = eval( "document.all." + id );
+		else
+			return false;
+
+		elemStyle = elem.style;
+		
+		if ( elemStyle.display != "block" ) {
+			elemStyle.display = "block"
+		} else {
+			elemStyle.display = "none"
+		}
+
+		return true;
+	}
+	
+	// Make codeblocks hidden by default
+	document.writeln( "<style type=\\"text/css\\">div.method-source-code { display: none }</style>" )
+	
+	// ]]>
+	</script>
+
+</head>
+<body>
+}
+
+
+#####################################################################
+###	C O N T E X T   C O N T E N T   T E M P L A T E
+#####################################################################
+
+CONTEXT_CONTENT = %{
+	<div id="contextContent">
+IF:diagram
+		<div id="diagram">
+			%diagram%
+		</div>
+ENDIF:diagram
+
+IF:description
+		<div id="description">
+			%description%
+		</div>
+ENDIF:description
+
+IF:requires
+		<div id="requires-list">
+			<h2 class="section-bar">Required files</h2>
+
+			<div class="name-list">
+START:requires
+			HREF:aref:name:&nbsp;&nbsp;
+END:requires
+			</div>
+		</div>
+ENDIF:requires
+
+IF:methods
+		<div id="method-list">
+			<h2 class="section-bar">Methods</h2>
+
+			<div class="name-list">
+START:methods
+			HREF:aref:name:&nbsp;&nbsp;
+END:methods
+			</div>
+		</div>
+ENDIF:methods
+
+IF:constants
+		<div id="constants-list">
+			<h2 class="section-bar">Constants</h2>
+
+			<div class="name-list">
+				<table summary="Constants">
+START:constants
+				<tr class="top-aligned-row context-row">
+					<td class="context-item-name">%name%</td>
+					<td>=</td>
+					<td class="context-item-value">%value%</td>
+				</tr>
+IF:desc
+				<tr class="top-aligned-row context-row">
+					<td>&nbsp;</td>
+					<td colspan="2" class="context-item-desc">%desc%</td>
+				</tr>
+ENDIF:desc
+END:constants
+				</table>
+			</div>
+		</div>
+ENDIF:constants
+
+IF:aliases
+		<div id="aliases-list">
+			<h2 class="section-bar">External Aliases</h2>
+
+			<div class="name-list">
+START:aliases
+			%old_name% -> %new_name% <br />
+END:aliases
+			</div>
+		</div>
+ENDIF:aliases
+
+
+IF:attributes
+		<div id="attribute-list">
+			<h2 class="section-bar">Attributes</h2>
+
+			<div class="name-list">
+				<table>
+START:attributes
+				<tr class="top-aligned-row context-row">
+					<td class="context-item-name">%name%</td>
+					<td class="context-item-value">&nbsp;[%rw%]&nbsp;</td>
+					<td class="context-item-desc">%a_desc%</td>
+				</tr>
+END:attributes
+				</table>
+			</div>
+		</div>
+ENDIF:attributes
+			
+IF:classlist
+		<div id="class-list">
+			<h2 class="section-bar">Classes and Modules</h2>
+
+			%classlist%
+		</div>
+ENDIF:classlist
+
+	</div>
+
+}
+
+
+#####################################################################
+###	F O O T E R   T E M P L A T E
+#####################################################################
+FOOTER = %{
+<div id="validator-badges">
+  <p><small><a href="http://validator.w3.org/check/referer">[Validate]</a></small></p>
+</div>
+
+</body>
+</html>
+}
+
+
+#####################################################################
+###	F I L E   P A G E   H E A D E R   T E M P L A T E
+#####################################################################
+
+FILE_PAGE = %{
+	<div id="fileHeader">
+		<h1>%short_name%</h1>
+		<table class="header-table">
+		<tr class="top-aligned-row">
+			<td><strong>Path:</strong></td>
+			<td>%full_path%
+IF:cvsurl
+				&nbsp;(<a href="%cvsurl%">CVS</a>)
+ENDIF:cvsurl
+			</td>
+		</tr>
+		<tr class="top-aligned-row">
+			<td><strong>Last Update:</strong></td>
+			<td>%dtm_modified%</td>
+		</tr>
+		</table>
+	</div>
+}
+
+
+#####################################################################
+###	C L A S S   P A G E   H E A D E R   T E M P L A T E
+#####################################################################
+
+CLASS_PAGE = %{
+    <div id="classHeader">
+        <h1>%full_name% <sup class="type-note">(%classmod%)</sup></h1>
+        <table class="header-table">
+        <tr class="top-aligned-row">
+            <td><strong>In:</strong></td>
+            <td>
+START:infiles
+IF:full_path_url
+                <a href="%full_path_url%">
+ENDIF:full_path_url
+                %full_path%
+IF:full_path_url
+                </a>
+ENDIF:full_path_url
+IF:cvsurl
+				&nbsp;(<a href="%cvsurl%">CVS</a>)
+ENDIF:cvsurl
+				<br />
+END:infiles
+            </td>
+        </tr>
+
+IF:parent
+        <tr class="top-aligned-row">
+            <td><strong>Parent:</strong></td>
+            <td>
+IF:par_url
+                <a href="%par_url%">
+ENDIF:par_url
+                %parent%
+IF:par_url
+               </a>
+ENDIF:par_url
+            </td>
+        </tr>
+ENDIF:parent
+        </table>
+    </div>
+}
+
+
+#####################################################################
+###	M E T H O D   L I S T   T E M P L A T E
+#####################################################################
+
+METHOD_LIST = %{
+
+		<!-- if includes -->
+IF:includes
+		<div id="includes">
+			<h2 class="section-bar">Included Modules</h2>
+
+			<div id="includes-list">
+START:includes
+		    <span class="include-name">HREF:aref:name:</span>
+END:includes
+			</div>
+		</div>
+ENDIF:includes
+
+
+		<!-- if method_list -->
+IF:method_list
+		<div id="methods">
+START:method_list
+IF:methods
+			<h2 class="section-bar">%type% %category% methods</h2>
+
+START:methods
+			<!-- %name%%params% -->
+			<div id="method-%aref%" class="method-detail">
+				<a name="%aref%"></a>
+
+				<div class="method-heading">
+IF:codeurl
+					<a href="%codeurl%" target="Code" class="method-signature"
+						onclick="popupCode('%codeurl%');return false;">
+ENDIF:codeurl
+IF:sourcecode
+					<a href="#%aref%" class="method-signature">
+ENDIF:sourcecode
+					<span class="method-name">%name%</span><span class="method-args">%params%</span>
+IF:codeurl
+					</a>
+ENDIF:codeurl
+IF:sourcecode
+					</a>
+ENDIF:sourcecode
+				</div>
+			
+				<div class="method-description">
+IF:m_desc
+					%m_desc%
+ENDIF:m_desc
+IF:sourcecode
+					<p><a class="source-toggle" href="#"
+					  onclick="toggleCode('%aref%-source');return false;">[Source]</a></p>
+					<div class="method-source-code" id="%aref%-source">
+<pre>
+%sourcecode%
+</pre>
+					</div>
+ENDIF:sourcecode
+				</div>
+			</div>
+
+END:methods
+ENDIF:methods
+END:method_list
+
+		</div>
+ENDIF:method_list
+}
+
+
+#####################################################################
+###	B O D Y   T E M P L A T E
+#####################################################################
+
+BODY = HEADER + %{
+
+!INCLUDE!  <!-- banner header -->
+
+	<div id="bodyContent">
+
+} + CONTEXT_CONTENT + METHOD_LIST + %{
+
+	</div>
+
+} + FOOTER
+
+
+
+#####################################################################
+###	S O U R C E   C O D E   T E M P L A T E
+#####################################################################
+
+SRC_PAGE = XHTML_PREAMBLE + %{
+<!--
+
+    %title%
+
+  -->
+<html>
+<head>
+	<title>%title%</title>
+	<meta http-equiv="Content-Type" content="text/html; charset=%charset%" />
+	<link rel="stylesheet" href="http://www.FaerieMUD.org/stylesheets/rdoc.css" type="text/css" />
+</head>
+<body>
+	<pre>%code%</pre>
+</body>
+</html>
+}
+
+
+#####################################################################
+###	I N D E X   F I L E   T E M P L A T E S
+#####################################################################
+
+FR_INDEX_BODY = %{
+!INCLUDE!
+}
+
+FILE_INDEX = XHTML_PREAMBLE + %{
+<!--
+
+    %list_title%
+
+  -->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+	<title>%list_title%</title>
+	<meta http-equiv="Content-Type" content="text/html; charset=%charset%" />
+	<link rel="stylesheet" href="%style_url%" type="text/css" />
+	<base target="docwin" />
+</head>
+<body>
+<div id="index">
+	<h1 class="section-bar">%list_title%</h1>
+	<div id="index-entries">
+START:entries
+		<a href="%href%">%name%</a><br />
+END:entries
+	</div>
+</div>
+</body>
+</html>
+}
+
+CLASS_INDEX = FILE_INDEX
+METHOD_INDEX = FILE_INDEX
+
+INDEX = %{<?xml version="1.0" encoding="%charset%"?>
+<!DOCTYPE html 
+     PUBLIC "-//W3C//DTD XHTML 1.0 Frameset//EN"
+     "DTD/xhtml1-frameset.dtd">
+
+<!--
+
+    %title%
+
+  -->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+	<title>%title%</title>
+	<meta http-equiv="Content-Type" content="text/html; charset=%charset%" />
+</head>
+<frameset rows="20%, 80%">
+    <frameset cols="25%,35%,45%">
+        <frame src="fr_file_index.html"   title="Files" name="Files" />
+        <frame src="fr_class_index.html"  name="Classes" />
+        <frame src="fr_method_index.html" name="Methods" />
+    </frameset>
+    <frame src="%initial_page%" name="docwin" />
+</frameset>
+</html>
+}
+
+
+	end # module Page
+end # class RDoc
+
diff --git a/lib/rdoc/generators/template/html/hefss.rb b/lib/rdoc/generators/template/html/hefss.rb
new file mode 100644
index 0000000000..ff4f557b71
--- /dev/null
+++ b/lib/rdoc/generators/template/html/hefss.rb
@@ -0,0 +1,418 @@
+module RDoc
+module Page
+
+
+FONTS = "Verdana, Arial, Helvetica, sans-serif"
+
+STYLE = %{
+body,p { font-family: Verdana, Arial, Helvetica, sans-serif; 
+       color: #000040; background: #BBBBBB;
+}
+
+td { font-family: Verdana, Arial, Helvetica, sans-serif; 
+       color: #000040;
+}
+
+.attr-rw { font-size: small; color: #444488 }
+
+.title-row {color:      #eeeeff;
+	    background: #BBBBDD;
+}
+
+.big-title-font { color: white;
+                  font-family: Verdana, Arial, Helvetica, sans-serif;
+                  font-size: large; 
+                  height: 50px}
+
+.small-title-font { color: purple;
+                    font-family: Verdana, Arial, Helvetica, sans-serif;
+                    font-size: small; }
+
+.aqua { color: purple }
+
+.method-name, attr-name {
+      font-family: monospace; font-weight: bold;
+}
+
+.tablesubtitle {
+   width: 100%;
+   margin-top: 1ex;
+   margin-bottom: .5ex;
+   padding: 5px 0px 5px 20px;
+   font-size: large;
+   color: purple;
+   background: #BBBBCC;
+}
+
+.tablesubsubtitle {
+   width: 100%;
+   margin-top: 1ex;
+   margin-bottom: .5ex;
+   padding: 5px 0px 5px 20px;
+   font-size: medium;
+   color: white;
+   background: #BBBBCC;
+}
+
+.name-list {
+  font-family: monospace;
+  margin-left: 40px;
+  margin-bottom: 2ex;
+  line-height: 140%;
+}
+
+.description {
+  margin-left: 40px;
+  margin-bottom: 2ex;
+  line-height: 140%;
+}
+
+.methodtitle {
+  font-size: medium;
+  text_decoration: none;
+  padding: 3px 3px 3px 20px;
+  color: #0000AA;
+}
+
+.column-title {
+  font-size: medium;
+  font-weight: bold;
+  text_decoration: none;
+  padding: 3px 3px 3px 20px;
+  color: #3333CC;
+  }
+
+.variable-name {
+  font-family: monospace;
+  font-size: medium;
+  text_decoration: none;
+  padding: 3px 3px 3px 20px;
+  color: #0000AA;
+}
+
+.row-name {
+  font-size: medium;
+  font-weight: medium;
+  font-family: monospace;
+  text_decoration: none;
+  padding: 3px 3px 3px 20px;
+}
+
+.paramsig {
+   font-size: small;
+}
+
+.srcbut { float: right }
+
+}
+
+
+############################################################################
+
+
+BODY = %{
+<html><head>
+  <title>%title%</title>
+  <meta http-equiv="Content-Type" content="text/html; charset=%charset%">
+  <link rel=StyleSheet href="%style_url%" type="text/css" media=screen>
+  <script type="text/javascript" language="JavaScript">
+  <!--
+  function popCode(url) {
+    parent.frames.source.location = url
+  }
+  //-->
+  </script>
+</head>
+<body bgcolor="#BBBBBB">
+
+!INCLUDE!  <!-- banner header -->
+
+IF:diagram
+<table width="100%"><tr><td align="center">
+%diagram%
+</td></tr></table>
+ENDIF:diagram
+
+IF:description
+<div class="description">%description%</div>
+ENDIF:description
+
+IF:requires
+<table cellpadding=5 width="100%">
+<tr><td class="tablesubtitle">Required files</td></tr>
+</table><br>
+<div class="name-list">
+START:requires
+HREF:aref:name:
+END:requires
+ENDIF:requires
+</div>
+
+IF:methods
+<table cellpadding=5 width="100%">
+<tr><td class="tablesubtitle">Subroutines and Functions</td></tr>
+</table><br>
+<div class="name-list">
+START:methods
+HREF:aref:name:,
+END:methods
+</div>
+ENDIF:methods
+
+IF:attributes
+<table cellpadding=5 width="100%">
+<tr><td class="tablesubtitle">Arguments</td></tr>
+</table><br>
+<table cellspacing=5>
+START:attributes
+     <tr valign="top">
+IF:rw
+       <td align="center" class="attr-rw">&nbsp;[%rw%]&nbsp;</td>
+ENDIF:rw
+IFNOT:rw
+       <td></td>
+ENDIF:rw
+       <td class="attr-name">%name%</td>
+       <td>%a_desc%</td>
+     </tr>
+END:attributes
+</table>
+ENDIF:attributes
+
+IF:classlist
+<table cellpadding=5 width="100%">
+<tr><td class="tablesubtitle">Modules</td></tr>
+</table><br>
+%classlist%<br>
+ENDIF:classlist
+
+  !INCLUDE!  <!-- method descriptions -->
+
+</body>
+</html>
+}
+
+###############################################################################
+
+FILE_PAGE = <<_FILE_PAGE_
+<table width="100%">
+ <tr class="title-row">
+ <td><table width="100%"><tr>
+   <td class="big-title-font" colspan=2><font size=-3><B>File</B><BR></font>%short_name%</td>
+   <td align="right"><table cellspacing=0 cellpadding=2>
+         <tr>
+           <td  class="small-title-font">Path:</td>
+           <td class="small-title-font">%full_path%
+IF:cvsurl
+				&nbsp;(<a href="%cvsurl%">CVS</a>)
+ENDIF:cvsurl
+           </td>
+         </tr>
+         <tr>
+           <td class="small-title-font">Modified:</td>
+           <td class="small-title-font">%dtm_modified%</td>
+         </tr>
+        </table>
+    </td></tr></table></td>
+  </tr>
+</table><br>
+_FILE_PAGE_
+
+###################################################################
+
+CLASS_PAGE = %{
+<table width="100%" border=0 cellspacing=0>
+ <tr class="title-row">
+ <td class="big-title-font">
+   <font size=-3><B>%classmod%</B><BR></font>%full_name%
+ </td>
+ <td align="right">
+   <table cellspacing=0 cellpadding=2>
+     <tr valign="top">
+      <td class="small-title-font">In:</td>
+      <td class="small-title-font">
+START:infiles
+HREF:full_path_url:full_path:
+IF:cvsurl
+&nbsp;(<a href="%cvsurl%">CVS</a>)
+ENDIF:cvsurl
+END:infiles
+      </td>
+     </tr>
+IF:parent
+     <tr>
+      <td class="small-title-font">Parent:</td>
+      <td class="small-title-font">
+IF:par_url
+        <a href="%par_url%" class="cyan">
+ENDIF:par_url
+%parent%
+IF:par_url
+         </a>
+ENDIF:par_url
+      </td>
+     </tr>
+ENDIF:parent
+   </table>
+  </td>
+  </tr>
+</table><br>
+}
+
+###################################################################
+
+METHOD_LIST = %{
+IF:includes
+<div class="tablesubsubtitle">Uses</div><br>
+<div class="name-list">
+START:includes
+    <span class="method-name">HREF:aref:name:</span>
+END:includes
+</div>
+ENDIF:includes
+
+IF:method_list
+START:method_list
+IF:methods
+<table cellpadding=5 width="100%">
+<tr><td class="tablesubtitle">%type% %category% methods</td></tr>
+</table>
+START:methods
+<table width="100%" cellspacing = 0 cellpadding=5 border=0>
+<tr><td class="methodtitle">
+<a name="%aref%">
+<b>%name%</b>%params% 
+IF:codeurl
+<a href="%codeurl%" target="source" class="srclink">src</a>
+ENDIF:codeurl
+</a></td></tr>
+</table>
+IF:m_desc
+<div class="description">
+%m_desc%
+</div>
+ENDIF:m_desc
+END:methods
+ENDIF:methods
+END:method_list
+ENDIF:method_list
+}
+
+=begin
+=end
+
+########################## Source code ##########################
+
+SRC_PAGE = %{
+<html>
+<head><title>%title%</title>
+<meta http-equiv="Content-Type" content="text/html; charset=%charset%">
+<style>
+  .kw { color: #3333FF; font-weight: bold }
+  .cmt { color: green; font-style: italic }
+  .str { color: #662222; font-style: italic }
+  .re  { color: #662222; }
+.ruby-comment    { color: green; font-style: italic }
+.ruby-constant   { color: #4433aa; font-weight: bold; }
+.ruby-identifier { color: #222222;  }
+.ruby-ivar       { color: #2233dd; }
+.ruby-keyword    { color: #3333FF; font-weight: bold }
+.ruby-node       { color: #777777; }
+.ruby-operator   { color: #111111;  }
+.ruby-regexp     { color: #662222; }
+.ruby-value      { color: #662222; font-style: italic }
+</style>
+</head>
+<body bgcolor="#BBBBBB">
+<pre>%code%</pre>
+</body>
+</html>
+}
+
+########################## Index ################################
+
+FR_INDEX_BODY = %{
+!INCLUDE!
+}
+
+FILE_INDEX = %{
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=%charset%">
+<style>
+<!--
+  body {
+background-color: #bbbbbb;
+     font-family: #{FONTS}; 
+       font-size: 11px; 
+      font-style: normal;
+     line-height: 14px; 
+           color: #000040;
+  }
+div.banner {
+  background: #bbbbcc;
+  color:      white;
+  padding: 1;
+  margin: 0;
+  font-size: 90%;
+  font-weight: bold;
+  line-height: 1.1;
+  text-align: center;
+  width: 100%;
+}
+  
+-->
+</style>
+<base target="docwin">
+</head>
+<body>
+<div class="banner">%list_title%</div>
+START:entries
+<a href="%href%">%name%</a><br>
+END:entries
+</body></html>
+}
+
+CLASS_INDEX = FILE_INDEX
+METHOD_INDEX = FILE_INDEX
+
+INDEX = %{
+<html>
+<head>
+  <title>%title%</title>
+  <meta http-equiv="Content-Type" content="text/html; charset=%charset%">
+</head>
+
+<frameset cols="20%,*">
+    <frameset rows="15%,35%,50%">
+        <frame src="fr_file_index.html"   title="Files" name="Files">
+        <frame src="fr_class_index.html"  name="Modules">
+        <frame src="fr_method_index.html" name="Subroutines and Functions">
+    </frameset>
+    <frameset rows="80%,20%">
+      <frame  src="%initial_page%" name="docwin">
+      <frame  src="blank.html" name="source">
+    </frameset>
+    <noframes>
+          <body bgcolor="#BBBBBB">
+            Click <a href="html/index.html">here</a> for a non-frames
+            version of this page.
+          </body>
+    </noframes>
+</frameset>
+
+</html>
+}
+
+# and a blank page to use as a target
+BLANK = %{
+<html><body bgcolor="#BBBBBB"></body></html>
+}
+
+def write_extra_pages
+  template = TemplatePage.new(BLANK)
+  File.open("blank.html", "w") { |f| template.write_html_on(f, {}) }
+end
+
+end
+end
diff --git a/lib/rdoc/generators/template/html/html.rb b/lib/rdoc/generators/template/html/html.rb
new file mode 100644
index 0000000000..18ad9dcb3b
--- /dev/null
+++ b/lib/rdoc/generators/template/html/html.rb
@@ -0,0 +1,762 @@
+module RDoc
+
+# This is how you define the HTML that RDoc generates. Simply create
+# a file in rdoc/generators/html_templates that creates the
+# module RDoc::Page and populate it as described below. Then invoke
+# rdoc using the --template <name of your file> option, and
+# your template will be used.
+#
+# The constants defining pages use a simple templating system:
+#
+# * The templating system is passed a hash. Keys in the hash correspond
+#   to tags on this page. The tag %abc% is looked up in the hash,
+#   and is replaced by the corresponding hash value.
+#
+# * Some tags are optional. You can detect this using IF/ENDIF
+#
+#      IF: title
+#      The value of title is %title%
+#      ENDIF: title
+#
+# * Some entries in the hash have values that are arrays, where each
+#   entry in the array is itself a hash. These are used to generate
+#   lists using the START: construct. For example, given a hash
+#   containing
+# 
+#      { 'people' => [ { 'name' => 'Fred', 'age' => '12' },
+#                      { 'name' => 'Mary', 'age' => '21' } ]
+#
+#   You could generate a simple table using
+#
+#      <table>
+#      START:people
+#        <tr><td>%name%<td>%age%</tr>
+#      END:people
+#      </table>
+#
+#   These lists can be nested to an arbitrary depth
+#
+# * the construct HREF:url:name: generates <a href="%url%">%name%</a>
+#   if +url+ is defined in the hash, or %name% otherwise.
+#
+#
+# Your file must contain the following constants
+#
+# [*FONTS*]  a list of fonts to be used
+# [*STYLE*]  a CSS section (without the <style> or comments). This is
+#            used to generate a style.css file
+#
+# [*BODY*] 
+#   The main body of all non-index RDoc pages. BODY will contain
+#   two !INCLUDE!s. The first is used to include a document-type
+#   specific header (FILE_PAGE or CLASS_PAGE). The second include
+#   is for the method list (METHOD_LIST). THe body is passed:
+#
+#   %title%::
+#       the page's title
+#
+#   %style_url%::
+#       the url of a style sheet for this page
+#
+#   %diagram%::
+#       the optional URL of a diagram for this page
+#
+#   %description%::
+#       a (potentially multi-paragraph) string containing the
+#       description for th file/class/module.
+#
+#   %requires%::
+#       an optional list of %aref%/%name% pairs, one for each module
+#       required by this file.
+#
+#   %methods%::
+#       an optional list of %aref%/%name%, one for each method
+#       documented on this page. This is intended to be an index.
+#
+#   %attributes%::  
+#       An optional list. For each attribute it contains:
+#       %name%::   the attribute name
+#       %rw%::     r/o, w/o, or r/w
+#       %a_desc%:: description of the attribute
+#
+#   %classlist%:: 
+#       An optional string containing an already-formatted list of
+#       classes and modules documented in this file
+#
+#   For FILE_PAGE entries, the body will be passed
+#
+#   %short_name%::  
+#       The name of the file
+#
+#   %full_path%::
+#       The full path to the file
+#
+#   %dtm_modified%::
+#       The date/time the file was last changed
+#
+#   For class and module pages, the body will be passed
+#
+#   %classmod%::
+#       The name of the class or module
+#
+#   %files%::
+#       A list. For each file this class is defined in, it contains:
+#       %full_path_url%:: an (optional) URL of the RDoc page
+#                         for this file
+#       %full_path%::     the name of the file
+#
+#   %par_url%::
+#       The (optional) URL of the RDoc page documenting this class's
+#       parent class
+#
+#   %parent%::
+#       The name of this class's parent.
+#
+#   For both files and classes, the body is passed the following information
+#   on includes and methods:
+#
+#   %includes%::    
+#       Optional list of included modules. For each, it receives
+#       %aref%:: optional URL to RDoc page for the module
+#       %name%:: the name of the module
+#
+#   %method_list%::
+#       Optional list of methods of a particular class and category.
+#
+#   Each method list entry contains:
+#
+#   %type%::        public/private/protected
+#   %category%::    instance/class
+#   %methods%::     a list of method descriptions
+#
+#   Each method description contains:
+#
+#   %aref%::        a target aref, used when referencing this method
+#                   description. You should code this as <a name="%aref%">
+#   %codeurl%::     the optional URL to the page containing this method's
+#                   source code.
+#   %name%::        the method's name
+#   %params%::      the method's parameters
+#   %m_desc%::      the (potentially multi-paragraph) description of
+#                   this method.
+#
+# [*CLASS_PAGE*]
+#         Header for pages documenting classes and modules. See
+#         BODY above for the available parameters.
+#
+# [*FILE_PAGE*]
+#         Header for pages documenting files. See
+#         BODY above for the available parameters.
+#
+# [*METHOD_LIST*]
+#         Controls the display of the listing of methods. See BODY for
+#         parameters.
+#
+# [*INDEX*]
+#         The top-level index page. For a browser-like environment
+#         define a frame set that includes the file, class, and 
+#         method indices. Passed
+#         %title%:: title of page
+#         %initial_page% :: url of initial page to display
+#
+# [*CLASS_INDEX*]
+#         Individual files for the three indexes. Passed:
+#         %index_url%:: URL of main index page
+#         %entries%::   List of
+#                       %name%:: name of an index entry
+#                       %href%:: url of corresponding page
+# [*METHOD_INDEX*]
+#         Same as CLASS_INDEX for methods
+#
+# [*FILE_INDEX*]
+#         Same as CLASS_INDEX for methods
+#
+# [*FR_INDEX_BODY*]
+#         A wrapper around CLASS_INDEX, METHOD_INDEX, and FILE_INDEX.
+#         If those index strings contain the complete HTML for the
+#         output, then FR_INDEX_BODY can simply be !INCLUDE!
+#
+# [*SRC_PAGE*]
+#         Page used to display source code. Passed %title% and %code%,
+#         the latter being a multi-line string of code.
+
+module Page
+
+FONTS = "Verdana, Arial, Helvetica, sans-serif"
+
+STYLE = %{
+body,td,p { font-family: %fonts%; 
+       color: #000040;
+}
+
+.attr-rw { font-size: x-small; color: #444488 }
+
+.title-row { background: #0000aa;
+             color:      #eeeeff;
+}
+
+.big-title-font { color: white;
+                  font-family: %fonts%;
+                  font-size: large; 
+                  height: 50px}
+
+.small-title-font { color: aqua;
+                    font-family: %fonts%;
+                    font-size: xx-small; }
+
+.aqua { color: aqua }
+
+.method-name, attr-name {
+      font-family: monospace; font-weight: bold;
+}
+
+.tablesubtitle, .tablesubsubtitle {
+   width: 100%;
+   margin-top: 1ex;
+   margin-bottom: .5ex;
+   padding: 5px 0px 5px 20px;
+   font-size: large;
+   color: aqua;
+   background: #3333cc;
+}
+
+.name-list {
+  font-family: monospace;
+  margin-left: 40px;
+  margin-bottom: 2ex;
+  line-height: 140%;
+}
+
+.description {
+  margin-left: 40px;
+  margin-top: -2ex;
+  margin-bottom: 2ex;
+}
+
+.description p {
+  line-height: 140%;
+}
+
+.aka {
+  margin-left: 40px;
+  margin-bottom: 2ex;
+  line-height: 100%;
+  font-size:   small;
+  color:       #808080;
+}
+
+.methodtitle {
+  font-size: medium;
+  text-decoration: none;
+  color: #0000AA;
+  background: white; 
+}
+
+.paramsig {
+   font-size: small;
+}
+
+.srcbut { float: right }
+
+pre { font-size: 1.2em; }
+tt  { font-size: 1.2em; }
+
+pre.source {
+  border-style: groove;
+  background-color: #ddddff;
+  margin-left:  40px;
+  padding: 1em 0em 1em 2em;
+}
+
+.classlist {
+  margin-left: 40px;
+  margin-bottom: 2ex;
+  line-height: 140%;
+}
+
+li {
+  display:    list-item;
+  margin-top: .6em;
+}
+
+.ruby-comment    { color: green; font-style: italic }
+.ruby-constant   { color: #4433aa; font-weight: bold; }
+.ruby-identifier { color: #222222;  }
+.ruby-ivar       { color: #2233dd; }
+.ruby-keyword    { color: #3333FF; font-weight: bold }
+.ruby-node       { color: #777777; }
+.ruby-operator   { color: #111111;  }
+.ruby-regexp     { color: #662222; }
+.ruby-value      { color: #662222; font-style: italic }
+
+}
+
+
+############################################################################
+
+
+HEADER = %{
+<?xml version="1.0" encoding="utf-8"?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
+"DTD/xhtml1-strict.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+  <title>%title%</title>
+  <meta http-equiv="Content-Type" content="text/html; charset=%charset%" />
+  <link rel=StyleSheet href="%style_url%" type="text/css" media="screen" />
+  <script type="text/javascript" language="JavaScript">
+  <!--
+  function popCode(url) {
+    window.open(url, "Code", 
+          "resizable=yes,scrollbars=yes,toolbar=no,status=no,height=150,width=400")
+  }
+  //-->
+  </script>
+</head>
+}
+
+
+###################################################################
+
+METHOD_LIST = %{
+IF:includes
+<table summary="Included modules" cellpadding="5" width="100%">
+<tr><td class="tablesubtitle">Included modules</td></tr>
+</table>
+<div class="name-list">
+START:includes
+    <span class="method-name">HREF:aref:name:</span>
+END:includes
+</div>
+ENDIF:includes
+
+IF:method_list
+START:method_list
+IF:methods
+<table summary="Method list" cellpadding="5" width="100%">
+<tr><td class="tablesubtitle">%type% %category% methods</td></tr>
+</table>
+START:methods
+<table summary="method"  width="100%" cellspacing="0" cellpadding="5" border="0">
+<tr><td class="methodtitle">
+<a name="%aref%"></a>
+IF:codeurl
+<a href="%codeurl%" target="Code" class="methodtitle"
+ onClick="popCode('%codeurl%');return false;">
+ENDIF:codeurl
+<b>%name%</b>%params%
+IF:codeurl
+</a>
+ENDIF:codeurl
+</td></tr>
+</table>
+IF:m_desc
+<div class="description">
+%m_desc%
+</div>
+ENDIF:m_desc
+IF:aka
+<div class="aka">
+This method is also aliased as
+START:aka
+<a href="%aref%">%name%</a>
+END:aka
+</div>
+ENDIF:aka
+IF:sourcecode
+<pre class="source">
+%sourcecode%
+</pre>
+ENDIF:sourcecode
+END:methods
+ENDIF:methods
+END:method_list
+ENDIF:method_list
+}
+
+###################################################################
+
+CONTEXT_CONTENT = %{
+IF:diagram
+<table summary="Diagram of classes and modules" width="100%">
+<tr><td align="center">
+%diagram%
+</td></tr></table>
+ENDIF:diagram
+
+
+IF:description
+<div class="description">%description%</div>
+ENDIF:description
+
+IF:requires
+<table summary="Requires" cellpadding="5" width="100%">
+<tr><td class="tablesubtitle">Required files</td></tr>
+</table>
+<div class="name-list">
+START:requires
+HREF:aref:name:&nbsp; &nbsp;
+END:requires
+</div>
+ENDIF:requires
+
+IF:methods
+<table summary="Methods" cellpadding="5" width="100%">
+<tr><td class="tablesubtitle">Methods</td></tr>
+</table>
+<div class="name-list">
+START:methods
+HREF:aref:name:&nbsp; &nbsp;
+END:methods
+</div>
+ENDIF:methods
+
+IF:constants
+<table summary="Constants" cellpadding="5" width="100%">
+<tr><td class="tablesubtitle">Constants</td></tr>
+</table>
+<table cellpadding="5">
+START:constants
+<tr valign="top"><td>%name%</td><td>=</td><td>%value%</td></tr>
+IF:desc
+<tr><td></td><td></td><td>%desc%</td></tr>
+ENDIF:desc
+END:constants
+</table>
+ENDIF:constants
+
+IF:aliases
+<table summary="Aliases" cellpadding="5" width="100%">
+<tr><td class="tablesubtitle">External Aliases</td></tr>
+</table>
+<div class="name-list">
+START:aliases
+%old_name% -> %new_name%<br />
+END:aliases
+</div>
+ENDIF:aliases
+
+IF:attributes
+<table summary="Attributes" cellpadding="5" width="100%">
+<tr><td class="tablesubtitle">Attributes</td></tr>
+</table>
+<table summary="Attribute details" cellspacing="5">
+START:attributes
+     <tr valign="top">
+       <td class="attr-name">%name%</td>
+IF:rw
+       <td align="center" class="attr-rw">&nbsp;[%rw%]&nbsp;</td>
+ENDIF:rw
+IFNOT:rw
+       <td></td>
+ENDIF:rw
+       <td>%a_desc%</td>
+     </tr>
+END:attributes
+</table>
+ENDIF:attributes
+
+IF:classlist
+<table summary="List of classes" cellpadding="5" width="100%">
+<tr><td class="tablesubtitle">Classes and Modules</td></tr>
+</table>
+<div class="classlist">
+%classlist%
+</div>
+ENDIF:classlist
+}
+
+###############################################################################
+
+BODY = HEADER + %{
+<body bgcolor="white">
+!INCLUDE!  <!-- banner header -->
+} +
+CONTEXT_CONTENT + METHOD_LIST +
+%{
+</body>
+</html>
+}
+
+
+###############################################################################
+
+FILE_PAGE = <<_FILE_PAGE_
+<table summary="Information on file" width="100%">
+ <tr class="title-row">
+ <td><table summary="layout" width="100%"><tr>
+   <td class="big-title-font" colspan="2">%short_name%</td>
+   <td align="right"><table summary="layout" cellspacing="0" cellpadding="2">
+         <tr>
+           <td  class="small-title-font">Path:</td>
+           <td class="small-title-font">%full_path%
+IF:cvsurl
+				&nbsp;(<a href="%cvsurl%">CVS</a>)
+ENDIF:cvsurl
+           </td>
+         </tr>
+         <tr>
+           <td class="small-title-font">Modified:</td>
+           <td class="small-title-font">%dtm_modified%</td>
+         </tr>
+        </table>
+    </td></tr></table></td>
+  </tr>
+</table>
+_FILE_PAGE_
+
+###################################################################
+
+CLASS_PAGE = %{
+<table summary="Information on class" width="100%" border="0" cellspacing="0">
+ <tr class="title-row">
+ <td class="big-title-font">
+   <sup><font color="aqua">%classmod%</font></sup> %full_name%
+ </td>
+ <td align="right">
+   <table summary="layout" cellspacing="0" cellpadding="2">
+     <tr valign="top">
+      <td class="small-title-font">In:</td>
+      <td class="small-title-font">
+START:infiles
+IF:full_path_url
+        <a href="%full_path_url%" class="aqua">
+ENDIF:full_path_url
+%full_path%
+IF:full_path_url
+         </a>
+ENDIF:full_path_url
+IF:cvsurl
+         &nbsp;(<a href="%cvsurl%">CVS</a>)
+ENDIF:cvsurl
+<br />
+END:infiles
+      </td>
+     </tr>
+IF:parent
+     <tr>
+      <td class="small-title-font">Parent:</td>
+      <td class="small-title-font">
+IF:par_url
+        <a href="%par_url%" class="aqua">
+ENDIF:par_url
+%parent%
+IF:par_url
+         </a>
+ENDIF:par_url
+      </td>
+     </tr>
+ENDIF:parent
+   </table>
+  </td>
+  </tr>
+</table>
+}
+
+=begin
+=end
+
+########################## Source code ##########################
+
+SRC_PAGE = %{
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=%charset%">
+<title>%title%</title>
+<link rel=StyleSheet href="%style_url%" type="text/css" media="screen" />
+</head>
+<body bgcolor="white">
+<pre>%code%</pre>
+</body>
+</html>
+}
+
+########################## Index ################################
+
+FR_INDEX_BODY = %{
+!INCLUDE!
+}
+
+FILE_INDEX = %{
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=%charset%">
+<title>%list_title%</title>
+<style type="text/css">
+<!--
+  body {
+background-color: #ddddff;
+     font-family: #{FONTS}; 
+       font-size: 11px; 
+      font-style: normal;
+     line-height: 14px; 
+           color: #000040;
+  }
+div.banner {
+  background: #0000aa;
+  color:      white;
+  padding: 1;
+  margin: 0;
+  font-size: 90%;
+  font-weight: bold;
+  line-height: 1.1;
+  text-align: center;
+  width: 100%;
+}
+
+A.xx { color: white; font-weight: bold; }
+-->
+</style>
+<base target="docwin">
+</head>
+<body>
+<div class="banner"><a href="%index_url%" class="xx">%list_title%</a></div>
+START:entries
+<a href="%href%">%name%</a><br />
+END:entries
+</body></html>
+}
+
+CLASS_INDEX = FILE_INDEX
+METHOD_INDEX = FILE_INDEX
+
+INDEX = %{
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Frameset//EN">
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=%charset%">
+<title>%title%</title></head>
+
+<frameset rows="20%, 80%">
+    <frameset cols="25%,35%,45%">
+        <frame src="fr_file_index.html"   title="Files" name="Files">
+        <frame src="fr_class_index.html"  name="Classes">
+        <frame src="fr_method_index.html" name="Methods">
+    </frameset>
+    <frame  src="%initial_page%" name="docwin">
+    <noframes>
+          <body bgcolor="white">
+            Sorry, RDoc currently only generates HTML using frames.
+          </body>
+    </noframes>
+</frameset>
+
+</html>
+}
+
+######################################################################
+#
+# The following is used for the -1 option
+#
+
+CONTENTS_XML = %{
+IF:description
+%description%
+ENDIF:description
+
+IF:requires
+<h4>Requires:</h4>
+<ul>
+START:requires
+IF:aref
+<li><a href="%aref%">%name%</a></li>
+ENDIF:aref
+IFNOT:aref
+<li>%name%</li>
+ENDIF:aref 
+END:requires
+</ul>
+ENDIF:requires
+
+IF:attributes
+<h4>Attributes</h4>
+<table>
+START:attributes
+<tr><td>%name%</td><td>%rw%</td><td>%a_desc%</td></tr>
+END:attributes
+</table>
+ENDIF:attributes
+
+IF:includes
+<h4>Includes</h4>
+<ul>
+START:includes
+IF:aref
+<li><a href="%aref%">%name%</a></li>
+ENDIF:aref
+IFNOT:aref
+<li>%name%</li>
+ENDIF:aref 
+END:includes
+</ul>
+ENDIF:includes
+
+IF:method_list
+<h3>Methods</h3>
+START:method_list
+IF:methods
+START:methods
+<h4>%type% %category% method: <a name="%aref%">%name%%params%</a></h4>
+
+IF:m_desc
+%m_desc%
+ENDIF:m_desc
+
+IF:sourcecode
+<blockquote><pre>
+%sourcecode%
+</pre></blockquote>
+ENDIF:sourcecode
+END:methods
+ENDIF:methods
+END:method_list
+ENDIF:method_list
+}
+
+########################################################################
+
+ONE_PAGE = %{
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+<head>
+  <title>%title%</title>
+  <meta http-equiv="Content-Type" content="text/html; charset=%charset%" />
+</head>
+<body>
+START:files
+<h2>File: %short_name%</h2>
+<table>
+  <tr><td>Path:</td><td>%full_path%</td></tr>
+  <tr><td>Modified:</td><td>%dtm_modified%</td></tr>
+</table>
+} + CONTENTS_XML + %{
+END:files
+
+IF:classes
+<h2>Classes</h2>
+START:classes
+IF:parent
+<h3>%classmod% %full_name% &lt; HREF:par_url:parent:</h3>
+ENDIF:parent
+IFNOT:parent
+<h3>%classmod% %full_name%</h3>
+ENDIF:parent
+
+IF:infiles
+(in files
+START:infiles
+HREF:full_path_url:full_path:
+END:infiles
+)
+ENDIF:infiles
+} + CONTENTS_XML + %{
+END:classes
+ENDIF:classes
+</body>
+</html>
+}
+
+end
+end
diff --git a/lib/rdoc/generators/template/html/kilmer.rb b/lib/rdoc/generators/template/html/kilmer.rb
new file mode 100644
index 0000000000..27fde1d29f
--- /dev/null
+++ b/lib/rdoc/generators/template/html/kilmer.rb
@@ -0,0 +1,398 @@
+module RDoc
+module Page
+
+
+FONTS = "Verdana, Arial, Helvetica, sans-serif"
+
+STYLE = %{
+body,td,p { font-family: %fonts%; 
+       color: #000040;
+}
+
+.attr-rw { font-size: xx-small; color: #444488 }
+
+.title-row { background-color: #CCCCFF;
+             color:      #000010;
+}
+
+.big-title-font { 
+  color: black;
+  font-weight: bold;
+  font-family: %fonts%; 
+  font-size: large; 
+  height: 60px;
+  padding: 10px 3px 10px 3px;
+}
+
+.small-title-font { color: black;
+                    font-family: %fonts%;
+                    font-size:10; }
+
+.aqua { color: black }
+
+.method-name, .attr-name {
+      font-family: font-family: %fonts%; 
+      font-weight: bold;
+      font-size: small;
+      margin-left: 20px;
+      color: #000033;
+}
+
+.tablesubtitle, .tablesubsubtitle {
+   width: 100%;
+   margin-top: 1ex;
+   margin-bottom: .5ex;
+   padding: 5px 0px 5px 3px;
+   font-size: large;
+   color: black;
+   background-color: #CCCCFF;
+   border: thin;
+}
+
+.name-list {
+  margin-left: 5px;
+  margin-bottom: 2ex;
+  line-height: 105%;
+}
+
+.description {
+  margin-left: 5px;
+  margin-bottom: 2ex;
+  line-height: 105%;
+  font-size: small;
+}
+
+.methodtitle {
+  font-size: small;
+  font-weight: bold;
+  text-decoration: none;
+  color: #000033;
+  background-color: white; 
+}
+
+.srclink {
+  font-size: small;
+  font-weight: bold;
+  text-decoration: none;
+  color: #0000DD;
+  background-color: white;
+}
+
+.paramsig {
+   font-size: small;
+}
+
+.srcbut { float: right }
+
+}
+
+
+############################################################################
+
+
+BODY = %{
+<html><head>
+  <title>%title%</title>
+  <meta http-equiv="Content-Type" content="text/html; charset=%charset%">
+  <link rel=StyleSheet href="%style_url%" type="text/css" media=screen>
+  <script type="text/javascript" language="JavaScript">
+  <!--
+  function popCode(url) {
+    parent.frames.source.location = url
+  }
+  //-->
+  </script>
+</head>
+<body bgcolor="white">
+
+!INCLUDE!  <!-- banner header -->
+
+IF:diagram
+<table width="100%"><tr><td align="center">
+%diagram%
+</td></tr></table>
+ENDIF:diagram
+
+IF:description
+<div class="description">%description%</div>
+ENDIF:description
+
+IF:requires
+<table cellpadding=5 width="100%">
+<tr><td class="tablesubtitle">Required files</td></tr>
+</table><br>
+<div class="name-list">
+START:requires
+HREF:aref:name:
+END:requires
+ENDIF:requires
+</div>
+
+IF:methods
+<table cellpadding=5 width="100%">
+<tr><td class="tablesubtitle">Methods</td></tr>
+</table><br>
+<div class="name-list">
+START:methods
+HREF:aref:name:,
+END:methods
+</div>
+ENDIF:methods
+
+IF:attributes
+<table cellpadding=5 width="100%">
+<tr><td class="tablesubtitle">Attributes</td></tr>
+</table><br>
+<table cellspacing=5>
+START:attributes
+     <tr valign="top">
+IF:rw
+       <td align="center" class="attr-rw">&nbsp;[%rw%]&nbsp;</td>
+ENDIF:rw
+IFNOT:rw
+       <td></td>
+ENDIF:rw
+       <td class="attr-name">%name%</td>
+       <td>%a_desc%</td>
+     </tr>
+END:attributes
+</table>
+ENDIF:attributes
+
+IF:classlist
+<table cellpadding=5 width="100%">
+<tr><td class="tablesubtitle">Classes and Modules</td></tr>
+</table><br>
+%classlist%<br>
+ENDIF:classlist
+
+  !INCLUDE!  <!-- method descriptions -->
+
+</body>
+</html>
+}
+
+###############################################################################
+
+FILE_PAGE = <<_FILE_PAGE_
+<table width="100%">
+ <tr class="title-row">
+ <td><table width="100%"><tr>
+   <td class="big-title-font" colspan=2><font size=-3><B>File</B><BR></font>%short_name%</td>
+   <td align="right"><table cellspacing=0 cellpadding=2>
+         <tr>
+           <td  class="small-title-font">Path:</td>
+           <td class="small-title-font">%full_path%
+IF:cvsurl
+				&nbsp;(<a href="%cvsurl%">CVS</a>)
+ENDIF:cvsurl
+           </td>
+         </tr>
+         <tr>
+           <td class="small-title-font">Modified:</td>
+           <td class="small-title-font">%dtm_modified%</td>
+         </tr>
+        </table>
+    </td></tr></table></td>
+  </tr>
+</table><br>
+_FILE_PAGE_
+
+###################################################################
+
+CLASS_PAGE = %{
+<table width="100%" border=0 cellspacing=0>
+ <tr class="title-row">
+ <td class="big-title-font">
+   <font size=-3><B>%classmod%</B><BR></font>%full_name%
+ </td>
+ <td align="right">
+   <table cellspacing=0 cellpadding=2>
+     <tr valign="top">
+      <td class="small-title-font">In:</td>
+      <td class="small-title-font">
+START:infiles
+HREF:full_path_url:full_path:
+IF:cvsurl
+&nbsp;(<a href="%cvsurl%">CVS</a>)
+ENDIF:cvsurl
+END:infiles
+      </td>
+     </tr>
+IF:parent
+     <tr>
+      <td class="small-title-font">Parent:</td>
+      <td class="small-title-font">
+IF:par_url
+        <a href="%par_url%" class="cyan">
+ENDIF:par_url
+%parent%
+IF:par_url
+         </a>
+ENDIF:par_url
+      </td>
+     </tr>
+ENDIF:parent
+   </table>
+  </td>
+  </tr>
+</table><br>
+}
+
+###################################################################
+
+METHOD_LIST = %{
+IF:includes
+<div class="tablesubsubtitle">Included modules</div><br>
+<div class="name-list">
+START:includes
+    <span class="method-name">HREF:aref:name:</span>
+END:includes
+</div>
+ENDIF:includes
+
+IF:method_list
+START:method_list
+IF:methods
+<table cellpadding=5 width="100%">
+<tr><td class="tablesubtitle">%type% %category% methods</td></tr>
+</table>
+START:methods
+<table width="100%" cellspacing = 0 cellpadding=5 border=0>
+<tr><td class="methodtitle">
+<a name="%aref%">
+<b>%name%</b>%params% 
+IF:codeurl
+<a href="%codeurl%" target="source" class="srclink">src</a>
+ENDIF:codeurl
+</a></td></tr>
+</table>
+IF:m_desc
+<div class="description">
+%m_desc%
+</div>
+ENDIF:m_desc
+END:methods
+ENDIF:methods
+END:method_list
+ENDIF:method_list
+}
+
+=begin
+=end
+
+########################## Source code ##########################
+
+SRC_PAGE = %{
+<html>
+<head><title>%title%</title>
+<meta http-equiv="Content-Type" content="text/html; charset=%charset%">
+<style>
+.ruby-comment    { color: green; font-style: italic }
+.ruby-constant   { color: #4433aa; font-weight: bold; }
+.ruby-identifier { color: #222222;  }
+.ruby-ivar       { color: #2233dd; }
+.ruby-keyword    { color: #3333FF; font-weight: bold }
+.ruby-node       { color: #777777; }
+.ruby-operator   { color: #111111;  }
+.ruby-regexp     { color: #662222; }
+.ruby-value      { color: #662222; font-style: italic }
+  .kw { color: #3333FF; font-weight: bold }
+  .cmt { color: green; font-style: italic }
+  .str { color: #662222; font-style: italic }
+  .re  { color: #662222; }
+</style>
+</head>
+<body bgcolor="white">
+<pre>%code%</pre>
+</body>
+</html>
+}
+
+########################## Index ################################
+
+FR_INDEX_BODY = %{
+!INCLUDE!
+}
+
+FILE_INDEX = %{
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=%charset%">
+<style>
+<!--
+  body {
+background-color: #ddddff;
+     font-family: #{FONTS}; 
+       font-size: 11px; 
+      font-style: normal;
+     line-height: 14px; 
+           color: #000040;
+  }
+div.banner {
+  background: #0000aa;
+  color:      white;
+  padding: 1;
+  margin: 0;
+  font-size: 90%;
+  font-weight: bold;
+  line-height: 1.1;
+  text-align: center;
+  width: 100%;
+}
+  
+-->
+</style>
+<base target="docwin">
+</head>
+<body>
+<div class="banner">%list_title%</div>
+START:entries
+<a href="%href%">%name%</a><br>
+END:entries
+</body></html>
+}
+
+CLASS_INDEX = FILE_INDEX
+METHOD_INDEX = FILE_INDEX
+
+INDEX = %{
+<html>
+<head>
+  <title>%title%</title>
+  <meta http-equiv="Content-Type" content="text/html; charset=%charset%">
+</head>
+
+<frameset cols="20%,*">
+    <frameset rows="15%,35%,50%">
+        <frame src="fr_file_index.html"   title="Files" name="Files">
+        <frame src="fr_class_index.html"  name="Classes">
+        <frame src="fr_method_index.html" name="Methods">
+    </frameset>
+    <frameset rows="80%,20%">
+      <frame  src="%initial_page%" name="docwin">
+      <frame  src="blank.html" name="source">
+    </frameset>
+    <noframes>
+          <body bgcolor="white">
+            Click <a href="html/index.html">here</a> for a non-frames
+            version of this page.
+          </body>
+    </noframes>
+</frameset>
+
+</html>
+}
+
+# and a blank page to use as a target
+BLANK = %{
+<html><body bgcolor="white"></body></html>
+}
+
+def write_extra_pages
+  template = TemplatePage.new(BLANK)
+  File.open("blank.html", "w") { |f| template.write_html_on(f, {}) }
+end
+
+end
+end
diff --git a/lib/rdoc/generators/template/xml/rdf.rb b/lib/rdoc/generators/template/xml/rdf.rb
new file mode 100644
index 0000000000..1545d81a2f
--- /dev/null
+++ b/lib/rdoc/generators/template/xml/rdf.rb
@@ -0,0 +1,112 @@
+module RDoc
+module Page
+
+
+
+CONTENTS_RDF = %{
+IF:description
+    <description rd:parseType="Literal">
+%description%
+    </description>
+ENDIF:description
+
+IF:requires
+START:requires
+         <rd:required-file rd:name="%name%" />
+END:requires
+ENDIF:requires
+
+IF:attributes
+START:attributes
+        <contents>
+        <Attribute rd:name="%name%">
+IF:rw
+          <attribute-rw>%rw%</attribute-rw>
+ENDIF:rw
+          <description rdf:parseType="Literal">%a_desc%</description>
+        </Attribute>
+        </contents>
+END:attributes
+ENDIF:attributes
+
+IF:includes
+      <IncludedModuleList>
+START:includes
+        <included-module rd:name="%name%"  />
+END:includes
+      </IncludedModuleList>
+ENDIF:includes
+
+IF:method_list
+START:method_list
+IF:methods
+START:methods
+	<contents>
+        <Method rd:name="%name%" rd:visibility="%type%"
+                rd:category="%category%" rd:id="%aref%">
+          <parameters>%params%</parameters>
+IF:m_desc
+          <description rdf:parseType="Literal">
+%m_desc%
+          </description>
+ENDIF:m_desc
+IF:sourcecode
+          <source-code-listing rdf:parseType="Literal">
+%sourcecode%
+          </source-code-listing>
+ENDIF:sourcecode
+        </Method>
+       </contents>
+END:methods
+ENDIF:methods
+END:method_list
+ENDIF:method_list
+     <!-- end method list -->
+}
+
+########################################################################
+
+ONE_PAGE = %{<?xml version="1.0" encoding="utf-8"?>
+<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+        xmlns="http://pragprog.com/rdoc/rdoc.rdf#"
+        xmlns:rd="http://pragprog.com/rdoc/rdoc.rdf#">
+
+<!-- RDoc -->
+START:files
+  <rd:File rd:name="%short_name%" rd:id="%href%">
+      <path>%full_path%</path>
+      <dtm-modified>%dtm_modified%</dtm-modified>
+} + CONTENTS_RDF + %{
+  </rd:File>
+END:files
+START:classes
+  <%classmod% rd:name="%full_name%" rd:id="%full_name%">
+    <classmod-info>
+IF:infiles
+      <InFiles>
+START:infiles
+        <infile>
+          <File rd:name="%full_path%"
+IF:full_path_url
+                rdf:about="%full_path_url%"
+ENDIF:full_path_url
+           />
+         </infile>
+END:infiles
+      </InFiles>
+ENDIF:infiles
+IF:parent
+     <superclass>HREF:par_url:parent:</superclass>
+ENDIF:parent
+    </classmod-info>
+} + CONTENTS_RDF + %{
+  </%classmod%>
+END:classes
+<!-- /RDoc -->
+</rdf:RDF>
+}
+
+
+end
+end
+
diff --git a/lib/rdoc/generators/template/xml/xml.rb b/lib/rdoc/generators/template/xml/xml.rb
new file mode 100644
index 0000000000..4a0c8c9ac4
--- /dev/null
+++ b/lib/rdoc/generators/template/xml/xml.rb
@@ -0,0 +1,112 @@
+module RDoc
+module Page
+
+
+
+CONTENTS_XML = %{
+IF:description
+    <description>
+%description%
+    </description>
+ENDIF:description
+    <contents>
+IF:requires
+      <required-file-list>
+START:requires
+         <required-file name="%name%"
+IF:aref 
+                        href="%aref%"
+ENDIF:aref
+         />
+END:requires
+      </required-file-list>
+ENDIF:requires
+IF:attributes
+      <attribute-list>
+START:attributes
+        <attribute name="%name%">
+IF:rw
+          <attribute-rw>%rw%</attribute-rw>
+ENDIF:rw
+          <description>%a_desc%</description>
+        </attribute>
+END:attributes
+      </attribute-list>
+ENDIF:attributes
+IF:includes
+      <included-module-list>
+START:includes
+        <included-module name="%name%"
+IF:aref
+                         href="%aref%"
+ENDIF:aref
+        />
+END:includes
+      </included-module-list>
+ENDIF:includes
+IF:method_list
+      <method-list>
+START:method_list
+IF:methods
+START:methods
+        <method name="%name%" type="%type%" category="%category%" id="%aref%">
+          <parameters>%params%</parameters>
+IF:m_desc
+          <description>
+%m_desc%
+          </description>
+ENDIF:m_desc
+IF:sourcecode
+          <source-code-listing>
+%sourcecode%
+          </source-code-listing>
+ENDIF:sourcecode
+        </method>
+END:methods
+ENDIF:methods
+END:method_list
+      </method-list>
+ENDIF:method_list
+     </contents>
+}
+
+########################################################################
+
+ONE_PAGE = %{<?xml version="1.0" encoding="utf-8"?>
+<rdoc>
+<file-list>
+START:files
+  <file name="%short_name%" id="%href%">
+    <file-info>
+      <path>%full_path%</path>
+      <dtm-modified>%dtm_modified%</dtm-modified>
+    </file-info>
+} + CONTENTS_XML + %{
+  </file>
+END:files
+</file-list>
+<class-module-list>
+START:classes
+  <%classmod% name="%full_name%" id="%full_name%">
+    <classmod-info>
+IF:infiles
+      <infiles>      
+START:infiles
+        <infile>HREF:full_path_url:full_path:</infile>
+END:infiles
+      </infiles>
+ENDIF:infiles
+IF:parent
+     <superclass>HREF:par_url:parent:</superclass>
+ENDIF:parent
+    </classmod-info>
+} + CONTENTS_XML + %{
+  </%classmod%>
+END:classes
+</class-module-list>
+</rdoc>
+}
+
+
+end
+end
diff --git a/lib/rdoc/generators/xml_generator.rb b/lib/rdoc/generators/xml_generator.rb
new file mode 100644
index 0000000000..8c1a76d62b
--- /dev/null
+++ b/lib/rdoc/generators/xml_generator.rb
@@ -0,0 +1,130 @@
+
+require 'ftools'
+
+require 'rdoc/options'
+require 'rdoc/markup/simple_markup'
+require 'rdoc/markup/simple_markup/to_html'
+require 'rdoc/generators/html_generator'
+
+module Generators
+
+  # Generate XML output as one big file
+
+  class XMLGenerator < HTMLGenerator
+
+    # Standard generator factory
+    def XMLGenerator.for(options)
+      XMLGenerator.new(options)
+    end
+
+    
+    def initialize(*args)
+      super
+    end
+
+    ##
+    # Build the initial indices and output objects
+    # based on an array of TopLevel objects containing
+    # the extracted information. 
+
+    def generate(info)
+      @info       = info
+      @files      = []
+      @classes    = []
+      @hyperlinks = {}
+
+      build_indices
+      generate_xml
+    end
+
+
+    ##
+    # Generate:
+    #
+    # * a list of HtmlFile objects for each TopLevel object.
+    # * a list of HtmlClass objects for each first level
+    #   class or module in the TopLevel objects
+    # * a complete list of all hyperlinkable terms (file,
+    #   class, module, and method names)
+
+    def build_indices
+
+      @info.each do |toplevel|
+        @files << HtmlFile.new(toplevel, @options, FILE_DIR)
+      end
+
+      RDoc::TopLevel.all_classes_and_modules.each do |cls|
+        build_class_list(cls, @files[0], CLASS_DIR)
+      end
+    end
+
+    def build_class_list(from, html_file, class_dir)
+      @classes << HtmlClass.new(from, html_file, class_dir, @options)
+      from.each_classmodule do |mod|
+        build_class_list(mod, html_file, class_dir)
+      end
+    end
+
+    ##
+    # Generate all the HTML. For the one-file case, we generate
+    # all the information in to one big hash
+    #
+    def generate_xml
+      values = { 
+        'charset' => @options.charset,
+        'files'   => gen_into(@files),
+        'classes' => gen_into(@classes)
+      }
+      
+      # this method is defined in the template file
+      write_extra_pages if defined? write_extra_pages
+
+      template = TemplatePage.new(RDoc::Page::ONE_PAGE)
+
+      if @options.op_name
+        opfile = File.open(@options.op_name, "w")
+      else
+        opfile = $stdout
+      end
+      template.write_html_on(opfile, values)
+    end
+
+    def gen_into(list)
+      res = []
+      list.each do |item|
+        res << item.value_hash
+      end
+      res
+    end
+
+    def gen_file_index
+      gen_an_index(@files, 'Files')
+    end
+
+    def gen_class_index
+      gen_an_index(@classes, 'Classes')
+    end
+
+    def gen_method_index
+      gen_an_index(HtmlMethod.all_methods, 'Methods')
+    end
+
+    
+    def gen_an_index(collection, title)
+      res = []
+      collection.sort.each do |f|
+        if f.document_self
+          res << { "href" => f.path, "name" => f.index_name }
+        end
+      end
+
+      return {
+        "entries" => res,
+        'list_title' => title,
+        'index_url'  => main_url,
+      }
+    end
+
+  end
+
+end
diff --git a/lib/rdoc/markup/sample/rdoc2latex.rb b/lib/rdoc/markup/sample/rdoc2latex.rb
new file mode 100644
index 0000000000..26563b75da
--- /dev/null
+++ b/lib/rdoc/markup/sample/rdoc2latex.rb
@@ -0,0 +1,16 @@
+#!/usr/local/bin/ruby
+# Illustration of a script to convert an RDoc-style file to a LaTeX
+# document
+
+require 'rdoc/markup/simple_markup'
+require 'rdoc/markup/simple_markup/to_latex'
+
+p = SM::SimpleMarkup.new
+h = SM::ToLaTeX.new
+
+#puts "\\documentclass{report}"
+#puts "\\usepackage{tabularx}"
+#puts "\\usepackage{parskip}"
+#puts "\\begin{document}"
+puts p.convert(ARGF.read, h)
+#puts "\\end{document}"
diff --git a/lib/rdoc/markup/sample/sample.rb b/lib/rdoc/markup/sample/sample.rb
new file mode 100644
index 0000000000..a375b54564
--- /dev/null
+++ b/lib/rdoc/markup/sample/sample.rb
@@ -0,0 +1,42 @@
+# This program illustrates the basic use of the SimpleMarkup
+# class. It extracts the first comment block from the 
+# simple_markup.rb file and converts it into HTML on
+# standard output. Run it using
+#
+#  % ruby sample.rb
+#
+# You should be in the sample/ directory when you do this,
+# as it hardwires the path to the files it needs to require.
+# This isn't necessary in the code you write once you've 
+# installed the package.
+#
+# For a better way of formatting code comment blocks (and more)
+# see the rdoc package.
+#
+
+$:.unshift "../../.."
+
+require 'rdoc/markup/simple_markup'
+require 'rdoc/markup/simple_markup/to_html'
+
+# Extract the comment block from the source file
+
+input_string = ""
+
+File.foreach("../simple_markup.rb") do |line|
+  break unless line.gsub!(/^\# ?/, '')
+  input_string << line
+end
+
+# Create a markup object
+markup = SM::SimpleMarkup.new
+
+# Attach it to an HTML formatter
+h = SM::ToHtml.new
+
+# And convert out comment block to html. Wrap it a body
+# tag pair to let browsers view it
+
+puts "<html><body>"
+puts markup.convert(input_string, h)
+puts "</body></html>"
diff --git a/lib/rdoc/markup/simple_markup.rb b/lib/rdoc/markup/simple_markup.rb
new file mode 100644
index 0000000000..18971e23e1
--- /dev/null
+++ b/lib/rdoc/markup/simple_markup.rb
@@ -0,0 +1,477 @@
+# = Introduction
+#
+# SimpleMarkup parses plain text documents and attempts to decompose
+# them into their constituent parts. Some of these parts are high-level:
+# paragraphs, chunks of verbatim text, list entries and the like. Other
+# parts happen at the character level: a piece of bold text, a word in
+# code font. This markup is similar in spirit to that used on WikiWiki
+# webs, where folks create web pages using a simple set of formatting
+# rules.
+#
+# SimpleMarkup itself does no output formatting: this is left to a
+# different set of classes.
+#
+# SimpleMarkup is extendable at runtime: you can add new markup
+# elements to be recognised in the documents that SimpleMarkup parses.
+#
+# SimpleMarkup is intended to be the basis for a family of tools which
+# share the common requirement that simple, plain-text should be
+# rendered in a variety of different output formats and media. It is
+# envisaged that SimpleMarkup could be the basis for formating RDoc
+# style comment blocks, Wiki entries, and online FAQs.
+#
+# = Basic Formatting
+#
+# * SimpleMarkup looks for a document's natural left margin. This is
+#   used as the initial margin for the document.
+#
+# * Consecutive lines starting at this margin are considered to be a
+#   paragraph.
+#
+# * If a paragraph starts with a "*", "-", or with "<digit>.", then it is
+#   taken to be the start of a list. The margin in increased to be the
+#   first non-space following the list start flag. Subsequent lines
+#   should be indented to this new margin until the list ends. For
+#   example:
+#
+#      * this is a list with three paragraphs in
+#        the first item. This is the first paragraph.
+#
+#        And this is the second paragraph.
+#
+#        1. This is an indented, numbered list.
+#        2. This is the second item in that list
+#
+#        This is the third conventional paragraph in the
+#        first list item.
+#
+#      * This is the second item in the original list
+#
+# * You can also construct labeled lists, sometimes called description
+#   or definition lists. Do this by putting the label in square brackets
+#   and indenting the list body:
+#
+#       [cat]  a small furry mammal
+#              that seems to sleep a lot
+#
+#       [ant]  a little insect that is known
+#              to enjoy picnics
+#
+#   A minor variation on labeled lists uses two colons to separate the
+#   label from the list body:
+#
+#       cat::  a small furry mammal
+#              that seems to sleep a lot
+#
+#       ant::  a little insect that is known
+#              to enjoy picnics
+#     
+#   This latter style guarantees that the list bodies' left margins are
+#   aligned: think of them as a two column table.
+#
+# * Any line that starts to the right of the current margin is treated
+#   as verbatim text. This is useful for code listings. The example of a
+#   list above is also verbatim text.
+#
+# * A line starting with an equals sign (=) is treated as a
+#   heading. Level one headings have one equals sign, level two headings
+#   have two,and so on.
+#
+# * A line starting with three or more hyphens (at the current indent)
+#   generates a horizontal rule. THe more hyphens, the thicker the rule
+#   (within reason, and if supported by the output device)
+#
+# * You can use markup within text (except verbatim) to change the
+#   appearance of parts of that text. Out of the box, SimpleMarkup
+#   supports word-based and general markup.
+#
+#   Word-based markup uses flag characters around individual words:
+#
+#   [\*word*]  displays word in a *bold* font
+#   [\_word_]  displays word in an _emphasized_ font
+#   [\+word+]  displays word in a +code+ font
+#
+#   General markup affects text between a start delimiter and and end
+#   delimiter. Not surprisingly, these delimiters look like HTML markup.
+#
+#   [\<b>text...</b>]    displays word in a *bold* font
+#   [\<em>text...</em>]  displays word in an _emphasized_ font
+#   [\<i>text...</i>]    displays word in an _emphasized_ font
+#   [\<tt>text...</tt>]  displays word in a +code+ font
+#
+#   Unlike conventional Wiki markup, general markup can cross line
+#   boundaries. You can turn off the interpretation of markup by
+#   preceding the first character with a backslash, so \\\<b>bold
+#   text</b> and \\\*bold* produce \<b>bold text</b> and \*bold
+#   respectively.
+#
+# = Using SimpleMarkup
+#
+# For information on using SimpleMarkup programatically, 
+# see SM::SimpleMarkup.
+#
+# Author::   Dave Thomas,  dave@pragmaticprogrammer.com
+# Version::  0.0
+# License::  Ruby license
+
+
+
+require 'rdoc/markup/simple_markup/fragments'
+require 'rdoc/markup/simple_markup/lines.rb'
+
+module SM  #:nodoc:
+
+  # == Synopsis
+  #
+  # This code converts <tt>input_string</tt>, which is in the format
+  # described in markup/simple_markup.rb, to HTML. The conversion
+  # takes place in the +convert+ method, so you can use the same
+  # SimpleMarkup object to convert multiple input strings.
+  #
+  #   require 'rdoc/markup/simple_markup'
+  #   require 'rdoc/markup/simple_markup/to_html'
+  #
+  #   p = SM::SimpleMarkup.new
+  #   h = SM::ToHtml.new
+  #
+  #   puts p.convert(input_string, h)
+  #
+  # You can extend the SimpleMarkup parser to recognise new markup
+  # sequences, and to add special processing for text that matches a
+  # regular epxression. Here we make WikiWords significant to the parser,
+  # and also make the sequences {word} and \<no>text...</no> signify
+  # strike-through text. When then subclass the HTML output class to deal
+  # with these:
+  #
+  #   require 'rdoc/markup/simple_markup'
+  #   require 'rdoc/markup/simple_markup/to_html'
+  #
+  #   class WikiHtml < SM::ToHtml
+  #     def handle_special_WIKIWORD(special)
+  #       "<font color=red>" + special.text + "</font>"
+  #     end
+  #   end
+  #
+  #   p = SM::SimpleMarkup.new
+  #   p.add_word_pair("{", "}", :STRIKE)
+  #   p.add_html("no", :STRIKE)
+  #
+  #   p.add_special(/\b([A-Z][a-z]+[A-Z]\w+)/, :WIKIWORD)
+  #
+  #   h = WikiHtml.new
+  #   h.add_tag(:STRIKE, "<strike>", "</strike>")
+  #
+  #   puts "<body>" + p.convert(ARGF.read, h) + "</body>"
+  #
+  # == Output Formatters
+  #
+  # _missing_
+  #
+  #
+
+  class SimpleMarkup
+
+    SPACE = ?\s
+
+    # List entries look like:
+    #  *       text
+    #  1.      text
+    #  [label] text
+    #  label:: text
+    #
+    # Flag it as a list entry, and
+    # work out the indent for subsequent lines
+
+    SIMPLE_LIST_RE = /^(
+                  (  \*          (?# bullet)
+                    |-           (?# bullet)
+                    |\d+\.       (?# numbered )
+                    |[A-Za-z]\.  (?# alphabetically numbered )
+                  )
+                  \s+
+                )\S/x
+
+    LABEL_LIST_RE = /^(
+                        (  \[.*?\]    (?# labeled  )
+                          |\S.*::     (?# note     )
+                        )(?=\s|$)
+                        \s*
+                      )/x
+
+
+    ##
+    # take a block of text and use various heuristics to determine
+    # it's structure (paragraphs, lists, and so on). Invoke an
+    # event handler as we identify significant chunks.
+    #
+
+    def initialize
+      @am = AttributeManager.new
+      @output = nil
+    end
+
+    ##
+    # Add to the sequences used to add formatting to an individual word 
+    # (such as *bold*). Matching entries will generate attibutes
+    # that the output formatters can recognize by their +name+
+
+    def add_word_pair(start, stop, name)
+      @am.add_word_pair(start, stop, name)
+    end
+
+    ##
+    # Add to the sequences recognized as general markup
+    #
+
+    def add_html(tag, name)
+      @am.add_html(tag, name)
+    end
+
+    ##
+    # Add to other inline sequences. For example, we could add
+    # WikiWords using something like:
+    #
+    #    parser.add_special(/\b([A-Z][a-z]+[A-Z]\w+)/, :WIKIWORD)
+    #
+    # Each wiki word will be presented to the output formatter 
+    # via the accept_special method
+    #
+
+    def add_special(pattern, name)
+      @am.add_special(pattern, name)
+    end
+
+
+    # We take a string, split it into lines, work out the type of
+    # each line, and from there deduce groups of lines (for example
+    # all lines in a paragraph). We then invoke the output formatter
+    # using a Visitor to display the result
+
+    def convert(str, op)
+      @lines = Lines.new(str.split(/\r?\n/).collect { |aLine| 
+                           Line.new(aLine) })
+      return "" if @lines.empty?
+      @lines.normalize
+      assign_types_to_lines
+      group = group_lines
+      # call the output formatter to handle the result
+      #      group.to_a.each {|i| p i}
+      group.accept(@am, op)
+    end
+
+
+    #######
+    private
+    #######
+
+
+    ##
+    # Look through the text at line indentation. We flag each line as being
+    # Blank, a paragraph, a list element, or verbatim text
+    #
+
+    def assign_types_to_lines(margin = 0, level = 0)
+
+      while line = @lines.next
+        if line.isBlank?
+          line.stamp(Line::BLANK, level)
+          next
+        end
+        
+        # if a line contains non-blanks before the margin, then it must belong
+        # to an outer level
+
+        text = line.text
+        
+        for i in 0...margin
+          if text[i] != SPACE
+            @lines.unget
+            return
+          end
+        end
+
+        active_line = text[margin..-1]
+
+        # Rules (horizontal lines) look like
+        #
+        #  ---   (three or more hyphens)
+        #
+        # The more hyphens, the thicker the rule
+        #
+
+        if /^(---+)\s*$/ =~ active_line
+          line.stamp(Line::RULE, level, $1.length-2)
+          next
+        end
+
+        # Then look for list entries. First the ones that have to have
+        # text following them (* xxx, - xxx, and dd. xxx)
+
+        if SIMPLE_LIST_RE =~ active_line
+
+          offset = margin + $1.length
+          prefix = $2
+          prefix_length = prefix.length
+
+          flag = case prefix
+                 when "*","-" then ListBase::BULLET
+                 when /^\d/   then ListBase::NUMBER
+                 when /^[A-Z]/ then ListBase::UPPERALPHA
+                 when /^[a-z]/ then ListBase::LOWERALPHA
+                 else raise "Invalid List Type: #{self.inspect}"
+                 end
+
+          line.stamp(Line::LIST, level+1, prefix, flag)
+          text[margin, prefix_length] = " " * prefix_length
+          assign_types_to_lines(offset, level + 1)
+          next
+        end
+
+
+        if LABEL_LIST_RE =~ active_line
+          offset = margin + $1.length
+          prefix = $2
+          prefix_length = prefix.length
+
+          next if handled_labeled_list(line, level, margin, offset, prefix)
+        end
+
+        # Headings look like
+        # = Main heading
+        # == Second level
+        # === Third
+        #
+        # Headings reset the level to 0
+
+        if active_line[0] == ?= and active_line =~ /^(=+)\s*(.*)/
+          prefix_length = $1.length
+          prefix_length = 6 if prefix_length > 6
+          line.stamp(Line::HEADING, 0, prefix_length)
+          line.strip_leading(margin + prefix_length)
+          next
+        end
+        
+        # If the character's a space, then we have verbatim text,
+        # otherwise 
+
+        if active_line[0] == SPACE
+          line.strip_leading(margin) if margin > 0
+          line.stamp(Line::VERBATIM, level)
+        else
+          line.stamp(Line::PARAGRAPH, level)
+        end
+      end
+    end
+
+    # Handle labeled list entries, We have a special case
+    # to deal with. Because the labels can be long, they force
+    # the remaining block of text over the to right:
+    #
+    # this is a long label that I wrote:: and here is the
+    #                                     block of text with
+    #                                     a silly margin
+    #
+    # So we allow the special case. If the label is followed
+    # by nothing, and if the following line is indented, then
+    # we take the indent of that line as the new margin
+    #
+    # this is a long label that I wrote::
+    #     here is a more reasonably indented block which
+    #     will ab attached to the label.
+    #
+    
+    def handled_labeled_list(line, level, margin, offset, prefix)
+      prefix_length = prefix.length
+      text = line.text
+      flag = nil
+      case prefix
+      when /^\[/
+        flag = ListBase::LABELED
+        prefix = prefix[1, prefix.length-2]
+      when /:$/
+        flag = ListBase::NOTE
+        prefix.chop!
+      else raise "Invalid List Type: #{self.inspect}"
+      end
+      
+      # body is on the next line
+      
+      if text.length <= offset
+        original_line = line
+        line = @lines.next
+        return(false) unless line
+        text = line.text
+        
+        for i in 0..margin
+          if text[i] != SPACE
+            @lines.unget
+            return false
+          end
+        end
+        i = margin
+        i += 1 while text[i] == SPACE
+        if i >= text.length
+          @lines.unget
+          return false
+        else
+          offset = i
+          prefix_length = 0
+          @lines.delete(original_line)
+        end
+      end
+      
+      line.stamp(Line::LIST, level+1, prefix, flag)
+      text[margin, prefix_length] = " " * prefix_length
+      assign_types_to_lines(offset, level + 1)
+      return true
+    end
+
+    # Return a block consisting of fragments which are
+    # paragraphs, list entries or verbatim text. We merge consecutive
+    # lines of the same type and level together. We are also slightly
+    # tricky with lists: the lines following a list introduction
+    # look like paragraph lines at the next level, and we remap them
+    # into list entries instead
+
+    def group_lines
+      @lines.rewind
+
+      inList = false
+      wantedType = wantedLevel = nil
+
+      block = LineCollection.new
+      group = nil
+
+      while line = @lines.next
+        if line.level == wantedLevel and line.type == wantedType
+          group.add_text(line.text)
+        else
+          group = block.fragment_for(line)
+          block.add(group)
+          if line.type == Line::LIST
+            wantedType = Line::PARAGRAPH
+          else
+            wantedType = line.type
+          end
+          wantedLevel = line.level
+        end
+      end
+
+      block.normalize
+      block
+    end
+
+    ## for debugging, we allow access to our line contents as text
+    def content
+      @lines.as_text
+    end
+    public :content
+
+    ## for debugging, return the list of line types
+    def get_line_types
+      @lines.line_types
+    end
+    public :get_line_types
+  end
+
+end
diff --git a/lib/rdoc/markup/simple_markup/fragments.rb b/lib/rdoc/markup/simple_markup/fragments.rb
new file mode 100644
index 0000000000..83388fcc0b
--- /dev/null
+++ b/lib/rdoc/markup/simple_markup/fragments.rb
@@ -0,0 +1,328 @@
+require 'rdoc/markup/simple_markup/lines.rb'
+require 'rdoc/markup/simple_markup/inline.rb'
+
+module SM
+
+  ##
+  # A Fragment is a chunk of text, subclassed as a paragraph, a list
+  # entry, or verbatim text
+
+  class Fragment
+    attr_reader   :level, :param, :txt
+    attr_accessor :type
+
+    def initialize(level, param, type, txt)
+      @level = level
+      @param = param
+      @type  = type
+      @txt   = ""
+      add_text(txt) if txt
+    end
+
+    def add_text(txt)
+      @txt << " " if @txt.length > 0
+      @txt << txt.tr_s("\n ", "  ").strip
+    end
+
+    def to_s
+      "L#@level: #{self.class.name.split('::')[-1]}\n#@txt"
+    end
+
+    ######
+    # This is a simple factory system that lets us associate fragement
+    # types (a string) with a subclass of fragment
+
+    TYPE_MAP = {}
+
+    def Fragment.type_name(name)
+      TYPE_MAP[name] = self
+    end
+
+    def Fragment.for(line)
+      klass =  TYPE_MAP[line.type] ||
+        raise("Unknown line type: '#{line.type.inspect}:' '#{line.text}'")
+      return klass.new(line.level, line.param, line.flag, line.text)
+    end
+  end
+
+  ##
+  # A paragraph is a fragment which gets wrapped to fit. We remove all
+  # newlines when we're created, and have them put back on output
+
+  class Paragraph < Fragment
+    type_name Line::PARAGRAPH
+  end
+
+  class BlankLine < Paragraph
+    type_name Line::BLANK
+  end
+
+  class Heading < Paragraph
+    type_name Line::HEADING
+
+    def head_level
+      @param.to_i
+    end
+  end
+
+  ##
+  # A List is a fragment with some kind of label
+  #
+
+  class ListBase < Paragraph
+    # List types
+    BULLET  = :BULLET
+    NUMBER  = :NUMBER
+    UPPERALPHA  = :UPPERALPHA
+    LOWERALPHA  = :LOWERALPHA
+    LABELED = :LABELED
+    NOTE    = :NOTE
+  end
+
+  class ListItem < ListBase
+    type_name Line::LIST
+
+    #  def label
+    #    am = AttributeManager.new(@param)
+    #    am.flow
+    #  end
+  end
+
+  class ListStart < ListBase
+    def initialize(level, param, type)
+      super(level, param, type, nil)
+    end
+  end
+
+  class ListEnd < ListBase
+    def initialize(level, type)
+      super(level, "", type, nil)
+    end
+  end
+
+  ##
+  # Verbatim code contains lines that don't get wrapped.
+
+  class Verbatim < Fragment
+    type_name  Line::VERBATIM
+
+    def add_text(txt)
+      @txt << txt.chomp << "\n"
+    end
+
+  end
+
+  ##
+  # A horizontal rule
+  class Rule < Fragment
+    type_name Line::RULE
+  end
+
+
+  # Collect groups of lines together. Each group
+  # will end up containing a flow of text
+
+  class LineCollection
+    
+    def initialize
+      @fragments = []
+    end
+
+    def add(fragment)
+      @fragments << fragment
+    end
+
+    def each(&b)
+      @fragments.each(&b)
+    end
+
+    # For testing
+    def to_a
+      @fragments.map {|fragment| fragment.to_s}
+    end
+
+    # Factory for different fragment types
+    def fragment_for(*args)
+      Fragment.for(*args)
+    end
+
+    # tidy up at the end
+    def normalize
+      change_verbatim_blank_lines
+      add_list_start_and_ends
+      add_list_breaks
+      tidy_blank_lines
+    end
+
+    def to_s
+      @fragments.join("\n----\n")
+    end
+
+    def accept(am, visitor)
+
+      visitor.start_accepting
+
+      @fragments.each do |fragment|
+        case fragment
+        when Verbatim
+          visitor.accept_verbatim(am, fragment)
+        when Rule
+          visitor.accept_rule(am, fragment)
+        when ListStart
+          visitor.accept_list_start(am, fragment)
+        when ListEnd
+          visitor.accept_list_end(am, fragment)
+        when ListItem
+          visitor.accept_list_item(am, fragment)
+        when BlankLine
+          visitor.accept_blank_line(am, fragment)
+        when Heading
+          visitor.accept_heading(am, fragment)
+        when Paragraph
+          visitor.accept_paragraph(am, fragment)
+        end
+      end
+
+      visitor.end_accepting
+    end
+    #######
+    private
+    #######
+
+    # If you have:
+    #
+    #    normal paragraph text.
+    #
+    #       this is code
+    #   
+    #       and more code
+    #
+    # You'll end up with the fragments Paragraph, BlankLine, 
+    # Verbatim, BlankLine, Verbatim, BlankLine, etc
+    #
+    # The BlankLine in the middle of the verbatim chunk needs to
+    # be changed to a real verbatim newline, and the two
+    # verbatim blocks merged
+    #
+    #    
+    def change_verbatim_blank_lines
+      frag_block = nil
+      blank_count = 0
+      @fragments.each_with_index do |frag, i|
+        if frag_block.nil?
+          frag_block = frag if Verbatim === frag
+        else
+          case frag
+          when Verbatim
+            blank_count.times { frag_block.add_text("\n") }
+            blank_count = 0
+            frag_block.add_text(frag.txt)
+            @fragments[i] = nil    # remove out current fragment
+          when BlankLine
+            if frag_block
+              blank_count += 1
+              @fragments[i] = nil
+            end
+          else
+            frag_block = nil
+            blank_count = 0
+          end
+        end
+      end
+      @fragments.compact!
+    end
+
+    # List nesting is implicit given the level of
+    # Make it explicit, just to make life a tad
+    # easier for the output processors
+
+    def add_list_start_and_ends
+      level = 0
+      res = []
+      type_stack = []
+
+      @fragments.each do |fragment|
+        # $stderr.puts "#{level} : #{fragment.class.name} : #{fragment.level}"
+        new_level = fragment.level
+        while (level < new_level)
+          level += 1
+          type = fragment.type
+          res << ListStart.new(level, fragment.param, type) if type
+          type_stack.push type
+          # $stderr.puts "Start: #{level}"
+        end
+
+        while level > new_level
+          type = type_stack.pop
+          res << ListEnd.new(level, type) if type
+          level -= 1
+          # $stderr.puts "End: #{level}, #{type}"
+        end
+
+        res << fragment
+        level = fragment.level
+      end
+      level.downto(1) do |i|
+        type = type_stack.pop
+        res << ListEnd.new(i, type) if type
+      end
+
+      @fragments = res
+    end
+
+    # now insert start/ends between list entries at the
+    # same level that have different element types
+
+    def add_list_breaks
+      res = @fragments
+
+      @fragments = []
+      list_stack = []
+
+      res.each do |fragment|
+        case fragment
+        when ListStart
+          list_stack.push fragment
+        when ListEnd
+          start = list_stack.pop
+          fragment.type = start.type
+        when ListItem
+          l = list_stack.last
+          if fragment.type != l.type
+            @fragments << ListEnd.new(l.level, l.type)
+            start = ListStart.new(l.level, fragment.param, fragment.type)
+            @fragments << start
+            list_stack.pop
+            list_stack.push start
+          end
+        else
+          ;
+        end
+        @fragments << fragment
+      end
+    end
+
+    # Finally tidy up the blank lines:
+    # * change Blank/ListEnd into ListEnd/Blank
+    # * remove blank lines at the front
+
+    def tidy_blank_lines
+      (@fragments.size - 1).times do |i|
+        if @fragments[i].kind_of?(BlankLine) and 
+            @fragments[i+1].kind_of?(ListEnd)
+          @fragments[i], @fragments[i+1] = @fragments[i+1], @fragments[i] 
+        end
+      end
+
+      # remove leading blanks
+      @fragments.each_with_index do |f, i|
+        break unless f.kind_of? BlankLine
+        @fragments[i] = nil
+      end
+
+      @fragments.compact!
+    end
+
+  end
+  
+end
diff --git a/lib/rdoc/markup/simple_markup/inline.rb b/lib/rdoc/markup/simple_markup/inline.rb
new file mode 100644
index 0000000000..684ff4b275
--- /dev/null
+++ b/lib/rdoc/markup/simple_markup/inline.rb
@@ -0,0 +1,348 @@
+module SM
+
+  # We manage a set of attributes. Each attribute has a symbol name
+  # and a bit value
+
+  class Attribute
+    SPECIAL = 1
+
+    @@name_to_bitmap = { :_SPECIAL_ => SPECIAL }
+    @@next_bitmap = 2
+
+    def Attribute.bitmap_for(name)
+      bitmap = @@name_to_bitmap[name]
+      if !bitmap
+        bitmap = @@next_bitmap
+        @@next_bitmap <<= 1
+        @@name_to_bitmap[name] = bitmap
+      end
+      bitmap
+    end
+
+    def Attribute.as_string(bitmap)
+      return "none" if bitmap.zero?
+      res = []
+      @@name_to_bitmap.each do |name, bit|
+        res << name if (bitmap & bit) != 0
+      end
+      res.join(",")
+    end
+
+    def Attribute.each_name_of(bitmap)
+      @@name_to_bitmap.each do |name, bit|
+        next if bit == SPECIAL
+        yield name.to_s if (bitmap & bit) != 0
+      end
+    end
+  end
+
+
+  # An AttrChanger records a change in attributes. It contains
+  # a bitmap of the attributes to turn on, and a bitmap of those to
+  # turn off
+
+  AttrChanger = Struct.new(:turn_on, :turn_off)
+  class AttrChanger
+    def to_s
+      "Attr: +#{Attribute.as_string(@turn_on)}/-#{Attribute.as_string(@turn_on)}"
+    end
+  end
+
+  # An array of attributes which parallels the characters in a string
+  class AttrSpan
+    def initialize(length)
+      @attrs = Array.new(length, 0)
+    end
+
+    def set_attrs(start, length, bits)
+      for i in start ... (start+length)
+        @attrs[i] |= bits
+      end
+    end
+
+    def [](n)
+      @attrs[n]
+    end
+  end
+
+  ##
+  # Hold details of a special sequence
+
+  class Special
+    attr_reader   :type
+    attr_accessor :text
+
+    def initialize(type, text)
+      @type, @text = type, text
+    end
+
+    def ==(o)
+      self.text == o.text && self.type == o.type
+    end
+
+    def to_s
+      "Special: type=#{type}, text=#{text.dump}"
+    end
+  end
+  
+  class AttributeManager
+
+    NULL = "\000".freeze
+
+    ##
+    # We work by substituting non-printing characters in to the
+    # text. For now I'm assuming that I can substitute
+    # a character in the range 0..8 for a 7 bit character
+    # without damaging the encoded string, but this might
+    # be optimistic
+    #
+
+=begin
+    ATTR_FLAG  = 001
+    A_START    = 002
+    A_END      = 003
+    A_SPECIAL_START = 005
+    A_SPECIAL_END   = 006
+
+    START_ATTR   = ATTR_FLAG.chr + A_START.chr
+    END_ATTR     = ATTR_FLAG.chr + A_END.chr
+
+    START_SPECIAL = ATTR_FLAG.chr + A_SPECIAL_START.chr
+    END_SPECIAL   = ATTR_FLAG.chr + A_SPECIAL_END.chr
+
+=end
+    A_PROTECT  = 004
+    PROTECT_ATTR  = A_PROTECT.chr
+
+    # This maps delimiters that occur around words (such as
+    # *bold* or +tt+) where the start and end delimiters
+    # and the same. This lets us optimize the regexp
+    MATCHING_WORD_PAIRS = {}
+
+    # And this is used when the delimiters aren't the same. In this
+    # case the hash maps a pattern to the attribute character
+    WORD_PAIR_MAP = {}
+
+    # This maps HTML tags to the corresponding attribute char
+    HTML_TAGS = {}
+
+    # And this maps _special_ sequences to a name. A special sequence
+    # is something like a WikiWord
+    SPECIAL = {}
+
+    # Return an attribute object with the given turn_on
+    # and turn_off bits set
+
+    def attribute(turn_on, turn_off)
+      AttrChanger.new(turn_on, turn_off)
+    end
+
+
+    def change_attribute(current, new)
+      diff = current ^ new
+      attribute(new & diff, current & diff)
+    end
+
+    def changed_attribute_by_name(current_set, new_set)
+      current = new = 0
+      current_set.each {|name| current |= Attribute.bitmap_for(name) }
+      new_set.each {|name| new |= Attribute.bitmap_for(name) }
+      change_attribute(current, new)
+    end
+
+    def copy_string(start_pos, end_pos)
+      res = @str[start_pos...end_pos]
+      res.gsub!(/\000/, '')
+      res
+    end
+
+    # Map attributes like <b>text</b>to the sequence \001\002<char>\001\003<char>,
+    # where <char> is a per-attribute specific character
+
+    def convert_attrs(str, attrs)
+      # first do matching ones
+      tags = MATCHING_WORD_PAIRS.keys.join("")
+      re = "(^|\\W)([#{tags}])([A-Za-z_]+?)\\2(\\W|\$)"
+#      re = "(^|\\W)([#{tags}])(\\S+?)\\2(\\W|\$)"
+      1 while str.gsub!(Regexp.new(re)) {
+        attr = MATCHING_WORD_PAIRS[$2];
+        attrs.set_attrs($`.length + $1.length + $2.length, $3.length, attr)
+        $1 + NULL*$2.length + $3 + NULL*$2.length + $4
+      }
+
+      # then non-matching
+      unless WORD_PAIR_MAP.empty?
+        WORD_PAIR_MAP.each do |regexp, attr|
+          str.gsub!(regexp) { 
+            attrs.set_attrs($`.length + $1.length, $2.length, attr)
+            NULL*$1.length + $2 + NULL*$3.length
+          }
+        end
+      end
+    end
+
+    def convert_html(str, attrs)
+      tags = HTML_TAGS.keys.join("|")
+      re = "<(#{tags})>(.*?)</\\1>"
+      1 while str.gsub!(Regexp.new(re, Regexp::IGNORECASE)) {
+        attr = HTML_TAGS[$1.downcase]
+        html_length = $1.length + 2
+        seq = NULL * html_length
+        attrs.set_attrs($`.length + html_length, $2.length, attr)
+        seq + $2 + seq + NULL
+      }
+    end
+
+    def convert_specials(str, attrs)
+      unless SPECIAL.empty?
+        SPECIAL.each do |regexp, attr|
+          str.scan(regexp) do
+            attrs.set_attrs($`.length, $1.length, attr | Attribute::SPECIAL)
+          end
+        end
+      end
+    end
+
+    # A \ in front of a character that would normally be
+    # processed turns off processing. We do this by turning
+    # \< into <#{PROTECT}
+    
+    PROTECTABLE = [ "<" << "\\" ]  #"
+
+
+    def mask_protected_sequences
+      protect_pattern = Regexp.new("\\\\([#{Regexp.escape(PROTECTABLE.join(''))}])")
+      @str.gsub!(protect_pattern, "\\1#{PROTECT_ATTR}")
+    end
+
+    def unmask_protected_sequences
+      @str.gsub!(/(.)#{PROTECT_ATTR}/, '\1')
+    end
+
+    def initialize
+      add_word_pair("*", "*", :BOLD)
+      add_word_pair("_", "_", :EM)
+      add_word_pair("+", "+", :TT)
+      
+      add_html("em", :EM)
+      add_html("i",  :EM)
+      add_html("b",  :BOLD)
+      add_html("tt", :TT)
+    end
+
+    def add_word_pair(start, stop, name)
+      raise "Word flags may not start '<'" if start[0] == ?<
+      bitmap = Attribute.bitmap_for(name)
+      if start == stop
+        MATCHING_WORD_PAIRS[start] = bitmap
+      else
+        pattern = Regexp.new("(" + Regexp.escape(start) + ")" +
+#                             "([A-Za-z]+)" +
+                             "(\\S+)" +
+                             "(" + Regexp.escape(stop) +")")
+        WORD_PAIR_MAP[pattern] = bitmap
+      end
+      PROTECTABLE << start[0,1]
+      PROTECTABLE.uniq!
+    end
+
+    def add_html(tag, name)
+      HTML_TAGS[tag.downcase] = Attribute.bitmap_for(name)
+    end
+
+    def add_special(pattern, name)
+      SPECIAL[pattern] = Attribute.bitmap_for(name)
+    end
+
+    def flow(str)
+      @str = str
+      @attrs = AttrSpan.new(str.length)
+
+      puts("Before flow, str='#{@str.dump}'") if $DEBUG
+      mask_protected_sequences
+      convert_attrs(@str, @attrs)
+      convert_html(@str, @attrs)
+      convert_specials(str, @attrs)
+      unmask_protected_sequences
+      puts("After flow, str='#{@str.dump}'") if $DEBUG
+      return split_into_flow
+    end
+
+    def display_attributes
+      puts
+      puts @str.tr(NULL, "!")
+      bit = 1
+      16.times do |bno|
+        line = ""
+        @str.length.times do |i|
+          if (@attrs[i] & bit) == 0
+            line << " "
+          else
+            if bno.zero?
+              line << "S"
+            else
+              line << ("%d" % (bno+1))
+            end
+          end
+        end
+        puts(line) unless line =~ /^ *$/
+        bit <<= 1
+      end
+    end
+
+    def split_into_flow
+
+      display_attributes if $DEBUG
+
+      res = []
+      current_attr = 0
+      str = ""
+
+      
+      str_len = @str.length
+
+      # skip leading invisible text
+      i = 0
+      i += 1 while i < str_len and @str[i].zero?
+      start_pos = i
+
+      # then scan the string, chunking it on attribute changes
+      while i < str_len
+        new_attr = @attrs[i]
+        if new_attr != current_attr
+          if i > start_pos
+            res << copy_string(start_pos, i)
+            start_pos = i
+          end
+
+          res << change_attribute(current_attr, new_attr)
+          current_attr = new_attr
+
+          if (current_attr & Attribute::SPECIAL) != 0
+            i += 1 while i < str_len and (@attrs[i] & Attribute::SPECIAL) != 0
+            res << Special.new(current_attr, copy_string(start_pos, i))
+            start_pos = i
+            next
+          end
+        end
+
+        # move on, skipping any invisible characters
+        begin
+          i += 1
+        end while i < str_len and @str[i].zero?
+      end
+      
+      # tidy up trailing text
+      if start_pos < str_len
+        res << copy_string(start_pos, str_len)
+      end
+
+      # and reset to all attributes off
+      res << change_attribute(current_attr, 0) if current_attr != 0
+
+      return res
+    end
+
+  end
+
+end
diff --git a/lib/rdoc/markup/simple_markup/lines.rb b/lib/rdoc/markup/simple_markup/lines.rb
new file mode 100644
index 0000000000..4e294f27dc
--- /dev/null
+++ b/lib/rdoc/markup/simple_markup/lines.rb
@@ -0,0 +1,151 @@
+##########################################################################
+#
+# We store the lines we're working on as objects of class Line.
+# These contain the text of the line, along with a flag indicating the
+# line type, and an indentation level
+
+module SM
+
+  class Line
+    INFINITY = 9999
+
+    BLANK     = :BLANK
+    HEADING   = :HEADING
+    LIST      = :LIST
+    RULE      = :RULE
+    PARAGRAPH = :PARAGRAPH
+    VERBATIM  = :VERBATIM
+    
+    # line type
+    attr_accessor :type
+
+    # The indentation nesting level
+    attr_accessor :level
+
+    # The contents
+    attr_accessor :text
+
+    # A prefix or parameter. For LIST lines, this is
+    # the text that introduced the list item (the label)
+    attr_accessor  :param
+
+    # A flag. For list lines, this is the type of the list
+    attr_accessor :flag
+
+    # the number of leading spaces
+    attr_accessor :leading_spaces
+
+    # true if this line has been deleted from the list of lines
+    attr_accessor :deleted
+    
+
+    def initialize(text)
+      @text    = text.dup
+      @deleted = false
+
+      # expand tabs
+      1 while @text.gsub!(/\t+/) { ' ' * (8*$&.length - $`.length % 8)}  && $~ #`
+
+      # Strip trailing whitespace
+      @text.sub!(/\s+$/, '')
+
+      # and look for leading whitespace
+      if @text.length > 0
+        @text =~ /^(\s*)/
+        @leading_spaces = $1.length
+      else
+        @leading_spaces = INFINITY
+      end
+    end
+
+    # Return true if this line is blank
+    def isBlank?
+      @text.length.zero?
+    end
+
+    # stamp a line with a type, a level, a prefix, and a flag
+    def stamp(type, level, param="", flag=nil)
+      @type, @level, @param, @flag = type, level, param, flag
+    end
+
+    ##
+    # Strip off the leading margin
+    #
+
+    def strip_leading(size)
+      if @text.size > size
+        @text[0,size] = ""
+      else
+        @text = ""
+      end
+    end
+
+    def to_s
+      "#@type#@level: #@text"
+    end
+  end
+
+  ###############################################################################
+  #
+  # A container for all the lines
+  #
+
+  class Lines
+    include Enumerable
+
+    attr_reader :lines   # for debugging
+
+    def initialize(lines)
+      @lines = lines
+      rewind
+    end
+
+    def empty?
+      @lines.size.zero?
+    end
+
+    def each
+      @lines.each do |line|
+        yield line unless line.deleted
+      end
+    end
+
+#    def [](index)
+#      @lines[index]
+#    end
+
+    def rewind
+      @nextline = 0
+    end
+
+    def next
+      begin
+        res = @lines[@nextline]
+        @nextline += 1 if @nextline < @lines.size
+      end while res and res.deleted and @nextline < @lines.size
+      res
+    end
+
+    def unget
+      @nextline -= 1
+    end
+
+    def delete(a_line)
+      a_line.deleted = true
+    end
+
+    def normalize
+      margin = @lines.collect{|l| l.leading_spaces}.min
+      margin = 0 if margin == Line::INFINITY
+      @lines.each {|line| line.strip_leading(margin) } if margin > 0
+    end
+
+    def as_text
+      @lines.map {|l| l.text}.join("\n")
+    end
+
+    def line_types
+      @lines.map {|l| l.type }
+    end
+  end
+end
diff --git a/lib/rdoc/markup/simple_markup/preprocess.rb b/lib/rdoc/markup/simple_markup/preprocess.rb
new file mode 100644
index 0000000000..09892c2b6c
--- /dev/null
+++ b/lib/rdoc/markup/simple_markup/preprocess.rb
@@ -0,0 +1,68 @@
+module SM
+
+  ## 
+  # Handle common directives that can occur in a block of text:
+  #
+  # : include : filename
+  #
+
+  class PreProcess
+
+    def initialize(input_file_name, include_path)
+      @input_file_name = input_file_name
+      @include_path = include_path
+    end
+
+    # Look for common options in a chunk of text. Options that
+    # we don't handle are passed back to our caller
+    # as |directive, param| 
+
+    def handle(text)
+      text.gsub!(/^([ \t#]*):(\w+):\s*(.+)?\n/) do 
+
+        directive = $2.downcase
+        param     = $3
+
+        case directive
+
+        when "include"
+          include_file($3, $1)
+
+        else
+          yield(directive, param)
+        end
+      end
+    end
+
+    #######
+    private
+    #######
+
+    # Include a file, indenting it correctly
+
+    def include_file(name, indent)
+      if (full_name = find_include_file(name))
+        content = File.open(full_name) {|f| f.read}
+        res = content.gsub(/^#?/, indent)
+      else
+        $stderr.puts "Couldn't find file to include: '#{name}'"
+        ''
+      end
+    end
+
+    # Look for the given file in the directory containing the current
+    # file, and then in each of the directories specified in the
+    # RDOC_INCLUDE path
+
+    def find_include_file(name)
+      to_search = [ File.dirname(@input_file_name) ].concat @include_path
+      to_search.each do |dir|
+        full_name = File.join(dir, name)
+        stat = File.stat(full_name) rescue next
+        return full_name if stat.readable?
+      end
+      nil
+    end
+
+  end
+end
diff --git a/lib/rdoc/markup/simple_markup/to_html.rb b/lib/rdoc/markup/simple_markup/to_html.rb
new file mode 100644
index 0000000000..26b5f4ce70
--- /dev/null
+++ b/lib/rdoc/markup/simple_markup/to_html.rb
@@ -0,0 +1,289 @@
+require 'rdoc/markup/simple_markup/fragments'
+require 'rdoc/markup/simple_markup/inline'
+
+require 'cgi'
+
+module SM
+
+  class ToHtml
+
+    LIST_TYPE_TO_HTML = {
+      ListBase::BULLET =>  [ "<ul>", "</ul>" ],
+      ListBase::NUMBER =>  [ "<ol>", "</ol>" ],
+      ListBase::UPPERALPHA =>  [ "<ol>", "</ol>" ],
+      ListBase::LOWERALPHA =>  [ "<ol>", "</ol>" ],
+      ListBase::LABELED => [ "<dl>", "</dl>" ],
+      ListBase::NOTE    => [ "<table>", "</table>" ],
+    }
+
+    InlineTag = Struct.new(:bit, :on, :off)
+
+    def initialize
+      init_tags
+    end
+
+    ##
+    # Set up the standard mapping of attributes to HTML tags
+    #
+    def init_tags
+      @attr_tags = [
+        InlineTag.new(SM::Attribute.bitmap_for(:BOLD), "<b>", "</b>"),
+        InlineTag.new(SM::Attribute.bitmap_for(:TT),   "<tt>", "</tt>"),
+        InlineTag.new(SM::Attribute.bitmap_for(:EM),   "<em>", "</em>"),
+      ]
+    end
+
+    ##
+    # Add a new set of HTML tags for an attribute. We allow
+    # separate start and end tags for flexibility
+    #
+    def add_tag(name, start, stop)
+      @attr_tags << InlineTag.new(SM::Attribute.bitmap_for(name), start, stop)
+    end
+
+    ##
+    # Given an HTML tag, decorate it with class information
+    # and the like if required. This is a no-op in the base
+    # class, but is overridden in HTML output classes that
+    # implement style sheets
+
+    def annotate(tag)
+      tag
+    end
+
+    ## 
+    # Here's the client side of the visitor pattern
+
+    def start_accepting
+      @res = ""
+      @in_list_entry = []
+    end
+
+    def end_accepting
+      @res
+    end
+
+    def accept_paragraph(am, fragment)
+      @res << annotate("<p>") + "\n"
+      @res << wrap(convert_flow(am.flow(fragment.txt)))
+      @res << annotate("</p>") + "\n"
+    end
+
+    def accept_verbatim(am, fragment)
+      @res << annotate("<pre>") + "\n"
+      @res << CGI.escapeHTML(fragment.txt)
+      @res << annotate("</pre>") << "\n"
+    end
+
+    def accept_rule(am, fragment)
+      size = fragment.param
+      size = 10 if size > 10
+      @res << "<hr size=\"#{size}\"></hr>"
+    end
+
+    def accept_list_start(am, fragment)
+      @res << html_list_name(fragment.type, true) <<"\n"
+      @in_list_entry.push false
+    end
+
+    def accept_list_end(am, fragment)
+      if tag = @in_list_entry.pop
+        @res << annotate(tag) << "\n"
+      end
+      @res << html_list_name(fragment.type, false) <<"\n"
+    end
+
+    def accept_list_item(am, fragment)
+      if tag = @in_list_entry.last
+        @res << annotate(tag) << "\n"
+      end
+      @res << list_item_start(am, fragment)
+      @res << wrap(convert_flow(am.flow(fragment.txt))) << "\n"
+      @in_list_entry[-1] = list_end_for(fragment.type)
+    end
+
+    def accept_blank_line(am, fragment)
+      # @res << annotate("<p />") << "\n"
+    end
+
+    def accept_heading(am, fragment)
+      @res << convert_heading(fragment.head_level, am.flow(fragment.txt))
+    end
+
+    # This is a higher speed (if messier) version of wrap
+
+    def wrap(txt, line_len = 76)
+      res = ""
+      sp = 0
+      ep = txt.length
+      while sp < ep
+        # scan back for a space
+        p = sp + line_len - 1
+        if p >= ep
+          p = ep
+        else
+          while p > sp and txt[p] != ?\s
+            p -= 1
+          end
+          if p <= sp
+            p = sp + line_len
+            while p < ep and txt[p] != ?\s
+              p += 1
+            end
+          end
+        end
+        res << txt[sp...p] << "\n"
+        sp = p
+        sp += 1 while sp < ep and txt[sp] == ?\s
+      end
+      res
+    end
+
+    #######################################################################
+
+    private
+
+    #######################################################################
+
+    def on_tags(res, item)
+      attr_mask = item.turn_on
+      return if attr_mask.zero?
+
+      @attr_tags.each do |tag|
+        if attr_mask & tag.bit != 0
+          res << annotate(tag.on)
+        end
+      end
+    end
+
+    def off_tags(res, item)
+      attr_mask = item.turn_off
+      return if attr_mask.zero?
+
+      @attr_tags.reverse_each do |tag|
+        if attr_mask & tag.bit != 0
+          res << annotate(tag.off)
+        end
+      end
+    end
+
+    def convert_flow(flow)
+      res = ""
+      flow.each do |item|
+        case item
+        when String
+          res << convert_string(item)
+        when AttrChanger
+          off_tags(res, item)
+          on_tags(res,  item)
+        when Special
+          res << convert_special(item)
+        else
+          raise "Unknown flow element: #{item.inspect}"
+        end
+      end
+      res
+    end
+
+    # some of these patterns are taken from SmartyPants...
+
+    def convert_string(item)
+      CGI.escapeHTML(item).
+      
+      
+      # convert -- to em-dash, (-- to en-dash)
+        gsub(/---?/, '&#8212;'). #gsub(/--/, '&#8211;').
+
+      # convert ... to elipsis (and make sure .... becomes .<elipsis>)
+        gsub(/\.\.\.\./, '.&#8230;').gsub(/\.\.\./, '&#8230;').
+
+      # convert single closing quote
+        gsub(%r{([^ \t\r\n\[\{\(])\'}) { "#$1&#8217;" }.
+        gsub(%r{\'(?=\W|s\b)}) { "&#8217;" }.
+
+      # convert single opening quote
+        gsub(/'/, '&#8216;').
+
+      # convert double closing quote
+        gsub(%r{([^ \t\r\n\[\{\(])\'(?=\W)}) { "#$1&#8221;" }.
+
+      # convert double opening quote
+        gsub(/'/, '&#8220;').
+
+      # convert copyright
+        gsub(/\(c\)/, '&#169;').
+
+      # convert and registered trademark
+        gsub(/\(r\)/, '&#174;')
+
+    end
+
+    def convert_special(special)
+      handled = false
+      Attribute.each_name_of(special.type) do |name|
+        method_name = "handle_special_#{name}"
+        if self.respond_to? method_name
+          special.text = send(method_name, special)
+          handled = true
+        end
+      end
+      raise "Unhandled special: #{special}" unless handled
+      special.text
+    end
+
+    def convert_heading(level, flow)
+      res =
+        annotate("<h#{level}>") + 
+        convert_flow(flow) + 
+        annotate("</h#{level}>\n")
+    end
+
+    def html_list_name(list_type, is_open_tag)
+      tags = LIST_TYPE_TO_HTML[list_type] || raise("Invalid list type: #{list_type.inspect}")
+      annotate(tags[ is_open_tag ? 0 : 1])
+    end
+
+    def list_item_start(am, fragment)
+      case fragment.type
+      when ListBase::BULLET, ListBase::NUMBER
+        annotate("<li>")
+
+      when ListBase::UPPERALPHA
+	annotate("<li type=\"A\">")
+
+      when ListBase::LOWERALPHA
+	annotate("<li type=\"a\">")
+
+      when ListBase::LABELED
+        annotate("<dt>") +
+          convert_flow(am.flow(fragment.param)) + 
+          annotate("</dt>") +
+          annotate("<dd>")
+
+      when ListBase::NOTE
+        annotate("<tr>") +
+          annotate("<td valign=\"top\">") +
+          convert_flow(am.flow(fragment.param)) + 
+          annotate("</td>") +
+          annotate("<td>")
+      else
+        raise "Invalid list type"
+      end
+    end
+
+    def list_end_for(fragment_type)
+      case fragment_type
+      when ListBase::BULLET, ListBase::NUMBER, ListBase::UPPERALPHA, ListBase::LOWERALPHA
+        "</li>"
+      when ListBase::LABELED
+        "</dd>"
+      when ListBase::NOTE
+        "</td></tr>"
+      else
+        raise "Invalid list type"
+      end
+    end
+
+  end
+
+end
diff --git a/lib/rdoc/markup/simple_markup/to_latex.rb b/lib/rdoc/markup/simple_markup/to_latex.rb
new file mode 100644
index 0000000000..6c16278652
--- /dev/null
+++ b/lib/rdoc/markup/simple_markup/to_latex.rb
@@ -0,0 +1,333 @@
+require 'rdoc/markup/simple_markup/fragments'
+require 'rdoc/markup/simple_markup/inline'
+
+require 'cgi'
+
+module SM
+
+  # Convert SimpleMarkup to basic LaTeX report format
+
+  class ToLaTeX
+
+    BS = "\020"   # \
+    OB = "\021"   # {
+    CB = "\022"   # }
+    DL = "\023"   # Dollar
+
+    BACKSLASH   = "#{BS}symbol#{OB}92#{CB}"
+    HAT         = "#{BS}symbol#{OB}94#{CB}"
+    BACKQUOTE   = "#{BS}symbol#{OB}0#{CB}"
+    TILDE       = "#{DL}#{BS}sim#{DL}"
+    LESSTHAN    = "#{DL}<#{DL}"
+    GREATERTHAN = "#{DL}>#{DL}"
+
+    def self.l(str)
+      str.tr('\\', BS).tr('{', OB).tr('}', CB).tr('$', DL)
+    end
+
+    def l(arg)
+      SM::ToLaTeX.l(arg)
+    end
+
+    LIST_TYPE_TO_LATEX = {
+      ListBase::BULLET =>  [ l("\\begin{itemize}"), l("\\end{itemize}") ],
+      ListBase::NUMBER =>  [ l("\\begin{enumerate}"), l("\\end{enumerate}"), "\\arabic" ],
+      ListBase::UPPERALPHA =>  [ l("\\begin{enumerate}"), l("\\end{enumerate}"), "\\Alph" ],
+      ListBase::LOWERALPHA =>  [ l("\\begin{enumerate}"), l("\\end{enumerate}"), "\\alph" ],
+      ListBase::LABELED => [ l("\\begin{description}"), l("\\end{description}") ],
+      ListBase::NOTE    => [
+        l("\\begin{tabularx}{\\linewidth}{@{} l X @{}}"), 
+        l("\\end{tabularx}") ],
+    }
+
+    InlineTag = Struct.new(:bit, :on, :off)
+
+    def initialize
+      init_tags
+      @list_depth = 0
+      @prev_list_types = []
+    end
+
+    ##
+    # Set up the standard mapping of attributes to LaTeX
+    #
+    def init_tags
+      @attr_tags = [
+        InlineTag.new(SM::Attribute.bitmap_for(:BOLD), l("\\textbf{"), l("}")),
+        InlineTag.new(SM::Attribute.bitmap_for(:TT),   l("\\texttt{"), l("}")),
+        InlineTag.new(SM::Attribute.bitmap_for(:EM),   l("\\emph{"), l("}")),
+      ]
+    end
+
+    ##
+    # Escape a LaTeX string
+    def escape(str)
+# $stderr.print "FE: ", str
+      s = str.
+#        sub(/\s+$/, '').
+        gsub(/([_\${}&%#])/, "#{BS}\\1").
+        gsub(/\\/, BACKSLASH).
+        gsub(/\^/, HAT).
+        gsub(/~/,  TILDE).
+        gsub(/</,  LESSTHAN).
+        gsub(/>/,  GREATERTHAN).
+        gsub(/,,/, ",{},").
+        gsub(/\`/,  BACKQUOTE)
+# $stderr.print "-> ", s, "\n"
+      s
+    end
+
+    ##
+    # Add a new set of LaTeX tags for an attribute. We allow
+    # separate start and end tags for flexibility
+    #
+    def add_tag(name, start, stop)
+      @attr_tags << InlineTag.new(SM::Attribute.bitmap_for(name), start, stop)
+    end
+
+
+    ## 
+    # Here's the client side of the visitor pattern
+
+    def start_accepting
+      @res = ""
+      @in_list_entry = []
+    end
+
+    def end_accepting
+      @res.tr(BS, '\\').tr(OB, '{').tr(CB, '}').tr(DL, '$')
+    end
+
+    def accept_paragraph(am, fragment)
+      @res << wrap(convert_flow(am.flow(fragment.txt)))
+      @res << "\n"
+    end
+
+    def accept_verbatim(am, fragment)
+      @res << "\n\\begin{code}\n"
+      @res << fragment.txt.sub(/[\n\s]+\Z/, '')
+      @res << "\n\\end{code}\n\n"
+    end
+
+    def accept_rule(am, fragment)
+      size = fragment.param
+      size = 10 if size > 10
+      @res << "\n\n\\rule{\\linewidth}{#{size}pt}\n\n"
+    end
+
+    def accept_list_start(am, fragment)
+      @res << list_name(fragment.type, true) <<"\n"
+      @in_list_entry.push false
+    end
+
+    def accept_list_end(am, fragment)
+      if tag = @in_list_entry.pop
+        @res << tag << "\n"
+      end
+      @res << list_name(fragment.type, false) <<"\n"
+    end
+
+    def accept_list_item(am, fragment)
+      if tag = @in_list_entry.last
+        @res << tag << "\n"
+      end
+      @res << list_item_start(am, fragment)
+      @res << wrap(convert_flow(am.flow(fragment.txt))) << "\n"
+      @in_list_entry[-1] = list_end_for(fragment.type)
+    end
+
+    def accept_blank_line(am, fragment)
+      # @res << "\n"
+    end
+
+    def accept_heading(am, fragment)
+      @res << convert_heading(fragment.head_level, am.flow(fragment.txt))
+    end
+
+    # This is a higher speed (if messier) version of wrap
+
+    def wrap(txt, line_len = 76)
+      res = ""
+      sp = 0
+      ep = txt.length
+      while sp < ep
+        # scan back for a space
+        p = sp + line_len - 1
+        if p >= ep
+          p = ep
+        else
+          while p > sp and txt[p] != ?\s
+            p -= 1
+          end
+          if p <= sp
+            p = sp + line_len
+            while p < ep and txt[p] != ?\s
+              p += 1
+            end
+          end
+        end
+        res << txt[sp...p] << "\n"
+        sp = p
+        sp += 1 while sp < ep and txt[sp] == ?\s
+      end
+      res
+    end
+
+    #######################################################################
+
+    private
+
+    #######################################################################
+
+    def on_tags(res, item)
+      attr_mask = item.turn_on
+      return if attr_mask.zero?
+
+      @attr_tags.each do |tag|
+        if attr_mask & tag.bit != 0
+          res << tag.on
+        end
+      end
+    end
+
+    def off_tags(res, item)
+      attr_mask = item.turn_off
+      return if attr_mask.zero?
+
+      @attr_tags.reverse_each do |tag|
+        if attr_mask & tag.bit != 0
+          res << tag.off
+        end
+      end
+    end
+
+    def convert_flow(flow)
+      res = ""
+      flow.each do |item|
+        case item
+        when String
+#          $stderr.puts "Converting '#{item}'"
+          res << convert_string(item)
+        when AttrChanger
+          off_tags(res, item)
+          on_tags(res,  item)
+        when Special
+          res << convert_special(item)
+        else
+          raise "Unknown flow element: #{item.inspect}"
+        end
+      end
+      res
+    end
+
+    # some of these patterns are taken from SmartyPants...
+
+    def convert_string(item)
+
+      escape(item).
+      
+      
+      # convert ... to elipsis (and make sure .... becomes .<elipsis>)
+        gsub(/\.\.\.\./, '.\ldots{}').gsub(/\.\.\./, '\ldots{}').
+
+      # convert single closing quote
+        gsub(%r{([^ \t\r\n\[\{\(])\'}) { "#$1'" }.
+        gsub(%r{\'(?=\W|s\b)}) { "'" }.
+
+      # convert single opening quote
+        gsub(/'/, '`').
+
+      # convert double closing quote
+        gsub(%r{([^ \t\r\n\[\{\(])\"(?=\W)}) { "#$1''" }.
+
+      # convert double opening quote
+        gsub(/"/, "``").
+
+      # convert copyright
+        gsub(/\(c\)/, '\copyright{}')
+
+    end
+
+    def convert_special(special)
+      handled = false
+      Attribute.each_name_of(special.type) do |name|
+        method_name = "handle_special_#{name}"
+        if self.respond_to? method_name
+          special.text = send(method_name, special)
+          handled = true
+        end
+      end
+      raise "Unhandled special: #{special}" unless handled
+      special.text
+    end
+
+    def convert_heading(level, flow)
+      res =
+        case level
+        when 1 then "\\chapter{"
+        when 2 then "\\section{"
+        when 3 then "\\subsection{"
+        when 4 then "\\subsubsection{"
+        else  "\\paragraph{"
+        end +
+        convert_flow(flow) + 
+        "}\n"
+    end
+
+    def list_name(list_type, is_open_tag)
+      tags = LIST_TYPE_TO_LATEX[list_type] || raise("Invalid list type: #{list_type.inspect}")
+      if tags[2] # enumerate
+        if is_open_tag
+          @list_depth += 1
+          if @prev_list_types[@list_depth] != tags[2]
+            case @list_depth
+            when 1
+              roman = "i"
+            when 2
+              roman = "ii"
+            when 3
+              roman = "iii"
+            when 4
+              roman = "iv"
+            else
+              raise("Too deep list: level #{@list_depth}")
+            end
+            @prev_list_types[@list_depth] = tags[2]
+            return l("\\renewcommand{\\labelenum#{roman}}{#{tags[2]}{enum#{roman}}}") + "\n" + tags[0]
+          end
+        else
+          @list_depth -= 1
+        end
+      end
+      tags[ is_open_tag ? 0 : 1]
+    end
+
+    def list_item_start(am, fragment)
+      case fragment.type
+      when ListBase::BULLET, ListBase::NUMBER, ListBase::UPPERALPHA, ListBase::LOWERALPHA
+        "\\item "
+
+      when ListBase::LABELED
+        "\\item[" + convert_flow(am.flow(fragment.param)) + "] "
+
+      when ListBase::NOTE
+          convert_flow(am.flow(fragment.param)) + " & "
+      else
+        raise "Invalid list type"
+      end
+    end
+
+    def list_end_for(fragment_type)
+      case fragment_type
+      when ListBase::BULLET, ListBase::NUMBER, ListBase::UPPERALPHA, ListBase::LOWERALPHA, ListBase::LABELED
+        ""
+      when ListBase::NOTE
+        "\\\\\n"
+      else
+        raise "Invalid list type"
+      end
+    end
+
+  end
+
+end
diff --git a/lib/rdoc/markup/test/AllTests.rb b/lib/rdoc/markup/test/AllTests.rb
new file mode 100644
index 0000000000..b9c8c9dfcc
--- /dev/null
+++ b/lib/rdoc/markup/test/AllTests.rb
@@ -0,0 +1,2 @@
+require 'TestParse.rb'
+require 'TestInline.rb'
diff --git a/lib/rdoc/markup/test/TestInline.rb b/lib/rdoc/markup/test/TestInline.rb
new file mode 100644
index 0000000000..c76f21dce5
--- /dev/null
+++ b/lib/rdoc/markup/test/TestInline.rb
@@ -0,0 +1,151 @@
+require "test/unit"
+
+$:.unshift "../../.."
+
+require "rdoc/markup/simple_markup/inline"
+
+class TestInline < Test::Unit::TestCase
+
+
+  def setup
+    @am = SM::AttributeManager.new
+
+    @bold_on  = @am.changed_attribute_by_name([], [:BOLD])
+    @bold_off = @am.changed_attribute_by_name([:BOLD], [])
+    
+    @tt_on    = @am.changed_attribute_by_name([], [:TT])
+    @tt_off   = @am.changed_attribute_by_name([:TT], [])
+    
+    @em_on    = @am.changed_attribute_by_name([], [:EM])
+    @em_off   = @am.changed_attribute_by_name([:EM], [])
+    
+    @bold_em_on   = @am.changed_attribute_by_name([], [:BOLD] | [:EM])
+    @bold_em_off  = @am.changed_attribute_by_name([:BOLD] | [:EM], [])
+    
+    @em_then_bold = @am.changed_attribute_by_name([:EM], [:EM] | [:BOLD])
+    
+    @em_to_bold   = @am.changed_attribute_by_name([:EM], [:BOLD])
+    
+    @am.add_word_pair("{", "}", :WOMBAT)
+    @wombat_on    = @am.changed_attribute_by_name([], [:WOMBAT])
+    @wombat_off   = @am.changed_attribute_by_name([:WOMBAT], [])
+  end
+
+  def crossref(text)
+    [ @am.changed_attribute_by_name([], [:CROSSREF] | [:_SPECIAL_]),
+      SM::Special.new(33, text),
+      @am.changed_attribute_by_name([:CROSSREF] | [:_SPECIAL_], [])
+    ]
+  end
+
+  def test_special
+    # class names, variable names, file names, or instance variables
+    @am.add_special(/(
+                       \b([A-Z]\w+(::\w+)*)
+                       | \#\w+[!?=]?
+                       | \b\w+([_\/\.]+\w+)+[!?=]?
+                      )/x, 
+                    :CROSSREF)
+    
+    assert_equal(["cat"], @am.flow("cat"))
+
+    assert_equal(["cat ", crossref("#fred"), " dog"].flatten,
+                  @am.flow("cat #fred dog"))
+
+    assert_equal([crossref("#fred"), " dog"].flatten,
+                  @am.flow("#fred dog"))
+
+    assert_equal(["cat ", crossref("#fred")].flatten, @am.flow("cat #fred"))
+  end
+
+  def test_basic
+    assert_equal(["cat"], @am.flow("cat"))
+
+    assert_equal(["cat ", @bold_on, "and", @bold_off, " dog"],
+                  @am.flow("cat *and* dog"))
+
+    assert_equal(["cat ", @bold_on, "AND", @bold_off, " dog"],
+                  @am.flow("cat *AND* dog"))
+
+    assert_equal(["cat ", @em_on, "And", @em_off, " dog"],
+                  @am.flow("cat _And_ dog"))
+
+    assert_equal(["cat *and dog*"], @am.flow("cat *and dog*"))
+
+    assert_equal(["*cat and* dog"], @am.flow("*cat and* dog"))
+
+    assert_equal(["cat *and ", @bold_on, "dog", @bold_off],
+                  @am.flow("cat *and *dog*"))
+
+    assert_equal(["cat ", @em_on, "and", @em_off, " dog"],
+                  @am.flow("cat _and_ dog"))
+
+    assert_equal(["cat_and_dog"],
+                  @am.flow("cat_and_dog"))
+
+    assert_equal(["cat ", @tt_on, "and", @tt_off, " dog"],
+                  @am.flow("cat +and+ dog"))
+
+    assert_equal(["cat ", @bold_on, "a_b_c", @bold_off, " dog"],
+                  @am.flow("cat *a_b_c* dog"))
+
+    assert_equal(["cat __ dog"],
+                  @am.flow("cat __ dog"))
+
+    assert_equal(["cat ", @em_on, "_", @em_off, " dog"],
+                  @am.flow("cat ___ dog"))
+
+  end
+
+  def test_combined
+    assert_equal(["cat ", @em_on, "and", @em_off, " ", @bold_on, "dog", @bold_off],
+                  @am.flow("cat _and_ *dog*"))
+
+    assert_equal(["cat ", @em_on, "a__nd", @em_off, " ", @bold_on, "dog", @bold_off], 
+                  @am.flow("cat _a__nd_ *dog*"))
+  end
+
+  def test_html_like
+    assert_equal(["cat ", @tt_on, "dog", @tt_off], @am.flow("cat <tt>dog</Tt>"))
+
+    assert_equal(["cat ", @em_on, "and", @em_off, " ", @bold_on, "dog", @bold_off], 
+                  @am.flow("cat <i>and</i> <B>dog</b>"))
+    
+    assert_equal(["cat ", @em_on, "and ", @em_then_bold, "dog", @bold_em_off], 
+                  @am.flow("cat <i>and <B>dog</B></I>"))
+    
+    assert_equal(["cat ", @em_on, "and ", @em_to_bold, "dog", @bold_off], 
+                  @am.flow("cat <i>and </i><b>dog</b>"))
+    
+    assert_equal(["cat ", @em_on, "and ", @em_to_bold, "dog", @bold_off], 
+                  @am.flow("cat <i>and <b></i>dog</b>"))
+    
+    assert_equal([@tt_on, "cat", @tt_off, " ", @em_on, "and ", @em_to_bold, "dog", @bold_off], 
+                  @am.flow("<tt>cat</tt> <i>and <b></i>dog</b>"))
+
+    assert_equal(["cat ", @em_on, "and ", @em_then_bold, "dog", @bold_em_off], 
+                  @am.flow("cat <i>and <b>dog</b></i>"))
+    
+    assert_equal(["cat ", @bold_em_on, "and", @bold_em_off, " dog"], 
+                  @am.flow("cat <i><b>and</b></i> dog"))
+    
+    
+  end
+
+  def test_protect
+    assert_equal(['cat \\ dog'], @am.flow('cat \\ dog'))
+
+    assert_equal(["cat <tt>dog</Tt>"], @am.flow("cat \\<tt>dog</Tt>"))
+
+    assert_equal(["cat ", @em_on, "and", @em_off, " <B>dog</b>"], 
+                  @am.flow("cat <i>and</i> \\<B>dog</b>"))
+    
+    assert_equal(["*word* or <b>text</b>"], @am.flow("\\*word* or \\<b>text</b>"))
+  end
+
+  def test_adding
+    assert_equal(["cat ", @wombat_on, "and", @wombat_off, " dog" ],
+                  @am.flow("cat {and} dog"))
+#    assert_equal(["cat {and} dog" ], @am.flow("cat \\{and} dog"))
+  end
+end
diff --git a/lib/rdoc/markup/test/TestParse.rb b/lib/rdoc/markup/test/TestParse.rb
new file mode 100644
index 0000000000..3ec541ce7a
--- /dev/null
+++ b/lib/rdoc/markup/test/TestParse.rb
@@ -0,0 +1,503 @@
+require 'test/unit'
+
+$:.unshift "../../.."
+
+require 'rdoc/markup/simple_markup'
+
+include SM
+
+class TestParse < Test::Unit::TestCase
+
+  class MockOutput
+    def start_accepting
+      @res = []
+      end
+    
+    def end_accepting
+      @res
+    end
+
+    def accept_paragraph(am, fragment)
+      @res << fragment.to_s
+    end
+
+    def accept_verbatim(am, fragment)
+      @res << fragment.to_s
+    end
+
+    def accept_list_start(am, fragment)
+      @res << fragment.to_s
+    end
+
+    def accept_list_end(am, fragment)
+      @res << fragment.to_s
+    end
+
+    def accept_list_item(am, fragment)
+      @res << fragment.to_s
+    end
+
+    def accept_blank_line(am, fragment)
+      @res << fragment.to_s
+    end
+
+    def accept_heading(am, fragment)
+      @res << fragment.to_s
+    end
+
+    def accept_rule(am, fragment)
+      @res << fragment.to_s
+    end
+
+  end
+
+  def basic_conv(str)
+    sm = SimpleMarkup.new
+    mock = MockOutput.new
+    sm.convert(str, mock)
+    sm.content
+  end
+
+  def line_types(str, expected)
+    p = SimpleMarkup.new
+    mock = MockOutput.new
+    p.convert(str, mock)
+    assert_equal(expected, p.get_line_types.map{|type| type.to_s[0,1]}.join(''))
+  end
+
+  def line_groups(str, expected)
+    p = SimpleMarkup.new
+    mock = MockOutput.new
+
+    block = p.convert(str, mock)
+
+    if block != expected
+      rows = (0...([expected.size, block.size].max)).collect{|i|
+        [expected[i]||"nil", block[i]||"nil"] 
+      }
+      printf "\n\n%35s %35s\n", "Expected", "Got"
+      rows.each {|e,g| printf "%35s %35s\n", e.dump, g.dump }
+    end
+
+    assert_equal(expected, block)
+  end
+
+  def test_tabs
+    str = "hello\n  dave"
+    assert_equal(str, basic_conv(str))
+    str = "hello\n\tdave"
+    assert_equal("hello\n        dave", basic_conv(str))
+    str = "hello\n \tdave"
+    assert_equal("hello\n        dave", basic_conv(str))
+    str = "hello\n  \tdave"
+    assert_equal("hello\n        dave", basic_conv(str))
+    str = "hello\n   \tdave"
+    assert_equal("hello\n        dave", basic_conv(str))
+    str = "hello\n    \tdave"
+    assert_equal("hello\n        dave", basic_conv(str))
+    str = "hello\n     \tdave"
+    assert_equal("hello\n        dave", basic_conv(str))
+    str = "hello\n      \tdave"
+    assert_equal("hello\n        dave", basic_conv(str))
+    str = "hello\n       \tdave"
+    assert_equal("hello\n        dave", basic_conv(str))
+    str = "hello\n        \tdave"
+    assert_equal("hello\n                dave", basic_conv(str))
+    str = ".\t\t."
+    assert_equal(".               .", basic_conv(str))
+  end
+
+  def test_whitespace
+    assert_equal("hello", basic_conv("hello"))
+    assert_equal("hello", basic_conv(" hello "))
+    assert_equal("hello", basic_conv(" \t \t hello\t\t"))
+
+    assert_equal("1\n 2\n  3", basic_conv("1\n 2\n  3"))
+    assert_equal("1\n 2\n  3", basic_conv("  1\n   2\n    3"))
+
+    assert_equal("1\n 2\n  3\n1\n 2", basic_conv("1\n 2\n  3\n1\n 2"))
+    assert_equal("1\n 2\n  3\n1\n 2", basic_conv("  1\n   2\n    3\n  1\n   2"))
+
+    assert_equal("1\n 2\n\n  3", basic_conv("  1\n   2\n\n    3"))
+  end
+
+  def test_types
+    str = "now is the time"
+    line_types(str, 'P')
+
+    str = "now is the time\nfor all good men"
+    line_types(str, 'PP')
+
+    str = "now is the time\n  code\nfor all good men"
+    line_types(str, 'PVP')
+
+    str = "now is the time\n  code\n more code\nfor all good men"
+    line_types(str, 'PVVP')
+
+    str = "now is\n---\nthe time"
+    line_types(str, 'PRP')
+
+    str = %{\
+       now is
+       * l1
+       * l2
+       the time}
+    line_types(str, 'PLLP')
+
+    str = %{\
+       now is
+       * l1
+         l1+
+       * l2
+       the time}
+    line_types(str, 'PLPLP')
+
+    str = %{\
+       now is
+       * l1
+         * l1.1
+       * l2
+       the time}
+    line_types(str, 'PLLLP')
+
+    str = %{\
+       now is
+       * l1
+         * l1.1
+           text
+             code
+             code
+
+           text
+       * l2
+       the time}
+    line_types(str, 'PLLPVVBPLP')
+
+    str = %{\
+       now is
+       1. l1
+          * l1.1
+       2. l2
+       the time}
+    line_types(str, 'PLLLP')
+
+    str = %{\
+       now is
+       [cat] l1
+             * l1.1
+       [dog] l2
+       the time}
+    line_types(str, 'PLLLP')
+
+    str = %{\
+       now is
+       [cat] l1
+             continuation
+       [dog] l2
+       the time}
+    line_types(str, 'PLPLP')
+  end
+
+  def test_groups
+    str = "now is the time"
+    line_groups(str, ["L0: Paragraph\nnow is the time"] )
+
+    str = "now is the time\nfor all good men"
+    line_groups(str, ["L0: Paragraph\nnow is the time for all good men"] )
+
+    str = %{\
+      now is the time
+        code _line_ here
+      for all good men}
+
+    line_groups(str,
+                [ "L0: Paragraph\nnow is the time",
+                  "L0: Verbatim\n  code _line_ here\n",
+                  "L0: Paragraph\nfor all good men"
+                ] )
+
+    str = "now is the time\n  code\n more code\nfor all good men"
+    line_groups(str,
+                [ "L0: Paragraph\nnow is the time",
+                  "L0: Verbatim\n  code\n more code\n",
+                  "L0: Paragraph\nfor all good men"
+                ] )
+
+    str = %{\
+       now is
+       * l1
+       * l2
+       the time}
+    line_groups(str,
+                [ "L0: Paragraph\nnow is",
+                  "L1: ListStart\n",
+                  "L1: ListItem\nl1",
+                  "L1: ListItem\nl2",
+                  "L1: ListEnd\n",
+                  "L0: Paragraph\nthe time"
+                ])
+
+    str = %{\
+       now is
+       * l1
+         l1+
+       * l2
+       the time}
+    line_groups(str,
+                [ "L0: Paragraph\nnow is",
+                  "L1: ListStart\n",
+                  "L1: ListItem\nl1 l1+",
+                  "L1: ListItem\nl2",
+                  "L1: ListEnd\n",
+                  "L0: Paragraph\nthe time"
+                ])
+
+    str = %{\
+       now is
+       * l1
+         * l1.1
+       * l2
+       the time}
+    line_groups(str,
+                [ "L0: Paragraph\nnow is",
+                  "L1: ListStart\n",
+                  "L1: ListItem\nl1",
+                  "L2: ListStart\n",
+                  "L2: ListItem\nl1.1",
+                  "L2: ListEnd\n",
+                  "L1: ListItem\nl2",
+                  "L1: ListEnd\n",
+                  "L0: Paragraph\nthe time"
+                ])
+
+
+    str = %{\
+       now is
+       * l1
+         * l1.1
+           text
+             code
+               code
+
+           text
+       * l2
+       the time}
+    line_groups(str,
+                [ "L0: Paragraph\nnow is",
+                  "L1: ListStart\n",
+                  "L1: ListItem\nl1",
+                  "L2: ListStart\n",
+                  "L2: ListItem\nl1.1 text",
+                  "L2: Verbatim\n  code\n    code\n",
+                  "L2: Paragraph\ntext",
+                  "L2: ListEnd\n",
+                  "L1: ListItem\nl2",
+                  "L1: ListEnd\n",
+                  "L0: Paragraph\nthe time"
+                ])
+
+
+    str = %{\
+       now is
+       1. l1
+          * l1.1
+       2. l2
+       the time}
+    line_groups(str,
+                [ "L0: Paragraph\nnow is",
+                  "L1: ListStart\n",
+                  "L1: ListItem\nl1",
+                  "L2: ListStart\n",
+                  "L2: ListItem\nl1.1",
+                  "L2: ListEnd\n",
+                  "L1: ListItem\nl2",
+                  "L1: ListEnd\n",
+                  "L0: Paragraph\nthe time"
+                ])
+
+    str = %{\
+       now is
+       [cat] l1
+             * l1.1
+       [dog] l2
+       the time}
+    line_groups(str,
+                [ "L0: Paragraph\nnow is",
+                  "L1: ListStart\n",
+                  "L1: ListItem\nl1",
+                  "L2: ListStart\n",
+                  "L2: ListItem\nl1.1",
+                  "L2: ListEnd\n",
+                  "L1: ListItem\nl2",
+                  "L1: ListEnd\n",
+                  "L0: Paragraph\nthe time"
+                ])
+
+    str = %{\
+       now is
+       [cat] l1
+             continuation
+       [dog] l2
+       the time}
+    line_groups(str,
+                [ "L0: Paragraph\nnow is",
+                  "L1: ListStart\n",
+                  "L1: ListItem\nl1 continuation",
+                  "L1: ListItem\nl2",
+                  "L1: ListEnd\n",
+                  "L0: Paragraph\nthe time"
+                ])
+
+    
+  end
+
+  def test_verbatim_merge
+    str = %{\
+       now is
+          code
+       the time}
+
+    line_groups(str,
+                [ "L0: Paragraph\nnow is",
+                  "L0: Verbatim\n   code\n",
+                  "L0: Paragraph\nthe time"
+                ])
+
+
+    str = %{\
+       now is
+          code
+          code1
+       the time}
+
+    line_groups(str,
+                [ "L0: Paragraph\nnow is",
+                  "L0: Verbatim\n   code\n   code1\n",
+                  "L0: Paragraph\nthe time"
+                ])
+
+
+    str = %{\
+       now is
+          code
+
+          code1
+       the time}
+
+    line_groups(str,
+                [ "L0: Paragraph\nnow is",
+                  "L0: Verbatim\n   code\n\n   code1\n",
+                  "L0: Paragraph\nthe time"
+                ])
+
+
+    str = %{\
+       now is
+          code
+
+          code1
+
+       the time}
+
+    line_groups(str,
+                [ "L0: Paragraph\nnow is",
+                  "L0: Verbatim\n   code\n\n   code1\n",
+                  "L0: Paragraph\nthe time"
+                ])
+
+
+    str = %{\
+       now is
+          code
+
+          code1
+
+          code2
+       the time}
+
+    line_groups(str,
+                [ "L0: Paragraph\nnow is",
+                  "L0: Verbatim\n   code\n\n   code1\n\n   code2\n",
+                  "L0: Paragraph\nthe time"
+                ])
+
+
+    # Folds multiple blank lines
+    str = %{\
+       now is
+          code
+
+
+          code1
+
+       the time}
+
+    line_groups(str,
+                [ "L0: Paragraph\nnow is",
+                  "L0: Verbatim\n   code\n\n   code1\n",
+                  "L0: Paragraph\nthe time"
+                ])
+
+
+  end
+ 
+  def test_list_split
+    str = %{\
+       now is
+       * l1
+       1. n1
+       2. n2
+       * l2
+       the time}
+    line_groups(str,
+                [ "L0: Paragraph\nnow is",
+                  "L1: ListStart\n",
+                  "L1: ListItem\nl1",
+                  "L1: ListEnd\n",
+                  "L1: ListStart\n",
+                  "L1: ListItem\nn1",
+                  "L1: ListItem\nn2",
+                  "L1: ListEnd\n",
+                  "L1: ListStart\n",
+                  "L1: ListItem\nl2",
+                  "L1: ListEnd\n",
+                  "L0: Paragraph\nthe time"
+                ])
+
+  end
+
+
+  def test_headings
+    str = "= heading one"
+    line_groups(str, 
+                [ "L0: Heading\nheading one"
+                ])
+
+    str = "=== heading three"
+    line_groups(str, 
+                [ "L0: Heading\nheading three"
+                ])
+
+    str = "text\n   === heading three"
+    line_groups(str, 
+                [ "L0: Paragraph\ntext",
+                  "L0: Verbatim\n   === heading three\n"
+                ])
+
+    str = "text\n   code\n   === heading three"
+    line_groups(str, 
+                [ "L0: Paragraph\ntext",
+                  "L0: Verbatim\n   code\n   === heading three\n"
+                ])
+
+    str = "text\n   code\n=== heading three"
+    line_groups(str, 
+                [ "L0: Paragraph\ntext",
+                  "L0: Verbatim\n   code\n",
+                  "L0: Heading\nheading three"
+                ])
+
+  end
+
+  
+end
diff --git a/lib/rdoc/options.rb b/lib/rdoc/options.rb
new file mode 100644
index 0000000000..fe75b10ce8
--- /dev/null
+++ b/lib/rdoc/options.rb
@@ -0,0 +1,526 @@
+# We handle the parsing of options, and subsequently as a singleton
+# object to be queried for option values
+
+class Options
+
+  require 'singleton'
+  require 'getoptlong'
+
+  include Singleton
+
+  # files matching this pattern will be excluded
+  attr_accessor :exclude
+
+  # the name of the output directory
+  attr_accessor :op_dir
+  
+  # the name to use for the output
+  attr_reader :op_name
+
+  # include private and protected methods in the
+  # output
+  attr_accessor :show_all
+  
+  # name of the file, class or module to display in
+  # the initial index page (if not specified
+  # the first file we encounter is used)
+  attr_accessor :main_page
+
+  # Don't display progress as we process the files
+  attr_reader :quiet
+
+  # description of the output generator (set with the <tt>-fmt</tt>
+  # option
+  attr_accessor :generator
+
+  # and the list of files to be processed
+  attr_reader :files
+
+  # array of directories to search for files to satisfy an :include:
+  attr_reader :rdoc_include
+
+  # title to be used out the output
+  #attr_writer :title
+
+  # template to be used when generating output
+  attr_reader :template
+
+  # should diagrams be drawn
+  attr_reader :diagram
+
+  # should we draw fileboxes in diagrams
+  attr_reader :fileboxes
+
+  # include the '#' at the front of hyperlinked instance method names
+  attr_reader :show_hash
+
+  # image format for diagrams
+  attr_reader :image_format
+
+  # character-set
+  attr_reader :charset
+
+  # should source code be included inline, or displayed in a popup
+  attr_reader :inline_source
+
+  # should the output be placed into a single file
+  attr_reader :all_one_file
+
+  # the number of columns in a tab
+  attr_reader :tab_width
+
+  # include line numbers in the source listings
+  attr_reader :include_line_numbers
+
+  # pattern for additional attr_... style methods
+  attr_reader :extra_accessors
+  attr_reader :extra_accessor_flags
+
+  # URL of stylesheet
+  attr_reader :css
+
+  # URL of web cvs frontend
+  attr_reader :webcvs
+
+  # Are we promiscuous about showing module contents across
+  # multiple files
+  attr_reader :promiscuous
+
+  module OptionList
+
+    OPTION_LIST = [
+      [ "--accessor",      "-A",   "accessorname[,..]",
+        "comma separated list of additional class methods\n" +
+        "that should be treated like 'attr_reader' and\n" +
+        "friends. Option may be repeated. Each accessorname\n" +
+        "may have '=text' appended, in which case that text\n" +
+        "appears where the r/w/rw appears for normal accessors."],
+                                                                   
+      [ "--all",           "-a",   nil,
+        "include all methods (not just public)\nin the output" ],
+
+      [ "--charset",       "-c",   "charset",
+        "specifies HTML character-set" ],
+
+      [ "--debug",         "-D",   nil,
+        "displays lots on internal stuff" ],
+
+      [ "--diagram",       "-d",   nil,
+        "Generate diagrams showing modules and classes.\n" +
+        "You need dot V1.8.6 or later to use the --diagram\n" +
+        "option correctly. Dot is available from\n"+
+        "http://www.research.att.com/sw/tools/graphviz/" ],
+
+      [ "--exclude",       "-x",   "pattern",
+        "do not process files or directories matching\n" +
+        "pattern. Files given explicitly on the command\n" +
+        "line will never be excluded." ],
+
+      [ "--extension",     "-E",   "new=old",
+        "Treat files ending with .new as if they ended with\n" +
+        ".old. Using '-E cgi=rb' will cause xxx.cgi to be\n" +
+        "parsed as a Ruby file"],
+
+      [ "--fileboxes",     "-F",   nil,
+        "classes are put in boxes which represents\n" +
+        "files, where these classes reside. Classes\n" +
+        "shared between more than one file are\n" +
+        "shown with list of files that sharing them.\n" +
+        "Silently discarded if --diagram is not given\n" +
+        "Experimental." ],
+
+      [ "--fmt",           "-f",   "format name",
+        "set the output formatter (see below)" ],
+
+      [ "--help",          "-h",   nil,
+        "you're looking at it" ],
+
+      [ "--help-output",   "-O",   nil,
+        "explain the various output options" ],
+
+      [ "--image-format",  "-I",   "gif/png/jpg/jpeg",
+        "Sets output image format for diagrams. Can\n" +
+        "be png, gif, jpeg, jpg. If this option is\n" +
+        "omitted, png is used. Requires --diagram." ],
+
+      [ "--include",       "-i",   "dir[,dir...]",
+        "set (or add to) the list of directories\n" +
+        "to be searched when satisfying :include:\n" +
+        "requests. Can be used more than once." ],
+
+      [ "--inline-source", "-S",   nil,
+        "Show method source code inline, rather\n" +
+        "than via a popup link" ],
+
+      [ "--line-numbers", "-N", nil,
+        "Include line numbers in the source code" ],
+
+      [ "--main",          "-m",   "name",
+        "'name' will be the initial page displayed" ],
+
+      [ "--one-file",      "-1",   nil,
+        "put all the output into a single file" ],
+
+      [ "--op",            "-o",   "dir",
+        "set the output directory" ],
+
+      [ "--opname",       "-n",    "name",
+        "Set the 'name' of the output. Has no\n" +
+        "effect for HTML." ],
+
+      [ "--promiscuous",   "-p",   nil,
+        "When documenting a file that contains a module\n" +
+        "or class also defined in other files, show\n" +
+        "all stuff for that module/class in each files\n" +
+        "page. By default, only show stuff defined in\n" +
+        "that particular file." ],
+
+      [ "--quiet",         "-q",   nil,
+        "don't show progress as we parse" ],
+
+      [ "--show-hash",     "-H",   nil,
+        "A name of the form #name in a comment\n" +
+        "is a possible hyperlink to an instance\n" +
+        "method name. When displayed, the '#' is\n" +
+        "removed unless this option is specified" ],
+
+      [ "--style",         "-s",   "stylesheet url",
+        "specifies the URL of a separate stylesheet." ],
+
+      [ "--tab-width",     "-w",   "n",
+        "Set the width of tab characters (default 8)"],
+
+      [ "--template",      "-T",   "template name",
+        "Set the template used when generating output" ],
+
+      [ "--title",         "-t",   "text",
+        "Set 'txt' as the title for the output" ],
+
+      [ "--version",       "-v",   nil,
+        "display  RDoc's version" ],
+
+      [ "--webcvs",        "-W",   "url",
+        "Specify a URL for linking to a web frontend\n" +
+        "to CVS. If the URL contains a '\%s', the\n" +
+        "name of the current file will be substituted;\n" +
+        "if the URL doesn't contain a '\%s', the\n" +
+        "filename will be appended to it." ],
+    ]
+
+    def OptionList.options
+      OPTION_LIST.map do |long, short, arg,|
+        [ long, 
+          short, 
+          arg ? GetoptLong::REQUIRED_ARGUMENT : GetoptLong::NO_ARGUMENT 
+        ]
+      end
+    end
+
+
+    def OptionList.strip_output(text)
+      text =~ /^\s+/
+      leading_spaces = $&
+      text.gsub!(/^#{leading_spaces}/, '')
+      $stdout.puts text
+    end
+
+
+    # Show an error and exit
+
+    def OptionList.error(msg)
+      $stderr.puts
+      $stderr.puts msg
+      $stderr.puts "\nFor help on options, try 'rdoc --help'\n\n"
+      exit 1
+    end
+
+    # Show usage and exit
+    
+    def OptionList.usage(generator_names)
+      
+      puts
+      puts(VERSION_STRING)
+      puts
+
+      name = File.basename($0)
+      OptionList.strip_output(<<-EOT)
+          Usage:
+
+            #{name} [options]  [names...]
+
+          Files are parsed, and the information they contain
+          collected, before any output is produced. This allows cross
+          references between all files to be resolved. If a name is a
+          directory, it is traversed. If no names are specified, all
+          Ruby files in the current directory (and subdirectories) are
+          processed.
+
+          Options:
+
+      EOT
+
+      OPTION_LIST.each do |long, short, arg, desc|
+        opt = sprintf("%20s", "#{long}, #{short}")
+        oparg = sprintf("%-7s", arg)
+        print "#{opt} #{oparg}"
+        desc = desc.split("\n")
+        if arg.nil? || arg.length < 7
+          puts desc.shift
+        else
+          puts
+        end
+        desc.each do |line|
+          puts(" "*28 + line)
+        end
+        puts
+      end
+
+      puts "\nAvailable output formatters: " +
+        generator_names.sort.join(', ') + "\n\n"
+
+      puts "For information on where the output goes, use\n\n"
+      puts "   rdoc --help-output\n\n"
+
+      exit 0
+    end
+
+    def OptionList.help_output
+      OptionList.strip_output(<<-EOT)
+      How RDoc generates output depends on the output formatter being
+      used, and on the options you give.
+
+      - HTML output is normally produced into a number of separate files
+        (one per class, module, and file, along with various indices). 
+        These files will appear in the directory given by the --op
+        option (doc/ by default).
+
+      - XML output by default is written to standard output. If a
+        --opname option is given, the output will instead be written
+        to a file with that name in the output directory.
+
+      - .chm files (Windows help files) are written in the --op directory.
+        If an --opname parameter is present, that name is used, otherwise
+        the file will be called rdoc.chm.
+
+      For information on other RDoc options, use "rdoc --help".
+      EOT
+      exit 0
+    end
+  end
+
+  # Parse command line options. We're passed a hash containing
+  # output generators, keyed by the generator name
+
+  def parse(argv, generators)
+    old_argv = ARGV.dup
+    begin
+      ARGV.replace(argv)
+      @op_dir = "doc"
+      @op_name = nil
+      @show_all = false
+      @main_page = nil
+      @exclude   = nil
+      @quiet = false
+      @generator_name = 'html'
+      @generator = generators[@generator_name]
+      @rdoc_include = []
+      @title = nil
+      @template = nil
+      @diagram = false
+      @fileboxes = false
+      @show_hash = false
+      @image_format = 'png'
+      @inline_source = false
+      @all_one_file  = false
+      @tab_width = 8
+      @include_line_numbers = false
+      @extra_accessor_flags = {}
+      @promiscuous = false
+
+      @css = nil
+      @webcvs = nil
+
+      @charset = case $KCODE
+                 when /^S/i
+                   'Shift_JIS'
+                 when /^E/i
+                   'EUC-JP'
+                 else
+                   'iso-8859-1'
+                 end
+
+      accessors = []
+
+      go = GetoptLong.new(*OptionList.options)
+      go.quiet = true
+
+      go.each do |opt, arg|
+	case opt
+        when "--all"           then @show_all      = true
+        when "--charset"       then @charset       = arg
+        when "--debug"         then $DEBUG         = true
+        when "--exclude"       then @exclude       = Regexp.new(arg)
+        when "--inline-source" then @inline_source = true
+        when "--line-numbers"  then @include_line_numbers = true
+        when "--main"          then @main_page     = arg
+        when "--one-file"      then @all_one_file  = true
+        when "--op"            then @op_dir        = arg
+        when "--opname"        then @op_name       = arg
+        when "--promiscuous"   then @promiscuous   = true
+        when "--quiet"         then @quiet         = true
+        when "--show-hash"     then @show_hash     = true
+        when "--style"         then @css           = arg
+        when "--template"      then @template      = arg
+        when "--title"         then @title         = arg
+        when "--webcvs"        then @webcvs        = arg
+
+        when "--accessor" 
+          arg.split(/,/).each do |accessor|
+            if accessor =~ /^(\w+)(=(.*))?$/
+              accessors << $1
+              @extra_accessor_flags[$1] = $3
+            end
+          end
+
+        when "--diagram"
+          check_diagram
+          @diagram = true
+
+        when "--fileboxes"
+          @fileboxes = true if @diagram
+
+	when "--fmt"
+          @generator_name = arg.downcase
+	  @generator = generators[@generator_name]
+          if !@generator
+            OptionList.error("Invalid output formatter")
+          end
+
+          if @generator_name == "xml"
+            @all_one_file = true
+            @inline_source = true
+          end
+
+        when "--help"      
+          OptionList.usage(generators.keys)
+
+        when "--help-output"      
+          OptionList.help_output
+
+        when "--image-format"
+          if ['gif', 'png', 'jpeg', 'jpg'].include?(arg)
+            @image_format = arg
+          else
+            raise GetoptLong::InvalidOption.new("unknown image format: #{arg}")
+          end
+
+        when "--include"   
+          @rdoc_include.concat arg.split(/\s*,\s*/)
+
+        when "--tab-width"
+          begin
+            @tab_width     = Integer(arg)
+          rescue 
+            $stderr.puts "Invalid tab width: '#{arg}'"
+            exit 1
+          end
+
+        when "--extension"
+          new, old = arg.split(/=/, 2)
+          OptionList.error("Invalid parameter to '-E'") unless new && old
+          unless RDoc::ParserFactory.alias_extension(old, new)
+            OptionList.error("Unknown extension .#{old} to -E")
+          end
+
+	when "--version"
+	  puts VERSION_STRING
+	  exit
+	end
+
+      end
+
+      @files = ARGV.dup
+
+      @rdoc_include << "." if @rdoc_include.empty?
+
+      check_files
+
+      # If no template was specified, use the default
+      # template for the output formatter
+
+      @template ||= @generator_name
+
+      # Generate a regexp from the accessors
+      unless accessors.empty?
+        re = '^(' + accessors.map{|a| Regexp.quote(a)}.join('|') + ')$' 
+        @extra_accessors = Regexp.new(re)
+      end
+
+    rescue GetoptLong::InvalidOption, GetoptLong::MissingArgument => error
+      OptionList.error(error.message)
+
+    ensure
+      ARGV.replace(old_argv)
+    end
+  end
+
+
+  def title
+    @title ||= "RDoc Documentation"
+  end
+  
+  # Set the title, but only if not already set. This means that a title set from 
+  # the command line trumps one set in a source file
+
+  def title=(string)
+    @title ||= string
+  end
+
+
+  private
+
+  # Check that the right version of 'dot' is available.
+  # Unfortuately this doesn't work correctly under Windows NT, 
+  # so we'll bypass the test under Windows
+
+  def check_diagram
+    return if RUBY_PLATFORM =~ /win/
+
+    ok = false
+    ver = nil
+    IO.popen("dot -V 2>&1") do |io|
+      ver = io.read
+      if ver =~ /dot\s+version(?:\s+gviz)?\s+(\d+)\.(\d+)/
+        ok = ($1.to_i > 1) || ($1.to_i == 1 && $2.to_i >= 8)
+      end
+    end
+    unless ok
+      if ver =~ /^dot version/
+        $stderr.puts "Warning: You may need dot V1.8.6 or later to use\n",
+          "the --diagram option correctly. You have:\n\n   ",
+          ver,
+          "\nDiagrams might have strange background colors.\n\n"
+      else
+        $stderr.puts "You need the 'dot' program to produce diagrams.",
+          "(see http://www.research.att.com/sw/tools/graphviz/)\n\n"
+        exit
+      end
+#      exit
+    end
+  end
+  
+  # Check that the files on the command line exist
+  
+  def check_files
+    @files.each do |f|
+      stat = File.stat f rescue error("File not found: #{f}")
+      error("File '#{f}' not readable") unless stat.readable?
+    end
+  end
+
+  def error(str)
+    $stderr.puts str
+    exit(1)
+  end
+
+end
diff --git a/lib/rdoc/parsers/parse_c.rb b/lib/rdoc/parsers/parse_c.rb
new file mode 100644
index 0000000000..3d9d7a247f
--- /dev/null
+++ b/lib/rdoc/parsers/parse_c.rb
@@ -0,0 +1,287 @@
+  # We attempt to parse C extension files. Basically we look for
+  # the standard patterns that you find in extensions: <tt>rb_define_class,
+  # rb_define_method</tt> and so on. We also try to find the corresponding
+  # C source for the methods and extract comments, but if we fail
+  # we don't worry too much.
+  #
+  # The comments associated with a Ruby method are extracted from the C
+  # comment block associated with the routine that _implements_ that
+  # method, that is to say the method whose name is given in the
+  # <tt>rb_define_method</tt> call. For example, you might write:
+  #
+  #  /*
+  #   * Returns a new array that is a one-dimensional flattening of this
+  #   * array (recursively). That is, for every element that is an array,
+  #   * extract its elements into the new array.
+  #   *
+  #   *    s = [ 1, 2, 3 ]           #=> [1, 2, 3]
+  #   *    t = [ 4, 5, 6, [7, 8] ]   #=> [4, 5, 6, [7, 8]]
+  #   *    a = [ s, t, 9, 10 ]       #=> [[1, 2, 3], [4, 5, 6, [7, 8]], 9, 10]
+  #   *    a.flatten                 #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
+  #   */
+  #   static VALUE
+  #   rb_ary_flatten(ary)
+  #       VALUE ary;
+  #   {
+  #       ary = rb_obj_dup(ary);
+  #       rb_ary_flatten_bang(ary);
+  #       return ary;
+  #   }
+  #
+  #   ...
+  #
+  #   void
+  #   Init_Array()
+  #   {
+  #     ...
+  #     rb_define_method(rb_cArray, "flatten", rb_ary_flatten, 0);
+  #
+  # Here RDoc will determine from the rb_define_method line that there's a
+  # method called "flatten" in class Array, and will look for the implementation
+  # in the method rb_ary_flatten. It will then use the comment from that
+  # method in the HTML output. This method must be in the same source file
+  # as the rb_define_method.
+  #
+  # C classes can be diagramed (see /tc/dl/ruby/ruby/error.c), and RDoc
+  # integrates C and Ruby source into one tree
+
+  # Classes and modules built in to the interpreter. We need
+  # these to define superclasses of user objects
+
+require "rdoc/code_objects"
+require "rdoc/parsers/parserfactory"
+
+
+module RDoc
+
+  KNOWN_CLASSES = {
+    "rb_cObject"           => "Object",
+    "rb_cArray"            => "Array",
+    "rb_cBignum"           => "Bignum",
+    "rb_cClass"            => "Class",
+    "rb_cDir"              => "Dir",
+    "rb_cData"             => "Data",
+    "rb_cFalseClass"       => "FalseClass",
+    "rb_cFile"             => "File",
+    "rb_cFixnum"           => "Fixnum",
+    "rb_cFloat"            => "Float",
+    "rb_cHash"             => "Hash",
+    "rb_cInteger"          => "Integer",
+    "rb_cIO"               => "IO",
+    "rb_cModule"           => "Module",
+    "rb_cNilClass"         => "NilClass",
+    "rb_cNumeric"          => "Numeric",
+    "rb_cProc"             => "Proc",
+    "rb_cRange"            => "Range",
+    "rb_cRegexp"           => "Regexp",
+    "rb_cString"           => "String",
+    "rb_cSymbol"           => "Symbol",
+    "rb_cThread"           => "Thread",
+    "rb_cTime"             => "Time",
+    "rb_cTrueClass"        => "TrueClass",
+    "rb_cStruct"           => "Struct",
+    "rb_eException"        => "Exception",
+    "rb_eStandardError"    => "StandardError",
+    "rb_eSystemExit"       => "SystemExit",
+    "rb_eInterrupt"        => "Interrupt",
+    "rb_eSignal"           => "Signal",
+    "rb_eFatal"            => "Fatal",
+    "rb_eArgError"         => "ArgError",
+    "rb_eEOFError"         => "EOFError",
+    "rb_eIndexError"       => "IndexError",
+    "rb_eRangeError"       => "RangeError",
+    "rb_eIOError"          => "IOError",
+    "rb_eRuntimeError"     => "RuntimeError",
+    "rb_eSecurityError"    => "SecurityError",
+    "rb_eSystemCallError"  => "SystemCallError",
+    "rb_eTypeError"        => "TypeError",
+    "rb_eZeroDivError"     => "ZeroDivError",
+    "rb_eNotImpError"      => "NotImpError",
+    "rb_eNoMemError"       => "NoMemError",
+    "rb_eFloatDomainError" => "FloatDomainError",
+    "rb_eScriptError"      => "ScriptError",
+    "rb_eNameError"        => "NameError",
+    "rb_eSyntaxError"      => "SyntaxError",
+    "rb_eLoadError"        => "LoadError",
+
+    "rb_mKernel"           => "Kernel",
+    "rb_mComparable"       => "Comparable",
+    "rb_mEnumerable"       => "Enumerable",
+    "rb_mPrecision"        => "Precision",
+    "rb_mErrno"            => "Errno",
+    "rb_mFileTest"         => "FileTest",
+    "rb_mGC"               => "GC",
+    "rb_mMath"             => "Math",
+    "rb_mProcess"          => "Process"
+
+  }
+
+  # See rdoc/c_parse.rb
+
+  class C_Parser
+
+
+    extend ParserFactory
+    parse_files_matching(/\.(c|cc|cpp|CC)$/)
+
+
+    # prepare to parse a C file
+    def initialize(top_level, file_name, body, options)
+      @known_classes = KNOWN_CLASSES.dup
+      @body = body
+      @options = options
+      @top_level = top_level
+      @classes = Hash.new
+    end
+
+    # Extract the classes/modules and methods from a C file
+    # and return the corresponding top-level object
+    def scan
+      remove_commented_out_lines
+      do_classes
+      do_methods
+      @top_level
+    end
+
+    #######
+    private
+    #######
+
+    # remove lines that are commented out that might otherwise get
+    # picked up when scanning for classes and methods
+
+    def remove_commented_out_lines
+      @body.gsub!(%r{//.*rb_define_}, '//')
+    end
+
+    def handle_class_module(var_name, class_mod, class_name, parent, in_module)
+      @known_classes[var_name] = class_name
+      parent_name = @known_classes[parent] || parent
+
+      if in_module
+        enclosure = @classes[in_module]
+        unless enclosure
+          $stderr.puts("Enclosing class/module '#{in_module}' for " +
+                        class_mod + " #{class_name} not known")
+          return
+        end
+      else
+        enclosure = @top_level
+      end
+      
+      if class_mod == "class" 
+        cm = enclosure.add_class(NormalClass, class_name, parent_name)
+      else
+        cm = enclosure.add_module(NormalModule, class_name)
+      end
+      cm.record_location(enclosure.toplevel)
+      @classes[var_name] = cm
+    end
+    
+    
+    
+    def do_classes
+      @body.scan(/(\w+)\s* = \s*rb_define_module\(\s*"(\w+)"\s*\)/mx) do 
+        |var_name, class_name|
+        handle_class_module(var_name, "module", class_name, nil, nil)
+      end
+      
+      @body.scan(/(\w+)\s* = \s*rb_define_module_under
+                \( 
+                   \s*(\w+),
+                   \s*"(\w+)"
+                \)/mx) do 
+        
+        |var_name, in_module, class_name|
+        handle_class_module(var_name, "module", class_name, nil, in_module)
+      end
+      
+      @body.scan(/(\w+)\s* = \s*rb_define_class
+                \( 
+                   \s*"(\w+)",
+                   \s*(\w+)\s*
+                \)/mx) do 
+        
+        |var_name, class_name, parent|
+        handle_class_module(var_name, "class", class_name, parent, nil)
+      end
+      
+      @body.scan(/(\w+)\s* = \s*rb_define_class_under
+                \( 
+                   \s*(\w+),
+                   \s*"(\w+)",
+                   \s*(\w+)\s*
+                \)/mx) do 
+        
+        |var_name, in_module, class_name, parent|
+        handle_class_module(var_name, "class", class_name, parent, in_module)
+      end
+      
+    end
+    
+    
+    def do_methods
+      @body.scan(/rb_define_(singleton_method|method|module_function)\(\s*(\w+),
+                               \s*"([^"]+)",
+                               \s*(?:RUBY_METHOD_FUNC\(|VALUEFUNC\()?(\w+)\)?,
+                               \s*(-?\w+)\s*\)/xm) do 
+        |type, var_name, meth_name, meth_body, param_count|
+        
+        class_name = @known_classes[var_name] || var_name
+        class_obj  = @classes[var_name]
+        if class_obj
+          meth_obj = AnyMethod.new("", meth_name)
+          meth_obj.singleton = type == "singleton_method"
+          
+          p_count = (Integer(param_count) rescue -1)
+          
+          if p_count < 0
+            meth_obj.params = "(...)"
+          elsif p_count == 0
+            meth_obj.params = "()"
+          else
+            meth_obj.params = "(" +
+              (1..p_count).map{|i| "p#{i}"}.join(", ") + 
+              ")"
+          end
+          
+          find_body(meth_body, meth_obj)
+          class_obj.add_method(meth_obj)
+        end
+      end
+    end
+    
+    # Find the C code corresponding to a c method
+    def find_body(meth_name, meth_obj)
+      if @body =~ %r{((?>/\*.*?\*/\s+))(static\s+)?VALUE\s+#{meth_name}
+                    \s*(\(.*?\)).*?^}xm
+        comment, params = $1, $3
+        body_text = $&
+
+        # see if we can find the whole body
+        
+        re = Regexp.escape(body_text) + "[^(]*^{.*?^}"
+        if Regexp.new(re, Regexp::MULTILINE).match(@body)
+          body_text = $&
+        end
+
+        meth_obj.params = params
+        meth_obj.start_collecting_tokens
+        meth_obj.add_token(RubyToken::Token.new(1,1).set_text(body_text))
+        meth_obj.comment = mangle_comment(comment)
+        
+      end
+    end
+    
+    # Remove the /*'s and leading asterisks from C comments
+    
+    def mangle_comment(comment)
+      comment.sub!(%r{/\*+}) { " " * $&.length }
+      comment.sub!(%r{\*+/}) { " " * $&.length }
+      comment.gsub!(/^[ \t]*\*/m) { " " * $&.length }
+      comment
+    end
+    
+  end
+  
+end
diff --git a/lib/rdoc/parsers/parse_f95.rb b/lib/rdoc/parsers/parse_f95.rb
new file mode 100644
index 0000000000..3adf29d933
--- /dev/null
+++ b/lib/rdoc/parsers/parse_f95.rb
@@ -0,0 +1,118 @@
+# Parse a Fortran 95 file.
+
+require "rdoc/code_objects"
+
+module RDoc
+
+  # See rdoc/parsers/parse_f95.rb
+ 
+  class Token
+
+    NO_TEXT = "??".freeze
+    
+    def initialize(line_no, char_no)
+      @line_no = line_no
+      @char_no = char_no
+      @text    = NO_TEXT
+    end
+    # Because we're used in contexts that expect to return a token,
+    # we set the text string and then return ourselves
+    def set_text(text)
+      @text = text
+      self
+    end
+
+    attr_reader :line_no, :char_no, :text
+
+  end
+
+  class Fortran95parser
+
+    extend ParserFactory
+    parse_files_matching(/\.(f9(0|5)|F)$/)
+    
+    # prepare to parse a Fortran 95 file
+    def initialize(top_level, file_name, body, options)
+      @body = body
+      @options = options
+      @top_level = top_level
+      @progress = $stderr unless options.quiet
+    end
+    
+    # devine code constructs
+    def scan
+
+      # modules and programs
+      if @body =~ /^(module|program)\s+(\w+)/i
+	progress "m"
+	f9x_module = @top_level.add_module NormalClass, $2
+	f9x_module.record_location @top_level
+	first_comment, second_comment = $`.gsub(/^!\s?/,"").split "\n\s*\n"
+	if second_comment
+	  @top_level.comment = first_comment if first_comment
+	  f9x_module.comment = second_comment
+	else
+	  f9x_module.comment = first_comment if first_comment
+	end
+      end
+
+      # use modules
+      remaining_code = @body
+      while remaining_code =~ /^\s*use\s+(\w+)/i
+	remaining_code = $~.post_match
+	progress "."
+	f9x_module.add_include Include.new($1, "") if f9x_module
+      end
+
+      # subroutines
+      remaining_code = @body
+      while remaining_code =~ /^\s*subroutine\s+(\w+)\s*\((.*?)\)/im
+	remaining_code = $~.post_match
+        subroutine = AnyMethod.new("Text", $1)
+	subroutine.singleton = false
+
+        prematchText = $~.pre_match
+        params = $2
+        params.gsub!(/&/,'')
+	subroutine.params = params
+        comment = find_comments prematchText
+	subroutine.comment = comment if comment
+
+        subroutine.start_collecting_tokens
+        remaining_code =~ /^\s*end\s+subroutine/i
+        code = "subroutine #{subroutine.name} (#{subroutine.params})\n"
+        code += $~.pre_match
+        code += "\nend subroutine\n"
+        subroutine.add_token Token.new(1,1).set_text(code)
+        
+        progress "s"
+        f9x_module.add_method subroutine if f9x_module
+      end
+
+      @top_level
+
+    end
+
+    def find_comments text
+      lines = text.split("\n").reverse
+      comment_block = Array.new
+      lines.each do |line|
+	break if line =~ /^\s*\w/
+        comment_block.unshift line.sub(/^!\s?/,"")
+      end
+      nice_lines = comment_block.join("\n").split "\n\s*\n"
+      nice_lines.shift
+      nice_lines.shift
+      nice_lines.shift
+    end
+
+    def progress(char)
+      unless @options.quiet
+        @progress.print(char)
+        @progress.flush
+      end
+    end
+
+  end # class Fortran95parser
+
+end # module RDoc
diff --git a/lib/rdoc/parsers/parse_rb.rb b/lib/rdoc/parsers/parse_rb.rb
new file mode 100644
index 0000000000..4c10bc4461
--- /dev/null
+++ b/lib/rdoc/parsers/parse_rb.rb
@@ -0,0 +1,2494 @@
+#!/usr/local/bin/ruby
+
+# Parse a Ruby source file, building a set of objects
+# representing the modules, classes, methods,
+# requires, and includes we find (these classes
+# are defined in code_objects.rb).
+
+# This file contains stuff stolen outright from:
+#
+#   rtags.rb - 
+#   ruby-lex.rb - ruby lexcal analizer
+#   ruby-token.rb - ruby tokens 
+#   	by Keiju ISHITSUKA (Nippon Rational Inc.)
+#
+
+require "tracer"
+require "e2mmap"
+require "irb/slex"
+
+require "rdoc/code_objects"
+require "rdoc/tokenstream"
+
+require "rdoc/markup/simple_markup/preprocess"
+
+require "rdoc/parsers/parserfactory"
+
+$TOKEN_DEBUG = $DEBUG
+
+# Definitions of all tokens involved in the lexical analysis
+
+module RubyToken
+  EXPR_BEG   = :EXPR_BEG
+  EXPR_MID   = :EXPR_MID
+  EXPR_END   = :EXPR_END
+  EXPR_ARG   = :EXPR_ARG
+  EXPR_FNAME = :EXPR_FNAME
+  EXPR_DOT   = :EXPR_DOT
+  EXPR_CLASS = :EXPR_CLASS
+  
+  class Token
+    NO_TEXT = "??".freeze
+    attr :text
+
+    def initialize(line_no, char_no)
+      @line_no = line_no
+      @char_no = char_no
+      @text    = NO_TEXT
+    end
+
+    # Because we're used in contexts that expect to return a token,
+    # we set the text string and then return ourselves
+    def set_text(text)
+      @text = text
+      self
+    end
+
+    attr_reader :line_no, :char_no, :text
+  end
+
+  class TkNode < Token
+    attr :node
+  end
+
+  class TkId < Token
+    def initialize(line_no, char_no, name)
+      super(line_no, char_no)
+      @name = name
+    end
+    attr :name
+  end
+
+  class TkKW < TkId
+  end
+
+  class TkVal < Token
+    def initialize(line_no, char_no, value = nil)
+      super(line_no, char_no)
+      set_text(value)
+    end
+  end
+
+  class TkOp < Token
+    def name
+      self.class.op_name
+    end
+  end
+
+  class TkOPASGN < TkOp
+    def initialize(line_no, char_no, op)
+      super(line_no, char_no)
+      op = TkReading2Token[op] unless op.kind_of?(Symbol)
+      @op = op
+    end
+    attr :op
+  end
+
+  class TkUnknownChar < Token
+    def initialize(line_no, char_no, id)
+      super(line_no, char_no)
+      @name = char_no.chr
+    end
+    attr :name
+  end
+
+  class TkError < Token
+  end
+
+  def set_token_position(line, char)
+    @prev_line_no = line
+    @prev_char_no = char
+  end
+
+  def Token(token, value = nil)
+    tk = nil
+    case token
+    when String, Symbol
+      source = token.kind_of?(String) ? TkReading2Token : TkSymbol2Token
+      if (tk = source[token]).nil?
+	IRB.fail TkReading2TokenNoKey, token
+      end
+      tk = Token(tk[0], value) 
+    else 
+      tk = if (token.ancestors & [TkId, TkVal, TkOPASGN, TkUnknownChar]).empty?
+             token.new(@prev_line_no, @prev_char_no)
+           else
+             token.new(@prev_line_no, @prev_char_no, value)
+           end
+    end
+    tk
+  end
+
+  TokenDefinitions = [
+    [:TkCLASS,      TkKW,  "class",  EXPR_CLASS],
+    [:TkMODULE,     TkKW,  "module", EXPR_BEG],
+    [:TkDEF,	    TkKW,  "def",    EXPR_FNAME],
+    [:TkUNDEF,      TkKW,  "undef",  EXPR_FNAME],
+    [:TkBEGIN,      TkKW,  "begin",  EXPR_BEG],
+    [:TkRESCUE,     TkKW,  "rescue", EXPR_MID],
+    [:TkENSURE,     TkKW,  "ensure", EXPR_BEG],
+    [:TkEND,	    TkKW,  "end",    EXPR_END],
+    [:TkIF,         TkKW,  "if",     EXPR_BEG, :TkIF_MOD],
+    [:TkUNLESS,     TkKW,  "unless", EXPR_BEG, :TkUNLESS_MOD],
+    [:TkTHEN,	    TkKW,  "then",   EXPR_BEG],
+    [:TkELSIF,      TkKW,  "elsif",  EXPR_BEG],
+    [:TkELSE,	    TkKW,  "else",   EXPR_BEG],
+    [:TkCASE,	    TkKW,  "case",   EXPR_BEG],
+    [:TkWHEN,	    TkKW,  "when",   EXPR_BEG],
+    [:TkWHILE,      TkKW,  "while",  EXPR_BEG, :TkWHILE_MOD],
+    [:TkUNTIL,      TkKW,  "until",  EXPR_BEG, :TkUNTIL_MOD],
+    [:TkFOR,	    TkKW,  "for",    EXPR_BEG],
+    [:TkBREAK,      TkKW,  "break",  EXPR_END],
+    [:TkNEXT,	    TkKW,  "next",   EXPR_END],
+    [:TkREDO,	    TkKW,  "redo",   EXPR_END],
+    [:TkRETRY,      TkKW,  "retry",  EXPR_END],
+    [:TkIN,	    TkKW,  "in",     EXPR_BEG],
+    [:TkDO,	    TkKW,  "do",     EXPR_BEG],
+    [:TkRETURN,     TkKW,  "return", EXPR_MID],
+    [:TkYIELD,      TkKW,  "yield",  EXPR_END],
+    [:TkSUPER,      TkKW,  "super",  EXPR_END],
+    [:TkSELF,	    TkKW,  "self",   EXPR_END],
+    [:TkNIL, 	    TkKW,  "nil",    EXPR_END],
+    [:TkTRUE,	    TkKW,  "true",   EXPR_END],
+    [:TkFALSE,      TkKW,  "false",  EXPR_END],
+    [:TkAND,	    TkKW,  "and",    EXPR_BEG],
+    [:TkOR, 	    TkKW,  "or",     EXPR_BEG],
+    [:TkNOT,	    TkKW,  "not",    EXPR_BEG],
+    [:TkIF_MOD,     TkKW],
+    [:TkUNLESS_MOD, TkKW],
+    [:TkWHILE_MOD,  TkKW],
+    [:TkUNTIL_MOD,  TkKW],
+    [:TkALIAS,      TkKW,  "alias",    EXPR_FNAME],
+    [:TkDEFINED,    TkKW,  "defined?", EXPR_END],
+    [:TklBEGIN,     TkKW,  "BEGIN",    EXPR_END],
+    [:TklEND,	    TkKW,  "END",      EXPR_END],
+    [:Tk__LINE__,   TkKW,  "__LINE__", EXPR_END],
+    [:Tk__FILE__,   TkKW,  "__FILE__", EXPR_END],
+
+    [:TkIDENTIFIER, TkId],
+    [:TkFID,	    TkId],
+    [:TkGVAR,	    TkId],
+    [:TkIVAR,	    TkId],
+    [:TkCONSTANT,   TkId],
+
+    [:TkINTEGER,    TkVal],
+    [:TkFLOAT,      TkVal],
+    [:TkSTRING,     TkVal],
+    [:TkXSTRING,    TkVal],
+    [:TkREGEXP,     TkVal],
+    [:TkCOMMENT,    TkVal],
+
+    [:TkDSTRING,    TkNode],
+    [:TkDXSTRING,   TkNode],
+    [:TkDREGEXP,    TkNode],
+    [:TkNTH_REF,    TkId],
+    [:TkBACK_REF,   TkId],
+
+    [:TkUPLUS,      TkOp,   "+@"],
+    [:TkUMINUS,     TkOp,   "-@"],
+    [:TkPOW,	    TkOp,   "**"],
+    [:TkCMP,	    TkOp,   "<=>"],
+    [:TkEQ,	    TkOp,   "=="],
+    [:TkEQQ,	    TkOp,   "==="],
+    [:TkNEQ,	    TkOp,   "!="],
+    [:TkGEQ,	    TkOp,   ">="],
+    [:TkLEQ,	    TkOp,   "<="],
+    [:TkANDOP,      TkOp,   "&&"],
+    [:TkOROP,	    TkOp,   "||"],
+    [:TkMATCH,      TkOp,   "=~"],
+    [:TkNMATCH,     TkOp,   "!~"],
+    [:TkDOT2,	    TkOp,   ".."],
+    [:TkDOT3,	    TkOp,   "..."],
+    [:TkAREF,	    TkOp,   "[]"],
+    [:TkASET,	    TkOp,   "[]="],
+    [:TkLSHFT,      TkOp,   "<<"],
+    [:TkRSHFT,      TkOp,   ">>"],
+    [:TkCOLON2,     TkOp],
+    [:TkCOLON3,     TkOp],
+#   [:OPASGN,	    TkOp],               # +=, -=  etc. #
+    [:TkASSOC,      TkOp,   "=>"],
+    [:TkQUESTION,   TkOp,   "?"],	 #?
+    [:TkCOLON,      TkOp,   ":"],        #:
+    
+    [:TkfLPAREN],         # func( #
+    [:TkfLBRACK],         # func[ #
+    [:TkfLBRACE],         # func{ #
+    [:TkSTAR],            # *arg
+    [:TkAMPER],           # &arg #
+    [:TkSYMBOL,     TkId],          # :SYMBOL
+    [:TkSYMBEG,     TkId], 
+    [:TkGT,	    TkOp,   ">"],
+    [:TkLT,	    TkOp,   "<"],
+    [:TkPLUS,	    TkOp,   "+"],
+    [:TkMINUS,      TkOp,   "-"],
+    [:TkMULT,	    TkOp,   "*"],
+    [:TkDIV,	    TkOp,   "/"],
+    [:TkMOD,	    TkOp,   "%"],
+    [:TkBITOR,      TkOp,   "|"],
+    [:TkBITXOR,     TkOp,   "^"],
+    [:TkBITAND,     TkOp,   "&"],
+    [:TkBITNOT,     TkOp,   "~"],
+    [:TkNOTOP,      TkOp,   "!"],
+
+    [:TkBACKQUOTE,  TkOp,   "`"],
+
+    [:TkASSIGN,     Token,  "="],
+    [:TkDOT,	    Token,  "."],
+    [:TkLPAREN,     Token,  "("],  #(exp)
+    [:TkLBRACK,     Token,  "["],  #[arry]
+    [:TkLBRACE,     Token,  "{"],  #{hash}
+    [:TkRPAREN,     Token,  ")"],
+    [:TkRBRACK,     Token,  "]"],
+    [:TkRBRACE,     Token,  "}"],
+    [:TkCOMMA,      Token,  ","],
+    [:TkSEMICOLON,  Token,  ";"],
+
+    [:TkRD_COMMENT],
+    [:TkSPACE],
+    [:TkNL],
+    [:TkEND_OF_SCRIPT],
+
+    [:TkBACKSLASH,  TkUnknownChar,  "\\"],
+    [:TkAT,	    TkUnknownChar,  "@"],
+    [:TkDOLLAR,     TkUnknownChar,  "\$"], #"
+  ]
+
+  # {reading => token_class}
+  # {reading => [token_class, *opt]}
+  TkReading2Token = {}
+  TkSymbol2Token = {}
+
+  def RubyToken.def_token(token_n, super_token = Token, reading = nil, *opts)
+    token_n = token_n.id2name unless token_n.kind_of?(String)
+    if RubyToken.const_defined?(token_n)
+      IRB.fail AlreadyDefinedToken, token_n
+    end
+
+    token_c =  Class.new super_token
+    RubyToken.const_set token_n, token_c
+#    token_c.inspect
+ 
+    if reading
+      if TkReading2Token[reading]
+	IRB.fail TkReading2TokenDuplicateError, token_n, reading
+      end
+      if opts.empty?
+	TkReading2Token[reading] = [token_c]
+      else
+	TkReading2Token[reading] = [token_c].concat(opts)
+      end
+    end
+    TkSymbol2Token[token_n.intern] = token_c
+
+    if token_c <= TkOp
+      token_c.class_eval %{
+        def self.op_name; "#{reading}"; end
+      }
+    end
+  end
+
+  for defs in TokenDefinitions
+    def_token(*defs)
+  end
+
+  NEWLINE_TOKEN = TkNL.new(0,0)
+  NEWLINE_TOKEN.set_text("\n")
+
+end
+
+
+
+# Lexical analyzer for Ruby source
+
+class RubyLex
+
+  ######################################################################
+  #
+  # Read an input stream character by character. We allow for unlimited
+  # ungetting of characters just read.
+  #
+  # We simplify the implementation greatly by reading the entire input
+  # into a buffer initially, and then simply traversing it using
+  # pointers.
+  #
+  # We also have to allow for the <i>here document diversion</i>. This
+  # little gem comes about when the lexer encounters a here
+  # document. At this point we effectively need to split the input
+  # stream into two parts: one to read the body of the here document,
+  # the other to read the rest of the input line where the here
+  # document was initially encountered. For example, we might have
+  #
+  #   do_something(<<-A, <<-B)
+  #     stuff
+  #     for
+  #   A
+  #     stuff
+  #     for
+  #   B
+  #
+  # When the lexer encounters the <<A, it reads until the end of the
+  # line, and keeps it around for later. It then reads the body of the
+  # here document.  Once complete, it needs to read the rest of the
+  # original line, but then skip the here document body.
+  #
+  
+  class BufferedReader
+    
+    attr_reader :line_num
+    
+    def initialize(content)
+      if /\t/ =~ content
+        tab_width = Options.instance.tab_width
+        content = content.split(/\n/).map do |line|
+          1 while line.gsub!(/\t+/) { ' ' * (tab_width*$&.length - $`.length % tab_width)}  && $~ #`
+          line
+        end .join("\n")
+      end
+      @content   = content
+      @content << "\n" unless @content[-1,1] == "\n"
+      @size      = @content.size
+      @offset    = 0
+      @hwm       = 0
+      @line_num  = 1
+      @read_back_offset = 0
+      @last_newline = 0
+      @newline_pending = false
+    end
+    
+    def column
+      @offset - @last_newline
+    end
+    
+    def getc
+      return nil if @offset >= @size
+      ch = @content[@offset, 1]
+      
+      @offset += 1
+      @hwm = @offset if @hwm < @offset
+      
+      if @newline_pending
+        @line_num += 1
+        @last_newline = @offset - 1
+        @newline_pending = false
+      end
+      
+      if ch == "\n"
+        @newline_pending = true
+      end
+      ch
+    end
+    
+    def getc_already_read
+      getc
+    end
+    
+    def ungetc(ch)
+      raise "unget past beginning of file" if @offset <= 0
+      @offset -= 1
+      if @content[@offset] == ?\n
+        @newline_pending = false
+      end
+    end
+    
+    def get_read
+      res = @content[@read_back_offset...@offset]
+      @read_back_offset = @offset
+      res
+    end
+    
+    def peek(at)
+      pos = @offset + at
+      if pos >= @size
+        nil
+      else
+        @content[pos, 1]
+      end
+    end
+    
+    def peek_equal(str)
+      @content[@offset, str.length] == str
+    end
+    
+    def divert_read_from(reserve)
+      @content[@offset, 0] = reserve
+      @size      = @content.size
+    end
+  end
+
+  # end of nested class BufferedReader
+
+  extend Exception2MessageMapper
+  def_exception(:AlreadyDefinedToken, "Already defined token(%s)")
+  def_exception(:TkReading2TokenNoKey, "key nothing(key='%s')")
+  def_exception(:TkSymbol2TokenNoKey, "key nothing(key='%s')")
+  def_exception(:TkReading2TokenDuplicateError, 
+		"key duplicate(token_n='%s', key='%s')")
+  def_exception(:SyntaxError, "%s")
+  
+  include RubyToken
+
+  attr_reader :continue
+  attr_reader :lex_state
+
+  def RubyLex.debug?
+    false
+  end
+
+  def initialize(content)
+    lex_init
+
+    @reader = BufferedReader.new(content)
+
+    @exp_line_no = @line_no = 1
+    @base_char_no = 0
+    @indent = 0
+
+    @ltype = nil
+    @quoted = nil
+    @lex_state = EXPR_BEG
+    @space_seen = false
+    
+    @continue = false
+    @line = ""
+
+    @skip_space = false
+    @read_auto_clean_up = false
+    @exception_on_syntax_error = true
+  end
+
+  attr :skip_space, true
+  attr :read_auto_clean_up, true
+  attr :exception_on_syntax_error, true
+
+  attr :indent
+
+  # io functions
+  def line_no
+    @reader.line_num
+  end
+
+  def char_no
+    @reader.column
+  end
+
+  def get_read
+    @reader.get_read
+  end
+
+  def getc
+    @reader.getc
+  end
+
+  def getc_of_rests
+    @reader.getc_already_read
+  end
+
+  def gets
+    c = getc or return
+    l = ""
+    begin
+      l.concat c unless c == "\r"
+      break if c == "\n"
+    end while c = getc
+    l
+  end
+
+
+  def ungetc(c = nil)
+    @reader.ungetc(c)
+  end
+
+  def peek_equal?(str)
+    @reader.peek_equal(str)
+  end
+
+  def peek(i = 0)
+    @reader.peek(i)
+  end
+
+  def lex
+    until (((tk = token).kind_of?(TkNL) || tk.kind_of?(TkEND_OF_SCRIPT)) &&
+	     !@continue or
+	     tk.nil?)
+    end
+    line = get_read
+
+    if line == "" and tk.kind_of?(TkEND_OF_SCRIPT) || tk.nil?
+      nil
+    else
+      line
+    end
+  end
+
+  def token
+    set_token_position(line_no, char_no)
+    begin
+      begin
+	tk = @OP.match(self)
+	@space_seen = tk.kind_of?(TkSPACE)
+      rescue SyntaxError
+	abort if @exception_on_syntax_error
+	tk = TkError.new(line_no, char_no)
+      end
+    end while @skip_space and tk.kind_of?(TkSPACE)
+    if @read_auto_clean_up
+      get_read
+    end
+#   throw :eof unless tk
+    p tk if $DEBUG
+    tk
+  end
+  
+  ENINDENT_CLAUSE = [
+    "case", "class", "def", "do", "for", "if",
+    "module", "unless", "until", "while", "begin" #, "when"
+  ]
+  DEINDENT_CLAUSE = ["end" #, "when"
+  ]
+
+  PERCENT_LTYPE = {
+    "q" => "\'",
+    "Q" => "\"",
+    "x" => "\`",
+    "r" => "\/",
+    "w" => "]"
+  }
+  
+  PERCENT_PAREN = {
+    "{" => "}",
+    "[" => "]",
+    "<" => ">",
+    "(" => ")"
+  }
+
+  Ltype2Token = {
+    "\'" => TkSTRING,
+    "\"" => TkSTRING,
+    "\`" => TkXSTRING,
+    "\/" => TkREGEXP,
+    "]" => TkDSTRING
+  }
+  Ltype2Token.default = TkSTRING
+
+  DLtype2Token = {
+    "\"" => TkDSTRING,
+    "\`" => TkDXSTRING,
+    "\/" => TkDREGEXP,
+  }
+
+  def lex_init()
+    @OP = SLex.new
+    @OP.def_rules("\0", "\004", "\032") do |chars, io|
+      Token(TkEND_OF_SCRIPT).set_text(chars)
+    end
+
+    @OP.def_rules(" ", "\t", "\f", "\r", "\13") do |chars, io|
+      @space_seen = TRUE
+      while (ch = getc) =~ /[ \t\f\r\13]/
+        chars << ch
+      end
+      ungetc
+      Token(TkSPACE).set_text(chars)
+    end
+
+    @OP.def_rule("#") do
+      |op, io|
+      identify_comment
+    end
+
+    @OP.def_rule("=begin", proc{@prev_char_no == 0 && peek(0) =~ /\s/}) do
+      |op, io|
+      str = op
+      @ltype = "="
+
+      begin
+        ch = getc
+        str << ch
+      end until ch == "\n"
+      
+      until peek_equal?("=end") && peek(4) =~ /\s/
+	begin
+          ch = getc
+          str << ch
+        end until ch == "\n"
+      end
+
+      str << "=end"
+      gets
+
+      @ltype = nil
+      Token(TkRD_COMMENT).set_text(str)
+    end
+
+    @OP.def_rule("\n") do
+      print "\\n\n" if RubyLex.debug?
+      case @lex_state
+      when EXPR_BEG, EXPR_FNAME, EXPR_DOT
+	@continue = TRUE
+      else
+	@continue = FALSE
+	@lex_state = EXPR_BEG
+      end
+      Token(TkNL).set_text("\n")
+    end
+
+    @OP.def_rules("*", "**",	
+		  "!", "!=", "!~",
+		  "=", "==", "===", 
+		  "=~", "<=>",	
+		  "<", "<=",
+		  ">", ">=", ">>") do
+      |op, io|
+      @lex_state = EXPR_BEG
+      Token(op).set_text(op)
+    end
+
+    @OP.def_rules("<<") do
+      |op, io|
+      tk = nil
+      if @lex_state != EXPR_END && @lex_state != EXPR_CLASS &&
+	  (@lex_state != EXPR_ARG || @space_seen)
+	c = peek(0)
+	if /[-\w_\"\'\`]/ =~ c
+	  tk = identify_here_document
+	end
+      end
+      if !tk
+        @lex_state = EXPR_BEG
+        tk = Token(op).set_text(op)
+      end
+      tk
+    end
+
+    @OP.def_rules("'", '"') do
+      |op, io|
+      identify_string(op)
+    end
+
+    @OP.def_rules("`") do
+      |op, io|
+      if @lex_state == EXPR_FNAME
+	Token(op).set_text(op)
+      else
+	identify_string(op)
+      end
+    end
+
+    @OP.def_rules('?') do
+      |op, io|
+      if @lex_state == EXPR_END
+	@lex_state = EXPR_BEG
+	Token(TkQUESTION).set_text(op)
+      else
+	ch = getc
+	if @lex_state == EXPR_ARG && ch !~ /\s/
+	  ungetc
+	  @lex_state = EXPR_BEG;
+	  Token(TkQUESTION).set_text(op)
+	else
+          str = op
+          str << ch
+	  if (ch == '\\') #'
+	    str << read_escape
+	  end
+	  @lex_state = EXPR_END
+	  Token(TkINTEGER).set_text(str)
+	end
+      end
+    end
+
+    @OP.def_rules("&", "&&", "|", "||") do
+      |op, io|
+      @lex_state = EXPR_BEG
+      Token(op).set_text(op)
+    end
+    
+    @OP.def_rules("+=", "-=", "*=", "**=", 
+		  "&=", "|=", "^=", "<<=", ">>=", "||=", "&&=") do
+      |op, io|
+      @lex_state = EXPR_BEG
+      op =~ /^(.*)=$/
+      Token(TkOPASGN, $1).set_text(op)
+    end
+
+    @OP.def_rule("+@", proc{@lex_state == EXPR_FNAME}) do |op, io|
+      Token(TkUPLUS).set_text(op)
+    end
+
+    @OP.def_rule("-@", proc{@lex_state == EXPR_FNAME}) do |op, io|
+      Token(TkUMINUS).set_text(op)
+    end
+
+    @OP.def_rules("+", "-") do
+      |op, io|
+      catch(:RET) do
+	if @lex_state == EXPR_ARG
+	  if @space_seen and peek(0) =~ /[0-9]/
+	    throw :RET, identify_number(op)
+	  else
+	    @lex_state = EXPR_BEG
+	  end
+	elsif @lex_state != EXPR_END and peek(0) =~ /[0-9]/
+	  throw :RET, identify_number(op)
+	else
+	  @lex_state = EXPR_BEG
+	end
+	Token(op).set_text(op)
+      end
+    end
+
+    @OP.def_rule(".") do
+      @lex_state = EXPR_BEG
+      if peek(0) =~ /[0-9]/
+	ungetc
+	identify_number("")
+      else
+	# for obj.if
+	@lex_state = EXPR_DOT
+	Token(TkDOT).set_text(".")
+      end
+    end
+
+    @OP.def_rules("..", "...") do
+      |op, io|
+      @lex_state = EXPR_BEG
+      Token(op).set_text(op)
+    end
+
+    lex_int2
+  end
+  
+  def lex_int2
+    @OP.def_rules("]", "}", ")") do
+      |op, io|
+      @lex_state = EXPR_END
+      @indent -= 1
+      Token(op).set_text(op)
+    end
+
+    @OP.def_rule(":") do
+      if @lex_state == EXPR_END || peek(0) =~ /\s/
+	@lex_state = EXPR_BEG
+	tk = Token(TkCOLON)
+      else
+	@lex_state = EXPR_FNAME;
+	tk = Token(TkSYMBEG)
+      end
+      tk.set_text(":")
+    end
+
+    @OP.def_rule("::") do
+#      p @lex_state.id2name, @space_seen
+      if @lex_state == EXPR_BEG or @lex_state == EXPR_ARG && @space_seen
+	@lex_state = EXPR_BEG
+	tk = Token(TkCOLON3)
+      else
+	@lex_state = EXPR_DOT
+	tk = Token(TkCOLON2)
+      end
+      tk.set_text("::")
+    end
+
+    @OP.def_rule("/") do
+      |op, io|
+      if @lex_state == EXPR_BEG || @lex_state == EXPR_MID
+	identify_string(op)
+      elsif peek(0) == '='
+	getc
+	@lex_state = EXPR_BEG
+	Token(TkOPASGN, :/).set_text("/=") #")
+      elsif @lex_state == EXPR_ARG and @space_seen and peek(0) !~ /\s/
+	identify_string(op)
+      else 
+	@lex_state = EXPR_BEG
+        Token("/").set_text(op)
+      end
+    end
+
+    @OP.def_rules("^") do
+      @lex_state = EXPR_BEG
+      Token("^").set_text("^")
+    end
+
+    #       @OP.def_rules("^=") do
+    # 	@lex_state = EXPR_BEG
+    # 	Token(TkOPASGN, :^)
+    #       end
+    
+    @OP.def_rules(",", ";") do
+      |op, io|
+      @lex_state = EXPR_BEG
+      Token(op).set_text(op)
+    end
+
+    @OP.def_rule("~") do
+      @lex_state = EXPR_BEG
+      Token("~").set_text("~")
+    end
+
+    @OP.def_rule("~@", proc{@lex_state = EXPR_FNAME}) do
+      @lex_state = EXPR_BEG
+      Token("~").set_text("~@")
+    end
+    
+    @OP.def_rule("(") do
+      @indent += 1
+      if @lex_state == EXPR_BEG || @lex_state == EXPR_MID
+	@lex_state = EXPR_BEG
+	tk = Token(TkfLPAREN)
+      else
+	@lex_state = EXPR_BEG
+	tk = Token(TkLPAREN)
+      end
+      tk.set_text("(")
+    end
+
+    @OP.def_rule("[]", proc{@lex_state == EXPR_FNAME}) do
+      Token("[]").set_text("[]")
+    end
+
+    @OP.def_rule("[]=", proc{@lex_state == EXPR_FNAME}) do
+      Token("[]=").set_text("[]=")
+    end
+
+    @OP.def_rule("[") do
+      @indent += 1
+      if @lex_state == EXPR_FNAME
+	t = Token(TkfLBRACK)
+      else
+	if @lex_state == EXPR_BEG || @lex_state == EXPR_MID
+	  t = Token(TkLBRACK)
+	elsif @lex_state == EXPR_ARG && @space_seen
+	  t = Token(TkLBRACK)
+	else
+	  t = Token(TkfLBRACK)
+	end
+	@lex_state = EXPR_BEG
+      end
+      t.set_text("[")
+    end
+
+    @OP.def_rule("{") do
+      @indent += 1
+      if @lex_state != EXPR_END && @lex_state != EXPR_ARG
+	t = Token(TkLBRACE)
+      else
+	t = Token(TkfLBRACE)
+      end
+      @lex_state = EXPR_BEG
+      t.set_text("{")
+    end
+
+    @OP.def_rule('\\') do   #'
+      if getc == "\n" 
+	@space_seen = true
+	@continue = true
+	Token(TkSPACE).set_text("\\\n")
+      else 
+	ungetc
+	Token("\\").set_text("\\")  #"
+      end 
+    end 
+
+    @OP.def_rule('%') do
+      |op, io|
+      if @lex_state == EXPR_BEG || @lex_state == EXPR_MID
+	identify_quotation('%')
+      elsif peek(0) == '='
+	getc
+	Token(TkOPASGN, "%").set_text("%=")
+      elsif @lex_state == EXPR_ARG and @space_seen and peek(0) !~ /\s/
+	identify_quotation('%')
+      else
+	@lex_state = EXPR_BEG
+	Token("%").set_text("%")
+      end
+    end
+
+    @OP.def_rule('$') do  #'
+      identify_gvar
+    end
+
+    @OP.def_rule('@') do
+      if peek(0) =~ /[\w_]/
+	ungetc
+	identify_identifier
+      else
+	Token("@").set_text("@")
+      end
+    end
+
+    #       @OP.def_rule("def", proc{|op, io| /\s/ =~ io.peek(0)}) do 
+    # 	|op, io|
+    # 	@indent += 1
+    # 	@lex_state = EXPR_FNAME
+    # #	@lex_state = EXPR_END
+    # #	until @rests[0] == "\n" or @rests[0] == ";"
+    # #	  rests.shift
+    # #	end
+    #       end
+
+    @OP.def_rule("__END__", proc{@prev_char_no == 0 && peek(0) =~ /[\r\n]/}) do
+      throw :eof
+    end
+
+    @OP.def_rule("") do
+      |op, io|
+      printf "MATCH: start %s: %s\n", op, io.inspect if RubyLex.debug?
+      if peek(0) =~ /[0-9]/
+	t = identify_number("")
+      elsif peek(0) =~ /[\w_]/
+	t = identify_identifier
+      end
+      printf "MATCH: end %s: %s\n", op, io.inspect if RubyLex.debug?
+      t
+    end
+    
+    p @OP if RubyLex.debug?
+  end
+  
+  def identify_gvar
+    @lex_state = EXPR_END
+    str = "$"
+
+    tk = case ch = getc
+         when /[~_*$?!@\/\\;,=:<>".]/   #"
+           str << ch
+           Token(TkGVAR, str)
+           
+         when "-"
+           str << "-" << getc
+           Token(TkGVAR, str)
+           
+         when "&", "`", "'", "+"
+           str << ch
+           Token(TkBACK_REF, str)
+           
+         when /[1-9]/
+           str << ch
+           while (ch = getc) =~ /[0-9]/
+             str << ch
+           end
+           ungetc
+           Token(TkNTH_REF)
+         when /\w/
+           ungetc
+           ungetc
+           return identify_identifier
+         else 
+           ungetc
+           Token("$")     
+         end
+    tk.set_text(str)
+  end
+  
+  def identify_identifier
+    token = ""
+    token.concat getc if peek(0) =~ /[$@]/
+    while (ch = getc) =~ /\w|_/
+      print ":", ch, ":" if RubyLex.debug?
+      token.concat ch
+    end
+    ungetc
+    
+    if ch == "!" or ch == "?"
+      token.concat getc
+    end
+    # fix token
+
+#    puts "identifier - #{token}, state = #@lex_state"
+
+    case token
+    when /^\$/
+      return Token(TkGVAR, token).set_text(token)
+    when /^\@/
+      @lex_state = EXPR_END
+      return Token(TkIVAR, token).set_text(token)
+    end
+    
+    if @lex_state != EXPR_DOT
+      print token, "\n" if RubyLex.debug?
+
+      token_c, *trans = TkReading2Token[token]
+      if token_c
+	# reserved word?
+
+	if (@lex_state != EXPR_BEG &&
+	    @lex_state != EXPR_FNAME &&
+	    trans[1])
+	  # modifiers
+	  token_c = TkSymbol2Token[trans[1]]
+	  @lex_state = trans[0]
+	else
+	  if @lex_state != EXPR_FNAME
+	    if ENINDENT_CLAUSE.include?(token)
+	      @indent += 1
+	    elsif DEINDENT_CLAUSE.include?(token)
+	      @indent -= 1
+	    end
+	    @lex_state = trans[0]
+	  else
+	    @lex_state = EXPR_END
+	  end
+	end
+	return Token(token_c, token).set_text(token)
+      end
+    end
+
+    if @lex_state == EXPR_FNAME
+      @lex_state = EXPR_END
+      if peek(0) == '='
+	token.concat getc
+      end
+    elsif @lex_state == EXPR_BEG || @lex_state == EXPR_DOT
+      @lex_state = EXPR_ARG
+    else
+      @lex_state = EXPR_END
+    end
+
+    if token[0, 1] =~ /[A-Z]/
+      return Token(TkCONSTANT, token).set_text(token)
+    elsif token[token.size - 1, 1] =~ /[!?]/
+      return Token(TkFID, token).set_text(token)
+    else
+      return Token(TkIDENTIFIER, token).set_text(token)
+    end
+  end
+
+  def identify_here_document
+    ch = getc
+    if ch == "-"
+      ch = getc
+      indent = true
+    end
+    if /['"`]/ =~ ch            # '
+      lt = ch
+      quoted = ""
+      while (c = getc) && c != lt
+	quoted.concat c
+      end
+    else
+      lt = '"'
+      quoted = ch.dup
+      while (c = getc) && c =~ /\w/
+	quoted.concat c
+      end
+      ungetc
+    end
+
+    ltback, @ltype = @ltype, lt
+    reserve = ""
+
+    while ch = getc
+      reserve << ch
+      if ch == "\\"    #"
+        ch = getc
+	reserve << ch
+      elsif ch == "\n"
+	break
+      end
+    end
+
+    str = ""
+    while (l = gets)
+      l.chomp!
+      l.strip! if indent
+      break if l == quoted
+      str << l.chomp << "\n"
+    end
+
+    @reader.divert_read_from(reserve)
+
+    @ltype = ltback
+    @lex_state = EXPR_END
+    Token(Ltype2Token[lt], str).set_text(str.dump)
+  end
+  
+  def identify_quotation(initial_char)
+    ch = getc
+    if lt = PERCENT_LTYPE[ch]
+      ch = getc
+    elsif ch =~ /\W/
+      lt = "\""
+    else
+      RubyLex.fail SyntaxError, "unknown type of %string ('#{ch}')"
+    end
+#     if ch !~ /\W/
+#       ungetc
+#       next
+#     end
+    #@ltype = lt
+    @quoted = ch unless @quoted = PERCENT_PAREN[ch]
+    identify_string(lt, @quoted, ch, initial_char)
+  end
+
+  def identify_number(start)
+    str = start.dup
+
+    if start == "+" or start == "-" or start == ""
+      start = getc
+      str << start
+    end
+
+    @lex_state = EXPR_END
+
+    if start == "0"
+      if peek(0) == "x"
+        ch = getc
+        str << ch
+        match = /[0-9a-f_]/
+      else
+        match = /[0-7_]/
+      end
+      while ch = getc
+        if ch !~ match
+          ungetc
+          break
+        else
+          str << ch
+        end
+      end
+      return Token(TkINTEGER).set_text(str)
+    end
+
+    type = TkINTEGER
+    allow_point = TRUE
+    allow_e = TRUE
+    while ch = getc
+      case ch
+      when /[0-9_]/
+        str << ch
+
+      when allow_point && "."
+	type = TkFLOAT
+	if peek(0) !~ /[0-9]/
+	  ungetc
+	  break
+	end
+        str << ch
+	allow_point = false
+
+      when allow_e && "e", allow_e && "E"
+        str << ch
+	type = TkFLOAT
+	if peek(0) =~ /[+-]/
+	  str << getc
+	end
+	allow_e = false
+	allow_point = false
+      else
+	ungetc
+	break
+      end
+    end
+    Token(type).set_text(str)
+  end
+  
+  def identify_string(ltype, quoted = ltype, opener=nil, initial_char = nil)
+    @ltype = ltype
+    @quoted = quoted
+    subtype = nil
+
+    str = ""
+    str << initial_char if initial_char
+    str << (opener||quoted)
+
+    nest = 0
+    begin
+      while ch = getc 
+	str << ch
+	if @quoted == ch 
+          if nest == 0
+            break
+          else
+            nest -= 1
+          end
+        elsif opener == ch
+          nest += 1
+	elsif @ltype != "'" && @ltype != "]" and ch == "#"
+          ch = getc
+          if ch == "{"
+            subtype = true
+            str << ch << skip_inner_expression
+          else
+            ungetc(ch)
+          end
+	elsif ch == '\\' #'
+	  str << read_escape
+	end
+      end
+      if @ltype == "/"
+	if peek(0) =~ /i|o|n|e|s/
+	  str << getc
+	end
+      end
+      if subtype
+	Token(DLtype2Token[ltype], str)
+      else
+	Token(Ltype2Token[ltype], str)
+      end.set_text(str)
+    ensure
+      @ltype = nil
+      @quoted = nil
+      @lex_state = EXPR_END
+    end
+  end
+
+  def skip_inner_expression
+    res = ""
+    nest = 0
+    while (ch = getc)
+      res << ch
+      if ch == '}'
+        break if nest.zero?
+        nest -= 1
+      elsif ch == '{'
+        nest += 1
+      end
+    end
+    res
+  end
+
+  def identify_comment
+    @ltype = "#"
+    comment = "#"
+    while ch = getc
+      if ch == "\\"
+        ch = getc
+        if ch == "\n"
+          ch = " "
+        else
+          comment << "\\" 
+        end
+      else
+        if ch == "\n"
+          @ltype = nil
+          ungetc
+          break
+        end
+      end
+      comment << ch
+    end
+    return Token(TkCOMMENT).set_text(comment)
+  end
+  
+  def read_escape
+    res = ""
+    case ch = getc
+    when /[0-7]/
+      ungetc ch
+      3.times do
+	case ch = getc
+	when /[0-7]/
+	when nil
+	  break
+	else
+	  ungetc
+	  break
+	end
+        res << ch
+      end
+      
+    when "x"
+      res << ch
+      2.times do
+	case ch = getc
+	when /[0-9a-fA-F]/
+	when nil
+	  break
+	else
+	  ungetc
+	  break
+	end
+        res << ch
+      end
+
+    when "M"
+      res << ch
+      if (ch = getc) != '-'
+	ungetc
+      else
+        res << ch
+	if (ch = getc) == "\\" #"
+          res << ch
+	  res << read_escape
+        else
+          res << ch
+	end
+      end
+
+    when "C", "c", "^"
+      res << ch
+      if ch == "C" and (ch = getc) != "-"
+	ungetc
+      else
+        res << ch
+        if (ch = getc) == "\\" #"
+          res << ch
+          res << read_escape
+        else
+          res << ch
+        end
+      end
+    else
+      res << ch
+    end
+    res
+  end
+end
+
+
+
+# Extract code elements from a source file, returning a TopLevel
+# object containing the constituent file elements.
+#
+# This file is based on rtags
+
+module RDoc
+
+  GENERAL_MODIFIERS = [ 'nodoc' ].freeze
+
+  CLASS_MODIFIERS = GENERAL_MODIFIERS
+
+  ATTR_MODIFIERS  = GENERAL_MODIFIERS
+
+  CONSTANT_MODIFIERS = GENERAL_MODIFIERS
+
+  METHOD_MODIFIERS = GENERAL_MODIFIERS + 
+    [ 'arg', 'args', 'yield', 'yields', 'notnew', 'not-new', 'not_new', 'doc' ]
+
+
+  class RubyParser
+    include RubyToken
+    include TokenStream
+
+    extend ParserFactory
+
+    parse_files_matching(/\.rbw?$/)
+
+
+    def initialize(top_level, file_name, content, options)
+      @options = options
+      @size = 0
+      @token_listeners = nil
+      @input_file_name = file_name
+      @scanner = RubyLex.new(content)
+      @scanner.exception_on_syntax_error = false
+      @top_level = top_level
+      @progress = $stderr unless options.quiet
+    end
+
+    def scan
+      @tokens = []
+      @unget_read = []
+      @read = []
+      catch(:eof) do
+        parse_statements(@top_level)
+      end
+      @top_level
+    end
+
+    private 
+
+    def warn(msg)
+      prefix = "\n" + @input_file_name + ":"
+      if @scanner
+        prefix << "#{@scanner.line_no}:#{@scanner.char_no}: "
+      end
+      $stderr.puts prefix + msg
+    end
+
+    def error(msg)
+      warn msg
+      exit(1)
+    end
+
+    def progress(char)
+      unless @options.quiet
+        @progress.print(char)
+	@progress.flush
+      end
+    end
+
+    def add_token_listener(obj)
+      @token_listeners ||= []
+      @token_listeners << obj
+    end
+
+    def remove_token_listener(obj)
+      @token_listeners.delete(obj)
+    end
+
+    def get_tk
+      tk = nil
+      if @tokens.empty?
+	tk = @scanner.token
+	@read.push @scanner.get_read
+	puts "get_tk1 => #{tk.inspect}" if $TOKEN_DEBUG
+      else
+	@read.push @unget_read.shift
+	tk = @tokens.shift
+	puts "get_tk2 => #{tk.inspect}" if $TOKEN_DEBUG
+      end
+
+      if tk.kind_of?(TkSYMBEG)
+        set_token_position(tk.line_no, tk.char_no)
+        tk1 = get_tk
+        if tk1.kind_of?(TkId) || tk1.kind_of?(TkOp)
+          tk = Token(TkSYMBOL).set_text(":" + tk1.name)
+          # remove the identifier we just read (we're about to
+          # replace it with a symbol)
+          @token_listeners.each do |obj|
+            obj.pop_token
+          end if @token_listeners
+        else
+          warn("':' not followed by identified or operator")
+          tk = tk1
+        end
+      end
+
+      # inform any listeners of our shiny new token
+      @token_listeners.each do |obj|
+        obj.add_token(tk)
+      end if @token_listeners
+
+      tk
+    end
+
+    def peek_tk
+      unget_tk(tk = get_tk)
+      tk
+    end
+
+    def unget_tk(tk)
+      @tokens.unshift tk
+      @unget_read.unshift @read.pop
+
+      # Remove this token from any listeners
+      @token_listeners.each do |obj|
+        obj.pop_token
+      end if @token_listeners
+    end
+
+    def skip_tkspace(skip_nl = true)
+      tokens = []
+      while ((tk = get_tk).kind_of?(TkSPACE) ||
+	     (skip_nl && tk.kind_of?(TkNL)))
+	tokens.push tk
+      end
+      unget_tk(tk)
+      tokens
+    end
+
+    def get_tkread
+      read = @read.join("")
+      @read = []
+      read
+    end
+
+    NORMAL = "::"
+    SINGLE = "<<"
+
+    # Look for the first comment in a file that isn't
+    # a shebang line.
+
+    def collect_first_comment
+      skip_tkspace
+      res = ''
+      first_line = true
+
+      tk = get_tk
+      while tk.kind_of?(TkCOMMENT)
+        if first_line && tk.text[0,2] == "#!"
+          skip_tkspace
+          tk = get_tk
+        else
+          res << tk.text << "\n"
+          tk = get_tk
+          if tk.kind_of? TkNL
+            skip_tkspace(false)
+            tk = get_tk
+          end
+        end
+        first_line = false
+      end
+      unget_tk(tk)
+      res
+    end
+
+    def parse_statements(container, single=NORMAL, current_method = nil)
+      nest = 1
+      save_visibility = container.visibility
+
+      if container.kind_of?(TopLevel)
+        comment = collect_first_comment
+        look_for_directives_in(container, comment)
+        container.comment = comment unless comment.empty?
+      else
+        comment = ''
+      end
+
+      non_comment_seen = true
+
+      while tk = get_tk
+
+        keep_comment = false
+
+        non_comment_seen = true unless tk.kind_of?(TkCOMMENT)
+
+#         $stderr.puts "===== #{tk.inspect}"
+#         blank_line_seen = true
+#         while tk.kind_of?(TkNL)
+#           skip_tkspace(false)
+#           if peek_tk.kind_of?(TkCOMMENT)
+#             if blank_line_seen
+#               comment = ''
+#               blank_line_seen = false
+#             end
+#             tk = get_tk
+#             comment << tk.text << "\n"
+#           else
+#             blank_line_seen = true
+#           end
+#           tk = get_tk
+#         end
+
+	case tk
+
+        when TkNL
+          skip_tkspace(true)   # Skip blanks and newlines
+          tk = get_tk
+          if tk.kind_of?(TkCOMMENT)
+            if non_comment_seen
+              comment = ''
+              non_comment_seen = false
+            end
+            while tk.kind_of?(TkCOMMENT)
+              comment << tk.text << "\n"
+              tk = get_tk          # this is the newline 
+              skip_tkspace(false)  # leading spaces
+              tk = get_tk
+            end
+            unless comment.empty?
+              look_for_directives_in(container, comment) 
+              if container.done_documenting
+                container.ongoing_visibility = save_visibility
+                return
+              end
+            end
+            keep_comment = true
+          else
+            non_comment_seen = true
+          end
+          unget_tk(tk)
+          keep_comment = true
+
+
+	when TkCLASS
+	  if container.document_children
+            parse_class(container, single, tk, comment)
+	  else
+	    nest += 1
+          end
+
+	when TkMODULE
+	  if container.document_children
+            parse_module(container, single, tk, comment)
+	  else
+	    nest += 1
+          end
+
+	when TkDEF
+	  if container.document_self
+	    parse_method(container, single, tk, comment)
+	  else
+	    nest += 1
+          end
+
+        when TkCONSTANT
+          if container.document_self
+            parse_constant(container, single, tk, comment)
+          end
+
+	when TkALIAS
+ 	  if container.document_self
+	    parse_alias(container, single, tk, comment)
+	  end
+
+        when TkYIELD
+          if current_method.nil?
+            warn("Warning: yield outside of method") if container.document_self
+          else
+            parse_yield(container, single, tk, current_method)
+          end
+
+          # Until and While can have a 'do', which shouldn't increas
+          # the nesting. We can't solve the general case, but we can
+          # handle most occurrences by ignoring a do at the end of a line
+
+        when  TkUNTIL, TkWHILE
+          nest += 1
+          puts "FOUND #{tk.class} in #{container.name}, nest = #{nest}, " +
+            "line #{tk.line_no}" if $DEBUG
+          skip_optional_do_after_expression
+
+	when TkCASE, TkDO, TkIF, TkUNLESS, TkBEGIN, TkFOR
+	  nest += 1
+          puts "Found #{tk.class} in #{container.name}, nest = #{nest}, " +
+            "line #{tk.line_no}" if $DEBUG
+
+	when TkIDENTIFIER
+          if nest == 1 and current_method.nil?
+            case tk.name
+            when "private", "protected", "public",
+                 "private_class_method", "public_class_method"
+              parse_visibility(container, single, tk)
+              keep_comment = true
+            when "attr"
+              parse_attr(container, single, tk, comment)
+            when /^attr_(reader|writer|accessor)$/, @options.extra_accessors
+              parse_attr_accessor(container, single, tk, comment)
+            when "alias_method"
+              if container.document_self
+	        parse_alias(container, single, tk, comment)
+	      end
+            end
+	  end
+	  
+	  case tk.name
+	  when "require"
+	    parse_require(container, comment)
+	  when "include"
+	    parse_include(container, comment)
+	  end
+
+
+	when TkEND
+          nest -= 1
+          puts "Found 'end' in #{container.name}, nest = #{nest}, line #{tk.line_no}" if $DEBUG
+          puts "Method = #{current_method.name}" if $DEBUG and current_method
+	  if nest == 0
+            read_documentation_modifiers(container, CLASS_MODIFIERS)
+            container.ongoing_visibility = save_visibility
+            return
+          end
+
+	end
+
+        comment = '' unless keep_comment
+	begin
+	  get_tkread
+	  skip_tkspace(false)
+	end while peek_tk == TkNL
+
+      end
+    end
+
+    def parse_class(container, single, tk, comment, &block)
+      progress("c")
+
+      container, name_t = get_class_or_module(container)
+
+      case name_t
+      when TkCONSTANT
+	name = name_t.name
+        superclass = "Object"
+        
+        if peek_tk.kind_of?(TkLT)
+          get_tk
+          skip_tkspace(true)
+          superclass = get_class_specification
+          superclass = "<unknown>" if superclass.empty?
+        end
+
+	if single == SINGLE
+	  cls_type = SingleClass
+	else
+	  cls_type = NormalClass
+	end
+        
+        cls = container.add_class(cls_type, name, superclass)
+        read_documentation_modifiers(cls, CLASS_MODIFIERS)
+        cls.record_location(@top_level)
+	parse_statements(cls)
+        cls.comment = comment
+
+      when TkLSHFT
+	case name = get_class_specification
+	when "self"
+	  parse_statements(container, SINGLE, &block)
+	else
+
+          # Special case: class << X inside class 'X' adds singleton methods
+          if name == container.name
+            parse_statements(container, SINGLE, &block)
+          else
+            other = TopLevel.find_class_named(name) || Context.new
+            parse_statements(other, SINGLE, &block)
+          end
+	end
+
+      else
+	warn("Expected class name or '<<'. Got #{name_t.class}: #{name_t.text.inspect}")
+      end
+    end
+
+    def parse_module(container, single, tk, comment)
+      progress("m")
+      container, name_t  = get_class_or_module(container)
+#      skip_tkspace
+      name = name_t.name
+      mod = container.add_module(NormalModule, name)
+      mod.record_location(@top_level)
+      read_documentation_modifiers(mod, CLASS_MODIFIERS)
+      parse_statements(mod)
+      mod.comment = comment
+    end
+
+    # Look for the name of a class of module (optionally with a leading :: or
+    # with :: separated named) and return the ultimate name and container
+
+    def get_class_or_module(container)
+      skip_tkspace
+      name_t = get_tk
+
+      # class ::A -> A is in the top level
+      if name_t.kind_of?(TkCOLON2)
+        name_t = get_tk
+        container = @top_level
+      end
+
+      skip_tkspace(false)
+
+      while peek_tk.kind_of?(TkCOLON2)
+        prev_container = container
+        container = container.find_module_named(name_t.name)
+        if !container
+#          warn("Couldn't find module #{name_t.name}")
+          container = prev_container.add_module(NormalModule, name_t.name)
+        end
+        get_tk
+        name_t = get_tk
+      end
+      return [container, name_t]
+    end
+
+    def parse_constant(container, single, tk, comment)
+      name = tk.name
+      skip_tkspace(false)
+      eq_tk = get_tk
+
+      unless eq_tk.kind_of?(TkASSIGN)
+        unget_tk(eq_tk)
+        return
+      end
+
+      tk = get_tk
+      if tk.kind_of? TkGT
+        unget_tk(tk)
+        unget_tk(eq_tk)
+        return
+      end
+
+      nest = 0
+      get_tkread
+      loop do
+        puts("Param: #{tk}, #{@scanner.continue} " +
+          "#{@scanner.lex_state} #{nest}")  if $DEBUG
+        case tk
+        when TkSEMICOLON
+          break
+        when TkLPAREN, TkfLPAREN
+          nest += 1
+        when TkRPAREN
+          nest -= 1
+        when TkCOMMENT
+          if nest <= 0 && @scanner.lex_state == EXPR_END
+            unget_tk(tk)
+            break
+          end
+        when TkNL
+          if (@scanner.lex_state == EXPR_END and nest <= 0) || !@scanner.continue
+            unget_tk(tk)
+            break
+          end
+        end
+        tk = get_tk
+      end
+      res = get_tkread.tr("\n", " ").strip
+      res = "" if res == ";"
+      con = Constant.new(name, res, comment)
+      read_documentation_modifiers(con, CONSTANT_MODIFIERS)
+      if con.document_self
+	container.add_constant(con)
+      end
+    end
+
+    def parse_method(container, single, tk, comment)
+      progress(".")
+      line_no = tk.line_no
+      column  = tk.char_no
+
+      start_collecting_tokens
+      add_token(tk)
+      add_token_listener(self)
+      
+      @scanner.instance_eval{@lex_state = EXPR_FNAME}
+      skip_tkspace(false)
+      name_t = get_tk
+      back_tk = skip_tkspace
+      meth = nil
+
+      dot = get_tk
+      if dot.kind_of?(TkDOT) or dot.kind_of?(TkCOLON2)
+	@scanner.instance_eval{@lex_state = EXPR_FNAME}
+	skip_tkspace
+	name_t2 = get_tk
+	case name_t
+	when TkSELF
+	  name = name_t2.name
+	when TkCONSTANT
+          name = name_t2.name
+          container = container.find_module_named(name_t.name)
+          if !container
+            error("Couldn't find module #{name_t.name}")
+          end
+	else
+	  # warn("Unexpected token '#{name_t2.inspect}'")
+	  # break
+          skip_method(container)
+          return
+	end
+	meth =  AnyMethod.new(get_tkread, name)
+        meth.singleton = true
+      else
+	unget_tk dot
+	back_tk.reverse_each do
+	  |tk|
+	  unget_tk tk
+	end
+	name = name_t.name
+
+        meth =  AnyMethod.new(get_tkread, name)
+        meth.singleton = (single == SINGLE)
+      end
+
+      remove_token_listener(self)
+
+      meth.start_collecting_tokens
+      indent = TkSPACE.new(1,1)
+      indent.set_text(" " * column)
+
+      meth.add_tokens([TkCOMMENT.new(line_no,
+                                     1,
+                                     "# File #{@top_level.file_absolute_name}, line #{line_no}"),
+                        NEWLINE_TOKEN,
+                        indent])
+
+      meth.add_tokens(@token_stream)
+
+      add_token_listener(meth)
+
+      @scanner.instance_eval{@continue = false}
+      parse_method_parameters(meth)
+
+      if meth.document_self
+	container.add_method(meth)
+      end
+
+      # Having now read the method parameters and documentation modifiers, we
+      # now know whether we have to rename #initialize to ::new
+
+      if name == "initialize" && !meth.singleton
+        if meth.dont_rename_initialize
+          meth.visibility = :protected
+        else
+          meth.singleton = true
+          meth.name = "new"
+          meth.visibility = :public
+        end
+      end
+
+      parse_statements(container, single, meth)
+
+      remove_token_listener(meth)
+
+      meth.comment = comment
+    end
+
+    def skip_method(container)
+      meth =  AnyMethod.new("", "anon")
+      parse_method_parameters(meth)
+      parse_statements(container, false, meth)
+    end
+
+    # Capture the method's parameters. Along the way,
+    # look for a comment containing 
+    #
+    #    # yields: ....
+    #
+    # and add this as the block_params for the method
+
+    def parse_method_parameters(method)
+      res = parse_method_or_yield_parameters(method)
+      method.params = res unless method.params
+      if method.block_params.nil?
+          skip_tkspace(false)
+	  read_documentation_modifiers(method, METHOD_MODIFIERS)
+      end
+    end
+
+    def parse_method_or_yield_parameters(method=nil, modifiers=METHOD_MODIFIERS)
+      skip_tkspace(false)
+      tk = get_tk
+
+      # Little hack going on here. In the statement
+      #  f = 2*(1+yield)
+      # We see the RPAREN as the next token, so we need
+      # to exit early. This still won't catch all cases
+      # (such as "a = yield + 1"
+      end_token = case tk
+                  when TkLPAREN, TkfLPAREN
+                    TkRPAREN
+                  when TkRPAREN
+                    return ""
+                  else
+                    TkNL
+                  end
+      nest = 0
+
+      loop do
+        puts("Param: #{tk}, #{@scanner.continue} " +
+          "#{@scanner.lex_state} #{nest}") if $DEBUG
+        case tk
+        when TkSEMICOLON
+          break
+        when TkLPAREN, TkfLPAREN
+          nest += 1
+        when end_token
+          if end_token == TkRPAREN
+            nest -= 1
+            break if @scanner.lex_state == EXPR_END and nest <= 0
+          else
+            break unless @scanner.continue
+          end
+        when method && method.block_params.nil? && TkCOMMENT
+	  unget_tk(tk)
+	  read_documentation_modifiers(method, modifiers)
+        end
+        tk = get_tk
+      end
+      res = get_tkread.tr("\n", " ").strip
+      res = "" if res == ";"
+      res
+    end
+
+    # while, until, and for have an optional 
+    def skip_optional_do_after_expression
+      skip_tkspace(false)
+      tk = get_tk
+      case tk
+      when TkLPAREN, TkfLPAREN
+        end_token = TkRPAREN
+      else
+        end_token = TkNL
+      end
+
+      nest = 0
+      @scanner.instance_eval{@continue = false}
+
+      loop do
+        puts("\nWhile: #{tk}, #{@scanner.continue} " +
+          "#{@scanner.lex_state} #{nest}") if $DEBUG
+        case tk
+        when TkSEMICOLON
+          break
+        when TkLPAREN, TkfLPAREN
+          nest += 1
+        when TkDO
+          break if nest.zero?
+        when end_token
+          if end_token == TkRPAREN
+            nest -= 1
+            break if @scanner.lex_state == EXPR_END and nest.zero?
+          else
+            break unless @scanner.continue
+          end
+        end
+        tk = get_tk
+      end
+      skip_tkspace(false)
+      if peek_tk.kind_of? TkDO
+        get_tk
+      end
+    end
+    
+    # Return a superclass, which can be either a constant
+    # of an expression
+
+    def get_class_specification
+      tk = get_tk
+      return "self" if tk.kind_of?(TkSELF)
+        
+      res = ""
+      while tk.kind_of?(TkCOLON2) ||
+          tk.kind_of?(TkCOLON3)   ||
+          tk.kind_of?(TkCONSTANT)   
+        
+        res += tk.text
+        tk = get_tk
+      end
+
+      unget_tk(tk)
+      skip_tkspace(false)
+
+      get_tkread # empty out read buffer
+
+      tk = get_tk
+
+      case tk
+      when TkNL, TkCOMMENT, TkSEMICOLON
+        unget_tk(tk)
+        return res
+      end
+
+      res += parse_call_parameters(tk)
+      res
+    end
+
+    def parse_call_parameters(tk)
+
+      end_token = case tk
+                  when TkLPAREN, TkfLPAREN
+                    TkRPAREN
+                  when TkRPAREN
+                    return ""
+                  else
+                    TkNL
+                  end
+      nest = 0
+
+      loop do
+        puts("Call param: #{tk}, #{@scanner.continue} " +
+          "#{@scanner.lex_state} #{nest}") if $DEBUG
+        case tk
+        when TkSEMICOLON
+          break
+        when TkLPAREN, TkfLPAREN
+          nest += 1
+        when end_token
+          if end_token == TkRPAREN
+            nest -= 1
+            break if @scanner.lex_state == EXPR_END and nest <= 0
+          else
+            break unless @scanner.continue
+          end
+        when TkCOMMENT
+	  unget_tk(tk)
+	  break
+        end
+        tk = get_tk
+      end
+      res = get_tkread.tr("\n", " ").strip
+      res = "" if res == ";"
+      res
+    end
+
+
+    # Parse a constant, which might be qualified by
+    # one or more class or module names
+
+    def get_constant
+      res = ""
+      skip_tkspace(false)
+      tk = get_tk
+
+      while tk.kind_of?(TkCOLON2) ||
+          tk.kind_of?(TkCOLON3)   ||
+          tk.kind_of?(TkCONSTANT)          
+        
+        res += tk.text
+        tk = get_tk
+      end
+
+#      if res.empty?
+#        warn("Unexpected token #{tk} in constant")
+#      end 
+      unget_tk(tk)
+      res
+    end
+
+    # Get a constant that may be surrounded by parens
+    
+    def get_constant_with_optional_parens
+      skip_tkspace(false)
+      nest = 0
+      while (tk = peek_tk).kind_of?(TkLPAREN)  || tk.kind_of?(TkfLPAREN)
+        get_tk
+        skip_tkspace(true)
+        nest += 1
+      end
+
+      name = get_constant
+
+      while nest > 0
+        skip_tkspace(true)
+        tk = get_tk
+        nest -= 1 if tk.kind_of?(TkRPAREN)
+      end
+      name
+    end
+
+    # Directives are modifier comments that can appear after class, module,
+    # or method names. For example
+    #
+    #   def fred    # :yields:  a, b
+    #
+    # or
+    #
+    #   class SM  # :nodoc:
+    #
+    # we return the directive name and any parameters as a two element array
+    
+    def read_directive(allowed)
+      tk = get_tk
+      puts "directive: #{tk.inspect}" if $DEBUG
+      result = nil
+      if tk.kind_of?(TkCOMMENT) 
+        if tk.text =~ /\s*:?(\w+):\s*(.*)/
+          directive = $1.downcase
+          if allowed.include?(directive)
+            result = [directive, $2]
+          end
+        end
+      else
+        unget_tk(tk)
+      end
+      result
+    end
+
+    
+    def read_documentation_modifiers(context, allow)
+      dir = read_directive(allow)
+
+      case dir[0]
+
+      when "notnew", "not_new", "not-new"
+        context.dont_rename_initialize = true
+
+      when "nodoc"
+        context.document_self = false
+	if dir[1].downcase == "all"
+	  context.document_children = false
+	end
+
+      when "doc"
+        context.document_self = true
+        context.force_documentation = true
+
+      when "yield", "yields"
+        unless context.params.nil?
+          context.params.sub!(/(,|)\s*&\w+/,'') # remove parameter &proc
+        end
+	context.block_params = dir[1]
+
+      when "arg", "args"
+        context.params = dir[1]
+      end if dir
+    end
+
+    
+    # Look for directives in a normal comment block:
+    #
+    #   #--       - don't display comment from this point forward
+    #  
+    #
+    # This routine modifies it's parameter
+
+    def look_for_directives_in(context, comment)
+
+      preprocess = SM::PreProcess.new(@input_file_name,
+                                      @options.rdoc_include)
+
+      preprocess.handle(comment) do |directive, param|
+        case directive
+        when "stopdoc"
+          context.stop_doc
+          ""
+        when "startdoc"
+          context.start_doc
+          context.force_documentation = true
+          ""
+
+        when "enddoc"
+          context.done_documenting = true
+          ""
+
+        when "main"
+          options = Options.instance
+          options.main_page = param
+	  ""
+
+        when "title"
+          options = Options.instance
+          options.title = param
+          ""
+
+        else
+          warn "Unrecognized directive '#{directive}'"
+          break
+        end
+      end
+
+      remove_private_comments(comment)
+    end
+
+    def remove_private_comments(comment)
+      comment.gsub!(/^#--.*?^#\+\+/m, '')
+      comment.sub!(/^#--.*/m, '')
+    end
+
+
+
+    def get_symbol_or_name
+      tk = get_tk
+      case tk
+      when  TkSYMBOL
+        tk.text.sub(/^:/, '')
+      when TkId, TkOp
+        tk.name
+      else
+        raise "Name or symbol expected (got #{tk})"
+      end
+    end
+    
+    def parse_alias(context, single, tk, comment)
+      skip_tkspace
+      new_name = get_symbol_or_name
+      @scanner.instance_eval{@lex_state = EXPR_FNAME}
+      skip_tkspace
+      if (peek_tk.kind_of? TkCOMMA)
+        get_tk
+        skip_tkspace
+      end
+      old_name = get_symbol_or_name
+      al = Alias.new(get_tkread, old_name, new_name, comment)
+      read_documentation_modifiers(al, ATTR_MODIFIERS)
+      if al.document_self
+	context.add_alias(al)
+      end
+    end
+
+    def parse_yield_parameters
+      parse_method_or_yield_parameters
+    end
+
+  def parse_yield(context, single, tk, method)
+    if method.block_params.nil?
+      get_tkread
+      @scanner.instance_eval{@continue = false}
+      method.block_params = parse_yield_parameters
+    end
+  end
+
+  def parse_require(context, comment)
+    skip_tkspace_comment
+    tk = get_tk
+    if tk.kind_of? TkLPAREN
+      skip_tkspace_comment
+      tk = get_tk
+    end
+
+    name = nil
+    case tk
+    when TkSTRING
+      name = tk.text
+    when TkCONSTANT, TkIDENTIFIER, TkIVAR, TkGVAR
+      name = tk.name
+    when TkDSTRING
+      warn "Skipping require of dynamic string: #{tk.text}"
+    else
+      error("Unknown argument type to require: #{tk}")
+    end
+    if name
+      context.add_require(Require.new(name, comment))
+    end
+  end
+
+  def parse_include(context, comment)
+    skip_tkspace_comment
+    name = get_constant_with_optional_parens
+    unless name.empty?
+      context.add_include(Include.new(name, comment))
+    end
+  end
+
+    def get_bool
+      skip_tkspace
+      tk = get_tk
+      case tk
+      when TkTRUE
+        true
+      when TkFALSE, TkNIL
+        false
+      else
+        unget_tk tk
+        true
+      end
+    end
+
+    def parse_attr(context, single, tk, comment)
+      args = parse_symbol_arg(1)
+      if args.size > 0
+	name = args[0]
+        rw = "R"
+        skip_tkspace(false)
+        tk = get_tk
+        if tk.kind_of? TkCOMMA
+          rw = "RW" if get_bool
+        else
+          unget_tk tk
+        end
+	att = Attr.new(get_tkread, name, rw, comment)
+	read_documentation_modifiers(att, ATTR_MODIFIERS)
+	if att.document_self
+	  context.add_attribute(att)
+	end
+      else
+	error("Missing argument to 'attr'")
+      end    
+
+    end
+
+    def parse_visibility(container, single, tk)
+      singleton = (single == SINGLE)
+      vis = case tk.name
+            when "private"   then :private
+            when "protected" then :protected
+            when "public"    then :public
+            when "private_class_method"
+              singleton = true
+              :private
+            when "public_class_method"
+              singleton = true
+              :public
+            else raise "Invalid visibility: #{tk.name}"
+            end
+            
+      skip_tkspace_comment(false)
+      if peek_tk.kind_of? TkNL
+#        error("Missing argument") if singleton        
+        container.ongoing_visibility = vis
+      else
+        args = parse_symbol_arg
+        container.set_visibility_for(args, vis, singleton)
+      end
+    end
+
+    def parse_attr_accessor(context, single, tk, comment)
+      args = parse_symbol_arg
+      read = get_tkread
+      rw = "?"
+
+      # If nodoc is given, don't document any of them
+
+      tmp = CodeObject.new
+      read_documentation_modifiers(tmp, ATTR_MODIFIERS)
+      return unless tmp.document_self
+
+      case tk.name
+      when "attr_reader"   then rw = "R"
+      when "attr_writer"   then rw = "W"
+      when "attr_accessor" then rw = "RW"
+      else
+        rw = @options.extra_accessor_flags[tk.name]
+      end
+      
+      for name in args
+	att = Attr.new(get_tkread, name, rw, comment)
+        context.add_attribute(att)
+      end    
+    end
+
+    def skip_tkspace_comment(skip_nl = true)
+      loop do
+        skip_tkspace(skip_nl)
+        return unless peek_tk.kind_of? TkCOMMENT
+        get_tk
+      end
+    end
+
+    def parse_symbol_arg(no = nil)
+
+      args = []
+      skip_tkspace_comment
+      case tk = get_tk
+      when TkLPAREN
+	loop do
+	  skip_tkspace_comment
+	  if tk1 = parse_symbol_in_arg
+	    args.push tk1
+	    break if no and args.size >= no
+	  end
+	  
+	  skip_tkspace_comment
+	  case tk2 = get_tk
+	  when TkRPAREN
+	    break
+	  when TkCOMMA
+	  else
+	    warn("unexpected token: '#{tk.inspect}'")
+	    break
+	  end
+	end
+      else
+	unget_tk tk
+	if tk = parse_symbol_in_arg
+	  args.push tk
+	  return args if no and args.size >= no
+	end
+
+	loop do
+#	  skip_tkspace_comment(false)
+	  skip_tkspace(false)
+
+	  tk1 = get_tk
+	  unless tk1.kind_of?(TkCOMMA) 
+	    unget_tk tk1
+	    break
+	  end
+	  
+	  skip_tkspace_comment
+	  if tk = parse_symbol_in_arg
+	    args.push tk
+	    break if no and args.size >= no
+	  end
+	end
+      end
+      args
+    end
+
+    def parse_symbol_in_arg
+      case tk = get_tk
+      when TkSYMBOL
+        tk.text.sub(/^:/, '')
+      when TkSTRING
+	eval @read[-1]
+      else
+	warn("Expected symbol or string, got #{tk.inspect}") if $DEBUG
+	nil
+      end
+    end
+  end
+
+end
diff --git a/lib/rdoc/parsers/parse_simple.rb b/lib/rdoc/parsers/parse_simple.rb
new file mode 100644
index 0000000000..754f650793
--- /dev/null
+++ b/lib/rdoc/parsers/parse_simple.rb
@@ -0,0 +1,37 @@
+# Parse a non-source file. We basically take the whole thing 
+# as one big comment. If the first character in the file
+# is '#', we strip leading pound signs.
+
+
+require "rdoc/code_objects"
+require "rdoc/markup/simple_markup/preprocess"
+
+module RDoc
+  # See rdoc/parsers/parse_c.rb
+
+  class SimpleParser
+    
+    # prepare to parse a plain file
+    def initialize(top_level, file_name, body, options)
+      
+      preprocess = SM::PreProcess.new(file_name, options.rdoc_include)
+      
+      preprocess.handle(body) do |directive, param|
+        $stderr.puts "Unrecognized directive '#{directive}' in #{file_name}"
+      end
+      
+      @body = body
+      @options = options
+      @top_level = top_level
+    end
+    
+    # Extract the file contents and attach them to the toplevel as a
+    # comment
+    
+    def scan
+      #    @body.gsub(/^(\s\n)+/, '')
+      @top_level.comment = @body
+      @top_level
+    end
+  end
+end
diff --git a/lib/rdoc/parsers/parserfactory.rb b/lib/rdoc/parsers/parserfactory.rb
new file mode 100644
index 0000000000..dc7d629903
--- /dev/null
+++ b/lib/rdoc/parsers/parserfactory.rb
@@ -0,0 +1,86 @@
+require "rdoc/parsers/parse_simple"
+
+module RDoc
+
+  # A parser is simple a class that implements
+  #
+  #   #initialize(file_name, body, options)
+  #
+  # and
+  #
+  #   #scan
+  #
+  # The initialize method takes a file name to be used, the body of the
+  # file, and an RDoc::Options object. The scan method is then called
+  # to return an appropriately parsed TopLevel code object.
+
+  # The ParseFactory is used to redirect to the correct parser given a filename
+  # extension. This magic works because individual parsers have to register 
+  # themselves with us as they are loaded in. The do this using the following
+  # incantation
+  #
+  #
+  #    require "rdoc/parsers/parsefactory"
+  #    
+  #    module RDoc
+  #    
+  #      class XyzParser
+  #        extend ParseFactory                  <<<<
+  #        parse_files_matching /\.xyz$/        <<<<
+  #    
+  #        def initialize(file_name, body, options)
+  #          ...
+  #        end
+  #    
+  #        def scan
+  #          ...
+  #        end
+  #      end
+  #    end
+
+
+
+  module ParserFactory
+
+    @@parsers = []
+
+    Parsers = Struct.new(:regexp, :parser)
+
+    # Record the fact that a particular class parses files that
+    # match a given extension
+
+    def parse_files_matching(regexp)
+      @@parsers.unshift Parsers.new(regexp, self)
+    end
+
+    # Return a parser that can handle a particular extension
+
+    def ParserFactory.can_parse(file_name)
+      @@parsers.find {|p| p.regexp.match(file_name) }
+    end
+
+    # Alias an extension to another extension. After this call,
+    # files ending "new_ext" will be parsed using the same parser
+    # as "old_ext"
+
+    def ParserFactory.alias_extension(old_ext, new_ext)
+      parser = ParserFactory.can_parse("xxx.#{old_ext}")
+      return false unless parser
+      @@parsers.unshift Parsers.new(Regexp.new("\\.#{new_ext}$"), parser.parser)
+      true
+    end
+
+    # Find the correct parser for a particular file name. Return a 
+    # SimpleParser for ones that we don't know
+
+    def ParserFactory.parser_for(top_level, file_name, body, options)
+      parser_description = can_parse(file_name)
+      if parser_description
+        parser = parser_description.parser 
+      else
+        parser = SimpleParser
+      end
+      parser.new(top_level, file_name, body, options)
+    end
+  end
+end
diff --git a/lib/rdoc/rdoc.rb b/lib/rdoc/rdoc.rb
new file mode 100644
index 0000000000..ca9c5da2bf
--- /dev/null
+++ b/lib/rdoc/rdoc.rb
@@ -0,0 +1,219 @@
+# See README.
+#
+ 
+
+RDOC_VERSION = "1.0pr1"
+
+rcs = '$Date$ $Revision$'.
+  gsub(/\$/, '').
+  sub(/Date: /, ': ').
+  sub(/ Revision: (\S+)/) { "(#$1)" }
+
+VERSION_STRING = %{RDoc V} + RDOC_VERSION + rcs
+
+
+require 'rdoc/parsers/parse_rb.rb'
+require 'rdoc/parsers/parse_c.rb'
+require 'rdoc/parsers/parse_f95.rb'
+
+require 'rdoc/parsers/parse_simple.rb'
+require 'rdoc/options'
+
+require 'rdoc/diagram'
+
+require 'find'
+require 'ftools'
+
+# We put rdoc stuff in the RDoc module to avoid namespace
+# clutter.
+#
+# ToDo: This isn't universally true.
+
+module RDoc
+
+  # Exception thrown by any rdoc error. Only the #message part is
+  # of use externally.
+
+  class RDocError < Exception
+  end
+
+  # Encapsulate the production of rdoc documentation. Basically
+  # you can use this as you would invoke rdoc from the command
+  # line:
+  #
+  #    rdoc = RDoc::RDoc.new
+  #    rdoc.document(args)
+  #
+  # where _args_ is an array of strings, each corresponding to
+  # an argument you'd give rdoc on the command line. See rdoc/rdoc.rb 
+  # for details.
+  
+  class RDoc
+
+    ##
+    # This is the list of output generators that we
+    # support
+    
+    Generator = Struct.new(:file_name, :class_name, :key)
+    
+    GENERATORS = {}
+    $:.collect {|d|
+      File::expand_path(d)
+    }.find_all {|d|
+      File::directory?("#{d}/rdoc/generators")
+    }.each {|dir|
+      Dir::entries("#{dir}/rdoc/generators").each {|gen|
+        next unless /(\w+)_generator.rb$/ =~ gen
+        type = $1
+        unless GENERATORS.has_key? type
+          GENERATORS[type] = Generator.new("rdoc/generators/#{gen}",
+                                           "#{type.upcase}Generator".intern,
+                                           type)
+        end
+      }
+    }                                                    
+
+    #######
+    private
+    #######
+
+    ##
+    # Report an error message and exit
+    
+    def error(msg)
+      raise RDocError.new(msg)
+    end
+    
+    ##
+    # Create an output dir if it doesn't exist. If it does
+    # exist, but doesn't contain the flag file <tt>created.rid</tt>
+    # then we refuse to use it, as we may clobber some
+    # manually generated documentation
+    
+    def setup_output_dir(op_dir)
+      flag_file = File.join(op_dir, "created.rid")
+      if File.exist?(op_dir)
+        unless File.directory?(op_dir)
+          error "'#{op_dir}' exists, and is not a directory" 
+        end
+        unless File.file?(flag_file)
+          error "\nDirectory #{op_dir} already exists, but it looks like it\n" +
+            "isn't an RDoc directory. Because RDoc doesn't want to risk\n" +
+            "destroying any of your existing files, you'll need to\n" +
+            "specify a different output directory name (using the\n" +
+            "--op <dir> option).\n\n"
+        end
+      else
+        File.makedirs(op_dir)
+      end
+      File.open(flag_file, "w") {|f| f.puts Time.now }
+    end
+    
+
+    # Given a list of files and directories, create a list
+    # of all the Ruby files they contain. 
+    
+    def normalized_file_list(options, *relative_files)
+      file_list = []
+
+      relative_files.each do |rel_file_name|
+
+        case type = File.stat(rel_file_name).ftype
+        when "file"
+          file_list << rel_file_name
+        when "directory"
+          next if options.exclude && options.exclude =~ rel_file_name
+          Find.find(rel_file_name) do |fn|
+            next if options.exclude && options.exclude =~ fn
+            next unless ParserFactory.can_parse(fn)
+            next unless File.file?(fn)
+            
+            file_list << fn.sub(%r{\./}, '')
+          end
+        else
+          raise RDocError.new("I can't deal with a #{type} #{rel_file_name}")
+        end
+      end
+      file_list
+    end
+
+    # Parse each file on the command line, recursively entering
+    # directories
+
+    def parse_files(options)
+ 
+      file_info = []
+
+      files = options.files
+      files = ["."] if files.empty?
+
+      file_list = normalized_file_list(options, *files)
+
+      file_list.each do |fn|
+        $stderr.printf("\n%35s: ", File.basename(fn)) unless options.quiet
+        
+        content = File.open(fn, "r") {|f| f.read}
+
+        top_level = TopLevel.new(fn)
+        parser = ParserFactory.parser_for(top_level, fn, content, options)
+        file_info << parser.scan
+      end
+
+      file_info
+    end
+
+
+    public
+
+    ###################################################################
+    #
+    # Format up one or more files according to the given arguments.
+    # For simplicity, _argv_ is an array of strings, equivalent to the
+    # strings that would be passed on the command line. (This isn't a
+    # coincidence, as we _do_ pass in ARGV when running
+    # interactively). For a list of options, see rdoc/rdoc.rb. By
+    # default, output will be stored in a directory called +doc+ below
+    # the current directory, so make sure you're somewhere writable
+    # before invoking.
+    #
+    # Throws: RDocError on error
+
+    def document(argv)
+
+      TopLevel::reset
+
+      options = Options.instance
+      options.parse(argv, GENERATORS)
+    
+      file_info = parse_files(options)
+
+      gen = options.generator
+      
+      $stderr.puts "\nGenerating #{gen.key.upcase}..." unless options.quiet
+      
+      require gen.file_name
+      
+      gen_class = Generators.const_get(gen.class_name)
+      
+      unless file_info.empty?
+        gen = gen_class.for(options)
+
+        pwd = Dir.pwd
+
+        unless options.all_one_file
+          setup_output_dir(options.op_dir)
+          Dir.chdir(options.op_dir)
+        end
+
+        begin
+          Diagram.new(file_info, options).draw if options.diagram
+          gen.generate(file_info)
+        ensure
+          Dir.chdir(pwd)
+        end
+
+      end
+    end
+  end
+end
+
diff --git a/lib/rdoc/template.rb b/lib/rdoc/template.rb
new file mode 100644
index 0000000000..469e10fb4b
--- /dev/null
+++ b/lib/rdoc/template.rb
@@ -0,0 +1,234 @@
+# Cheap-n-cheerful HTML page template system. You create a 
+# template containing:
+#
+# * variable names between percent signs (<tt>%fred%</tt>)
+# * blocks of repeating stuff:
+#
+#     START:key
+#       ... stuff
+#     END:key
+#
+# You feed the code a hash. For simple variables, the values
+# are resolved directly from the hash. For blocks, the hash entry
+# corresponding to +key+ will be an array of hashes. The block will
+# be generated once for each entry. Blocks can be nested arbitrarily
+# deeply.
+#
+# The template may also contain
+#
+#   IF:key
+#     ... stuff
+#   ENDIF:key
+#
+# _stuff_ will only be included in the output if the corresponding
+# key is set in the value hash.
+#
+# Usage:  Given a set of templates <tt>T1, T2,</tt> etc
+#
+#            values = { "name" => "Dave", state => "TX" }
+#
+#            t = TemplatePage.new(T1, T2, T3)
+#            File.open(name, "w") {|f| t.write_html_on(f, values)}
+#         or
+#            res = ''
+#            t.write_html_on(res, values)
+#
+#
+
+class TemplatePage
+
+  ##########
+  # A context holds a stack of key/value pairs (like a symbol
+  # table). When asked to resolve a key, it first searches the top of
+  # the stack, then the next level, and so on until it finds a match
+  # (or runs out of entries)
+
+  class Context
+    def initialize
+      @stack = []
+    end
+
+    def push(hash)
+      @stack.push(hash)
+    end
+
+    def pop
+      @stack.pop
+    end
+
+    # Find a scalar value, throwing an exception if not found. This
+    # method is used when substituting the %xxx% constructs
+
+    def find_scalar(key)
+      @stack.reverse_each do |level|
+        if val = level[key]
+          return val unless val.kind_of? Array
+        end
+      end
+      raise "Template error: can't find variable '#{key}'"
+    end
+
+    # Lookup any key in the stack of hashes
+
+    def lookup(key)
+      @stack.reverse_each do |level|
+        val = level[key]
+        return val if val
+      end
+      nil
+    end
+  end
+
+  #########
+  # Simple class to read lines out of a string
+
+  class LineReader
+    # we're initialized with an array of lines
+    def initialize(lines)
+      @lines = lines
+    end
+
+    # read the next line 
+    def read
+      @lines.shift
+    end
+
+    # Return a list of lines up to the line that matches
+    # a pattern. That last line is discarded.
+    def read_up_to(pattern)
+      res = []
+      while line = read
+        if pattern.match(line)
+          return LineReader.new(res) 
+        else
+          res << line
+        end
+      end
+      raise "Missing end tag in template: #{pattern.source}"
+    end
+
+    # Return a copy of ourselves that can be modified without
+    # affecting us
+    def dup
+      LineReader.new(@lines.dup)
+    end
+  end
+
+
+
+  # +templates+ is an array of strings containing the templates.
+  # We start at the first, and substitute in subsequent ones
+  # where the string <tt>!INCLUDE!</tt> occurs. For example,
+  # we could have the overall page template containing
+  #
+  #   <html><body>
+  #     <h1>Master</h1>
+  #     !INCLUDE!
+  #   </bost></html>
+  #
+  # and substitute subpages in to it by passing [master, sub_page].
+  # This gives us a cheap way of framing pages
+
+  def initialize(*templates)
+    result = "!INCLUDE!"
+    templates.each do |content|
+      result.sub!(/!INCLUDE!/, content)
+    end
+    @lines = LineReader.new(result.split($/))
+  end
+
+  # Render the templates into HTML, storing the result on +op+ 
+  # using the method <tt><<</tt>. The <tt>value_hash</tt> contains
+  # key/value pairs used to drive the substitution (as described above)
+
+  def write_html_on(op, value_hash)
+    @context = Context.new
+    op << substitute_into(@lines, value_hash).tr("\000", '\\')
+  end
+
+
+  # Substitute a set of key/value pairs into the given template. 
+  # Keys with scalar values have them substituted directly into
+  # the page. Those with array values invoke <tt>substitute_array</tt>
+  # (below), which examples a block of the template once for each 
+  # row in the array.
+  #
+  # This routine also copes with the <tt>IF:</tt>_key_ directive,
+  # removing chunks of the template if the corresponding key
+  # does not appear in the hash, and the START: directive, which
+  # loops its contents for each value in an array
+
+  def substitute_into(lines, values)
+    @context.push(values)
+    skip_to = nil
+    result = []
+
+    while line = lines.read
+
+      case line
+
+      when /^IF:(\w+)/
+        lines.read_up_to(/^ENDIF:#$1/) unless @context.lookup($1)
+
+    when /^IFNOT:(\w+)/
+        lines.read_up_to(/^ENDIF:#$1/) if @context.lookup($1)
+
+      when /^ENDIF:/
+        ;
+
+      when /^START:(\w+)/
+        tag = $1
+        body = lines.read_up_to(/^END:#{tag}/)
+        inner_values = @context.lookup(tag)
+        raise "unknown tag: #{tag}" unless inner_values
+        raise "not array: #{tag}"   unless inner_values.kind_of?(Array)
+        inner_values.each do |vals|
+          result << substitute_into(body.dup, vals)
+        end
+      else
+        result << expand_line(line.dup)
+      end
+    end
+
+    @context.pop
+
+    result.join("\n")
+  end
+
+  # Given an individual line, we look for %xxx% constructs and 
+  # HREF:ref:name: constructs, substituting for each.
+
+  def expand_line(line)
+    # Generate a cross reference if a reference is given,
+    # otherwise just fill in the name part
+
+    line.gsub!(/HREF:(\w+?):(\w+?):/) {
+      ref = @context.lookup($1)
+      name = @context.find_scalar($2)
+
+      if ref and !ref.kind_of?(Array)
+	"<a href=\"#{ref}\">#{name}</a>"
+      else
+	name
+      end
+    }
+
+    # Substitute in values for %xxx% constructs.  This is made complex
+    # because the replacement string may contain characters that are
+    # meaningful to the regexp (like \1)
+
+    line = line.gsub(/%([a-zA-Z]\w*)%/) {
+      val = @context.find_scalar($1) 
+      val.tr('\\', "\000")
+    }
+
+
+    line
+  rescue Exception => e
+    $stderr.puts "Error in template: #{e}"
+    $stderr.puts "Original line: #{line}"
+    exit
+  end
+
+end
+
diff --git a/lib/rdoc/tokenstream.rb b/lib/rdoc/tokenstream.rb
new file mode 100644
index 0000000000..0a0720d8a9
--- /dev/null
+++ b/lib/rdoc/tokenstream.rb
@@ -0,0 +1,25 @@
+# A TokenStream is a list of tokens, gathered during the parse
+# of some entity (say a method). Entities populate these streams
+# by being registered with the lexer. Any class can collect tokens
+# by including TokenStream. From the outside, you use such an object
+# by calling the start_collecting_tokens method, followed by calls
+# to add_token and pop_token
+
+module TokenStream
+  def token_stream
+    @token_stream
+  end
+
+  def start_collecting_tokens
+    @token_stream = []
+  end
+  def add_token(tk)
+    @token_stream << tk
+  end
+  def add_tokens(tks)
+    tks.each  {|tk| add_token(tk)}
+  end
+  def pop_token
+    @token_stream.pop
+  end
+end