added tag v1_9_0_4

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/tags/v1_9_0_4@18845 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

56 files changed, 18060 insertions, 0 deletions

diff --git a/trunk/lib/rdoc/README b/trunk/lib/rdoc/README
new file mode 100644
index 0000000000..f183c61f8d
--- /dev/null
+++ b/trunk/lib/rdoc/README
@@ -0,0 +1,232 @@
+= RDOC - Ruby Documentation System
+
+This package contains RDoc and RDoc::Markup.  RDoc is an application that
+produces documentation for one or more Ruby source files.  We work similarly to
+JavaDoc, parsing the source, and extracting the definition for classes,
+modules, and methods (along with includes and requires).  We associate with
+these optional documentation contained in the immediately preceding comment
+block, and then render the result using a pluggable output formatter.
+RDoc::Markup is a library that converts plain text into various output formats.
+The markup library is used to interpret the comment blocks that RDoc uses to
+document methods, classes, and so on.
+
+== Roadmap
+
+* If you want to use RDoc to create documentation for your Ruby source files,
+  read on.
+* If you want to include extensions written in C, see RDoc::C_Parser
+* For information on the various markups available in comment blocks, see
+  RDoc::Markup.
+* If you want to drive RDoc programmatically, see RDoc::RDoc.
+* If you want to use the library to format text blocks into HTML, have a look
+  at RDoc::Markup.
+* If you want to try writing your own HTML output template, see
+  RDoc::Generator::HTML
+
+== Summary
+
+Once installed, you can create documentation using the 'rdoc' command
+(the command is 'rdoc.bat' under Windows)
+
+  % rdoc [options] [names...]
+
+Type "rdoc --help" for an up-to-date option summary.
+
+A typical use might be to generate documentation for a package of Ruby
+source (such as rdoc itself). 
+
+  % rdoc
+
+This command generates documentation for all the Ruby and C source
+files in and below the current directory.  These will be stored in a
+documentation tree starting in the subdirectory 'doc'.
+
+You can make this slightly more useful for your readers by having the
+index page contain the documentation for the primary file.  In our
+case, we could type
+
+  % rdoc --main rdoc.rb
+
+You'll find information on the various formatting tricks you can use
+in comment blocks in the documentation this generates.
+
+RDoc uses file extensions to determine how to process each file.  File names
+ending +.rb+ and <tt>.rbw</tt> are assumed to be Ruby source.  Files
+ending +.c+ are parsed as C files.  All other files are assumed to
+contain just Markup-style markup (with or without leading '#' comment markers).
+If directory names are passed to RDoc, they are scanned recursively for C and
+Ruby source files only.
+
+= Markup
+
+For information on how to make lists, hyperlinks, & etc. with RDoc, see
+RDoc::Markup.
+
+Comment blocks can be written fairly naturally, either using '#' on successive
+lines of the comment, or by including the comment in an =begin/=end block.  If
+you use the latter form, the =begin line must be flagged with an RDoc tag:
+
+  =begin rdoc
+  Documentation to be processed by RDoc.
+  
+  ...
+  =end
+
+RDoc stops processing comments if it finds a comment line containing '+#--+'.
+This can be used to separate external from internal comments, or to stop a
+comment being associated with a method, class, or module.  Commenting can be
+turned back on with a line that starts '+#+++'.
+
+  ##
+  # Extract the age and calculate the date-of-birth.
+  #--
+  # FIXME: fails if the birthday falls on February 29th
+  #++
+  # The DOB is returned as a Time object.
+  
+  def get_dob(person)
+    # ...
+  end
+
+Names of classes, source files, and any method names containing an underscore
+or preceded by a hash character are automatically hyperlinked from comment text
+to their description. 
+
+Method parameter lists are extracted and displayed with the method description.
+If a method calls +yield+, then the parameters passed to yield will also be
+displayed:
+
+  def fred
+    ...
+    yield line, address
+
+This will get documented as:
+
+  fred() { |line, address| ... }
+
+You can override this using a comment containing ':yields: ...' immediately
+after the method definition
+
+  def fred # :yields: index, position
+    # ...
+  
+    yield line, address
+
+which will get documented as
+
+   fred() { |index, position| ... }
+
++:yields:+ is an example of a documentation directive.  These appear immediately
+after the start of the document element they are modifying.
+
+== Directives
+
+[+:nodoc:+ / +:nodoc:+ all]
+  Don't include this element in the documentation.  For classes
+  and modules, the methods, aliases, constants, and attributes
+  directly within the affected class or module will also be
+  omitted.  By default, though, modules and classes within that
+  class of module _will_ be documented.  This is turned off by
+  adding the +all+ modifier.
+  
+    module MyModule # :nodoc:
+      class Input
+      end
+    end
+    
+    module OtherModule # :nodoc: all
+      class Output
+      end
+    end
+  
+  In the above code, only class +MyModule::Input+ will be documented.
+
+[+:doc:+]
+  Force a method or attribute to be documented even if it wouldn't otherwise
+  be.  Useful if, for example, you want to include documentation of a
+  particular private method.
+
+[+:notnew:+]
+  Only applicable to the +initialize+ instance method.  Normally RDoc assumes
+  that the documentation and parameters for #initialize are actually for the
+  ::new method, and so fakes out a ::new for the class.  The :notnew: modifier
+  stops this.  Remember that #initialize is protected, so you won't see the
+  documentation unless you use the -a command line option.
+
+Comment blocks can contain other directives:
+
+[+:section: title+]
+  Starts a new section in the output.  The title following +:section:+ is used
+  as the section heading, and the remainder of the comment containing the
+  section is used as introductory text.  Subsequent methods, aliases,
+  attributes, and classes will be documented in this section.  A :section:
+  comment block may have one or more lines before the :section: directive.
+  These will be removed, and any identical lines at the end of the block are
+  also removed.  This allows you to add visual cues such as:
+    
+    # ----------------------------------------
+    # :section: My Section
+    # This is the section that I wrote.
+    # See it glisten in the noon-day sun.
+    # ----------------------------------------
+
+[+:call-seq:+]
+  Lines up to the next blank line in the comment are treated as the method's
+  calling sequence, overriding the default parsing of method parameters and
+  yield arguments.
+
+[+:include:+ _filename_]
+  Include the contents of the named file at this point.  The file will be
+  searched for in the directories listed by the +--include+ option, or in the
+  current directory by default.  The contents of the file will be shifted to
+  have the same indentation as the ':' at the start of the :include: directive.
+
+[+:title:+ _text_]
+  Sets the title for the document.  Equivalent to the --title command line
+  parameter.  (The command line parameter overrides any :title: directive in
+  the source).
+
+[+:enddoc:+]
+  Document nothing further at the current level.
+
+[+:main:+ _name_]
+  Equivalent to the --main command line parameter.
+
+[+:stopdoc:+ / +:startdoc:+]
+  Stop and start adding new documentation elements to the current container.
+  For example, if a class has a number of constants that you don't want to
+  document, put a +:stopdoc:+ before the first, and a +:startdoc:+ after the
+  last.  If you don't specify a +:startdoc:+ by the end of the container,
+  disables documentation for the entire class or module.
+
+= Other stuff
+
+Author::   Dave Thomas <dave@pragmaticprogrammer.com>
+
+== Credits
+
+* The Ruby parser in rdoc/parse.rb is based heavily on the outstanding
+  work of Keiju ISHITSUKA of Nippon Rational Inc, who produced the Ruby
+  parser for irb and the rtags package.
+
+* Code to diagram classes and modules was written by Sergey A Yanovitsky
+  (Jah) of Enticla. 
+
+* Charset patch from MoonWolf.
+
+* Rich Kilmer wrote the kilmer.rb output template.
+
+* Dan Brickley led the design of the RDF format.
+
+== License
+
+RDoc is Copyright (c) 2001-2003 Dave Thomas, The Pragmatic Programmers.  It
+is free software, and may be redistributed under the terms specified
+in the README file of the Ruby distribution.
+
+== Warranty
+
+This software is provided "as is" and without any express or implied
+warranties, including, without limitation, the implied warranties of
+merchantibility and fitness for a particular purpose.
+
diff --git a/trunk/lib/rdoc/code_objects.rb b/trunk/lib/rdoc/code_objects.rb
new file mode 100644
index 0000000000..fbdb612b92
--- /dev/null
+++ b/trunk/lib/rdoc/code_objects.rb
@@ -0,0 +1,995 @@
+# We represent the various high-level code constructs that appear
+# in Ruby programs: classes, modules, methods, and so on.
+
+require 'rdoc/tokenstream'
+
+module RDoc
+
+  ##
+  # We contain the common stuff for contexts (which are containers) and other
+  # elements (methods, attributes and so on)
+
+  class CodeObject
+
+    attr_accessor :parent
+
+    # We are the model of the code, but we know that at some point
+    # we will be worked on by viewers. By implementing the Viewable
+    # protocol, viewers can associated themselves with these objects.
+
+    attr_accessor :viewer
+
+    # are we done documenting (ie, did we come across a :enddoc:)?
+
+    attr_accessor :done_documenting
+
+    # Which section are we in
+
+    attr_accessor :section
+
+    # do we document ourselves?
+
+    attr_reader :document_self
+
+    def initialize
+      @document_self = true
+      @document_children = true
+      @force_documentation = false
+      @done_documenting = false
+    end
+
+    def document_self=(val)
+      @document_self = val
+      if !val
+	remove_methods_etc
+      end
+    end
+
+    # set and cleared by :startdoc: and :enddoc:, this is used to toggle
+    # the capturing of documentation
+    def start_doc
+      @document_self = true
+      @document_children = true
+    end
+
+    def stop_doc
+      @document_self = false
+      @document_children = false
+    end
+
+    # do we document ourselves and our children
+
+    attr_reader :document_children
+
+    def document_children=(val)
+      @document_children = val
+      if !val
+	remove_classes_and_modules
+      end
+    end
+
+    # Do we _force_ documentation, even is we wouldn't normally show the entity
+    attr_accessor :force_documentation
+
+    def parent_file_name
+      @parent ? @parent.file_base_name : '(unknown)'
+    end
+
+    def parent_name
+      @parent ? @parent.name : '(unknown)'
+    end
+
+    # Default callbacks to nothing, but this is overridden for classes
+    # and modules
+    def remove_classes_and_modules
+    end
+
+    def remove_methods_etc
+    end
+
+    # Access the code object's comment
+    attr_reader :comment
+
+    # Update the comment, but don't overwrite a real comment with an empty one
+    def comment=(comment)
+      @comment = comment unless comment.empty?
+    end
+
+    # There's a wee trick we pull. Comment blocks can have directives that
+    # override the stuff we extract during the parse. So, we have a special
+    # class method, attr_overridable, that lets code objects list
+    # those directives. Wehn a comment is assigned, we then extract
+    # out any matching directives and update our object
+
+    def self.attr_overridable(name, *aliases)
+      @overridables ||= {}
+
+      attr_accessor name
+
+      aliases.unshift name
+      aliases.each do |directive_name|
+        @overridables[directive_name.to_s] = name
+      end
+    end
+
+  end
+
+  ##
+  # A Context is something that can hold modules, classes, methods,
+  # attributes, aliases, requires, and includes. Classes, modules, and files
+  # are all Contexts.
+
+  class Context < CodeObject
+
+    attr_reader :aliases
+    attr_reader :attributes
+    attr_reader :constants
+    attr_reader :current_section
+    attr_reader :in_files
+    attr_reader :includes
+    attr_reader :method_list
+    attr_reader :name
+    attr_reader :requires
+    attr_reader :sections
+    attr_reader :visibility
+
+    class Section
+      attr_reader :title, :comment, :sequence
+
+      @@sequence = "SEC00000"
+
+      def initialize(title, comment)
+        @title = title
+        @@sequence.succ!
+        @sequence = @@sequence.dup
+        @comment = nil
+        set_comment(comment)
+      end
+
+      def ==(other)
+        self.class === other and @sequence == other.sequence
+      end
+
+      def inspect
+        "#<%s:0x%x %s %p>" % [
+          self.class, object_id,
+          @sequence, title
+        ]
+      end
+
+      ##
+      # Set the comment for this section from the original comment block If
+      # the first line contains :section:, strip it and use the rest.
+      # Otherwise remove lines up to the line containing :section:, and look
+      # for those lines again at the end and remove them. This lets us write
+      #
+      #   # ---------------------
+      #   # :SECTION: The title
+      #   # The body
+      #   # ---------------------
+
+      def set_comment(comment)
+        return unless comment
+
+        if comment =~ /^#[ \t]*:section:.*\n/
+          start = $`
+          rest = $'
+
+          if start.empty?
+            @comment = rest
+          else
+            @comment = rest.sub(/#{start.chomp}\Z/, '')
+          end
+        else
+          @comment = comment
+        end
+        @comment = nil if @comment.empty?
+      end
+
+    end
+
+    def initialize
+      super
+
+      @in_files = []
+
+      @name    ||= "unknown"
+      @comment ||= ""
+      @parent  = nil
+      @visibility = :public
+
+      @current_section = Section.new(nil, nil)
+      @sections = [ @current_section ]
+
+      initialize_methods_etc
+      initialize_classes_and_modules
+    end
+
+    ##
+    # map the class hash to an array externally
+
+    def classes
+      @classes.values
+    end
+
+    ##
+    # map the module hash to an array externally
+
+    def modules
+      @modules.values
+    end
+
+    ##
+    # Change the default visibility for new methods
+
+    def ongoing_visibility=(vis)
+      @visibility = vis
+    end
+
+    ##
+    # Yields Method and Attr entries matching the list of names in +methods+.
+    # Attributes are only returned when +singleton+ is false.
+
+    def methods_matching(methods, singleton = false)
+      count = 0
+
+      @method_list.each do |m|
+        if methods.include? m.name and m.singleton == singleton then
+          yield m
+          count += 1
+        end
+      end
+
+      return if count == methods.size || singleton
+
+      # perhaps we need to look at attributes
+
+      @attributes.each do |a|
+        yield a if methods.include? a.name
+      end
+    end
+
+    ##
+    # Given an array +methods+ of method names, set the visibility of the
+    # corresponding AnyMethod object
+
+    def set_visibility_for(methods, vis, singleton = false)
+      methods_matching methods, singleton do |m|
+        m.visibility = vis
+      end
+    end
+
+    ##
+    # Record the file that we happen to find it in
+
+    def record_location(toplevel)
+      @in_files << toplevel unless @in_files.include?(toplevel)
+    end
+
+    # Return true if at least part of this thing was defined in +file+
+    def defined_in?(file)
+      @in_files.include?(file)
+    end
+
+    def add_class(class_type, name, superclass)
+      add_class_or_module @classes, class_type, name, superclass
+    end
+
+    def add_module(class_type, name)
+      add_class_or_module(@modules, class_type, name, nil)
+    end
+
+    def add_method(a_method)
+      a_method.visibility = @visibility
+      add_to(@method_list, a_method)
+    end
+
+    def add_attribute(an_attribute)
+      add_to(@attributes, an_attribute)
+    end
+
+    def add_alias(an_alias)
+      meth = find_instance_method_named(an_alias.old_name)
+
+      if meth then
+        new_meth = AnyMethod.new(an_alias.text, an_alias.new_name)
+        new_meth.is_alias_for = meth
+        new_meth.singleton    = meth.singleton
+        new_meth.params       = meth.params
+        new_meth.comment = "Alias for \##{meth.name}"
+        meth.add_alias(new_meth)
+        add_method(new_meth)
+      else
+        add_to(@aliases, an_alias)
+      end
+
+      an_alias
+    end
+
+    def add_include(an_include)
+      add_to(@includes, an_include)
+    end
+
+    def add_constant(const)
+      add_to(@constants, const)
+    end
+
+    # Requires always get added to the top-level (file) context
+    def add_require(a_require)
+      if TopLevel === self then
+        add_to @requires, a_require
+      else
+        parent.add_require a_require
+      end
+    end
+
+    def add_class_or_module(collection, class_type, name, superclass=nil)
+      cls = collection[name]
+
+      if cls then
+        cls.superclass = superclass unless cls.module?
+        puts "Reusing class/module #{name}" if $DEBUG_RDOC
+      else
+        cls = class_type.new(name, superclass)
+#        collection[name] = cls if @document_self  && !@done_documenting
+        collection[name] = cls if !@done_documenting
+        cls.parent = self
+        cls.section = @current_section
+      end
+      cls
+    end
+
+    def add_to(array, thing)
+      array << thing if @document_self and not @done_documenting
+      thing.parent = self
+      thing.section = @current_section
+    end
+
+    # If a class's documentation is turned off after we've started
+    # collecting methods etc., we need to remove the ones
+    # we have
+
+    def remove_methods_etc
+      initialize_methods_etc
+    end
+
+    def initialize_methods_etc
+      @method_list = []
+      @attributes  = []
+      @aliases     = []
+      @requires    = []
+      @includes    = []
+      @constants   = []
+    end
+
+    # and remove classes and modules when we see a :nodoc: all
+    def remove_classes_and_modules
+      initialize_classes_and_modules
+    end
+
+    def initialize_classes_and_modules
+      @classes     = {}
+      @modules     = {}
+    end
+
+    # Find a named module
+    def find_module_named(name)
+      return self if self.name == name
+      res = @modules[name] || @classes[name]
+      return res if res
+      find_enclosing_module_named(name)
+    end
+
+    # find a module at a higher scope
+    def find_enclosing_module_named(name)
+      parent && parent.find_module_named(name)
+    end
+
+    # Iterate over all the classes and modules in
+    # this object
+
+    def each_classmodule
+      @modules.each_value {|m| yield m}
+      @classes.each_value {|c| yield c}
+    end
+
+    def each_method
+      @method_list.each {|m| yield m}
+    end
+
+    def each_attribute 
+      @attributes.each {|a| yield a}
+    end
+
+    def each_constant
+      @constants.each {|c| yield c}
+    end
+
+    # Return the toplevel that owns us
+
+    def toplevel
+      return @toplevel if defined? @toplevel
+      @toplevel = self
+      @toplevel = @toplevel.parent until TopLevel === @toplevel
+      @toplevel
+    end
+
+    # allow us to sort modules by name
+    def <=>(other)
+      name <=> other.name
+    end
+
+    ##
+    # Look up +symbol+.  If +method+ is non-nil, then we assume the symbol
+    # references a module that contains that method.
+
+    def find_symbol(symbol, method = nil)
+      result = nil
+
+      case symbol
+      when /^::(.*)/ then
+        result = toplevel.find_symbol($1)
+      when /::/ then
+        modules = symbol.split(/::/)
+
+        unless modules.empty? then
+          module_name = modules.shift
+          result = find_module_named(module_name)
+          if result then
+            modules.each do |name|
+              result = result.find_module_named(name)
+              break unless result
+            end
+          end
+        end
+
+      else
+        # if a method is specified, then we're definitely looking for
+        # a module, otherwise it could be any symbol
+        if method
+          result = find_module_named(symbol)
+        else
+          result = find_local_symbol(symbol)
+          if result.nil?
+            if symbol =~ /^[A-Z]/
+              result = parent
+              while result && result.name != symbol
+                result = result.parent
+              end
+            end
+          end
+        end
+      end
+
+      if result and method then
+        fail unless result.respond_to? :find_local_symbol
+        result = result.find_local_symbol(method)
+      end
+
+      result
+    end
+
+    def find_local_symbol(symbol)
+      res = find_method_named(symbol) ||
+            find_constant_named(symbol) ||
+            find_attribute_named(symbol) ||
+            find_module_named(symbol) ||
+            find_file_named(symbol)
+    end
+
+    # Handle sections
+
+    def set_current_section(title, comment)
+      @current_section = Section.new(title, comment)
+      @sections << @current_section
+    end
+
+    private
+
+    # Find a named method, or return nil
+    def find_method_named(name)
+      @method_list.find {|meth| meth.name == name}
+    end
+
+    # Find a named instance method, or return nil
+    def find_instance_method_named(name)
+      @method_list.find {|meth| meth.name == name && !meth.singleton}
+    end
+
+    # Find a named constant, or return nil
+    def find_constant_named(name)
+      @constants.find {|m| m.name == name}
+    end
+
+    # Find a named attribute, or return nil
+    def find_attribute_named(name)
+      @attributes.find {|m| m.name == name}
+    end
+
+    ##
+    # Find a named file, or return nil
+
+    def find_file_named(name)
+      toplevel.class.find_file_named(name)
+    end
+
+  end
+
+  ##
+  # A TopLevel context is a source file
+
+  class TopLevel < Context
+    attr_accessor :file_stat
+    attr_accessor :file_relative_name
+    attr_accessor :file_absolute_name
+    attr_accessor :diagram
+
+    @@all_classes = {}
+    @@all_modules = {}
+    @@all_files   = {}
+
+    def self.reset
+      @@all_classes = {}
+      @@all_modules = {}
+      @@all_files   = {}
+    end
+
+    def initialize(file_name)
+      super()
+      @name = "TopLevel"
+      @file_relative_name    = file_name
+      @file_absolute_name    = file_name
+      @file_stat             = File.stat(file_name)
+      @diagram               = nil
+      @@all_files[file_name] = self
+    end
+
+    def file_base_name
+      File.basename @file_absolute_name
+    end
+
+    def full_name
+      nil
+    end
+
+    ##
+    # Adding a class or module to a TopLevel is special, as we only want one
+    # copy of a particular top-level class. For example, if both file A and
+    # file B implement class C, we only want one ClassModule object for C.
+    # This code arranges to share classes and modules between files.
+
+    def add_class_or_module(collection, class_type, name, superclass)
+      cls = collection[name]
+
+      if cls then
+        cls.superclass = superclass unless cls.module?
+        puts "Reusing class/module #{cls.full_name}" if $DEBUG_RDOC
+      else
+        if class_type == NormalModule then
+          all = @@all_modules
+        else
+          all = @@all_classes
+        end
+
+        cls = all[name]
+
+        unless cls then
+          cls = class_type.new name, superclass
+          all[name] = cls unless @done_documenting
+        end
+
+        collection[name] = cls unless @done_documenting
+
+        cls.parent = self
+      end
+
+      cls
+    end
+
+    def self.all_classes_and_modules
+      @@all_classes.values + @@all_modules.values
+    end
+
+    def self.find_class_named(name)
+     @@all_classes.each_value do |c|
+        res = c.find_class_named(name) 
+        return res if res
+      end
+      nil
+    end
+
+    def self.find_file_named(name)
+      @@all_files[name]
+    end
+
+    def find_local_symbol(symbol)
+      find_class_or_module_named(symbol) || super
+    end
+
+    def find_class_or_module_named(symbol)
+      @@all_classes.each_value {|c| return c if c.name == symbol}
+      @@all_modules.each_value {|m| return m if m.name == symbol}
+      nil
+    end
+
+    ##
+    # Find a named module
+
+    def find_module_named(name)
+      find_class_or_module_named(name) || find_enclosing_module_named(name)
+    end
+
+    def inspect
+      "#<%s:0x%x %p modules: %p classes: %p>" % [
+        self.class, object_id,
+        file_base_name,
+        @modules.map { |n,m| m },
+        @classes.map { |n,c| c }
+      ]
+    end
+
+  end
+
+  ##
+  # ClassModule is the base class for objects representing either a class or a
+  # module.
+
+  class ClassModule < Context
+
+    attr_accessor :diagram
+
+    def initialize(name, superclass = nil)
+      @name       = name
+      @diagram    = nil
+      @superclass = superclass
+      @comment    = ""
+      super()
+    end
+
+    def find_class_named(name)
+      return self if full_name == name
+      @classes.each_value {|c| return c if c.find_class_named(name) }
+      nil
+    end
+
+    ##
+    # Return the fully qualified name of this class or module
+
+    def full_name
+      if @parent && @parent.full_name
+        @parent.full_name + "::" + @name
+      else
+        @name
+      end
+    end
+
+    def http_url(prefix)
+      path = full_name.split("::")
+      File.join(prefix, *path) + ".html"
+    end
+
+    ##
+    # Does this object represent a module?
+
+    def module?
+      false
+    end
+
+    ##
+    # Get the superclass of this class.  Attempts to retrieve the superclass'
+    # real name by following module nesting.
+
+    def superclass
+      raise NoMethodError, "#{full_name} is a module" if module?
+
+      scope = self
+
+      begin
+        superclass = scope.classes.find { |c| c.name == @superclass }
+
+        return superclass.full_name if superclass
+        scope = scope.parent
+      end until scope.nil? or TopLevel === scope
+
+      @superclass
+    end
+
+    ##
+    # Set the superclass of this class
+
+    def superclass=(superclass)
+      raise NoMethodError, "#{full_name} is a module" if module?
+
+      if @superclass.nil? or @superclass == 'Object' then
+        @superclass = superclass 
+      end
+    end
+
+    def to_s
+      "#{self.class}: #{@name} #{@comment} #{super}"
+    end
+
+  end
+
+  ##
+  # Anonymous classes
+
+  class AnonClass < ClassModule
+  end
+
+  ##
+  # Normal classes
+
+  class NormalClass < ClassModule
+
+    def inspect
+      superclass = @superclass ? " < #{@superclass}" : nil
+      "<%s:0x%x class %s%s includes: %p attributes: %p methods: %p aliases: %p>" % [
+        self.class, object_id,
+        @name, superclass, @includes, @attributes, @method_list, @aliases
+      ]
+    end
+
+  end
+
+  ##
+  # Singleton classes
+
+  class SingleClass < ClassModule
+  end
+
+  ##
+  # Module
+
+  class NormalModule < ClassModule
+
+    def comment=(comment)
+      return if comment.empty?
+      comment = @comment << "# ---\n" << comment unless @comment.empty?
+
+      super
+    end
+
+    def inspect
+      "#<%s:0x%x module %s includes: %p attributes: %p methods: %p aliases: %p>" % [
+        self.class, object_id,
+        @name, @includes, @attributes, @method_list, @aliases
+      ]
+    end
+
+    def module?
+      true
+    end
+
+  end
+
+  ##
+  # AnyMethod is the base class for objects representing methods
+
+  class AnyMethod < CodeObject
+
+    attr_accessor :name
+    attr_accessor :visibility
+    attr_accessor :block_params
+    attr_accessor :dont_rename_initialize
+    attr_accessor :singleton
+    attr_reader :text
+
+    # list of other names for this method
+    attr_reader   :aliases
+
+    # method we're aliasing
+    attr_accessor :is_alias_for
+
+    attr_overridable :params, :param, :parameters, :parameter
+
+    attr_accessor :call_seq
+
+    include TokenStream
+
+    def initialize(text, name)
+      super()
+      @text = text
+      @name = name
+      @token_stream  = nil
+      @visibility    = :public
+      @dont_rename_initialize = false
+      @block_params  = nil
+      @aliases       = []
+      @is_alias_for  = nil
+      @comment = ""
+      @call_seq = nil
+    end
+
+    def <=>(other)
+      @name <=> other.name
+    end
+
+    def add_alias(method)
+      @aliases << method
+    end
+
+    def inspect
+      alias_for = @is_alias_for ? " (alias for #{@is_alias_for.name})" : nil
+      "#<%s:0x%x %s%s%s (%s)%s>" % [
+        self.class, object_id,
+        parent_name,
+        singleton ? '::' : '#',
+        name,
+        visibility,
+        alias_for,
+      ]
+    end
+
+    def param_seq
+      params = params.gsub(/\s*\#.*/, '')
+      params = params.tr("\n", " ").squeeze(" ")
+      params = "(#{params})" unless p[0] == ?(
+
+      if block = block_params then # yes, =
+        # If this method has explicit block parameters, remove any explicit
+        # &block
+        params.sub!(/,?\s*&\w+/)
+
+        block.gsub!(/\s*\#.*/, '')
+        block = block.tr("\n", " ").squeeze(" ")
+        if block[0] == ?(
+          block.sub!(/^\(/, '').sub!(/\)/, '')
+        end
+        params << " { |#{block}| ... }"
+      end
+
+      params
+    end
+
+    def to_s
+      res = self.class.name + ": " + @name + " (" + @text + ")\n"
+      res << @comment.to_s
+      res
+    end
+
+  end
+
+  ##
+  # GhostMethod represents a method referenced only by a comment
+
+  class GhostMethod < AnyMethod
+  end
+
+  ##
+  # MetaMethod represents a meta-programmed method
+
+  class MetaMethod < AnyMethod
+  end
+
+  ##
+  # Represent an alias, which is an old_name/ new_name pair associated with a
+  # particular context
+
+  class Alias < CodeObject
+
+    attr_accessor :text, :old_name, :new_name, :comment
+
+    def initialize(text, old_name, new_name, comment)
+      super()
+      @text = text
+      @old_name = old_name
+      @new_name = new_name
+      self.comment = comment
+    end
+
+    def inspect
+      "#<%s:0x%x %s.alias_method %s, %s>" % [
+        self.class, object_id,
+        parent.name, @old_name, @new_name,
+      ]
+    end
+
+    def to_s
+      "alias: #{self.old_name} ->  #{self.new_name}\n#{self.comment}"
+    end
+
+  end
+
+  ##
+  # Represent a constant
+
+  class Constant < CodeObject
+    attr_accessor :name, :value
+
+    def initialize(name, value, comment)
+      super()
+      @name = name
+      @value = value
+      self.comment = comment
+    end
+  end
+
+  ##
+  # Represent attributes
+
+  class Attr < CodeObject
+    attr_accessor :text, :name, :rw, :visibility
+
+    def initialize(text, name, rw, comment)
+      super()
+      @text = text
+      @name = name
+      @rw = rw
+      @visibility = :public
+      self.comment = comment
+    end
+
+    def <=>(other)
+      self.name <=> other.name
+    end
+
+    def inspect
+      attr = case rw
+             when 'RW' then :attr_accessor
+             when 'R'  then :attr_reader
+             when 'W'  then :attr_writer
+             else
+               " (#{rw})"
+             end
+
+      "#<%s:0x%x %s.%s :%s>" % [
+        self.class, object_id,
+        parent_name, attr, @name,
+      ]
+    end
+
+    def to_s
+      "attr: #{self.name} #{self.rw}\n#{self.comment}"
+    end
+
+  end
+
+  ##
+  # A required file
+
+  class Require < CodeObject
+    attr_accessor :name
+
+    def initialize(name, comment)
+      super()
+      @name = name.gsub(/'|"/, "") #'
+      self.comment = comment
+    end
+
+    def inspect
+      "#<%s:0x%x require '%s' in %s>" % [
+        self.class,
+        object_id,
+        @name,
+        parent_file_name,
+      ]
+    end
+
+  end
+
+  ##
+  # An included module
+
+  class Include < CodeObject
+
+    attr_accessor :name
+
+    def initialize(name, comment)
+      super()
+      @name = name
+      self.comment = comment
+
+    end
+
+    def inspect
+      "#<%s:0x%x %s.include %s>" % [
+        self.class,
+        object_id,
+        parent_name, @name,
+      ]
+    end
+
+  end
+
+end
diff --git a/trunk/lib/rdoc/diagram.rb b/trunk/lib/rdoc/diagram.rb
new file mode 100644
index 0000000000..e235e043dc
--- /dev/null
+++ b/trunk/lib/rdoc/diagram.rb
@@ -0,0 +1,338 @@
+# A wonderful hack by to draw package diagrams using the dot package.
+# Originally written by  Jah, team Enticla.
+#
+# You must have the V1.7 or later in your path
+# http://www.research.att.com/sw/tools/graphviz/
+
+require 'rdoc/dot'
+
+module RDoc
+
+  ##
+  # Draw a set of diagrams representing the modules and classes in the
+  # system. We draw one diagram for each file, and one for each toplevel
+  # class or module. This means there will be overlap. However, it also
+  # means that you'll get better context for objects.
+  #
+  # To use, simply
+  #
+  #   d = Diagram.new(info)   # pass in collection of top level infos
+  #   d.draw
+  #
+  # The results will be written to the +dot+ subdirectory. The process
+  # also sets the +diagram+ attribute in each object it graphs to
+  # the name of the file containing the image. This can be used
+  # by output generators to insert images.
+
+  class Diagram
+
+    FONT = "Arial"
+
+    DOT_PATH = "dot"
+
+    ##
+    # Pass in the set of top level objects. The method also creates the
+    # subdirectory to hold the images
+
+    def initialize(info, options)
+      @info = info
+      @options = options
+      @counter = 0
+      FileUtils.mkdir_p(DOT_PATH)
+      @diagram_cache = {}
+    end
+
+    ##
+    # Draw the diagrams. We traverse the files, drawing a diagram for each. We
+    # also traverse each top-level class and module in that file drawing a
+    # diagram for these too.
+
+    def draw
+      unless @options.quiet
+        $stderr.print "Diagrams: "
+        $stderr.flush
+      end
+
+      @info.each_with_index do |i, file_count|
+        @done_modules = {}
+        @local_names = find_names(i)
+        @global_names = []
+        @global_graph = graph = DOT::Digraph.new('name' => 'TopLevel',
+                                                 'fontname' => FONT,
+                                                 'fontsize' => '8',
+                                                 'bgcolor'  => 'lightcyan1',
+                                                 'compound' => 'true')
+
+        # it's a little hack %) i'm too lazy to create a separate class
+        # for default node
+        graph << DOT::Node.new('name' => 'node',
+                               'fontname' => FONT,
+                               'color' => 'black',
+                               'fontsize' => 8)
+
+        i.modules.each do |mod|
+          draw_module(mod, graph, true, i.file_relative_name)
+        end
+        add_classes(i, graph, i.file_relative_name)
+
+        i.diagram = convert_to_png("f_#{file_count}", graph)
+
+        # now go through and document each top level class and
+        # module independently
+        i.modules.each_with_index do |mod, count|
+          @done_modules = {}
+          @local_names = find_names(mod)
+          @global_names = []
+
+          @global_graph = graph = DOT::Digraph.new('name' => 'TopLevel',
+                                                   'fontname' => FONT,
+                                                   'fontsize' => '8',
+                                                   'bgcolor'  => 'lightcyan1',
+                                                   'compound' => 'true')
+
+          graph << DOT::Node.new('name' => 'node',
+                                 'fontname' => FONT,
+                                 'color' => 'black',
+                                 'fontsize' => 8)
+          draw_module(mod, graph, true)
+          mod.diagram = convert_to_png("m_#{file_count}_#{count}",
+                                       graph)
+        end
+      end
+      $stderr.puts unless @options.quiet
+    end
+
+    private
+
+    def find_names(mod)
+      return [mod.full_name] + mod.classes.collect{|cl| cl.full_name} +
+        mod.modules.collect{|m| find_names(m)}.flatten
+    end
+
+    def find_full_name(name, mod)
+      full_name = name.dup
+      return full_name if @local_names.include?(full_name)
+      mod_path = mod.full_name.split('::')[0..-2]
+      unless mod_path.nil?
+        until mod_path.empty?
+          full_name = mod_path.pop + '::' + full_name
+          return full_name if @local_names.include?(full_name)
+        end
+      end
+      return name
+    end
+
+    def draw_module(mod, graph, toplevel = false, file = nil)
+      return if  @done_modules[mod.full_name] and not toplevel
+
+      @counter += 1
+      url = mod.http_url("classes")
+      m = DOT::Subgraph.new('name' => "cluster_#{mod.full_name.gsub( /:/,'_' )}",
+                            'label' => mod.name,
+                            'fontname' => FONT,
+                            'color' => 'blue',
+                            'style' => 'filled',
+                            'URL'   => %{"#{url}"},
+                            'fillcolor' => toplevel ? 'palegreen1' : 'palegreen3')
+
+      @done_modules[mod.full_name] = m
+      add_classes(mod, m, file)
+      graph << m
+
+      unless mod.includes.empty?
+        mod.includes.each do |inc|
+          m_full_name = find_full_name(inc.name, mod)
+          if @local_names.include?(m_full_name)
+            @global_graph << DOT::Edge.new('from' => "#{m_full_name.gsub( /:/,'_' )}",
+                                           'to' => "#{mod.full_name.gsub( /:/,'_' )}",
+                                           'ltail' => "cluster_#{m_full_name.gsub( /:/,'_' )}",
+                                           'lhead' => "cluster_#{mod.full_name.gsub( /:/,'_' )}")
+          else
+            unless @global_names.include?(m_full_name)
+              path = m_full_name.split("::")
+              url = File.join('classes', *path) + ".html"
+              @global_graph << DOT::Node.new('name' => "#{m_full_name.gsub( /:/,'_' )}",
+                                             'shape' => 'box',
+                                             'label' => "#{m_full_name}",
+                                             'URL'   => %{"#{url}"})
+              @global_names << m_full_name
+            end
+            @global_graph << DOT::Edge.new('from' => "#{m_full_name.gsub( /:/,'_' )}",
+                                           'to' => "#{mod.full_name.gsub( /:/,'_' )}",
+                                           'lhead' => "cluster_#{mod.full_name.gsub( /:/,'_' )}")
+          end
+        end
+      end
+    end
+
+    def add_classes(container, graph, file = nil )
+
+      use_fileboxes = @options.fileboxes
+
+      files = {}
+
+      # create dummy node (needed if empty and for module includes)
+      if container.full_name
+        graph << DOT::Node.new('name'     => "#{container.full_name.gsub( /:/,'_' )}",
+                               'label'    => "",
+                               'width'  => (container.classes.empty? and
+                                            container.modules.empty?) ?
+                               '0.75' : '0.01',
+                               'height' => '0.01',
+                               'shape' => 'plaintext')
+      end
+
+      container.classes.each_with_index do |cl, cl_index|
+        last_file = cl.in_files[-1].file_relative_name
+
+        if use_fileboxes && !files.include?(last_file)
+          @counter += 1
+          files[last_file] =
+            DOT::Subgraph.new('name'     => "cluster_#{@counter}",
+                                 'label'    => "#{last_file}",
+                                 'fontname' => FONT,
+                                 'color'=>
+                                 last_file == file ? 'red' : 'black')
+        end
+
+        next if cl.name == 'Object' || cl.name[0,2] == "<<"
+
+        url = cl.http_url("classes")
+
+        label = cl.name.dup
+        if use_fileboxes && cl.in_files.length > 1
+          label <<  '\n[' +
+                        cl.in_files.collect {|i|
+                             i.file_relative_name
+                        }.sort.join( '\n' ) +
+                    ']'
+        end
+
+        attrs = {
+          'name' => "#{cl.full_name.gsub( /:/, '_' )}",
+          'fontcolor' => 'black',
+          'style'=>'filled',
+          'color'=>'palegoldenrod',
+          'label' => label,
+          'shape' => 'ellipse',
+          'URL'   => %{"#{url}"}
+        }
+
+        c = DOT::Node.new(attrs)
+
+        if use_fileboxes
+          files[last_file].push c
+        else
+          graph << c
+        end
+      end
+
+      if use_fileboxes
+        files.each_value do |val|
+          graph << val
+        end
+      end
+
+      unless container.classes.empty?
+        container.classes.each_with_index do |cl, cl_index|
+          cl.includes.each do |m|
+            m_full_name = find_full_name(m.name, cl)
+            if @local_names.include?(m_full_name)
+              @global_graph << DOT::Edge.new('from' => "#{m_full_name.gsub( /:/,'_' )}",
+                                             'to' => "#{cl.full_name.gsub( /:/,'_' )}",
+                                             'ltail' => "cluster_#{m_full_name.gsub( /:/,'_' )}")
+            else
+              unless @global_names.include?(m_full_name)
+                path = m_full_name.split("::")
+                url = File.join('classes', *path) + ".html"
+                @global_graph << DOT::Node.new('name' => "#{m_full_name.gsub( /:/,'_' )}",
+                                               'shape' => 'box',
+                                               'label' => "#{m_full_name}",
+                                               'URL'   => %{"#{url}"})
+                @global_names << m_full_name
+              end
+              @global_graph << DOT::Edge.new('from' => "#{m_full_name.gsub( /:/,'_' )}",
+                                             'to' => "#{cl.full_name.gsub( /:/, '_')}")
+            end
+          end
+
+          sclass = cl.superclass
+          next if sclass.nil? || sclass == 'Object'
+          sclass_full_name = find_full_name(sclass,cl)
+          unless @local_names.include?(sclass_full_name) or @global_names.include?(sclass_full_name)
+            path = sclass_full_name.split("::")
+            url = File.join('classes', *path) + ".html"
+            @global_graph << DOT::Node.new('name' => "#{sclass_full_name.gsub( /:/, '_' )}",
+                                           'label' => sclass_full_name,
+                                           'URL'   => %{"#{url}"})
+            @global_names << sclass_full_name
+          end
+          @global_graph << DOT::Edge.new('from' => "#{sclass_full_name.gsub( /:/,'_' )}",
+                                         'to' => "#{cl.full_name.gsub( /:/, '_')}")
+        end
+      end
+
+      container.modules.each do |submod|
+        draw_module(submod, graph)
+      end
+
+    end
+
+    def convert_to_png(file_base, graph)
+      str = graph.to_s
+      return @diagram_cache[str] if @diagram_cache[str]
+      op_type = @options.image_format
+      dotfile = File.join(DOT_PATH, file_base)
+      src = dotfile + ".dot"
+      dot = dotfile + "." + op_type
+
+      unless @options.quiet
+        $stderr.print "."
+        $stderr.flush
+      end
+
+      File.open(src, 'w+' ) do |f|
+        f << str << "\n"
+      end
+
+      system "dot", "-T#{op_type}", src, "-o", dot
+
+      # Now construct the imagemap wrapper around
+      # that png
+
+      ret = wrap_in_image_map(src, dot)
+      @diagram_cache[str] = ret
+      return ret
+    end
+
+    ##
+    # Extract the client-side image map from dot, and use it to generate the
+    # imagemap proper. Return the whole <map>..<img> combination, suitable for
+    # inclusion on the page
+
+    def wrap_in_image_map(src, dot)
+      res = %{<map id="map" name="map">\n}
+      dot_map = `dot -Tismap #{src}`
+      dot_map.split($/).each do |area|
+        unless area =~ /^rectangle \((\d+),(\d+)\) \((\d+),(\d+)\) ([\/\w.]+)\s*(.*)/
+          $stderr.puts "Unexpected output from dot:\n#{area}"
+          return nil
+        end
+
+        xs, ys = [$1.to_i, $3.to_i], [$2.to_i, $4.to_i]
+        url, area_name = $5, $6
+
+        res <<  %{  <area shape="rect" coords="#{xs.min},#{ys.min},#{xs.max},#{ys.max}" }
+        res <<  %{     href="#{url}" alt="#{area_name}" />\n}
+      end
+      res << "</map>\n"
+#      map_file = src.sub(/.dot/, '.map')
+#      system("dot -Timap #{src} -o #{map_file}")
+      res << %{<img src="#{dot}" usemap="#map" border="0" alt="#{dot}">}
+      return res
+    end
+
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/dot.rb b/trunk/lib/rdoc/dot.rb
new file mode 100644
index 0000000000..fbd2cfba02
--- /dev/null
+++ b/trunk/lib/rdoc/dot.rb
@@ -0,0 +1,249 @@
+module RDoc; end
+
+module RDoc::DOT
+
+  TAB = '  '
+  TAB2 = TAB * 2
+
+  # options for node declaration
+  NODE_OPTS = [
+    'bgcolor',
+    'color',
+    'fontcolor',
+    'fontname',
+    'fontsize',
+    'height',
+    'width',
+    'label',
+    'layer',
+    'rank',
+    'shape',
+    'shapefile',
+    'style',
+    'URL',
+  ]
+
+  # options for edge declaration
+  EDGE_OPTS = [
+    'color',
+    'decorate',
+    'dir',
+    'fontcolor',
+    'fontname',
+    'fontsize',
+    'id',
+    'label',
+    'layer',
+    'lhead',
+    'ltail',
+    'minlen',
+    'style',
+    'weight'
+  ]
+
+  # options for graph declaration
+  GRAPH_OPTS = [
+    'bgcolor',
+    'center',
+    'clusterrank',
+    'color',
+    'compound',
+    'concentrate',
+    'fillcolor',
+    'fontcolor',
+    'fontname',
+    'fontsize',
+    'label',
+    'layerseq',
+    'margin',
+    'mclimit',
+    'nodesep',
+    'nslimit',
+    'ordering',
+    'orientation',
+    'page',
+    'rank',
+    'rankdir',
+    'ranksep',
+    'ratio',
+    'size',
+    'style',
+    'URL'
+  ]
+
+  # a root class for any element in dot notation
+  class SimpleElement
+    attr_accessor :name
+
+    def initialize( params = {} )
+      @label = params['name'] ? params['name'] : ''
+    end
+
+    def to_s
+      @name
+    end
+  end
+
+  # an element that has options ( node, edge or graph )
+  class Element < SimpleElement
+    #attr_reader :parent
+    attr_accessor :name, :options
+
+    def initialize( params = {}, option_list = [] )
+      super( params )
+      @name = params['name'] ? params['name'] : nil
+      @parent = params['parent'] ? params['parent'] : nil
+      @options = {}
+      option_list.each{ |i|
+        @options[i] = params[i] if params[i]
+      }
+      @options['label'] ||= @name if @name != 'node'
+    end
+
+    def each_option
+      @options.each{ |i| yield i }
+    end
+
+    def each_option_pair
+      @options.each_pair{ |key, val| yield key, val }
+    end
+
+    #def parent=( thing )
+    #    @parent.delete( self ) if defined?( @parent ) and @parent
+    #    @parent = thing
+    #end
+  end
+
+
+  # this is used when we build nodes that have shape=record
+  # ports don't have options :)
+  class Port < SimpleElement
+    attr_accessor :label
+
+    def initialize( params = {} )
+      super( params )
+      @name = params['label'] ? params['label'] : ''
+    end
+    def to_s
+      ( @name && @name != "" ? "<#{@name}>" : "" ) + "#{@label}"
+    end
+  end
+
+  # node element
+  class Node < Element
+
+    def initialize( params = {}, option_list = NODE_OPTS )
+      super( params, option_list )
+      @ports = params['ports'] ? params['ports'] : []
+    end
+
+    def each_port
+      @ports.each{ |i| yield i }
+    end
+
+    def << ( thing )
+      @ports << thing
+    end
+
+    def push ( thing )
+      @ports.push( thing )
+    end
+
+    def pop
+      @ports.pop
+    end
+
+    def to_s( t = '' )
+
+      label = @options['shape'] != 'record' && @ports.length == 0 ?
+        @options['label'] ?
+        t + TAB + "label = \"#{@options['label']}\"\n" :
+                    '' :
+                    t + TAB + 'label = "' + " \\\n" +
+                    t + TAB2 + "#{@options['label']}| \\\n" +
+                    @ports.collect{ |i|
+        t + TAB2 + i.to_s
+      }.join( "| \\\n" ) + " \\\n" +
+        t + TAB + '"' + "\n"
+
+        t + "#{@name} [\n" +
+        @options.to_a.collect{ |i|
+        i[1] && i[0] != 'label' ?
+          t + TAB + "#{i[0]} = #{i[1]}" : nil
+      }.compact.join( ",\n" ) + ( label != '' ? ",\n" : "\n" ) +
+        label +
+        t + "]\n"
+    end
+  end
+
+  # subgraph element is the same to graph, but has another header in dot
+  # notation
+  class Subgraph < Element
+
+    def initialize( params = {}, option_list = GRAPH_OPTS )
+      super( params, option_list )
+      @nodes = params['nodes'] ? params['nodes'] : []
+      @dot_string = 'subgraph'
+    end
+
+    def each_node
+      @nodes.each{ |i| yield i }
+    end
+
+    def << ( thing )
+      @nodes << thing
+    end
+
+    def push( thing )
+      @nodes.push( thing )
+    end
+
+    def pop
+      @nodes.pop
+    end
+
+    def to_s( t = '' )
+      hdr = t + "#{@dot_string} #{@name} {\n"
+
+      options = @options.to_a.collect{ |name, val|
+        val && name != 'label' ?
+          t + TAB + "#{name} = #{val}" :
+        name ? t + TAB + "#{name} = \"#{val}\"" : nil
+      }.compact.join( "\n" ) + "\n"
+
+      nodes = @nodes.collect{ |i|
+        i.to_s( t + TAB )
+      }.join( "\n" ) + "\n"
+      hdr + options + nodes + t + "}\n"
+    end
+  end
+
+  # this is graph
+  class Digraph < Subgraph
+    def initialize( params = {}, option_list = GRAPH_OPTS )
+      super( params, option_list )
+      @dot_string = 'digraph'
+    end
+  end
+
+  # this is edge
+  class Edge < Element
+    attr_accessor :from, :to
+    def initialize( params = {}, option_list = EDGE_OPTS )
+      super( params, option_list )
+      @from = params['from'] ? params['from'] : nil
+      @to = params['to'] ? params['to'] : nil
+    end
+
+    def to_s( t = '' )
+      t + "#{@from} -> #{to} [\n" +
+        @options.to_a.collect{ |i|
+        i[1] && i[0] != 'label' ?
+          t + TAB + "#{i[0]} = #{i[1]}" :
+        i[1] ? t + TAB + "#{i[0]} = \"#{i[1]}\"" : nil
+      }.compact.join( "\n" ) + "\n" + t + "]\n"
+    end
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/generator.rb b/trunk/lib/rdoc/generator.rb
new file mode 100644
index 0000000000..fbc08c4e20
--- /dev/null
+++ b/trunk/lib/rdoc/generator.rb
@@ -0,0 +1,1076 @@
+require 'cgi'
+require 'rdoc'
+require 'rdoc/options'
+require 'rdoc/markup/to_html_crossref'
+require 'rdoc/template'
+
+module RDoc::Generator
+
+  ##
+  # Name of sub-directory that holds file descriptions
+
+  FILE_DIR  = "files"
+
+  ##
+  # Name of sub-directory that holds class descriptions
+
+  CLASS_DIR = "classes"
+
+  ##
+  # Name of the RDoc CSS file
+
+  CSS_NAME  = "rdoc-style.css"
+
+  ##
+  # Build a hash of all items that can be cross-referenced.  This is used when
+  # we output required and included names: if the names appear in this hash,
+  # we can generate an html cross reference to the appropriate description.
+  # We also use this when parsing comment blocks: any decorated words matching
+  # an entry in this list are hyperlinked.
+
+  class AllReferences
+    @@refs = {}
+
+    def AllReferences::reset
+      @@refs = {}
+    end
+
+    def AllReferences.add(name, html_class)
+      @@refs[name] = html_class
+    end
+
+    def AllReferences.[](name)
+      @@refs[name]
+    end
+
+    def AllReferences.keys
+      @@refs.keys
+    end
+  end
+
+  ##
+  # Handle common markup tasks for the various Context subclasses
+
+  module MarkUp
+
+    ##
+    # Convert a string in markup format into HTML.
+
+    def markup(str, remove_para = false)
+      return '' unless str
+
+      # Convert leading comment markers to spaces, but only if all non-blank
+      # lines have them
+      if str =~ /^(?>\s*)[^\#]/ then
+        content = str
+      else
+        content = str.gsub(/^\s*(#+)/) { $1.tr '#', ' ' }
+      end
+
+      res = formatter.convert content
+
+      if remove_para then
+        res.sub!(/^<p>/, '')
+        res.sub!(/<\/p>$/, '')
+      end
+
+      res
+    end
+
+    ##
+    # Qualify a stylesheet URL; if if +css_name+ does not begin with '/' or
+    # 'http[s]://', prepend a prefix relative to +path+. Otherwise, return it
+    # unmodified.
+
+    def style_url(path, css_name=nil)
+#      $stderr.puts "style_url( #{path.inspect}, #{css_name.inspect} )"
+      css_name ||= CSS_NAME
+      if %r{^(https?:/)?/} =~ css_name
+        css_name
+      else
+        RDoc::Markup::ToHtml.gen_relative_url path, css_name
+      end
+    end
+
+    ##
+    # Build a webcvs URL with the given 'url' argument. URLs with a '%s' in them
+    # get the file's path sprintfed into them; otherwise they're just catenated
+    # together.
+
+    def cvs_url(url, full_path)
+      if /%s/ =~ url
+        return sprintf( url, full_path )
+      else
+        return url + full_path
+      end
+    end
+
+  end
+
+  ##
+  # A Context is built by the parser to represent a container: contexts hold
+  # classes, modules, methods, require lists and include lists.  ClassModule
+  # and TopLevel are the context objects we process here
+
+  class Context
+
+    include MarkUp
+
+    attr_reader :context
+
+    ##
+    # Generate:
+    #
+    # * a list of RDoc::Generator::File objects for each TopLevel object
+    # * a list of RDoc::Generator::Class objects for each first level class or
+    #   module in the TopLevel objects
+    # * a complete list of all hyperlinkable terms (file, class, module, and
+    #   method names)
+
+    def self.build_indicies(toplevels, options)
+      files = []
+      classes = []
+
+      toplevels.each do |toplevel|
+        files << RDoc::Generator::File.new(toplevel, options,
+                                           RDoc::Generator::FILE_DIR)
+      end
+
+      RDoc::TopLevel.all_classes_and_modules.each do |cls|
+        build_class_list(classes, options, cls, files[0], 
+                         RDoc::Generator::CLASS_DIR)
+      end
+
+      return files, classes
+    end
+
+    def self.build_class_list(classes, options, from, html_file, class_dir)
+      classes << RDoc::Generator::Class.new(from, html_file, class_dir, options)
+
+      from.each_classmodule do |mod|
+        build_class_list(classes, options, mod, html_file, class_dir)
+      end
+    end
+
+    def initialize(context, options)
+      @context = context
+      @options = options
+
+      # HACK ugly
+      @template = options.template_class
+    end
+
+    def formatter
+      @formatter ||= @options.formatter ||
+        RDoc::Markup::ToHtmlCrossref.new(path, self, @options.show_hash)
+    end
+
+    ##
+    # convenience method to build a hyperlink
+
+    def href(link, cls, name)
+      %{<a href="#{link}" class="#{cls}">#{name}</a>} #"
+    end
+
+    ##
+    # Returns a reference to outselves to be used as an href= the form depends
+    # on whether we're all in one file or in multiple files
+
+    def as_href(from_path)
+      if @options.all_one_file
+        "#" + path
+      else
+        RDoc::Markup::ToHtml.gen_relative_url from_path, path
+      end
+    end
+
+    ##
+    # Create a list of Method objects for each method in the corresponding
+    # context object. If the @options.show_all variable is set (corresponding
+    # to the <tt>--all</tt> option, we include all methods, otherwise just the
+    # public ones.
+
+    def collect_methods
+      list = @context.method_list
+
+      unless @options.show_all then
+        list = list.select do |m|
+          m.visibility == :public or
+            m.visibility == :protected or
+            m.force_documentation
+        end
+      end
+
+      @methods = list.collect do |m|
+        RDoc::Generator::Method.new m, self, @options
+      end
+    end
+
+    ##
+    # Build a summary list of all the methods in this context
+
+    def build_method_summary_list(path_prefix = "")
+      collect_methods unless @methods
+
+      @methods.sort.map do |meth|
+        {
+          "name" => CGI.escapeHTML(meth.name),
+          "aref" => "#{path_prefix}\##{meth.aref}"
+        }
+      end
+    end
+
+    ##
+    # Build a list of aliases for which we couldn't find a
+    # corresponding method
+
+    def build_alias_summary_list(section)
+      @context.aliases.map do |al|
+        next unless al.section == section
+
+        res = {
+          'old_name' => al.old_name,
+          'new_name' => al.new_name,
+        }
+
+        if al.comment and not al.comment.empty? then
+          res['desc'] = markup al.comment, true
+        end
+
+        res
+      end.compact
+    end
+
+    ##
+    # Build a list of constants
+
+    def build_constants_summary_list(section)
+      @context.constants.map do |co|
+        next unless co.section == section
+
+        res = {
+          'name'  => co.name,
+          'value' => CGI.escapeHTML(co.value)
+        }
+
+        if co.comment and not co.comment.empty? then
+          res['desc'] = markup co.comment, true
+        end
+
+        res
+      end.compact
+    end
+
+    def build_requires_list(context)
+      potentially_referenced_list(context.requires) {|fn| [fn + ".rb"] }
+    end
+
+    def build_include_list(context)
+      potentially_referenced_list(context.includes)
+    end
+
+    ##
+    # Build a list from an array of Context items. Look up each in the
+    # AllReferences hash: if we find a corresponding entry, we generate a
+    # hyperlink to it, otherwise just output the name.  However, some names
+    # potentially need massaging. For example, you may require a Ruby file
+    # without the .rb extension, but the file names we know about may have it.
+    # To deal with this, we pass in a block which performs the massaging,
+    # returning an array of alternative names to match
+
+    def potentially_referenced_list(array)
+      res = []
+      array.each do |i|
+        ref = AllReferences[i.name]
+#         if !ref
+#           container = @context.parent
+#           while !ref && container
+#             name = container.name + "::" + i.name
+#             ref = AllReferences[name]
+#             container = container.parent
+#           end
+#         end
+
+        ref = @context.find_symbol(i.name)
+        ref = ref.viewer if ref
+
+        if !ref && block_given?
+          possibles = yield(i.name)
+          while !ref and !possibles.empty?
+            ref = AllReferences[possibles.shift]
+          end
+        end
+        h_name = CGI.escapeHTML(i.name)
+        if ref and ref.document_self
+          path = url(ref.path)
+          res << { "name" => h_name, "aref" => path }
+        else
+          res << { "name" => h_name }
+        end
+      end
+      res
+    end
+
+    ##
+    # Build an array of arrays of method details. The outer array has up
+    # to six entries, public, private, and protected for both class
+    # methods, the other for instance methods. The inner arrays contain
+    # a hash for each method
+
+    def build_method_detail_list(section)
+      outer = []
+
+      methods = @methods.sort.select do |m|
+        m.document_self and m.section == section
+      end
+
+      for singleton in [true, false]
+        for vis in [ :public, :protected, :private ]
+          res = []
+          methods.each do |m|
+            next unless m.visibility == vis and m.singleton == singleton
+
+            row = {}
+
+            if m.call_seq then
+              row["callseq"] = m.call_seq.gsub(/->/, '&rarr;')
+            else
+              row["name"]        = CGI.escapeHTML(m.name)
+              row["params"]      = m.params
+            end
+
+            desc = m.description.strip
+            row["m_desc"]      = desc unless desc.empty?
+            row["aref"]        = m.aref
+            row["visibility"]  = m.visibility.to_s
+
+            alias_names = []
+
+            m.aliases.each do |other|
+              if other.viewer then # won't be if the alias is private
+                alias_names << {
+                  'name' => other.name,
+                  'aref'  => other.viewer.as_href(path)
+                }
+              end
+            end
+
+            row["aka"] = alias_names unless alias_names.empty?
+
+            if @options.inline_source then
+              code = m.source_code
+              row["sourcecode"] = code if code
+            else
+              code = m.src_url
+              if code then
+                row["codeurl"] = code
+                row["imgurl"]  = m.img_url
+              end
+            end
+
+            res << row
+          end
+
+          if res.size > 0 then
+            outer << {
+              "type"     => vis.to_s.capitalize,
+              "category" => singleton ? "Class" : "Instance",
+              "methods"  => res
+            }
+          end
+        end
+      end
+
+      outer
+    end
+
+    ##
+    # Build the structured list of classes and modules contained
+    # in this context.
+
+    def build_class_list(level, from, section, infile=nil)
+      prefix = '&nbsp;&nbsp;::' * level;
+      res = ''
+
+      from.modules.sort.each do |mod|
+        next unless mod.section == section
+        next if infile && !mod.defined_in?(infile)
+        if mod.document_self
+          res <<
+            prefix <<
+            'Module ' <<
+            href(url(mod.viewer.path), 'link', mod.full_name) <<
+            "<br />\n" <<
+            build_class_list(level + 1, mod, section, infile)
+        end
+      end
+
+      from.classes.sort.each do |cls|
+        next unless cls.section == section
+        next if infile and not cls.defined_in?(infile)
+
+        if cls.document_self
+          res <<
+            prefix <<
+            'Class ' <<
+            href(url(cls.viewer.path), 'link', cls.full_name) <<
+            "<br />\n" <<
+            build_class_list(level + 1, cls, section, infile)
+        end
+      end
+
+      res
+    end
+
+    def url(target)
+      RDoc::Markup::ToHtml.gen_relative_url path, target
+    end
+
+    def aref_to(target)
+      if @options.all_one_file
+        "#" + target
+      else
+        url(target)
+      end
+    end
+
+    def document_self
+      @context.document_self
+    end
+
+    def diagram_reference(diagram)
+      res = diagram.gsub(/((?:src|href)=")(.*?)"/) {
+        $1 + url($2) + '"'
+      }
+      res
+    end
+
+    ##
+    # Find a symbol in ourselves or our parent
+
+    def find_symbol(symbol, method=nil)
+      res = @context.find_symbol(symbol, method)
+      if res
+        res = res.viewer
+      end
+      res
+    end
+
+    ##
+    # create table of contents if we contain sections
+
+    def add_table_of_sections
+      toc = []
+      @context.sections.each do |section|
+        if section.title then
+          toc << {
+            'secname' => section.title,
+            'href'    => section.sequence
+          }
+        end
+      end
+
+      @values['toc'] = toc unless toc.empty?
+    end
+
+  end
+
+  ##
+  # Wrap a ClassModule context
+
+  class Class < Context
+
+    attr_reader :methods
+    attr_reader :path
+    attr_reader :values
+
+    def initialize(context, html_file, prefix, options)
+      super context, options
+
+      @html_file = html_file
+      @html_class = self
+      @is_module = context.module?
+      @values    = {}
+
+      context.viewer = self
+
+      if options.all_one_file
+        @path = context.full_name
+      else
+        @path = http_url(context.full_name, prefix)
+      end
+
+      collect_methods
+
+      AllReferences.add(name, self)
+    end
+
+    ##
+    # Returns the relative file name to store this class in, which is also its
+    # url
+
+    def http_url(full_name, prefix)
+      path = full_name.dup
+
+      path.gsub!(/<<\s*(\w*)/, 'from-\1') if path['<<']
+
+      ::File.join(prefix, path.split("::")) + ".html"
+    end
+
+    def name
+      @context.full_name
+    end
+
+    def parent_name
+      @context.parent.full_name
+    end
+
+    def index_name
+      name
+    end
+
+    def write_on(f, file_list, class_list, method_list, overrides = {})
+      value_hash
+
+      @values['file_list'] = file_list
+      @values['class_list'] = class_list
+      @values['method_list'] = method_list
+
+      @values.update overrides
+
+      template = RDoc::TemplatePage.new(@template::BODY,
+                                        @template::CLASS_PAGE,
+                                        @template::METHOD_LIST)
+
+      template.write_html_on(f, @values)
+    end
+
+    def value_hash
+      class_attribute_values
+      add_table_of_sections
+
+      @values["charset"] = @options.charset
+      @values["style_url"] = style_url(path, @options.css)
+
+      d = markup(@context.comment)
+      @values["description"] = d unless d.empty?
+
+      ml = build_method_summary_list @path
+      @values["methods"] = ml unless ml.empty?
+
+      il = build_include_list @context
+      @values["includes"] = il unless il.empty?
+
+      @values["sections"] = @context.sections.map do |section|
+        secdata = {
+          "sectitle" => section.title,
+          "secsequence" => section.sequence,
+          "seccomment" => markup(section.comment),
+        }
+
+        al = build_alias_summary_list section
+        secdata["aliases"] = al unless al.empty?
+
+        co = build_constants_summary_list section
+        secdata["constants"] = co unless co.empty?
+
+        al = build_attribute_list section
+        secdata["attributes"] = al unless al.empty?
+
+        cl = build_class_list 0, @context, section
+        secdata["classlist"] = cl unless cl.empty?
+
+        mdl = build_method_detail_list section
+        secdata["method_list"] = mdl unless mdl.empty?
+
+        secdata
+      end
+
+      @values
+    end
+
+    def build_attribute_list(section)
+      @context.attributes.sort.map do |att|
+        next unless att.section == section
+
+        if att.visibility == :public or att.visibility == :protected or
+           @options.show_all then
+
+          entry = {
+            "name"   => CGI.escapeHTML(att.name),
+            "rw"     => att.rw,
+            "a_desc" => markup(att.comment, true)
+          }
+
+          unless att.visibility == :public or att.visibility == :protected then
+            entry["rw"] << "-"
+          end
+
+          entry
+        end
+      end.compact
+    end
+
+    def class_attribute_values
+      h_name = CGI.escapeHTML(name)
+
+      @values["path"]      = @path
+      @values["classmod"]  = @is_module ? "Module" : "Class"
+      @values["title"]     = "#{@values['classmod']}: #{h_name}"
+
+      c = @context
+      c = c.parent while c and not c.diagram
+
+      if c and c.diagram then
+        @values["diagram"] = diagram_reference(c.diagram)
+      end
+
+      @values["full_name"] = h_name
+
+      if not @context.module? and @context.superclass then
+        parent_class = @context.superclass
+        @values["parent"] = CGI.escapeHTML(parent_class)
+
+        if parent_name
+          lookup = parent_name + "::" + parent_class
+        else
+          lookup = parent_class
+        end
+
+        parent_url = AllReferences[lookup] || AllReferences[parent_class]
+
+        if parent_url and parent_url.document_self
+          @values["par_url"] = aref_to(parent_url.path)
+        end
+      end
+
+      files = []
+      @context.in_files.each do |f|
+        res = {}
+        full_path = CGI.escapeHTML(f.file_absolute_name)
+
+        res["full_path"]     = full_path
+        res["full_path_url"] = aref_to(f.viewer.path) if f.document_self
+
+        if @options.webcvs
+          res["cvsurl"] = cvs_url( @options.webcvs, full_path )
+        end
+
+        files << res
+      end
+
+      @values['infiles'] = files
+    end
+
+    def <=>(other)
+      self.name <=> other.name
+    end
+
+  end
+
+  ##
+  # Handles the mapping of a file's information to HTML. In reality, a file
+  # corresponds to a +TopLevel+ object, containing modules, classes, and
+  # top-level methods. In theory it _could_ contain attributes and aliases,
+  # but we ignore these for now.
+
+  class File < Context
+
+    attr_reader :path
+    attr_reader :name
+    attr_reader :values
+
+    def initialize(context, options, file_dir)
+      super context, options
+
+      @values = {}
+
+      if options.all_one_file
+        @path = filename_to_label
+      else
+        @path = http_url(file_dir)
+      end
+
+      @name = @context.file_relative_name
+
+      collect_methods
+      AllReferences.add(name, self)
+      context.viewer = self
+    end
+
+    def http_url(file_dir)
+      ::File.join file_dir, "#{@context.file_relative_name.tr '.', '_'}.html"
+    end
+
+    def filename_to_label
+      @context.file_relative_name.gsub(/%|\/|\?|\#/) do
+        '%%%x' % $&[0].unpack('C')
+      end
+    end
+
+    def index_name
+      name
+    end
+
+    def parent_name
+      nil
+    end
+
+    def value_hash
+      file_attribute_values
+      add_table_of_sections
+
+      @values["charset"]   = @options.charset
+      @values["href"]      = path
+      @values["style_url"] = style_url(path, @options.css)
+
+      if @context.comment
+        d = markup(@context.comment)
+        @values["description"] = d if d.size > 0
+      end
+
+      ml = build_method_summary_list
+      @values["methods"] = ml unless ml.empty?
+
+      il = build_include_list(@context)
+      @values["includes"] = il unless il.empty?
+
+      rl = build_requires_list(@context)
+      @values["requires"] = rl unless rl.empty?
+
+      if @options.promiscuous
+        file_context = nil
+      else
+        file_context = @context
+      end
+
+
+      @values["sections"] = @context.sections.map do |section|
+
+        secdata = {
+          "sectitle" => section.title,
+          "secsequence" => section.sequence,
+          "seccomment" => markup(section.comment)
+        }
+
+        cl = build_class_list(0, @context, section, file_context)
+        secdata["classlist"] = cl unless cl.empty?
+
+        mdl = build_method_detail_list(section)
+        secdata["method_list"] = mdl unless mdl.empty?
+
+        al = build_alias_summary_list(section)
+        secdata["aliases"] = al unless al.empty?
+
+        co = build_constants_summary_list(section)
+        secdata["constants"] = co unless co.empty?
+
+        secdata
+      end
+
+      @values
+    end
+
+    def write_on(f, file_list, class_list, method_list, overrides = {})
+      value_hash
+
+      @values['file_list'] = file_list
+      @values['class_list'] = class_list
+      @values['method_list'] = method_list
+
+      @values.update overrides
+
+      template = RDoc::TemplatePage.new(@template::BODY,
+                                        @template::FILE_PAGE,
+                                        @template::METHOD_LIST)
+
+      template.write_html_on(f, @values)
+    end
+
+    def file_attribute_values
+      full_path = @context.file_absolute_name
+      short_name = ::File.basename full_path
+
+      @values["title"] = CGI.escapeHTML("File: #{short_name}")
+
+      if @context.diagram then
+        @values["diagram"] = diagram_reference(@context.diagram)
+      end
+
+      @values["short_name"]   = CGI.escapeHTML(short_name)
+      @values["full_path"]    = CGI.escapeHTML(full_path)
+      @values["dtm_modified"] = @context.file_stat.mtime.to_s
+
+      if @options.webcvs then
+        @values["cvsurl"] = cvs_url @options.webcvs, @values["full_path"]
+      end
+    end
+
+    def <=>(other)
+      self.name <=> other.name
+    end
+
+  end
+
+  class Method
+
+    include MarkUp
+
+    attr_reader :context
+    attr_reader :src_url
+    attr_reader :img_url
+    attr_reader :source_code
+
+    @@seq = "M000000"
+
+    @@all_methods = []
+
+    def self.all_methods
+      @@all_methods
+    end
+
+    def self.reset
+      @@all_methods = []
+    end
+
+    def initialize(context, html_class, options)
+      # TODO: rethink the class hierarchy here...
+      @context    = context
+      @html_class = html_class
+      @options    = options
+
+      @@seq       = @@seq.succ
+      @seq        = @@seq
+
+      # HACK ugly
+      @template = options.template_class
+
+      @@all_methods << self
+
+      context.viewer = self
+
+      if (ts = @context.token_stream)
+        @source_code = markup_code(ts)
+        unless @options.inline_source
+          @src_url = create_source_code_file(@source_code)
+          @img_url = RDoc::Markup::ToHtml.gen_relative_url path, 'source.png'
+        end
+      end
+
+      AllReferences.add(name, self)
+    end
+
+    ##
+    # Returns a reference to outselves to be used as an href= the form depends
+    # on whether we're all in one file or in multiple files
+
+    def as_href(from_path)
+      if @options.all_one_file
+        "#" + path
+      else
+        RDoc::Markup::ToHtml.gen_relative_url from_path, path
+      end
+    end
+
+    def formatter
+      @formatter ||= @options.formatter ||
+        RDoc::Markup::ToHtmlCrossref.new(path, self, @options.show_hash)
+    end
+
+    def inspect
+      alias_for = if @context.is_alias_for then
+                    " (alias_for #{@context.is_alias_for})"
+                  else
+                    nil
+                  end
+
+      "#<%s:0x%x %s%s%s (%s)%s>" % [
+        self.class, object_id,
+        @context.parent.name,
+        @context.singleton ? '::' : '#',
+        name,
+        @context.visibility,
+        alias_for
+      ]
+    end
+
+    def name
+      @context.name
+    end
+
+    def section
+      @context.section
+    end
+
+    def index_name
+      "#{@context.name} (#{@html_class.name})"
+    end
+
+    def parent_name
+      if @context.parent.parent
+        @context.parent.parent.full_name
+      else
+        nil
+      end
+    end
+
+    def aref
+      @seq
+    end
+
+    def path
+      if @options.all_one_file
+        aref
+      else
+        @html_class.path + "#" + aref
+      end
+    end
+
+    def description
+      markup(@context.comment)
+    end
+
+    def visibility
+      @context.visibility
+    end
+
+    def singleton
+      @context.singleton
+    end
+
+    def call_seq
+      cs = @context.call_seq
+      if cs
+        cs.gsub(/\n/, "<br />\n")
+      else
+        nil
+      end
+    end
+
+    def params
+      # params coming from a call-seq in 'C' will start with the
+      # method name
+      params = @context.params
+      if params !~ /^\w/
+        params = @context.params.gsub(/\s*\#.*/, '')
+        params = params.tr("\n", " ").squeeze(" ")
+        params = "(" + params + ")" unless params[0] == ?(
+
+        if (block = @context.block_params)
+         # If this method has explicit block parameters, remove any
+         # explicit &block
+
+         params.sub!(/,?\s*&\w+/, '')
+
+          block.gsub!(/\s*\#.*/, '')
+          block = block.tr("\n", " ").squeeze(" ")
+          if block[0] == ?(
+            block.sub!(/^\(/, '').sub!(/\)/, '')
+          end
+          params << " {|#{block.strip}| ...}"
+        end
+      end
+      CGI.escapeHTML(params)
+    end
+
+    def create_source_code_file(code_body)
+      meth_path = @html_class.path.sub(/\.html$/, '.src')
+      FileUtils.mkdir_p(meth_path)
+      file_path = ::File.join meth_path, "#{@seq}.html"
+
+      template = RDoc::TemplatePage.new(@template::SRC_PAGE)
+
+      open file_path, 'w' do |f|
+        values = {
+          'title'     => CGI.escapeHTML(index_name),
+          'code'      => code_body,
+          'style_url' => style_url(file_path, @options.css),
+          'charset'   => @options.charset
+        }
+        template.write_html_on(f, values)
+      end
+
+      RDoc::Markup::ToHtml.gen_relative_url path, file_path
+    end
+
+    def <=>(other)
+      @context <=> other.context
+    end
+
+    ##
+    # Given a sequence of source tokens, mark up the source code
+    # to make it look purty.
+
+    def markup_code(tokens)
+      src = ""
+      tokens.each do |t|
+        next unless t
+#        style = STYLE_MAP[t.class]
+        style = case t
+                when RDoc::RubyToken::TkCONSTANT then "ruby-constant"
+                when RDoc::RubyToken::TkKW       then "ruby-keyword kw"
+                when RDoc::RubyToken::TkIVAR     then "ruby-ivar"
+                when RDoc::RubyToken::TkOp       then "ruby-operator"
+                when RDoc::RubyToken::TkId       then "ruby-identifier"
+                when RDoc::RubyToken::TkNode     then "ruby-node"
+                when RDoc::RubyToken::TkCOMMENT  then "ruby-comment cmt"
+                when RDoc::RubyToken::TkREGEXP   then "ruby-regexp re"
+                when RDoc::RubyToken::TkSTRING   then "ruby-value str"
+                when RDoc::RubyToken::TkVal      then "ruby-value"
+                else
+                    nil
+                end
+
+        text = CGI.escapeHTML(t.text)
+
+        if style
+          src << "<span class=\"#{style}\">#{text}</span>"
+        else
+          src << text
+        end
+      end
+
+      add_line_numbers(src) if @options.include_line_numbers
+      src
+    end
+
+    ##
+    # We rely on the fact that the first line of a source code listing has
+    #    # File xxxxx, line dddd
+
+    def add_line_numbers(src)
+      if src =~ /\A.*, line (\d+)/
+        first = $1.to_i - 1
+        last  = first + src.count("\n")
+        size = last.to_s.length
+        real_fmt = "%#{size}d: "
+        fmt = " " * (size+2)
+        src.gsub!(/^/) do
+          res = sprintf(fmt, first)
+          first += 1
+          fmt = real_fmt
+          res
+        end
+      end
+    end
+
+    def document_self
+      @context.document_self
+    end
+
+    def aliases
+      @context.aliases
+    end
+
+    def find_symbol(symbol, method=nil)
+      res = @context.parent.find_symbol(symbol, method)
+      if res
+        res = res.viewer
+      end
+      res
+    end
+
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/generator/chm.rb b/trunk/lib/rdoc/generator/chm.rb
new file mode 100644
index 0000000000..7537365842
--- /dev/null
+++ b/trunk/lib/rdoc/generator/chm.rb
@@ -0,0 +1,113 @@
+require 'rdoc/generator/html'
+
+class RDoc::Generator::CHM < RDoc::Generator::HTML
+
+  HHC_PATH = "c:/Program Files/HTML Help Workshop/hhc.exe"
+
+  ##
+  # Standard generator factory
+
+  def self.for(options)
+    new(options)
+  end
+
+  def initialize(*args)
+    super
+    @op_name = @options.op_name || "rdoc"
+    check_for_html_help_workshop
+  end
+
+  def check_for_html_help_workshop
+    stat = File.stat(HHC_PATH)
+  rescue
+    $stderr <<
+      "\n.chm output generation requires that Microsoft's Html Help\n" <<
+      "Workshop is installed. RDoc looks for it in:\n\n    " <<
+      HHC_PATH <<
+      "\n\nYou can download a copy for free from:\n\n" <<
+      "    http://msdn.microsoft.com/library/default.asp?" <<
+      "url=/library/en-us/htmlhelp/html/hwMicrosoftHTMLHelpDownloads.asp\n\n"
+  end
+
+  ##
+  # Generate the html as normal, then wrap it in a help project
+
+  def generate(info)
+    super
+    @project_name = @op_name + ".hhp"
+    create_help_project
+  end
+
+  ##
+  # The project contains the project file, a table of contents and an index
+
+  def create_help_project
+    create_project_file
+    create_contents_and_index
+    compile_project
+  end
+
+  ##
+  # The project file links together all the various
+  # files that go to make up the help.
+
+  def create_project_file
+    template = RDoc::TemplatePage.new @template::HPP_FILE
+    values = { "title" => @options.title, "opname" => @op_name }
+    files = []
+    @files.each do |f|
+      files << { "html_file_name" => f.path }
+    end
+
+    values['all_html_files'] = files
+
+    File.open(@project_name, "w") do |f|
+      template.write_html_on(f, values)
+    end
+  end
+
+  ##
+  # The contents is a list of all files and modules.
+  # For each we include  as sub-entries the list
+  # of methods they contain. As we build the contents
+  # we also build an index file
+
+  def create_contents_and_index
+    contents = []
+    index    = []
+
+    (@files+@classes).sort.each do |entry|
+      content_entry = { "c_name" => entry.name, "ref" => entry.path }
+      index << { "name" => entry.name, "aref" => entry.path }
+
+      internals = []
+
+      methods = entry.build_method_summary_list(entry.path)
+
+      content_entry["methods"] = methods unless methods.empty?
+      contents << content_entry
+      index.concat methods
+    end
+
+    values = { "contents" => contents }
+    template = RDoc::TemplatePage.new @template::CONTENTS
+    File.open("contents.hhc", "w") do |f|
+      template.write_html_on(f, values)
+    end
+
+    values = { "index" => index }
+    template = RDoc::TemplatePage.new @template::CHM_INDEX
+    File.open("index.hhk", "w") do |f|
+      template.write_html_on(f, values)
+    end
+  end
+
+  ##
+  # Invoke the windows help compiler to compiler the project
+
+  def compile_project
+    system(HHC_PATH, @project_name)
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/generator/chm/chm.rb b/trunk/lib/rdoc/generator/chm/chm.rb
new file mode 100644
index 0000000000..0a17a9e1ea
--- /dev/null
+++ b/trunk/lib/rdoc/generator/chm/chm.rb
@@ -0,0 +1,98 @@
+require 'rdoc/generator/chm'
+require 'rdoc/generator/html/html'
+
+module RDoc::Generator::CHM::CHM
+
+  HTML = RDoc::Generator::HTML::HTML
+
+  INDEX = HTML::INDEX
+
+  CLASS_INDEX = HTML::CLASS_INDEX
+  CLASS_PAGE = HTML::CLASS_PAGE
+  FILE_INDEX = HTML::FILE_INDEX
+  FILE_PAGE = HTML::FILE_PAGE
+  METHOD_INDEX = HTML::METHOD_INDEX
+  METHOD_LIST = HTML::METHOD_LIST
+
+  FR_INDEX_BODY = HTML::FR_INDEX_BODY
+
+  # This is a nasty little hack, but hhc doesn't support the <?xml tag, so...
+  BODY = HTML::BODY.sub!(/<\?xml.*\?>/, '')
+  SRC_PAGE = HTML::SRC_PAGE.sub!(/<\?xml.*\?>/, '')
+
+  HPP_FILE = <<-EOF
+[OPTIONS]
+Auto Index = Yes
+Compatibility=1.1 or later
+Compiled file=<%= values["opname"] %>.chm
+Contents file=contents.hhc
+Full-text search=Yes
+Index file=index.hhk
+Language=0x409 English(United States)
+Title=<%= values["title"] %>
+
+[FILES]
+<% values["all_html_files"].each do |all_html_files| %>
+<%= all_html_files["html_file_name"] %>
+<% end # values["all_html_files"] %>
+  EOF
+
+  CONTENTS = <<-EOF
+<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
+<HTML>
+<HEAD>
+<meta name="GENERATOR" content="Microsoft&reg; HTML Help Workshop 4.1">
+<!-- Sitemap 1.0 -->
+</HEAD><BODY>
+<OBJECT type="text/site properties">
+	<param name="Foreground" value="0x80">
+	<param name="Window Styles" value="0x800025">
+	<param name="ImageType" value="Folder">
+</OBJECT>
+<UL>
+<% values["contents"].each do |contents| %>
+	<LI> <OBJECT type="text/sitemap">
+		<param name="Name" value="<%= contents["c_name"] %>">
+		<param name="Local" value="<%= contents["ref"] %>">
+		</OBJECT>
+<% if contents["methods"] then %>
+<ul>
+<% contents["methods"].each do |methods| %>
+	<LI> <OBJECT type="text/sitemap">
+		<param name="Name" value="<%= methods["name"] %>">
+		<param name="Local" value="<%= methods["aref"] %>">
+		</OBJECT>
+<% end # contents["methods"] %>
+</ul>
+<% end %>
+        </LI>
+<% end # values["contents"] %>
+</UL>
+</BODY></HTML>
+  EOF
+
+  CHM_INDEX = <<-EOF
+<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
+<HTML>
+<HEAD>
+<meta name="GENERATOR" content="Microsoft&reg; HTML Help Workshop 4.1">
+<!-- Sitemap 1.0 -->
+</HEAD><BODY>
+<OBJECT type="text/site properties">
+	<param name="Foreground" value="0x80">
+	<param name="Window Styles" value="0x800025">
+	<param name="ImageType" value="Folder">
+</OBJECT>
+<UL>
+<% values["index"].each do |index| %>
+	<LI> <OBJECT type="text/sitemap">
+		<param name="Name" value="<%= index["name"] %>">
+		<param name="Local" value="<%= index["aref"] %>">
+		</OBJECT>
+<% end # values["index"] %>
+</UL>
+</BODY></HTML>
+  EOF
+
+end
+
diff --git a/trunk/lib/rdoc/generator/html.rb b/trunk/lib/rdoc/generator/html.rb
new file mode 100644
index 0000000000..a9e030a896
--- /dev/null
+++ b/trunk/lib/rdoc/generator/html.rb
@@ -0,0 +1,397 @@
+require 'fileutils'
+
+require 'rdoc/generator'
+require 'rdoc/markup/to_html'
+
+##
+# We're responsible for generating all the HTML files from the object tree
+# defined in code_objects.rb. We generate:
+#
+# [files]   an html file for each input file given. These
+#           input files appear as objects of class
+#           TopLevel
+#
+# [classes] an html file for each class or module encountered.
+#           These classes are not grouped by file: if a file
+#           contains four classes, we'll generate an html
+#           file for the file itself, and four html files
+#           for the individual classes.
+#
+# [indices] we generate three indices for files, classes,
+#           and methods. These are displayed in a browser
+#           like window with three index panes across the
+#           top and the selected description below
+#
+# Method descriptions appear in whatever entity (file, class, or module) that
+# contains them.
+#
+# We generate files in a structure below a specified subdirectory, normally
+# +doc+.
+#
+#  opdir
+#     |
+#     |___ files
+#     |       |__  per file summaries
+#     |
+#     |___ classes
+#             |__ per class/module descriptions
+#
+# HTML is generated using the Template class.
+
+class RDoc::Generator::HTML
+
+  include RDoc::Generator::MarkUp
+
+  ##
+  # Generator may need to return specific subclasses depending on the
+  # options they are passed. Because of this we create them using a factory
+
+  def self.for(options)
+    RDoc::Generator::AllReferences.reset
+    RDoc::Generator::Method.reset
+
+    if options.all_one_file
+      RDoc::Generator::HTMLInOne.new options
+    else
+      new options
+    end
+  end
+
+  class << self
+    protected :new
+  end
+
+  ##
+  # Set up a new HTML generator. Basically all we do here is load up the
+  # correct output temlate
+
+  def initialize(options) #:not-new:
+    @options = options
+    load_html_template
+    @main_page_path = nil
+  end
+
+  ##
+  # Build the initial indices and output objects
+  # based on an array of TopLevel objects containing
+  # the extracted information.
+
+  def generate(toplevels)
+    @toplevels  = toplevels
+    @files      = []
+    @classes    = []
+
+    write_style_sheet
+    gen_sub_directories
+    build_indices
+    generate_html
+  end
+
+  private
+
+  ##
+  # Load up the HTML template specified in the options.
+  # If the template name contains a slash, use it literally
+
+  def load_html_template
+    template = @options.template
+
+    unless template =~ %r{/|\\} then
+      template = File.join('rdoc', 'generator', @options.generator.key,
+                           template)
+    end
+
+    require template
+
+    @template = self.class.const_get @options.template.upcase
+    @options.template_class = @template
+
+  rescue LoadError
+    $stderr.puts "Could not find HTML template '#{template}'"
+    exit 99
+  end
+
+  ##
+  # Write out the style sheet used by the main frames
+
+  def write_style_sheet
+    return unless @template.constants.include? :STYLE or
+                  @template.constants.include? 'STYLE'
+
+    template = RDoc::TemplatePage.new @template::STYLE
+
+    unless @options.css then
+      open RDoc::Generator::CSS_NAME, 'w' do |f|
+        values = {}
+
+        if @template.constants.include? :FONTS or
+           @template.constants.include? 'FONTS' then
+          values["fonts"] = @template::FONTS
+        end
+
+        template.write_html_on(f, values)
+      end
+    end
+  end
+
+  ##
+  # See the comments at the top for a description of the directory structure
+
+  def gen_sub_directories
+    FileUtils.mkdir_p RDoc::Generator::FILE_DIR
+    FileUtils.mkdir_p RDoc::Generator::CLASS_DIR
+  rescue
+    $stderr.puts $!.message
+    exit 1
+  end
+
+  def build_indices
+    @files, @classes = RDoc::Generator::Context.build_indicies(@toplevels,
+                                                               @options)
+  end
+
+  ##
+  # Generate all the HTML
+
+  def generate_html
+    # the individual descriptions for files and classes
+    gen_into(@files)
+    gen_into(@classes)
+
+    # and the index files
+    gen_file_index
+    gen_class_index
+    gen_method_index
+    gen_main_index
+
+    # this method is defined in the template file
+    write_extra_pages if defined? write_extra_pages
+  end
+
+  def gen_into(list)
+    @file_list ||= index_to_links @files
+    @class_list ||= index_to_links @classes
+    @method_list ||= index_to_links RDoc::Generator::Method.all_methods
+
+    list.each do |item|
+      next unless item.document_self
+
+      op_file = item.path
+
+      FileUtils.mkdir_p File.dirname(op_file)
+
+      open op_file, 'w' do |io|
+        item.write_on io, @file_list, @class_list, @method_list
+      end
+    end
+  end
+
+  def gen_file_index
+    gen_an_index @files, 'Files', @template::FILE_INDEX, "fr_file_index.html"
+  end
+
+  def gen_class_index
+    gen_an_index(@classes, 'Classes', @template::CLASS_INDEX,
+                 "fr_class_index.html")
+  end
+
+  def gen_method_index
+    gen_an_index(RDoc::Generator::Method.all_methods, 'Methods',
+                 @template::METHOD_INDEX, "fr_method_index.html")
+  end
+
+  def gen_an_index(collection, title, template, filename)
+    template = RDoc::TemplatePage.new @template::FR_INDEX_BODY, template
+    res = []
+    collection.sort.each do |f|
+      if f.document_self
+        res << { "href" => f.path, "name" => f.index_name }
+      end
+    end
+
+    values = {
+      "entries"    => res,
+      'list_title' => CGI.escapeHTML(title),
+      'index_url'  => main_url,
+      'charset'    => @options.charset,
+      'style_url'  => style_url('', @options.css),
+    }
+
+    open filename, 'w' do |f|
+      template.write_html_on(f, values)
+    end
+  end
+
+  ##
+  # The main index page is mostly a template frameset, but includes the
+  # initial page. If the <tt>--main</tt> option was given, we use this as
+  # our main page, otherwise we use the first file specified on the command
+  # line.
+
+  def gen_main_index
+    if @template.const_defined? :FRAMELESS then
+      main = @files.find do |file|
+        @main_page == file.name
+      end
+
+      if main.nil? then
+        main = @classes.find do |klass|
+          main_page == klass.context.full_name
+        end
+      end
+    else
+      main = RDoc::TemplatePage.new @template::INDEX
+    end
+
+    open 'index.html', 'w'  do |f|
+      style_url = style_url '', @options.css
+
+      classes = @classes.sort.map { |klass| klass.value_hash }
+
+      values = {
+        'main_page'     => @main_page,
+        'initial_page'  => main_url,
+        'style_url'     => style_url('', @options.css),
+        'title'         => CGI.escapeHTML(@options.title),
+        'charset'       => @options.charset,
+        'classes'       => classes,
+      }
+
+      values['inline_source'] = @options.inline_source
+
+      if main.respond_to? :write_on then
+        main.write_on f, @file_list, @class_list, @method_list, values
+      else
+        main.write_html_on f, values
+      end
+    end
+  end
+
+  def index_to_links(collection)
+    collection.sort.map do |f|
+      next unless f.document_self
+      { "href" => f.path, "name" => f.index_name }
+    end.compact
+  end
+
+  ##
+  # Returns the url of the main page
+
+  def main_url
+    @main_page = @options.main_page
+    @main_page_ref = nil
+
+    if @main_page then
+      @main_page_ref = RDoc::Generator::AllReferences[@main_page]
+
+      if @main_page_ref then
+        @main_page_path = @main_page_ref.path
+      else
+        $stderr.puts "Could not find main page #{@main_page}"
+      end
+    end
+
+    unless @main_page_path then
+      file = @files.find { |context| context.document_self }
+      @main_page_path = file.path if file
+    end
+
+    unless @main_page_path then
+      $stderr.puts "Couldn't find anything to document"
+      $stderr.puts "Perhaps you've used :stopdoc: in all classes"
+      exit 1
+    end
+
+    @main_page_path
+  end
+
+end
+
+class RDoc::Generator::HTMLInOne < RDoc::Generator::HTML
+
+  def initialize(*args)
+    super
+  end
+
+  ##
+  # Build the initial indices and output objects
+  # based on an array of TopLevel objects containing
+  # the extracted information.
+
+  def generate(info)
+    @toplevels  = info
+    @hyperlinks = {}
+
+    build_indices
+    generate_xml
+  end
+
+  ##
+  # Generate:
+  #
+  # * a list of RDoc::Generator::File objects for each TopLevel object.
+  # * a list of RDoc::Generator::Class objects for each first level
+  #   class or module in the TopLevel objects
+  # * a complete list of all hyperlinkable terms (file,
+  #   class, module, and method names)
+
+  def build_indices
+    @files, @classes = RDoc::Generator::Context.build_indices(@toplevels,
+                                                              @options)
+  end
+
+  ##
+  # Generate all the HTML. For the one-file case, we generate
+  # all the information in to one big hash
+
+  def generate_xml
+    values = {
+      'charset' => @options.charset,
+      'files'   => gen_into(@files),
+      'classes' => gen_into(@classes),
+      'title'        => CGI.escapeHTML(@options.title),
+    }
+
+    # this method is defined in the template file
+    write_extra_pages if defined? write_extra_pages
+
+    template = RDoc::TemplatePage.new @template::ONE_PAGE
+
+    if @options.op_name
+      opfile = open @options.op_name, 'w'
+    else
+      opfile = $stdout
+    end
+    template.write_html_on(opfile, values)
+  end
+
+  def gen_into(list)
+    res = []
+    list.each do |item|
+      res << item.value_hash
+    end
+    res
+  end
+
+  def gen_file_index
+    gen_an_index(@files, 'Files')
+  end
+
+  def gen_class_index
+    gen_an_index(@classes, 'Classes')
+  end
+
+  def gen_method_index
+    gen_an_index(RDoc::Generator::Method.all_methods, 'Methods')
+  end
+
+  def gen_an_index(collection, title)
+    return {
+      "entries" => index_to_links(collection),
+      'list_title' => title,
+      'index_url'  => main_url,
+    }
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/generator/html/frameless.rb b/trunk/lib/rdoc/generator/html/frameless.rb
new file mode 100644
index 0000000000..2af890ce04
--- /dev/null
+++ b/trunk/lib/rdoc/generator/html/frameless.rb
@@ -0,0 +1,795 @@
+require 'rdoc/generator/html'
+require 'rdoc/generator/html/one_page_html'
+
+##
+# = CSS2 RDoc HTML template
+#
+# This is a template for RDoc that uses XHTML 1.0 Transitional and dictates a
+# bit more of the appearance of the output to cascading stylesheets than the
+# default. It was designed for clean inline code display, and uses DHTMl to
+# toggle the visbility of each method's source with each click on the '[source]'
+# link.
+#
+# == Authors
+#
+# * Michael Granger <ged@FaerieMUD.org>
+#
+# Copyright (c) 2002, 2003 The FaerieMUD Consortium. Some rights reserved.
+#
+# This work is licensed under the Creative Commons Attribution License. To view
+# a copy of this license, visit http://creativecommons.org/licenses/by/1.0/ or
+# send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California
+# 94305, USA.
+
+module RDoc::Generator::HTML::FRAMELESS
+
+  FRAMELESS = true
+
+  FONTS = "Verdana,Arial,Helvetica,sans-serif"
+
+  STYLE = <<-EOF
+body {
+  font-family: #{FONTS};
+  font-size: 90%;
+  margin: 0;
+  margin-left: 40px;
+  padding: 0;
+  background: white;
+}
+
+h1, h2, h3, h4 {
+  margin: 0;
+  color: #efefef;
+  background: transparent;
+}
+
+h1 {
+  font-size: 150%;
+}
+
+h2,h3,h4 {
+  margin-top: 1em;
+}
+
+:link, :visited {
+  background: #eef;
+  color: #039;
+  text-decoration: none;
+}
+
+:link:hover, :visited:hover {
+  background: #039;
+  color: #eef;
+}
+
+/* Override the base stylesheet's Anchor inside a table cell */
+td > :link, td > :visited {
+  background: transparent;
+  color: #039;
+  text-decoration: none;
+}
+
+/* and inside a section title */
+.section-title > :link, .section-title > :visited {
+  background: transparent;
+  color: #eee;
+  text-decoration: none;
+}
+
+/* === Structural elements =================================== */
+
+.index {
+  margin: 0;
+  margin-left: -40px;
+  padding: 0;
+  font-size: 90%;
+}
+
+.index :link, .index :visited {
+  margin-left: 0.7em;
+}
+
+.index .section-bar {
+  margin-left: 0px;
+  padding-left: 0.7em;
+  background: #ccc;
+  font-size: small;
+}
+
+#classHeader, #fileHeader {
+  width: auto;
+  color: white;
+  padding: 0.5em 1.5em 0.5em 1.5em;
+  margin: 0;
+  margin-left: -40px;
+  border-bottom: 3px solid #006;
+}
+
+#classHeader :link, #fileHeader :link,
+#classHeader :visited, #fileHeader :visited {
+  background: inherit;
+  color: white;
+}
+
+#classHeader td, #fileHeader td {
+  background: inherit;
+  color: white;
+}
+
+#fileHeader {
+  background: #057;
+}
+
+#classHeader {
+  background: #048;
+}
+
+.class-name-in-header {
+  font-size:  180%;
+  font-weight: bold;
+}
+
+#bodyContent {
+  padding: 0 1.5em 0 1.5em;
+}
+
+#description {
+  padding: 0.5em 1.5em;
+  background: #efefef;
+  border: 1px dotted #999;
+}
+
+#description h1, #description h2, #description h3,
+#description h4, #description h5, #description h6 {
+  color: #125;
+  background: transparent;
+}
+
+#copyright {
+  color: #333;
+  background: #efefef;
+  font: 0.75em sans-serif;
+  margin-top: 5em;
+  margin-bottom: 0;
+  padding: 0.5em 2em;
+}
+
+/* === Classes =================================== */
+
+table.header-table {
+  color: white;
+  font-size: small;
+}
+
+.type-note {
+  font-size: small;
+  color: #dedede;
+}
+
+.xxsection-bar {
+  background: #eee;
+  color: #333;
+  padding: 3px;
+}
+
+.section-bar {
+  color: #333;
+  border-bottom: 1px solid #999;
+  margin-left: -20px;
+}
+
+.section-title {
+  background: #79a;
+  color: #eee;
+  padding: 3px;
+  margin-top: 2em;
+  margin-left: -30px;
+  border: 1px solid #999;
+}
+
+.top-aligned-row {
+  vertical-align: top
+}
+
+.bottom-aligned-row {
+  vertical-align: bottom
+}
+
+/* --- Context section classes ----------------------- */
+
+.context-row { }
+
+.context-item-name {
+  font-family: monospace;
+  font-weight: bold;
+  color: black;
+}
+
+.context-item-value {
+  font-size: small;
+  color: #448;
+}
+
+.context-item-desc {
+  color: #333;
+  padding-left: 2em;
+}
+
+/* --- Method classes -------------------------- */
+
+.method-detail {
+  background: #efefef;
+  padding: 0;
+  margin-top: 0.5em;
+  margin-bottom: 1em;
+  border: 1px dotted #ccc;
+}
+
+.method-heading {
+  color: black;
+  background: #ccc;
+  border-bottom: 1px solid #666;
+  padding: 0.2em 0.5em 0 0.5em;
+}
+
+.method-signature {
+  color: black;
+  background: inherit;
+}
+
+.method-name {
+  font-weight: bold;
+}
+
+.method-args {
+  font-style: italic;
+}
+
+.method-description {
+  padding: 0 0.5em 0 0.5em;
+}
+
+/* --- Source code sections -------------------- */
+
+:link.source-toggle, :visited.source-toggle {
+  font-size: 90%;
+}
+
+div.method-source-code {
+  background: #262626;
+  color: #ffdead;
+  margin: 1em;
+  padding: 0.5em;
+  border: 1px dashed #999;
+  overflow: hidden;
+}
+
+div.method-source-code pre {
+  color: #ffdead;
+  overflow: hidden;
+}
+
+/* --- Ruby keyword styles --------------------- */
+
+.standalone-code {
+  background: #221111;
+  color: #ffdead;
+  overflow: hidden;
+}
+
+.ruby-constant {
+  color: #7fffd4;
+  background: transparent;
+}
+
+.ruby-keyword {
+  color: #00ffff;
+  background: transparent;
+}
+
+.ruby-ivar {
+  color: #eedd82;
+  background: transparent;
+}
+
+.ruby-operator {
+  color: #00ffee;
+  background: transparent;
+}
+
+.ruby-identifier {
+  color: #ffdead;
+  background: transparent;
+}
+
+.ruby-node {
+  color: #ffa07a;
+  background: transparent;
+}
+
+.ruby-comment {
+  color: #b22222;
+  font-weight: bold;
+  background: transparent;
+}
+
+.ruby-regexp {
+  color: #ffa07a;
+  background: transparent;
+}
+
+.ruby-value {
+  color: #7fffd4;
+  background: transparent;
+}
+
+EOF
+
+  ##
+  # Header template
+
+  XHTML_PREAMBLE = <<-EOF
+<?xml version="1.0" encoding="<%= values["charset"] %>"?>
+<!DOCTYPE html
+     PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+  EOF
+
+  HEADER = XHTML_PREAMBLE + <<-EOF
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+  <title><%= values["title"] %></title>
+  <meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>" />
+  <meta http-equiv="Content-Script-Type" content="text/javascript" />
+  <link rel="stylesheet" href="<%= values["style_url"] %>" type="text/css" media="screen" />
+  <script type="text/javascript">
+  // <![CDATA[
+
+  function popupCode( url ) {
+    window.open(url, "Code", "resizable=yes,scrollbars=yes,toolbar=no,status=no,height=150,width=400")
+  }
+
+  function toggleCode( id ) {
+    if ( document.getElementById )
+      elem = document.getElementById( id );
+    else if ( document.all )
+      elem = eval( "document.all." + id );
+    else
+      return false;
+
+    elemStyle = elem.style;
+
+    if ( elemStyle.display != "block" ) {
+      elemStyle.display = "block"
+    } else {
+      elemStyle.display = "none"
+    }
+
+    return true;
+  }
+
+  // Make codeblocks hidden by default
+  document.writeln( "<style type=\\"text/css\\">div.method-source-code { display: none }</style>" )
+
+  // ]]>
+  </script>
+
+</head>
+<body>
+EOF
+
+  ##
+  # Context content template
+
+  CONTEXT_CONTENT = %{
+}
+
+  ##
+  # Footer template
+
+  FOOTER = <<-EOF
+  <div id="popupmenu" class="index">
+    <ul>
+    <li class="index-entries section-bar">Classes
+      <ul>
+<% values["class_list"].each do |klass| %>
+        <li><a href="<%= klass["href"] %>"><%= klass["name"] %></a>
+<% end %>
+      </ul>
+    </li>
+
+    <li class="index-entries section-bar">Methods
+      <ul>
+<% values["method_list"].each do |file| %>
+        <li><a href="<%= file["href"] %>"><%= file["name"] %></a>
+<% end %>
+      </ul>
+    </li>
+
+    <li class="index-entries section-bar">Files
+      <ul>
+<% values["file_list"].each do |file| %>
+        <li><a href="<%= file["href"] %>"><%= file["name"] %></a>
+<% end %>
+      </ul>
+    </li>
+    </ul>
+  </li>
+
+</body>
+</html>
+  EOF
+
+  ##
+  # File page header template
+
+  FILE_PAGE = <<-EOF
+  <div id="fileHeader">
+    <h1><%= values["short_name"] %></h1>
+
+    <table class="header-table">
+    <tr class="top-aligned-row">
+      <td><strong>Path:</strong></td>
+      <td><%= values["full_path"] %>
+<% if values["cvsurl"] then %>
+        &nbsp;(<a href="<%= values["cvsurl"] %>"><acronym title="Concurrent Versioning System">CVS</acronym></a>)
+<% end %>
+      </td>
+    </tr>
+
+    <tr class="top-aligned-row">
+      <td><strong>Last Update:</strong></td>
+      <td><%= values["dtm_modified"] %></td>
+    </tr>
+    </table>
+  </div>
+  EOF
+
+  ##
+  # Class page header template
+
+  CLASS_PAGE = <<-EOF
+    <div id="classHeader">
+      <table class="header-table">
+      <tr class="top-aligned-row">
+        <td><strong><%= values["classmod"] %></strong></td>
+        <td class="class-name-in-header"><%= values["full_name"] %></td>
+      </tr>
+
+      <tr class="top-aligned-row">
+        <td><strong>In:</strong></td>
+        <td>
+<% values["infiles"].each do |infiles| %>
+<% if infiles["full_path_url"] then %>
+          <a href="<%= infiles["full_path_url"] %>">
+<% end %>
+            <%= infiles["full_path"] %>
+<% if infiles["full_path_url"] then %>
+          </a>
+<% end %>
+<% if infiles["cvsurl"] then %>
+          &nbsp;(<a href="<%= infiles["cvsurl"] %>"><acronym title="Concurrent Versioning System">CVS</acronym></a>)
+<% end %>
+          <br />
+<% end %><%# values["infiles"] %>
+        </td>
+      </tr>
+
+<% if values["parent"] then %>
+      <tr class="top-aligned-row">
+        <td><strong>Parent:</strong></td>
+        <td>
+<% if values["par_url"] then %>
+          <a href="<%= values["par_url"] %>">
+<% end %>
+            <%= values["parent"] %>
+<% if values["par_url"] then %>
+          </a>
+<% end %>
+        </td>
+      </tr>
+<% end %>
+    </table>
+  </div>
+  EOF
+
+  ##
+  # Method list template
+
+  METHOD_LIST = <<-EOF
+
+  <div id="contextContent">
+<% if values["diagram"] then %>
+    <div id="diagram">
+      <%= values["diagram"] %>
+    </div>
+<% end %>
+
+<% if values["description"] then %>
+    <div id="description">
+      <%= values["description"] %>
+    </div>
+<% end %>
+
+<% if values["requires"] then %>
+    <div id="requires-list">
+      <h3 class="section-bar">Required files</h3>
+
+      <div class="name-list">
+<% values["requires"].each do |requires| %>
+        <%= href requires["aref"], requires["name"] %>&nbsp;&nbsp;
+<% end %><%# values["requires"] %>
+      </div>
+    </div>
+<% end %>
+
+<% if values["toc"] then %>
+    <div id="contents-list">
+      <h3 class="section-bar">Contents</h3>
+      <ul>
+<% values["toc"].each do |toc| %>
+      <li><a href="#<%= values["href"] %>"><%= values["secname"] %></a></li>
+<% end %><%# values["toc"] %>
+     </ul>
+<% end %>
+   </div>
+
+<% if values["methods"] then %>
+    <div id="method-list">
+      <h3 class="section-bar">Methods</h3>
+
+      <div class="name-list">
+<% values["methods"].each do |methods| %>
+        <%= href methods["aref"], methods["name"] %>&nbsp;&nbsp;
+<% end %><%# values["methods"] %>
+      </div>
+    </div>
+<% end %>
+
+  </div>
+
+
+    <!-- if includes -->
+<% if values["includes"] then %>
+    <div id="includes">
+      <h3 class="section-bar">Included Modules</h3>
+
+      <div id="includes-list">
+<% values["includes"].each do |includes| %>
+        <span class="include-name"><%= href includes["aref"], includes["name"] %></span>
+<% end %><%# values["includes"] %>
+      </div>
+    </div>
+<% end %>
+
+<% values["sections"].each do |sections| %>
+    <div id="section">
+<% if sections["sectitle"] then %>
+      <h2 class="section-title"><a name="<%= sections["secsequence"] %>"><%= sections["sectitle"] %></a></h2>
+<% if sections["seccomment"] then %>
+      <div class="section-comment">
+        <%= sections["seccomment"] %>
+      </div>
+<% end %>
+<% end %>
+
+<% if values["classlist"] then %>
+    <div id="class-list">
+      <h3 class="section-bar">Classes and Modules</h3>
+
+      <%= values["classlist"] %>
+    </div>
+<% end %>
+
+<% if values["constants"] then %>
+    <div id="constants-list">
+      <h3 class="section-bar">Constants</h3>
+
+      <div class="name-list">
+        <table summary="Constants">
+<% values["constants"].each do |constants| %>
+        <tr class="top-aligned-row context-row">
+          <td class="context-item-name"><%= constants["name"] %></td>
+          <td>=</td>
+          <td class="context-item-value"><%= constants["value"] %></td>
+<% if values["desc"] then %>
+          <td width="3em">&nbsp;</td>
+          <td class="context-item-desc"><%= constants["desc"] %></td>
+<% end %>
+        </tr>
+<% end %><%# values["constants"] %>
+        </table>
+      </div>
+    </div>
+<% end %>
+
+<% if values["aliases"] then %>
+    <div id="aliases-list">
+      <h3 class="section-bar">External Aliases</h3>
+
+      <div class="name-list">
+                        <table summary="aliases">
+<% values["aliases"].each do |aliases| $stderr.puts({ :aliases => aliases }.inspect) %>
+        <tr class="top-aligned-row context-row">
+          <td class="context-item-name"><%= values["old_name"] %></td>
+          <td>-&gt;</td>
+          <td class="context-item-value"><%= values["new_name"] %></td>
+        </tr>
+<% if values["desc"] then %>
+      <tr class="top-aligned-row context-row">
+        <td>&nbsp;</td>
+        <td colspan="2" class="context-item-desc"><%= values["desc"] %></td>
+      </tr>
+<% end %>
+<% end %><%# values["aliases"] %>
+        </table>
+      </div>
+    </div>
+<% end %>
+
+
+<% if values["attributes"] then %>
+    <div id="attribute-list">
+      <h3 class="section-bar">Attributes</h3>
+
+      <div class="name-list">
+        <table>
+<% values["attributes"].each do |attributes| $stderr.puts({ :attributes => attributes }.inspect) %>
+        <tr class="top-aligned-row context-row">
+          <td class="context-item-name"><%= values["name"] %></td>
+<% if values["rw"] then %>
+          <td class="context-item-value">&nbsp;[<%= values["rw"] %>]&nbsp;</td>
+<% end %>
+<% unless values["rw"] then %>
+          <td class="context-item-value">&nbsp;&nbsp;</td>
+<% end %>
+          <td class="context-item-desc"><%= values["a_desc"] %></td>
+        </tr>
+<% end %><%# values["attributes"] %>
+        </table>
+      </div>
+    </div>
+<% end %>
+
+    <!-- if method_list -->
+<% if sections["method_list"] then %>
+    <div id="methods">
+<% sections["method_list"].each do |method_list| %>
+<% if method_list["methods"] then %>
+      <h3 class="section-bar"><%= method_list["type"] %> <%= method_list["category"] %> methods</h3>
+
+<% method_list["methods"].each do |methods| %>
+      <div id="method-<%= methods["aref"] %>" class="method-detail">
+        <a name="<%= methods["aref"] %>"></a>
+
+        <div class="method-heading">
+<% if methods["codeurl"] then %>
+          <a href="<%= methods["codeurl"] %>" target="Code" class="method-signature"
+            onclick="popupCode('<%= methods["codeurl"] %>');return false;">
+<% end %>
+<% if methods["sourcecode"] then %>
+          <a href="#<%= methods["aref"] %>" class="method-signature">
+<% end %>
+<% if methods["callseq"] then %>
+          <span class="method-name"><%= methods["callseq"] %></span>
+<% end %>
+<% unless methods["callseq"] then %>
+          <span class="method-name"><%= methods["name"] %></span><span class="method-args"><%= methods["params"] %></span>
+<% end %>
+<% if methods["codeurl"] then %>
+          </a>
+<% end %>
+<% if methods["sourcecode"] then %>
+          </a>
+<% end %>
+        </div>
+
+        <div class="method-description">
+<% if methods["m_desc"] then %>
+          <%= methods["m_desc"] %>
+<% end %>
+<% if methods["sourcecode"] then %>
+          <p><a class="source-toggle" href="#"
+            onclick="toggleCode('<%= methods["aref"] %>-source');return false;">[Source]</a></p>
+          <div class="method-source-code" id="<%= methods["aref"] %>-source">
+<pre>
+<%= methods["sourcecode"] %>
+</pre>
+          </div>
+<% end %>
+        </div>
+      </div>
+
+<% end %><%# method_list["methods"] %>
+<% end %>
+<% end %><%# sections["method_list"] %>
+
+    </div>
+<% end %>
+<% end %><%# values["sections"] %>
+  EOF
+
+  ##
+  # Body template
+
+  BODY = HEADER + %{
+
+<%= template_include %>  <!-- banner header -->
+
+  <div id="bodyContent">
+
+} +  METHOD_LIST + %{
+
+  </div>
+
+} + FOOTER
+
+  ##
+  # Source code template
+
+  SRC_PAGE = XHTML_PREAMBLE + <<-EOF
+<html>
+<head>
+  <title><%= values["title"] %></title>
+  <meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>" />
+  <link rel="stylesheet" href="<%= values["style_url"] %>" type="text/css" media="screen" />
+</head>
+<body class="standalone-code">
+  <pre><%= values["code"] %></pre>
+</body>
+</html>
+  EOF
+
+  ##
+  # Index file templates
+
+  FR_INDEX_BODY = %{
+<%= template_include %>
+}
+
+  FILE_INDEX = XHTML_PREAMBLE + <<-EOF
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+  <title><%= values["list_title"] %></title>
+  <meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>" />
+  <link rel="stylesheet" href="<%= values["style_url"] %>" type="text/css" />
+  <base target="docwin" />
+</head>
+<body>
+<div class="index">
+  <h1 class="section-bar"><%= values["list_title"] %></h1>
+  <div class="index-entries">
+<% values["entries"].each do |entries| %>
+    <a href="<%= entries["href"] %>"><%= entries["name"] %></a><br />
+<% end %><%# values["entries"] %>
+  </div>
+</div>
+</body>
+</html>
+  EOF
+
+  CLASS_INDEX = FILE_INDEX
+  METHOD_INDEX = FILE_INDEX
+
+  INDEX = <<-EOF
+<?xml version="1.0" encoding="<%= values["charset"] %>"?>
+<!DOCTYPE html
+     PUBLIC "-//W3C//DTD XHTML 1.0 Frameset//EN"
+     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-frameset.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+  <title><%= values["title"] %></title>
+  <meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>" />
+</head>
+<frameset rows="20%, 80%">
+  <frameset cols="45%,55%">
+    <frame src="fr_class_index.html"  name="Classes" />
+    <frame src="fr_method_index.html" name="Methods" />
+  </frameset>
+  <frame src="<%= values["initial_page"] %>" name="docwin" />
+</frameset>
+</html>
+  EOF
+
+end
+
diff --git a/trunk/lib/rdoc/generator/html/hefss.rb b/trunk/lib/rdoc/generator/html/hefss.rb
new file mode 100644
index 0000000000..e186a40384
--- /dev/null
+++ b/trunk/lib/rdoc/generator/html/hefss.rb
@@ -0,0 +1,414 @@
+require 'rdoc/generator/html'
+require 'rdoc/generator/html/html'
+
+module RDoc::Generator::HTML::HEFSS
+
+  FONTS = "Verdana, Arial, Helvetica, sans-serif"
+
+STYLE = <<-EOF
+body,p { font-family: Verdana, Arial, Helvetica, sans-serif;
+       color: #000040; background: #BBBBBB;
+}
+
+td { font-family: Verdana, Arial, Helvetica, sans-serif;
+       color: #000040;
+}
+
+.attr-rw { font-size: small; color: #444488 }
+
+.title-row {color:      #eeeeff;
+	    background: #BBBBDD;
+}
+
+.big-title-font { color: white;
+                  font-family: Verdana, Arial, Helvetica, sans-serif;
+                  font-size: large;
+                  height: 50px}
+
+.small-title-font { color: purple;
+                    font-family: Verdana, Arial, Helvetica, sans-serif;
+                    font-size: small; }
+
+.aqua { color: purple }
+
+.method-name, attr-name {
+      font-family: monospace; font-weight: bold;
+}
+
+.tablesubtitle {
+   width: 100%;
+   margin-top: 1ex;
+   margin-bottom: .5ex;
+   padding: 5px 0px 5px 20px;
+   font-size: large;
+   color: purple;
+   background: #BBBBCC;
+}
+
+.tablesubsubtitle {
+   width: 100%;
+   margin-top: 1ex;
+   margin-bottom: .5ex;
+   padding: 5px 0px 5px 20px;
+   font-size: medium;
+   color: white;
+   background: #BBBBCC;
+}
+
+.name-list {
+  font-family: monospace;
+  margin-left: 40px;
+  margin-bottom: 2ex;
+  line-height: 140%;
+}
+
+.description {
+  margin-left: 40px;
+  margin-bottom: 2ex;
+  line-height: 140%;
+}
+
+.methodtitle {
+  font-size: medium;
+  text_decoration: none;
+  padding: 3px 3px 3px 20px;
+  color: #0000AA;
+}
+
+.column-title {
+  font-size: medium;
+  font-weight: bold;
+  text_decoration: none;
+  padding: 3px 3px 3px 20px;
+  color: #3333CC;
+  }
+
+.variable-name {
+  font-family: monospace;
+  font-size: medium;
+  text_decoration: none;
+  padding: 3px 3px 3px 20px;
+  color: #0000AA;
+}
+
+.row-name {
+  font-size: medium;
+  font-weight: medium;
+  font-family: monospace;
+  text_decoration: none;
+  padding: 3px 3px 3px 20px;
+}
+
+.paramsig {
+   font-size: small;
+}
+
+.srcbut { float: right }
+
+  EOF
+
+  BODY = <<-EOF
+<html><head>
+  <title><%= values["title"] %></title>
+  <meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>">
+  <link rel="stylesheet" href="<%= values["style_url"] %>" type="text/css" media="screen" />
+  <script type="text/javascript" language="JavaScript">
+  <!--
+  function popCode(url) {
+    parent.frames.source.location = url
+  }
+  //-->
+  </script>
+</head>
+<body bgcolor="#BBBBBB">
+
+<%= template_include %>  <!-- banner header -->
+
+<% if values["diagram"] then %>
+<table width="100%"><tr><td align="center">
+<%= values["diagram"] %>
+</td></tr></table>
+<% end %>
+
+<% if values["description"] then %>
+<div class="description"><%= values["description"] %></div>
+<% end %>
+
+<% if values["requires"] then %>
+<table cellpadding="5" width="100%">
+<tr><td class="tablesubtitle">Required files</td></tr>
+</table><br />
+<div class="name-list">
+<% values["requires"].each do |requires| %>
+<%= href requires["aref"], requires["name"] %>
+<% end %><%# values["requires"] %>
+<% end %>
+</div>
+
+<% if values["sections"] then %>
+<% values["sections"].each do |sections| %>
+<% if sections["method_list"] then %>
+<% sections["method_list"].each do |method_list| %>
+<% if method_list["methods"] then %>
+<table cellpadding="5" width="100%">
+<tr><td class="tablesubtitle">Subroutines and Functions</td></tr>
+</table><br />
+<div class="name-list">
+<% method_list["methods"].each do |methods| %>
+<a href="<%= methods["codeurl"] %>" target="source"><%= methods["name"] %></a>
+<% end %><%# values["methods"] %>
+</div>
+<% end %>
+<% end %><%# values["method_list"] %>
+<% end %>
+
+<% if sections["attributes"] then %>
+<table cellpadding="5" width="100%">
+<tr><td class="tablesubtitle">Arguments</td></tr>
+</table><br />
+<table cellspacing="5">
+<% sections["attributes"].each do |attributes| %>
+     <tr valign="top">
+<% if attributes["rw"] then %>
+       <td align="center" class="attr-rw">&nbsp;[<%= attributes["rw"] %>]&nbsp;</td>
+<% end %>
+<% unless attributes["rw"] then %>
+       <td></td>
+<% end %>
+       <td class="attr-name"><%= attributes["name"] %></td>
+       <td><%= attributes["a_desc"] %></td>
+     </tr>
+<% end %><%# values["attributes"] %>
+</table>
+<% end %>
+<% end %><%# values["sections"] %>
+<% end %>
+
+<% if values["classlist"] then %>
+<table cellpadding="5" width="100%">
+<tr><td class="tablesubtitle">Modules</td></tr>
+</table><br />
+<%= values["classlist"] %><br />
+<% end %>
+
+  <%= template_include %>  <!-- method descriptions -->
+
+</body>
+</html>
+  EOF
+
+  FILE_PAGE = <<-EOF
+<table width="100%">
+ <tr class="title-row">
+ <td><table width="100%"><tr>
+   <td class="big-title-font" colspan="2"><font size="-3"><b>File</b><br /></font><%= values["short_name"] %></td>
+   <td align="right"><table cellspacing="0" cellpadding="2">
+         <tr>
+           <td  class="small-title-font">Path:</td>
+           <td class="small-title-font"><%= values["full_path"] %>
+<% if values["cvsurl"] then %>
+				&nbsp;(<a href="<%= values["cvsurl"] %>"><acronym title="Concurrent Versioning System">CVS</acronym></a>)
+<% end %>
+           </td>
+         </tr>
+         <tr>
+           <td class="small-title-font">Modified:</td>
+           <td class="small-title-font"><%= values["dtm_modified"] %></td>
+         </tr>
+        </table>
+    </td></tr></table></td>
+  </tr>
+</table><br />
+  EOF
+
+  CLASS_PAGE = <<-EOF
+<table width="100%" border="0" cellspacing="0">
+ <tr class="title-row">
+ <td class="big-title-font">
+   <font size="-3"><b><%= values["classmod"] %></b><br /></font><%= values["full_name"] %>
+ </td>
+ <td align="right">
+   <table cellspacing="0" cellpadding="2">
+     <tr valign="top">
+      <td class="small-title-font">In:</td>
+      <td class="small-title-font">
+<% values["infiles"].each do |infiles| %>
+<%= href infiles["full_path_url"], infiles["full_path"] %>
+<% if infiles["cvsurl"] then %>
+&nbsp;(<a href="<%= infiles["cvsurl"] %>"><acronym title="Concurrent Versioning System">CVS</acronym></a>)
+<% end %>
+<% end %><%# values["infiles"] %>
+      </td>
+     </tr>
+<% if values["parent"] then %>
+     <tr>
+      <td class="small-title-font">Parent:</td>
+      <td class="small-title-font">
+<% if values["par_url"] then %>
+        <a href="<%= values["par_url"] %>" class="cyan">
+<% end %>
+<%= values["parent"] %>
+<% if values["par_url"] then %>
+         </a>
+<% end %>
+      </td>
+     </tr>
+<% end %>
+   </table>
+  </td>
+  </tr>
+</table><br />
+  EOF
+
+  METHOD_LIST = <<-EOF
+<% if values["includes"] then %>
+<div class="tablesubsubtitle">Uses</div><br />
+<div class="name-list">
+<% values["includes"].each do |includes| %>
+    <span class="method-name"><%= href includes["aref"], includes["name"] %></span>
+<% end %><%# values["includes"] %>
+</div>
+<% end %>
+
+<% if values["sections"] then %>
+<% values["sections"].each do |sections| %>
+<% if sections["method_list"] then %>
+<% sections["method_list"].each do |method_list| %>
+<% if method_list["methods"] then %>
+<table cellpadding="5" width="100%">
+<tr><td class="tablesubtitle"><%= method_list["type"] %> <%= method_list["category"] %> methods</td></tr>
+</table>
+<% method_list["methods"].each do |methods| %>
+<table width="100%" cellspacing="0" cellpadding="5" border="0">
+<tr><td class="methodtitle">
+<a name="<%= methods["aref"] %>">
+<b><%= methods["name"] %></b><%= methods["params"] %>
+<% if methods["codeurl"] then %>
+<a href="<%= methods["codeurl"] %>" target="source" class="srclink">src</a>
+<% end %>
+</a></td></tr>
+</table>
+<% if method_list["m_desc"] then %>
+<div class="description">
+<%= method_list["m_desc"] %>
+</div>
+<% end %>
+<% end %><%# method_list["methods"] %>
+<% end %>
+<% end %><%# sections["method_list"] %>
+<% end %>
+<% end %><%# values["sections"] %>
+<% end %>
+  EOF
+
+  SRC_PAGE = <<-EOF
+<html>
+<head><title><%= values["title"] %></title>
+<meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>">
+<style type="text/css">
+  .kw { color: #3333FF; font-weight: bold }
+  .cmt { color: green; font-style: italic }
+  .str { color: #662222; font-style: italic }
+  .re  { color: #662222; }
+.ruby-comment    { color: green; font-style: italic }
+.ruby-constant   { color: #4433aa; font-weight: bold; }
+.ruby-identifier { color: #222222;  }
+.ruby-ivar       { color: #2233dd; }
+.ruby-keyword    { color: #3333FF; font-weight: bold }
+.ruby-node       { color: #777777; }
+.ruby-operator   { color: #111111;  }
+.ruby-regexp     { color: #662222; }
+.ruby-value      { color: #662222; font-style: italic }
+</style>
+</head>
+<body bgcolor="#BBBBBB">
+<pre><%= values["code"] %></pre>
+</body>
+</html>
+  EOF
+
+  FR_INDEX_BODY = %{
+<%= template_include %>
+}
+
+  FILE_INDEX = <<-EOF
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>">
+<style type="text/css">
+<!--
+  body {
+background-color: #bbbbbb;
+     font-family: #{FONTS};
+       font-size: 11px;
+      font-style: normal;
+     line-height: 14px;
+           color: #000040;
+  }
+div.banner {
+  background: #bbbbcc;
+  color:      white;
+  padding: 1;
+  margin: 0;
+  font-size: 90%;
+  font-weight: bold;
+  line-height: 1.1;
+  text-align: center;
+  width: 100%;
+}
+
+-->
+</style>
+<base target="docwin">
+</head>
+<body>
+<div class="banner"><%= values["list_title"] %></div>
+<% values["entries"].each do |entries| %>
+<a href="<%= entries["href"] %>"><%= entries["name"] %></a><br />
+<% end %><%# values["entries"] %>
+</body></html>
+  EOF
+
+  CLASS_INDEX = FILE_INDEX
+  METHOD_INDEX = FILE_INDEX
+
+  INDEX = <<-EOF
+<html>
+<head>
+  <title><%= values["title"] %></title>
+  <meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>">
+</head>
+
+<frameset cols="20%,*">
+    <frameset rows="15%,35%,50%">
+        <frame src="fr_file_index.html"   title="Files" name="Files">
+        <frame src="fr_class_index.html"  name="Modules">
+        <frame src="fr_method_index.html" name="Subroutines and Functions">
+    </frameset>
+    <frameset rows="80%,20%">
+      <frame  src="<%= values["initial_page"] %>" name="docwin">
+      <frame  src="blank.html" name="source">
+    </frameset>
+    <noframes>
+          <body bgcolor="#BBBBBB">
+            Click <a href="html/index.html">here</a> for a non-frames
+            version of this page.
+          </body>
+    </noframes>
+</frameset>
+
+</html>
+  EOF
+
+  # Blank page to use as a target
+  BLANK = %{
+<html><body bgcolor="#BBBBBB"></body></html>
+}
+
+  def write_extra_pages
+    template = TemplatePage.new(BLANK)
+    File.open("blank.html", "w") { |f| template.write_html_on(f, {}) }
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/generator/html/html.rb b/trunk/lib/rdoc/generator/html/html.rb
new file mode 100644
index 0000000000..1ab90c6264
--- /dev/null
+++ b/trunk/lib/rdoc/generator/html/html.rb
@@ -0,0 +1,698 @@
+require 'rdoc/generator/html'
+require 'rdoc/generator/html/one_page_html'
+
+##
+# = CSS2 RDoc HTML template
+#
+# This is a template for RDoc that uses XHTML 1.0 Transitional and dictates a
+# bit more of the appearance of the output to cascading stylesheets than the
+# default. It was designed for clean inline code display, and uses DHTMl to
+# toggle the visibility of each method's source with each click on the
+# '[source]' link.
+#
+# == Authors
+#
+# * Michael Granger <ged@FaerieMUD.org>
+#
+# Copyright (c) 2002, 2003 The FaerieMUD Consortium. Some rights reserved.
+#
+# This work is licensed under the Creative Commons Attribution License. To
+# view a copy of this license, visit
+# http://creativecommons.org/licenses/by/1.0/ or send a letter to Creative
+# Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.
+
+module RDoc::Generator::HTML::HTML
+
+  FONTS = "Verdana,Arial,Helvetica,sans-serif"
+
+  STYLE = <<-EOF
+body {
+    font-family: Verdana,Arial,Helvetica,sans-serif;
+    font-size:   90%;
+    margin: 0;
+    margin-left: 40px;
+    padding: 0;
+    background: white;
+}
+
+h1,h2,h3,h4 { margin: 0; color: #efefef; background: transparent; }
+h1 { font-size: 150%; }
+h2,h3,h4 { margin-top: 1em; }
+
+a { background: #eef; color: #039; text-decoration: none; }
+a:hover { background: #039; color: #eef; }
+
+/* Override the base stylesheet's Anchor inside a table cell */
+td > a {
+  background: transparent;
+  color: #039;
+  text-decoration: none;
+}
+
+/* and inside a section title */
+.section-title > a {
+  background: transparent;
+  color: #eee;
+  text-decoration: none;
+}
+
+/* === Structural elements =================================== */
+
+div#index {
+    margin: 0;
+    margin-left: -40px;
+    padding: 0;
+    font-size: 90%;
+}
+
+
+div#index a {
+    margin-left: 0.7em;
+}
+
+div#index .section-bar {
+   margin-left: 0px;
+   padding-left: 0.7em;
+   background: #ccc;
+   font-size: small;
+}
+
+
+div#classHeader, div#fileHeader {
+    width: auto;
+    color: white;
+    padding: 0.5em 1.5em 0.5em 1.5em;
+    margin: 0;
+    margin-left: -40px;
+    border-bottom: 3px solid #006;
+}
+
+div#classHeader a, div#fileHeader a {
+    background: inherit;
+    color: white;
+}
+
+div#classHeader td, div#fileHeader td {
+    background: inherit;
+    color: white;
+}
+
+
+div#fileHeader {
+    background: #057;
+}
+
+div#classHeader {
+    background: #048;
+}
+
+
+.class-name-in-header {
+  font-size:  180%;
+  font-weight: bold;
+}
+
+
+div#bodyContent {
+    padding: 0 1.5em 0 1.5em;
+}
+
+div#description {
+    padding: 0.5em 1.5em;
+    background: #efefef;
+    border: 1px dotted #999;
+}
+
+div#description h1,h2,h3,h4,h5,h6 {
+    color: #125;;
+    background: transparent;
+}
+
+div#validator-badges {
+    text-align: center;
+}
+div#validator-badges img { border: 0; }
+
+div#copyright {
+    color: #333;
+    background: #efefef;
+    font: 0.75em sans-serif;
+    margin-top: 5em;
+    margin-bottom: 0;
+    padding: 0.5em 2em;
+}
+
+
+/* === Classes =================================== */
+
+table.header-table {
+    color: white;
+    font-size: small;
+}
+
+.type-note {
+    font-size: small;
+    color: #DEDEDE;
+}
+
+.xxsection-bar {
+    background: #eee;
+    color: #333;
+    padding: 3px;
+}
+
+.section-bar {
+   color: #333;
+   border-bottom: 1px solid #999;
+    margin-left: -20px;
+}
+
+
+.section-title {
+    background: #79a;
+    color: #eee;
+    padding: 3px;
+    margin-top: 2em;
+    margin-left: -30px;
+    border: 1px solid #999;
+}
+
+.top-aligned-row {  vertical-align: top }
+.bottom-aligned-row { vertical-align: bottom }
+
+/* --- Context section classes ----------------------- */
+
+.context-row { }
+.context-item-name { font-family: monospace; font-weight: bold; color: black; }
+.context-item-value { font-size: small; color: #448; }
+.context-item-desc { color: #333; padding-left: 2em; }
+
+/* --- Method classes -------------------------- */
+.method-detail {
+    background: #efefef;
+    padding: 0;
+    margin-top: 0.5em;
+    margin-bottom: 1em;
+    border: 1px dotted #ccc;
+}
+.method-heading {
+  color: black;
+  background: #ccc;
+  border-bottom: 1px solid #666;
+  padding: 0.2em 0.5em 0 0.5em;
+}
+.method-signature { color: black; background: inherit; }
+.method-name { font-weight: bold; }
+.method-args { font-style: italic; }
+.method-description { padding: 0 0.5em 0 0.5em; }
+
+/* --- Source code sections -------------------- */
+
+a.source-toggle { font-size: 90%; }
+div.method-source-code {
+    background: #262626;
+    color: #ffdead;
+    margin: 1em;
+    padding: 0.5em;
+    border: 1px dashed #999;
+    overflow: hidden;
+}
+
+div.method-source-code pre { color: #ffdead; overflow: hidden; }
+
+/* --- Ruby keyword styles --------------------- */
+
+.standalone-code { background: #221111; color: #ffdead; overflow: hidden; }
+
+.ruby-constant  { color: #7fffd4; background: transparent; }
+.ruby-keyword { color: #00ffff; background: transparent; }
+.ruby-ivar    { color: #eedd82; background: transparent; }
+.ruby-operator  { color: #00ffee; background: transparent; }
+.ruby-identifier { color: #ffdead; background: transparent; }
+.ruby-node    { color: #ffa07a; background: transparent; }
+.ruby-comment { color: #b22222; font-weight: bold; background: transparent; }
+.ruby-regexp  { color: #ffa07a; background: transparent; }
+.ruby-value   { color: #7fffd4; background: transparent; }
+EOF
+
+
+#####################################################################
+### H E A D E R   T E M P L A T E
+#####################################################################
+
+  XHTML_PREAMBLE = <<-EOF
+<?xml version="1.0" encoding="<%= values["charset"] %>"?>
+<!DOCTYPE html
+     PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+  EOF
+
+  HEADER = XHTML_PREAMBLE + <<-EOF
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+  <title><%= values["title"] %></title>
+  <meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>" />
+  <meta http-equiv="Content-Script-Type" content="text/javascript" />
+  <link rel="stylesheet" href="<%= values["style_url"] %>" type="text/css" media="screen" />
+  <script type="text/javascript">
+  // <![CDATA[
+
+  function popupCode( url ) {
+    window.open(url, "Code", "resizable=yes,scrollbars=yes,toolbar=no,status=no,height=150,width=400")
+  }
+
+  function toggleCode( id ) {
+    if ( document.getElementById )
+      elem = document.getElementById( id );
+    else if ( document.all )
+      elem = eval( "document.all." + id );
+    else
+      return false;
+
+    elemStyle = elem.style;
+
+    if ( elemStyle.display != "block" ) {
+      elemStyle.display = "block"
+    } else {
+      elemStyle.display = "none"
+    }
+
+    return true;
+  }
+
+  // Make codeblocks hidden by default
+  document.writeln( "<style type=\\"text/css\\">div.method-source-code { display: none }</style>" )
+
+  // ]]>
+  </script>
+
+</head>
+<body>
+EOF
+
+#####################################################################
+### C O N T E X T   C O N T E N T   T E M P L A T E
+#####################################################################
+
+  CONTEXT_CONTENT = %{
+}
+
+#####################################################################
+### F O O T E R   T E M P L A T E
+#####################################################################
+
+  FOOTER = <<-EOF
+<div id="validator-badges">
+  <p><small><a href="http://validator.w3.org/check/referer">[Validate]</a></small></p>
+</div>
+
+</body>
+</html>
+  EOF
+
+
+#####################################################################
+### F I L E   P A G E   H E A D E R   T E M P L A T E
+#####################################################################
+
+  FILE_PAGE = <<-EOF
+  <div id="fileHeader">
+    <h1><%= values["short_name"] %></h1>
+    <table class="header-table">
+    <tr class="top-aligned-row">
+      <td><strong>Path:</strong></td>
+      <td><%= values["full_path"] %>
+<% if values["cvsurl"] then %>
+        &nbsp;(<a href="<%= values["cvsurl"] %>"><acronym title="Concurrent Versioning System">CVS</acronym></a>)
+<% end %>
+      </td>
+    </tr>
+    <tr class="top-aligned-row">
+      <td><strong>Last Update:</strong></td>
+      <td><%= values["dtm_modified"] %></td>
+    </tr>
+    </table>
+  </div>
+  EOF
+
+#####################################################################
+### C L A S S   P A G E   H E A D E R   T E M P L A T E
+#####################################################################
+
+  CLASS_PAGE = <<-EOF
+    <div id="classHeader">
+        <table class="header-table">
+        <tr class="top-aligned-row">
+          <td><strong><%= values["classmod"] %></strong></td>
+          <td class="class-name-in-header"><%= values["full_name"] %></td>
+        </tr>
+        <tr class="top-aligned-row">
+            <td><strong>In:</strong></td>
+            <td>
+<% values["infiles"].each do |infiles| %>
+<% if infiles["full_path_url"] then %>
+                <a href="<%= infiles["full_path_url"] %>">
+<% end %>
+                <%= infiles["full_path"] %>
+<% if infiles["full_path_url"] then %>
+                </a>
+<% end %>
+<% if infiles["cvsurl"] then %>
+        &nbsp;(<a href="<%= infiles["cvsurl"] %>"><acronym title="Concurrent Versioning System">CVS</acronym></a>)
+<% end %>
+        <br />
+<% end %><%# values["infiles"] %>
+            </td>
+        </tr>
+
+<% if values["parent"] then %>
+        <tr class="top-aligned-row">
+            <td><strong>Parent:</strong></td>
+            <td>
+<% if values["par_url"] then %>
+                <a href="<%= values["par_url"] %>">
+<% end %>
+                <%= values["parent"] %>
+<% if values["par_url"] then %>
+               </a>
+<% end %>
+            </td>
+        </tr>
+<% end %>
+        </table>
+    </div>
+  EOF
+
+#####################################################################
+### M E T H O D   L I S T   T E M P L A T E
+#####################################################################
+
+  METHOD_LIST = <<-EOF
+  <div id="contextContent">
+<% if values["diagram"] then %>
+    <div id="diagram">
+      <%= values["diagram"] %>
+    </div>
+<% end
+
+   if values["description"] then %>
+    <div id="description">
+      <%= values["description"] %>
+    </div>
+<% end
+
+   if values["requires"] then %>
+    <div id="requires-list">
+      <h3 class="section-bar">Required files</h3>
+
+      <div class="name-list">
+<% values["requires"].each do |requires| %>
+        <%= href requires["aref"], requires["name"] %>&nbsp;&nbsp;
+<% end %><%# values["requires"] %>
+      </div>
+    </div>
+<% end
+
+   if values["toc"] then %>
+    <div id="contents-list">
+      <h3 class="section-bar">Contents</h3>
+      <ul>
+<% values["toc"].each do |toc| %>
+      <li><a href="#<%= toc["href"] %>"><%= toc["secname"] %></a></li>
+<% end %><%# values["toc"] %>
+     </ul>
+<% end %>
+   </div>
+
+<% if values["methods"] then %>
+    <div id="method-list">
+      <h3 class="section-bar">Methods</h3>
+
+      <div class="name-list">
+<%   values["methods"].each do |methods| %>
+        <%= href methods["aref"], methods["name"] %>&nbsp;&nbsp;
+<%   end %><%# values["methods"] %>
+      </div>
+    </div>
+<% end %>
+  </div>
+
+    <!-- if includes -->
+<% if values["includes"] then %>
+    <div id="includes">
+      <h3 class="section-bar">Included Modules</h3>
+
+      <div id="includes-list">
+<% values["includes"].each do |includes| %>
+        <span class="include-name"><%= href includes["aref"], includes["name"] %></span>
+<% end %><%# values["includes"] %>
+      </div>
+    </div>
+<% end
+
+   values["sections"].each do |sections| %>
+    <div id="section">
+<%   if sections["sectitle"] then %>
+      <h2 class="section-title"><a name="<%= sections["secsequence"] %>"><%= sections["sectitle"] %></a></h2>
+<%     if sections["seccomment"] then %>
+      <div class="section-comment">
+        <%= sections["seccomment"] %>
+      </div>
+<%     end
+     end
+
+     if sections["classlist"] then %>
+    <div id="class-list">
+      <h3 class="section-bar">Classes and Modules</h3>
+
+      <%= sections["classlist"] %>
+    </div>
+<%   end
+
+     if sections["constants"] then %>
+    <div id="constants-list">
+      <h3 class="section-bar">Constants</h3>
+
+      <div class="name-list">
+        <table summary="Constants">
+<%     sections["constants"].each do |constants| %>
+        <tr class="top-aligned-row context-row">
+          <td class="context-item-name"><%= constants["name"] %></td>
+          <td>=</td>
+          <td class="context-item-value"><%= constants["value"] %></td>
+<%       if sections["desc"] then %>
+          <td width="3em">&nbsp;</td>
+          <td class="context-item-desc"><%= constants["desc"] %></td>
+<%       end %>
+        </tr>
+<%     end %><%# sections["constants"] %>
+        </table>
+      </div>
+    </div>
+<%   end
+
+     if sections["aliases"] then %>
+    <div id="aliases-list">
+      <h3 class="section-bar">External Aliases</h3>
+
+      <div class="name-list">
+      <table summary="aliases">
+<%     sections["aliases"].each do |aliases| %>
+        <tr class="top-aligned-row context-row">
+          <td class="context-item-name"><%= aliases["old_name"] %></td>
+          <td>-&gt;</td>
+          <td class="context-item-value"><%= aliases["new_name"] %></td>
+        </tr>
+<%       if aliases["desc"] then %>
+      <tr class="top-aligned-row context-row">
+        <td>&nbsp;</td>
+        <td colspan="2" class="context-item-desc"><%= aliases["desc"] %></td>
+      </tr>
+<%       end
+       end %><%# sections["aliases"] %>
+        </table>
+      </div>
+    </div>
+<%   end %>
+
+<%   if sections["attributes"] then %>
+    <div id="attribute-list">
+      <h3 class="section-bar">Attributes</h3>
+
+      <div class="name-list">
+        <table>
+<%     sections["attributes"].each do |attribute| %>
+        <tr class="top-aligned-row context-row">
+          <td class="context-item-name"><%= attribute["name"] %></td>
+<%       if attribute["rw"] then %>
+          <td class="context-item-value">&nbsp;[<%= attribute["rw"] %>]&nbsp;</td>
+<%       end
+         unless attribute["rw"] then %>
+          <td class="context-item-value">&nbsp;&nbsp;</td>
+<%       end %>
+          <td class="context-item-desc"><%= attribute["a_desc"] %></td>
+        </tr>
+<%     end %><%# sections["attributes"] %>
+        </table>
+      </div>
+    </div>
+<%   end %>
+
+    <!-- if method_list -->
+<%   if sections["method_list"] then %>
+    <div id="methods">
+<%     sections["method_list"].each do |method_list|
+         if method_list["methods"] then %>
+      <h3 class="section-bar"><%= method_list["type"] %> <%= method_list["category"] %> methods</h3>
+
+<%         method_list["methods"].each do |methods| %>
+      <div id="method-<%= methods["aref"] %>" class="method-detail">
+        <a name="<%= methods["aref"] %>"></a>
+
+        <div class="method-heading">
+<%           if methods["codeurl"] then %>
+          <a href="<%= methods["codeurl"] %>" target="Code" class="method-signature"
+            onclick="popupCode('<%= methods["codeurl"] %>');return false;">
+<%           end
+             if methods["sourcecode"] then %>
+          <a href="#<%= methods["aref"] %>" class="method-signature">
+<%           end
+             if methods["callseq"] then %>
+          <span class="method-name"><%= methods["callseq"] %></span>
+<%           end
+             unless methods["callseq"] then %>
+          <span class="method-name"><%= methods["name"] %></span><span class="method-args"><%= methods["params"] %></span>
+<%           end
+             if methods["codeurl"] then %>
+          </a>
+<%           end
+             if methods["sourcecode"] then %>
+          </a>
+<%           end %>
+        </div>
+
+        <div class="method-description">
+<%           if methods["m_desc"] then %>
+          <%= methods["m_desc"] %>
+<%           end
+               if methods["sourcecode"] then %>
+          <p><a class="source-toggle" href="#"
+            onclick="toggleCode('<%= methods["aref"] %>-source');return false;">[Source]</a></p>
+          <div class="method-source-code" id="<%= methods["aref"] %>-source">
+<pre>
+<%= methods["sourcecode"] %>
+</pre>
+          </div>
+<%           end %>
+        </div>
+      </div>
+
+<%         end %><%# method_list["methods"] %><%
+         end
+       end %><%# sections["method_list"] %>
+
+    </div>
+<%   end %>
+<% end %><%# values["sections"] %>
+  EOF
+
+#####################################################################
+### B O D Y   T E M P L A T E
+#####################################################################
+
+  BODY = HEADER + %{
+
+<%= template_include %>  <!-- banner header -->
+
+  <div id="bodyContent">
+
+} +  METHOD_LIST + %{
+
+  </div>
+
+} + FOOTER
+
+#####################################################################
+### S O U R C E   C O D E   T E M P L A T E
+#####################################################################
+
+  SRC_PAGE = XHTML_PREAMBLE + <<-EOF
+<html>
+<head>
+  <title><%= values["title"] %></title>
+  <meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>" />
+  <link rel="stylesheet" href="<%= values["style_url"] %>" type="text/css" media="screen" />
+</head>
+<body class="standalone-code">
+  <pre><%= values["code"] %></pre>
+</body>
+</html>
+  EOF
+
+
+#####################################################################
+### I N D E X   F I L E   T E M P L A T E S
+#####################################################################
+
+  FR_INDEX_BODY = %{
+<%= template_include %>
+}
+
+  FILE_INDEX = XHTML_PREAMBLE + <<-EOF
+<!--
+
+    <%= values["list_title"] %>
+
+  -->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+  <title><%= values["list_title"] %></title>
+  <meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>" />
+  <link rel="stylesheet" href="<%= values["style_url"] %>" type="text/css" />
+  <base target="docwin" />
+</head>
+<body>
+<div id="index">
+  <h1 class="section-bar"><%= values["list_title"] %></h1>
+  <div id="index-entries">
+<% values["entries"].each do |entries| %>
+    <a href="<%= entries["href"] %>"><%= entries["name"] %></a><br />
+<% end %><%#  values["entries"] %>
+  </div>
+</div>
+</body>
+</html>
+  EOF
+
+  CLASS_INDEX = FILE_INDEX
+  METHOD_INDEX = FILE_INDEX
+
+  INDEX = <<-EOF
+<?xml version="1.0" encoding="<%= values["charset"] %>"?>
+<!DOCTYPE html
+     PUBLIC "-//W3C//DTD XHTML 1.0 Frameset//EN"
+     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-frameset.dtd">
+
+<!--
+
+    <%= values["title"] %>
+
+  -->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+  <title><%= values["title"] %></title>
+  <meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>" />
+</head>
+<frameset rows="20%, 80%">
+    <frameset cols="25%,35%,45%">
+        <frame src="fr_file_index.html"   title="Files" name="Files" />
+        <frame src="fr_class_index.html"  name="Classes" />
+        <frame src="fr_method_index.html" name="Methods" />
+    </frameset>
+    <frame src="<%= values["initial_page"] %>" name="docwin" />
+</frameset>
+</html>
+  EOF
+
+end
+
diff --git a/trunk/lib/rdoc/generator/html/kilmer.rb b/trunk/lib/rdoc/generator/html/kilmer.rb
new file mode 100644
index 0000000000..6479abaf8b
--- /dev/null
+++ b/trunk/lib/rdoc/generator/html/kilmer.rb
@@ -0,0 +1,418 @@
+require 'rdoc/generator/html'
+
+module RDoc::Generator::HTML::KILMER
+
+  FONTS = "Verdana, Arial, Helvetica, sans-serif"
+
+  STYLE = <<-EOF
+body,td,p { font-family: <%= values["fonts"] %>;
+       color: #000040;
+}
+
+.attr-rw { font-size: xx-small; color: #444488 }
+
+.title-row { background-color: #CCCCFF;
+             color:      #000010;
+}
+
+.big-title-font {
+  color: black;
+  font-weight: bold;
+  font-family: <%= values["fonts"] %>;
+  font-size: large;
+  height: 60px;
+  padding: 10px 3px 10px 3px;
+}
+
+.small-title-font { color: black;
+                    font-family: <%= values["fonts"] %>;
+                    font-size:10; }
+
+.aqua { color: black }
+
+.method-name, .attr-name {
+      font-family: font-family: <%= values["fonts"] %>;
+      font-weight: bold;
+      font-size: small;
+      margin-left: 20px;
+      color: #000033;
+}
+
+.tablesubtitle, .tablesubsubtitle {
+   width: 100%;
+   margin-top: 1ex;
+   margin-bottom: .5ex;
+   padding: 5px 0px 5px 3px;
+   font-size: large;
+   color: black;
+   background-color: #CCCCFF;
+   border: thin;
+}
+
+.name-list {
+  margin-left: 5px;
+  margin-bottom: 2ex;
+  line-height: 105%;
+}
+
+.description {
+  margin-left: 5px;
+  margin-bottom: 2ex;
+  line-height: 105%;
+  font-size: small;
+}
+
+.methodtitle {
+  font-size: small;
+  font-weight: bold;
+  text-decoration: none;
+  color: #000033;
+  background-color: white;
+}
+
+.srclink {
+  font-size: small;
+  font-weight: bold;
+  text-decoration: none;
+  color: #0000DD;
+  background-color: white;
+}
+
+.paramsig {
+   font-size: small;
+}
+
+.srcbut { float: right }
+  EOF
+
+  BODY = <<-EOF
+<html><head>
+  <title><%= values["title"] %></title>
+  <meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>">
+  <link rel="stylesheet" href="<%= values["style_url"] %>" type="text/css" media="screen" />
+  <script type="text/javascript" language="JavaScript">
+  <!--
+  function popCode(url) {
+    parent.frames.source.location = url
+  }
+  //-->
+  </script>
+</head>
+<body bgcolor="white">
+
+<%= template_include %>  <!-- banner header -->
+
+<% if values["diagram"] then %>
+<table width="100%"><tr><td align="center">
+<%= values["diagram"] %>
+</td></tr></table>
+<% end %>
+
+<% if values["description"] then %>
+<div class="description"><%= values["description"] %></div>
+<% end %>
+
+<% if values["requires"] then %>
+<table cellpadding="5" width="100%">
+<tr><td class="tablesubtitle">Required files</td></tr>
+</table><br />
+<div class="name-list">
+<% values["requires"].each do |requires| %>
+<%= href requires["aref"], requires["name"] %>
+<% end %><%# values["requires"] %>
+<% end %>
+</div>
+
+<% if values["methods"] then %>
+<table cellpadding="5" width="100%">
+<tr><td class="tablesubtitle">Methods</td></tr>
+</table><br />
+<div class="name-list">
+<% values["methods"].each do |methods| %>
+<%= href methods["aref"], methods["name"] %>,
+<% end %><%# values["methods"] %>
+</div>
+<% end %>
+
+
+<% values["sections"].each do |sections| %>
+    <div id="section">
+<% if sections["sectitle"] then %>
+      <h2 class="section-title"><a name="<%= sections["secsequence"] %>"><%= sections["sectitle"] %></a></h2>
+<% if sections["seccomment"] then %>
+      <div class="section-comment">
+        <%= sections["seccomment"] %>
+      </div>
+<% end %>
+<% end %>
+
+<% if sections["attributes"] then %>
+<table cellpadding="5" width="100%">
+<tr><td class="tablesubtitle">Attributes</td></tr>
+</table><br />
+<table cellspacing="5">
+<% sections["attributes"].each do |attributes| %>
+     <tr valign="top">
+<% if attributes["rw"] then %>
+       <td align="center" class="attr-rw">&nbsp;[<%= attributes["rw"] %>]&nbsp;</td>
+<% end %>
+<% unless attributes["rw"] then %>
+       <td></td>
+<% end %>
+       <td class="attr-name"><%= attributes["name"] %></td>
+       <td><%= attributes["a_desc"] %></td>
+     </tr>
+<% end %><%# sections["attributes"] %>
+</table>
+<% end %>
+
+<% if sections["classlist"] then %>
+<table cellpadding="5" width="100%">
+<tr><td class="tablesubtitle">Classes and Modules</td></tr>
+</table><br />
+<%= sections["classlist"] %><br />
+<% end %>
+
+  <%= template_include %>  <!-- method descriptions -->
+
+<% end %><%# values["sections"] %>
+
+</body>
+</html>
+  EOF
+
+  FILE_PAGE = <<-EOF
+<table width="100%">
+ <tr class="title-row">
+ <td><table width="100%"><tr>
+   <td class="big-title-font" colspan="2"><font size="-3"><b>File</b><br /></font><%= values["short_name"] %></td>
+   <td align="right"><table cellspacing="0" cellpadding="2">
+         <tr>
+           <td  class="small-title-font">Path:</td>
+           <td class="small-title-font"><%= values["full_path"] %>
+<% if values["cvsurl"] then %>
+				&nbsp;(<a href="<%= values["cvsurl"] %>"><acronym title="Concurrent Versioning System">CVS</acronym></a>)
+<% end %>
+           </td>
+         </tr>
+         <tr>
+           <td class="small-title-font">Modified:</td>
+           <td class="small-title-font"><%= values["dtm_modified"] %></td>
+         </tr>
+        </table>
+    </td></tr></table></td>
+  </tr>
+</table><br />
+  EOF
+
+  CLASS_PAGE = <<-EOF
+<table width="100%" border="0" cellspacing="0">
+ <tr class="title-row">
+ <td class="big-title-font">
+   <font size="-3"><b><%= values["classmod"] %></b><br /></font><%= values["full_name"] %>
+ </td>
+ <td align="right">
+   <table cellspacing="0" cellpadding="2">
+     <tr valign="top">
+      <td class="small-title-font">In:</td>
+      <td class="small-title-font">
+<% values["infiles"].each do |infiles| %>
+<%= href infiles["full_path_url"], infiles["full_path"] %>
+<% if infiles["cvsurl"] then %>
+&nbsp;(<a href="<%= infiles["cvsurl"] %>"><acronym title="Concurrent Versioning System">CVS</acronym></a>)
+<% end %>
+<% end %><%# values["infiles"] %>
+      </td>
+     </tr>
+<% if values["parent"] then %>
+     <tr>
+      <td class="small-title-font">Parent:</td>
+      <td class="small-title-font">
+<% if values["par_url"] then %>
+        <a href="<%= values["par_url"] %>" class="cyan">
+<% end %>
+<%= values["parent"] %>
+<% if values["par_url"] then %>
+         </a>
+<% end %>
+      </td>
+     </tr>
+<% end %>
+   </table>
+  </td>
+  </tr>
+</table><br />
+  EOF
+
+  METHOD_LIST = <<-EOF
+<% if values["includes"] then %>
+<div class="tablesubsubtitle">Included modules</div><br />
+<div class="name-list">
+<% values["includes"].each do |includes| %>
+    <span class="method-name"><%= href includes["aref"], includes["name"] %></span>
+<% end %><%# values["includes"] %>
+</div>
+<% end %>
+
+<% if values["method_list"] then %>
+<% values["method_list"].each do |method_list| $stderr.puts({ :method_list => method_list }.inspect) %>
+<% if values["methods"] then %>
+<table cellpadding=5 width="100%">
+<tr><td class="tablesubtitle"><%= values["type"] %> <%= values["category"] %> methods</td></tr>
+</table>
+<% values["methods"].each do |methods| $stderr.puts({ :methods => methods }.inspect) %>
+<table width="100%" cellspacing="0" cellpadding="5" border="0">
+<tr><td class="methodtitle">
+<a name="<%= values["aref"] %>">
+<% if values["callseq"] then %>
+<b><%= values["callseq"] %></b>
+<% end %>
+<% unless values["callseq"] then %>
+ <b><%= values["name"] %></b><%= values["params"] %>
+<% end %>
+<% if values["codeurl"] then %>
+<a href="<%= values["codeurl"] %>" target="source" class="srclink">src</a>
+<% end %>
+</a></td></tr>
+</table>
+<% if values["m_desc"] then %>
+<div class="description">
+<%= values["m_desc"] %>
+</div>
+<% end %>
+<% if values["aka"] then %>
+<div class="aka">
+This method is also aliased as
+<% values["aka"].each do |aka| $stderr.puts({ :aka => aka }.inspect) %>
+<a href="<%= values["aref"] %>"><%= values["name"] %></a>
+<% end %><%# values["aka"] %>
+</div>
+<% end %>
+<% if values["sourcecode"] then %>
+<pre class="source">
+<%= values["sourcecode"] %>
+</pre>
+<% end %>
+<% end %><%# values["methods"] %>
+<% end %>
+<% end %><%# values["method_list"] %>
+<% end %>
+  EOF
+
+  SRC_PAGE = <<-EOF
+<html>
+<head><title><%= values["title"] %></title>
+<meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>">
+<style type="text/css">
+.ruby-comment    { color: green; font-style: italic }
+.ruby-constant   { color: #4433aa; font-weight: bold; }
+.ruby-identifier { color: #222222;  }
+.ruby-ivar       { color: #2233dd; }
+.ruby-keyword    { color: #3333FF; font-weight: bold }
+.ruby-node       { color: #777777; }
+.ruby-operator   { color: #111111;  }
+.ruby-regexp     { color: #662222; }
+.ruby-value      { color: #662222; font-style: italic }
+  .kw { color: #3333FF; font-weight: bold }
+  .cmt { color: green; font-style: italic }
+  .str { color: #662222; font-style: italic }
+  .re  { color: #662222; }
+</style>
+</head>
+<body bgcolor="white">
+<pre><%= values["code"] %></pre>
+</body>
+</html>
+  EOF
+
+  FR_INDEX_BODY = %{
+<%= template_include %>
+}
+
+  FILE_INDEX = <<-EOF
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>">
+<style>
+<!--
+body {
+  background-color: #ddddff;
+  font-family: #{FONTS};
+  font-size: 11px;
+  font-style: normal;
+  line-height: 14px;
+  color: #000040;
+}
+
+div.banner {
+  background: #0000aa;
+  color: white;
+  padding: 1;
+  margin: 0;
+  font-size: 90%;
+  font-weight: bold;
+  line-height: 1.1;
+  text-align: center;
+  width: 100%;
+}
+
+-->
+</style>
+<base target="docwin">
+</head>
+<body>
+<div class="banner"><%= values["list_title"] %></div>
+<% values["entries"].each do |entries| %>
+<a href="<%= entries["href"] %>"><%= entries["name"] %></a><br />
+<% end %><%# values["entries"] %>
+</body></html>
+  EOF
+
+  CLASS_INDEX = FILE_INDEX
+  METHOD_INDEX = FILE_INDEX
+
+  INDEX = <<-EOF
+<html>
+<head>
+  <title><%= values["title"] %></title>
+  <meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>">
+</head>
+
+<frameset cols="20%,*">
+    <frameset rows="15%,35%,50%">
+        <frame src="fr_file_index.html"   title="Files" name="Files">
+        <frame src="fr_class_index.html"  name="Classes">
+        <frame src="fr_method_index.html" name="Methods">
+    </frameset>
+<% if values["inline_source"] then %>
+      <frame  src="<%= values["initial_page"] %>" name="docwin">
+<% end %>
+<% unless values["inline_source"] then %>
+    <frameset rows="80%,20%">
+      <frame  src="<%= values["initial_page"] %>" name="docwin">
+      <frame  src="blank.html" name="source">
+    </frameset>
+<% end %>
+    <noframes>
+          <body bgcolor="white">
+            Click <a href="html/index.html">here</a> for a non-frames
+            version of this page.
+          </body>
+    </noframes>
+</frameset>
+
+</html>
+  EOF
+
+  # A blank page to use as a target
+  BLANK = %{
+<html><body bgcolor="white"></body></html>
+}
+
+  def write_extra_pages
+    template = TemplatePage.new(BLANK)
+    File.open("blank.html", "w") { |f| template.write_html_on(f, {}) }
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/generator/html/one_page_html.rb b/trunk/lib/rdoc/generator/html/one_page_html.rb
new file mode 100644
index 0000000000..c4dd95529d
--- /dev/null
+++ b/trunk/lib/rdoc/generator/html/one_page_html.rb
@@ -0,0 +1,121 @@
+require 'rdoc/generator/html'
+
+module RDoc::Generator::HTML::ONE_PAGE_HTML
+
+  CONTENTS_XML = <<-EOF
+<% if defined? classes and classes["description"] then %>
+<%= classes["description"] %>
+<% end %>
+
+<% if defined? files and files["requires"] then %>
+<h4>Requires:</h4>
+<ul>
+<% files["requires"].each do |requires| %>
+<% if requires["aref"] then %>
+<li><a href="<%= requires["aref"] %>"><%= requires["name"] %></a></li>
+<% end %>
+<% unless requires["aref"] then %>
+<li><%= requires["name"] %></li>
+<% end %>
+<% end %><%# files["requires"] %>
+</ul>
+<% end %>
+
+<% if defined? classes and classes["includes"] then %>
+<h4>Includes</h4>
+<ul>
+<% classes["includes"].each do |includes| %>
+<% if includes["aref"] then %>
+<li><a href="<%= includes["aref"] %>"><%= includes["name"] %></a></li>
+<% end %>
+<% unless includes["aref"] then %>
+<li><%= includes["name"] %></li>
+<% end %>
+<% end %><%# classes["includes"] %>
+</ul>
+<% end %>
+
+<% if defined? classes and classes["sections"] then %>
+<% classes["sections"].each do |sections| %>
+<% if sections["attributes"] then %>
+<h4>Attributes</h4>
+<table>
+<% sections["attributes"].each do |attributes| %>
+<tr><td><%= attributes["name"] %></td><td><%= attributes["rw"] %></td><td><%= attributes["a_desc"] %></td></tr>
+<% end %><%# sections["attributes"] %>
+</table>
+<% end %>
+
+<% if sections["method_list"] then %>
+<h3>Methods</h3>
+<% sections["method_list"].each do |method_list| %>
+<% if method_list["methods"] then %>
+<% method_list["methods"].each do |methods| %>
+<h4><%= methods["type"] %> <%= methods["category"] %> method: 
+<% if methods["callseq"] then %>
+<a name="<%= methods["aref"] %>"><%= methods["callseq"] %></a>
+<% end %>
+<% unless methods["callseq"] then %>
+<a name="<%= methods["aref"] %>"><%= methods["name"] %><%= methods["params"] %></a></h4>
+<% end %>
+
+<% if methods["m_desc"] then %>
+<%= methods["m_desc"] %>
+<% end %>
+
+<% if methods["sourcecode"] then %>
+<blockquote><pre>
+<%= methods["sourcecode"] %>
+</pre></blockquote>
+<% end %>
+<% end %><%# method_list["methods"] %>
+<% end %>
+<% end %><%# sections["method_list"] %>
+<% end %>
+<% end %><%# classes["sections"] %>
+<% end %>
+  EOF
+
+  ONE_PAGE = %{
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html>
+<head>
+  <title><%= values["title"] %></title>
+  <meta http-equiv="Content-Type" content="text/html; charset=<%= values["charset"] %>" />
+</head>
+<body>
+<% values["files"].each do |files| %>
+<h2>File: <%= files["short_name"] %></h2>
+<table>
+  <tr><td>Path:</td><td><%= files["full_path"] %></td></tr>
+  <tr><td>Modified:</td><td><%= files["dtm_modified"] %></td></tr>
+</table>
+} + CONTENTS_XML + %{
+<% end %><%# values["files"] %>
+
+<% if values["classes"] then %>
+<h2>Classes</h2>
+<% values["classes"].each do |classes| %>
+<% if classes["parent"] then %>
+<h3><%= classes["classmod"] %> <%= classes["full_name"] %> &lt; <%= href classes["par_url"], classes["parent"] %></h3>
+<% end %>
+<% unless classes["parent"] then %>
+<h3><%= classes["classmod"] %> <%= classes["full_name"] %></h3>
+<% end %>
+
+<% if classes["infiles"] then %>
+(in files
+<% classes["infiles"].each do |infiles| %>
+<%= href infiles["full_path_url"], infiles["full_path"] %>
+<% end %><%# classes["infiles"] %>
+)
+<% end %>
+} + CONTENTS_XML + %{
+<% end %><%# values["classes"] %>
+<% end %>
+</body>
+</html>
+}
+
+end
+
diff --git a/trunk/lib/rdoc/generator/ri.rb b/trunk/lib/rdoc/generator/ri.rb
new file mode 100644
index 0000000000..6b7a5932f8
--- /dev/null
+++ b/trunk/lib/rdoc/generator/ri.rb
@@ -0,0 +1,226 @@
+require 'rdoc/generator'
+require 'rdoc/markup/to_flow'
+
+require 'rdoc/ri/cache'
+require 'rdoc/ri/reader'
+require 'rdoc/ri/writer'
+require 'rdoc/ri/descriptions'
+
+class RDoc::Generator::RI
+
+  ##
+  # Generator may need to return specific subclasses depending on the
+  # options they are passed. Because of this we create them using a factory
+
+  def self.for(options)
+    new(options)
+  end
+
+  ##
+  # Set up a new ri generator
+
+  def initialize(options) #:not-new:
+    @options   = options
+    @ri_writer = RDoc::RI::Writer.new "."
+    @markup    = RDoc::Markup.new
+    @to_flow   = RDoc::Markup::ToFlow.new
+
+    @generated = {}
+  end
+
+  ##
+  # Build the initial indices and output objects based on an array of
+  # TopLevel objects containing the extracted information.
+
+  def generate(toplevels)
+    RDoc::TopLevel.all_classes_and_modules.each do |cls|
+      process_class cls
+    end
+  end
+
+  def process_class(from_class)
+    generate_class_info(from_class)
+
+    # now recurse into this class' constituent classes
+    from_class.each_classmodule do |mod|
+      process_class(mod)
+    end
+  end
+
+  def generate_class_info(cls)
+    case cls
+    when RDoc::NormalModule then
+      cls_desc = RDoc::RI::ModuleDescription.new
+    else
+      cls_desc = RDoc::RI::ClassDescription.new
+      cls_desc.superclass = cls.superclass
+    end
+
+    cls_desc.name        = cls.name
+    cls_desc.full_name   = cls.full_name
+    cls_desc.comment     = markup(cls.comment)
+
+    cls_desc.attributes = cls.attributes.sort.map do |a|
+      RDoc::RI::Attribute.new(a.name, a.rw, markup(a.comment))
+    end
+
+    cls_desc.constants = cls.constants.map do |c|
+      RDoc::RI::Constant.new(c.name, c.value, markup(c.comment))
+    end
+
+    cls_desc.includes = cls.includes.map do |i|
+      RDoc::RI::IncludedModule.new(i.name)
+    end
+
+    class_methods, instance_methods = method_list(cls)
+
+    cls_desc.class_methods = class_methods.map do |m|
+      RDoc::RI::MethodSummary.new(m.name)
+    end
+
+    cls_desc.instance_methods = instance_methods.map do |m|
+      RDoc::RI::MethodSummary.new(m.name)
+    end
+
+    update_or_replace(cls_desc)
+
+    class_methods.each do |m|
+      generate_method_info(cls_desc, m)
+    end
+
+    instance_methods.each do |m|
+      generate_method_info(cls_desc, m)
+    end
+  end
+
+  def generate_method_info(cls_desc, method)
+    meth_desc = RDoc::RI::MethodDescription.new
+    meth_desc.name = method.name
+    meth_desc.full_name = cls_desc.full_name
+    if method.singleton
+      meth_desc.full_name += "::"
+    else
+      meth_desc.full_name += "#"
+    end
+    meth_desc.full_name << method.name
+
+    meth_desc.comment = markup(method.comment)
+    meth_desc.params = params_of(method)
+    meth_desc.visibility = method.visibility.to_s
+    meth_desc.is_singleton = method.singleton
+    meth_desc.block_params = method.block_params
+
+    meth_desc.aliases = method.aliases.map do |a|
+      RDoc::RI::AliasName.new(a.name)
+    end
+
+    @ri_writer.add_method(cls_desc, meth_desc)
+  end
+
+  private
+
+  ##
+  # Returns a list of class and instance methods that we'll be documenting
+
+  def method_list(cls)
+    list = cls.method_list
+    unless @options.show_all
+      list = list.find_all do |m|
+        m.visibility == :public || m.visibility == :protected || m.force_documentation
+      end
+    end
+
+    c = []
+    i = []
+    list.sort.each do |m|
+      if m.singleton
+        c << m
+      else
+        i << m
+      end
+    end
+    return c,i
+  end
+
+  def params_of(method)
+    if method.call_seq
+      method.call_seq
+    else
+      params = method.params || ""
+
+      p = params.gsub(/\s*\#.*/, '')
+      p = p.tr("\n", " ").squeeze(" ")
+      p = "(" + p + ")" unless p[0] == ?(
+
+      if (block = method.block_params)
+        block.gsub!(/\s*\#.*/, '')
+        block = block.tr("\n", " ").squeeze(" ")
+        if block[0] == ?(
+          block.sub!(/^\(/, '').sub!(/\)/, '')
+        end
+        p << " {|#{block.strip}| ...}"
+      end
+      p
+    end
+  end
+
+  def markup(comment)
+    return nil if !comment || comment.empty?
+
+    # Convert leading comment markers to spaces, but only
+    # if all non-blank lines have them
+
+    if comment =~ /^(?>\s*)[^\#]/
+      content = comment
+    else
+      content = comment.gsub(/^\s*(#+)/)  { $1.tr('#',' ') }
+    end
+    @markup.convert(content, @to_flow)
+  end
+
+  ##
+  # By default we replace existing classes with the same name. If the
+  # --merge option was given, we instead merge this definition into an
+  # existing class. We add our methods, aliases, etc to that class, but do
+  # not change the class's description.
+
+  def update_or_replace(cls_desc)
+    old_cls = nil
+
+    if @options.merge
+      rdr = RDoc::RI::Reader.new RDoc::RI::Cache.new(@options.op_dir)
+
+      namespace = rdr.top_level_namespace
+      namespace = rdr.lookup_namespace_in(cls_desc.name, namespace)
+      if namespace.empty?
+        $stderr.puts "You asked me to merge this source into existing "
+        $stderr.puts "documentation. This file references a class or "
+        $stderr.puts "module called #{cls_desc.name} which I don't"
+        $stderr.puts "have existing documentation for."
+        $stderr.puts
+        $stderr.puts "Perhaps you need to generate its documentation first"
+        exit 1
+      else
+        old_cls = namespace[0]
+      end
+    end
+
+    prev_cls = @generated[cls_desc.full_name]
+
+    if old_cls and not prev_cls then
+      old_desc = rdr.get_class old_cls
+      cls_desc.merge_in old_desc
+    end
+
+    if prev_cls then
+      cls_desc.merge_in prev_cls
+    end
+
+    @generated[cls_desc.full_name] = cls_desc
+
+    @ri_writer.remove_class cls_desc
+    @ri_writer.add_class cls_desc
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/generator/texinfo.rb b/trunk/lib/rdoc/generator/texinfo.rb
new file mode 100644
index 0000000000..0b79820228
--- /dev/null
+++ b/trunk/lib/rdoc/generator/texinfo.rb
@@ -0,0 +1,84 @@
+require 'rdoc/rdoc'
+require 'rdoc/generator'
+require 'rdoc/markup/to_texinfo'
+
+module RDoc
+  RDoc::GENERATORS['texinfo'] = RDoc::Generator.new("rdoc/generator/texinfo",
+                                                    :Texinfo,
+                                                    'texinfo')
+  module Generator
+    # This generates Texinfo files for viewing with GNU Info or Emacs
+    # from RDoc extracted from Ruby source files.
+    class Texinfo
+      # What should the .info file be named by default?
+      DEFAULT_INFO_FILENAME = 'rdoc.info'
+
+      include Generator::MarkUp
+
+      # Accept some options
+      def initialize(options)
+        @options = options
+        @options.inline_source = true
+        @options.op_name ||= 'rdoc.texinfo'
+        @options.formatter = ::RDoc::Markup::ToTexInfo.new
+      end
+
+      # Generate the +texinfo+ files
+      def generate(toplevels)
+        @toplevels = toplevels
+        @files, @classes = ::RDoc::Generator::Context.build_indicies(@toplevels,
+                                                                     @options)
+
+        (@files + @classes).each { |x| x.value_hash }
+
+        open(@options.op_name, 'w') do |f|
+          f.puts TexinfoTemplate.new('files' => @files,
+                                     'classes' => @classes,
+                                     'filename' => @options.op_name.gsub(/texinfo/, 'info'),
+                                     'title' => @options.title).render
+        end
+        # TODO: create info files and install?
+      end
+
+      class << self
+        # Factory? We don't need no stinkin' factory!
+        alias_method :for, :new
+      end
+    end
+
+    # Basically just a wrapper around ERB.
+    # Should probably use RDoc::TemplatePage instead
+    class TexinfoTemplate
+      BASE_DIR = ::File.expand_path(::File.dirname(__FILE__)) # have to calculate this when the file's loaded.
+
+      def initialize(values, file = 'texinfo.erb')
+        @v, @file = [values, file]
+      end
+     
+      def template
+        ::File.read(::File.join(BASE_DIR, 'texinfo', @file))
+      end
+
+      # Go!
+      def render
+        ERB.new(template).result binding
+      end
+
+      def href(location, text)
+        text # TODO: how does texinfo do hyperlinks?
+      end
+
+      def target(name, text)
+        text # TODO: how do hyperlink targets work?
+      end
+
+      # TODO: this is probably implemented elsewhere?
+      def method_prefix(section)
+        { 'Class' => '.',
+          'Module' => '::',
+          'Instance' => '#',
+        }[section['category']]
+      end
+    end
+  end
+end
diff --git a/trunk/lib/rdoc/generator/texinfo/class.texinfo.erb b/trunk/lib/rdoc/generator/texinfo/class.texinfo.erb
new file mode 100644
index 0000000000..07f17eaef2
--- /dev/null
+++ b/trunk/lib/rdoc/generator/texinfo/class.texinfo.erb
@@ -0,0 +1,44 @@
+@node <%= @v['class']['full_name'].gsub(/::/, '-') %>
+@chapter <%= @v['class']["classmod"] %> <%= @v['class']['full_name'] %>
+
+<% if @v['class']["parent"] and @v['class']['par_url'] %>
+Inherits <%= href @v['class']["par_url"], @v['class']["parent"] %><% end %>
+
+<%= @v['class']["description"] %>
+
+<% if @v['class']["includes"] %>
+Includes
+<%   @v['class']["includes"].each do |include| %>
+* <%=  href include["aref"], include["name"] %>
+<%   end # @v['class']["includes"] %>
+<% end %>
+
+<% if @v['class']["sections"] %>
+<%   @v['class']["sections"].each do |section| %>
+<%     if section["attributes"] %>
+Attributes
+<%       section["attributes"].each do |attributes| %>
+* <%=      attributes["name"] %> <%= attributes["rw"] %> <%= attributes["a_desc"] %>
+<%       end # section["attributes"] %>
+<%     end %>
+<%   end %>
+
+<%   @v['class']["sections"].each do |section| %>
+<%     if section["method_list"] %>
+Methods
+@menu
+<%     section["method_list"].each_with_index do |method_list, i| %>
+<%= i %>
+<%       (method_list["methods"] || []).each do |method| %>
+* <%= @v['class']['full_name'].gsub(/::/, '-') %><%= method_prefix method_list %><%= method['name'] %>::<% end %>
+<% end %>
+@end menu
+
+<%     section["method_list"].each do |method_list| %>
+<%       (method_list["methods"] || []).uniq.each do |method| %>
+<%=        TexinfoTemplate.new(@v.merge({'method' => method, 'list' => method_list}),
+                               'method.texinfo.erb').render %><% end %>
+<%     end # section["method_list"] %>
+<%   end %>
+<% end # @v['class']["sections"] %>
+<% end %>
diff --git a/trunk/lib/rdoc/generator/texinfo/file.texinfo.erb b/trunk/lib/rdoc/generator/texinfo/file.texinfo.erb
new file mode 100644
index 0000000000..b619b94bd2
--- /dev/null
+++ b/trunk/lib/rdoc/generator/texinfo/file.texinfo.erb
@@ -0,0 +1,6 @@
+<% if false %>
+<h2>File: <%= @v['file']["short_name"] %></h2>
+Path: <%= @v['file']["full_path"] %>
+
+<%= TexinfoTemplate.new(@v, 'content.texinfo.erb').render %>
+<% end %>
diff --git a/trunk/lib/rdoc/generator/texinfo/method.texinfo.erb b/trunk/lib/rdoc/generator/texinfo/method.texinfo.erb
new file mode 100644
index 0000000000..f5c2b73a4b
--- /dev/null
+++ b/trunk/lib/rdoc/generator/texinfo/method.texinfo.erb
@@ -0,0 +1,6 @@
+@node <%= @v['class']['full_name'].gsub(/::/, '-') %><%= method_prefix @v['list'] %><%= @v['method']['name'] %>
+@section <%= @v['class']["classmod"] %> <%= @v['class']['full_name'] %><%= method_prefix @v['list'] %><%= @v['method']['name'] %>
+<%= @v['method']["type"] %> <%= @v['method']["category"] %> method:
+<%= target @v['method']["aref"], @v['method']['callseq'] ||
+                                 @v['method']["name"] + @v['method']["params"] %>
+<%= @v['method']["m_desc"] %>
diff --git a/trunk/lib/rdoc/generator/texinfo/texinfo.erb b/trunk/lib/rdoc/generator/texinfo/texinfo.erb
new file mode 100644
index 0000000000..235f63d73c
--- /dev/null
+++ b/trunk/lib/rdoc/generator/texinfo/texinfo.erb
@@ -0,0 +1,28 @@
+\input texinfo   @c -*-texinfo-*-
+@c %**start of header
+@setfilename <%= @v['filename'] %>
+@settitle <%= @v['title'] %>
+@c %**end of header
+
+@contents @c TODO: whitespace is a mess... =\
+
+@ifnottex
+@node Top
+
+@top <%= @v['title'] %>
+@end ifnottex
+
+<% if @f = @v['files'].detect { |f| f.name =~ /Readme/i }  %>
+<%= @f.values['description'] %><% end %>
+
+@menu
+<% @v['classes'].each do |klass| %>
+* <%= klass.name.gsub(/::/, '-') %>::<% end %>
+@c TODO: add files
+@end menu
+
+<% (@v['classes'] || []).each_with_index do |klass, i| %>
+<%= TexinfoTemplate.new(@v.merge('class' => klass.values), 
+                        'class.texinfo.erb').render %><% end %>
+
+@bye
diff --git a/trunk/lib/rdoc/generator/xml.rb b/trunk/lib/rdoc/generator/xml.rb
new file mode 100644
index 0000000000..3335f2ce7c
--- /dev/null
+++ b/trunk/lib/rdoc/generator/xml.rb
@@ -0,0 +1,120 @@
+require 'rdoc/generator/html'
+
+##
+# Generate XML output as one big file
+
+class RDoc::Generator::XML < RDoc::Generator::HTML
+
+  ##
+  # Standard generator factory
+
+  def self.for(options)
+    new(options)
+  end
+
+  def initialize(*args)
+    super
+  end
+
+  ##
+  # Build the initial indices and output objects
+  # based on an array of TopLevel objects containing
+  # the extracted information.
+
+  def generate(info)
+    @info       = info
+    @files      = []
+    @classes    = []
+    @hyperlinks = {}
+
+    build_indices
+    generate_xml
+  end
+
+  ##
+  # Generate:
+  #
+  # * a list of HtmlFile objects for each TopLevel object.
+  # * a list of HtmlClass objects for each first level
+  #   class or module in the TopLevel objects
+  # * a complete list of all hyperlinkable terms (file,
+  #   class, module, and method names)
+
+  def build_indices
+    @info.each do |toplevel|
+      @files << RDoc::Generator::HtmlFile.new(toplevel, @options, RDoc::Generator::FILE_DIR)
+    end
+
+    RDoc::TopLevel.all_classes_and_modules.each do |cls|
+      build_class_list(cls, @files[0], RDoc::Generator::CLASS_DIR)
+    end
+  end
+
+  def build_class_list(from, html_file, class_dir)
+    @classes << RDoc::Generator::HtmlClass.new(from, html_file, class_dir, @options)
+    from.each_classmodule do |mod|
+      build_class_list(mod, html_file, class_dir)
+    end
+  end
+
+  ##
+  # Generate all the HTML. For the one-file case, we generate
+  # all the information in to one big hash
+
+  def generate_xml
+    values = {
+      'charset' => @options.charset,
+      'files'   => gen_into(@files),
+      'classes' => gen_into(@classes)
+    }
+
+    # this method is defined in the template file
+    write_extra_pages if defined? write_extra_pages
+
+    template = RDoc::TemplatePage.new @template::ONE_PAGE
+
+    if @options.op_name
+      opfile = File.open(@options.op_name, "w")
+    else
+      opfile = $stdout
+    end
+    template.write_html_on(opfile, values)
+  end
+
+  def gen_into(list)
+    res = []
+    list.each do |item|
+      res << item.value_hash
+    end
+    res
+  end
+
+  def gen_file_index
+    gen_an_index(@files, 'Files')
+  end
+
+  def gen_class_index
+    gen_an_index(@classes, 'Classes')
+  end
+
+  def gen_method_index
+    gen_an_index(RDoc::Generator::HtmlMethod.all_methods, 'Methods')
+  end
+
+  def gen_an_index(collection, title)
+    res = []
+    collection.sort.each do |f|
+      if f.document_self
+        res << { "href" => f.path, "name" => f.index_name }
+      end
+    end
+
+    return {
+      "entries" => res,
+      'list_title' => title,
+      'index_url'  => main_url,
+    }
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/generator/xml/rdf.rb b/trunk/lib/rdoc/generator/xml/rdf.rb
new file mode 100644
index 0000000000..7b15c69a18
--- /dev/null
+++ b/trunk/lib/rdoc/generator/xml/rdf.rb
@@ -0,0 +1,113 @@
+require 'rdoc/generator/xml'
+
+module RDoc::Generator::XML::RDF
+
+  CONTENTS_RDF = <<-EOF
+<% if defined? classes and classes["description"] then %>
+    <description rd:parseType="Literal">
+<%= classes["description"] %>
+    </description>
+<% end %>
+
+<% if defined? files and files["requires"] then %>
+<% files["requires"].each do |requires| %>
+         <rd:required-file rd:name="<%= requires["name"] %>" />
+<% end # files["requires"] %>
+<% end %>
+
+<% if defined? classes and classes["includes"] then %>
+      <IncludedModuleList>
+<% classes["includes"].each do |includes| %>
+        <included-module rd:name="<%= includes["name"] %>"  />
+<% end # includes["includes"] %>
+      </IncludedModuleList>
+<% end %>
+
+<% if defined? classes and classes["sections"] then %>
+<% classes["sections"].each do |sections| %>
+<% if sections["attributes"] then %>
+<% sections["attributes"].each do |attributes| %>
+        <contents>
+        <Attribute rd:name="<%= attributes["name"] %>">
+<% if attributes["rw"] then %>
+          <attribute-rw><%= attributes["rw"] %></attribute-rw>
+<% end %>
+          <description rdf:parseType="Literal"><%= attributes["a_desc"] %></description>
+        </Attribute>
+        </contents>
+<% end # sections["attributes"] %>
+<% end %>
+
+<% if sections["method_list"] then %>
+<% sections["method_list"].each do |method_list| %>
+<% if method_list["methods"] then %>
+<% method_list["methods"].each do |methods| %>
+	<contents>
+        <Method rd:name="<%= methods["name"] %>" rd:visibility="<%= methods["type"] %>"
+                rd:category="<%= methods["category"] %>" rd:id="<%= methods["aref"] %>">
+          <parameters><%= methods["params"] %></parameters>
+<% if methods["m_desc"] then %>
+          <description rdf:parseType="Literal">
+<%= methods["m_desc"] %>
+          </description>
+<% end %>
+<% if methods["sourcecode"] then %>
+          <source-code-listing rdf:parseType="Literal">
+<%= methods["sourcecode"] %>
+          </source-code-listing>
+<% end %>
+        </Method>
+       </contents>
+<% end # method_list["methods"] %>
+<% end %>
+<% end # sections["method_list"] %>
+<% end %>
+     <!-- end method list -->
+<% end # classes["sections"] %>
+<% end %>
+  EOF
+
+########################################################################
+
+  ONE_PAGE = %{<?xml version="1.0" encoding="utf-8"?>
+<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+        xmlns="http://pragprog.com/rdoc/rdoc.rdf#"
+        xmlns:rd="http://pragprog.com/rdoc/rdoc.rdf#">
+
+<!-- RDoc -->
+<% values["files"].each do |files| %>
+  <rd:File rd:name="<%= files["short_name"] %>" rd:id="<%= files["href"] %>">
+      <path><%= files["full_path"] %></path>
+      <dtm-modified><%= files["dtm_modified"] %></dtm-modified>
+} + CONTENTS_RDF + %{
+  </rd:File>
+<% end # values["files"] %>
+<% values["classes"].each do |classes| %>
+  <<%= values["classmod"] %> rd:name="<%= classes["full_name"] %>" rd:id="<%= classes["full_name"] %>">
+    <classmod-info>
+<% if classes["infiles"] then %>
+      <InFiles>
+<% classes["infiles"].each do |infiles| %>
+        <infile>
+          <File rd:name="<%= infiles["full_path"] %>"
+<% if infiles["full_path_url"] then %>
+                rdf:about="<%= infiles["full_path_url"] %>"
+<% end %>
+           />
+         </infile>
+<% end # classes["infiles"] %>
+      </InFiles>
+<% end %>
+<% if classes["parent"] then %>
+     <superclass><%= href classes["par_url"], classes["parent"] %></superclass>
+<% end %>
+    </classmod-info>
+} + CONTENTS_RDF + %{
+  </<%= classes["classmod"] %>>
+<% end # values["classes"] %>
+<!-- /RDoc -->
+</rdf:RDF>
+}
+
+end
+
diff --git a/trunk/lib/rdoc/generator/xml/xml.rb b/trunk/lib/rdoc/generator/xml/xml.rb
new file mode 100644
index 0000000000..ffb1329c4e
--- /dev/null
+++ b/trunk/lib/rdoc/generator/xml/xml.rb
@@ -0,0 +1,111 @@
+require 'rdoc/generator/xml'
+
+module RDoc::Generator::XML::XML
+
+  CONTENTS_XML = <<-EOF
+<% if defined? classes and classes["description"] then %>
+    <description>
+<%= classes["description"] %>
+    </description>
+<% end %>
+    <contents>
+<% if defined? files and files["requires"] then %>
+      <required-file-list>
+<% files["requires"].each do |requires| %>
+         <required-file name="<%= requires["name"] %>"
+<% if requires["aref"] then %>
+                        href="<%= requires["aref"] %>"
+<% end %>
+         />
+<% end # files["requires"] %>
+      </required-file-list>
+<% end %>
+<% if defined? classes and classes["sections"] then %>
+<% classes["sections"].each do |sections| %>
+<% if sections["attributes"] then %>
+      <attribute-list>
+<% sections["attributes"].each do |attributes| %>
+        <attribute name="<%= attributes["name"] %>">
+<% if attributes["rw"] then %>
+          <attribute-rw><%= attributes["rw"] %></attribute-rw>
+<% end %>
+          <description><%= attributes["a_desc"] %></description>
+        </attribute>
+<% end # sections["attributes"] %>
+      </attribute-list>
+<% end %>
+<% if sections["method_list"] then %>
+      <method-list>
+<% sections["method_list"].each do |method_list| %>
+<% if method_list["methods"] then %>
+<% method_list["methods"].each do |methods| %>
+        <method name="<%= methods["name"] %>" type="<%= methods["type"] %>" category="<%= methods["category"] %>" id="<%= methods["aref"] %>">
+          <parameters><%= methods["params"] %></parameters>
+<% if methods["m_desc"] then %>
+          <description>
+<%= methods["m_desc"] %>
+          </description>
+<% end %>
+<% if methods["sourcecode"] then %>
+          <source-code-listing>
+<%= methods["sourcecode"] %>
+          </source-code-listing>
+<% end %>
+        </method>
+<% end # method_list["methods"] %>
+<% end %>
+<% end # sections["method_list"] %>
+      </method-list>
+<% end %>
+<% end # classes["sections"] %>
+<% end %>
+<% if defined? classes and classes["includes"] then %>
+      <included-module-list>
+<% classes["includes"].each do |includes| %>
+        <included-module name="<%= includes["name"] %>"
+<% if includes["aref"] then %>
+                         href="<%= includes["aref"] %>"
+<% end %>
+        />
+<% end # classes["includes"] %>
+      </included-module-list>
+<% end %>
+    </contents>
+  EOF
+
+  ONE_PAGE = %{<?xml version="1.0" encoding="utf-8"?>
+<rdoc>
+<file-list>
+<% values["files"].each do |files| %>
+  <file name="<%= files["short_name"] %>" id="<%= files["href"] %>">
+    <file-info>
+      <path><%= files["full_path"] %></path>
+      <dtm-modified><%= files["dtm_modified"] %></dtm-modified>
+    </file-info>
+} + CONTENTS_XML + %{
+  </file>
+<% end # values["files"] %>
+</file-list>
+<class-module-list>
+<% values["classes"].each do |classes| %>
+  <<%= classes["classmod"] %> name="<%= classes["full_name"] %>" id="<%= classes["full_name"] %>">
+    <classmod-info>
+<% if classes["infiles"] then %>
+      <infiles>
+<% classes["infiles"].each do |infiles|  %>
+        <infile><%= href infiles["full_path_url"], infiles["full_path"] %></infile>
+<% end # classes["infiles"] %>
+      </infiles>
+<% end %>
+<% if classes["parent"] then %>
+     <superclass><%= href classes["par_url"], classes["parent"] %></superclass>
+<% end %>
+    </classmod-info>
+} + CONTENTS_XML + %{
+  </<%= classes["classmod"] %>>
+<% end # values["classes"] %>
+</class-module-list>
+</rdoc>
+}
+
+end
diff --git a/trunk/lib/rdoc/known_classes.rb b/trunk/lib/rdoc/known_classes.rb
new file mode 100644
index 0000000000..4c52f58ad2
--- /dev/null
+++ b/trunk/lib/rdoc/known_classes.rb
@@ -0,0 +1,69 @@
+module RDoc
+
+  ##
+  # Ruby's built-in classes, modules and exceptions
+
+  KNOWN_CLASSES = {
+    "rb_cArray"            => "Array",
+    "rb_cBignum"           => "Bignum",
+    "rb_cClass"            => "Class",
+    "rb_cData"             => "Data",
+    "rb_cDir"              => "Dir",
+    "rb_cFalseClass"       => "FalseClass",
+    "rb_cFile"             => "File",
+    "rb_cFixnum"           => "Fixnum",
+    "rb_cFloat"            => "Float",
+    "rb_cHash"             => "Hash",
+    "rb_cIO"               => "IO",
+    "rb_cInteger"          => "Integer",
+    "rb_cModule"           => "Module",
+    "rb_cNilClass"         => "NilClass",
+    "rb_cNumeric"          => "Numeric",
+    "rb_cObject"           => "Object",
+    "rb_cProc"             => "Proc",
+    "rb_cRange"            => "Range",
+    "rb_cRegexp"           => "Regexp",
+    "rb_cRubyVM"           => "RubyVM",
+    "rb_cString"           => "String",
+    "rb_cStruct"           => "Struct",
+    "rb_cSymbol"           => "Symbol",
+    "rb_cThread"           => "Thread",
+    "rb_cTime"             => "Time",
+    "rb_cTrueClass"        => "TrueClass",
+
+    "rb_eArgError"         => "ArgError",
+    "rb_eEOFError"         => "EOFError",
+    "rb_eException"        => "Exception",
+    "rb_eFatal"            => "Fatal",
+    "rb_eFloatDomainError" => "FloatDomainError",
+    "rb_eIOError"          => "IOError",
+    "rb_eIndexError"       => "IndexError",
+    "rb_eInterrupt"        => "Interrupt",
+    "rb_eLoadError"        => "LoadError",
+    "rb_eNameError"        => "NameError",
+    "rb_eNoMemError"       => "NoMemError",
+    "rb_eNotImpError"      => "NotImpError",
+    "rb_eRangeError"       => "RangeError",
+    "rb_eRuntimeError"     => "RuntimeError",
+    "rb_eScriptError"      => "ScriptError",
+    "rb_eSecurityError"    => "SecurityError",
+    "rb_eSignal"           => "Signal",
+    "rb_eStandardError"    => "StandardError",
+    "rb_eSyntaxError"      => "SyntaxError",
+    "rb_eSystemCallError"  => "SystemCallError",
+    "rb_eSystemExit"       => "SystemExit",
+    "rb_eTypeError"        => "TypeError",
+    "rb_eZeroDivError"     => "ZeroDivError",
+
+    "rb_mComparable"       => "Comparable",
+    "rb_mEnumerable"       => "Enumerable",
+    "rb_mErrno"            => "Errno",
+    "rb_mFileTest"         => "FileTest",
+    "rb_mGC"               => "GC",
+    "rb_mKernel"           => "Kernel",
+    "rb_mMath"             => "Math",
+    "rb_mPrecision"        => "Precision",
+    "rb_mProcess"          => "Process"
+  }
+
+end
diff --git a/trunk/lib/rdoc/markup.rb b/trunk/lib/rdoc/markup.rb
new file mode 100644
index 0000000000..0e1b596255
--- /dev/null
+++ b/trunk/lib/rdoc/markup.rb
@@ -0,0 +1,473 @@
+require 'rdoc'
+
+##
+# RDoc::Markup parses plain text documents and attempts to decompose them into
+# their constituent parts.  Some of these parts are high-level: paragraphs,
+# chunks of verbatim text, list entries and the like.  Other parts happen at
+# the character level: a piece of bold text, a word in code font.  This markup
+# is similar in spirit to that used on WikiWiki webs, where folks create web
+# pages using a simple set of formatting rules.
+#
+# RDoc::Markup itself does no output formatting: this is left to a different
+# set of classes.
+#
+# RDoc::Markup is extendable at runtime: you can add \new markup elements to
+# be recognised in the documents that RDoc::Markup parses.
+#
+# RDoc::Markup is intended to be the basis for a family of tools which share
+# the common requirement that simple, plain-text should be rendered in a
+# variety of different output formats and media.  It is envisaged that
+# RDoc::Markup could be the basis for formatting RDoc style comment blocks,
+# Wiki entries, and online FAQs.
+#
+# = Basic Formatting
+#
+# * RDoc::Markup looks for a document's natural left margin.  This is
+#   used as the initial margin for the document.
+#
+# * Consecutive lines starting at this margin are considered to be a
+#   paragraph.
+#
+# * If a paragraph starts with a "*", "-", or with "<digit>.", then it is
+#   taken to be the start of a list.  The margin in increased to be the first
+#   non-space following the list start flag.  Subsequent lines should be
+#   indented to this \new margin until the list ends.  For example:
+#
+#      * this is a list with three paragraphs in
+#        the first item.  This is the first paragraph.
+#
+#        And this is the second paragraph.
+#
+#        1. This is an indented, numbered list.
+#        2. This is the second item in that list
+#
+#        This is the third conventional paragraph in the
+#        first list item.
+#
+#      * This is the second item in the original list
+#
+# * You can also construct labeled lists, sometimes called description
+#   or definition lists.  Do this by putting the label in square brackets
+#   and indenting the list body:
+#
+#       [cat]  a small furry mammal
+#              that seems to sleep a lot
+#
+#       [ant]  a little insect that is known
+#              to enjoy picnics
+#
+#   A minor variation on labeled lists uses two colons to separate the
+#   label from the list body:
+#
+#       cat::  a small furry mammal
+#              that seems to sleep a lot
+#
+#       ant::  a little insect that is known
+#              to enjoy picnics
+#
+#   This latter style guarantees that the list bodies' left margins are
+#   aligned: think of them as a two column table.
+#
+# * Any line that starts to the right of the current margin is treated
+#   as verbatim text.  This is useful for code listings.  The example of a
+#   list above is also verbatim text.
+#
+# * A line starting with an equals sign (=) is treated as a
+#   heading.  Level one headings have one equals sign, level two headings
+#   have two,and so on.
+#
+# * A line starting with three or more hyphens (at the current indent)
+#   generates a horizontal rule.  The more hyphens, the thicker the rule
+#   (within reason, and if supported by the output device)
+#
+# * You can use markup within text (except verbatim) to change the
+#   appearance of parts of that text.  Out of the box, RDoc::Markup
+#   supports word-based and general markup.
+#
+#   Word-based markup uses flag characters around individual words:
+#
+#   [\*word*]  displays word in a *bold* font
+#   [\_word_]  displays word in an _emphasized_ font
+#   [\+word+]  displays word in a +code+ font
+#
+#   General markup affects text between a start delimiter and and end
+#   delimiter.  Not surprisingly, these delimiters look like HTML markup.
+#
+#   [\<b>text...</b>]    displays word in a *bold* font
+#   [\<em>text...</em>]  displays word in an _emphasized_ font
+#   [\<i>text...</i>]    displays word in an _emphasized_ font
+#   [\<tt>text...</tt>]  displays word in a +code+ font
+#
+#   Unlike conventional Wiki markup, general markup can cross line
+#   boundaries.  You can turn off the interpretation of markup by
+#   preceding the first character with a backslash, so \\\<b>bold
+#   text</b> and \\\*bold* produce \<b>bold text</b> and \*bold*
+#   respectively.
+#
+# * Hyperlinks to the web starting http:, mailto:, ftp:, or www. are
+#   recognized.  An HTTP url that references an external image file is
+#   converted into an inline <IMG..>.  Hyperlinks starting 'link:' are
+#   assumed to refer to local files whose path is relative to the --op
+#   directory.
+#
+#   Hyperlinks can also be of the form <tt>label</tt>[url], in which
+#   case the label is used in the displayed text, and <tt>url</tt> is
+#   used as the target.  If <tt>label</tt> contains multiple words,
+#   put it in braces: <em>{multi word label}[</em>url<em>]</em>.
+#
+# == Synopsis
+#
+# This code converts +input_string+ to HTML.  The conversion takes place in
+# the +convert+ method, so you can use the same RDoc::Markup converter to
+# convert multiple input strings.
+#
+#   require 'rdoc/markup/to_html'
+#   
+#   h = RDoc::Markup::ToHtml.new
+#   
+#   puts h.convert(input_string)
+#
+# You can extend the RDoc::Markup parser to recognise new markup
+# sequences, and to add special processing for text that matches a
+# regular expression.  Here we make WikiWords significant to the parser,
+# and also make the sequences {word} and \<no>text...</no> signify
+# strike-through text.  When then subclass the HTML output class to deal
+# with these:
+#
+#   require 'rdoc/markup'
+#   require 'rdoc/markup/to_html'
+#   
+#   class WikiHtml < RDoc::Markup::ToHtml
+#     def handle_special_WIKIWORD(special)
+#       "<font color=red>" + special.text + "</font>"
+#     end
+#   end
+#   
+#   m = RDoc::Markup.new
+#   m.add_word_pair("{", "}", :STRIKE)
+#   m.add_html("no", :STRIKE)
+#   
+#   m.add_special(/\b([A-Z][a-z]+[A-Z]\w+)/, :WIKIWORD)
+#   
+#   wh = WikiHtml.new
+#   wh.add_tag(:STRIKE, "<strike>", "</strike>")
+#   
+#   puts "<body>#{wh.convert ARGF.read}</body>"
+#
+#--
+# Author::   Dave Thomas,  dave@pragmaticprogrammer.com
+# License::  Ruby license
+
+class RDoc::Markup
+
+  SPACE = ?\s
+
+  # List entries look like:
+  #   *       text
+  #   1.      text
+  #   [label] text
+  #   label:: text
+  #
+  # Flag it as a list entry, and work out the indent for subsequent lines
+
+  SIMPLE_LIST_RE = /^(
+                (  \*          (?# bullet)
+                  |-           (?# bullet)
+                  |\d+\.       (?# numbered )
+                  |[A-Za-z]\.  (?# alphabetically numbered )
+                )
+                \s+
+              )\S/x
+
+  LABEL_LIST_RE = /^(
+                      (  \[.*?\]    (?# labeled  )
+                        |\S.*::     (?# note     )
+                      )(?:\s+|$)
+                    )/x
+
+  ##
+  # Take a block of text and use various heuristics to determine it's
+  # structure (paragraphs, lists, and so on).  Invoke an event handler as we
+  # identify significant chunks.
+
+  def initialize
+    @am = RDoc::Markup::AttributeManager.new
+    @output = nil
+  end
+
+  ##
+  # Add to the sequences used to add formatting to an individual word (such
+  # as *bold*).  Matching entries will generate attributes that the output
+  # formatters can recognize by their +name+.
+
+  def add_word_pair(start, stop, name)
+    @am.add_word_pair(start, stop, name)
+  end
+
+  ##
+  # Add to the sequences recognized as general markup.
+
+  def add_html(tag, name)
+    @am.add_html(tag, name)
+  end
+
+  ##
+  # Add to other inline sequences.  For example, we could add WikiWords using
+  # something like:
+  #
+  #    parser.add_special(/\b([A-Z][a-z]+[A-Z]\w+)/, :WIKIWORD)
+  #
+  # Each wiki word will be presented to the output formatter via the
+  # accept_special method.
+
+  def add_special(pattern, name)
+    @am.add_special(pattern, name)
+  end
+
+  ##
+  # We take a string, split it into lines, work out the type of each line,
+  # and from there deduce groups of lines (for example all lines in a
+  # paragraph).  We then invoke the output formatter using a Visitor to
+  # display the result.
+
+  def convert(str, op)
+    lines = str.split(/\r?\n/).map { |line| Line.new line }
+    @lines = Lines.new lines
+
+    return "" if @lines.empty?
+    @lines.normalize
+    assign_types_to_lines
+    group = group_lines
+    # call the output formatter to handle the result
+    #group.each { |line| p line }
+    group.accept @am, op
+  end
+
+  private
+
+  ##
+  # Look through the text at line indentation.  We flag each line as being
+  # Blank, a paragraph, a list element, or verbatim text.
+
+  def assign_types_to_lines(margin = 0, level = 0)
+    while line = @lines.next
+      if line.blank? then
+        line.stamp :BLANK, level
+        next
+      end
+
+      # if a line contains non-blanks before the margin, then it must belong
+      # to an outer level
+
+      text = line.text
+
+      for i in 0...margin
+        if text[i] != SPACE
+          @lines.unget
+          return
+        end
+      end
+
+      active_line = text[margin..-1]
+
+      # Rules (horizontal lines) look like
+      #
+      #  ---   (three or more hyphens)
+      #
+      # The more hyphens, the thicker the rule
+      #
+
+      if /^(---+)\s*$/ =~ active_line
+        line.stamp :RULE, level, $1.length-2
+        next
+      end
+
+      # Then look for list entries.  First the ones that have to have
+      # text following them (* xxx, - xxx, and dd. xxx)
+
+      if SIMPLE_LIST_RE =~ active_line
+        offset = margin + $1.length
+        prefix = $2
+        prefix_length = prefix.length
+
+        flag = case prefix
+               when "*","-" then :BULLET
+               when /^\d/   then :NUMBER
+               when /^[A-Z]/ then :UPPERALPHA
+               when /^[a-z]/ then :LOWERALPHA
+               else raise "Invalid List Type: #{self.inspect}"
+               end
+
+        line.stamp :LIST, level+1, prefix, flag
+        text[margin, prefix_length] = " " * prefix_length
+        assign_types_to_lines(offset, level + 1)
+        next
+      end
+
+      if LABEL_LIST_RE =~ active_line
+        offset = margin + $1.length
+        prefix = $2
+        prefix_length = prefix.length
+
+        next if handled_labeled_list(line, level, margin, offset, prefix)
+      end
+
+      # Headings look like
+      # = Main heading
+      # == Second level
+      # === Third
+      #
+      # Headings reset the level to 0
+
+      if active_line[0] == ?= and active_line =~ /^(=+)\s*(.*)/
+        prefix_length = $1.length
+        prefix_length = 6 if prefix_length > 6
+        line.stamp :HEADING, 0, prefix_length
+        line.strip_leading(margin + prefix_length)
+        next
+      end
+
+      # If the character's a space, then we have verbatim text,
+      # otherwise
+
+      if active_line[0] == SPACE
+        line.strip_leading(margin) if margin > 0
+        line.stamp :VERBATIM, level
+      else
+        line.stamp :PARAGRAPH, level
+      end
+    end
+  end
+
+  ##
+  # Handle labeled list entries, We have a special case to deal with.
+  # Because the labels can be long, they force the remaining block of text
+  # over the to right:
+  #
+  #   this is a long label that I wrote:: and here is the
+  #                                       block of text with
+  #                                       a silly margin
+  #
+  # So we allow the special case.  If the label is followed by nothing, and
+  # if the following line is indented, then we take the indent of that line
+  # as the new margin.
+  #
+  #   this is a long label that I wrote::
+  #       here is a more reasonably indented block which
+  #       will be attached to the label.
+  #
+
+  def handled_labeled_list(line, level, margin, offset, prefix)
+    prefix_length = prefix.length
+    text = line.text
+    flag = nil
+
+    case prefix
+    when /^\[/ then
+      flag = :LABELED
+      prefix = prefix[1, prefix.length-2]
+    when /:$/ then
+      flag = :NOTE
+      prefix.chop!
+    else
+      raise "Invalid List Type: #{self.inspect}"
+    end
+
+    # body is on the next line
+    if text.length <= offset then
+      original_line = line
+      line = @lines.next
+      return false unless line
+      text = line.text
+
+      for i in 0..margin
+        if text[i] != SPACE
+          @lines.unget
+          return false
+        end
+      end
+
+      i = margin
+      i += 1 while text[i] == SPACE
+
+      if i >= text.length then
+        @lines.unget
+        return false
+      else
+        offset = i
+        prefix_length = 0
+
+        if text[offset..-1] =~ SIMPLE_LIST_RE then
+          @lines.unget
+          line = original_line
+          line.text = ''
+        else
+          @lines.delete original_line
+        end
+      end
+    end
+
+    line.stamp :LIST, level+1, prefix, flag
+    text[margin, prefix_length] = " " * prefix_length
+    assign_types_to_lines(offset, level + 1)
+    return true
+  end
+
+  ##
+  # Return a block consisting of fragments which are paragraphs, list
+  # entries or verbatim text.  We merge consecutive lines of the same type
+  # and level together.  We are also slightly tricky with lists: the lines
+  # following a list introduction look like paragraph lines at the next
+  # level, and we remap them into list entries instead.
+
+  def group_lines
+    @lines.rewind
+
+    in_list = false
+    wanted_type = wanted_level = nil
+
+    block = LineCollection.new
+    group = nil
+
+    while line = @lines.next
+      if line.level == wanted_level and line.type == wanted_type
+        group.add_text(line.text)
+      else
+        group = block.fragment_for(line)
+        block.add(group)
+
+        if line.type == :LIST
+          wanted_type = :PARAGRAPH
+        else
+          wanted_type = line.type
+        end
+
+        wanted_level = line.type == :HEADING ? line.param : line.level
+      end
+    end
+
+    block.normalize
+    block
+  end
+
+  ##
+  # For debugging, we allow access to our line contents as text.
+
+  def content
+    @lines.as_text
+  end
+  public :content
+
+  ##
+  # For debugging, return the list of line types.
+
+  def get_line_types
+    @lines.line_types
+  end
+  public :get_line_types
+
+end
+
+require 'rdoc/markup/fragments'
+require 'rdoc/markup/inline'
+require 'rdoc/markup/lines'
diff --git a/trunk/lib/rdoc/markup/attribute_manager.rb b/trunk/lib/rdoc/markup/attribute_manager.rb
new file mode 100644
index 0000000000..d13b79376c
--- /dev/null
+++ b/trunk/lib/rdoc/markup/attribute_manager.rb
@@ -0,0 +1,265 @@
+require 'rdoc/markup/inline'
+
+class RDoc::Markup::AttributeManager
+
+  NULL = "\000".freeze
+
+  ##
+  # We work by substituting non-printing characters in to the text. For now
+  # I'm assuming that I can substitute a character in the range 0..8 for a 7
+  # bit character without damaging the encoded string, but this might be
+  # optimistic
+
+  A_PROTECT  = 004
+  PROTECT_ATTR  = A_PROTECT.chr
+
+  ##
+  # This maps delimiters that occur around words (such as *bold* or +tt+)
+  # where the start and end delimiters and the same. This lets us optimize
+  # the regexp
+
+  MATCHING_WORD_PAIRS = {}
+
+  ##
+  # And this is used when the delimiters aren't the same. In this case the
+  # hash maps a pattern to the attribute character
+
+  WORD_PAIR_MAP = {}
+
+  ##
+  # This maps HTML tags to the corresponding attribute char
+
+  HTML_TAGS = {}
+
+  ##
+  # And this maps _special_ sequences to a name. A special sequence is
+  # something like a WikiWord
+
+  SPECIAL = {}
+
+  ##
+  # Return an attribute object with the given turn_on and turn_off bits set
+
+  def attribute(turn_on, turn_off)
+    RDoc::Markup::AttrChanger.new turn_on, turn_off
+  end
+
+  def change_attribute(current, new)
+    diff = current ^ new
+    attribute(new & diff, current & diff)
+  end
+
+  def changed_attribute_by_name(current_set, new_set)
+    current = new = 0
+    current_set.each do |name|
+      current |= RDoc::Markup::Attribute.bitmap_for(name)
+    end
+
+    new_set.each do |name|
+      new |= RDoc::Markup::Attribute.bitmap_for(name)
+    end
+
+    change_attribute(current, new)
+  end
+
+  def copy_string(start_pos, end_pos)
+    res = @str[start_pos...end_pos]
+    res.gsub!(/\000/, '')
+    res
+  end
+
+  ##
+  # Map attributes like <b>text</b>to the sequence
+  # \001\002<char>\001\003<char>, where <char> is a per-attribute specific
+  # character
+
+  def convert_attrs(str, attrs)
+    # first do matching ones
+    tags = MATCHING_WORD_PAIRS.keys.join("")
+
+    re = /(^|\W)([#{tags}])([#:\\]?[\w.\/-]+?\S?)\2(\W|$)/
+
+    1 while str.gsub!(re) do
+      attr = MATCHING_WORD_PAIRS[$2]
+      attrs.set_attrs($`.length + $1.length + $2.length, $3.length, attr)
+      $1 + NULL * $2.length + $3 + NULL * $2.length + $4
+    end
+
+    # then non-matching
+    unless WORD_PAIR_MAP.empty? then
+      WORD_PAIR_MAP.each do |regexp, attr|
+        str.gsub!(regexp) {
+          attrs.set_attrs($`.length + $1.length, $2.length, attr)
+          NULL * $1.length + $2 + NULL * $3.length
+        }
+      end
+    end
+  end
+
+  def convert_html(str, attrs)
+    tags = HTML_TAGS.keys.join '|'
+
+    1 while str.gsub!(/<(#{tags})>(.*?)<\/\1>/i) {
+      attr = HTML_TAGS[$1.downcase]
+      html_length = $1.length + 2
+      seq = NULL * html_length
+      attrs.set_attrs($`.length + html_length, $2.length, attr)
+      seq + $2 + seq + NULL
+    }
+  end
+
+  def convert_specials(str, attrs)
+    unless SPECIAL.empty?
+      SPECIAL.each do |regexp, attr|
+        str.scan(regexp) do
+          attrs.set_attrs($`.length, $&.length,
+                          attr | RDoc::Markup::Attribute::SPECIAL)
+        end
+      end
+    end
+  end
+
+  ##
+  # A \ in front of a character that would normally be processed turns off
+  # processing. We do this by turning \< into <#{PROTECT}
+
+  PROTECTABLE = %w[<\\]
+
+  def mask_protected_sequences
+    protect_pattern = Regexp.new("\\\\([#{Regexp.escape(PROTECTABLE.join(''))}])")
+    @str.gsub!(protect_pattern, "\\1#{PROTECT_ATTR}")
+  end
+
+  def unmask_protected_sequences
+    @str.gsub!(/(.)#{PROTECT_ATTR}/, "\\1\000")
+  end
+
+  def initialize
+    add_word_pair("*", "*", :BOLD)
+    add_word_pair("_", "_", :EM)
+    add_word_pair("+", "+", :TT)
+
+    add_html("em", :EM)
+    add_html("i",  :EM)
+    add_html("b",  :BOLD)
+    add_html("tt",   :TT)
+    add_html("code", :TT)
+  end
+
+  def add_word_pair(start, stop, name)
+    raise ArgumentError, "Word flags may not start with '<'" if
+      start[0,1] == '<'
+
+    bitmap = RDoc::Markup::Attribute.bitmap_for name
+
+    if start == stop then
+      MATCHING_WORD_PAIRS[start] = bitmap
+    else
+      pattern = /(#{Regexp.escape start})(\S+)(#{Regexp.escape stop})/
+      WORD_PAIR_MAP[pattern] = bitmap
+    end
+
+    PROTECTABLE << start[0,1]
+    PROTECTABLE.uniq!
+  end
+
+  def add_html(tag, name)
+    HTML_TAGS[tag.downcase] = RDoc::Markup::Attribute.bitmap_for name
+  end
+
+  def add_special(pattern, name)
+    SPECIAL[pattern] = RDoc::Markup::Attribute.bitmap_for name
+  end
+
+  def flow(str)
+    @str = str
+
+    mask_protected_sequences
+
+    @attrs = RDoc::Markup::AttrSpan.new @str.length
+
+    convert_attrs(@str, @attrs)
+    convert_html(@str, @attrs)
+    convert_specials(str, @attrs)
+
+    unmask_protected_sequences
+
+    return split_into_flow
+  end
+
+  def display_attributes
+    puts
+    puts @str.tr(NULL, "!")
+    bit = 1
+    16.times do |bno|
+      line = ""
+      @str.length.times do |i|
+        if (@attrs[i] & bit) == 0
+          line << " "
+        else
+          if bno.zero?
+            line << "S"
+          else
+            line << ("%d" % (bno+1))
+          end
+        end
+      end
+      puts(line) unless line =~ /^ *$/
+      bit <<= 1
+    end
+  end
+
+  def split_into_flow
+    res = []
+    current_attr = 0
+    str = ""
+
+    str_len = @str.length
+
+    # skip leading invisible text
+    i = 0
+    i += 1 while i < str_len and @str[i].chr == "\0"
+    start_pos = i
+
+    # then scan the string, chunking it on attribute changes
+    while i < str_len
+      new_attr = @attrs[i]
+      if new_attr != current_attr
+        if i > start_pos
+          res << copy_string(start_pos, i)
+          start_pos = i
+        end
+
+        res << change_attribute(current_attr, new_attr)
+        current_attr = new_attr
+
+        if (current_attr & RDoc::Markup::Attribute::SPECIAL) != 0 then
+          i += 1 while
+            i < str_len and (@attrs[i] & RDoc::Markup::Attribute::SPECIAL) != 0
+
+          res << RDoc::Markup::Special.new(current_attr,
+                                           copy_string(start_pos, i))
+          start_pos = i
+          next
+        end
+      end
+
+      # move on, skipping any invisible characters
+      begin
+        i += 1
+      end while i < str_len and @str[i].chr == "\0"
+    end
+
+    # tidy up trailing text
+    if start_pos < str_len
+      res << copy_string(start_pos, str_len)
+    end
+
+    # and reset to all attributes off
+    res << change_attribute(current_attr, 0) if current_attr != 0
+
+    return res
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/markup/formatter.rb b/trunk/lib/rdoc/markup/formatter.rb
new file mode 100644
index 0000000000..14cbae59f9
--- /dev/null
+++ b/trunk/lib/rdoc/markup/formatter.rb
@@ -0,0 +1,14 @@
+require 'rdoc/markup'
+
+class RDoc::Markup::Formatter
+
+  def initialize
+    @markup = RDoc::Markup.new
+  end
+
+  def convert(content)
+    @markup.convert content, self
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/markup/fragments.rb b/trunk/lib/rdoc/markup/fragments.rb
new file mode 100644
index 0000000000..b7f9b605c8
--- /dev/null
+++ b/trunk/lib/rdoc/markup/fragments.rb
@@ -0,0 +1,337 @@
+require 'rdoc/markup'
+require 'rdoc/markup/lines'
+
+class RDoc::Markup
+
+  ##
+  # A Fragment is a chunk of text, subclassed as a paragraph, a list
+  # entry, or verbatim text.
+
+  class Fragment
+    attr_reader   :level, :param, :txt
+    attr_accessor :type
+
+    ##
+    # This is a simple factory system that lets us associate fragement
+    # types (a string) with a subclass of fragment
+
+    TYPE_MAP = {}
+
+    def self.type_name(name)
+      TYPE_MAP[name] = self
+    end
+
+    def self.for(line)
+      klass =  TYPE_MAP[line.type] ||
+        raise("Unknown line type: '#{line.type.inspect}:' '#{line.text}'")
+      return klass.new(line.level, line.param, line.flag, line.text)
+    end
+
+    def initialize(level, param, type, txt)
+      @level = level
+      @param = param
+      @type  = type
+      @txt   = ""
+      add_text(txt) if txt
+    end
+
+    def add_text(txt)
+      @txt << " " if @txt.length > 0
+      @txt << txt.tr_s("\n ", "  ").strip
+    end
+
+    def to_s
+      "L#@level: #{self.class.name.split('::')[-1]}\n#@txt"
+    end
+
+  end
+
+  ##
+  # A paragraph is a fragment which gets wrapped to fit. We remove all
+  # newlines when we're created, and have them put back on output.
+
+  class Paragraph < Fragment
+    type_name :PARAGRAPH
+  end
+
+  class BlankLine < Paragraph
+    type_name :BLANK
+  end
+
+  class Heading < Paragraph
+    type_name :HEADING
+
+    def head_level
+      @param.to_i
+    end
+  end
+
+  ##
+  # A List is a fragment with some kind of label
+
+  class ListBase < Paragraph
+    LIST_TYPES = [
+      :BULLET,
+      :NUMBER,
+      :UPPERALPHA,
+      :LOWERALPHA,
+      :LABELED,
+      :NOTE,
+    ]
+  end
+
+  class ListItem < ListBase
+    type_name :LIST
+
+    def to_s
+      text = if [:NOTE, :LABELED].include? type then
+               "#{@param}: #{@txt}"
+             else
+               @txt
+             end
+
+      "L#@level: #{type} #{self.class.name.split('::')[-1]}\n#{text}"
+    end
+
+  end
+
+  class ListStart < ListBase
+    def initialize(level, param, type)
+      super(level, param, type, nil)
+    end
+  end
+
+  class ListEnd < ListBase
+    def initialize(level, type)
+      super(level, "", type, nil)
+    end
+  end
+
+  ##
+  # Verbatim code contains lines that don't get wrapped.
+
+  class Verbatim < Fragment
+    type_name  :VERBATIM
+
+    def add_text(txt)
+      @txt << txt.chomp << "\n"
+    end
+
+  end
+
+  ##
+  # A horizontal rule
+
+  class Rule < Fragment
+    type_name :RULE
+  end
+
+  ##
+  # Collect groups of lines together. Each group will end up containing a flow
+  # of text.
+
+  class LineCollection
+
+    def initialize
+      @fragments = []
+    end
+
+    def add(fragment)
+      @fragments << fragment
+    end
+
+    def each(&b)
+      @fragments.each(&b)
+    end
+
+    def to_a # :nodoc:
+      @fragments.map {|fragment| fragment.to_s}
+    end
+
+    ##
+    # Factory for different fragment types
+
+    def fragment_for(*args)
+      Fragment.for(*args)
+    end
+
+    ##
+    # Tidy up at the end
+
+    def normalize
+      change_verbatim_blank_lines
+      add_list_start_and_ends
+      add_list_breaks
+      tidy_blank_lines
+    end
+
+    def to_s
+      @fragments.join("\n----\n")
+    end
+
+    def accept(am, visitor)
+      visitor.start_accepting
+
+      @fragments.each do |fragment|
+        case fragment
+        when Verbatim
+          visitor.accept_verbatim(am, fragment)
+        when Rule
+          visitor.accept_rule(am, fragment)
+        when ListStart
+          visitor.accept_list_start(am, fragment)
+        when ListEnd
+          visitor.accept_list_end(am, fragment)
+        when ListItem
+          visitor.accept_list_item(am, fragment)
+        when BlankLine
+          visitor.accept_blank_line(am, fragment)
+        when Heading
+          visitor.accept_heading(am, fragment)
+        when Paragraph
+          visitor.accept_paragraph(am, fragment)
+        end
+      end
+
+      visitor.end_accepting
+    end
+
+    private
+
+    # If you have:
+    #
+    #    normal paragraph text.
+    #
+    #       this is code
+    #   
+    #       and more code
+    #
+    # You'll end up with the fragments Paragraph, BlankLine, Verbatim,
+    # BlankLine, Verbatim, BlankLine, etc.
+    #
+    # The BlankLine in the middle of the verbatim chunk needs to be changed to
+    # a real verbatim newline, and the two verbatim blocks merged
+
+    def change_verbatim_blank_lines
+      frag_block = nil
+      blank_count = 0
+      @fragments.each_with_index do |frag, i|
+        if frag_block.nil?
+          frag_block = frag if Verbatim === frag
+        else
+          case frag
+          when Verbatim
+            blank_count.times { frag_block.add_text("\n") }
+            blank_count = 0
+            frag_block.add_text(frag.txt)
+            @fragments[i] = nil    # remove out current fragment
+          when BlankLine
+            if frag_block
+              blank_count += 1
+              @fragments[i] = nil
+            end
+          else
+            frag_block = nil
+            blank_count = 0
+          end
+        end
+      end
+      @fragments.compact!
+    end
+
+    ##
+    # List nesting is implicit given the level of indentation. Make it
+    # explicit, just to make life a tad easier for the output processors
+
+    def add_list_start_and_ends
+      level = 0
+      res = []
+      type_stack = []
+
+      @fragments.each do |fragment|
+        # $stderr.puts "#{level} : #{fragment.class.name} : #{fragment.level}"
+        new_level = fragment.level
+        while (level < new_level)
+          level += 1
+          type = fragment.type
+          res << ListStart.new(level, fragment.param, type) if type
+          type_stack.push type
+          # $stderr.puts "Start: #{level}"
+        end
+
+        while level > new_level
+          type = type_stack.pop
+          res << ListEnd.new(level, type) if type
+          level -= 1
+          # $stderr.puts "End: #{level}, #{type}"
+        end
+
+        res << fragment
+        level = fragment.level
+      end
+      level.downto(1) do |i|
+        type = type_stack.pop
+        res << ListEnd.new(i, type) if type
+      end
+
+      @fragments = res
+    end
+
+    ##
+    # Inserts start/ends between list entries at the same level that have
+    # different element types
+
+    def add_list_breaks
+      res = @fragments
+
+      @fragments = []
+      list_stack = []
+
+      res.each do |fragment|
+        case fragment
+        when ListStart
+          list_stack.push fragment
+        when ListEnd
+          start = list_stack.pop
+          fragment.type = start.type
+        when ListItem
+          l = list_stack.last
+          if fragment.type != l.type
+            @fragments << ListEnd.new(l.level, l.type)
+            start = ListStart.new(l.level, fragment.param, fragment.type)
+            @fragments << start
+            list_stack.pop
+            list_stack.push start
+          end
+        else
+          ;
+        end
+        @fragments << fragment
+      end
+    end
+
+    ##
+    # Tidy up the blank lines:
+    # * change Blank/ListEnd into ListEnd/Blank
+    # * remove blank lines at the front
+
+    def tidy_blank_lines
+      (@fragments.size - 1).times do |i|
+        if BlankLine === @fragments[i] and ListEnd === @fragments[i+1] then
+          @fragments[i], @fragments[i+1] = @fragments[i+1], @fragments[i]
+        end
+      end
+
+      # remove leading blanks
+      @fragments.each_with_index do |f, i|
+        break unless f.kind_of? BlankLine
+        @fragments[i] = nil
+      end
+
+      @fragments.compact!
+    end
+
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/markup/inline.rb b/trunk/lib/rdoc/markup/inline.rb
new file mode 100644
index 0000000000..ee77679a11
--- /dev/null
+++ b/trunk/lib/rdoc/markup/inline.rb
@@ -0,0 +1,101 @@
+require 'rdoc/markup'
+
+class RDoc::Markup
+
+  ##
+  # We manage a set of attributes. Each attribute has a symbol name and a bit
+  # value.
+
+  class Attribute
+    SPECIAL = 1
+
+    @@name_to_bitmap = { :_SPECIAL_ => SPECIAL }
+    @@next_bitmap = 2
+
+    def self.bitmap_for(name)
+      bitmap = @@name_to_bitmap[name]
+      unless bitmap then
+        bitmap = @@next_bitmap
+        @@next_bitmap <<= 1
+        @@name_to_bitmap[name] = bitmap
+      end
+      bitmap
+    end
+
+    def self.as_string(bitmap)
+      return "none" if bitmap.zero?
+      res = []
+      @@name_to_bitmap.each do |name, bit|
+        res << name if (bitmap & bit) != 0
+      end
+      res.join(",")
+    end
+
+    def self.each_name_of(bitmap)
+      @@name_to_bitmap.each do |name, bit|
+        next if bit == SPECIAL
+        yield name.to_s if (bitmap & bit) != 0
+      end
+    end
+  end
+
+  AttrChanger = Struct.new(:turn_on, :turn_off)
+
+  ##
+  # An AttrChanger records a change in attributes. It contains a bitmap of the
+  # attributes to turn on, and a bitmap of those to turn off.
+
+  class AttrChanger
+    def to_s
+      "Attr: +#{Attribute.as_string(@turn_on)}/-#{Attribute.as_string(@turn_on)}"
+    end
+  end
+
+  ##
+  # An array of attributes which parallels the characters in a string.
+
+  class AttrSpan
+    def initialize(length)
+      @attrs = Array.new(length, 0)
+    end
+
+    def set_attrs(start, length, bits)
+      for i in start ... (start+length)
+        @attrs[i] |= bits
+      end
+    end
+
+    def [](n)
+      @attrs[n]
+    end
+  end
+
+  ##
+  # Hold details of a special sequence
+
+  class Special
+    attr_reader   :type
+    attr_accessor :text
+
+    def initialize(type, text)
+      @type, @text = type, text
+    end
+
+    def ==(o)
+      self.text == o.text && self.type == o.type
+    end
+
+    def inspect
+      "#<RDoc::Markup::Special:0x%x @type=%p, name=%p @text=%p>" % [
+        object_id, @type, RDoc::Markup::Attribute.as_string(type), text.dump]
+    end
+
+    def to_s
+      "Special: type=#{type}, name=#{RDoc::Markup::Attribute.as_string type}, text=#{text.dump}"
+    end
+
+  end
+
+end
+
+require 'rdoc/markup/attribute_manager'
diff --git a/trunk/lib/rdoc/markup/lines.rb b/trunk/lib/rdoc/markup/lines.rb
new file mode 100644
index 0000000000..069492122f
--- /dev/null
+++ b/trunk/lib/rdoc/markup/lines.rb
@@ -0,0 +1,152 @@
+class RDoc::Markup
+
+  ##
+  # We store the lines we're working on as objects of class Line.  These
+  # contain the text of the line, along with a flag indicating the line type,
+  # and an indentation level.
+
+  class Line
+    INFINITY = 9999
+
+    LINE_TYPES = [
+      :BLANK,
+      :HEADING,
+      :LIST,
+      :PARAGRAPH,
+      :RULE,
+      :VERBATIM,
+    ]
+
+    # line type
+    attr_accessor :type
+
+    # The indentation nesting level
+    attr_accessor :level
+
+    # The contents
+    attr_accessor :text
+
+    # A prefix or parameter. For LIST lines, this is
+    # the text that introduced the list item (the label)
+    attr_accessor  :param
+
+    # A flag. For list lines, this is the type of the list
+    attr_accessor :flag
+
+    # the number of leading spaces
+    attr_accessor :leading_spaces
+
+    # true if this line has been deleted from the list of lines
+    attr_accessor :deleted
+
+    def initialize(text)
+      @text    = text.dup
+      @deleted = false
+
+      # expand tabs
+      1 while @text.gsub!(/\t+/) { ' ' * (8*$&.length - $`.length % 8)}  && $~ #`
+
+      # Strip trailing whitespace
+      @text.sub!(/\s+$/, '')
+
+      # and look for leading whitespace
+      if @text.length > 0
+        @text =~ /^(\s*)/
+        @leading_spaces = $1.length
+      else
+        @leading_spaces = INFINITY
+      end
+    end
+
+    # Return true if this line is blank
+    def blank?
+      @text.empty?
+    end
+
+    # stamp a line with a type, a level, a prefix, and a flag
+    def stamp(type, level, param="", flag=nil)
+      @type, @level, @param, @flag = type, level, param, flag
+    end
+
+    ##
+    # Strip off the leading margin
+
+    def strip_leading(size)
+      if @text.size > size
+        @text[0,size] = ""
+      else
+        @text = ""
+      end
+    end
+
+    def to_s
+      "#@type#@level: #@text"
+    end
+  end
+
+  ##
+  # A container for all the lines.
+
+  class Lines
+
+    include Enumerable
+
+    attr_reader :lines # :nodoc:
+
+    def initialize(lines)
+      @lines = lines
+      rewind
+    end
+
+    def empty?
+      @lines.size.zero?
+    end
+
+    def each
+      @lines.each do |line|
+        yield line unless line.deleted
+      end
+    end
+
+#    def [](index)
+#      @lines[index]
+#    end
+
+    def rewind
+      @nextline = 0
+    end
+
+    def next
+      begin
+        res = @lines[@nextline]
+        @nextline += 1 if @nextline < @lines.size
+      end while res and res.deleted and @nextline < @lines.size
+      res
+    end
+
+    def unget
+      @nextline -= 1
+    end
+
+    def delete(a_line)
+      a_line.deleted = true
+    end
+
+    def normalize
+      margin = @lines.collect{|l| l.leading_spaces}.min
+      margin = 0 if margin == :INFINITY
+      @lines.each {|line| line.strip_leading(margin) } if margin > 0
+    end
+
+    def as_text
+      @lines.map {|l| l.text}.join("\n")
+    end
+
+    def line_types
+      @lines.map {|l| l.type }
+    end
+
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/markup/preprocess.rb b/trunk/lib/rdoc/markup/preprocess.rb
new file mode 100644
index 0000000000..00dd4be4ad
--- /dev/null
+++ b/trunk/lib/rdoc/markup/preprocess.rb
@@ -0,0 +1,75 @@
+require 'rdoc/markup'
+
+##
+# Handle common directives that can occur in a block of text:
+#
+# : include : filename
+
+class RDoc::Markup::PreProcess
+
+  def initialize(input_file_name, include_path)
+    @input_file_name = input_file_name
+    @include_path = include_path
+  end
+
+  ##
+  # Look for common options in a chunk of text. Options that we don't handle
+  # are yielded to the caller.
+
+  def handle(text)
+    text.gsub!(/^([ \t]*#?[ \t]*):(\w+):([ \t]*)(.+)?\n/) do
+      next $& if $3.empty? and $4 and $4[0, 1] == ':'
+
+      prefix    = $1
+      directive = $2.downcase
+      param     = $4
+
+      case directive
+      when 'include' then
+        filename = param.split[0]
+        include_file filename, prefix
+
+      else
+        result = yield directive, param
+        result = "#{prefix}:#{directive}: #{param}\n" unless result
+        result
+      end
+    end
+  end
+
+  private
+
+  ##
+  # Include a file, indenting it correctly.
+
+  def include_file(name, indent)
+    if full_name = find_include_file(name) then
+      content = File.open(full_name) {|f| f.read}
+      # strip leading '#'s, but only if all lines start with them
+      if content =~ /^[^#]/
+        content.gsub(/^/, indent)
+      else
+        content.gsub(/^#?/, indent)
+      end
+    else
+      $stderr.puts "Couldn't find file to include: '#{name}'"
+      ''
+    end
+  end
+
+  ##
+  # Look for the given file in the directory containing the current file,
+  # and then in each of the directories specified in the RDOC_INCLUDE path
+
+  def find_include_file(name)
+    to_search = [ File.dirname(@input_file_name) ].concat @include_path
+    to_search.each do |dir|
+      full_name = File.join(dir, name)
+      stat = File.stat(full_name) rescue next
+      return full_name if stat.readable?
+    end
+    nil
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/markup/to_flow.rb b/trunk/lib/rdoc/markup/to_flow.rb
new file mode 100644
index 0000000000..3d87b3e9c3
--- /dev/null
+++ b/trunk/lib/rdoc/markup/to_flow.rb
@@ -0,0 +1,185 @@
+require 'rdoc/markup/formatter'
+require 'rdoc/markup/fragments'
+require 'rdoc/markup/inline'
+require 'cgi'
+
+class RDoc::Markup
+
+  module Flow
+    P = Struct.new(:body)
+    VERB = Struct.new(:body)
+    RULE = Struct.new(:width)
+    class LIST
+      attr_reader :type, :contents
+      def initialize(type)
+        @type = type
+        @contents = []
+      end
+      def <<(stuff)
+        @contents << stuff
+      end
+    end
+    LI = Struct.new(:label, :body)
+    H = Struct.new(:level, :text)
+  end
+
+  class ToFlow < RDoc::Markup::Formatter
+    LIST_TYPE_TO_HTML = {
+      :BULLET     =>  [ "<ul>", "</ul>" ],
+      :NUMBER     =>  [ "<ol>", "</ol>" ],
+      :UPPERALPHA =>  [ "<ol>", "</ol>" ],
+      :LOWERALPHA =>  [ "<ol>", "</ol>" ],
+      :LABELED    =>  [ "<dl>", "</dl>" ],
+      :NOTE       =>  [ "<table>", "</table>" ],
+    }
+
+    InlineTag = Struct.new(:bit, :on, :off)
+
+    def initialize
+      super
+
+      init_tags
+    end
+
+    ##
+    # Set up the standard mapping of attributes to HTML tags
+
+    def init_tags
+      @attr_tags = [
+        InlineTag.new(RDoc::Markup::Attribute.bitmap_for(:BOLD), "<b>", "</b>"),
+        InlineTag.new(RDoc::Markup::Attribute.bitmap_for(:TT),   "<tt>", "</tt>"),
+        InlineTag.new(RDoc::Markup::Attribute.bitmap_for(:EM),   "<em>", "</em>"),
+      ]
+    end
+
+    ##
+    # Add a new set of HTML tags for an attribute. We allow separate start and
+    # end tags for flexibility
+
+    def add_tag(name, start, stop)
+      @attr_tags << InlineTag.new(RDoc::Markup::Attribute.bitmap_for(name), start, stop)
+    end
+
+    ##
+    # Given an HTML tag, decorate it with class information and the like if
+    # required. This is a no-op in the base class, but is overridden in HTML
+    # output classes that implement style sheets
+
+    def annotate(tag)
+      tag
+    end
+
+    ##
+    # Here's the client side of the visitor pattern
+
+    def start_accepting
+      @res = []
+      @list_stack = []
+    end
+
+    def end_accepting
+      @res
+    end
+
+    def accept_paragraph(am, fragment)
+      @res << Flow::P.new((convert_flow(am.flow(fragment.txt))))
+    end
+
+    def accept_verbatim(am, fragment)
+      @res << Flow::VERB.new((convert_flow(am.flow(fragment.txt))))
+    end
+
+    def accept_rule(am, fragment)
+      size = fragment.param
+      size = 10 if size > 10
+      @res << Flow::RULE.new(size)
+    end
+
+    def accept_list_start(am, fragment)
+      @list_stack.push(@res)
+      list = Flow::LIST.new(fragment.type)
+      @res << list
+      @res = list
+    end
+
+    def accept_list_end(am, fragment)
+      @res = @list_stack.pop
+    end
+
+    def accept_list_item(am, fragment)
+      @res << Flow::LI.new(fragment.param, convert_flow(am.flow(fragment.txt)))
+    end
+
+    def accept_blank_line(am, fragment)
+      # @res << annotate("<p />") << "\n"
+    end
+
+    def accept_heading(am, fragment)
+      @res << Flow::H.new(fragment.head_level, convert_flow(am.flow(fragment.txt)))
+    end
+
+    private
+
+    def on_tags(res, item)
+      attr_mask = item.turn_on
+      return if attr_mask.zero?
+
+      @attr_tags.each do |tag|
+        if attr_mask & tag.bit != 0
+          res << annotate(tag.on)
+        end
+      end
+    end
+
+    def off_tags(res, item)
+      attr_mask = item.turn_off
+      return if attr_mask.zero?
+
+      @attr_tags.reverse_each do |tag|
+        if attr_mask & tag.bit != 0
+          res << annotate(tag.off)
+        end
+      end
+    end
+
+    def convert_flow(flow)
+      res = ""
+      flow.each do |item|
+        case item
+        when String
+          res << convert_string(item)
+        when AttrChanger
+          off_tags(res, item)
+          on_tags(res,  item)
+        when Special
+          res << convert_special(item)
+        else
+          raise "Unknown flow element: #{item.inspect}"
+        end
+      end
+      res
+    end
+
+    def convert_string(item)
+      CGI.escapeHTML(item)
+    end
+
+    def convert_special(special)
+      handled = false
+      Attribute.each_name_of(special.type) do |name|
+        method_name = "handle_special_#{name}"
+        if self.respond_to? method_name
+          special.text = send(method_name, special)
+          handled = true
+        end
+      end
+
+      raise "Unhandled special: #{special}" unless handled
+
+      special.text
+    end
+
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/markup/to_html.rb b/trunk/lib/rdoc/markup/to_html.rb
new file mode 100644
index 0000000000..ca29373db1
--- /dev/null
+++ b/trunk/lib/rdoc/markup/to_html.rb
@@ -0,0 +1,400 @@
+require 'rdoc/markup/formatter'
+require 'rdoc/markup/fragments'
+require 'rdoc/markup/inline'
+
+require 'cgi'
+
+class RDoc::Markup::ToHtml < RDoc::Markup::Formatter
+
+  LIST_TYPE_TO_HTML = {
+    :BULLET =>     %w[<ul> </ul>],
+    :NUMBER =>     %w[<ol> </ol>],
+    :UPPERALPHA => %w[<ol> </ol>],
+    :LOWERALPHA => %w[<ol> </ol>],
+    :LABELED =>    %w[<dl> </dl>],
+    :NOTE    =>    %w[<table> </table>],
+  }
+
+  InlineTag = Struct.new(:bit, :on, :off)
+
+  def initialize
+    super
+
+    # @in_tt - tt nested levels count
+    # @tt_bit - cache
+    @in_tt = 0
+    @tt_bit = RDoc::Markup::Attribute.bitmap_for :TT
+
+    # external hyperlinks
+    @markup.add_special(/((link:|https?:|mailto:|ftp:|www\.)\S+\w)/, :HYPERLINK)
+
+    # and links of the form  <text>[<url>]
+    @markup.add_special(/(((\{.*?\})|\b\S+?)\[\S+?\.\S+?\])/, :TIDYLINK)
+
+    init_tags
+  end
+
+  ##
+  # Converts a target url to one that is relative to a given path
+
+  def self.gen_relative_url(path, target)
+    from        = File.dirname path
+    to, to_file = File.split target
+
+    from = from.split "/"
+    to   = to.split "/"
+
+    while from.size > 0 and to.size > 0 and from[0] == to[0] do
+      from.shift
+      to.shift
+    end
+
+    from.fill ".."
+    from.concat to
+    from << to_file
+    File.join(*from)
+  end
+
+  ##
+  # Generate a hyperlink for url, labeled with text. Handle the
+  # special cases for img: and link: described under handle_special_HYPEDLINK
+
+  def gen_url(url, text)
+    if url =~ /([A-Za-z]+):(.*)/ then
+      type = $1
+      path = $2
+    else
+      type = "http"
+      path = url
+      url  = "http://#{url}"
+    end
+
+    if type == "link" then
+      url = if path[0, 1] == '#' then # is this meaningful?
+              path
+            else
+              self.class.gen_relative_url @from_path, path
+            end
+    end
+
+    if (type == "http" or type == "link") and
+       url =~ /\.(gif|png|jpg|jpeg|bmp)$/ then
+      "<img src=\"#{url}\" />"
+    else
+      "<a href=\"#{url}\">#{text.sub(%r{^#{type}:/*}, '')}</a>"
+    end
+  end
+
+  ##
+  # And we're invoked with a potential external hyperlink mailto:
+  # just gets inserted. http: links are checked to see if they
+  # reference an image. If so, that image gets inserted using an
+  # <img> tag. Otherwise a conventional <a href> is used.  We also
+  # support a special type of hyperlink, link:, which is a reference
+  # to a local file whose path is relative to the --op directory.
+
+  def handle_special_HYPERLINK(special)
+    url = special.text
+    gen_url url, url
+  end
+
+  ##
+  # Here's a hypedlink where the label is different to the URL
+  #  <label>[url] or {long label}[url]
+
+  def handle_special_TIDYLINK(special)
+    text = special.text
+
+    return text unless text =~ /\{(.*?)\}\[(.*?)\]/ or text =~ /(\S+)\[(.*?)\]/
+
+    label = $1
+    url   = $2
+    gen_url url, label
+  end
+
+  ##
+  # are we currently inside <tt> tags?
+
+  def in_tt?
+    @in_tt > 0
+  end
+
+  ##
+  # is +tag+ a <tt> tag?
+
+  def tt?(tag)
+    tag.bit == @tt_bit
+  end
+
+  ##
+  # Set up the standard mapping of attributes to HTML tags
+
+  def init_tags
+    @attr_tags = [
+      InlineTag.new(RDoc::Markup::Attribute.bitmap_for(:BOLD), "<b>", "</b>"),
+      InlineTag.new(RDoc::Markup::Attribute.bitmap_for(:TT),   "<tt>", "</tt>"),
+      InlineTag.new(RDoc::Markup::Attribute.bitmap_for(:EM),   "<em>", "</em>"),
+    ]
+  end
+
+  ##
+  # Add a new set of HTML tags for an attribute. We allow separate start and
+  # end tags for flexibility.
+
+  def add_tag(name, start, stop)
+    @attr_tags << InlineTag.new(RDoc::Markup::Attribute.bitmap_for(name), start, stop)
+  end
+
+  ##
+  # Given an HTML tag, decorate it with class information and the like if
+  # required. This is a no-op in the base class, but is overridden in HTML
+  # output classes that implement style sheets.
+
+  def annotate(tag)
+    tag
+  end
+
+  ##
+  # Here's the client side of the visitor pattern
+
+  def start_accepting
+    @res = ""
+    @in_list_entry = []
+  end
+
+  def end_accepting
+    @res
+  end
+
+  def accept_paragraph(am, fragment)
+    @res << annotate("<p>") + "\n"
+    @res << wrap(convert_flow(am.flow(fragment.txt)))
+    @res << annotate("</p>") + "\n"
+  end
+
+  def accept_verbatim(am, fragment)
+    @res << annotate("<pre>") + "\n"
+    @res << CGI.escapeHTML(fragment.txt)
+    @res << annotate("</pre>") << "\n"
+  end
+
+  def accept_rule(am, fragment)
+    size = fragment.param
+    size = 10 if size > 10
+    @res << "<hr size=\"#{size}\"></hr>"
+  end
+
+  def accept_list_start(am, fragment)
+    @res << html_list_name(fragment.type, true) << "\n"
+    @in_list_entry.push false
+  end
+
+  def accept_list_end(am, fragment)
+    if tag = @in_list_entry.pop
+      @res << annotate(tag) << "\n"
+    end
+    @res << html_list_name(fragment.type, false) << "\n"
+  end
+
+  def accept_list_item(am, fragment)
+    if tag = @in_list_entry.last
+      @res << annotate(tag) << "\n"
+    end
+
+    @res << list_item_start(am, fragment)
+
+    @res << wrap(convert_flow(am.flow(fragment.txt))) << "\n"
+
+    @in_list_entry[-1] = list_end_for(fragment.type)
+  end
+
+  def accept_blank_line(am, fragment)
+    # @res << annotate("<p />") << "\n"
+  end
+
+  def accept_heading(am, fragment)
+    @res << convert_heading(fragment.head_level, am.flow(fragment.txt))
+  end
+
+  ##
+  # This is a higher speed (if messier) version of wrap
+
+  def wrap(txt, line_len = 76)
+    res = ""
+    sp = 0
+    ep = txt.length
+    while sp < ep
+      # scan back for a space
+      p = sp + line_len - 1
+      if p >= ep
+        p = ep
+      else
+        while p > sp and txt[p] != ?\s
+          p -= 1
+        end
+        if p <= sp
+          p = sp + line_len
+          while p < ep and txt[p] != ?\s
+            p += 1
+          end
+        end
+      end
+      res << txt[sp...p] << "\n"
+      sp = p
+      sp += 1 while sp < ep and txt[sp] == ?\s
+    end
+    res
+  end
+
+  private
+
+  def on_tags(res, item)
+    attr_mask = item.turn_on
+    return if attr_mask.zero?
+
+    @attr_tags.each do |tag|
+      if attr_mask & tag.bit != 0
+        res << annotate(tag.on)
+        @in_tt += 1 if tt?(tag)
+      end
+    end
+  end
+
+  def off_tags(res, item)
+    attr_mask = item.turn_off
+    return if attr_mask.zero?
+
+    @attr_tags.reverse_each do |tag|
+      if attr_mask & tag.bit != 0
+        @in_tt -= 1 if tt?(tag)
+        res << annotate(tag.off)
+      end
+    end
+  end
+
+  def convert_flow(flow)
+    res = ""
+
+    flow.each do |item|
+      case item
+      when String
+        res << convert_string(item)
+      when RDoc::Markup::AttrChanger
+        off_tags(res, item)
+        on_tags(res,  item)
+      when RDoc::Markup::Special
+        res << convert_special(item)
+      else
+        raise "Unknown flow element: #{item.inspect}"
+      end
+    end
+
+    res
+  end
+
+  def convert_string(item)
+    in_tt? ? convert_string_simple(item) : convert_string_fancy(item)
+  end
+
+  def convert_string_simple(item)
+    CGI.escapeHTML item
+  end
+
+  ##
+  # some of these patterns are taken from SmartyPants...
+
+  def convert_string_fancy(item)
+    # convert -- to em-dash, (-- to en-dash)
+    item.gsub(/---?/, '&#8212;'). #gsub(/--/, '&#8211;').
+
+    # convert ... to elipsis (and make sure .... becomes .<elipsis>)
+      gsub(/\.\.\.\./, '.&#8230;').gsub(/\.\.\./, '&#8230;').
+
+    # convert single closing quote
+      gsub(%r{([^ \t\r\n\[\{\(])\'}, '\1&#8217;'). # }
+      gsub(%r{\'(?=\W|s\b)}, '&#8217;').
+
+    # convert single opening quote
+      gsub(/'/, '&#8216;').
+
+    # convert double closing quote
+      gsub(%r{([^ \t\r\n\[\{\(])\'(?=\W)}, '\1&#8221;'). # }
+
+    # convert double opening quote
+      gsub(/'/, '&#8220;').
+
+    # convert copyright
+      gsub(/\(c\)/, '&#169;').
+
+    # convert and registered trademark
+      gsub(/\(r\)/, '&#174;')
+  end
+
+  def convert_special(special)
+    handled = false
+    RDoc::Markup::Attribute.each_name_of(special.type) do |name|
+      method_name = "handle_special_#{name}"
+      if self.respond_to? method_name
+        special.text = send(method_name, special)
+        handled = true
+      end
+    end
+    raise "Unhandled special: #{special}" unless handled
+    special.text
+  end
+
+  def convert_heading(level, flow)
+    res =
+      annotate("<h#{level}>") +
+      convert_flow(flow) +
+      annotate("</h#{level}>\n")
+  end
+
+  def html_list_name(list_type, is_open_tag)
+    tags = LIST_TYPE_TO_HTML[list_type] || raise("Invalid list type: #{list_type.inspect}")
+    annotate(tags[ is_open_tag ? 0 : 1])
+  end
+
+  def list_item_start(am, fragment)
+    case fragment.type
+    when :BULLET, :NUMBER then
+      annotate("<li>")
+
+    when :UPPERALPHA then
+      annotate("<li type=\"A\">")
+
+    when :LOWERALPHA then
+      annotate("<li type=\"a\">")
+
+    when :LABELED then
+      annotate("<dt>") +
+        convert_flow(am.flow(fragment.param)) +
+        annotate("</dt>") +
+        annotate("<dd>")
+
+    when :NOTE then
+      annotate("<tr>") +
+        annotate("<td valign=\"top\">") +
+        convert_flow(am.flow(fragment.param)) +
+        annotate("</td>") +
+        annotate("<td>")
+    else
+      raise "Invalid list type"
+    end
+  end
+
+  def list_end_for(fragment_type)
+    case fragment_type
+    when :BULLET, :NUMBER, :UPPERALPHA, :LOWERALPHA then
+      "</li>"
+    when :LABELED then
+      "</dd>"
+    when :NOTE then
+      "</td></tr>"
+    else
+      raise "Invalid list type"
+    end
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/markup/to_html_crossref.rb b/trunk/lib/rdoc/markup/to_html_crossref.rb
new file mode 100644
index 0000000000..a6f29c5c2c
--- /dev/null
+++ b/trunk/lib/rdoc/markup/to_html_crossref.rb
@@ -0,0 +1,102 @@
+require 'rdoc/markup/to_html'
+
+##
+# Subclass of the RDoc::Markup::ToHtml class that supports looking up words in
+# the AllReferences list. Those that are found (like AllReferences in this
+# comment) will be hyperlinked
+
+class RDoc::Markup::ToHtmlCrossref < RDoc::Markup::ToHtml
+
+  attr_accessor :context
+
+  ##
+  # We need to record the html path of our caller so we can generate
+  # correct relative paths for any hyperlinks that we find
+
+  def initialize(from_path, context, show_hash)
+    raise ArgumentError, 'from_path cannot be nil' if from_path.nil?
+    super()
+
+    # class names, variable names, or instance variables
+    @markup.add_special(/(
+                           # A::B.meth(**) (for operator in Fortran95)
+                           \w+(::\w+)*[.\#]\w+(\([\.\w+\*\/\+\-\=\<\>]+\))?
+                           # meth(**) (for operator in Fortran95)
+                         | \#\w+(\([.\w\*\/\+\-\=\<\>]+\))?
+                         | \b([A-Z]\w*(::\w+)*[.\#]\w+)  #    A::B.meth
+                         | \b([A-Z]\w+(::\w+)*)          #    A::B
+                         | \#\w+[!?=]?                   #    #meth_name
+                         | \\?\b\w+([_\/\.]+\w+)*[!?=]?  #    meth_name
+                         )/x,
+                        :CROSSREF)
+
+    @from_path = from_path
+    @context = context
+    @show_hash = show_hash
+
+    @seen = {}
+  end
+
+  ##
+  # We're invoked when any text matches the CROSSREF pattern
+  # (defined in MarkUp). If we fine the corresponding reference,
+  # generate a hyperlink. If the name we're looking for contains
+  # no punctuation, we look for it up the module/class chain. For
+  # example, HyperlinkHtml is found, even without the Generator::
+  # prefix, because we look for it in module Generator first.
+
+  def handle_special_CROSSREF(special)
+    name = special.text
+
+    return name if name =~ /\A[a-z]*\z/
+
+    return @seen[name] if @seen.include? name
+
+    if name[0, 1] == '#' then
+      lookup = name[1..-1]
+      name = lookup unless @show_hash
+    else
+      lookup = name
+    end
+
+
+    # Find class, module, or method in class or module.
+    #
+    # Do not, however, use an if/elsif/else chain to do so.  Instead, test
+    # each possible pattern until one matches.  The reason for this is that a
+    # string like "YAML.txt" could be the txt() class method of class YAML (in
+    # which case it would match the first pattern, which splits the string
+    # into container and method components and looks up both) or a filename
+    # (in which case it would match the last pattern, which just checks
+    # whether the string as a whole is a known symbol).
+
+    if /([A-Z][\w:]*)[.\#](\w+[!?=]?)/ =~ lookup then
+      container = $1
+      method = $2
+      ref = @context.find_symbol container, method
+    end
+
+    if !ref and
+       /([A-Za-z][\w:]*)[.\#](\w+(\([\.\w+\*\/\+\-\=\<\>]+\))?)/ =~ lookup then
+      container = $1
+      method = $2
+      ref = @context.find_symbol container, method
+    end
+
+    ref = @context.find_symbol lookup unless ref
+
+    out = if lookup =~ /^\\/ then
+            $'
+          elsif ref and ref.document_self then
+            "<a href=\"#{ref.as_href(@from_path)}\">#{name}</a>"
+          else
+            name
+          end
+
+    @seen[name] = out
+
+    out
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/markup/to_latex.rb b/trunk/lib/rdoc/markup/to_latex.rb
new file mode 100644
index 0000000000..bbf958f2ed
--- /dev/null
+++ b/trunk/lib/rdoc/markup/to_latex.rb
@@ -0,0 +1,328 @@
+require 'rdoc/markup/formatter'
+require 'rdoc/markup/fragments'
+require 'rdoc/markup/inline'
+
+require 'cgi'
+
+##
+# Convert SimpleMarkup to basic LaTeX report format.
+
+class RDoc::Markup::ToLaTeX < RDoc::Markup::Formatter
+
+  BS = "\020"   # \
+  OB = "\021"   # {
+  CB = "\022"   # }
+  DL = "\023"   # Dollar
+
+  BACKSLASH   = "#{BS}symbol#{OB}92#{CB}"
+  HAT         = "#{BS}symbol#{OB}94#{CB}"
+  BACKQUOTE   = "#{BS}symbol#{OB}0#{CB}"
+  TILDE       = "#{DL}#{BS}sim#{DL}"
+  LESSTHAN    = "#{DL}<#{DL}"
+  GREATERTHAN = "#{DL}>#{DL}"
+
+  def self.l(str)
+    str.tr('\\', BS).tr('{', OB).tr('}', CB).tr('$', DL)
+  end
+
+  def l(arg)
+    RDoc::Markup::ToLaTeX.l(arg)
+  end
+
+  LIST_TYPE_TO_LATEX = {
+    :BULLET =>  [ l("\\begin{itemize}"), l("\\end{itemize}") ],
+    :NUMBER =>  [ l("\\begin{enumerate}"), l("\\end{enumerate}"), "\\arabic" ],
+    :UPPERALPHA =>  [ l("\\begin{enumerate}"), l("\\end{enumerate}"), "\\Alph" ],
+    :LOWERALPHA =>  [ l("\\begin{enumerate}"), l("\\end{enumerate}"), "\\alph" ],
+    :LABELED => [ l("\\begin{description}"), l("\\end{description}") ],
+    :NOTE    => [
+      l("\\begin{tabularx}{\\linewidth}{@{} l X @{}}"),
+      l("\\end{tabularx}") ],
+  }
+
+  InlineTag = Struct.new(:bit, :on, :off)
+
+  def initialize
+    init_tags
+    @list_depth = 0
+    @prev_list_types = []
+  end
+
+  ##
+  # Set up the standard mapping of attributes to LaTeX
+
+  def init_tags
+    @attr_tags = [
+      InlineTag.new(RDoc::Markup::Attribute.bitmap_for(:BOLD), l("\\textbf{"), l("}")),
+      InlineTag.new(RDoc::Markup::Attribute.bitmap_for(:TT),   l("\\texttt{"), l("}")),
+      InlineTag.new(RDoc::Markup::Attribute.bitmap_for(:EM),   l("\\emph{"), l("}")),
+    ]
+  end
+
+  ##
+  # Escape a LaTeX string
+
+  def escape(str)
+    $stderr.print "FE: ", str if $DEBUG_RDOC
+    s = str.
+       sub(/\s+$/, '').
+      gsub(/([_\${}&%#])/, "#{BS}\\1").
+      gsub(/\\/, BACKSLASH).
+      gsub(/\^/, HAT).
+      gsub(/~/,  TILDE).
+      gsub(/</,  LESSTHAN).
+      gsub(/>/,  GREATERTHAN).
+      gsub(/,,/, ",{},").
+      gsub(/\`/,  BACKQUOTE)
+    $stderr.print "-> ", s, "\n" if $DEBUG_RDOC
+    s
+  end
+
+  ##
+  # Add a new set of LaTeX tags for an attribute. We allow
+  # separate start and end tags for flexibility
+
+  def add_tag(name, start, stop)
+    @attr_tags << InlineTag.new(RDoc::Markup::Attribute.bitmap_for(name), start, stop)
+  end
+
+  ##
+  # Here's the client side of the visitor pattern
+
+  def start_accepting
+    @res = ""
+    @in_list_entry = []
+  end
+
+  def end_accepting
+    @res.tr(BS, '\\').tr(OB, '{').tr(CB, '}').tr(DL, '$')
+  end
+
+  def accept_paragraph(am, fragment)
+    @res << wrap(convert_flow(am.flow(fragment.txt)))
+    @res << "\n"
+  end
+
+  def accept_verbatim(am, fragment)
+    @res << "\n\\begin{code}\n"
+    @res << fragment.txt.sub(/[\n\s]+\Z/, '')
+    @res << "\n\\end{code}\n\n"
+  end
+
+  def accept_rule(am, fragment)
+    size = fragment.param
+    size = 10 if size > 10
+    @res << "\n\n\\rule{\\linewidth}{#{size}pt}\n\n"
+  end
+
+  def accept_list_start(am, fragment)
+    @res << list_name(fragment.type, true) << "\n"
+    @in_list_entry.push false
+  end
+
+  def accept_list_end(am, fragment)
+    if tag = @in_list_entry.pop
+      @res << tag << "\n"
+    end
+    @res << list_name(fragment.type, false) << "\n"
+  end
+
+  def accept_list_item(am, fragment)
+    if tag = @in_list_entry.last
+      @res << tag << "\n"
+    end
+    @res << list_item_start(am, fragment)
+    @res << wrap(convert_flow(am.flow(fragment.txt))) << "\n"
+    @in_list_entry[-1] = list_end_for(fragment.type)
+  end
+
+  def accept_blank_line(am, fragment)
+    # @res << "\n"
+  end
+
+  def accept_heading(am, fragment)
+    @res << convert_heading(fragment.head_level, am.flow(fragment.txt))
+  end
+
+  ##
+  # This is a higher speed (if messier) version of wrap
+
+  def wrap(txt, line_len = 76)
+    res = ""
+    sp = 0
+    ep = txt.length
+    while sp < ep
+      # scan back for a space
+      p = sp + line_len - 1
+      if p >= ep
+        p = ep
+      else
+        while p > sp and txt[p] != ?\s
+          p -= 1
+        end
+        if p <= sp
+          p = sp + line_len
+          while p < ep and txt[p] != ?\s
+            p += 1
+          end
+        end
+      end
+      res << txt[sp...p] << "\n"
+      sp = p
+      sp += 1 while sp < ep and txt[sp] == ?\s
+    end
+    res
+  end
+
+  private
+
+  def on_tags(res, item)
+    attr_mask = item.turn_on
+    return if attr_mask.zero?
+
+    @attr_tags.each do |tag|
+      if attr_mask & tag.bit != 0
+        res << tag.on
+      end
+    end
+  end
+
+  def off_tags(res, item)
+    attr_mask = item.turn_off
+    return if attr_mask.zero?
+
+    @attr_tags.reverse_each do |tag|
+      if attr_mask & tag.bit != 0
+        res << tag.off
+      end
+    end
+  end
+
+  def convert_flow(flow)
+    res = ""
+    flow.each do |item|
+      case item
+      when String
+        $stderr.puts "Converting '#{item}'" if $DEBUG_RDOC
+        res << convert_string(item)
+      when AttrChanger
+        off_tags(res, item)
+        on_tags(res,  item)
+      when Special
+        res << convert_special(item)
+      else
+        raise "Unknown flow element: #{item.inspect}"
+      end
+    end
+    res
+  end
+
+  ##
+  # some of these patterns are taken from SmartyPants...
+
+  def convert_string(item)
+    escape(item).
+
+    # convert ... to elipsis (and make sure .... becomes .<elipsis>)
+      gsub(/\.\.\.\./, '.\ldots{}').gsub(/\.\.\./, '\ldots{}').
+
+    # convert single closing quote
+      gsub(%r{([^ \t\r\n\[\{\(])\'}, '\1\'').
+      gsub(%r{\'(?=\W|s\b)}, "'" ).
+
+    # convert single opening quote
+      gsub(/'/, '`').
+
+    # convert double closing quote
+      gsub(%r{([^ \t\r\n\[\{\(])\"(?=\W)}, "\\1''").
+
+    # convert double opening quote
+      gsub(/"/, "``").
+
+    # convert copyright
+      gsub(/\(c\)/, '\copyright{}')
+
+  end
+
+  def convert_special(special)
+    handled = false
+    Attribute.each_name_of(special.type) do |name|
+      method_name = "handle_special_#{name}"
+      if self.respond_to? method_name
+        special.text = send(method_name, special)
+        handled = true
+      end
+    end
+    raise "Unhandled special: #{special}" unless handled
+    special.text
+  end
+
+  def convert_heading(level, flow)
+    res =
+      case level
+      when 1 then "\\chapter{"
+      when 2 then "\\section{"
+      when 3 then "\\subsection{"
+      when 4 then "\\subsubsection{"
+      else  "\\paragraph{"
+      end +
+      convert_flow(flow) +
+      "}\n"
+  end
+
+  def list_name(list_type, is_open_tag)
+    tags = LIST_TYPE_TO_LATEX[list_type] || raise("Invalid list type: #{list_type.inspect}")
+    if tags[2] # enumerate
+      if is_open_tag
+        @list_depth += 1
+        if @prev_list_types[@list_depth] != tags[2]
+          case @list_depth
+          when 1
+            roman = "i"
+          when 2
+            roman = "ii"
+          when 3
+            roman = "iii"
+          when 4
+            roman = "iv"
+          else
+            raise("Too deep list: level #{@list_depth}")
+          end
+          @prev_list_types[@list_depth] = tags[2]
+          return l("\\renewcommand{\\labelenum#{roman}}{#{tags[2]}{enum#{roman}}}") + "\n" + tags[0]
+        end
+      else
+        @list_depth -= 1
+      end
+    end
+    tags[ is_open_tag ? 0 : 1]
+  end
+
+  def list_item_start(am, fragment)
+    case fragment.type
+    when :BULLET, :NUMBER, :UPPERALPHA, :LOWERALPHA then
+      "\\item "
+
+    when :LABELED then
+      "\\item[" + convert_flow(am.flow(fragment.param)) + "] "
+
+    when :NOTE then
+        convert_flow(am.flow(fragment.param)) + " & "
+    else
+      raise "Invalid list type"
+    end
+  end
+
+  def list_end_for(fragment_type)
+    case fragment_type
+    when :BULLET, :NUMBER, :UPPERALPHA, :LOWERALPHA, :LABELED then
+      ""
+    when :NOTE
+      "\\\\\n"
+    else
+      raise "Invalid list type"
+    end
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/markup/to_test.rb b/trunk/lib/rdoc/markup/to_test.rb
new file mode 100644
index 0000000000..ce6aff6e9a
--- /dev/null
+++ b/trunk/lib/rdoc/markup/to_test.rb
@@ -0,0 +1,50 @@
+require 'rdoc/markup'
+require 'rdoc/markup/formatter'
+
+##
+# This Markup outputter is used for testing purposes.
+
+class RDoc::Markup::ToTest < RDoc::Markup::Formatter
+
+  def start_accepting
+    @res = []
+  end
+
+  def end_accepting
+    @res
+  end
+
+  def accept_paragraph(am, fragment)
+    @res << fragment.to_s
+  end
+
+  def accept_verbatim(am, fragment)
+    @res << fragment.to_s
+  end
+
+  def accept_list_start(am, fragment)
+    @res << fragment.to_s
+  end
+
+  def accept_list_end(am, fragment)
+    @res << fragment.to_s
+  end
+
+  def accept_list_item(am, fragment)
+    @res << fragment.to_s
+  end
+
+  def accept_blank_line(am, fragment)
+    @res << fragment.to_s
+  end
+
+  def accept_heading(am, fragment)
+    @res << fragment.to_s
+  end
+
+  def accept_rule(am, fragment)
+    @res << fragment.to_s
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/markup/to_texinfo.rb b/trunk/lib/rdoc/markup/to_texinfo.rb
new file mode 100644
index 0000000000..533d3e34a0
--- /dev/null
+++ b/trunk/lib/rdoc/markup/to_texinfo.rb
@@ -0,0 +1,69 @@
+require 'rdoc/markup/formatter'
+require 'rdoc/markup/fragments'
+require 'rdoc/markup/inline'
+
+require 'rdoc/markup'
+require 'rdoc/markup/formatter'
+
+##
+# Convert SimpleMarkup to basic TexInfo format
+#
+# TODO: WTF is AttributeManager for?
+#
+class RDoc::Markup::ToTexInfo < RDoc::Markup::Formatter
+
+  def start_accepting
+    @text = []
+  end
+
+  def end_accepting
+    @text.join("\n")
+  end
+
+  def accept_paragraph(attributes, text)
+    @text << format(text)
+  end
+
+  def accept_verbatim(attributes, text)
+    @text << "@verb{|#{format(text)}|}"
+  end
+
+  def accept_heading(attributes, text)
+    heading = ['@majorheading', '@chapheading'][text.head_level - 1] || '@heading'
+    @text << "#{heading}{#{format(text)}}"
+  end
+
+  def accept_list_start(attributes, text)
+    @text << '@itemize @bullet'
+  end
+
+  def accept_list_end(attributes, text)
+    @text << '@end itemize'
+  end
+
+  def accept_list_item(attributes, text)
+    @text << "@item\n#{format(text)}"
+  end
+
+  def accept_blank_line(attributes, text)
+    @text << "\n"
+  end
+
+  def accept_rule(attributes, text)
+    @text << '-----'
+  end
+
+  def format(text)
+    text.txt.
+      gsub(/@/, "@@").
+      gsub(/\{/, "@{").
+      gsub(/\}/, "@}").
+      # gsub(/,/, "@,"). # technically only required in cross-refs
+      gsub(/\+([\w]+)\+/, "@code{\\1}").
+      gsub(/\<tt\>([^<]+)\<\/tt\>/, "@code{\\1}").
+      gsub(/\*([\w]+)\*/, "@strong{\\1}").
+      gsub(/\<b\>([^<]+)\<\/b\>/, "@strong{\\1}").
+      gsub(/_([\w]+)_/, "@emph{\\1}").
+      gsub(/\<em\>([^<]+)\<\/em\>/, "@emph{\\1}")
+  end
+end
diff --git a/trunk/lib/rdoc/options.rb b/trunk/lib/rdoc/options.rb
new file mode 100644
index 0000000000..d683a14022
--- /dev/null
+++ b/trunk/lib/rdoc/options.rb
@@ -0,0 +1,639 @@
+# We handle the parsing of options, and subsequently as a singleton
+# object to be queried for option values
+
+require "rdoc/ri/paths"
+require 'optparse'
+
+class RDoc::Options
+
+  ##
+  # Should the output be placed into a single file
+
+  attr_reader :all_one_file
+
+  ##
+  # Character-set
+
+  attr_reader :charset
+
+  ##
+  # URL of stylesheet
+
+  attr_reader :css
+
+  ##
+  # Should diagrams be drawn
+
+  attr_reader :diagram
+
+  ##
+  # Files matching this pattern will be excluded
+
+  attr_accessor :exclude
+
+  ##
+  # Additional attr_... style method flags
+
+  attr_reader :extra_accessor_flags
+
+  ##
+  # Pattern for additional attr_... style methods
+
+  attr_accessor :extra_accessors
+
+  ##
+  # Should we draw fileboxes in diagrams
+
+  attr_reader :fileboxes
+
+  ##
+  # The list of files to be processed
+
+  attr_accessor :files
+
+  ##
+  # Scan newer sources than the flag file if true.
+
+  attr_reader :force_update
+
+  ##
+  # Description of the output generator (set with the <tt>-fmt</tt> option)
+
+  attr_accessor :generator
+
+  ##
+  # Formatter to mark up text with
+
+  attr_accessor :formatter
+  
+  ##
+  # image format for diagrams
+
+  attr_reader :image_format
+
+  ##
+  # Include line numbers in the source listings
+
+  attr_reader :include_line_numbers
+
+  ##
+  # Should source code be included inline, or displayed in a popup
+
+  attr_accessor :inline_source
+
+  ##
+  # Name of the file, class or module to display in the initial index page (if
+  # not specified the first file we encounter is used)
+
+  attr_accessor :main_page
+
+  ##
+  # Merge into classes of the same name when generating ri
+
+  attr_reader :merge
+
+  ##
+  # The name of the output directory
+
+  attr_accessor :op_dir
+
+  ##
+  # The name to use for the output
+
+  attr_accessor :op_name
+
+  ##
+  # Are we promiscuous about showing module contents across multiple files
+
+  attr_reader :promiscuous
+
+  ##
+  # Array of directories to search for files to satisfy an :include:
+
+  attr_reader :rdoc_include
+
+  ##
+  # Include private and protected methods in the output
+
+  attr_accessor :show_all
+
+  ##
+  # Include the '#' at the front of hyperlinked instance method names
+
+  attr_reader :show_hash
+
+  ##
+  # The number of columns in a tab
+
+  attr_reader :tab_width
+
+  ##
+  # template to be used when generating output
+
+  attr_reader :template
+
+  ##
+  # Template class for file generation
+  #--
+  # HACK around dependencies in lib/rdoc/generator/html.rb
+
+  attr_accessor :template_class # :nodoc:
+
+  ##
+  # Documentation title
+
+  attr_reader :title
+
+  ##
+  # Verbosity, zero means quiet
+
+  attr_accessor :verbosity
+
+  ##
+  # URL of web cvs frontend
+
+  attr_reader :webcvs
+
+  def initialize(generators = {}) # :nodoc:
+    @op_dir = "doc"
+    @op_name = nil
+    @show_all = false
+    @main_page = nil
+    @merge = false
+    @exclude = []
+    @generators = generators
+    @generator_name = 'html'
+    @generator = @generators[@generator_name]
+    @rdoc_include = []
+    @title = nil
+    @template = nil
+    @template_class = nil
+    @diagram = false
+    @fileboxes = false
+    @show_hash = false
+    @image_format = 'png'
+    @inline_source = false
+    @all_one_file = false
+    @tab_width = 8
+    @include_line_numbers = false
+    @extra_accessor_flags = {}
+    @promiscuous = false
+    @force_update = false
+    @verbosity = 1
+
+    @css = nil
+    @webcvs = nil
+
+    @charset = 'iso-8859-1'
+  end
+
+  ##
+  # Parse command line options.
+
+  def parse(argv)
+    accessors = []
+
+    opts = OptionParser.new do |opt|
+      opt.program_name = File.basename $0
+      opt.version = RDoc::VERSION
+      opt.summary_indent = ' ' * 4
+      opt.banner = <<-EOF
+Usage: #{opt.program_name} [options] [names...]
+
+  Files are parsed, and the information they contain collected, before any
+  output is produced. This allows cross references between all files to be
+  resolved. If a name is a directory, it is traversed. If no names are
+  specified, all Ruby files in the current directory (and subdirectories) are
+  processed.
+
+  How RDoc generates output depends on the output formatter being used, and on
+  the options you give.
+
+  - HTML output is normally produced into a number of separate files
+    (one per class, module, and file, along with various indices).
+    These files will appear in the directory given by the --op
+    option (doc/ by default).
+
+  - XML output by default is written to standard output. If a
+    --opname option is given, the output will instead be written
+    to a file with that name in the output directory.
+
+  - .chm files (Windows help files) are written in the --op directory.
+    If an --opname parameter is present, that name is used, otherwise
+    the file will be called rdoc.chm.
+      EOF
+
+      opt.separator nil
+      opt.separator "Options:"
+      opt.separator nil
+
+      opt.on("--accessor=ACCESSORS", "-A", Array,
+             "A comma separated list of additional class",
+             "methods that should be treated like",
+             "'attr_reader' and friends.",
+             " ",
+             "Option may be repeated.",
+             " ",
+             "Each accessorname may have '=text'",
+             "appended, in which case that text appears",
+             "where the r/w/rw appears for normal.",
+             "accessors") do |value|
+        value.each do |accessor|
+          if accessor =~ /^(\w+)(=(.*))?$/
+            accessors << $1
+            @extra_accessor_flags[$1] = $3
+          end
+        end
+      end
+
+      opt.separator nil
+
+      opt.on("--all", "-a",
+             "Include all methods (not just public) in",
+             "the output.") do |value|
+        @show_all = value
+      end
+
+      opt.separator nil
+
+      opt.on("--charset=CHARSET", "-c",
+             "Specifies the HTML character-set.") do |value|
+        @charset = value
+      end
+
+      opt.separator nil
+
+      opt.on("--debug", "-D",
+             "Displays lots on internal stuff.") do |value|
+        $DEBUG_RDOC = value
+      end
+
+      opt.separator nil
+
+      opt.on("--diagram", "-d",
+             "Generate diagrams showing modules and",
+             "classes. You need dot V1.8.6 or later to",
+             "use the --diagram option correctly. Dot is",
+             "available from http://graphviz.org") do |value|
+        check_diagram
+        @diagram = true
+      end
+
+      opt.separator nil
+
+      opt.on("--exclude=PATTERN", "-x", Regexp,
+             "Do not process files or directories",
+             "matching PATTERN. Files given explicitly",
+             "on the command line will never be",
+             "excluded.") do |value|
+        @exclude << value
+      end
+
+      opt.separator nil
+
+      opt.on("--extension=NEW=OLD", "-E",
+             "Treat files ending with .new as if they",
+             "ended with .old. Using '-E cgi=rb' will",
+             "cause xxx.cgi to be parsed as a Ruby file.") do |value|
+        new, old = value.split(/=/, 2)
+
+        unless new and old then
+          raise OptionParser::InvalidArgument, "Invalid parameter to '-E'"
+        end
+
+        unless RDoc::ParserFactory.alias_extension old, new then
+          raise OptionParser::InvalidArgument, "Unknown extension .#{old} to -E"
+        end
+      end
+
+      opt.separator nil
+
+      opt.on("--fileboxes", "-F",
+             "Classes are put in boxes which represents",
+             "files, where these classes reside. Classes",
+             "shared between more than one file are",
+             "shown with list of files that are sharing",
+             "them. Silently discarded if --diagram is",
+             "not given.") do |value|
+        @fileboxes = value
+      end
+
+      opt.separator nil
+
+      opt.on("--force-update", "-U",
+             "Forces rdoc to scan all sources even if",
+             "newer than the flag file.") do |value|
+        @force_update = value
+      end
+
+      opt.separator nil
+
+      opt.on("--fmt=FORMAT", "--format=FORMAT", "-f", @generators.keys,
+             "Set the output formatter.") do |value|
+        @generator_name = value.downcase
+        setup_generator
+      end
+
+      opt.separator nil
+
+      image_formats = %w[gif png jpg jpeg]
+      opt.on("--image-format=FORMAT", "-I", image_formats,
+             "Sets output image format for diagrams. Can",
+             "be #{image_formats.join ', '}. If this option",
+             "is omitted, png is used. Requires",
+             "diagrams.") do |value|
+        @image_format = value
+      end
+
+      opt.separator nil
+
+      opt.on("--include=DIRECTORIES", "-i", Array,
+             "set (or add to) the list of directories to",
+             "be searched when satisfying :include:",
+             "requests. Can be used more than once.") do |value|
+        @rdoc_include.concat value.map { |dir| dir.strip }
+      end
+
+      opt.separator nil
+
+      opt.on("--inline-source", "-S",
+             "Show method source code inline, rather than",
+             "via a popup link.") do |value|
+        @inline_source = value
+      end
+
+      opt.separator nil
+
+      opt.on("--line-numbers", "-N",
+             "Include line numbers in the source code.") do |value|
+        @include_line_numbers = value
+      end
+
+      opt.separator nil
+
+      opt.on("--main=NAME", "-m",
+             "NAME will be the initial page displayed.") do |value|
+        @main_page = value
+      end
+
+      opt.separator nil
+
+      opt.on("--merge", "-M",
+             "When creating ri output, merge previously",
+             "processed classes into previously",
+             "documented classes of the same name.") do |value|
+        @merge = value
+      end
+
+      opt.separator nil
+
+      opt.on("--one-file", "-1",
+             "Put all the output into a single file.") do |value|
+        @all_one_file = value
+        @inline_source = value if value
+        @template = 'one_page_html'
+      end
+
+      opt.separator nil
+
+      opt.on("--op=DIR", "-o",
+             "Set the output directory.") do |value|
+        @op_dir = value
+      end
+
+      opt.separator nil
+
+      opt.on("--opname=NAME", "-n",
+             "Set the NAME of the output. Has no effect",
+             "for HTML.") do |value|
+        @op_name = value
+      end
+
+      opt.separator nil
+
+      opt.on("--promiscuous", "-p",
+             "When documenting a file that contains a",
+             "module or class also defined in other",
+             "files, show all stuff for that module or",
+             "class in each files page. By default, only",
+             "show stuff defined in that particular file.") do |value|
+        @promiscuous = value
+      end
+
+      opt.separator nil
+
+      opt.on("--quiet", "-q",
+             "Don't show progress as we parse.") do |value|
+        @verbosity = 0
+      end
+
+      opt.on("--verbose", "-v",
+             "Display extra progress as we parse.") do |value|
+        @verbosity = 2
+      end
+
+
+      opt.separator nil
+
+      opt.on("--ri", "-r",
+             "Generate output for use by `ri`. The files",
+             "are stored in the '.rdoc' directory under",
+             "your home directory unless overridden by a",
+             "subsequent --op parameter, so no special",
+             "privileges are needed.") do |value|
+        @generator_name = "ri"
+        @op_dir = RDoc::RI::Paths::HOMEDIR
+        setup_generator
+      end
+
+      opt.separator nil
+
+      opt.on("--ri-site", "-R",
+             "Generate output for use by `ri`. The files",
+             "are stored in a site-wide directory,",
+             "making them accessible to others, so",
+             "special privileges are needed.") do |value|
+        @generator_name = "ri"
+        @op_dir = RDoc::RI::Paths::SITEDIR
+        setup_generator
+      end
+
+      opt.separator nil
+
+      opt.on("--ri-system", "-Y",
+             "Generate output for use by `ri`. The files",
+             "are stored in a site-wide directory,",
+             "making them accessible to others, so",
+             "special privileges are needed.  This",
+             "option is intended to be used during Ruby",
+             "installation.") do |value|
+        @generator_name = "ri"
+        @op_dir = RDoc::RI::Paths::SYSDIR
+        setup_generator
+      end
+
+      opt.separator nil
+
+      opt.on("--show-hash", "-H",
+             "A name of the form #name in a comment is a",
+             "possible hyperlink to an instance method",
+             "name. When displayed, the '#' is removed",
+             "unless this option is specified.") do |value|
+        @show_hash = value
+      end
+
+      opt.separator nil
+
+      opt.on("--style=URL", "-s",
+             "Specifies the URL of a separate stylesheet.") do |value|
+        @css = value
+      end
+
+      opt.separator nil
+
+      opt.on("--tab-width=WIDTH", "-w", OptionParser::DecimalInteger,
+             "Set the width of tab characters.") do |value|
+        @tab_width = value
+      end
+
+      opt.separator nil
+
+      opt.on("--template=NAME", "-T",
+             "Set the template used when generating",
+             "output.") do |value|
+        @template = value
+      end
+
+      opt.separator nil
+
+      opt.on("--title=TITLE", "-t",
+             "Set TITLE as the title for HTML output.") do |value|
+        @title = value
+      end
+
+      opt.separator nil
+
+      opt.on("--webcvs=URL", "-W",
+             "Specify a URL for linking to a web frontend",
+             "to CVS. If the URL contains a '\%s', the",
+             "name of the current file will be",
+             "substituted; if the URL doesn't contain a",
+             "'\%s', the filename will be appended to it.") do |value|
+        @webcvs = value
+      end
+    end
+
+    argv.insert(0, *ENV['RDOCOPT'].split) if ENV['RDOCOPT']
+
+    opts.parse! argv
+
+    @files = argv.dup
+
+    @rdoc_include << "." if @rdoc_include.empty?
+
+    if @exclude.empty? then
+      @exclude = nil
+    else
+      @exclude = Regexp.new(@exclude.join("|"))
+    end
+
+    check_files
+
+    # If no template was specified, use the default template for the output
+    # formatter
+
+    @template ||= @generator_name
+
+    # Generate a regexp from the accessors
+    unless accessors.empty? then
+      re = '^(' + accessors.map { |a| Regexp.quote a }.join('|') + ')$'
+      @extra_accessors = Regexp.new re
+    end
+
+  rescue OptionParser::InvalidArgument, OptionParser::InvalidOption => e
+    puts opts
+    puts
+    puts e
+    exit 1
+  end
+
+  ##
+  # Set the title, but only if not already set. This means that a title set
+  # from the command line trumps one set in a source file
+
+  def title=(string)
+    @title ||= string
+  end
+
+  ##
+  # Don't display progress as we process the files
+
+  def quiet
+    @verbosity.zero?
+  end
+
+  def quiet=(bool)
+    @verbosity = bool ? 0 : 1
+  end
+
+  private
+
+  ##
+  # Set up an output generator for the format in @generator_name
+
+  def setup_generator
+    @generator = @generators[@generator_name]
+
+    unless @generator then
+      raise OptionParser::InvalidArgument, "Invalid output formatter"
+    end
+
+    if @generator_name == "xml" then
+      @all_one_file = true
+      @inline_source = true
+    end
+  end
+
+  # Check that the right version of 'dot' is available.  Unfortunately this
+  # doesn't work correctly under Windows NT, so we'll bypass the test under
+  # Windows.
+
+  def check_diagram
+    return if RUBY_PLATFORM =~ /mswin|cygwin|mingw|bccwin/
+
+    ok = false
+    ver = nil
+
+    IO.popen "dot -V 2>&1" do |io|
+      ver = io.read
+      if ver =~ /dot.+version(?:\s+gviz)?\s+(\d+)\.(\d+)/ then
+        ok = ($1.to_i > 1) || ($1.to_i == 1 && $2.to_i >= 8)
+      end
+    end
+
+    unless ok then
+      if ver =~ /^dot.+version/ then
+        $stderr.puts "Warning: You may need dot V1.8.6 or later to use\n",
+          "the --diagram option correctly. You have:\n\n   ",
+          ver,
+          "\nDiagrams might have strange background colors.\n\n"
+      else
+        $stderr.puts "You need the 'dot' program to produce diagrams.",
+          "(see http://www.research.att.com/sw/tools/graphviz/)\n\n"
+        exit
+      end
+    end
+  end
+
+  ##
+  # Check that the files on the command line exist
+
+  def check_files
+    @files.each do |f|
+      stat = File.stat f
+      raise RDoc::Error, "file '#{f}' not readable" unless stat.readable?
+    end
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/parser.rb b/trunk/lib/rdoc/parser.rb
new file mode 100644
index 0000000000..794fad00e9
--- /dev/null
+++ b/trunk/lib/rdoc/parser.rb
@@ -0,0 +1,109 @@
+require 'rdoc'
+require 'rdoc/code_objects'
+require 'rdoc/markup/preprocess'
+require 'rdoc/stats'
+
+##
+# A parser is simple a class that implements
+#
+#   #initialize(file_name, body, options)
+#
+# and
+#
+#   #scan
+#
+# The initialize method takes a file name to be used, the body of the file,
+# and an RDoc::Options object. The scan method is then called to return an
+# appropriately parsed TopLevel code object.
+#
+# The ParseFactory is used to redirect to the correct parser given a
+# filename extension. This magic works because individual parsers have to
+# register themselves with us as they are loaded in. The do this using the
+# following incantation
+#
+#   require "rdoc/parser"
+#   
+#   class RDoc::Parser::Xyz < RDoc::Parser
+#     parse_files_matching /\.xyz$/ # <<<<
+#   
+#     def initialize(file_name, body, options)
+#       ...
+#     end
+#   
+#     def scan
+#       ...
+#     end
+#   end
+#
+# Just to make life interesting, if we suspect a plain text file, we also
+# look for a shebang line just in case it's a potential shell script
+
+class RDoc::Parser
+
+  @parsers = []
+
+  class << self
+    attr_reader :parsers
+  end
+
+  ##
+  # Alias an extension to another extension. After this call, files ending
+  # "new_ext" will be parsed using the same parser as "old_ext"
+
+  def self.alias_extension(old_ext, new_ext)
+    old_ext = old_ext.sub(/^\.(.*)/, '\1')
+    new_ext = new_ext.sub(/^\.(.*)/, '\1')
+
+    parser = can_parse "xxx.#{old_ext}"
+    return false unless parser
+
+    RDoc::Parser.parsers.unshift [/\.#{new_ext}$/, parser]
+
+    true
+  end
+
+  ##
+  # Return a parser that can handle a particular extension
+
+  def self.can_parse(file_name)
+    RDoc::Parser.parsers.find { |regexp, parser| regexp =~ file_name }.last
+  end
+
+  ##
+  # Find the correct parser for a particular file name. Return a SimpleParser
+  # for ones that we don't know
+
+  def self.for(top_level, file_name, body, options, stats)
+    # If no extension, look for shebang
+    if file_name !~ /\.\w+$/ && body =~ %r{\A#!(.+)} then
+      shebang = $1
+      case shebang
+      when %r{env\s+ruby}, %r{/ruby}
+        file_name = "dummy.rb"
+      end
+    end
+
+    parser = can_parse file_name
+
+    parser.new top_level, file_name, body, options, stats
+  end
+
+  ##
+  # Record which file types this parser can understand.
+
+  def self.parse_files_matching(regexp)
+    RDoc::Parser.parsers.unshift [regexp, self]
+  end
+
+  def initialize(top_level, file_name, content, options, stats)
+    @top_level = top_level
+    @file_name = file_name
+    @content = content
+    @options = options
+    @stats = stats
+  end
+
+end
+
+require 'rdoc/parser/simple'
+
diff --git a/trunk/lib/rdoc/parser/c.rb b/trunk/lib/rdoc/parser/c.rb
new file mode 100644
index 0000000000..43bb767da9
--- /dev/null
+++ b/trunk/lib/rdoc/parser/c.rb
@@ -0,0 +1,656 @@
+require 'rdoc/parser'
+require 'rdoc/known_classes'
+
+##
+# We attempt to parse C extension files. Basically we look for
+# the standard patterns that you find in extensions: <tt>rb_define_class,
+# rb_define_method</tt> and so on. We also try to find the corresponding
+# C source for the methods and extract comments, but if we fail
+# we don't worry too much.
+#
+# The comments associated with a Ruby method are extracted from the C
+# comment block associated with the routine that _implements_ that
+# method, that is to say the method whose name is given in the
+# <tt>rb_define_method</tt> call. For example, you might write:
+#
+#  /*
+#   * Returns a new array that is a one-dimensional flattening of this
+#   * array (recursively). That is, for every element that is an array,
+#   * extract its elements into the new array.
+#   *
+#   *    s = [ 1, 2, 3 ]           #=> [1, 2, 3]
+#   *    t = [ 4, 5, 6, [7, 8] ]   #=> [4, 5, 6, [7, 8]]
+#   *    a = [ s, t, 9, 10 ]       #=> [[1, 2, 3], [4, 5, 6, [7, 8]], 9, 10]
+#   *    a.flatten                 #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
+#   */
+#   static VALUE
+#   rb_ary_flatten(ary)
+#       VALUE ary;
+#   {
+#       ary = rb_obj_dup(ary);
+#       rb_ary_flatten_bang(ary);
+#       return ary;
+#   }
+#
+#   ...
+#
+#   void
+#   Init_Array()
+#   {
+#     ...
+#     rb_define_method(rb_cArray, "flatten", rb_ary_flatten, 0);
+#
+# Here RDoc will determine from the rb_define_method line that there's a
+# method called "flatten" in class Array, and will look for the implementation
+# in the method rb_ary_flatten. It will then use the comment from that
+# method in the HTML output. This method must be in the same source file
+# as the rb_define_method.
+#
+# C classes can be diagrammed (see /tc/dl/ruby/ruby/error.c), and RDoc
+# integrates C and Ruby source into one tree
+#
+# The comment blocks may include special directives:
+#
+# [Document-class: <i>name</i>]
+#   This comment block is documentation for the given class. Use this
+#   when the <tt>Init_xxx</tt> method is not named after the class.
+#
+# [Document-method: <i>name</i>]
+#   This comment documents the named method. Use when RDoc cannot
+#   automatically find the method from it's declaration
+#
+# [call-seq:  <i>text up to an empty line</i>]
+#   Because C source doesn't give descripive names to Ruby-level parameters,
+#   you need to document the calling sequence explicitly
+#
+# In addition, RDoc assumes by default that the C method implementing a
+# Ruby function is in the same source file as the rb_define_method call.
+# If this isn't the case, add the comment:
+#
+#    rb_define_method(....);  // in: filename
+#
+# As an example, we might have an extension that defines multiple classes
+# in its Init_xxx method. We could document them using
+#
+#   /*
+#    * Document-class:  MyClass
+#    *
+#    * Encapsulate the writing and reading of the configuration
+#    * file. ...
+#    */
+#   
+#   /*
+#    * Document-method: read_value
+#    *
+#    * call-seq:
+#    *   cfg.read_value(key)            -> value
+#    *   cfg.read_value(key} { |key| }  -> value
+#    *
+#    * Return the value corresponding to +key+ from the configuration.
+#    * In the second form, if the key isn't found, invoke the
+#    * block and return its value.
+#    */
+
+class RDoc::Parser::C < RDoc::Parser
+
+  parse_files_matching(/\.(?:([CcHh])\1?|c([+xp])\2|y)\z/)
+
+  @@enclosure_classes = {}
+  @@known_bodies = {}
+
+  ##
+  # Prepare to parse a C file
+
+  def initialize(top_level, file_name, content, options, stats)
+    super
+
+    @known_classes = RDoc::KNOWN_CLASSES.dup
+    @content = handle_tab_width handle_ifdefs_in(@content)
+    @classes = Hash.new
+    @file_dir = File.dirname(@file_name)
+  end
+
+  def do_aliases
+    @content.scan(%r{rb_define_alias\s*\(\s*(\w+),\s*"([^"]+)",\s*"([^"]+)"\s*\)}m) do
+      |var_name, new_name, old_name|
+      class_name = @known_classes[var_name] || var_name
+      class_obj  = find_class(var_name, class_name)
+
+      as = class_obj.add_alias RDoc::Alias.new("", old_name, new_name, "")
+
+      @stats.add_alias as
+    end
+ end
+
+  def do_classes
+    @content.scan(/(\w+)\s* = \s*rb_define_module\s*\(\s*"(\w+)"\s*\)/mx) do 
+      |var_name, class_name|
+      handle_class_module(var_name, "module", class_name, nil, nil)
+    end
+
+    # The '.' lets us handle SWIG-generated files
+    @content.scan(/([\w\.]+)\s* = \s*rb_define_class\s*
+              \(
+                 \s*"(\w+)",
+                 \s*(\w+)\s*
+              \)/mx) do |var_name, class_name, parent|
+      handle_class_module(var_name, "class", class_name, parent, nil)
+    end
+
+    @content.scan(/(\w+)\s*=\s*boot_defclass\s*\(\s*"(\w+?)",\s*(\w+?)\s*\)/) do
+      |var_name, class_name, parent|
+      parent = nil if parent == "0"
+      handle_class_module(var_name, "class", class_name, parent, nil)
+    end
+
+    @content.scan(/(\w+)\s* = \s*rb_define_module_under\s*
+              \(
+                 \s*(\w+),
+                 \s*"(\w+)"
+              \s*\)/mx) do |var_name, in_module, class_name|
+      handle_class_module(var_name, "module", class_name, nil, in_module)
+    end
+
+    @content.scan(/([\w\.]+)\s* = \s*rb_define_class_under\s*
+              \(
+                 \s*(\w+),
+                 \s*"(\w+)",
+                 \s*(\w+)\s*
+              \s*\)/mx) do |var_name, in_module, class_name, parent|
+      handle_class_module(var_name, "class", class_name, parent, in_module)
+    end
+  end
+
+  def do_constants
+    @content.scan(%r{\Wrb_define_
+                   (
+                      variable |
+                      readonly_variable |
+                      const |
+                      global_const |
+                    )
+               \s*\(
+                 (?:\s*(\w+),)?
+                 \s*"(\w+)",
+                 \s*(.*?)\s*\)\s*;
+                 }xm) do |type, var_name, const_name, definition|
+      var_name = "rb_cObject" if !var_name or var_name == "rb_mKernel"
+      handle_constants(type, var_name, const_name, definition)
+    end
+  end
+
+  ##
+  # Look for includes of the form:
+  #
+  #   rb_include_module(rb_cArray, rb_mEnumerable);
+
+  def do_includes
+    @content.scan(/rb_include_module\s*\(\s*(\w+?),\s*(\w+?)\s*\)/) do |c,m|
+      if cls = @classes[c]
+        m = @known_classes[m] || m
+        cls.add_include RDoc::Include.new(m, "")
+      end
+    end
+  end
+
+  def do_methods
+    @content.scan(%r{rb_define_
+                   (
+                      singleton_method |
+                      method           |
+                      module_function  |
+                      private_method
+                   )
+                   \s*\(\s*([\w\.]+),
+                     \s*"([^"]+)",
+                     \s*(?:RUBY_METHOD_FUNC\(|VALUEFUNC\()?(\w+)\)?,
+                     \s*(-?\w+)\s*\)
+                   (?:;\s*/[*/]\s+in\s+(\w+?\.[cy]))?
+                 }xm) do
+      |type, var_name, meth_name, meth_body, param_count, source_file|
+
+      # Ignore top-object and weird struct.c dynamic stuff
+      next if var_name == "ruby_top_self"
+      next if var_name == "nstr"
+      next if var_name == "envtbl"
+      next if var_name == "argf"   # it'd be nice to handle this one
+
+      var_name = "rb_cObject" if var_name == "rb_mKernel"
+      handle_method(type, var_name, meth_name,
+                    meth_body, param_count, source_file)
+    end
+
+    @content.scan(%r{rb_define_attr\(
+                             \s*([\w\.]+),
+                             \s*"([^"]+)",
+                             \s*(\d+),
+                             \s*(\d+)\s*\);
+                }xm) do |var_name, attr_name, attr_reader, attr_writer|
+      #var_name = "rb_cObject" if var_name == "rb_mKernel"
+      handle_attr(var_name, attr_name,
+                  attr_reader.to_i != 0,
+                  attr_writer.to_i != 0)
+    end
+
+    @content.scan(%r{rb_define_global_function\s*\(
+                             \s*"([^"]+)",
+                             \s*(?:RUBY_METHOD_FUNC\(|VALUEFUNC\()?(\w+)\)?,
+                             \s*(-?\w+)\s*\)
+                (?:;\s*/[*/]\s+in\s+(\w+?\.[cy]))?
+                }xm) do |meth_name, meth_body, param_count, source_file|
+      handle_method("method", "rb_mKernel", meth_name,
+                    meth_body, param_count, source_file)
+    end
+
+    @content.scan(/define_filetest_function\s*\(
+                             \s*"([^"]+)",
+                             \s*(?:RUBY_METHOD_FUNC\(|VALUEFUNC\()?(\w+)\)?,
+                             \s*(-?\w+)\s*\)/xm) do
+      |meth_name, meth_body, param_count|
+
+      handle_method("method", "rb_mFileTest", meth_name, meth_body, param_count)
+      handle_method("singleton_method", "rb_cFile", meth_name, meth_body, param_count)
+    end
+ end
+
+  def find_attr_comment(attr_name)
+    if @content =~ %r{((?>/\*.*?\*/\s+))
+                   rb_define_attr\((?:\s*(\w+),)?\s*"#{attr_name}"\s*,.*?\)\s*;}xmi
+      $1
+    elsif @content =~ %r{Document-attr:\s#{attr_name}\s*?\n((?>.*?\*/))}m
+      $1
+    else
+      ''
+    end
+  end
+
+  ##
+  # Find the C code corresponding to a Ruby method
+
+  def find_body(meth_name, meth_obj, body, quiet = false)
+    case body
+    when %r"((?>/\*.*?\*/\s*))(?:static\s+)?VALUE\s+#{meth_name}
+            \s*(\([^)]*\))\s*\{.*?^\}"xm
+      comment, params = $1, $2
+      body_text = $&
+
+      remove_private_comments(comment) if comment
+
+      # see if we can find the whole body
+
+      re = Regexp.escape(body_text) + '[^(]*^\{.*?^\}'
+      if Regexp.new(re, Regexp::MULTILINE).match(body)
+        body_text = $&
+      end
+
+      # The comment block may have been overridden with a 'Document-method'
+      # block. This happens in the interpreter when multiple methods are
+      # vectored through to the same C method but those methods are logically
+      # distinct (for example Kernel.hash and Kernel.object_id share the same
+      # implementation
+
+      override_comment = find_override_comment(meth_obj.name)
+      comment = override_comment if override_comment
+
+      find_modifiers(comment, meth_obj) if comment
+
+#        meth_obj.params = params
+      meth_obj.start_collecting_tokens
+      meth_obj.add_token(RDoc::RubyToken::Token.new(1,1).set_text(body_text))
+      meth_obj.comment = mangle_comment(comment)
+    when %r{((?>/\*.*?\*/\s*))^\s*\#\s*define\s+#{meth_name}\s+(\w+)}m
+      comment = $1
+      find_body($2, meth_obj, body, true)
+      find_modifiers(comment, meth_obj)
+      meth_obj.comment = mangle_comment(comment) + meth_obj.comment
+    when %r{^\s*\#\s*define\s+#{meth_name}\s+(\w+)}m
+      unless find_body($1, meth_obj, body, true)
+        warn "No definition for #{meth_name}" unless @options.quiet
+        return false
+      end
+    else
+
+      # No body, but might still have an override comment
+      comment = find_override_comment(meth_obj.name)
+
+      if comment
+        find_modifiers(comment, meth_obj)
+        meth_obj.comment = mangle_comment(comment)
+      else
+        warn "No definition for #{meth_name}" unless @options.quiet
+        return false
+      end
+    end
+    true
+  end
+
+  def find_class(raw_name, name)
+    unless @classes[raw_name]
+      if raw_name =~ /^rb_m/
+        container = @top_level.add_module RDoc::NormalModule, name
+      else
+        container = @top_level.add_class RDoc::NormalClass, name, nil
+      end
+
+      container.record_location @top_level
+      @classes[raw_name] = container
+    end
+    @classes[raw_name]
+  end
+
+  ##
+  # Look for class or module documentation above Init_+class_name+(void),
+  # in a Document-class +class_name+ (or module) comment or above an
+  # rb_define_class (or module).  If a comment is supplied above a matching
+  # Init_ and a rb_define_class the Init_ comment is used.
+  #
+  #   /*
+  #    * This is a comment for Foo
+  #    */
+  #   Init_Foo(void) {
+  #       VALUE cFoo = rb_define_class("Foo", rb_cObject);
+  #   }
+  #
+  #   /*
+  #    * Document-class: Foo
+  #    * This is a comment for Foo
+  #    */
+  #   Init_foo(void) {
+  #       VALUE cFoo = rb_define_class("Foo", rb_cObject);
+  #   }
+  #
+  #   /*
+  #    * This is a comment for Foo
+  #    */
+  #   VALUE cFoo = rb_define_class("Foo", rb_cObject);
+
+  def find_class_comment(class_name, class_meth)
+    comment = nil
+    if @content =~ %r{((?>/\*.*?\*/\s+))
+                   (static\s+)?void\s+Init_#{class_name}\s*(?:_\(\s*)?\(\s*(?:void\s*)\)}xmi
+      comment = $1
+    elsif @content =~ %r{Document-(class|module):\s#{class_name}\s*?\n((?>.*?\*/))}m
+      comment = $2
+    else
+      if @content =~ /rb_define_(class|module)/m then
+        class_name = class_name.split("::").last
+        comments = []
+        @content.split(/(\/\*.*?\*\/)\s*?\n/m).each_with_index do |chunk, index|
+          comments[index] = chunk
+          if chunk =~ /rb_define_(class|module).*?"(#{class_name})"/m then
+            comment = comments[index-1]
+            break
+          end
+        end
+      end
+    end
+    class_meth.comment = mangle_comment(comment) if comment
+  end
+
+  ##
+  # Finds a comment matching +type+ and +const_name+ either above the
+  # comment or in the matching Document- section.
+
+  def find_const_comment(type, const_name)
+    if @content =~ %r{((?>^\s*/\*.*?\*/\s+))
+                   rb_define_#{type}\((?:\s*(\w+),)?\s*"#{const_name}"\s*,.*?\)\s*;}xmi
+      $1
+    elsif @content =~ %r{Document-(?:const|global|variable):\s#{const_name}\s*?\n((?>.*?\*/))}m
+      $1
+    else
+      ''
+    end
+  end
+
+  ##
+  # If the comment block contains a section that looks like:
+  #
+  #    call-seq:
+  #        Array.new
+  #        Array.new(10)
+  #
+  # use it for the parameters.
+
+  def find_modifiers(comment, meth_obj)
+    if comment.sub!(/:nodoc:\s*^\s*\*?\s*$/m, '') or
+       comment.sub!(/\A\/\*\s*:nodoc:\s*\*\/\Z/, '')
+      meth_obj.document_self = false
+    end
+    if comment.sub!(/call-seq:(.*?)^\s*\*?\s*$/m, '') or
+       comment.sub!(/\A\/\*\s*call-seq:(.*?)\*\/\Z/, '')
+      seq = $1
+      seq.gsub!(/^\s*\*\s*/, '')
+      meth_obj.call_seq = seq
+    end
+  end
+
+  def find_override_comment(meth_name)
+    name = Regexp.escape(meth_name)
+    if @content =~ %r{Document-method:\s#{name}\s*?\n((?>.*?\*/))}m
+      $1
+    end
+  end
+
+  def handle_attr(var_name, attr_name, reader, writer)
+    rw = ''
+    if reader
+      #@stats.num_methods += 1
+      rw << 'R'
+    end
+    if writer
+      #@stats.num_methods += 1
+      rw << 'W'
+    end
+
+    class_name = @known_classes[var_name]
+
+    return unless class_name
+
+    class_obj  = find_class(var_name, class_name)
+
+    if class_obj
+      comment = find_attr_comment(attr_name)
+      unless comment.empty?
+        comment = mangle_comment(comment)
+      end
+      att = RDoc::Attr.new '', attr_name, rw, comment
+      class_obj.add_attribute(att)
+    end
+  end
+
+  def handle_class_module(var_name, class_mod, class_name, parent, in_module)
+    parent_name = @known_classes[parent] || parent
+
+    if in_module
+      enclosure = @classes[in_module] || @@enclosure_classes[in_module]
+      unless enclosure
+        if enclosure = @known_classes[in_module]
+          handle_class_module(in_module, (/^rb_m/ =~ in_module ? "module" : "class"),
+                              enclosure, nil, nil)
+          enclosure = @classes[in_module]
+        end
+      end
+      unless enclosure
+        warn("Enclosing class/module '#{in_module}' for " +
+              "#{class_mod} #{class_name} not known")
+        return
+      end
+    else
+      enclosure = @top_level
+    end
+
+    if class_mod == "class" then
+      cm = enclosure.add_class RDoc::NormalClass, class_name, parent_name
+      @stats.add_class cm
+    else
+      cm = enclosure.add_module RDoc::NormalModule, class_name
+      @stats.add_module cm
+    end
+
+    cm.record_location(enclosure.toplevel)
+
+    find_class_comment(cm.full_name, cm)
+    @classes[var_name] = cm
+    @@enclosure_classes[var_name] = cm
+    @known_classes[var_name] = cm.full_name
+  end
+
+  ##
+  # Adds constant comments.  By providing some_value: at the start ofthe
+  # comment you can override the C value of the comment to give a friendly
+  # definition.
+  #
+  #   /* 300: The perfect score in bowling */
+  #   rb_define_const(cFoo, "PERFECT", INT2FIX(300);
+  #
+  # Will override +INT2FIX(300)+ with the value +300+ in the output RDoc.
+  # Values may include quotes and escaped colons (\:).
+
+  def handle_constants(type, var_name, const_name, definition)
+    #@stats.num_constants += 1
+    class_name = @known_classes[var_name]
+
+    return unless class_name
+
+    class_obj  = find_class(var_name, class_name)
+
+    unless class_obj
+      warn("Enclosing class/module '#{const_name}' for not known")
+      return
+    end
+
+    comment = find_const_comment(type, const_name)
+
+    # In the case of rb_define_const, the definition and comment are in
+    # "/* definition: comment */" form.  The literal ':' and '\' characters
+    # can be escaped with a backslash.
+    if type.downcase == 'const' then
+       elements = mangle_comment(comment).split(':')
+       if elements.nil? or elements.empty? then
+          con = RDoc::Constant.new(const_name, definition,
+                                   mangle_comment(comment))
+       else
+          new_definition = elements[0..-2].join(':')
+          if new_definition.empty? then # Default to literal C definition
+             new_definition = definition
+          else
+             new_definition.gsub!("\:", ":")
+             new_definition.gsub!("\\", '\\')
+          end
+          new_definition.sub!(/\A(\s+)/, '')
+          new_comment = $1.nil? ? elements.last : "#{$1}#{elements.last.lstrip}"
+          con = RDoc::Constant.new(const_name, new_definition,
+                                   mangle_comment(new_comment))
+       end
+    else
+       con = RDoc::Constant.new const_name, definition, mangle_comment(comment)
+    end
+
+    class_obj.add_constant(con)
+  end
+
+  ##
+  # Removes #ifdefs that would otherwise confuse us
+
+  def handle_ifdefs_in(body)
+    body.gsub(/^#ifdef HAVE_PROTOTYPES.*?#else.*?\n(.*?)#endif.*?\n/m, '\1')
+  end
+
+  def handle_method(type, var_name, meth_name, meth_body, param_count,
+                    source_file = nil)
+    class_name = @known_classes[var_name]
+
+    return unless class_name
+
+    class_obj  = find_class(var_name, class_name)
+
+    if class_obj
+      if meth_name == "initialize"
+        meth_name = "new"
+        type = "singleton_method"
+      end
+      meth_obj = RDoc::AnyMethod.new("", meth_name)
+      meth_obj.singleton =
+  %w{singleton_method module_function}.include?(type)
+
+      @stats.add_method meth_obj
+
+      p_count = (Integer(param_count) rescue -1)
+
+      if p_count < 0
+        meth_obj.params = "(...)"
+      elsif p_count == 0
+        meth_obj.params = "()"
+      else
+        meth_obj.params = "(" + (1..p_count).map{|i| "p#{i}"}.join(", ") + ")"
+      end
+
+      if source_file
+        file_name = File.join(@file_dir, source_file)
+        body = (@@known_bodies[source_file] ||= File.read(file_name))
+      else
+        body = @content
+      end
+      if find_body(meth_body, meth_obj, body) and meth_obj.document_self
+        class_obj.add_method(meth_obj)
+      end
+    end
+  end
+
+  def handle_tab_width(body)
+    if /\t/ =~ body
+      tab_width = @options.tab_width
+      body.split(/\n/).map do |line|
+        1 while line.gsub!(/\t+/) { ' ' * (tab_width*$&.length - $`.length % tab_width)}  && $~ #`
+        line
+      end .join("\n")
+    else
+      body
+    end
+  end
+
+  ##
+  # Remove the /*'s and leading asterisks from C comments
+
+  def mangle_comment(comment)
+    comment.sub!(%r{/\*+}) { " " * $&.length }
+    comment.sub!(%r{\*+/}) { " " * $&.length }
+    comment.gsub!(/^[ \t]*\*/m) { " " * $&.length }
+    comment
+  end
+
+  ##
+  # Removes lines that are commented out that might otherwise get picked up
+  # when scanning for classes and methods
+
+  def remove_commented_out_lines
+    @content.gsub!(%r{//.*rb_define_}, '//')
+  end
+
+  def remove_private_comments(comment)
+     comment.gsub!(/\/?\*--(.*?)\/?\*\+\+/m, '')
+     comment.sub!(/\/?\*--.*/m, '')
+  end
+
+  ##
+  # Extract the classes/modules and methods from a C file and return the
+  # corresponding top-level object
+
+  def scan
+    remove_commented_out_lines
+    do_classes
+    do_constants
+    do_methods
+    do_includes
+    do_aliases
+    @top_level
+  end
+
+  def warn(msg)
+    $stderr.puts
+    $stderr.puts msg
+    $stderr.flush
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/parser/f95.rb b/trunk/lib/rdoc/parser/f95.rb
new file mode 100644
index 0000000000..fd372b098b
--- /dev/null
+++ b/trunk/lib/rdoc/parser/f95.rb
@@ -0,0 +1,1835 @@
+require 'rdoc/parser'
+
+##
+# = Fortran95 RDoc Parser
+#
+# == Overview
+#
+# This parser parses Fortran95 files with suffixes "f90", "F90", "f95" and
+# "F95". Fortran95 files are expected to be conformed to Fortran95 standards.
+#
+# == Rules
+#
+# Fundamental rules are same as that of the Ruby parser.  But comment markers
+# are '!' not '#'.
+#
+# === Correspondence between RDoc documentation and Fortran95 programs
+#
+# F95 parses main programs, modules, subroutines, functions, derived-types,
+# public variables, public constants, defined operators and defined
+# assignments.  These components are described in items of RDoc documentation,
+# as follows.
+#
+# Files :: Files (same as Ruby)
+# Classes:: Modules
+# Methods:: Subroutines, functions, variables, constants, derived-types,
+#           defined operators, defined assignments
+# Required files:: Files in which imported modules, external subroutines and
+#                  external functions are defined.
+# Included Modules:: List of imported modules
+# Attributes:: List of derived-types, List of imported modules all of whose
+#              components are published again
+#
+# Components listed in 'Methods' (subroutines, functions, ...) defined in
+# modules are described in the item of 'Classes'.  On the other hand,
+# components defined in main programs or as external procedures are described
+# in the item of 'Files'.
+#
+# === Components parsed by default
+#
+# By default, documentation on public components (subroutines, functions,
+# variables, constants, derived-types, defined operators, defined assignments)
+# are generated.
+#
+# With "--all" option, documentation on all components are generated (almost
+# same as the Ruby parser).
+#
+# === Information parsed automatically
+#
+# The following information is automatically parsed.
+#
+# * Types of arguments
+# * Types of variables and constants
+# * Types of variables in the derived types, and initial values
+# * NAMELISTs and types of variables in them, and initial values
+#
+# Aliases by interface statement are described in the item of 'Methods'.
+#
+# Components which are imported from other modules and published again are
+# described in the item of 'Methods'.
+#
+# === Format of comment blocks
+#
+# Comment blocks should be written as follows.
+#
+# Comment blocks are considered to be ended when the line without '!' appears.
+#
+# The indentation is not necessary.
+#
+#   ! (Top of file)
+#   !
+#   ! Comment blocks for the files.
+#   !
+#   !--
+#   ! The comment described in the part enclosed by
+#   ! "!--" and "!++" is ignored.
+#   !++
+#   !
+#   module hogehoge
+#     !
+#     ! Comment blocks for the modules (or the programs).
+#     !
+#   
+#     private
+#   
+#     logical            :: a     ! a private variable
+#     real, public       :: b     ! a public variable
+#     integer, parameter :: c = 0 ! a public constant
+#   
+#     public :: c
+#     public :: MULTI_ARRAY
+#     public :: hoge, foo
+#   
+#     type MULTI_ARRAY
+#       !
+#       ! Comment blocks for the derived-types.
+#       !
+#       real, pointer :: var(:) =>null() ! Comments block for the variables.
+#       integer       :: num = 0
+#     end type MULTI_ARRAY
+#   
+#   contains
+#   
+#     subroutine hoge( in,   &   ! Comment blocks between continuation lines are ignored.
+#         &            out )
+#       !
+#       ! Comment blocks for the subroutines or functions
+#       !
+#       character(*),intent(in):: in ! Comment blocks for the arguments.
+#       character(*),intent(out),allocatable,target  :: in
+#                                    ! Comment blocks can be
+#                                    ! written under Fortran statements.
+#   
+#       character(32) :: file ! This comment parsed as a variable in below NAMELIST.
+#       integer       :: id
+#   
+#       namelist /varinfo_nml/ file, id
+#               !
+#               ! Comment blocks for the NAMELISTs.
+#               ! Information about variables are described above.
+#               !
+#   
+#     ....
+#   
+#     end subroutine hoge
+#   
+#     integer function foo( in )
+#       !
+#       ! This part is considered as comment block.
+#   
+#       ! Comment blocks under blank lines are ignored.
+#       !
+#       integer, intent(in):: inA ! This part is considered as comment block.
+#   
+#                                 ! This part is ignored.
+#   
+#     end function foo
+#   
+#     subroutine hide( in,   &
+#       &              out )      !:nodoc:
+#       !
+#       ! If "!:nodoc:" is described at end-of-line in subroutine
+#       ! statement as above, the subroutine is ignored.
+#       ! This assignment can be used to modules, subroutines,
+#       ! functions, variables, constants, derived-types,
+#       ! defined operators, defined assignments,
+#       ! list of imported modules ("use" statement).
+#       !
+#   
+#     ....
+#   
+#     end subroutine hide
+#   
+#   end module hogehoge
+
+class RDoc::Parser::F95 < RDoc::Parser
+
+  parse_files_matching(/\.((f|F)9(0|5)|F)$/)
+
+  class Token
+
+    NO_TEXT = "??".freeze
+
+    def initialize(line_no, char_no)
+      @line_no = line_no
+      @char_no = char_no
+      @text    = NO_TEXT
+    end
+    # Because we're used in contexts that expect to return a token,
+    # we set the text string and then return ourselves
+    def set_text(text)
+      @text = text
+      self
+    end
+
+    attr_reader :line_no, :char_no, :text
+
+  end
+
+  @@external_aliases = []
+  @@public_methods   = []
+
+  ##
+  # "false":: Comments are below source code
+  # "true" :: Comments are upper source code
+
+  COMMENTS_ARE_UPPER  = false
+
+  ##
+  # Internal alias message
+
+  INTERNAL_ALIAS_MES = "Alias for"
+
+  ##
+  # External alias message
+
+  EXTERNAL_ALIAS_MES = "The entity is"
+
+  ##
+  # Define code constructs
+
+  def scan
+    # remove private comment
+    remaining_code = remove_private_comments(@content)
+
+    # continuation lines are united to one line
+    remaining_code = united_to_one_line(remaining_code)
+
+    # semicolons are replaced to line feed
+    remaining_code = semicolon_to_linefeed(remaining_code)
+
+    # collect comment for file entity
+    whole_comment, remaining_code = collect_first_comment(remaining_code)
+    @top_level.comment = whole_comment
+
+    # String "remaining_code" is converted to Array "remaining_lines"
+    remaining_lines = remaining_code.split("\n")
+
+    # "module" or "program" parts are parsed (new)
+    #
+    level_depth = 0
+    block_searching_flag = nil
+    block_searching_lines = []
+    pre_comment = []
+    module_program_trailing = ""
+    module_program_name = ""
+    other_block_level_depth = 0
+    other_block_searching_flag = nil
+    remaining_lines.collect!{|line|
+      if !block_searching_flag && !other_block_searching_flag
+        if line =~ /^\s*?module\s+(\w+)\s*?(!.*?)?$/i
+          block_searching_flag = :module
+          block_searching_lines << line
+          module_program_name = $1
+          module_program_trailing = find_comments($2)
+          next false
+        elsif line =~ /^\s*?program\s+(\w+)\s*?(!.*?)?$/i ||
+               line =~ /^\s*?\w/ && !block_start?(line)
+          block_searching_flag = :program
+          block_searching_lines << line
+          module_program_name = $1 || ""
+          module_program_trailing = find_comments($2)
+          next false
+
+        elsif block_start?(line)
+          other_block_searching_flag = true
+          next line
+
+        elsif line =~ /^\s*?!\s?(.*)/
+          pre_comment << line
+          next line
+        else
+          pre_comment = []
+          next line
+        end
+      elsif other_block_searching_flag
+        other_block_level_depth += 1 if block_start?(line)
+        other_block_level_depth -= 1 if block_end?(line)
+        if other_block_level_depth < 0
+          other_block_level_depth = 0
+          other_block_searching_flag = nil
+        end
+        next line
+      end
+
+      block_searching_lines << line
+      level_depth += 1 if block_start?(line)
+      level_depth -= 1 if block_end?(line)
+      if level_depth >= 0
+        next false
+      end
+
+      # "module_program_code" is formatted.
+      # ":nodoc:" flag is checked.
+      #
+      module_program_code = block_searching_lines.join("\n")
+      module_program_code = remove_empty_head_lines(module_program_code)
+      if module_program_trailing =~ /^:nodoc:/
+        # next loop to search next block
+        level_depth = 0
+        block_searching_flag = false
+        block_searching_lines = []
+        pre_comment = []
+        next false
+      end
+
+      # NormalClass is created, and added to @top_level
+      #
+      if block_searching_flag == :module
+        module_name = module_program_name
+        module_code = module_program_code
+        module_trailing = module_program_trailing
+
+        f9x_module = @top_level.add_module NormalClass, module_name
+        f9x_module.record_location @top_level
+
+        @stats.add_module f9x_module
+
+        f9x_comment = COMMENTS_ARE_UPPER ? 
+          find_comments(pre_comment.join("\n"))  + "\n" + module_trailing :
+            module_trailing + "\n" + find_comments(module_code.sub(/^.*$\n/i, ''))
+        f9x_module.comment = f9x_comment
+        parse_program_or_module(f9x_module, module_code)
+
+        TopLevel.all_files.each do |name, toplevel|
+          if toplevel.include_includes?(module_name, @options.ignore_case)
+            if !toplevel.include_requires?(@file_name, @options.ignore_case)
+              toplevel.add_require(Require.new(@file_name, ""))
+            end
+          end
+          toplevel.each_classmodule{|m|
+            if m.include_includes?(module_name, @options.ignore_case)
+              if !m.include_requires?(@file_name, @options.ignore_case)
+                m.add_require(Require.new(@file_name, ""))
+              end
+            end
+          }
+        end
+      elsif block_searching_flag == :program
+        program_name = module_program_name
+        program_code = module_program_code
+        program_trailing = module_program_trailing
+        # progress "p" # HACK what stats thingy does this correspond to?
+        program_comment = COMMENTS_ARE_UPPER ? 
+          find_comments(pre_comment.join("\n")) + "\n" + program_trailing : 
+            program_trailing + "\n" + find_comments(program_code.sub(/^.*$\n/i, ''))
+        program_comment = "\n\n= <i>Program</i> <tt>#{program_name}</tt>\n\n" \
+                          + program_comment
+        @top_level.comment << program_comment
+        parse_program_or_module(@top_level, program_code, :private)
+      end
+
+      # next loop to search next block
+      level_depth = 0
+      block_searching_flag = false
+      block_searching_lines = []
+      pre_comment = []
+      next false
+    }
+
+    remaining_lines.delete_if{ |line|
+      line == false
+    }
+
+    # External subprograms and functions are parsed
+    #
+    parse_program_or_module(@top_level, remaining_lines.join("\n"),
+                            :public, true)
+
+    @top_level
+  end  # End of scan
+
+  private
+
+  def parse_program_or_module(container, code,
+                              visibility=:public, external=nil)
+    return unless container
+    return unless code
+    remaining_lines = code.split("\n")
+    remaining_code = "#{code}"
+
+    #
+    # Parse variables before "contains" in module
+    #
+    level_depth = 0
+    before_contains_lines = []
+    before_contains_code = nil
+    before_contains_flag = nil
+    remaining_lines.each{ |line|
+      if !before_contains_flag
+        if line =~ /^\s*?module\s+\w+\s*?(!.*?)?$/i
+          before_contains_flag = true
+        end
+      else
+        break if line =~ /^\s*?contains\s*?(!.*?)?$/i
+        level_depth += 1 if block_start?(line)
+        level_depth -= 1 if block_end?(line)
+        break if level_depth < 0
+        before_contains_lines << line
+      end
+    }
+    before_contains_code = before_contains_lines.join("\n")
+    if before_contains_code
+      before_contains_code.gsub!(/^\s*?interface\s+.*?\s+end\s+interface.*?$/im, "")
+      before_contains_code.gsub!(/^\s*?type[\s\,]+.*?\s+end\s+type.*?$/im, "")
+    end
+
+    #
+    # Parse global "use"
+    #
+    use_check_code = "#{before_contains_code}"
+    cascaded_modules_list = []
+    while use_check_code =~ /^\s*?use\s+(\w+)(.*?)(!.*?)?$/i
+      use_check_code = $~.pre_match
+      use_check_code << $~.post_match
+      used_mod_name = $1.strip.chomp
+      used_list = $2 || ""
+      used_trailing = $3 || ""
+      next if used_trailing =~ /!:nodoc:/
+      if !container.include_includes?(used_mod_name, @options.ignore_case)
+        # progress "." # HACK what stats thingy does this correspond to?
+        container.add_include Include.new(used_mod_name, "")
+      end
+      if ! (used_list =~ /\,\s*?only\s*?:/i )
+        cascaded_modules_list << "\#" + used_mod_name
+      end
+    end
+
+    #
+    # Parse public and private, and store information.
+    # This information is used when "add_method" and
+    # "set_visibility_for" are called.
+    #
+    visibility_default, visibility_info = 
+              parse_visibility(remaining_lines.join("\n"), visibility, container)
+    @@public_methods.concat visibility_info
+    if visibility_default == :public
+      if !cascaded_modules_list.empty?
+        cascaded_modules = 
+          Attr.new("Cascaded Modules",
+                   "Imported modules all of whose components are published again",
+                   "",
+                   cascaded_modules_list.join(", "))
+        container.add_attribute(cascaded_modules)
+      end
+    end
+
+    #
+    # Check rename elements
+    #
+    use_check_code = "#{before_contains_code}"
+    while use_check_code =~ /^\s*?use\s+(\w+)\s*?\,(.+)$/i
+      use_check_code = $~.pre_match
+      use_check_code << $~.post_match
+      used_mod_name = $1.strip.chomp
+      used_elements = $2.sub(/\s*?only\s*?:\s*?/i, '')
+      used_elements.split(",").each{ |used|
+        if /\s*?(\w+)\s*?=>\s*?(\w+)\s*?/ =~ used
+          local = $1
+          org = $2
+          @@public_methods.collect!{ |pub_meth|
+            if local == pub_meth["name"] ||
+                local.upcase == pub_meth["name"].upcase &&
+                @options.ignore_case
+              pub_meth["name"] = org
+              pub_meth["local_name"] = local
+            end
+            pub_meth
+          }
+        end
+      }
+    end
+
+    #
+    # Parse private "use"
+    #
+    use_check_code = remaining_lines.join("\n")
+    while use_check_code =~ /^\s*?use\s+(\w+)(.*?)(!.*?)?$/i
+      use_check_code = $~.pre_match
+      use_check_code << $~.post_match
+      used_mod_name = $1.strip.chomp
+      used_trailing = $3 || ""
+      next if used_trailing =~ /!:nodoc:/
+      if !container.include_includes?(used_mod_name, @options.ignore_case)
+        # progress "." # HACK what stats thingy does this correspond to?
+        container.add_include Include.new(used_mod_name, "")
+      end
+    end
+
+    container.each_includes{ |inc|
+      TopLevel.all_files.each do |name, toplevel|
+        indicated_mod = toplevel.find_symbol(inc.name,
+                                             nil, @options.ignore_case)
+        if indicated_mod
+          indicated_name = indicated_mod.parent.file_relative_name
+          if !container.include_requires?(indicated_name, @options.ignore_case)
+            container.add_require(Require.new(indicated_name, ""))
+          end
+          break
+        end
+      end
+    }
+
+    #
+    # Parse derived-types definitions
+    #
+    derived_types_comment = ""
+    remaining_code = remaining_lines.join("\n")
+    while remaining_code =~ /^\s*?
+                                  type[\s\,]+(public|private)?\s*?(::)?\s*?
+                                  (\w+)\s*?(!.*?)?$
+                                  (.*?)
+                                  ^\s*?end\s+type.*?$
+                            /imx
+      remaining_code = $~.pre_match
+      remaining_code << $~.post_match
+      typename = $3.chomp.strip
+      type_elements = $5 || ""
+      type_code = remove_empty_head_lines($&)
+      type_trailing = find_comments($4)
+      next if type_trailing =~ /^:nodoc:/
+      type_visibility = $1
+      type_comment = COMMENTS_ARE_UPPER ? 
+        find_comments($~.pre_match) + "\n" + type_trailing :
+          type_trailing + "\n" + find_comments(type_code.sub(/^.*$\n/i, ''))
+      type_element_visibility_public = true
+      type_code.split("\n").each{ |line|
+        if /^\s*?private\s*?$/ =~ line
+          type_element_visibility_public = nil
+          break
+        end
+      } if type_code
+
+      args_comment = ""
+      type_args_info = nil
+
+      if @options.show_all
+        args_comment = find_arguments(nil, type_code, true)
+      else
+        type_public_args_list = []
+        type_args_info = definition_info(type_code)
+        type_args_info.each{ |arg|
+          arg_is_public = type_element_visibility_public
+          arg_is_public = true if arg.include_attr?("public")
+          arg_is_public = nil if arg.include_attr?("private")
+          type_public_args_list << arg.varname if arg_is_public
+        }
+        args_comment = find_arguments(type_public_args_list, type_code)
+      end
+
+      type = AnyMethod.new("type #{typename}", typename)
+      type.singleton = false
+      type.params = ""
+      type.comment = "<b><em> Derived Type </em></b> :: <tt></tt>\n"
+      type.comment << args_comment if args_comment
+      type.comment << type_comment if type_comment
+
+      @stats.add_method type
+
+      container.add_method type
+
+      set_visibility(container, typename, visibility_default, @@public_methods)
+
+      if type_visibility
+        type_visibility.gsub!(/\s/,'')
+        type_visibility.gsub!(/\,/,'')
+        type_visibility.gsub!(/:/,'')
+        type_visibility.downcase!
+        if type_visibility == "public"
+          container.set_visibility_for([typename], :public)
+        elsif type_visibility == "private"
+          container.set_visibility_for([typename], :private)
+        end
+      end
+
+      check_public_methods(type, container.name)
+
+      if @options.show_all
+        derived_types_comment << ", " unless derived_types_comment.empty?
+        derived_types_comment << typename
+      else
+        if type.visibility == :public
+        derived_types_comment << ", " unless derived_types_comment.empty?
+        derived_types_comment << typename
+        end
+      end
+
+    end
+
+    if !derived_types_comment.empty?
+      derived_types_table = 
+        Attr.new("Derived Types", "Derived_Types", "", 
+                 derived_types_comment)
+      container.add_attribute(derived_types_table)
+    end
+
+    #
+    # move interface scope
+    #
+    interface_code = ""
+    while remaining_code =~ /^\s*?
+                                 interface(
+                                            \s+\w+                      |
+                                            \s+operator\s*?\(.*?\)       |
+                                            \s+assignment\s*?\(\s*?=\s*?\)
+                                          )?\s*?$
+                                 (.*?)
+                                 ^\s*?end\s+interface.*?$
+                            /imx
+      interface_code << remove_empty_head_lines($&) + "\n"
+      remaining_code = $~.pre_match
+      remaining_code << $~.post_match
+    end
+
+    #
+    # Parse global constants or variables in modules
+    #
+    const_var_defs = definition_info(before_contains_code)
+    const_var_defs.each{|defitem|
+      next if defitem.nodoc
+      const_or_var_type = "Variable"
+      const_or_var_progress = "v"
+      if defitem.include_attr?("parameter")
+        const_or_var_type = "Constant"
+        const_or_var_progress = "c"
+      end
+      const_or_var = AnyMethod.new(const_or_var_type, defitem.varname)
+      const_or_var.singleton = false
+      const_or_var.params = ""
+      self_comment = find_arguments([defitem.varname], before_contains_code)
+      const_or_var.comment = "<b><em>" + const_or_var_type + "</em></b> :: <tt></tt>\n"
+      const_or_var.comment << self_comment if self_comment
+
+      @stats.add_method const_or_var_progress
+
+      container.add_method const_or_var
+
+      set_visibility(container, defitem.varname, visibility_default, @@public_methods)
+
+      if defitem.include_attr?("public")
+        container.set_visibility_for([defitem.varname], :public)
+      elsif defitem.include_attr?("private")
+        container.set_visibility_for([defitem.varname], :private)
+      end
+
+      check_public_methods(const_or_var, container.name)
+
+    } if const_var_defs
+
+    remaining_lines = remaining_code.split("\n")
+
+    # "subroutine" or "function" parts are parsed (new)
+    #
+    level_depth = 0
+    block_searching_flag = nil
+    block_searching_lines = []
+    pre_comment = []
+    procedure_trailing = ""
+    procedure_name = ""
+    procedure_params = ""
+    procedure_prefix = ""
+    procedure_result_arg = ""
+    procedure_type = ""
+    contains_lines = []
+    contains_flag = nil
+    remaining_lines.collect!{|line|
+      if !block_searching_flag
+        # subroutine
+        if line =~ /^\s*?
+                         (recursive|pure|elemental)?\s*?
+                         subroutine\s+(\w+)\s*?(\(.*?\))?\s*?(!.*?)?$
+                   /ix
+          block_searching_flag = :subroutine
+          block_searching_lines << line
+
+          procedure_name = $2.chomp.strip
+          procedure_params = $3 || ""
+          procedure_prefix = $1 || ""
+          procedure_trailing = $4 || "!"
+          next false
+
+        # function
+        elsif line =~ /^\s*?
+                       (recursive|pure|elemental)?\s*?
+                       (
+                           character\s*?(\([\w\s\=\(\)\*]+?\))?\s+
+                         | type\s*?\([\w\s]+?\)\s+
+                         | integer\s*?(\([\w\s\=\(\)\*]+?\))?\s+
+                         | real\s*?(\([\w\s\=\(\)\*]+?\))?\s+
+                         | double\s+precision\s+
+                         | logical\s*?(\([\w\s\=\(\)\*]+?\))?\s+
+                         | complex\s*?(\([\w\s\=\(\)\*]+?\))?\s+
+                       )?
+                       function\s+(\w+)\s*?
+                       (\(.*?\))?(\s+result\((.*?)\))?\s*?(!.*?)?$
+                      /ix
+          block_searching_flag = :function
+          block_searching_lines << line
+
+          procedure_prefix = $1 || ""
+          procedure_type = $2 ? $2.chomp.strip : nil
+          procedure_name = $8.chomp.strip
+          procedure_params = $9 || ""
+          procedure_result_arg = $11 ? $11.chomp.strip : procedure_name
+          procedure_trailing = $12 || "!"
+          next false
+        elsif line =~ /^\s*?!\s?(.*)/
+          pre_comment << line
+          next line
+        else
+          pre_comment = []
+          next line
+        end
+      end
+      contains_flag = true if line =~ /^\s*?contains\s*?(!.*?)?$/
+      block_searching_lines << line
+      contains_lines << line if contains_flag
+
+      level_depth += 1 if block_start?(line)
+      level_depth -= 1 if block_end?(line)
+      if level_depth >= 0
+        next false
+      end
+
+      # "procedure_code" is formatted.
+      # ":nodoc:" flag is checked.
+      #
+      procedure_code = block_searching_lines.join("\n")
+      procedure_code = remove_empty_head_lines(procedure_code)
+      if procedure_trailing =~ /^!:nodoc:/
+        # next loop to search next block
+        level_depth = 0
+        block_searching_flag = nil
+        block_searching_lines = []
+        pre_comment = []
+        procedure_trailing = ""
+        procedure_name = ""
+        procedure_params = ""
+        procedure_prefix = ""
+        procedure_result_arg = ""
+        procedure_type = ""
+        contains_lines = []
+        contains_flag = nil
+        next false
+      end
+
+      # AnyMethod is created, and added to container
+      #
+      subroutine_function = nil
+      if block_searching_flag == :subroutine
+        subroutine_prefix   = procedure_prefix
+        subroutine_name     = procedure_name
+        subroutine_params   = procedure_params
+        subroutine_trailing = procedure_trailing
+        subroutine_code     = procedure_code
+
+        subroutine_comment = COMMENTS_ARE_UPPER ? 
+          pre_comment.join("\n") + "\n" + subroutine_trailing : 
+            subroutine_trailing + "\n" + subroutine_code.sub(/^.*$\n/i, '')
+        subroutine = AnyMethod.new("subroutine", subroutine_name)
+        parse_subprogram(subroutine, subroutine_params,
+                         subroutine_comment, subroutine_code,
+                         before_contains_code, nil, subroutine_prefix)
+
+        @stats.add_method subroutine
+
+        container.add_method subroutine
+        subroutine_function = subroutine
+
+      elsif block_searching_flag == :function
+        function_prefix     = procedure_prefix
+        function_type       = procedure_type
+        function_name       = procedure_name
+        function_params_org = procedure_params
+        function_result_arg = procedure_result_arg
+        function_trailing   = procedure_trailing
+        function_code_org   = procedure_code
+
+        function_comment = COMMENTS_ARE_UPPER ?
+          pre_comment.join("\n") + "\n" + function_trailing :
+            function_trailing + "\n " + function_code_org.sub(/^.*$\n/i, '')
+
+        function_code = "#{function_code_org}"
+        if function_type
+          function_code << "\n" + function_type + " :: " + function_result_arg
+        end
+
+        function_params =
+          function_params_org.sub(/^\(/, "\(#{function_result_arg}, ")
+
+        function = AnyMethod.new("function", function_name)
+        parse_subprogram(function, function_params,
+                         function_comment, function_code,
+                         before_contains_code, true, function_prefix)
+
+        # Specific modification due to function
+        function.params.sub!(/\(\s*?#{function_result_arg}\s*?,\s*?/, "\( ")
+        function.params << " result(" + function_result_arg + ")"
+        function.start_collecting_tokens
+        function.add_token Token.new(1,1).set_text(function_code_org)
+
+        @stats.add_method function
+
+        container.add_method function
+        subroutine_function = function
+
+      end
+
+      # The visibility of procedure is specified
+      #
+      set_visibility(container, procedure_name, 
+                     visibility_default, @@public_methods)
+
+      # The alias for this procedure from external modules
+      #
+      check_external_aliases(procedure_name,
+                             subroutine_function.params,
+                             subroutine_function.comment, subroutine_function) if external
+      check_public_methods(subroutine_function, container.name)
+
+
+      # contains_lines are parsed as private procedures
+      if contains_flag
+        parse_program_or_module(container,
+                                contains_lines.join("\n"), :private)
+      end
+
+      # next loop to search next block
+      level_depth = 0
+      block_searching_flag = nil
+      block_searching_lines = []
+      pre_comment = []
+      procedure_trailing = ""
+      procedure_name = ""
+      procedure_params = ""
+      procedure_prefix = ""
+      procedure_result_arg = ""
+      contains_lines = []
+      contains_flag = nil
+      next false
+    } # End of remaining_lines.collect!{|line|
+
+    # Array remains_lines is converted to String remains_code again
+    #
+    remaining_code = remaining_lines.join("\n")
+
+    #
+    # Parse interface
+    #
+    interface_scope = false
+    generic_name = ""
+    interface_code.split("\n").each{ |line|
+      if /^\s*?
+               interface(
+                          \s+\w+|
+                          \s+operator\s*?\(.*?\)|
+                          \s+assignment\s*?\(\s*?=\s*?\)
+                        )?
+               \s*?(!.*?)?$
+         /ix =~ line
+        generic_name = $1 ? $1.strip.chomp : nil
+        interface_trailing = $2 || "!"
+        interface_scope = true
+        interface_scope = false if interface_trailing =~ /!:nodoc:/
+#        if generic_name =~ /operator\s*?\((.*?)\)/i
+#          operator_name = $1
+#          if operator_name && !operator_name.empty?
+#            generic_name = "#{operator_name}"
+#          end
+#        end
+#        if generic_name =~ /assignment\s*?\((.*?)\)/i
+#          assignment_name = $1
+#          if assignment_name && !assignment_name.empty?
+#            generic_name = "#{assignment_name}"
+#          end
+#        end
+      end
+      if /^\s*?end\s+interface/i =~ line
+        interface_scope = false
+        generic_name = nil
+      end
+      # internal alias
+      if interface_scope && /^\s*?module\s+procedure\s+(.*?)(!.*?)?$/i =~ line
+        procedures = $1.strip.chomp
+        procedures_trailing = $2 || "!"
+        next if procedures_trailing =~ /!:nodoc:/
+        procedures.split(",").each{ |proc|
+          proc.strip!
+          proc.chomp!
+          next if generic_name == proc || !generic_name
+          old_meth = container.find_symbol(proc, nil, @options.ignore_case)
+          next if !old_meth
+          nolink = old_meth.visibility == :private ? true : nil
+          nolink = nil if @options.show_all
+          new_meth = 
+             initialize_external_method(generic_name, proc, 
+                                        old_meth.params, nil, 
+                                        old_meth.comment, 
+                                        old_meth.clone.token_stream[0].text, 
+                                        true, nolink)
+          new_meth.singleton = old_meth.singleton
+
+          @stats.add_method new_meth
+
+          container.add_method new_meth
+
+          set_visibility(container, generic_name, visibility_default, @@public_methods)
+
+          check_public_methods(new_meth, container.name)
+
+        }
+      end
+
+      # external aliases
+      if interface_scope
+        # subroutine
+        proc = nil
+        params = nil
+        procedures_trailing = nil
+        if line =~ /^\s*?
+                         (recursive|pure|elemental)?\s*?
+                         subroutine\s+(\w+)\s*?(\(.*?\))?\s*?(!.*?)?$
+                   /ix
+          proc = $2.chomp.strip
+          generic_name = proc unless generic_name
+          params = $3 || ""
+          procedures_trailing = $4 || "!"
+
+        # function
+        elsif line =~ /^\s*?
+                       (recursive|pure|elemental)?\s*?
+                       (
+                           character\s*?(\([\w\s\=\(\)\*]+?\))?\s+
+                         | type\s*?\([\w\s]+?\)\s+
+                         | integer\s*?(\([\w\s\=\(\)\*]+?\))?\s+
+                         | real\s*?(\([\w\s\=\(\)\*]+?\))?\s+
+                         | double\s+precision\s+
+                         | logical\s*?(\([\w\s\=\(\)\*]+?\))?\s+
+                         | complex\s*?(\([\w\s\=\(\)\*]+?\))?\s+
+                       )?
+                       function\s+(\w+)\s*?
+                       (\(.*?\))?(\s+result\((.*?)\))?\s*?(!.*?)?$
+                      /ix
+          proc = $8.chomp.strip
+          generic_name = proc unless generic_name
+          params = $9 || ""
+          procedures_trailing = $12 || "!"
+        else
+          next
+        end
+        next if procedures_trailing =~ /!:nodoc:/
+        indicated_method = nil
+        indicated_file   = nil
+        TopLevel.all_files.each do |name, toplevel|
+          indicated_method = toplevel.find_local_symbol(proc, @options.ignore_case)
+          indicated_file = name
+          break if indicated_method
+        end
+
+        if indicated_method
+          external_method = 
+            initialize_external_method(generic_name, proc, 
+                                       indicated_method.params, 
+                                       indicated_file, 
+                                       indicated_method.comment)
+
+          @stats.add_method external_method
+
+          container.add_method external_method
+          set_visibility(container, generic_name, visibility_default, @@public_methods)
+          if !container.include_requires?(indicated_file, @options.ignore_case)
+            container.add_require(Require.new(indicated_file, ""))
+          end
+          check_public_methods(external_method, container.name)
+
+        else
+          @@external_aliases << {
+            "new_name"  => generic_name,
+            "old_name"  => proc,
+            "file_or_module" => container,
+            "visibility" => find_visibility(container, generic_name, @@public_methods) || visibility_default
+          }
+        end
+      end
+
+    } if interface_code # End of interface_code.split("\n").each ...
+
+    #
+    # Already imported methods are removed from @@public_methods.
+    # Remainders are assumed to be imported from other modules.
+    #
+    @@public_methods.delete_if{ |method| method["entity_is_discovered"]}
+
+    @@public_methods.each{ |pub_meth|
+      next unless pub_meth["file_or_module"].name == container.name
+      pub_meth["used_modules"].each{ |used_mod|
+        TopLevel.all_classes_and_modules.each{ |modules|
+          if modules.name == used_mod ||
+              modules.name.upcase == used_mod.upcase &&
+              @options.ignore_case
+            modules.method_list.each{ |meth|
+              if meth.name == pub_meth["name"] ||
+                  meth.name.upcase == pub_meth["name"].upcase &&
+                  @options.ignore_case
+                new_meth = initialize_public_method(meth,
+                                                    modules.name)
+                if pub_meth["local_name"]
+                  new_meth.name = pub_meth["local_name"]
+                end
+
+                @stats.add_method new_meth
+
+                container.add_method new_meth
+              end
+            }
+          end
+        }
+      }
+    }
+
+    container
+  end  # End of parse_program_or_module
+
+  ##
+  # Parse arguments, comment, code of subroutine and function.  Return
+  # AnyMethod object.
+
+  def parse_subprogram(subprogram, params, comment, code, 
+                       before_contains=nil, function=nil, prefix=nil)
+    subprogram.singleton = false
+    prefix = "" if !prefix
+    arguments = params.sub(/\(/, "").sub(/\)/, "").split(",") if params
+    args_comment, params_opt = 
+      find_arguments(arguments, code.sub(/^s*?contains\s*?(!.*?)?$.*/im, ""),
+                     nil, nil, true)
+    params_opt = "( " + params_opt + " ) " if params_opt
+    subprogram.params = params_opt || ""
+    namelist_comment = find_namelists(code, before_contains)
+
+    block_comment = find_comments comment
+    if function
+      subprogram.comment = "<b><em> Function </em></b> :: <em>#{prefix}</em>\n"
+    else
+      subprogram.comment = "<b><em> Subroutine </em></b> :: <em>#{prefix}</em>\n"
+    end
+    subprogram.comment << args_comment if args_comment
+    subprogram.comment << block_comment if block_comment
+    subprogram.comment << namelist_comment if namelist_comment
+
+    # For output source code
+    subprogram.start_collecting_tokens
+    subprogram.add_token Token.new(1,1).set_text(code)
+
+    subprogram
+  end
+
+  ##
+  # Collect comment for file entity
+
+  def collect_first_comment(body)
+    comment = ""
+    not_comment = ""
+    comment_start = false
+    comment_end   = false
+    body.split("\n").each{ |line|
+      if comment_end
+        not_comment << line
+        not_comment << "\n"
+      elsif /^\s*?!\s?(.*)$/i =~ line
+        comment_start = true
+        comment << $1
+        comment << "\n"
+      elsif /^\s*?$/i =~ line
+        comment_end = true if comment_start && COMMENTS_ARE_UPPER
+      else
+        comment_end = true
+        not_comment << line
+        not_comment << "\n"
+      end
+    }
+    return comment, not_comment
+  end
+
+
+  ##
+  # Return comments of definitions of arguments
+  #
+  # If "all" argument is true, information of all arguments are returned.
+  #
+  # If "modified_params" is true, list of arguments are decorated, for
+  # example, optional arguments are parenthetic as "[arg]".
+
+  def find_arguments(args, text, all=nil, indent=nil, modified_params=nil)
+    return unless args || all
+    indent = "" unless indent
+    args = ["all"] if all
+    params = "" if modified_params
+    comma = ""
+    return unless text
+    args_rdocforms = "\n"
+    remaining_lines = "#{text}"
+    definitions = definition_info(remaining_lines)
+    args.each{ |arg|
+      arg.strip!
+      arg.chomp!
+      definitions.each { |defitem|
+        if arg == defitem.varname.strip.chomp || all
+          args_rdocforms << <<-"EOF"
+
+#{indent}<tt><b>#{defitem.varname.chomp.strip}#{defitem.arraysuffix}</b> #{defitem.inivalue}</tt> :: 
+#{indent}   <tt>#{defitem.types.chomp.strip}</tt>
+EOF
+          if !defitem.comment.chomp.strip.empty?
+            comment = ""
+            defitem.comment.split("\n").each{ |line|
+              comment << "       " + line + "\n"
+            }
+            args_rdocforms << <<-"EOF"
+
+#{indent}   <tt></tt> :: 
+#{indent}       <tt></tt>
+#{indent}       #{comment.chomp.strip}
+EOF
+          end
+
+          if modified_params
+            if defitem.include_attr?("optional")
+              params << "#{comma}[#{arg}]"
+            else
+              params << "#{comma}#{arg}"
+            end
+            comma = ", "
+          end
+        end
+      }
+    }
+    if modified_params
+      return args_rdocforms, params
+    else
+      return args_rdocforms
+    end
+  end
+
+  ##
+  # Return comments of definitions of namelists
+
+  def find_namelists(text, before_contains=nil)
+    return nil if !text
+    result = ""
+    lines = "#{text}"
+    before_contains = "" if !before_contains
+    while lines =~ /^\s*?namelist\s+\/\s*?(\w+)\s*?\/([\s\w\,]+)$/i
+      lines = $~.post_match
+      nml_comment = COMMENTS_ARE_UPPER ? 
+          find_comments($~.pre_match) : find_comments($~.post_match)
+      nml_name = $1
+      nml_args = $2.split(",")
+      result << "\n\n=== NAMELIST <tt><b>" + nml_name + "</tt></b>\n\n"
+      result << nml_comment + "\n" if nml_comment
+      if lines.split("\n")[0] =~ /^\//i
+        lines = "namelist " + lines
+      end
+      result << find_arguments(nml_args, "#{text}" + "\n" + before_contains)
+    end
+    return result
+  end
+
+  ##
+  # Comments just after module or subprogram, or arguments are returned. If
+  # "COMMENTS_ARE_UPPER" is true, comments just before modules or subprograms
+  # are returnd
+
+  def find_comments text
+    return "" unless text
+    lines = text.split("\n")
+    lines.reverse! if COMMENTS_ARE_UPPER
+    comment_block = Array.new
+    lines.each do |line|
+      break if line =~ /^\s*?\w/ || line =~ /^\s*?$/
+      if COMMENTS_ARE_UPPER
+        comment_block.unshift line.sub(/^\s*?!\s?/,"")
+      else
+        comment_block.push line.sub(/^\s*?!\s?/,"")
+      end
+    end
+    nice_lines = comment_block.join("\n").split "\n\s*?\n"
+    nice_lines[0] ||= ""
+    nice_lines.shift
+  end
+
+  ##
+  # Create method for internal alias
+
+  def initialize_public_method(method, parent)
+    return if !method || !parent
+
+    new_meth = AnyMethod.new("External Alias for module", method.name)
+    new_meth.singleton    = method.singleton
+    new_meth.params       = method.params.clone
+    new_meth.comment      = remove_trailing_alias(method.comment.clone)
+    new_meth.comment      << "\n\n#{EXTERNAL_ALIAS_MES} #{parent.strip.chomp}\##{method.name}"
+
+    return new_meth
+  end
+
+  ##
+  # Create method for external alias
+  #
+  # If argument "internal" is true, file is ignored.
+
+  def initialize_external_method(new, old, params, file, comment, token=nil,
+                                 internal=nil, nolink=nil)
+    return nil unless new || old
+
+    if internal
+      external_alias_header = "#{INTERNAL_ALIAS_MES} "
+      external_alias_text   = external_alias_header + old 
+    elsif file
+      external_alias_header = "#{EXTERNAL_ALIAS_MES} "
+      external_alias_text   = external_alias_header + file + "#" + old
+    else
+      return nil
+    end
+    external_meth = AnyMethod.new(external_alias_text, new)
+    external_meth.singleton    = false
+    external_meth.params       = params
+    external_comment = remove_trailing_alias(comment) + "\n\n" if comment
+    external_meth.comment = external_comment || ""
+    if nolink && token
+      external_meth.start_collecting_tokens
+      external_meth.add_token Token.new(1,1).set_text(token)
+    else
+      external_meth.comment << external_alias_text
+    end
+
+    return external_meth
+  end
+
+  ##
+  # Parse visibility
+
+  def parse_visibility(code, default, container)
+    result = []
+    visibility_default = default || :public
+
+    used_modules = []
+    container.includes.each{|i| used_modules << i.name} if container
+
+    remaining_code = code.gsub(/^\s*?type[\s\,]+.*?\s+end\s+type.*?$/im, "")
+    remaining_code.split("\n").each{ |line|
+      if /^\s*?private\s*?$/ =~ line
+        visibility_default = :private
+        break
+      end
+    } if remaining_code
+
+    remaining_code.split("\n").each{ |line|
+      if /^\s*?private\s*?(::)?\s+(.*)\s*?(!.*?)?/i =~ line
+        methods = $2.sub(/!.*$/, '')
+        methods.split(",").each{ |meth|
+          meth.sub!(/!.*$/, '')
+          meth.gsub!(/:/, '')
+          result << {
+            "name" => meth.chomp.strip,
+            "visibility" => :private,
+            "used_modules" => used_modules.clone,
+            "file_or_module" => container,
+            "entity_is_discovered" => nil,
+            "local_name" => nil
+          }
+        }
+      elsif /^\s*?public\s*?(::)?\s+(.*)\s*?(!.*?)?/i =~ line
+        methods = $2.sub(/!.*$/, '')
+        methods.split(",").each{ |meth|
+          meth.sub!(/!.*$/, '')
+          meth.gsub!(/:/, '')
+          result << {
+            "name" => meth.chomp.strip,
+            "visibility" => :public,
+            "used_modules" => used_modules.clone,
+            "file_or_module" => container,
+            "entity_is_discovered" => nil,
+            "local_name" => nil
+          }
+        }
+      end
+    } if remaining_code
+
+    if container
+      result.each{ |vis_info|
+        vis_info["parent"] = container.name
+      }
+    end
+
+    return visibility_default, result
+  end
+
+  ##
+  # Set visibility
+  #
+  # "subname" element of "visibility_info" is deleted.
+
+  def set_visibility(container, subname, visibility_default, visibility_info)
+    return unless container || subname || visibility_default || visibility_info
+    not_found = true
+    visibility_info.collect!{ |info|
+      if info["name"] == subname ||
+          @options.ignore_case && info["name"].upcase == subname.upcase
+        if info["file_or_module"].name == container.name
+          container.set_visibility_for([subname], info["visibility"])
+          info["entity_is_discovered"] = true
+          not_found = false
+        end
+      end
+      info
+    }
+    if not_found
+      return container.set_visibility_for([subname], visibility_default)
+    else
+      return container
+    end
+  end
+
+  ##
+  # Find visibility
+
+  def find_visibility(container, subname, visibility_info)
+    return nil if !subname || !visibility_info
+    visibility_info.each{ |info|
+      if info["name"] == subname ||
+          @options.ignore_case && info["name"].upcase == subname.upcase
+        if info["parent"] == container.name
+          return info["visibility"]
+        end
+      end
+    }
+    return nil
+  end
+
+  ##
+  # Check external aliases
+
+  def check_external_aliases(subname, params, comment, test=nil)
+    @@external_aliases.each{ |alias_item|
+      if subname == alias_item["old_name"] ||
+                  subname.upcase == alias_item["old_name"].upcase &&
+                          @options.ignore_case
+
+        new_meth = initialize_external_method(alias_item["new_name"], 
+                                              subname, params, @file_name, 
+                                              comment)
+        new_meth.visibility = alias_item["visibility"]
+
+        @stats.add_method new_meth
+
+        alias_item["file_or_module"].add_method(new_meth)
+
+        if !alias_item["file_or_module"].include_requires?(@file_name, @options.ignore_case)
+          alias_item["file_or_module"].add_require(Require.new(@file_name, ""))
+        end
+      end
+    }
+  end
+
+  ##
+  # Check public_methods
+
+  def check_public_methods(method, parent)
+    return if !method || !parent
+    @@public_methods.each{ |alias_item|
+      parent_is_used_module = nil
+      alias_item["used_modules"].each{ |used_module|
+        if used_module == parent ||
+            used_module.upcase == parent.upcase &&
+            @options.ignore_case
+          parent_is_used_module = true
+        end
+      }
+      next if !parent_is_used_module
+
+      if method.name == alias_item["name"] ||
+          method.name.upcase == alias_item["name"].upcase &&
+          @options.ignore_case
+
+        new_meth = initialize_public_method(method, parent)
+        if alias_item["local_name"]
+          new_meth.name = alias_item["local_name"]
+        end
+
+        @stats.add_method new_meth
+
+        alias_item["file_or_module"].add_method new_meth
+      end
+    }
+  end
+
+  ##
+  # Continuous lines are united.
+  #
+  # Comments in continuous lines are removed.
+
+  def united_to_one_line(f90src)
+    return "" unless f90src
+    lines = f90src.split("\n")
+    previous_continuing = false
+    now_continuing = false
+    body = ""
+    lines.each{ |line|
+      words = line.split("")
+      next if words.empty? && previous_continuing
+      commentout = false
+      brank_flag = true ; brank_char = ""
+      squote = false    ; dquote = false
+      ignore = false
+      words.collect! { |char|
+        if previous_continuing && brank_flag
+          now_continuing = true
+          ignore         = true
+          case char
+          when "!"                       ; break
+          when " " ; brank_char << char  ; next ""
+          when "&"
+            brank_flag = false
+            now_continuing = false
+            next ""
+          else 
+            brank_flag     = false
+            now_continuing = false
+            ignore         = false
+            next brank_char + char
+          end
+        end
+        ignore = false
+
+        if now_continuing
+          next ""
+        elsif !(squote) && !(dquote) && !(commentout)
+          case char
+          when "!" ; commentout = true     ; next char
+          when "\""; dquote = true         ; next char
+          when "\'"; squote = true         ; next char
+          when "&" ; now_continuing = true ; next ""
+          else next char
+          end
+        elsif commentout
+          next char
+        elsif squote
+          case char
+          when "\'"; squote = false ; next char
+          else next char
+          end
+        elsif dquote
+          case char
+          when "\""; dquote = false ; next char
+          else next char
+          end
+        end
+      }
+      if !ignore && !previous_continuing || !brank_flag
+        if previous_continuing
+          body << words.join("")
+        else
+          body << "\n" + words.join("")
+        end
+      end
+      previous_continuing = now_continuing ? true : nil
+      now_continuing = nil
+    }
+    return body
+  end
+
+
+  ##
+  # Continuous line checker
+
+  def continuous_line?(line)
+    continuous = false
+    if /&\s*?(!.*)?$/ =~ line
+      continuous = true
+      if comment_out?($~.pre_match)
+        continuous = false
+      end
+    end
+    return continuous
+  end
+
+  ##
+  # Comment out checker
+
+  def comment_out?(line)
+    return nil unless line
+    commentout = false
+    squote = false ; dquote = false
+    line.split("").each { |char|
+      if !(squote) && !(dquote)
+        case char
+        when "!" ; commentout = true ; break
+        when "\""; dquote = true
+        when "\'"; squote = true
+        else next
+        end
+      elsif squote
+        case char
+        when "\'"; squote = false
+        else next
+        end
+      elsif dquote
+        case char
+        when "\""; dquote = false
+        else next
+        end
+      end
+    }
+    return commentout
+  end
+
+  ##
+  # Semicolons are replaced to line feed.
+
+  def semicolon_to_linefeed(text)
+    return "" unless text
+    lines = text.split("\n")
+    lines.collect!{ |line|
+      words = line.split("")
+      commentout = false
+      squote = false ; dquote = false
+      words.collect! { |char|
+        if !(squote) && !(dquote) && !(commentout)
+          case char
+          when "!" ; commentout = true ; next char
+          when "\""; dquote = true     ; next char
+          when "\'"; squote = true     ; next char
+          when ";" ;                     "\n"
+          else next char
+          end
+        elsif commentout
+          next char
+        elsif squote
+          case char
+          when "\'"; squote = false ; next char
+          else next char
+          end
+        elsif dquote
+          case char
+          when "\""; dquote = false ; next char
+          else next char
+          end
+        end
+      }
+      words.join("")
+    }
+    return lines.join("\n")
+  end
+
+  ##
+  # Which "line" is start of block (module, program, block data, subroutine,
+  # function) statement ?
+
+  def block_start?(line)
+    return nil if !line
+
+    if line =~ /^\s*?module\s+(\w+)\s*?(!.*?)?$/i    ||
+        line =~ /^\s*?program\s+(\w+)\s*?(!.*?)?$/i  ||
+        line =~ /^\s*?block\s+data(\s+\w+)?\s*?(!.*?)?$/i     ||
+        line =~ \
+                /^\s*?
+                 (recursive|pure|elemental)?\s*?
+                 subroutine\s+(\w+)\s*?(\(.*?\))?\s*?(!.*?)?$
+                /ix ||
+        line =~ \
+                /^\s*?
+                 (recursive|pure|elemental)?\s*?
+                 (
+                     character\s*?(\([\w\s\=\(\)\*]+?\))?\s+
+                   | type\s*?\([\w\s]+?\)\s+
+                   | integer\s*?(\([\w\s\=\(\)\*]+?\))?\s+
+                   | real\s*?(\([\w\s\=\(\)\*]+?\))?\s+
+                   | double\s+precision\s+
+                   | logical\s*?(\([\w\s\=\(\)\*]+?\))?\s+
+                   | complex\s*?(\([\w\s\=\(\)\*]+?\))?\s+
+                 )?
+                 function\s+(\w+)\s*?
+                 (\(.*?\))?(\s+result\((.*?)\))?\s*?(!.*?)?$
+                /ix
+      return true
+    end
+
+    return nil
+  end
+
+  ##
+  # Which "line" is end of block (module, program, block data, subroutine,
+  # function) statement ?
+
+  def block_end?(line)
+    return nil if !line
+
+    if line =~ /^\s*?end\s*?(!.*?)?$/i                 ||
+        line =~ /^\s*?end\s+module(\s+\w+)?\s*?(!.*?)?$/i       ||
+        line =~ /^\s*?end\s+program(\s+\w+)?\s*?(!.*?)?$/i      ||
+        line =~ /^\s*?end\s+block\s+data(\s+\w+)?\s*?(!.*?)?$/i  ||
+        line =~ /^\s*?end\s+subroutine(\s+\w+)?\s*?(!.*?)?$/i   ||
+        line =~ /^\s*?end\s+function(\s+\w+)?\s*?(!.*?)?$/i
+      return true
+    end
+
+    return nil
+  end
+
+  ##
+  # Remove "Alias for" in end of comments
+
+  def remove_trailing_alias(text)
+    return "" if !text
+    lines = text.split("\n").reverse
+    comment_block = Array.new
+    checked = false
+    lines.each do |line|
+      if !checked 
+        if /^\s?#{INTERNAL_ALIAS_MES}/ =~ line ||
+            /^\s?#{EXTERNAL_ALIAS_MES}/ =~ line
+          checked = true
+          next
+        end
+      end
+      comment_block.unshift line
+    end
+    nice_lines = comment_block.join("\n")
+    nice_lines ||= ""
+    return nice_lines
+  end
+
+  ##
+  # Empty lines in header are removed
+
+  def remove_empty_head_lines(text)
+    return "" unless text
+    lines = text.split("\n")
+    header = true
+    lines.delete_if{ |line|
+      header = false if /\S/ =~ line
+      header && /^\s*?$/ =~ line
+    }
+    lines.join("\n")
+  end
+
+  ##
+  # header marker "=", "==", ... are removed
+
+  def remove_header_marker(text)
+    return text.gsub(/^\s?(=+)/, '<tt></tt>\1')
+  end
+
+  def remove_private_comments(body)
+    body.gsub!(/^\s*!--\s*?$.*?^\s*!\+\+\s*?$/m, '')
+    return body
+  end
+
+  ##
+  # Information of arguments of subroutines and functions in Fortran95
+
+  class Fortran95Definition
+
+    # Name of variable
+    #
+    attr_reader   :varname
+
+    # Types of variable
+    #
+    attr_reader   :types
+
+    # Initial Value
+    #
+    attr_reader   :inivalue
+
+    # Suffix of array
+    #
+    attr_reader   :arraysuffix
+
+    # Comments
+    #
+    attr_accessor   :comment
+
+    # Flag of non documentation
+    #
+    attr_accessor   :nodoc
+
+    def initialize(varname, types, inivalue, arraysuffix, comment,
+                   nodoc=false)
+      @varname = varname
+      @types = types
+      @inivalue = inivalue
+      @arraysuffix = arraysuffix
+      @comment = comment
+      @nodoc = nodoc
+    end
+
+    def to_s
+      return <<-EOF
+<Fortran95Definition: 
+varname=#{@varname}, types=#{types},
+inivalue=#{@inivalue}, arraysuffix=#{@arraysuffix}, nodoc=#{@nodoc}, 
+comment=
+#{@comment}
+>
+EOF
+    end
+
+    #
+    # If attr is included, true is returned
+    #
+    def include_attr?(attr)
+      return if !attr
+      @types.split(",").each{ |type|
+        return true if type.strip.chomp.upcase == attr.strip.chomp.upcase
+      }
+      return nil
+    end
+
+  end # End of Fortran95Definition
+
+  ##
+  # Parse string argument "text", and Return Array of Fortran95Definition
+  # object
+
+  def definition_info(text)
+    return nil unless text
+    lines = "#{text}"
+    defs = Array.new
+    comment = ""
+    trailing_comment = ""
+    under_comment_valid = false
+    lines.split("\n").each{ |line|
+      if /^\s*?!\s?(.*)/ =~ line
+        if COMMENTS_ARE_UPPER
+          comment << remove_header_marker($1)
+          comment << "\n"
+        elsif defs[-1] && under_comment_valid
+          defs[-1].comment << "\n"
+          defs[-1].comment << remove_header_marker($1)
+        end
+        next
+      elsif /^\s*?$/ =~ line
+        comment = ""
+        under_comment_valid = false
+        next
+      end
+      type = ""
+      characters = ""
+      if line =~ /^\s*?
+                  (
+                      character\s*?(\([\w\s\=\(\)\*]+?\))?[\s\,]*
+                    | type\s*?\([\w\s]+?\)[\s\,]*
+                    | integer\s*?(\([\w\s\=\(\)\*]+?\))?[\s\,]*
+                    | real\s*?(\([\w\s\=\(\)\*]+?\))?[\s\,]*
+                    | double\s+precision[\s\,]*
+                    | logical\s*?(\([\w\s\=\(\)\*]+?\))?[\s\,]*
+                    | complex\s*?(\([\w\s\=\(\)\*]+?\))?[\s\,]*
+                  )
+                  (.*?::)?
+                  (.+)$
+                 /ix
+        characters = $8
+        type = $1
+        type << $7.gsub(/::/, '').gsub(/^\s*?\,/, '') if $7
+      else
+        under_comment_valid = false
+        next
+      end
+      squote = false ; dquote = false ; bracket = 0
+      iniflag = false; commentflag = false
+      varname = "" ; arraysuffix = "" ; inivalue = ""
+      start_pos = defs.size
+      characters.split("").each { |char|
+        if !(squote) && !(dquote) && bracket <= 0 && !(iniflag) && !(commentflag)
+          case char
+          when "!" ; commentflag = true
+          when "(" ; bracket += 1       ; arraysuffix = char
+          when "\""; dquote = true
+          when "\'"; squote = true
+          when "=" ; iniflag = true     ; inivalue << char
+          when ","
+            defs << Fortran95Definition.new(varname, type, inivalue, arraysuffix, comment)
+            varname = "" ; arraysuffix = "" ; inivalue = ""
+            under_comment_valid = true
+          when " " ; next
+          else     ; varname << char
+          end
+        elsif commentflag
+          comment << remove_header_marker(char)
+          trailing_comment << remove_header_marker(char)
+        elsif iniflag
+          if dquote
+            case char
+            when "\"" ; dquote = false ; inivalue << char
+            else      ; inivalue << char
+            end
+          elsif squote
+            case char
+            when "\'" ; squote = false ; inivalue << char
+            else      ; inivalue << char
+            end
+          elsif bracket > 0
+            case char
+            when "(" ; bracket += 1 ; inivalue << char
+            when ")" ; bracket -= 1 ; inivalue << char
+            else     ; inivalue << char
+            end
+          else
+            case char
+            when ","
+              defs << Fortran95Definition.new(varname, type, inivalue, arraysuffix, comment)
+              varname = "" ; arraysuffix = "" ; inivalue = ""
+              iniflag = false
+              under_comment_valid = true
+            when "(" ; bracket += 1 ; inivalue << char
+            when "\""; dquote = true  ; inivalue << char
+            when "\'"; squote = true  ; inivalue << char
+            when "!" ; commentflag = true
+            else     ; inivalue << char
+            end
+          end
+        elsif !(squote) && !(dquote) && bracket > 0
+          case char
+          when "(" ; bracket += 1 ; arraysuffix << char
+          when ")" ; bracket -= 1 ; arraysuffix << char
+          else     ; arraysuffix << char
+          end
+        elsif squote
+          case char
+          when "\'"; squote = false ; inivalue << char
+          else     ; inivalue << char
+          end
+        elsif dquote
+          case char
+          when "\""; dquote = false ; inivalue << char
+          else     ; inivalue << char
+          end
+        end
+      }
+      defs << Fortran95Definition.new(varname, type, inivalue, arraysuffix, comment)
+      if trailing_comment =~ /^:nodoc:/
+        defs[start_pos..-1].collect!{ |defitem|
+          defitem.nodoc = true
+        }
+      end
+      varname = "" ; arraysuffix = "" ; inivalue = ""
+      comment = ""
+      under_comment_valid = true
+      trailing_comment = ""
+    }
+    return defs
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/parser/ruby.rb b/trunk/lib/rdoc/parser/ruby.rb
new file mode 100644
index 0000000000..abbc85bde7
--- /dev/null
+++ b/trunk/lib/rdoc/parser/ruby.rb
@@ -0,0 +1,2829 @@
+##
+# This file contains stuff stolen outright from:
+#
+#   rtags.rb -
+#   ruby-lex.rb - ruby lexcal analyzer
+#   ruby-token.rb - ruby tokens
+#       by Keiju ISHITSUKA (Nippon Rational Inc.)
+#
+
+require 'e2mmap'
+require 'irb/slex'
+
+require 'rdoc/code_objects'
+require 'rdoc/tokenstream'
+require 'rdoc/markup/preprocess'
+require 'rdoc/parser'
+
+$TOKEN_DEBUG ||= nil
+#$TOKEN_DEBUG = $DEBUG_RDOC
+
+##
+# Definitions of all tokens involved in the lexical analysis
+
+module RDoc::RubyToken
+
+  EXPR_BEG   = :EXPR_BEG
+  EXPR_MID   = :EXPR_MID
+  EXPR_END   = :EXPR_END
+  EXPR_ARG   = :EXPR_ARG
+  EXPR_FNAME = :EXPR_FNAME
+  EXPR_DOT   = :EXPR_DOT
+  EXPR_CLASS = :EXPR_CLASS
+
+  class Token
+    NO_TEXT = "??".freeze
+
+    attr_accessor :text
+    attr_reader :line_no
+    attr_reader :char_no
+
+    def initialize(line_no, char_no)
+      @line_no = line_no
+      @char_no = char_no
+      @text    = NO_TEXT
+    end
+
+    def ==(other)
+      self.class == other.class and
+        other.line_no == @line_no and
+        other.char_no == @char_no and
+        other.text == @text
+    end
+
+    ##
+    # Because we're used in contexts that expect to return a token, we set the
+    # text string and then return ourselves
+
+    def set_text(text)
+      @text = text
+      self
+    end
+
+  end
+
+  class TkNode < Token
+    attr :node
+  end
+
+  class TkId < Token
+    def initialize(line_no, char_no, name)
+      super(line_no, char_no)
+      @name = name
+    end
+    attr :name
+  end
+
+  class TkKW < TkId
+  end
+
+  class TkVal < Token
+    def initialize(line_no, char_no, value = nil)
+      super(line_no, char_no)
+      set_text(value)
+    end
+  end
+
+  class TkOp < Token
+    def name
+      self.class.op_name
+    end
+  end
+
+  class TkOPASGN < TkOp
+    def initialize(line_no, char_no, op)
+      super(line_no, char_no)
+      op = TkReading2Token[op] unless Symbol === op
+      @op = op
+    end
+    attr :op
+  end
+
+  class TkUnknownChar < Token
+    def initialize(line_no, char_no, id)
+      super(line_no, char_no)
+      @name = char_no.chr
+    end
+    attr :name
+  end
+
+  class TkError < Token
+  end
+
+  def set_token_position(line, char)
+    @prev_line_no = line
+    @prev_char_no = char
+  end
+
+  def Token(token, value = nil)
+    tk = nil
+    case token
+    when String, Symbol
+      source = String === token ? TkReading2Token : TkSymbol2Token
+      raise TkReading2TokenNoKey, token if (tk = source[token]).nil?
+      tk = Token(tk[0], value)
+    else
+      tk = if (token.ancestors & [TkId, TkVal, TkOPASGN, TkUnknownChar]).empty?
+             token.new(@prev_line_no, @prev_char_no)
+           else
+             token.new(@prev_line_no, @prev_char_no, value)
+           end
+    end
+    tk
+  end
+
+  TokenDefinitions = [
+    [:TkCLASS,      TkKW,  "class",  EXPR_CLASS],
+    [:TkMODULE,     TkKW,  "module", EXPR_BEG],
+    [:TkDEF,        TkKW,  "def",    EXPR_FNAME],
+    [:TkUNDEF,      TkKW,  "undef",  EXPR_FNAME],
+    [:TkBEGIN,      TkKW,  "begin",  EXPR_BEG],
+    [:TkRESCUE,     TkKW,  "rescue", EXPR_MID],
+    [:TkENSURE,     TkKW,  "ensure", EXPR_BEG],
+    [:TkEND,        TkKW,  "end",    EXPR_END],
+    [:TkIF,         TkKW,  "if",     EXPR_BEG, :TkIF_MOD],
+    [:TkUNLESS,     TkKW,  "unless", EXPR_BEG, :TkUNLESS_MOD],
+    [:TkTHEN,       TkKW,  "then",   EXPR_BEG],
+    [:TkELSIF,      TkKW,  "elsif",  EXPR_BEG],
+    [:TkELSE,       TkKW,  "else",   EXPR_BEG],
+    [:TkCASE,       TkKW,  "case",   EXPR_BEG],
+    [:TkWHEN,       TkKW,  "when",   EXPR_BEG],
+    [:TkWHILE,      TkKW,  "while",  EXPR_BEG, :TkWHILE_MOD],
+    [:TkUNTIL,      TkKW,  "until",  EXPR_BEG, :TkUNTIL_MOD],
+    [:TkFOR,        TkKW,  "for",    EXPR_BEG],
+    [:TkBREAK,      TkKW,  "break",  EXPR_END],
+    [:TkNEXT,       TkKW,  "next",   EXPR_END],
+    [:TkREDO,       TkKW,  "redo",   EXPR_END],
+    [:TkRETRY,      TkKW,  "retry",  EXPR_END],
+    [:TkIN,         TkKW,  "in",     EXPR_BEG],
+    [:TkDO,         TkKW,  "do",     EXPR_BEG],
+    [:TkRETURN,     TkKW,  "return", EXPR_MID],
+    [:TkYIELD,      TkKW,  "yield",  EXPR_END],
+    [:TkSUPER,      TkKW,  "super",  EXPR_END],
+    [:TkSELF,       TkKW,  "self",   EXPR_END],
+    [:TkNIL,        TkKW,  "nil",    EXPR_END],
+    [:TkTRUE,       TkKW,  "true",   EXPR_END],
+    [:TkFALSE,      TkKW,  "false",  EXPR_END],
+    [:TkAND,        TkKW,  "and",    EXPR_BEG],
+    [:TkOR,         TkKW,  "or",     EXPR_BEG],
+    [:TkNOT,        TkKW,  "not",    EXPR_BEG],
+    [:TkIF_MOD,     TkKW],
+    [:TkUNLESS_MOD, TkKW],
+    [:TkWHILE_MOD,  TkKW],
+    [:TkUNTIL_MOD,  TkKW],
+    [:TkALIAS,      TkKW,  "alias",    EXPR_FNAME],
+    [:TkDEFINED,    TkKW,  "defined?", EXPR_END],
+    [:TklBEGIN,     TkKW,  "BEGIN",    EXPR_END],
+    [:TklEND,       TkKW,  "END",      EXPR_END],
+    [:Tk__LINE__,   TkKW,  "__LINE__", EXPR_END],
+    [:Tk__FILE__,   TkKW,  "__FILE__", EXPR_END],
+
+    [:TkIDENTIFIER, TkId],
+    [:TkFID,        TkId],
+    [:TkGVAR,       TkId],
+    [:TkIVAR,       TkId],
+    [:TkCONSTANT,   TkId],
+
+    [:TkINTEGER,    TkVal],
+    [:TkFLOAT,      TkVal],
+    [:TkSTRING,     TkVal],
+    [:TkXSTRING,    TkVal],
+    [:TkREGEXP,     TkVal],
+    [:TkCOMMENT,    TkVal],
+
+    [:TkDSTRING,    TkNode],
+    [:TkDXSTRING,   TkNode],
+    [:TkDREGEXP,    TkNode],
+    [:TkNTH_REF,    TkId],
+    [:TkBACK_REF,   TkId],
+
+    [:TkUPLUS,      TkOp,   "+@"],
+    [:TkUMINUS,     TkOp,   "-@"],
+    [:TkPOW,        TkOp,   "**"],
+    [:TkCMP,        TkOp,   "<=>"],
+    [:TkEQ,         TkOp,   "=="],
+    [:TkEQQ,        TkOp,   "==="],
+    [:TkNEQ,        TkOp,   "!="],
+    [:TkGEQ,        TkOp,   ">="],
+    [:TkLEQ,        TkOp,   "<="],
+    [:TkANDOP,      TkOp,   "&&"],
+    [:TkOROP,       TkOp,   "||"],
+    [:TkMATCH,      TkOp,   "=~"],
+    [:TkNMATCH,     TkOp,   "!~"],
+    [:TkDOT2,       TkOp,   ".."],
+    [:TkDOT3,       TkOp,   "..."],
+    [:TkAREF,       TkOp,   "[]"],
+    [:TkASET,       TkOp,   "[]="],
+    [:TkLSHFT,      TkOp,   "<<"],
+    [:TkRSHFT,      TkOp,   ">>"],
+    [:TkCOLON2,     TkOp],
+    [:TkCOLON3,     TkOp],
+#   [:OPASGN,       TkOp],               # +=, -=  etc. #
+    [:TkASSOC,      TkOp,   "=>"],
+    [:TkQUESTION,   TkOp,   "?"],        #?
+    [:TkCOLON,      TkOp,   ":"],        #:
+
+    [:TkfLPAREN],         # func( #
+    [:TkfLBRACK],         # func[ #
+    [:TkfLBRACE],         # func{ #
+    [:TkSTAR],            # *arg
+    [:TkAMPER],           # &arg #
+    [:TkSYMBOL,     TkId],          # :SYMBOL
+    [:TkSYMBEG,     TkId],
+    [:TkGT,         TkOp,   ">"],
+    [:TkLT,         TkOp,   "<"],
+    [:TkPLUS,       TkOp,   "+"],
+    [:TkMINUS,      TkOp,   "-"],
+    [:TkMULT,       TkOp,   "*"],
+    [:TkDIV,        TkOp,   "/"],
+    [:TkMOD,        TkOp,   "%"],
+    [:TkBITOR,      TkOp,   "|"],
+    [:TkBITXOR,     TkOp,   "^"],
+    [:TkBITAND,     TkOp,   "&"],
+    [:TkBITNOT,     TkOp,   "~"],
+    [:TkNOTOP,      TkOp,   "!"],
+
+    [:TkBACKQUOTE,  TkOp,   "`"],
+
+    [:TkASSIGN,     Token,  "="],
+    [:TkDOT,        Token,  "."],
+    [:TkLPAREN,     Token,  "("],  #(exp)
+    [:TkLBRACK,     Token,  "["],  #[arry]
+    [:TkLBRACE,     Token,  "{"],  #{hash}
+    [:TkRPAREN,     Token,  ")"],
+    [:TkRBRACK,     Token,  "]"],
+    [:TkRBRACE,     Token,  "}"],
+    [:TkCOMMA,      Token,  ","],
+    [:TkSEMICOLON,  Token,  ";"],
+
+    [:TkRD_COMMENT],
+    [:TkSPACE],
+    [:TkNL],
+    [:TkEND_OF_SCRIPT],
+
+    [:TkBACKSLASH,  TkUnknownChar,  "\\"],
+    [:TkAT,         TkUnknownChar,  "@"],
+    [:TkDOLLAR,     TkUnknownChar,  "\$"], #"
+  ]
+
+  # {reading => token_class}
+  # {reading => [token_class, *opt]}
+  TkReading2Token = {}
+  TkSymbol2Token = {}
+
+  def self.def_token(token_n, super_token = Token, reading = nil, *opts)
+    token_n = token_n.id2name unless String === token_n
+
+    fail AlreadyDefinedToken, token_n if const_defined?(token_n)
+
+    token_c =  Class.new super_token
+    const_set token_n, token_c
+#    token_c.inspect
+
+    if reading
+      if TkReading2Token[reading]
+        fail TkReading2TokenDuplicateError, token_n, reading
+      end
+      if opts.empty?
+        TkReading2Token[reading] = [token_c]
+      else
+        TkReading2Token[reading] = [token_c].concat(opts)
+      end
+    end
+    TkSymbol2Token[token_n.intern] = token_c
+
+    if token_c <= TkOp
+      token_c.class_eval %{
+        def self.op_name; "#{reading}"; end
+      }
+    end
+  end
+
+  for defs in TokenDefinitions
+    def_token(*defs)
+  end
+
+  NEWLINE_TOKEN = TkNL.new(0,0)
+  NEWLINE_TOKEN.set_text("\n")
+
+end
+
+##
+# Lexical analyzer for Ruby source
+
+class RDoc::RubyLex
+
+  ##
+  # Read an input stream character by character. We allow for unlimited
+  # ungetting of characters just read.
+  #
+  # We simplify the implementation greatly by reading the entire input
+  # into a buffer initially, and then simply traversing it using
+  # pointers.
+  #
+  # We also have to allow for the <i>here document diversion</i>. This
+  # little gem comes about when the lexer encounters a here
+  # document. At this point we effectively need to split the input
+  # stream into two parts: one to read the body of the here document,
+  # the other to read the rest of the input line where the here
+  # document was initially encountered. For example, we might have
+  #
+  #   do_something(<<-A, <<-B)
+  #     stuff
+  #     for
+  #   A
+  #     stuff
+  #     for
+  #   B
+  #
+  # When the lexer encounters the <<A, it reads until the end of the
+  # line, and keeps it around for later. It then reads the body of the
+  # here document.  Once complete, it needs to read the rest of the
+  # original line, but then skip the here document body.
+  #
+
+  class BufferedReader
+
+    attr_reader :line_num
+
+    def initialize(content, options)
+      @options = options
+
+      if /\t/ =~ content
+        tab_width = @options.tab_width
+        content = content.split(/\n/).map do |line|
+          1 while line.gsub!(/\t+/) { ' ' * (tab_width*$&.length - $`.length % tab_width)}  && $~ #`
+          line
+        end .join("\n")
+      end
+      @content   = content
+      @content << "\n" unless @content[-1,1] == "\n"
+      @size      = @content.size
+      @offset    = 0
+      @hwm       = 0
+      @line_num  = 1
+      @read_back_offset = 0
+      @last_newline = 0
+      @newline_pending = false
+    end
+
+    def column
+      @offset - @last_newline
+    end
+
+    def getc
+      return nil if @offset >= @size
+      ch = @content[@offset, 1]
+
+      @offset += 1
+      @hwm = @offset if @hwm < @offset
+
+      if @newline_pending
+        @line_num += 1
+        @last_newline = @offset - 1
+        @newline_pending = false
+      end
+
+      if ch == "\n"
+        @newline_pending = true
+      end
+      ch
+    end
+
+    def getc_already_read
+      getc
+    end
+
+    def ungetc(ch)
+      raise "unget past beginning of file" if @offset <= 0
+      @offset -= 1
+      if @content[@offset] == ?\n
+        @newline_pending = false
+      end
+    end
+
+    def get_read
+      res = @content[@read_back_offset...@offset]
+      @read_back_offset = @offset
+      res
+    end
+
+    def peek(at)
+      pos = @offset + at
+      if pos >= @size
+        nil
+      else
+        @content[pos, 1]
+      end
+    end
+
+    def peek_equal(str)
+      @content[@offset, str.length] == str
+    end
+
+    def divert_read_from(reserve)
+      @content[@offset, 0] = reserve
+      @size      = @content.size
+    end
+  end
+
+  # end of nested class BufferedReader
+
+  extend Exception2MessageMapper
+  def_exception(:AlreadyDefinedToken, "Already defined token(%s)")
+  def_exception(:TkReading2TokenNoKey, "key nothing(key='%s')")
+  def_exception(:TkSymbol2TokenNoKey, "key nothing(key='%s')")
+  def_exception(:TkReading2TokenDuplicateError,
+                "key duplicate(token_n='%s', key='%s')")
+  def_exception(:SyntaxError, "%s")
+
+  include RDoc::RubyToken
+  include IRB
+
+  attr_reader :continue
+  attr_reader :lex_state
+
+  def self.debug?
+    false
+  end
+
+  def initialize(content, options)
+    lex_init
+
+    @options = options
+
+    @reader = BufferedReader.new content, @options
+
+    @exp_line_no = @line_no = 1
+    @base_char_no = 0
+    @indent = 0
+
+    @ltype = nil
+    @quoted = nil
+    @lex_state = EXPR_BEG
+    @space_seen = false
+
+    @continue = false
+    @line = ""
+
+    @skip_space = false
+    @read_auto_clean_up = false
+    @exception_on_syntax_error = true
+  end
+
+  attr_accessor :skip_space
+  attr_accessor :read_auto_clean_up
+  attr_accessor :exception_on_syntax_error
+  attr_reader :indent
+
+  # io functions
+  def line_no
+    @reader.line_num
+  end
+
+  def char_no
+    @reader.column
+  end
+
+  def get_read
+    @reader.get_read
+  end
+
+  def getc
+    @reader.getc
+  end
+
+  def getc_of_rests
+    @reader.getc_already_read
+  end
+
+  def gets
+    c = getc or return
+    l = ""
+    begin
+      l.concat c unless c == "\r"
+      break if c == "\n"
+    end while c = getc
+    l
+  end
+
+
+  def ungetc(c = nil)
+    @reader.ungetc(c)
+  end
+
+  def peek_equal?(str)
+    @reader.peek_equal(str)
+  end
+
+  def peek(i = 0)
+    @reader.peek(i)
+  end
+
+  def lex
+    until (TkNL === (tk = token) or TkEND_OF_SCRIPT === tk) and
+           not @continue or tk.nil?
+    end
+
+    line = get_read
+
+    if line == "" and TkEND_OF_SCRIPT === tk or tk.nil? then
+      nil
+    else
+      line
+    end
+  end
+
+  def token
+    set_token_position(line_no, char_no)
+    begin
+      begin
+        tk = @OP.match(self)
+        @space_seen = TkSPACE === tk
+      rescue SyntaxError => e
+        raise RDoc::Error, "syntax error: #{e.message}" if
+          @exception_on_syntax_error
+
+        tk = TkError.new(line_no, char_no)
+      end
+    end while @skip_space and TkSPACE === tk
+    if @read_auto_clean_up
+      get_read
+    end
+#   throw :eof unless tk
+    tk
+  end
+
+  ENINDENT_CLAUSE = [
+    "case", "class", "def", "do", "for", "if",
+    "module", "unless", "until", "while", "begin" #, "when"
+  ]
+  DEINDENT_CLAUSE = ["end" #, "when"
+  ]
+
+  PERCENT_LTYPE = {
+    "q" => "\'",
+    "Q" => "\"",
+    "x" => "\`",
+    "r" => "/",
+    "w" => "]"
+  }
+
+  PERCENT_PAREN = {
+    "{" => "}",
+    "[" => "]",
+    "<" => ">",
+    "(" => ")"
+  }
+
+  Ltype2Token = {
+    "\'" => TkSTRING,
+    "\"" => TkSTRING,
+    "\`" => TkXSTRING,
+    "/" => TkREGEXP,
+    "]" => TkDSTRING
+  }
+  Ltype2Token.default = TkSTRING
+
+  DLtype2Token = {
+    "\"" => TkDSTRING,
+    "\`" => TkDXSTRING,
+    "/" => TkDREGEXP,
+  }
+
+  def lex_init()
+    @OP = IRB::SLex.new
+    @OP.def_rules("\0", "\004", "\032") do |chars, io|
+      Token(TkEND_OF_SCRIPT).set_text(chars)
+    end
+
+    @OP.def_rules(" ", "\t", "\f", "\r", "\13") do |chars, io|
+      @space_seen = TRUE
+      while (ch = getc) =~ /[ \t\f\r\13]/
+        chars << ch
+      end
+      ungetc
+      Token(TkSPACE).set_text(chars)
+    end
+
+    @OP.def_rule("#") do
+      |op, io|
+      identify_comment
+    end
+
+    @OP.def_rule("=begin", proc{@prev_char_no == 0 && peek(0) =~ /\s/}) do
+      |op, io|
+      str = op
+      @ltype = "="
+
+
+      begin
+        line = ""
+        begin
+          ch = getc
+          line << ch
+        end until ch == "\n"
+        str << line
+      end until line =~ /^=end/
+
+      ungetc
+
+      @ltype = nil
+
+      if str =~ /\A=begin\s+rdoc/i
+        str.sub!(/\A=begin.*\n/, '')
+        str.sub!(/^=end.*/m, '')
+        Token(TkCOMMENT).set_text(str)
+      else
+        Token(TkRD_COMMENT)#.set_text(str)
+      end
+    end
+
+    @OP.def_rule("\n") do
+      print "\\n\n" if RDoc::RubyLex.debug?
+      case @lex_state
+      when EXPR_BEG, EXPR_FNAME, EXPR_DOT
+        @continue = TRUE
+      else
+        @continue = FALSE
+        @lex_state = EXPR_BEG
+      end
+      Token(TkNL).set_text("\n")
+    end
+
+    @OP.def_rules("*", "**",
+                  "!", "!=", "!~",
+                  "=", "==", "===",
+                  "=~", "<=>",
+                  "<", "<=",
+                  ">", ">=", ">>") do
+      |op, io|
+      @lex_state = EXPR_BEG
+      Token(op).set_text(op)
+    end
+
+    @OP.def_rules("<<") do
+      |op, io|
+      tk = nil
+      if @lex_state != EXPR_END && @lex_state != EXPR_CLASS &&
+        (@lex_state != EXPR_ARG || @space_seen)
+        c = peek(0)
+        if /[-\w_\"\'\`]/ =~ c
+          tk = identify_here_document
+        end
+      end
+      if !tk
+        @lex_state = EXPR_BEG
+        tk = Token(op).set_text(op)
+      end
+      tk
+    end
+
+    @OP.def_rules("'", '"') do
+      |op, io|
+      identify_string(op)
+    end
+
+    @OP.def_rules("`") do
+      |op, io|
+      if @lex_state == EXPR_FNAME
+        Token(op).set_text(op)
+      else
+        identify_string(op)
+      end
+    end
+
+    @OP.def_rules('?') do
+      |op, io|
+      if @lex_state == EXPR_END
+        @lex_state = EXPR_BEG
+        Token(TkQUESTION).set_text(op)
+      else
+        ch = getc
+        if @lex_state == EXPR_ARG && ch !~ /\s/
+          ungetc
+          @lex_state = EXPR_BEG
+          Token(TkQUESTION).set_text(op)
+        else
+          str = op
+          str << ch
+          if (ch == '\\') #'
+            str << read_escape
+          end
+          @lex_state = EXPR_END
+          Token(TkINTEGER).set_text(str)
+        end
+      end
+    end
+
+    @OP.def_rules("&", "&&", "|", "||") do
+      |op, io|
+      @lex_state = EXPR_BEG
+      Token(op).set_text(op)
+    end
+
+    @OP.def_rules("+=", "-=", "*=", "**=",
+                  "&=", "|=", "^=", "<<=", ">>=", "||=", "&&=") do
+      |op, io|
+      @lex_state = EXPR_BEG
+      op =~ /^(.*)=$/
+      Token(TkOPASGN, $1).set_text(op)
+    end
+
+    @OP.def_rule("+@", proc{@lex_state == EXPR_FNAME}) do |op, io|
+      Token(TkUPLUS).set_text(op)
+    end
+
+    @OP.def_rule("-@", proc{@lex_state == EXPR_FNAME}) do |op, io|
+      Token(TkUMINUS).set_text(op)
+    end
+
+    @OP.def_rules("+", "-") do
+      |op, io|
+      catch(:RET) do
+        if @lex_state == EXPR_ARG
+          if @space_seen and peek(0) =~ /[0-9]/
+            throw :RET, identify_number(op)
+          else
+            @lex_state = EXPR_BEG
+          end
+        elsif @lex_state != EXPR_END and peek(0) =~ /[0-9]/
+          throw :RET, identify_number(op)
+        else
+          @lex_state = EXPR_BEG
+        end
+        Token(op).set_text(op)
+      end
+    end
+
+    @OP.def_rule(".") do
+      @lex_state = EXPR_BEG
+      if peek(0) =~ /[0-9]/
+        ungetc
+        identify_number("")
+      else
+        # for obj.if
+        @lex_state = EXPR_DOT
+        Token(TkDOT).set_text(".")
+      end
+    end
+
+    @OP.def_rules("..", "...") do
+      |op, io|
+      @lex_state = EXPR_BEG
+      Token(op).set_text(op)
+    end
+
+    lex_int2
+  end
+
+  def lex_int2
+    @OP.def_rules("]", "}", ")") do
+      |op, io|
+      @lex_state = EXPR_END
+      @indent -= 1
+      Token(op).set_text(op)
+    end
+
+    @OP.def_rule(":") do
+      if @lex_state == EXPR_END || peek(0) =~ /\s/
+        @lex_state = EXPR_BEG
+        tk = Token(TkCOLON)
+      else
+        @lex_state = EXPR_FNAME
+        tk = Token(TkSYMBEG)
+      end
+      tk.set_text(":")
+    end
+
+    @OP.def_rule("::") do
+      if @lex_state == EXPR_BEG or @lex_state == EXPR_ARG && @space_seen
+        @lex_state = EXPR_BEG
+        tk = Token(TkCOLON3)
+      else
+        @lex_state = EXPR_DOT
+        tk = Token(TkCOLON2)
+      end
+      tk.set_text("::")
+    end
+
+    @OP.def_rule("/") do
+      |op, io|
+      if @lex_state == EXPR_BEG || @lex_state == EXPR_MID
+        identify_string(op)
+      elsif peek(0) == '='
+        getc
+        @lex_state = EXPR_BEG
+        Token(TkOPASGN, :/).set_text("/=") #")
+      elsif @lex_state == EXPR_ARG and @space_seen and peek(0) !~ /\s/
+        identify_string(op)
+      else
+        @lex_state = EXPR_BEG
+        Token("/").set_text(op)
+      end
+    end
+
+    @OP.def_rules("^") do
+      @lex_state = EXPR_BEG
+      Token("^").set_text("^")
+    end
+
+    @OP.def_rules(",", ";") do
+      |op, io|
+      @lex_state = EXPR_BEG
+      Token(op).set_text(op)
+    end
+
+    @OP.def_rule("~") do
+      @lex_state = EXPR_BEG
+      Token("~").set_text("~")
+    end
+
+    @OP.def_rule("~@", proc{@lex_state = EXPR_FNAME}) do
+      @lex_state = EXPR_BEG
+      Token("~").set_text("~@")
+    end
+
+    @OP.def_rule("(") do
+      @indent += 1
+      if @lex_state == EXPR_BEG || @lex_state == EXPR_MID
+        @lex_state = EXPR_BEG
+        tk = Token(TkfLPAREN)
+      else
+        @lex_state = EXPR_BEG
+        tk = Token(TkLPAREN)
+      end
+      tk.set_text("(")
+    end
+
+    @OP.def_rule("[]", proc{@lex_state == EXPR_FNAME}) do
+      Token("[]").set_text("[]")
+    end
+
+    @OP.def_rule("[]=", proc{@lex_state == EXPR_FNAME}) do
+      Token("[]=").set_text("[]=")
+    end
+
+    @OP.def_rule("[") do
+      @indent += 1
+      if @lex_state == EXPR_FNAME
+        t = Token(TkfLBRACK)
+      else
+        if @lex_state == EXPR_BEG || @lex_state == EXPR_MID
+          t = Token(TkLBRACK)
+        elsif @lex_state == EXPR_ARG && @space_seen
+          t = Token(TkLBRACK)
+        else
+          t = Token(TkfLBRACK)
+        end
+        @lex_state = EXPR_BEG
+      end
+      t.set_text("[")
+    end
+
+    @OP.def_rule("{") do
+      @indent += 1
+      if @lex_state != EXPR_END && @lex_state != EXPR_ARG
+        t = Token(TkLBRACE)
+      else
+        t = Token(TkfLBRACE)
+      end
+      @lex_state = EXPR_BEG
+      t.set_text("{")
+    end
+
+    @OP.def_rule('\\') do   #'
+      if getc == "\n"
+        @space_seen = true
+        @continue = true
+        Token(TkSPACE).set_text("\\\n")
+      else
+        ungetc
+        Token("\\").set_text("\\")  #"
+      end
+    end
+
+    @OP.def_rule('%') do
+      |op, io|
+      if @lex_state == EXPR_BEG || @lex_state == EXPR_MID
+        identify_quotation('%')
+      elsif peek(0) == '='
+        getc
+        Token(TkOPASGN, "%").set_text("%=")
+      elsif @lex_state == EXPR_ARG and @space_seen and peek(0) !~ /\s/
+        identify_quotation('%')
+      else
+        @lex_state = EXPR_BEG
+        Token("%").set_text("%")
+      end
+    end
+
+    @OP.def_rule('$') do  #'
+      identify_gvar
+    end
+
+    @OP.def_rule('@') do
+      if peek(0) =~ /[@\w_]/
+        ungetc
+        identify_identifier
+      else
+        Token("@").set_text("@")
+      end
+    end
+
+    @OP.def_rule("__END__", proc{@prev_char_no == 0 && peek(0) =~ /[\r\n]/}) do
+      throw :eof
+    end
+
+    @OP.def_rule("") do
+      |op, io|
+      printf "MATCH: start %s: %s\n", op, io.inspect if RDoc::RubyLex.debug?
+      if peek(0) =~ /[0-9]/
+        t = identify_number("")
+      elsif peek(0) =~ /[\w_]/
+        t = identify_identifier
+      end
+      printf "MATCH: end %s: %s\n", op, io.inspect if RDoc::RubyLex.debug?
+      t
+    end
+  end
+
+  def identify_gvar
+    @lex_state = EXPR_END
+    str = "$"
+
+    tk = case ch = getc
+         when /[~_*$?!@\/\\;,=:<>".]/   #"
+           str << ch
+           Token(TkGVAR, str)
+
+         when "-"
+           str << "-" << getc
+           Token(TkGVAR, str)
+
+         when "&", "`", "'", "+"
+           str << ch
+           Token(TkBACK_REF, str)
+
+         when /[1-9]/
+           str << ch
+           while (ch = getc) =~ /[0-9]/
+             str << ch
+           end
+           ungetc
+           Token(TkNTH_REF)
+         when /\w/
+           ungetc
+           ungetc
+           return identify_identifier
+         else
+           ungetc
+           Token("$")
+         end
+    tk.set_text(str)
+  end
+
+  def identify_identifier
+    token = ""
+    token.concat getc if peek(0) =~ /[$@]/
+    token.concat getc if peek(0) == "@"
+
+    while (ch = getc) =~ /\w|_/
+      print ":", ch, ":" if RDoc::RubyLex.debug?
+      token.concat ch
+    end
+    ungetc
+
+    if ch == "!" or ch == "?"
+      token.concat getc
+    end
+    # fix token
+
+    # $stderr.puts "identifier - #{token}, state = #@lex_state"
+
+    case token
+    when /^\$/
+      return Token(TkGVAR, token).set_text(token)
+    when /^\@/
+      @lex_state = EXPR_END
+      return Token(TkIVAR, token).set_text(token)
+    end
+
+    if @lex_state != EXPR_DOT
+      print token, "\n" if RDoc::RubyLex.debug?
+
+      token_c, *trans = TkReading2Token[token]
+      if token_c
+        # reserved word?
+
+        if (@lex_state != EXPR_BEG &&
+            @lex_state != EXPR_FNAME &&
+            trans[1])
+          # modifiers
+          token_c = TkSymbol2Token[trans[1]]
+          @lex_state = trans[0]
+        else
+          if @lex_state != EXPR_FNAME
+            if ENINDENT_CLAUSE.include?(token)
+              @indent += 1
+            elsif DEINDENT_CLAUSE.include?(token)
+              @indent -= 1
+            end
+            @lex_state = trans[0]
+          else
+            @lex_state = EXPR_END
+          end
+        end
+        return Token(token_c, token).set_text(token)
+      end
+    end
+
+    if @lex_state == EXPR_FNAME
+      @lex_state = EXPR_END
+      if peek(0) == '='
+        token.concat getc
+      end
+    elsif @lex_state == EXPR_BEG || @lex_state == EXPR_DOT
+      @lex_state = EXPR_ARG
+    else
+      @lex_state = EXPR_END
+    end
+
+    if token[0, 1] =~ /[A-Z]/
+      return Token(TkCONSTANT, token).set_text(token)
+    elsif token[token.size - 1, 1] =~ /[!?]/
+      return Token(TkFID, token).set_text(token)
+    else
+      return Token(TkIDENTIFIER, token).set_text(token)
+    end
+  end
+
+  def identify_here_document
+    ch = getc
+    if ch == "-"
+      ch = getc
+      indent = true
+    end
+    if /['"`]/ =~ ch            # '
+      lt = ch
+      quoted = ""
+      while (c = getc) && c != lt
+        quoted.concat c
+      end
+    else
+      lt = '"'
+      quoted = ch.dup
+      while (c = getc) && c =~ /\w/
+        quoted.concat c
+      end
+      ungetc
+    end
+
+    ltback, @ltype = @ltype, lt
+    reserve = ""
+
+    while ch = getc
+      reserve << ch
+      if ch == "\\"    #"
+        ch = getc
+        reserve << ch
+      elsif ch == "\n"
+        break
+      end
+    end
+
+    str = ""
+    while (l = gets)
+      l.chomp!
+      l.strip! if indent
+      break if l == quoted
+      str << l.chomp << "\n"
+    end
+
+    @reader.divert_read_from(reserve)
+
+    @ltype = ltback
+    @lex_state = EXPR_END
+    Token(Ltype2Token[lt], str).set_text(str.dump)
+  end
+
+  def identify_quotation(initial_char)
+    ch = getc
+    if lt = PERCENT_LTYPE[ch]
+      initial_char += ch
+      ch = getc
+    elsif ch =~ /\W/
+      lt = "\""
+    else
+      fail SyntaxError, "unknown type of %string ('#{ch}')"
+    end
+#     if ch !~ /\W/
+#       ungetc
+#       next
+#     end
+    #@ltype = lt
+    @quoted = ch unless @quoted = PERCENT_PAREN[ch]
+    identify_string(lt, @quoted, ch, initial_char)
+  end
+
+  def identify_number(start)
+    str = start.dup
+
+    if start == "+" or start == "-" or start == ""
+      start = getc
+      str << start
+    end
+
+    @lex_state = EXPR_END
+
+    if start == "0"
+      if peek(0) == "x"
+        ch = getc
+        str << ch
+        match = /[0-9a-f_]/
+      else
+        match = /[0-7_]/
+      end
+      while ch = getc
+        if ch !~ match
+          ungetc
+          break
+        else
+          str << ch
+        end
+      end
+      return Token(TkINTEGER).set_text(str)
+    end
+
+    type = TkINTEGER
+    allow_point = TRUE
+    allow_e = TRUE
+    while ch = getc
+      case ch
+      when /[0-9_]/
+        str << ch
+
+      when allow_point && "."
+        type = TkFLOAT
+        if peek(0) !~ /[0-9]/
+          ungetc
+          break
+        end
+        str << ch
+        allow_point = false
+
+      when allow_e && "e", allow_e && "E"
+        str << ch
+        type = TkFLOAT
+        if peek(0) =~ /[+-]/
+          str << getc
+        end
+        allow_e = false
+        allow_point = false
+      else
+        ungetc
+        break
+      end
+    end
+    Token(type).set_text(str)
+  end
+
+  def identify_string(ltype, quoted = ltype, opener=nil, initial_char = nil)
+    @ltype = ltype
+    @quoted = quoted
+    subtype = nil
+
+    str = ""
+    str << initial_char if initial_char
+    str << (opener||quoted)
+
+    nest = 0
+    begin
+      while ch = getc
+        str << ch
+        if @quoted == ch
+          if nest == 0
+            break
+          else
+            nest -= 1
+          end
+        elsif opener == ch
+          nest += 1
+        elsif @ltype != "'" && @ltype != "]" and ch == "#"
+          ch = getc
+          if ch == "{"
+            subtype = true
+            str << ch << skip_inner_expression
+          else
+            ungetc(ch)
+          end
+        elsif ch == '\\' #'
+          str << read_escape
+        end
+      end
+      if @ltype == "/"
+        if peek(0) =~ /i|o|n|e|s/
+          str << getc
+        end
+      end
+      if subtype
+        Token(DLtype2Token[ltype], str)
+      else
+        Token(Ltype2Token[ltype], str)
+      end.set_text(str)
+    ensure
+      @ltype = nil
+      @quoted = nil
+      @lex_state = EXPR_END
+    end
+  end
+
+  def skip_inner_expression
+    res = ""
+    nest = 0
+    while (ch = getc)
+      res << ch
+      if ch == '}'
+        break if nest.zero?
+        nest -= 1
+      elsif ch == '{'
+        nest += 1
+      end
+    end
+    res
+  end
+
+  def identify_comment
+    @ltype = "#"
+    comment = "#"
+    while ch = getc
+      if ch == "\\"
+        ch = getc
+        if ch == "\n"
+          ch = " "
+        else
+          comment << "\\"
+        end
+      else
+        if ch == "\n"
+          @ltype = nil
+          ungetc
+          break
+        end
+      end
+      comment << ch
+    end
+    return Token(TkCOMMENT).set_text(comment)
+  end
+
+  def read_escape
+    res = ""
+    case ch = getc
+    when /[0-7]/
+      ungetc ch
+      3.times do
+        case ch = getc
+        when /[0-7]/
+        when nil
+          break
+        else
+          ungetc
+          break
+        end
+        res << ch
+      end
+
+    when "x"
+      res << ch
+      2.times do
+        case ch = getc
+        when /[0-9a-fA-F]/
+        when nil
+          break
+        else
+          ungetc
+          break
+        end
+        res << ch
+      end
+
+    when "M"
+      res << ch
+      if (ch = getc) != '-'
+        ungetc
+      else
+        res << ch
+        if (ch = getc) == "\\" #"
+          res << ch
+          res << read_escape
+        else
+          res << ch
+        end
+      end
+
+    when "C", "c" #, "^"
+      res << ch
+      if ch == "C" and (ch = getc) != "-"
+        ungetc
+      else
+        res << ch
+        if (ch = getc) == "\\" #"
+          res << ch
+          res << read_escape
+        else
+          res << ch
+        end
+      end
+    else
+      res << ch
+    end
+    res
+  end
+end
+
+##
+# Extracts code elements from a source file returning a TopLevel object
+# containing the constituent file elements.
+#
+# This file is based on rtags
+#
+# RubyParser understands how to document:
+# * classes
+# * modules
+# * methods
+# * constants
+# * aliases
+# * private, public, protected
+# * private_class_function, public_class_function
+# * module_function
+# * attr, attr_reader, attr_writer, attr_accessor
+# * extra accessors given on the command line
+# * metaprogrammed methods
+# * require
+# * include
+#
+# == Method Arguments
+#
+#--
+# NOTE: I don't think this works, needs tests, remove the paragraph following
+# this block when known to work
+#
+# The parser extracts the arguments from the method definition.  You can
+# override this with a custom argument definition using the :args: directive:
+#
+#   ##
+#   # This method tries over and over until it is tired
+#   
+#   def go_go_go(thing_to_try, tries = 10) # :args: thing_to_try
+#     puts thing_to_try
+#     go_go_go thing_to_try, tries - 1
+#   end
+#
+# If you have a more-complex set of overrides you can use the :call-seq:
+# directive:
+#++
+#
+# The parser extracts the arguments from the method definition.  You can
+# override this with a custom argument definition using the :call-seq:
+# directive:
+#
+#   ##
+#   # This method can be called with a range or an offset and length
+#   #
+#   # :call-seq:
+#   #   my_method(Range)
+#   #   my_method(offset, length)
+#   
+#   def my_method(*args)
+#   end
+#
+# The parser extracts +yield+ expressions from method bodies to gather the
+# yielded argument names.  If your method manually calls a block instead of
+# yielding or you want to override the discovered argument names use
+# the :yields: directive:
+#
+#   ##
+#   # My method is awesome
+#   
+#   def my_method(&block) # :yields: happy, times
+#     block.call 1, 2
+#   end
+#
+# == Metaprogrammed Methods
+#
+# To pick up a metaprogrammed method, the parser looks for a comment starting
+# with '##' before an identifier:
+#
+#   ##
+#   # This is a meta-programmed method!
+#   
+#   add_my_method :meta_method, :arg1, :arg2
+#
+# The parser looks at the token after the identifier to determine the name, in
+# this example, :meta_method.  If a name cannot be found, a warning is printed
+# and 'unknown is used.
+#
+# You can force the name of a method using the :method: directive:
+#
+#   ##
+#   # :method: woo_hoo!
+#
+# By default, meta-methods are instance methods.  To indicate that a method is
+# a singleton method instead use the :singleton-method: directive:
+#
+#   ##
+#   # :singleton-method:
+#
+# You can also use the :singleton-method: directive with a name:
+#
+#   ##
+#   # :singleton-method: woo_hoo!
+#
+# == Hidden methods
+#
+# You can provide documentation for methods that don't appear using
+# the :method: and :singleton-method: directives:
+#
+#   ##
+#   # :method: ghost_method
+#   # There is a method here, but you can't see it!
+#   
+#   ##
+#   # this is a comment for a regular method
+#   
+#   def regular_method() end
+#
+# Note that by default, the :method: directive will be ignored if there is a
+# standard rdocable item following it.
+
+class RDoc::Parser::Ruby < RDoc::Parser
+
+  parse_files_matching(/\.rbw?$/)
+
+  include RDoc::RubyToken
+  include RDoc::TokenStream
+
+  NORMAL = "::"
+  SINGLE = "<<"
+
+  def initialize(top_level, file_name, content, options, stats)
+    super
+
+    @size = 0
+    @token_listeners = nil
+    @scanner = RDoc::RubyLex.new content, @options
+    @scanner.exception_on_syntax_error = false
+
+    reset
+  end
+
+  def add_token_listener(obj)
+    @token_listeners ||= []
+    @token_listeners << obj
+  end
+
+  ##
+  # Look for the first comment in a file that isn't a shebang line.
+
+  def collect_first_comment
+    skip_tkspace
+    res = ''
+    first_line = true
+
+    tk = get_tk
+
+    while TkCOMMENT === tk
+      if first_line and tk.text =~ /\A#!/ then
+        skip_tkspace
+        tk = get_tk
+      elsif first_line and tk.text =~ /\A#\s*-\*-/ then
+        first_line = false
+        skip_tkspace
+        tk = get_tk
+      else
+        first_line = false
+        res << tk.text << "\n"
+        tk = get_tk
+
+        if TkNL === tk then
+          skip_tkspace false
+          tk = get_tk
+        end
+      end
+    end
+
+    unget_tk tk
+
+    res
+  end
+
+  def error(msg)
+    msg = make_message msg
+    $stderr.puts msg
+    exit(1)
+  end
+
+  ##
+  # Look for a 'call-seq' in the comment, and override the normal parameter
+  # stuff
+
+  def extract_call_seq(comment, meth)
+    if comment.sub!(/:?call-seq:(.*?)^\s*\#?\s*$/m, '') then
+      seq = $1
+      seq.gsub!(/^\s*\#\s*/, '')
+      meth.call_seq = seq
+    end
+
+    meth
+  end
+
+  def get_bool
+    skip_tkspace
+    tk = get_tk
+    case tk
+    when TkTRUE
+      true
+    when TkFALSE, TkNIL
+      false
+    else
+      unget_tk tk
+      true
+    end
+  end
+
+  ##
+  # Look for the name of a class of module (optionally with a leading :: or
+  # with :: separated named) and return the ultimate name and container
+
+  def get_class_or_module(container)
+    skip_tkspace
+    name_t = get_tk
+
+    # class ::A -> A is in the top level
+    if TkCOLON2 === name_t then
+      name_t = get_tk
+      container = @top_level
+    end
+
+    skip_tkspace(false)
+
+    while TkCOLON2 === peek_tk do
+      prev_container = container
+      container = container.find_module_named(name_t.name)
+      if !container
+#          warn("Couldn't find module #{name_t.name}")
+        container = prev_container.add_module RDoc::NormalModule, name_t.name
+      end
+      get_tk
+      name_t = get_tk
+    end
+    skip_tkspace(false)
+    return [container, name_t]
+  end
+
+  ##
+  # Return a superclass, which can be either a constant of an expression
+
+  def get_class_specification
+    tk = get_tk
+    return "self" if TkSELF === tk
+
+    res = ""
+    while TkCOLON2 === tk or TkCOLON3 === tk or TkCONSTANT === tk do
+      res += tk.text
+      tk = get_tk
+    end
+
+    unget_tk(tk)
+    skip_tkspace(false)
+
+    get_tkread # empty out read buffer
+
+    tk = get_tk
+
+    case tk
+    when TkNL, TkCOMMENT, TkSEMICOLON then
+      unget_tk(tk)
+      return res
+    end
+
+    res += parse_call_parameters(tk)
+    res
+  end
+
+  ##
+  # Parse a constant, which might be qualified by one or more class or module
+  # names
+
+  def get_constant
+    res = ""
+    skip_tkspace(false)
+    tk = get_tk
+
+    while TkCOLON2 === tk or TkCOLON3 === tk or TkCONSTANT === tk do
+      res += tk.text
+      tk = get_tk
+    end
+
+#      if res.empty?
+#        warn("Unexpected token #{tk} in constant")
+#      end
+    unget_tk(tk)
+    res
+  end
+
+  ##
+  # Get a constant that may be surrounded by parens
+
+  def get_constant_with_optional_parens
+    skip_tkspace(false)
+    nest = 0
+    while TkLPAREN === (tk = peek_tk) or TkfLPAREN === tk do
+      get_tk
+      skip_tkspace(true)
+      nest += 1
+    end
+
+    name = get_constant
+
+    while nest > 0
+      skip_tkspace(true)
+      tk = get_tk
+      nest -= 1 if TkRPAREN === tk
+    end
+    name
+  end
+
+  def get_symbol_or_name
+    tk = get_tk
+    case tk
+    when  TkSYMBOL
+      tk.text.sub(/^:/, '')
+    when TkId, TkOp
+      tk.name
+    when TkSTRING
+      tk.text
+    else
+      raise "Name or symbol expected (got #{tk})"
+    end
+  end
+
+  def get_tk
+    tk = nil
+    if @tokens.empty?
+      tk = @scanner.token
+      @read.push @scanner.get_read
+      puts "get_tk1 => #{tk.inspect}" if $TOKEN_DEBUG
+    else
+      @read.push @unget_read.shift
+      tk = @tokens.shift
+      puts "get_tk2 => #{tk.inspect}" if $TOKEN_DEBUG
+    end
+
+    if TkSYMBEG === tk then
+      set_token_position(tk.line_no, tk.char_no)
+      tk1 = get_tk
+      if TkId === tk1 or TkOp === tk1 or TkSTRING === tk1 then
+        if tk1.respond_to?(:name)
+          tk = Token(TkSYMBOL).set_text(":" + tk1.name)
+        else
+          tk = Token(TkSYMBOL).set_text(":" + tk1.text)
+        end
+        # remove the identifier we just read (we're about to
+        # replace it with a symbol)
+        @token_listeners.each do |obj|
+          obj.pop_token
+        end if @token_listeners
+      else
+        warn("':' not followed by identifier or operator")
+        tk = tk1
+      end
+    end
+
+    # inform any listeners of our shiny new token
+    @token_listeners.each do |obj|
+      obj.add_token(tk)
+    end if @token_listeners
+
+    tk
+  end
+
+  def get_tkread
+    read = @read.join("")
+    @read = []
+    read
+  end
+
+  ##
+  # Look for directives in a normal comment block:
+  #
+  #   #-- - don't display comment from this point forward
+  #
+  # This routine modifies it's parameter
+
+  def look_for_directives_in(context, comment)
+    preprocess = RDoc::Markup::PreProcess.new(@file_name,
+                                              @options.rdoc_include)
+
+    preprocess.handle(comment) do |directive, param|
+      case directive
+      when 'enddoc' then
+        throw :enddoc
+      when 'main' then
+        @options.main_page = param
+        ''
+      when 'method', 'singleton-method' then
+        false # ignore
+      when 'section' then
+        context.set_current_section(param, comment)
+        comment.replace ''
+        break
+      when 'startdoc' then
+        context.start_doc
+        context.force_documentation = true
+        ''
+      when 'stopdoc' then
+        context.stop_doc
+        ''
+      when 'title' then
+        @options.title = param
+        ''
+      else
+        warn "Unrecognized directive '#{directive}'"
+        false
+      end
+    end
+
+    remove_private_comments(comment)
+  end
+
+  def make_message(msg)
+    prefix = "\n" + @file_name + ":"
+    if @scanner
+      prefix << "#{@scanner.line_no}:#{@scanner.char_no}: "
+    end
+    return prefix + msg
+  end
+
+  def parse_attr(context, single, tk, comment)
+    args = parse_symbol_arg(1)
+    if args.size > 0
+      name = args[0]
+      rw = "R"
+      skip_tkspace(false)
+      tk = get_tk
+      if TkCOMMA === tk then
+        rw = "RW" if get_bool
+      else
+        unget_tk tk
+      end
+      att = RDoc::Attr.new get_tkread, name, rw, comment
+      read_documentation_modifiers att, RDoc::ATTR_MODIFIERS
+      if att.document_self
+        context.add_attribute(att)
+      end
+    else
+      warn("'attr' ignored - looks like a variable")
+    end
+  end
+
+  def parse_attr_accessor(context, single, tk, comment)
+    args = parse_symbol_arg
+    read = get_tkread
+    rw = "?"
+
+    # If nodoc is given, don't document any of them
+
+    tmp = RDoc::CodeObject.new
+    read_documentation_modifiers tmp, RDoc::ATTR_MODIFIERS
+    return unless tmp.document_self
+
+    case tk.name
+    when "attr_reader"   then rw = "R"
+    when "attr_writer"   then rw = "W"
+    when "attr_accessor" then rw = "RW"
+    else
+      rw = @options.extra_accessor_flags[tk.name]
+      rw = '?' if rw.nil?
+    end
+
+    for name in args
+      att = RDoc::Attr.new get_tkread, name, rw, comment
+      context.add_attribute att
+    end
+  end
+
+  def parse_alias(context, single, tk, comment)
+    skip_tkspace
+    if TkLPAREN === peek_tk then
+      get_tk
+      skip_tkspace
+    end
+    new_name = get_symbol_or_name
+    @scanner.instance_eval{@lex_state = EXPR_FNAME}
+    skip_tkspace
+    if TkCOMMA === peek_tk then
+      get_tk
+      skip_tkspace
+    end
+    old_name = get_symbol_or_name
+
+    al = RDoc::Alias.new get_tkread, old_name, new_name, comment
+    read_documentation_modifiers al, RDoc::ATTR_MODIFIERS
+    if al.document_self
+      context.add_alias(al)
+    end
+  end
+
+  def parse_call_parameters(tk)
+    end_token = case tk
+                when TkLPAREN, TkfLPAREN
+                  TkRPAREN
+                when TkRPAREN
+                  return ""
+                else
+                  TkNL
+                end
+    nest = 0
+
+    loop do
+        case tk
+        when TkSEMICOLON
+          break
+        when TkLPAREN, TkfLPAREN
+          nest += 1
+        when end_token
+          if end_token == TkRPAREN
+            nest -= 1
+            break if @scanner.lex_state == EXPR_END and nest <= 0
+          else
+            break unless @scanner.continue
+          end
+        when TkCOMMENT
+          unget_tk(tk)
+          break
+        end
+        tk = get_tk
+    end
+    res = get_tkread.tr("\n", " ").strip
+    res = "" if res == ";"
+    res
+  end
+
+  def parse_class(container, single, tk, comment)
+    container, name_t = get_class_or_module(container)
+
+    case name_t
+    when TkCONSTANT
+      name = name_t.name
+      superclass = "Object"
+
+      if TkLT === peek_tk then
+        get_tk
+        skip_tkspace(true)
+        superclass = get_class_specification
+        superclass = "<unknown>" if superclass.empty?
+      end
+
+      cls_type = single == SINGLE ? RDoc::SingleClass : RDoc::NormalClass
+      cls = container.add_class cls_type, name, superclass
+
+      @stats.add_class cls
+
+      read_documentation_modifiers cls, RDoc::CLASS_MODIFIERS
+      cls.record_location @top_level
+
+      parse_statements cls
+      cls.comment = comment
+
+    when TkLSHFT
+      case name = get_class_specification
+      when "self", container.name
+        parse_statements(container, SINGLE)
+      else
+        other = RDoc::TopLevel.find_class_named(name)
+        unless other
+          #            other = @top_level.add_class(NormalClass, name, nil)
+          #            other.record_location(@top_level)
+          #            other.comment = comment
+          other = RDoc::NormalClass.new "Dummy", nil
+        end
+
+        @stats.add_class other
+
+        read_documentation_modifiers other, RDoc::CLASS_MODIFIERS
+        parse_statements(other, SINGLE)
+      end
+
+    else
+      warn("Expected class name or '<<'. Got #{name_t.class}: #{name_t.text.inspect}")
+    end
+  end
+
+  def parse_constant(container, single, tk, comment)
+    name = tk.name
+    skip_tkspace(false)
+    eq_tk = get_tk
+
+    unless TkASSIGN === eq_tk then
+      unget_tk(eq_tk)
+      return
+    end
+
+
+    nest = 0
+    get_tkread
+
+    tk = get_tk
+    if TkGT === tk then
+      unget_tk(tk)
+      unget_tk(eq_tk)
+      return
+    end
+
+    loop do
+        case tk
+        when TkSEMICOLON
+          break
+        when TkLPAREN, TkfLPAREN
+          nest += 1
+        when TkRPAREN
+          nest -= 1
+        when TkCOMMENT
+          if nest <= 0 && @scanner.lex_state == EXPR_END
+            unget_tk(tk)
+            break
+          end
+        when TkNL
+          if (@scanner.lex_state == EXPR_END and nest <= 0) || !@scanner.continue
+            unget_tk(tk)
+            break
+          end
+        end
+        tk = get_tk
+    end
+
+    res = get_tkread.tr("\n", " ").strip
+    res = "" if res == ";"
+
+    con = RDoc::Constant.new name, res, comment
+    read_documentation_modifiers con, RDoc::CONSTANT_MODIFIERS
+
+    if con.document_self
+      container.add_constant(con)
+    end
+  end
+
+  def parse_comment(container, tk, comment)
+    line_no = tk.line_no
+    column  = tk.char_no
+
+    singleton = !!comment.sub!(/(^# +:?)(singleton-)(method:)/, '\1\3')
+
+    if comment.sub!(/^# +:?method: *(\S*).*?\n/i, '') then
+      name = $1 unless $1.empty?
+    else
+      return nil
+    end
+
+    meth = RDoc::GhostMethod.new get_tkread, name
+    meth.singleton = singleton
+
+    @stats.add_method meth
+
+    meth.start_collecting_tokens
+    indent = TkSPACE.new 1, 1
+    indent.set_text " " * column
+
+    position_comment = TkCOMMENT.new(line_no, 1, "# File #{@top_level.file_absolute_name}, line #{line_no}")
+    meth.add_tokens [position_comment, NEWLINE_TOKEN, indent]
+
+    meth.params = ''
+
+    extract_call_seq comment, meth
+
+    container.add_method meth if meth.document_self
+
+    meth.comment = comment
+  end
+
+  def parse_include(context, comment)
+    loop do
+      skip_tkspace_comment
+
+      name = get_constant_with_optional_parens
+      context.add_include RDoc::Include.new(name, comment) unless name.empty?
+
+      return unless TkCOMMA === peek_tk
+      get_tk
+    end
+  end
+
+  ##
+  # Parses a meta-programmed method
+
+  def parse_meta_method(container, single, tk, comment)
+    line_no = tk.line_no
+    column  = tk.char_no
+
+    start_collecting_tokens
+    add_token tk
+    add_token_listener self
+
+    skip_tkspace false
+
+    singleton = !!comment.sub!(/(^# +:?)(singleton-)(method:)/, '\1\3')
+
+    if comment.sub!(/^# +:?method: *(\S*).*?\n/i, '') then
+      name = $1 unless $1.empty?
+    end
+
+    if name.nil? then
+      name_t = get_tk
+      case name_t
+      when TkSYMBOL then
+        name = name_t.text[1..-1]
+      when TkSTRING then
+        name = name_t.text[1..-2]
+      else
+        warn "#{container.top_level.file_relative_name}:#{name_t.line_no} unknown name token #{name_t.inspect} for meta-method"
+        name = 'unknown'
+      end
+    end
+
+    meth = RDoc::MetaMethod.new get_tkread, name
+    meth.singleton = singleton
+
+    @stats.add_method meth
+
+    remove_token_listener self
+
+    meth.start_collecting_tokens
+    indent = TkSPACE.new 1, 1
+    indent.set_text " " * column
+
+    position_comment = TkCOMMENT.new(line_no, 1, "# File #{@top_level.file_absolute_name}, line #{line_no}")
+    meth.add_tokens [position_comment, NEWLINE_TOKEN, indent]
+    meth.add_tokens @token_stream
+
+    add_token_listener meth
+
+    meth.params = ''
+
+    extract_call_seq comment, meth
+
+    container.add_method meth if meth.document_self
+
+    last_tk = tk
+
+    while tk = get_tk do
+      case tk
+      when TkSEMICOLON then
+        break
+      when TkNL then
+        break unless last_tk and TkCOMMA === last_tk
+      when TkSPACE then
+        # expression continues
+      else
+        last_tk = tk
+      end
+    end
+
+    remove_token_listener meth
+
+    meth.comment = comment
+  end
+
+  ##
+  # Parses a method
+
+  def parse_method(container, single, tk, comment)
+    line_no = tk.line_no
+    column  = tk.char_no
+
+    start_collecting_tokens
+    add_token(tk)
+    add_token_listener(self)
+
+    @scanner.instance_eval do @lex_state = EXPR_FNAME end
+
+    skip_tkspace(false)
+    name_t = get_tk
+    back_tk = skip_tkspace
+    meth = nil
+    added_container = false
+
+    dot = get_tk
+    if TkDOT === dot or TkCOLON2 === dot then
+      @scanner.instance_eval do @lex_state = EXPR_FNAME end
+      skip_tkspace
+      name_t2 = get_tk
+
+      case name_t
+      when TkSELF then
+        name = name_t2.name
+      when TkCONSTANT then
+        name = name_t2.name
+        prev_container = container
+        container = container.find_module_named(name_t.name)
+        unless container then
+          added_container = true
+          obj = name_t.name.split("::").inject(Object) do |state, item|
+            state.const_get(item)
+          end rescue nil
+
+          type = obj.class == Class ? RDoc::NormalClass : RDoc::NormalModule
+
+          unless [Class, Module].include?(obj.class) then
+            warn("Couldn't find #{name_t.name}. Assuming it's a module")
+          end
+
+          if type == RDoc::NormalClass then
+            container = prev_container.add_class(type, name_t.name, obj.superclass.name)
+          else
+            container = prev_container.add_module(type, name_t.name)
+          end
+
+          container.record_location @top_level
+        end
+      else
+        # warn("Unexpected token '#{name_t2.inspect}'")
+        # break
+        skip_method(container)
+        return
+      end
+
+      meth = RDoc::AnyMethod.new(get_tkread, name)
+      meth.singleton = true
+    else
+      unget_tk dot
+      back_tk.reverse_each do |token|
+        unget_tk token
+      end
+      name = name_t.name
+
+      meth = RDoc::AnyMethod.new get_tkread, name
+      meth.singleton = (single == SINGLE)
+    end
+
+    @stats.add_method meth
+
+    remove_token_listener self
+
+    meth.start_collecting_tokens
+    indent = TkSPACE.new 1, 1
+    indent.set_text " " * column
+
+    token = TkCOMMENT.new(line_no, 1, "# File #{@top_level.file_absolute_name}, line #{line_no}")
+    meth.add_tokens [token, NEWLINE_TOKEN, indent]
+    meth.add_tokens @token_stream
+
+    add_token_listener meth
+
+    @scanner.instance_eval do @continue = false end
+    parse_method_parameters meth
+
+    if meth.document_self then
+      container.add_method meth
+    elsif added_container then
+      container.document_self = false
+    end
+
+    # Having now read the method parameters and documentation modifiers, we
+    # now know whether we have to rename #initialize to ::new
+
+    if name == "initialize" && !meth.singleton then
+      if meth.dont_rename_initialize then
+        meth.visibility = :protected
+      else
+        meth.singleton = true
+        meth.name = "new"
+        meth.visibility = :public
+      end
+    end
+
+    parse_statements(container, single, meth)
+
+    remove_token_listener(meth)
+
+    extract_call_seq comment, meth
+
+    meth.comment = comment
+  end
+
+  def parse_method_or_yield_parameters(method = nil,
+                                       modifiers = RDoc::METHOD_MODIFIERS)
+    skip_tkspace(false)
+    tk = get_tk
+
+    # Little hack going on here. In the statement
+    #  f = 2*(1+yield)
+    # We see the RPAREN as the next token, so we need
+    # to exit early. This still won't catch all cases
+    # (such as "a = yield + 1"
+    end_token = case tk
+                when TkLPAREN, TkfLPAREN
+                  TkRPAREN
+                when TkRPAREN
+                  return ""
+                else
+                  TkNL
+                end
+    nest = 0
+
+    loop do
+        case tk
+        when TkSEMICOLON
+          break
+        when TkLBRACE
+          nest += 1
+        when TkRBRACE
+          # we might have a.each {|i| yield i }
+          unget_tk(tk) if nest.zero?
+          nest -= 1
+          break if nest <= 0
+        when TkLPAREN, TkfLPAREN
+          nest += 1
+        when end_token
+          if end_token == TkRPAREN
+            nest -= 1
+            break if @scanner.lex_state == EXPR_END and nest <= 0
+          else
+            break unless @scanner.continue
+          end
+        when method && method.block_params.nil? && TkCOMMENT
+          unget_tk(tk)
+          read_documentation_modifiers(method, modifiers)
+        end
+      tk = get_tk
+    end
+    res = get_tkread.tr("\n", " ").strip
+    res = "" if res == ";"
+    res
+  end
+
+  ##
+  # Capture the method's parameters. Along the way, look for a comment
+  # containing:
+  #
+  #    # yields: ....
+  #
+  # and add this as the block_params for the method
+
+  def parse_method_parameters(method)
+    res = parse_method_or_yield_parameters(method)
+    res = "(" + res + ")" unless res[0] == ?(
+    method.params = res unless method.params
+    if method.block_params.nil?
+      skip_tkspace(false)
+      read_documentation_modifiers method, RDoc::METHOD_MODIFIERS
+    end
+  end
+
+  def parse_module(container, single, tk, comment)
+    container, name_t = get_class_or_module(container)
+
+    name = name_t.name
+
+    mod = container.add_module RDoc::NormalModule, name
+    mod.record_location @top_level
+
+    @stats.add_module mod
+
+    read_documentation_modifiers mod, RDoc::CLASS_MODIFIERS
+    parse_statements(mod)
+    mod.comment = comment
+  end
+
+  def parse_require(context, comment)
+    skip_tkspace_comment
+    tk = get_tk
+    if TkLPAREN === tk then
+      skip_tkspace_comment
+      tk = get_tk
+    end
+
+    name = nil
+    case tk
+    when TkSTRING
+      name = tk.text
+      #    when TkCONSTANT, TkIDENTIFIER, TkIVAR, TkGVAR
+      #      name = tk.name
+    when TkDSTRING
+      warn "Skipping require of dynamic string: #{tk.text}"
+      #   else
+      #     warn "'require' used as variable"
+    end
+    if name
+      context.add_require RDoc::Require.new(name, comment)
+    else
+      unget_tk(tk)
+    end
+  end
+
+  def parse_statements(container, single = NORMAL, current_method = nil,
+                       comment = '')
+    nest = 1
+    save_visibility = container.visibility
+
+    non_comment_seen = true
+
+    while tk = get_tk do
+      keep_comment = false
+
+      non_comment_seen = true unless TkCOMMENT === tk
+
+      case tk
+      when TkNL then
+        skip_tkspace true # Skip blanks and newlines
+        tk = get_tk
+
+        if TkCOMMENT === tk then
+          if non_comment_seen then
+            # Look for RDoc in a comment about to be thrown away
+            parse_comment container, tk, comment unless comment.empty?
+
+            comment = ''
+            non_comment_seen = false
+          end
+
+          while TkCOMMENT === tk do
+            comment << tk.text << "\n"
+            tk = get_tk          # this is the newline
+            skip_tkspace(false)  # leading spaces
+            tk = get_tk
+          end
+
+          unless comment.empty? then
+            look_for_directives_in container, comment
+
+            if container.done_documenting then
+              container.ongoing_visibility = save_visibility
+            end
+          end
+
+          keep_comment = true
+        else
+          non_comment_seen = true
+        end
+
+        unget_tk tk
+        keep_comment = true
+
+      when TkCLASS then
+        if container.document_children then
+          parse_class container, single, tk, comment
+        else
+          nest += 1
+        end
+
+      when TkMODULE then
+        if container.document_children then
+          parse_module container, single, tk, comment
+        else
+          nest += 1
+        end
+
+      when TkDEF then
+        if container.document_self then
+          parse_method container, single, tk, comment
+        else
+          nest += 1
+        end
+
+      when TkCONSTANT then
+        if container.document_self then
+          parse_constant container, single, tk, comment
+        end
+
+      when TkALIAS then
+        if container.document_self then
+          parse_alias container, single, tk, comment
+        end
+
+      when TkYIELD then
+        if current_method.nil? then
+          warn "Warning: yield outside of method" if container.document_self
+        else
+          parse_yield container, single, tk, current_method
+        end
+
+      # Until and While can have a 'do', which shouldn't increase the nesting.
+      # We can't solve the general case, but we can handle most occurrences by
+      # ignoring a do at the end of a line.
+
+      when  TkUNTIL, TkWHILE then
+        nest += 1
+        skip_optional_do_after_expression
+
+      # 'for' is trickier
+      when TkFOR then
+        nest += 1
+        skip_for_variable
+        skip_optional_do_after_expression
+
+      when TkCASE, TkDO, TkIF, TkUNLESS, TkBEGIN then
+        nest += 1
+
+      when TkIDENTIFIER then
+        if nest == 1 and current_method.nil? then
+          case tk.name
+          when 'private', 'protected', 'public', 'private_class_method',
+               'public_class_method', 'module_function' then
+            parse_visibility container, single, tk
+            keep_comment = true
+          when 'attr' then
+            parse_attr container, single, tk, comment
+          when /^attr_(reader|writer|accessor)$/, @options.extra_accessors then
+            parse_attr_accessor container, single, tk, comment
+          when 'alias_method' then
+            if container.document_self then
+              parse_alias container, single, tk, comment
+            end
+          else
+            if container.document_self and comment =~ /\A#\#$/ then
+              parse_meta_method container, single, tk, comment
+            end
+          end
+        end
+
+        case tk.name
+        when "require" then
+          parse_require container, comment
+        when "include" then
+          parse_include container, comment
+        end
+
+      when TkEND then
+        nest -= 1
+        if nest == 0 then
+          read_documentation_modifiers container, RDoc::CLASS_MODIFIERS
+          container.ongoing_visibility = save_visibility
+          return
+        end
+
+      end
+
+      comment = '' unless keep_comment
+
+      begin
+        get_tkread
+        skip_tkspace(false)
+      end while peek_tk == TkNL
+    end
+  end
+
+  def parse_symbol_arg(no = nil)
+    args = []
+    skip_tkspace_comment
+    case tk = get_tk
+    when TkLPAREN
+      loop do
+        skip_tkspace_comment
+        if tk1 = parse_symbol_in_arg
+          args.push tk1
+          break if no and args.size >= no
+        end
+
+        skip_tkspace_comment
+        case tk2 = get_tk
+        when TkRPAREN
+          break
+        when TkCOMMA
+        else
+          warn("unexpected token: '#{tk2.inspect}'") if $DEBUG_RDOC
+          break
+        end
+      end
+    else
+      unget_tk tk
+      if tk = parse_symbol_in_arg
+        args.push tk
+        return args if no and args.size >= no
+      end
+
+      loop do
+        skip_tkspace(false)
+
+        tk1 = get_tk
+        unless TkCOMMA === tk1 then
+          unget_tk tk1
+          break
+        end
+
+        skip_tkspace_comment
+        if tk = parse_symbol_in_arg
+          args.push tk
+          break if no and args.size >= no
+        end
+      end
+    end
+    args
+  end
+
+  def parse_symbol_in_arg
+    case tk = get_tk
+    when TkSYMBOL
+      tk.text.sub(/^:/, '')
+    when TkSTRING
+      eval @read[-1]
+    else
+      warn("Expected symbol or string, got #{tk.inspect}") if $DEBUG_RDOC
+      nil
+    end
+  end
+
+  def parse_toplevel_statements(container)
+    comment = collect_first_comment
+    look_for_directives_in(container, comment)
+    container.comment = comment unless comment.empty?
+    parse_statements container, NORMAL, nil, comment
+  end
+
+  def parse_visibility(container, single, tk)
+    singleton = (single == SINGLE)
+
+    vis_type = tk.name
+
+    vis = case vis_type
+          when 'private'   then :private
+          when 'protected' then :protected
+          when 'public'    then :public
+          when 'private_class_method' then
+            singleton = true
+            :private
+          when 'public_class_method' then
+            singleton = true
+            :public
+          when 'module_function' then
+            singleton = true
+            :public
+          else
+            raise "Invalid visibility: #{tk.name}"
+          end
+
+    skip_tkspace_comment false
+
+    case peek_tk
+      # Ryan Davis suggested the extension to ignore modifiers, because he
+      # often writes
+      #
+      #   protected unless $TESTING
+      #
+    when TkNL, TkUNLESS_MOD, TkIF_MOD, TkSEMICOLON then
+      container.ongoing_visibility = vis
+    else
+      if vis_type == 'module_function' then
+        args = parse_symbol_arg
+        container.set_visibility_for args, :private, false
+
+        module_functions = []
+
+        container.methods_matching args do |m|
+          s_m = m.dup
+          s_m.singleton = true if RDoc::AnyMethod === s_m
+          s_m.visibility = :public
+          module_functions << s_m
+        end
+
+        module_functions.each do |s_m|
+          case s_m
+          when RDoc::AnyMethod then
+            container.add_method s_m
+          when RDoc::Attr then
+            container.add_attribute s_m
+          end
+        end
+      else
+        args = parse_symbol_arg
+        container.set_visibility_for args, vis, singleton
+      end
+    end
+  end
+
+  def parse_yield_parameters
+    parse_method_or_yield_parameters
+  end
+
+  def parse_yield(context, single, tk, method)
+    if method.block_params.nil?
+      get_tkread
+      @scanner.instance_eval{@continue = false}
+      method.block_params = parse_yield_parameters
+    end
+  end
+
+  def peek_read
+    @read.join('')
+  end
+
+  ##
+  # Peek at the next token, but don't remove it from the stream
+
+  def peek_tk
+    unget_tk(tk = get_tk)
+    tk
+  end
+
+  ##
+  # Directives are modifier comments that can appear after class, module, or
+  # method names. For example:
+  #
+  #   def fred # :yields: a, b
+  #
+  # or:
+  #
+  #   class MyClass # :nodoc:
+  #
+  # We return the directive name and any parameters as a two element array
+
+  def read_directive(allowed)
+    tk = get_tk
+    result = nil
+    if TkCOMMENT === tk
+      if tk.text =~ /\s*:?(\w+):\s*(.*)/
+        directive = $1.downcase
+        if allowed.include?(directive)
+          result = [directive, $2]
+        end
+      end
+    else
+      unget_tk(tk)
+    end
+    result
+  end
+
+  def read_documentation_modifiers(context, allow)
+    dir = read_directive(allow)
+
+    case dir[0]
+    when "notnew", "not_new", "not-new" then
+      context.dont_rename_initialize = true
+
+    when "nodoc" then
+      context.document_self = false
+      if dir[1].downcase == "all"
+        context.document_children = false
+      end
+
+    when "doc" then
+      context.document_self = true
+      context.force_documentation = true
+
+    when "yield", "yields" then
+      unless context.params.nil?
+        context.params.sub!(/(,|)\s*&\w+/,'') # remove parameter &proc
+      end
+
+      context.block_params = dir[1]
+
+    when "arg", "args" then
+      context.params = dir[1]
+    end if dir
+  end
+
+  def remove_private_comments(comment)
+    comment.gsub!(/^#--.*?^#\+\+/m, '')
+    comment.sub!(/^#--.*/m, '')
+  end
+
+  def remove_token_listener(obj)
+    @token_listeners.delete(obj)
+  end
+
+  def reset
+    @tokens = []
+    @unget_read = []
+    @read = []
+  end
+
+  def scan
+    reset
+
+    catch(:eof) do
+      catch(:enddoc) do
+        begin
+          parse_toplevel_statements(@top_level)
+        rescue Exception => e
+          $stderr.puts <<-EOF
+
+
+RDoc failure in #{@file_name} at or around line #{@scanner.line_no} column
+#{@scanner.char_no}
+
+Before reporting this, could you check that the file you're documenting
+compiles cleanly--RDoc is not a full Ruby parser, and gets confused easily if
+fed invalid programs.
+
+The internal error was:
+
+          EOF
+
+          e.set_backtrace(e.backtrace[0,4])
+          raise
+        end
+      end
+    end
+
+    @top_level
+  end
+
+  ##
+  # while, until, and for have an optional do
+
+  def skip_optional_do_after_expression
+    skip_tkspace(false)
+    tk = get_tk
+    case tk
+    when TkLPAREN, TkfLPAREN
+      end_token = TkRPAREN
+    else
+      end_token = TkNL
+    end
+
+    nest = 0
+    @scanner.instance_eval{@continue = false}
+
+    loop do
+      case tk
+      when TkSEMICOLON
+        break
+      when TkLPAREN, TkfLPAREN
+        nest += 1
+      when TkDO
+        break if nest.zero?
+      when end_token
+        if end_token == TkRPAREN
+          nest -= 1
+          break if @scanner.lex_state == EXPR_END and nest.zero?
+        else
+          break unless @scanner.continue
+        end
+      end
+      tk = get_tk
+    end
+    skip_tkspace(false)
+
+    get_tk if TkDO === peek_tk
+  end
+
+  ##
+  # skip the var [in] part of a 'for' statement
+
+  def skip_for_variable
+    skip_tkspace(false)
+    tk = get_tk
+    skip_tkspace(false)
+    tk = get_tk
+    unget_tk(tk) unless TkIN === tk
+  end
+
+  def skip_method(container)
+    meth = RDoc::AnyMethod.new "", "anon"
+    parse_method_parameters(meth)
+    parse_statements(container, false, meth)
+  end
+
+  ##
+  # Skip spaces
+
+  def skip_tkspace(skip_nl = true)
+    tokens = []
+
+    while TkSPACE === (tk = get_tk) or (skip_nl and TkNL === tk) do
+      tokens.push tk
+    end
+
+    unget_tk(tk)
+    tokens
+  end
+
+  ##
+  # Skip spaces until a comment is found
+
+  def skip_tkspace_comment(skip_nl = true)
+    loop do
+      skip_tkspace(skip_nl)
+      return unless TkCOMMENT === peek_tk
+      get_tk
+    end
+  end
+
+  def unget_tk(tk)
+    @tokens.unshift tk
+    @unget_read.unshift @read.pop
+
+    # Remove this token from any listeners
+    @token_listeners.each do |obj|
+      obj.pop_token
+    end if @token_listeners
+  end
+
+  def warn(msg)
+    return if @options.quiet
+    msg = make_message msg
+    $stderr.puts msg
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/parser/simple.rb b/trunk/lib/rdoc/parser/simple.rb
new file mode 100644
index 0000000000..6e123a4655
--- /dev/null
+++ b/trunk/lib/rdoc/parser/simple.rb
@@ -0,0 +1,38 @@
+require 'rdoc/parser'
+
+##
+# Parse a non-source file. We basically take the whole thing as one big
+# comment. If the first character in the file is '#', we strip leading pound
+# signs.
+
+class RDoc::Parser::Simple < RDoc::Parser
+
+  parse_files_matching(//)
+
+  ##
+  # Prepare to parse a plain file
+
+  def initialize(top_level, file_name, content, options, stats)
+    super
+
+    preprocess = RDoc::Markup::PreProcess.new @file_name, @options.rdoc_include
+
+    preprocess.handle @content do |directive, param|
+      warn "Unrecognized directive '#{directive}' in #{@file_name}"
+    end
+  end
+
+  ##
+  # Extract the file contents and attach them to the toplevel as a comment
+
+  def scan
+    @top_level.comment = remove_private_comments(@content)
+    @top_level
+  end
+
+  def remove_private_comments(comment)
+    comment.gsub(/^--[^-].*?^\+\+/m, '').sub(/^--.*/m, '')
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/rdoc.rb b/trunk/lib/rdoc/rdoc.rb
new file mode 100644
index 0000000000..bc0a32f407
--- /dev/null
+++ b/trunk/lib/rdoc/rdoc.rb
@@ -0,0 +1,293 @@
+require 'rdoc'
+
+require 'rdoc/parser'
+
+# Simple must come first
+require 'rdoc/parser/simple'
+require 'rdoc/parser/ruby'
+require 'rdoc/parser/c'
+require 'rdoc/parser/f95'
+
+require 'rdoc/stats'
+require 'rdoc/options'
+
+require 'rdoc/diagram'
+
+require 'find'
+require 'fileutils'
+require 'time'
+
+module RDoc
+
+  ##
+  # Encapsulate the production of rdoc documentation. Basically you can use
+  # this as you would invoke rdoc from the command line:
+  #
+  #   rdoc = RDoc::RDoc.new
+  #   rdoc.document(args)
+  #
+  # Where +args+ is an array of strings, each corresponding to an argument
+  # you'd give rdoc on the command line. See rdoc/rdoc.rb for details.
+
+  class RDoc
+
+    Generator = Struct.new(:file_name, :class_name, :key)
+
+    ##
+    # Accessor for statistics.  Available after each call to parse_files
+
+    attr_reader :stats
+
+    ##
+    # This is the list of output generator that we support
+
+    GENERATORS = {}
+
+    $LOAD_PATH.collect do |d|
+      File.expand_path d
+    end.find_all do |d|
+      File.directory? "#{d}/rdoc/generator"
+    end.each do |dir|
+      Dir.entries("#{dir}/rdoc/generator").each do |gen|
+        next unless /(\w+)\.rb$/ =~ gen
+        type = $1
+        unless GENERATORS.has_key? type
+          GENERATORS[type] = Generator.new("rdoc/generator/#{gen}",
+                                           "#{type.upcase}".intern,
+                                           type)
+        end
+      end
+    end
+
+    def initialize
+      @stats = nil
+    end
+
+    ##
+    # Report an error message and exit
+
+    def error(msg)
+      raise ::RDoc::Error, msg
+    end
+
+    ##
+    # Create an output dir if it doesn't exist. If it does exist, but doesn't
+    # contain the flag file <tt>created.rid</tt> then we refuse to use it, as
+    # we may clobber some manually generated documentation
+
+    def setup_output_dir(op_dir, force)
+      flag_file = output_flag_file(op_dir)
+      if File.exist?(op_dir)
+        unless File.directory?(op_dir)
+          error "'#{op_dir}' exists, and is not a directory"
+        end
+        begin
+          created = File.read(flag_file)
+        rescue SystemCallError
+          error "\nDirectory #{op_dir} already exists, but it looks like it\n" +
+            "isn't an RDoc directory. Because RDoc doesn't want to risk\n" +
+            "destroying any of your existing files, you'll need to\n" +
+            "specify a different output directory name (using the\n" +
+            "--op <dir> option).\n\n"
+        else
+          last = (Time.parse(created) unless force rescue nil)
+        end
+      else
+        FileUtils.mkdir_p(op_dir)
+      end
+      last
+    end
+
+    ##
+    # Update the flag file in an output directory.
+
+    def update_output_dir(op_dir, time)
+      File.open(output_flag_file(op_dir), "w") {|f| f.puts time.rfc2822 }
+    end
+
+    ##
+    # Return the path name of the flag file in an output directory.
+
+    def output_flag_file(op_dir)
+      File.join(op_dir, "created.rid")
+    end
+
+    ##
+    # The .document file contains a list of file and directory name patterns,
+    # representing candidates for documentation. It may also contain comments
+    # (starting with '#')
+
+    def parse_dot_doc_file(in_dir, filename, options)
+      # read and strip comments
+      patterns = File.read(filename).gsub(/#.*/, '')
+
+      result = []
+
+      patterns.split.each do |patt|
+        candidates = Dir.glob(File.join(in_dir, patt))
+        result.concat(normalized_file_list(options,  candidates))
+      end
+      result
+    end
+
+    ##
+    # Given a list of files and directories, create a list of all the Ruby
+    # files they contain.
+    #
+    # If +force_doc+ is true we always add the given files, if false, only
+    # add files that we guarantee we can parse.  It is true when looking at
+    # files given on the command line, false when recursing through
+    # subdirectories.
+    #
+    # The effect of this is that if you want a file with a non-standard
+    # extension parsed, you must name it explicitly.
+
+    def normalized_file_list(options, relative_files, force_doc = false,
+                             exclude_pattern = nil)
+      file_list = []
+
+      relative_files.each do |rel_file_name|
+        next if exclude_pattern && exclude_pattern =~ rel_file_name
+        stat = File.stat(rel_file_name)
+        case type = stat.ftype
+        when "file"
+          next if @last_created and stat.mtime < @last_created
+
+          if force_doc or ::RDoc::Parser.can_parse(rel_file_name) then
+            file_list << rel_file_name.sub(/^\.\//, '')
+          end
+        when "directory"
+          next if rel_file_name == "CVS" || rel_file_name == ".svn"
+          dot_doc = File.join(rel_file_name, DOT_DOC_FILENAME)
+          if File.file?(dot_doc)
+            file_list.concat(parse_dot_doc_file(rel_file_name, dot_doc, options))
+          else
+            file_list.concat(list_files_in_directory(rel_file_name, options))
+          end
+        else
+          raise RDoc::Error, "I can't deal with a #{type} #{rel_file_name}"
+        end
+      end
+
+      file_list
+    end
+
+    ##
+    # Return a list of the files to be processed in a directory. We know that
+    # this directory doesn't have a .document file, so we're looking for real
+    # files. However we may well contain subdirectories which must be tested
+    # for .document files.
+
+    def list_files_in_directory(dir, options)
+      files = Dir.glob File.join(dir, "*")
+
+      normalized_file_list options, files, false, options.exclude
+    end
+
+    ##
+    # Parse each file on the command line, recursively entering directories.
+
+    def parse_files(options)
+      @stats = Stats.new options.verbosity
+      
+      files = options.files
+      files = ["."] if files.empty?
+
+      file_list = normalized_file_list(options, files, true)
+
+      return [] if file_list.empty?
+
+      file_info = []
+
+      file_list.each do |filename|
+        @stats.add_file filename
+
+        content = if RUBY_VERSION >= '1.9' then
+                    File.open(filename, "r:ascii-8bit") { |f| f.read }
+                  else
+                    File.read filename
+                  end
+
+        if defined? Encoding then
+          if /coding:\s*(\S+)/ =~ content[/\A(?:.*\n){0,2}/]
+            if enc = ::Encoding.find($1)
+              content.force_encoding(enc)
+            end
+          end
+        end
+
+        top_level = ::RDoc::TopLevel.new filename
+
+        parser = ::RDoc::Parser.for top_level, filename, content, options,
+                                    @stats
+
+        file_info << parser.scan
+      end
+
+      file_info
+    end
+
+    ##
+    # Format up one or more files according to the given arguments.
+    #
+    # For simplicity, _argv_ is an array of strings, equivalent to the strings
+    # that would be passed on the command line. (This isn't a coincidence, as
+    # we _do_ pass in ARGV when running interactively). For a list of options,
+    # see rdoc/rdoc.rb. By default, output will be stored in a directory
+    # called +doc+ below the current directory, so make sure you're somewhere
+    # writable before invoking.
+    #
+    # Throws: RDoc::Error on error
+
+    def document(argv)
+      TopLevel::reset
+
+      @options = Options.new GENERATORS
+      @options.parse argv
+
+      @last_created = nil
+
+      unless @options.all_one_file then
+        @last_created = setup_output_dir @options.op_dir, @options.force_update
+      end
+
+      start_time = Time.now
+
+      file_info = parse_files @options
+
+      @options.title = "RDoc Documentation"
+
+      if file_info.empty?
+        $stderr.puts "\nNo newer files." unless @options.quiet
+      else
+        @gen = @options.generator
+
+        $stderr.puts "\nGenerating #{@gen.key.upcase}..." unless @options.quiet
+
+        require @gen.file_name
+
+        gen_class = ::RDoc::Generator.const_get @gen.class_name
+        @gen = gen_class.for @options
+
+        pwd = Dir.pwd
+
+        Dir.chdir @options.op_dir unless @options.all_one_file
+
+        begin
+          Diagram.new(file_info, @options).draw if @options.diagram
+          @gen.generate(file_info)
+          update_output_dir(".", start_time)
+        ensure
+          Dir.chdir(pwd)
+        end
+      end
+
+      unless @options.quiet
+        puts
+        @stats.print
+      end
+    end
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/ri.rb b/trunk/lib/rdoc/ri.rb
new file mode 100644
index 0000000000..a3a858e673
--- /dev/null
+++ b/trunk/lib/rdoc/ri.rb
@@ -0,0 +1,8 @@
+require 'rdoc'
+
+module RDoc::RI
+
+  class Error < RDoc::Error; end
+
+end
+
diff --git a/trunk/lib/rdoc/ri/cache.rb b/trunk/lib/rdoc/ri/cache.rb
new file mode 100644
index 0000000000..2e267d95fb
--- /dev/null
+++ b/trunk/lib/rdoc/ri/cache.rb
@@ -0,0 +1,188 @@
+require 'rdoc/ri'
+
+class RDoc::RI::ClassEntry
+
+  attr_reader :name
+  attr_reader :path_names
+
+  def initialize(path_name, name, in_class)
+    @path_names = [ path_name ]
+    @name = name
+    @in_class = in_class
+    @class_methods    = []
+    @instance_methods = []
+    @inferior_classes = []
+  end
+
+  # We found this class in more tha one place, so add
+  # in the name from there.
+  def add_path(path)
+    @path_names << path
+  end
+
+  # read in our methods and any classes
+  # and modules in our namespace. Methods are
+  # stored in files called name-c|i.yaml,
+  # where the 'name' portion is the external
+  # form of the method name and the c|i is a class|instance
+  # flag
+
+  def load_from(dir)
+    Dir.foreach(dir) do |name|
+      next if name =~ /^\./
+
+      # convert from external to internal form, and
+      # extract the instance/class flag
+
+      if name =~ /^(.*?)-(c|i).yaml$/
+        external_name = $1
+        is_class_method = $2 == "c"
+        internal_name = RiWriter.external_to_internal(external_name)
+        list = is_class_method ? @class_methods : @instance_methods
+        path = File.join(dir, name)
+        list << MethodEntry.new(path, internal_name, is_class_method, self)
+      else
+        full_name = File.join(dir, name)
+        if File.directory?(full_name)
+          inf_class = @inferior_classes.find {|c| c.name == name }
+          if inf_class
+            inf_class.add_path(full_name)
+          else
+            inf_class = ClassEntry.new(full_name, name, self)
+            @inferior_classes << inf_class
+          end
+          inf_class.load_from(full_name)
+        end
+      end
+    end
+  end
+
+  # Return a list of any classes or modules that we contain
+  # that match a given string
+
+  def contained_modules_matching(name)
+    @inferior_classes.find_all {|c| c.name[name]}
+  end
+
+  def classes_and_modules
+    @inferior_classes
+  end
+
+  # Return an exact match to a particular name
+  def contained_class_named(name)
+    @inferior_classes.find {|c| c.name == name}
+  end
+
+  # return the list of local methods matching name
+  # We're split into two because we need distinct behavior
+  # when called from the _toplevel_
+  def methods_matching(name, is_class_method)
+    local_methods_matching(name, is_class_method)
+  end
+
+  # Find methods matching 'name' in ourselves and in
+  # any classes we contain
+  def recursively_find_methods_matching(name, is_class_method)
+    res = local_methods_matching(name, is_class_method)
+    @inferior_classes.each do |c|
+      res.concat(c.recursively_find_methods_matching(name, is_class_method))
+    end
+    res
+  end
+
+
+  # Return our full name
+  def full_name
+    res = @in_class.full_name
+    res << "::" unless res.empty?
+    res << @name
+  end
+
+  # Return a list of all out method names
+  def all_method_names
+    res = @class_methods.map {|m| m.full_name }
+    @instance_methods.each {|m| res << m.full_name}
+    res
+  end
+
+  private
+
+  # Return a list of all our methods matching a given string.
+  # Is +is_class_methods+ if 'nil', we don't care if the method
+  # is a class method or not, otherwise we only return
+  # those methods that match
+  def local_methods_matching(name, is_class_method)
+
+    list = case is_class_method
+           when nil then  @class_methods + @instance_methods
+           when true then @class_methods
+           when false then @instance_methods
+           else fail "Unknown is_class_method: #{is_class_method.inspect}"
+           end
+
+    list.find_all {|m| m.name;  m.name[name]}
+  end
+end
+
+##
+# A TopLevelEntry is like a class entry, but when asked to search for methods
+# searches all classes, not just itself
+
+class RDoc::RI::TopLevelEntry < RDoc::RI::ClassEntry
+  def methods_matching(name, is_class_method)
+    res = recursively_find_methods_matching(name, is_class_method)
+  end
+
+  def full_name
+      ""
+  end
+
+  def module_named(name)
+
+  end
+
+end
+
+class RDoc::RI::MethodEntry
+  attr_reader :name
+  attr_reader :path_name
+
+  def initialize(path_name, name, is_class_method, in_class)
+    @path_name = path_name
+    @name = name
+    @is_class_method = is_class_method
+    @in_class = in_class
+  end
+
+  def full_name
+    res = @in_class.full_name
+    unless res.empty?
+      if @is_class_method
+        res << "::"
+      else
+        res << "#"
+      end
+    end
+    res << @name
+  end
+end
+
+##
+# We represent everything know about all 'ri' files accessible to this program
+
+class RDoc::RI::Cache
+
+  attr_reader :toplevel
+
+  def initialize(dirs)
+    # At the top level we have a dummy module holding the
+    # overall namespace
+    @toplevel = RDoc::RI::TopLevelEntry.new('', '::', nil)
+
+    dirs.each do |dir|
+      @toplevel.load_from(dir)
+    end
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/ri/descriptions.rb b/trunk/lib/rdoc/ri/descriptions.rb
new file mode 100644
index 0000000000..0d8560323a
--- /dev/null
+++ b/trunk/lib/rdoc/ri/descriptions.rb
@@ -0,0 +1,153 @@
+require 'yaml'
+require 'rdoc/markup/fragments'
+require 'rdoc/ri'
+
+##
+# Descriptions are created by RDoc (in ri_generator) and written out in
+# serialized form into the documentation tree. ri then reads these to generate
+# the documentation
+
+class RDoc::RI::NamedThing
+  attr_reader :name
+  def initialize(name)
+    @name = name
+  end
+
+  def <=>(other)
+    @name <=> other.name
+  end
+
+  def hash
+    @name.hash
+  end
+
+  def eql?(other)
+    @name.eql?(other)
+  end
+end
+
+class RDoc::RI::AliasName < RDoc::RI::NamedThing; end
+
+class RDoc::RI::Attribute < RDoc::RI::NamedThing
+  attr_reader :rw, :comment
+
+  def initialize(name, rw, comment)
+    super(name)
+    @rw = rw
+    @comment = comment
+  end
+end
+
+class RDoc::RI::Constant < RDoc::RI::NamedThing
+  attr_reader :value, :comment
+
+  def initialize(name, value, comment)
+    super(name)
+    @value = value
+    @comment = comment
+  end
+end
+
+class RDoc::RI::IncludedModule < RDoc::RI::NamedThing; end
+
+class RDoc::RI::MethodSummary < RDoc::RI::NamedThing
+  def initialize(name="")
+    super
+  end
+end
+
+class RDoc::RI::Description
+  attr_accessor :name
+  attr_accessor :full_name
+  attr_accessor :comment
+
+  def serialize
+    self.to_yaml
+  end
+
+  def self.deserialize(from)
+    YAML.load(from)
+  end
+
+  def <=>(other)
+    @name <=> other.name
+  end
+end
+
+class RDoc::RI::ModuleDescription < RDoc::RI::Description
+
+  attr_accessor :class_methods
+  attr_accessor :instance_methods
+  attr_accessor :attributes
+  attr_accessor :constants
+  attr_accessor :includes
+
+  # merge in another class description into this one
+  def merge_in(old)
+    merge(@class_methods, old.class_methods)
+    merge(@instance_methods, old.instance_methods)
+    merge(@attributes, old.attributes)
+    merge(@constants, old.constants)
+    merge(@includes, old.includes)
+    if @comment.nil? || @comment.empty?
+      @comment = old.comment
+    else
+      unless old.comment.nil? or old.comment.empty? then
+        if @comment.nil? or @comment.empty? then
+          @comment = old.comment
+        else
+          @comment << RDoc::Markup::Flow::RULE.new
+          @comment.concat old.comment
+        end
+      end
+    end
+  end
+
+  def display_name
+      "Module"
+  end
+
+  # the 'ClassDescription' subclass overrides this
+  # to format up the name of a parent
+  def superclass_string
+    nil
+  end
+
+  private
+
+  def merge(into, from)
+    names = {}
+    into.each {|i| names[i.name] = i }
+    from.each {|i| names[i.name] = i }
+    into.replace(names.keys.sort.map {|n| names[n]})
+  end
+end
+
+class RDoc::RI::ClassDescription < RDoc::RI::ModuleDescription
+  attr_accessor :superclass
+
+  def display_name
+      "Class"
+  end
+
+  def superclass_string
+    if @superclass && @superclass != "Object"
+      @superclass
+    else
+      nil
+    end
+  end
+end
+
+class RDoc::RI::MethodDescription < RDoc::RI::Description
+
+  attr_accessor :is_class_method
+  attr_accessor :visibility
+  attr_accessor :block_params
+  attr_accessor :is_singleton
+  attr_accessor :aliases
+  attr_accessor :is_alias_for
+  attr_accessor :params
+
+end
+
diff --git a/trunk/lib/rdoc/ri/display.rb b/trunk/lib/rdoc/ri/display.rb
new file mode 100644
index 0000000000..379cef11b3
--- /dev/null
+++ b/trunk/lib/rdoc/ri/display.rb
@@ -0,0 +1,274 @@
+require 'rdoc/ri'
+
+##
+# This is a kind of 'flag' module. If you want to write your own 'ri' display
+# module (perhaps because you're writing an IDE), you write a class which
+# implements the various 'display' methods in RDoc::RI::DefaultDisplay, and
+# include the RDoc::RI::Display module in that class.
+#
+# To access your class from the command line, you can do
+#
+#    ruby -r <your source file>  ../ri ....
+
+module RDoc::RI::Display
+
+  @@display_class = nil
+
+  def self.append_features(display_class)
+    @@display_class = display_class
+  end
+
+  def self.new(*args)
+    @@display_class.new(*args)
+  end
+
+end
+
+##
+# A paging display module. Uses the RDoc::RI::Formatter class to do the actual
+# presentation.
+
+class RDoc::RI::DefaultDisplay
+
+  include RDoc::RI::Display
+
+  def initialize(formatter, width, use_stdout, output = $stdout)
+    @use_stdout = use_stdout
+    @formatter = formatter.new output, width, "     "
+  end
+
+  ##
+  # Display information about +klass+.  Fetches additional information from
+  # +ri_reader+ as necessary.
+
+  def display_class_info(klass, ri_reader)
+    page do
+      superclass = klass.superclass_string
+
+      if superclass
+        superclass = " < " + superclass
+      else
+        superclass = ""
+      end
+
+      @formatter.draw_line(klass.display_name + ": " +
+                           klass.full_name + superclass)
+
+      display_flow(klass.comment)
+      @formatter.draw_line
+
+      unless klass.includes.empty?
+        @formatter.blankline
+        @formatter.display_heading("Includes:", 2, "")
+        incs = []
+        klass.includes.each do |inc|
+          inc_desc = ri_reader.find_class_by_name(inc.name)
+          if inc_desc
+            str = inc.name + "("
+            str << inc_desc.instance_methods.map{|m| m.name}.join(", ")
+            str << ")"
+            incs << str
+          else
+            incs << inc.name
+          end
+      end
+        @formatter.wrap(incs.sort.join(', '))
+      end
+
+      unless klass.constants.empty?
+        @formatter.blankline
+        @formatter.display_heading("Constants:", 2, "")
+
+        constants = klass.constants.sort_by { |constant| constant.name }
+
+        constants.each do |constant|
+          if constant.comment then
+            @formatter.wrap "#{constant.name}:"
+
+            @formatter.indent do
+              @formatter.display_flow constant.comment
+            end
+          else
+            @formatter.wrap constant.name
+          end
+        end
+      end
+
+      class_data = [
+        :class_methods,
+        :class_method_extensions,
+        :instance_methods,
+        :instance_method_extensions,
+      ]
+
+      class_data.each do |data_type|
+        data = klass.send data_type
+
+        unless data.empty? then
+          @formatter.blankline
+
+          heading = data_type.to_s.split('_').join(' ').capitalize << ':'
+          @formatter.display_heading heading, 2, ''
+
+          data = data.map { |item| item.name }.sort.join ', '
+          @formatter.wrap data
+        end
+      end
+
+      unless klass.attributes.empty? then
+        @formatter.blankline
+
+        @formatter.display_heading 'Attributes:', 2, ''
+
+        attributes = klass.attributes.sort_by { |attribute| attribute.name }
+
+        attributes.each do |attribute|
+          if attribute.comment then
+            @formatter.wrap "#{attribute.name} (#{attribute.rw}):"
+            @formatter.indent do
+              @formatter.display_flow attribute.comment
+            end
+          else
+            @formatter.wrap "#{attribute.name} (#{attribute.rw})"
+          end
+        end
+      end
+    end
+  end
+
+  ##
+  # Display an Array of RDoc::Markup::Flow objects, +flow+.
+
+  def display_flow(flow)
+    if flow and not flow.empty? then
+      @formatter.display_flow flow
+    else
+      @formatter.wrap '[no description]'
+    end
+  end
+
+  ##
+  # Display information about +method+.
+
+  def display_method_info(method)
+    page do
+      @formatter.draw_line(method.full_name)
+      display_params(method)
+
+      @formatter.draw_line
+      display_flow(method.comment)
+
+      if method.aliases and not method.aliases.empty? then
+        @formatter.blankline
+        aka = "(also known as #{method.aliases.map { |a| a.name }.join(', ')})"
+        @formatter.wrap aka
+      end
+    end
+  end
+
+  ##
+  # Display the list of +methods+.
+
+  def display_method_list(methods)
+    page do
+      @formatter.wrap "More than one method matched your request.  You can refine your search by asking for information on one of:"
+
+      @formatter.blankline
+
+      @formatter.wrap methods.map { |m| m.full_name }.join(", ")
+    end
+  end
+
+  ##
+  # Display the params for +method+.
+
+  def display_params(method)
+    params = method.params
+
+    if params[0,1] == "(" then
+      if method.is_singleton
+        params = method.full_name + params
+      else
+        params = method.name + params
+      end
+    end
+
+    params.split(/\n/).each do |param|
+      @formatter.wrap param
+      @formatter.break_to_newline
+    end
+
+    if method.source_path then
+      @formatter.blankline
+      @formatter.wrap("Extension from #{method.source_path}")
+    end
+  end
+
+  ##
+  # List the classes in +classes+.
+
+  def list_known_classes(classes)
+    if classes.empty?
+      warn_no_database
+    else
+      page do
+        @formatter.draw_line "Known classes and modules"
+        @formatter.blankline
+
+        @formatter.wrap classes.sort.join(', ')
+      end
+    end
+  end
+
+  ##
+  # Paginates output through a pager program.
+
+  def page
+    if pager = setup_pager then
+      begin
+        orig_output = @formatter.output
+        @formatter.output = pager
+        yield
+      ensure
+        @formatter.output = orig_output
+        pager.close
+      end
+    else
+      yield
+    end
+  rescue Errno::EPIPE
+  end
+
+  ##
+  # Sets up a pager program to pass output through.
+
+  def setup_pager
+    unless @use_stdout then
+      for pager in [ ENV['PAGER'], "less", "more", 'pager' ].compact.uniq
+        return IO.popen(pager, "w") rescue nil
+      end
+      @use_stdout = true
+      nil
+    end
+  end
+
+  ##
+  # Displays a message that describes how to build RI data.
+
+  def warn_no_database
+    output = @formatter.output
+
+    output.puts "No ri data found"
+    output.puts
+    output.puts "If you've installed Ruby yourself, you need to generate documentation using:"
+    output.puts
+    output.puts "  make install-doc"
+    output.puts
+    output.puts "from the same place you ran `make` to build ruby."
+    output.puts
+    output.puts "If you installed Ruby from a packaging system, then you may need to"
+    output.puts "install an additional package, or ask the packager to enable ri generation."
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/ri/driver.rb b/trunk/lib/rdoc/ri/driver.rb
new file mode 100644
index 0000000000..dfc5f2f98a
--- /dev/null
+++ b/trunk/lib/rdoc/ri/driver.rb
@@ -0,0 +1,551 @@
+require 'optparse'
+require 'yaml'
+
+require 'rdoc/ri'
+require 'rdoc/ri/paths'
+require 'rdoc/ri/formatter'
+require 'rdoc/ri/display'
+require 'fileutils'
+require 'rdoc/markup'
+require 'rdoc/markup/to_flow'
+
+class RDoc::RI::Driver
+
+  class Hash < ::Hash
+    def self.convert(hash)
+      hash = new.update hash
+
+      hash.each do |key, value|
+        hash[key] = case value
+                    when ::Hash then
+                      convert value
+                    when Array then
+                      value = value.map do |v|
+                        ::Hash === v ? convert(v) : v
+                      end
+                      value
+                    else
+                      value
+                    end
+      end
+
+      hash
+    end
+
+    def method_missing method, *args
+      self[method.to_s]
+    end
+
+    def merge_enums(other)
+      other.each do |k, v|
+        if self[k] then
+          case v
+          when Array then
+            # HACK dunno
+            if String === self[k] and self[k].empty? then
+              self[k] = v
+            else
+              self[k] += v
+            end
+          when Hash then
+            self[k].update v
+          else
+            # do nothing
+          end
+        else
+          self[k] = v
+        end
+      end
+    end
+  end
+
+  class Error < RDoc::RI::Error; end
+
+  class NotFoundError < Error
+    def message
+      "Nothing known about #{super}"
+    end
+  end
+
+  attr_accessor :homepath # :nodoc:
+
+  def self.process_args(argv)
+    options = {}
+    options[:use_stdout] = !$stdout.tty?
+    options[:width] = 72
+    options[:formatter] = RDoc::RI::Formatter.for 'plain'
+    options[:list_classes] = false
+    options[:list_names] = false
+
+    # By default all paths are used.  If any of these are true, only those
+    # directories are used.
+    use_system = false
+    use_site = false
+    use_home = false
+    use_gems = false
+    doc_dirs = []
+
+    opts = OptionParser.new do |opt|
+      opt.program_name = File.basename $0
+      opt.version = RDoc::VERSION
+      opt.summary_indent = ' ' * 4
+
+      directories = [
+        RDoc::RI::Paths::SYSDIR,
+        RDoc::RI::Paths::SITEDIR,
+        RDoc::RI::Paths::HOMEDIR
+      ]
+
+      if RDoc::RI::Paths::GEMDIRS then
+        Gem.path.each do |dir|
+          directories << "#{dir}/doc/*/ri"
+        end
+      end
+
+      opt.banner = <<-EOT
+Usage: #{opt.program_name} [options] [names...]
+
+Where name can be:
+
+  Class | Class::method | Class#method | Class.method | method
+
+All class names may be abbreviated to their minimum unambiguous form. If a name
+is ambiguous, all valid options will be listed.
+
+The form '.' method matches either class or instance methods, while #method
+matches only instance and ::method matches only class methods.
+
+For example:
+
+    #{opt.program_name} Fil
+    #{opt.program_name} File
+    #{opt.program_name} File.new
+    #{opt.program_name} zip
+
+Note that shell quoting may be required for method names containing
+punctuation:
+
+    #{opt.program_name} 'Array.[]'
+    #{opt.program_name} compact\\!
+
+By default ri searches for documentation in the following directories:
+
+    #{directories.join "\n    "}
+
+Specifying the --system, --site, --home, --gems or --doc-dir options will
+limit ri to searching only the specified directories.
+
+Options may also be set in the 'RI' environment variable.
+      EOT
+
+      opt.separator nil
+      opt.separator "Options:"
+      opt.separator nil
+
+      opt.on("--classes", "-c",
+             "Display the names of classes and modules we",
+             "know about.") do |value|
+        options[:list_classes] = value
+      end
+
+      opt.separator nil
+
+      opt.on("--doc-dir=DIRNAME", "-d", Array,
+             "List of directories to search for",
+             "documentation. If not specified, we search",
+             "the standard rdoc/ri directories. May be",
+             "repeated.") do |value|
+        value.each do |dir|
+          unless File.directory? dir then
+            raise OptionParser::InvalidArgument, "#{dir} is not a directory"
+          end
+        end
+
+        doc_dirs.concat value
+      end
+
+      opt.separator nil
+
+      opt.on("--fmt=FORMAT", "--format=FORMAT", "-f",
+             RDoc::RI::Formatter::FORMATTERS.keys,
+             "Format to use when displaying output:",
+             "   #{RDoc::RI::Formatter.list}",
+             "Use 'bs' (backspace) with most pager",
+             "programs. To use ANSI, either disable the",
+             "pager or tell the pager to allow control",
+             "characters.") do |value|
+        options[:formatter] = RDoc::RI::Formatter.for value
+      end
+
+      opt.separator nil
+
+      unless RDoc::RI::Paths::GEMDIRS.empty? then
+        opt.on("--[no-]gems",
+               "Include documentation from RubyGems.") do |value|
+          use_gems = value
+        end
+      end
+
+      opt.separator nil
+
+      opt.on("--[no-]home",
+             "Include documentation stored in ~/.rdoc.") do |value|
+        use_home = value
+      end
+
+      opt.separator nil
+
+      opt.on("--[no-]list-names", "-l",
+             "List all the names known to RDoc, one per",
+             "line.") do |value|
+        options[:list_names] = value
+      end
+
+      opt.separator nil
+
+      opt.on("--no-pager", "-T",
+             "Send output directly to stdout.") do |value|
+        options[:use_stdout] = !value
+      end
+
+      opt.separator nil
+
+      opt.on("--[no-]site",
+             "Include documentation from libraries",
+             "installed in site_lib.") do |value|
+        use_site = value
+      end
+
+      opt.separator nil
+
+      opt.on("--[no-]system",
+             "Include documentation from Ruby's standard",
+             "library.") do |value|
+        use_system = value
+      end
+
+      opt.separator nil
+
+      opt.on("--width=WIDTH", "-w", OptionParser::DecimalInteger,
+             "Set the width of the output.") do |value|
+        options[:width] = value
+      end
+    end
+
+    argv = ENV['RI'].to_s.split.concat argv
+
+    opts.parse! argv
+
+    options[:names] = argv
+
+    options[:path] = RDoc::RI::Paths.path(use_system, use_site, use_home,
+                                          use_gems, *doc_dirs)
+    options[:raw_path] = RDoc::RI::Paths.raw_path(use_system, use_site,
+                                                  use_home, use_gems, *doc_dirs)
+
+    options
+
+  rescue OptionParser::InvalidArgument, OptionParser::InvalidOption => e
+    puts opts
+    puts
+    puts e
+    exit 1
+  end
+
+  def self.run(argv = ARGV)
+    options = process_args argv
+    ri = new options
+    ri.run
+  end
+
+  def initialize(options={})
+    options[:formatter] ||= RDoc::RI::Formatter.for('plain')
+    options[:use_stdout] ||= !$stdout.tty?
+    options[:width] ||= 72
+    @names = options[:names]
+
+    @class_cache_name = 'classes'
+    @all_dirs = RDoc::RI::Paths.path(true, true, true, true)
+    @homepath = RDoc::RI::Paths.raw_path(false, false, true, false).first
+    @homepath = @homepath.sub(/\.rdoc/, '.ri')
+    @sys_dirs = RDoc::RI::Paths.raw_path(true, false, false, false)
+
+    FileUtils.mkdir_p cache_file_path unless File.directory? cache_file_path
+
+    @class_cache = nil
+
+    @display = RDoc::RI::DefaultDisplay.new(options[:formatter],
+                                            options[:width],
+                                            options[:use_stdout])
+  end
+
+  def class_cache
+    return @class_cache if @class_cache
+
+    newest = map_dirs('created.rid', :all) do |f|
+      File.mtime f if test ?f, f
+    end.max
+
+    up_to_date = (File.exist?(class_cache_file_path) and
+                  newest and newest < File.mtime(class_cache_file_path))
+
+    @class_cache = if up_to_date then
+                     load_cache_for @class_cache_name
+                   else
+                     class_cache = RDoc::RI::Driver::Hash.new
+
+                     classes = map_dirs('**/cdesc*.yaml', :sys) { |f| Dir[f] }
+                     populate_class_cache class_cache, classes
+
+                     classes = map_dirs('**/cdesc*.yaml') { |f| Dir[f] }
+                     warn "Updating class cache with #{classes.size} classes..."
+
+                     populate_class_cache class_cache, classes, true
+                     write_cache class_cache, class_cache_file_path
+                   end
+
+    @class_cache = RDoc::RI::Driver::Hash.convert @class_cache
+    @class_cache
+  end
+
+  def class_cache_file_path
+    File.join cache_file_path, @class_cache_name
+  end
+
+  def cache_file_for(klassname)
+    File.join cache_file_path, klassname.gsub(/:+/, "-")
+  end
+
+  def cache_file_path
+    File.join @homepath, 'cache'
+  end
+
+  def display_class(name)
+    klass = class_cache[name]
+    klass = RDoc::RI::Driver::Hash.convert klass
+    @display.display_class_info klass, class_cache
+  end
+
+  def get_info_for(arg)
+    @names = [arg]
+    run
+  end
+
+  def load_cache_for(klassname)
+    path = cache_file_for klassname
+
+    cache = nil
+
+    if File.exist? path and
+       File.mtime(path) >= File.mtime(class_cache_file_path) then
+      open path, 'rb' do |fp|
+        cache = Marshal.load fp.read
+      end
+    else
+      class_cache = nil
+
+      open class_cache_file_path, 'rb' do |fp|
+        class_cache = Marshal.load fp.read
+      end
+
+      klass = class_cache[klassname]
+      return nil unless klass
+
+      method_files = klass["sources"]
+      cache = RDoc::RI::Driver::Hash.new
+
+      sys_dir = @sys_dirs.first
+      method_files.each do |f|
+        system_file = f.index(sys_dir) == 0
+        Dir[File.join(File.dirname(f), "*")].each do |yaml|
+          next unless yaml =~ /yaml$/
+          next if yaml =~ /cdesc-[^\/]+yaml$/
+          method = read_yaml yaml
+          name = method["full_name"]
+          ext_path = f
+          ext_path = "gem #{$1}" if f =~ %r%gems/[\d.]+/doc/([^/]+)%
+          method["source_path"] = ext_path unless system_file
+          cache[name] = RDoc::RI::Driver::Hash.convert method
+        end
+      end
+
+      write_cache cache, path
+    end
+
+    RDoc::RI::Driver::Hash.convert cache
+  end
+
+  ##
+  # Finds the next ancestor of +orig_klass+ after +klass+.
+
+  def lookup_ancestor(klass, orig_klass)
+    cache = class_cache[orig_klass]
+
+    return nil unless cache
+
+    ancestors = [orig_klass]
+    ancestors.push(*cache.includes.map { |inc| inc['name'] })
+    ancestors << cache.superclass
+
+    ancestor = ancestors[ancestors.index(klass) + 1]
+
+    return ancestor if ancestor
+
+    lookup_ancestor klass, cache.superclass
+  end
+
+  ##
+  # Finds the method
+
+  def lookup_method(name, klass)
+    cache = load_cache_for klass
+    return nil unless cache
+
+    method = cache[name.gsub('.', '#')]
+    method = cache[name.gsub('.', '::')] unless method
+    method
+  end
+
+  def map_dirs(file_name, system=false)
+    dirs = if system == :all then
+             @all_dirs
+           else
+             if system then
+               @sys_dirs
+             else
+               @all_dirs - @sys_dirs
+             end
+           end
+
+    dirs.map { |dir| yield File.join(dir, file_name) }.flatten.compact
+  end
+
+  ##
+  # Extract the class and method name parts from +name+ like Foo::Bar#baz
+
+  def parse_name(name)
+    parts = name.split(/(::|\#|\.)/)
+
+    if parts[-2] != '::' or parts.last !~ /^[A-Z]/ then
+      meth = parts.pop
+      parts.pop
+    end
+
+    klass = parts.join
+
+    [klass, meth]
+  end
+
+  def populate_class_cache(class_cache, classes, extension = false)
+    classes.each do |cdesc|
+      desc = read_yaml cdesc
+      klassname = desc["full_name"]
+
+      unless class_cache.has_key? klassname then
+        desc["display_name"] = "Class"
+        desc["sources"] = [cdesc]
+        desc["instance_method_extensions"] = []
+        desc["class_method_extensions"] = []
+        class_cache[klassname] = desc
+      else
+        klass = class_cache[klassname]
+
+        if extension then
+          desc["instance_method_extensions"] = desc.delete "instance_methods"
+          desc["class_method_extensions"] = desc.delete "class_methods"
+        end
+
+        klass = RDoc::RI::Driver::Hash.convert klass
+
+        klass.merge_enums desc
+        klass["sources"] << cdesc
+      end
+    end
+  end
+
+  def read_yaml(path)
+    data = File.read path
+    data = data.gsub(/ \!ruby\/(object|struct):(RDoc::RI|RI).*/, '')
+    data = data.gsub(/ \!ruby\/(object|struct):SM::(\S+)/,
+                     ' !ruby/\1:RDoc::Markup::\2')
+    YAML.load data
+  end
+
+  def run
+    if @names.empty? then
+      @display.list_known_classes class_cache.keys.sort
+    else
+      @names.each do |name|
+        case name
+        when /::|\#|\./ then
+          if class_cache.key? name then
+            display_class name
+          else
+            klass, = parse_name name
+
+            orig_klass = klass
+            orig_name = name
+
+            until klass == 'Kernel' do
+              method = lookup_method name, klass
+
+              break method if method
+
+              ancestor = lookup_ancestor klass, orig_klass
+
+              break unless ancestor
+
+              name = name.sub klass, ancestor
+              klass = ancestor
+            end
+
+            raise NotFoundError, orig_name unless method
+
+            @display.display_method_info method
+          end
+        else
+          if class_cache.key? name then
+            display_class name
+          else
+            methods = select_methods(/^#{name}/)
+
+            if methods.size == 0
+              raise NotFoundError, name
+            elsif methods.size == 1
+              @display.display_method_info methods.first
+            else
+              @display.display_method_list methods
+            end
+          end
+        end
+      end
+    end
+  rescue NotFoundError => e
+    abort e.message
+  end
+
+  def select_methods(pattern)
+    methods = []
+    class_cache.keys.sort.each do |klass|
+      class_cache[klass]["instance_methods"].map{|h|h["name"]}.grep(pattern) do |name|
+        method = load_cache_for(klass)[klass+'#'+name]
+        methods << method if method
+      end
+      class_cache[klass]["class_methods"].map{|h|h["name"]}.grep(pattern) do |name|
+        method = load_cache_for(klass)[klass+'::'+name]
+        methods << method if method
+      end
+    end
+    methods
+  end
+
+  def write_cache(cache, path)
+    File.open path, "wb" do |cache_file|
+      Marshal.dump cache, cache_file
+    end
+
+    cache
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/ri/formatter.rb b/trunk/lib/rdoc/ri/formatter.rb
new file mode 100644
index 0000000000..0a0c3f7380
--- /dev/null
+++ b/trunk/lib/rdoc/ri/formatter.rb
@@ -0,0 +1,616 @@
+require 'rdoc/ri'
+require 'rdoc/markup'
+
+class RDoc::RI::Formatter
+
+  attr_writer :indent
+  attr_accessor :output
+
+  FORMATTERS = { }
+
+  def self.for(name)
+    FORMATTERS[name.downcase]
+  end
+
+  def self.list
+    FORMATTERS.keys.sort.join ", "
+  end
+
+  def initialize(output, width, indent)
+    @output = output
+    @width  = width
+    @indent = indent
+    @original_indent = indent.dup
+  end
+
+  def draw_line(label=nil)
+    len = @width
+    len -= (label.size + 1) if label
+
+    if len > 0 then
+      @output.print '-' * len
+      if label
+        @output.print ' '
+        bold_print label
+      end
+
+      @output.puts
+    else
+      @output.print '-' * @width
+      @output.puts
+
+      @output.puts label
+    end
+  end
+
+  def indent
+    return @indent unless block_given?
+
+    begin
+      indent = @indent.dup
+      @indent += @original_indent
+      yield
+    ensure
+      @indent = indent
+    end
+  end
+
+  def wrap(txt, prefix=@indent, linelen=@width)
+    return unless txt && !txt.empty?
+
+    work = conv_markup(txt)
+    textLen = linelen - prefix.length
+    patt = Regexp.new("^(.{0,#{textLen}})[ \n]")
+    next_prefix = prefix.tr("^ ", " ")
+
+    res = []
+
+    while work.length > textLen
+      if work =~ patt
+        res << $1
+        work.slice!(0, $&.length)
+      else
+        res << work.slice!(0, textLen)
+      end
+    end
+    res << work if work.length.nonzero?
+    @output.puts(prefix + res.join("\n" + next_prefix))
+  end
+
+  def blankline
+    @output.puts
+  end
+
+  ##
+  # Called when we want to ensure a new 'wrap' starts on a newline.  Only
+  # needed for HtmlFormatter, because the rest do their own line breaking.
+
+  def break_to_newline
+  end
+
+  def bold_print(txt)
+    @output.print txt
+  end
+
+  def raw_print_line(txt)
+    @output.puts txt
+  end
+
+  ##
+  # Convert HTML entities back to ASCII
+
+  def conv_html(txt)
+    txt = txt.gsub(/&gt;/, '>')
+    txt.gsub!(/&lt;/, '<')
+    txt.gsub!(/&quot;/, '"')
+    txt.gsub!(/&amp;/, '&')
+    txt
+  end
+
+  ##
+  # Convert markup into display form
+
+  def conv_markup(txt)
+    txt = txt.gsub(%r{<tt>(.*?)</tt>}, '+\1+')
+    txt.gsub!(%r{<code>(.*?)</code>}, '+\1+')
+    txt.gsub!(%r{<b>(.*?)</b>}, '*\1*')
+    txt.gsub!(%r{<em>(.*?)</em>}, '_\1_') 
+    txt
+  end
+
+  def display_list(list)
+    case list.type
+    when :BULLET
+      prefixer = proc { |ignored| @indent + "*   " }
+
+    when :NUMBER, :UPPERALPHA, :LOWERALPHA then
+      start = case list.type
+              when :NUMBER     then 1
+              when :UPPERALPHA then 'A'
+              when :LOWERALPHA then 'a'
+              end
+
+      prefixer = proc do |ignored|
+        res = @indent + "#{start}.".ljust(4)
+        start = start.succ
+        res
+      end
+
+    when :LABELED, :NOTE then
+      longest = 0
+
+      list.contents.each do |item|
+        if RDoc::Markup::Flow::LI === item and item.label.length > longest then
+          longest = item.label.length
+        end
+      end
+
+      longest += 1
+
+      prefixer = proc { |li| @indent + li.label.ljust(longest) }
+
+    else
+      raise ArgumentError, "unknown list type #{list.type}"
+    end
+
+    list.contents.each do |item|
+      if RDoc::Markup::Flow::LI === item then
+        prefix = prefixer.call item
+        display_flow_item item, prefix
+      else
+        display_flow_item item
+      end
+    end
+  end
+
+  def display_flow_item(item, prefix = @indent)
+    case item
+    when RDoc::Markup::Flow::P, RDoc::Markup::Flow::LI
+      wrap(conv_html(item.body), prefix)
+      blankline
+
+    when RDoc::Markup::Flow::LIST
+      display_list(item)
+
+    when RDoc::Markup::Flow::VERB
+      display_verbatim_flow_item(item, @indent)
+
+    when RDoc::Markup::Flow::H
+      display_heading(conv_html(item.text), item.level, @indent)
+
+    when RDoc::Markup::Flow::RULE
+      draw_line
+
+    else
+      raise RDoc::Error, "Unknown flow element: #{item.class}"
+    end
+  end
+
+  def display_verbatim_flow_item(item, prefix=@indent)
+    item.body.split(/\n/).each do |line|
+      @output.print @indent, conv_html(line), "\n"
+    end
+    blankline
+  end
+
+  def display_heading(text, level, indent)
+    text = strip_attributes text
+
+    case level
+    when 1 then
+      ul = "=" * text.length
+      @output.puts
+      @output.puts text.upcase
+      @output.puts ul
+
+    when 2 then
+      ul = "-" * text.length
+      @output.puts
+      @output.puts text
+      @output.puts ul
+    else
+      @output.print indent, text, "\n"
+    end
+
+    @output.puts
+  end
+
+  def display_flow(flow)
+    flow.each do |f|
+      display_flow_item(f)
+    end
+  end
+
+  def strip_attributes(text)
+    text.gsub(/(<\/?(?:b|code|em|i|tt)>)/, '')
+  end
+
+end
+
+##
+# Handle text with attributes. We're a base class: there are different
+# presentation classes (one, for example, uses overstrikes to handle bold and
+# underlining, while another using ANSI escape sequences.
+
+class RDoc::RI::AttributeFormatter < RDoc::RI::Formatter
+
+  BOLD      = 1
+  ITALIC    = 2
+  CODE      = 4
+
+  ATTR_MAP = {
+    "b"    => BOLD,
+    "code" => CODE,
+    "em"   => ITALIC,
+    "i"    => ITALIC,
+    "tt"   => CODE
+  }
+
+  AttrChar = Struct.new :char, :attr
+
+  class AttributeString
+    attr_reader :txt
+
+    def initialize
+      @txt = []
+      @optr = 0
+    end
+
+    def <<(char)
+      @txt << char
+    end
+
+    def empty?
+      @optr >= @txt.length
+    end
+
+    # accept non space, then all following spaces
+    def next_word
+      start = @optr
+      len = @txt.length
+
+      while @optr < len && @txt[@optr].char != " "
+        @optr += 1
+      end
+
+      while @optr < len && @txt[@optr].char == " "
+        @optr += 1
+      end
+
+      @txt[start...@optr]
+    end
+  end
+
+  ##
+  # Overrides base class.  Looks for <tt>...</tt> etc sequences and generates
+  # an array of AttrChars.  This array is then used as the basis for the
+  # split.
+
+  def wrap(txt, prefix=@indent, linelen=@width)
+    return unless txt && !txt.empty?
+
+    txt = add_attributes_to(txt)
+    next_prefix = prefix.tr("^ ", " ")
+    linelen -= prefix.size
+
+    line = []
+
+    until txt.empty?
+      word = txt.next_word
+      if word.size + line.size > linelen
+        write_attribute_text(prefix, line)
+        prefix = next_prefix
+        line = []
+      end
+      line.concat(word)
+    end
+
+    write_attribute_text(prefix, line) if line.length > 0
+  end
+
+  protected
+
+  def write_attribute_text(prefix, line)
+    @output.print prefix
+    line.each do |achar|
+      @output.print achar.char
+    end
+    @output.puts
+  end
+
+  def bold_print(txt)
+    @output.print txt
+  end
+
+  private
+
+  def add_attributes_to(txt)
+    tokens = txt.split(%r{(</?(?:b|code|em|i|tt)>)})
+    text = AttributeString.new
+    attributes = 0
+    tokens.each do |tok|
+      case tok
+      when %r{^</(\w+)>$} then attributes &= ~(ATTR_MAP[$1]||0)
+      when %r{^<(\w+)>$}  then attributes  |= (ATTR_MAP[$1]||0)
+      else
+        tok.split(//).each {|ch| text << AttrChar.new(ch, attributes)}
+      end
+    end
+    text
+  end
+
+end
+
+##
+# This formatter generates overstrike-style formatting, which works with
+# pagers such as man and less.
+
+class RDoc::RI::OverstrikeFormatter < RDoc::RI::AttributeFormatter
+
+  BS = "\C-h"
+
+  def write_attribute_text(prefix, line)
+    @output.print prefix
+
+    line.each do |achar|
+      attr = achar.attr
+      @output.print "_", BS if (attr & (ITALIC + CODE)) != 0
+      @output.print achar.char, BS if (attr & BOLD) != 0
+      @output.print achar.char
+    end
+
+    @output.puts
+  end
+
+  ##
+  # Draw a string in bold
+
+  def bold_print(text)
+    text.split(//).each do |ch|
+      @output.print ch, BS, ch
+    end
+  end
+
+end
+
+##
+# This formatter uses ANSI escape sequences to colorize stuff works with
+# pagers such as man and less.
+
+class RDoc::RI::AnsiFormatter < RDoc::RI::AttributeFormatter
+
+  def initialize(*args)
+    super
+    @output.print "\033[0m"
+  end
+
+  def write_attribute_text(prefix, line)
+    @output.print prefix
+    curr_attr = 0
+    line.each do |achar|
+      attr = achar.attr
+      if achar.attr != curr_attr
+        update_attributes(achar.attr)
+        curr_attr = achar.attr
+      end
+      @output.print achar.char
+    end
+    update_attributes(0) unless curr_attr.zero?
+    @output.puts
+  end
+
+  def bold_print(txt)
+    @output.print "\033[1m#{txt}\033[m"
+  end
+
+  HEADINGS = {
+    1 => ["\033[1;32m", "\033[m"],
+    2 => ["\033[4;32m", "\033[m"],
+    3 => ["\033[32m",   "\033[m"],
+  }
+
+  def display_heading(text, level, indent)
+    level = 3 if level > 3
+    heading = HEADINGS[level]
+    @output.print indent
+    @output.print heading[0]
+    @output.print strip_attributes(text)
+    @output.puts heading[1]
+  end
+
+  private
+
+  ATTR_MAP = {
+    BOLD   => "1",
+    ITALIC => "33",
+    CODE   => "36"
+  }
+
+  def update_attributes(attr)
+    str = "\033["
+    for quality in [ BOLD, ITALIC, CODE]
+      unless (attr & quality).zero?
+        str << ATTR_MAP[quality]
+      end
+    end
+    @output.print str, "m"
+  end
+
+end
+
+##
+# This formatter uses HTML.
+
+class RDoc::RI::HtmlFormatter < RDoc::RI::AttributeFormatter
+
+  def write_attribute_text(prefix, line)
+    curr_attr = 0
+    line.each do |achar|
+      attr = achar.attr
+      if achar.attr != curr_attr
+        update_attributes(curr_attr, achar.attr)
+        curr_attr = achar.attr
+      end
+      @output.print(escape(achar.char))
+    end
+    update_attributes(curr_attr, 0) unless curr_attr.zero?
+  end
+
+  def draw_line(label=nil)
+    if label != nil
+      bold_print(label)
+    end
+    @output.puts("<hr>")
+  end
+
+  def bold_print(txt)
+    tag("b") { txt }
+  end
+
+  def blankline()
+    @output.puts("<p>")
+  end
+
+  def break_to_newline
+    @output.puts("<br>")
+  end
+
+  def display_heading(text, level, indent)
+    level = 4 if level > 4
+    tag("h#{level}") { text }
+    @output.puts
+  end
+
+  def display_list(list)
+    case list.type
+    when :BULLET then
+      list_type = "ul"
+      prefixer = proc { |ignored| "<li>" }
+
+    when :NUMBER, :UPPERALPHA, :LOWERALPHA then
+      list_type = "ol"
+      prefixer = proc { |ignored| "<li>" }
+
+    when :LABELED then
+      list_type = "dl"
+      prefixer = proc do |li|
+        "<dt><b>" + escape(li.label) + "</b><dd>"
+      end
+
+    when :NOTE then
+      list_type = "table"
+      prefixer = proc do |li|
+        %{<tr valign="top"><td>#{li.label.gsub(/ /, '&nbsp;')}</td><td>}
+      end
+    else
+      fail "unknown list type"
+    end
+
+    @output.print "<#{list_type}>"
+    list.contents.each do |item|
+      if item.kind_of? RDoc::Markup::Flow::LI
+        prefix = prefixer.call(item)
+        @output.print prefix
+        display_flow_item(item, prefix)
+      else
+        display_flow_item(item)
+      end
+    end
+    @output.print "</#{list_type}>"
+  end
+
+  def display_verbatim_flow_item(item, prefix=@indent)
+    @output.print("<pre>")
+    item.body.split(/\n/).each do |line|
+      @output.puts conv_html(line)
+    end
+    @output.puts("</pre>")
+  end
+
+  private
+
+  ATTR_MAP = {
+    BOLD   => "b>",
+    ITALIC => "i>",
+    CODE   => "tt>"
+  }
+
+  def update_attributes(current, wanted)
+    str = ""
+    # first turn off unwanted ones
+    off = current & ~wanted
+    for quality in [ BOLD, ITALIC, CODE]
+      if (off & quality) > 0
+        str << "</" + ATTR_MAP[quality]
+      end
+    end
+
+    # now turn on wanted
+    for quality in [ BOLD, ITALIC, CODE]
+      unless (wanted & quality).zero?
+        str << "<" << ATTR_MAP[quality]
+      end
+    end
+    @output.print str
+  end
+
+  def tag(code)
+    @output.print("<#{code}>")
+    @output.print(yield)
+    @output.print("</#{code}>")
+  end
+
+  def escape(str)
+    str = str.gsub(/&/n, '&amp;')
+    str.gsub!(/\"/n, '&quot;')
+    str.gsub!(/>/n, '&gt;')
+    str.gsub!(/</n, '&lt;')
+    str
+  end
+
+end
+
+##
+# This formatter reduces extra lines for a simpler output.  It improves way
+# output looks for tools like IRC bots.
+
+class RDoc::RI::SimpleFormatter < RDoc::RI::Formatter
+
+  ##
+  # No extra blank lines
+
+  def blankline
+  end
+
+  ##
+  # Display labels only, no lines
+
+  def draw_line(label=nil)
+    unless label.nil? then
+      bold_print(label)
+      @output.puts
+    end
+  end
+
+  ##
+  # Place heading level indicators inline with heading.
+
+  def display_heading(text, level, indent)
+    text = strip_attributes(text)
+    case level
+    when 1
+      @output.puts "= " + text.upcase
+    when 2
+      @output.puts "-- " + text
+    else
+      @output.print indent, text, "\n"
+    end
+  end
+
+end
+
+RDoc::RI::Formatter::FORMATTERS['plain']  = RDoc::RI::Formatter
+RDoc::RI::Formatter::FORMATTERS['simple'] = RDoc::RI::SimpleFormatter
+RDoc::RI::Formatter::FORMATTERS['bs']     = RDoc::RI::OverstrikeFormatter
+RDoc::RI::Formatter::FORMATTERS['ansi']   = RDoc::RI::AnsiFormatter
+RDoc::RI::Formatter::FORMATTERS['html']   = RDoc::RI::HtmlFormatter
diff --git a/trunk/lib/rdoc/ri/paths.rb b/trunk/lib/rdoc/ri/paths.rb
new file mode 100644
index 0000000000..b4b6c64925
--- /dev/null
+++ b/trunk/lib/rdoc/ri/paths.rb
@@ -0,0 +1,102 @@
+require 'rdoc/ri'
+
+##
+# Encapsulate all the strangeness to do with finding out where to find RDoc
+# files
+#
+# We basically deal with three directories:
+#
+# 1. The 'system' documentation directory, which holds the documentation
+#    distributed with Ruby, and which is managed by the Ruby install process
+# 2. The 'site' directory, which contains site-wide documentation added
+#    locally.
+# 3. The 'user' documentation directory, stored under the user's own home
+#    directory.
+#
+# There's contention about all this, but for now:
+#
+# system:: $datadir/ri/<ver>/system/...
+# site::   $datadir/ri/<ver>/site/...
+# user::   ~/.rdoc
+
+module RDoc::RI::Paths
+
+  #:stopdoc:
+  require 'rbconfig'
+
+  DOC_DIR  = "doc/rdoc"
+
+  version = RbConfig::CONFIG['ruby_version']
+
+  base    = File.join(RbConfig::CONFIG['datadir'], "ri", version)
+  SYSDIR  = File.join(base, "system")
+  SITEDIR = File.join(base, "site")
+  homedir = ENV['HOME'] || ENV['USERPROFILE'] || ENV['HOMEPATH']
+
+  if homedir then
+    HOMEDIR = File.join(homedir, ".rdoc")
+  else
+    HOMEDIR = nil
+  end
+
+  # This is the search path for 'ri'
+  PATH = [ SYSDIR, SITEDIR, HOMEDIR ].find_all {|p| p && File.directory?(p)}
+
+  begin
+    require 'rubygems' unless defined?(Gem) and defined?(Gem::Enable) and
+                              Gem::Enable
+
+    # HACK dup'd from Gem.latest_partials and friends
+    all_paths = []
+
+    all_paths = Gem.path.map do |dir|
+      Dir[File.join(dir, 'doc', '*', 'ri')]
+    end.flatten
+
+    ri_paths = {}
+
+    all_paths.each do |dir|
+      base = File.basename File.dirname(dir)
+      if base =~ /(.*)-((\d+\.)*\d+)/ then
+        name, version = $1, $2
+        ver = Gem::Version.new version
+        if ri_paths[name].nil? or ver > ri_paths[name][0] then
+          ri_paths[name] = [ver, dir]
+        end
+      end
+    end
+
+    GEMDIRS = ri_paths.map { |k,v| v.last }.sort
+    GEMDIRS.each { |dir| PATH << dir }
+  rescue LoadError
+    GEMDIRS = []
+  end
+
+  # Returns the selected documentation directories as an Array, or PATH if no
+  # overriding directories were given.
+
+  def self.path(use_system, use_site, use_home, use_gems, *extra_dirs)
+    path = raw_path(use_system, use_site, use_home, use_gems, *extra_dirs)
+    return path.select { |directory| File.directory? directory }
+  end
+
+  # Returns the selected documentation directories including nonexistent
+  # directories.  Used to print out what paths were searched if no ri was
+  # found.
+
+  def self.raw_path(use_system, use_site, use_home, use_gems, *extra_dirs)
+    return PATH unless use_system or use_site or use_home or use_gems or
+                       not extra_dirs.empty?
+
+    path = []
+    path << extra_dirs unless extra_dirs.empty?
+    path << SYSDIR if use_system
+    path << SITEDIR if use_site
+    path << HOMEDIR if use_home
+    path << GEMDIRS if use_gems
+
+    return path.flatten.compact
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/ri/reader.rb b/trunk/lib/rdoc/ri/reader.rb
new file mode 100644
index 0000000000..986bb75954
--- /dev/null
+++ b/trunk/lib/rdoc/ri/reader.rb
@@ -0,0 +1,106 @@
+require 'rdoc/ri'
+require 'rdoc/ri/descriptions'
+require 'rdoc/ri/writer'
+require 'rdoc/markup/to_flow'
+
+class RDoc::RI::Reader
+
+  def initialize(ri_cache)
+    @cache = ri_cache
+  end
+
+  def top_level_namespace
+    [ @cache.toplevel ]
+  end
+
+  def lookup_namespace_in(target, namespaces)
+    result = []
+    for n in namespaces
+      result.concat(n.contained_modules_matching(target))
+    end
+    result
+  end
+
+  def find_class_by_name(full_name)
+    names = full_name.split(/::/)
+    ns = @cache.toplevel
+    for name in names
+      ns = ns.contained_class_named(name)
+      return nil if ns.nil?
+    end
+    get_class(ns)
+  end
+
+  def find_methods(name, is_class_method, namespaces)
+    result = []
+    namespaces.each do |ns|
+      result.concat ns.methods_matching(name, is_class_method)
+    end
+    result
+  end
+
+  ##
+  # Return the MethodDescription for a given MethodEntry by deserializing the
+  # YAML
+
+  def get_method(method_entry)
+    path = method_entry.path_name
+    File.open(path) { |f| RI::Description.deserialize(f) }
+  end
+
+  ##
+  # Return a class description
+
+  def get_class(class_entry)
+    result = nil
+    for path in class_entry.path_names
+      path = RiWriter.class_desc_path(path, class_entry)
+      desc = File.open(path) {|f| RI::Description.deserialize(f) }
+      if result
+        result.merge_in(desc)
+      else
+        result = desc
+      end
+    end
+    result
+  end
+
+  ##
+  # Return the names of all classes and modules
+
+  def full_class_names
+    res = []
+    find_classes_in(res, @cache.toplevel)
+  end
+
+  ##
+  # Return a list of all classes, modules, and methods
+
+  def all_names
+    res = []
+    find_names_in(res, @cache.toplevel)
+  end
+
+  private
+
+  def find_classes_in(res, klass)
+    classes = klass.classes_and_modules
+    for c in classes
+      res << c.full_name
+      find_classes_in(res, c)
+    end
+    res
+  end
+
+  def find_names_in(res, klass)
+    classes = klass.classes_and_modules
+    for c in classes
+      res << c.full_name
+      res.concat c.all_method_names
+      find_names_in(res, c)
+    end
+    res
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/ri/util.rb b/trunk/lib/rdoc/ri/util.rb
new file mode 100644
index 0000000000..34277f2594
--- /dev/null
+++ b/trunk/lib/rdoc/ri/util.rb
@@ -0,0 +1,81 @@
+require 'rdoc/ri'
+
+class RDoc::RI::Error < RuntimeError; end
+
+##
+# Break argument into its constituent class or module names, an
+# optional method type, and a method name
+
+class RDoc::RI::NameDescriptor
+
+  attr_reader :class_names
+  attr_reader :method_name
+
+  ##
+  # true and false have the obvious meaning. nil means we don't care
+
+  attr_reader :is_class_method
+
+  ##
+  # +arg+ may be
+  #
+  # 1. A class or module name (optionally qualified with other class or module
+  #    names (Kernel, File::Stat etc)
+  # 2. A method name
+  # 3. A method name qualified by a optionally fully qualified class or module
+  #    name
+  #
+  # We're fairly casual about delimiters: folks can say Kernel::puts,
+  # Kernel.puts, or Kernel\#puts for example. There's one exception: if you
+  # say IO::read, we look for a class method, but if you say IO.read, we look
+  # for an instance method
+
+  def initialize(arg)
+    @class_names = []
+    separator = nil
+
+    tokens = arg.split(/(\.|::|#)/)
+
+    # Skip leading '::', '#' or '.', but remember it might
+    # be a method name qualifier
+    separator = tokens.shift if tokens[0] =~ /^(\.|::|#)/
+
+    # Skip leading '::', but remember we potentially have an inst
+
+    # leading stuff must be class names
+
+    while tokens[0] =~ /^[A-Z]/
+      @class_names << tokens.shift
+      unless tokens.empty?
+        separator = tokens.shift
+        break unless separator == "::"
+      end
+    end
+
+    # Now must have a single token, the method name, or an empty array
+    unless tokens.empty?
+      @method_name = tokens.shift
+      # We may now have a trailing !, ?, or = to roll into
+      # the method name
+      if !tokens.empty? && tokens[0] =~ /^[!?=]$/
+        @method_name << tokens.shift
+      end
+
+      if @method_name =~ /::|\.|#/ or !tokens.empty?
+        raise RDoc::RI::Error.new("Bad argument: #{arg}") 
+      end
+      if separator && separator != '.'
+        @is_class_method = separator == "::"
+      end
+    end
+  end
+
+  # Return the full class name (with '::' between the components) or "" if
+  # there's no class name
+
+  def full_class_name
+    @class_names.join("::")
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/ri/writer.rb b/trunk/lib/rdoc/ri/writer.rb
new file mode 100644
index 0000000000..92aaa1c2da
--- /dev/null
+++ b/trunk/lib/rdoc/ri/writer.rb
@@ -0,0 +1,68 @@
+require 'fileutils'
+require 'rdoc/ri'
+
+class RDoc::RI::Writer
+
+  def self.class_desc_path(dir, class_desc)
+    File.join(dir, "cdesc-" + class_desc.name + ".yaml")
+  end
+
+  ##
+  # Convert a name from internal form (containing punctuation) to an external
+  # form (where punctuation is replaced by %xx)
+
+  def self.internal_to_external(name)
+    if ''.respond_to? :ord then
+      name.gsub(/\W/) { "%%%02x" % $&[0].ord }
+    else
+      name.gsub(/\W/) { "%%%02x" % $&[0] }
+    end
+  end
+
+  ##
+  # And the reverse operation
+
+  def self.external_to_internal(name)
+    name.gsub(/%([0-9a-f]{2,2})/) { $1.to_i(16).chr }
+  end
+
+  def initialize(base_dir)
+    @base_dir = base_dir
+  end
+
+  def remove_class(class_desc)
+    FileUtils.rm_rf(path_to_dir(class_desc.full_name))
+  end
+
+  def add_class(class_desc)
+    dir = path_to_dir(class_desc.full_name)
+    FileUtils.mkdir_p(dir)
+    class_file_name = self.class.class_desc_path(dir, class_desc)
+    File.open(class_file_name, "w") do |f|
+      f.write(class_desc.serialize)
+    end
+  end
+
+  def add_method(class_desc, method_desc)
+    dir = path_to_dir(class_desc.full_name)
+    file_name = self.class.internal_to_external(method_desc.name)
+    meth_file_name = File.join(dir, file_name)
+    if method_desc.is_singleton
+      meth_file_name += "-c.yaml"
+    else
+      meth_file_name += "-i.yaml"
+    end
+
+    File.open(meth_file_name, "w") do |f|
+      f.write(method_desc.serialize)
+    end
+  end
+
+  private
+
+  def path_to_dir(class_name)
+    File.join(@base_dir, *class_name.split('::'))
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/stats.rb b/trunk/lib/rdoc/stats.rb
new file mode 100644
index 0000000000..e18e3c23d7
--- /dev/null
+++ b/trunk/lib/rdoc/stats.rb
@@ -0,0 +1,115 @@
+require 'rdoc'
+
+##
+# Simple stats collector
+
+class RDoc::Stats
+
+  attr_reader :num_classes
+  attr_reader :num_files
+  attr_reader :num_methods
+  attr_reader :num_modules
+
+  def initialize(verbosity = 1)
+    @num_classes = 0
+    @num_files   = 0
+    @num_methods = 0
+    @num_modules = 0
+
+    @start = Time.now
+
+    @display = case verbosity
+               when 0 then Quiet.new
+               when 1 then Normal.new
+               else        Verbose.new
+               end
+  end
+
+  def add_alias(as)
+    @display.print_alias as
+    @num_methods += 1
+  end
+
+  def add_class(klass)
+    @display.print_class klass
+    @num_classes += 1
+  end
+
+  def add_file(file)
+    @display.print_file file
+    @num_files += 1
+  end
+
+  def add_method(method)
+    @display.print_method method
+    @num_methods += 1
+  end
+
+  def add_module(mod)
+    @display.print_module mod
+    @num_modules += 1
+  end
+
+  def print
+    puts "Files:   #@num_files"
+    puts "Classes: #@num_classes"
+    puts "Modules: #@num_modules"
+    puts "Methods: #@num_methods"
+    puts "Elapsed: " + sprintf("%0.1fs", Time.now - @start)
+  end
+
+  class Quiet
+    def print_alias(*) end
+    def print_class(*) end
+    def print_file(*) end
+    def print_method(*) end
+    def print_module(*) end
+  end
+
+  class Normal
+    def print_alias(as)
+      print 'a'
+    end
+
+    def print_class(klass)
+      print 'C'
+    end
+
+    def print_file(file)
+      print "\n#{file}: "
+    end
+
+    def print_method(method)
+      print 'm'
+    end
+
+    def print_module(mod)
+      print 'M'
+    end
+  end
+
+  class Verbose
+    def print_alias(as)
+      puts "\t\talias #{as.new_name} #{as.old_name}"
+    end
+
+    def print_class(klass)
+      puts "\tclass #{klass.full_name}"
+    end
+
+    def print_file(file)
+      puts file
+    end
+
+    def print_method(method)
+      puts "\t\t#{method.singleton ? '::' : '#'}#{method.name}"
+    end
+
+    def print_module(mod)
+      puts "\tmodule #{mod.full_name}"
+    end
+  end
+
+end
+
+
diff --git a/trunk/lib/rdoc/template.rb b/trunk/lib/rdoc/template.rb
new file mode 100644
index 0000000000..53d0e3ce68
--- /dev/null
+++ b/trunk/lib/rdoc/template.rb
@@ -0,0 +1,64 @@
+require 'erb'
+
+module RDoc; end
+
+##
+# An ERb wrapper that allows nesting of one ERb template inside another.
+#
+# This TemplatePage operates similarly to RDoc 1.x's TemplatePage, but uses
+# ERb instead of a custom template language.
+#
+# Converting from a RDoc 1.x template to an RDoc 2.x template is fairly easy.
+#
+# * %blah% becomes <%= values["blah"] %>
+# * !INCLUDE! becomes <%= template_include %>
+# * HREF:aref:name becomes <%= href values["aref"], values["name"] %>
+# * IF:blah becomes <% if values["blah"] then %>
+# * IFNOT:blah becomes <% unless values["blah"] then %>
+# * ENDIF:blah becomes <% end %>
+# * START:blah becomes <% values["blah"].each do |blah| %>
+# * END:blah becomes <% end %>
+#
+# To make nested loops easier to convert, start by converting START statements
+# to:
+#
+#   <% values["blah"].each do |blah| $stderr.puts blah.keys %>
+#
+# So you can see what is being used inside which loop.
+
+class RDoc::TemplatePage
+
+  ##
+  # Create a new TemplatePage that will use +templates+.
+
+  def initialize(*templates)
+    @templates = templates
+  end
+
+  ##
+  # Returns "<a href=\"#{ref}\">#{name}</a>"
+
+  def href(ref, name)
+    if ref then
+      "<a href=\"#{ref}\">#{name}</a>"
+    else
+      name
+    end
+  end
+
+  ##
+  # Process the template using +values+, writing the result to +io+.
+
+  def write_html_on(io, values)
+    b = binding
+    template_include = ""
+
+    @templates.reverse_each do |template|
+      template_include = ERB.new(template).result b
+    end
+
+    io.write template_include
+  end
+
+end
+
diff --git a/trunk/lib/rdoc/tokenstream.rb b/trunk/lib/rdoc/tokenstream.rb
new file mode 100644
index 0000000000..0a1eb9130b
--- /dev/null
+++ b/trunk/lib/rdoc/tokenstream.rb
@@ -0,0 +1,33 @@
+module RDoc; end
+
+##
+# A TokenStream is a list of tokens, gathered during the parse of some entity
+# (say a method). Entities populate these streams by being registered with the
+# lexer. Any class can collect tokens by including TokenStream. From the
+# outside, you use such an object by calling the start_collecting_tokens
+# method, followed by calls to add_token and pop_token.
+
+module RDoc::TokenStream
+
+  def token_stream
+    @token_stream
+  end
+
+  def start_collecting_tokens
+    @token_stream = []
+  end
+
+  def add_token(tk)
+    @token_stream << tk
+  end
+
+  def add_tokens(tks)
+    tks.each  {|tk| add_token(tk)}
+  end
+
+  def pop_token
+    @token_stream.pop
+  end
+
+end
+