library(sgml), a foreign library for SWI-Prolog to parse 
SGML and XML documents, returning information on both the document and 
the document's DTD. The parser is designed to be small, fast and 
flexible.
Markup languages have recently regained popularity for two reasons. One is document exchange, which is largely based on HTML, an instance of SGML, and the other is for data exchange between programs, which is often based on XML, which can be considered a simplified and rationalised version of SGML.
James Clark's SP parser is a flexible SGML and XML parser. Unfortunately it has some drawbacks. It is very big, not very fast, cannot work under event-driven input and is generally hard to program beyond the scope of the well designed generic interface. The generic interface however does not provide access to the DTD, does not allow for flexible handling of input or parsing the DTD independently of a document instance.
The parser described in this document is small (less than 100 kBytes executable on a Pentium), fast (between 2 and 5 times faster than SP), provides access to the DTD, and provides flexible input handling.
The document output is equal to the output produced by xml2pl, an SP interface to SWI-Prolog written by Anjo Anjewierden.
This package allows you to parse SGML, XML and HTML data into a 
Prolog data structure. The high-level interface defined in library(sgml) 
provides access at the file-level, while the low-level interface defined 
in the foreign module works with Prolog streams. Please use the source 
of sgml.pl as a starting point for dealing with data from 
other sources than files, such as SWI-Prolog resources, network-sockets, 
character strings, etc. The first example below loads an HTML 
file.
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN"> <html> <head> <title>Demo</title> </head> <body> <h1 align=center>This is a demo</title> Paragraphs in HTML need not be closed. This is called `omitted-tag' handling. </body> </html>
?- load_html_file('test.html', Term),
   pretty_print(Term).
[ element(html,
          [],
          [ element(head,
                    [],
                    [ element(title,
                              [],
                              [ 'Demo'
                              ])
                    ]),
            element(body,
                    [],
                    [ '\n',
                      element(h1,
                              [ align = center
                              ],
                              [ 'This is a demo'
                              ]),
                      '\n\n',
                      element(p,
                              [],
                              [ 'Paragraphs in HTML need not be closed.\n'
                              ]),
                      element(p,
                              [],
                              [ 'This is called `omitted-tag\' handling.'
                              ])
                    ])
          ])
].
The document is represented as a list, each element being an atom to 
represent CDATA or a term element(Name, Attributes, 
Content). Entities (e.g. <) are expanded and 
included in the atom representing the element content or attribute 
value.1Up to SWI-Prolog 5.4.x, 
Prolog could not represent wide characters and entities that 
did not fit in the Prolog characters set were emitted as a term number(+Code). 
With the introduction of wide characters in the 5.5 branch this is no 
longer needed.
These predicates are for basic use of the library, converting entire and self-contained files in SGML, HTML, or XML into a structured term. They are based on load_structure/3.
load_structure(Source, ListOfContent, [dialect(sgml)]).load_structure(Source, ListOfContent, [dialect(xml)](load_structure(Source, 
ListOfContent, [dialect(xml)])stream(Handle). Implemented as 
below. Note that load_html_file/2 
re-uses a cached DTD object as defined by dtd/2. 
As DTD objects may be corrupted while loading errornous documents 
sharing is undesirable if the documents are not known to be correct. See dtd/2 
for details.
load_html_file(Source, Term) :-
        dtd(html, DTD),
        load_structure(Source, Term,
                       [ dtd(DTD),
                         dialect(sgml),
                         shorttag(false)
                       ]).
SGML or XML files are loaded through the common predicate load_structure/3. This is a predicate with many options. For simplicity a number of commonly used shorthands are provided: load_sgml_file/2, load_xml_file/2, and load_html_file/2.
stream(StreamHandle) or a file-name. Options is 
a list of options controlling the conversion process.
A proper XML document contains only a single toplevel element whose name matches the document type. Nevertheless, a list is returned for consistency with the representation of element content. The <aref/ ListOfContent/ consists of the following types:
CDATA. Note this is possible in 
SWI-Prolog, as there is no length-limit on atoms and atom garbage 
collection is provided.ListOfAttributes is a list of Name=Value 
pairs for attributes. Attributes of type CDATA are returned 
literal. Multi-valued attributes (NAMES, etc.) are 
returned as a list of atoms. Handling attributes of the types NUMBER 
and NUMBERS depends on the setting of the number(+NumberMode) 
attribute through
set_sgml_parser/2 
or load_structure/3. 
By default they are returned as atoms, but automatic conversion to 
Prolog integers is supported. ListOfContent defines the 
content for the element.
SDATA is 
encountered, this term is returned holding the data in Text.NDATA is 
encountered, this term is returned holding the data in Text.
<?...?>), 
<aref/ Text/ holds the text of the processing instruction. Please 
note that the
<?xml ...?> instruction is handled internally.
The Options list controls the conversion process. Currently defined options are:
<!DOCTYPE ...> 
declaration is ignored and the document is parsed and validated against 
the provided DTD. If provided as a variable, the created DTD is 
returned. See section 3.5.sgml (default), xml 
and xmlns. See section 3.3 
for details on the differences.NUMBER and NUMBERS 
are handled. If token (default) they are passed as an atom. 
If
integer the parser attempts to convert the value to an 
integer. If successful, the attribute is passed as a Prolog integer. 
Otherwise it is still passed as an atom. Note that SGML defines a 
numeric attribute to be a sequence of digits. The - sign is 
not allowed and
1 is different from 01. For this reason the 
default is to handle numeric attributes as tokens. If conversion to 
integer is enabled, negative values are silently accepted.false, only the attributes occurring in the 
source are emitted.CDATA entities can be specified with this construct. 
Multiple entity options are allowed.max_errors(-1) makes the parser continue, no matter how 
many errors it encounters.
error(limit_exceeded(max_errors, Max), _)
SGML2PL has four modes for handling white-space. The initial mode can 
be switched using the space(SpaceMode) option to
load_structure/3 
and set_sgml_parser/2. 
In XML mode, the mode is further controlled by the <elem/xml:space/ 
attribute, which may be specified both in the DTD and in the document. 
The defined modes are:
sgml space-mode, all consequtive white-space 
is reduced to a single space-character. This mode canonises all white 
space.default, all leading and trailing 
white-space is removed from CDATA objects. If, as a result, 
the CDATA becomes empty, nothing is passed to the 
application. This mode is especially handy for processing 
`data-oriented' documents, such as RDF. It is not suitable for normal 
text documents. Consider the HTML fragment below. When processed in this 
mode, the spaces between the three modified words are lost. This mode is 
not part of any standard; XML 1.0 allows only default and preserve.
Consider adjacent <b>bold</b> <ul>and</ul> <it>italic</it> words.
The parser can operate in two modes: sgml mode and xml 
mode, as defined by the dialect(Dialect) option. Regardless 
of this option, if the first line of the document reads as below, the 
parser is switched automatically into XML mode.
<?xml ... ?>
Currently switching to XML mode implies:
<element [attribute...] /> is 
recognised as an empty element.
lt (<), gt 
(>), amp (&), apos 
(') and quot (").
ELEMENT, etc.).
_) and colon (:) are 
allowed in names.
preserve. In addition to setting 
white-space handling at the toplevel the XML reserved attribute 
<elem/xml:space/ is honoured. It may appear both in the document and 
the DTD. The remove extension is honoured as 
<elem/xml:space/ value. For example, the DTD statement below ensures 
that the <elem/pre/ element preserves space, regardless of the 
default processing mode.
<!ATTLIST pre xml:space nmtoken #fixed preserve>
Using the dialect xmlns, the parser will 
interpret XML namespaces. In this case, the names of elements are 
returned as a term of the format
URL:LocalName
If an identifier has no namespace and there is no default namespace it is returned as a simple atom. If an identifier has a namespace but this namespace is undeclared, the namespace name rather than the related URL is returned.
Attributes declaring namespaces (xmlns:<ns>=<url>) 
are reported as if xmlns were not a defined resource.
In many cases, getting attribute-names as url:name is not desirable. Such terms are hard to unify and sometimes multiple URLs may be mapped to the same identifier. This may happen due to poor version management, poor standardisation or because the the application doesn't care too much about versions. This package defines two call-backs that can be set using set_sgml_parser/2 to deal with this problem.
The call-back xmlns is called as XML namespaces are 
noticed. It can be used to extend a canonical mapping for later use by 
the urlns call-back. The following illustrates this 
behaviour. Any namespace containing rdf-syntax in its URL 
or that is used as
rdf namespace is canonised to rdf. This 
implies that any attribute and element name from the RDF namespace 
appears as
rdf:\bnfmeta{name}
:- dynamic
        xmlns/3.
on_xmlns(rdf, URL, _Parser) :- !,
        asserta(xmlns(URL, rdf, _)).
on_xmlns(_, URL, _Parser) :-
        sub_atom(URL, _, _, _, 'rdf-syntax'), !,
        asserta(xmlns(URL, rdf, _)).
load_rdf_xml(File, Term) :-
        load_structure(File, Term,
                       [ dialect(xmlns),
                         call(xmlns, on_xmlns),
                         call(urlns, xmlns)
                       ]).
The library provides iri_xml_namespace/3 to break down an IRI into its namespace and localname:
# or /. Note however that this can 
produce unexpected results. E.g., in the example below, one might expect 
the namespace to be http://example.com/images\#, 
but an XML name cannot start with a digit.
?- iri_xml_namespace('http://example.com/images#12345', NS, L).
NS = 'http://example.com/images#12345',
L = ''.
As we see from the example above, the Localname can be the empty atom. Similarly, Namespace can be the empty atom if IRI is an XML name. Applications will often have to check for either or both these conditions. We decided against failing in these conditions because the application typically wants to know which of the two conditions (empty namespace or empty localname) holds. This predicate is often used for generating RDF/XML from an RDF graph.
The DTD (Document Type Definition) is a separate entity in sgml2pl, that can be created, freed, defined and inspected. Like the parser itself, it is filled by opening it as a Prolog output stream and sending data to it. This section summarises the predicates for handling the DTD.
dialect option from open_dtd/3 
and the encoding option from open/4. 
Notably the dialect option must match the dialect used for 
subsequent parsing using this DTD.sgml. Using xml 
or
xmlns processes the DTD case-sensitive.
dtd using 
the call:
...,
absolute_file_name(dtd(Type),
                   [ extensions([dtd]),
                     access(read)
                   ], DtdFile),
...
Note that DTD objects may be modified while processing errornous 
documents. For example, loading an SGML document starting with
<?xml ...?> switches the DTD to XML mode and 
encountering unknown elements adds these elements to the DTD object. 
Re-using a DTD object to parse multiple documents should be restricted 
to situations where the documents processed are known to be error-free.
omit(OmitOpen, OmitClose), where both 
arguments are booleans (true or false 
representing whether the open- or close-tag may be omitted. Content 
is the content-model of the element represented as a Prolog term. This 
term takes the following form:
cdata, but entity-references are expanded.|(SubModel1, SubModel2)cdata, entity,
id, idref, name, nmtoken,
notation, number or nutoken. For 
DTD types that allow for a list, the notation list(Type) is 
used. Finally, the DTD construct (a|b|...) is mapped to the 
term
nameof(ListOfValues).
Default describes the sgml default. It is one required,
current, conref or implied. If a 
real default is present, it is one of default(Value) or fixed(Value).
NOTATION 
declarations.system(+File) and/or
public(+PublicId).
As this parser allows for processing partial documents and process the DTD separately, the DOCTYPE declaration plays a special role.
If a document has no DOCTYPE declaraction, the parser returns a list holding all elements and CDATA found. If the document has a DOCTYPE declaraction, the parser will open the element defined in the DOCTYPE as soon as the first real data is encountered.
Some documents have no DTD. One of the neat facilities of this 
library is that it builds a DTD while parsing a document with an 
<jargon/ implicit/ DTD. The resulting DTD contains all elements 
encountered in the document. For each element the content model is a 
disjunction of elements and possibly #PCDATA that can be 
repeated. Thus, if we found element <elem/y/ and CDATA in element 
<elem/x/, the model is:
<!ELEMENT x - - (y|#PCDATA)*>
Any encountered attribute is added to the attribute list with the 
type
CDATA and default #IMPLIED.
The example below extracts the elements used in an unknown XML document.
elements_in_xml_document(File, Elements) :-
        load_structure(File, _,
                       [ dialect(xml),
                         dtd(DTD)
                       ]),
        dtd_property(DTD, elements(Elements)),
        free_dtd(DTD).
dtd(DTD) option.file(File) 
option.
<?xml ...> is encountered. See section 
3.3 for details.qualify_attributes 
option below.
xmlns) mode. Default and standard 
compliant is not to qualify such elements. If true, such 
attributes are qualified with the namespace of the element they appear 
in. This option is for backward compatibility as this is the behaviour 
of older versions. In addition, the namespace document suggests 
unqualified attributes are often interpreted in the namespace of their 
element.token (default), attributes of type number are passed as 
a Prolog atom. If integer, such attributes are translated 
into Prolog integers. If the conversion fails (e.g. due to overflow) a 
warning is issued and the value is passed as an atom.iso-8859-1 
and utf-8.<!DOCTYPE 
declaration has been parsed, the default is the defined doctype. The 
parser can be instructed to accept the first element encountered as the 
toplevel using <xmp>doctype(_)</xmp>. This 
feature is especially useful when parsing part of a document (see the parse 
option to
sgml_parse/2.
on_begin, etc. 
callbacks from sgml_parse/2.sgml, xml 
or xmlns).begin or end) is caused by 
an element written down using the shorttag notation (<tag/value/>.#pcdata is part of
Elements. If no element is open, the doctype is 
returned.
This option is intended to support syntax-sensitive editors. Such an editor should load the DTD, find an appropriate starting point and then feed all data between the starting point and the caret into the parser. Next it can use this option to determine the elements allowed at this point. Below is a code fragment illustrating this use given a parser with loaded DTD, an input stream and a start-location.
        ...,
        seek(In, Start, bof, _),
        set_sgml_parser(Parser, charpos(Start)),
        set_sgml_parser(Parser, doctype(_)),
        Len is Caret - Start,
        sgml_parse(Parser,
                   [ source(In),
                     content_length(Len),
                     parse(input)       % do not complete document
                   ]),
        get_sgml_parser(Parser, allowed(Allowed)),
        ...
Input is a stream. A full description of the option-list is below.
source(Stream), this implies reading is stopped as soon as 
the element is complete, and another call may be issued on the same 
stream to read the next element.content is like element but assumes 
the element has already been opened. It may be used in a call-back from
call(on_begin, Pred) to parse individual 
elements after validating their headers.doctype 
declaration.
allowed(Elements) option of get_sgml_parser/2. 
It disables the parser's default to complete the parse-tree by closing 
all open elements.
informational.
quiet, the error is suppressed. Can be used 
together with call(urlns, Closure) to provide external 
expansion of namespaces. See also section 
3.3.1.Handler(+Tag, +Attributes, +Parser).
Handler(+Tag, +Parser).Handler(+CDATA, +Parser), where CDATA is an atom 
representing the data.Handler(+Text, +Parser), 
where
Text is the text of the processing instruction.<!...>) has been read. The named 
handler is called with two arguments: Handler(+Text, 
+Parser), where Text is the text of the declaration 
with comments removed.
This option is expecially useful for highlighting declarations and comments in editor support, where the location of the declaration is extracted using get_sgml_parser/2.
Handler(+Severity, +Message, +Parser), 
where
Severity is one of warning or error 
and
Message is an atom representing the diagnostic message. The 
location of the error can be determined using get_sgml_parser/2
If this option is present, errors and warnings are not reported using print_message/3
xmlns mode, a new namespace declaraction 
is pushed on the environment. The named handler is called with three 
arguments: Handler(+NameSpace, +URL, +Parser). 
See section 3.3.1 for details.xmlns mode, this predicate can be used 
to map a url into either a canonical URL for this namespace or another 
internal identifier. See section 3.3.1 
for details.
In some cases, part of a document needs to be parsed. One option is 
to use load_structure/2 
or one of its variations and extract the desired elements from the 
returned structure. This is a clean solution, especially on small and 
medium-sized documents. It however is unsuitable for parsing really big 
documents. Such documents can only be handled with the call-back output 
interface realised by the
call(Event, Action) option of sgml_parse/2. 
Event-driven processing is not very natural in Prolog.
The SGML2PL library allows for a mixed approach. Consider the case where we want to process all descriptions from RDF elements in a document. The code below calls <xmp>process_rdf_description(Element)</xmp> on each element that is directly inside an RDF element.
:- dynamic
        in_rdf/0.
load_rdf(File) :-
        retractall(in_rdf),
        open(File, read, In),
        new_sgml_parser(Parser, []),
        set_sgml_parser(Parser, file(File)),
        set_sgml_parser(Parser, dialect(xml)),
        sgml_parse(Parser,
                   [ source(In),
                     call(begin, on_begin),
                     call(end, on_end)
                   ]),
        close(In).
on_end('RDF', _) :-
        retractall(in_rdf).
on_begin('RDF', _, _) :-
        assert(in_rdf).
on_begin(Tag, Attr, Parser) :-
        in_rdf, !,
        sgml_parse(Parser,
                   [ document(Content),
                     parse(content)
                   ]),
        process_rdf_description(element(Tag, Attr, Content)).
The parser can deal with ISO Latin-1 and UTF-8 encoded files, doing 
decoding based on the encoding argument provided to
set_sgml_parser/2 
or, for XML, based on the encoding attribute of the XML 
header. The parser reads from SWI-Prolog streams, which also provide 
encoding handling. Therefore, there are two modes for parsing. If the 
SWI-Prolog stream has encoding octet (which is the default 
for binary streams), the decoder of the SGML parser will be used and 
positions reported by the parser are octet offsets in the stream. In 
other cases, the Prolog stream decoder is used and offsets are character 
code counts.
In some cases applications wish to process small portions of large SGML, XML or RDF files. For example, the OpenDirectory project by Netscape has produced a 90MB RDF file representing the main index. The parser described here can process this document as a unit, but loading takes 85 seconds on a Pentium-II 450 and the resulting term requires about 70MB global stack. One option is to process the entire document and output it as a Prolog fact-base of RDF triplets, but in many cases this is undesirable. Another example is a large SGML file containing online documentation. The application normally wishes to provide only small portions at a time to the user. Loading the entire document into memory is then undesirable.
Using the parse(element) option, we open a file, seek 
(using <pref builtin>seek/4) to the position of the element and 
read the desired element.
The index can be built using the call-back interface of
sgml_parse/2. 
For example, the following code makes an index of the structure.rdf 
file of the OpenDirectory project:
:- dynamic
        location/3.                     % Id, File, Offset
rdf_index(File) :-
        retractall(location(_,_)),
        open(File, read, In, [type(binary)]),
        new_sgml_parser(Parser, []),
        set_sgml_parser(Parser, file(File)),
        set_sgml_parser(Parser, dialect(xml)),
        sgml_parse(Parser,
                   [ source(In),
                     call(begin, index_on_begin)
                   ]),
        close(In).
index_on_begin(_Element, Attributes, Parser) :-
        memberchk('r:id'=Id, Attributes),
        get_sgml_parser(Parser, charpos(Offset)),
        get_sgml_parser(Parser, file(File)),
        assert(location(Id, File, Offset)).
The following code extracts the RDF element with required id:
rdf_element(Id, Term) :-
        location(Id, File, Offset),
        load_structure(File, Term,
                       [ dialect(xml),
                         offset(Offset),
                         parse(element)
                       ]).
While processing an SGML document the document may refer to external 
data. This occurs in three places: external parameter entities, normal 
external entities and the DOCTYPE declaration. The current 
version of this tool deals rather primitively with external data. 
External entities can only be loaded from a file and the mapping between 
the entity names and the file is done using a catalog file in a 
format compatible with that used by James Clark's SP Parser, based on 
the SGML Open (now OASIS) specification.
Catalog files can be specified using two primitives: the predicate
sgml_register_catalog_file/2 
or the environment variable
SGML_CATALOG_FILES (compatible with the SP package).
start or end and defines whether the 
catalog is considered first or last. This predicate has no effect if File 
is already part of the catalog.
If no files are registered using this predicate, the first query on 
the catalog examines SGML_CATALOG_FILES and fills the 
catalog with all files in this path.
Two types of lines are used by this package.
DOCTYPEdoctype file
PUBLIC"Id"file
The specified file path is taken relative to the location 
of the catolog file. For the DOCTYPE declaraction, library(sgml) 
first makes an attempt to resolve the SYSTEM or PUBLIC 
identifier. If this fails it tries to resolve the doctype 
using the provided catalog files.
Strictly speaking, library(sgml) breaks the rules for 
XML, where system identifiers must be Universal Resource Indicators, not 
local file names. Simple uses of relative URIs will work correctly under 
UNIX and Windows.
In the future we will design a call-back mechanism for locating and processing external entities, so Prolog-based file-location and Prolog resources can be used to store external entities.
The library library(sgml_write) provides the inverse of 
the parser, converting the parser's output back into a file. This 
process is fairly simple for XML, but due to the power of the SGML DTD 
it is much harder to achieve a reasonable generic result for SGML.
These predicates can write the output in two encoding schemas depending on the encoding of the Stream. In UTF-8 mode, all characters are encoded using UTF-8 sequences. In ISO Latin-1 mode, characters outside the ISO Latin-1 range are represented using a named character entity if provided by the DTD or a numeric character entity.
false, the XML header is suppressed. 
Useful for embedding in other XML streams.
false, 
no layout characters are added. As this mode does not need to analyse 
the document it is faster and guarantees correct output when read back. 
Unfortunately the output is hardly human readable and causes problems 
with many editors.
header 
and
ident is added to use xml_write/3 
to generate XML that is embedded in a larger XML document.
Use/do not use Null End Tags. For XML, this applies only to 
empty elements, so you get <foo/> (default,
net(true)) or \bnfmeta{foo}</foo> (net(false)). 
For SGML, this applies to empty elements, so you get \bnfmeta{foo} 
(if foo is declared to be EMPTY in the DTD),
\bnfmeta{foo}</foo> (default, net(false)) 
or
<foo// (net(true)). In SGML code, short 
character content not containing <c>/</c> can be 
emitted as \bnfmeta{b}xxx</b> (default, net(false) 
or <b/xxx/ (net(true))
DOCTYPE header and the content of the 
document as represented by Term to Stream. The Options 
are described with xml_write/3.The library(sgml) package is a parser. Output is 
generally much easier achieved directly from Prolog. Nevertheless, it 
contains a few building blocks for emitting markup data. The quote 
funtions return a version of the input text into one that contains 
entities for characters that need to be escaped. These are the XML meta 
characters and the characters that cannot be expressed by the document 
encoding. Therefore these predicates accept an encoding 
argument. Accepted values are ascii, iso_latin_1, utf8 
and
unicode. Versions with two arguments are provided for 
backward compatibility, making the safe ascii encoding 
assumption.
<>&".2Older 
versions also mapped ' to '. 
Characters that cannot represented in Encoding are mapped to 
XML character entities.ascii encoding.ascii encoding.ascii encoding.
The current parser is rather limited. While it is able to deal with many serious documents, it omits several less-used features of SGML and XML. Known missing SGML features include
<!ATTLIST #NOTATION name attributes>. Those data 
attributes are provided when you declare an external CDATA, NDATA, or 
SDATA entity.
XML does not include external CDATA, NDATA, or SDATA entities, nor any of the other uses to which data attributes are put in SGML, so it doesn't include data attributes for notations either.
Sgml2pl does not support this feature and is unlikely to; you should be aware that SGML documents using this feature cannot be converted faithfully to XML.
<tag/content/ is a valid abbreviation for
\bnfmeta{tag}content</tag>, which can also be written 
as
\bnfmeta{tag}content</>. Empty start tags (<>), 
unclosed start tags (<a<b</verb>) and unclosed end tags (\bnfmeta{verb}</a<b) 
are not supported.
In XML mode the parser recognises SGML constructs that are not allowed in XML. Also various extensions of XML over SGML are not yet realised. In particular, XInclude is not implemented because the designers of XInclude can't make up their minds whether to base it on elements or attributes yet, let alone details.
Installation on Unix system uses the commonly found configure, 
<make and make install sequence. SWI-Prolog should be 
installed before building this package. If SWI-Prolog is not installed 
as pl, the environment variable PL must be set to 
the name of the SWI-Prolog executable. Installation is now accomplished 
using:
% ./configure % make % make install
This installs the foreign libraries in $PLBASE/lib/$PLARCH 
and the Prolog library files in $PLBASE/library, where $PLBASE 
refers to the SWI-Prolog `home-directory'.
The Prolog representation for parsed documents is based on the SWI-Prolog interface to SP by Anjo Anjewierden.
Richard O'Keefe has put a lot of effort testing and providing bug reports consisting of an illustrative example and explanation of the standard. He also made many suggestions for improving this document.