jgm/pandoc: pandoc 2.0

New features New output format ms (groff ms). Complete support, including tables, math, syntax highlighting, and PDF bookmarks. The writer uses texmath's new eqn writer to convert math to eqn format, so a ms file produced with this writer should be processed with groff -ms -e if it contains math. New output format jats (Journal Article Tag Suite). This is an XML format used in archiving and publishing articles. Note that a URI-encoded CSL stylesheet (data/jats.csl) is added automatically unless a stylesheet is specified using --css. New output format gfm (GitHub-flavored CommonMark) (#3841). This uses bindings to GitHub's fork of cmark, so it should parse gfm exactly as GitHub does (excepting certain postprocessing steps, involving notifications, emojis, etc.). markdown_github has been deprecated in favor of gfm. New output format muse (Emacs Muse) (Alexander Krotov, #3489). New input format gfm (GitHub-flavored CommonMark) (#3841). This uses bindings to GitHub's fork of cmark. markdown_github has been deprecated in favor of gfm. New input format muse (Emacs Muse) reader (Alexander Krotov, #3620). New input format tikiwiki (TikiWiki markup) (rlpowell, #3800). New input format vimwiki (Vimwiki markup) (Yuchen Pei, #3705). Note that there is a new data file, data/vimwiki.css, which can be used to display the HTML produced by this reader and pandoc's HTML writer in the style of vimwiki's own HTML export. New input format creole (Creole 1.0) (#3994, Sascha Wilde). New syntax for Divs, with fenced_divs extension enabled by default (#168). This gives an attractive, plain-text way to create containers for block-level content. Added new syntax for including raw content in any output format, enabled by the raw_attribute extension (which is on by default for markdown and multimarkdown). The syntax is the same as for fenced code blocks or code inlines, only with {=FORMAT} for attributes, where FORMAT is the name of the output format (e.g., ms, html). Implement multicolumn support for slide formats (#1710). The structure expected is: <div class="columns"> <div class="column" width="40%"> contents... </div> <div class="column" width="60%"> contents... </div> </div> Support has been added for beamer and all HTML slide formats. Allows line comments in templates, beginning with $-- (#3806). (Requires doctemplates 0.2.1.) Add --eol=crlf|lf|native flag and writer option to control line endings (Stefan Dresselhaus, #3663, #2097). Add --log option to save log messages in JSON format to a file (#3392). Add --request-header option, to set request headers when pandoc makes HTTP requests to fetch external resources. For example: --request-header User-Agent:blah. Added lua filters (Albert Krewinkel, #3514). The new --lua-filter option works like --filter but takes pathnames of special lua filters and uses the lua interpreter baked into pandoc, so that no external interpreter is needed. Note that lua filters are all applied after regular filters, regardless of their position on the command line. For documentation of lua filters, see doc/lua-filters.md. Set PANDOC_READER_OPTIONS in environment where filters are run. This contains a JSON representation of ReaderOptions, so filters can access it. Support creation of pdf via groff ms and pdfroff. pandoc -t ms -o output.pdf input.txt. Support for PDF generation via HTML and weasyprint or prince (Mauro Bieg, #3909). pandoc -t html5 -o output.pdf --pdf-engine=prince. Added --epub-subdirectory option (#3720). This specifies the subdirectory in the OCF container that holds the EPUB specific content. We now put all EPUB related content in an EPUB/ subdirectory by default (later this will be configurable). mimetype META-INF/ com.apple.ibooks.display-options.xml container.xml EPUB/ <<--configurable-->> fonts/ <<--static-->> font.otf media/ <<--static-->> cover.jpg fig1.jpg styles/ <<--static-->> stylesheet.css content.opf toc.ncx text/ <<--static-->> ch001.xhtml Added --resource-path=SEARCHPATH command line option (#852). SEARCHPATH is separated by the usual character, depending on OS (: on unix, ; on windows). Default resource path is just working directory. However, the working directory must be explicitly specified if the --resource-path option is used. Added –abbreviations=FILE option for custom abbreviations file (#256). Dfault abbreviations file (data/abbreviations) contains a list of strings that will be recognized by pandoc's Markdown parser as abbreviations. (A nonbreaking space will be inserted after the period, preventing a sentence space in formats like LaTeX.) Users can override the default by putting a file abbreviations in their user data directory (~/.pandoc on *nix). Allow a theme file as argument to --highlight-style. Also include a sample, default.theme, in data/. Allow --syntax-definition option for dynamic loading of syntax highlighting definitions (#3334). Lists in markdown by default now use the CommonMark variable nesting rules (#3511). The indentation required for a block-level item to be included in a list item is no longer fixed, but is determined by the first line of the list item. To be included in the list item, a block must be indented to the level of the first non-space content after the list marker. Exception: if are 5 or more spaces after the list marker, then the content is interpreted as an indented code block, and continuation paragraphs must be indented two spaces beyond the end of the list marker. See the CommonMark spec for more details and examples. Documents that adhere to the four-space rule should, in most cases, be parsed the same way by the new rules. Here are some examples of texts that will be parsed differently: - a - b will be parsed as a list item with a sublist; under the four-space rule, it would be a list with two items. - a code Here we have an indented code block under the list item, even though it is only indented six spaces from the margin, because it is four spaces past the point where a continuation paragraph could begin. With the four-space rule, this would be a regular paragraph rather than a code block. - a code Here the code block will start with two spaces, whereas under the four-space rule, it would start with code. With the four-space rule, indented code under a list item always must be indented eight spaces from the margin, while the new rules require only that it be indented four spaces from the beginning of the first non-space text after the list marker (here, a). This change was motivated by a slew of bug reports from people who expected lists to work differently (#3125, #2367, #2575, #2210, #1990, #1137, #744, #172, #137, #128) and by the growing prevalance of CommonMark (now used by GitHub, for example). Those who prefer the old behavior can use -f markdown+four_space_rule. Added four_space_rule extension. This triggers the old pandoc parsing rule for content nested under list items (the "four space rule"). Added spaced_reference_links extension (#2602). It allows whitespace between the two parts of a reference link: e.g. [a] [b] [b]: url This was previously enabled by default; now it is now forbidden by default. Add space_in_atx_header extension (#3512). This is enabled by default in pandoc and GitHub markdown but not the other flavors. This requirse a space between the opening #'s and the header text in ATX headers (as CommonMark does but many other implementations do not). This is desirable to avoid falsely capturing things ilke #hashtag or #5 Add sourcefile and outputfile template variables (Roland Hieber, #3431). Allow ibooks-specific metadata in epubs (#2693). You can now have the following fields in your YAML metadata, and it will be treated appropriately in the generated EPUB: ibooks: version: 1.3.4 specified-fonts: false ipad-orientation-lock: portrait-only iphone-orientation-lock: landscape-only binding: true scroll-axis: vertical Behavior changes Reader functions no longer presuppose that CRs have been stripped from the input. (They strip CRs themselves, before parsing, to simplify the parsers.) Added support for translations (localization) (#3559). Currently this only affects the LaTeX reader, for things like \figurename. Translation data files for 46 languages can be found in data/translations. Make --ascii work with DocBook output too. Rename --latex-engine to --pdf-engine, and --latex-engine-opt to --pdf-engine-opt. Removed --parse-raw and readerParseRaw. These were confusing. Now we rely on the +raw_tex or +raw_html extension with latex or html input. Thus, instead of --parse-raw -f latex we use -f latex+raw_tex, and instead of --parse-raw -f html we use -f html+raw_html. With --filter R filters are now recognized, even if they are not executable (#3940, #3941, Andrie de Vries). Support SVG in PDF output, converting with rsvg2pdf (#1793). Make epub an alias for epub3, not epub2. Removed --epub-stylesheet; use --css instead (#3472, #847). Multiple stylesheets may be used. Stylesheets will be taken both from --css and from the stylesheet metadata field (which can contain either a file path or a list of them). --mathml and MathML in HTMLMathMethod no longer take an argument. The argument was for a bridge JavaScript that used to be necessary in 2004. We have removed the script already. --katex improvements. The latest version is used, and the autoload script is loaded by default. Change MathJax CDN default since old one is shutting down (#3544). Note: The new URL requires a version number, which we'll have to update manually in subsequent pandoc releases in order to take advantage of mathjax improvements. --self-contained: don't incorporate elements with data-external="1" (#2656). You can leave an external link as it is by adding the attribute data-external="1" to the element. Pandoc will then not try to incorporate its content when --self-contained is used. This is similar to a feature already supported by the EPUB writer. Allow --extract-media to work with non-binary input formats (#1583, #2289). If --extract-media is supplied with a non-binary input format, pandoc will attempt to extract the contents of all linked images, whether in local files, data: uris, or external uris. They will be named based on the sha1 hash of the contents. Make papersize: a4 work regardless of the case of a4. It is converted to a4 in LaTeX and A4 in ConTeXt. Make east_asian_line_breaks affect all readers/writers (#3703). Underlined elements are now treated consistently by readers (#2270, hftf); they are always put in a Span with class underline. This allows the user to treat them differently from other emphasis, using a filter. Docx, Org, Textile, Txt2Tags, and HTML readers have been changed. Improved behavior of auto_identifiers when there are explicit ids (#1745). Previously only autogenerated ids were added to the list of header identifiers in state, so explicit ids weren't taken into account when generating unique identifiers. Duplicated identifiers could result. This simple fix ensures that explicitly given identifiers are also taken into account. Use table-of-contents for contents of toc, make toc a boolean (#2872). Changed markdown, rtf, and HTML-based templates accordingly. This allows you to set toc: true in the metadata; this previously produced strange results in some output formats. For backwards compatibility, toc is still set to the toc contents. But it is recommended that you update templates to use table-of-contents for the toc contents and toc for a boolean flag. Change behavior with binary format output to stdout. Previously, for binary formats, output to stdout was disabled unless we could detect that the output was being piped (and not sent to the terminal). Unfortunately, such detection is not possible on Windows, leaving windows users no way to pipe binary output. So we have changed the behavior in the following way: Output to stdout is allowed when it can be determined that the output is being piped (on non-Windows platforms). If the -o option is not used, binary output is never sent to stdout by default; instead, an error is raised. If -o - is used, binary output is sent to stdout, regardless of whether it is being piped. This works on Windows too. Better error behavior: uses of error have been replaced by raising of PandocError, which can be trapped and handled by the calling program. Removed hard_line_breaks extension from markdown_github (#3594). GitHub has two Markdown modes, one for long-form documents like READMEs and one for short things like issue coments. In issue comments, a line break is treated as a hard line break. In README, wikis, etc., it is treated as a space as in regular Markdown. Since pandoc is more likely to be used to convert long-form documents from GitHub Markdown, -hard_line_breaks is a better default. Include backtick_code_blocks extension in mardkown_mmd (#3637). Escape MetaString values (as added with -M/--metadata flag) (#3792). Previously they would be transmitted to the template without any escaping. Note that --M title='*foo*' yields a different result from --- title: *foo* --- In the latter case, we have emphasis; in the former case, just a string with literal asterisks (which will be escaped in formats, like Markdown, that require it). Allow em, cm, in for image height/width in HTML, LaTeX (#3450). HTML writer: Insert data- in front of unsupported attributes. Thus, a span with attribute foo gets written to HTML5 with data-foo, so it is valid HTML5. HTML4 is not affected. This will allow us to use custom attributes in pandoc without producing invalid HTML. (With help from Wandmalfarbe, #3817.) Plain writer: improved super/subscript rendering. We now handle more non-digit characters for which there are sub/superscripted unicode characters. When unicode sub/superscripted characters are not available, we use _(..) or ^(..) (#3518). Docbook, JATS, TEI writers: print INFO message when omitting interior header (#3750). This only applies to section headers inside list items, e.g., which were otherwise silently omitted. Change to --reference-links in Markdown writer (#3701). With --reference-location of section or block, pandoc will now repeat references that have been used in earlier sections. The Markdown reader has also been modified, so that exactly repeated references do not generate a warning, only references with the same label but different targets. The idea is that, with references after every block, one might want to repeat references sometimes. ODT/OpenDocument writer: Support lang attribute (#1667). Added support for --toc (#2836). Thanks to @anayrat. Docx writer: lang meta, see #1667 (Mauro Bieg, #3515). Change FigureWithCaption to CaptionedFigure (iandol, #3658). Use Table rather than Table Normal for table style (#3275). Table Normal is the default table style and can't be modified. Pass through comments (#2994). We assume that comments are defined as parsed by the docx reader: I want I left a comment.some text to have a comment on it. We assume also that the id attributes are unique and properly matched between comment-start and comment-end. Bookmark improvements. Bookmark start/end now surrounds content rather than preceding it. Bookmarks generated for Div with id (jgm/pandoc-citeproc#205). Add keywords metadata to docx document properties (Ian). RST writer: support unknown interpreted text roles by parsing them as Span with role attributes (#3407). This way they can be manipulated in the AST. HTML writer: Line block: Use class instead of style attribute (#1623). We now issue <div class="line-block"> and include a default definition for line-block in the default templates, instead of hard-coding a style on the div. Add class footnoteBack to footnote back references (Timm Albers). This allows for easier CSS styling. Render SmallCaps as span with smallcaps class (#1592), rather than using a style attribute directly. This gives the user more flexibility in styling small caps in CSS. With reveal.js we use data-src instead of src for images for lazy loading. Special-case .stretch class for images in reveal.js (#1291). Now in reveal.js, an image with class stretch in a paragraph by itself will stretch to fill the whole screen, with no caption or figure environment. Added warnings for non-rendered blocks to writers. Writers now raise an error on template failure. When creating a PDF via LaTeX, warn if the font is missing some characters (#3742). Remove initial check for PDF-creating program (#3819). Instead, just try running it and raise the exception if it isn't found at that point. This improves things for users of Cygwin on Windows, where the executable won't be found by findExecutable unless .exe is added. The same exception is raised as before, but at a later point. Readers issue warning for duplicate header identifiers (#1745). Autogenerated header identifiers are given suffixes so as not to clash with previously used header identifiers. But they may still coincide with an explicit identifier that is given for a header later in the document, or with an identifier on a div, span, link, or image. We now issue a warning in this case, so users can supply an explicit identifier. CommonMark reader now supports emoji, hard_line_breaks, smart, and raw_html extensions. Markdown reader: Don't allow backslash + newline to affect block structure (#3730). Note that as a result of this change, the following, which formerly produced a header with two lines separated by a line break, will now produce a header followed by a paragraph: # Hi there This may affect some existing documents that relied on this undocumented and unintended behavior. This change makes pandoc more consistent with other Markdown implementations, and with itself (since the two-space version of a line break doesn't work inside ATX headers, and neither version works inside Setext headers). Org reader (Albert Krewinkel, unless noted): Support table.el tables (#3314). Support macros (#3401). Support the #+INCLUDE: file inclusion mechanism (#3510). Recognized include types are example, export, src, and normal org file inclusion. Advanced features like line numbers and level selection are not implemented yet. Interpret more meta value as inlines. The values of the following meta variables are now interpreted using org-markup instead of treating them as pure strings: keywords (comma-separated list of inlines), subtitle (inline values), nocite (inline values, can be repeated). Support \n export option (#3940). This turns all newlines in the text into hard linebreaks. RST reader: Improved admonition support (#223). We no longer add an admonition class, we just use the class for the type of admonition, note for example. We put the word corresponding to the label in a paragraph inside a Div at the beginning of the admonition with class admonition-title. This is about as close as we can get to RST's own output. Initial support of .. table directive. This allows adding captions to tables. Support .. line-block directive. This is deprecated but may still be in older documents. Support scale and align attributes of images (#2662). Implemented implicit internal header links (#3475). Support RST-style citations (#853). The citations appear at the end of the document as a definition list in a special div with id citations. Citations link to the definitions. Recurse into bodies of unknown directives (#3432). In most cases it's better to preserve the content than to emit it. This isn't guaranteed to have good results; it will fail spectacularly for unknown raw or verbatim directives. Handle chained link definitions (#262). For example, .. _hello: .. _goodbye: example.com Here both hello and goodbye should link to example.com. Support anchors (#262). E.g. `hello` .. _hello: paragraph This is supported by putting "paragraph" in a Div with id hello. Support :widths: attribute for table directive. Implement csv-table directive (#3533). Most attributes are supported, including :file: and :url:. Support unknown interpreted text roles by parsing them as Span with "role" attributes (#3407). This way they can be manipulated in the AST. HTML reader: parse a span with class smallcaps as SmallCaps. LaTeX reader: Implemented \graphicspath (#736). Properly handle column prefixes/suffixes. For example, in \begin{tabular}{>{$}l<{$}>{$}l<{$} >{$}l<{$}} each cell will be interpreted as if it has a $ before its content and a $ after (math mode). Handle komascript \dedication (#1845). It now adds a dedication field to metadata. It is up to the user to supply a template that uses this variable. Support all \textXX commands, where XX = rm, tt, up, md, sf, bf (#3488). Spans with a class are used when there is nothing better. Expand \newenvironment macros (#987). Add support for LaTeX subfiles package (Marc Schreiber, #3530). Better support for subfigure package (#3577). A figure with two subfigures turns into two pandoc figures; the subcaptions are used and the main caption ignored, unless there are no subcaptions. Add support for \vdots (Marc Schreiber, #3607). Add basic support for hyphenat package (Marc Schreiber, #3603). Add basic \textcolor support (Marc Schreiber). Add support for tabularx environment (Marc Schreiber, #3632). Better handling of comments inside math environments (#3113). This solves a problem with commented out \end{eqnarray} inside an eqnarray (among other things). Parse tikzpicture as raw verbatim environment if raw_tex extension is selected (#3692). Otherwise skip with a warning. This is better than trying to parse it as text! Add \colorbox support (Marc Schreiber). Set identifiers on Spans used for \label