Skip to content

Importing documents

SciFlow provides an in-browser document importer built on Pandoc-WASM. It converts DOCX files, including files created with the Zotero or Mendeley citation plugins, Markdown files, and LaTeX files into ProseMirror snapshots for <sciflow-editor>.

There is no server-side step. The conversion runs entirely in the user's browser.

The two packages

Package What it does
@sciflow/pandoc-web Browser-side wrapper around the official pandoc-wasm build. Lazy-loads the Pandoc WASM binary on the first conversion, extracts media, and exposes a Promise helper plus a Lit web component.
@sciflow/pandoc-ast Pure-JSON Pandoc-AST → ProseMirror translator. No DOM, no filesystem, no network. Pairs with @sciflow/pandoc-web (which calls it internally) or with any other source of a Pandoc JSON AST.

Most consumers only depend on @sciflow/pandoc-web — it depends on @sciflow/pandoc-ast and re-exports its result types, so a single install covers the whole import flow.

npm install @sciflow/pandoc-web

Install @sciflow/pandoc-ast on its own when the Pandoc AST comes from somewhere else (a server-side pandoc -t json, a stored AST, a worker that already ran Pandoc):

npm install @sciflow/pandoc-ast

Both packages are ESM-only and are published to the public npm registry as of 0.1.0.

Choose an integration layer

1. Drop-zone web component

Add the element to the page and listen for the sciflow-document event.

<script type="module">
  import '@sciflow/pandoc-web';   // side-effect import registers <sciflow-pandoc-drop>
  import '@sciflow/editor-start';
</script>

<sciflow-pandoc-drop
  accept=".docx,.md,.markdown,.tex,.latex"
  headline="Drop a manuscript here"
></sciflow-pandoc-drop>

<sciflow-editor id="ed"></sciflow-editor>

<script type="module">
  const drop = document.querySelector('sciflow-pandoc-drop');
  const editor = document.getElementById('ed');

  drop.addEventListener('sciflow-document', (event) => {
    const { parseResult, ast, media, timings, warnings, stderr } = event.detail;
    // `doc` is null when the document could not be assembled — the run still
    // ends in `done`, so check it here. See "Handling failures" below.
    if (!parseResult.doc) return;
    // The editor takes document and bibliography together; `references` on the
    // element is a getter, not a setter. `toSnapshotReferences` is the two-line
    // mapping in "Loading a result into the editor" below.
    editor.doc = {
      doc: parseResult.doc,
      references: toSnapshotReferences(parseResult.references),
    };
  });

  drop.addEventListener('sciflow-status', (event) => {
    // event.detail.text is a human-readable message, not a machine-readable phase
    console.log('[import]', event.detail.text);
  });

  drop.addEventListener('sciflow-error', (event) => {
    console.error('[import] failed:', event.detail.message);
  });
</script>

Importing @sciflow/pandoc-web for its side effect registers the element. The element module calls customElements.define when evaluated. Importing a named export such as convertFile from the package entry point loads the same module graph and also registers the element.

Attributes and properties

Every property below is a Lit @property(), so it can be set as an HTML attribute or assigned in JavaScript.

Property Attribute Type Default Notes
accept accept string .docx,.md,.markdown,.tex,.latex Forwarded to the internal <input type="file">.
headline headline string Drop a .docx, .md, or .tex file here Also used as the drop zone's aria-label.
subline subline string or click to choose a file Secondary line shown when idle.
hideStatus hide-status boolean false Suppresses the built-in status overlay. The sciflow-status event still fires, so you can render your own.
hideError hide-error boolean false Suppresses the built-in role="alert" error box. The sciflow-error event still fires, and so does sciflow-status with phase: 'error'.
compact compact boolean false Reflected to the host attribute. Renders a slim horizontal bar instead of the full drop card — useful for "drop another file" once a document is loaded.
engineUrl engine-url string unset Where to fetch the Pandoc WebAssembly binary from, instead of the copy your bundler emitted. Absolute http(s) URL, or a relative URL resolved against the page. See Providing the engine yourself.
drop.compact = true;      // shrinks to a slim "drop another file" bar
drop.hideStatus = true;   // hide the built-in overlay, drive your own from sciflow-status
drop.hideError = true;    // hide the built-in alert box, render failures yourself

The element has no other public methods; it owns the file input, drag-and-drop state, the WASM lifecycle, and keyboard activation (Enter / Space).

Events

All three events are bubbles: true, composed: true, so they cross the shadow boundary and can be caught on an ancestor.

Event event.detail
sciflow-document { parseResult, ast, media, pandocVersion, file, timings, warnings, stderr } — typed as PandocDropEventDetail.
sciflow-status { text: string, phase: PandocDropPhase }text is a human-readable message, phase is the machine-readable stage. Branch on phase; treat text as display copy that may be reworded at any time.
sciflow-error { message: string } — the Error.message of whatever failed (unknown format, Pandoc failure, WASM fetch failure). Fires instead of sciflow-document, never alongside it.

sciflow-status's phase is one of five values, and a run walks them in this order:

phase Fires when Default text
loading-pandoc The file was accepted; the engine is being fetched and instantiated. Only the first import on a page pays the download. Loading Pandoc-WASM (~58 MB, first time only)…
converting The engine is ready; Pandoc is reading the file and writing the JSON AST. Converting the document…
parsing Pandoc is done; the AST is being translated into a SciFlow document. Building the SciFlow document…
done The document is ready. Fires immediately before sciflow-document. Done
error The import failed. Fires immediately before sciflow-error. The error message.

Two guarantees make these events suitable for a state machine. Every run starts with loading-pandoc and ends with either done or error, including runs that fail before the engine loads. The same phase is never announced twice in succession.

drop.addEventListener('sciflow-status', (event) => {
  const { phase, text } = event.detail;
  progress.hidden = phase === 'done' || phase === 'error';
  progress.textContent = text;
});

sciflow-document's detail in full:

Field Type Meaning
parseResult ParsePandocAstResult { doc, parts?, title?, subtitle?, abstract?, references, affiliations, altTitles?, media, errors, schemaName, timings? }.
ast unknown The raw Pandoc JSON AST, already JSON.parsed.
media MediaBlob[] { path, blob, mimeType? } entries extracted from the source document.
pandocVersion string Version reported by the WASM build, for example, 3.9.
file { name, size, format } format is 'docx' \| 'markdown' \| 'latex'.
timings { convertMs, parseMs } Wall-clock split between Pandoc and the AST translation.
warnings string[] Pandoc warnings, stringified.
stderr string Pandoc stderr; usually empty on success.

2. convertFile() — Promise helper

When you have your own UI but want the full file → ProseMirror pipeline as one call:

import { convertFile } from '@sciflow/pandoc-web';

const result = await convertFile(file, {
  format: 'docx',      // optional — auto-detected from File.name
  extractMedia: true,  // default: true for DOCX, false for markdown/latex
  citeproc: true,      // default true — runs Pandoc's CSL processor
  citations: true,     // default true for DOCX — reads Zotero/Mendeley citation XML
  styles: true,        // default true for DOCX — preserves Word style names
  parse: { timings: true },  // forwarded to parsePandocAST
});
Option Type Default Notes
format 'docx' \| 'markdown' \| 'latex' Detected from File.name (.docx; .md/.markdown; .tex/.latex) Required when the input is a Blob, ArrayBuffer, Uint8Array or string — those carry no filename. Throws if it cannot be resolved.
extractMedia boolean true for docx, false otherwise Asks Pandoc for embedded media and returns them as result.media.
citeproc boolean true Passes --citeproc.
styles boolean true Adds the +styles reader extension. DOCX only; ignored for other formats.
citations boolean true for docx, false otherwise Adds the +citations reader extension. DOCX only.
parse Omit<ParsePandocAstOptions, 'media'> {} Forwarded to parsePandocAST. media is supplied by convertFile itself and cannot be overridden. parse.timings defaults to true here, unlike parsePandocAST's own default of false.
onPhase (phase: ConvertFilePhase) => void none Called as each stage is entered: loading-pandoc, then converting, then parsing. The first is reported synchronously, before the returned promise exists, so a progress UI is never a frame behind. An input whose format cannot be resolved reports nothing — it never reaches the engine.
engineUrl string unset Where to fetch the Pandoc WebAssembly binary from, instead of the copy your bundler emitted. Forwarded to loadPandoc() — see Providing the engine yourself.
debug boolean false Logs the resolved Pandoc invocation to console.debug.

The resolved ConvertFileResult:

result.parseResult   // ParsePandocAstResult — { doc, references, affiliations, media, errors, … }
result.ast           // raw Pandoc JSON AST
result.media         // MediaBlob[] — images extracted from the source
result.timings       // { convertMs, parseMs }; parseResult.timings has the per-phase breakdown
result.pandocVersion // e.g. "3.9"
result.warnings      // string[]
result.stderr        // string, usually empty

The function accepts File | Blob | ArrayBuffer | Uint8Array | string. It reads a DOCX File or Blob as an ArrayBuffer and Markdown or LaTeX input as text.

Pass { debug: true } to log the resolved reader string, extractMedia, citeproc, Pandoc version, stderr, and warning count with console.debug('[pandoc-web] convert: …'). Use this output to determine whether references are missing because citations was disabled or because the file contains none. Debug logging is disabled by default.

3. loadPandoc() — direct WASM access

If you want only Pandoc (no AST translation, no media manifest, no parseResult), use the loader directly. This is the layer to use for input or output formats outside the three that convertFile() supports.

import { loadPandoc } from '@sciflow/pandoc-web';

const pandoc = await loadPandoc();
pandoc.version; // '3.9'

const { ast, media, warnings, stderr } = await pandoc.convert(docxBlob, {
  from: 'docx+styles+citations',
  to: 'json',
  extractMedia: true,
  citeproc: true,
});

loadPandoc() memoizes by engine location. The first call for an engine fetches and instantiates the WASM binary. Later calls for the same location return the same instance. You cannot retry a failed load without reloading the page; see Handling failures.

Pass loadPandoc({ engineUrl }) to fetch the binary from a URL you host rather than from the asset your bundler emitted — see Providing the engine yourself.

PandocWasm.convert(input, opts)input is Blob | ArrayBuffer | Uint8Array | string:

Option Type Default Notes
from string 'markdown' Any Pandoc reader string, extensions included.
to string 'json' The result is JSON.parsed into result.ast, so a non-JSON target throws. Use convertOutput() for other targets.
extractMedia boolean false Off by default here — most callers still hold the source archive. convertFile() turns it on for DOCX.
citeproc boolean false Off by default here. convertFile() turns it on.

PandocWasm.convertOutput(input, opts) — the AST → deliverable direction, used by convertAstToBlob():

Option Type Default Notes
from string required for example, 'json'.
to string required for example, 'docx'.
outputName string output.<to> Surfaced as result.filename.
media PandocMediaFile[] none { path, bytes, mimeType? }. Paths must match the AST's image URLs.
citeproc boolean false Forwarded as --citeproc.
standalone boolean false Forwarded as --standalone.
toc boolean false Forwarded as --table-of-contents.

Returns { blob, filename, stderr, warnings }. It throws if Pandoc produced no bytes at outputName, with Pandoc's stderr in the message.

@sciflow/pandoc-ast — the AST layer

@sciflow/pandoc-web calls this package internally. Use it directly when the Pandoc AST reaches you some other way.

import { parsePandocAST, manuscript } from '@sciflow/pandoc-ast';
import { Node } from 'prosemirror-model';

const result = await parsePandocAST(ast, { media });

const doc = Node.fromJSON(manuscript, result.doc);   // rehydrate when you need a live PM node
const snapshot = JSON.stringify(result);             // or keep it as plain JSON

parsePandocAST(pandocDocument, opts?, schemaOverride?)pandocDocument must have a blocks array (it throws otherwise). The third argument replaces the default manuscript schema.

Option Type Default Notes
media MediaBlob[] [] { path, blob, mimeType? }. The library never reads the bytes; it matches path against the AST's image URLs.
renderStandalone boolean false Splits the document into parts on H1 boundaries, returned in result.parts[].
skipInvalid boolean false Skip blocks that fail schema validation instead of inserting an error placeholder.
docxStats { customProperties?, template? } undefined DOCX metadata side-channel. Normally absent in the browser flow. customProperties become result.altTitles.
captionTelemetry CaptionTelemetryOptions undefined Opt-in figure-caption-pairing diagnostics.
idGenerator () => string uuid v4, with a leading digit replaced by a so IDs are valid identifiers Inject for deterministic IDs in tests.
referenceRenderer ReferenceRendererInjector undefined A { renderLabels(references): Map<string, string> } object. Without it, references carry full CSL data but no label.
timings boolean false Collect per-phase timings into result.timings. convertFile() sets this to true.

ParsePandocAstResult:

Field Type Notes
doc ParsedDoc \| null ProseMirror document as JSON. null on a fatal parse failure.
parts ParsedDoc[] \| undefined Only when renderStandalone: true.
title, subtitle, abstract string \| undefined Omitted when absent.
references ReferenceEntry[] { id, csl, label? }.
affiliations AffiliationEntry[] { name?, … }.
altTitles Record<string, unknown> \| undefined DOCX customProperties mapped to alternative titles.
media MediaManifestEntry[] { path, used, mimeType?, referencedBy } — see Media and images.
errors ImportErrorPayload[] Non-fatal, per-block. See Handling failures.
schemaName string Always manuscript. A name, not a live Schema.
timings ParsePandocAstTimings \| undefined Only when timings: true.

The result contains no live runtime objects: doc is JSON, the schema is a name, and errors are plain objects. You can pass the result to JSON.stringify().

The renderer surface

For pipelines that need to translate a fragment rather than a whole document, or to add handling for a Pandoc node type:

Export What it is
pandocRenderers The default renderer map, keyed by Pandoc node tag (Para, Header, Image, …). Spread it to override individual entries.
render(node, opts, parentNode?, labels?) Renders one Pandoc node. Throws ImportError when no renderer is registered for the tag.
renderContent(nodes, opts, parentNode?, labels?) Renders an array of Pandoc nodes and flattens the result.
ImportError Error subclass carrying a payload and a toJSON(). Thrown by the renderer surface; parsePandocAST catches these and records them as ImportErrorPayload instead of throwing.
assignIds(node, schema, createId) Walks a ProseMirror node and fills every attrs.id that is null.
SfNodeType, SfMarkType String enums for the manuscript schema's node and mark names (SfNodeType.figure === 'figure', SfMarkType.emphasis === 'em'). Use these instead of hardcoding names.
manuscript Re-export of the schema from @sciflow/schema-prosemirror, so an importer needs only this one dependency.

RendererOptions (the opts argument) carries renderers, schema, and optionally marks, skipInvalid, media, docxStats, captionTelemetry, bibliographyBlocks and a per-pass context. skipInvalid is acted on by the top-level block loop (placeholder or omission, see above), not by individual renderers.

Media and images

When media extraction is on, Pandoc returns the embedded files and the importer rewrites each image node's src to media:<path>, where <path> is exactly the path Pandoc wrote into the AST. Nothing is inlined or uploaded — resolving media: to a real URL is the application's job.

drop.addEventListener('sciflow-document', (event) => {
  const { parseResult, media } = event.detail;

  const urlByPath = new Map(
    media.map((m) => [m.path, URL.createObjectURL(m.blob)]),
  );

  // Rewrite `media:<path>` src attributes to object URLs before handing the doc over.
  const resolved = JSON.parse(
    JSON.stringify(parseResult.doc).replaceAll(
      /"src":"media:([^"]+)"/g,
      (match, path) => `"src":"${urlByPath.get(path) ?? ''}"`,
    ),
  );

  editor.doc = resolved;
  // Revoke the object URLs when the document is discarded.
});

parseResult.media is a manifest, not the bytes: each entry is { path, used, mimeType?, referencedBy }. used tells you whether the document actually references the file, and referencedBy lists the ProseMirror node IDs that do — enough to upload only what the document needs, and to spot dangling references (an entry with used: true that has no matching blob).

Loading a result into the editor

A ParsePandocAstResult is not a complete editor snapshot. Complete these 3 steps before passing it to <sciflow-editor>:

  1. Check doc. It is null when assembly failed — see Handling failures.
  2. Hand document and bibliography over in one assignment. editor.references is a read-only getter; the writable property is editor.doc, which accepts either bare document JSON or the bundle { doc, references?, files?, selection?, version? }.
  3. Convert the references. The importer returns { id, csl, label? } per entry; the editor's SnapshotReference is { id, rawReference, mimeType?, … }, where rawReference is the rendered bibliography string. Extra keys (such as the original csl) are preserved, so nothing is lost in the mapping.
/** ReferenceEntry[] (importer) → SnapshotReference[] (editor). */
function toSnapshotReferences(references = []) {
  return references
    .filter((reference) => typeof reference.id === 'string' && reference.id.trim())
    .map((reference) => ({
      id: reference.id.trim(),
      // `label` is only populated when a `referenceRenderer` was injected;
      // without one, fall back to something a human can read.
      rawReference: reference.label ?? String(reference.csl?.title ?? reference.id),
      csl: reference.csl ?? {},
      mimeType: 'application/vnd.citationstyles.csl+json',
    }));
}

drop.addEventListener('sciflow-document', (event) => {
  const { parseResult, media } = event.detail;
  if (!parseResult.doc) return;

  // `withResolvedMedia` is the `media:<path>` → object-URL rewrite shown under
  // "Media and images" above, wrapped in a function.
  const doc = withResolvedMedia(parseResult.doc, media);

  editor.doc = {
    doc,
    references: toSnapshotReferences(parseResult.references),
    files: [],
  };
});

Extracted media does not become files. A SnapshotFile describes an asset by URL{ id, type?, url?, previewSrc?, mimeType?, name?, dimensions? } — while media[] carries Blobs. Rewrite each image's media:<path> src to a URL you control (an object URL, or the URL you uploaded the bytes to) and pass files: [], as above. If you do list assets in files, each entry's url (or previewSrc) must be the same string the document's src uses: the editor keeps only the file entries a figure in the document actually points at and drops the rest.

Persisting media and references

Object URLs last only for the lifetime of the page. A host that persists the manuscript, such as a journal system or CMS, must store the media and references during import, before the first save.

Store media. Upload every extracted file the document uses, rewrite the image src to the URL you got back, and list the descriptor in files. The figure feature already has the upload hook for exactly this: the uploadFile handler you registered with createFigureFeature({ imageUpload }) (see Figure File API) takes a File and returns { id, mimeType, url?, previewSrc? }, which is the SnapshotFile shape. Pushing the import's media through the same handler means figures that arrived by import and figures the author adds later are stored the same way. uploadImageFile from @sciflow/editor-start calls whichever handler is registered.

import { uploadImageFile } from '@sciflow/editor-start';

async function persistMedia(parseResult, media) {
  const blobByPath = new Map(media.map((m) => [m.path, m]));
  const files = [];
  const srcByPath = new Map();

  for (const entry of parseResult.media) {
    if (!entry.used) continue; // not referenced by the document: nothing to store
    const item = blobByPath.get(entry.path);
    if (!item) continue; // dangling reference: keep the node, the src stays `media:<path>`
    const file = new File([item.blob], entry.path.split('/').pop(), {
      type: item.mimeType ?? entry.mimeType ?? 'application/octet-stream',
    });
    const stored = await uploadImageFile(file); // your handler; returns { id, mimeType, url?, previewSrc? }
    const src = stored.previewSrc ?? stored.url;
    srcByPath.set(entry.path, src);
    files.push({ id: stored.id, type: 'image', mimeType: stored.mimeType, url: stored.url, previewSrc: stored.previewSrc });
  }

  const doc = JSON.parse(
    JSON.stringify(parseResult.doc).replaceAll(
      /"src":"media:([^"]+)"/g,
      (match, path) => (srcByPath.has(path) ? `"src":"${srcByPath.get(path)}"` : match),
    ),
  );
  return { doc, files };
}

Store references. parseResult.references is the structured result of the import: one { id, csl, label? } per bibliography entry, csl being CSL-JSON. The editor is not the system of record for it. What you hand over as references is what the editor keeps and gives back on every editor-change (detail.references, next to detail.files and detail.doc), so store the CSL where your platform keeps bibliographic data at import time, and keep it on the reference object you pass to the editor — SnapshotReference preserves extra keys. rawReference is the rendered bibliography string; platforms that store one under another name map it both ways (OJS, for instance, calls it rawCitation).

drop.addEventListener('sciflow-document', async (event) => {
  const { parseResult, media } = event.detail;
  if (!parseResult.doc) return;

  const references = toSnapshotReferences(parseResult.references); // keeps `csl`
  await myStore.saveReferences(references.map((r) => ({ id: r.id, rawCitation: r.rawReference, csl: r.csl })));

  const { doc, files } = await persistMedia(parseResult, media);
  editor.doc = { doc, references, files };
});

Do the uploads and the reference write before assigning editor.doc: the first editor-change your save loop sees then already carries hosted URLs and the references you stored, and nothing in the persisted document points at a blob: or media: address.

Zotero, Mendeley, and --citeproc

DOCX files produced by the Zotero or Mendeley Word plugins embed citation data as custom XML inside the document. Two flags decide whether the importer picks it up — both are on by default for DOCX:

Flag What it does
citations: true Adds Pandoc's +citations reader extension, which reads the citation custom XML and emits structured Cite nodes. Without it the importer only sees the pre-formatted citation text the plugin injected, and references stays empty.
citeproc: true Passes --citeproc, resolving those Cite nodes against the in-document bibliography and populating references.

If references are missing from a document you know was written with Zotero or Mendeley, one of these being off is the most likely cause. Re-run the conversion with { debug: true } and the console.debug('[pandoc-web] convert: …') line shows exactly which flags reached Pandoc — nothing is logged otherwise.

Only citations is DOCX-only: convertFile() resolves it to true for DOCX and to false for every other format. citeproc defaults to true and is passed to Pandoc for every format — it is what resolves whatever Cite nodes the reader produced, in Markdown and LaTeX as much as in DOCX. At the loadPandoc() layer both defaults flip off: you compose the reader string yourself (from: 'docx+styles+citations') and pass citeproc: true explicitly.

Round-trip: AST → DOCX / HTML / EPUB

The same WASM instance produces binary output formats from any Pandoc AST — useful for a "Download as DOCX" button without a server:

import { convertAstToBlob } from '@sciflow/pandoc-web';

// The two directions speak different media shapes: import hands you
// `MediaBlob[]` (`{ path, blob }`), export wants `PandocMediaFile[]`
// (`{ path, bytes }`). Convert before passing them on.
const exportMedia = await Promise.all(
  media.map(async (entry) => ({
    path: entry.path,
    bytes: new Uint8Array(await entry.blob.arrayBuffer()),
    mimeType: entry.mimeType,
  })),
);

const { blob, filename, stderr, warnings } = await convertAstToBlob(ast, {
  to: 'docx',
  filename: 'paper',   // default 'document'; the extension is appended from `to`
  media: exportMedia,  // pass the media so images embed
  citeproc: true,
  toc: false,
  // Required when the import used a hosted engine: this call loads the engine
  // too, and an omitted `engineUrl` fetches a second one.
  engineUrl: '/pandoc-1.0.1.wasm',
});

const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = filename;   // 'paper.docx'
a.click();
URL.revokeObjectURL(url);
Option Type Default Notes
to AstOutputFormat required docx, odt, epub, epub3, html, html5, markdown, gfm, commonmark, latex, rtf, rst, asciidoc, plain.
filename string document Base name without extension; result.filename is <filename>.<to>.
media PandocMediaFile[] none { path, bytes, mimeType? }. Paths must match the AST's image URLs, typically pandoc-extracted-media.zip/media/imageN.ext.
citeproc boolean off Forwarded as --citeproc.
standalone boolean true for html, html5, epub, epub3, docx, odt, rtf; false for everything else Set false explicitly to get an HTML body fragment instead of a full page.
toc boolean off Forwarded as --table-of-contents.

The returned blob is re-typed with the format's MIME type (application/vnd.openxmlformats-officedocument.wordprocessingml.document for DOCX, and so on) so the browser's download dialog picks a sensible application.

convertAstToBlob reuses the cached WASM instance, so it does not re-fetch the binary — as long as it names the same engine. A page that imported with engineUrl must pass the same engineUrl here; each distinct engine location is downloaded and instantiated separately.

The output is the AST you pass, not what the editor now holds

convertAstToBlob renders the Pandoc AST passed to it. An imported AST represents the source file at conversion time and does not include subsequent editor changes. These packages do not provide a ProseMirror → Pandoc AST converter. To export edited content, use the editor's JATS XML export or another export pipeline.

Media paths differ between the two directions

MediaBlob ({ path, blob }) is what convertFile() and parsePandocAST() speak; PandocMediaFile ({ path, bytes }) is what convertAstToBlob() and convertOutput() want. Convert with new Uint8Array(await blob.arrayBuffer()) when round-tripping.

Per-phase timings

convertFile() gives you the coarse split (convertMs vs parseMs) and enables the per-phase breakdown inside the AST translation by default:

const { timings } = result.parseResult;

console.table({
  metadata:                  timings.metadataMs,              // references, affiliations, abstract
  'render body':             timings.renderBodyMs,            // the block render loop
  'materialize references':  timings.materializeReferencesMs, // CSL → reference nodes
  'header (title/subtitle)': timings.headerMs,
  'assemble doc + check':    timings.assembleDocMs,           // schema validation
  total:                     timings.totalMs,
});

renderBodyMs is usually the dominant phase for long documents. When calling parsePandocAST directly, opt in with { timings: true } — it defaults to off there.

Bundling

@sciflow/pandoc-web depends on pandoc-wasm like any other npm dependency and reaches it through its package entry point — import('pandoc-wasm'), resolved by your bundler the same way as lit or any other dependency. There is no path into node_modules anywhere in the shipped code, and nothing for you to copy by hand.

The engine's entry point imports ./pandoc.wasm and expects a URL back, which it then fetches and instantiates against its own WASI shim. That gives you exactly one obligation:

Configure your bundler to treat .wasm as an asset, not as a module. A bundler that compiles and links a bare .wasm import fails on this binary, because it imports the WASI interface (wasi_snapshot_preview1) that the engine supplies at instantiation time — you will see an unresolved wasi_snapshot_preview1 at build time.

The import is dynamic, so the ~58 MB binary lands in a chunk of its own and is fetched when a conversion actually starts — never on page load.

If your page has no bundler at all, name the binary yourself: Providing the engine yourself. If you have a bundler but must not ship a 58 MB asset, that setting alone is not enough — see Keeping the binary out of your build.

Vite

Give Vite a small pre plugin that emits the binary and hands the importer its URL, and keep .wasm in assetsInclude alongside it:

import { readFile } from 'node:fs/promises';
import path from 'node:path';

/** Emit every .wasm as a plain asset and hand the importer its URL. */
function wasmAsAsset() {
  return {
    name: 'wasm-as-asset',
    enforce: 'pre',
    async load(id) {
      const file = id.split('?')[0];
      if (!file.endsWith('.wasm')) return null;
      const ref = this.emitFile({
        type: 'asset',
        name: path.basename(file),
        source: await readFile(file),
      });
      return `export default new URL(import.meta.ROLLUP_FILE_URL_${ref}, import.meta.url).href;`;
    },
  };
}

export default {
  plugins: [wasmAsAsset()],
  assetsInclude: ['**/*.wasm'],
  // Development only: keep the dependency out of pre-bundling, which would
  // otherwise rewrite the asset URL out from under the engine.
  optimizeDeps: {
    exclude: ['pandoc-wasm', '@sciflow/pandoc-web'],
  },
};

The plugin runs first because Vite 8.2 and newer processes a plain .wasm import as a WebAssembly ESM integration before the asset pipeline can copy it. The build then stops with failed to resolve import "wasi_snapshot_preview1". On older Vite versions, assetsInclude alone was sufficient. A pre plugin produces the same result on both versions: one content-hashed asset and a module whose default export is its URL. The configuration therefore does not need to branch by Vite version.

assetsInclude stays as the companion: it costs nothing once the plugin has taken the binary, and it still covers any .wasm that reaches the asset pipeline by another route.

Webpack 5 / Rspack

module.exports = {
  module: {
    rules: [
      {
        test: /\.(wasm)$/,
        type: 'asset/resource',
      },
    ],
  },
};

Rollup

pandoc-wasm documents @rollup/plugin-wasm for plain Rollup:

import { wasm } from '@rollup/plugin-wasm';

export default {
  plugins: [wasm()],
};

Check the plugin's maxFileSize / targetEnv settings against the rule above — the binary has to come back as a URL to fetch, and it must never be inlined as base64.

Anything else

The requirement is the same in every toolchain: emit the pandoc-wasm dependency's .wasm binary into your output directory as a static asset and serve it. Serving it from a CDN or a versioned static path is fine — the engine fetches it like any other asset.

Size and caching

Size
pandoc.wasm, uncompressed ~58 MB
pandoc.wasm, over the wire with gzip ~16 MB
pandoc.wasm, over the wire with brotli ~11 MB
@sciflow/pandoc-web's own JavaScript ~33 KB (~10 KB gzipped)

The binary is fetched once, lazily, on the first conversion. Nothing is downloaded when the page loads, when the module is imported, or when <sciflow-pandoc-drop> is mounted — only when loadPandoc(), convertFile(), convertAstToBlob() or a file drop actually runs. Within a page, loadPandoc() memoizes the instance, so subsequent conversions cost nothing extra.

Across page loads, the browser's HTTP cache does the work, which means the response headers on pandoc.wasm decide whether a returning user pays 58 MB again. Serve it with a long Cache-Control: max-age — the file is content-hashed by Vite, Webpack and Rspack, so it is safe to treat as immutable — and enable compression for it.

Show a loading state

A first conversion on a cold cache takes as long as the download. Always give the user something to look at. <sciflow-pandoc-drop> renders its own status by default; when you drive the pipeline yourself, wire your own:

async function importFile(file) {
  setBusy(true, 'Loading the converter (one-time download)…');
  try {
    const result = await convertFile(file);
    setBusy(false);
    return result;
  } catch (error) {
    setBusy(false);
    showError(error.message);
  }
}

If you know an import is likely — a user opened the "Import" screen, say — you can warm the cache ahead of the file picker by calling loadPandoc() and ignoring the result. The instance is memoized, so the later conversion reuses it.

Providing the engine yourself

Everything above assumes a bundler emits pandoc.wasm for you. When that is not how your page is built, name the binary instead: @sciflow/pandoc-web then fetches it from your URL and instantiates it itself. At runtime the copy that came with pandoc-wasm is never fetched or instantiated. Your bundler still resolves the loader's import('pandoc-wasm') at build time, so either keep the asset rule or keep the binary out of the build as described below.

Reach for this when:

  • The page has no bundler. A plain <script type="module"> against a CDN or a checked-in file, with no build step to emit an asset.
  • The deployment must not carry 58 MB. A documentation site, a demo, a preview environment — anything whose artifact size is budgeted. Setting engineUrl is the first half of that; the second is keeping the binary out of the build, because a bundler still resolves the import even though nothing ever calls it.
  • A Content-Security-Policy restricts where code may come from. One origin you chose and can list in script-src, rather than whichever path the bundler happened to emit.
  • The deployment is offline or air-gapped. The binary becomes a file you place on your own host, next to everything else you already serve.

It changes nothing else. The same PandocWasm, the same convertFile() result, the same events — only the origin of the bytes.

Keeping the binary out of your build

engineUrl is a runtime setting. It decides where the bytes come from when a conversion runs; it does not change what your bundler does with the code. The loader's import('pandoc-wasm') is dynamic but its specifier is a literal, so a bundler still resolves the dependency at build time — and resolving it means walking into its ./pandoc.wasm import. Two consequences, both of which have surprised integrators:

  • Without a .wasm rule the build fails, exactly as described under Bundling — on current Vite the message is failed to resolve import "wasi_snapshot_preview1", even though the page never intends to use that copy of the engine.
  • With the asset rule from Bundling the build succeeds and emits the 58 MB file anyway. It is dead weight — nothing fetches it — but it is in your artifact.

To have neither, tell the bundler the dependency will never be needed. Mark it external:

// vite.config.js
export default {
  build: {
    rollupOptions: {
      // Nothing imports it at runtime — `engineUrl` supplies the engine.
      external: ['pandoc-wasm'],
    },
  },
};

…or alias it to an empty module, which also works in bundlers with no externals concept:

// vite.config.js
export default {
  resolve: {
    alias: { 'pandoc-wasm': new URL('./empty-module.js', import.meta.url).pathname },
  },
};

Either way the .wasm rule becomes unnecessary and no binary is emitted. Both are only safe when every call sets engineUrlloadPandoc(), convertFile(), convertAstToBlob() and the element's engine-url attribute alike. One call that falls back to the bundled engine fails at runtime: an unresolved module with the external route, an engine-less stub with the alias route.

The demo copy on the documentation site takes a third route, because it is a copy of a build rather than a build of its own: the demo is built with the asset plugin from Bundling, so its own output does contain the binary, and the script that copies it into the documentation site (docs/scripts/create-demo.mjs) leaves the .wasm behind and rewrites the demo's engine configuration to a hosted URL. Same result — a small artifact pointed at an engine it does not carry — reached without touching the demo's build.

Where the binary comes from

pandoc.wasm ships inside the pandoc-wasm npm tarball at src/pandoc.wasm. It is about 58 MB uncompressed, and it must come from pandoc-wasm@1.0.1 — the version @sciflow/pandoc-web pins. This package drives the engine through a calling convention that is not guaranteed across Pandoc-WASM builds; a mismatched binary fails at instantiation rather than producing a wrong document, but it fails.

Pull it out of the registry without installing anything:

npm pack pandoc-wasm@1.0.1
tar -xzf pandoc-wasm-1.0.1.tgz package/src/pandoc.wasm
mv package/src/pandoc.wasm public/pandoc-1.0.1.wasm

Or, if pandoc-wasm is already in your tree as a dependency of @sciflow/pandoc-web:

cp node_modules/pandoc-wasm/src/pandoc.wasm public/pandoc-1.0.1.wasm

Put the version in the filename or the path. You will want to be able to change it without fighting a cache — see below.

Hosting it

Requirement Why it matters
Content-Type: application/wasm The loader rejects content types other than application/wasm and octet-stream. This check identifies HTML error pages before the WebAssembly compiler reports a less specific “magic word” error.
Access-Control-Allow-Origin when the URL is cross-origin It is an ordinary fetch(). Same-origin needs nothing.
A long Cache-Control: max-age, immutable 58 MB is a first-visit cost you want the browser to pay once. This is why the version belongs in the URL: the file at a given URL never changes, so it can be cached forever and a new engine gets a new URL.
Compression (gzip or brotli) About 16 MB gzipped, about 11 MB with brotli. Configure it — the uncompressed transfer is the single largest thing your page will ever download.
A byte-exact copy Do not rewrite, minify or transform it. It is a binary.

Naming it

Every entry point takes the same option, and each is optional — leave it out anywhere and that call uses the bundled binary.

<sciflow-pandoc-drop engine-url="/pandoc-1.0.1.wasm"></sciflow-pandoc-drop>
drop.engineUrl = '/pandoc-1.0.1.wasm';

await convertFile(file, { engineUrl: '/pandoc-1.0.1.wasm' });
await convertAstToBlob(ast, { to: 'docx', engineUrl: '/pandoc-1.0.1.wasm' });
const pandoc = await loadPandoc({ engineUrl: '/pandoc-1.0.1.wasm' });
Surface Option Type
<sciflow-pandoc-drop> engine-url attribute / engineUrl property string
convertFile(input, options) options.engineUrl string
convertAstToBlob(ast, options) options.engineUrl string
loadPandoc(options) options.engineUrl string

An absolute http: or https: URL is used as given. A relative URL is resolved against the page (document.baseURI), so the same markup survives a move to a different path prefix. Anything else — a file: URL, an empty string, a string that is not a URL at all — is rejected before anything is fetched.

loadPandoc() memoizes one engine per resolved URL: two calls naming the same file share an instance and one download, two calls naming different files get one each. Two spellings of the same location count as one. As with the bundled engine, a load that failed stays failed for the life of the page — reload after fixing the deployment.

Use one engine URL per page. Mixing a bundled call with a hosted one, or two different hosted URLs, downloads and instantiates the engine more than once.

Licensing

Hosting the binary yourself does not change what it is: you are serving Pandoc, licensed GPL-2.0-or-later, from your own host. See Licensing below and check what that means for your distribution with whoever advises you on licensing.

About public CDNs

A public package CDN can serve the file — https://unpkg.com/pandoc-wasm@1.0.1/src/pandoc.wasm answers with Content-Type: application/wasm and permissive CORS — and that is a reasonable way to stand up a demo or a documentation page in one line.

It is not the recommended production path. A CDN you do not control decides your availability, your latency and your privacy story, on a 58 MB request every visitor makes. For anything real, serve the binary from your own host.

What is and is not supported

Input formats

convertFile() and <sciflow-pandoc-drop> accept three formats:

Format Extensions Notes
DOCX .docx Media extraction, Word style names and citation custom XML are all on by default.
Markdown .md, .markdown Pandoc's markdown reader.
LaTeX .tex, .latex Pandoc's latex reader.

Anything else — ODT, EPUB, HTML, RTF, JATS, reStructuredText — is reachable through loadPandoc(), which passes your from string straight to Pandoc. Only the three formats above have a translation path that has been exercised end-to-end against the manuscript schema.

DOCX citation custom XML

Documents written with the Zotero or Mendeley Word plugins are supported: the +citations reader extension reads their citation custom XML and --citeproc resolves it. Both are on by default for DOCX. Citations that were pasted as plain text, or flattened by "remove field codes", carry no custom XML and cannot be recovered — they arrive as ordinary text.

Media extraction

Supported for DOCX. Pandoc-WASM's WASI filesystem has no real directories, so the loader asks Pandoc to write a single zip and unpacks it in-process. The primary path reads the media the WASM wrapper already exposes; the zip fallback uses the browser's built-in DecompressionStream('deflate-raw'), which needs Chrome 103+, Firefox 113+ or Safari 16.4+.

Pandoc-WASM sandbox limits

These come from the WASM build itself and apply to every layer of this package:

  1. No HTTP requests from inside Pandoc. It cannot fetch a remote image, stylesheet or bibliography. Everything must be passed in as bytes.
  2. No external programs. Filters must be Lua; no executable filters, no shell-outs.
  3. No PDF output. PDF needs LaTeX, ConTeXt or Typst, which cannot run in the sandbox. Convert to LaTeX or HTML and generate the PDF elsewhere.

Output formats

convertAstToBlob() accepts docx, odt, epub, epub3, html, html5, markdown, gfm, commonmark, latex, rtf, rst, asciidoc and plain. PDF is not among them, per the sandbox limit above.

Browsers

The same baseline as the rest of SciFlow — see Browser & Mobile Compatibility — with one exception: the zip fallback used for DOCX media extraction needs DecompressionStream('deflate-raw'), which is Chrome 103+, Firefox 113+ or Safari 16.4+, above the editor's own Chrome 94 / Firefox 93 / Safari 15 floor. The fallback runs only when the engine does not hand the media over directly, so a browser between the two floors imports most documents normally — but an import that does reach the fallback there fails rather than degrading.

Both packages are ESM-only; there is no CommonJS or UMD build. Conversion is CPU- and memory-heavy, so a large DOCX on a low-memory mobile device may fail where the same file converts fine on a desktop.

What gets imported

Source feature Mapping
Word headings (Heading 1–6) heading with a level attribute
Numbered and bulleted lists ordered_list / bullet_list
Word tables figure containing a table with a caption
Embedded images figure whose src is media:<path>
LaTeX math ($…$, $$…$$) math carrying the TeX source
Footnotes footnote
Hyperlinks anchor mark with href
Zotero / Mendeley citations citation resolved against references (Pandoc emits CSL JSON)
Title / subtitle metadata result.title / result.subtitle, and the document header
Affiliations (YAML metadata or DOCX custom XML) result.affiliations
DOCX custom properties result.altTitles

A construct with no mapping does not silently become prose. A top-level block the importer cannot render — a Pandoc node type no renderer covers, or content the manuscript schema rejects — is recorded in parseResult.errors[] with the block's complete source Pandoc JSON, and, by default, replaced in the document by a paragraph that says the block could not be imported. Pass skipInvalid: true to leave it out of the document instead; either way the entry in errors[] is the same. See Handling failures below, Schema Reference for the resulting node and mark shapes, and Import & Export for the wider conversion picture.

Handling failures

Failures come in three flavours, and they surface in different places. Two of them are worth stating up front, because they are the ones that catch integrators out:

  • A fatal failure does not throw — it resolves with doc: null. parsePandocAST returns a normal result whose doc is null and whose errors[] carries the fatal entry. convertFile() passes that result through, so its promise resolves too, and <sciflow-pandoc-drop> announces phase: 'done' and fires sciflow-document with a null parseResult.doc. No sciflow-error is fired. A handler that assumes sciflow-document means success will hand null to the editor.
  • The one thing that does throw is a malformed AST. parsePandocAST throws a plain Error when pandocDocument.blocks is missing — that is a caller mistake, not a document problem.

So: check parseResult.doc before using it, in every loading path, whether you drive the element, convertFile(), or parsePandocAST() directly.

drop.addEventListener('sciflow-document', (event) => {
  const { parseResult } = event.detail;
  if (!parseResult.doc) {
    // Fatal — the reason is the last entry in errors[].
    showBanner(parseResult.errors.at(-1)?.message ?? 'The document could not be imported.');
    return;
  }
  // …load it
});

Per-block problems: parseResult.errors[]

The importer does not abandon a document because one block is unusable. Each problem is recorded and the rest of the document still loads:

for (const err of result.parseResult.errors) {
  console.warn(err.severity, err.message, 'block', err.blockIndex);
  if (err.pandocNode) {
    // The complete, untruncated source Pandoc JSON for the block.
    console.debug(JSON.parse(err.pandocNode));
  }
}

ImportErrorPayload fields:

Field Type Meaning
message string What went wrong.
severity 'error' \| 'warning'
blockIndex number \| undefined Index of the top-level block in the source AST.
pandocType string \| undefined The Pandoc node tag that triggered it.
pandocNode string \| undefined The complete, untruncated source Pandoc JSON for the affected block, as a string. Present whenever the diagnostic concerns one specific block — so nothing about the block is lost, even when it could not be rendered.
imageCount number \| undefined Images found recursively inside the block.
hasUnplacedImages boolean \| undefined true when the block carries two or more images that were rendered inline but not turned into a figure. The content is left untouched — a human has to decide the layout (one figure with several graphics, or several figures).

What happens to the block itself is up to skipInvalid: by default (false) the document keeps a paragraph in its place saying it could not be imported, so the gap is visible to the person reading the document; with skipInvalid: true the block is omitted and only errors[] records it. The entry is identical either way, and it carries the block's complete source Pandoc JSON — nothing is destroyed.

Show this list in your UI so users can review documents that import with warnings.

Thrown errors: ImportError

At the renderer level, render() and renderContent() throw ImportError, an Error subclass with a payload describing what could not be rendered and a toJSON() for logging. parsePandocAST catches these and converts them into errors[] entries, so you only meet ImportError directly if you call the renderer surface yourself.

parsePandocAST throws a plain Error for exactly one situation: pandocDocument.blocks missing. A fatal failure while assembling the document does not throw — it resolves with doc: null and the reason appended to errors[], as above.

convertFile() throws when the input format cannot be resolved, when the input is not one of the accepted types, or when Pandoc itself fails. The <sciflow-pandoc-drop> element catches all of these and re-emits them as sciflow-error:

drop.addEventListener('sciflow-error', (event) => {
  showBanner(event.detail.message);
});

The element also renders the message itself in a role="alert" box unless you set hide-error. The same failure is announced on sciflow-status with phase: 'error', so a host that drives its UI from status events alone still sees the run close.

The WASM never arrives

If pandoc.wasm 404s, the engine's own fetch resolves with a non-OK response and the failure surfaces as a WASM instantiation error — typically CompileError or magic word wording, because the 404's HTML body is handed to the WASM compiler. This is almost always a bundling problem, not a runtime one: see WASM fetch fails with 404 after bundling.

Because loadPandoc() memoizes its promise, a failed load stays failed for the life of the page. Reload the page after fixing the deployment.

The engine URL returns HTML, or the wrong MIME type

When you host the binary yourself, a mistyped path or a single-page-app fallback answers 200 OK with a page instead of a binary. The loader checks the response before handing 58 MB to the WebAssembly compiler and fails with the URL and the type it got:

@sciflow/pandoc-web: https://example.org/pandoc.wasm answered with content type
"text/html", not a WebAssembly binary. Serve pandoc.wasm with
"Content-Type: application/wasm"; a host answering with HTML is usually serving
an error or index page instead of the file.

Two causes, in order of likelihood:

  1. The file is not there. Your host is serving its index page or its 404 page for every unknown path. curl -I <your engine url> and look at the status and the content type; a real hit answers 200 with application/wasm (or an octet-stream, which the loader also accepts).
  2. The file is there but the MIME type is wrong. Some static hosts do not know the .wasm extension and send text/plain or text/html. Configure application/wasm for it.

A missing file usually gets a clearer message still — HTTP 404 Not Found with the URL in it — because most hosts do answer 404 for a path they cannot serve.

If the fetch fails outright rather than answering, the message names reachability and CORS: a cross-origin engine URL needs Access-Control-Allow-Origin on the response. If the bytes arrive but will not instantiate, the message says so and names the version the file has to be — that is almost always a binary from a different Pandoc-WASM build.

Licensing

  • @sciflow/pandoc-ast and @sciflow/pandoc-web are MIT.
  • They depend on pandoc-wasm, whose JavaScript wrapper is MIT and whose bundled pandoc.wasm binary is Pandoc itself, licensed GPL-2.0-or-later. The pandoc-wasm npm package declares GPL-2.0-or-later as its license.

Shipping @sciflow/pandoc-web distributes that GPL-2.0-or-later binary to your users, whether your bundler emits it or you host it and specify engineUrl. Consult your licensing adviser about the implications for your distribution.

Try it

Two demos in this repository exercise the import path:

  • The main editor demo (packages/editor/start/demo/) has an import action that runs a file through this pipeline and loads the result into the editor — the shortest look at what a user experiences.
  • The pandoc-import developer demo (packages/editor/start/demo/pandoc-import/) is the wiring reference. It shows the converted document alongside tabs for the raw Pandoc AST, the parsed parseResult source, the extracted media, the references, and the generated JATS XML — the view to open when you want to see what a specific DOCX actually turned into.

Both are served by the same dev server; see the demo folder's own README for how to run it.