Skip to content

Content import & export

SciFlow documents are stored as JSON snapshots. The schema package provides export capabilities for scholarly publishing formats and validation tools for document integrity.

Snapshot format

Every SciFlow document is wrapped in a snapshot:

{
  "doc": { "type": "doc", "content": [...] },
  "files": [],
  "references": [],
  "version": 1
}
  • doc — The ProseMirror document tree as JSON. See Schema Reference.
  • files — Metadata for attached media (images, data files).
  • references — Bibliography entries used by citation nodes.
  • version — Optional version counter for optimistic locking.

JATS XML export

SciFlow can export the document body to JATS 1.4 (Blue) XML, the standard format for scholarly article interchange.

Usage

import { generateJatsBody } from '@sciflow/schema-prosemirror';

// Generate JATS <body> XML straight from the snapshot's doc JSON
const xml = generateJatsBody(snapshot.doc, {
  pretty: true,
  indent: 2,
});

generateJatsBody takes document JSON, not a live ProseMirror Node: its parameter type is PMNode, the plain { type, attrs?, content?, marks?, text? } shape Node.toJSON() produces. If what you hold is a live node — from editor.document, say — serialize it first:

const xml = generateJatsBody(pmDoc.toJSON(), { pretty: true });

What gets exported

Document element JATS output
Parts (chapter, abstract, and so on) <sec> with appropriate sec-type
Headings Auto-sectioning: headings open nested <sec> elements
Paragraphs <p>
Bold / italic / sup / sub <bold>, <italic>, <sup>, <sub>
Citations <xref ref-type="bibr"> with decoded source IDs
Footnotes Collected into an <fn-group> at the end of the last top-level <sec>
Math (TeX) <disp-formula> or <inline-formula> with <tex-math>
Math inside a heading or subtitle Always <inline-formula>, whatever the node's style — <title>/<subtitle> admit no <disp-formula>. Math inside a footnote that sits in a heading is unaffected.
Poetry <verse-group>; each child paragraph becomes one <verse-line>, nested poetry stays a nested group
Verbatim <preformat preformat-type="verbatim"> with the child paragraphs joined by newlines (their ids are dropped — <preformat> admits no <p> to carry them)
Bookmark Empty <target> carrying the node's id
Figures <fig> with <caption>, <label> and <graphic> (accessible text as <alt-text>)
Tables <table-wrap> with HTML-style table markup
Lists <list list-type="bullet"> or <list list-type="order">
Blockquotes <disp-quote>
Code blocks <code>
Hyperlinks <ext-link>

Options

interface JatsBodyOptions {
  pretty?: boolean;  // Indent the output for reading (default: false)
  indent?: number;   // Spaces per indent level (default: 2)
}

Indentation is off by default because the output is meant for machines. When you turn it on, line breaks are inserted only between block-level elements: elements holding mixed content (title, p, preformat, verse-line, td, …) are written on a single line, so no whitespace is ever added inside their text.

Known limitations

generateJatsBody targets the JATS 1.4 Publishing ("Blue") DTD. Poetry, verbatim blocks, figures, footnotes, and element ids all now produce DTD-valid markup. The following constructs do not yet validate against the Publishing DTD:

  • The document title (from a header node) renders as a bare <title> (and <subtitle>) directly under <body>, not inside <front><title-group><article-title>.
  • A <sec> without a <title> — a part with no heading — is invalid under Publishing, though the more permissive Archiving DTD accepts it.
  • A bibliography part renders <ref> elements directly, not wrapped in a <ref-list>.
  • An index term wraps the marked (indexed) text inside <index-term>, rather than leaving that text in the running prose and confining the machine-readable index string to <term>.
  • A hard line break renders as <break/> inside <p>.
  • An ordered list starting above 1 emits a start attribute the DTD does not declare.
  • A subtitle nested inside a section is not valid JATS in that position.
  • A link whose href is an external URL renders as an <xref> with a rid derived from that URL, which does not resolve to anything in the document.

Treat generateJatsBody's output as a starting point for further validation and post-processing against your target DTD, not a drop-in Publishing-valid document.

Semantic notes (valid, but maybe not what you expected)

These produce DTD-valid markup and still deserve a decision on your side:

  • A generated citation (citationMode: "generated") renders as an empty <xref ref-type="bibr" rid="…"/>. An empty <xref/> is valid — the label is meant to be produced downstream from rid — but a pipeline that copies the XML straight into a rendering surface shows nothing where the citation was. citationMode: "custom" carries the authored inline content instead.

Element ids are repaired, and the repair is not reversible

JATS types id as ID and rid as IDREFS, so both must be XML Names: they cannot begin with a digit and cannot contain arbitrary punctuation. Document identifiers routinely are neither — a reference keyed 299, a figure keyed ref 1. Every id and every rid the generator writes therefore goes through one repair:

  1. The identifier is trimmed; a blank or non-string id produces no attribute at all.
  2. Every character an XML Name may not contain is replaced by -. A colon counts as one of those: : is legal in an XML Name but reads as a namespace prefix, so it is replaced too.
  3. If the result still opens with a character that cannot start a Name — a digit, a hyphen, a dot — id- is prefixed.

An identifier that already is an XML Name is written unchanged. The mapping is pure, so the same input always yields the same output and cross-references inside one generated <body> keep resolving.

It is not injective. Distinct document ids can collide after repair:

Document ids Both become
299, id-299 id-299
ref 1, ref:1, ref-1 ref-1

Two consequences for anything assembling a JATS article around this <body>:

  • Apply the same mapping to ids you write yourself. A <ref id="…"> in your <ref-list>, or any <fig id>/<fn id> you generate outside generateJatsBody, must be repaired identically or the rids in the body will not resolve.
  • Check for collisions before publishing. Duplicate ID values are a validity error. Real documents rarely carry both spellings of the same id, but a merge of several sources can — repair your ids first, then verify they are still unique.

JSON schema validation

The schema package can generate JSON Schema definitions from the live ProseMirror schema. Use these to validate documents outside the editor. Both generators emit Draft 2020-12 ($schema: https://json-schema.org/draft/2020-12/schema, definitions under $defs), so point a 2020-12-capable validator at them.

Generating schemas

Two functions produce them, both from a live ProseMirror Schema:

Function Validates
generateJsonSchema(schema) The doc portion of a snapshot
generateSnapshotSchema(schema) The full snapshot (doc + version + selection + files + references)

The package ships no pre-generated .json files and exposes no deep import paths — its exports map publishes only the package entry — so generate the schema at build time or at startup from the schema you actually use.

Validating a document

// Ajv's default entry point is Draft-07; the 2020-12 build is a separate export.
import Ajv2020 from 'ajv/dist/2020.js';
import { generateSnapshotSchema, manuscript } from '@sciflow/schema-prosemirror';

// Generate once and reuse the compiled validator; it is not cheap to rebuild.
const ajv = new Ajv2020();
const validate = ajv.compile(generateSnapshotSchema(manuscript));

if (!validate(snapshot)) {
  console.error('Invalid document:', validate.errors);
}

Pass your own Schema instead of manuscript when the editor runs with schema extensions — a document is only valid against the schema that produced it.

To keep the schemas as files (to check them into a repository, or to hand them to a validator in another language), run the generator script in this workspace:

npx nx run @sciflow/schema-prosemirror:generate-schema

It writes manuscript.schema.json and manuscript-snapshot.schema.json into the package's dist/ directory. The underlying script (packages/schema/prosemirror/scripts/generate-json-schema.ts) takes --out-dir <path> to put them somewhere else. Neither file is part of the published package — generate them where you need them.

Importing content

SciFlow ships two complementary packages for ingesting external documents:

  • @sciflow/pandoc-ast — pure-JSON translator. Pandoc JSON AST in, ProseMirror document out. No DOM, no filesystem, no network.
  • @sciflow/pandoc-web — browser-side wrapper around Pandoc-WASM. File / Blob in, fully-resolved snapshot out. Pairs with @sciflow/pandoc-ast for the AST translation step.

Together they cover DOCX (including Zotero / Mendeley citation XML), Markdown, and LaTeX. See the Importing documents guide for end-to-end examples and the pandoc-import demo (packages/editor/start/demo/pandoc-import in the repository) for a working drag-and-drop integration.

One-shot: file → snapshot

import { convertFile } from '@sciflow/pandoc-web';

const file = await fileInput.files[0];
const { parseResult, ast, media, timings } = await convertFile(file, {
  // Defaults: citeproc + DOCX styles + Zotero citation XML are all on.
  // Tweak per-call if you want to skip a stage.
});

// A parse result is not an editor snapshot. `doc` is null on a fatal failure,
// and the editor's `references` is a read-only getter — document and
// bibliography go in together through the writable `doc` property.
if (parseResult.doc) {
  editor.doc = {
    doc: parseResult.doc,
    // ReferenceEntry `{ id, csl, label? }` → SnapshotReference `{ id, rawReference }`.
    references: parseResult.references.map((reference) => ({
      id: reference.id,
      rawReference: reference.label ?? String(reference.csl?.title ?? reference.id),
      csl: reference.csl,
      mimeType: 'application/vnd.citationstyles.csl+json',
    })),
    files: [],
  };
}

Extracted media does not travel in files: a SnapshotFile names an asset by URL, while media[] carries Blobs. Rewrite each image's media:<path> src to a URL you control first. The full recipe — media rewriting, the reference mapping, and what files is for — is in Importing documents.

Drop-zone web component

<script type="module">
  import '@sciflow/pandoc-web';
</script>

<sciflow-pandoc-drop accept=".docx,.md,.tex"></sciflow-pandoc-drop>

<script type="module">
  document
    .querySelector('sciflow-pandoc-drop')
    .addEventListener('sciflow-document', (event) => {
      const { parseResult } = event.detail;
      // Fires with a null `doc` when assembly failed — no error event follows.
      if (parseResult.doc) editor.doc = { doc: parseResult.doc };
    });
</script>

AST → other formats

The same Pandoc-WASM instance can serve AST → DOCX / HTML / EPUB / Markdown round-trips via convertAstToBlob:

import { convertAstToBlob } from '@sciflow/pandoc-web';

// Import hands you `MediaBlob[]` (`{ path, blob }`); export wants
// `PandocMediaFile[]` (`{ path, bytes }`). Convert before passing them on.
const exportMedia = await Promise.all(
  media.map(async (entry) => ({
    path: entry.path,
    bytes: new Uint8Array(await entry.blob.arrayBuffer()),
    mimeType: entry.mimeType,
  })),
);

const { blob, filename } = await convertAstToBlob(ast, {
  to: 'docx',
  media: exportMedia,
  // Name the same engine the import used, when it named one at all.
  engineUrl,
});

Two things this is not: it renders the Pandoc AST you pass, so edits made in the editor after the import are not in the output; and it loads an engine like every other entry point, so a page that imported with engineUrl must pass the same engineUrl here or it downloads and instantiates a second one.

Building a custom importer

If you need to import from a format Pandoc doesn't cover, construct a valid ProseMirror JSON tree directly and validate it against the generated JSON Schema:

// Minimal valid document
const imported = {
  doc: {
    type: 'doc',
    content: [
      {
        type: 'heading',
        attrs: { level: 1, id: 'title-1' },
        content: [{ type: 'text', text: 'Imported Article' }],
      },
      {
        type: 'paragraph',
        content: [{ type: 'text', text: 'First paragraph of imported content.' }],
      },
    ],
  },
  files: [],
  references: [],
};

Schema migration

When the ProseMirror schema changes between versions (new attributes, renamed nodes, removed fields), existing documents may need migration.

Migration strategy

  1. Regenerate the JSON schema after any schema change: 

    npx nx run @sciflow/schema-prosemirror:generate-schema

  2. Validate existing documents against the new schema to identify breaking changes.

  3. Write a migration function that transforms old JSON to the new format: 

    function migrateV1toV2(snapshot: any): SyncSnapshot {
  // Walk the document tree and transform nodes
  const migrateNode = (node: any) => {
    if (node.type === 'old_node_name') {
      node.type = 'new_node_name';
    }
    if (node.content) {
      node.content.forEach(migrateNode);
    }
    return node;
  };

  migrateNode(snapshot.doc);
  return snapshot;
}

  4. Run migrations at load time in your sync strategy's load() method, before returning the snapshot to the editor.