Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ferro-hgvs

A high-performance HGVS variant nomenclature parser and normalizer, written in Rust with Python bindings. It supports every HGVS coordinate system (g / c / n / r / p / m / o) and edit type (substitution, deletion, insertion, duplication, inversion, repeat).

Use it as a Python package, a Rust crate, or a command-line tool. API docs are on docs.rs; source is on GitHub.

Start here

  • Installationpip install ferro-hgvs, cargo install ferro-hgvs, or the Rust crate.
  • Quickstart — parse a variant, then normalize one.

Guides

Interpretation

The Interpretation section is a per-line reading of the HGVS recommendations that records exactly how ferro interprets each clause — with runnable example spellings, the form ferro normalizes each input to, and the reasoning behind each decision. It is under construction, one spec page at a time.

Installation

ferro ships as a Python package, a Rust crate, and a standalone command-line tool. Install whichever fits your workflow — they share the same engine.

Python

pip install ferro-hgvs

Pre-built wheels are available for Linux (x86_64, aarch64), macOS (x86_64 and Apple Silicon), and Windows (x86_64) on Python 3.10+.

Command-line tool

cargo install ferro-hgvs

This builds the ferro binary from source and places it on your PATH. It needs a Rust toolchain (rustup).

Rust library

Add the crate to your project:

[dependencies]
ferro-hgvs = "0.14"

Verify the install

ferro parse "NM_000088.3:c.459A>G"
import ferro_hgvs
print(ferro_hgvs.parse("NM_000088.3:c.459A>G"))

Parsing needs no reference data. Normalization against real transcripts does — see Reference data.

Quickstart

This walks through parsing a variant, then normalizing one against reference data.

Parse a variant

Parsing validates a description and returns its structure. It needs no reference data.

ferro parse "NM_000088.3:c.459A>G"
import ferro_hgvs

variant = ferro_hgvs.parse("NM_000088.3:c.459A>G")
print(variant.variant_type)  # "coding"
print(variant.reference)     # "NM_000088.3"
print(str(variant))          # "NM_000088.3:c.459A>G"

Normalize a variant

Normalization rewrites a description into its canonical form. It needs reference sequences, so first prepare a reference (a one-time download — see Reference data):

ferro prepare --output-dir ferro-reference
ferro check --reference ferro-reference

Then normalize:

ferro normalize "NM_000088.3:c.459del" --reference ferro-reference/
import ferro_hgvs

normalizer = ferro_hgvs.Normalizer.from_manifest("ferro-reference/manifest.json")
print(normalizer.normalize("NM_000088.3:c.459del"))

Read the warnings

Normalization sometimes repairs a description in a way the normalized string does not itself record — separately reported members merged into one delins (MEMBERS_COALESCED_FROM_REPORTED_FORM), a reference-range insertion payload replaced by the bases it denotes (INSERTED_SEQUENCE_EXPANDED), or a stated reference base that contradicted the reference and was accepted anyway (REFSEQ_MISMATCH).

These are reported as warning[CODE]: message on stderr (and in the warnings array under --format json, the detail column under --format tsv), so a pipeline reading only stdout will not see them. The message on each warning[CODE]: line says what the code means.

Next steps

Normalize variants

Normalization rewrites a description into its canonical form — 3′-shifting, collapsing equivalent spellings, and applying the recommendations’ preferred representation. This guide covers batches, files, output formats, and error modes.

Normalization needs reference sequences. Prepare a reference first (see Reference data) and pass --reference.

A single variant

ferro normalize "NM_000088.3:c.459del" --reference ferro-reference/

Read from stdin instead:

echo "NC_000001.11:g.12345A>G" | ferro normalize --reference ferro-reference/

A batch from a file

One description per line:

ferro normalize -i variants.txt --reference ferro-reference/ -o normalized.txt

Sort the input by transcript accession (or genomic position) for large batches. ferro caches each resolved transcript, so consecutive variants on the same transcript skip the dominant cost of re-reading it. Sorted input keeps the working set resident and is markedly faster.

Use several workers for large batches:

ferro normalize -i variants.txt --reference ferro-reference/ -j 8

Output formats

-f/--format selects the output:

  • text (default) — the normalized description, one per line.
  • json — a structured record per input, including a warnings array.
  • tsv — a table with header line, input, normalized, changed, status, detail, plus a summary line on stderr. This is the format for answering “which of my variants changed?”
ferro normalize -i variants.txt --reference ferro-reference/ -f tsv > normalized.tsv

Error modes

--error-mode controls how strict ferro is about its input and output:

  • strict (default) — validates that the input conforms to the recommendations and fails on a violation.
  • lenient — accepts a wider range of inputs and repairs where it can; fails only when it cannot normalize.
  • silent — lenient, but without the diagnostic messages.

Regardless of mode, normalization may repair a description in a way the output string does not record. Those repairs are reported as warning[CODE]: message on stderr (and in the warnings array under -f json, the detail column under -f tsv). A pipeline reading only stdout will not see them, so capture stderr or use -f json/-f tsv. The message on each warning[CODE]: line says what the code means; use --ignore / --reject to tune specific codes.

Normalize or rederive?

ferro exposes two entry points for canonicalizing an HGVS description, but they are different operations with different guarantees. This guide says what each does, which axes each supports, and when to reach for one over the other.

normalizerederive
what it takesan HGVS descriptionan HGVS description
what it doesshifts and re-spells the given description in placethrows the spelling away and re-derives from the denoted bases
axesevery axis — g/c/n/r/p/mgenomic only — g. (and m. on the two rCRS accessions)
confluencebest-effort (rule 3) — converges equivalent inputsconfluent by construction, over one window
needs a referenceyesyes

normalize — canonicalize a description you were handed

Use normalize when you already have a description and want its canonical form: 3′/5′ shifting and the recommendations’ preferred representation, applied to the spelling you passed in. It works on every axis, because a c./r./p. description carries information — reading frame, which transcript, a protein consequence — that a genomic base window cannot recover.

import ferro_hgvs

nz = ferro_hgvs.Normalizer(reference_json="…")
nz.normalize("NM_000088.3:c.100delA")   # -> "NM_000088.3:c.100del"

Converging equivalent inputs to one output is confluence — rule 3 of the normalization rules, which normalize targets on a best-effort basis: a handful of known residual defects remain, where two spellings of one variant still normalize to two strings. Those are tracked bugs, not the intended contract.

rederive — one canonical description per variant

Use rederive when the input’s spelling should carry no weight — when you want the description for a variant, regardless of how it happened to be written. rederive expresses the variant as the bases it denotes (a reference/alternate window) and derives a description from those bases alone, so two spellings of one variant reach one result:

nz.rederive("NC_TEST.1:g.14_15insACGTACGT")   # -> "NC_TEST.1:g.7_14dup"
nz.rederive("NC_TEST.1:g.7_14dup")            # -> "NC_TEST.1:g.7_14dup"

Because it re-derives from a genomic sequence window, rederive is genomic-only and refuses a transcript or protein axis. Project onto the axis you need afterwards if required.

rederive returns the alignment-derived form by default. Pass recommended_form=True to route the result through normalize for ferro’s recommended, reference-anchored form:

nz.rederive("NC_TEST.1:g.11del", recommended_form=True)   # 3′-shifted per the recommendations

rederive is the single-call form of the to_sequencesfrom_sequences round trip described in Deriving a description from sequences; read that guide for the mechanics and for the read-dependence cost of a window-local derivation.

Which one?

  • You have a description and want it in canonical form on its own axis → normalize.
  • You want the same description for a variant no matter how two callers spelled it, on a genomic axis → rederive.
  • You have raw bases (a reference/alternate window) rather than a description → from_sequences.

Project a variant to another axis

ferro project re-expresses a variant on a chosen output axis — genomic (g), coding (c), non-coding (n), protein (p), or RNA (r) — against a transcript. It is how you move a genomic call onto a transcript, or read the protein consequence of a coding one.

ferro project --axis c --transcript NM_000059.4 --reference <dir> \
  'NC_000013.11:g.32316466T>G'
# NC_000013.11(NM_000059.4):c.6T>G

Coding-axis rules apply on the projected axis

The projected c./r. axis is normalized on the transcript, so it applies the coding-axis rules a bare genomic axis cannot — and produces the same string ferro normalize gives for the coding-authored form of the variant, whichever axis you started from.

The clearest case is the coding one-amino-acid delins exception (DNA/delins.md:18): two substitutions one nucleotide apart within a single codon are written as one delins, not as two members.

# BRCA2 codon 2 (c.4_6 = CCT): c.4 C>A and c.6 T>G, with c.5 unchanged.
ferro project --axis c --transcript NM_000059.4 --reference <dir> \
  'NC_000013.11:g.[32316464C>A;32316466T>G]'
# NC_000013.11(NM_000059.4):c.4_6delinsACG

Because both members fall in one codon, the coding axis merges them into c.4_6delinsACG — identical to ferro normalize on NM_000059.4:c.[4C>A;6T>G]. The genomic axis has no reading frame, so it keeps the members individual, and a pair that straddles a codon boundary or an exon junction stays split on every axis.

Pairs with deriving from sequences

If you hold bases rather than a description — a window from a BAM, a VCF row — derive the genomic description with from_sequences first (see Deriving a description from sequences), then project it onto the transcript:

# 1. derive the g. description from the bases
# 2. project it onto the transcript
ferro project --axis c --transcript NM_000059.4 --reference <dir> '<the g. description>'

Deriving a description from sequences

If what you have is bases rather than a description — a window out of a BAM, a VCF row, an aligner’s output — from_sequences derives the description instead of asking you to spell one:

import ferro_hgvs

ferro_hgvs.from_sequences("NC_000001.11", 1000, "AGCGT", "AGT")
# NC_000001.11:g.1002_1003del
use ferro_hgvs::{from_sequences, FromSequencesOptions};

let variant = from_sequences("NC_000001.11", 1000, "AGCGT", "AGT",
                             &FromSequencesOptions::default())?;

The axis follows the accession: g., or m. on NC_012920 / NC_001807, which HGVS requires the m. coordinate system for. Every other accession class — transcript, protein, UniProt — is refused with a message naming it.

It reads no reference sequence. The output is a pure function of its arguments — the accession, the position, the two sequences and the options — so the same bases give the same description on any machine, against any reference build, with no hidden input. position is 1-based, and reference is taken on trust: verifying it would need the reference and would make the provider a hidden input, costing exactly the determinism the function exists to provide.

That is five values, not four. max_grid_cells is not inert — it decides whether an answer is produced at all — so the “four arguments” this section and the Rust docs both used to claim is withdrawn. Purity is the property; the count was wrong. (FromSequencesOptions carries a direction too, but it is not a caller-facing knob: it is #[doc(hidden)], always 3’ on every shipped path, and exists for the internal differential oracle described under rule 6.)

How to: one canonical description per variant

The job this exists for — a pipeline that aligns reads, post-processes a BAM, and wants one description per variant, decided by the observed bases and nothing else.

1. Get a window. From a pileup, a VCF row, or an aligner’s output: the reference bases over some interval, the observed bases over the same interval, and the 1-based position of the window’s first base. If what you hold is a description rather than bases, Normalizer::to_sequences produces the same window — see Going the other way.

2. Derive, and read the flag.

d = ferro_hgvs.from_sequences_detailed("NC_000001.11", 1000, "AGCGT", "AGT")
d.variant                      # NC_000001.11:g.1002_1003del
d.placement_bounded_by_window  # False — the bases settled the placement, not the window

# The same variant, read through a window that stops at it:
d = ferro_hgvs.from_sequences_detailed("NC_000001.11", 1000, "AGCG", "AG")
d.variant                      # NC_000001.11:g.1002_1003del — the same answer
d.placement_bounded_by_window  # True — the deletion is flush with the window's 3' edge

The two rows are the flag’s whole meaning: same description, different confidence that a wider read would agree. The first window has a base to spare 3’ of the deletion, so the placement is settled by the bases; the second ends exactly where the deletion does, so the flag fires even though the answer is unchanged. This example used to show only the second window while claiming False, which inverted both the value and its explanation.

Prefer from_sequences_detailed to from_sequences in a pipeline. The flag is the only thing that tells you the window may have decided the answer, and a bare from_sequences discards it.

3. Store it, or normalize first. What you have is already conformant (rule 1) and deterministic (rule 4), so it is safe to store and to compare between runs and between machines. It is not necessarily the recommended form, and it is not guaranteed to agree with a description derived from a different window — those are rules 2 and 3, and both need the reference. If you want them, run normalize on the result. That is the whole offer: derive now, normalize later, or never.

4. If the flag is set and you need the reference-anchored answer, either re-derive from a wider window or run normalize. Do not treat a flagged result as wrong — see below.

How wide should the window be?

Wide enough to contain the whole interval over which the change could legally be placed. That is the exact condition, and it is what the cost section pins.

Operationally: pad on both sides by at least the length of the longest ambiguous run — a homopolymer or tandem repeat — the variant might sit in. An insertion or duplication needs one further base 5’, since that is where a 5’-most insertion anchors. to_sequences pads by 128 on each side, so its window is span + 2 * pad; that covers ordinary repeats comfortably, and is worth raising if you work with long tracts.

Two things follow that are easy to get backwards:

  • A window that cuts the interval does not give a wrong answer, it gives a bounded one. The description still denotes the same bases and carries the same canonical SPDI; it is simply placed at the window’s edge rather than the run’s. What you lose is the preferred spelling, not correctness.
  • placement_bounded_by_window is conservative on purpose. It reports “this could have moved”, not “this is wrong” — a window flush with a tract is flagged and is nonetheless the same answer a whole-sequence derivation gives. Distinguishing the two needs the reference, which this function does not read.

The refusals, and what to do about each

The policy is to refuse rather than quietly answer with a weaker rule, so each of these is actionable rather than fatal:

refusalwhywhat to do
the accession is not genomicthis surface emits g. (and m. on the two rCRS mitochondrial accessions) and nothing else; NM_…:g.9_10del would be well-formed and denote nothingpass the genomic accession, and project afterwards if you need a transcript axis
an inserted payload sits against the window’s 5’ edgeHGVS writes an insertion between two positions, so it would have to anchor at position - 1 — outside the windowre-fetch with more 5’ flank. How much is direction-dependent, so widen rather than adding a fixed one base
the alignment grid exceeds max_grid_cellsa cost bound — a cell is roughly 18 bytes, and the default admits a window of about 4 096 basesraise max_grid_cells if you have the memory, or narrow the window. Real structural alleles are far past any sane budget: LRG_542:g.[101177_102434delins36;107248_127198delins21] spans 26 kb
a symbol outside the IUPAC-IUBMB set, a zero position, an empty referencethe input cannot denote anythingfix the input. X and - are refused deliberately (standards.md:39) — they are alignment symbols, not bases
U in either sequencethis surface’s axis is DNA; a g./m. description naming U would be well-formed and wrongpass T, and project onto an r. axis afterwards if you need RNA

Case is not a refusal: a soft-masked (lower-case) window derives exactly as its upper-case twin does, and both sequences are folded before anything reads them.

Which rules it delivers

This is the whole design, and it falls straight out of the four normalization rules:

rules deliveredforceneeds
from_sequences1 (conformant), 4 (deterministic)both absolutethe caller’s four arguments
normalize, afterwards2 (recommended form), 3 (confluent)both best effortthe reference

Rules 1 and 4 are the two the normalization rules call always achievable, so a function that has only the caller’s arguments can still deliver both in full. Rules 2 and 3 need the reference: rule 2’s scope names the 3’ rule explicitly, and a reference-anchored shift is precisely what a window-local function cannot perform.

So an output may be 3’-shiftable further than the window allowed, and that is not a defect. Run normalize afterwards if you want it — Normalizer::from_sequences(..., recommended_form = true) does both in one call. (Normalizer::rederive is the one-call path when you have a description rather than a sequence pair — see Normalize or rederive?.)

Run it unless you have a reason not to. In an internal sweep of many synthetic shapes, normalize moved a meaningful share of derived descriptions — in three classes: repeat notation (g.27_28insAAAg.27A[4]), reference-anchored member re-derivation, and an inversion spread across several members (g.[17C>A;19T>A;21T>G]g.17_21inv, which the alignment DAG partitions before anything can see it, since it minimises edit distance and an inversion is not in that cost model). All three are rule 2 and rule 3 — the recommended form and agreement with a wider view — which is exactly the pair this design assigns to normalize. Rules 1 and 4 hold either way.

That movement rate is a qualitative claim about a narrow sweep, not a benchmark: one synthetic contig, a handful of shape generators, genomic axis only, from a one-off measurement with no committed harness — so it is deliberately not quoted here as a number.

What you get for it

Two spellings of one variant, over one window, reach one description — because the derivation never sees a spelling. Over the cis confluence corpus that is 5 636 classes with no divergence — its genomic half, and all of what this surface can reach: the corpus is generated --axes g,c at 11 272 classes, and the 5 636 c. classes are drawn against NM_TEST.1, which the g.-only gate refuses, so not one of them enters the comparison. The exclusion is structural and is asserted as such in tests/it/from_sequences_corpus.rs.

Over the nine externally-reported confluence pairs (#1419 / #1420 / #1421) it is nine of nine, in both shuffle directions — where normalize, handed the same pairs as descriptions, currently converges none of them.

Read “over one window” as load-bearing, not as hedging. It is what makes the claim arithmetic: to_sequences computes its window from the denoted bases, so both spellings of a variant get byte-identical (position, reference, alternate) triples, and a pure function of that triple can only give one answer. It is also the exact limit — the claim is confluence over spellings, and rule 3’s scope is confluence over inputs. Two reads covering one variant differently are two inputs, and the section below is what happens then.

Read the comparison with normalize as two functions answering different questions rather than as one beating the other. A caller who has a description and wants it normalized still needs normalize to converge; the pairs are simply the case where being handed a description is itself the problem.

The cost, stated plainly

A window-local derivation is read-dependent. Nothing may shift outside the bases you supplied, so a read that stops partway through an ambiguous run places the change at the end of the read rather than the end of the run. One deletion from the AAAA at 12–15 of a test contig, seen through three windows:

windowreferencealternatederivedplacement_bounded_by_window
10–16GCAAAAGGCAAAGg.15delfalse
12–15AAAAAAAg.15deltrue
10–14GCAAAGCAAg.14deltrue

None of these is wrong. All three carry the same canonical SPDI (14:A:) and denote the same bases — g.14del is a conformant description of exactly the same variant, it is simply not the 3’-most spelling. So what a truncating read costs you is the recommended form (rule 2) and agreement with a wider read (rule 3): precisely the two rules this function never claimed, because both need the reference. Rules 1 and 4 hold in every row. normalize closes the gap, shifting to 15 regardless of what the read covered.

The boundary is exact, and pinned in tests/it/from_sequences_window_condition.rs:

Two windows that both contain the whole interval over which the change can be placed derive the same description. A window that cuts that interval places the change at its own edge instead.

Note that for an insertion or duplication that interval already reaches one base 5’ of the tract, since that is where a 5’-most insertion anchors — so “contains the interval” subsumes the flank requirement rather than needing a separate clause.

placement_bounded_by_window is a “could move” flag, not a “is wrong” flag, and it is conservative in that direction on purpose. Row 2 is flagged and already correct; row 3 is flagged and merely non-preferred. Telling those apart requires knowing what lies outside the window — the reference — which this function does not read, so it reports the uncertainty rather than resolving it. Treat a true as “re-derive from a wider window, or run normalize, if you need a reference-anchored answer”.

Two refusals are worth knowing about in advance, both deliberate — the policy is to refuse rather than degrade to a weaker rule:

  • An alignment grid over budget. The default admits a window of about 4 096 bases; a cell costs roughly 18 bytes. max_grid_cells is the knob, and the refusal names it. Real structural alleles are well past it — LRG_542:g.[101177_102434delins36;107248_127198delins21] spans 26 kb.

    Do not read the multi-member census as a measure of this refusal. Of the 592 multi-member alleles harvested from ClinVar, CMRG and Paraphase, 443 windows were captured and 59 derive — but splitting the 384 refusals by message gives 384 accession refusals and 0 grid refusals, every one of them an NM_ transcript hitting the g.-only gate above. The structural rows are filtered out of the capture before the grid is ever consulted. That figure was quoted here as evidence for the grid bound and is not.

  • An inserted payload against the window’s 5’ edge. HGVS writes an insertion between two positions, so such a payload can only be anchored at position - 1 — outside the window, and non-existent when position is 1. Supply more 5’ flank; Normalizer::to_sequences pads both sides for you.

Going the other way

Normalizer::to_sequences is the inverse, so a caller who already holds descriptions needs no new plumbing to reach the derivation:

pair = normalizer.to_sequences(variant, pad=128)
derived = normalizer.from_sequences(pair.accession, pair.position, pair.reference, pair.alternate)

The pad is not decoration: dup typing reads the reference bases immediately 5’ of an insertion point (DNA/duplication.md:18), so a member flush with the window’s 5’ edge comes back as an ins instead of a dup. It is applied to both sides — the window is span + 2 * pad — and the bases come back upper-cased, so a soft-masked region does not produce a mixed-case pair.

Bounding a derivation to a region it must not leave

When a variant must stay inside a target region, an amplicon or a tiling window, anchor every raw pair to that region first. The derivation is a pure function of the window it is handed, so one window gives one answer:

pair = ferro_hgvs.SequencePair("chr1", 10, "GCAAAAG", "GCAAAG")   # straight from a BAM

str(pair.derive().variant)                  # 'chr1:g.15del' — rolls to the run's end
bounded = pair.trim_to(end=14)              # hold it at 14
str(bounded.derive().variant)               # 'chr1:g.14del'

trim_to needs no reference and can only narrow. To widen, use Normalizer::reanchor, which reads the padding bases from the reference:

anchored = normalizer.reanchor(pair, start=5, end=25)   # 5' widened, 3' widened
str(anchored.derive().variant)                          # 'chr1:g.15del'

both = normalizer.reanchor(pair, start=5, end=14)       # 5' widened, 3' narrowed
str(both.derive().variant)                              # 'chr1:g.14del'

reanchor moves a window’s edges; it does not relocate the window. Each edge may go outwards (padded from the reference) or inwards (trimmed), in any combination — but the window you ask for must overlap the pair’s own, and the overlap must still hold the bases the two sequences disagree on. reanchor(pair, start=1000, end=1200) on the pair above is refused, not fetched: the changed bases exist only in the pair, so there is nothing to carry to a region the pair does not cover. So “anchor every raw pair to my target region” works exactly when every raw pair overlaps that region — which is the case the feature is for, and is worth checking rather than assuming.

Prefer pair.derive() to re-spreading the four fields: a pair returned by trim_to or reanchor carries its own position, and pairing a pre-trim position with post-trim bases is the mistake the method exists to prevent.

Both take 1-based inclusive bounds, and None leaves that edge where it is.

Both refuse rather than clamp, in every case: a bound that would cut a base the two sequences disagree on (naming the coordinate), a bound that would empty the reference, start past end, and — for reanchor — a bound outside the sequence, or a window disjoint from the pair’s. A window silently pulled back to the contig would hide a bug upstream of the call.

Case is not a disagreement: a soft-masked reference against an upper-case alternate trims normally. trim_to fetches nothing and so leaves your bases as you passed them; reanchor reads flank from the provider and therefore returns the whole window upper-cased, exactly as to_sequences does, rather than splicing provider bases onto caller bases and handing back a mixed-case pair.

Reach for this when the bound is a requirement, not to make heterogeneous inputs agree. For that, Normalizer::from_sequences(..., recommended_form = true) and a to_sequences round trip both already converge, and both reach the reference-anchored placement — which can shift as far as the sequence allows rather than as far as your window allows. Anchoring to a window that cuts an ambiguous run makes every caller using that window agree with each other and disagree with the reference. That is a legitimate contract and a poor default; placement_bounded_by_window reports it either way.

Reference data

Parsing needs no reference data. Normalization does — it must read the transcript and genome sequences a description refers to. ferro prepare downloads and assembles them into a reference directory.

Prepare a reference

ferro prepare --output-dir ferro-reference

This downloads RefSeq transcripts, genome FASTAs, and cdot metadata, and writes them under ferro-reference/. A bare prepare builds a RefSeq-only reference — accessions NM_ / NR_ / NP_ / NG_.

Verify it

ferro check --reference ferro-reference

Optionally pre-build the on-disk cdot cache as a setup step, so the one-time cache build does not slow the start of a real (or timed) run:

ferro check --reference ferro-reference --build-cache

Optional data

Two opt-in flags provision more than RefSeq. Pass them at prepare time; both are incremental, so re-running prepare over an existing reference adds the requested data and preserves what is already there.

Ensembl support (accessions ENST / ENSG / ENSP) — downloads Ensembl cdot metadata and cDNA FASTAs (~1 GB+); off by default. Without it, an Ensembl input reports “Reference not found”:

ferro prepare --output-dir ferro-reference --ensembl

RefSeqGene placements — derives version-independent NG_ placements and the NG_→transcript-version map, required to resolve legacy gene-symbol selectors (NG_(GENE):c.…) and bare-NG_ hosted lookups:

ferro prepare --output-dir ferro-reference \
  --derive-ng-placements path/to/ng_accessions.txt

A fully-provisioned reference combines both in one run:

ferro prepare --output-dir ferro-reference --ensembl \
  --derive-ng-placements path/to/ng_accessions.txt

Using it

Point any normalizing command at the directory:

ferro normalize -i variants.txt --reference ferro-reference/

To guard against a reference that has drifted on disk, add --strict-reference, which hard-fails if the reference’s content no longer matches its recorded identity (the default warns and proceeds).

Error Handling

ferro-hgvs provides configurable error handling with three modes:

ModeBehavior
strictReject non-conformant input (default)
lenientAuto-correct with warnings
silentAuto-correct silently
# Use lenient mode to auto-correct common issues
ferro parse --error-mode lenient "p.val600glu"  # Corrects to p.Val600Glu

# Ignore specific warnings
ferro parse --ignore W1001,W2001 "p.val600glu"

# Get help on any error/warning code
ferro explain W1001
ferro explain --list

Configuration File

Create .ferro.toml in your project directory:

[error-handling]
mode = "lenient"
ignore = ["W1001", "W2001"]  # Silently correct these
reject = ["W3003"]           # Always reject these

Comparing normalization rules (FERRO_PARTITION)

Unstable. FERRO_PARTITION is an evaluation switch, not a supported feature. It is not covered by semantic versioning, its values may change, and it is expected to be removed once the normalization rule is settled. Do not depend on it in production pipelines.

Normalization cuts each changed region of sequence into allele members. FERRO_PARTITION selects which rule does that cutting, so a candidate rule can be measured against the shipped one over a real corpus before anything changes for users.

ValueRule
unset / empty / canonical-coalescedThe shipped rule, and what every normal invocation uses. canonical, plus the delins.md:44-47 merge: a split whose payload realigns as one block is re-spelled as a single delins. Applied after the downstream passes rather than at partition time, so it cuts identically to canonical and differs only in what survives.
liveThe rule shipped up to and including v0.14.0: a single-gap alignment search plus two narrow escapes. No longer the default — set it by name to reproduce pre-flip output.
shadowCut only at alignment steps common to every minimal alignment.
canonicalThe member-count-minimal minimal alignment, without the merge above.

With the variable unset — or set to the empty string — output is byte-identical to a build with no switch at all.

The default moved in v0.15.0, from live to canonical-coalesced. If you are comparing against a stored corpus normalized by v0.14.0 or earlier, the like-for-like arm is now FERRO_PARTITION=live, not the unset one. The change is disclosed as a representation change in CHANGELOG.md; the ruling behind it is that a description is derived from the resulting sequence rather than preserved from the input’s spelling, which partition_blocklive’s cutter — cannot do.

Running an A/B comparison

Run the same input twice and diff the normalized descriptions. In tsv format the columns are line, input, normalized, changed, status, detail, so column 3 is the normalized string:

# 1. a baseline arm, named explicitly. `live` is the pre-v0.15.0 rule, which is
#    the one to use when the question is "what moved for my stored corpus";
#    name the arm rather than relying on unset, which is now the NEW rule.
FERRO_PARTITION=live ferro normalize --input variants.txt --reference /path/to/reference \
  --format tsv --error-mode lenient -j 10 > shipped.tsv

# 2. the candidate rule — here, the shipped default
ferro normalize --input variants.txt --reference /path/to/reference \
  --format tsv --error-mode lenient -j 10 > candidate.tsv

# 3. what moved
diff <(cut -f3 shipped.tsv) <(cut -f3 candidate.tsv) | head

# 4. how many moved, counting only rows that succeeded on both sides
#    (columns 3/5 are `normalized`/`status`; a row that failed carries no
#     normalized string, so comparing column 3 alone would score two different
#     failures as identical)
paste <(cut -f3,5 shipped.tsv) <(cut -f3,5 candidate.tsv) \
  | awk -F'\t' '$2 == "ok" && $4 == "ok" && $1 != $3' | wc -l

# and, separately, rows whose status itself changed
paste <(cut -f5 shipped.tsv) <(cut -f5 candidate.tsv) | awk -F'\t' '$1 != $2' | wc -l

This works on a stock release build — the switch is not behind a build feature, so no special binary or wheel is needed.

Two traps worth knowing

A misspelled value is refused, loudly. FERRO_PARTITION=canonicl makes ferro exit with an error before it reads any input, naming the value you gave and the arms this build has:

FERRO_PARTITION="canonicl" is not a partitioner this build has. This build's arms
are: live, shadow, canonical, canonical-coalesced. Refusing rather than falling
back to `live`, because a bake-off served the shipped rule under a candidate's
name reports that the candidate changes nothing.

Naming this build’s arms is the point: a value that exists on some other branch, or that used to exist, is reported as absent here rather than quietly answered as live.

The refusal comes from the CLI, not from the normalizer. Up to and including v0.14.0 it was a panic raised deep inside normalization, which meant a development-only switch could abort any process — a long-running service, or a Python caller, across the FFI boundary — that merely happened to have the variable set. A release build of the library now falls safe to live for a value that names no arm and keeps the refusal for its caller to report. Every binary and every example in this repository reports it — ferro, ferro-web, ferro-benchmark, both spec generators, and all 23 declared examples, which are the bake-off harnesses, the window extractors and the artifact generators — and that is pinned by a test whose denominator is Cargo.toml’s own [[bin]] and [[example]] tables rather than by this sentence, so a target added later cannot quietly skip it. The one class of cargo target still outside that obligation is [[bench]]: criterion’s criterion_main! generates the main, so there is no hand-written entry point to put the call in. If you embed ferro as a library and run bake-offs through it, read ferro_hgvs::normalize::partition_switch_startup_error() at startup and do the same.

Builds up to and including v0.13.1 fell back to live instead, and produced a clean, empty diff that read as “the candidate changes nothing”. The fallback emitted a warning through the log facade, but the ferro CLI installs no logger, so that warning reached no stream and RUST_LOG could not surface it — there was no signal at all. If you are on an older build, treat every empty diff as unproven.

A positive control on an input known to differ is still worth running, because it catches the other way a comparison can be vacuous: a variable that never reached the process at all (a lost export, a sudo that scrubbed the environment, a container that did not forward it). That case is indistinguishable from unset, which is legitimately live, so no amount of validation inside ferro can catch it. On your own corpus a zero remains ambiguous between “the switch is not taking effect” and “this corpus has no affected variants”.

These three inputs run against the built-in test data, so they need no --reference and no prepared reference directory. Between them they separate all four arms — every pair of arms disagrees on at least one row, so the control tells you which arm you got, not merely that something changed:

printf 'NM_001234.1:c.[5_6insAC;9del]\nNM_001234.1:c.2_6delinsGA\nNM_001234.1:c.4_10delinsAC\n' > control.txt

for arm in live shadow canonical canonical-coalesced; do
  echo "== $arm"
  if FERRO_PARTITION=$arm ferro normalize --input control.txt --format tsv \
       --error-mode lenient > "control.$arm.tsv"; then
    tail -n +2 "control.$arm.tsv" | cut -f3
  else
    echo "   FAILED (exit $?) -- ferro did not produce this arm's column"
  fi
done

The status check is not boilerplate. An unrecognised arm now aborts the process, and a pipeline reports the exit status of its last command — so ferro … | tail | cut reports cut’s success and the loop prints an empty column under the arm’s heading. An empty column and an aborted run look identical, which is the same “a broken measurement reads as a result” failure this whole section is about. Redirecting first and testing the status makes the abort say so.

inputliveshadowcanonicalcanonical-coalesced (= unset)
c.[5_6insAC;9del]c.[5_6insA;7_9delinsCAA]c.[5_6insAC;11del]c.[5_6insAC;11del]c.[5_6insAC;11del]
c.2_6delinsGAc.2_6delinsGAc.2_6delinsGAc.[2del;4_6delinsA]c.2_6delinsGA
c.4_10delinsACc.4_10delinsACc.4_10delinsACc.[4C>A;6_10del]c.[4C>A;6_10del]

All six pairs of arms are separated, which is what makes this tell you which arm you got rather than merely that something changed: row 1 separates live from the other three, row 2 isolates canonical (it is the only arm that does not merge that block), and row 3 separates {live, shadow} from {canonical, canonical-coalesced}. Rows 1 and 3 together separate live from canonical-coalesced; row 3 alone separates shadow from it.

Because canonical-coalesced is now the default, running with the variable unset must reproduce that last column exactly. If it reproduces the live column instead, you are on a pre-v0.15.0 build.

If the arm you selected does not produce its column above, the variable is not reaching ferro and any comparison you run is meaningless. Only once the control behaves is a zero on your own corpus informative.

This control has now been wrong twice, so check it rather than trusting it. The first version offered c.[2del;9del], c.[3del;9del] and c.[2del;9dup] and claimed canonical answers c.[2del;33del]; on those three inputs no arm differed from live at all. Its replacement — c.[5_6insAC;9del], c.[2del;5del], c.[2del;9del] — was also wrong in four of its twelve cells when re-measured: live row 1 read c.[5_6insA;7_9delinsCAA] and not c.6_9delinsACCAA, and rows 2 and 3 separated nothing, every arm answering c.[2del;6del] and c.[2del;11del]. Both failures share one cause: NM_001234.1 is a G homopolymer from c.9 to c.33, so deletion pairs inside it shuffle to a common form on every arm instead of partitioning differently. A discriminating row has to be a delins whose payload re-aligns, which is what the three above are. The table is measured, not composed.

From Python, it must be set before the first normalization. The value is read once per process and cached, so:

import os
os.environ["FERRO_PARTITION"] = "canonical"   # must precede the first normalize call
import ferro_hgvs

Setting it after any variant has been normalized silently does nothing, and it cannot be changed within a running process. Comparing two rules therefore means two separate processes (or two CLI runs, as above), not two calls in one script.

Benchmarking

ferro ships a benchmark harness for measuring its own throughput and comparing it against other HGVS tools (mutalyzer, biocommons/hgvs, hgvs-rs). This guide covers running it; for the published figures and full method, see docs/BENCHMARK_RUNBOOK.md.

Timing ferro alone

The main ferro binary needs no special build. Prepare a reference (see Reference data), then time a normalize run:

ferro prepare --output-dir data/ferro
ferro check --reference data/ferro
ferro normalize -i patterns.txt --reference data/ferro -j 8 -t timing.json

-t/--timing writes timing information to JSON. Two things matter for a fair number:

  • Warm the cache first. ferro check --reference data/ferro --build-cache builds the on-disk cdot cache as a setup step, so the one-time build does not land inside the timed region.
  • Sort the input by transcript accession (or genomic position). ferro caches each resolved transcript, so sorted input keeps the working set resident and is markedly faster on large batches.

Comparing against other tools

Tool comparison uses the separate ferro-benchmark binary, built with the benchmark feature:

cargo build --release --features benchmark

ferro-benchmark prepares each tool’s reference data (reusing ferro’s for transcripts), runs parse/normalize, and compares results:

# Prepare another tool against the same transcript data
ferro-benchmark prepare mutalyzer --ferro-reference data/ferro --output-dir data/mutalyzer

# Normalize a shared pattern set with each tool
ferro-benchmark normalize mutalyzer -i patterns.txt -o mutalyzer.json \
  --mutalyzer-settings data/mutalyzer/mutalyzer_settings.conf

# Compare two tools' outputs
ferro-benchmark compare results normalize ferro.json mutalyzer.json -o comparison.json

Supported tool values: ferro, mutalyzer, biocommons, hgvs-rs, and all.

The Python-based tools (mutalyzer, biocommons/hgvs, seqrepo) run in a pixi environment defined by the repository’s pixi.toml; run pixi shell to activate it before preparing them.

Reading the numbers honestly

All tools are benchmarked offline — reference data preloaded locally, network disabled — so the figures measure parse/normalize compute, not per-variant I/O. Out of the box the other tools resolve each variant against a remote service (a network round-trip per variant), which is far slower than any local figure. That local, offline setup is exactly what ferro prepare builds. When you quote a benchmark, say which configuration it was measured in, and confirm the run succeeded and produced correct output before trusting a timing — a crashed run still prints a wall-clock line.

Normalization rules

ferro’s normalizer follows four rules about its output, and three about how it handles the gaps.

The output contract

  1. Conformant. Output follows the HGVS recommendations. Absolute — never traded. Scope: the syntax.yaml grammar, plus every prohibition, read from prose force rather than keyword casing — “not allowed”, “not correct”, “can not be used”, “by definition”, the class="invalid" markup, and the set checklist.md enumerates.

  2. Recommended form. Where the spec prefers among conformant forms, ferro produces it. Best effort. Scope: “recommended”, “preferred”, lowercase “should”, the 3’ rule. A preference clause outranks maintainer judgment, but not rule 3: where it is not evaluable on what a normalizer holds, rule 3 governs.

  3. Confluent. Inputs denoting one variant produce one output. Best effort. Every rule — rule 2 included — is evaluated over the resulting sequence, never over the input’s spelling. Reference context counts as sequence: transcript model, exon boundaries, reading frame, strand and topology are functions of the accession, not of the description.

  4. Deterministic. Same input, same output. Absolute. Note that 4 does not imply 3 — a deterministic normalizer can be arbitrarily non-confluent.

The procedure

  1. Where the spec is silent, ambiguous, or self-contradictory: the issue is filed upstream first and cited, and only then does ferro ship a provisional choice.

    • Self-contradictory — two clauses that cannot both hold — is a defect. Every conforming tool must pick a side and none of them can be right, so filing is a bug report and is not optional.
    • Silent is merely incomplete. Ferro decides under rule 6 and violates nothing; filing is a feature request, worth making but never a reason to hold a release.
  2. Among multiple conformant forms: the maintainers choose. There are no user options for normalization form. Error mode is an orthogonal axis and stays available. The 3’/5’ shuffle direction was the one exception, and it is now removed from the public surface rather than excused: it is not orthogonal — it selects the frame every rule is evaluated in — so it was a user option for normalization form sitting inside this rule. ferro shifts 3’, the only direction the HGVS recommendations describe, and no CLI flag, Python keyword or service config key selects otherwise. The 5’ arm survives internally as a differential oracle over ferro’s own test suite; an instrument is not a user option, and rules 2 and 3 are now claimed once rather than per direction.

  3. Disclosure. Any change to these rules, and any different choice made under 5 or 6, is disclosed: in the changelog before v1, by a major version bump after. Output that violates rules 1-4 is a bug, not a disclosure.

Why 2 and 3 are best effort, and 1 and 4 are not

Rules 1 and 4 are always achievable. Conformance is checkable against the spec text, and determinism is a property of ferro’s own code; nothing external can prevent us honouring them.

Rules 2 and 3 depend on the spec determining an answer, and sometimes it does not:

  • No preference exists. The spec ranks substitution, deletion, inversion, duplication and insertion, but says nothing about competing delins forms.
  • Two preferences disagree. general.md ranks duplication above insertion; DNA/inversion.md prefers an insertion for inverted copies. No single output satisfies both.
  • The same clause has two versions. general.md’s current text and its forthcoming NOTE give opposite answers for variants separated by one nucleotide.
  • The variant’s decomposition is not recoverable. Recovering one means choosing an alignment, and the spec does not say which, so there is no derivable form to converge on. Block length does not settle it: an equal-length block can still carry a balanced del+ins pair, so its column correspondence need not be unique — CAG -> AGA is equal length with edit distance 2, not the position-wise 3. What decides whether a reference base is unchanged is whether every minimal alignment matches it, so the property to key on is edit distance against block length, never length alone. See rulings[unchanged-is-read-over-every-minimal-alignment].
  • The preference keys on information a normalizer does not hold. A “frequently occurring variant” (RNA/delins.md:41); a repeat “variable in the population” (RNA/repeated.md:33, protein/repeated.md:22); two variants “reported (or might occur) individually” (DNA/delins.md:83). The spec determines an answer; ferro cannot see what it keys on.

“Best effort” is bounded by the spec’s determinacy and by what a normalizer’s inputs can decide — not by ferro’s implementation quality. A failure of rule 2 or 3 caused by ferro’s code is a bug under rule 7; one caused by the spec not determining an answer triggers rule 5; one caused by a clause keying on provenance or population data is a declared deviation, and a permanent one — no upstream answer can put that information into a normalizer’s hands.

Permanent is the only thing the last case adds, and it does not exempt the question from rule 5. The grain matters, because the two grains give opposite answers. Read on its own, such a clause is not silent, ambiguous or self-contradictory — it determines an answer perfectly well, and ferro is the one that cannot see what the answer keys on. Read against the spec’s own re-derivation mandategeneral.md:157-160, which has a protein description derived by comparing the variant and reference protein sequences and says knowledge of the underlying DNA change “should not be used”, with general.md:13 extending the method to RNA — it is one half of a pair that cannot both hold. That is rule 5’s self-contradiction limb exactly, and the ruling ledger says so in those words. What the carve-out states is that rule 5’s escalation cannot end the matter here: a choice made under rule 5 is provisional pending an upstream ruling, and no ruling upstream makes provenance visible, so the deviation outlives the filing.

Where it is recorded is worth stating precisely, because the obvious pointer does not resolve. canonical-form-choice-when-both-legal has no deviates_from field: it carries all four clauses in its rationale as the recorded counter-evidence to re-derivation, opens that paragraph with “The spec contradicts itself here”, and adopts re-derivation over them deliberately. The one of the four that is recorded as a deviation is DNA/delins.md:83, through the deviates_from: ["docs/recommendations/DNA/delins.md:79-84"] on separation-is-a-property-of-the-spelling-not-of-the-variant — the record whose ruling the rule 3 example below turns on.

A worked example of reading force from prose

DNA/duplication.md says a variant that can be described as a duplication must be — but the “must” is scoped by the preceding clause, which defines when a duplication can be used at all: only when the additional copy is directly 3’-flanking the original. So the rule ranks the label for one span; it does not require that a partition be chosen so as to produce a duplication. Reading the force without the scope inverts the rule.

What rule 3 excludes

“Never over the input’s spelling” is narrower than it sounds. Most of a description is context.

Carried by the descriptionTreatment
Accession, axis (g./c./n.), versionUsed — it is the reference context
Which bases end up differentUsed — this is the variant
Cis against trans ([a;b] against [a];[b])Used — different variants, not two spellings of one
Type label (dup against ins, inv against delins)Re-derived, then ranked by general.md
How the edit set is cut into membersExcluded — a property of who wrote the string
Which copy in a run of identical residues a member namesExcluded — the 3’ rule assigns this “arbitrarily” (general.md:41)
Repeat unit and phase, where several are equivalentExcluded

So three rows read Excluded, and they are three spellings of the one thing the input does not get to decide: the partition, the run-position choice that feeds it, and — where several unit-and-phase pairs describe one tract equally well — which pair a repeat member names.

NC_000001.11:g.1001002_1001016 reads ATGAGGGGCCACTGT: a GGGG run at 1001006-1001009, a CC run at 1001010-1001011, a lone C at 1001013. Two spellings, one denoted sequence (ATGAGGGCATGT), because 1001010 and 1001011 are both C:

g.[1001009del;1001010del;1001013del]     written gaps of 0 and 2
g.[1001009del;1001011del;1001013del]     written gaps of 1 and 1

general.md:34’s “two variants separated by one or more nucleotides should be described individually” reads those gaps, so it answers twice for one variant. Rule 3 reads them off the partition ferro derives instead: both give g.[1001009_1001010del;1001013del]. Rule 2 then keeps that over the spanning g.1001009_1001013delinsCA, which merges across two unchanged nucleotides. Pinned in tests/it/cis_confluence_adjudication.rs.

Known limitation

ferro cannot today guarantee that every input form is normalized according to these rules. They are enforced intent, not a claim of current completeness.

Rule 7’s disclosure mechanism — the Representation-Change: trailer and how it reaches the changelog — is documented in CONTRIBUTING.md.

Where the individual decisions live

Rules 5 and 6 above say how a question is decided where the recommendations are silent, ambiguous or self-contradictory. What was decided, case by case, is recorded separately, as adjudication records — each naming the clauses in tension, which one governs, which is deviated from, and why. Those records are published in full, with their clause quotes, in docs/NORMALIZATION_CONTRACT.md.

That document is generated from the records and gated against them, so it cannot drift; it deliberately does not restate the seven rules above, which are stated only here.

The inverse index — for each stage of the normalizer, which record or clause governs it, and which decisions are governed by nothing — is docs/NORMALIZATION_STAGE_AUDIT.md. Read that one if what you want to know is whether a behaviour you are looking at was chosen or merely happened.

CLI reference

The ferro command groups its work into subcommands. Run ferro <command> --help for the full, authoritative options of any one — this page is a map, not a substitute.

Core

CommandWhat it does
parseParse and validate an HGVS description (no reference needed).
normalizeRewrite descriptions into canonical form (needs a reference). See Normalize variants.
projectRe-express a variant on a chosen output axis (g / c / n / p / r).
explainExplain an error or warning code (e.g. ferro explain W3003).

Reference data

CommandWhat it does
prepareDownload and assemble reference data. See Reference data.
checkVerify a reference directory is ready (optionally pre-build the cache).
convert-gffConvert a GFF3/GTF annotation to transcripts.json.
build-transcriptBuild a single-exon transcripts.json from a FASTA + CDS coordinates.

VCF and interchange

CommandWhat it does
annotate-vcfAnnotate a VCF file with HGVS notation.
vcf-to-hgvs / hgvs-to-vcfConvert between VCF and HGVS.
extract-hgvsExtract HGVS patterns from VEP-annotated VCF files.
liftoverLift genomic coordinates between genome builds.

Generation and prediction

CommandWhat it does
generateGenerate an HGVS description from components.
describeGenerate a description from reference and observed sequences.
effectPredict the protein effect of a variant.
backtranslateBacktranslate a protein variant to possible DNA variants.

Common options

Most commands accept -i/--input <file> (one description per line) and -o/--output <file>. The output format is set with -f/--format, and the accepted values depend on the command: text|json for parse and project, text|json|tsv for normalize, and text|json|markdown for explain. Commands that normalize also accept --reference <dir> and --error-mode <strict|lenient|silent>.

Supported HGVS Syntax

TypePrefixExample
Genomicg.NC_000001.11:g.12345A>G
Coding DNAc.NM_000088.3:c.459A>G
Non-codingn.NR_000001.1:n.100A>G
RNAr.NM_000088.3:r.459a>g
Proteinp.NP_000079.2:p.Val600Glu
Mitochondrialm.NC_012920.1:m.3243A>G

Edit Types

  • Substitution: A>G, Val600Glu
  • Deletion: del, 100_200del
  • Insertion: 100_101insATG
  • Deletion-Insertion: 100_102delinsATG
  • Duplication: 100_102dup
  • Inversion: 100_200inv
  • Repeat: 100CAG[20]

Why ferro-hgvs?

ferro-hgvs provides the most comprehensive HGVS variant normalization across all pattern types, with performance orders of magnitude faster than alternatives.

Normalization Capabilities Comparison

Pattern Typeferromutalyzerbiocommonshgvs-rs
Genomic (g.)
Coding (c.) exonic
Coding (c.) intronic✓**
Non-coding (n.)
RNA (r.)
Protein (p.)Net*

* mutalyzer protein normalization requires network access for NP_→NM_ lookups (cannot be cached locally). ** mutalyzer intronic support is enabled by default via genomic-context rewriting; disable with –no-rewrite-intronic.

Performance Comparison

All tools are benchmarked in ferro’s offline configuration — best case for every tool. Reference data is preloaded locally (a local UTA database and SeqRepo) and the network is disabled, so the figures below measure parse/normalize compute, not I/O. Out of the box, hgvs-rs, biocommons/hgvs, and mutalyzer resolve each variant against a remote UTA/SeqRepo or the Mutalyzer web API — a network round-trip per variant (~100–1000 ms), i.e. roughly 1–10 variants/sec, hundreds to thousands of times slower than shown here (an order-of-magnitude estimate from per-call network latency, not separately benchmarked). That local, offline setup is exactly what ferro’s prepare command builds; ferro needs no external service.

Median patterns/sec over 5 reps on an Apple M2 Max, local/offline. All tools draw from one stratified ClinVar population; per-tool sample sizes are calibrated so each tool is measured over a meaningful interval — fast cells (e.g. ferro/hgvs-rs parse) draw from millions of patterns, while slower cells (e.g. the per-tool normalize columns) draw from as few as tens to thousands. All tools exclude process/interpreter startup from the timed region — the mutalyzer/biocommons Python subprocesses are timed by their own internal startup-excluded timer, matching ferro/hgvs-rs. Only ferro parallelizes natively (rayon); the other tools are single-threaded libraries, so their normalize @8 workers figures come from the benchmark harness running 8 independent instances in parallel, while parsing is not sharded for them — hence the single-threaded label in their parse @8 workers column (mutalyzer normalize likewise shows no gain at 8 workers: per-call cache and IPC overhead dominate, so sharding does not help). Every tool runs fully offline against local reference data — a local UTA database and SeqRepo, with mutalyzer’s network lookups disabled — the configuration ferro’s prepare command enables; the figures therefore reflect compute throughput, not per-variant network latency. Reference-data load is excluded for all tools. ferro full-population peak: parse 20.0M/s, normalize 77.0k/s. See docs/BENCHMARK_RUNBOOK.md for the full method.

Parse

ToolThroughput @ 1 workerThroughput @ 8 workersferro speedup @ 8w
ferro5.1M/s12.2M/s
mutalyzer352/ssingle-threaded35,000×
biocommons3.9k/ssingle-threaded3,100×
hgvs-rs3.6M/ssingle-threaded

Normalize

ToolThroughput @ 1 workerThroughput @ 8 workersferro speedup @ 8w
ferro78.1k/s260.2k/s
mutalyzer4/s4/s73,000×
biocommons368/s818/s320×
hgvs-rs195/s1.3k/s200×

ferro thread scaling

Threads1248
ferro parse5.1M/s9.4M/s16.0M/s12.0M/s

Input ordering matters for batch throughput. Resolving a transcript (reading its full sequence from the reference and rebuilding its CDS/exon metadata) dominates per-variant cost. ferro memoizes resolved transcripts in a bounded in-memory cache, so repeated lookups of the same transcript are near-free. Providing variants sorted by transcript accession — or by genomic position, which clusters variants onto the same transcripts — maximizes the cache hit rate and can speed up large batches by an order of magnitude versus randomly-ordered input. Ordering matters most when the number of distinct transcripts in the run exceeds the cache capacity (very large or genome-wide inputs); below that, the working set stays resident regardless of order.

Reference Data: What ferro Prepares

The ferro prepare command downloads and organizes all reference data needed for comprehensive normalization. This data is then shared with other tools (mutalyzer, biocommons, hgvs-rs) to enable their local operation.

Data TypeSourceSizeEnables
RefSeq transcriptsNCBI~1GBNM_/NR_/XM_ normalization
cdot metadataMANE~200MBTranscript-to-genome mappings
GRCh38 + GRCh37 genomesNCBI~4GBNC_ genomic normalization
RefSeqGene (sequences + genome alignments)NCBI~600MBNG_ gene-region normalization; projecting c./n. variants into an NG_ parent’s own g. frame (via the RefSeqGene→genome alignment GFF3)
LRG sequences + XMLEBI~50MBLRG_ stable-reference normalization; projecting c./n. variants into an LRG_ parent’s own g. frame (via the LRG XML genomic mapping)
Protein sequencesDerived from CDS~200MBNP_/XP_ protein normalization
Legacy transcript versionsNCBI~50MBHistorical ClinVar variants

Key insight: Without ferro’s reference preparation, other tools require network access for each variant lookup (adding 100-1000ms latency per variant). With ferro’s cached reference data, all tools can operate fully offline with consistent, reproducible results.

Deriving version-independent NG_ placements (#728)

ferro prepare --derive-ng-placements <accessions.txt> derives genomic placements for the listed NG_ versions (one exact accession per line, e.g. NG_012337.3; blank lines and # comments ignored), writing derived_refseqgene_placements.json into the reference directory and wiring the manifest’s derived_refseqgene_placements field. This fills version gaps the archived RefSeqGene→genome GFF3 snapshots do not cover. It needs cdot + the genome in the same prepare run and uses NCBI EFetch per accession; accessions that cannot be validated are skipped with a warning. The field is preserved across subsequent prepare runs.

Interpreting the HGVS Recommendations

Under construction. Pages are added one HGVS spec page at a time; this section will fill in as they land.

What it is

The HGVS nomenclature recommendations are the standard for naming sequence variants. They are large and carefully written, but — like any natural-language standard — in places they leave a question open, or two passages can be read as pointing different ways. A normalizer still has to emit exactly one string. Where ferro has had to settle such a question, we record the decision (and, where a passage looks inconsistent, file it upstream with the SVD Working Group).

This section mirrors the recommendations page for page and records, per clause, how ferro reads each one. Each page gives, for a span of the spec:

  • the spec text it interprets, quoted verbatim;
  • ferro’s reading of that clause;
  • example spellings, each with the form ferro normalizes it to and a verdict describing that output against the recommendations (see below);
  • which spellings converge on one output;
  • the reason ferro reads the clause the way it does, and a link to the governing decision.

How to read the verdicts

The recommendations are HGVS’s, not ferro’s. ferro’s job is to take any input and produce the form the recommendations prefer. The verdict describes ferro’s output against that goal:

  • recommended — ferro’s output is the form the HGVS recommendations prefer. This is the target, whether the input was already that form or ferro normalized it there.
  • conformant — ferro’s output is a valid HGVS string but not yet the recommended form. This is a known ferro limitation, and it links to the issue tracking it.
  • refused — the input is not valid HGVS; ferro rejects it in strict mode. Refusing is the correct behavior.
  • bug — ferro’s output is not valid HGVS. This should never happen; where a page shows one, it links to the open defect.

What it is not

It is not a second copy of the rules. The canonical ruleset lives in the normalization rules page, and every adjudication is recorded once in the ruling ledger (hgvs_spec_normalization_overrides.json, rendered for reading as NORMALIZATION_CONTRACT.md). These pages draw their explanations from the ledger rather than restating them — a rule written in two places is a rule that drifts.

How it stays honest

Every example is executed through ferro by a build check: each spec quote is verified against the pinned spec checkout, each spelling is parsed (and strict-refused where marked refused), and each normalized output is asserted against a real GRCh38 reference. A wrong verdict, a moved quote, or a changed normalization fails the build — so a page cannot claim behavior ferro does not have.

Pages

DNA

Substitution — ferro’s reading

ferro’s reading of substitution.md. The rules are HGVS’s; ferro’s job is to produce the form the recommendations prefer. Verdicts describe ferro’s output:

  • recommended — ferro’s output is the form the recommendations prefer (whether the input was already that form, or ferro normalized it there).
  • conformant — ferro’s output is valid HGVS but not yet the recommended form — a ferro limitation, with a tracking issue.
  • refused — the input is not valid HGVS; ferro rejects it in strict mode (correct behavior).
  • bug — ferro’s output is not valid HGVS (a defect). None on this page.

Each Why block is transcluded from the ruling ledger — the record’s own one-line summary, rendered here and linked to its full entry in NORMALIZATION_CONTRACT.md. The reasoning lives once, in the ledger; it is never re-typed here.

substitution.md:5 — definition: one nucleotide for one

one nucleotide is replaced by one other nucleotide

Ferro: a substitution is exactly 1→1; IUPAC ambiguity codes stand in for the single replacing base.

InputVerdictNormalizes toNotes
NM_004006.3:c.76A>Grecommendedselfcanonical single-base substitution
NM_004006.3:c.54G>Hrecommendedselfreplacement is IUPAC H = A, C, or T
NC_000023.10:g.33038255C>Arecommendedthe spec’s genomic example (GRCh37 accession — parse-only here)
NM_004006.3:c.79GC>TTrefusedtwo nt — violates 1→1; rejected as “deprecated multi-base substitution syntax”

substitution.md:15 — two or more nucleotides is a delins

two or more consecutive nucleotides are described as deletion/insertion (delins) variants

Ferro: a multi-base replacement is never a substitution; write it as a delins.

Why.

absolute-prohibition-enforcement-stage — Spellings the spec prohibits are rejected — at parse in strict mode; lenient mode instead repairs the input where it can and fails only if it cannot normalize.

InputVerdictNormalizes toNotes
NM_004006.3:c.79_80delinsTTrecommendedselfthe two-base change written as a delins
NM_004006.3:c.79_80GC>TTrefusedrejected: “deprecated multi-base substitution syntax” — use delins

substitution.md:16-18 — separation, and the one-codon exception

two variants separated by one or more nucleotides should be described individually

Ferro: the separation rule (ruleset rule 2); the exception folds two subs into one delins only when they sit one nucleotide apart and together change one amino acid.

Why.

separation-rule-force-modal-or-negation — Two changes a nucleotide or more apart are described individually — this is the spec’s preference (ruleset rule 2), not an outright ban; the only spelling the recommendations forbid is the split at separation zero.

codon-carve-out-shape-restriction — Two changes one nucleotide apart that together alter a single amino acid are written as one delins, whatever the edit types — because “together affecting one amino acid” is a fact about the resulting sequence, not about how the input was spelled.

InputVerdictNormalizes toNotes
NM_004006.3:c.145_147delinsTGGrecommendedselfone-codon delins, CGCTGG
NM_004006.3:c.[145C>T;147C>G]recommendedNM_004006.3:c.145_147delinsTGGferro normalizes the split to the recommended delins (warns MEMBERS_COALESCED_FROM_REPORTED_FORM)
NM_004006.3:c.235_237delinsTATrecommendedselfthe Lys79 codon; the split predicts p.[Lys79*;Lys79Asn], the delins p.Lys79Tyr
NM_004006.3:c.[235A>T;237G>T]recommendedNM_004006.3:c.235_237delinsTATsplit → the recommended delins (same codon exception)

Confluence. Each split and its delins converge on one recommended output — {c.[145C>T;147C>G], c.145_147delinsTGG} → c.145_147delinsTGG, and the Lys79 pair → c.235_237delinsTAT.

substitution.md:19 — no change is =, not a substitution

found not changed are described as

Ferro: a tested, unchanged position is =; a no-change substitution is normalized to it.

InputVerdictNormalizes toNotes
NM_004006.3:c.123=recommendedselfc.123 screened, unchanged (reference C)
NM_004006.3:c.123C>CrecommendedNM_004006.3:c.123=a no-change substitution is normalized to the recommended =

substitution.md:20 — polymorphisms are not A/G

it is not correct to describe

Ferro: the slash form for “polymorphism” is not valid HGVS; ferro rejects it.

InputVerdictNormalizes toNotes
NM_004006.3:c.76A>Grecommendedselfthe correct form
NM_004006.3:c.76A/Grefusedthe disallowed “polymorphism” slash — rejected at parse

substitution.md:32 — adjacent substitutions are one delins

changes involving two or more consecutive nucleotides are described as deletion-insertion (delins) so the description

Ferro: at separation zero the split is not correct; two adjacent members that both consume reference bases are one delins. The same rule licenses re-merging adjacency that per-member 3′ shifting creates (protein p.[Gly16Ala;Gly17del]p.Gly16_Gly17delinsAla), keeping normalization idempotent.

Why.

delins-adjacent-members-when-both-consume-reference — Two adjacent changes that both consume reference bases are written as a single delins; the spec marks the split spelling “not correct” at separation zero.

canonical-form-choice-when-both-legal — When two descriptions of one variant are both legal and no clause chooses between them, ferro derives the form from the resulting sequence rather than preserving the input’s spelling.

InputVerdictNormalizes toNotes
NM_004006.3:c.79_80delinsTTrecommendedselfadjacent c.79,c.80 written as one delins
NM_004006.3:c.[79G>T;80C>T]recommendedNM_004006.3:c.79_80delinsTTferro normalizes the adjacent split to the recommended delins
NM_004006.3:c.79_80GC>TTrefusedthe multi-base substitution spelling — rejected

Confluence. {c.[79G>T;80C>T], c.79_80delinsTT} → c.79_80delinsTT.

See also → substitution.md:15, substitution.md:16-18.

substitution.md:47-49 — mosaic and chimeric

a mosaic case where at position

Ferro: mosaic (/) and chimeric (//) mixtures are valid; the recommendations write the reference allele first.

InputVerdictNormalizes toNotes
NM_004006.3:c.85=/T>Crecommendedselfmosaic: reference = written first, then T>C
NM_004006.3:c.85=//T>Crecommendedselfchimeric: a mix of c.85= and c.85T>C cells
NM_004006.3:c.85T>C/=conformantselfvalid, but substitution.md:49 writes the reference first; ferro does not yet reorder this to the recommended c.85=/T>C. Tracked by #2034.