Results files

A results file stores every FrameResult from one indexing run in one HDF5 file. Keep this file for later analysis: visualization functions load it directly, and its datasets are accessible with standard HDF5 tools. XML output remains available for software that reads that format; see Write XML.

The XML document stores every number as text in a nested element tree, so reading a 100,000-frame wire scan takes tens of seconds and several gigabytes. A results file stores each quantity as one typed array across all frames, and the same scan loads in a fraction of a second.

Write a results file

Indexer.write_results() writes an iterable of results in iteration order, together with the crystal, the geometry, and the peak and indexing parameters from the indexer:

from pathlib import Path

from lauelab.indexing import Indexer

indexer = Indexer("geometry.xml", "crystal.xml")
frames = sorted(Path("frames").glob("*.h5"))
results = indexer.index_many(frames)
indexer.write_results(results, "results.h5")

An existing file is replaced only with overwrite=True. Pass frame_ids to record an identifier for each frame instead of its zero-based position; identifiers must be unique and all strings or all integers:

indexer.write_results(results, "results.h5", frame_ids=[path.stem for path in frames])

Write while indexing

Indexer.results_writer() returns a streaming writer for scans too large to hold every FrameResult in memory. Each append() extends the file in order, and the file is complete when the with block exits:

with indexer.results_writer("results.h5") as writer:
    for path in frames:
        result = indexer.index(path, keep_image=False)
        writer.append(result, frame_id=path.stem)

append() checks the result type and frame ID before writing. If either is invalid, such as a duplicate ID, it raises an exception and the writer remains usable.

A failure during writing can leave a partial frame in the file. In that case, writer.failed becomes True, writer.error contains the exception, and further appends raise RuntimeError. Stop writing and recreate the results file from the beginning. Validate the completed file as described below.

Write XML alongside

XmlResultsWriter appends each result to one LaueGo AllSteps document as it arrives, with the same bytes Indexer.write_many_xml() would produce and without holding earlier steps in memory. Feed both writers from one stream of results:

import warnings

from lauelab.indexing import XmlResultsWriter

with indexer.results_writer("results.h5") as writer, \
        XmlResultsWriter("results.xml") as xml:
    for path in frames:
        result = indexer.index(path, keep_image=False)
        writer.append(result, frame_id=path.stem)
        if not xml.failed:
            try:
                xml.append(result)
            except OSError as error:
                warnings.warn(f"XML output stopped: {error}")

This example continues writing HDF5 results if XML output fails. Each XML step is flushed after writing. Following a failed append, the XML writer rejects further steps. On close, it attempts to truncate the document to the last complete step and add the closing tag; this recovery depends on the filesystem remaining writable. An HDF5 write failure stops the loop.

Validate a results file

validate_results_file() checks a closed file’s structure with bounded reads and returns a ResultsFileSummary:

from lauelab.indexing import validate_results_file

summary = validate_results_file("results.h5", frame_ids=[path.stem for path in frames])
print(summary.n_frames, summary.n_peaks, summary.n_patterns)

Validation checks the format and version, required datasets, dtypes, shapes, and lengths. It also checks that offsets and counts agree, pattern ranks start at zero in each frame, and frame IDs are unique. Assignment peak indices are checked in bounded chunks to ensure they refer to peaks in the corresponding frame. Peak measurements, reciprocal lattices, and other assignment values are excluded from this structural check.

Pass frame_ids or n_frames to compare the output with the expected inputs. An ID mismatch reports the first differing frame.

A structural defect raises InvalidResultsFile with the file path and the failing check. lauelab.is_results_file() reads only the format marker; a file interrupted while being written still carries the marker, so use the validator before treating a file as complete. A structurally valid file can contain only part of a run. Compare its frame IDs or count with the expected inputs to check completeness.

Write from a process pool

iter_index() runs frames in worker processes and yields outcomes in input order, so one writer can append them as they arrive:

from lauelab.indexing import FrameInput

inputs = [FrameInput(path, input_id=path.stem) for path in frames]
failures = []

with indexer.iter_index(inputs, workers=8) as outcomes, \
        indexer.results_writer("results.h5") as writer:
    for outcome in outcomes:
        if outcome.ok:
            writer.append(outcome.result, frame_id=outcome.input_id)
        else:
            failures.append((outcome.input_id, outcome.error))

Results arrive without images by default, so only peaks, patterns, and provenance cross between processes. See Batch indexing for the in-flight bound, error handling, and cooperative stopping.

Load a results file

load_results() returns the same VisualizationDataset as the other loaders:

from lauelab.visualization import load_results, plot_map

dataset = load_results("results.h5")
figure = plot_map(dataset, axes=("X", "Y"), color="cubic_ipf")

The crystal comes from the file. The geometry is, in order, an explicit geometry argument, the geometry text embedded in the file, or the recorded geometry path. An updated external calibration therefore does not change an existing results file; missing geometry prevents only detector views. Rotation matrices are not stored; the loader derives them from each reciprocal lattice and the crystal in the basis described in Patterns.

is_results_file() distinguishes a results file from other HDF5 files, such as detector frames.

Convert an existing XML document

convert_xml() reads a LaueGo AllSteps document once and writes the equivalent results file, by default beside the XML with the .h5 suffix:

from lauelab.visualization import convert_xml

output = convert_xml("indexed-scan.xml")

An existing output raises FileExistsError unless you pass output_path or overwrite=True. Conversion costs one XML load, so convert once and load the results file afterwards.

Conversion writes a unique <destination>.partial-* file in the destination directory, then closes and validates it before renaming it to the requested path. A failure removes the partial file and preserves any existing destination. If two conversions target the same path, the second to finish raises FileExistsError unless overwrite=True.

The exception identifies the failure: a missing document raises FileNotFoundError, a malformed one xml.etree.ElementTree.ParseError, conflicting run parameters ValueError, and a written file that fails validation InvalidResultsFile. A document without geometry or crystal context converts; validate_results_file on the output reports has_geometry_text and has_crystal.

The same publication step is available for your own outputs: write to lauelab.partial_path(), validate, then lauelab.publish_file() renames it to the final name and refuses, by default, to replace a file that appeared in the meantime.

The converter records the geometry path from the XML and embeds the geometry text only when that path is readable. A document without crystal information converts without a crystal, and the loaded dataset then rejects pole figures and orientation colors. Run parameters recorded in the XML become run attributes; parameters absent from the XML stay absent, and parameters that conflict between steps raise ValueError, so convert separate configurations separately.

File contents

Group	One row per	Contents
crystal		Space group, cell, and atoms; absent without a crystal
geometry		Geometry path and, when available, the geometry XML text
run		Program name, detector selection, and every peak and indexing parameter, as attributes
frames	frame	Sample position, depth, energy, acquisition metadata, image shape and region, frame statistics, timing, and offsets into peaks and patterns
peaks	detected peak	The fields of FrameResult.peaks
patterns	pattern	Rank within its frame, reciprocal lattice, quality values, and offsets into assignments
assignments	pattern-to-peak assignment	Peak index, hkl, angular error, energy, and predicted intensity

Peak and pattern rows are grouped by frame; assignment rows are grouped by pattern. The peaks of frame i are rows peak_offsets[i] to peak_offsets[i + 1]; pattern_offsets and assignment_offsets follow the same rule.

Reciprocal lattices, cell parameters, atom positions, sample positions, depths, and intensity sums are numpy.float64. Other floating-point values are numpy.float32, which keeps about seven significant digits, or about 0.0001 px for a coordinate on a 2048-pixel detector. Hutch temperature and sample distance keep the values supplied by acquisition with units="unspecified", because the input metadata does not establish a unit. Missing flags are -1 and missing floating-point values are NaN. In the run attributes, a parameter given as None is also stored as NaN: threshold for automatic thresholding, threshold_ratio for the native default, and max_peaks for an unlimited peak search.

The reference lists every dataset with its dtype, shape, and units. Provenance that VisualizationDataset does not carry, such as the run attributes, is read directly with h5py.