Testing and validation

Three validation levels

Recommended commands

Run from the directory containing kopr.py # R92+ application ephemeris selection python3 check_nbody_status.py # Optional external N-body/Horizons validation # Use the local corpus utility and require used_model=nbody for every case. with the KOPR environment activated.

# Syntax of the principal modules
python -m py_compile kopr.py wstack.py wstack_common.py wstack_functions.py   wccd.py autophot.py photom.py koprfunc.py

# Correct KOPR regression target
python -m pytest -q tests

# Targeted Windows/WSL solver contract
python -m pytest -q tests/test_windows_wsl_astrometry_solver.py

# Static Autophot/WCCD build contract
python verify_autophot_build.py

# R22 platform-neutral performance/release contract
python3 validate_performance_release.py
python3 validate_performance_release.py --json

# R22 performance and multiplatform regression contract retained by R27
python -m pytest -q tests/test_performance_release_batch10.py

# R27 photometry release contract
python3 validate_photometry_release.py
python -m pytest -q tests/test_photometry_release_batch4.py

Do not use an unqualified repository-root pytest command as the only release command when unrelated nested projects are present. Target the maintained KOPR tests/ directory.

Current automated test families

R74 adds a focused 58-test gate for automatic DSLR discovery, materialization, session rebuilding, preview selection and WCCD/WStack hand-off. The last complete deterministic baseline remains R73 with 1125 passed and 3 environment-gated skips. A target desktop is still required for real PyQt5 rendering, complete Astropy FITS I/O and rawpy/LibRaw decoding.

R79 cumulative release gate

tests/test_r79_cumulative_release.py contains 13 tests for current release identity, production parsing, automatic RAW materialization, R75 guards, R76 identity/binning/scale/light-time wiring, the strict 0.150-mag global pairwise boundary, lossless AutoStars/Manual Stars parity and the unified Reference stars/Process obs./Af-rho solution.

Scientific and Compact validation

Use the production validator rather than only inspecting header keywords:

python validate_scientific_compact.py \
  --json validation.json \
  --csv validation.csv \
  --markdown validation.md

python validate_scientific_compact.py \
  --compare scientific.fits compact.fits \
  --json pair-validation.json

python validate_scientific_compact.py \
  --source input.fits \
  --output-dir validation-products \
  --keep-files

Optional --center, --aperture-radius, --sky-inner and --sky-outer arguments add a fixed-aperture storage comparison. Required report evidence includes requested/actual mode, fallback reason, QSTEP, local noise, safety ratio, max/RMS/median pixel deviation, optional flux/magnitude/background/centroid differences, primary payload/full FITS size and real-file read/write timings.

Release-blocking cases

• normal coma/star field accepted as Compact with bounded decoded error;
• normalized master flat accepted with sub-response-scale error;
• extreme dynamic range triggers exact Scientific fallback;
• NaN/inf map to BLANK without colliding with valid pixels;
• WCS header update preserves Compact scaling and extensions without requantization;
• manual WCCD results remain consistent across supported integer and float FITS;
• native Scientific and Compact pairs pass the same automatic contract;
• external FITS does not receive invented total exposure or automatic-pair provenance.

Release-blocking scientific invariants

• Manual WCCD and retained shared ACF primitives use production pa-adaptive-upper-v1; the legacy symmetric selector is rollback-only.
• Only positive PA-local outliers are clipped; negative samples are not clipped.
• A Review preview can change only ACF analysis derived from the frozen context, not centroid, background, catalogue stars or zero point.
• Apply is accepted only when context_id and run_id match the current measurement.
• The final signal is the cumulative PA-adaptive ACF value at the selected radius and is not overwritten by ordinary aperture photometry.
• Every accepted measurement retains detailed kopr-acf-provenance-v1 evidence and a concise KACF log/FITS HISTORY line.
• Manual ordinary photometry, ACF, curve-of-growth points and transferred comparison stars use 40 - 2.5 log10(net_flux / EXPTIME); local sky is additive subtraction only.
• Comet and Stars EXPTIME values are validated within 0.001 s during preflight and the accepted value is frozen for the observation; TOTEXP is reporting metadata only.
• Sigma-clipped mean uses one pass and never restores clipped samples.
• Complete source saturation provenance—not the aggregate stack peak—is authoritative in automatic policy.
• An explicit Saturation level edit creates a role-specific manual threshold, intentionally bypasses the map for that slot and cannot leak to a different source path.
• Relative-WCS diagnostics may be expanded, but acceptance thresholds cannot change unintentionally.
• The result plot and ICQ comment use the same physical curve-of-growth model.

A refactor that changes any of these outputs or decision paths is a scientific-method change and requires new reference data, explicit review and release notes—not merely updated unit tests.

• Master-flat correction remains guarded and Stack Parameters uses one preflight pass.
• Physical scale follows the R76 shared source priority and uses validated observation binning.
• DE440s comet positions retain down-leg light-time; external residuals do not define a universal arcsecond gate.
• The canonical reference-star solver applies extinction before global pairwise selection and rejects a pair at exactly 0.150 mag.
• user_enabled and pairwise_accepted remain separate; Process obs. and Af-rho use one final accepted set.

Required GUI and pipeline smoke tests

1. Host tabs. Launch KOPR, open all five production tabs in different orders and confirm each widget is created once. Invoke legacy Analyzer/Comparison routes and confirm both resolve to the existing Comet Analyzer widget without a duplicate top-level window or cache.
2. Settings propagation and persistence. Change active location, telescope, eyepiece and camera; confirm an already-open WStack context refreshes, WCCD uses the host menu state, restart KOPR and verify all four selections are restored rather than reset to index zero.
3. WStack preference persistence. Save different Comet and Stars methods/sigma values and a Comet output mode. Reopen both dialogs and confirm the separate values. Cancel or submit invalid input and verify the prior values remain unchanged.
4. Worker shutdown. Start a cancellable WStack task, close the application and verify bounded cancellation without an orphaned thread or destroyed worker.
5. Windows/WSL preflight. On a native Windows installation with Astrometry.net only in WSL, verify that solver preflight: WSL solve-field (...) appears before any index dialog.
6. Missing Windows/WSL solver. Temporarily select a distribution without Astrometry.net and verify one actionable error appears and no index download begins.
7. Windows/WSL cancellation. Cancel a deliberately long solve and verify pgrep -a solve-field returns no remaining solver process in the selected distribution.
8. Path handling. Solve a frame from a local Windows path containing spaces and verify WCS/AXY/config/index paths translate correctly and output returns to the Windows directory.
9. Calibration. Build dark/flat masters from a known series, inspect QC reports and verify a calibrated LIGHT header references the selected masters.
10. Astrometry/WCS QC. Solve a known series, inspect AXY diagnostics, deliberately include one bad WCS and confirm rejection persists as Astro OFF.
11. R48 blind online path. Inspect the upload payload and confirm all position, radius, scale, parity and downsampling constraints are absent. Verify queued/solving heartbeats and that calibration is requested only after status=success.
12. R48 offline overrides. Open the Astrometry group dialog and accept unchanged automatic RA/DEC and scale values; confirm no override is stored and per-frame ephemerides remain independent. Edit both fields to create an override, then clear both scale limits to restore automatic scale.
13. Stacks. Produce mean, sum and sigma-clipped star/comet pairs; verify names, pair ID, frame counts, NVALID/NCONTRIB, saturation extensions and provenance.
14. Saturation and camera integration. Verify complete-map authority and all scalar base priorities, then use a FLOAT32 SUM fixture with base 65535 and maximum 1036396.5625 to require multiplier 16 and effective limit 1048560. Confirm automatic/manual base parity, non-fatal SATCOMP=NONE, continuation without a source-peak provider and independent flat-top/bleeding rejection. Reconstruct 1370 × 1002 at 3 × 3 as 4110 × 3006 and verify VERIFIED_EXACT; verify a unique 4096 × 3000 active-area configuration returns VERIFIED_COMPATIBLE. Neither verified case may open the camera dialog.
15. COBS source codes. Parameterize all 15 WCCD magnitude-source labels and assert exact two-character case. In particular, APASS Sloan g′/r′/i′ are AG/AR/AI and Gaia mixed-case identifiers remain unchanged.
16. Manual photometry. Reproduce a reference comet measurement with the expected centroid, background, ACF curve and magnitude.
17. Review dialog. Open and resize it; verify zoom/pan, ring selection, active-ring map and raw/upper/filtered profile redraw.
18. Review Close/Reset. Confirm Close without Apply changes nothing and Reset restores exactly the original or last-applied measurement state.
19. Manual Review Apply. Apply a preview while preserving the user-selected fixed aperture radius and then complete the normal save flow.
20. Review lifecycle. Open Review on Comet and Stars images and after reference-star/catalogue/photometric-option changes; confirm these do not invalidate it. Verify new observation/reset/different Comet FITS or a successful non-ACF measurement does invalidate it, while cancelled/failed replacement attempts preserve it.
21. ACF provenance. Confirm detailed diagnostics and the concise KACF line in the log and compatible FITS HISTORY.
22. R45 robust astrometry. Run the supplied problematic frames with local indexes; compare Attempt 1–3 terminal lines and full logs, confirm the calibrated FITS is unchanged, then force an anchor failure and verify the default skip and explicit continuation policies.

• Point style preserves identical observation rows and X/Y coordinates.
• Observation markers remain visually above model curves.
• Single-target reverse mode uses line-only formula legends.
• Manual ranges invalidate only for R163 identity changes; style-only changes, including Point size, preserve them.
• Point size changes both observation sides, persists across restart, keeps formula/legend markers unchanged and resets to 3.0 with display defaults.

WCCD Automatic Helpers manual acceptance

• Open a new observation with all helpers disabled and confirm legacy manual behaviour.
• Run finder with internal KOPR position and with JPL Horizons; disconnect the network and verify fallback.
• Test paired Comet + Stars and Comet-only finder modes.
• Place a valid candidate between 30 and 120 arcsec from the expected position. Verify interactive WCCD review finds it while the unattended automatic-photometry gate remains 30 arcsec.
• Verify displayed automatic rows are consecutive C1..Cn and Primary/Extended classification remains present; click a circle to select its row.
• Click another object and verify M1 uses the same bounded Gaussian comet centroid as manual WCCD and automatic candidates; enable pan/zoom and verify clicks no longer select.
• Force zero numerical candidates and verify the valid cutout still opens for manual visual selection.
• Confirm that accepting Cn or M1 stores only the centre. Verify the same accepted coordinates drive the marker, Slice, aperture measurement and ACF; rejection changes nothing.
• Process a result with automatic and manual saturation policies and verify no automatic Saturation: sentence is added to Results, ICQ or COBS comments while diagnostics retain saturation provenance.
• Run Auto Star Measure with elongations around 2.2 and 2.7, with and without a valid plateau, and with several removable NaNs in the sky annulus.
• Use a crowded field and confirm raw median FWHM can exceed the robust isolated/lower-distribution FWHM; inspect method, source count and 25/50/75% widths.
• Verify the aperture fallback is based on robust FWHM, the final blend gate is no smaller than aperture radius plus 0.75× robust FWHM, and post-aperture neighbour rejection is reported.
• Place NaN in a non-zero-weight aperture pixel and require invalid_aperture; separately test a partially masked/cropped annulus and the guarded balanced_boxes fallback.
• Force each detailed centroid/background rejection family and verify the specific subreason appears.
• Verify <3 final stars fails, 3–7 warns, >40% clipping warns without automatic failure, scatter 0.08–0.15 mag warns and >0.15 mag fails with computed diagnostics retained.
• Verify the result report ordering, expanded FWHM/aperture/blend/background diagnostics, selectable read-only text and persistent window lifetime.
• Run Auto Star Measure on Stars and Comet images, then force saturation and coverage failures and continue manually.
• Edit automatically added stars and verify each displayed aperture and quality flag.
• Enter Measure Stars while finder is active and verify queue behaviour.
• Test Slice and persistent ACF Review lifecycle, then replace/reset the observation and verify stale workers cannot commit and comet_center_meta is empty.
• Open Reference stars with accepted, pairwise-rejected, quality-rejected and user-disabled stars. Verify Use is checked only for the accepted ensemble, the Pairwise status column is absent and Reason explains each exclusion.
• Recheck an incompatible star, verify it remains selected while the solution is pending, then press Recalculate and verify it becomes unchecked again.
• Open a DSLR directory containing direct TG FITS plus CALIB/WCS/CO/ST derivatives with historical RAW cards. Verify no false conflict; then introduce a second genuine direct representation and require a blocking conflict.
• Zoom and pan the active Comet image, choose Measure comet again and verify the viewport is preserved. Repeat for Stars and Tail; verify a wrong-role start still switches and redraws once.

Reported automated status

• R74 focused automatic DSLR gate: 58 passed, 0 failed; the complete pytest run reached 12% without failure before the environment time limit.
• R73 focused saturation/camera integration gate: 75 passed, 0 failed.
• R73 complete deterministic QA: 1125 passed, 3 skipped, 0 failed (157; 324; 327 + 3 skipped; 317).
• R61 focused cumulative R55–R61 gate: 75 passed.
• R61 focused centroid, Comet Finder, hand-off and ACF gate: 78 passed.
• R61 updated historical astrometry/integration gate: 26 passed.
• Retained R50 Automatic Star Measure policy: 13 tests OK.
• R69 complete grouped QA archive: 1062 passed, 3 skipped, 0 failed (236; 421 + 3 skipped; 405).
• The three skips require a complete Astropy FITS runtime.
• Production and QA compilation, release manifest validation, forbidden cache/artifact scan and ZIP integrity: PASS.

R102 ephemeris architecture gate

The schema-5 inventory must show exactly one application-engine call and no direct engine, widget-era selector or two-body range-solver routes:

engine_application_compute: 1
engine_direct_compute: 0
legacy_observation_wrapper calls: 0
element_selection (FindElements calls): 0
two_body_range_solver (finddt calls): 0
unified_ephemeris_service: 13
unified_distance_solver: 8
widget_ephemeris_entry: 4

Historical shadow-comparison helpers are imported from tests/ephemeris_shadow_validation_support.py; ephemeris_shadow_validation.py must not reappear in the production root. Public compatibility APIs remain present and documented even though production callers are zero.

R114 Analyzer/Comparison performance gate

python3 analysis_release_gate.py .
python3 validate_performance_release.py
python3 benchmark_analysis_pipeline.py

The first two commands are release gates; the benchmark is informational. Tests enforce one geometry computation for three formulas of one apparition, zero new work for an identical redraw, zero repeated distance solves, two jobs for four formulas across two apparitions and at least 90% sampling reduction for the fixed 5,000-day Automatic fixture.

A live Qt heartbeat test starts a 250 ms analytical worker and requires the GUI timer to continue firing when complete PyQt5 is available.

• Focused analytical/ephemeris gate: 159 passed, 1 skipped.
• Complete R114 QA: 1664 passed, 7 skipped, 0 failed.

R121 modal ephemeris release gate

python3 modal_ephemeris_release_gate.py
python3 validate_performance_release.py

The standard-library-only gate checks the exact widget matrix, profile capabilities, point/interval scheduling, persistent cache ownership, nested-pool exclusion and transactional guards.

Focused result: 71 passed, 1 skipped, 0 failed. Complete deterministic QA: 1,720 passed, 7 skipped, 0 failed.

Full gate documentation · R121 report

R130 WPlanner performance release gate

• Immutable QThread operation route; no direct heavy compute or QApplication.processEvents().
• At most one spawn pool per operation and no nested pools.
• One job per comet/apparition interval; at most one forward and one backward integration.
• Vector Astropy transform per element batch.
• Deferred bounded persistence cannot invalidate a completed ephemeris.
• Deterministic job/worker scenarios and cold 1 / warm 0 integration probe.
• WStack isolation and existing analytical transactional safeguards.

Complete R130 QA: 1767 passed, 8 skipped, 0 failed. All final validators passed.

R133 WAnalyzer formula/render release gate

• H0/n and dust edits inside prepared coverage produce local results and no analytical worker.
• Axis/filter/Compare/asymmetry changes are render-only.
• Formula validity extension submits only uncovered T intervals.
• View changes during compute accept valid geometry; data/element changes reject stale results.
• Cancel, error and incomplete results preserve the previous graph.
• R123 exact binding and completeness checks remain enforced.
• WComparison stays on its established request-scoped route.

Focused analytical result: 103 passed, 1 skipped, 0 failed; R133-specific module: 8 passed.

R144 WLightCurves final cutover gate

• Exactly one Comet Analyzer tab and one wlightcurves.py compute/render implementation.
• wanalyzer.py and wcomparison.py remain import-only transition aliases.
• WLIGHTCURVES_POLICY is the sole approved analytical policy identity; historical policy names are object-identical aliases.
• The production Comparison editor is absent while the internal legacy parser can load compatible stored data.
• R131–R143 N-target, channels, FitGroups, rendering, fitting, adaptive-break and colour-index contracts remain active.

Source-reported results: split regression suite 489 passed, 2 skipped; final R135–R144 focused set 87 passed.

R149 multi-target formula-document release gate

• Single-target line syntax remains header-free.
• Multi-target Regular and Additional editors contain all selected targets in exact canonical blocks.
• Text before the first block, duplicate/unknown/missing targets, malformed rows and empty Regular blocks fail before mutation.
• The target combobox navigates both complete documents instead of replacing their contents.
• Unchanged Additional blocks retain typed FormulaSeries identity and metadata.
• MagParams.dat replacement uses exact designation matching, same-directory temporary write, flush, fsync and one os.replace().
• Parse or replace failure preserves the old file and all in-memory formula state.
• Successful Save advances only the formula revision and must not restart observation loading.

Source-reported cumulative result: 405 passed, 2 skipped. Both skips require unavailable PyQt5 runtime support. CLEAN and QA manifests contain 124 and 391 files respectively.

R149 report

R154 Comet Analyzer state-machine and interactive gate

The release contract has two non-substitutable layers.

1. The Qt-free matrix verifies deterministic EMPTY, SINGLE_TARGET, MULTI_TARGET and BUSY projections plus adaptive completeness and worker terminal-state invariants.
2. The real PyQt5 runner instantiates WLightCurves and exercises state transitions, topology change while BUSY, adaptive completeness, worker success, exception and cooperative cancellation.

QT_QPA_PLATFORM=offscreen \
python3 r154_comet_analyzer_interactive_release_gate.py --require-qt

QT_QPA_PLATFORM=offscreen python3 r154_comet_analyzer_qt_smoke.py
python3 r154_comet_analyzer_qt_smoke.py --native

Source-reported focused status: 13 passed, 1 skipped; R130–R154 selection: 189 passed, 1 skipped; deterministic state checks: 14/14 PASS. The strict gate failed with qt_runtime_unavailable, which is expected but release-blocking.

Manual/native visual acceptance

Record one-target black rendering, stable target colours after adding/reordering targets, channel-only legend entries, opposite upper-corner legends, read-only fit-group labels, dropdown-only assignments, ordered Multi-colour projections, responsive Load/Analyse/Cancel/error flows, graph preservation after invalid Refresh/error/cancellation, and strict release_ready: true.

R169 point-size release gate

The gate must verify the compact QDoubleSpinBox contract, default 3.0, range 1.0–12.0, step 0.5, identical sizing of pre/post observation artists, unchanged formula/legend marker sizes, persistence compatibility and the zero data/fit/range invalidation boundary.

Source report: 7 focused tests, 10/10 PASS, and 152 passed, 1 skipped for R151–R169. The skip is the real PyQt5 offscreen smoke test.

R169 report

R179 formula interactive qualification gate

Run the deterministic gate on every platform and the strict Qt gate on the release host:

python3 r179_formula_interactive_release_gate.py

QT_QPA_PLATFORM=offscreen python3 r179_formula_interactive_release_gate.py --require-qt

python3 r179_formula_interactive_release_gate.py --require-qt --native

Acceptance covers character-by-character incomplete text, no mutation while typing, exactly one valid Refresh commit, invalid Refresh preservation, byte-identical invalid Save, corrected Save/reload and Analyse immediate commit.

A missing PyQt5 runtime must fail the strict command. Historical R141–R148 source-token tests that require removed implementation details are not current release evidence.

R187 final integration release gate

Use the current Analyzer behavioral suite rather than the R100–R134 historical characterization set:

python3 -m pytest -q \
  tests/test_r13[5-9]*.py \
  tests/test_r14*.py \
  tests/test_r15*.py \
  tests/test_r16*.py \
  tests/test_r17*.py \
  tests/test_r18*.py

Source-reported result: 384 passed, 3 skipped, 0 failed. The R187 gate composes cumulative cutover, R181 automatic Y envelope, R182 Auto range, R183 scenario contract, R184 scenario GUI/runtime, R185 1..N unification and R186 typed-target cleanup.

Release evidence

Current R187 authority

• Current Analyzer regression: 384 passed, 3 skipped, 0 failed.
• All headless release gates R150–R187 passed.
• R187 component gates from cumulative cutover through final integration all report PASS.
• QA/CLEAN common production/distribution files: 120, byte differences: 0.
• QA manifest: 502 entries; CLEAN manifest: 120 entries; ZIP integrity PASS.

Historical evidence

Earlier release counts remain useful for regression history, but intermediate R100–R134 source assertions are not the R187 release criterion.

Adding a regression test

1. Reduce the bug to the smallest deterministic input that still fails.
2. Assert the scientific invariant or serialized contract, not incidental GUI coordinates or implementation order.
3. Include a negative/control case so the test cannot pass by returning a constant.
4. When changing FITS output, test both numeric arrays and header/extension metadata.
5. When changing orchestration, add a static contract check plus a live smoke-test step.
6. Record why the previous behavior was wrong in a short report tied to the build baseline.

R107 comet-designation release gate

python3 comet_designation_inventory.py --enforce .

The AST-only inventory must pass without PyQt5. Tests must include synthetic violations for private text dialogs, legacy validators, non-allowlisted LoadComet(), external regexes, obsolete UI terminology, duplicated messages/examples and missing shared-widget imports.

• Focused R103–R107 suite: 67 passed.
• Extended designation/ephemeris/compatibility gate: 95 passed.
• Full release total: 1589 passed, 6 skipped, 0 failed.

R156 Fit Group and CSV release gate

The dedicated gate verifies that inline renaming uses the controlled registry operation and that CSV export serializes an existing ColourIndexResult without invoking calculation or ephemeris paths.

python3 r156_comet_analyzer_inline_fit_group_csv_release_gate.py
pytest -q tests/test_r156_fit_group_inline_rename_csv_export.py tests/test_r156_release_gate.py

Required assertions include ordinary-click read-only behavior, double-click column restriction, empty/case-insensitive duplicate rejection, immutable group_id, assignment-combobox refresh, export enable/invalidation behavior, render-only axis ordering, exact CSV header order, UTF-8 BOM and absence of compute calls during export.

Source-reported results: 6/6 PASS, 23 passed for the focused R152/R153/R155/R156 set, and 203 passed, 1 skipped for R130–R156. The skip is the existing real-PyQt5 state-machine test.

R161 display-encoding release gate

r161_comet_analyzer_display_encoding_release_gate.py executes the production style resolvers with real Matplotlib artists. The deterministic matrix covers default/reverse encoding, perihelion fill, reverse formula artists, rendered-only legends, palette behavior, identity after reordering, marker capacity, persistence restoration, transactionality, view-only routing and presence of the Qt artifacts.

Source-reported evidence:

• 12/12 PASS deterministic checks;
• R158–R161: 31 passed, 1 skipped;
• R130–R161 Analyzer: 234 passed, 2 skipped;
• R158, R159 and R160 prerequisite gates: PASS;
• R161 gate: PASS_HEADLESS;
• aggregate performance validator: PASS.

The skipped tests are the real PyQt5 state-machine and display-encoding modules. Strict qualification requires:

QT_QPA_PLATFORM=offscreen python3 r161_comet_analyzer_display_encoding_release_gate.py --require-qt

python3 r161_comet_analyzer_qt_smoke.py --native

A missing PyQt5 binding must produce qt_runtime_unavailable, not a false release PASS.

R163 automatic range-invalidation gate

Required checks cover AUTO/USER transitions, complete identity fingerprinting, ordering normalization, field-intent tracking, central plot-entry resolution, settings classification and exclusion of style/FitGroup/Bortle-only changes.

Source evidence: 22 targeted tests, 11/11 PASS, and 260 passed, 2 skipped cumulatively.

R164 style-artist invariance gate

Render identical observation arrays in both Point style modes and assert exact X/Y sequence equality. Also assert formula z-order 2.0, observation z-order 4.0, no single-target reverse formula markers and retained sparse multi-target markers.

Source evidence: 6 targeted tests, 6/6 PASS, and 266 passed, 2 skipped cumulatively.