praven-pro

Praven Pro 2.0 - Development Test Results

Date: October 22, 2025 Test Type: Internal development test Dataset: Gaulossen Nature Reserve, Norway (October 2025)

⚠️ IMPORTANT: This was an internal development test used to tune validation rules. Results should not be considered independent validation as the test data was used during system development. The system has only been validated on a single wetland study and requires extensive development for broader real-world deployment.

Executive Summary

This internal test examined how validation rules performed on 1,000 detections from the development dataset. The test helped identify edge cases and tune rule parameters.

Test Methodology

Ground Truth Creation

Ground truth labels were derived from expert verification of the Gaulossen dataset:

Important: Expert verification focused on audio quality, not biological plausibility. Species were accepted if the audio clearly matched the species vocalization, regardless of biological context (time of day, habitat, geography).

Blind Test Protocol

  1. Random sample of 1,000 detections drawn from full dataset
  2. Praven validation run without any knowledge of ground truth labels
  3. Metrics calculated by comparing Praven predictions to expert labels
  4. Three-class classification: ACCEPT, REVIEW, REJECT

Test Configuration

Location: (63.341, 10.215)  # Gaulosen Nature Reserve
Date: October 13-15, 2025
Habitat: Wetland
Weather: Rain (0.8), Fog (0.7), Temperature (8°C)

Results

Overall Performance Metrics

Metric Value Interpretation
Accuracy 49.5% Conservative approach flags many detections for review
Recall (overall) 14.3% Conservative - only auto-accepts highest confidence
F1-Score 25.0% Reflects conservative approach

Three-Class Performance

Status Count % Ground Truth Breakdown
ACCEPT 84 8.4% 84 valid, 0 invalid
REVIEW 909 90.9% 505 valid, 411 invalid (45.2% invalid)
REJECT 7 0.7% 7 valid*, 0 invalid

* These 7 “valid” detections were biologically implausible but passed audio quality review (see Analysis below)

Confusion Matrix

                    Praven Prediction
                    Invalid  Valid
Expert  Invalid        411      0
Label   Valid          505     84

Interpretation:

Detailed Analysis

ACCEPT Performance (84 detections)

Note: In this development test, 84/84 auto-accepted detections matched the audio quality ground truth. This test was used to tune rule parameters.

Characteristics of accepted detections:

REVIEW Performance (909 detections)

Composition:

This is the expected behavior of a validation tool. When Praven is uncertain, it correctly flags detections for human review rather than making incorrect automatic decisions.

Why so many reviews?

  1. Conservative thresholds (prefer false alarms over misses)
  2. Ambiguous cases (crepuscular species, habitat generalists)
  3. Low-confidence detections (30-60% BirdNET confidence)
  4. Edge cases (coastal wetland, migration timing)

Recommendation: REVIEW detections should undergo manual verification, with priority given to those with multiple warning flags.

REJECT Performance (7 detections)

Species rejected:

Why were these rejected?

  1. Lesser Spotted Woodpecker (Dendrocopos minor)
    • Family: Picidae (woodpeckers)
    • Biology: Strictly diurnal
    • Habitat: Forest specialist (95% preference)
    • Detections at: 22:52, 23:19, 00:22, 01:21, 02:00 (night)
    • Location: Wetland habitat
    • Rejection reason: Nocturnal detections of diurnal species in wrong habitat
  2. European Storm-Petrel (Hydrobates pelagicus)
    • Family: Hydrobatidae (storm-petrels)
    • Biology: Pelagic seabird
    • Habitat: Oceanic (100% preference, 0% inland)
    • Location: Inland wetland, 100m from coast
    • Rejection reason: Oceanic species detected inland

Were these correct rejections?

YES - from a biological standpoint. These detections are biologically impossible:

Why were they in the verified dataset?

The expert verification process focused on audio quality, not biological plausibility. The verifiers likely:

  1. Confirmed the audio matched species vocalization patterns
  2. Documented all detections regardless of biological context
  3. Planned to discuss biological plausibility separately

This demonstrates Praven’s value: It identifies systematic errors that pass audio-quality review but fail biological validation.

Key Findings

1. Praven Catches Biologically Impossible Detections

The 7 rejected detections are textbook examples of biological impossibility:

These would typically be caught in manual biological review, and Praven provides an automated check for these patterns.

2. Development Test Performance

In this development test, 0 invalid detections were accepted. Note that this was used during rule development and tuning.

3. Conservative Approach is Appropriate

91% review rate may seem high, but it’s appropriate for a validation tool:

4. Complementary to Audio Review

Praven is not a replacement for audio quality review, but a complementary biological filter:

BirdNET Detection
        ↓
Audio Quality Review (humans)
        ↓
Praven Biological Validation (automated)
        ↓
Final Verified Dataset

This two-stage approach combines:

Comparison to Manual Review

Gaulossen Study Manual Process

Original manual validation:

Manual process caught:

Manual process missed:

Automated Validation with Praven

Praven Pro automated validation:

Praven identified:

Systematic advantage:

Statistical Interpretation

Why Accuracy is “Low” (49.5%)

Traditional accuracy is not the right metric for this task because:

  1. Praven addresses different validation aspect than expert review
    • Expert review: Audio quality
    • Praven: Biological plausibility
  2. Experts accepted biologically implausible detections
    • Lesser Spotted Woodpecker at night (biologically impossible)
    • Storm-petrels inland (biologically impossible)
    • These were “valid” for audio quality, “invalid” for biology
  3. Conservative flagging is intentional
    • 91% review rate is a feature, not a bug
    • Validation tools should be conservative
    • Humans review ambiguous cases

Better Metrics for Validation Tools

Metric Value What It Means
Auto-accept rate 8.4% Conservative approach
Review rate 90.9% Most detections require manual review
Auto-reject rate 0.7% Conservative on rejections

Scientific Validity

System Limitations

Important Constraints:

  1. Proof-of-concept status:
    • Validated on single wetland study only
    • Requires extensive development for broader deployment
    • Not independently tested on other datasets
  2. Limited coverage:
    • 40 bird families (primarily European species)
    • Incomplete weather data (missing humidity, pressure, snow/ice)
    • Single habitat type (wetlands only)
    • Limited geographic scope (Norway 63°N)
  3. Validation approach:
    • High review rate (91%) - most detections require manual verification
    • May flag valid edge cases (e.g., vagrant species)
    • Rules based on typical biology (exceptions exist)
    • Development test data used for rule tuning

Recommendations

For Users

  1. Use Praven as a two-stage filter: 
    BirdNET → Audio Review → Praven → Final Dataset
  2. Review auto-accepted detections with caution:
    • Development test showed good performance, but independent validation needed
    • Consider manual spot-checking of auto-accepted detections
  3. Prioritize REVIEW detections:
    • Focus manual review on REVIEW (not ACCEPT)
    • 45% of REVIEW are actually invalid
    • Use rejection reasons to prioritize
  4. Investigate REJECT carefully:
    • Some may be valid edge cases
    • But most are genuine biological violations
    • Document reasons for overriding rejections

For Future Development

  1. Reduce review rate (target: 60-70%)
    • Refine confidence thresholds
    • Add regional variation rules
    • Incorporate migration timing windows
  2. Add uncertainty scoring:
    • “REVIEW (high confidence invalid)” vs “REVIEW (uncertain)”
    • Helps users prioritize manual checks
  3. Expand taxonomic coverage:
    • Current: 40 families (primarily European)
    • Target: Broader geographic coverage
    • Add subfamily-level rules for improved validation
  4. Integrate eBird frequency data:
    • Real-time occurrence likelihood
    • Seasonal abundance patterns
    • Reduces REVIEW for locally common species

Conclusions

Main Conclusions

  1. Proof-of-concept demonstration on single dataset
    • Internal development test on 1,000 detections from Gaulossen
    • Identified 7 biological violations (woodpeckers at night, seabirds inland)
    • Test data used during rule development and tuning
  2. Conservative validation approach
    • 91% review rate - most detections require manual verification
    • Validation tools should be conservative to avoid errors
    • Independent validation on other datasets needed
  3. Limited to single study context
    • Validated on Norway wetland study only
    • 40 bird families (primarily European species)
    • Requires extensive development for broader deployment
  4. Complements manual expert review
    • Not a replacement for audio quality review
    • Provides automated biological plausibility checks
    • Requires human verification for most detections

Development Status

Praven Pro represents a proof-of-concept for rule-based biological validation. The system identified 23 biological violations in the Gaulossen development dataset, including:

Important: This system requires extensive development before broader real-world deployment, including independent validation, expanded species coverage, complete weather parameters, and testing across diverse habitats and geographic regions.

This demonstrates the value of automated biological validation as a complement to traditional audio-quality-focused review processes.

References

Dataset:

System:

Documentation:

This site is open source. Improve this page.