# Filter Documentation

Filters transform data during the inference pipeline. They are configured in YAML and executed in order of `weight` (lower values run first).

## Quick Start for Reflectometry

Typical Reflectorch configs use these preprocessing filters:

1. `ErrorBarFilter.preprocess_error_bar_filter` (`weight: -10`)
2. `InterpolationFilter.preprocess_interpolate` (`weight: 0`)

Run an example:

```bash
vipr --config @vipr_reflectometry/reflectorch/examples/configs/Ni500.yaml inference run
```

Discover available filters:

```bash
vipr discovery filters
```

---

## Available Filters

### Normalization Filters (`INFERENCE_PREPROCESS_PRE_FILTER`)

Normalization is part of the preprocessing filter chain in the current 5-step workflow (there is no separate normalize step anymore).

These filters are discovered with `enabled_in_config=False` and `weight=-10`.

#### MinMaxNormalizer
Scales `y` values to `[0, 1]`.

**Formula:**
```text
y_norm = (y - min(y)) / (max(y) - min(y))
dy_norm = dy / (max(y) - min(y))
```

**YAML configuration:**
```yaml
vipr:
  inference:
    filters:
      INFERENCE_PREPROCESS_PRE_FILTER:
      - class: vipr.plugins.normalizers.minmax_normalizer.MinMaxNormalizer
        enabled: true
        method: normalize_filter
        weight: -10
```

#### ZScoreNormalizer
Standardizes `y` to mean `0` and standard deviation `1`.

**Formula:**
```text
y_norm = (y - mean(y)) / std(y)
dy_norm = dy / std(y)
```

**YAML configuration:**
```yaml
vipr:
  inference:
    filters:
      INFERENCE_PREPROCESS_PRE_FILTER:
      - class: vipr.plugins.normalizers.zscore_normalizer.ZScoreNormalizer
        enabled: true
        method: normalize_filter
        weight: -10
```

#### LogNormalizer
Applies logarithmic transformation to `y`.

**Formula:**
```text
y_norm = log(y + offset)   # offset is used if min(y) <= 0
dy_norm = dy / (y + offset)
```

**YAML configuration:**
```yaml
vipr:
  inference:
    filters:
      INFERENCE_PREPROCESS_PRE_FILTER:
      - class: vipr.plugins.normalizers.log_normalizer.LogNormalizer
        enabled: true
        method: normalize_filter
        weight: -10
```

---

### Reflectometry Preprocessing Filters (`INFERENCE_PREPROCESS_PRE_FILTER`)

These run in preprocessing (step 3), before prediction (step 4).

#### ErrorBarFilter
Filters experimental curves based on relative error bars and truncation rules.

**Class/method:**
- `vipr_reflectometry.reflectorch.preprocess.error_bar_filter.ErrorBarFilter`
- `preprocess_error_bar_filter`

**Default:** disabled in generated discovery config (`enabled_in_config=False`)

**Parameters:**
- `filter_threshold` (float, default `0.3`): Relative error threshold `dR/R`
- `filter_remove_singles` (bool, default `true`): Remove isolated high-error points
- `filter_remove_consecutives` (bool, default `true`): Truncate at consecutive high-error points
- `filter_consecutive` (int, default `3`): Consecutive point count for truncation trigger
- `filter_q_start_trunc` (float, default `0.1`): Start checking truncation only above this `Q`

**YAML configuration:**
```yaml
vipr:
  inference:
    filters:
      INFERENCE_PREPROCESS_PRE_FILTER:
      - class: vipr_reflectometry.reflectorch.preprocess.error_bar_filter.ErrorBarFilter
        enabled: true
        method: preprocess_error_bar_filter
        weight: -10
        parameters:
          filter_threshold: 0.3
          filter_consecutive: 3
          filter_remove_singles: true
          filter_remove_consecutives: true
          filter_q_start_trunc: 0.1
```

#### InterpolationFilter (Reflectorch)
Interpolates experimental data to the model Q-grid.

**Class/method:**
- `vipr_reflectometry.reflectorch.preprocess.interpolation_filter.InterpolationFilter`
- `preprocess_interpolate`

**Default:** enabled in generated discovery config (`enabled_in_config=True`)

**YAML configuration:**
```yaml
vipr:
  inference:
    filters:
      INFERENCE_PREPROCESS_PRE_FILTER:
      - class: vipr_reflectometry.reflectorch.preprocess.interpolation_filter.InterpolationFilter
        enabled: true
        method: preprocess_interpolate
        weight: 0
```

#### FlowPreprocessor
Preprocessing for NSF-based flow models.

**Class/method:**
- `vipr_reflectometry.flow_models.preprocess.flow_preprocessor.FlowPreprocessor`
- `preprocess_flow`

**Default:** disabled in generated discovery config (`enabled_in_config=False`)

**YAML configuration:**
```yaml
vipr:
  inference:
    filters:
      INFERENCE_PREPROCESS_PRE_FILTER:
      - class: vipr_reflectometry.flow_models.preprocess.flow_preprocessor.FlowPreprocessor
        enabled: true
        method: preprocess_flow
        weight: 0
```

---

## Filter Chaining

All preprocessing filters share the same namespace and are ordered by `weight`.

```yaml
vipr:
  inference:
    filters:
      INFERENCE_PREPROCESS_PRE_FILTER:
      - class: vipr_reflectometry.reflectorch.preprocess.error_bar_filter.ErrorBarFilter
        enabled: true
        method: preprocess_error_bar_filter
        weight: -10
        parameters:
          filter_threshold: 0.3

      - class: vipr_reflectometry.reflectorch.preprocess.interpolation_filter.InterpolationFilter
        enabled: true
        method: preprocess_interpolate
        weight: 0
```

---

## Practical Examples

### Reflectorch default preprocessing
```bash
vipr --config @vipr_reflectometry/reflectorch/examples/configs/Ni500.yaml inference run
```

This config uses `ErrorBarFilter` + `InterpolationFilter`.

### Discovery
```bash
vipr discovery filters
```

---

## Best Practices

### Reflectorch models
- Usually do not add external normalization filters.
- Reflectorch models already apply their own internal scaling.

### ErrorBarFilter tuning
- Start with `filter_threshold=0.3`
- Keep `filter_consecutive=3` as robust default
- Use `filter_remove_singles=true` only if isolated outliers are common

### Notes
- `enabled_in_config` is discovery metadata (default generation), not a runtime override by itself.
- Runtime activation is controlled via YAML `enabled: true/false`.
- `INFERENCE_NORMALIZE_PRE_FILTER` is legacy naming; current normalization filters run under `INFERENCE_PREPROCESS_PRE_FILTER`.
