# Discovery Plugin Documentation

The Discovery Plugin provides a decorator-based registry system for all VIPR components, enabling automatic discovery and introspection of available handlers, hooks, and filters.

## Overview

The Discovery Plugin serves as the central registry for:
- **Hooks & Filters**: Readonly (hooks) and transformative (filters) callbacks for workflow extensions
- **Data Loaders**: Components for loading input data
- **Model Loaders**: Components for loading ML models
- **Predictors**: Components for executing predictions
- **Postprocessors**: Components for post-processing results

**Key Features:**
- **Decorator-based registration**: Simple `@discover_*` decorators
- **CLI introspection**: Query available components via command line
- **API-ready commands**: Controller methods exposable as FastAPI endpoints via `@api` decorator
- **Plugin filtering**: Filter components by plugin name
- **Automatic discovery**: Components self-register on import

## Architecture

### Registry System

The plugin maintains separate registries for each component type:

```python
# Central registries (in decorators.py)
hooks_discovery: dict[str, list[dict]]
filters_discovery: dict[str, list[dict]]
data_loaders_discovery: dict[str, dict]
model_loaders_discovery: dict[str, dict]
predictors_discovery: dict[str, dict]
postprocessors_discovery: dict[str, dict]
```

### Registration Flow

```
1. Plugin loads and imports modules
2. Decorators execute on import (@discover_filter, @discover_data_loader, etc.)
3. Components register in central registries (hooks_discovery, filters_discovery, etc.)
4. Registries ready for introspection via CLI/API and programmatic access
```

## Available Decorators

### Hook & Filter Registration

| Decorator | Purpose | Registry |
|-----------|---------|----------|
| `@discover_hook()` | Register workflow hooks | `hooks_discovery` |
| `@discover_filter()` | Register workflow filters | `filters_discovery` |

**Parameters:**
- `hook_type` / `filter_type`: Hook/filter name (e.g., `INFERENCE_NORMALIZE_PRE_FILTER`)
- `weight`: Execution priority (default: 0, lower runs first)
- `enabled_in_config`: Enable in generated configs (default: `True`)
- `parameters`: Optional Pydantic model class (via `parameters=MyParams`) for validation/defaults

### Handler Registration

| Decorator | Purpose | Registry |
|-----------|---------|----------|
| `@discover_data_loader()` | Register data loaders | `data_loaders_discovery` |
| `@discover_model_loader()` | Register model loaders | `model_loaders_discovery` |
| `@discover_predictor()` | Register predictors | `predictors_discovery` |
| `@discover_postprocessor()` | Register postprocessors | `postprocessors_discovery` |

**Parameters:**
- `handler_name`: Handler identifier (e.g., `csv_spectrareader`)
- `parameters`: Optional Pydantic model class for validation/default extraction

## Usage Examples

### Registering Hooks/Filters

Hook and filter callbacks are attached to these extension points via YAML configuration, which is passed to the CLI workflow command (e.g., `vipr --config ./config.yaml inference run`).
See [Dynamic Hooks and Filters Extension](../../extensions/dynamic_hooks_filters.md) for details on registering callbacks.

**Example of decorator registration:**
```python
from vipr.plugins.discovery.decorators import discover_filter

class LogNormalizer:
    @discover_filter('INFERENCE_NORMALIZE_PRE_FILTER', enabled_in_config=False)
    def normalize_filter(self, data, **kwargs):
        return data
```

### Registering Data Loader

Real example from Reflectometry Plugin:

```python
from vipr.plugins.inference.handlers.data_loader import DataLoaderHandler
from vipr.plugins.discovery.decorators import discover_data_loader
from vipr.plugins.inference.dataset import DataSet
from pydantic import BaseModel, Field

class CSVSpectraReaderParams(BaseModel):
    data_path: str = Field(..., description="Path to single CSV/DAT/TXT file")
    column_mapping: dict[str, int] = Field(
        default_factory=lambda: {"q": 0, "I": 1},
        description="Column mapping for CSV data",
    )

@discover_data_loader('csv_spectrareader', parameters=CSVSpectraReaderParams)
class CSVSpectraReaderDataLoader(DataLoaderHandler):
    """CSV/DAT/TXT data loader for reflectometry data."""
    
    class Meta:
        label = 'csv_spectrareader'
    
    def _load_data(self, data_path: str, column_mapping: dict = None, **kwargs) -> DataSet:
        # Load CSV using SpectraReader
        # Returns DataSet with batch_size=1
        ...
```

### Registering Model Loader

Real example from Reflectometry Plugin:

```python
from vipr.handlers.model_loader import ModelLoaderHandler
from vipr.plugins.discovery.decorators import discover_model_loader
from typing import Optional
from pydantic import BaseModel, ConfigDict, Field

from vipr_reflectometry.paths import get_model_family_root
from vipr_reflectometry.reflectorch.load_model.timed_inference_model import TimedInferenceModel

class ReflectorchModelLoaderParams(BaseModel):
    """Parameters for the reflectorch model loader."""

    model_config = ConfigDict(extra='forbid')

    config_name: str = Field(default='b_mc_point_xray_conv_standard_L2_InputQ')
    model_name: Optional[str] = Field(default=None)
    root_dir: Optional[str] = Field(default=None)
    weights_format: str = Field(default='safetensors')
    repo_id: str = Field(default='valentinsingularity/reflectivity')
    device: str = Field(default='cpu')

@discover_model_loader('reflectorch', ReflectorchModelLoaderParams)
class ReflectorchModelLoader(ModelLoaderHandler):
    class Meta:
        label = 'reflectorch'
        
    def _load_model(self, **kwargs):
        """
        Loads a reflectorch model.
        
        Args:
            model_type: Type of reflectorch model ('reflectorch' expected)
            **kwargs: Model parameters such as config_name, device, etc.
            
        Returns:
            Loaded reflectorch model
        """
        params = ReflectorchModelLoaderParams.model_validate(kwargs)
        config_name = params.config_name
        device = params.device

        root_dir = get_model_family_root('reflectorch')
        if root_dir is None:
            raise ValueError("REFLECTOMETRY_ROOT_DIR environment variable must be set")
        
        # Load and return the model with timing measurements
        model = TimedInferenceModel(
            config_name=config_name,
            root_dir=str(root_dir),
            device=device
        )
        return model
```

## CLI Commands

### Query Components Summary

```bash
# All components
vipr discovery plugins

# Filter by plugin
vipr discovery plugins --plugin normalizers
```

**Output:**
```json
{
  "components_summary": {
    "model_loaders": 2,
    "data_loaders": 3,
    "hooks": 15,
    "filters": 12,
    "predictors": 1,
    "postprocessors": 0
  }
}
```

### Query Specific Component Types

```bash
# List all model loaders
vipr discovery model-loaders

# List data loaders for specific plugin
vipr discovery data-loaders --plugin reflectometry

# List all hooks
vipr discovery hooks

# List hooks of specific type
vipr discovery hooks --hook-type INFERENCE_POSTPROCESS_PRE_PRE_FILTER_HOOK

# List hooks for specific plugin
vipr discovery hooks --plugin reflectometry

# List all filters
vipr discovery filters

# List filters of specific type
vipr discovery filters --filter-type INFERENCE_NORMALIZE_PRE_FILTER

# List predictors
vipr discovery predictors

```

### Query All Components

```bash
# Complete registry dump
vipr discovery components

# Filter by plugin
vipr discovery components --plugin reflectometry
```

### Query Available Types

```bash
# List all hook types (inference workflow)
vipr discovery hook-types

# List all filter types (inference workflow)
vipr discovery filter-types
```

## API Integration

All discovery commands are available as HTTP endpoints via the API plugin.

### Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/discovery/plugins` | GET | Components summary |
| `/api/discovery/model-loaders` | GET | List model loaders |
| `/api/discovery/data-loaders` | GET | List data loaders |
| `/api/discovery/hooks` | GET | List hooks |
| `/api/discovery/filters` | GET | List filters |
| `/api/discovery/predictors` | GET | List predictors |
| `/api/discovery/postprocessors` | GET | List postprocessors |
| `/api/discovery/components` | GET | All components |
| `/api/discovery/hook-types` | GET | Available hook types |
| `/api/discovery/filter-types` | GET | Available filter types |
| `/api/discovery/example-configs` | GET | List usable packaged example configs |
| `/api/discovery/example-config` | GET | Get one packaged example config by id |

### Example Config Discovery

Discovery also exposes packaged example configurations for plugins.

Unlike handlers, hooks, and filters, example configs are not stored in the decorator registries. Instead, VIPR scans installed plugin packages for packaged YAML files under:

```text
<plugin_package>/<group>/examples/configs/*.yaml
```

or

```text
<plugin_package>/<group>/examples/configs/*.yml
```

For each matching file:

- `plugin` comes from the plugin entry point name
- `group` comes from the path segment immediately before `examples/configs`
- `id` is built as `<plugin>.<group>.<file_stem>`

Only usable examples are returned. The referenced handlers, predictors, hooks, and filters must be available in the current runtime. Callbacks with `enabled: false` are ignored during availability checks.

See [Developing VIPR Plugins](../../plugins/developing.md#packaged-example-configs) for the packaging convention and requirements.

### Example API Call

```bash
# Get all model loaders
curl http://localhost:8000/api/discovery/model-loaders

# Get hooks for specific plugin
curl "http://localhost:8000/api/discovery/hooks?plugin=normalizers"

# Get example configs for one plugin
curl "http://localhost:8000/api/discovery/example-configs?plugin=reflectometry"

# Get one packaged example config by id
curl "http://localhost:8000/api/discovery/example-config?id=reflectometry.reflectorch.Fe_Pt_DN"
```

## Plugin Filtering

The Discovery system supports filtering by plugin name for both internal and external plugins.

### Internal Plugins

Pattern: `vipr.plugins.{plugin_name}.`

```bash
vipr discovery hooks --plugin normalizers
# Finds: vipr.plugins.normalizers.log_normalizer.LogNormalizer
```

### External Plugins

External plugins are discovered via Python entry points. See [Plugin Development Guide](../README.md#naming-convention) for details on entry point configuration and naming conventions.

**Plugin Filtering Example:**
```bash
vipr discovery hooks --plugin reflectometry
# Example match: vipr_reflectometry.reflectorch.reflectorch_extension.Reflectorch
```

**How it works:**
- **Input:** `--plugin reflectometry` (plugin name from entry point)
- **Pattern constructed:** `vipr_reflectometry.*` (automatic `vipr_` prefix added)
- **Matches:** All components in the `vipr_reflectometry` package

This is why the `vipr_{plugin_name}` naming convention is recommended – it enables automatic filtering by plugin name.

## Integration with Dynamic Hooks Extension

The Discovery Plugin provides the registry foundation for the [Dynamic Hooks and Filters Extension](../../extensions/dynamic_hooks_filters.md):

1. **Decorator Registration**: Components use `@discover_hook`/`@discover_filter`
2. **Registry Validation**: Dynamic Hooks Extension validates YAML configs against registries
3. **Security**: Only registered components can be loaded from configuration
4. **Parameter Validation**: Parameter schemas from decorators enforce type safety

**Flow:**
```
@discover_filter decorator
    ↓
Register in filters_discovery
    ↓
Dynamic Hooks Extension reads registry
    ↓
Validates YAML config against registry
    ↓
Imports and registers callback
```

## Registry Data Structure

### Hook/Filter Entry

```python
{
    'enabled_in_config': False,
    'weight': 0,
    'method': 'normalize_filter',
    'class': 'vipr.plugins.normalizers.log_normalizer.LogNormalizer',
    'component_id': 'vipr.plugins.normalizers.log_normalizer.LogNormalizer.normalize_filter',
    'parameters': {'scale_factor': 1.0},
    'parameters_schema': {
        'type': 'object',
        'properties': {'scale_factor': {'type': 'number', 'default': 1.0}}
    }
}
```

### Handler Entry

```python
{
    'handler': 'csv_loader',
    'class': 'vipr.plugins.data_loader.csv_loader.CSVDataLoader',
    'parameters': {'delimiter': ','},
    'parameters_schema': {
        'type': 'object',
        'properties': {
            'file_path': {'type': 'string'},
            'delimiter': {'type': 'string', 'default': ','}
        },
        'required': ['file_path']
    }
}
```

## Best Practices

1. **Always use decorators**: Register all components via `@discover_*` decorators
2. **Unique handler names**: Use descriptive, unique names for handlers
3. **Document parameters**: Include parameter schemas in decorators
4. **Class-only callbacks**: Hooks/filters must be class methods (enforced)
5. **Set enabled_in_config appropriately**: Use `False` for optional components

## Programmatic Access

### Query Registries in Code

```python
from vipr.plugins.discovery.decorators import (
    get_discovered_hooks,
    get_discovered_filters,
    get_discovered_data_loaders,
    get_discovered_model_loaders
)

# Get all hooks
all_hooks = get_discovered_hooks()

# Get hooks of specific type
start_hooks = get_discovered_hooks(hook_type='INFERENCE_START_HOOK')

# Get hooks for plugin
plugin_hooks = get_discovered_hooks(plugin_name='normalizers')

# Get data loaders
loaders = get_discovered_data_loaders()

# Filter by plugin
reflectometry_loaders = get_discovered_data_loaders(plugin_name='reflectometry')
```

## Configuration Generation

The Discovery system can be used to automatically generate YAML/JSON configurations based on registered components. This is particularly useful for:

- **Default configurations**: Generate ready-to-use starter configs for plugins
- **User templates**: Provide users with pre-configured, customizable setups

### Use Case: Plugin Configuration Templates

Plugins can provide commands to generate standard configurations by querying the Discovery registries for their components.

**Example from Reflectometry Plugin:**

```python
from vipr.plugins.discovery.decorators import (
    get_discovered_filters, get_discovered_hooks,
    get_discovered_data_loaders, get_discovered_model_loaders,
    get_discovered_predictors
)

def generate_standard_config(self):
    """Builds a standard configuration based on registry components."""
    # Query registered components for this plugin
    all_model_loaders = get_discovered_model_loaders('reflectometry')
    all_data_loaders = get_discovered_data_loaders('reflectometry')
    all_predictors = get_discovered_predictors('reflectometry')
    
    # Find specific handlers by name
    model_loader = self._find_handler(all_model_loaders, 'reflectorch')
    data_loader = self._find_handler(all_data_loaders, 'csv_spectrareader')
    predictor = self._find_handler(all_predictors, 'reflectorch_predictor')
    
    # Get all hooks and filters
    hooks = get_discovered_hooks()
    filters = get_discovered_filters()
    
    # Build configuration with default parameter values
    config = {
        'inference': {
            'load_model': self._component_to_config(model_loader),
            'load_data': self._component_to_config(data_loader),
            'prediction': self._component_to_config(predictor),
            'hooks': self._generate_hooks_config(hooks),
            'filters': self._generate_filters_config(filters)
        }
    }
    
    return config

def _component_to_config(self, component):
    """Converts a component to configuration format."""
    config = {'handler': component.get('handler')}
    
    # 'parameters' already contains default values extracted from the Pydantic model
    parameters = component.get('parameters', {})
    
    if parameters:
        config['parameters'] = parameters
    
    return config
```

### CLI Usage

```bash
# Generate standard config (returns JSON)
vipr reflectometry generate-standard-config

# Write config to YAML file
vipr reflectometry write-standard-config-to-yaml --output config/my_config.yaml
```

## See Also

- [Dynamic Hooks and Filters Extension](../../extensions/dynamic_hooks_filters.md) - Uses Discovery registries for validation
- [Inference Plugin](./inference.md) - Uses handlers registered via Discovery
- [Normalizers Plugin](./normalizers.md) - Example of filter registration
- `vipr-core/vipr/plugins/discovery/decorators.py` - Decorator implementations
- `vipr-core/vipr/plugins/discovery/controllers.py` - CLI commands
