API Reference

ConformalTuner

class confopt.tuning.ConformalTuner(objective_function, search_space, minimize=True, n_candidates=5000, warm_starts=None, dynamic_sampling=True)

Conformal prediction-based hyperparameter optimization framework.

Implements a sophisticated hyperparameter optimization system that combines random search initialization with conformal prediction-guided exploration. The tuner uses uncertainty quantification to make statistically principled decisions about which configurations to evaluate next, providing both efficiency improvements and theoretical guarantees.

The optimization process follows a two-phase strategy: 1. Random search phase: Explores the search space randomly to establish baseline performance 2. Conformal search phase: Uses conformal prediction models to guide configuration selection

The framework supports adaptive retraining of prediction models, dynamic configuration sampling, and multi-armed bandit optimization for automatically tuning searcher parameters. Statistical validity is maintained through proper conformal prediction procedures that provide distribution-free coverage guarantees.

Parameters:

• objective_function (callable) – Function to optimize, must accept ‘configuration’ dict parameter

• search_space (Dict[str, IntRange | FloatRange | CategoricalRange]) – Dictionary mapping parameter names to ParameterRange objects

• minimize (bool) – Whether to minimize (True) or maximize (False) the objective function

• n_candidates (int) – Number of candidate configurations to sample from the search space at each iteration of conformal search

• warm_starts (List[Tuple[Dict, float]] | None) – Pre-evaluated (configuration, performance) pairs to seed the search

• dynamic_sampling (bool) – Whether to dynamically resample configuration candidates at each iteration of conformal search

• random_state – Random seed for reproducible results. Default: None.

study

Container for storing trial results and optimization history

config_manager

Handles configuration sampling and tracking

search_timer

Tracks total optimization runtime

check_objective_function()

Validate objective function signature and type annotations.

Ensures the objective function conforms to the required interface: single parameter named ‘configuration’ of type Dict, returning numeric value. This validation prevents runtime errors and ensures compatibility with the optimization framework.

Raises:

• ValueError – If function signature doesn’t match requirements

• TypeError – If type annotations are incorrect

Return type:

None

process_warm_starts()

Initialize optimization with pre-evaluated configurations.

Processes warm start configurations by marking them as searched and creating corresponding trial records. This allows the optimization to begin with prior knowledge, potentially accelerating convergence by skipping known poor configurations and leveraging good starting points.

The warm start configurations are treated as iteration 0 data and assigned the ‘warm_start’ acquisition source for tracking purposes.

Return type:

None

initialize_tuning_resources()

Initialize core optimization components and data structures.

Sets up the study container for trial tracking, configuration manager for handling search space sampling, and processes any warm start configurations. The configuration manager uses the optimized incremental approach for maximum performance.

Return type:

None

random_search(max_random_iter, max_runtime=None, max_searches=None, verbose=True)

Execute random search phase to initialize optimization with baseline data.

Performs uniform random sampling of configurations to establish initial performance landscape understanding. This phase is crucial for subsequent conformal prediction model training, as it provides the foundational dataset for uncertainty quantification.

Parameters:

• max_random_iter (int) – Maximum number of random configurations to evaluate

• max_runtime (int | None) – Optional runtime budget in seconds

• max_searches (int | None) – Optional total iteration limit

• verbose (bool) – Whether to display progress information

Return type:

None

setup_conformal_search_resources(verbose, max_runtime, max_searches)

Initialize progress tracking and iteration limits for conformal search.

Sets up the progress bar manager for displaying search progress and calculates the maximum number of conformal search iterations based on total limits and already completed trials from previous phases.

Parameters:

• verbose (bool) – Whether to display progress information

• max_runtime (int | None) – Optional maximum runtime in seconds

• max_searches (int | None) – Optional maximum total iterations

Returns:

Tuple of (progress_manager, conformal_max_searches)

Return type:

Tuple[ProgressBarManager, float]

initialize_searcher_optimizer(optimizer_framework)

Initialize searcher parameter tuner.

Parameters:

optimizer_framework (str | None) – Tuning strategy (‘decaying’, ‘fixed’, None)

Returns:

Configured optimizer instance

retrain_searcher(searcher, X, y, tuning_count)

Train conformal prediction searcher on accumulated data.

Fits the conformal prediction model using the provided data, tracking training time and model performance for adaptive parameter optimization. The tuning_count parameter controls internal hyperparameter optimization within the searcher.

Parameters:

• searcher (BaseConformalSearcher) – Conformal searcher instance to train

• X (array) – Feature matrix (sign-adjusted)

• y (array) – Target values (sign-adjusted)

• tuning_count (int) – Number of internal tuning iterations

Returns:

Training runtime in seconds

Return type:

float

select_next_configuration(searcher, searchable_configs, transformed_configs)

Select the most promising configuration using conformal predictions.

Uses the conformal searcher to predict lower bounds for all available configurations and selects the one with the minimum predicted lower bound. This implements a pessimistic acquisition strategy that favors configurations with high confidence of good performance.

Parameters:

• searcher (BaseConformalSearcher) – Trained conformal searcher for predictions

• searchable_configs (List) – List of available configuration dictionaries

• transformed_configs (array) – Scaled feature matrix for configurations

Returns:

Selected configuration dictionary

Return type:

Dict

get_interval_if_applicable(searcher, transformed_config)

Get prediction interval bounds if supported by searcher.

Returns the lower and upper bounds of the prediction interval for configurations using lower bound samplers. This provides the raw interval information for storage and analysis.

Parameters:

• searcher (BaseConformalSearcher) – Conformal searcher instance

• transformed_config (array) – Scaled configuration features

Returns:

Tuple of (lower_bound, upper_bound) if applicable, (None, None) otherwise

Return type:

Tuple[float | None, float | None]

update_optimizer_parameters(optimizer, search_iter)

Update multi-armed bandit optimizer and select new parameter values.

Updates the parameter optimizer with the current search iteration and selects new parameter values for subsequent iterations.

Parameters:

• optimizer – Multi-armed bandit optimizer instance

• search_iter (int) – Current search iteration number

Returns:

Tuple of (new_tuning_count, new_searcher_retuning_frequency)

Return type:

Tuple[int, int]

conformal_search(searcher, verbose, max_searches, max_runtime, optimizer_framework=None)

Execute conformal prediction-guided hyperparameter search.

Implements the main conformal search loop that iteratively trains conformal prediction models, selects promising configurations based on uncertainty quantification, and updates the models with new observations.

Parameters:

• searcher (BaseConformalSearcher) – Conformal prediction searcher for configuration selection

• verbose (bool) – Whether to display search progress

• max_searches (int | None) – Maximum total iterations including previous phases

• max_runtime (int | None) – Maximum total runtime budget in seconds

• optimizer_framework (str | None) – Parameter tuning strategy

Return type:

None

tune(max_searches=100, max_runtime=None, searcher=None, n_random_searches=15, optimizer_framework=None, random_state=None, verbose=True)

Execute hyperparameter optimization using conformal prediction surrogate models.

Performs intelligent hyperparameter search by randomly sampling an initial number of hyperparameter configurations, then activating surrogate based search according to the specified searcher.

Parameters:

• max_searches (int | None) – Maximum total configurations to search (random + conformal searches). Default: 100.

• max_runtime (int | None) – Maximum search time in seconds. Search will terminate after this time, regardless of iterations. Default: None (no time limit).

• searcher (QuantileConformalSearcher | None) – Conformal searcher object responsible for the selection of candidate hyperparameter configurations. When none is provided, the searcher defaults to a QGBM surrogate with a Thompson Sampler. Should you want to use a custom searcher, see confopt.selection.acquisition for searcher instantiation and confopt.selection.acquisition.samplers to set the searcher’s sampler. Default: None.

• n_random_searches (int) – Number of random configurations to evaluate before conformal search. Provides initial training data for the surrogate model. Default: 15.

• optimizer_framework (Literal['decaying', 'fixed'] | None) – Controls how and when the surrogate model tunes its own parameters (this is different from tuning your target model). Options are ‘decaying’ for adaptive tuning with increasing intervals over time, ‘fixed’ for deterministic tuning at fixed intervals, or None for no tuning. Surrogate tuning adds computational cost and is recommended only if your target model takes more than 5 minutes to train. Default: None.

• random_state (int | None) – Random seed for reproducible results. Default: None.

• verbose (bool) – Whether to enable progress display. Default: True.

Return type:

None

Example

Basic usage:

import numpy as np
from confopt.tuning import ConformalTuner
from confopt.wrapping import FloatRange

def objective(configuration):
    x1 = configuration['x1']
    x2 = configuration['x2']
    A = 10
    n = 2
    return A * n + (x1**2 - A * np.cos(2 * np.pi * x1)) + (x2**2 - A * np.cos(2 * np.pi * x2))

search_space = {
    'x1': FloatRange(min_value=-5.12, max_value=5.12),
    'x2': FloatRange(min_value=-5.12, max_value=5.12)
}

tuner = ConformalTuner(
    objective_function=objective,
    search_space=search_space,
    minimize=True
)

tuner.tune(n_random_searches=10, max_searches=50)

best_config = tuner.get_best_params()
best_score = tuner.get_best_value()

get_best_params()

Retrieve the best configuration found during optimization.

Returns the parameter configuration that achieved the optimal objective function value, according to the specified optimization direction.

Returns:

Dictionary containing the optimal parameter configuration

Return type:

Dict

get_best_value()

Retrieve the best objective function value achieved during optimization.

Returns the optimal performance value found across all evaluated configurations, according to the specified optimization direction.

Returns:

Best objective function value achieved

Return type:

float

Parameter Ranges

IntRange

class confopt.wrapping.IntRange(*, min_value, max_value, log_scale=False)

Range of integer values for hyperparameter optimization.

Parameters:

• min_value (int)

• max_value (int)

• log_scale (bool)

min_value: int

max_value: int

log_scale: bool

classmethod max_gt_min(v, info)

Parameters:

info (ValidationInfo)

classmethod log_scale_positive_values(v, info)

Parameters:

info (ValidationInfo)

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

FloatRange

class confopt.wrapping.FloatRange(*, min_value, max_value, log_scale=False)

Range of float values for hyperparameter optimization.

Parameters:

• min_value (float)

• max_value (float)

• log_scale (bool)

min_value: float

max_value: float

log_scale: bool

classmethod max_gt_min(v, info)

Parameters:

info (ValidationInfo)

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

CategoricalRange

class confopt.wrapping.CategoricalRange(*, choices)

Categorical values for hyperparameter optimization.

Parameters:

choices (list[str | int | float | bool])

choices: list[str | int | float | bool]

classmethod non_empty_choices(v)

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

QuantileConformalSearcher

class confopt.selection.acquisition.QuantileConformalSearcher(quantile_estimator_architecture, sampler, n_pre_conformal_trials=32, n_calibration_folds=3, calibration_split_strategy='adaptive')

Conformal acquisition function using quantile-based prediction intervals.

Implements acquisition functions based on quantile conformal prediction, directly estimating prediction quantiles and applying conformal adjustments when sufficient calibration data is available. Provides flexible acquisition strategies while maintaining coverage guarantees.

This approach is particularly effective when the objective function exhibits asymmetric uncertainty or when specific quantile behaviors are of interest. Automatically switches between conformalized and non-conformalized modes based on data availability.

Parameters:

• quantile_estimator_architecture (Literal['qgbm', 'qgp', 'qrf', 'qknn', 'ql', 'qleaf', 'qens5']) – Architecture identifier for the quantile estimator. Must be registered in ESTIMATOR_REGISTRY and support simultaneous multi-quantile estimation.

• sampler (LowerBoundSampler | ThompsonSampler | PessimisticLowerBoundSampler | ExpectedImprovementSampler) – Acquisition strategy that defines point selection behavior.

• n_pre_conformal_trials (int) – Minimum total samples required for conformal mode. Below this threshold, uses direct quantile predictions.

• n_calibration_folds (int)

• calibration_split_strategy (Literal['cv', 'train_test_split', 'adaptive'])

quantile_estimator_architecture

Quantile estimator configuration.

n_pre_conformal_trials

Threshold for conformal vs non-conformal mode.

conformal_estimator

Fitted QuantileConformalEstimator instance.

point_estimator

Optional point estimator for optimistic Thompson sampling.

Mathematical Foundation:

Uses quantile conformal prediction where intervals have the form:

Conformalized: [q̂_{α/2}(x) - C_α, q̂_{1-α/2}(x) + C_α] Non-conformalized: [q̂_{α/2}(x), q̂_{1-α/2}(x)]

Where: - q̂_τ(x): τ-quantile estimate at location x - C_α: Conformal adjustment based on nonconformity scores - Mode selection based on n_pre_conformal_trials threshold

Adaptive Behavior:

Supports sampler-specific adaptation mechanisms including upper quantile capping for conservative samplers and point estimator integration for optimistic Thompson sampling when enabled.

fit(X, y, tuning_iterations=0, random_state=None)

Fit the quantile conformal estimator for acquisition.

Trains the quantile estimator and sets up conformal calibration, with automatic mode selection based on data availability. Handles sampler-specific configurations and point estimator setup for optimistic Thompson sampling and median estimation for bound samplers.

Parameters:

• X (array) – Input features for estimator fitting, shape (n_samples, n_features).

• y (array) – Target values for estimator fitting, shape (n_samples,).

• tuning_iterations (int | None) – Number of hyperparameter tuning iterations (0 disables tuning).

• random_state (int | None) – Random seed for reproducible results.

Implementation Process:

1. Store data for potential use by acquisition strategies

2. Configure sampler-specific quantile estimation and point estimators

3. Set default random state for Information Gain Sampler if not provided

4. Fit QuantileConformalEstimator with internal data splitting

5. Store estimator performance metrics for quality assessment

Sampler-Specific Setup:

• Bound samplers: Median (0.5 quantile) estimator for UCB point estimates

• Optimistic Thompson: Additional point estimator training

• Information-based: Full quantile range support

Samplers

PessimisticLowerBoundSampler

class confopt.selection.sampling.bound_samplers.PessimisticLowerBoundSampler(interval_width=0.8, adapter=None)

Conservative acquisition strategy using pessimistic lower bounds.

This sampler implements a conservative approach to uncertainty quantification by focusing exclusively on the lower bounds of prediction intervals. The strategy prioritizes risk-averse decision making by assuming pessimistic scenarios, making it suitable for applications where conservative estimates are preferred over aggressive exploration.

The approach provides simple, interpretable acquisition values while maintaining proper uncertainty quantification through conformal prediction intervals. The single-interval design offers computational efficiency and straightforward interpretation.

Methodological characteristics: - Single confidence level with configurable interval width - Direct lower bound extraction for acquisition decisions - Optional adaptive interval width adjustment - Conservative bias suitable for risk-averse optimization

Parameters:

• interval_width (float)

• adapter (Literal['DtACI', 'ACI'] | None)

fetch_alphas()

Retrieve current alpha value for interval construction.

Returns:

Single-element list containing the current alpha value (miscoverage rate).

Return type:

List[float]

update_interval_width(beta)

Update interval width based on observed coverage rate.

This method applies adaptive interval width adjustment using empirical coverage feedback. The alpha parameter is updated to maintain target coverage while optimizing interval efficiency for conservative bound estimation.

Parameters:

beta (float) – Observed coverage rate for the prediction interval, representing the fraction of true values falling within the interval.

Return type:

None

LowerBoundSampler

class confopt.selection.sampling.bound_samplers.LowerBoundSampler(interval_width=0.8, adapter=None, beta_decay='logarithmic_decay', c=1, beta_max=10)

Lower Confidence Bound acquisition strategy with adaptive exploration.

This sampler implements a Lower Confidence Bound (LCB) strategy adapted for minimization problems. The approach balances exploitation of promising regions with exploration of uncertain areas through an adaptive exploration parameter that decays over time, providing theoretical guarantees for convergence.

The strategy extends the pessimistic lower bound approach with sophisticated exploration control, making it suitable for efficient optimization under uncertainty with provable regret bounds.

Mathematical formulation: LCB(x) = μ(x) - β(t) * σ(x) where μ(x) is the point estimate, σ(x) is the interval width, and β(t) is the time-dependent exploration parameter.

Exploration decay strategies: - Inverse square root: β(t) = sqrt(c/t) for aggressive decay - Logarithmic: β(t) = sqrt(c*log(t)/t) for balanced exploration

Performance characteristics: - Theoretical regret guarantees under appropriate decay schedules - Adaptive exploration balancing exploitation and uncertainty quantification - Efficient single-interval computation with optional adaptation

Parameters:

• interval_width (float)

• adapter (Literal['DtACI', 'ACI'] | None)

• beta_decay (Literal['inverse_square_root_decay', 'logarithmic_decay'] | None)

• c (float)

• beta_max (float)

update_exploration_step()

Update exploration parameter based on decay schedule and time step.

This method advances the time step and computes the new exploration parameter according to the specified decay strategy. The decay ensures that exploration decreases over time as confidence in the model increases, following theoretical requirements for convergence guarantees.

calculate_ucb_predictions(point_estimates=None, half_width=None)

Calculate Lower Confidence Bound predictions for acquisition.

This method computes LCB values by combining point estimates with exploration bonuses based on interval widths and the current exploration parameter. The result provides acquisition values that balance exploitation of promising regions with exploration of uncertain areas.

Parameters:

• point_estimates (ndarray) – Point predictions (e.g., posterior means) for each candidate. These represent the exploitation component.

• half_width (ndarray) – Uncertainty estimates (e.g., half interval widths) for each candidate. These drive the exploration component.

Returns:

Array of LCB acquisition values. Lower values indicate more attractive candidates for minimization problems.

Return type:

ndarray

ThompsonSampler

class confopt.selection.sampling.thompson_samplers.ThompsonSampler(n_quantiles=4, adapter=None, enable_optimistic_sampling=False)

Thompson sampling acquisition strategy for conformal prediction optimization.

This class implements Thompson sampling using conformal prediction intervals as approximations to posterior distributions. The sampler randomly draws values from prediction intervals to balance exploration and exploitation, providing a principled approach to acquisition function optimization under uncertainty.

The implementation supports multiple confidence levels through quantile-based interval construction, adaptive interval width adjustment based on coverage feedback, and optional optimistic sampling for enhanced exploration.

Methodological approach: - Constructs nested prediction intervals using symmetric quantile pairing - Samples randomly from flattened interval representations - Optionally incorporates point estimates for optimistic exploration - Adapts interval widths using empirical coverage rates

Performance characteristics: - O(n_intervals * n_observations) sampling complexity - Efficient vectorized operations for large candidate sets - Minimal memory overhead through flattened representations

Parameters:

• n_quantiles (int)

• adapter (Literal['DtACI', 'ACI'] | None)

• enable_optimistic_sampling (bool)

fetch_alphas()

Retrieve current alpha values for interval construction.

Returns:

List of alpha values (miscoverage rates) for each confidence level, ordered from lowest to highest confidence (decreasing alpha values).

Return type:

List[float]

update_interval_width(betas)

Update interval widths using observed coverage rates.

This method applies adaptive interval width adjustment based on empirical coverage feedback. Each interval’s alpha parameter is updated independently using its corresponding observed coverage rate, allowing for fine-grained control over uncertainty quantification accuracy.

Parameters:

betas (List[float]) – Observed coverage rates for each interval, in the same order as the alpha values. Values should be in [0, 1] representing the fraction of true values falling within each interval.

calculate_thompson_predictions(predictions_per_interval, point_predictions=None)

Generate Thompson sampling predictions through random interval sampling.

This method implements the core Thompson sampling logic by randomly selecting values from the available prediction intervals. The sampling process approximates drawing from posterior distributions over the objective function, enabling principled exploration-exploitation trade-offs.

Methodology: 1. Flatten prediction intervals into efficient matrix representation 2. Randomly sample column indices for each observation 3. Extract corresponding interval bounds 4. Optionally apply optimistic capping using point estimates

Parameters:

• predictions_per_interval (List[ConformalBounds]) – List of ConformalBounds objects containing lower and upper bounds for each confidence level. All bounds must have the same number of observations.

• point_predictions (ndarray | None) – Optional point estimates for optimistic sampling. When provided and optimistic sampling is enabled, sampled values are capped at point estimates to encourage exploitation.

Returns:

Array of sampled predictions with shape (n_observations,). Each value represents a random draw from the corresponding observation’s prediction intervals, potentially capped by point estimates.

Return type:

ndarray

ExpectedImprovementSampler

class confopt.selection.sampling.expected_improvement_samplers.ExpectedImprovementSampler(n_quantiles=4, adapter=None, current_best_value=inf, num_ei_samples=20)

Expected Improvement acquisition strategy using conformal prediction intervals.

This class implements Expected Improvement for optimization under uncertainty using conformal prediction intervals as uncertainty quantification. The sampler estimates expected improvement through Monte Carlo sampling from prediction intervals, providing a principled approach to balancing exploration and exploitation without requiring explicit posterior models.

Methodological approach: - Constructs nested prediction intervals using symmetric quantile pairing - Estimates expected improvement via Monte Carlo sampling from intervals - Tracks current best value for improvement computation - Adapts interval widths using empirical coverage feedback

The acquisition function naturally balances exploration (high uncertainty regions) with exploitation (promising low-value regions) by computing expected improvements over the current best observation.

Performance characteristics: - O(n_samples * n_intervals * n_observations) for EI computation - Efficient vectorized operations for batch evaluation - Adaptive complexity through configurable sample count

Parameters:

• n_quantiles (int)

• adapter (Literal['DtACI', 'ACI'] | None)

• current_best_value (float)

• num_ei_samples (int)

update_best_value(value)

Update current best observed value for improvement computation.

This method should be called after each new observation to maintain accurate improvement calculations. The best value serves as the baseline for computing expected improvements in subsequent acquisition decisions.

Parameters:

value (float) – Newly observed objective value to compare with current best. For minimization problems, this updates the minimum observed value.

fetch_alphas()

Retrieve current alpha values for interval construction.

Returns:

List of alpha values (miscoverage rates) for each confidence level, ordered from lowest to highest confidence (decreasing alpha values).

Return type:

List[float]

update_interval_width(betas)

Update interval widths using observed coverage rates.

This method applies adaptive interval width adjustment based on empirical coverage feedback. Each interval’s alpha parameter is updated independently to maintain target coverage while optimizing interval efficiency for accurate expected improvement estimation.

Parameters:

betas (List[float]) – Observed coverage rates for each interval, in the same order as alpha values. Values should be in [0, 1] representing the fraction of true values falling within each interval.

calculate_expected_improvement(predictions_per_interval)

Calculate Expected Improvement for each candidate point using Monte Carlo sampling.

This method estimates the expected improvement acquisition function by Monte Carlo sampling from prediction intervals. For each candidate point, multiple samples are drawn from its prediction intervals, improvements over the current best are computed, and the expectation is estimated as the sample mean.

Methodology: 1. Flatten prediction intervals into efficient matrix representation 2. Generate random samples from intervals for each observation 3. Compute improvements: max(0, current_best - sampled_value) 4. Estimate expected improvement as sample mean 5. Return negated values for minimization compatibility

Parameters:

predictions_per_interval (List[ConformalBounds]) – List of ConformalBounds objects containing lower and upper bounds for each confidence level. All bounds must have the same number of observations.

Returns:

Array of expected improvement values with shape (n_observations,). Values are negated for minimization (higher EI = more negative value). Points with higher expected improvement are more attractive for next evaluation.

Return type:

ndarray