src.environment.optimizer.les_optimizer

Module Contents

Classes

SelfAttn
LrNet
LES_Optimizer	Introduction Learned Evolution Strategy (LES) is a novel self-attention-based evolution strategies parametrization, and discover effective update rules for ES via meta-learning.

Functions

vector2nn

Introduction Maps a flat parameter vector to the parameters of a given neural network, updating the network’s weights in-place.

API

src.environment.optimizer.les_optimizer.vector2nn(x, net, device)

Introduction

Maps a flat parameter vector to the parameters of a given neural network, updating the network’s weights in-place.

Args:

• x (list or numpy.ndarray): A 1D array or list containing all the parameters to be assigned to the network.

• net (torch.nn.Module): The neural network whose parameters will be updated.

• device (torch.device or str): The device on which the parameters should be allocated (e.g., ‘cpu’ or ‘cuda’).

Returns:

• torch.nn.Module: The neural network with updated parameters.

Raises:

• AssertionError: If the length of x does not match the total number of parameters in net.

class src.environment.optimizer.les_optimizer.SelfAttn

Bases: torch.nn.Module

Initialization

Introduction

Initializes the optimizer class and sets up the linear transformation layers for query, key, and value projections.

Built-in Attribute:

• Wq (nn.Linear): Linear layer mapping input of size 3 to output of size 8 for query projection.

• Wk (nn.Linear): Linear layer mapping input of size 3 to output of size 8 for key projection.

• Wv (nn.Linear): Linear layer mapping input of size 3 to output of size 1 for value projection.

Raises:

• None

forward(X)

Introduction

Computes the output of a single-head self-attention mechanism using the input tensor X. Applies learned linear projections to obtain queries, keys, and values, computes attention scores, and returns the attended output.

Args:

• X (torch.Tensor): Input tensor of shape (batch_size, input_dim).

Returns:

• torch.Tensor: The output tensor after applying self-attention and softmax, with the last singleton dimension removed.

Raises:

• None

class src.environment.optimizer.les_optimizer.LrNet

Bases: torch.nn.Module

Initialization

Introduction

Initializes the optimizer neural network with two linear layers and a sigmoid activation function.

Built-in Attribute:

• ln1 (nn.Linear): First linear layer transforming input of size 19 to 8.

• ln2 (nn.Linear): Second linear layer transforming input of size 8 to 2.

• sm (nn.Sigmoid): Sigmoid activation function applied to the output.

Returns:

• None

forward(X)

Introduction

Applies two sequential linear transformations with normalization and a softmax activation to the input tensor.

Args:

• X (torch.Tensor): The input tensor to be processed.

Returns:

• torch.Tensor: The output tensor after normalization, linear transformation, and softmax activation.

Raises:

• None

class src.environment.optimizer.les_optimizer.LES_Optimizer(config)

Bases: src.environment.optimizer.learnable_optimizer.Learnable_Optimizer

Introduction

Learned Evolution Strategy (LES) is a novel self-attention-based evolution strategies parametrization, and discover effective update rules for ES via meta-learning.

Original paper

“Discovering evolution strategies via meta-black-box optimization.” The Eleventh International Conference on Learning Representations. (2023).

Official Implementation

LES

Initialization

Introduction

Initializes the optimizer with the given configuration, setting up neural network components, device, and various hyperparameters for the optimization process.

Args:

• config (object): Config object containing optimizer settings.

  • Attributes needed for the LES_Optimizer are the following:

    • device (str or torch.device): Device on which computations will be performed.Default is ‘cpu’.

    • maxFEs (int): Maximum number of function evaluations allowed.

    • log_interval (int): Interval for logging progress.Default is 100.

    • n_logpoint (int): Number of log points for logging.Default is 50.

Built-in Attribute:

• self.device (torch.device): Device on which computations will be performed.

• self.max_fes (int): Maximum number of function evaluations allowed.

• self.attn (SelfAttn): Self-attention neural network module.

• self.mlp (LrNet): Multi-layer perceptron neural network module.

• self.alpha (list of float): List of alpha time-scale values.

• self.timestamp (np.ndarray): Array of timestamps for tracking progress.

• self.save_time (int): Counter for save operations.Default is 0.

• self.NP (int): Population size or related parameter.Default is 16.

• self.sigma_ratio (float): Ratio used for sigma calculation. Default is 0.2.

• self.fes (int or None): Current function evaluation count. Default is None.

• self.cost (any): Variable for recording cost or loss. Default is None.

• self.log_index (any): Index for logging. Default is None.

• self.log_interval (int): Interval for logging progress.

Returns:

• None

Raises:

• None

__str__()

Introduction

Returns a string representation of the LES_Optimizer object.

Returns:

• str: The string “LES_Optimizer”, representing the class name.

init_population(problem)

Introduction

Initializes the population for the optimizer using a normal distribution based on the problem’s bounds and dimension. Sets up initial evolution information, including parent solutions, their costs, and statistical parameters for the optimization process.

Args:

• problem (object): The optimization problem object, which has attributes ub (upper bounds), lb (lower bounds), dim (problem dimensionality), and a method eval for evaluating a population.

Built-in Attributes:

• self.ub (np.ndarray): Upper bounds for the variables.

• self.lb (np.ndarray): Lower bounds for the variables.

• self.problem (object): The optimization problem instance.

• self.evolution_info (dict): Dictionary to store evolution information, including parents, costs, and generation counter.

• self.cost (list): List to store the best costs found during the optimization process.

• self.log_index (int): Index for logging progress.Default is 1.

• self.fes (int): Total number of function evaluations performed.Default is 0.

• self.meta_X (list, optional): List to store population snapshots for meta data logging.Default is empty.

• self.meta_Cost (list, optional): List to store cost snapshots for meta data logging.Default is empty.

Returns:

• None

Side Effects:

• Sets several instance attributes such as ub, lb, problem, evolution_info, fes, cost, log_index, and optionally meta_X and meta_Cost if full meta data logging is enabled.

Notes:

• The initial population is generated using a normal distribution clipped to the problem’s bounds.

• The method assumes that self.rng is a random number generator and self.sigma_ratio, self.NP, and self.__config.full_meta_data are properly initialized instance attributes.

cal_attn_feature()

Introduction

Computes attention features for the current population in the evolutionary optimization process. The features include the z-score of population costs, shifted normalized ranking, and an improvement indicator, which are concatenated into a single tensor.

Returns:

• torch.FloatTensor: A tensor of shape (N, 3), where N is the population size. Each row contains the z-score of the cost, the shifted normalized rank, and a boolean indicator of improvement for each individual.

Notes:

• The z-score is calculated to standardize the population costs.

• The shifted rank normalizes the ranking of costs and centers it around zero.

• The improvement indicator is a boolean array indicating whether each individual’s cost is better than the global best.

cal_mlp_feature(W)

Introduction

Calculates multi-layer perceptron (MLP) features based on the current evolutionary state, including evolution paths and timestamp embeddings.

Args:

• W (np.ndarray): Weight vector or matrix used to compute weighted sums of evolutionary information.

Returns:

• A torch tensor containing the concatenated feature vector (shape: [dim, 19]).

• Numpy array of updated evolution paths for the mean (c, shape: [3, dim]).

• Numpy array of updated evolution paths for the standard deviation (s, shape: [3, dim]).

Notes:

The function computes updated evolution paths (Pc and Ps) for each alpha value, generates a timestamp embedding, and concatenates these features for use in an MLP. The output tensor is suitable for input into a neural network.

update(action, problem)

Introduction

Updates the optimizer’s internal state by performing one or more evolutionary optimization steps using the provided action and problem. The method adapts model parameters, generates new populations, evaluates them, and logs progress until a stopping criterion is met.

Args:

• action (dict): Dictionary containing new model parameters for attention and MLP networks, and optionally a ‘skip_step’ key to limit the number of steps.

• problem (object): The optimization problem object, which has a dim attribute and an eval method for evaluating populations.

Returns:

• float: The best cost (fitness) found in the current optimization run.

• float: The normalized improvement from the initial to the best cost.

• bool: Whether the stopping criterion was met.

• dict: Additional information (currently empty).

Raises:

• None explicitly, but may raise exceptions if the action or problem objects are malformed or if numerical errors occur during optimization.