Skip to content

gnm.fitting

fitting

Parameter fitting and analysis for generative network models.

This subpackage provides tools for systematically exploring parameter spaces, running experiments with generative network models, and analyzing the results. The module implements functionality for fitting model parameters to observed networks and evaluating how well generated networks match real-world data.

The module includes:

  • Data structures for defining parameter sweeps and storing experiment results
  • Functions for running model simulations and parameter explorations
  • Analysis tools for finding optimal parameter combinations
  • Aggregation methods for summarising results across simulations

These tools enable users to:

  1. Define parameter spaces to explore systematically
  2. Run generative models with different parameter combinations
  3. Evaluate network similarity using various criteria
  4. Identify parameter values that produce the most realistic networks

Dataclasses

gnm.fitting.BinarySweepParameters(eta, gamma, lambdah, distance_relationship_type, preferential_relationship_type, heterochronicity_relationship_type, generative_rule, num_iterations, prob_offset=lambda: [1e-06](), binary_updates_per_iteration=lambda: [1]()) dataclass

Parameter space definition for binary generative network models.

This class defines a multidimensional parameter space to explore for binary network generation. It contains lists of parameter values that will be combined to create different configurations of binary generative models.

When iterated, this class yields all possible combinations of parameters as BinaryGenerativeParameters instances, creating a comprehensive parameter sweep.

Attributes:

Name Type Description
eta Tensor

Parameter values (\(\eta\)) controlling the influence of Euclidean distances \(D_{ij}\) on wiring probability. More negative values indicate lower wiring probabilities between nodes that are further away.
gamma Tensor

Parameter values (\(\gamma\)) controlling the influence of preferential attachment. Higher values increase the influence of the preferential attachment factor.
lambdah Tensor

Parameter values (\(\lambda\)) controlling the influence of heterochronicity or temporal distance between nodes. Affects the probability of connections between nodes with different developmental timing.
distance_relationship_type List[str]

Types of distance-dependent relationships to use (e.g., "powerlaw", "exponential"). Defines the functional form of distance dependence in the model.
preferential_relationship_type List[str]

Types of preferential attachment relationships to use (e.g., "powerlaw", "exponential"). Defines the functional form of degree dependence in the model.
heterochronicity_relationship_type List[str]

Types of heterochronicity relationships to use (e.g., "powerlaw", "exponential"). Defines the functional form of temporal dependence in the model.
generative_rule List[GenerativeRule]

Generative rules to use for network creation. These define the rule by which the preferential attachment factor is computed.
num_iterations List[int]

Numbers of iterations to run the generative process for each parameter set. Controls the number of new connections added to the network.
prob_offset List[float]

Small probability offsets added to avoid numerical issues with zero probabilities. Defaults to [1e-6].
binary_updates_per_iteration List[int]

Number of connection updates to perform in each iteration of the binary network generation process. Don't touch this unless you know what you're doing. Defaults to [1].

Examples:

>>> import torch
>>> from gnm.generative_rules import MatchingIndex
>>> from gnm.fitting import BinarySweepParameters
>>> # Define parameter ranges to explore
>>> eta_values = torch.tensor([-3.0, -2.0, -1.0])
>>> gamma_values = torch.tensor([0.1, 0.2, 0.3])
>>> lambda_values = torch.tensor([0.0])
>>> sweep_params = BinarySweepParameters(
...     eta=eta_values,
...     gamma=gamma_values,
...     lambdah=lambda_values,
...     distance_relationship_type=["powerlaw"],
...     preferential_relationship_type=["powerlaw"],
...     heterochronicity_relationship_type=["powerlaw"],
...     generative_rule=[MatchingIndex()],
...     num_iterations=[100],
... )
>>> # Count total parameter combinations
>>> len(list(sweep_params))
9
See Also

gnm.fitting.WeightedSweepParameters(alpha, optimisation_criterion, weight_lower_bound=lambda: [0.0](), weight_upper_bound=lambda: [float('inf')](), maximise_criterion=lambda: [False](), weight_updates_per_iteration=lambda: [1]()) dataclass

Parameter space definition for weighted generative network models.

This class defines a multidimensional parameter space to explore for the weight optimization phase of weighted network generation. It contains lists of parameter values that will be combined to create different configurations of weighted generative models.

When iterated, this class yields all possible combinations of parameters as WeightedGenerativeParameters instances, creating a comprehensive parameter sweep.

Attributes:

Name Type Description
alpha Tensor

Parameter values (\(\alpha\)) controlling the size of the gradient step during weight optimization. Higher values lead to larger updates to edge weights.
optimisation_criterion List[OptimisationCriterion]

Optimisation criteria to use for weight optimization. These define the objective function on which gradients are taken during the weight optimization process.
weight_lower_bound List[float]

Lower bounds for edge weights. Defaults to [0.0].
weight_upper_bound List[float]

Upper bounds for edge weights. Defaults to [inf].
maximise_criterion List[bool]

Whether to maximise the optimisation criterion. Defaults to [False].
weight_updates_per_iteration List[int]

Number of weight updates to perform in each iteration of the weight optimisation process. Defaults to [1].

Examples:

>>> import torch
>>> from gnm.weight_criteria import Communicability
>>> from gnm.fitting import WeightedSweepParameters
>>> # Define parameter ranges to explore
>>> sweep_params = WeightedSweepParameters(
...     alpha=torch.tensor([0.01, 0.05, 0.1]),
...     optimisation_criterion=[Communicability(omega=1.0)],
...     maximise_criterion=[True, False],
...     weight_updates_per_iteration=[1, 5, 10],
... )
>>> # Count total parameter combinations
>>> len(list(sweep_params))
18
See Also

gnm.fitting.SweepConfig(binary_sweep_parameters, num_simulations=None, seed_adjacency_matrix=None, distance_matrix=None, weighted_sweep_parameters=None, seed_weight_matrix=None, heterochronous_matrix=None) dataclass

Configuration for a comprehensive parameter sweep.

This class defines a complete parameter sweep by combining binary parameter spaces, weighted parameter spaces, and various input matrices. When iterated, it yields RunConfig instances for each unique parameter combination in the sweep.

The sweep includes all combinations of binary parameters, weighted parameters (if provided), and input matrices, creating a thorough exploration of the parameter space.

Attributes:

Name Type Description
binary_sweep_parameters BinarySweepParameters

Parameters for binary network generation. These define the rules and relationships used to generate binary networks.
num_simulations Optional[int]

Number of simulations to run in parallel. Each simulation generates a separate network using the same parameters.
seed_adjacency_matrix Optional[List[Float[Tensor, 'num_simulations num_nodes num_nodes']]]

Seed adjacency matrices for the binary network generation process. If provided, these matrices are used as the starting points for network generation. If unspecified, the networks are generated from scratch.
distance_matrices Optional[List[Float[Tensor, 'num_nodes num_nodes']]]

Distance matrices for the network. These matrices define the spatial relationships between nodes and are used in the generative process. If unspecified, constant distances are used.
weighted_sweep_parameters Optional[WeightedSweepParameters]

Parameters for weight optimisation. If provided, the model will perform a weight optimisation phase after generating the binary network. If unspecified, the model will only generate binary networks.
seed_weight_matrix Optional[List[Float[Tensor, 'num_simulations num_nodes num_nodes']]]

Seed weight matrices for the weight optimisation process. If provided, these matrices are used as the starting points for weight optimisation. If unspecified, the weights are optimised from scratch.
heterochronous_matrix Optional[List[Float[Tensor, 'num_binary_updates num_simulations num_nodes num_nodes']]]

The heterochronous development matrices for each binary update step. Can be provided for each simulation in the batch or as a single matrix to be used across all simulations. Defaults to None, which means that there is no heterochronicity.

Examples:

>>> import torch
>>> from gnm.generative_rules import MatchingIndex
>>> from gnm.weight_criteria import NormalisedCommunicability, WeightedDistance
>>> from gnm.fitting import BinarySweepParameters, WeightedSweepParameters, SweepConfig
>>> from gnm.defaults import get_distance_matrix
>>> # Define binary parameter space
>>> binary_sweep = BinarySweepParameters(
...     eta=torch.tensor([-3.0, -2.0, -1.0]),
...     gamma=torch.tensor([0.2]),
...     lambdah=torch.tensor([0.0]),
...     distance_relationship_type=["powerlaw"],
...     preferential_relationship_type=["powerlaw"],
...     heterochronicity_relationship_type=["powerlaw"],
...     generative_rule=[MatchingIndex()],
...     num_iterations=[100],
... )
>>> # Define weighted parameter space
>>> weighted_sweep = WeightedSweepParameters(
...     alpha=torch.tensor([0.01, 0.02, 0.03, 0.04, 0.05]),
...     optimisation_criterion=[NormalisedCommunicability(), WeightedDistance()],
... )
>>> # Create sweep configuration
>>> sweep_config = SweepConfig(
...     binary_sweep_parameters=binary_sweep,
...     weighted_sweep_parameters=weighted_sweep,
...     num_simulations=10,
...     distance_matrices=[get_distance_matrix()],
... )
>>> # Count total run configurations
>>> len(list(sweep_config))
30
See Also

gnm.fitting.Experiment(run_config, evaluation_results, model=None, run_history=None) dataclass

Complete record of a generative network model experiment.

This class encapsulates the entire experiment, including the configuration used, the results of evaluations, the model instance, and the history of network evolution. It provides a comprehensive record that can be saved, loaded, and analysed.

The to_device method allows moving all tensors in the experiment to a specified device, which is useful for efficient computation or visualization.

Attributes:

Name Type Description
run_config RunConfig

Configuration for the experiment, including parameters and input matrices.
evaluation_results EvaluationResults

Results of evaluating the generated networks against real networks.
model Optional[GenerativeNetworkModel]

Instance of the generative network model used in the experiment. If the model was not saved, this field is None.
run_history Optional[RunHistory]

History of network evolution during the model run. If the history was not saved, this field is None.

Examples:

>>> from gnm.fitting import RunConfig, EvaluationResults, Experiment
>>> from gnm import GenerativeNetworkModel, BinaryGenerativeParameters
>>> from gnm.generative_rules import MatchingIndex
>>> # Create minimal example (without actual data)
>>> config = RunConfig(
...     binary_parameters=BinaryGenerativeParameters(
...         eta=-2.0,
...         gamma=0.3,
...         lambdah=0.0,
...         distance_relationship_type="powerlaw",
...         preferential_relationship_type="powerlaw",
...         heterochronicity_relationship_type="powerlaw",
...         generative_rule=MatchingIndex(),
...         num_iterations=100,
...     )
... )
>>> results = EvaluationResults(
...     binary_evaluations={},
...     weighted_evaluations={},
... )
>>> experiment = Experiment(
...     run_config=config,
...     evaluation_results=results,
... )
See Also

to_device(device)

Move all tensors in the experiment, including the model, to a specified device.

Parameters:

Name Type Description Default
device Union[device, str]

The device to move all tensors to.

 required
Source code in src/gnm/fitting/experiment_dataclasses.py 
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
def to_device(self, device: Union[torch.device, str]):
    r"""Move all tensors in the experiment, including the model, to a specified device.

    Args:
        device: The device to move all tensors to.
    """
    self.evaluation_results.to_device(device)
    self.run_config.to_device(device)
    if self.model is not None:
        self.model.to_device(device)
    if self.run_history is not None:
        self.run_history.to_device(device)

    gc.collect()
    torch.cuda.empty_cache()

gnm.fitting.RunConfig(binary_parameters, num_simulations=None, seed_adjacency_matrix=None, distance_matrix=None, weighted_parameters=None, seed_weight_matrix=None, heterochronous_matrix=None) dataclass

Configuration for a single generative network model run.

This class encapsulates all the parameters and inputs needed for a single run of the generative network model. It contains both binary and (optionally) weighted parameters, as well as input matrices like distance matrices and seed networks.

Attributes:

Name Type Description
binary_parameters BinaryGenerativeParameters

Parameters for binary network generation. These define the rules and relationships used to generate binary networks.
num_simulations int

Number of simulations to run in parallel. Each simulation generates a separate network using the same parameters.
seed_adjacency_matrix Optional[Float[Tensor, 'num_simulations num_nodes num_nodes']]

Seed adjacency matrix for the binary network generation process. If provided, this matrix is used as the starting point for network generation. If unspecified, the network is generated from scratch.
distance_matrix Optional[Float[Tensor, 'num_nodes num_nodes']]

Distance matrix for the network. This matrix defines the spatial relationships between nodes and is used in the generative process. If unspecified, constant distances are used.
weighted_parameters Optional[WeightedGenerativeParameters]

Parameters for weight optimization. If provided, the model will perform a weight optimization phase after generating the binary network. If unspecified, the model will only generate binary networks.
seed_weight_matrix Optional[Float[Tensor, 'num_simulations num_nodes num_nodes']]

Seed weight matrix for the weight optimization process. If provided, this matrix is used as the starting point for weight optimization. If unspecified, the weights are optimised from scratch.
heterochronous_matrix Optional[Float[Tensor, 'num_binary_updates num_nodes num_nodes']]

The heterochronous development matrix for each binary update step. Can be provided for each simulation in the batch or as a single matrix to be used across all simulations. Defaults to None, which means that there is no heterochronicity.

Examples:

>>> from gnm import BinaryGenerativeParameters
>>> from gnm.generative_rules import ClusteringMin
>>> from gnm.fitting import RunConfig
>>> from gnm.defaults import get_distance_matrix
>>> # Create binary parameters
>>> binary_params = BinaryGenerativeParameters(
...     eta=-2.0,
...     gamma=0.3,
...     lambdah=0.0,
...     distance_relationship_type="powerlaw",
...     preferential_relationship_type="powerlaw",
...     heterochronicity_relationship_type="powerlaw",
...     generative_rule=ClusteringMin(),
...     num_iterations=200,
... )
>>> # Create run configuration
>>> config = RunConfig(
...     binary_parameters=binary_params,
...     num_simulations=100,
...     distance_matrix=get_distance_matrix(),
... )
See Also

gnm.fitting.RunHistory(added_edges, adjacency_snapshots, weight_snapshots) dataclass

Record of network evolution during a model run.

This class stores the history of how networks evolved during the generative process. It records which edges were added at each step and maintains snapshots of the adjacency and weight matrices at regular intervals.

This history can be used to visualize network growth, analyse the order in which connections formed, and track how weights evolved over time.

Attributes:

Name Type Description
added_edges Int[torch.Tensor, "num_binary_updates num_simulations 2]

Tensor containing the edges added at each binary update step. Each row corresponds to a single update, with columns [source, target] indicating the nodes that were connected in that step.
adjacency_snapshots Float[Tensor, 'num_binary_updates num_simulations num_nodes num_nodes']

Tensor containing snapshots of the adjacency matrix at each binary update step.
weight_snapshots Optional[Float[Tensor, 'num_weight_updates num_simulations num_nodes num_nodes']]

Tensor containing snapshots of the weight matrix at each weight update step. If the model did not perform weight optimisation, this tensor is None.

Examples:

>>> from gnm import BinaryGenerativeParameters, WeightedGenerativeParameters, GenerativeNetworkModel
>>> from gnm.defaults import get_distance_matrix
>>> from gnm.generative_rules import Neighbours
>>> from gnm.weight_criteria import WeightedDistance
>>> binary_parameters = BinaryGenerativeParameters(
...     eta=1.0,
...     gamma=-0.5,
...     lambdah=1.0,
...     distance_relationship_type='exponential',
...     preferential_relationship_type='powerlaw',
...     heterochronicity_relationship_type='powerlaw',
...     generative_rule=Neighbours(),
...     num_iterations=250,
...     binary_updates_per_iteration=1,
... )
>>> weighted_parameters = WeightedGenerativeParameters(
...     alpha=0.003,
...     optimisation_criterion=WeightedDistance(),
...     weighted_updates_per_iteration=200,
... )
... distance_matrix = get_distance_matrix()
>>> model = GenerativeNetworkModel(
...     binary_parameters=binary_parameters,
...     num_simulations=100, # Run 100 networks in parallel
...     distance_matrix=distance_matrix,
...     weighted_parameters=weighted_parameters,
... )
>>> added_edges, adjacency_snapshots, weight_snapshots = model.run_model()
>>> history = RunHistory(
...     added_edges=added_edges,
...     adjacency_snapshots=adjacency_snapshots,
...     weight_snapshots=weight_snapshots,
... )
See Also

to_device(device)

Moves the run history to a specified device.

Parameters:

Name Type Description Default
device Union[device, str]

The device to move all tensors to.

 required
Source code in src/gnm/fitting/experiment_dataclasses.py 
665
666
667
668
669
670
671
672
673
674
675
676
def to_device(self, device: Union[torch.device, str]):
    r"""Moves the run history to a specified device.

    Args:
        device: The device to move all tensors to.
    """
    if self.added_edges is not None:
        self.added_edges = self.added_edges.to(device)
    if self.adjacency_snapshots is not None:
        self.adjacency_snapshots = self.adjacency_snapshots.to(device)
    if self.weight_snapshots is not None:
        self.weight_snapshots = self.weight_snapshots.to(device)

gnm.fitting.EvaluationResults(binary_evaluations, weighted_evaluations) dataclass

Storage for network evaluation results.

This class stores the results of evaluating generated networks against real networks using various evaluation criteria. It contains separate dictionaries for binary and weighted evaluation results, where each entry maps a criterion name to a tensor of evaluation scores.

Each evaluation tensor has shape [num_real_networks, num_simulations], containing the evaluation score for each combination of real network and simulated network.

Attributes:

Name Type Description
binary_evaluations Dict[str, Float[Tensor, 'num_real_binary_networks num_simulations']]

Dictionary of binary evaluation results. Each entry maps a criterion name to a tensor of evaluation scores for binary networks.
weighted_evaluations Dict[str, Float[Tensor, 'num_real_weighted_networks num_simulations']]

Dictionary of weighted evaluation results. Each entry maps a criterion name to a tensor of evaluation scores for weighted networks.

Examples:

>>> import torch
>>> from gnm.fitting import EvaluationResults
>>> from gnm.evaluation import DegreeKS
>>> from gnm.defaults import get_binary_network
>>> from gnm.utils import get_control
>>> # Create evaluation results
>>> real_matrices = get_binary_network()
>>> control_matrices = get_control(real_matrices)
>>> degree_ks_eval = DegreeKS()
>>> binary_evaluations = {str(degree_ks_eval): degree_ks_eval(real_matrices, control_matrices)}
>>> results = EvaluationResults(
...     binary_evaluations=binary_evaluations,
...     weighted_evaluations={},
... )
>>> # Get evaluation scores for a specific criterion
>>> results.binary_evaluations[str(degree_ks_eval)]
See Also

to_device(device)

Moves the evalution results to a specified device.

Parameters:

Name Type Description Default
device Union[device, str]

The device to move all tensors to.

 required
Source code in src/gnm/fitting/experiment_dataclasses.py 
580
581
582
583
584
585
586
587
588
589
def to_device(self, device: Union[torch.device, str]):
    r"""Moves the evalution results to a specified device.

    Args:
        device: The device to move all tensors to.
    """
    for key, value in self.binary_evaluations.items():
        self.binary_evaluations[key] = value.to(device)
    for key, value in self.weighted_evaluations.items():
        self.weighted_evaluations[key] = value.to(device)

Performing sweeps and evaluations

gnm.fitting.perform_run(run_config, binary_evaluations=None, weighted_evaluations=None, real_binary_matrices=None, real_weighted_matrices=None, save_model=True, save_run_history=True, device=None)

Perform a single run of the generative network model.

This function executes a generative network model with the specified configuration, creates synthetic networks, and evaluates them against real networks using provided evaluation criteria. It returns an Experiment object containing the results.

Parameters:

Name Type Description Default
run_config RunConfig

Configuration for the run, specifying parameters and input matrices.

 required
binary_evaluations Optional[List[Union[BinaryEvaluationCriterion, CompositeCriterion]]]

List of criteria for evaluating binary network properties. Defaults to None (no binary evaluation).

 None
weighted_evaluations Optional[List[Union[WeightedEvaluationCriterion, CompositeCriterion]]]

List of criteria for evaluating weighted network properties. Defaults to None (no weighted evaluation).

 None
real_binary_matrices Optional[Float[Tensor, 'num_real_binary_networks num_nodes num_nodes']]

Real binary networks to compare synthetic networks against. Required if binary_evaluations is provided.

 None
real_weighted_matrices Optional[Float[Tensor, 'num_real_weighted_networks num_nodes num_nodes']]

Real weighted networks to compare synthetic networks against. Required if weighted_evaluations is provided.

 None
save_model bool

If True, saves the model in the experiment. Set this argument to False to save on memory. Defaults to True.

 True
save_run_history bool

If True, saves the adjacency and weight snapshots in the run history. Set this argument to False to save on memory. Defaults to True.

 True
device Optional[Union[device, str]]

Device to run the model on. If unspecified, uses CUDA if available, else CPU.

 None

Returns:

Type Description
Experiment

An Experiment object containing the run configuration, evaluation results,
Experiment

and optionally the model and run history.

Examples:

>>> from gnm import BinaryGenerativeParameters
>>> from gnm.generative_rules import MatchingIndex
>>> from gnm.fitting import RunConfig, perform_run
>>> from gnm.evaluation import ClusteringKS
>>> from gnm.defaults import get_binary_network, get_distance_matrix
>>> # Create run configuration
>>> binary_params = BinaryGenerativeParameters(
...     eta=-2.0,
...     gamma=0.3,
...     lambdah=0.0,
...     distance_relationship_type="powerlaw",
...     preferential_relationship_type="powerlaw",
...     heterochronicity_relationship_type="powerlaw",
...     generative_rule=MatchingIndex(),
...     num_iterations=100,
... )
>>> config = RunConfig(
...     binary_parameters=binary_params,
...     num_simulations=5,
...     distance_matrix=get_distance_matrix(),
... )
>>> # Define evaluation
>>> binary_evals = [ClusteringKS()]
>>> real_networks = get_binary_network()
>>> # Run the model
>>> experiment = perform_run(
...     run_config=config,
...     binary_evaluations=binary_evals,
...     real_binary_matrices=real_networks,
... )
See Also
Source code in src/gnm/fitting/sweep.py 
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
@jaxtyped(typechecker=typechecked)
def perform_run(
    run_config: RunConfig,
    binary_evaluations: Optional[
        List[Union[BinaryEvaluationCriterion, CompositeCriterion]]
    ] = None,
    weighted_evaluations: Optional[
        List[
            Union[
                WeightedEvaluationCriterion,
                CompositeCriterion,
            ]
        ]
    ] = None,
    real_binary_matrices: Optional[
        Float[torch.Tensor, "num_real_binary_networks num_nodes num_nodes"]
    ] = None,
    real_weighted_matrices: Optional[
        Float[torch.Tensor, "num_real_weighted_networks num_nodes num_nodes"]
    ] = None,
    save_model: bool = True,
    save_run_history: bool = True,
    device: Optional[Union[torch.device, str]] = None,
) -> Experiment:
    r"""Perform a single run of the generative network model.

    This function executes a generative network model with the specified configuration,
    creates synthetic networks, and evaluates them against real networks using provided
    evaluation criteria. It returns an Experiment object containing the results.

    Args:
        run_config:
            Configuration for the run, specifying parameters and input matrices.
        binary_evaluations:
            List of criteria for evaluating binary network properties.
            Defaults to None (no binary evaluation).
        weighted_evaluations:
            List of criteria for evaluating weighted network properties.
            Defaults to None (no weighted evaluation).
        real_binary_matrices:
            Real binary networks to compare synthetic networks against.
            Required if binary_evaluations is provided.
        real_weighted_matrices:
            Real weighted networks to compare synthetic networks against.
            Required if weighted_evaluations is provided.
        save_model:
            If True, saves the model in the experiment. Set this argument to False to save on memory.
            Defaults to True.
        save_run_history:
            If True, saves the adjacency and weight snapshots in the run history.
            Set this argument to False to save on memory. Defaults to True.
        device:
            Device to run the model on. If unspecified, uses CUDA if available, else CPU.

    Returns:
        An Experiment object containing the run configuration, evaluation results,
        and optionally the model and run history.

    Examples:
        >>> from gnm import BinaryGenerativeParameters
        >>> from gnm.generative_rules import MatchingIndex
        >>> from gnm.fitting import RunConfig, perform_run
        >>> from gnm.evaluation import ClusteringKS
        >>> from gnm.defaults import get_binary_network, get_distance_matrix
        >>> # Create run configuration
        >>> binary_params = BinaryGenerativeParameters(
        ...     eta=-2.0,
        ...     gamma=0.3,
        ...     lambdah=0.0,
        ...     distance_relationship_type="powerlaw",
        ...     preferential_relationship_type="powerlaw",
        ...     heterochronicity_relationship_type="powerlaw",
        ...     generative_rule=MatchingIndex(),
        ...     num_iterations=100,
        ... )
        >>> config = RunConfig(
        ...     binary_parameters=binary_params,
        ...     num_simulations=5,
        ...     distance_matrix=get_distance_matrix(),
        ... )
        >>> # Define evaluation
        >>> binary_evals = [ClusteringKS()]
        >>> real_networks = get_binary_network()
        >>> # Run the model
        >>> experiment = perform_run(
        ...     run_config=config,
        ...     binary_evaluations=binary_evals,
        ...     real_binary_matrices=real_networks,
        ... )

    See Also:
        - [`fitting.RunConfig`][gnm.fitting.RunConfig]: Configuration for a single run
        - [`fitting.Experiment`][gnm.fitting.Experiment]: Result container for experiments
        - [`fitting.perform_sweep`][gnm.fitting.perform_sweep]: Function for running multiple parameter combinations
        - [`GenerativeNetworkModel`][gnm.GenerativeNetworkModel]: The network model being executed
    """

    model = GenerativeNetworkModel(
        binary_parameters=run_config.binary_parameters,
        num_simulations=run_config.num_simulations,
        seed_adjacency_matrix=run_config.seed_adjacency_matrix,
        distance_matrix=run_config.distance_matrix,
        weighted_parameters=run_config.weighted_parameters,
        seed_weight_matrix=run_config.seed_weight_matrix,
        device=device,
        verbose=False,
    )

    added_edges, adjacency_snapshots, weight_snapshots = model.run_model(
        heterochronous_matrix=run_config.heterochronous_matrix
    )

    run_history = RunHistory(
        added_edges=added_edges,
        adjacency_snapshots=adjacency_snapshots,
        weight_snapshots=weight_snapshots,
    )

    evaluation_results = perform_evaluations(
        model=model,
        binary_evaluations=binary_evaluations,
        weighted_evaluations=weighted_evaluations,
        real_binary_matrices=real_binary_matrices,
        real_weighted_matrices=real_weighted_matrices,
        device=device,
    )

    experiment = Experiment(
        run_config=run_config,
        model=model if save_model else None,
        run_history=run_history if save_run_history else None,
        evaluation_results=evaluation_results,
    )

    experiment.to_device("cpu")

    gc.collect()
    torch.cuda.empty_cache()

    return experiment

gnm.fitting.perform_sweep(sweep_config, binary_evaluations=None, weighted_evaluations=None, real_binary_matrices=None, real_weighted_matrices=None, save_model=True, save_run_history=True, device=None, verbose=False, wandb_logging=False, method='grid', num_bayesian_runs=30, metric_to_optimise=None)

Perform a parameter sweep over multiple model configurations.

This function systematically explores a parameter space by running the generative network model with different parameter combinations. It generates and evaluates synthetic networks for each configuration, returning a list of experiments.

Parameters:

Name Type Description Default
sweep_config SweepConfig

Configuration for the parameter sweep, defining the parameter space to explore.

 required
binary_evaluations Optional[List[Union[BinaryEvaluationCriterion, CompositeCriterion]]]

List of criteria for evaluating binary network properties. Defaults to None (no binary evaluation).

 None
weighted_evaluations Optional[List[Union[WeightedEvaluationCriterion, CompositeCriterion]]]

List of criteria for evaluating weighted network properties. Defaults to None (no weighted evaluation).

 None
real_binary_matrices Optional[Float[Tensor, 'num_real_binary_networks num_nodes num_nodes']]

Real binary networks to compare synthetic networks against. Required if binary_evaluations is provided.

 None
real_weighted_matrices Optional[Float[Tensor, 'num_real_weighted_networks num_nodes num_nodes']]

Real weighted networks to compare synthetic networks against. Required if weighted_evaluations is provided.

 None
save_model bool

If True, saves the model in the experiment. Set this argument to False to save on memory. Defaults to True.

 True
save_run_history bool

If True, saves the adjacency and weight snapshots in the run history. Set this argument to False to save on memory. Defaults to True.

 True
device Optional[Union[device, str]]

Device to run the models on. If None, uses CUDA if available, else CPU.

 None
verbose Optional[bool]

If True, displays a progress bar for the sweep. Defaults to False.

 False
wandb_logging Optional[bool]

If True, logs the experiment to Weights & Biases. Defaults to False. May reqire a login.

 False
method Literal['bayesian', 'grid']

The method to use for the sweep. Options are 'bayesian' or 'grid'. Defaults to 'grid'. - 'bayesian': Uses the Bayesian optimisation method within Weights and Biases to explore the parameter space. - 'grid': Performs a grid search over the parameter space.

 'grid'
num_bayesian_runs Optional[int]

The number of runs to perform for the Bayesian sweep. Defaults to 30. This is only used if method is 'bayesian'.

 30
metric_to_optimise Optional[Union[str, EvaluationCriterion]]

Which evaluation metric to optimise for the Bayesian sweep. This is only used if method is 'bayesian'. If unspecified, the first binary evaluation will be used if available, otherwise the first weighted evaluation will be used.

 None

Returns:

Type Description
List[Experiment]

A list of Experiment objects, one for each parameter combination in the sweep.

Examples:

>>> import torch
>>> from gnm.generative_rules import MatchingIndex
>>> from gnm.fitting import BinarySweepParameters, SweepConfig, perform_sweep
>>> from gnm.evaluation import ClusteringKS
>>> from gnm.defaults import get_binary_network, get_distance_matrix
>>> # Define parameter space
>>> binary_sweep = BinarySweepParameters(
...     eta=torch.tensor([-3.0, -2.0, -1.0]),
...     gamma=torch.tensor([0.2, 0.3]),
...     lambdah=torch.tensor([0.0]),
...     distance_relationship_type=["powerlaw"],
...     preferential_relationship_type=["powerlaw"],
...     heterochronicity_relationship_type=["powerlaw"],
...     generative_rule=[MatchingIndex()],
...     num_iterations=[100],
... )
>>> # Create sweep configuration
>>> sweep_config = SweepConfig(
...     binary_sweep_parameters=binary_sweep,
...     num_simulations=50,
...     distance_matrices=[get_distance_matrix()],
... )
>>> # Define evaluation
>>> binary_evals = [ClusteringKS()]
>>> real_networks = get_binary_network()
>>> # Run the sweep
>>> experiments = perform_sweep(
...     sweep_config=sweep_config,
...     binary_evaluations=binary_evals,
...     real_binary_matrices=real_networks,
...     save_only_evaluations=True,  # Save memory during sweep
... )
>>> len(experiments)
6
See Also
Source code in src/gnm/fitting/sweep.py 
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
@jaxtyped(typechecker=typechecked)
def perform_sweep(
    sweep_config: SweepConfig,
    binary_evaluations: Optional[
        List[Union[BinaryEvaluationCriterion, CompositeCriterion]]
    ] = None,
    weighted_evaluations: Optional[
        List[
            Union[
                WeightedEvaluationCriterion,
                CompositeCriterion,
            ]
        ]
    ] = None,
    real_binary_matrices: Optional[
        Float[torch.Tensor, "num_real_binary_networks num_nodes num_nodes"]
    ] = None,
    real_weighted_matrices: Optional[
        Float[torch.Tensor, "num_real_weighted_networks num_nodes num_nodes"]
    ] = None,
    save_model: bool = True,
    save_run_history: bool = True,
    device: Optional[Union[torch.device, str]] = None,
    verbose: Optional[bool] = False,
    wandb_logging: Optional[bool] = False,
    method: Literal["bayesian", "grid"] = "grid",
    num_bayesian_runs: Optional[int] = 30,
    metric_to_optimise: Optional[Union[str, EvaluationCriterion]] = None,
) -> List[Experiment]:
    r"""Perform a parameter sweep over multiple model configurations.

    This function systematically explores a parameter space by running the generative
    network model with different parameter combinations. It generates and evaluates
    synthetic networks for each configuration, returning a list of experiments.

    Args:
        sweep_config:
            Configuration for the parameter sweep, defining the parameter space to explore.

        binary_evaluations:
            List of criteria for evaluating binary network properties.
            Defaults to None (no binary evaluation).

        weighted_evaluations:
            List of criteria for evaluating weighted network properties.
            Defaults to None (no weighted evaluation).

        real_binary_matrices:
            Real binary networks to compare synthetic networks against.
            Required if binary_evaluations is provided.

        real_weighted_matrices:
            Real weighted networks to compare synthetic networks against.
            Required if weighted_evaluations is provided.

        save_model:
            If True, saves the model in the experiment. Set this argument to False to save on memory.
            Defaults to True.

        save_run_history:
            If True, saves the adjacency and weight snapshots in the run history.
            Set this argument to False to save on memory. Defaults to True.

        device:
            Device to run the models on. If None, uses CUDA if available, else CPU.

        verbose:
            If True, displays a progress bar for the sweep. Defaults to False.

        wandb_logging:
            If True, logs the experiment to Weights & Biases. Defaults to False. May reqire a login.

        method:
            The method to use for the sweep. Options are 'bayesian' or 'grid'.
            Defaults to 'grid'.
            - 'bayesian': Uses the Bayesian optimisation method within Weights and Biases to explore the parameter space.
            - 'grid': Performs a grid search over the parameter space.

        num_bayesian_runs:
            The number of runs to perform for the Bayesian sweep. Defaults to 30.
            This is only used if method is 'bayesian'.

        metric_to_optimise:
            Which evaluation metric to optimise for the Bayesian sweep.
            This is only used if method is 'bayesian'.
            If unspecified, the first binary evaluation will be used if available,
            otherwise the first weighted evaluation will be used.

    Returns:
        A list of Experiment objects, one for each parameter combination in the sweep.

    Examples:
        >>> import torch
        >>> from gnm.generative_rules import MatchingIndex
        >>> from gnm.fitting import BinarySweepParameters, SweepConfig, perform_sweep
        >>> from gnm.evaluation import ClusteringKS
        >>> from gnm.defaults import get_binary_network, get_distance_matrix
        >>> # Define parameter space
        >>> binary_sweep = BinarySweepParameters(
        ...     eta=torch.tensor([-3.0, -2.0, -1.0]),
        ...     gamma=torch.tensor([0.2, 0.3]),
        ...     lambdah=torch.tensor([0.0]),
        ...     distance_relationship_type=["powerlaw"],
        ...     preferential_relationship_type=["powerlaw"],
        ...     heterochronicity_relationship_type=["powerlaw"],
        ...     generative_rule=[MatchingIndex()],
        ...     num_iterations=[100],
        ... )
        >>> # Create sweep configuration
        >>> sweep_config = SweepConfig(
        ...     binary_sweep_parameters=binary_sweep,
        ...     num_simulations=50,
        ...     distance_matrices=[get_distance_matrix()],
        ... )
        >>> # Define evaluation
        >>> binary_evals = [ClusteringKS()]
        >>> real_networks = get_binary_network()
        >>> # Run the sweep
        >>> experiments = perform_sweep(
        ...     sweep_config=sweep_config,
        ...     binary_evaluations=binary_evals,
        ...     real_binary_matrices=real_networks,
        ...     save_only_evaluations=True,  # Save memory during sweep
        ... )
        >>> len(experiments)
        6

    See Also:
        - [`fitting.SweepConfig`][gnm.fitting.SweepConfig]: Configuration for parameter sweeps
        - [`fitting.perform_run`][gnm.fitting.perform_run]: Function for running a single configuration
        - [`fitting.optimise_evaluation`][gnm.fitting.optimise_evaluation]: Function for finding optimal parameters
    """

    def perform_grid_sweep():
        run_results = []
        run_times = []
        for run_config in tqdm(
            sweep_config,
            desc="Configuration Iterations",
            total=config_count,
            disable=not verbose,
        ):
            start_time = time.perf_counter()
            experiment = perform_run(
                run_config=run_config,
                binary_evaluations=binary_evaluations,
                weighted_evaluations=weighted_evaluations,
                real_binary_matrices=real_binary_matrices,
                real_weighted_matrices=real_weighted_matrices,
                save_model=save_model,
                save_run_history=save_run_history,
                device=device,
            )

            end_time = time.perf_counter()
            run_time = end_time - start_time
            run_times.append(run_time)

            run_results.append(experiment)

            if wandb_logging:
                exp = ExperimentEvaluation(save=False)
                experiment_data_config = exp._save_experiment(experiment)
                wandb.init(project=project_name, config=experiment_data_config)
                wandb.log(experiment_data_config)
                wandb.finish()

            gc.collect()
            torch.cuda.empty_cache()

        return run_results, run_times

    def wandb_agent_single_run():
        with wandb.init() as run:
            config = wandb.config

            # Insert config values into run config
            run_config.eta = torch.Tensor([config.eta])
            run_config.gamma = torch.Tensor([config.gamma])

            experiment = perform_run(
                run_config=run_config,
                binary_evaluations=binary_evaluations,
                weighted_evaluations=weighted_evaluations,
                real_binary_matrices=real_binary_matrices,
                real_weighted_matrices=real_weighted_matrices,
                save_model=save_model,
                save_run_history=save_run_history,
                device=device,
            )

            run_results.append(experiment)

            # eval_binary_score = experiment.evaluation_results.binary_evaluations
            # eval_weighted_score = experiment.evaluation_results.weighted_evaluations

            # wandb.log({
            #     metric_to_optimise: eval_binary_score,
            # })

            experiment_data_config = exp._save_experiment(experiment)
            wandb.log(experiment_data_config)

    def perform_bayesian_sweep(sweep_config_dict):
        sweep_id = wandb.sweep(sweep=sweep_config_dict, project=project_name)

        start_time = time.perf_counter()
        wandb.agent(
            sweep_id,
            function=wandb_agent_single_run,
            project=project_name,
            count=num_bayesian_runs,
        )
        end_time = time.perf_counter()

        run_time = [end_time - start_time]

        api = wandb.Api()
        sweep = api.sweep(f"{project_name}/{sweep_id}")

        return run_time

    print(f"Using device: {device} for GNM simulations")
    if wandb_logging:
        # for experiment logging if wandb is used - ignore if not
        exp = ExperimentEvaluation(save=False)
        print("Logging experiment to wandb - login may be required.")
        wandb.login()
        project_name = input("Enter wandb project name: ")

    if method == "bayesian":
        if metric_to_optimise is None:
            if len(binary_evaluations) != 0:
                metric_to_optimise = binary_evaluations[0]
            elif len(weighted_evaluations) != 0:
                metric_to_optimise = weighted_evaluations[0]
            else:
                raise ValueError(
                    "No evaluation criteria provided for Bayesian optimisation."
                )

        if isinstance(metric_to_optimise, EvaluationCriterion):
            metric_to_optimise = str(metric_to_optimise)

        run_results = []
        eta_min = float(sweep_config.binary_sweep_parameters.eta.min().item())
        eta_max = float(sweep_config.binary_sweep_parameters.eta.max().item())

        gamma_min = float(sweep_config.binary_sweep_parameters.gamma.min().item())
        gamma_max = float(sweep_config.binary_sweep_parameters.gamma.max().item())

        binary_sweep_configuration = {
            "name": project_name,
            "method": "bayes",
            "metric": {"goal": "minimize", "name": metric_to_optimise},
            "parameters": {
                "eta": {"min": eta_min, "max": eta_max},
                "gamma": {"min": gamma_min, "max": gamma_max},
            },
        }

        run_config = next(iter(sweep_config))
        run_times = perform_bayesian_sweep(binary_sweep_configuration)

    else:
        config_count = len(list(sweep_config))
        run_results, run_times = perform_grid_sweep()

    if verbose:
        avg_time = sum(run_times) / len(run_times)
        print(f"Average time per run: {avg_time:.2f} seconds")
        print(f"Total time for sweep: {sum(run_times):.2f} seconds")

    return run_results

gnm.fitting.perform_evaluations(model, binary_evaluations=None, weighted_evaluations=None, real_binary_matrices=None, real_weighted_matrices=None, device=None)

Evaluate synthetic networks against real networks using various criteria.

This function compares networks generated by a model against real networks using the specified evaluation criteria. It performs both binary and weighted evaluations if the corresponding parameters are provided.

Parameters:

Name Type Description Default
model GenerativeNetworkModel

The generative network model containing the synthetic networks to evaluate.

 required
binary_evaluations Optional[List[Union[BinaryEvaluationCriterion, CompositeCriterion]]]

List of criteria for evaluating binary network properties. Defaults to None (no binary evaluation).

 None
weighted_evaluations Optional[List[Union[WeightedEvaluationCriterion, CompositeCriterion]]]

List of criteria for evaluating weighted network properties. Defaults to None (no weighted evaluation).

 None
real_binary_matrices Optional[Float[Tensor, 'num_real_binary_networks num_nodes num_nodes']]

Real binary networks to compare synthetic networks against. Required if binary_evaluations is provided.

 None
real_weighted_matrices Optional[Float[Tensor, 'num_real_weighted_networks num_nodes num_nodes']]

Real weighted networks to compare synthetic networks against. Required if weighted_evaluations is provided.

 None
device Optional[Union[device, str]]

Device to perform the evaluations on. If None, uses CUDA if available, else CPU.

 None

Returns:

Type Description
EvaluationResults

An EvaluationResults object containing the results of all evaluations.

Examples:

>>> from gnm import GenerativeNetworkModel, BinaryGenerativeParameters
>>> from gnm.generative_rules import MatchingIndex
>>> from gnm.fitting import perform_evaluations
>>> from gnm.evaluation import ClusteringKS, DegreeKS
>>> from gnm.defaults import get_binary_network, get_distance_matrix
>>> # Create and run a model
>>> binary_params = BinaryGenerativeParameters(
...     eta=-2.0,
...     gamma=0.3,
...     lambdah=0.0,
...     distance_relationship_type="powerlaw",
...     preferential_relationship_type="powerlaw",
...     heterochronicity_relationship_type="powerlaw",
...     generative_rule=MatchingIndex(),
...     num_iterations=100,
... )
>>> model = GenerativeNetworkModel(
...     binary_parameters=binary_params,
...     num_simulations=15,
...     distance_matrix=get_distance_matrix(),
... )
>>> _, _, _ = model.run_model()
>>> # Define evaluations
>>> binary_evals = [ClusteringKS(), DegreeKS()]
>>> real_networks = get_binary_network()
>>> # Perform evaluations
>>> eval_results = perform_evaluations(
...     model=model,
...     binary_evaluations=binary_evals,
...     real_binary_matrices=real_networks,
... )
>>> # Access results
>>> clustering_scores = eval_results.binary_evaluations["ClusteringKS"]
>>> degree_scores = eval_results.binary_evaluations["DegreeKS"]
See Also
Source code in src/gnm/fitting/sweep.py 
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
@jaxtyped(typechecker=typechecked)
def perform_evaluations(
    model: GenerativeNetworkModel,
    binary_evaluations: Optional[
        List[Union[BinaryEvaluationCriterion, CompositeCriterion]]
    ] = None,
    weighted_evaluations: Optional[
        List[
            Union[
                WeightedEvaluationCriterion,
                CompositeCriterion,
            ]
        ]
    ] = None,
    real_binary_matrices: Optional[
        Float[torch.Tensor, "num_real_binary_networks num_nodes num_nodes"]
    ] = None,
    real_weighted_matrices: Optional[
        Float[torch.Tensor, "num_real_weighted_networks num_nodes num_nodes"]
    ] = None,
    device: Optional[Union[torch.device, str]] = None,
) -> EvaluationResults:
    r"""Evaluate synthetic networks against real networks using various criteria.

    This function compares networks generated by a model against real networks using
    the specified evaluation criteria. It performs both binary and weighted evaluations
    if the corresponding parameters are provided.

    Args:
        model:
            The generative network model containing the synthetic networks to evaluate.

        binary_evaluations:
            List of criteria for evaluating binary network properties.
            Defaults to None (no binary evaluation).
        weighted_evaluations:
            List of criteria for evaluating weighted network properties.
            Defaults to None (no weighted evaluation).
        real_binary_matrices:
            Real binary networks to compare synthetic networks against.
            Required if binary_evaluations is provided.
        real_weighted_matrices:
            Real weighted networks to compare synthetic networks against.
            Required if weighted_evaluations is provided.
        device:
            Device to perform the evaluations on. If None, uses CUDA if available, else CPU.

    Returns:
        An EvaluationResults object containing the results of all evaluations.

    Examples:
        >>> from gnm import GenerativeNetworkModel, BinaryGenerativeParameters
        >>> from gnm.generative_rules import MatchingIndex
        >>> from gnm.fitting import perform_evaluations
        >>> from gnm.evaluation import ClusteringKS, DegreeKS
        >>> from gnm.defaults import get_binary_network, get_distance_matrix
        >>> # Create and run a model
        >>> binary_params = BinaryGenerativeParameters(
        ...     eta=-2.0,
        ...     gamma=0.3,
        ...     lambdah=0.0,
        ...     distance_relationship_type="powerlaw",
        ...     preferential_relationship_type="powerlaw",
        ...     heterochronicity_relationship_type="powerlaw",
        ...     generative_rule=MatchingIndex(),
        ...     num_iterations=100,
        ... )
        >>> model = GenerativeNetworkModel(
        ...     binary_parameters=binary_params,
        ...     num_simulations=15,
        ...     distance_matrix=get_distance_matrix(),
        ... )
        >>> _, _, _ = model.run_model()
        >>> # Define evaluations
        >>> binary_evals = [ClusteringKS(), DegreeKS()]
        >>> real_networks = get_binary_network()
        >>> # Perform evaluations
        >>> eval_results = perform_evaluations(
        ...     model=model,
        ...     binary_evaluations=binary_evals,
        ...     real_binary_matrices=real_networks,
        ... )
        >>> # Access results
        >>> clustering_scores = eval_results.binary_evaluations["ClusteringKS"]
        >>> degree_scores = eval_results.binary_evaluations["DegreeKS"]

    See Also:
        - [`evaluation.BinaryEvaluationCriterion`][gnm.evaluation.BinaryEvaluationCriterion]: Criteria for binary networks
        - [`evaluation.WeightedEvaluationCriterion`][gnm.evaluation.WeightedEvaluationCriterion]: Criteria for weighted networks
        - [`fitting.EvaluationResults`][gnm.fitting.EvaluationResults]: Container for evaluation results
        - [`GenerativeNetworkModel`][gnm.GenerativeNetworkModel]: The model being evaluated
    """

    if binary_evaluations is not None:
        for evaluation in binary_evaluations:
            assert (
                evaluation.accepts == "binary"
            ), f"Binary evaluations must accept binary matrices. Evaluation {evaluation} accepts {evaluation.accepts}."

    if weighted_evaluations is not None:
        for evaluation in weighted_evaluations:
            assert (
                evaluation.accepts == "weighted"
            ), f"Weighted evaluations must accept weighted matrices. Evaluation {evaluation} accepts {evaluation.accepts}."

    if real_binary_matrices is not None:
        try:
            binary_checks(real_binary_matrices)
        except AssertionError as e:
            raise AssertionError(f"real_binary_matrices are not valid. {e}")
    if real_weighted_matrices is not None:
        try:
            weighted_checks(real_weighted_matrices)
        except AssertionError as e:
            raise AssertionError(f"real_weighted_matrices are not valid. {e}")

    # Move the experiment onto the desired device.
    if device is not None:
        model.to_device(device)
        if isinstance(device, str):
            device = torch.device(device)

        real_binary_matrices = real_binary_matrices.to(device)

        if real_weighted_matrices is not None:
            real_weighted_matrices = real_weighted_matrices.to(device)

    if binary_evaluations is not None and real_binary_matrices is not None:
        synthetic_adjacency_matrices = model.adjacency_matrix
        binary_evaluations_results = {
            str(evaluation): evaluation(
                synthetic_adjacency_matrices,
                real_binary_matrices,
            )
            for evaluation in binary_evaluations
        }
    else:
        binary_evaluations_results = {}

    if (
        weighted_evaluations is not None
        and real_weighted_matrices is not None
        and model.weight_matrix is not None
    ):
        synthetic_weight_matrices = model.weight_matrix
        weighted_evaluations_results = {
            str(evaluation): evaluation(
                synthetic_weight_matrices,
                real_weighted_matrices,
            )
            for evaluation in weighted_evaluations
        }
    else:
        weighted_evaluations_results = {}

    return EvaluationResults(
        binary_evaluations=binary_evaluations_results,
        weighted_evaluations=weighted_evaluations_results,
    )

gnm.fitting.optimise_evaluation(experiments, criterion, maximise_criterion=False, aggregation=MeanAggregator())

Find the optimal experiments based on evaluation criteria.

This function searches through a list of experiments to find the ones that best satisfy a given criterion for each real network. It can either minimise or maximise the criterion value, depending on the desired optimisation direction.

The function handles both binary and weighted evaluation criteria, and can work with criteria specified either by name (string) or object instance.

Parameters:

Name Type Description Default
experiments List[Experiment]

A list of experiments to search through for the optimal ones.

 required
criterion Union[BinaryEvaluationCriterion, WeightedEvaluationCriterion, CompositeCriterion, str]

The criterion to optimise. Can either be specified by name (string) or by passing in the criterion object directly.

 required
maximise_criterion bool

Whether to maximise the criterion. If True, the experiment with the highest criterion value is considered optimal. If False (default), the experiment with the lowest criterion value is considered optimal.

 False
aggregation Aggregator

The method to aggregate evaluation scores across synthetic networks. Default is the MeanAggregator, which averages the evaluation values across all synthetic networks for each real network.

 MeanAggregator()

Returns:

Name Type Description
optimal_experiments List[Experiment]

A list of experiments, one for each real network, where each experiment is the one that best satisfies the criterion for that particular real network.
current_best Float[Tensor, num_real_networks]

The evaluation values of the optimal experiments for each real network.

Examples:

>>> from gnm.fitting import perform_sweep, perform_evaluations, optimise_evaluation
>>> from gnm.evaluation import ClusteringKS
>>> # Run a parameter sweep and get experiments.
>>> experiments = perform_sweep(...)
>>> # Find the experiments that best match clustering coefficients
>>> criterion = ClusteringKS()
>>> best_experiments, best_scores = optimise_evaluation(
...     experiments=experiments,
...     criterion=criterion,
...     maximise_criterion=False,
... )
>>> # For the first real network, show the optimal parameters
>>> best_exp = best_experiments[0]
>>> print(f"Best eta: {best_exp.run_config.binary_parameters.eta}")
>>> print(f"Best gamma: {best_exp.run_config.binary_parameters.gamma}")
>>> print(f"Best score: {best_scores[0]}")
See Also
Source code in src/gnm/fitting/analysis.py 
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
@jaxtyped(typechecker=typechecked)
def optimise_evaluation(
    experiments: List[Experiment],
    criterion: Union[
        BinaryEvaluationCriterion, WeightedEvaluationCriterion, CompositeCriterion, str
    ],
    maximise_criterion: bool = False,
    aggregation: Aggregator = MeanAggregator(),
) -> Tuple[List[Experiment], Float[torch.Tensor, "num_real_networks"]]:
    r"""Find the optimal experiments based on evaluation criteria.

    This function searches through a list of experiments to find the ones that best
    satisfy a given criterion for each real network. It can either minimise or maximise
    the criterion value, depending on the desired optimisation direction.

    The function handles both binary and weighted evaluation criteria, and can work
    with criteria specified either by name (string) or object instance.

    Args:
        experiments:
            A list of experiments to search through for the optimal ones.
        criterion:
            The criterion to optimise. Can either be specified by name (string) or
            by passing in the criterion object directly.
        maximise_criterion:
            Whether to maximise the criterion. If True, the experiment with the highest
            criterion value is considered optimal. If False (default), the experiment
            with the lowest criterion value is considered optimal.
        aggregation:
            The method to aggregate evaluation scores across synthetic networks.
            Default is the MeanAggregator, which averages the evaluation values
            across all synthetic networks for each real network.

    Returns:
        optimal_experiments: A list of experiments, one for each real network, where
                           each experiment is the one that best satisfies the criterion
                           for that particular real network.
        current_best: The evaluation values of the optimal experiments for each real network.

    Examples:
        >>> from gnm.fitting import perform_sweep, perform_evaluations, optimise_evaluation
        >>> from gnm.evaluation import ClusteringKS
        >>> # Run a parameter sweep and get experiments.
        >>> experiments = perform_sweep(...)
        >>> # Find the experiments that best match clustering coefficients
        >>> criterion = ClusteringKS()
        >>> best_experiments, best_scores = optimise_evaluation(
        ...     experiments=experiments,
        ...     criterion=criterion,
        ...     maximise_criterion=False,
        ... )
        >>> # For the first real network, show the optimal parameters
        >>> best_exp = best_experiments[0]
        >>> print(f"Best eta: {best_exp.run_config.binary_parameters.eta}")
        >>> print(f"Best gamma: {best_exp.run_config.binary_parameters.gamma}")
        >>> print(f"Best score: {best_scores[0]}")

    See Also:
        - [`fitting.Aggregator`][gnm.fitting.Aggregator]: Base class for score aggregation methods
        - [`fitting.perform_evaluations`][gnm.fitting.perform_evaluations]: Function to evaluate networks against criteria
        - [`evaluation.BinaryEvaluationCriterion`][gnm.evaluation.BinaryEvaluationCriterion]: Criteria for binary networks
        - [`evaluation.WeightedEvaluationCriterion`][gnm.evaluation.WeightedEvaluationCriterion]: Criteria for weighted networks
    """
    assert len(experiments) > 0, "No experiments provided."

    available_criteria = set(
        experiments[0].evaluation_results.binary_evaluations.keys()
    ).union(experiments[0].evaluation_results.weighted_evaluations.keys())

    if isinstance(criterion, str):
        criterion_name = criterion

        if (
            criterion_name
            in experiments[0].evaluation_results.binary_evaluations.keys()
        ):
            criterion_type = "binary"
        elif (
            criterion_name
            in experiments[0].evaluation_results.weighted_evaluations.keys()
        ):
            criterion_type = "weighted"
        else:
            raise ValueError(
                f"Criterion not found in experiments. Available criteria are {available_criteria}. You may wish to call 'fitting.perform_evaluations' with the desired criterion before this function."
            )
    else:
        criterion_name = str(criterion)
        criterion_type = criterion.accepts

        if criterion_type == "binary":
            assert (
                criterion_name
                in experiments[0].evaluation_results.binary_evaluations.keys()
            ), f"Criterion not found in experiments. Available criteria are {available_criteria}. You may wish to call 'fitting.perform_evaluations' with the desired criterion before this function."
        elif criterion_type == "weighted":
            assert (
                criterion_name
                in experiments[0].evaluation_results.weighted_evaluations.keys()
            ), f"Criterion not found in experiments. Available criteria are {available_criteria}. You may wish to call 'fitting.perform_evaluations' with the desired criterion before this function."
        else:
            raise ValueError(f"Do not recognise criterion type {criterion_type}.")

    num_real_networks = (
        (experiments[0].evaluation_results.binary_evaluations[criterion_name].shape[-1])
        if criterion_type == "binary"
        else experiments[0]
        .evaluation_results.weighted_evaluations[criterion_name]
        .shape[-1]
    )

    optimal_experiments = [experiments[0]] * num_real_networks
    current_best = aggregation(
        experiments[0].evaluation_results.binary_evaluations[criterion_name]
        if criterion_type == "binary"
        else experiments[0].evaluation_results.weighted_evaluations[criterion_name]
    )

    for experiment in experiments[1:]:
        current_evaluation = aggregation(
            experiment.evaluation_results.binary_evaluations[criterion_name]
            if criterion_type == "binary"
            else experiment.evaluation_results.weighted_evaluations[criterion_name]
        )  # This has shape [num_real_networks]

        if maximise_criterion:
            for idx in range(num_real_networks):
                if current_evaluation[idx] > current_best[idx]:
                    optimal_experiments[idx] = experiment
                    current_best[idx] = current_evaluation[idx]
        else:
            for idx in range(num_real_networks):
                if current_evaluation[idx] < current_best[idx]:
                    optimal_experiments[idx] = experiment
                    current_best[idx] = current_evaluation[idx]

    return optimal_experiments, current_best

Aggregating evaluations

gnm.fitting.Aggregator

Bases: ABC

Abstract base class for aggregating evaluation scores across simulations.

Aggregators reduce a matrix of evaluation scores from multiple simulations into a single score for each real network. Different aggregation methods (mean, max, min, quantile) provide different perspectives on model performance.

All aggregators transform a scores tensor with shape [num_synthetic_networks, num_real_networks] into a reduced tensor with shape [num_real_networks], applying their specific aggregation method along the first dimension, i.e., across the synthetic networks.

See Also

__call__(scores) abstractmethod

Source code in src/gnm/fitting/analysis.py 
49
50
51
52
53
54
@abstractmethod
@jaxtyped(typechecker=typechecked)
def __call__(
    self, scores: Float[torch.Tensor, "num_synthetic_networks num_real_networks"]
) -> Float[torch.Tensor, "num_real_networks"]:
    pass

gnm.fitting.MeanAggregator

Bases: Aggregator

Aggregates scores by taking the mean across simulations.

This aggregator computes the average score across all synthetic networks for each real network. It provides a measure of central tendency in model performance.

Examples:

>>> import torch
>>> from gnm.fitting import MeanAggregator
>>> # Create some example scores
>>> scores = torch.tensor([
...     [0.1, 0.2, 0.3],  # Scores for synthetic network 1
...     [0.2, 0.3, 0.4],  # Scores for synthetic network 2
...     [0.3, 0.4, 0.5],  # Scores for synthetic network 3
... ])
>>> # Aggregate using the mean
>>> aggregator = MeanAggregator()
>>> mean_scores = aggregator(scores)
>>> mean_scores
tensor([0.2000, 0.3000, 0.4000])
See Also

gnm.fitting.MaxAggregator

Bases: Aggregator

Aggregates scores by taking the maximum across simulations.

This aggregator selects the maximum score across all synthetic networks for each real network. It provides a measure of the worst-case performance when the score represents dissimilarity (higher is worse).

Examples:

>>> import torch
>>> from gnm.fitting import MaxAggregator
>>> # Create some example scores
>>> scores = torch.tensor([
...     [0.1, 0.2, 0.3],  # Scores for synthetic network 1
...     [0.2, 0.3, 0.4],  # Scores for synthetic network 2
...     [0.3, 0.4, 0.5],  # Scores for synthetic network 3
... ])
>>> # Aggregate using the maximum
>>> aggregator = MaxAggregator()
>>> max_scores = aggregator(scores)
>>> max_scores
tensor([0.3000, 0.4000, 0.5000])
See Also

gnm.fitting.MinAggregator

Bases: Aggregator

Aggregates scores by taking the minimum across simulations.

This aggregator selects the minimum score across all synthetic networks for each real network. It provides a measure of the best-case performance when the score represents dissimilarity (lower is better).

Examples:

>>> import torch
>>> from gnm.fitting import MinAggregator
>>> # Create some example scores
>>> scores = torch.tensor([
...     [0.1, 0.2, 0.3],  # Scores for synthetic network 1
...     [0.2, 0.3, 0.4],  # Scores for synthetic network 2
...     [0.3, 0.4, 0.5],  # Scores for synthetic network 3
... ])
>>> # Aggregate using the minimum
>>> aggregator = MinAggregator()
>>> min_scores = aggregator(scores)
>>> min_scores
tensor([0.1000, 0.2000, 0.3000])
See Also

gnm.fitting.QuantileAggregator(quantile=0.5)

Bases: Aggregator

Aggregates scores by computing a specific quantile across simulations.

This aggregator calculates a specified quantile (e.g., median, 75th percentile) across all synthetic networks for each real network. It provides a flexible way to characterize the distribution of scores beyond simple mean, min, or max. Defaults to the median, which is a more robust measure of central tendency than the mean.

Examples:

>>> import torch
>>> from gnm.fitting import QuantileAggregator
>>> # Create some example scores
>>> scores = torch.tensor([
...     [0.1, 0.2, 0.3],  # Scores for synthetic network 1
...     [0.2, 0.3, 0.4],  # Scores for synthetic network 2
...     [0.3, 0.4, 0.5],  # Scores for synthetic network 3
... ])
>>> # Aggregate using the median (0.5 quantile)
>>> aggregator = QuantileAggregator(quantile=0.5)
>>> median_scores = aggregator(scores)
>>> median_scores
tensor([0.2000, 0.3000, 0.4000])
>>> # Aggregate using the 75th percentile
>>> aggregator = QuantileAggregator(quantile=0.75)
>>> q75_scores = aggregator(scores)
>>> q75_scores
tensor([0.2500, 0.3500, 0.4500])
See Also

Parameters:

Name Type Description Default
quantile float

The quantile to compute. Must be in the range [0, 1]. Default is 0.5 for the median.

 0.5
Source code in src/gnm/fitting/analysis.py 
188
189
190
191
192
193
194
def __init__(self, quantile: float = 0.5):
    r"""
    Args:
        quantile:
            The quantile to compute. Must be in the range [0, 1]. Default is 0.5 for the median.
    """
    self.quantile = quantile

Saving Experiments

gnm.fitting.experiment_saving.ExperimentEvaluation(path=None, index_file_path=None, variables_to_ignore=[], save=True)

The ExperimentEvaluation class provides functionality for managing and saving experiment data for generative network models. It handles creating directories, managing index files, saving experiment configurations, and querying experiments.

Attributes:

Name Type Description
- path (str

Directory where experiment data is stored. Defaults to 'generative_model_experiments'.
- index_path (str

Path to the index file that tracks experiment configurations.
- variables_to_save (list

List of variables to save, excluding those specified in variables_to_ignore.

Methods:

Name Description
- __init__

Initializes the class, sets up paths, and prepares the index file.
- _refresh_index_file

Reloads the index file from disk or creates it if it doesn't exist.
- _make_index_file

Creates a new index file with initial data.
- save_experiments

Saves a list of Experiment objects to disk.
- _save_experiment

Saves a single experiment and updates the index file.
- view_experiments

Placeholder for viewing experiments as a table or saving them as CSV.
- _sort_experiments

Sorts experiments by a specified variable.
- clean_index_file

Placeholder for cleaning up the index file.
- _ask_loop

Prompts the user for confirmation with a yes/no question.
- delete_experiment

Deletes an experiment and removes it from the index file.
- purge_index_file

Placeholder for purging the index file.
- _is_similar_wording

Suggests the most similar variable name if a given name is not found.
- query_experiments

Queries experiments based on a variable and value.
- open_experiments_by_name

Opens experiments by their names and returns their data.
Usage

evaluator = ExperimentEvaluation(path="experiment_data", index_file_path="index.json") evaluator.save_experiments([experiment1, experiment2]) results = evaluator.query_experiments(value=0.5, by="alpha")

Source code in src/gnm/fitting/experiment_saving.py 
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
def __init__(self, path=None, index_file_path=None, variables_to_ignore=[], save=True):#
    self.save = save

    if path is None:
        path = 'generative_model_experiments'

    if index_file_path is None:
        index_file_path = 'gnm_index.json'

    # create path to experiment data and index file if it doesn't exist already
    if not os.path.exists(path) and save:
        os.mkdir(path)

    self.path = path
    self.index_path = os.path.join(self.path, index_file_path)      

    # get the variables we want to save, i.e. alpha, gamma etc (some will be in list format)
    binary_variables_to_save = [f.name for f in fields(BinarySweepParameters)]
    weighted_variables_to_save = [f.name for f in fields(WeightedSweepParameters)]
    variables_to_save = binary_variables_to_save + weighted_variables_to_save
    self.variables_to_save = [i for i in variables_to_save if i not in variables_to_ignore]

    if self.save:
        self._refresh_index_file()