credit.scheduler

credit.scheduler#

Attributes#

`update_on_batch`
`update_on_epoch`
`num_epochs`

Classes#

`LinearWarmupCosineScheduler`	Linear warmup followed by cosine decay to min_lr.
`CosineAnnealingWarmupRestarts`	optimizer (Optimizer): Wrapped optimizer.

Functions#

`load_scheduler`(optimizer, conf)	Load a learning rate scheduler based on the configuration.
`lr_lambda_phase2`(step[, total_updates_phase2])
`phased_lr_lambda`(step[, total_updates_phase1, ...])
`lr_lambda_phase1`(epoch[, num_epochs, warmup_epochs])
`annealed_probability`(epoch[, max_epochs, ...])	Anneal the termination probability from 1 to a small value.

Module Contents#

credit.scheduler.update_on_batch = ['cosine-annealing-restarts', 'linear-warmup-cosine']#

credit.scheduler.update_on_epoch = ['lambda', 'plateau', 'cosine-annealing']#

class credit.scheduler.LinearWarmupCosineScheduler(optimizer, warmup_steps: int = 1000, total_steps: int = 500000, min_lr: float = 1e-05, last_epoch: int = -1)#

Bases: torch.optim.lr_scheduler.LRScheduler

Linear warmup followed by cosine decay to min_lr.

Steps are batch-level (call scheduler.step() after each optimizer.step()).

Parameters:

optimizer – Wrapped optimizer.
warmup_steps – Number of steps to linearly ramp lr from 0 to base_lr.
total_steps – Total number of steps (warmup + cosine decay).
min_lr – Minimum learning rate at the end of decay (absolute value).
last_epoch – Used by LRScheduler for resume; -1 means start from scratch.

warmup_steps = 1000#

total_steps = 500000#

min_lr = 1e-05#

get_lr()#

Compute the next learning rate for each of the optimizer’s param_groups.

Returns:: A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"]s.
Return type:: list[float | Tensor]

Note

If you’re trying to inspect the most recent learning rate, use get_last_lr() instead.

Note

The returned Tensors are copies, and never alias the optimizer’s group["lr"]s.

credit.scheduler.load_scheduler(optimizer, conf)#

Load a learning rate scheduler based on the configuration.

Parameters: - optimizer: The PyTorch optimizer. - conf: The configuration dictionary.

Returns: - scheduler: The PyTorch learning rate scheduler.

credit.scheduler.lr_lambda_phase2(step, total_updates_phase2=299000)#

credit.scheduler.phased_lr_lambda(step, total_updates_phase1=1000, total_updates_phase2=299000)#

credit.scheduler.lr_lambda_phase1(epoch, num_epochs=100, warmup_epochs=10)#

class credit.scheduler.CosineAnnealingWarmupRestarts(optimizer: torch.optim.Optimizer, first_cycle_steps: int, cycle_mult: float = 1.0, max_lr: float = 0.1, min_lr: float = 0.001, warmup_steps: int = 0, gamma: float = 1.0, last_epoch: int = -1)#

Bases: torch.optim.lr_scheduler.LRScheduler

optimizer (Optimizer): Wrapped optimizer. first_cycle_steps (int): First cycle step size. cycle_mult(float): Cycle steps magnification. Default: -1. max_lr(float): First cycle’s max learning rate. Default: 0.1. min_lr(float): Min learning rate. Default: 0.001. warmup_steps(int): Linear warmup step size. Default: 0. gamma(float): Decrease rate of max learning rate by cycle. Default: 1. last_epoch (int): The index of last epoch. Default: -1.

first_cycle_steps#

cycle_mult = 1.0#

base_max_lr = 0.1#

max_lr = 0.1#

min_lr = 0.001#

warmup_steps = 0#

gamma = 1.0#

cur_cycle_steps#

cycle = 0#

step_in_cycle = -1#

init_lr()#

get_lr()#

Compute the next learning rate for each of the optimizer’s param_groups.

Returns:: A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"]s.
Return type:: list[float | Tensor]

Note

If you’re trying to inspect the most recent learning rate, use get_last_lr() instead.

Note

The returned Tensors are copies, and never alias the optimizer’s group["lr"]s.

step(epoch=None)#

Step the scheduler.

Parameters:

epoch (int, optional) –

Deprecated since version 1.4: If provided, sets last_epoch to epoch and uses _get_closed_form_lr() if it is available. This is not universally supported. Use step() without arguments instead.

Note

Call this method after calling the optimizer’s step().

credit.scheduler.annealed_probability(epoch, max_epochs=100, min_probability=0.01, max_probability=1.0)#

Anneal the termination probability from 1 to a small value.

Parameters: - epoch: The current epoch. - max_epochs: The maximum number of epochs for annealing. - min_probability: The minimum termination probability. - max_probability: The maximum termination probability.

Returns: - termination_probability: The annealed termination probability.

credit.scheduler.num_epochs = 100#