o hv@sddlmZddlZddlZddlmZmZddlmZddl m Z m Z m Z m Z mZmZddlZer8ddlmZddgZGd d d ZGd ddeZdddZGdddZdS)) annotationsN)abc defaultdict)Enum)AnycastOptionaloverload TYPE_CHECKINGUnion)IterableOptState GradScalerc@s$eZdZdZd ddZdd d Zd S)_MultiDeviceReplicatorz^Lazily serves copies of a tensor to requested devices. Copies are cached per-device. master_tensor torch.TensorreturnNonecCs||_i|_dSN)master_per_device_tensors)selfrri/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/torch/amp/grad_scaler.py__init__s z_MultiDeviceReplicator.__init__device torch.devicecCs6|j|d}|dur|jj|ddd}||j|<|S)NT)r non_blockingcopy)rgetrto)rrretvalrrrrs  z_MultiDeviceReplicator.getN)rrrr)rrrr)__name__ __module__ __qualname____doc__rrrrrrrs rc@seZdZdZdZdZdS)r rN)r"r#r$READYUNSCALEDSTEPPEDrrrrr +srdict[str, Any]cCs tjidS)N)stagefound_inf_per_device)r r(rrrr_refresh_per_optimizer_state1s r.c@sReZdZdZ      dhdiddZdjddZdkddZedld d!Zedmd#d!Zednd%d!Zedod'd!Zdpd)d!Zdqd0d1Z drd2d3Z dsd:d;Z dtdS)raXAn instance ``scaler`` of :class:`GradScaler`. Helps perform the steps of gradient scaling conveniently. * ``scaler.scale(loss)`` multiplies a given loss by ``scaler``'s current scale factor. * ``scaler.step(optimizer)`` safely unscales gradients and calls ``optimizer.step()``. * ``scaler.update()`` updates ``scaler``'s scale factor. Example:: # Creates a GradScaler once at the beginning of training. scaler = GradScaler() for epoch in epochs: for input, target in data: optimizer.zero_grad() output = model(input) loss = loss_fn(output, target) # Scales loss. Calls backward() on scaled loss to create scaled gradients. scaler.scale(loss).backward() # scaler.step() first unscales gradients of the optimizer's params. # If gradients don't contain infs/NaNs, optimizer.step() is then called, # otherwise, optimizer.step() is skipped. scaler.step(optimizer) # Updates the scale for next iteration. scaler.update() See the :ref:`Automatic Mixed Precision examples` for usage (along with autocasting) in more complex cases like gradient clipping, gradient accumulation, gradient penalty, and multiple losses/optimizers. ``scaler`` dynamically estimates the scale factor each iteration. To minimize gradient underflow, a large scale factor should be used. However, ``float16`` values can "overflow" (become inf or NaN) if the scale factor is too large. Therefore, the optimal scale factor is the largest factor that can be used without incurring inf or NaN gradient values. ``scaler`` approximates the optimal scale factor over time by checking the gradients for infs and NaNs during every ``scaler.step(optimizer)`` (or optional separate ``scaler.unscale_(optimizer)``, see :meth:`unscale_`). * If infs/NaNs are found, ``scaler.step(optimizer)`` skips the underlying ``optimizer.step()`` (so the params themselves remain uncorrupted) and ``update()`` multiplies the scale by ``backoff_factor``. * If no infs/NaNs are found, ``scaler.step(optimizer)`` runs the underlying ``optimizer.step()`` as usual. If ``growth_interval`` unskipped iterations occur consecutively, ``update()`` multiplies the scale by ``growth_factor``. The scale factor often causes infs/NaNs to appear in gradients for the first few iterations as its value calibrates. ``scaler.step`` will skip the underlying ``optimizer.step()`` for these iterations. After that, step skipping should occur rarely (once every few hundred or thousand iterations). Args: device (str, optional, default="cuda"): Device type to use. Possible values are: 'cuda' and 'cpu'. The type is the same as the `type` attribute of a :class:`torch.device`. Thus, you may obtain the device type of a tensor using `Tensor.device.type`. init_scale (float, optional, default=2.**16): Initial scale factor. growth_factor (float, optional, default=2.0): Factor by which the scale is multiplied during :meth:`update` if no inf/NaN gradients occur for ``growth_interval`` consecutive iterations. backoff_factor (float, optional, default=0.5): Factor by which the scale is multiplied during :meth:`update` if inf/NaN gradients occur in an iteration. growth_interval (int, optional, default=2000): Number of consecutive iterations without inf/NaN gradients that must occur for the scale to be multiplied by ``growth_factor``. enabled (bool, optional): If ``False``, disables gradient scaling. :meth:`step` simply invokes the underlying ``optimizer.step()``, and other methods become no-ops. Default: ``True`` cuda@@?Trstr init_scalefloat growth_factorbackoff_factorgrowth_intervalintenabledboolrrcCs||_||_|jdkr|rtjjjrtdd|_|jrK|dks'Jd|dks/Jd||_ d|_ ||_ ||_ ||_ d|_d|_tt|_dSdS)Nr/zLtorch.cuda.amp.GradScaler is enabled, but CUDA is not available. Disabling.F?z The growth factor must be > 1.0.z!The backoff factor must be < 1.0.r)_device_enabledtorchr/ampcommonamp_definitely_not_availablewarningswarn _init_scale_scale_growth_factor_backoff_factor_growth_interval_init_growth_tracker_growth_trackerrr._per_optimizer_states)rrr5r7r8r9r;rrrr{s,   zGradScaler.__init__funcname!tuple[torch.Tensor, torch.Tensor]cCsLd}|jdusJd|d||jdus Jd|d||j|jfS)NzaThis may indicate your script did not use scaler.scale(loss or outputs) earlier in the iteration.z Attempted z but _scale is None. z but _growth_tracker is None. )rGrL)rrNfixrrr_check_scale_growth_trackers   z&GradScaler._check_scale_growth_trackerdevrcCsF|jdus Jdtjd|jtj|d|_tjd|jtj|d|_dS)Nz)_growth_tracker initialized before _scalerdtyper)rLr@fullrFfloat32rGrKint32)rrRrrr_lazy_init_scale_growth_trackers   z*GradScaler._lazy_init_scale_growth_trackeroutputsrcCdSrrrrYrrrscalezGradScaler.scalelist[torch.Tensor]cCrZrrr[rrrr\r]tuple[torch.Tensor, ...]cCrZrrr[rrrr\r]Iterable[torch.Tensor]cCrZrrr[rrrr\r]+Union[torch.Tensor, Iterable[torch.Tensor]]csnjs|St|tjr(jdur|jjdusJ|jj|jddSgdfdd |S) a2 Multiplies ('scales') a tensor or list of tensors by the scale factor. Returns scaled outputs. If this instance of :class:`GradScaler` is not enabled, outputs are returned unmodified. Args: outputs (Tensor or iterable of Tensors): Outputs to scale. NTrrvalracst|tjr0tdkr&jdur|jjdusJtj|d |jSt|t j rJt |}t|t tfrHt||S|Std)Nrz2outputs must be a Tensor or an iterable of Tensors) isinstancer@TensorlenrGrXrappendrrrr maplisttupletype ValueError)rciterable apply_scalerstashrrros       z%GradScaler.scale..apply_scale)rcra)r?rdr@rerGrXrr r[rrnrr\s     optimizertorch.optim.Optimizer inv_scale found_inf allow_fp16 dict[torch.device, torch.Tensor]c Cs"t|}t|}tdd}tt|jD]I}|dD]B} t| tjs&J| jdur,q|s9| jjtj kr9t d| jj rP| jjtj urJ| j | _| j } n| j} || j| j| qq|D]\} } | D]} t| || || qlqdWd|jS1swY|jS)NcSsttSr)rrirrrrsz,GradScaler._unscale_grads_..paramsz%Attempting to unscale FP16 gradients.)rrr@no_grad param_groupsrdregradrTfloat16rl is_sparsecoalesce_valuesrrgitemsvalues*_amp_foreach_non_finite_check_and_unscale_rr)rrqrsrtruper_device_inv_scaleper_device_found_infper_device_and_dtype_gradsgroupparam to_unscalerper_dtype_gradsgradsrrr_unscale_grads_sF            zGradScaler._unscale_grads_cCs|jsdS|d|jt|}|dtjurtd|dtjur'td|jdus.J|j }t j ddt j|jjd}||||d |d <tj|d<dS) as Divides ("unscales") the optimizer's gradient tensors by the scale factor. :meth:`unscale_` is optional, serving cases where you need to :ref:`modify or inspect gradients` between the backward pass(es) and :meth:`step`. If :meth:`unscale_` is not called explicitly, gradients will be unscaled automatically during :meth:`step`. Simple example, using :meth:`unscale_` to enable clipping of unscaled gradients:: ... scaler.scale(loss).backward() scaler.unscale_(optimizer) torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm) scaler.step(optimizer) scaler.update() Args: optimizer (torch.optim.Optimizer): Optimizer that owns the gradients to be unscaled. .. note:: :meth:`unscale_` does not incur a CPU-GPU sync. .. warning:: :meth:`unscale_` should only be called once per optimizer per :meth:`step` call, and only after all gradients for that optimizer's assigned parameters have been accumulated. Calling :meth:`unscale_` twice for a given optimizer between each :meth:`step` triggers a RuntimeError. .. warning:: :meth:`unscale_` may unscale sparse gradients out of place, replacing the ``.grad`` attribute. Nunscale_r,zMunscale_() has already been called on this optimizer since the last update().z(unscale_() is being called after step().rrSFr-)r?rQrMidr r) RuntimeErrorr*rGdouble reciprocalr6r@rUrVrr)rrqoptimizer_statersrtrrrr#s"  zGradScaler.unscale_rr+argsrkwargsOptional[float]cOs2d}tdd|dDs|j|i|}|S)Ncss|]}|VqdSr)item).0vrrr csz-GradScaler._maybe_opt_step..r-)sumrstep)rrqrrrr!rrr_maybe_opt_step[szGradScaler._maybe_opt_stepc s|js |j|i|Sd|vrtd|d|jt|}|dtjur*tdd}t|ddr|}d t |jj v}|rMt d t|d |inB|dtjurY|||duscJttjtfd d |d D}|dtjkrt|ddnt|dd|_||_|j|i|}tj|d<|s|`|`|S|dtjur||t|d dksJd|j||g|Ri|}tj|d<|S)aInvoke ``unscale_(optimizer)`` followed by parameter update, if gradients are not infs/NaN. :meth:`step` carries out the following two operations: 1. Internally invokes ``unscale_(optimizer)`` (unless :meth:`unscale_` was explicitly called for ``optimizer`` earlier in the iteration). As part of the :meth:`unscale_`, gradients are checked for infs/NaNs. 2. If no inf/NaN gradients are found, invokes ``optimizer.step()`` using the unscaled gradients. Otherwise, ``optimizer.step()`` is skipped to avoid corrupting the params. ``*args`` and ``**kwargs`` are forwarded to ``optimizer.step()``. Returns the return value of ``optimizer.step(*args, **kwargs)``. Args: optimizer (torch.optim.Optimizer): Optimizer that applies the gradients. args: Any arguments. kwargs: Any keyword arguments. .. warning:: Closure use is not currently supported. closurez@Closure use is not currently supported if GradScaler is enabled.rr,z7step() has already been called since the last update().N_step_supports_amp_scalingF grad_scalerzGradScaler is going to stop passing itself as a keyword argument to the passed optimizer. In the near future GradScaler registers `grad_scale: Tensor` and `found_inf: Tensor` to the passed optimizer and let the optimizer use them directly.csg|] }|jjddqS)T)r)r r)rtscalerrr sz#GradScaler.step..r- grad_scaler&rz/No inf checks were recorded for this optimizer.)r?rrrQrMrr r*getattrinspect signature parametersrDrE FutureWarningupdater(_check_inf_per_device_get_scale_asyncrr@rerrr)rrtrrfr) rrqrrrr!kwargs_has_grad_scaler_kwargrtrrrrgsl           zGradScaler.stepN new_scale$Optional[Union[float, torch.Tensor]]cs|jsdS|d\}|durJ|jdusJt|tr#|j|ndd}|jj|jks0J|| dks:J||j dusCJ||j |n=fdd|j D}t|dks`Jd |d}t|dkrztdt|D]}|||7}qqt|||j|j|jtt|_ dS) a?Update the scale factor. If any optimizer steps were skipped the scale is multiplied by ``backoff_factor`` to reduce it. If ``growth_interval`` unskipped iterations occurred consecutively, the scale is multiplied by ``growth_factor`` to increase it. Passing ``new_scale`` sets the new scale value manually. (``new_scale`` is not used directly, it's used to fill GradScaler's internal scale tensor. So if ``new_scale`` was a tensor, later in-place changes to that tensor will not further affect the scale GradScaler uses internally.) Args: new_scale (float or :class:`torch.Tensor`, optional, default=None): New scale factor. .. warning:: :meth:`update` should only be called at the end of the iteration, after ``scaler.step(optimizer)`` has been invoked for all optimizers used this iteration. .. warning:: For performance reasons, we do not check the scale factor value to avoid synchronizations, so the scale factor is not guaranteed to be above 1. If the scale falls below 1 and/or you are seeing NaNs in your gradients or loss, something is likely wrong. For example, bf16-pretrained models are often incompatible with AMP/fp16 due to differing dynamic ranges. Nrznew_scale should be a float or a 1-element torch.cuda.FloatTensor or torch.FloatTensor with requires_grad=False.r&Fcs.g|]}|dD] }|jjddq qS)r-Trb)rr r)rstatertrGrrrs z%GradScaler.update..rz,No inf checks were recorded prior to update.)r?rQrGrdr6fill_rrkr>numel requires_gradcopy_rMrrfranger@_amp_update_scale_rHrIrJrr.)rrrLreason found_infsfound_inf_combinedirrrrs:    zGradScaler.updateOptional[torch.Tensor]cC|jSrrrrrrrszGradScaler._get_scale_asynccCs.|jr|}dur|jStt|SdS)zReturn a Python float containing the current scale, or 1.0 if scaling is disabled. .. warning:: :meth:`get_scale` incurs a CPU-GPU sync. Nr=)r?rrFrr6r)rr\rrr get_scales zGradScaler.get_scalecCr)z9Return a Python float containing the scale growth factor.rHrrrrget_growth_factor*zGradScaler.get_growth_factor new_factorcC ||_dS)zSet a new scale growth factor. Args: new_scale (float): Value to use as the new scale growth factor. Nrrrrrrset_growth_factor. zGradScaler.set_growth_factorcCr)z:Return a Python float containing the scale backoff factor.rIrrrrget_backoff_factor6rzGradScaler.get_backoff_factorcCr)zSet a new scale backoff factor. Args: new_scale (float): Value to use as the new scale backoff factor. Nrrrrrset_backoff_factor:rzGradScaler.set_backoff_factorcCr)z3Return a Python int containing the growth interval.rJrrrrget_growth_intervalBrzGradScaler.get_growth_interval new_intervalcCr)z|Set a new growth interval. Args: new_interval (int): Value to use as the new growth interval. Nr)rrrrrset_growth_intervalFrzGradScaler.set_growth_intervalcCs*|jr|jdur |jStt|jSdS)Nr)r?rLrKrr:rrrrr_get_growth_trackerNs zGradScaler._get_growth_trackercCr)z:Return a bool indicating whether this instance is enabled.)r?rrrr is_enabledWrzGradScaler.is_enabledcCs(|jr||j|j|j|dSiS)aReturn the state of the scaler as a :class:`dict`. It contains five entries: * ``"scale"`` - a Python float containing the current scale * ``"growth_factor"`` - a Python float containing the current growth factor * ``"backoff_factor"`` - a Python float containing the current backoff factor * ``"growth_interval"`` - a Python int containing the current growth interval * ``"_growth_tracker"`` - a Python int containing the number of recent consecutive unskipped steps. If this instance is not enabled, returns an empty dict. .. note:: If you wish to checkpoint the scaler's state after a particular iteration, :meth:`state_dict` should be called after :meth:`update`. )r\r7r8r9rL)r?rrHrIrJrrrrr state_dict[szGradScaler.state_dictrcCs|jsdSt|dkrtdtt|d|_|jdur$|j|dtt|d|_tt|d|_ tt |d|_ tt |d|_ |j durS|j |ddSdS) zLoad the scaler state. If this instance is disabled, :meth:`load_state_dict` is a no-op. Args: state_dict(dict): scaler state. Should be an object returned from a call to :meth:`state_dict`. NrzeThe source state dict is empty, possibly because it was saved from a disabled instance of GradScaler.r\r7r8r9rL)r?rfrrr6rFrGrrHrIr:rJrKrL)rrrrrload_state_dictvs    zGradScaler.load_state_dictcCsR|j}|jr't|jdksJd||d<||d<d|d<d|d<|S)NrzpA GradScaler instance may only be pickled at the beginning of an iteration, or at the end after scaler.update().rFrKrGrL)__dict__rr?rfrMrrrrrrr __getstate__s   zGradScaler.__getstate__rcCs|j|dSr)rrrrrr __setstate__szGradScaler.__setstate__cCsj|d\}}tjddtj|jd}tjddtj|jd}||||d|jt|d<|jt|dS)Nrrr=rSrTr-)rQr@rUrVrrrMr)rrqrG_dummy_inv_scalertrrrrs z GradScaler._check_inf_per_devicecCs|jt|dS)Nr-)rMr)rrqrrr_found_inf_per_devicesz GradScaler._found_inf_per_device)r/r0r1r2r3T)rr4r5r6r7r6r8r6r9r:r;r<rr)rNr4rrO)rRrrr)rYrrr)rYr^rr^)rYr_rr_)rYr`rr`)rYrarra) rqrrrsrrtrrur<rrv)rqrrrr) rqrrrr+rrrrrr)rqrrrrrrrrr)rrrr)rr)rr6)rr6rr)rr:)rr:rr)rr<rr+)rr+rr)rr+rr)rqrrrr+)r"r#r$r%rrQrXr r\rrrrrrrrrrrrrrrrrrrrrrrrrr5sRG #       - 5 8 l F              r) __future__rrrD collectionsrrenumrtypingrrrr r r r@collections.abcr __all__rr r.rrrrrs