o hw@sddlZddlZddlZddlZddlZddlmmmZ ddl mZddl m Z ddl mZddlmZmZmZmZmZmZddlmZmZmZmZmZmZmZmZddlm Z m!Z!ddl"m#Z#dd l$m%Z%m&Z&gd Z'eZ(ej)ej*j)ej+ej*j+iej*j)ejj)ej*j+ejj+id Z,d d Z-   d7ddZ.d8ddZ/ddZ0ddZ1d9ddZ2    d:ddZ3ddZ4ddZ5    d;d d!Z6d"d#Z7d$d%Z8d     d>d1d2Z? d9d3d4Z@d?d5d6ZAdS)@N) _FusedModule)_is_activation_post_process)_activation_is_memoryless_add_module_to_qconfig_obs_ctrdefault_dynamic_qconfigfloat16_dynamic_qconfig!float_qparams_weight_only_qconfig&float_qparams_weight_only_qconfig_4bit)_get_special_act_post_process_has_special_act_post_process)get_default_dynamic_quant_module_mappingsget_default_qat_module_mappings$get_default_qconfig_propagation_list(get_default_static_quant_module_mappings2get_default_static_quant_reference_module_mappingsno_observer_set) DeQuantStub QuantWrapper)type_before_parametrizations)get_qparam_dict)has_no_children_ignoring_parametrizations) get_default_custom_config_dictpropagate_qconfig_add_quant_dequantpreparequantizequantize_dynamic prepare_qat quantize_qatconvert swap_module)%float_to_observed_custom_module_class)observed_to_quantized_custom_module_classcCstS)z'Defines the default custom config dict.)_DEFAULT_CUSTOM_CONFIG_DICTr%r%r/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/torch/ao/quantization/quantize.pyrBsrc Cs|t||}|||}t|d|}tjjj||t||}||_| D]+\}}|r5|d|n|} |dusM||dgvsTt ||dgvsTt |||| q)dS)aThis is a helper function for `propagate_qconfig_` Args: module: input module qconfig_dict: dictionary that maps from name of submodule to quantization configuration qconfig_parent: quantization config of parent module, we will fallback to this config when there is no specified config for current module prefix: corresponding prefix of the current module, used as key in qconfig_dict prepare_custom_config_dict: dictionary for custom handling of modules see docs for :func:`~torch.ao.quantization.prepare_fx` Return: None, module is modified inplace with qconfig attached qconfig.Nnon_traceable_module_namenon_traceable_module_class) getrgetattrtorchao quantizationr(_assert_valid_qconfigrnamed_childrentype_propagate_qconfig_helper) module qconfig_dictqconfig_parentprefixprepare_custom_config_dictmodule_qconfigqconfig_with_device_checknamechild module_prefixr%r%r&r4Gs(    r4cCs*|duri}|dur i}t|||ddS)aPropagate qconfig through the module hierarchy and assign `qconfig` attribute on each leaf module Args: module: input module qconfig_dict: dictionary that maps from name or type of submodule to quantization configuration, qconfig applies to all submodules of a given module unless qconfig for the submodules are specified (when the submodule already has qconfig attribute) prepare_custom_config_dict: dictionary for custom handling of modules see docs for :func:`~torch.ao.quantization.prepare_fx` Return: None, module is modified inplace with qconfig attached N)r9)r4)r5r6r9r%r%r&rxs rcCs ||S)z.Forward hook that calls observer on the outputactivation_post_process)selfinputoutputr%r%r&_observer_forward_hooks rDcCs||dS)z2Forward pre hook that calls observer on the outputrr?)rArBr%r%r&_observer_forward_pre_hooksrEFcCs:t|ds Jd|r|jtdddS|jtdddS)Nr@zGExpect activation_post_process attribute already attached to the moduleT)prepend)hasattrregister_forward_pre_hookrEregister_forward_hookrD)r5pre_hookr%r%r&&_register_activation_post_process_hooksrKc s|durt}|dur i}dur0t|}t|dks"Jd|t|dkr.tt|ndddddddfd d }|D]\}}t|tjfvrSqFt t|t j t j fry|rxt |d sqJd t|d |j|_qFt|tr|r||qF|durt||vr|r||qFt|rt|} ||| qF|rt||vr|t|} | |} t||| t | tts|| qFt||||qFt|rt|tjjst||vr||t |dr t|tjjs t||vr ||dSdSdSdS)asAdd observer for the leaf child of the module. This function insert observer module to all leaf child module that has a valid qconfig attribute. Args: module: input module with qconfig attributes for all the leaf modules that we want to quantize qconfig_propagation_list: a list of quantizable modules that will have observers added to them if they are leaf nodes device: parent device, if any non_leaf_module_list: list of non-leaf modules we want to add observer Return: None, module is modified inplace with added observer modules and forward_hooks NrzR_add_observer_ only works with cpu or single-device CUDA modules, but got devices rcSs,|dur|n|}|dur|||SN) activationto)r(devicespecial_act_post_processrMr%r%r&get_activation_post_processs z3_add_observer_..get_activation_post_processcSst|do |jduS)Nr(rGr()mr%r%r&needs_observationz)_add_observer_..needs_observationcsF|rt|ts!|d|j|t|t|jddSdSdS)zmAdds an activation post process module and register a pre or post hook that calls the module r@rJN) isinstancer add_moduler(rKr)rSrPrOrQrTr%r&insert_activation_post_processs  z6_add_observer_..insert_activation_post_processr@zfunctional class z- has no pre-defined `activation_post_process`weight_fake_quantrL)r_get_unique_devices_lennextiterr2rnnDropout issubclassnnqFloatFunctional QFunctionalrGr(r@rWrr r from_floatsetattrtupler_add_observer_rr. Sequential) r5qconfig_propagation_listnon_leaf_module_listrOcustom_module_class_mappingdevicesrZr<r=rPobserved_classobserved_childr%rYr&ris             ricCs$dd|Ddd|DBS)NcSsh|] }|jjdkr|jqS)meta)rOr3).0pr%r%r& /sz'_get_unique_devices_..) parametersbuffersr5r%r%r&r\.sr\cCsDt|rt|dr|jrt|S|D] \}}t||j|<q|S)a{Wrap the leaf child module in QuantWrapper if it has a valid qconfig Note that this function will modify the children of module inplace and it can return a new module which wraps the input module as well. Args: module: input module with qconfig attributes for all the leaf modules that we want to quantize Return: Either the inplace modified module with submodules wrapped in `QuantWrapper` based on qconfig or a new `QuantWrapper` module which wraps the input module, the latter case only happens when the input module is a leaf module and we want to quantize it. r()rrGr(rr2r_modules)r5r<r=r%r%r&r4srcCstjd|dur t}|di}|st|}|}|dur#t}t|ddt dd| Ds9t dt ||||d|S) aPrepares a copy of the model for quantization calibration or quantization-aware training. Quantization configuration should be assigned preemptively to individual submodules in `.qconfig` attribute. The model will be attached with observer or fake quant modules, and qconfig will be propagated. Args: `model`: input model to be modified in-place `inplace`: carry out model transformations in-place, the original module is mutated `allow_list`: list of quantizable modules `observer_non_leaf_module_list`: list of non-leaf modules we want to add observer `prepare_custom_config_dict`: customization configuration dictionary for prepare function .. code-block:: python # Example of prepare_custom_config_dict: prepare_custom_config_dict = { # user will manually define the corresponding observed # module class which has a from_float class method that converts # float custom module to observed custom module "float_to_observed_custom_module_class": { CustomModule: ObservedCustomModule } } z!quantization_api.quantize.prepareNr"r6css |] }t|do |jVqdS)r(NrR)rrrSr%r%r& szprepare..zNone of the submodule got qconfig applied. Make sure you passed correct configuration through `qconfig_dict` or by assigning the `.qconfig` attribute directly on submodules)rm)r._C_log_api_usage_oncerr,copydeepcopyrranymoduleswarningswarnri)modelinplace allow_listobserver_non_leaf_module_listr9rmrkr%r%r&rOs. #  rcsDtdrtjrtddfdd }|dd|dddS)Nr@Fcs^|rjnj}|r tnt}t}|D] \}}||ur"||q|D]}||q%dSrL)_forward_pre_hooks_forward_hooksrErDsetitemsaddpop)rJhook_map observer_hookhandle_ids_to_remove handle_idhook_fnrwr%r& remove_hookss   z5_remove_activation_post_process..remove_hooksTrVF)rGrr@delattr)r5rr%rwr&_remove_activation_post_processs   rcCs0|D]}t|qt|dr|`t|dS)zClean up the qconfig left in the module so that new qconfig can be propagated. Args: module: module to be cleaned up r(N)children_remove_qconfigrGr(r)r5r=r%r%r&rs    rcCs^tjd|dur t}|st|}|t|dd||g|Rt||dd|S)aQuantize the input float model with post training static quantization. First it will prepare the model for calibration, then it calls `run_fn` which will run the calibration step, after that we will convert the model to a quantized model. Args: model: input float model run_fn: a calibration function for calibrating the prepared model run_args: positional arguments for `run_fn` inplace: carry out model transformations in-place, the original module is mutated mapping: correspondence between original module types and quantized counterparts Return: Quantized model. z"quantization_api.quantize.quantizeNTr) r.r{r|rr}r~evalrr )rrun_fnrun_argsmappingrr%r%r&rs   rc Csjtjd|dur_|tjkr$tjttjttjttj ttj ttj ti}nq|tj kr>tjt tjt tjt tj t tj t tj t i}nW|tjkrLtjttjti}nI|tjkrWtjti}n>td|dt|tr|tjurlt}n|tj urtt }n|tjur|t}n|tjurt}ntdt|tt|t|}|durt}|st|}| t!||t"||dd|S)avConverts a float model to dynamic (i.e. weights-only) quantized model. Replaces specified modules with dynamic weight-only quantized versions and output the quantized model. For simplest usage provide `dtype` argument that can be float16 or qint8. Weight-only quantization by default is performed for layers with large weights size - i.e. Linear and RNN variants. Fine grained control is possible with `qconfig` and `mapping` that act similarly to `quantize()`. If `qconfig` is provided, the `dtype` argument is ignored. Args: model: input model qconfig_spec: Either: - A dictionary that maps from name or type of submodule to quantization configuration, qconfig applies to all submodules of a given module unless qconfig for the submodules are specified (when the submodule already has qconfig attribute). Entries in the dictionary need to be QConfig instances. - A set of types and/or submodule names to apply dynamic quantization to, in which case the `dtype` argument is used to specify the bit-width inplace: carry out model transformations in-place, the original module is mutated mapping: maps type of a submodule to a type of corresponding dynamically quantized version with which the submodule needs to be replaced z*quantization_api.quantize.quantize_dynamicNz5Don't know how to quantize with default settings for z. Provide full qconfig pleasez.Unknown dtype specified for quantize_dynamic: Tr)#r.r{r|qint8r`LinearrLSTMGRULSTMCellRNNCellGRUCellfloat16rquint8 EmbeddingBagr Embeddingquint4x2r ValueErrorrWr RuntimeErrorstrdictzip itertoolsrepeatr r}r~rrr )r qconfig_specdtyperrdefault_qconfigr%r%r&rsb             rcCsltjd|js Jd|durt}|st|}t|ddt||dddt |t | dd|S) a Prepares a copy of the model for quantization calibration or quantization-aware training and converts it to quantized version. Quantization configuration should be assigned preemptively to individual submodules in `.qconfig` attribute. Args: model: input model to be modified in-place mapping: dictionary that maps float modules to quantized modules to be replaced. inplace: carry out model transformations in-place, the original module is mutated z%quantization_api.quantize.prepare_qatz1prepare_qat only works on models in training modeNryTF)rrremove_qconfig)rr) r.r{r|trainingr r}r~rr rrvalues)rrrr%r%r&r4s   rcCsNtjd|s t|}|t|dd||g|Rt|dd|S)agDo quantization aware training and output a quantized model Args: model: input model run_fn: a function for evaluating the prepared model, can be a function that simply runs the prepared model or a training loop run_args: positional arguments for `run_fn` Return: Quantized model. z&quantization_api.quantize.quantize_qatTr)r.r{r|r}r~trainrr )rrrrr%r%r&rQs    rTcCs>tjd|s t|}t||d|||d|rt||S)aConverts submodules in input module to a different module according to `mapping` by calling `from_float` method on the target module class. And remove qconfig at the end if remove_qconfig is set to True. Args: `module`: prepared and calibrated module `mapping`: a dictionary that maps from source module type to target module type, can be overwritten to allow swapping user defined Modules `inplace`: carry out model transformations in-place, the original module is mutated `convert_custom_config_dict`: custom configuration dictionary for convert function `use_precomputed_fake_quant`: a flag to enable use of precomputed fake quant .. code-block:: python # Example of convert_custom_config_dict: convert_custom_config_dict = { # user will manually define the corresponding quantized # module class which has a from_observed class method that converts # observed custom module to quantized custom module "observed_to_quantized_custom_module_class": { ObservedCustomModule: QuantizedCustomModule } } z!quantization_api.quantize.convertT)r is_referenceconvert_custom_config_dictuse_precomputed_fake_quant)r.r{r|r}r~_convertr)r5rrrrrrr%r%r&r hs $ r c Cs|dur |r tnt}|durt}|di}|s t|}i}|D]"\}} t| ts?t | |vr?t | |d|||dt | |||||<q&| D] \} } | |j | <qM|S)aoConverts submodules in input module to a different module according to `mapping` by calling `from_float` method on the target module class Args: module: input module mapping: a dictionary that maps from source module type to target module type, can be overwritten to allow swapping user defined Modules inplace: carry out model transformations in-place, the original module is mutated is_reference: a flag to enable quantized reference module use_precomputed_fake_quant: a flag to enable use of precomputed fake quant Nr#Tr)rrrr,r}r~r2rWrrrr!rrx) r5rrrrrrmreassignr<modkeyvaluer%r%r&rs>    rcCs~|}t|dr|jdurd}t||vr |t||}d}nJt||vrj|t|}t|drP|jrP|jdus;J|j}||jt|}|||}nt |j} d| j vrc|j||d}n||}d}|r|j D]} | | qq|j D] } | tur|| q~t|} t| dkst| d krtd | vsJd | t| d krtt| nd} | r|| |S) a Swaps the module if it has a quantized counterpart and it has an `observer` attached. Args: mod: input module mapping: a dictionary that maps from nn module to nnq module Return: The corresponding quantized module of `mod` r(NFT _IS_REFERENCErrrrqzOswap_module only works with cpu or single-device CUDA modules, but got devices r)rGr(r from_observedrweightrrfinspect signaturerurrrHrrDrIr\r]r.rOr^r_rN)rrrmrnew_modswappedqmodweight_post_processweight_qparamssig pre_hook_fnrrnrOr%r%r&r!sR             r!cCsZdd}t|dr|j|||d<|D]\}}|r"|||n|}t|||qdS)a,Traverse the modules and save all observers into dict. This is mainly used for quantization accuracy debug Args: mod: the top module we want to save all observers prefix: the prefix for the current module target_dict: the dictionary used to save all the observers cSs|dkr|S|dS)Nr'r)r%)r8r%r%r& get_prefixrUz&_get_observer_dict..get_prefixr@N)rGr@r2_get_observer_dict)r target_dictr8rr<r=r>r%r%r&rs  r)Nr'N)NNr)NNNN)FNNN)NF)NFTFNF)NFFNF)r')Br}rrrr.torch.ao.nn.quantizedr/r` quantizedrctorch.nntorch.ao.nn.intrinsicrtorch.ao.quantization.observerrtorch.ao.quantization.qconfigrrrrrr +torch.ao.quantization.quantization_mappingsr r r r rrrrtorch.ao.quantization.stubsrrtorch.nn.utils.parametrizerutilsrr__all__is_activation_post_processr quantizableMultiheadAttentionr$rr4rrDrErKrir\rrrrrrrrrr rr!rr%r%r%r&s    (       1     D   Z  6 ? <