'jj0JdZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l m Z mZe r dd lmZd d lmZiZeed GddZGddZedZd+dZd,d!Zd-d.d&ZejZd/d'Zd-d(d)d*ZdS)0z Contains the logic for automatic additional output capture with our forward decorators. This mostly describe the hooks used and the logic to make capture thread/context safe. ) annotationsN) ContextVar) dataclasswraps) TYPE_CHECKING)is_torchdynamo_compilingrequires)nnPreTrainedModel)torch)backendscHeZdZUdZded<dZded<dZded <dZded <dS) OutputRecordera Configuration for recording outputs from a model via hooks. Attributes: target_class (Type): The class (e.g., nn.Module) to which the hook will be attached. index (Optional[int]): If the output is a tuple/list, optionally record only at a specific index. layer_name (Optional[str]): Name of the submodule to target (if needed), e.g., "transformer.layer.3.attn". class_name (Optional[str]): Name of the class to which the hook will be attached. Could be the suffix of class name in some cases. ztype[nn.Module] target_classrintindexN str | None layer_name class_name)__name__ __module__ __qualname____doc____annotations__rrrk/home/rurouni/.hermes/hermes-agent/venv/lib/python3.11/site-packages/transformers/utils/output_capturing.pyrr'sY"!!!ENNNN!J!!!!!J!!!!!!r rc*eZdZdZdZdZdZdZdS)CompileableContextVara Convenience wrapper around a ContextVar for usage with `torch.compile`. This behaves exactly as a `ContextVar`, except when compilation is triggered in which case it behaves as a simple global variable. This is useful as `torch.compile` cannot trace the `get` method of `ContextVar`. This however means that the access to the underlying variable is not thread-safe when compilation is triggered. cNt|d|_d|_d|_dS)N)defaultF)r context_var global_var compiling)selfnames r!__init__zCompileableContextVar.__init__Bs)%dD999r cP|jr|jS|jSN)r(r'r&get)r)s r!r.zCompileableContextVar.getGs) > *? "#'')) )r crtr||_d|_dS|j|S)NT)r r'r(r&set)r)values r!r0zCompileableContextVar.setNs: # % % /#DO!DN4#''.. .r cl|js|d|_d|_dS|j|dS)NF)r(r'r&reset)r)tokens r!r3zCompileableContextVar.resetVs? > *U]"DO"DNNN   " "5 ) ) ) ) )r N)rrrrr+r.r0r3rr r!r#r#:sZ ***///*****r r#output_collectormodule nn.ModulekeystrrrreturnNonec@fd}||dS)zaInstall the forward hook needed to capture the output described by `key` and `index` in `module`.ct}||vrdSdkr:t|dkr!||dt |t s||dS|#||dSdS)N hidden_statesr)_active_collectorr.keyslenappend isinstancetuple)r6argsoutputcollected_outputsrr8s r!output_capturing_hookz.output_capturing_hookes-1133  $3D3I3I3K3K(K(K F / ! !c*;C*@&A&AQ&F&F c " ) )$q' 2 2 2&%(( 9 c " ) )& 1 1 1 1 1 E] & c " ) )&- 8 8 8 8 8' &r N)register_forward_hook)r6r8rrHs `` r!install_output_capturing_hookrJbs? 9 9 9 9 9 9   !677777r parent_module module_name capture_tasks list[tuple[str, OutputRecorder]]cddlm}|D]B\}}t||st ||d||,t ||d|C|D]i\}}|jt||js!|jA||jr'|j |j |vrSt|||j jdS)a Recursively install all output capturing hooks on all submodules of `parent_module`. Note that we need to use this recursive approach instead of simply iterating over all modules, because we want to respect the `capture_tasks` of all individual submodels (`PreTrainedModel` instances) in the graph. That is, once we reach a submodel in the graph, its children should use this submodel's `capture_tasks`, but other parts of the graph should not. r r.)prefixN) modeling_utilsrnamed_childrenrCrecursively_install_hooks"install_all_output_capturing_hooksrrendswithrrJr)rKrLrMrr*r6r8specss r!rTrTvs 100000&4466WW f&/22 W %f.E.Et.E.E} U U U U /v>U>Ut>U>U V V V V V$KK U   *z-I[/\/\ *   ([-A-A%BR-S-S (+0@ 0S0S )-ek J J JKKr modelrrQrctt|jpi}g}|D]\}}t |t s|g}|D]z}t |tsLd|vrdnd}t |tsdn|}t |ts|nd} t| ||}|||f{||nd}t|||t|dddS) z Install the output recording hooks on all the modules in `model`. This will take care of correctly dispatching the `_can_record_outputs` property of each individual submodels in case of composite models. r>rr N)rrr!_output_capturing_hooks_installedT) _CAN_RECORD_REGISTRYr.r9 __class__itemsrClistrrBrTsetattr) rXrQ capture_flagsrMr8 layer_specsrWrrrs r!rUrUs9 ),,S-A-ABBHbMM)//11 / /[+t,, (&-K  / /Ee^44 f,33)3E3)?)?JTTU ,6uc,B,BLuu &LZdeee  #u . . . .  /)VVrFeV];;; E6=====r ct|ddrdSt5t|ddr ddddSt|ddddS#1swxYwYdS)z Check if the model already has output capturing hooks installed, and install them if it is not already the case. Note that this is thread-safe, in case 2 (or more) threads want to install them concurrently. r[FN)getattr_hook_installation_lockrU)rXs r!maybe_install_capturing_hooksrfsu95AA 22 5=u E E   22222222 +5111 222222222222222222sAAAAT)tie_last_hidden_statesc,fd}| ||S|S)a Decorator to intercept specific layer outputs through hooks. The hooks are installed only once and lazily, the first time output capture is requested with the `output_xxx` kwargs/config. The implementation is fully context/thread safe, except when using `torch.compile`, as dynamo is unable to trace through `ContextVar` methods. Args: tie_last_hidden_states (`bool`, *optional*, defaults to `True`): Whether to overwrite `out.hidden_states[-1]` with the `out.last_hidden_state`. This is true for all language models and should be toggled off only if `out.hidden_states[-1]` has to be the hidden state before last layer norm, which is needed for some vision models (e.g. CLIP, SigLIP) c@tfd}|S)Ncdtjdd}tt jpi}fd|D}d|vr-dtjdd|d<d|vr-dtjdd|d <d |D}t|d krtt |}  g|Ri}t |n#t |wxYw|D]_} | d kr snt|d r4|| dd|| <|| |jnCt|dr3|| dd|| <|| |jt#|| || <| dkrt%|| t&r`t|| dkrGt#|| d dd|| <t#|| ddd|d| z<-t#|| || <Gt#|| || <a|dur|}|S)N return_dictTc xi|]6}d|d|tjd|d7S)output_F)r.rdconfig).0kkwargsr)s r! zHcapture_outputs..wrapped_fn..wrapper..s[! vzz-A--mXYmm]b9c9cddr cross_attentionsoutput_attentionsFoutput_cross_attentionsmask_decoder_attentionsoutput_mask_decoder_attentionscFi|]\}}||ddgS)rmrZ)replace)rorpvs r!rrzHcapture_outputs..wrapped_fn..wrapper..s2 g g g$!Qef g9b!9!92 g g gr rr>vision_hidden_stateslast_hidden_state attentionsr r cross_)poprdrnr\r.r9r]r^rArfr?r0r3hasattrrBr{r}rDrCr_to_tuple) r)rErqrkcapturable_flagsrecordable_keysrG output_tokenoutputsr8funcrgs ` ` r!wrapperz4capture_outputs..wrapped_fn..wrappers]!**]GDKX\4]4]^^K 477DN8K8KLLRPR )O "%555=CZZ'>QSX)Y)Y>> 9:),<<QSX)Y)YEE @A!h g/J_J_JaJa g g g $%%))-d333,001BCCL 6$t5d555f55"'' 5555!'' 5555) A A/))1Q *@AAQ1B31G1L)#.)#.55g6RSSSS *=>>Q1B31G1L)#.)#.55g6OPPP#():3)?#@#@GCLLL((!"23"7>>E3GWX[G\C]C]abCbCb',->s-CADqD-I'J'J 278I#8NqtRSt8T2U2U3//',->s-C'D'D #():3)?#@#@GCLLe##!**,,Ns ( EE+r)rrrgs` r! wrapped_fnz#capture_outputs..wrapped_fns: t= = = = =  = ~r r)rrgrs ` r!capture_outputsrs@AAAAAF z$ r )r6r7r8r9rrr:r;)rKr7rLr9rMrNr:r;r-)rXrrQrr:r;)rXrr:r;)r __future__r threading contextvarsr dataclassesr functoolsrtypingr import_utilsr r rr rRrr\rr#r?rJrTrULockrerfrrr r!rs #"""""""""""!!!!!! <<<<<<<<1000000  :""""""" ""!*!*!*!*!*!*!*!*J*)*<==8888(KKKK@>>>>>:)).**2222&TTTTTTTTr