$j)pdZddlmZddlZddlZddlmZddlmZm Z ddl m Z ddl Z ddlZddlmZmZmZmZmZmZ ddlmZn #e$rdZYnwxYw dd lmZmZn#eef$rdZdZYnwxYwejeZ d d d d d Z!d.dZ"d/dZ#d.dZ$d.dZ%d.dZ&d0dZ'd1dZ(d2d!Z)Gd"d#Z* d3d4d(Z+ d3d5d+Z,d6d-Z-dS)7zAModality detection, input parsing, and message format conversion.) annotationsN) defaultdict)AnyLiteral)urlparse) AudioInput MessageFormatModality PairInput SingleInput VideoInput)Image) AudioDecoder VideoDecoderflat)apertus deepseek_v3gpt_ossseed_osstextstrreturnboolc4|dod|vS)zQCheck if a string looks like a valid URL (starts with http(s) and has no spaces).)zhttp://zhttps:// ) startswithrs k/home/rurouni/.hermes/hermes-agent/venv/lib/python3.11/site-packages/sentence_transformers/base/modality.py_looks_like_urlr.s ??2 3 3 G4G extensionstuple[str, ...]cHt|rN t|j}n#t$rYdSwxYw||S||ot j|S)zOCheck if a string is a URL or local file path with one of the given extensions.F)rrpathlower ValueErrorendswithosisfile)rr!r$s r_is_media_url_or_pathr*3st) D>>&,,..DD   55 }}Z((( ::<<  , , E1E1EEs&8 AAcP|drdSt|dS)z:Check if a string is an image URL, file path, or data URI.z data:image/T)z.jpgz.jpegz.pngz.gifz.bmpz.tiffz.webp)rr*rs ris_image_url_or_pathr,?s, }%%t 'b c ccr ct|drdSt|sdS t|j}n#t$rYdSwxYw|dvS)z.Check if a string is a video URL or file path.)z.mp4z.aviz.movz.wmvz.flvz.mkvTF)zwww.youtube.comz youtube.comzyoutu.bez m.youtube.com)r*rrnetlocr&)rr.s ris_video_url_or_pathr/FsxT#STTt 4 u$& uu  s: AAc"t|dS)z/Check if a string is an audio URL or file path.)z.mp3z.wavz.oggz.flacz.aac)r*rs ris_audio_url_or_pathr1Xs 'P Q QQr samplerct|ttfrt|dkrdSt|dtrt|dtrdS|D]W}t|t r d|vrd|vrdSt|tr |rt|dt rdSXdS)aCheck if a sample is a non-text pair (2-element tuple/list with at least one non-string element). Text pairs ``(str, str)`` are handled natively by tokenizers and detected as ``"text"`` modality by :func:`infer_modality`. This helper detects pairs that contain at least one non-string element (e.g. an image, audio array, or dict), which require conversion to message format. FrrolecontentT) isinstancetuplelistlenrdict)r2elems r_is_non_text_pairr>]s fudm , ,F q0@0@u&)S!!jC&@&@u dD ! ! fnnd9J9J55 dD ! ! d z$q'4/H/H 55 4r audio_valuerextra_modality_kwargsdict[str, dict[str, Any]]cDt|trd|vr|d|dd<|dStft|trQ|}|j|dd<|jdS|S)zUnwrap dict-wrapped audio or an ``AudioDecoder`` into a raw array, collecting ``sampling_rate``. Passes through unchanged if ``audio_value`` is already a raw array/tensor/URL/path. sampling_rateaudioarrayNr)dim)r8r<rget_all_samples sample_ratedatameannumpy)r?r@sampless r _unwrap_audiorMrs +t$$$ k ) )>I/>Z !' *? ;7##J{L$I$I--//:A:Mg&7|  Q ''--/// r video_valuer c  t|trAd|vr5|ddg|d|dStt|tr|dt |}|ddg|jj|jj tt|j j dd|j S|S)zUnwrap dict-wrapped video or a ``VideoDecoder`` into a raw array, collecting ``video_metadata``. Passes through unchanged if ``video_value`` is already a raw array/tensor/URL/path. video_metadatavideorENr)fpstotal_num_framesframes_indices)r8r< setdefaultappendrget_frames_in_ranger;metadata average_fps num_framesr:rangerIshape)rNr@ frame_batchs r _unwrap_videor^s +t$$$ { * * !' * 5 56F K K R RS^_oSp q q q7##J{L$I$I!55a[9I9IJJ g&112BBGGNN"+7$/$8$C"&u[-=-CA-F'G'G"H"H      r c|eZdZdZ d.d/d Zd0dZd1dZd2dZd3d4dZd5d!Z e d6d%Z d7d'Z d8d)Z d9d-ZdS):InputFormatteraHandles input parsing, modality detection, and message format conversion. This class manages the complete input preprocessing pipeline: 1. Parsing raw inputs to detect their modality (text, image, audio, video, message) 2. Converting inputs to different chat template formats 3. Normalizing mixed-modality inputs Different models require different message/chat template formats: - **Structured format**: Content is a list of dicts with type annotations [{"role": "user", "content": [{"type": "text", "text": "hello"}]}] - **Flat format**: Content is the direct value [{"role": "user", "content": "hello"}] Args: model_type: The model type string (e.g. from ``config.model_type``). message_format: Message format to use. Options: - ``"structured"``: Content is a list of dicts with type/modality keys - ``"flat"``: Content is the direct value - ``"auto"``: Automatically infer from processor (default) processor: Optional processor to infer format from when ``message_format="auto"``. supported_modalities: Optional list of modalities supported by the model. When provided, string inputs that look like media URLs/paths are only classified as non-text if the model actually supports that modality. This prevents text-only models from misclassifying text containing media URLs. autoN model_typermessage_formatr supported_modalitieslist[Modality] | NonerNonec||_||_||_|dkr |r||nd|_dS||_dS)Nra structured)rb processorrd _infer_formatrc)selfrbrcrirds r__init__zInputFormatter.__init__sZ%"$8! V # #CL"^$"4"4Y"?"?"?R^D   "0D   r Literal['structured', 'flat']c|jtvrt|jSt|ddttrsdSgd}t fd|DrdSdS)aInfer the message format expected by the processor. Checks known model types first, then inspects the processor's chat template for patterns indicating structured format. Defaults to ``"structured"`` if neither approach is conclusive. Args: processor: The processor/tokenizer to inspect. Returns: ``"structured"`` or ``"flat"`` message format. chat_templateNrh)z content[0]z.typez'type'z"type"z item.typezmessage.content[c3 K|]}|vV dSN).0patterntemplates r z/InputFormatter._infer_format..s(FFww("FFFFFFr r)rb!KNOWN_MODEL_TYPES_MESSAGE_FORMATSgetattrr8rany)rkristructured_patternsrus @rrjzInputFormatter._infer_formats ?? ? ?4T_E E9ot<<(C((  <    FFFF2EFFF F F <vr inputslist[SingleInput | PairInput]Btuple[Modality, dict[str, list], defaultdict[str, dict[str, Any]]]c|sddgittfSg}tt}d}|D]}t|r|d|fd},t |j}|dkrt ||}n|dkrt||}nt|trJt|}d|vrt |d||d<d|vrt|d||d<n!|dkrt|tr|g}n|}|||f |rg}|D]\} }| dkr)| |4| dkrXt|ttfrz:InputFormatter.parse_inputs...Ds)S)S)S%*)S)S)Sr rr)rsrprocessed_inputss @r z/InputFormatter.parse_inputs..Ds3#l#l#lX[C)S)S)S)SBR)S)S)S#l#l#lr zMixed modalities detected: z!. Converting to 'message' format.crg|]3\}}t|tr|n||i4Srr) to_messager8r9)rsmodalityvaluerks rrz/InputFormatter.parse_inputs..HsR'%OOZ%-H-H$_EExY^N_``r )rr<r>rVinfer_modalityrdrMr^r8r9pair_to_messagesr:r;rzipsetpoprkeysloggerdebug)rkr{ typed_inputsr@ has_pairsitemrrmessagesrtyped modalitiesunique_modalities ordered_keysrs` @r parse_inputszInputFormatter.parse_inputss8 ;FB<T):):: :EG +D 1 1  3 3D!&& ##VTN333 %dAZ[[[H 7""%d,ABBW$$%d,ABBHe,, T e##%25>CX%Y%YE'Ne##%25>CX%Y%YE'NY&&:dD+A+A&   5 1 2 2 2 2  KH* < < U&==OOD$9$9%$@$@AAAAF]]z%%'G'G]CPUJJZ[OOOOD$9$9%$@$@AAAAI%%OOE****%/U%;%;MEE#uEOODOOE$:$:;;;;y(35JJ J'*L'9$ $ 011 OO  ! !Q & &(,,..H(C(( m$,.>#?   027799 #l#l#l#l_k#l#l#l  LLk7Hkkk l l l+7   !H)+@@@r r tuple | listlist[dict[str, Any]]c|\}}t|}t|}|jdkr d|dd|dgSd}d|||dd|||dgS)aConvert a pair of inputs to query/document message format. Each element of the pair is wrapped in a message with role ``"query"`` (first element) or ``"document"`` (second element). The modality of each element is inferred individually via :func:`infer_modality`. Args: pair: A 2-element tuple or list of inputs (e.g. ``(image, text)``). Returns: List of two message dictionaries with ``"query"`` and ``"document"`` roles. rqueryr6r7documentct|tr#ttrfd|DSd||igS)Nc0g|]}|vd|||iStyperr)rsrrs rrzHInputFormatter.pair_to_messages.._to_content..ls,YYY#STX[[c495[[[r r)r8r9r<)rrs `r _to_contentz4InputFormatter.pair_to_messages.._to_contenthsV(E** Zz$/E/E ZYYYYYYYYXx67 7r )rrc)rkr query_itemdoc_itemquery_modality doc_modalityrs rrzInputFormatter.pair_to_messagesQs $ H' 33%h//  & ( ( Z88#99   8 8 8^Z)P)P Q QKK h,O,O P P  r user typed_inputdict[Modality, Any]r6c$|jdkrdt|dkr7tt|\}}||dgSt d|d|DdgS)aConvert a typed input dictionary to message format. Produces a single message with the given ``role``. For pair/multi-value inputs, use :meth:`pair_to_messages` instead (which is called automatically by :meth:`parse_inputs`). Args: typed_input: Dictionary mapping modality to input value (single value per modality). role: Role for the message (default: ``"user"``). Returns: List of message dictionaries (single message). rr5rzbFlat message format requested but multiple modalities detected. Falling back to structured format.c g|] \}}d|||i Srrr)rsrrs rrz-InputFormatter.to_message..s%kkkOHeVXx?kkkr )rcr;nextiteritemsrwarning)rkrr6_rs rrzInputFormatter.to_messagets  & ( (;1$$[%6%6%8%8 9 9::5!%%88999kkWbWhWhWjWjkkk    r rr processor_inputsr<*tuple[Literal['message'], dict[str, list]]csddgifSt|tr|fn|ttt }g}t |D]fdD}t|dkrtt |}t|ttfrUt|dkrBtd|Dr)| | || | |dd|ifS)acConvert a batch of modality-specific inputs into the unified message format. Args: modality: The modality key (string) or tuple of modality keys. processor_inputs: Dictionary mapping modality names to lists of inputs. Returns: Tuple of ``("message", {"message": [messages_per_sample, ...]})`` rc6i|]}|v||Srrrr)rsrirrs rrz3InputFormatter.batch_to_message..s2jjjSX[_iXiXi3 0 5a 8XiXiXir r5r4c3@K|]}t|tVdSrq)r8r)rsvs rrvz2InputFormatter.batch_to_message..s0OrOrghPZ[\^aPbPbOrOrOrOrOrOrr ) r8rr;rrvaluesr[r9r:allrVrr) rkrr batch_sizerrrrrs ` @@rbatch_to_messagezInputFormatter.batch_to_messagesf  .y"o- -$.x$=$=Kh[[8 d#3#:#:#<#<==>>?? z"" : :AjjjjjjDTjjjK;1$$T+"4"4"6"67788eeT]33E aCOrOrlqOrOrOrLrLrOOD$9$9%$@$@AAA OODOOK88 9 9 9 99h///r messages_batchlist[list[dict[str, Any]]]rc|D]h}|D]c}|d}t|tr-t|trt d|DrdS`dSidS)aCheck whether all messages in a batch contain only text content. Works with both flat format (``{"content": "hello"}``) and structured format (``{"content": [{"type": "text", "text": "hello"}]}``). Args: messages_batch: List of message lists, one per sample. Returns: True if every message contains only text, False if any contain non-text content. r7c3JK|]}|dddkVdS)rrN)get)rsrs rrvz7InputFormatter.is_text_only_messages..s5RR$488FF33v=RRRRRRr FT)rr8rr:ry)rrrr7s ris_text_only_messagesz$InputFormatter.is_text_only_messagess' ! !H# ! !!++i00gs++gt,,!RR'RRRRR%$uuu%!555 !tr rcg}|D]f}d|vsd|vrtd|d*|d}t|to|ot|dt}|jdkr|rt |dkr2d|dvr(|i|d|dditd t |d |||jd krO|sMt|tr!|i|dd|d gi:||Q||h|S) a;Normalize messages to the target format (``self.message_format``). Extra keys beyond ``"role"`` and ``"content"`` are preserved during conversion. Args: messages: List of message dictionaries to normalize. Returns: Normalized list of message dictionaries. r6r7zInvalid message format: z . Skipping.rrr5rz;Cannot convert structured message to flat format: contains z# content items. Keeping structured.rhrr) rrr8r:r<rcr;rVr)rkr normalizedrr7is_currently_structureds rnormalize_messagesz!InputFormatter.normalize_messagess  + +GW$$ (@(@N'NNNOOOi(G&0$&?&?&lG&lPZ[bcd[egkPlPl #"f,,1H,w<<1$$71:)=)=%%&P&P)WQZ=O&P&PQQQQNNV$'LLVVV%%g....$ 44=T4gs++/%%&a&a)vW^>_>_=`&a&abbbb%%g....!!'****r promptcR|jdkrfd|DSfd|DS)a'Prepend a system prompt to message format inputs. Args: messages: List of message lists (each message list represents one input). prompt: System prompt to prepend. Returns: Messages with system prompt prepended to each message list. rc"g|] }ddg|z S)systemrrrrs message_listrs rrz=InputFormatter.prepend_prompt_to_messages..s)hhh|h6::;lJhhhr c*g|]}dddgdg|zS)rrrrrrrs rrz=InputFormatter.prepend_prompt_to_messages..sE   VV,L,L+M N N OR^ ^   r )rc)rkrrs `rprepend_prompt_to_messagesz)InputFormatter.prepend_prompt_to_messagessY  & ( (hhhh_ghhh h    (    r texts'list[str | tuple[str, str] | list[str]]list[str | list[str]]c g}|D]g}t|tr|||z0|||dzgt|ddzh|S)abPrepend a prompt to text format inputs. For single texts, prepends the prompt directly. For text pairs (cross-encoder inputs), prepends only to the first text. Args: texts: List of text inputs (strings or pairs) prompt: Prompt to prepend Returns: Texts with prompt prepended rr5N)r8rrVr:)rkrrresultrs rprepend_prompt_to_textsz&InputFormatter.prepend_prompt_to_textss C CD$$$ C ftm,,,, vQ/04QRR>>ABBBB r )raNN)rbrrcr rdrerrf)rrm)r{r|rr})rrrr)r)rrr6rrr)rr rr<rr)rrrr)rrrr)rrrrrr)rrrrrr)__name__ __module__ __qualname____doc__rlrjrrrr staticmethodrrrrrrr rr`r`s<)/6: 1 1 1 1 1    DiAiAiAiAV! ! ! ! F     <0000:\0%%%%N    &r r`SingleInput | PairInput | Anyrdrer c^tt|trdStt|trdStt|trdS|xtdx\t |r |d|vrdSdSxtdx\t |r |d|vrdSdSxtdx\t|r |d|vrdSdSxxtdx\nOx r"dkr\tdx\tdx\n*x r"dkr\tdx\tdx\nndSxtdx \d|vrd |vrd Sxtdx6\|r3t|d trd|d vrd |d vrd Sxtdx \d |vrd |vrdSxtdx \d |vrd|vrdSxtdx8\d |vr3tdt| xtdxr\|rohd}t| |z }|rtd|d|tt| Sxtdx\tdxtjdx\nxt"jdx\nn<|jdvrdS|jdkrdS|jdvrdStd|jd tdt)|jd)aInfer the modality of a single input sample by inspecting its type/structure. Pure type-based detection, does not require a processor or tokenizer. Args: sample: A single input sample to inspect. supported_modalities: Optional list of modalities the model supports. When provided, string inputs that would be classified as image/video/audio based on URL/path heuristics are instead classified as ``"text"`` if that modality is not supported. This prevents misclassification of text that happens to contain media URLs. Returns: The detected modality string, or a tuple of modality strings for multimodal dict inputs. Raises: ValueError: If the input type/structure is not recognized. NimagerDrQrrrr4r6r7rrrErCrPzuDict input with 'array' key must also include 'sampling_rate' (for audio) or 'video_metadata' (for video). Got keys: >rrDrrQz;Multimodal dict input contains unrecognized modality keys: z. Expected keys from: z-Empty dict input is not a valid input sample.)r5r4)z#Unsupported tensor dimensionality: zz'infer_batch_modality..s(jjjX^.>RSSSjjjr r5r)r;r)rLrdrs ` rinfer_batch_modalityrusQ$ vjjjjbijjjJ":!33:>>   Br rcZt|trd|S|S)zSFormat a modality for display, e.g. ``("text", "image")`` becomes ``"text+image"``.+)r8r9join)rs rformat_modalityrs+(E"""xx!!! Or )rrrr)rrr!r"rr)r2rrr)r?rr@rArr)rNr r@rArrrq)r2rrdrerr )rLr|rdrerr )rr rr).r __future__rloggingr( collectionsrtypingrr urllib.parserrKrr)sentence_transformers.base.modality_typesrr r r r r PIL.Imagerr ImportErrortorchcodec.decodersrrOSError getLoggerrrrwrr*r,r/r1r>rMr^r`rrrrrr rrsGG"""""" ######!!!!!! +++++++HHH>>>>>>>>>WLLLL  8 $ $  %%!HHHH F F F Fdddd$RRRR *",~~~~~~~~F 37XXXXXz37CCCCC0s#AAAA A*)A*