%j< d Z ddlZddlZddlZddlZddlZddlmZ ddlm Z m
Z
m Z m Z m
Z
mZmZmZ ddlZddlmZ ddlZddlmZ ddlmZmZm Z m Z m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z% dZ& G d d Z'dS )
zMagika (the Python library).
This module provides the `Magika` class, the main entry point for using Magika
to identify file content types.
N)Path)BinaryIODictListOptionalSequenceSetTupleUnion)
get_logger) ContentTypeInfoContentTypeLabel MagikaErrorMagikaPrediction MagikaResult ModelConfig
ModelFeatures ModelOutputOverwriteReasonPredictionModeSeekableStatus
standard_v3_3c e Zd ZdZdej ddddfdee dedededed ed
dfd Z d
e
fd Z d
e
fd
Z d
e
fdZ
d
e
fdZdee
ej f d
efdZdeee
ej f d
ee fdZded
efdZded
efdZd
ee fdZ d
ee fdZ e d
e
fd Z e ded
e ee!f fd Z"e d ed
e#fd Z$d
e%j& fd Z'd ed
e!fd!Z(dee d
ee fd"Z)ded
efd#Z*d$e+d
efd%Z,e d$e+d&e-d'e-d(e-d)e-d*e-d+ed
e.fd, Z/e d-ed&e-d)e-d
ee- fd. Z0e d/ed(e-d)e-d
ee- fd0 Z1d1ee2ee.f d
ee2ee3f fd2Z4d1ee2ee.f d
e e
ef fd3Z5d4ed5e6d
e2ee7f fd6Z8e7j9 fded4ed7ed5e6d8e7d
ef d9Z:ded
e2ee ee. f fd:Z; ed; fd$e+ded
e2ee ee. f fd<Z< ed; fdeded
efd=Z=ded
efd>Z>d?ee2ee.f d
e?j@ fd@ZAdS )AMagikazMain Magika class for content type identification.
This class provides methods to identify the content type of files, bytes,
and streams.
NF model_dirprediction_modeno_dereferenceverbosedebug
use_colorsreturnc t | | _ |r$| j t j |r$| j t j ||| _ n6t t j dz |
z | _ | j dz | _ | j dz | _ | j
s$t dt | j | j s$t dt | j | j s$t dt | j t" | j | _ t) j t- t/ t | j j | _ || _ || _ t t j d z d
z }t" | | _ | | _ dS ) ai Initializes the Magika instance.
Args:
model_dir: Path to the directory containing the model and its
configuration. If None, the default model is used.
prediction_mode: The prediction mode to use. Defaults to
PredictionMode.HIGH_CONFIDENCE.
no_dereference: If True, do not follow symlinks. Defaults to False.
verbose: If True, enable verbose logging. Defaults to False.
debug: If True, enable debug logging. Defaults to False.
use_colors: If True, use colors in the logger. Defaults to False.
)r! Nmodelsz
model.onnxzconfig.min.jsonzmodel dir not found at zmodel not found at zmodel config not found at configzcontent_types_kb.min.json) r _logsetLevelloggingINFODEBUG
_model_dirr __file__parent_get_default_model_name _model_path_model_config_pathis_dirr stris_filer _load_model_config
_model_confignparraylistmaptarget_labels_space_target_labels_space_np_prediction_mode_no_dereference_load_content_types_kb
_cts_infos_init_onnx_session
_onnx_session)selfr r r r r r! content_types_kb_paths U/home/rurouni/.hermes/hermes-agent/venv/lib/python3.11/site-packages/magika/magika.py__init__zMagika.__init__; s * *555 - I w| , , , . I w} - - - 'DOO X%043O3O3Q3QQ
O ?\9"&/4E"E %%'' P NDO8L8L N NOO O '')) M KC8H4I4I K KLL L&..00 KS1H-I-IKK
+1*C*C #+
+
(*x S$,@AA B B(
(
$ !0 -
NN !H ,/J J !778MNN ! 4 4 6 6 c t | S N)r2 rB s rD __repr__zMagika.__repr__ s 4yyrF c \ d| d| dS )NzMagika(module_version="z", model_name="z"))get_module_versionget_model_namerI s rD __str__zMagika.__str__ s4 l)@)@)B)BllSWSfSfShShllllrF c N t t | j j S )z-Gets the version of the Magika Python module.)r2
__import__
__module__ __version__rI s rD rL zMagika.get_module_version s :do..:;;;rF c | j j S )z"Gets the name of the loaded model.)r+ namerI s rD rM zMagika.get_model_name s
##rF pathc t |t st |t j rt | }nt d| d | | S )z3Identify the content type of a file given its path.zPath 'D' is invalid: input path should be of type `Union[str, os.PathLike]`)
isinstancer2 osPathLiker TypeError_get_result_from_pathrB rU s rD
identify_pathzMagika.identify_path sj dC JtR[$A$A ::DDcccc
))$///rF pathsc J t |t st d g }|D ]f}t |t st |t j r#| t | Tt d| d | | S )z?Identify the content type of a list of files given their paths.z,Input paths should be of type Sequence[Path]zInput 'rW ) rX r r[ r2 rY rZ appendr _get_results_from_paths)rB r_ paths_rU s rD identify_pathszMagika.identify_paths s %** L JKK K D$$$
4(E(E
d4jj)))) hdhhh ++F333rF contentc t |t s t dt | d | t t
j | S )z'Identify the content type of raw bytes.z-Input content should be of type 'bytes', not .)rX bytesr[ type_get_result_from_seekabler ioBytesIO)rB re s rD identify_byteszMagika.identify_bytes sb '5)) PW
PPP
--hrz'7J7J.K.KLLLrF streamc p t |t j r| st d t |t j rt d t |t j st d t |d r t |d rt |d st d | }| t | }| | n# | | w xY w|S )aR Identify the content type of a BinaryIO stream.
Identifies the content type from an already-open binary file-like object
(e.g., the output of `open(file_path, 'rb')`). Note: 1) Magika will
`seek()` around the stream; 2) the stream _is not closed_ (closing it is
the responsibility of the caller).
z0Input stream must be a readable BinaryIO object.z &"),, PFOO4E4E P NOO O fbm , , N
&""344 P NOO O '' T66** T 66** T
RSS S * %{{}} 33HV4D4DEEF
KK( ) ) ) )FKK( ) ) ) )
s 6D D3c " | j j }| j j }t j t j t j t j t j h}|D ]-}| || }|
| .t | S )a This method returns the list of all possible output content types.
I.e., all possible values for `MagikaResult.prediction.output.label`.
This considers the list of possible outputs from the model itself, but
also keeps into account additional configuration such as `override_map`
and special content types such as `empty` or `symlink`.
Consult the documentation for more details.
) r5 r:
overwrite_mapr DIRECTORYEMPTYSYMLINKTXTUNKNOWNgetaddsorted)rB r: r| output_content_typesct output_cts rD get_output_content_typesz Magika.get_output_content_types s # 0 D *8
& " $ $ 7
& 0 0B &))"b11I $ $Y / / / /*+++rF c x t j h}| | j j t | S )aD This method returns the list of all possible output of the model.
I.e., all possible values for `MagikaResult.prediction.dl.label`. Note
that, in general, the list of "model outputs" is different than the
"tool outputs" as in some cases the model is not even used, or the
model's output is overwritten due to a low-confidence score, or other
reasons. This API is useful mostly for debugging purposes; the vast
majority of client should use `get_output_content_types()`.
Consult the documentation for more details.
)r UNDEFINEDupdater5 r: r )rB model_content_typess rD get_model_content_typesz Magika.get_model_content_types s=
&6
""4#5#IJJJ)***rF c t S )zReturns the default model name.
This method is static so that it can be used by external clients/tests
without the need to instantiate a Magika object.
)_DEFAULT_MODEL_NAME rF rD r. z Magika._get_default_model_name s
#"rF content_types_kb_json_pathc d}d}d}i }t j | D ]\ }}|d }|r|}n|}|d |n|d } |d |n|d }
|d |n|d } |d } t t | | |
| | | |t | < |S )
Nz
text/plainzapplication/octet-streamunknownis_text mime_typegroup description
extensions)labelr r r r r )jsonloads read_textitemsr
r )
r
TXT_MIME_TYPEUNKNOWN_MIME_TYPE
UNKNOWN_GROUPoutct_namect_infor default_mime_typer r r r s
rD r> z Magika._load_content_types_kb s %
6!
$
& 0 0 2 2!
!
%'' GW i(G
6$1!!$5! ;'/ "! [)
&-W%5%=MM77CSE"=19w}?U
! .J-<&w//# '%
. . .C )) * *
rF model_config_pathc t j | }t |d |d |d |d |d |d |d |d d |d
D d |d D d
|d D S )Nbeg_sizemid_sizeend_sizeuse_inputs_at_offsetsmedium_confidence_thresholdmin_file_size_for_dl
padding_token
block_sizec , g | ]}t | S r r ).0ct_strs rD
z-Magika._load_model_config..@ s. ! ! !-3 ((! ! !rF r: c 4 i | ]\ }}t | |S r r r kvs rD
z-Magika._load_model_config..C s4 +/1a ##Q rF
thresholdsc N i | ]"\ }}t | t | #S r r r s rD r z-Magika._load_model_config..F s? Aq !##%5a%8%8 rF r| ) r r r r r r r r r: r r| )r r r r r )r r% s rD r4 zMagika._load_model_config3 s - 7 7 9 9::J'J'J'"()@"A(./L(M!'(>!? 1 l+! !7=>S7T! ! ! 39,3G3M3M3O3O "? 3 9 9 ; ;
rF c
t j }t j t j | j dg }dt j |z
z }| j d| j d|dd |S )NCPUExecutionProvider) providers zONNX DL model "z " loaded in .03f ms)timertdisable_telemetry_eventsInferenceSessionr/ r& r )rB
start_time onnx_session elapsed_times rD r@ zMagika._init_onnx_sessionL s Y[[
#%%%* -.
ty{{Z 78 Rd . R RL R R R R
rF content_typec | j | S rH )r? )rB r s rD _get_ct_infozMagika._get_ct_infoZ s |,,rF c V i }g }| j dt | d t j }|D ]J}| | \ }}|||t | < /|J | ||f Kdt j |z
z }| j d|dd | | D ]
\ } }
|
|| < g } |D ]*}| |t | +| S )ao Get results for a list of paths.
Given a list of paths, returns a list of MagikaResult objects, which
contain relevant information, such as: file path, the output of the DL
model, the confidence score, the output of the tool, and associated
metadata. The order of the predictions matches the order of the input
paths.
z3Processing input files and extracting features for z samplesNr z%First pass and features extracted in r r ) r& r lenr !_get_result_or_features_from_pathr2 ra _get_results_from_featuresr ) rB r_ all_outputs all_featuresr rU outputfeaturesr path_strry sorted_outputss rD rb z Magika._get_results_from_paths] sd 02 :< V#e** V V V
Y[[
6 6D # E Ed K K FH!)/ CII&& +++ ##T8$45555ty{{Z 78 V VVVVWWW !% ? ? M M S S U U + + Hf$*K ! ! : :D ! !+c$ii"8 9 9 9 9 rF c : | |g d S )Nr )rb r] s rD r\ z Magika._get_result_from_path s ++TF33A66rF seekablec | | \ }}||S |J | t d |fg d S )N-)%_get_result_or_features_from_seekabler r )rB r ry r s rD rj z Magika._get_result_from_seekable sX EEhOO M###..cH0E/FGGLLrF r r r r r r c ||k sJ |dk sJ ||k sJ |rJ t || j }|dk rG| d| }| }t ||| } ng } |dk rO| | j |z
| }
|
}
t |
|| } ng } t | g | g g g g S )a Extract features from an input seekable.
This implements features extraction v2 from a seekable, which is an
abstraction about anything that has a size and that can be "read_at" a
specific offset, such as a file or a buffer. This is implemented so that
we do not need to load the entire file in memory or scan the entire
buffer.
High-level overview on what we do:
- We read (at most) `block_size` bytes from the beginning and from the
end.
- We normalize these bytes by stripping whitespaces.
- We consider `beg_size` and `end_size` bytes as `beg` and `end`
features. If we don't have enough bytes, we use `padding_token` as
padding.
See comments below for the specifics and handling of corner cases.
NOTE: This implementation does not support extraction of `mid` features
and `use_inputs_at_offsets`.
r )begmidendoffset_0x8000_0x8007offset_0x8800_0x8807offset_0x9000_0x9007offset_0x9800_0x9807) minsizeread_atlstripr _get_beg_ints_with_paddingrstrip_get_end_ints_with_paddingr ) r r r r r r r bytes_num_to_read beg_contentbeg_ints end_contentend_intss rD _extract_features_from_seekablez&Magika._extract_features_from_seekable s< > *$$$$1}}}}*$$$$((((
HM : : a<<
#**1.?@@K%,,..K 88X} HH H a<< #**
113D K &,,..K 88X} HH H !#!#!#!#
rF r c |t | k r
| d| } t t t | }t | |k r||g|t | z
z z }t | |k sJ |S )a Take an (already-stripped) buffer as input and extract beg ints.
This returns a list of integers whose length is exactly beg_size. If
the buffer is bigger than required, take only the initial portion. If
the buffer is shorter, add padding at the end.
r r r8 r9 int)r r r r s rD r z!Magika._get_beg_ints_with_padding s c+&& & &%aj1KC--.. x==8 # # M?hX 6N#OPH8}}((((rF r c @ |t | k r'| t | |z
t | } t t t | }t | |k r|g|t | z
z |z }t | |k sJ |S )a Take an (already-stripped) buffer as input and extract end ints.
This returns a list of integers whose length is exactly end_size. If the
buffer is bigger than required, take only the last portion. If the
buffer is shorter, add padding at the beginning.
r )r r r r s rD r z!Magika._get_end_ints_with_padding s c+&& & &%c+&6&6&AC DTDT&TUKC--.. x==8 # #&8c(mm+CDPH8}}((((rF r c | | }t j |d }| j | }t j |d }d t ||| D S )N )axisc v g | ]6\ \ }}}}|t t | t | f7S ))r score)r r float)r rU _r r s rD r z;Magika._get_model_outputs_from_features.. sS
' q5% ;%5e%<%> & (:
+L* "&!G!G%+ )"(!1 "H " "GCII rF r r c ( t j }| j j || }||k r t j }| j t j k rn| j t j k r/|| j j
|| j j k rnw| j t j k r|| j j k rnQt j
}| | j r
t j }n t j }||k r t j }||fS rH )r NONEr5 r| r
OVERWRITE_MAPr< r
BEST_GUESSHIGH_CONFIDENCEr r MEDIUM_CONFIDENCELOW_CONFIDENCEr r r r r )rB r r r r s rD r z0Magika._get_output_label_from_dl_label_and_scoreD s +/ )7;;HhOO 8 # # . < N$= = =
!^%C C C!,00$,H
!^%E E E+GGG
/ = ..6
8 / 3 / 7 <''
$3#7 ---rF r r c t |t | | | | || S )N)dlr r r )rU
prediction)r r r )rB rU r r r r s rD r z(Magika._get_result_from_labels_and_score~ sX '$$X..(( 66!1
rF c > | j rE| r1| |t j t j d }|dfS | s t |t j dfS |
rt j |t j
s t |t j dfS t |d 5 }| t# | | cddd S # 1 swxY w Y nv| r1| |t j t j d }|dfS | |t j t j d }|dfS t+ d )a Given a path, we return either a MagikaOutput or a MagikaFeatures.
There are some files and corner cases for which we do not need to use
deep learning to get the output; in these cases, we already return a
MagikaOutput object.
For some other files, we do need to use deep learning, in which case we
return a MagikaFeatures object. Note that for now we just collect the
features instead of already performing inference because we want to use
batching.
?rU r r r N)rU statusrb unreachable)r=
is_symlinkr r r r existsr r FILE_NOT_FOUND_ERRORr3 rY accessR_OKPERMISSION_ERRORopenr r r1 r} r Exception)rB rU ry rn s rD r z(Magika._get_result_or_features_from_path s DOO$5$5 ;;)3 - 5 < F 4< {{}} U T&2MNNNPTT T <<>> 9T27++
#f6MNNNPTTT
$%% EE (($
[[]] ;;)3 - 7 < F 4< ;;)3 - 5 < F 4<
&&&s #DDDr c |j dk r1| |t j t j d }|dfS |j | j j k r6| d|j }| || }|dfS t
|| j j | j j | j j
| j j | j j | j j }|j | j j dz
| j j k rPt% |j | j j }| d| }| || }|dfS d|fS )ag Get result or features from a seekable object.
Given a Seekable object (which is a wrapper of BinaryIO), we return
either a MagikaOutput or a MagikaFeatures.
There are some corner cases for which we do not need to use deep
learning to get the output; in these cases, we return directly a
MagikaOutput object.
For all other cases, we do need to use deep learning, in which case we
return a MagikaFeatures object. Note that for now we just collect the
features instead of already performing inference because we want to use
batching.
r r r N)rU r )r r r r r~ r5 r r _get_result_from_few_bytesr r r r r r r r r r r" )rB r rU ry re
file_features
bytes_to_reads rD r z,Magika._get_result_or_features_from_seekable sm " =A ;;)3 - 3 < F 4<
]T / D
D
D &&q(-88G44W44HHF4< # B B"+"+"+"0"-"8 M !$"4"IA"MN%34 4 !$HM43E3P Q Q
"**1m<< 88t8LL t|#
]**rF c t | d| j j z k sJ | | }| |t
j |d S )N r r )r r5 r _get_label_from_few_bytesr r r )rB re rU r s rD r$ z!Magika._get_result_from_few_bytes se 7||q4#5#@ @@@@@..w7755%/ 6
rF c t j }| d }n # t $ r t j }Y nw xY w|S )Nzutf-8)r r decodeUnicodeDecodeErrorr )rB re r r s rD r) z Magika._get_label_from_few_bytes sN -$(Ew''AA! - - -$,EEE - s !$ ==r c t j }g }|D ]\ }}g }| j j dk r,| |j d| j j | j j dk r,| |j d| j j | j j dk r-| |j | j j d | | t j |t j }dt j |z
z }| j
d|dd g } |j d }
d} |
| z } |
| z dk r| dz
} t! | D ]}
| j
d |
dz d
| |
| z }t# |
dz | z |
}t j }| j d gd |||ddf i d }dt j |z
z }| j
d
|dd | | t j | S )zGet raw predictions from features.
Given a list of (path, features), return a (files_num, features_size)
matrix encoding the predictions.
r N)dtyper zDL input prepared in r r r z-Getting raw predictions for (internal) batch / target_labelrh zDL raw prediction in )r r5 r extendr r r r r ra r6 r7 int32r& r shaperanger rA run concatenate)rB r r X_bytesr fs sample_bytesXr raw_predictions_list samples_nummax_internal_batch_size batches_num batch_idx start_idxend_idxbatch_raw_predictionss rD r zMagika._get_raw_predictions s Y[[
) )EAr L!*Q.. ##BF+HT-?-H+H$IJJJ!*Q.. ##BF+HT-?-H+H$IJJJ!*Q.. ##BFD,>,G+G+I+I$JKKK NN< ( ( ( ( HWBH - - -ty{{Z 78 F FFFFGGG ! gaj "& !%<< 0 0A 5 5 1 K {++ ? ?I IOO[ ! [[k[[
"$;;I9q=,CC[QQG J$($6$:$: 7Ai.?.B,C"D% %% ! 49;;#; > > >~ 2333rF )B__name__rQ __qualname____doc__r r r r boolrE r2 rJ rN rL rM r rY rZ r r^ r r rd rh rm r rz r r r staticmethodr. r r
r> r r4 r r r@ r rb r\ r rj r r r r r r
r r r r r r r r r r r$ r) nptNDArrayr r rF rD r r 4 s %)*8*H $ B7 B7D>B7 (B7 B7
B7
B7 B7
B7 B7 B7 B7H# m m m m m)? + + + +$ #S # # # \# "$("
/ 0" " " \"H
d
{
\
0 B$7 -)9 -o - - - -, T$Z , D > ?8' 8' 8' 8'v 04tCyy<' <' <'(,<' x %x
'>> ?<' <' <' <'~ ,0499
$(
;K .4 U4 #6 78.4 .4 .4 .4 .4 .4 .4rF r )(rE rk r r( rY r pathlibr typingr r r r r r r
r numpyr6 numpy.typingrH onnxruntimer
magika.loggerr magika.typesr
r r r r r r r r r r r r r r rF rD rQ s
N N N N N N N N N N N N N N N N N N N N $ $ $ $ $ $
& V 4 V 4 V 4 V 4 V 4 V 4 V 4 V 4 V 4 V 4rF