.i}PddlmZddlmZmZddlZddlmZmZm Z m Z ddl Z ddl mZddlmZddlZddlmZmZerddlmZdd lmZmZdd Ze d dd Ze d dd Z d dd Z ddZddZ d ddZddZed d d dZ y)!) annotations)abc defaultdictN) TYPE_CHECKINGAny DefaultDictoverloadconvert_json_to_lines) set_module) DataFrameSeries)Iterable) IgnoreRaiseScalarcF|ddk(s |ddk(r|S|dd}t|S)zJ Helper function that converts JSON lists to line delimited JSON. r[]r )ss W/var/www/app/trading-bot/venv/lib/python3.12/site-packages/pandas/io/json/_normalize.pyconvert_to_line_delimitsr%s4 Q43;1R5C< !BA  ##cyNdsprefixseplevel max_levels rnested_to_recordr$2srcyrrrs rr$r$<src d}t|tr|g}d}g}|D]}tj|}|j D]\} } t| t s t | } |dk(r| } n||z| z} t| tr|!||k\r|dk7r|j | } | || <d|j | } |jt| | ||dz||j||r|dS|S)a A simplified json_normalize Converts a nested dict into a flat dict ("record"), unlike json_normalize, it does not attempt to extract a subset of the data. Parameters ---------- ds : dict or list of dicts prefix: the prefix, optional, default: "" sep : str, default '.' Nested records will generate names separated by sep, e.g., for sep='.', { 'foo' : { 'bar' : 0 } } -> foo.bar level: int, optional, default: 0 The number of levels in the json string. max_level: int, optional, default: None The max depth to normalize. Returns ------- d - dict or list of dicts, matching `ds` Examples -------- >>> nested_to_record( ... dict(flat1=1, dict1=dict(c=1, d=2), nested=dict(e=dict(c=1, d=2), d=2)) ... ) {'flat1': 1, 'dict1.c': 1, 'dict1.d': 2, 'nested.e.c': 1, 'nested.e.d': 2, 'nested.d': 2} FTrr) isinstancedictcopydeepcopyitemsstrpopupdater$append) rr r!r"r# singletonnew_dsdnew_dkvnewkeys rr$r$FsXI"dT F  a GGI QDAqa%Fz#) a&%%9*<A: ! A$%E&M ! A LL)!VS%!)YO P- Q.  e36ay Mrct|trD|jD]/\}}|||}|s|j|}t ||||1|S|||<|S)a3 Main recursive function Designed for the most basic use case of pd.json_normalize(data) intended as a performance improvement, see #15621 Parameters ---------- data : Any Type dependent on types contained within nested Json key_string : str New key (with separator(s) in) for data normalized_dict : dict The new normalized/flattened Json dict separator : str, default '.' Nested records will generate names separated by sep, e.g., for sep='.', { 'foo' : { 'bar' : 0 } } -> foo.bar data key_stringnormalized_dict separator)r'r(r+ removeprefix_normalize_json)r9r:r;r<keyvaluenew_keys rr>r>sz.$**, JC# YKu5G!..y9 " /#    '+ # rc |jDcic]\}}t|tr||}}}t|jDcic]\}}t|ts||c}}di|}i||Scc}}wcc}}w)aw Order the top level keys and then recursively go to depth Parameters ---------- data : dict or list of dicts separator : str, default '.' Nested records will generate names separated by sep, e.g., for sep='.', { 'foo' : { 'bar' : 0 } } -> foo.bar Returns ------- dict or list of dicts, matching `normalized_json_object` r8)r+r'r(r>)r9r<r4r5 top_dict_ nested_dict_s r_normalize_json_orderedrFs#'**,J$!QjD6IAJIJ"#zz| Ctq!z!T/Bad C L )i (< ((K CsA=A= B &B ci}t|trt||}|St|tr|Dcgc]}t ||}}|S|Scc}w)a An optimized basic json_normalize Converts a nested dict into a flat dict ("record"), unlike json_normalize and nested_to_record it doesn't do anything clever. But for the most basic use cases it enhances performance. E.g. pd.json_normalize(data) Parameters ---------- ds : dict or list of dicts sep : str, default '.' Nested records will generate names separated by sep, e.g., for sep='.', { 'foo' : { 'bar' : 0 } } -> foo.bar Returns ------- frame : DataFrame d - dict or list of dicts, matching `normalized_json_object` Examples -------- >>> _simple_json_normalize( ... { ... "flat1": 1, ... "dict1": {"c": 1, "d": 2}, ... "nested": {"e": {"c": 1, "d": 2}, "d": 2}, ... } ... ) {'flat1': 1, 'dict1.c': 1, 'dict1.d': 2, 'nested.e.c': 1, 'nested.e.d': 2, 'nested.d': 2} )r9r<r!)r'r(rFlist_simple_json_normalize)rr!normalized_json_objectrownormalized_json_lists rrJrJsdV "d!8bC!P "! B PRS 6s DSS## !! TsAc:|yt|try|D]}t|tr<|D]6}t|trtdt |j d|Ot|tr`tdt |j d|y)aD Validate that meta parameter contains only strings or lists of strings. Parameters ---------- meta : str or list of str or list of list of str or None The meta parameter to validate. Raises ------ TypeError If meta contains elements that are not strings or lists of strings. Nz9All elements in nested meta paths must be strings. Found z: zBAll elements in 'meta' must be strings or lists of strings. Found )r'r,rI TypeErrortype__name__)metaitemsubitems r_validate_metarUs |$  dD ! !'3/#!!%g!7!7 87+G  D#&d,,-Rx9  rpandasc t| d dfd dfd t|tr |j}nd}t|tr |s t St|t r|g}nvt|tjrVt|tsFt |}|D]5} t| t rdt| j} t| nt| ||t t||S|Ftd|Drt!|}t ||} | j#fd  } | St|ts|g}|g}nt|ts|g}|D cgc]} t| tr| n| gc} ggt%tD cgc]} j'| c} ddf d ||id t } | j#fd } j)D]\}}|||z}|| vrt+d|dt-j.|t0}|j2dkDr=t-j4t7|ft0}t9|D] \}} | ||< |j;| |<||j;| _| Scc} wcc} w)a> Normalize semi-structured JSON data into a flat table. This method is designed to transform semi-structured JSON data, such as nested dictionaries or lists, into a flat table. This is particularly useful when handling JSON-like data structures that contain deeply nested fields. Parameters ---------- data : dict, list of dicts, or Series of dicts Unserialized JSON objects. record_path : str or list of str, default None Path in each object to list of records. If not passed, data will be assumed to be an array of records. meta : list of paths (str or list of str), default None Fields to use as metadata for each record in resulting table. meta_prefix : str, default None String to prefix records with dotted path, e.g. foo.bar.field if meta is ['foo', 'bar']. record_prefix : str, default None String to prefix records with dotted path, e.g. foo.bar.field if path to records is ['foo', 'bar']. errors : {'raise', 'ignore'}, default 'raise' Configures error handling. * 'ignore' : will ignore KeyError if keys listed in meta are not always present. * 'raise' : will raise KeyError if keys listed in meta are not always present. sep : str, default '.' Nested records will generate names separated by sep. e.g., for sep='.', {'foo': {'bar': 0}} -> foo.bar. max_level : int, default None Max number of levels(depth of dict) to normalize. if None, normalizes all levels. Returns ------- DataFrame The normalized data, represented as a pandas DataFrame. See Also -------- DataFrame : Two-dimensional, size-mutable, potentially heterogeneous tabular data. Series : One-dimensional ndarray with axis labels (including time series). Examples -------- >>> data = [ ... {"id": 1, "name": {"first": "Coleen", "last": "Volk"}}, ... {"name": {"given": "Mark", "family": "Regner"}}, ... {"id": 2, "name": "Faye Raker"}, ... ] >>> pd.json_normalize(data) id name.first name.last name.given name.family name 0 1.0 Coleen Volk NaN NaN NaN 1 NaN NaN NaN Mark Regner NaN 2 2.0 NaN NaN NaN NaN Faye Raker >>> data = [ ... { ... "id": 1, ... "name": "Cole Volk", ... "fitness": {"height": 130, "weight": 60}, ... }, ... {"name": "Mark Reg", "fitness": {"height": 130, "weight": 60}}, ... { ... "id": 2, ... "name": "Faye Raker", ... "fitness": {"height": 130, "weight": 60}, ... }, ... ] >>> pd.json_normalize(data, max_level=0) id name fitness 0 1.0 Cole Volk {'height': 130, 'weight': 60} 1 NaN Mark Reg {'height': 130, 'weight': 60} 2 2.0 Faye Raker {'height': 130, 'weight': 60} Normalizes nested data up to level 1. >>> data = [ ... { ... "id": 1, ... "name": "Cole Volk", ... "fitness": {"height": 130, "weight": 60}, ... }, ... {"name": "Mark Reg", "fitness": {"height": 130, "weight": 60}}, ... { ... "id": 2, ... "name": "Faye Raker", ... "fitness": {"height": 130, "weight": 60}, ... }, ... ] >>> pd.json_normalize(data, max_level=1) id name fitness.height fitness.weight 0 1.0 Cole Volk 130 60 1 NaN Mark Reg 130 60 2 2.0 Faye Raker 130 60 >>> data = [ ... { ... "id": 1, ... "name": "Cole Volk", ... "fitness": {"height": 130, "weight": 60}, ... }, ... {"name": "Mark Reg", "fitness": {"height": 130, "weight": 60}}, ... { ... "id": 2, ... "name": "Faye Raker", ... "fitness": {"height": 130, "weight": 60}, ... }, ... ] >>> series = pd.Series(data, index=pd.Index(["a", "b", "c"])) >>> pd.json_normalize(series) id name fitness.height fitness.weight a 1.0 Cole Volk 130 60 b NaN Mark Reg 130 60 c 2.0 Faye Raker 130 60 >>> data = [ ... { ... "state": "Florida", ... "shortname": "FL", ... "info": {"governor": "Rick Scott"}, ... "counties": [ ... {"name": "Dade", "population": 12345}, ... {"name": "Broward", "population": 40000}, ... {"name": "Palm Beach", "population": 60000}, ... ], ... }, ... { ... "state": "Ohio", ... "shortname": "OH", ... "info": {"governor": "John Kasich"}, ... "counties": [ ... {"name": "Summit", "population": 1234}, ... {"name": "Cuyahoga", "population": 1337}, ... ], ... }, ... ] >>> result = pd.json_normalize( ... data, "counties", ["state", "shortname", ["info", "governor"]] ... ) >>> result name population state shortname info.governor 0 Dade 12345 Florida FL Rick Scott 1 Broward 40000 Florida FL Rick Scott 2 Palm Beach 60000 Florida FL Rick Scott 3 Summit 1234 Ohio OH John Kasich 4 Cuyahoga 1337 Ohio OH John Kasich >>> data = {"A": [1, 2]} >>> pd.json_normalize(data, "A", record_prefix="Prefix.") Prefix.0 0 1 1 2 Returns normalized data with columns prefixed with the given string. c|} t|tr|D]}| t|||} |S||} |S#t$rD}|rtd|d|dk(rtjcYd}~Std|d|d|d}~wwxYw)zInternal function to pull fieldNzKey zS not found. If specifying a record_path, all elements of data should have the path.ignorez) not found. To replace missing values of z% with np.nan, pass in errors='ignore')r'rIKeyErrornpnan)jsspecextract_recordresultfieldeerrorss r _pull_fieldz#json_normalize.._pull_fields $%!+E~&uo-#E]F+(    1#12!vv 1#FqcJ67 s%)99 B&B(B.BBc||d}t|ts=tj|rg}|St dt |j d||S)z Internal function to pull field for records, and similar to _pull_field, but require to return list. And will raise error if has non iterable value. T)r_z(Path must contain list or null, but got z at )r'rIpdisnullrOrPrQ)r]r^r`rds r _pull_recordsz%json_normalize.._pull_recordssn Rd;&$'yy    #F|445T$C rNz.All items in data must be of type dict, found rH)indexc3~K|]0}|jDcgc]}t|tc}2ycc}wwr)valuesr'r().0yxs r z!json_normalize../s+GQQXXZ8 1d#8G8s =8 =r!r#c|Srrrn record_prefixs rz json_normalize..:s qc5Jr)columnsrc p t|tr|g}t|dkDrW|D]Q}t dD]&\}}|dzt|k(s ||d||<( ||d|dd||dzSy|D]} ||d}|Dcgc]"}t|trt |n|$}}j t|t dD]<\}}|dzt|kDr||} n  |||d} |j | >j |ycc}w)NrT)strictrrr"rp)r'r(lenzipr$r/extend)r9path seen_metar"objvalr?recsrmeta_val_metardrh_recursive_extractlengthsr# meta_keys meta_valsrecordsr!s rrz*json_normalize.._recursive_extractMsr dD !6D t9q= W #E9T BCHCqyCH,)4S#b')B #C#3tAw<ab9ETUIV  W %$S$q'2 " "!T*%QC9Es4y) #E9T B4HCqy3s8+#,S>#.sCK#@cN))(3 4 t$# %s'D3rxc|Srrrrs rrtz json_normalize..psM?1#1FrzConflicting metadata name z, need distinguishing prefix )dtyper)F)r]dict[str, Any]r^ list | strr_boolreturnzScalar | Iterable)r]rr^rrrI)r)r"intrNone)rUr'rrirIr r(rrr,rPrQrONotImplementedErrorrJanyr$renamerjoinr+ ValueErrorr[arrayobjectndimemptyry enumeraterepeat)r9 record_pathrR meta_prefixrsrcr!r#rirSmsgr`mrr4r5rkirrdrhrrrrrs ```` @@@@@@@@rjson_normalizer-sT4FK ",>B :($ $d{ D$ v D#,, ' 40EDz %DdD)!$Z0013 n$  %"!  L    !  /#>eLL G$G G$DcYGD4u-  $]]+J]KF  T *"m  | d #v8< =1*Q%QA3 . =EGG(.I*/03#0I%%<t["A6 w F 'FG!+1  "aA ;,QC/LM  !6* ;;?XXs1viv6F#A, 3q  MM'*q %+& ||G, MI >1s K!K&)rr,rr,)....) rr(r r,r!r,r"rr# int | Nonerr) rz list[dict]r r,r!r,r"rr#rrzlist[dict[str, Any]])rC.rN) rdict | list[dict]r r,r!r,r"rr#rrz%dict[str, Any] | list[dict[str, Any]]) r9rr:r,r;rr<r,rr)r9rr<r,rr)r)rrr!r,rzdict | list[dict] | Any)rR"str | list[str | list[str]] | Nonerr)NNNNraiserN)r9zdict | list[dict] | Seriesrzstr | list | NonerRrr str | Nonersrrcrr!r,r#rrr )! __future__r collectionsrrr)typingrrrr numpyr[pandas._libs.writersr pandas.util._decoratorsr rVrfr rcollections.abcrpandas._typingrrrr$r>rFrJrUrrrrrs># 6. ( $                        NN N N  N  N + Nb& &&$& &  &R)62"2" 2"2"j> H&*/3" $! Z $Z"Z -Z Z  Z  Z ZZZZr