o h @sddlZddlZddlZddlmZddlmZmZddlmZddl m Z ddl m Z ddl mZmZmZmZmZmZmZmZddlmZmZddlZdd lmZmZmZdd lmZdd l m!Z!d d l"m#Z#d dl$m%Z%m&Z&d dl'm(Z(m)Z)m*Z*d dl+m,Z,m-Z-d dl.m/Z/m0Z0m1Z1d dl2m3Z3m4Z4m5Z5e6dej7Z8eGdddZ9Gdddej:Z;GdddejGdddejZ$ded efd?d@Z%e&dfdNfdAdB Z'e(dCdDZ)dEdFZ*Z+S)O HfFileSystema8 Access a remote Hugging Face Hub repository as if were a local file system. [`HfFileSystem`] provides fsspec compatibility, which is useful for libraries that require it (e.g., reading Hugging Face datasets directly with `pandas`). However, it introduces additional overhead due to this compatibility layer. For better performance and reliability, it's recommended to use `HfApi` methods when possible. Args: token (`str` or `bool`, *optional*): A valid user access token (string). Defaults to the locally saved token, which is the recommended method for authentication (see https://huggingface.co/docs/huggingface_hub/quick-start#authentication). To disable authentication, pass `False`. endpoint (`str`, *optional*): Endpoint of the Hub. Defaults to . Usage: ```python >>> from huggingface_hub import HfFileSystem >>> fs = HfFileSystem() >>> # List files >>> fs.glob("my-username/my-model/*.bin") ['my-username/my-model/pytorch_model.bin'] >>> fs.ls("datasets/my-username/my-dataset", detail=False) ['datasets/my-username/my-dataset/.gitattributes', 'datasets/my-username/my-dataset/README.md', 'datasets/my-username/my-dataset/data.json'] >>> # Read/write files >>> with fs.open("my-username/my-model/pytorch_model.bin") as f: ... data = f.read() >>> with fs.open("my-username/my-model/pytorch_model.bin", "wb") as f: ... f.write(data) ``` r/hfNendpointtokenrErFcs<tj|i||p tj|_||_t||d|_i|_dS)NrD) super__init__rENDPOINTrErFr _api_repo_and_revision_exists_cache)r7rErFargsstorage_options __class__r9r:rHfs  zHfFileSystem.__init__r'r(r)r.c Cs|||f|jvrqz |jj|||tjdWnKttfy=}zd|f|j|||f<d|f|j||df<WYd}~n8d}~wty`}zd|f|j|||f<d|j||df<WYd}~nd}~wwd|j|||f<d|j||df<|j|||fS)N)r)r'timeoutF)TN)rKrJ repo_inforHF_HUB_ETAG_TIMEOUTrr#r)r7r'r(r)er9r9r:_repo_and_revision_existxs"   z%HfFileSystem._repo_and_revision_existpathcCspdttdttdttfdd}||}|std|dddtjvr@d|vr2td|dd \}}tj|}ntj }| ddkrd |vr|d d \}}d|vrt |}|d ury|d | fvryt d |d}| }n |dd \}}nd }|t||}||||\} } | st|| nd }d|dd d } d|dd d } |dd} d|dd d }| }| }||||\} } | st| ttfr| }|}||||\} }| st|| n1t|| n+|}d }d |vr|d d \}}|t||}nd }||||\} }| s%td|d ur,|ntj}t|||||dS)a Resolve a Hugging Face file system path into its components. Args: path (`str`): Path to resolve. revision (`str`, *optional*): The revision of the repo to resolve. Defaults to the revision specified in the path. Returns: [`HfFileSystemResolvedPath`]: Resolved path information containing `repo_type`, `repo_id`, `revision` and `path_in_repo`. Raises: `ValueError`: If path contains conflicting revision information. `NotImplementedError`: If trying to list repositories. revision_in_pathr)r.cSs:|dur|dur||krtd|d|d|S|}|S)NzRevision specified in path ("z ") and in `revision` argument ("z") are not the same.) ValueError)rVr)r9r9r:%_align_revision_in_path_with_revisionszHHfFileSystem.resolve_path.._align_revision_in_path_with_revisionz0Access to repositories lists is not implemented.r1rrr0Nr/)r-)r r@_strip_protocolNotImplementedErrorsplitrr2valuesREPO_TYPES_MAPPINGREPO_TYPE_MODELcountSPECIAL_REFS_REVISION_REGEXsearchgroupsublstriprrT_raise_file_not_foundjoin isinstancerr#r5r&)r7rUr)rXr'r(rVmatchr*repo_and_revision_existerrrepo_id_with_namespacepath_in_repo_with_namespacerepo_id_without_namespacepath_in_repo_without_namespace_r9r9r: resolve_pathsx        zHfFileSystem.resolve_pathcCs|s|j|jdS||}|}|r'|j|d||}|s|jsE|j|j|j dfd|j|j|j |j fddSdS)ac Clear the cache for a given path. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.invalidate_cache). Args: path (`str`, *optional*): Path to clear from cache. If not provided, clear the entire cache. N) dircacheclearrKrqr;pop_parentr*r'r(r))r7rU resolved_pathr9r9r:invalidate_caches   zHfFileSystem.invalidate_cacherbmode block_sizeHfFileSystemFilecKsLd|vrtd|dkrt||f|||d|St||f|||d|S)Naz/Appending to remote files is not yet supported.r)ryr)rz)r[HfFileSystemStreamFiler{)r7rUryr)rzkwargsr9r9r:_opens zHfFileSystem._openc KsR|j||d}|jj|j|j|j|j|j|d|dd|j | ddS)Nr)commit_messagecommit_description)r*r(rFr'r)rrrU) rqrJ delete_filer*r(rFr'r)r3rwr;)r7rUr)r~rvr9r9r:_rms zHfFileSystem._rmF recursivemaxdepthc sj||d}j||||d}fdd|D}dd|D} d|d} | |r+dnd 7} | |d ur9d |dnd 7} jj|j|jj| |j|d | |d dj | dd S)a Delete files from a repository. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.rm). Note: When possible, use `HfApi.delete_file()` for better performance. Args: path (`str`): Path to delete. recursive (`bool`, *optional*): If True, delete directory and all its contents. Defaults to False. maxdepth (`int`, *optional*): Maximum number of subdirectories to visit when deleting recursively. revision (`str`, *optional*): The git revision to delete from. r)rrr)cs"g|] }|s|jqSr9)isdirrqr*).0rUr7r9r: B"z#HfFileSystem.rm..cSsg|]}t|dqS))r*)r)rr*r9r9r:rCzDelete  z recursively r/Nz up to depth rr)r(r'rF operationsr)rrr) rq expand_pathrJ create_commitr(r'rFr)r3rwr;) r7rUrrr)r~rvpaths paths_in_reporrr9rr:rm"s"   zHfFileSystem.rmTdetailrefreshc s|j|d}|d|i|}z|jf||d|}Wn4tyS|js.td|j|f||d|}fdd|D}t|dkrQtdYnw|rX|Sdd|DS) a List the contents of a directory. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.ls). Note: When possible, use `HfApi.list_repo_tree()` for better performance. Args: path (`str`): Path to the directory. detail (`bool`, *optional*): If True, returns a list of dictionaries containing file information. If False, returns a list of file paths. Defaults to True. refresh (`bool`, *optional*): If True, bypass the cache and fetch the latest data. Defaults to False. revision (`str`, *optional*): The git revision to list from. Returns: `List[Union[str, Dict[str, Any]]]`: List of file paths (if detail=False) or list of file information dictionaries (if detail=True). r expand_info)rr)Ncg|] }|dkr|qSnamer9rorr9r:rzz#HfFileSystem.ls..rcSsg|]}|dqSrr9rr9r9r:r}s)rqr;_ls_treerr*rfrulen)r7rUrrr)r~rvoutr9rr:lsSs     zHfFileSystem.lsrc sNj||d}|}t|j|j|jd|jd}g}|jvr|sшj|} || g} |rgt dd| D} | rg| } | djvrO| | dnj| d} || | dd| D| s.rcSrrr9rr9r9r:rrcs&g|]}|ddur|dqS) last_commitNr)rurrr9r:rs&r1cs"g|] }|dds|qS)rr1) startswithr) common_pathr9r:rrT)rrr)r)rexpandr)r'filersizerblob_idlfsrsecurityrrrrrtree_idr)"rqr;r&r'r(r)r-rrextendrpopleftappendosrU commonprefixendswithrr4rurrtrrJlist_repo_treer*rhr"rrrrrr setdefault)r7rUrrr)rrv root_pathrcached_path_infosdirs_not_in_dircache dirs_to_visitdir_infodirs_not_expanded common_prefix cached_pathtreercache_path_info parent_pathr9)rr7r:rs        (    zHfFileSystem._ls_treec/sRd|ddi|}|j||dd}tj|g|Ri|EdHdS)a Return all files below the given path. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.walk). Args: path (`str`): Root path to list files from. Returns: `Iterator[Tuple[str, List[str], List[str]]]`: An iterator of (path, list of directory names, list of file names) tuples. rrFr)rN)r3rqr;rGwalk)r7rUrLr~rNr9r:rs$zHfFileSystem.walkc s@d|ddi|}|j||dd}tj|fi|S)ah Find files by glob-matching. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.glob). Args: path (`str`): Path pattern to match. Returns: `List[str]`: List of paths matching the pattern. rrFr)r)r3rqr;rGglobr7rUr~rNr9r:rszHfFileSystem.globwithdirsc  s|rtj|f|||||d|S|j||d}|}d|i|}z|j|fd||jd|WntyR|j|fd|i|ddkrN|iiniYn+w|s]d d Dn|j|fd|ji|} | dd krt| gnd d Dt} |s| Sfdd | DS)a List all files below path. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.find). Args: path (`str`): Root path to list files from. maxdepth (`int`, *optional*): Maximum depth to descend into subdirectories. withdirs (`bool`, *optional*): Include directory paths in the output. Defaults to False. detail (`bool`, *optional*): If True, returns a dict mapping paths to file information. Defaults to False. refresh (`bool`, *optional*): If True, bypass the cache and fetch the latest data. Defaults to False. revision (`str`, *optional*): The git revision to list from. Returns: `Union[List[str], Dict[str, Dict[str, Any]]]`: List of paths or dict of file information. )rrrrr)rrT)rrr)r)rrcSsg|] }|ddkr|qSrr9rr9r9r:r<rz%HfFileSystem.find..rcSsi|]}|d|qSrr9rr9r9r: Arz%HfFileSystem.find..ci|]}||qSr9r9)rrrr9r:rFr) rGfindrqr;rr)rinfosorted) r7rUrrrrr)r~rvrnamesrNrr:r s8      zHfFileSystem.findpath1path2c Ks,|j||d}|j||d}|j|jko|j|jk}|rEd|d|}|jj|j|j|j|d||ddt|j|j|jdgdn?|j |d |jd } | } Wd n1s]wYd|d|}|jj | |j|j|j |j|j|d||dd |j |d |j |d d S) a Copy a file within or between repositories. Note: When possible, use `HfApi.upload_file()` for better performance. Args: path1 (`str`): Source path to copy from. path2 (`str`): Destination path to copy to. revision (`str`, *optional*): The git revision to copy from. rzCopy z to rrr/)src_path_in_repor* src_revision)r(r'r)rrrrxNpath_or_fileobjr*r(rFr'r)rrr)rqr'r(rJrr)r3rr*openread upload_filerFrwr;) r7rrr)r~resolved_path1resolved_path2 same_reporfcontentr9r9r:cp_fileHsF     zHfFileSystem.cp_filecKs|j|fi|}|ddS)ai Get the last modified time of a file. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.modified). Args: path (`str`): Path to the file. Returns: `datetime`: Last commit date of the file. rdater)r7rUr~rr9r9r:modifieds zHfFileSystem.modifiedc  s|j|d}||dd}|js<ddd|r;|jj|j|j|jdd}id t |j |j |j d d nd | }|sQ||jvrQ|j|d d ||jvrmfdd|j|D} | sitd | d|s}d us}|rڈrڈdd ur|jj|j|j||j|jd} | std | d} t|j|j|jd|jd} t| tr| d| j| jd| j| j| j| jdn| d| jdd| j| jd|sڇfdddDd usJS)a Get information about a file or directory. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.info). Note: When possible, use `HfApi.get_paths_info()` or `HfApi.repo_info()` for better performance. Args: path (`str`): Path to get info for. refresh (`bool`, *optional*): If True, bypass the cache and fetch the latest data. Defaults to False. revision (`str`, *optional*): The git revision to get info from. Returns: `Dict[str, Any]`: Dictionary containing file information (type, size, commit info, etc.). rrTrr)rrr)r'r)N)oidtitler)rrFrcrrr9rrr9r:rrz%HfFileSystem.info..r)rr)r'r/rr1rrrcrr9r9)rkrr9r:rrz%HfFileSystem.info..)rqr;r3r*rJlist_repo_commitsr(r'r)r! commit_idr created_atrurrrrfget_paths_infor&r-rhr"rUrrrrrr) r7rUrr)r~rvrrrout1 paths_inforrr9)rrUr:rs           zHfFileSystem.infocKsHz|ddr |||j|fii|ddiWdSYdS)a Check if a file exists. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.exists). Note: When possible, use `HfApi.file_exists()` for better performance. Args: path (`str`): Path to check. Returns: `bool`: True if file exists, False otherwise. rFrT)r3rwrrr9r9r:existss  zHfFileSystem.existscCs.z |j|ddddkWStyYdSw)ai Check if a path is a directory. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.isdir). Args: path (`str`): Path to check. Returns: `bool`: True if path is a directory, False otherwise. Frrr)rOSErrorr7rUr9r9r:rs   zHfFileSystem.isdircCs&z |j|ddddkWSYdS)a` Check if a path is a file. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.isfile). Args: path (`str`): Path to check. Returns: `bool`: True if path is a file, False otherwise. Frrrrrr9r9r:isfile%s zHfFileSystem.isfilecCsB||}t|j|j|j|j|jd}||r|ddd}|S)z Get the HTTP URL of the given path. Args: path (`str`): Path to get URL for. Returns: `str`: HTTP URL to access the file or directory on the Hub. )r'r)rEz /resolve/z/tree/r) rqrr(r*r'r)rErreplace)r7rUrvurlr9r9r:r7s  zHfFileSystem.urlc sb|d}t|dh}t|ttfrt|dkr)tj||f||d|St |r0|}n| |r>t j |dddSt|t tfrPt j t j|ddd}|dur]t|d}d}|} |j||d } |j||d d } || z2tt| j| j| j| j|jd ||| d|jt|tr|jndd | | W|r|!dSdS|r|!ww) a^ Copy single remote file to local. Note: When possible, use `HfApi.hf_hub_download()` for better performance. Args: rpath (`str`): Remote path to download from. lpath (`str`): Local path to download to. callback (`Callback`, *optional*): Optional callback to track download progress. Defaults to no callback. outfile (`IO`, *optional*): Optional file-like object to write to. If provided, `lpath` is ignored. r)r)callbackoutfileT)exist_okNFwbrrr(r)filenamer'rE)r temp_filedisplayed_filename expected_size resume_sizeheaders _tqdm_bar)"r3setkeysrhrrrrGget_filerrrmakedirsr@rrUdirnamertellrqrset_sizerrr(r)r*r'rErJ_build_hf_headerstqdmseekclose) r7rpathlpathrrr~r)unhandled_kwargs close_file initial_posresolve_remote_pathrrNr9r:rNsR       zHfFileSystem.get_filecCtd)zA context within which files are committed together upon exit Requires the file class to implement `.commit()` and `.discard()` for the normal and exception cases. (Transactional commits are not supported.r[rr9r9r: transactions zHfFileSystem.transactioncCr)z@Begin write transaction for deferring files, non-context versionrrrr9r9r:start_transactionszHfFileSystem.start_transactionN)rxNN)FNN)TFN)FFNT)NFFFN)FNr.N),r<r=r>r? root_markerprotocolr r@rboolrHr ExceptionrTr&rqrwintrrrr r rrrr rrrrrrrrrrrrrpropertyrr __classcell__r9r9rNr:rB:s(  _  2 / ,h  =;(dG  rBcseZdZddededeeffdd ZfddZd ed ed e fd d Z dddZ dde d dfddZ dfdd Zd efddZZS)r{NfsrUr)c sz |j||d|_Wnty'}zd|ddvr"t|d|d}~ww|dddkr;|j|jdd|_tj||jfi||dS) Nrwryr/B. Make sure the repository and revision exist before writing data.rxFr) rqrvFileNotFoundErrorr3rr;detailsrGrH)r7r!rUr)r~rSrNr9r:rHs zHfFileSystemFile.__init__ct|dsdStSNrvhasattrrG__del__rrNr9r:r*  zHfFileSystemFile.__del__startendr.cCsldd|d|di|jj}t|jj|jj|jj|jj|jj d}t d||dt j d}t ||jS) Nrangezbytes=-rrGETiiii)rretry_on_status_codesrP)r!rJr rrvr(r)r*r'rEr%rHF_HUB_DOWNLOAD_TIMEOUTr$r)r7r,r-rrrr9r9r: _fetch_ranges( zHfFileSystemFile._fetch_rangecCstjddd|_dS)Nzhffs-F)prefixdelete)tempfileNamedTemporaryFilerrr9r9r:_initiate_uploadsz!HfFileSystemFile._initiate_uploadFfinalc Cs|jd|j}|j||rN|j|jjj|jj |j j |j j |jj |j j|j j|jd|jddt|jj |jj|j ddSdS)Nrrrrr)bufferr rrwriter r!rJrrrvr*r(rFr'r)r~r3rremoverwr;)r7r;blockr9r9r: _upload_chunks(        zHfFileSystemFile._upload_chunkrcsl|jdkr0|dus |dkr0|jdkr0|jj|jddd }|WdS1s+wYt|S)a)Read remote file. If `length` is not provided or is -1, the entire file is downloaded and read. On POSIX systems and if `hf_transfer` is not enabled, the file is loaded in memory directly. Otherwise, the file is downloaded to a temporary file and read from there. rxNrr)rz)rylocr!rrUrrG)r7lengthrrNr9r:rs $ zHfFileSystemFile.readcC|j|jSrr!rrUrr9r9r:rzHfFileSystemFile.urlrr)Fr)r<r=r>rBr@r rHr*rbytesr5r:rr@rrr r9r9rNr:r{s    r{c seZdZ    ddedededeed ed ef fd d Zdd edefddZddefddZ defddZ fddZ ddZ Z S)r}rxNrnoner!rUryr)rz cache_typec s|dkr td||dkrtd|d|vr"td|dz |j||d|_Wn"tyN}zd|d d vrDt|d |WYd}~nd}~ww|jdd |_tj||jf|||d |d|_ |dS)Nrz:HfFileSystemStreamFile only supports block_size=0 but got rHz?HfFileSystemStreamFile only supports cache_type='none' but got r"z;HfFileSystemStreamFile only supports reading but got mode=''rryr/r#)rrryrzrI) rWrqrvr$r3r;r%rGrHresponse) r7r!rUryr)rzrIr~rSrNr9r:rHs8   zHfFileSystemStreamFile.__init__rAwhencecCs2|dkr |dkr dS||jkr|dkrdStd)NrrzCannot seek streaming HF file)rArW)r7rArMr9r9r:r s zHfFileSystemStreamFile.seekrrBc CsJ|dkr|fnd}|jdus|jjrrBr@r rrHr rrr*rVr r9r9rNr:r}s.1 r}r)r.cCst|r|St|Sr)rari safe_quoterr9r9r:r6`sr6scCs t|ddS)Nr/)safe)r)rXr9r9r:rWds rWrUrkcCsN|}t|tr |d}nt|tr|d}n t|tr"|d}t||)Nz (repository not found)z (revision not found)z (invalid repository id))rhrrr#r$)rUrkmsgr9r9r:rfhs       rfr!ryrzrIcCs|j||||dS)NrK)r)r!rUryrzrIr9r9r:rSssrS)Grrer8 collectionsr dataclassesrrr itertoolsrpathlibrtypingrr r r r r rr urllib.parserrfsspecfsspec.callbacksrrr fsspec.utilsrrequestsrr/r _commit_apirrerrorsrrr file_downloadrrhf_apir r!r"utilsr#r$r%compileVERBOSErar&AbstractFileSystemrBspecAbstractBufferedFiler{r}r@r6rWrrfrrSr9r9r9r:sN    (    sTe"