o h'*-@sUddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z ddlmZmZmZmZmZmZmZmZddlmZmZddlZddlmZmZddlmZm Z m!Z!dd lm"Z"m#Z#dd l$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+dd l,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;mZ>m?Z?m@Z@mAZAmBZBmCZCmDZDmEZEmFZFmGZGdd lHmIZIdd lJmKZKmLZLddlMmNZNddlOmPZPddlQmRZReBSeTZUeVZWeZXeYdZZeYdZ[eYdZ\iZ]ee^e_fe`d<d}dee^e dfde_fddZae ddGdddZbeGdddddde^de^d ee^d!ee^d"ee^d#ee^de^fd$d%Zcd&d'd(eNd)e^d*e_dejdfd+d,Zedddddd-dd.d)e^d/ed0eed1efd2eee^efd3eefd4ee^d5efd6eeFddfd7d8Zgdddd9d:e d;e0d2ee^e^fd3eefd4ee^d6eeFddfdee^dee^fd?d@Zid~dAe^dBe^dCe_ddfdDdEZjd~dAe^dBe^dCe_ddfdFdGZkdHe^d"e^dIe^ddfdJdKZleGde^d!e^de^fdLdMZmd3efdNee^e fddfdOdPZneGddddddddd&dejodd&dddddQdRde^de^d ee^d!ee^d"ee^dSee^dTee^dee^e dfdUee^e dfdVeee^dfdWe_d0eedXepdYee_e^dfdZe_d2eee^e^fd#ee^d[ee_d\ee^d]ee_edQfde^f*d^d_Zqde^de^de^d!e^d"e^d#ee^dXepd2ee^e^fd0eedYeee_e^fdZe_dWe_de^fd`daZrdUee^e fde^d!e^de^d"e^d#ee^dXepd2ee^e^fd0eedYee_e^dfde^dWe_dZe_de^fdbdcZseG   dde^de^dee^e dfd"ee^d!ee^dee^eXdff dddeZteGddejuddddfd)e^dYee_e^dfd0eedfeepdSee^dTee^dVeee^dfd2eee^e^fdebfdgdhZvdddide^de^d!e^d"e^d#ee^d0eedXeepd2ee^e^fdYee_e^dfdZe_djee^dHee^deedddddewfee^e^e^efee0dfffdkdlZxdmewdWe_dZe_defdndoZyd:e dpe dqe^d0eed2ee^e^fd3eefde^dWe_d>ee^d;ee0ddfdrdsZzdtee^deeffdudvZ{dAe dBe ddfdwdxZ|dAe^dBe^ddfdydzZ}dHe^d"e^dje^de^fd{d|Z~dS)N) dataclass)Path)AnyBinaryIODictLiteralNoReturnOptionalTupleUnion)quoteurlparse) __version__ constants)get_local_download_pathsread_download_metadatawrite_download_metadata)HUGGINGFACE_CO_URL_TEMPLATEHUGGINGFACE_HUB_CACHE)EntryNotFoundErrorFileMetadataErrorGatedRepoErrorHfHubHTTPErrorLocalEntryNotFoundErrorRepositoryNotFoundErrorRevisionNotFoundError)OfflineModeIsEnabledSoftTemporaryDirectory WeakFileLock XetFileDatabuild_hf_headersget_fastai_versionget_fastcore_versionget_graphviz_versionget_jinja_versionget_pydot_version get_sessionget_tf_versionget_torch_versionhf_raise_for_statusis_fastai_availableis_fastcore_availableis_graphviz_availableis_jinja_availableis_pydot_availableis_tf_availableis_torch_availablelogging!parse_xet_file_data_from_responserefresh_xet_connection_inforeset_sessionstqdmvalidate_hf_hub_args)_adjust_range_header) _PY_VERSIONis_xet_available) HTTP_METHOD_T) sha_fileobj)_get_progress_bar_contextzfilename="(?P.*?)";z^[0-9a-f]{40}$z^[0-9a-f]{64}$_are_symlinks_supported_in_dir cache_dirreturnc Cs|durtj}tt|}|tvrdt|<tj|ddt |d^}t|d}| t|d}tj j |tj |d}zt||Wn$tyndt|<tjsld |d }tjd krg|d 7}t|Yn wWdt|SWdt|S1swYt|S) aReturn whether the symlinks are supported on the machine. Since symlinks support can change depending on the mounted disk, we need to check on the precise cache folder. By default, the default HF cache directory is checked. Args: cache_dir (`str`, `Path`, *optional*): Path to the folder where cached files are stored. Returns: [bool] Whether symlinks are supported in the directory. NTexist_ok)dirdummy_file_srcdummy_file_dst)startFz`huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in a%. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.nta To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development)r HF_HUB_CACHEstrr expanduserresolver>osmakedirsrtouchpathrelpathdirnamesymlinkOSErrorHF_HUB_DISABLE_SYMLINKS_WARNINGnamewarningswarn)r?tmpdirsrc_pathdst_path relative_srcmessager]q/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/huggingface_hub/file_download.pyare_symlinks_supportedZsF        " ""r_T)frozenc@sJeZdZUdZeeed<eeed<eed<eeed<eeed<dS)HfFileMetadataaData structure containing information about a file versioned on the Hub. Returned by [`get_hf_file_metadata`] based on a URL. Args: commit_hash (`str`, *optional*): The commit_hash related to the file. etag (`str`, *optional*): Etag of the file on the server. location (`str`): Location where to download the file. Can be a Hub url or not (CDN). size (`size`): Size of the file. In case of an LFS file, contains the size of the actual LFS file, not the pointer. xet_file_data (`XetFileData`, *optional*): Xet information for the file. This is only set if the file is stored using Xet storage. commit_hashetaglocationsize xet_file_dataN) __name__ __module__ __qualname____doc__r rI__annotations__intr r]r]r]r^ras    ra) subfolder repo_typerevisionendpointrepo_idfilenamermrnrorpcCs|dkrd}|dur|d|}|tjvrtd|tjvr&tj||}|dur-tj}tj|t|ddt|d}|durP|tj rP||t tj d}|S)a|Construct the URL of a file from the given information. The resolved address can either be a huggingface.co-hosted url, or a link to Cloudfront (a Content Delivery Network, or CDN) for large files which are more than a few MBs. Args: repo_id (`str`): A namespace (user or an organization) name and a repo name separated by a `/`. filename (`str`): The name of the file in the repo. subfolder (`str`, *optional*): An optional value corresponding to a folder inside the repo. repo_type (`str`, *optional*): Set to `"dataset"` or `"space"` if downloading from a dataset or space, `None` or `"model"` if downloading from a model. Default is `None`. revision (`str`, *optional*): An optional Git revision id which can be a branch name, a tag, or a commit hash. Example: ```python >>> from huggingface_hub import hf_hub_url >>> hf_hub_url( ... repo_id="julien-c/EsperBERTo-small", filename="pytorch_model.bin" ... ) 'https://huggingface.co/julien-c/EsperBERTo-small/resolve/main/pytorch_model.bin' ``` Notes: Cloudfront is replicated over the globe so downloads are way faster for the end user (and it also lowers our bandwidth costs). Cloudfront aggressively caches files by default (default TTL is 24 hours), however this is not an issue here because we implement a git-based versioning system on huggingface.co, which means that we store the files on S3/Cloudfront in a content-addressable way (i.e., the file name is its hash). Using content-addressable filenames means cache can't ever be stale. In terms of client-side caching from this library, we base our caching on the objects' entity tag (`ETag`), which is an identifier of a specific version of a resource [1]_. An object's ETag is: its git-sha1 if stored in git, or its sha256 if stored in git-lfs. References: - [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag N/zInvalid repo type)safe)rqrorr) r REPO_TYPES ValueErrorREPO_TYPES_URL_PREFIXESDEFAULT_REVISIONrformatr startswithENDPOINTlen)rqrrrmrnrorpurlr]r]r^ hf_hub_urls C  rF)follow_relative_redirectsmethodr~rcKs|r?td ||dd|}d|jkrdkr=n|St|jd}|jdkr=t|j|jd}td ||dd|S|Stj d ||d |}t ||S) aWrapper around requests methods to follow relative redirects if `follow_relative_redirects=True` even when `allow_redirection=False`. Args: method (`str`): HTTP method, such as 'GET' or 'HEAD'. url (`str`): The URL of the resource to fetch. follow_relative_redirects (`bool`, *optional*, defaults to `False`) If True, relative redirection (redirection to the same site) will be resolved even when `allow_redirection` kwarg is set to False. Useful when we want to follow a redirection to a renamed repository without following redirection to a CDN. **params (`dict`, *optional*): Params to pass to `requests.request`. F)rr~ri,iLocationrs)rOT)rr~Nr]) _request_wrapper status_coder headersnetloc_replacerOgeturlr'requestr*)rr~rparamsresponse parsed_targetnext_urlr]r]r^r s$   r)proxies resume_sizer expected_sizedisplayed_filename _nb_retries _tqdm_bar temp_filerrrrrrrcCs|dur ||kr dSd} tjr5|dkrtdn|dur#tdnzddl} Wn ty4tdw|} t|p=i}|dkrLt | d||d<t d|d||tj d } t | | j d } | durk|t| nd} |dur|}| j d }|durt|}|dur|d }t|d krd|dd}d|d|d}t|t| |d|d}|}| r/| dur/| dtjkr/dt| jjv}|stdz| jd!||jtjtj|ddd|rd|jiniWnty}zt d|d}~ww|s || |dur&|t!j"#|jkr&t$|j%t!j"#|jd WddS|}z$| j&tjdD]}|rR|t||'||t|7}d}q9WnLt(j)t(j*fy}z;|dkrqt+d|t,|t+d|t,|t-.dt/t0||||| ||d|d WYd}~WdSd}~wwWdn 1swY|dur||1krt$|j%|1ddSdS)"a Download a remote file. Do not gobble up errors, and will return errors tailored to the Hugging Face Hub. If ConnectionError (SSLError) or ReadTimeout happen while streaming data from the server, it is most likely a transient error (network outage?). We log a warning message and try to resume the download a few times before giving up. The method gives up after 5 attempts if no new data has being received from the server. Args: url (`str`): The URL of the file to download. temp_file (`BinaryIO`): The file-like object where to save the file. proxies (`dict`, *optional*): Dictionary mapping protocol to the URL of the proxy passed to `requests.request`. resume_size (`int`, *optional*): The number of bytes already downloaded. If set to 0 (default), the whole file is download. If set to a positive number, the download will resume at the given position. headers (`dict`, *optional*): Dictionary of HTTP Headers to send with the request. expected_size (`int`, *optional*): The expected size of the file to download. If set, the download will raise an error if the size of the received content is different from the expected one. displayed_filename (`str`, *optional*): The filename of the file that is being downloaded. Value is used only to display a nice progress bar. If not set, the filename is guessed from the URL or the `Content-Disposition` header. NrzU'hf_transfer' does not support `resume_size`: falling back to regular download methodzQ'hf_transfer' does not support `proxies`: falling back to regular download methodzFast download using 'hf_transfer' is enabled (HF_HUB_ENABLE_HF_TRANSFER=1) but 'hf_transfer' package is not available in your environment. Try `pip install hf_transfer`.RangeGETT)rr~streamrrtimeoutContent-LengthzContent-Dispositionrr((…)iz1Consistency check failed: file should be of size z but has size {actual_size} (zm). This is usually due to network issues while downloading the file. Please retry with `force_download=True`.zhuggingface_hub.http_getdesc log_leveltotalinitialrUrrcallbackzYou are using an outdated version of `hf_transfer`. Consider upgrading to latest version to enable progress bars using `pip install -U hf_transfer`.)r~rr max_files chunk_sizerparallel_failures max_retrieszAn error occurred while downloading using `hf_transfer`. Consider disabling HF_HUB_ENABLE_HF_TRANSFER for better error handling.) actual_size)rz9Error while downloading from %s: %s Max retries exceeded.z@Error while downloading from %s: %s Trying to resume download...r)r~rrrrrrrr])2rHF_HUB_ENABLE_HF_TRANSFERrVrW hf_transfer ImportErrorrwcopydeepcopyr8getrHF_HUB_DOWNLOAD_TIMEOUTr*rrlHEADER_FILENAME_PATTERNsearch groupdictr}r=loggergetEffectiveLevelDOWNLOAD_CHUNK_SIZEinspect signaturedownload parametersrUHF_TRANSFER_CONCURRENCYupdate Exception RuntimeErrorrLrOgetsizeEnvironmentErrorrz iter_contentwriterequestsConnectionError ReadTimeoutwarningrItimesleepr5http_gettell)r~rrrrrrrrrinitial_headersrcontent_lengthrcontent_dispositionmatchconsistency_error_message progress_cmprogresssupports_callbackenew_resume_sizechunkr]r]r^r9s&              "    +@r)rrrincomplete_pathrfc sz ddlm}m}Wn tytdwtd}dtttfffdd } |t| j |dg} |s<|j }t |d krK|d d d }t |t|dd |d } | !dtffdd } || |j|j|jf| | gdWd d S1s}wYd S)a+ Download a file using Xet storage service. Args: incomplete_path (`Path`): The path to the file to download. xet_file_data (`XetFileData`): The file metadata needed to make the request to the xet storage service. headers (`Dict[str, str]`): The headers to send to the xet storage service. expected_size (`int`, *optional*): The expected size of the file to download. If set, the download will raise an error if the size of the received content is different from the expected one. displayed_filename (`str`, *optional*): The filename of the file that is being downloaded. Value is used only to display a nice progress bar. If not set, the filename is guessed from the URL or the `Content-Disposition` header. **How it works:** The file download system uses Xet storage, which is a content-addressable storage system that breaks files into chunks for efficient storage and transfer. `hf_xet.download_files` manages downloading files by: - Taking a list of files to download (each with its unique content hash) - Connecting to a storage server (CAS server) that knows how files are chunked - Using authentication to ensure secure access - Providing progress updates during download Authentication works by regularly refreshing access tokens through `refresh_xet_connection_info` to maintain a valid connection to the storage server. The download process works like this: 1. Create a local cache folder at `~/.cache/huggingface/xet/chunk-cache` to store reusable file chunks 2. Download files in parallel: 2.1. Prepare to write the file to disk 2.2. Ask the server "how is this file split into chunks?" using the file's unique hash The server responds with: - Which chunks make up the complete file - Where each chunk can be downloaded from 2.3. For each needed chunk: - Checks if we already have it in our local cache - If not, download it from cloud storage (S3) - Save it to cache for future use - Assemble the chunks in order to recreate the original file r) PyPointerFiledownload_fileszTo use optimized download using Xet storage, you need to install the hf_xet package. Try `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`. file_datarr@cs(td}|durtd|j|jfS)Nrz+Failed to refresh token using xet metadata.)r4rw access_tokenexpiration_unix_epoch)connection_info)rrfr]r^token_refresher%s  z xet_get..token_refresher)rOhashfilesizerNrzhuggingface_hub.xet_getrprogress_bytescs|dSN)r)r)rr]r^progress_updaterAsz!xet_get..progress_updater)rp token_inforr)hf_xetrrrrwr4r rIrlabsolute file_hashrUr}r=rrfloatrprr) rrfrrrrrrrr pointer_filesrrr])rrrfr^xet_getsB6     "rrccCs|durdS|ddS)aONormalize ETag HTTP header, so it can be used to create nice filepaths. The HTTP spec allows two forms of ETag: ETag: W/"" ETag: "" For now, we only expect the second form from the server, but we want to be future-proof so we support both. For more context, see `TestNormalizeEtag` tests and https://github.com/huggingface/huggingface_hub/pull/1428. Args: etag (`str`, *optional*): HTTP header Returns: `str` or `None`: string that can be used as a nice directory name. Returns `None` if input is None. NzW/")lstripstrip)rcr]r]r^_normalize_etagMsrsrcdstnew_blobcCst|||dS)z6Alias method used in `transformers` conversion script.rrr)_create_symlinkrr]r]r^_create_relative_symlinkcsrc Cszt|Wn tyYnwtjtj|}tjtj|}tj|}z tj||}Wn ty?d}Ynwztj ||g}t |}Wn7ty\tj dk}Yn*t yht |}Ynty} z| j t jkrzt |}nWYd} ~ nd} ~ ww|r|p|} td| d|z t| |WdStytj|rtj|tj|krYdSt yYnw|rtd|d|tj||tddStd|d|t||dS)uCreate a symbolic link named dst pointing to src. By default, it will try to create a symlink using a relative path. Relative paths have 2 advantages: - If the cache_folder is moved (example: back-up on a shared drive), relative paths within the cache folder will not break. - Relative paths seems to be better handled on Windows. Issue was reported 3 times in less than a week when changing from relative to absolute paths. See https://github.com/huggingface/huggingface_hub/issues/1398, https://github.com/huggingface/diffusers/issues/2729 and https://github.com/huggingface/transformers/pull/22228. NOTE: The issue with absolute paths doesn't happen on admin mode. When creating a symlink from the cache to a local folder, it is possible that a relative path cannot be created. This happens when paths are not on the same volume. In that case, we use absolute paths. The result layout looks something like └── [ 128] snapshots ├── [ 128] 2439f60ef33a0d46d85da5001d52aeda5b00ce9f │ ├── [ 52] README.md -> ../../../blobs/d7edf6bd2a681fb0175f7735299831ee1b22b812 │ └── [ 76] pytorch_model.bin -> ../../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd If symlinks cannot be created on this platform (most likely to be Windows), the workaround is to avoid symlinks by having the actual file in `dst`. If it is a new file (`new_blob=True`), we move it to `dst`. If it is not a new file (`new_blob=False`), we don't know if the blob file is already referenced elsewhere. To avoid breaking existing cache, the file is duplicated on the disk. In case symlinks are not supported, a warning message is displayed to the user once when loading `huggingface_hub`. The warning message can be disabled with the `DISABLE_SYMLINKS_WARNING` environment variable. NrGzCreating pointer from z to z(Symlink not supported. Moving file from  copy_functionz)Symlink not supported. Copying file from )rLremoverSrOabspathrJrQrPrw commonpathr_rUPermissionErrorerrnoEROFSrdebugrRFileExistsErrorislinkrealpathinfoshutilmove_copy_no_matter_whatcopyfile) rrrabs_srcabs_dstabs_dst_folderr[r_support_symlinksrsrc_rel_or_absr]r]r^rhsZ            $ rstorage_folderrbcCsR||kr%t|d|}|jjddd|r||kr'||dSdSdS)zCache reference between a revision (tag, branch or truncated commit hash) and the corresponding commit hash. Does nothing if `revision` is already a proper `commit_hash` or reference is already cached. refsTparentsrBN)rparentmkdirexists read_text write_text)r rorbref_pathr]r]r^(_cache_commit_hash_for_specific_revisionsrcCs"|dg|d}tj|S)zReturn a serialized version of a hf.co repo name and type, safe for disk storage as a single non-nested folder. Example: models--julien-c--EsperBERTo-small srt)splitrREPO_ID_SEPARATORjoin)rqrnpartsr]r]r^repo_folder_names r target_dirc Cszt|}|gt|jD].}z#t|j}||kr.td|ddd|d|dddWdSty:Yq wdS)a$Check disk usage and log a warning if there is not enough disk space to download the file. Args: expected_size (`int`): The expected size of the file in bytes. target_dir (`str`): The directory where the file will be stored after downloading. zLNot enough free disk space to download the file. The expected file size is: g.Az.2fz MB. The target location z only has z MB free disk space.N) rlistrr disk_usagefreerVrWrS)rrrOtarget_dir_freer]r]r^_check_disk_spaces&   r!auto)rmrnro library_namelibrary_versionr? local_dir user_agentforce_downloadr etag_timeouttokenlocal_files_onlyrrpresume_downloadforce_filenamelocal_dir_use_symlinksr#r$r%r&r'r(r)r*r+r,r-cCs:tjtjkr tj} |durtdt|durtdt|dur$tj}|dur+tj}t|t r4t |}t|t r=t |}|dkrCd}|durN|d|}|durTd}|tj vrft d|dt tj t | ||| |d }|dur|d kr|td t||||||| || | || |d St||||||| || | || d S)uDownload a given file if it's not already present in the local cache. The new cache file layout looks like this: - The cache directory contains one subfolder per repo_id (namespaced by repo type) - inside each repo folder: - refs is a list of the latest known revision => commit_hash pairs - blobs contains the actual file blobs (identified by their git-sha or sha256, depending on whether they're LFS files or not) - snapshots contains one subfolder per commit, each "commit" contains the subset of the files that have been resolved at that particular commit. Each filename is a symlink to the blob at that particular commit. ``` [ 96] . └── [ 160] models--julien-c--EsperBERTo-small ├── [ 160] blobs │ ├── [321M] 403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd │ ├── [ 398] 7cb18dc9bafbfcf74629a4b760af1b160957a83e │ └── [1.4K] d7edf6bd2a681fb0175f7735299831ee1b22b812 ├── [ 96] refs │ └── [ 40] main └── [ 128] snapshots ├── [ 128] 2439f60ef33a0d46d85da5001d52aeda5b00ce9f │ ├── [ 52] README.md -> ../../blobs/d7edf6bd2a681fb0175f7735299831ee1b22b812 │ └── [ 76] pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd └── [ 128] bbc77c8132af1cc5cf678da3f1ddf2de43606d48 ├── [ 52] README.md -> ../../blobs/7cb18dc9bafbfcf74629a4b760af1b160957a83e └── [ 76] pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd ``` If `local_dir` is provided, the file structure from the repo will be replicated in this location. When using this option, the `cache_dir` will not be used and a `.cache/huggingface/` folder will be created at the root of `local_dir` to store some metadata related to the downloaded files. While this mechanism is not as robust as the main cache-system, it's optimized for regularly pulling the latest version of a repository. Args: repo_id (`str`): A user or an organization name and a repo name separated by a `/`. filename (`str`): The name of the file in the repo. subfolder (`str`, *optional*): An optional value corresponding to a folder inside the model repo. repo_type (`str`, *optional*): Set to `"dataset"` or `"space"` if downloading from a dataset or space, `None` or `"model"` if downloading from a model. Default is `None`. revision (`str`, *optional*): An optional Git revision id which can be a branch name, a tag, or a commit hash. library_name (`str`, *optional*): The name of the library to which the object corresponds. library_version (`str`, *optional*): The version of the library. cache_dir (`str`, `Path`, *optional*): Path to the folder where cached files are stored. local_dir (`str` or `Path`, *optional*): If provided, the downloaded file will be placed under this directory. user_agent (`dict`, `str`, *optional*): The user-agent info in the form of a dictionary or a string. force_download (`bool`, *optional*, defaults to `False`): Whether the file should be downloaded even if it already exists in the local cache. proxies (`dict`, *optional*): Dictionary mapping protocol to the URL of the proxy passed to `requests.request`. etag_timeout (`float`, *optional*, defaults to `10`): When fetching ETag, how many seconds to wait for the server to send data before giving up which is passed to `requests.request`. token (`str`, `bool`, *optional*): A token to be used for the download. - If `True`, the token is read from the HuggingFace config folder. - If a string, it's used as the authentication token. local_files_only (`bool`, *optional*, defaults to `False`): If `True`, avoid downloading the file and return the path to the local cached file if it exists. headers (`dict`, *optional*): Additional headers to be sent with the request. Returns: `str`: Local path of file or if networking is off, last version of file cached on disk. Raises: [`~utils.RepositoryNotFoundError`] If the repository to download from cannot be found. This may be because it doesn't exist, or because it is set to `private` and you do not have access. [`~utils.RevisionNotFoundError`] If the revision to download from cannot be found. [`~utils.EntryNotFoundError`] If the file to download cannot be found. [`~utils.LocalEntryNotFoundError`] If network is disabled or unavailable and file is not found in cache. [`EnvironmentError`](https://docs.python.org/3/library/exceptions.html#EnvironmentError) If `token=True` but the token cannot be found. [`OSError`](https://docs.python.org/3/library/exceptions.html#OSError) If ETag cannot be determined. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If some parameter value is invalid. NzThe `force_filename` parameter is deprecated as a new caching system, which keeps the filenames as they are on the Hub, is now in place.z`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.rsrtmodelInvalid repo type: . Accepted repo types are: r)r#r$r&rr"ae`local_dir_use_symlinks` parameter is deprecated and will be ignored. The process to download files to a local folder has been updated and do not rely on symlinks anymore. You only need to pass a destination folder as`local_dir`. For more details, check out https://huggingface.co/docs/huggingface_hub/main/en/guides/download#download-files-to-local-folder.) r%rqrnrrrorpr(rrr)r?r'r*) r?rqrrrnrorpr(rrr)r*r')rHF_HUB_ETAG_TIMEOUTDEFAULT_ETAG_TIMEOUTrVrW FutureWarningrHry isinstancerrIrvrwr!_hf_hub_download_to_local_dir_hf_hub_download_to_cache_dir)rqrrrmrnror#r$r?r%r&r'rr(r)r*rrpr+r,r- hf_headersr]r]r^hf_hub_downloads {   r9c Cs>tj|d} tj|t||d} tjj|d}tjdkr1|ds)d|vr1td|dt |rFt | ||}tj |rF| sF|St ||||||||| | | |d \}}}}}}|d ur| sd }t |rl|}n&tj| d |}tj |rt| }|}Wd n1swY|d urt | ||}tj |r| s|St|| | |d usJd |d usJd |d usJd|d usJdtj| d|}t | ||}tjtj|ddtjtj|ddt| ||| stj |r|Stj |rt||dd|Stj| t||d|d}tjdkr:ttj|dkr:dtj|}tjdkrSttj|dkrSdtj|}t|jjdddt|3tt|dt||||||| ||d tj |st||ddWd |SWd |S1swY|S)zDownload a given file to a cache folder, if not already present. Method should not be called directly. Please use `hf_hub_download` instead. z.locks)rqrnrtrGz..\z\..\z*Invalid filename: cannot handle filename 'zB' on Windows. Please ask the repository owner to rename this file.) rqrrrnrorprr(rr)r*r relative_filenameNr )etag must have been retrieved from server0commit_hash must have been retrieved from server2file location must have been retrieved from server2expected_size must have been retrieved from serverblobsTrAF)rz.lock\\?\r z .incomplete rdestination_pathurl_to_downloadrrrrrr'rcrf)rLrOrrrrUr{rwREGEX_COMMIT_HASHr_get_pointer_pathr_get_metadata_or_catch_errorisfileopenread_raise_on_head_call_errorrMrQrrr}rrrrr_download_to_tmp_and_move)r?rqrrrnrorpr(rrr)r*r' locks_dirr r: pointer_pathrDrcrbrrfhead_call_errorrf blob_path lock_pathr]r]r^r7s            ""     r7c  Cstjdkrttj|dkrdtj|}t|}t||d} t||d}| sAt |rA| j rA|durA|j |krAt | j St||||||||| | d \}}}}}}|durt| sn| j rntd|dt | j St|| | |dus|Jd |dusJd |dusJd |dusJd | s| j r|dur|j|krt||||d t | j S|durt |durt| j d}t|}Wdn1swY||krt||||d t | j S| s.t||| ||d}t|t r.t| j| j jjdddt|| j Wdn 1swYt||||d t | j St| j!| j j ddt!| "|| j |||||| ||d Wdn 1sXwYt||||d t | j S)zDownload a given file to a local folder, if not already present. Method should not be called directly. Please use `hf_hub_download` instead. rGr@rA)r%rrN) rqrrrnrorprr(rr)r*zpCouldn't access the Hub to check for update but local file already exists. Defaulting to existing file. (error: )r;r<r=r>)r%rrrbrcrb)rqrrr?rornTr  missing_okrB)#rLrUr}rOrrrrrEr file_pathis_filerbrIrGrrrKrcr REGEX_SHA256rIr<hextry_to_load_from_cacher5rrRrrrrunlinkrLr)r%rqrnrrrorpr(rrr)r?r'r*pathslocal_metadatarDrcrbrrfrOrPr cached_pathr]r]r^r6js               r6cCsd|durd}|dur d}|tjvrtd|dttj|dur%tj}|dd}tj||d|}tj |s?dStj|d }tj|d }tj|d } tj |rtj||} tj | rt |  } | }Wdn1szwYtj tj| ||rt Stj|sdSt|} || vrdStj|||} tj | r| SdS) a Explores the cache to return the latest cached file for a given revision if found. This function will not raise any exception if the file in not cached. Args: cache_dir (`str` or `os.PathLike`): The folder where the cached files lie. repo_id (`str`): The ID of the repo on huggingface.co. filename (`str`): The filename to look for inside `repo_id`. revision (`str`, *optional*): The specific model version to use. Will default to `"main"` if it's not provided and no `commit_hash` is provided either. repo_type (`str`, *optional*): The type of the repository. Will default to `"model"`. Returns: `Optional[str]` or `_CACHED_NO_EXIST`: Will return `None` if the file was not cached. Otherwise: - The exact path to the cached file if it's found in the cache - A special value `_CACHED_NO_EXIST` if the file does not exist at the given commit hash and this fact was cached. Example: ```python from huggingface_hub import try_to_load_from_cache, _CACHED_NO_EXIST filepath = try_to_load_from_cache() if isinstance(filepath, str): # file exists and is cached ... elif filepath is _CACHED_NO_EXIST: # non-existence of file is cached ... else: # file is not cached ... ``` Nmainr.r/r0rtz--zs--r  snapshots .no_exist)rrvrwrIrHreplacerLrOrisdirrHrIrJ_CACHED_NO_EXISTrlistdir)rqrrr?rorn object_id repo_cacherefs_dir snapshots_dir no_exist_dir revision_filerP cached_shas cached_filer]r]r^r[s<2         r[rc Cst|||||d}d|d<td||dd||d} t| t| jtjt| jtj p0| jd| jd p:| j j t | jtj pG| jd t| d S) a!Fetch metadata of a file versioned on the Hub for a given url. Args: url (`str`): File url, for example returned by [`hf_hub_url`]. token (`str` or `bool`, *optional*): A token to be used for the download. - If `True`, the token is read from the HuggingFace config folder. - If `False` or `None`, no token is provided. - If a string, it's used as the authentication token. proxies (`dict`, *optional*): Dictionary mapping protocol to the URL of the proxy passed to `requests.request`. timeout (`float`, *optional*, defaults to 10): How many seconds to wait for the server to send metadata before giving up. library_name (`str`, *optional*): The name of the library to which the object corresponds. library_version (`str`, *optional*): The version of the library. user_agent (`dict`, `str`, *optional*): The user-agent info in the form of a dictionary or a string. headers (`dict`, *optional*): Additional headers to be sent with the request. Returns: A [`HfFileMetadata`] object containing metadata such as location, etag, size and commit_hash. r1identityzAccept-EncodingHEADFT)rr~rallow_redirectsrrrETagrr)rbrcrdrerf)r!rr*rarrr HUGGINGFACE_HEADER_X_REPO_COMMITr HUGGINGFACE_HEADER_X_LINKED_ETAGrr~ _int_or_none HUGGINGFACE_HEADER_X_LINKED_SIZEr3) r~r)rrr#r$r&rr8rr]r]r^get_hf_file_metadataGs6(  rw)r:r r:c Cs| rdddddtd|d|d|d|d fSt|||||d} | } d}d}d}d}d}| s7zz t| ||||d}WnZty}zN| dur| dur|jjtj}|durt | d || }z|j j d d d | Wnt y}ztd |WYd}~nd}~wwt| ||d}~ww|j}|durtd |j}|durtd|j}|durtd|j}|dur| |jkr|j} t| jt|jjkr|ddWn\tjjtjjfytjjtjjtfy}z|}WYd}~n8d}~wt tfy tj!y"}z|}WYd}~nd}~wty6}z|}WYd}~nd}~ww| sH|dusH|dusHt"d| |||||fS)aGet metadata for a file on the Hub, safely handling network issues. Returns either the etag, commit_hash and expected size of the file, or the error raised while fetching the metadata. NOTE: This function mutates `headers` inplace! It removes the `authorization` header if the file is a LFS blob and the domain of the url is different from the domain of the location (typically an S3 bucket). NzHCannot access file since 'local_files_only=True' as been set. (repo_id: z , repo_type: z , revision: z , filename: rS)rnrorp)r~rrrr)rbTr zNCould not cache non-existence of file. Will ignore error and continue. Error: aDistant resource does not seem to be on huggingface.co. It is possible that a configuration issue prevents you from downloading resources from https://huggingface.co. Please check your firewall and proxy settings and make sure your SSL certificates are updated.z\Distant resource does not have an ETag, we won't be able to reliably ensure reproducibility.z0Distant resource does not have a Content-Length. authorizationz'etag is empty due to uncovered problems)#rrrwrrrrrrsrrrrNrSrerrorrrbrrcrerfrdr rpopr exceptionsSSLError ProxyErrorrTimeoutr HTTPErrorr)rqrrrnrorprr(rr)r*r:r r~rDrcrbrhead_error_callrfmetadata http_errorno_exist_file_pathrryr]r]r^rGs      rGrOcCsl|r|rtdt|trtd|td||rtdt|ttfs/t|tr1|jjdkr1|td|)zWRaise an appropriate error when the HEAD call failed and we cannot locate a local file.zOCannot pass 'force_download=True' and 'local_files_only=True' at the same time.z?Cannot pass 'force_download=True' when offline mode is enabled.z-Force download failed due to the above error.zCannot find the requested files in the disk cache and outgoing traffic has been disabled. To enable hf.co look-ups and downloads online, set 'local_files_only' to False.izAn error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.) rwr5rrrrrrr)rOr'r*r]r]r^rK&s(    rKrCrDc Csl|r|sdS|r4|stjr4|s4d|d} |r | d7} n tjr)|s)| d7} t| |jdd|df} | } d |d |d} | d krZ|durZ| d | d |d7} t| |durot||j t||j | durt rtdt || |||dn| durt dt || || ||dWdn1swYtd|t||dS)a0Download content from a URL to a destination path. Internal logic: - return early if file is already downloaded - resume download if possible (from incomplete file) - do not resume download if `force_download=True` or `HF_HUB_ENABLE_HF_TRANSFER=True` - check disk space before downloading - download content to a temporary file - set correct permissions on temporary file - move the temporary file to the destination path Both `incomplete_path` and `destination_path` must be on the same volume to avoid a local copy. NzRemoving incomplete file ''z (force_download=True)z (hf_transfer=True)TrUabz Downloading 'z' to 'rz (resume from rtrSzIXet Storage is enabled for this repo. Downloading file from Xet Storage..)rrfrrrzXet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`)rrrrz"Download complete. Moving file to )rrrrrr\rIrr!rr:rrr_chmod_and_move) rrCrDrrrrrr'rcrfr\rPrr]r]r^rLFsX           %rLvaluec Cs$zt|WSttfyYdSwr)rl TypeErrorrw)rr]r]r^rus  rucCs|jjdt}zHz|t|j}tt |t |Wnt yB}zt d|d|dWYd}~nd}~wwWz|Wnt ySYnwz|Wwt ycYwwtjt |t |tddS)aVSet correct permission before moving a blob from tmp directory to cache dir. Do not take into account the `umask` from the process as there is no convenient way to get it that is thread-safe. See: - About umask: https://docs.python.org/3/library/os.html#os.umask - Thread-safety: https://stackoverflow.com/a/70343066 - About solution: https://github.com/huggingface/huggingface_hub/pull/1220#issuecomment-1326211591 - Fix issue: https://github.com/huggingface/huggingface_hub/issues/1141 - Fix issue: https://github.com/huggingface/huggingface_hub/issues/1215 tmp_z+Could not set the permissions on the file 'z '. Error: z). Continuing without setting permissions.Nr)ruuiduuid4rNrstatst_moderLchmodrIS_IMODErSrrr\rrr)rrtmp_filecache_dir_moderr]r]r^rs.    rcCs4z t||WdStyt||YdSw)z\Copy file from src to dst. If `shutil.copy2` fails, fallback to `shutil.copyfile`. N)rcopy2rSr)rrr]r]r^rs  rcCs`tj|d}tj|||}ttj|ttj|jvr.td|d|d|d|S)NrazXInvalid pointer path: cannot create pointer path in snapshot folder if `storage_folder='z'`, `revision='z'` and `relative_filename='z'`.)rLrOrrrrrw)r ror: snapshot_pathrNr]r]r^rFs"rFr)F)NNN)rrrrLrerrrrrV dataclassesrpathlibrtypingrrrrrr r r urllib.parser r rrsrr _local_folderrrrrrerrorsrrrrrrrutilsrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7 utils._httpr8utils._runtimer9r: utils._typingr; utils.shar< utils.tqdmr= get_loggerrgrobjectre_CACHED_NO_EXIST_TcompilerrErYr>rIboolrkr_rarResponserrlrrrrrrrr!r3rr9r7r6r[DEFAULT_REQUEST_TIMEOUTrwrrGrKrLrurrrFr]r]r]r^s   ($t        ; Y 4   2  h^        [               \   Y           R"