o hy@s|dZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z ddl mZddlmZmZddlmZmZmZmZmZmZmZmZmZmZmZdd lmZd d l m!Z!d d l"m#Z#m$Z$m%Z%m&Z&d d l'm(Z(d dl)m*Z*m+Z+m,Z,d dl-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7d dl-m8Z9d dl:m;Z;erd dle?Z@edZAdZBdZCe GdddZDe GdddZEe GdddZFdeGdeGfddZHeeFeEeDfZId eeIddfd!d"ZJe7dd#dd$d%eeFd&eGd'eGd(eeGeGfd)eeGd*eKd+eeGfd,d-ZLe7dddd.d%eeFd&eGd'eGd(eeGeGfd)eeGd+eeGd/eeMfd0d1ZNd2eOfd3d4ZPe7  5 dCd%eeFd&eGd'eGd(eeGeGfd+eGd)eeGd/eMd6eeGddfd7d8ZQe7 dDd9eeEd&eGd'eGd(eeGeGfd+eGd)eeGdeeeGeeGfed:eRfffd;d<ZS  dEd eeId=eeeGeeGfed:eRffd>eGd?eeGd@eeGdeeeGeff dAdBZTdS)Fz< Type definitions and utilities for the `create_commit` API N) defaultdict)contextmanager) dataclassfield)groupby)Path PurePosixPath) TYPE_CHECKINGAnyBinaryIODictIterableIteratorListLiteralOptionalTupleUnion) thread_map) constants)EntryNotFoundErrorHfHubHTTPErrorXetAuthorizationErrorXetRefreshTokenError) hf_hub_url) UploadInfo lfs_uploadpost_lfs_batch_info) FORBIDDEN_FOLDERS XetTokenTypechunk_iterable(fetch_xet_connection_info_from_repo_info get_sessionhf_raise_for_statusloggingshatqdm_stream_filevalidate_hf_hub_args)tqdm)_get_progress_bar_context)RepoFilelfsregularic@s:eZdZUdZeed<dZeee dfed<ddZ dS)CommitOperationDeletea Data structure holding necessary info to delete a file or a folder from a repository on the Hub. Args: path_in_repo (`str`): Relative filepath in the repo, for example: `"checkpoints/1fec34a/weights.bin"` for a file or `"checkpoints/1fec34a/"` for a folder. is_folder (`bool` or `Literal["auto"]`, *optional*) Whether the Delete Operation applies to a folder or not. If "auto", the path type (file or folder) is guessed automatically by looking if path ends with a "/" (folder) or not (file). To explicitly set the path type, you can set `is_folder=True` or `is_folder=False`. path_in_repoauto is_foldercCsFt|j|_|jdkr|jd|_t|jts!td|jddS)Nr2/zNWrong value for `is_folder`. Must be one of [`True`, `False`, `'auto'`]. Got 'z'.)_validate_path_in_repor1r3endswith isinstancebool ValueErrorselfr<o/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/huggingface_hub/_commit_api.py __post_init__Ms    z#CommitOperationDelete.__post_init__N) __name__ __module__ __qualname____doc__str__annotations__r3rr8rr>r<r<r<r=r09s  r0c@sZeZdZUdZeed<eed<dZeeed<dZeeed<dZ eeed<dd Z dS) CommitOperationCopyab Data structure holding necessary info to copy a file in a repository on the Hub. Limitations: - Only LFS files can be copied. To copy a regular file, you need to download it locally and re-upload it - Cross-repository copies are not supported. Note: you can combine a [`CommitOperationCopy`] and a [`CommitOperationDelete`] to rename an LFS file on the Hub. Args: src_path_in_repo (`str`): Relative filepath in the repo of the file to be copied, e.g. `"checkpoints/1fec34a/weights.bin"`. path_in_repo (`str`): Relative filepath in the repo where to copy the file, e.g. `"checkpoints/1fec34a/weights_copy.bin"`. src_revision (`str`, *optional*): The git revision of the file to be copied. Can be any valid git revision. Default to the target commit revision. src_path_in_repor1N src_revision_src_oid _dest_oidcCst|j|_t|j|_dSN)r5rFr1r:r<r<r=r>ws z!CommitOperationCopy.__post_init__) r?r@rArBrCrDrGrrHrIr>r<r<r<r=rEXs  rEc@seZdZUdZeed<eeeee fed<e dddZ e ed<e ddddZ eeed <e ddddZeeed <e ddddZeeed <e ddddZeed <e ddddZeed <dddZeddedee fddZdefddZedeefddZdS)CommitOperationAdda Data structure holding necessary info to upload a file to a repository on the Hub. Args: path_in_repo (`str`): Relative filepath in the repo, for example: `"checkpoints/1fec34a/weights.bin"` path_or_fileobj (`str`, `Path`, `bytes`, or `BinaryIO`): Either: - a path to a local file (as `str` or `pathlib.Path`) to upload - a buffer of bytes (`bytes`) holding the content of the file to upload - a "file object" (subclass of `io.BufferedIOBase`), typically obtained with `open(path, "rb")`. It must support `seek()` and `tell()` methods. Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `path_or_fileobj` is not one of `str`, `Path`, `bytes` or `io.BufferedIOBase`. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `path_or_fileobj` is a `str` or `Path` but not a path to an existing file. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `path_or_fileobj` is a `io.BufferedIOBase` but it doesn't support both `seek()` and `tell()`. r1path_or_fileobjF)initrepr upload_infoN)rMrNdefault _upload_mode_should_ignore _remote_oid _is_uploaded _is_committedreturnc Cs t|j|_t|jtrt|j|_t|jtr2tjtj |j}tj |s1t d|dn t|jt j tfs?t dt|jt j riz|j|jdtjWnttfyh}zt d|d}~wwt|jtrxt|j|_dSt|jtrt|j|_dSt|j|_dS)z6Validates `path_or_fileobj` and compute `upload_info`.zProvided path: 'z(' is not a file on the local file systemzpath_or_fileobj must be either an instance of str, bytes or io.BufferedIOBase. If you passed a file-like object, make sure it is in binary mode.rzNpath_or_fileobj is a file-like object but does not implement seek() and tell()N)r5r1r7rLrrCospathnormpath expanduserisfiler9ioBufferedIOBasebytestellseekSEEK_CUROSErrorAttributeErrorr from_pathrO from_bytes from_fileobj)r;rLexcr<r<r=r>s:        z CommitOperationAdd.__post_init__ with_tqdmccst|jts t|jtrF|r*t|j }|VWddS1s#wYdSt|jd }|VWddS1s?wYdSt|jtrUt|jVdSt|jtj ro|j }|jV|j |tj dSdS)u A context manager that yields a file-like object allowing to read the underlying data behind `path_or_fileobj`. Args: with_tqdm (`bool`, *optional*, defaults to `False`): If True, iterating over the file object will display a progress bar. Only works if the file-like object is a path to a file. Pure bytes and buffers are not supported. Example: ```python >>> operation = CommitOperationAdd( ... path_in_repo="remote/dir/weights.h5", ... path_or_fileobj="./local/weights.h5", ... ) CommitOperationAdd(path_in_repo='remote/dir/weights.h5', path_or_fileobj='./local/weights.h5') >>> with operation.as_file() as file: ... content = file.read() >>> with operation.as_file(with_tqdm=True) as file: ... while True: ... data = file.read(1024) ... if not data: ... break config.json: 100%|█████████████████████████| 8.19k/8.19k [00:02<00:00, 3.72kB/s] >>> with operation.as_file(with_tqdm=True) as file: ... requests.put(..., data=file) config.json: 100%|█████████████████████████| 8.19k/8.19k [00:02<00:00, 3.72kB/s] ``` Nrb) r7rLrCrr'openr^r\BytesIOr]r_r`SEEK_SET)r;rhfileprev_posr<r<r=as_files $ ""  zCommitOperationAdd.as_filecCs:|}t|WdS1swYdS)z[ The base64-encoded content of `path_or_fileobj` Returns: `bytes` N)robase64 b64encodereadr;rmr<r<r= b64contents  $zCommitOperationAdd.b64contentcCs^|jdurdS|jdkr|jjS|}t|WdS1s(wYdS)aReturn the OID of the local file. This OID is then compared to `self._remote_oid` to check if the file has changed compared to the remote one. If the file did not change, we won't upload it again to prevent empty commits. For LFS files, the OID corresponds to the SHA256 of the file content (used a LFS ref). For regular files, the OID corresponds to the SHA1 of the file content. Note: this is slightly different to git OID computation since the oid of an LFS file is usually the git-SHA1 of the pointer file content (not the actual file content). However, using the SHA256 is enough to detect changes and more convenient client-side. Nr-)rQrOsha256hexror&git_hashrrrsr<r<r= _local_oid s     $zCommitOperationAdd._local_oidrVN)F)r?r@rArBrCrDrrr^r rrOrrQr UploadModerRr8rSrTrUr>rrrortpropertyrxr<r<r<r=rK|s  #1 rKr1rVcs|dr |dd}|dks|dks|dr td|d|dr+|d d}tD]tfd d |dDrHtd d |dq-|S)Nr4r.z..z../z,Invalid `path_in_repo` in CommitOperation: ''z./c3s|]}|kVqdSrJr<).0part forbiddenr<r= +sz)_validate_path_in_repo..zHInvalid `path_in_repo` in CommitOperation: cannot update files under a 'z/' folder (path: 'z').) startswithr9ranysplitr1r<rr=r5"s    r5 operationscCstt}|D]X}|j}t|tr9||dkrtd|d||d7<t|jD] }|t |d7<q,t|t r^|t t|dkr^|j rUtd|dqtd|dqdS)a Warn user when a list of operations is expected to overwrite itself in a single commit. Rules: - If a filepath is updated by multiple `CommitOperationAdd` operations, a warning message is triggered. - If a filepath is updated at least once by a `CommitOperationAdd` and then deleted by a `CommitOperationDelete`, a warning is triggered. - If a `CommitOperationDelete` deletes a filepath that is then updated by a `CommitOperationAdd`, no warning is triggered. This is usually useless (no need to delete before upload) but can happen if a user deletes an entire folder and then add new files to it. rzBAbout to update multiple times the same file in the same commit: 'z9'. This can cause undesired inconsistencies in your repo.rz_About to delete a folder containing files that have just been updated within the same commit: 'zLAbout to delete a file that have just been updated within the same commit: 'N) rintr1r7rKwarningswarnrparentsrCr0r3)rnb_additions_per_path operationr1parentr<r<r=_warn_on_overwriting_operations6s8   r)endpoint num_threadsrevision additions repo_typerepo_idheadersrrrc sg}t|tdD]+}tdd|D|||dd\} } | r/ddd| D} td| || 7}qd d |Dg} |D]} | d durWtd | d jdq?| | q?t | dkrjtddSdfdd }t j rtdt | dt | ddD]} || qdSt | dkrtd|| ddStdt | d|dt|| dt | d|t ddS) a2 Uploads the content of `additions` to the Hub using the large file storage protocol. Relevant external documentation: - LFS Batch API: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: additions (`List` of `CommitOperationAdd`): The files to be uploaded repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. headers (`Dict[str, str]`): Headers to use for the request, including authorization headers and user agent. num_threads (`int`, *optional*): The number of concurrent threads to use when uploading. Defaults to 5. revision (`str`, *optional*): The git revision to upload to. Raises: [`EnvironmentError`](https://docs.python.org/3/library/exceptions.html#EnvironmentError) If an upload failed for any reason [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If the server returns malformed responses [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the LFS batch endpoint returned an HTTP error.  chunk_sizecSg|]}|jqSr<)rOropr<r<r= z%_upload_lfs_files..N) upload_infosrrrrrtoken c Ss0g|]}d|dd|didqS)z$Encountered error for file with OID oidz: `errormessageget)rerrr<r<r=rs"z$LFS batch endpoint returned errors: cSsi|] }|jj|qSr<)rOrurv)radd_opr<r<r= z%_upload_lfs_files..actionszContent of file rz/ is already present upstream - skipping upload.rzNo LFS files to upload.rVc sPz|d}t||dWdSty'}z td|jd|d}~ww)Nr)rlfs_batch_actionrrzError while uploading 'z ' to the Hub.)r Exception RuntimeErrorr1) batch_actionrrgrr oid2addopr<r=_wrapped_lfs_uploads z._upload_lfs_files.._wrapped_lfs_uploadz Uploading z* LFS files to the Hub using `hf_transfer`.zhuggingface_hub.lfs_upload)namerzUploading 1 LFS file to the Hubz" LFS files to the Hub using up to z threads concurrentlyzUpload z LFS files)desc max_workers tqdm_classry)r!UPLOAD_BATCH_MAX_NUM_FILESrjoinr9rloggerdebugr1appendlenrHF_HUB_ENABLE_HF_TRANSFERhf_tqdmr)rrrrrrr batch_actionschunkbatch_actions_chunkbatch_errors_chunkrfiltered_actionsactionrr<rr=_upload_lfs_filesdsb+           r)rr create_prrc st|dkrdSddlm}zttjrddindd}WntyA} z| jjdkr.token_refresherrrcSsg|]}|qSr<r<rr<r<r=r=sz%_upload_xet_files..cSsg|]}t|jqSr<)rCrLrr<r<r=r>scSsg|]}tj|qSr<)rWrXgetsize)rrXr<r<r=r?szUploading Batch [z]...z Uploading...BTzhuggingface_hub.xet_put)rtotalinitialunit unit_scaler log_level incrementcs|dSrJ)update)r)progressr<r=update_progressPsz*_upload_xet_files..update_progress)rhf_xetrr"r rrresponse status_coderrrrrrCrmathceilrlog10 enumerater!sumzfillr*rgetEffectiveLevel)rrrrrrrrxet_connection_infoe xet_endpointaccess_token_infor num_chunksnum_chunks_num_digitsir_chunkpaths expected_size description progress_cmrr<)rrrrrrrr=_upload_xet_filessd ?    $" rpreupload_infocCsh|d}t|tstd|D]!}t|tr-t|dtr-t|dtr-|ddvs1tdq|S)Nfilesz&preupload_info is improperly formattedrX uploadModer,z'preupload_info is improperly formatted:)rr7listr9dictrC)rr file_infor<r<r=_validate_preupload_infoWs   rFgitignore_contentc CsB|dur|ntj}i}i} i} t|dD]e} ddd| Di} |dur'|| d<tj|d|d|d || ||r>d d indd } t| t| }|jdid d|dD| jdidd|dD| jdidd|dDq|D]}||j |_ | |j |_ | |j |_ q||D] }|j jdkrd|_ qdS)a Requests the Hub "preupload" endpoint to determine whether each input file should be uploaded as a regular git blob, as a git LFS blob, or as a XET file. Input `additions` are mutated in-place with the upload mode. Args: additions (`Iterable` of :class:`CommitOperationAdd`): Iterable of :class:`CommitOperationAdd` describing the files to upload to the Hub. repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. headers (`Dict[str, str]`): Headers to use for the request, including authorization headers and user agent. revision (`str`): The git revision to upload the files to. Can be any valid git revision. gitignore_content (`str`, *optional*): The content of the `.gitignore` file to know which files should be ignored. The order of priority is to first check if `gitignore_content` is passed, then check if the `.gitignore` file is present in the list of files to commit and finally default to the `.gitignore` file already hosted on the Hub (if any). Raises: [`~utils.HfHubHTTPError`] If the Hub API returned an error. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If the Hub API response is improperly formatted. Nr/rcSs.g|]}|jt|jjd|jjdqS)ascii)rXsamplesize)r1rprqrOrdecoderrr<r<r=rsz'_fetch_upload_modes.. gitIgnorez/api/zs/z /preupload/rr)jsonrrcSi|] }|d|dqS)rXrr<rrmr<r<r=rrz'_fetch_upload_modes..cSr)rX shouldIgnorer<rr<r<r=rrcSsi|] }|d|dqS)rXrrrr<r<r=rsrr.r<)rENDPOINTr!r#postr$rrrr1rQrRrSrOr)rrrrrrrr upload_modesshould_ignore_infooid_inforpayloadrespradditionr<r<r=_fetch_upload_modesfs>'      r copiesr+c Csddlm}m}|||d}i} i} dd|D} tdt| tD]#} |j|| | | t||d} | D]}t||sC|j| |j |f<q4q!t |dd d D]\}}t |}d d|D}tdt|tD]T} |j||| | t|ps||d}|D]>}t||rt d |j| |j |f<|j r|| |j |f<qyt||||p||j d }tj||d}t||j| |j |f<qyqd|D]*}|j|f| vrtd|jd|p|d| |j|jf|_| |j|f|_qqM| S)a; Fetch information about the files to copy. For LFS files, we only need their metadata (file size and sha256) while for regular files we need to download the raw content from the Hub. Args: copies (`Iterable` of :class:`CommitOperationCopy`): Iterable of :class:`CommitOperationCopy` describing the files to copy on the Hub. repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. headers (`Dict[str, str]`): Headers to use for the request, including authorization headers and user agent. revision (`str`): The git revision to upload the files to. Can be any valid git revision. Returns: `Dict[Tuple[str, Optional[str]], Union[RepoFile, bytes]]]` Key is the file path and revision of the file to copy. Value is the raw content as bytes (for regular files) or the file information as a RepoFile (for LFS files). Raises: [`~utils.HfHubHTTPError`] If the Hub API returned an error. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If the Hub API response is improperly formatted. r)HfApi RepoFolder)rrcSrr<rrr<r<r=rrz(_fetch_files_to_copy..r)rrrrcSs|jSrJ)rG)rr<r<r=sz&_fetch_files_to_copy..)keycSrr<)rFrr<r<r=rrz$Copying a folder is not implemented.)rrrrfilename)rz Cannot copy z at revision z: file is missing on repo.)hf_apirrrangerFETCH_LFS_BATCH_SIZEget_paths_infor7blob_idrXrrNotImplementedErrorr-rr#rr$contentrFrrGrHr1rI)rrrrrrrrr files_to_copyr  dest_pathsoffsetdest_repo_filesrmrGr src_pathssrc_repo_files src_repo_fileurlrrr<r<r=_fetch_files_to_copysn'    r#rcommit_messagecommit_description parent_commitc cs|dur|nd}||d}|dur||d<d|dVd}|D]}t|tr9|jr9td|jd |d 7}q t|trS|jd krSd ||jd ddVq t|trp|jdkrpd|jd|j j |j j ddVq t|t r|jrzdndd|jidVq t|tr||j|jf}t|trd t||jd ddVq |jrd|jd|jj ddVq tdtd|dt|dd|dkrtd|ddSdS)aG Builds the payload to POST to the `/commit` API of the Hub. Payload is returned as an iterator so that it can be streamed as a ndjson in the POST request. For more information, see: - https://github.com/huggingface/huggingface_hub/issues/1085#issuecomment-1265208073 - http://ndjson.org/ N)summaryr parentCommitheader)rvaluerzSkipping file 'z(' in commit (ignored by gitignore file).rr.rmrp)rrXencodingr-lfsFileru)rXalgorr deletedFolder deletedFilerX)rXr.rz_Malformed files_to_copy (should be raw file content as bytes or RepoFile objects with LFS info.z(Unknown operation to commit. Operation: z. Upload mode: rQzSkipped z/ file(s) in commit (ignored by gitignore file).)r7rKrRrrr1rQrtrrOrurvrr0r3rErFrGr^rprqr-r9getattrinfo) rrr$r%r& header_valuenb_ignored_filesr file_to_copyr<r<r=_prepare_commit_payload!st              r6)NFNrJ)NN)UrBrpr\rrWr collectionsr contextlibr dataclassesrr itertoolsrpathlibrrtypingr r r r r rrrrrrtqdm.contrib.concurrentrr'rerrorsrrrr file_downloadrr-rrrutilsrr r!r"r#r$r%r&r'r(r)r utils.tqdmr*rr+ get_loggerr?rrzrrr0rErKrCr5CommitOperationrrrr8rrrr r^r#r6r<r<r<r=s   4   0   #&.  n    T  i