o h@ @sVdZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z eeZeGdd d Zed d Gd d d ZeGdddZeGdddZde dedefddZde dedefddZde dede efddZde dedefddZde dedededdf dd Zde de fd!d"Zdedefd#d$ZdS)%u$Contains utilities to handle the `../.cache/huggingface` folder in local directories. First discussed in https://github.com/huggingface/huggingface_hub/issues/1738 to store download metadata when downloading files from the hub to a local directory (without using the cache). ./.cache/huggingface folder structure: [4.0K] data ├── [4.0K] .cache │ └── [4.0K] huggingface │ └── [4.0K] download │ ├── [ 16] file.parquet.metadata │ ├── [ 16] file.txt.metadata │ └── [4.0K] folder │ └── [ 16] file.parquet.metadata │ ├── [6.5G] file.parquet ├── [1.5K] file.txt └── [4.0K] folder └── [ 16] file.parquet Download metadata file structure: ``` # file.txt.metadata 11c5a3d5811f50298f278a704980280950aedb10 a16a55fda99d2f2e7b69cce5cf93ff4ad3049930 1712656091.123 # file.parquet.metadata 11c5a3d5811f50298f278a704980280950aedb10 7c5d3f4b8b76583b422fcb9189ad6c89d5d97a094541ce8932dce3ecabde1421 1712656091.123 } ``` N) dataclass)Path)Optional) WeakFileLockc@s<eZdZUdZeed<eed<eed<dedefddZd S) LocalDownloadFilePathsa Paths to the files related to a download process in a local dir. Returned by [`get_local_download_paths`]. Attributes: file_path (`Path`): Path where the file will be saved. lock_path (`Path`): Path to the lock file used to ensure atomicity when reading/writing metadata. metadata_path (`Path`): Path to the metadata file. file_path lock_path metadata_pathetagreturncCs |jjt|jjd|dS)z^Return the path where a file will be temporarily downloaded before being moved to `file_path`..z .incomplete)r parent _short_hashname)selfr rq/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/huggingface_hub/_local_folder.pyincomplete_pathWs z&LocalDownloadFilePaths.incomplete_pathN)__name__ __module__ __qualname____doc__r__annotations__strrrrrrrCs rT)frozenc@s2eZdZUdZeed<eed<eed<eed<dS)LocalUploadFilePathsa Paths to the files related to an upload process in a local dir. Returned by [`get_local_upload_paths`]. Attributes: path_in_repo (`str`): Path of the file in the repo. file_path (`Path`): Path where the file will be saved. lock_path (`Path`): Path to the lock file used to ensure atomicity when reading/writing metadata. metadata_path (`Path`): Path to the metadata file. path_in_reporr r N)rrrrrrrrrrrr\s  rc@s2eZdZUdZeed<eed<eed<eed<dS)LocalDownloadFileMetadataaG Metadata about a file in the local directory related to a download process. Attributes: filename (`str`): Path of the file in the repo. commit_hash (`str`): Commit hash of the file in the repo. etag (`str`): ETag of the file in the repo. Used to check if the file has changed. For LFS files, this is the sha256 of the file. For regular files, it corresponds to the git hash. timestamp (`int`): Unix timestamp of when the metadata was saved i.e. when the metadata was accurate. filename commit_hashr timestampN)rrrrrrfloatrrrrrts  rc@seZdZUdZeed<dZeeed<dZ ee ed<dZ ee ed<dZ ee ed<dZe ed <dZe ed <d ed dfd dZdS)LocalUploadFileMetadatazT Metadata about a file in the local directory related to an upload process. sizeNr! should_ignoresha256 upload_modeF is_uploaded is_committedpathsr c Cs.t|j|jdn}t}|t|d|t|j|d|jdur6|tt |j|d|j durF||j |d|j durV||j |d|tt |j d|tt |j dWdn1s}wY||_WddS1swYdS)zSave the metadata to disk.w N)rr r opentimewriterr$r%intr&r'r(r)r!)rr*f new_timestamprrrsaves(          "zLocalUploadFileMetadata.save)rrrrr0rr!rr"r%boolr&rr'r(r)rr3rrrrr#s   r# local_dirrr cCstjj|d}tjdkr|dsd|vrtd|d||}t|d|d}|d }tjdkrgt |d sgt tj |d krgt d tj |}t d tj |}t d tj |}|j jd d d |j jd d d t|||dS)aCompute paths to the files related to a download process. Folders containing the paths are all guaranteed to exist. Args: local_dir (`Path`): Path to the local directory in which files are downloaded. filename (`str`): Path of the file in the repo. Return: [`LocalDownloadFilePaths`]: the paths to the files (file_path, lock_path, metadata_path, incomplete_path). /nt..\\..\*Invalid filename: cannot handle filename 'B' on Windows. Please ask the repository owner to rename this file.download .metadata.lock\\?\Tparentsexist_ok)rr r )ospathjoinsplitr startswith ValueError_huggingface_dir with_suffixrlenabspathrrmkdirrr5rsanitized_filenamerr r rrrget_local_download_pathss"    "rQcCstjj|d}tjdkr|dsd|vrtd|d||}t|d|d}|d }tjdkrgt |d sgt tj |d krgt d tj |}t d tj |}t d tj |}|j jd d d |j jd d d t||||dS)aCompute paths to the files related to an upload process. Folders containing the paths are all guaranteed to exist. Args: local_dir (`Path`): Path to the local directory that is uploaded. filename (`str`): Path of the file in the repo. Return: [`LocalUploadFilePaths`]: the paths to the files (file_path, lock_path, metadata_path). r6r7r8r9r:r;uploadr=r>r?r@TrA)rrr r )rDrErFrGrrHrIrJrKrrLrMrrrNrrOrrrget_local_upload_pathss&    "rSc Cst||}t|j|jrz3|j$}|}|}t|}t ||||d}Wdn1s=wYWnEt y}z9t d|jd|dz|j Wnt y}}zt d|jd|WYd}~nd}~wwWYd}~nd}~wwz"|j} | jd|jkr|WWdSt d|d WntyYWddSwWddSWddS1swYdS) arRead metadata about a file in the local directory related to a download process. Args: local_dir (`Path`): Path to the local directory in which files are downloaded. filename (`str`): Path of the file in the repo. Return: `[LocalDownloadFileMetadata]` or `None`: the metadata if it exists, `None` otherwise. )rr r r!NInvalid metadata file : %. Removing it from disk and continue.)Could not remove corrupted metadata file rIgnored metadata for '#' (outdated). Will re-compute hash.)rQrr r existsr-readlinestripr"r Exceptionloggerwarningunlinkrstatst_mtimer!infoFileNotFoundError) r5rr*r1r r r!metadataerarrrread_download_metadata s\      $   " ""rgcCsvt||}t|j|jr"z|jt}t|}t |}|}|dkr5dnt t |}|}|dkrGdn|} |} | dkrUdn| } | dvrft d|j d| t t |} t t |} t |||| | | | d}Wdn1swYWnEty}z9td|jd|dz|jWnty}ztd |jd|WYd}~nd}~wwWYd}~nd}~ww|jdur|jr|jst|jd krd |_z%|jdur |jj|jkr |WWdStd |d Wn ty!YnwWdn 1s-wYt |jjdS)aRead metadata about a file in the local directory related to an upload process. TODO: factorize logic with `read_download_metadata`. Args: local_dir (`Path`): Path to the local directory in which files are downloaded. filename (`str`): Path of the file in the repo. Return: `[LocalUploadFileMetadata]` or `None`: the metadata if it exists, `None` otherwise. N)Nregularlfsz Invalid upload mode in metadata rU)r!r$r%r&r'r(r)rTrVrWi@FrXrY)r$)rSrr r rZr-r"r[r\r0r4rIrr#r]r^r_r`r!r(r)r.rrarbrcrdst_size)r5rr*r1r!r$_should_ignorer%_sha256r& _upload_moder'r(r)rerfrrrread_upload_metadata;st      $  6s4%     +(*2N