o
hW9 @ s d dl Z d dlZd dlZd dlZd dlmZ d dlmZ d dlm Z m
Z
m Z d dl m
Z
d dlmZmZ d dlmZmZmZmZ dd lmZmZmZmZ dd
lm Z e e Z ed d G d
d dZ G dd dZ!G dd de Z"dS ) N)Future) dataclass)SEEK_ENDSEEK_SETBytesIO)Path)LockThread)DictListOptionalUnion )DEFAULT_IGNORE_PATTERNS
CommitInfoCommitOperationAddHfApi)filter_repo_objectsT)frozenc @ s2 e Zd ZU dZeed< eed< eed< eed< dS )
_FileToUploadzWTemporary dataclass to store info about files to upload. Not meant to be used directly.
local_path path_in_repo
size_limit
last_modifiedN) __name__
__module__ __qualname____doc__r __annotations__strintfloat r" r" u/var/www/html/construction_image-detection-poc/venv/lib/python3.10/site-packages/huggingface_hub/_commit_scheduler.pyr s
r c @ s e Zd ZdZddddddddddd
dedeeef deeef d e e d
e e d e e d e e
d
e e de ee e ef de ee e ef de
de d ddfddZ d$ddZ
d%ddZd$ddZd$d d Zdefd d Zde e fd d!Zde e fd"d#ZdS )&CommitSchedulera
Scheduler to upload a local folder to the Hub at regular intervals (e.g. push to hub every 5 minutes).
The recommended way to use the scheduler is to use it as a context manager. This ensures that the scheduler is
properly stopped and the last commit is triggered when the script ends. The scheduler can also be stopped manually
with the `stop` method. Checkout the [upload guide](https://huggingface.co/docs/huggingface_hub/guides/upload#scheduled-uploads)
to learn more about how to use it.
Args:
repo_id (`str`):
The id of the repo to commit to.
folder_path (`str` or `Path`):
Path to the local folder to upload regularly.
every (`int` or `float`, *optional*):
The number of minutes between each commit. Defaults to 5 minutes.
path_in_repo (`str`, *optional*):
Relative path of the directory in the repo, for example: `"checkpoints/"`. Defaults to the root folder
of the repository.
repo_type (`str`, *optional*):
The type of the repo to commit to. Defaults to `model`.
revision (`str`, *optional*):
The revision of the repo to commit to. Defaults to `main`.
private (`bool`, *optional*):
Whether to make the repo private. If `None` (default), the repo will be public unless the organization's default is private. This value is ignored if the repo already exists.
token (`str`, *optional*):
The token to use to commit to the repo. Defaults to the token saved on the machine.
allow_patterns (`List[str]` or `str`, *optional*):
If provided, only files matching at least one pattern are uploaded.
ignore_patterns (`List[str]` or `str`, *optional*):
If provided, files matching any of the patterns are not uploaded.
squash_history (`bool`, *optional*):
Whether to squash the history of the repo after each commit. Defaults to `False`. Squashing commits is
useful to avoid degraded performances on the repo when it grows too large.
hf_api (`HfApi`, *optional*):
The [`HfApi`] client to use to commit to the Hub. Can be set with custom settings (user agent, token,...).
Example:
```py
>>> from pathlib import Path
>>> from huggingface_hub import CommitScheduler
# Scheduler uploads every 10 minutes
>>> csv_path = Path("watched_folder/data.csv")
>>> CommitScheduler(repo_id="test_scheduler", repo_type="dataset", folder_path=csv_path.parent, every=10)
>>> with csv_path.open("a") as f:
... f.write("first line")
# Some time later (...)
>>> with csv_path.open("a") as f:
... f.write("second line")
```
Example using a context manager:
```py
>>> from pathlib import Path
>>> from huggingface_hub import CommitScheduler
>>> with CommitScheduler(repo_id="test_scheduler", repo_type="dataset", folder_path="watched_folder", every=10) as scheduler:
... csv_path = Path("watched_folder/data.csv")
... with csv_path.open("a") as f:
... f.write("first line")
... (...)
... with csv_path.open("a") as f:
... f.write("second line")
# Scheduler is now stopped and last commit have been triggered
```
NF)
everyr repo_typerevisionprivatetokenallow_patternsignore_patternssquash_historyhf_apirepo_id folder_pathr& r r' r( r) r* r+ r, r- r. r returnc C s: | pt |d| _t| | _|pd| _| | _|
d u r g }
nt|
t r(|
g}
|
t
| _ | j r;t
d| j d| jjddd | jj|||dd}
|
j| _|| _|| _|| _i | _|dksit
d | dt | _|| _| | _td
| j d | j d | j d
t| j dd| _ | j t | j! d| _"d S )N)r* z0'folder_path' must be a directory, not a file: ''.T)parentsexist_ok)r/ r) r' r5 r z)'every' must be a positive integer, not 'zScheduled job to push 'z' to 'z' every z minutes.)targetdaemonF)#r apir
expanduserresolver0 r r+
isinstancer r r, is_file
ValueErrormkdir create_repor/ r' r( r*
last_uploadedr lockr& r- loggerinfor _run_scheduler_scheduler_threadstartatexitregister _push_to_hub_CommitScheduler__stopped)selfr/ r0 r& r r' r( r) r* r+ r, r- r. repo_urlr" r" r# __init__d s8
$
zCommitScheduler.__init__c C s
d| _ dS )ziStop the scheduler.
A stopped scheduler cannot be restarted. Mostly for tests purposes.
TN)rJ rK r" r" r# stop s
zCommitScheduler.stopc C s | S Nr" rN r" r" r# __enter__ s zCommitScheduler.__enter__c C s | | d S rP )triggerresultrO )rK exc_type exc_value tracebackr" r" r# __exit__ s zCommitScheduler.__exit__c C s( | | _t| jd | jrdS q)z7Dumb thread waiting between each scheduled push to Hub.T< N)rR last_futuretimesleepr& rJ rN r" r" r# rD s
z CommitScheduler._run_schedulerc C s | j | jS )zTrigger a `push_to_hub` and return a future.
This method is automatically called every `every` minutes. You can also call it manually to trigger a commit
immediately, without waiting for the next scheduled commit.
)r8
run_as_futurerI rN r" r" r# rR s zCommitScheduler.triggerc
C sv | j rd S td z| }| jr#td | jj| j| j| j d |W S t
y: } z t d| d }~ww )Nz((Background) scheduled commit triggered.z$(Background) squashing repo history.)r/ r' branchz Error while pushing to Hub: ) rJ rB rC push_to_hubr- r8 super_squash_historyr/ r' r( Exceptionerror)rK valueer" r" r# rI s
z CommitScheduler._push_to_hubc
sL j a td fddt jdD } jr$ jd dnd}g }t| j
j dD ])}|| }| } j
|du sM j
| |jkr\|t||| |j|jd q3W d n1 sgw Y t|d
krytd dS td d
d |D }td jj j j|d jd}|D ] } | j j
| j< q|S )a
Push folder to the Hub and return the commit info.
This method is not meant to be called directly. It is run in the background by the scheduler, respecting a
queue mechanism to avoid concurrent commits. Making a direct call to the method might lead to concurrency
issues.
The default behavior of `push_to_hub` is to assume an append-only folder. It lists all files in the folder and
uploads only changed files. If no changes are found, the method returns without committing anything. If you want
to change this behavior, you can inherit from [`CommitScheduler`] and override this method. This can be useful
for example to compress data together in a single file before committing. For more details and examples, check
out our [integration guide](https://huggingface.co/docs/huggingface_hub/main/en/guides/upload#scheduled-uploads).
z-Listing files to upload for scheduled commit.c s& i | ]}| r| j |qS r" )r< relative_tor0 as_posix).0pathrN r" r#
s z/CommitScheduler.push_to_hub..z**/*/r2 )r+ r, N)r r r r r z4Dropping schedule commit: no changed file to upload.z9Removing unchanged files since previous scheduled commit.c S s& g | ]}t t|j|jd |jdqS ))r )path_or_fileobjr )r
PartialFileIOr r r )rf file_to_uploadr" r" r#
s z/CommitScheduler.push_to_hub..z%Uploading files for scheduled commit.zScheduled Commit)r/ r'
operationscommit_messager( )rA rB debugsortedr0 globr stripr keysr+ r, statr@ getst_mtimeappendr st_sizelenr8
create_commitr/ r' r( r r )
rK relpath_to_abspathprefixfiles_to_uploadrelpathr ru add_operations commit_infofiler" rN r# r^ sT
zCommitScheduler.push_to_hubr1 N)r1 r$ )r r r r r r
r r r! r boolr rM rO rQ rW rD r rR r rI r^ r" r" r" r# r$ s\ K
9
r$ c s e Zd ZdZdeeef deddfddZd fdd Z defd
d Z
defd d
Z def fdd Z defddZ
efdededefddZd dee defddZ ZS ) rk a A file-like object that reads only the first part of a file.
Useful to upload a file to the Hub when the user might still be appending data to it. Only the first part of the
file is uploaded (i.e. the part that was available when the filesystem was first scanned).
In practice, only used internally by the CommitScheduler to regularly push a folder to the Hub with minimal
disturbance for the user. The object is passed to `CommitOperationAdd`.
Only supports `read`, `tell` and `seek` methods.
Args:
file_path (`str` or `Path`):
Path to the file to read.
size_limit (`int`):
The maximum number of bytes to read from the file. If the file is larger than this, only the first part
will be read (and uploaded).
file_pathr r1 Nc C s6 t || _| jd| _t|t| j j| _ d S )Nrb)
r
_file_pathopen_fileminosfstatfilenory _size_limit)rK r r r" r" r# rM . s
zPartialFileIO.__init__c s | j t S rP )r closesuper__del__rN __class__r" r# r 3 s
zPartialFileIO.__del__c C s d| j d| j dS )Nz)r r rN r" r" r# __repr__7 s zPartialFileIO.__repr__c C s | j S rP )r rN r" r" r# __len__: s zPartialFileIO.__len__namec s. | ds |dv rt |S td| d)N_)readtellseekz PartialFileIO does not support 'r3 )
startswithr __getattribute__NotImplementedError)rK r r r" r# r = s z PartialFileIO.__getattribute__c C s
| j S )z!Return the current file position.)r r rN r" r" r# r B s
zPartialFileIO.tell_PartialFileIO__offset_PartialFileIO__whencec C sB |t kr t| | }t}| j||}|| jkr | j| jS |S )zChange the stream position to the given offset.
Behavior is the same as a regular file, except that the position is capped to the size limit.
)r rz r r r r )rK r r posr" r" r# r F s
zPartialFileIO.seek_PartialFileIO__sizec C sB | j }|du s
|dk r| j| }nt|| j| }| j |S )zRead at most `__size` bytes from the file.
Behavior is the same as a regular file, except that it is capped to the size limit.
Nr )r r r r r )rK r currenttruncated_sizer" r" r# r U s
zPartialFileIO.readr )r )r r r r r
r r r rM r r r r r r r r bytesr
__classcell__r" r" r r# rk s rk )#rG loggingr rZ concurrent.futuresr dataclassesr ior r r pathlibr threadingr r typingr
r r r
r. r r r r utilsr getLoggerr rB r r$ rk r" r" r" r# s$