progiter package

Module contents

ProgIter lets you measure and print the progress of an iterative process. This can be done either via an iterable interface or using the manual api. Using the iterable inferface is most common.

ProgIter was originally developed independantly of tqdm, but the newer versions of this library have been designed to be compatible with tqdm-API. ProgIter is now a (mostly) drop-in alternative to tqdm_. The tqdm library may be more appropriate in some cases. The main advantage of ``ProgIter`` is that it does not use any python threading, and therefore can be safer with code that makes heavy use of multiprocessing. `The reason`_ for this is that threading before forking may cause locks to be duplicated across processes, which may lead to deadlocks.

ProgIter is simpler than tqdm, which may be desirable for some applications. However, this also means ProgIter is not as extensible as tqdm. If you want a pretty bar, use tqdm; if you want useful information (rate, fraction-complete, estimated time remaining, time taken so far, current wall time) about your iteration by default, use progiter.

Example

The basic usage of ProgIter is simple and intuitive. Just wrap a python iterable. The following example wraps a range iterable and prints reported progress to stdout as the iterable is consumed. The ProgIter object accepts various keyword arguments to modify the details of how progress is measured and reported. See API documentation of the ProgIter classs here: https://progiter.readthedocs.io/en/latest/progiter.progiter.html#progiter.progiter.ProgIter

Note that by default ProgIter reports information about iteration-rate, fraction-complete, estimated time remaining, time taken so far, and the current wall time.

>>> from progiter import ProgIter
>>> def is_prime(n):
...     return n >= 2 and not any(n % i == 0 for i in range(2, n))
>>> for n in ProgIter(range(1000), verbose=2):
>>>     # do some work
>>>     is_prime(n)
    0/1000... rate=0.00 Hz, eta=?, total=0:00:00, wall=14:05 EST
    1/1000... rate=82241.25 Hz, eta=0:00:00, total=0:00:00, wall=14:05 EST
  257/1000... rate=177204.69 Hz, eta=0:00:00, total=0:00:00, wall=14:05 EST
  642/1000... rate=94099.22 Hz, eta=0:00:00, total=0:00:00, wall=14:05 EST
 1000/1000... rate=71886.74 Hz, eta=0:00:00, total=0:00:00, wall=14:05 EST
class progiter.ProgIter(iterable=None, desc=None, total=None, freq=1, initial=0, eta_window=64, clearline=True, adjust=True, time_thresh=2.0, show_times=True, enabled=True, verbose=None, stream=None, chunksize=None, **kwargs)[source]

Bases: progiter.progiter._TQDMCompat, progiter.progiter._BackwardsCompat

Prints progress as an iterator progresses

ProgIter is an alternative to tqdm. ProgIter implements much of the tqdm-API. The main difference between ProgIter and tqdm is that ProgIter does not use threading where as tqdm does.

Variables:
  • iterable (iterable) – An iterable iterable
  • desc (str) – description label to show with progress
  • total (int) – Maximum length of the process. If not specified, we estimate it from the iterable, if possible.
  • freq (int, default=1) – How many iterations to wait between messages.
  • adjust (bool, default=True) – if True freq is adjusted based on time_thresh
  • eta_window (int, default=64) – number of previous measurements to use in eta calculation
  • clearline (bool, default=True) – if True messages are printed on the same line otherwise each new progress message is printed on new line.
  • adjust – if True freq is adjusted based on time_thresh
  • time_thresh (float, default=2.0) – desired amount of time to wait between messages if adjust is True otherwise does nothing
  • show_times (bool, default=True) – shows rate, eta, and wall (defaults to True)
  • initial (int, default=0) – starting index offset (defaults to 0)
  • stream (file, default=sys.stdout) – stream where progress information is written to
  • enabled (bool, default=True) – if False nothing happens.
  • chunksize (int, optional) – indicates that each iteration processes a batch of this size. Iteration rate is displayed in terms of single-items.
  • verbose (int) – verbosity mode, which controls clearline, adjust, and enabled. The following maps the value of verbose to its effect. 0: enabled=False, 1: enabled=True with clearline=True and adjust=True, 2: enabled=True with clearline=False and adjust=True, 3: enabled=True with clearline=False and adjust=False

Note

Either use ProgIter in a with statement or call prog.end() at the end of the computation if there is a possibility that the entire iterable may not be exhausted.

Note

ProgIter is an alternative to tqdm. The main difference between ProgIter and tqdm is that ProgIter does not use threading where as tqdm does. ProgIter is simpler than tqdm and thus more stable in certain circumstances.

SeeAlso:
tqdm - https://pypi.python.org/pypi/tqdm

References

http://datagenetics.com/blog/february12017/index.html

Example

>>> # doctest: +SKIP
>>> def is_prime(n):
...     return n >= 2 and not any(n % i == 0 for i in range(2, n))
>>> for n in ProgIter(range(100), verbose=1):
>>>     # do some work
>>>     is_prime(n)
100/100... rate=... Hz, total=..., wall=... EST
set_extra(extra)[source]

specify a custom info appended to the end of the next message

Todo

  • [ ] extra is a bad name; come up with something better and rename

Example

>>> prog = ProgIter(range(100, 300, 100), show_times=False, verbose=3)
>>> for n in prog:
>>>     prog.set_extra('processesing num {}'.format(n))
0/2...
1/2...processesing num 100
2/2...processesing num 200
step(inc=1)[source]

Manually step progress update, either directly or by an increment.

Parameters:inc (int, default=1) – number of steps to increment

Example

>>> n = 3
>>> prog = ProgIter(desc='manual', total=n, verbose=3)
>>> # Need to manually begin and end in this mode
>>> prog.begin()
>>> for _ in range(n):
...     prog.step()
>>> prog.end()

Example

>>> n = 3
>>> # can be used as a context manager in manual mode
>>> with ProgIter(desc='manual', total=n, verbose=3) as prog:
...     for _ in range(n):
...         prog.step()
begin()[source]

Initializes information used to measure progress

This only needs to be used if this ProgIter is not wrapping an iterable. Does nothing if the this ProgIter is disabled.

end()[source]

Signals that iteration has ended and displays the final message.

This only needs to be used if this ProgIter is not wrapping an iterable. Does nothing if the this ProgIter object is disabled or has already finished.

format_message()[source]

builds a formatted progres message with the current values. This contains the special characters needed to clear lines.

Example

>>> self = ProgIter(clearline=False, show_times=False)
>>> print(repr(self.format_message()))
'    0/?... \n'
>>> self.begin()
>>> self.step()
>>> print(repr(self.format_message()))
' 1/?... \n'

Example

>>> self = ProgIter(chunksize=10, total=100, clearline=False,
>>>                 show_times=False, microseconds=True)
>>> # hack, microseconds=True for coverage, needs real test
>>> print(repr(self.format_message()))
' 0.00% of 10x100... \n'
>>> self.begin()
>>> self.update()  # tqdm alternative to step
>>> print(repr(self.format_message()))
' 1.00% of 10x100... \n'
ensure_newline()[source]

use before any custom printing when using the progress iter to ensure your print statement starts on a new line instead of at the end of a progress line

Example

>>> # Unsafe version may write your message on the wrong line
>>> prog = ProgIter(range(4), show_times=False, verbose=1)
>>> for n in prog:
...     print('unsafe message')
 0/4...  unsafe message
 1/4...  unsafe message
unsafe message
unsafe message
 4/4...
>>> # apparently the safe version does this too.
>>> print('---')
---
>>> prog = ProgIter(range(4), show_times=False, verbose=1)
>>> for n in prog:
...     prog.ensure_newline()
...     print('safe message')
 0/4...
safe message
 1/4...
safe message
safe message
safe message
 4/4...
display_message()[source]

Writes current progress to the output stream