--- jupytext: formats: md:myst text_representation: extension: .md format_name: myst kernelspec: display_name: Python 3 language: python name: python3 --- # Iterators ## Introduction **Iterating** means doing something repeatedly {cite}`CambridgeDict_iteration`, and in software development, it typically refers to accessing data elements one by one. An **iterator** is an object that enables iteration over a collection or stream of data {cite}`PythonDocs_Iterator`. ````{admonition} Iterator :class: tip Iterators provide a mechanism to **access items sequentially** (one-by-one) from an **iterable source of data**, such as a stream or collection (e.g., a dictionary or list). ```` This raises an important question: what qualifies as an *iterable source of data*? ````{admonition} Iterable source of data :class: hint An iterable source of data is any source from which elements can be retrieved sequentially. Syntactically, it is an object that implements the `__iter__` method (see [Iterable sources of data](#iterable-sources-of-data)). ```` Iterators are most commonly encountered in `for` loops. By convention, when looping over an object: ```{code-block} python :lineno-start: 1 for item in my_collection: print(item) ``` Python internally transforms this into the following equivalent code: ```{code-block} python :lineno-start: 1 :emphasize-lines: 1,4,6 iterator = iter(my_collection) while True: try: item = next(iterator) print(item) except StopIteration: break ``` This transformation demonstrates how Python uses iterators under the hood. To access successive values from a collection, we use an **iterator** object that must comply with the iterator protocol (see [Iterator protocol](#iterator-protocol)). This logic can be seen in the example below: ```{raw} html

Iterating through data accessible via iterator

Data Stream/Collection

💧 Iterator is exhausted - no more data to retrieve

# Click "Call next(iterator)" to retrieve values from the collection

ℹ️ How it works: Each droplet represents a data value in the collection. Calling next(iterator) points to and retrieves the next value in sequence. The original collection remains unchanged.

``` ## Iterating over an object An **iterator** object has a straightforward requirement: it must implement the dunder `__next__` method with the following signature: ```{code-block} python :lineno-start: 1 from typing import Any class MyDummyIterator: def __next__(self) -> Any: pass ``` This is an argumentless[^argumentless] method whose purpose is to produce successive elements from a collection or stream of data. When the iterator is exhausted, it must raise a [`StopIteration`](https://docs.python.org/3/library/exceptions.html#StopIteration) exception. ````{admonition} Exhausted iterator :class: important An **exhausted** iterator has no more data to return. Once exhausted, each successive call to the `__next__` method must raise the built-in [`StopIteration`](https://docs.python.org/3/library/exceptions.html#StopIteration) exception. Violating this requirement will cause the iterator to malfunction and prevent the `for` loop from terminating properly. ```` Consider the following example, which appears to work correctly: ```{code-block} python :lineno-start: 1 :emphasize-lines: 8 class MyIterator: def __next__(self): return 1 class SomeIterableCollection: def __iter__(self): return MyIterator() for item in SomeIterableCollection(): print(item) ``` However, Python will not recognize this as a proper iterator: ```{code-block} python :lineno-start: 1 :tags: ["raises-exception"] from collections.abc import Iterator class MyIterator: def __next__(self): return 1 assert isinstance(MyIterator(), Iterator) ``` Why does this assertion fail? Python requires iterators to follow the complete iterator protocol (see [Iterator protocol](#iterator-protocol)), which demands that **iterators must themselves be iterable objects**. This ensures iterators can be used seamlessly in `for` loops. The example above violates this requirement, as demonstrated when attempting to iterate directly over the iterator: ```{code-block} python :lineno-start: 1 class MyIterator: def __next__(self): return 1 for item in MyIterator(): print(item) ``` ```{code-cell} python :tags: ["remove-input","raises-exception"] from collections.abc import Iterator class MyIterator: def __next__(self): return 1 for item in MyIterator(): print(item) ``` ## Iterator protocol To make an iterator a proper iterable object, two methods must be implemented: 1. `__next__(self)` — returns the next element of an iterator 2. `__iter__(self)` — returns an iterator (conventionally, the object itself) With both methods implemented correctly, the following code executes without error: ```{code-block} python :lineno-start: 1 :emphasize-lines: 3,4 class MyIterator: def __iter__(self): return self def __next__(self): # This is an infinite iterator that never raises StopIteration return 1 for item in MyIterator(): print(item) ``` We can verify this implementation satisfies the iterator protocol: ```{code-block} python :lineno-start: 1 from collections.abc import Iterator assert isinstance(MyIterator(), Iterator) ``` ````{admonition} Iterator subclassing :class: important As demonstrated above, subclassing the [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator) abstract class is not required to create an iterator[^no_subclass] {cite}`python:abc_collections`. However, using the [`collections.abc`](https://docs.python.org/3/library/collections.abc.html) module is recommended when working with collections and iterators. When explicitly subclassing [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator), the `__iter__` method need not be implemented explicitly, as the superclass provides it as a mixin method. ```{code-block} python :lineno-start: 1 :emphasize-lines: 3 from collections.abc import Iterator class MyIterator(Iterator): def __next__(self): # Note: this is an infinite iterator # as it never raises StopIteration exception return 1 ``` ```` [^argumentless]: An argumentless method is one that takes no explicit arguments beyond the implicit `self` or `cls` argument, which refers to the instance or type invoking the method. [^abstract]: An abstract method is one that must be implemented in a subclass before an instance of that class can be created. [^no_subclass]: The ability to check `isinstance` against [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator) without explicit subclassing is not magical. The [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator) abstract class implements a special `__subclasshook__` method that checks whether a class implements both `__iter__` and `__next__` methods. ````{admonition} Iterator can be asynchronous :class: hint To write asynchronous iterator, you should use methods `__anext__` and `__aiter__` ```` ## Iterable sources of data Having established how iterators work, let us examine iterable sources of data in greater detail. An object is iterable if it satisfies at least one of the following conditions {cite}`pep-0234`: 1. The object implements the `__getitem__` dunder method for data access (i.e., it is a sequential or mapping collection {cite}`python:abc_collections`): ```{code-cell} python my_list = [1, 2, 3] idx = 1 item = my_list[idx] # equivalent to my_list.__getitem__(idx) assert item == 2 ``` 2. The object implements the `__iter__` dunder method, which returns an **iterator** object (an instance of a type compliant with the [Iterator Protocol](#iterator-protocol)): ```{code-cell} python from collections.abc import Iterator my_list = [1, 2, 3] list_iter = iter(my_list) # or my_list.__iter__() assert isinstance(list_iter, Iterator) ```