what is an iterable ?

In python , objects are abstraction of data , they have methods that work with data , and help us to manipulate it . If we take a look at a list , and see all of its methods

>>> import json

>>> _list = []
# an empty list

>>> json_list = json.dumps(dir(_list), sort_keys=True, indent=4)
# dump the methods and properties of a list to a json object

>>> print(json_list)
[
    "__add__",
    "__class__",
    "__contains__",
    "__delattr__",
    "__delitem__",
    "__dir__",
    "__doc__",
    "__eq__",
    "__format__",
    "__ge__",
    "__getattribute__",
    "__getitem__",
    "__gt__",
    "__hash__",
    "__iadd__",
    "__imul__",
    "__init__",
    "__init_subclass__",
    "__iter__",
    "__le__",
    "__len__",
    "__lt__",
    "__mul__",
    "__ne__",
    "__new__",
    "__reduce__",
    "__reduce_ex__",
    "__repr__",
    "__reversed__",
    "__rmul__",
    "__setattr__",
    "__setitem__",
    "__sizeof__",
    "__str__",
    "__subclasshook__",
    "append",
    "clear",
    "copy",
    "count",
    "extend",
    "index",
    "insert",
    "pop",
    "remove",
    "reverse",
    "sort"
]
# print the json object

we can see that we can add elements to a list using add , and that we can remove an element from a list using remove , and we can also see that we have some methods which start with a double underscore . These methods are called special methods , and they are implemented by default on some of python data types such as list , and dictionaries ..

in python everything is an object , an integer a string , a double , a function a class .. they are all objects . A instance of a class is also called an object . An object has a type , it has data [properties , attributes] , and it has some methods . An iterable object is an object that defines a way to iterate over its data .

But what does iterate mean ? Iterate simply means to loop over each of its elements , and do something with , like printing or calculating the sum , or whatever. Let us give an example of an iterable , let us say we have some electronics in our house , and we want to store them in python , where should we store them ?

We can store them in a list . A list is just a collection of elements , that we can iterate over , hence we can access the elements that we have stored through looping .

# what is an iterable ?
# want to print the gadgets that we have
# store them in an iterable ?
#   yes
#       ?
#       -> an iterable enable looping through an object

gadgets = ['i7', 's8', 'samsung screen']  
# store the gadget in a list , which is an iterable
# to loop through the gadgets , because we want to print
# them

for gadget in gadgets:  
    # print the gadget
    print(f'{gadget}')

# output
i7
s8
samsung screen

Let us give another example , let us say , we have a list of pencil boxes , and each pencil box has a price . We want to store the pencil boxes with their price . Which python type should we use ? The answer is a dictionary : a dictionary is formed of a key and a value , and it helps us to loop through the data .

>>> pencilBoxes = {'pencilBoxA': 2.3, 'pencilBoxB': 1.5}
#  pencil boxes ?
#  price ?
#   {pencilBoxA : price }
#       ~ pencilBoxA : name of the pencil box
#       ~ price      : The price of the pencil in $


>>> print(f'The sum of the prices of the Pencil Boxes is :'
      f' {sum(pencilBoxes.values())}'
      # we use values to get the values inside the dictionary
      )

The sum of the prices of the Pencil Boxes is : 3.8
# calculate total Price

What is an iterator ?

As we have seen in the previous section , an iterable is an object that define a way to loop over its data , so its properties or attributes or methods .

An iterator is an object that enable us to traverse the data of an iterable. An object might define more than one way to loop over its data , hence one or more iterator .

Let us give an example . we want to search for something by using google . for example we want to search for some slides , as such we enter the keyword slide , in google search .

Google has all the data that is related to the keyword slide , this is the search object slide . The search object slide is an iterable since it defines a way to loop over its data . Google provide various iterators to traverse the search object slide , we can traverse it and filter it by date , by language ..

What is the iter function ?

The iter function is used to get an iterator from an object in python . It is used by the for loop , the sum … to get an iterator which is used to loop over the data . the iter function syntax is the following

iter(iterable) -> iterator
iter(callable, sentinel) -> iterator

How to create an iterator in python

In python everything is an object , an object has a type , it can can have some data and some methods . An iterable is an object which type defines a way to loop over its data by the use of an iterator which is an object that enable us to traverse the data of an iterable .

we can get an iterator from an iterable object in python through the use of the iter method .

if an object is an instance of a class

• • we can implement in this class the __iter__ method . The __iter__ method must return an instance of a class that has the __next__ method. The __next__method returns the next data or raises a StopIteration exception when there is no more data to get . The iterator is gotten from this object by using iter(object)
  • we can implement the __getitem__ method . __getitem__ returns data based on a provided index , the index will start from zero . __getitem__ raises a IndexError , when there is no more data. The iterator is gotten by using iter(obj)

if the object is a function or is a callable .

• • • A callable is an instance of a class that implements the __callable__ method , so a callable is an instance of a class that can be called like a regular function , a regular function is callable .
    • we can get an iterator from this object by using the iter(callable, sentinel) -> iterator method . This method takes the callable , and call it without providing any value , until the value returned by this callable is equal to the sentinel . A sentinel is just a regular value to stop the iteration .When the returned value is equal to the sentinel , the iterator will raise a StopIteration , to stop the iteration over this iterable . The iterable in this case is the function that we have provided , the iterator is created by using iter which specify when to stop the iteration through the sentinel .

Object is instance of a class

implement the __iter__ and __next__ methods

If we want to create an iterable from an object which is an instance of a class , we must define the __iter__ method . The __iter__ method is called by the iter function , and it must return an iterator . An iterator is simply an object which implements the __next__ method .

class Loop:

    def __init__(self, maxLoop=5):
        self.maxLoop = maxLoop

    def __iter__(self):
        self.loop = 0  
        # reset the iterator when __iter__ 
        # is called
        return self

    def __next__(self):
        if self.loop >= self.maxLoop:
            raise StopIteration
        else:
            self.loop += 1  
            return self.loop

Let us say we want to create the loop object , that iterate over some numbers. The loop object is an instance of the Loop class which as such is an iterable .

The Loop class , defines the __iter__ function which creates a loop counter which starts at zero , and return the instance of this class itself . The iter function will call the __iter__ function in order to get an iterator . This __iter__function returns the class instance itself , as such on this class we have defined the __next__ method which returns the next data , in this case the data from 1 till 5 .

In order to get the data from the loop object , we can either use it with the next method.

>>> loopIterable = Loop()
>>> next(loopIterable)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 10, in __next__
AttributeError: 'Loop' object has no attribute 'loop'
# to use an class that has an iterator , 
#  with next method , we must first 
# create the iterator , or else this will 
# throw an error

>>> iterator_loop = iter(loopIterable)
# create an iterator from loopIterable
# we can now use next with either iterator_loop
# or with loopIterable

>>> next(iterator_loop)
1

>>> next(loopIterable)
2

>>> next(iterator_loop)
3

>>> next(loopIterable)
4

>>> next(loopIterable)
5

>>> next(loopIterable)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 11, in __next__
StopIteration
# once we have iterated through all the elements
# the next function will raise a StopIteration 
# if we want to reuse the next function we must 
# recreate an iterator

>>> iter(loopIterable)
<__main__.Loop object at 0x104e2b050>
>>> next(loopIterable)
1
>>> next(iterator_loop)
2

or we can use it with a for loop or with a list .. this will automatically call the iter function in order to get an iterator from the iterable .

>>> loopIterable = Loop()
>>> iterator_loop = iter(loopIterable)

>>> next(loopIterable)
1

>>> list(loopIterable)
[1, 2, 3, 4, 5]
# this will automatically call iter(loopIterable)
# as such we restart the looping

>>> next(loopIterable)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 11, in __next__
StopIteration
#  there are no more items in the iterable 
# as such next raises an error

for data in loopIterable : 
    print(data , end='')
    if data == 2:
        print()
        break
# output : 12 
# the for loop will call the iter function
# to create an iterator from the iterable 
# we print the first two elements in the 
# iterator and break 

>>> next(loopIterable)
3
# we can use next to continue iterating
# through the iterator

>>> list(loopIterable)
[1, 2, 3, 4, 5]
# if we call list with the iterable 
# it will call the iter function 
# to get an iterator , and we start 
# iterating from the start of the iterable

implement the __getitem__ methods

The iter function can also create an iterator from an object that has the __getitem__ method .

The __getitem__ methods simply returns an element from an object based on a numerical index , if the numerical index does not exist , it raises an IndexError .

The iter function will create an iterator from an instance of a class that has a __getitem__ method . it will pass the indexes starting 0 till an IndexError is raised , in this case it will raise a StopIteration error .

class Paper:
    def __getitem__(self, key):
        '''
            a key can be between 0 and 2
                or
            a key can be larger than 7
        '''
        if 0 <= key < 2 or key > 7:
            return key
        raise IndexError

>>> paper = Paper()

>>> list(paper)
[0, 1]
# the iter function will create an iterator
# for data 0 , 1 only

>>> paper[8]
8
# paper have data which exists at index 
# larger than 7  , but the iter function
# will stop at the first IndexError that 
# is raised

>>> next(paper)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'Paper' object is not an iterator
# we cannot use the paper instance
# with next  , 

>>> iter_paper = iter(paper)
# create an iterator from paper

>>> next(iter_paper)
0

>>> next(paper)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'Paper' object is not an iterator
# we cannot use the paper instance
# with next  , 


>>> next(iter_paper)
1

>>> next(iter_paper)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
# when there are no more elements 
# a StopIteration exception is raised

object is a Function

We can use the iter function with a function and a sentinel value in order to create an iterator . The sentinel value is used to stop the iteration .

import random
# import the random module

def choice():
    # return a random choice from the list "yes" , "no"  ,"perhaps"
    return random.choice(["yes", "no", "perhaps"])

choice_iter = iter(choice , "no")
# create an iterator , which will stop 
# when the value returned by the choice
# function is no 

>>> list(choice_iter)
['yes', 'perhaps', 'yes', 'perhaps', 'yes', 'yes', 'yes']
list(choice_iter) #[]

>>> list(choice_iter)
[]
# once the iterator is used , we cannot reuse it
# we must create a new one


>>> choice_iter = iter(choice , "no")
>>> list(choice_iter)
['yes']

object is a callable

a callable is an object which is an instance from a class that implements the __call__ method. it can be used as a regular function .

we can use the iter function with a callable , and a sentinel value , the sentinel value is used to stop the iteration .

import random 

class RandomWords:
    ''' return some random words
    '''

    def __init__(self):
        self.words = ["statement", "has", "no", "effect", "stop"]

    def __call__(self):
        return random.choice(self.words)

randomWords = RandomWords()

>>> randomWords = RandomWords()

>>> iter_random_words = iter(randomWords , "stop")

>>> list(iter_random_words)
['no']
# an iterator can only be used once

>>> list(iter_random_words)
[]

>>> iter_random_words = iter(randomWords , "stop")
>>> list(iter_random_words)
['no', 'no']

Complex iterables & Real world usage

iterables can be used to iterate through data , or to generate data without statically storing it . let us give an example . Let us say that we want to generate data for series , so we decided to create a class to do that , this class can generate data for two series , the harmonic series

and the sum series .

This class name is the Series class and it has two inner classes , the HARMONIC and the SUM class , which are used to return an iterator based on if we want to calculate the sum of harmonic or a sum series .

class Series:
    '''
        return an iterator , that return  the sum
        of a harmonic and a sum series
    '''
    SERIES = ['SUM', 'HARMONIC']
    # the series that we can calculate their sum

    _SUM = "SUM"
    _HARMONIC = "HARMONIC"

    # Name of the series to be set by the user

    def __init__(self, series='SUM'):
        '''default series to return , its iterator
            is the SUM Series'''
        self.series = series

    def __iter__(self):
        # return a SUM or HARMONIC instance
        return getattr(self, self.series)()

    @property
    # series property to set and get the series
    def series(self):
        return self.__serie

    @series.setter
    def series(self, series):
        if series not in Series.SERIES:
            # can only get the sum of series defined in SERIES
            raise ValueError("series must be in SERIES")
        self.__serie = series

    class SUM:
        '''
            Iterator to return the sum of sum series
        '''
        def __init__(self):
            self.n = -1

        def __iter__(self):
            # not necessary if we don't want SUM to be iterable
            return self

        def __next__(self):
            # infinite series no StopIteration
            self.n += 1
            return self.n * (self.n + 1) / 2

    class HARMONIC:
        '''
            Iterator to return the sum of
            HARMONIC series
        '''
        def __init__(self):
            self.currentSum = 0
            self.n = 0

        def __iter__(self):
            # not necessary if we don't want SUM to be iterable
            return self

        def __next__(self):
            # infinite series no StopIteration
            self.n += 1
            self.currentSum += 1 / self.n
            return self.currentSum



>>> serie = Series()
>>> sum_iter = iter(serie)
# default iterator is SUM series iterator
   
>>> serie.serie = serie._HARMONIC 
# set serie to harmonic
>>> harm_iter = iter(serie)
# create a HARMONIC series iterator
    
    
for _r in range(10):
     print(f'SUM({_r}) : {next(sum_iter)} ')
     print(f'HARMONIC({_r}) : {next(harm_iter)} ')

SUM(0) : 210.0 
HARMONIC(0) : 210.0 
SUM(1) : 231.0 
HARMONIC(1) : 231.0 
SUM(2) : 253.0 
HARMONIC(2) : 253.0 
SUM(3) : 276.0 
HARMONIC(3) : 276.0 
SUM(4) : 300.0 
HARMONIC(4) : 300.0 
SUM(5) : 325.0 
HARMONIC(5) : 325.0 
SUM(6) : 351.0 
HARMONIC(6) : 351.0 
SUM(7) : 378.0 
HARMONIC(7) : 378.0 
SUM(8) : 406.0 
HARMONIC(8) : 406.0 
SUM(9) : 435.0 
HARMONIC(9) : 435.0

Iterator vs generator object

an iterator is created by using the iter function , while a generator object is created by either a generator function or a generator expression . A generator object can act like an iterator , and can be used wherever an iterator can be used .