Generators

https://docs.python.org/3/reference/expressions.html#yieldexpr

https://rmariano.eu/posts/exploring-generators-and-coroutines/
Exploring Generators and Coroutines
PEP-255 (Simple Generators) [yield is a statement; lazy evaluation]
PEP-342 (Coroutines via Enhanced Generators) [yield is an expression; send(), throw(), close()]
PEP-380 (Syntax for delegating to a Sub-Generator) [return in generators; yield from]
PEP-525 (Asynchronous Generators)

https://snarky.ca/how-the-heck-does-async-await-work-in-python-3-5/
How the heck does async/await work in Python 3.5? [long essey]

INTRODUCTION

Use cases of generators:
(a) Working with data streams or large files, like CSV files.
(b) Generating an infinite sequence.

GENERATOR EXPRESSIONS

This is an expression that returns an iterator.

# sum of squares 0, 1, 4, ... 81

alist = [i*i for i in range(10)]   # list comprehension

gen = (i*i for i in range(10))   # generator expression

result = sum(i*i for i in range(10))

GENERATOR FUNCTIONS

This is a function which returns a generator iterator. It looks like a normal function except that it contains yield expressions (instead of return) for producing a series of values.

def iter_squares(n):
    for i in range(n):
        yield i*i

result = sum(iter_squares(10))

def my_range(stop):
    """My version of range(stop)."""
    value = 0
    while value < stop:
        yield value           # yield instead of return
        value = value + 1

# Example 1 - using for loop (hidden iteration protocole).
for i in my_range(10):
    print(i)

# Example 2 - explicit iteration protocole.
gen = my_range(3)
print(next(gen))   # 0
print(next(gen))   # 1
print(next(gen))   # 2
print(next(gen))   # StopIteration, gen is exhausted
print(next(gen))   # StopIteration

INFINITE ITERATORS

def iter_squares2():   # an infinite iterator
    i = 0
    while True:
        yield i*i
        i += 1

def fibonacci():   # an infinite iterator
    """The Fibonacci sequence."""
    a, b = 0, 1
    yield a
    yield b
    while True:
        a, b = b, a + b
        yield b

# Example 1 - using for loop (hidden iteration protocole).
for i in fibonacci():
    print(i)
    if i > 100:   # we have to break the loop manually
        break
# 0 1 1 2 3 5 8 13 21 34 55 89 144

# Example 2 - the generator is alive!
fib = fibonacci()   # fib is a generator object
for i in fib:
    print(i)
    if i > 100:
        break
print(next(fib))   # 233
print(next(fib))   # 377

SPEED VS MEMORY

List comprehensions can be faster to evaluate than the equivalent generator expression.

import sys
import timeit

sqr_list = [i*i for i in range(1000000)]
print(sys.getsizeof(sqr_list))   # 8697464 bytes

sqr_gen = (i*i for i in range(1000000))
print(sys.getsizeof(sqr_gen))   # 120 bytes

t1 = timeit.Timer(lambda: sum(sqr_list))
print(t1.timeit(1))   # 0.00961924200237263

t2 = timeit.Timer(lambda: sum(sqr_gen))
print(t2.timeit(1))   # 0.05263053500675596 (5 times slower)

EXAMPLES

https://realpython.com/introduction-to-python-generators/

# Problem: couting rows in a text file.

txt_gen = txt_reader("some_file.txt")
line_count = 0
for line in txt_gen:
    line_count += 1

def txt_reader(file_name):   # possible MemoryError
    file = open(file_name, "r")
    result = file.read()   # loads everything into memory at once
    result = result.rstrip()   # remove last "\n" if present
    result = result.split("\n")   # divide to lines 
    return result   # list of strings without "\n"

def txt_reader(file_name):   # a generator function
    for line in open(file_name, "r"):
        yield line   # lines with "\n" (the last line may be an exception)