datapad.Sequence

class datapad.Sequence(_iterable=None)

The core object in datapad used to wrap sequence-like data types in a fluent-style API.

__init__(_iterable=None)

Instantiates a new Sequence object.

Parameters:_iterable (List, Set, Tuple, Iterator) – Any object that conforms to the Iterable API.

Methods

__init__([_iterable]) Instantiates a new Sequence object.
all() Returns a standard python iterator that you can use to lazily iterate over your sequence of data
batch(size) Lazily combines elements in sequence into a list of length size.
collect() Eagerly returns all elements in sequence
concat(seq) Concatenates another sequence to the end of this sequence
count([distinct]) Eagerly count number of elements in sequence
distinct() Eagerly returns a new sequence with unique values
drop(count) Lazily skip or drop over count elements.
drop_if(fn) Lazily apply fn function to every element of iterable and drop sequence elements where the function fn evaluates to True.
filter(fn) This is an alias for the Sequence.keep_if function
first() Eagerly returns first element in sequence
flatmap(fn) Lazily apply fn function to every element of iterable and chain the output into a single flattend sequence.
groupby([key, getter, eager_group]) Groups sequence using key function,
join(other[, key, other_key]) Joins two sequences based on common field matches between the sequence and other.
keep_if(fn) Lazily apply fn function to every element of iterable and keep only sequence elements where the function fn evaluates to True.
map(fn) Lazily apply fn function to every element of iterable
peek([count]) Returns list of count elements without advancing sequence iterator.
pmap(fn[, workers, ordered]) Lazily apply fn function to every element of iterable, in parallel using multiprocess.dummy.Pool .
reduce(fn[, initial]) Eagerly apply a function of two arguments cumulatively to the items of a sequence, from left to right, so as to reduce the sequence to a single value.
shuffle() Eagerly shuffles your sequence and returns a newly created sequence containing the shuffled items.
sort([key]) Eagerly sorts your sequence and returns a newly created sequence containing the sorted items.
take(count) Lazily returns a sequence of the first count elements.
window(size[, stride]) Lazily slides and yields a window of length size over sequence.
zip_with_index() Add an to each item in sequence
all()

Returns a standard python iterator that you can use to lazily iterate over your sequence of data

>>> seq = Sequence(range(10))
>>> seq = seq.map(lambda v: v*2)
>>> i = 0
>>> for item in seq.all():
...     i += item
>>> i
90
batch(size)

Lazily combines elements in sequence into a list of length size. This function will drop any remainder if the sequence ends before a batch with size has been created.

Parameters:size (int) – The batch size.

Examples

>>> seq = Sequence(range(10))
>>> seq.batch(3).collect()
[[0, 1, 2], [3, 4, 5], [6, 7, 8]]
collect()

Eagerly returns all elements in sequence

>>> seq = Sequence(range(10))
>>> seq.collect()
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
concat(seq)

Concatenates another sequence to the end of this sequence

Examples

Concat two sequences together:

>>> s1 = Sequence(['a', 'b', 'c'])
>>> s2 = Sequence(range(3))
>>> s3 = s2.concat(s1)
>>> s3.collect()
[0, 1, 2, 'a', 'b', 'c']

Concat sequence with itself:

>>> seq = Sequence(range(5))
>>> seq = seq.concat(seq)
>>> seq.collect()
[0, 1, 2, 3, 4, 0, 1, 2, 3, 4]
count(distinct=False)

Eagerly count number of elements in sequence

Parameters:distinct – bool If True, counts occurence of each distinct value in sequence.
Returns:Either an integer count or a new sequence of tuples where the first value is the unique element and the second value is the number of times that element appeared in the sequence.
>>> seq = Sequence(range(5))
>>> seq.count()
5
>>> seq = Sequence(['a', 'a', 'b', 'b', 'c', 'c'])
>>> seq.count(distinct=True).collect()
[('a', 2), ('b', 2), ('c', 2)]
distinct()

Eagerly returns a new sequence with unique values

>>> seq = Sequence(['a', 'a', 'b', 'b', 'c', 'c'])
>>> seq.distinct().collect()
['a', 'b', 'c']
drop(count)

Lazily skip or drop over count elements.

>>> seq = Sequence(range(5))
>>> seq.collect()
[0, 1, 2, 3, 4]
>>> seq = Sequence(range(5))
>>> seq = seq.drop(2)
>>> seq.collect()
[2, 3, 4]
drop_if(fn)

Lazily apply fn function to every element of iterable and drop sequence elements where the function fn evaluates to True.

Parameters:fn – function Function with signature fn(element) -> bool to apply to every element of sequence. Drop all elements in the sequence where the fn function evaluates to True.
>>> seq = Sequence(range(5))
>>> seq = seq.drop_if(lambda v: v > 1)
>>> seq.collect()
[0, 1]
filter(fn)

This is an alias for the Sequence.keep_if function

>>> seq = Sequence(range(5))
>>> seq = seq.filter(lambda v: v > 1)
>>> seq.collect()
[2, 3, 4]
first()

Eagerly returns first element in sequence

Examples

Get first value in sequence:

>>> seq = Sequence(range(5))
>>> seq.first()
0
>>> seq.first()
1

Calling first on empty sequence returns None:

>>> seq = Sequence([])
>>> seq.first()
flatmap(fn)

Lazily apply fn function to every element of iterable and chain the output into a single flattend sequence.

Parameters:fn (function) – Function with signature fn(element) -> iterable(element) to apply to every element of sequence.

Examples

>>> seq = Sequence(range(5))
>>> seq = seq.flatmap(lambda v: [v,v])
>>> seq.collect()
[0, 0, 1, 1, 2, 2, 3, 3, 4, 4]
groupby(key=None, getter=None, eager_group=True)

Groups sequence using key function,

Note: you must ensure elements are sorted by groups before calling this function.

Parameters:
  • key – function Function used to determine what to use as a key for grouping
  • getter – function Function to be applied to each element of a group
  • eager_group – bool, default=True If true, eagerly convert a group from a lazy Sequence to a fully-realized list.

Examples

Simple usage:

>>> from pprint import pprint
>>> seq = Sequence(['a', 'b', 'c', 'd', 'a', 'b', 'a', 'd'])
>>> res = seq.sort().groupby(key=lambda x: x).collect()
>>> res == [
...    ('a', ['a', 'a', 'a']),
...    ('b', ['b', 'b']),
...    ('c', ['c']),
...    ('d', ['d', 'd']),
... ]
True

Grouping with getter function:

>>> things = [("animal", "lion"),
...           ("plant", "maple tree"),
...           ("animal", "walrus"),
...           ("plant", "grass")]
>>> seq = Sequence(things)
>>> res = seq.sort().groupby(key=lambda x: x[0], getter=lambda x: x[1]).collect()
>>> res == [
...    ('animal', ['lion', 'walrus']),
...    ('plant', ['grass', 'maple tree'])
... ]
True
join(other, key=None, other_key=None)

Joins two sequences based on common field matches between the sequence and other. This is known as an “inner” join in SQL terminology.

Parameters:
  • other (Sequence) – A Sequence to join with the calling sequence.
  • key (function) – A function to retrieve the field to be used for matching between the two sequence. If key is None, then key will default to lambda x: x.
  • other_key (function) – A function to retrieve the field in other to be used for matching between the two sequence. If other_key is None, use key.
Returns:

A sequence of 2-tuples (a, b) where a is an element in self that matched element b in other (based on the given field keys).

Examples

>>> a = Sequence([
...     {'id': 1, 'name': 'John'},
...     {'id': 2, 'name': 'Nayeon'},
...     {'id': 3, 'name': 'Reza'}
... ])
>>> b = Sequence([
...     {'id': 1, 'age': 2},
...     {'id': 2, 'age': 3}
... ])
>>> res = a.join(b, key=lambda x: x['id']).collect()
>>> res == [
...     ({'id': 1, 'name': 'John'}, {'id': 1, 'age': 2}),
...     ({'id': 2, 'name': 'Nayeon'}, {'id': 2, 'age': 3})
... ]
True
keep_if(fn)

Lazily apply fn function to every element of iterable and keep only sequence elements where the function fn evaluates to True.

Parameters:fn – function Function with signature fn(element) -> bool to apply to every element of sequence. Keep all elements in the sequence where the fn function evaluates to True.
>>> seq = Sequence(range(5))
>>> seq = seq.keep_if(lambda v: v > 1)
>>> seq.collect()
[2, 3, 4]
map(fn)

Lazily apply fn function to every element of iterable

Parameters:fn (function) – Function with signature fn(element) to apply to every element of sequence.
>>> seq = Sequence(range(10))
>>> seq = seq.map(lambda v: v*2)
>>> seq.collect()
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
peek(count=None)

Returns list of count elements without advancing sequence iterator. If count is None, return only the first element.

WARNING: this function will load up to count elements of your sequence into memory.

Examples

Peek at first element (notice iterator does not advance):

>>> seq = Sequence(range(10))
>>> seq.peek()
0
>>> seq.peek()
0

Peek at first 3 elements:

>>> seq = Sequence(range(10))
>>> seq.peek(3)
[0, 1, 2]
>>> seq.peek(3)
[0, 1, 2]
pmap(fn, workers=3, ordered=True)

Lazily apply fn function to every element of iterable, in parallel using multiprocess.dummy.Pool . The returned sequence may appear in a different order than the input sequence if you set ordered to False

THIS FUNCTION IS EXPERIMENTAL

Parameters:
  • fn (function) – Function with signature fn(element) -> element to apply to every element of sequence.
  • workers (int) – Number of parallel workers to use (default: 3). These workers are implemented as python threads.
  • ordered (bool) – Whether to yield results in the same order in which items arrive. You may get better performance by setting this to false (default: True).
>>> seq = Sequence(range(10))
>>> seq = seq.pmap(lambda v: v*2)
>>> seq.collect()
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
>>> seq = Sequence(range(10))
>>> seq = seq.pmap(lambda v: v*2, workers=1, ordered=False)
>>> seq.collect()
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
reduce(fn, initial=None)

Eagerly apply a function of two arguments cumulatively to the items of a sequence, from left to right, so as to reduce the sequence to a single value. For example, reduce(lambda x, y: x+y, [1, 2, 3, 4, 5]) calculates ((((1+2)+3)+4)+5). If initial is present, it is placed before the items of the sequence in the calculation, and serves as a default when the sequence is empty.

Parameters:
  • fn (function) – Function with signature fn(acc, current_item) -> acc_next
  • initial (Any) – An initial value that acc will be set to. If not provided, this function will set the first element of the sequence as the initial value.

Examples

Reduce with accumulator initialized to first element:

>>> seq = Sequence(range(3))
>>> seq.reduce(lambda acc, item: acc + item)
3

Reduce with accumulator set to a custom initial value:

>>> seq = Sequence(range(3))
>>> seq.reduce(lambda acc, item: acc + item, initial=10)
13
shuffle()

Eagerly shuffles your sequence and returns a newly created sequence containing the shuffled items. WARNING: this function loads the entirety of your sequence into memory.

>>> import random
>>> random.seed(0)
>>> seq = Sequence(range(5))
>>> seq.shuffle().collect()
[2, 1, 0, 4, 3]
sort(key=None)

Eagerly sorts your sequence and returns a newly created sequence containing the sorted items. WARNING: this function loads the entirety of your sequence into memory.

>>> seq = Sequence([2, 1, 0, 4, 3])
>>> seq.sort().collect()
[0, 1, 2, 3, 4]
take(count)

Lazily returns a sequence of the first count elements.

>>> seq = Sequence(range(5))
>>> seq.take(2).collect()
[0, 1]
window(size, stride=1)

Lazily slides and yields a window of length size over sequence. This function will drop any remainder if the sequence ends before a window with size has been filled.

Parameters:
  • size (int) – The window size.
  • stride (int) – How many elements to skip for each advancement in window position.

Examples

>>> seq = Sequence(range(10))
>>> seq.window(2).collect()
[[0, 1], [1, 2], [2, 3], [3, 4], [4, 5], [5, 6], [6, 7], [7, 8], [8, 9]]
>>> seq = Sequence(range(10))
>>> seq.window(3, stride=2).collect()
[[0, 1, 2], [2, 3, 4], [4, 5, 6], [6, 7, 8]]
>>> seq = Sequence(range(10))
>>> seq.window(2, stride=4).collect()
[[0, 1], [4, 5], [8, 9]]
>>> seq = Sequence(range(10))
>>> seq.window(1, stride=1).collect()
[[0], [1], [2], [3], [4], [5], [6], [7], [8], [9]]
zip_with_index()

Add an to each item in sequence

>>> seq = Sequence(['a', 'b', 'c'])
>>> seq.zip_with_index().collect()
[(0, 'a'), (1, 'b'), (2, 'c')]