Mercurial > hg > Lemuriformes

annotate lemuriformes/chunk.py @ 12:82cd4e0b66cf

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
csv2sql
author Jeff Hammel <k0scist@gmail.com>
date Sun, 10 Dec 2017 15:18:00 -0800
parents 244c29f46554
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6
244c29f46554 add module for chunking data
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
1 def chunk(data, chunksize=100000):
244c29f46554 add module for chunking data
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
2 """
244c29f46554 add module for chunking data
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
3 chunk `data` into smaller sizes of `chunksize`
244c29f46554 add module for chunking data
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
4 """
244c29f46554 add module for chunking data
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
5
244c29f46554 add module for chunking data
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
6 offset = 0
244c29f46554 add module for chunking data
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
7 chunk = data[0:chunksize]
244c29f46554 add module for chunking data
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
8 while chunk:
244c29f46554 add module for chunking data
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
9 yield chunk
244c29f46554 add module for chunking data
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
10 offset += chunksize
244c29f46554 add module for chunking data
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
11 chunk = data[offset:offset+chunksize]
244c29f46554 add module for chunking data
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
12