Mercurial > hg > Lemuriformes

diff lemuriformes/chunk.py @ 6:244c29f46554

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
add module for chunking data
author Jeff Hammel <k0scist@gmail.com>
date Sun, 10 Dec 2017 11:52:37 -0800
parents
children
line wrap: on
line diff
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/lemuriformes/chunk.py	Sun Dec 10 11:52:37 2017 -0800
@@ -0,0 +1,12 @@
+def chunk(data, chunksize=100000):
+    """
+    chunk `data` into smaller sizes of `chunksize`
+    """
+
+    offset = 0
+    chunk = data[0:chunksize]
+    while chunk:
+        yield chunk
+        offset += chunksize
+        chunk = data[offset:offset+chunksize]
+