Python iterators, loading data in chunks with Pandas
Python iterators
loading data in chunks with pandas
Iterators, load file in chunks¶
Iterators vs Iterables¶
an iterable is an object that can return an iterator¶
- Examples: lists, strings, dictionaries, file connections
- An object with an associated iter() method
- Applying iter() to an iterable creates an iterator
an iterator is an object that keeps state and produces the next value when you call next() on it.¶
- Produces next value with next()
In [30]:
a=[1,2,3,4]
b=iter([1,2,3,4])
c=iter([5,6,7,8])
In [31]:
print a
print b
print next(b),next(b),next(b),next(b)
print list(c)
Iterating over iterables¶
- Python 2 does NOT work
range() doesn't actually create the list; instead, it creates a range object with an iterator that produces the values until it reaches the limit
If range() created the actual list, calling it with a value of 10^100 may not work, especially since a number as big as that may go over a regular computer's memory. The value 10^100 is actually what's called a Googol which is a 1 followed by a hundred 0s. That's a huge number!
- calling range() with 10^100 won't actually pre-create the list.
In [15]:
# Create an iterator for range(10 ** 100): googol
googol = iter(range(10 ** 100))
Iterating over dictionaries¶
In [32]:
a={1:9, 'what':'why?'}
In [34]:
for key,value in a.items(): print key,value
Iterating over file connections¶
In [40]:
f = open('university_towns.txt')
type(f)
Out[40]:
In [47]:
iter(f)
Out[47]:
In [48]:
iter(f)==f
Out[48]:
In [49]:
next(f)
Out[49]:
In [50]:
next(iter(f))
Out[50]:
In [65]:
# Create a list of strings: mutants
mutants = ['charles xavier', 'bobby drake', 'kurt wagner', 'max eisenhardt', 'kitty pride']
# Create a list of tuples: mutant_list
mutant_list = list(enumerate(mutants))
# Print the list of tuples
print(mutant_list)
print
# Unpack and print the tuple pairs
for index1, value1 in enumerate(mutants):
print(index1, value1)
print "\nChange the start index\n"
for index2, value2 in enumerate(mutants, start=3):
print(index2, value2)
Using zip¶
zip(), which takes any number of iterables and returns a zip object that is an iterator of tuples.¶
- If you wanted to print the values of a zip object, you can convert it into a list and then print it.
- Printing just a zip object will not return the values unless you unpack it first.
In Python 2 , zip() returns a list¶
Docstring: zip(seq1 [, seq2 [...]]) -> [(seq1[0], seq2[0] ...), (...)]
Return a list of tuples, where each tuple contains the i-th element from each of the argument sequences. The returned list is truncated in length to the length of the shortest argument sequence.
In [73]:
aliases = ['prof x', 'iceman', 'nightcrawler', 'magneto', 'shadowcat']
powers = ['telepathy','thermokinesis','teleportation','magnetokinesis','intangibility']
# Create a list of tuples: mutant_data
mutant_data = list(zip(mutants, aliases, powers))
# Print the list of tuples
print(mutant_data)
print
# Create a zip object using the three lists: mutant_zip
mutant_zip = zip(mutants, aliases, powers)
# Print the zip object
print(type(mutant_zip))
# Unpack the zip object and print the tuple values
for value1, value2, value3 in mutant_zip:
print(value1, value2, value3)
Loading data in chunks¶
- There can be too much data to hold in memory
- Solution: load data in chunks!
- Pandas function: read_csv()
- Specify the chunk: chunksize
In [78]:
import pandas as pd
from time import time
In [110]:
start = time()
df = pd.read_csv('kamcord_data.csv')
print 'used {:.2f} s'.format(time()-start)
print df.shape
df.head(1)
Out[110]:
explore¶
In [ ]:
a=pd.read_csv('kamcord_data.csv',chunksize=4)
b=pd.read_csv('kamcord_data.csv',iterator=True)
In [139]:
a.next()
Out[139]:
In [140]:
x=a.next()
y=a.next()
In [143]:
y.append(x, ignore_index=True)
Out[143]:
In [144]:
pd.concat([x,y], ignore_index=True)
Out[144]:
1st way of loading data in chunks¶
In [146]:
start = time()
c=0
for chuck in pd.read_csv('kamcord_data.csv',chunksize=50000):
if c==0:
df=chuck
c+=1
else:
df=df.append(chuck, ignore_index=True)
c+=1
print c
print 'used {:.2f} s'.format(time()-start)
print df.shape
df.head(1)
Out[146]:
2ed way of loading data in chunks¶
In [149]:
start = time()
want=[]
for chuck in pd.read_csv('kamcord_data.csv',chunksize=50000):
want.append(chuck)
print len(want)
df=pd.concat(want, ignore_index=True)
print 'used {:.2f} s'.format(time()-start)
print df.shape
df.head(1)
Out[149]:
3rd way of loading data in chunks¶
In [172]:
start = time()
want=[]
f = pd.read_csv('kamcord_data.csv',iterator = True)
go = True
while go:
try:
want.append(f.get_chunk(50000))
except Exception as e:
print type(e)
go = False
print len(want)
df=pd.concat(want, ignore_index=True)
print 'used {:.2f} s'.format(time()-start)
print df.shape
df.head(1)
Out[172]:
Processing large amounts of Twitter data by chunks¶
In [183]:
import pandas as pd
# Import package
import json
# Initialize empty list to store tweets: tweets_data
tweets_data = []
# Open connection to file
h=open('tweets.txt','r')
# Read in tweets and store in list: tweets_data
for i in h:
try:
print 'O',
tmp=json.loads(i)
tweets_data.append(tmp)
except:
print 'X',
h.close()
t_df = pd.DataFrame(tweets_data)
print
print t_df.shape
t_df.to_csv('tweets.csv',index=False, encoding= 'utf-8')
t_df.head(1)
Out[183]:
Processing large amounts of data by chunks¶
In [184]:
# Initialize an empty dictionary: counts_dict
counts_dict = {}
# Iterate over the file chunk by chunk
for chunk in pd.read_csv('tweets.csv', chunksize=10):
# Iterate over the column in dataframe
for entry in chunk['lang']:
if entry in counts_dict.keys():
counts_dict[entry] += 1
else:
counts_dict[entry] = 1
# Print the populated dictionary
print(counts_dict)
Extracting information for large amounts of Twitter data¶
- reusable
- def func
In [185]:
# Define count_entries()
def count_entries(csv_file, c_size, colname):
"""Return a dictionary with counts of
occurrences as value for each key."""
# Initialize an empty dictionary: counts_dict
counts_dict = {}
# Iterate over the file chunk by chunk
for chunk in pd.read_csv(csv_file, chunksize=c_size):
# Iterate over the column in dataframe
for entry in chunk[colname]:
if entry in counts_dict.keys():
counts_dict[entry] += 1
else:
counts_dict[entry] = 1
# Return counts_dict
return counts_dict
# Call count_entries(): result_counts
result_counts = count_entries('tweets.csv', 10, 'lang')
# Print result_counts
print(result_counts)
In [ ]: