Pickling and Scaling
https://pythonprogramming.net/pickling-scaling-machine-learning-tutorial/
I’ve covered Pickling before and gone through the steps – so not a lot of new stuff to cover.
Key elements being:-
Writing a Pickle
clf.fit(X_train, y_train) # fit the data to the training data with open('LinearRegression.pickle','wb') as f: pickle.dump(clf, f)
Reading a Pickle
pickle_in = open ('LinearRegression.pickle', 'rb') clf = pickle.load(pickle_in)
The other key concept on this tutorial was scaling on rented host systems. With the work flow being:-
- Rent a small part of a server to write the code (minimal cost)
- Once coding is complete – scale up the server to do the hard data crunching
- Save the output as pickles
- Scale the server back down to minimum needed
This means that you don’t need to pay for the large server all the time.
Nice idea.