Generic selectors
Exact matches only
Search in title
Search in content
wb_sunny

Best Ways to Save Data in Python

Python Best Ways save Data in Python

Hello readers! In this tutorial, we’ll be dealing with how we can effectively save data in Python.

How to Save Data in Python?

When we’re working on Python applications, we’ll be dealing with Python objects directly, since everything is an object in Python. Let’s look at some ways by which we can store them easily!

1. Using Pickle to store Python Objects

If we want to keep things simple, we can use the pickle module, which is a part of the standard library to save data in Python.

We can “pickle” Python objects to a pickle file, which we can use to save/load data.

So if you have a custom object which you might need to store / retrieve, you can use this format:

import pickle

class MyClass():
    def __init__(self, param):
        self.param = param

def save_object(obj):
    try:
        with open("data.pickle", "wb") as f:
            pickle.dump(obj, f, protocol=pickle.HIGHEST_PROTOCOL)
    except Exception as ex:
        print("Error during pickling object (Possibly unsupported):", ex)

obj = MyClass(10)
save_object(obj)

If you run this script, you’ll notice a file called data.pickle, which contains the saved data.

To load the same object back again, we could use pickle.load() using similar logic.

import pickle

class MyClass():
    def __init__(self, param):
        self.param = param

def load_object(filename):
    try:
        with open(filename, "rb") as f:
            return pickle.load(f)
    except Exception as ex:
        print("Error during unpickling object (Possibly unsupported):", ex)

obj = load_object("data.pickle")

print(obj.param)
print(isinstance(obj, MyClass))

Output

10
True

We’ve just retrieved our old data successfully!

2. Using Sqlite3 to save data in Python persistently

If you want to use a persistent database to save data in Python, you can use the sqlite3 library which provides you APIs for using Sqlite databases.

Again, this is a part of the standard library, so there’s no need to pip install anything!

However, since this is a Relational Database, you can’t directly dump Python objects like in pickle.

You’d have to serialize and de-serialize them to their appropriate Database types.

To look at some examples, you can refer to this article on using sqlite in Python.

3. Using SqliteDict as a persistent cache

If you find using sqlite3 too tedious, there’s a much better solution! You can use sqlitedict for storing persistent data, and this internally uses an sqlite3 database to handle the storage.

You have to install this package using pip:

pip install sqlitedict

The only thing you need to keep in mind is that you need to use key:value mappings to store / retrieve data, just like a dictionary!

Here’s a very simple example using the MyClass instance.

from sqlitedict import SqliteDict

class MyClass():
    def __init__(self, param):
        self.param = param

def save(key, value, cache_file="cache.sqlite3"):
    try:
        with SqliteDict(cache_file) as mydict:
            mydict[key] = value # Using dict[key] to store
            mydict.commit() # Need to commit() to actually flush the data
    except Exception as ex:
        print("Error during storing data (Possibly unsupported):", ex)

def load(key, cache_file="cache.sqlite3"):
    try:
        with SqliteDict(cache_file) as mydict:
            value = mydict[key] # No need to use commit(), since we are only loading data!
        return value
    except Exception as ex:
        print("Error during loading data:", ex)

obj1 = MyClass(10)
save("MyClass_key", obj1)

obj2 = load("MyClass_key")

print(obj1.param, obj2.param)
print(isinstance(obj1, MyClass), isinstance(obj2, MyClass))

Output

10 10
True True

Indeed, we’ve just loaded our Python object successfully! If you notice, sqlitedict will create a database cache.sqlite3 automatically, if it doesn’t exist, and then use it to store / load data.


Conclusion

In this article, we looked at how we can use Python to store data in different ways.