Mongodb Optimize Multiple Find_one + Insert Inside Loop

I'm using MongoDB 4.0.1 and Pymongo with pyhton 3.5. I have to loop over 12000 items every 30 - 60 seconds and add new data into MongoDB. For this example we will talk about User,

Solution 1:

Don´t do 12000 find_one queries, do 1 query to bring all that exist with $in operator. Code would be something like:

pet_codes = []
pet_names = []
while dictionary != False:
    pet_codes.append(dictionary['pet_code'])
    pet_names.append(dictionary['pet_name'])

pets = dict()
for pet in pet.find({"code": {$in: pet_codes}}):
    pets[pet['code']] = pet

new_pets = []
for code, name inzip(pet_codes, pet_names):
    if code notin pets:
        new_pets.add({'pet_code': code, 'name': name})

pet.insert_many(new_pets)

As you already have an index on pet_code making it unique, we can do better: just try to insert them all, because if we try to insert an existing one that record will get an error, but the rest will succeed by using the ordered=False from the docs:

new_pets = []
while dictionary != False:
    new_pets.add({
        "code" : dictionary['pet_code'],
        "name" : dictionary['name']
    })
pet.insert_many(new_pets, ordered=False)

In the case where you do not have a unique restriction set, another method is batching the operations