Dealing with db.Timeout on Google App Engine
Rise to the top 3% as a developer or hire one of them at Toptal: https://topt.al/25cXVn
--------------------------------------------------
Music by Eric Matyas
https://www.soundimage.org
Track title: Light Drops
--
Chapters
00:00 Dealing With Db.Timeout On Google App Engine
00:41 Accepted Answer Score 6
01:15 Answer 2 Score 1
01:37 Answer 3 Score 7
02:16 Answer 4 Score 2
02:33 Thank you
--
Full question
https://stackoverflow.com/questions/1456...
--
Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...
--
Tags
#python #googleappengine
#avk47
ANSWER 1
Score 7
Here's a decorator to retry on db.Timeout, adapted from one from Kay framework:
import logging, time
from google.appengine.ext import db
def retry_on_timeout(retries=3, interval=1.0, exponent=2.0):
"""A decorator to retry a given function performing db operations."""
def _decorator(func):
def _wrapper(*args, **kwargs):
count = 0
while True:
try:
return func(*args, **kwargs)
except db.Timeout, e:
logging.debug(e)
if count >= retries:
raise e
else:
sleep_time = (exponent ** count) * interval
logging.warning("Retrying function %r in %d secs" %
(func, sleep_time))
time.sleep(sleep_time)
count += 1
return _wrapper
return _decorator
To use it, simply decorate any function that performs db operations and you'd like to do retries:
@retry_on_timeout()
def do_the_stuff(models):
return db.put(models)
ACCEPTED ANSWER
Score 6
Queries will occasionally fail. You can either show an error message to the user, or retry, as you're doing above. If you retry, however, you should use thread.sleep to add increasing amounts of delay (starting at, say, 50ms) on each retry - retries are more likely to succeed if they're not retried as fast as possible.
40 queries per request is a lot, though. You should consider refactoring your code - it must be possible to eliminate most of those!
ANSWER 3
Score 2
Take a look at this this python Autoretry Datastore Timeouts recipe. Similar to moraes answer, but you only need to call it once at initialization time, rather than decorate functions that perform datastore operations.
http://appengine-cookbook.appspot.com/recipe/autoretry-datastore-timeouts
ANSWER 4
Score 1
See the new ROTModel in GAE-Utilities. The second discussion below shows how it does retries. It's a subclass of db.Model, so your classes can inherit from ROTModel instead, and take advantage of it's retries.