Forums

google drive

first of all, let me say that i love the site. thanks so much for making this available.

now, i've had a little web app up and running for a little while. i'd like to use my daily scheduled script execution to save backups of my DB to google drive, but i wasn't able to get the authentication working. from what i can gather, *.google.com (and googleapis.com and googlecode.com) should be whitelisted for HTTP requests, which is what the google API uses to handle authentication requests. but when i tried to run the quickstart sample from

https://developers.google.com/drive/quickstart-python

i got an error. (yes, i did hard-code my own client ID and client "secret" string.) here's the trace:

Traceback (most recent call last):
  File "./quickstart.py", line 29, in <module>
    credentials = flow.step2_exchange(code)
  File "/usr/local/lib/python2.7/site-packages/oauth2client/util.py", line 120, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/oauth2client/client.py", line 1131, in step2_exchange
    headers=headers)
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1597, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, 
cachekey)
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1345, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1281, in _conn_request
    conn.connect()
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1010, in connect
    sock.connect((self.host, self.port))
  File "/usr/local/lib/python2.7/site-packages/httplib2/socks.py", line 424, in connect
    self.__negotiatehttp(destpair[0], destpair[1])
  File "/usr/local/lib/python2.7/site-packages/httplib2/socks.py", line 390, in __negotiatehttp
    raise HTTPError((statuscode, statusline[2]))
httplib2.socks.HTTPError: (403, 'Forbidden')

any idea what i'm doing wrong? much thanks.

EDIT by admin to format stacktrace

Well, you're getting a "403 Forbidden" response from somewhere, but it's not clear to me whether that's the server you're contacting or the local proxy (presumably due to an intermediate site used during the open auth process not being whitelisted even if the original requested site is).

I guess the options are either to try and add some tracing to see which sites are being requested, or perhaps if the PA staff are feeling generous they could open your account for free internet access for 24 hours or something so you can try again and check if you get the same error without the proxy.

Tracing it would be a bit tricky because it's all done in the open auth library which is system installed - you could install your own in a virtualenv and modify it to add logging, or you could theoretically monkey-patch in some logging wrappers into your script, but either way it's a bit of a faff.

@vjmatchmaker, it seems likely that it's because of trying to run oauth through our proxy. I have given your account access to the internet for a week so we can confirm that. Let us know if that helps.

@glenn, thanks for the quick reply, and the access. i ran the script again and got a different error: 500 this time instead of a 403.

Traceback (most recent call last):
  File "./quickstart.py", line 45, in <module>
file = drive_service.files().insert(body=body, media_body=media_body).execute()
  File "/usr/local/lib/python2.7/site-packages/oauth2client/util.py", line 120, in positional_wrapper
return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/apiclient/http.py", line 656, in execute
_, body = self.next_chunk(http=http)
  File "/usr/local/lib/python2.7/site-packages/oauth2client/util.py", line 120, in positional_wrapper
return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/apiclient/http.py", line 789, in next_chunk
return self._process_response(resp, content)
  File "/usr/local/lib/python2.7/site-packages/apiclient/http.py", line 816, in _process_response
raise HttpError(resp, content, uri=self.uri)
apiclient.errors.HttpError: <HttpError 500 when requesting https://www.googleapis.com/upload/drive/v2/files?uploadType=resumable&alt=json returned "Internal Error">

unfortunately, 500 isn't very informative. am i at a dead end?

5xx errors should only be returned when there's an internal error on the remote server. Sometimes sites will return it for cases where there is a fault with the request, but I doubt that Google would make that sort of mistake.

Is it possible they're experiencing issues at the moment? Have you seen this code run successfully on, say, your home machine? Also, have you tried running your script from a fresh login shell just to make sure the environment isn't causing your script to still use the proxy or something similar?

It looks like Google Drive are having some problems at the moment

Hm, yes, I can access mine, but it's awfully slow. Haven't tried saving anything yet, just in case. Looks like Google are now aware of the issue.

UPDATE: They're claiming that it should be fixed for everyone within the next hour or so.

cool, my script is now working, too. thanks!

now i just need to see if i can save the authentication credentials locally so that i can push files onto my google drive without going through the UI. otherwise it won't be very useful as a daily scheduled script.

Excellent, thanks for confirming! We'd love to hear how your GDrive integration turns out in the end, BTW.

as far as i can tell, everything is working fine now: the script i've written successfully pushes my db onto google drive by using a serialized credential that i've saved as a JSON. no user interaction is required, so it can run as a scheduled script.

now, i guess the only thing i don't know is whether it will continue to work after the temporary internet access expires, or whether authentication will start failing again. i'll check back in a few days.

I can remove internet access today, if you like, so you can test it. Just let me know if that's what you want. Otherwise, we'll turn it off some time on the 25th, as planned.

sure, go ahead and shut it off.

Done, you'll need to reload any wep apps or consoles for it to take effect though.

sigh well, the 403 is back. pity. looks like i'll have to dig through the oauth code and figure out where it's making an HTTP request to some other domain.

If you could give us the system time (you can get that from the "Schedule" tab) at the point when you get that error, we could look through the console logs and see if anything obvious was making it generate a 403 -- no promises that it would show up, but there's at least a chance :-)

i've been digging through the code and it does seem to be a proxy issue. anyway, here's the timestamp from the last unsuccessful run.

2013-03-21 22:13:05 -- Completed task, took 1.00 seconds, return code was 1.

Here are the logs from around then:

1363903503.780      0 10.220.41.7 TCP_DENIED/501 1509 GET NONE:// - NONE/- text/html
1363903736.192      0 10.124.230.159 TCP_DENIED/403 874 CONNECT 74.125.26.84:443 - NONE/- text/html
tion/x-tar
1363903985.663      0 10.220.41.7 TCP_DENIED/403 874 CONNECT 74.125.26.84:443 - NONE/- text/html
1363904103.583      0 10.220.41.7 TCP_DENIED/403 874 CONNECT portal.qntest.com:443 - NONE/- text/html

Those timestamps are in unix time. 2013-03-21 22:13:05 == 1363903985. So I guess that 74. IP address is the most likely culprit?

Seems likely - that IP address reverse-resolves to vh-in-f84.1e100.net and that's the domain Google use for internal servers.

Unfortunately I suspect this might make it tricky to add to the whitelist - I'm guessing that requests are coming in for the IP address directly (rather than some hostname alias) and adding all the IP addresses in the intetionally opaque domain is somewhat infeasible (and dubious from a security perspective even were it feasible).

Bit of a tricky one - I can't think of any practical ways around it, I'm afraid. Maybe someone else has a cunning idea.

thanks for the info. cartroo may well be right, but as a first attempt, whitelisting *.1e100.net doesn't seem like a bad shot. worth a try, anyway?

Hey, I've added it .1e100.net to the whitelist on a trial basis. Does it fix your problem?

yes, it does. thanks! would it be too much to ask to keep it on the whitelist?

Ah, that's good news - I guess Google must be using hostnames after all, unless the proxy is doing the reverse-lookup on demand. Either way, great to hear it's working!

I guess the only thing now is to consider whether there's anything else in Google's domain that shouldn't be whitelisted (i.e. any blacklist holes to punch in the whitelist)... Although personally I'd be less worried about abuse of Google's services, they've got the resources to protect themselves against that fairly effectively. I doubt any miscreants looking to take on Google would be interested in using infrastructure like PA, they're much more likely to grab themselves a botnet or four.

Not at all! We'll leave it there.

This topic would be perfect for clicking on the transfer to Wiki button that has been discussed.

Absolutely!

@glenn Estoy tratando de automatizar respaldos de la db, pero no he logrado hacerlo. SerĂ­a genial si pudieras compartir tu script para hacer ese respaldo en google drive.

Thanks for the suggestion. We do not have scripts like that ready.