Forums

I use a simple pos tagger from NLTK. Until yesterday it was working fine, this morning it suddenly produced no result. 200 OK yesterday was 403 this morning. And nothing changed in the script.

from
37.48.67.208 - - [20/Aug/2014:15:59:55 +0000] "POST /rest/words-pos/ HTTP/1.1" 200 18320 "http://antsent.net/" "Httpful/0.1.7 (cURL/7.22.0 PHP/5.4.31-1+deb.sury.org~precise+1 (Linux) Apache/2.2.22 (Ubuntu))" "37.48.67.208" to
37.48.67.208 - - [21/Aug/2014:10:32:55 +0000] "POST /rest/words-pos/ HTTP/1.1" 403 1105 "http://antsent.net/" "Httpful/0.1.7 (cURL/7.22.0 PHP/5.4.31-1+deb.sury.org~precise+1 (Linux) Apache/2.2.22 (Ubuntu))" "37.48.67.208"

And from a rest client in firefox it produced 301
178.84.70.48 - - [21/Aug/2014:11:32:17 +0000] "POST /rest/words-pos HTTP/1.1" 301 5 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:16.0) Gecko/20100101 Firefox/16.0" "178.84.70.48"
178.84.70.48 - - [21/Aug/2014:11:32:18 +0000] "GET /rest/words-pos/ HTTP/1.1" 200 37 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:16.0) Gecko/20100101 Firefox/16.0" "178.84.70.48"

Has anything changed?

Well it seems to be back...
Sorry

That looks like the site that you were hitting may have had some changes. Both the 403 (not authorised) and 301 (redirect) came from the sites that you were hitting, not from PythonAnywhere.

We moved the application to a new but identical server and the problem is back... Since it solved itself last time I have no clue...
But I don't understand your comment... the 403 error is produced by pythonanywhere... it is from the the access log.

https://www.pythonanywhere.com/user/capsence/files/var/log/capsence.pythonanywhere.com.access.log

37.48.78.102 - - [11/Sep/2014:08:24:25 +0000] "POST /rest/words-pos/ HTTP/1.1" 403 1105 "http://37.48.78.102/" "Httpful/0.2.17 (cURL/7.35.0 PHP/5.5.9-1ubuntu4.3 (Linux) Apache/2.4.7 (Ubuntu) Mozilla/5.0 (Macintosh; Intel Mac OS X 10_5_8) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.90 Safari/537.1)" "37.48.78.102"

You're right. I misread the log. But the my original assertion still stands with a little modification: The 403 and 301 are coming from your app -- that's why you see them in the logs.

Could the framework you're using have some sort of CSRF or referrer checking that might only kick in for some resources? Another possibility, since you're accessing the site with what looks like cURL, is that your login cookie has expired (or been cleared down by a web app reload).

Well... you are fucking with my brain... aren't you... :)
Because again right after your answer the service is up again :)

Well I'll look into your suggestions anyway...
Thanks

Since september 2014 when I last asked about this problem the service has been running without problems. Since last week the 403 error is back.
CSRF verification failed. Request aborted.
You mention this in your answer, but the app we are using does not use CSRF anywhere... also we do not use a name / password... For some services we use API keys...
So I'm totally lost how to solve a problem that gives a CSRF verification failed if I do not use this at all...
Or is CSRF somehow mandatory if you use a post request?
Or is https mandatory from the server that sends requests to pythonanywhere?

We don't apply any sort of CSRF protection that would affect your web apps. So it must be coming from something in your code, or in one of the libraries you're using?

Well that makes two of us. It is very hard to debug a problem that with identical code on all sides produces a 200 and then suddenly a 403.
I need to have a think... maybe move the code and see what happens.
Thanx

Which framework is your app build on?

Sorry i have been away. IT is build on Django. We Used the CSRF for login on the test bench nut that has been explicitly commented out. There is no use of CSRF any where in the code any more.

What I have found in the meantime is that using GET instead of POST solved the problem. Although I would prefer to use POST.

I have also noticed today that the test bench which was not working is working again.

This is what's making it so difficult to address this issue...

What version of django? Newer ones tend to want csrf everywhere...

I use the system default 1.3.7, could that be the problem?

You need to enable CSRF in Django 1.3. Perhaps the docs are useful.

Now I finally understand (more or less) what is going on. I did what might seem obvious, I looked in the browser if there actually is a cookie csrftoken on http://capsence.pythonanywhere.com/ and, yes there is. Then looking for info on Django and CSRF I found this comment on a forum: "Django now has Cross Site Request Forgery protection built in, and it is automatically enabled in new projects." I assumed all the time that taking it out of the settings file would disable it, well apparently it doesn't, you can not disable it.

So I added the token by copying the value of the cookie in a csrfmiddlewaretoken header and it seems to be working.

It does bring up some interesting questions.
1 On march 14 we tested our app over 10 times without a problem so a 200 and on march 17 without any change from our side we get a 403.
I understand from the documentation that a check is done on any POST httprequest so why 200 one day and 403 the next.

2 Since the site is public I assume that everybody who knows how developer tools work on a browser can just have a look at the csrftoken cookie value which seems quit strange.

3 and it does conflict with you answer a while back that is it our service that uses capsence on pythonanywhere and not pythonanywhere that causes the 403... it is capsence on pythonanywhere that causes the 403.
Or I just did not understand you correctly.

I thought I let you know,

37.48.78.102 - - [14/Mar/2015:15:27:00 +0000] "POST /rest/batch/ HTTP/1.1" 200 23576 "http://antsent.net/" "Httpful/0.2.17 (cURL/7.35.0 PHP/5.5.9-1ubuntu4.3 (Linux) Apache/2.4.7 (Ubuntu))" "37.48.78.102"
37.48.78.102 - - [14/Mar/2015:15:27:43 +0000] "POST /rest/batch/ HTTP/1.1" 200 5543 "http://antsent.net/" "Httpful/0.2.17 (cURL/7.35.0 PHP/5.5.9-1ubuntu4.3 (Linux) Apache/2.4.7 (Ubuntu))" "37.48.78.102"
37.48.78.102 - - [14/Mar/2015:16:10:26 +0000] "POST /rest/batch/ HTTP/1.1" 200 5549 "http://antsent.net/" "Httpful/0.2.17 (cURL/7.35.0 PHP/5.5.9-1ubuntu4.3 (Linux) Apache/2.4.7 (Ubuntu))" "37.48.78.102"
37.48.78.102 - - [14/Mar/2015:16:33:39 +0000] "POST /rest/batch/ HTTP/1.1" 200 5549 "http://antsent.net/" "Httpful/0.2.17 (cURL/7.35.0 PHP/5.5.9-1ubuntu4.3 (Linux) Apache/2.4.7 (Ubuntu))" "37.48.78.102"
37.48.78.102 - - [14/Mar/2015:16:34:46 +0000] "POST /rest/batch/ HTTP/1.1" 200 5543 "http://antsent.net/" "Httpful/0.2.17 (cURL/7.35.0 PHP/5.5.9-1ubuntu4.3 (Linux) Apache/2.4.7 (Ubuntu))" "37.48.78.102"
37.48.78.102 - - [14/Mar/2015:16:35:27 +0000] "POST /rest/batch/ HTTP/1.1" 200 5543 "http://antsent.net/" "Httpful/0.2.17 (cURL/7.35.0 PHP/5.5.9-1ubuntu4.3 (Linux) Apache/2.4.7 (Ubuntu))" "37.48.78.102"
37.48.78.102 - - [14/Mar/2015:16:40:45 +0000] "POST /rest/batch/ HTTP/1.1" 200 5543 "http://antsent.net/" "Httpful/0.2.17 (cURL/7.35.0 PHP/5.5.9-1ubuntu4.3 (Linux) Apache/2.4.7 (Ubuntu))" "37.48.78.102"
37.48.78.102 - - [14/Mar/2015:16:41:00 +0000] "POST /rest/batch/ HTTP/1.1" 200 5543 "http://antsent.net/" "Httpful/0.2.17 (cURL/7.35.0 PHP/5.5.9-1ubuntu4.3 (Linux) Apache/2.4.7 (Ubuntu))" "37.48.78.102"
37.48.78.102 - - [14/Mar/2015:17:08:48 +0000] "POST /rest/batch/ HTTP/1.1" 200 14758 "http://antsent.net/" "Httpful/0.2.17 (cURL/7.35.0 PHP/5.5.9-1ubuntu4.3 (Linux) Apache/2.4.7 (Ubuntu))" "37.48.78.102"
37.48.78.102 - - [14/Mar/2015:17:10:50 +0000] "POST /rest/batch/ HTTP/1.1" 200 14758 "http://antsent.net/" "Httpful/0.2.17 (cURL/7.35.0 PHP/5.5.9-1ubuntu4.3 (Linux) Apache/2.4.7 (Ubuntu))" "37.48.78.102"
37.48.78.102 - - [14/Mar/2015:17:12:54 +0000] "POST /rest/batch/ HTTP/1.1" 200 14758 "http://antsent.net/" "Httpful/0.2.17 (cURL/7.35.0 PHP/5.5.9-1ubuntu4.3 (Linux) Apache/2.4.7 (Ubuntu))" "37.48.78.102"
37.48.78.102 - - [14/Mar/2015:17:19:37 +0000] "POST /rest/batch/ HTTP/1.1" 200 14758 "http://antsent.net/" "Httpful/0.2.17 (cURL/7.35.0 PHP/5.5.9-1ubuntu4.3 (Linux) Apache/2.4.7 (Ubuntu))" "37.48.78.102"
37.48.78.102 - - [14/Mar/2015:17:38:32 +0000] "POST /rest/batch/ HTTP/1.1" 200 14758 "http://antsent.net/" "Httpful/0.2.17 (cURL/7.35.0 PHP/5.5.9-1ubuntu4.3 (Linux) Apache/2.4.7 (Ubuntu))" "37.48.78.102"
37.48.78.102 - - [17/Mar/2015:11:54:26 +0000] "POST /rest/batch/ HTTP/1.1" 403 1105 "http://antsent.net/" "Httpful/0.2.17 (cURL/7.35.0 PHP/5.5.9-1ubuntu4.3 (Linux) Apache/2.4.7 (Ubuntu))" "37.48.78.102"
37.48.78.102 - - [17/Mar/2015:11:54:35 +0000] "POST /rest/batch/ HTTP/1.1" 403 1105 "http://antsent.net/" "Httpful/0.2.17 (cURL/7.35.0 PHP/5.5.9-1ubuntu4.3 (Linux) Apache/2.4.7 (Ubuntu))" "37.48.78.102"

1 It is possible that the csrf token expired, or if you dropped the session data from your db, then new csrf tokens are required, which would explain why all the 200s turned into 403s.

2 the csrf token is supposed to be generated as a different one for each client when they initiate a session. you probably should not be artificially injecting a preset token via middleware. (because then tokens are useless)

3 yes. you/capsense trying to send a get/post request to your pythonanywhere webapp was failing because your pythonanywhere django webapp is not getting the csrf token it wants. Glenn was saying that we do not do anything with csrf (ie. we do not modify your webapp in anyway, so if there is a csrf error, it's solely due to programming problems/bugs/unaccounted for scenarios on you webapp)

I would hypothesize that your tagging code/whatever trying to get/post to pythonanywhere does NOT generate a cookie automatically. So by default your webapp rejects the request. However, if you had say navigated to your url etc manually while you were trying to debug, this would have generated a cookie for you. And maybe your code then is able to get/post successfully using that cookie.

Well I don't want to drag on the conversation, now I have some understanding of the problem I will close this.

But I think you are missing the point.

1 The CSRF compares the value of the csrftoken cookie with the csrfmiddlewaretoken value in the header. You have to incorporate this in your code.
And all the time weather I got a 200 or a 403 I never ever supplied a csrfmiddlewaretoken value.
I never ever used CSRF in any way.
Something apparently decided to say 403 one time and 200 another time.
That is what is so frustrating, that I can not find any reason why.
This issue started when I moved the app to pythonanywhere.

2 I don't want to use CSRF... I have never ever used or wanted to use CSRF because we use an api-key.
So that the token is pointless, I don't really care, because I don't want them in the first place.

3 The comment I found confusing was:
That looks like the site that you were hitting may have had some changes. Both the 403 (not authorised) and 301 (redirect) came from the sites that you were hitting, not from PythonAnywhere.

Well, the really strange thing is that the docs say that, in django 1.3, csrf protection is only active if you explicitly switch it on via the middleware settings. And you say you've switched it off. But then again, django 1.3 isn't supported any more, so it's possible their docs are incaccurate...

I think this may have been caused by a bug we just fixed.

Here's what it was: if a web app was shut down for some reason (system reboot, certain kinds of glitch, excessive resource usage, maybe hibernation) then only a GET request would wake it up. POST requests, in particular, would be rejected with a CSRF error (generated by our code that's meant to start up the web app), and the app wouldn't be woken up. So if your app is one that processes mostly POST requests, you'd see this problem. This definitely seems to fit the issue as you describe it.

Our new code wakes up the app when a POST is received. One slight issue remains -- the first POST request that wakes it up will receive a "503 Service Unavailable" response with the "retry-after" header set to "5". If you handle this and do the retry, then the next request will work. We believe that browsers do that automatically, but unfortunately the requests library doesn't by default.

Well it is good to know that I was not mad after all. My app did only POST. I Will test everything again and switch back from GET to POST. unfortunatly we did move some of the logic to other services. But of posible we will switch back.

Sorry it took so long to track down!

Just a quick note to say that now we've improved the website startup code further; regardless of whether the first hit to a shut-down website is a GET or a POST, it will now start up normally. No more 503 status codes :-)