Forums

eggswithcheese.pythonanywhere.com is my project.

I'm using jsonpickle to load data into memory and pass it around between views. Installed it in the bash console using pip. Whenever I try to upload data files to the site I get an error code.

I did try reloading. No, there's nothing in the server error logs. Yes, I commented out app.run

Not sure if jsonpickle is what's causing it, but it seems likely. The version of the program that didn't use jsonpickle worked fine on PythonAnywhere.

Not sure what's going on. Anyone got any ideas?

What is the error code that you get? What does flask say when you turn debug mode on?

Error code: 502-backend

I didn't have debug mode on. In fact, I commented out everything in the main method.

Perhaps I should mention that I've been testing it successfully on Windows using PyDev. It runs fine there, with no errors.

How big are the data files? I know that there are some limits on uploads with PythonAnywhere.

~350 KB

How long does it take for that error to come back after you submit the form?

~ 5-6 seconds.

Perhaps it's worth trying some manual logging -- that is, wrap the view that's giving the error in a try...except and in the except block, write the exception to a file?

How do I do that on PythonAnywhere? What do I put in the Except?

Like this

try:
    # do stuff...
except Exception as e:
    with open("/tmp/mylog.txt", "a") as f:
        f.write("Caught exception {}\n".format(e))

Unfortunately, I can't seem to get any exceptions written to the file. This technique is still useful though. Using this technique I peppered my code with

        with open("/home/eggswithcheese/WSU-PAL-PPC/mysite/tmp/mylog.txt", "a") as f:
                f.write("program has reached such-and-such section \n")

Which at least allowed me to figure out where it's crashing, which has now confused me even more.

In my index view I have this line:

return redirect(url_for("primary_production"))

Which should result in the program executing this:

@app.route('/primary_production', methods=['GET', 'POST'])
@app.route('/primary_production.html', methods=['GET', 'POST'])
def primary_production():
    '''
    Renders the primary_production template, which shows calculated values and a button to download them.
    '''
    with open("/home/eggswithcheese/WSU-PAL-PPC/mysite/tmp/mylog.txt", "a") as f:
        f.write("in primary production view \n")
    try:
        with open("/home/eggswithcheese/WSU-PAL-PPC/mysite/tmp/mylog.txt", "a") as f:
            f.write("trying to render template of primary_production.html \n")
        return render_template("primary_production.html")
    except Exception as e:
        with open("/home/eggswithcheese/WSU-PAL-PPC/mysite/tmp/mylog.txt", "a") as f:
            f.write("Caught exception {}\n".format(e))
        print str(e)
        return render_template(INTERNAL_SERVER_ERROR_TEMPLATE_ROUTE, error = str(e))

But the program somehow never gets there. The line "in primary production view" never gets written to the file! Why would it crash in between views?

Interesting! Well, the return redirect call is returning a redirect message to the browser, and then the browser will be making a request to the primary_production view. So perhaps if you watch what's going on in your browser's developer tools you'll see what's going on?

Well, I did look at the development tools, and messed around with the timeline. Not sure what most of it means, but it did note that there was one error:

Failed to load resource: the server responded with a status of 502 (Bad Gateway)

A quick note: I do indeed have a template for it to find, so I don't know why the server cannot find it:

primary_production.html

{% extends "base.html" %}

{% block head %}
    <title>Primary Production</title>
{% endblock %}
{% block content %}

    <h1>Primary Production from user data</h1>

    <p>Primary Production Values: </p>
    {#http://jinja.pocoo.org/docs/dev/templates/#list-of-control-structures#}
    {#http://blog.bouni.de/blog/2013/04/24/call-functions-out-of-jinjs2-templates/#}
    <ul>
    {% for pond in ponds() %}
        <li>
        Year: {{ pond.get_year() |e }},
        Day of Year: {{ pond.get_day_of_year() |e }},
        Lake ID:{{ pond.get_lake_id() |e }},
        </li>
    {% endfor %}
    </ul>


    <a href={{( url_for("export_view",filename=request.args.get('filename'))) }}>Click here to download</a>




{% endblock %}

base.html

<html>
<link rel="shortcut icon" href="{{ url_for('static', filename='favicon.ico') }}">
    <head>
        {% block head %}
        <link rel="stylesheet" href="style.css" />
        <title>{% block title %}{% endblock %} - My Webpage</title>
        {% endblock %}
    </head>
  <body>

<script type="text/javascript">
<!--
    function toggle_visibility(id) {
       var e = document.getElementById(id);
       if(e.style.display == 'block')
          e.style.display = 'none';
       else
          e.style.display = 'block';
    }
//-->
</script>



    <div><a href="/index">Home</a></div>
    <hr>
    {% block content %}{% endblock %}
    <p class = "source_code_statement">WSU Pond And Lake Primary Production Calculator. Source code can be found <a href = "https://github.com/EggsWithCheese/WSU-PAL-PPC">here.</a></p>



    <input type="button" value="Toggle Pond Picture" onclick="toggle_visibility('pond_picture');"/>     
    <p></p>
    <img id="pond_picture" src="{{ url_for('static', filename = 'pond.JPG') }}" alt="WSU pond" style="width:1024;height:768;display:none;">
    <!--<p>WSU Nutter Center Pond</p>-->
  </body>
</html>

Wait. Apparently it actually runs this before crashing? I'm confused as to why this would happen.

@app.errorhandler(404)
def pageNotFound(error):
    with open("/home/eggswithcheese/WSU-PAL-PPC/mysite/tmp/mylog.txt", "a") as f:
                f.write("pageNotFound view \n")
    return "Page not found"

When it does the redirect, what is the URL it's using (as shown in the JavaScript console)?

It's completely blank. Am I looking in the right place? error page, with console

http://imgur.com/7WtPfgA

Here's another, with the timeline.

link

http://i.imgur.com/We5F72P.png

In this one, it just shows the url as http://eggswithcheese.pythonanywhere.com/

Is there a way I can try this out myself?

Well, you could go to http://eggswithcheese.pythonanywhere.com/

download the example data file, then upload the example data file.

I think the issue is the multipily-routed primary_production view. Have a look at this SO discussion

See, that theory confuses me because (1) that exact thing works on my home computer, and (2) the multiple routes were working on PythonAnywhere before.

All right, I tested it. Removed one route, reloaded the site, tried it. Same error.

Removed the other route, reloaded the site, tried it. Same error.

Ladies and gentlemen, I am now convinced it has something to do with jsonpickle. When I comment out the parts of the code related to pickling my Python objects into the session dict, the program ceases to crash. I no longer get a 502 error.

Have you considered just using standard pickle, and pickling to text?

Also, I had a number of problems in the past, where I ended up with objects - which I thought just contained data - also contained a bunch of methods. The consequence of which was that my picked objects were huge.

Well, I should have probably gone that route from the beginning. I wrote the class before I realized it would have to be pickled. At that point, jsonpickle seemed like a wonderful solution due to how simple it was: just do jsonpickle.encode(pond,keys=True), and later jsonpickle.decode(pickled_pond, keys=True) no need to rewrite my classes at all.

I'm confused as to why this would even cause a problem, though. Why doesn't this work with PythonAnywhere? If I do go and rewrite everything using standard pickle, will I get the same error?

Also, will I be able to call the object's methods when I unpickle it? That's kind of important.

My guess is that on your local file the pickled object was 5mb (or whatever), and that really wasn't a problem.

But on PythonAnywhere, there are memory limits for free accounts and you're blowing through them,

Why not simply add "save" and a "load" methods to your class that just pickle the data parts of the object.

Why not simply add "save" and a "load" methods to your class that just pickle the data parts of the object.

Not a bad plan! I'll investigate it.

So basically we're saying that (on PythonAnywhere) I have to deal with file I/O then? I have to save the objects to a data file between views? Won't that build up over time, cluttering up the disk?

I don't know what kind of cleanup PythonAnywhere does, but when I get this running on the server I don't want every user creating data files I have to delete periodically or something.

The whole point of serializing the objects was to avoid this problem. Before I started throwing objects in the session dict I was saving all the data to disk, and it was cluttering up my hard drive. I did investigate "temporary files", but it turns out you still need to delete them manually (on some OS, the tmp folder is only cleaned up on restart. On Windows, I don't think it ever gets cleaned up). And I don't know when the user is done with them. And I don't want to restart the server every day. I suppose I could schedule a Cron job to do it periodically, but that seems inelegant.

Maybe I don't know what I'm talking about though. Is there some solution to this I don't know about?

Now: I don't work for PythonAnywhere, so you shouldn't take my word as gospel...

My understanding is that you are running your app on a distributed cluster of web servers. This means that you can scale to 10,000 people using your site at once, in a way you simply couldn't in a traditional "I've got an Amazon EC2 instance".

But there is a tradeoff with this. Because you are running your instance over multiple machines, that can be rebooted, or upgraded or the like, you need to ensure permanence of data between sessions. (My understanding is that individual sessions always go back to the same server.)

To ensure permanence you can use a database for that (sqlite is super, super easy), or you can use the filesystem (which is also pretty easy).