Forums

Basically, I have some trained models that I want to expose via an API. The models can be quite large and they need to be constantly kept in memory for speed. Is PythonAnywhere a good choice for this type of use case?

More details: - The main application would handle user management, measuring API usage, billing - Another application would be an API gateway - The other applications would be microservices that expose access to some ML models. (3-4 apps)

Thanks.

It's probably not going to be ideal for the machine learning code itself :-( Our web-serving infrastructure is optimised for typical websites, which have relatively small memory footprints, and normally respond to a typical request within seconds, rather than minutes.

Awesome, thank you for the quick and honest answer :)

No problem -- always happy to help (and loath to mislead :-)

Unfortunately, I think we have two ships passing in the night here.

If you already have a trained model object that you've serialized - and want to create a Web app to load/deserialize that model and make it available via an API - I think PythonAnywhere is a perfect choice. Serving a (Python) machine-learning model object is not resource-intensive, and requests should return in milliseconds.

If, however, you're looking to upload large datasets and train new models, then I'd take giles' advice... that part takes significant CPU cycles.

When serving an existing model, all you're doing is sending some parameters to your model object; it applies some internal logic and returns a result. When training a new model, you're trying to get an algorithm to figure out specifically what that logic needs to be - which takes some horsepower.

Just kind of some vague guidelines: a logistic regression model may occupy 2KB RAM; my last Gradient Boosted Machine weighed in around 200KB; and I've recently worked with a Random Forest model roughly 1.5MB in size. About to find out how big the CatBoost objects are :)

thanks @jebstone for the clarification!

I agree- serving results from an existing model is not resource intensive, but if you want to use the webapp platform to train models / allow your webapp users to build / train models then that would not work well.

Dear @jebstone and @conrad, I must humbly disagree even though I always respect your professional answers.

The reason is that in the last two days, I tried to deploy a very simple flask app for image classification with Keras/Tensorflow that worked perfectly on my local machine. Porting them involved many changes but I eventually got a version that leaves no error messages in the server log and - does not execute the model predictions!

I swear that there is nothing special in my script that could prevent that.

Please have a look - it's a 100-liner. If you find the cause that prevents the model predictions, I'll buy you a coffee - serious about that!

from keras import backend
import tensorflow as tf

session = tf.Session(config=tf.ConfigProto(
    intra_op_parallelism_threads=1,
    inter_op_parallelism_threads=1,
    allow_soft_placement=True
    ))

backend.set_session(session)

from flask import Flask, render_template, request

from keras.applications import resnet50, inception_v3, mobilenet
from keras.preprocessing import image

import numpy as np
import os
# ERROR Initializing libiomp5.dylib, but found libiomp5.dylib already initialized.
os.environ['KMP_DUPLICATE_LIB_OK']='True'

# file processing
from werkzeug.utils import secure_filename

# import global variables (model_mobilenet, model_resnet50, model_inception_v3)
import mymodels

# create Flask app
app = Flask(__name__)

def predict_dog(image_input, model) -> str:
    if (model.name == 'mobilenet_1.00_224'):
        model_preprocess_input = mobilenet.preprocess_input
        model_decode_predictions = mobilenet.decode_predictions
    elif (model.name == 'resnet50'):
        model_preprocess_input = resnet50.preprocess_input
        model_decode_predictions = resnet50.decode_predictions
    elif (model.name == 'inception_v3'):
        model_preprocess_input = inception_v3.preprocess_input
        model_decode_predictions = inception_v3.decode_predictions

    # read image input
    img = image.img_to_array(image_input)
    # reshape data for the model
    img = np.expand_dims(img, axis=0)
    # prepare the image for the VGG model
    img = model_preprocess_input(img)
    # predict the probability across all output classes
    predictions = model.predict(img)
    # convert the probabilities to class labels
    label = model_decode_predictions(predictions, top=1)
    # retrieve the most likely result, e.g. highest probability
    label = label[0][0]
    # print the classification
    classification = '%s predicts: %s (%.2f%%)' % (model.name, label[1], label[2]*100)
    return (classification)


def get_file_path_and_save(request):
    # Get the file from post request
    f = request.files['file']

    # Save the file to ./uploads
    basepath = os.path.dirname(__file__)
    file_path = os.path.join(
        basepath, 'uploads', secure_filename(f.filename))
    f.save(file_path)
    return file_path

@app.route('/', methods = ['GET'])
def show_index():
    return render_template('index.html')

@app.route('/predict', methods = ['GET', 'POST'])
def predict_image():
    if request.method == 'POST':
        file_path = get_file_path_and_save(request)
        img_224 = image.load_img(file_path, target_size = (224,224))
        img_299 = image.load_img(file_path, target_size = (299,299))

        result = '<p>'
        result += predict_dog(img_224, mymodels.model_mobilenet)
        result += '</p><p>'
        result += predict_dog(img_224, mymodels.model_resnet50)
        result += '</p><p>'
        result += predict_dog(img_299, mymodels.model_inception_v3)
        result += '</p>'

        return result
    return None

if (__name__ == '__main__'):
    print('* Loading Keras models and starting Flask server...')
    print('.....')
    # get wsgi server to replace flask app.run()
    from gevent.pywsgi import WSGIServer
    web_server = WSGIServer(('', 5000), app)
    web_server.serve_forever()

    print('Success! Server available at http://127.0.0.1:5000')

Dear @jebstone and @conrad,

I am now almost sure that pythonanywhere is unable to run deep learning with Keras models.

I have spent the whole day debugging, and finally figured out why my above script works locally but not on pythonanywhere: My script calls the Keras models' predict function

predictions = model.predict(img)

This takes 1-4 sec locally. On pythonanywere, however, this starts but never finishes.

Some print() before each function blocks confirms that:

2018-12-02 09:20:43 entered predict image
2018-12-02 09:20:43 predict_image() entered
2018-12-02 09:20:43 image preprocessed
2018-12-02 09:20:46 Sun Dec  2 09:20:46 2018 - received message 1 from emperor
2018-12-02 09:20:46 ...gracefully killing workers...
2018-12-02 09:20:46 Gracefully killing worker 1 (pid: 44)..

So according to all threads in this forum, Keras with Tensorflow backend must be executed in single thread mode which I did successfully with the threads parameters set to 1.

@conrad @jebstone @giles : Can you find a possible solution or at least a potential explanation why pyhtonanywhere cannot execute the Model.predict() ?

This is a strategically important question because I intended to teach deep learning in my graduate course with pythonanywhere. For students, it will be so much more fun to see deep learning run on a web server. I guess there will be many who are attending to this question too. Would be great if you have a look. Please do access my code on agilebean.pythonanywere.com.

Keras models run fine on PythonAnywhere, just not necessarily in web apps. In general the issue appears to be that the prediction tries to start thread and that is not allowed in web apps on PythonAnywhere.

You can try to run your models outside of a web app. See http://help.pythonanywhere.com/pages/AsyncInWebApps/

Hm, I know that PythonAnywhere doesn't allow multi-threading that's why my script prevents that by setting the intra_op_parallelism_threads=1 parameter. So PythonAnywhere should normally run my code. Unfortunately, your answer makes the conclusion that PythonAnywhere is unable to run deep learning models with Keras. That's really sad, as it would have been the ideal learning environment for my students.

From what we've been able to determine, Tensorflow will always try to start multiple threads even if you tell it not to, so you're right that it won't work in PythonAnywhere websites. However, I know that some people were able to just switch the Keras backend over to Theano and it worked fine -- I don't know enough about Keras to say whether that's a generally-usable solution, though.