Forums

First of all, I have to say that I just found out about PythonAnywhere and I'm blown away. Great service, guys!

I would like to use PA for a web app that does some number crunching behind the scenes. Ideally I would run my CI builds on the same machines and environment as the web site. Even better would be to run the CI master on one of the PA nodes so it's available to all devs. Is that allowed? If so, what are other people using for this? Given the choice I would like to use jenkins, for ease of maintenance, but it looks like java is not installed.

Any advice? Thanks.

Hello, welcome to PA.

When you're talking about using Jenkins, I assume you're doing builds of Java or C/C++ projects on the PA infrastructure? This isn't going to get far, I'm afraid, because PA has no compilers for any of those languages installed. There has been discussion of adding a C/C++ compilation environment, primarily for compiling Python extension modules, but it may be some time before this becomes reality. I'm not aware of any plans at all for either a Java VM or compiler, and also I haven't seen any requests for one on the forums, but there's nothing to stop you putting in a request for the PA developers to consider.

If you're not talking about compilation, but rather something more like unit testing, then I would suggest that a full CI system like Jenkins is overkill - just have developers run the unit tests prior to committing the code, and perhaps have a daily scheduled job which runs them over the repository overnight. PA's batteries included list includes nose and unittest2 are the two most popular libraries (and the built-in unittest, of course).

If none of that seems to apply to what you're trying to do, perhaps it would be best to explain what you're trying to achieve, including whether there are any non-Python elements to your server.

Some sort if CI infrastructure sounds like a really good feature we could add. As it stands, I doubt any of the standard ones will work very well (they seem to depend quite heavily on being able to open sockets and listen on them). That siad, you could build a really simple CI out of a built-in Python test framework and a git commit hook. (Left as an exercise for the reader)

I actually started work on something like that a while back.

Isn't that a 1-line Git hook which runs python -m unittest discover on each commit and pipes the results into an email? I guess I must be missing something... (^_^)

Yes I think we would all be interested in hearing more details. Would be a worthwhile thought experiment to see if we could make something work.

It's certainly possible to run headless Selenium Webdriver tests on Firefox. Essentially that is what we use for our integration tests. The fact that they are launched by buildbot is not really that important.

It's a little bit more than that ("a little bit" meant literally rather than as understatement ;-). It's a hook that checks out the new commit, then runs a run_integration_tests script taken from the repo. If that succeeds then it calls a script called promote_to_live, and if it fails then it calls handle_integration_error with the output of the test. So basically a minimum viable continuous deployment script. Nothing terribly complicated, but there were a bunch of weird edge cases that made it harder than I'd expected.

The fun bit was making it self-hosting. Basically, I have a repo called development and a bare repo called integration. The origin remote on both of them is the GitHub repo, but development has the integration repo as a remote called integration. The integration repo has the hook installed. The codebase defines its own run_integration_tests (which in turn creates repositories with appropriate relationships and checks that pushing between them runs the scripts appropriately), handle_integration_error (which emails me test failures) and promote_to_live, which pushes the integration repo to GitHub.

All that means that I can work on it in the development repo, and do a git push integration master every now and then. That runs the tests and pushes to GitHub if they pass. Which means I can be sure that the version on GitHub always passes its own tests.

Not rocket science, but it was interesting how many bits of weirdness cropped up when I was trying to make it all hang together properly.

Yeah, OK, that's fair enough, it's doing a bit more than I first thought. Nice idea thinking of the integration stage as a "proxy" for the master repo.

I've always set stuff up where the integration environment pulls from top-of-tree in the master repo, does a build and if things succeed it tags the state of the tree at that point and commits the updated tag back to the central repo. So, developers can choose to sync down the tagged version for a guaranteed good build, or top-of-tree for bleeding edge. Still, I've never had occasion to set up CI with a distributed source control like Git, and I guess that allows you some more flexibility.

My concern in a large team would be whether changes would all get clogged up in the integration server if someone pushes something broken and then goes on holiday. Or were you assuming that in a team each developer would have their own integration repo? I guess that works well as long as it's fairly easy to build from a set of different repos (assuming the build server is shared), but it shouldn't be hard to do a clone each time (presumably with --depth=1) - in fact, that's a neat way to make sure every build is done from clean.

@Cartroo To clarify, I am only interested in python development; I only mentioned java because jenkins depends on it.

@glenn, giles, hansel I understand I could manually run my unit tests with a script, and maybe have that send me an email or something. But I would rather have the history be browsable, which is what a CI system gives you. (Like buildbot, for example, http://buildbot.buildbot.net/ )

Buildbot would be great, and it looks like all the dependencies are already installed(?). Running the master on the PA server would be fantastically convenient for us users, of course, but especially important would be to run a slave there so it tests everything in the place where it matters.

Buildbot was exactly the system I was thinking of when I said they wouldn't work very well. Buildbot uses various socket connections between the master and the slaves. Sockets are one of the weird, dark, don't-look-there corners of PythonAnywhere. We have plans that should change this, but they're still in their infancy.

Okay, I think I understand more of the issues regarding ports and the way PA operates. I think I can see a way to do most of what I would like, but I'd like to know if the staff approve.

I could strip down a jenkins server (running straight out of the .war with a local jre) so that it only uses one port, for the http interface on 8080 default. Then (with a paid account) I could port forward over ssh as in https://www.pythonanywhere.com/forums/topic/437/#id_post_2986 and start jenkins. Then I point my browser at localhost:8080 to connect to jenkins on the remote host.

The downsides are that I could not use slave builders; all tests would run on the master. Also, no one else would be able to access that jenkins server, and I would have to keep that ssh session going while I run a build or browse around.

Does that sound like it could work? I tried starting hudson on the bash console with my free account, but it failed with this, I assume because it tried to listen on 8080 with my N00b account: WARNING: Could not intialize the host network interface on nullbecause of an error: glenn-liveconsole: glenn-liveconsole java.net.UnknownHostException: glenn-liveconsole: glenn-liveconsole

The only ports accessible to free accounts are 80 and 443 via the proxy server. But that is access not hosting services.

Sorry duhctaep,

As Glenn said, this won't work reliably at the moment. We are currently thinking about a fundamental re-arch of PythonAnywhere that would make this possible. But at the moment I wouldn't advise doing it.

@hansel: The traditional approach in software is to thoroughly re-plan every last aspect of the service from top to bottom, employing the phrases "learning from our mistakes" and "no sacred cows" as often as possible and always preferring to redesign from scratch anything with the slightest flaw. After all, the software industry is still pretty young - we're never going to develop a consistent culture if we don't continue to observe these quaint little traditions.

Then take all those discussions and notes, drop them in a folder clearly marked "version 2.0". Then burn it and get on with something sensible for version 3.

(^_^)

Don't worry, when I say re-arch, I mean incremental step in the right direction :-) But something that would present more of a 'here is a machine that you have root access to' interface.

Yes, I guess if you can listen on arbitrary ports and the like then it wouldn't be a million miles away from a VPS - except that the P in this case would of course stand for Python.

PythonRootAnywhere.com...FTW

☺

Oooh, a Virtual Python Server. Now that sounds like it has marketing legs :-)

I'd buy it for a dollar...☺