Forums

I found a really cool Django toolset: github.com/django-extensions/django-extensions. Amongst other tricks, it can generate a visual data diagram of your models.

Its in pypy, and installed with pip into my virtual env without error. Just don't forget to import it as "django_extensions"....

But it depends upon https://pypi.python.org/pypi/pygraphviz/1.1

Which depends upon http://www.graphviz.org/

pip tried to install pygraphviz but it complained it could not find graphviz.

I definitely agree that Graphviz is a handy tool, if not too well known, and well worth making available. There are some sites which allow you to generate the images via a web service (this is one example and another) and I'm sure I remember one of these being set up as a proper API where you just provide the graph as a query parameter - I'm damned if I can find that site now, however. This pupports to be an AJAX version, but doesn't look all that polished. (EDIT: this looks to be a more professional-looking Javascript implementation).

Besides, a local version would be quite useful. I'm not sure if any of the dependencies might be tricky, however. I'm pretty sure there's an Ubuntu package, which should hopefully make life easier - bit old but it hasn't changed in any major way in years.

I'll add Graphviz to the list, if it's something we can easily install then it should get in pretty quickly.

Hm. I just checked the Ubuntu package's dependencies and they do include libx11 - not sure if that's likely to cause a problem:

Depends: libc6 (>= 2.15), libcdt4, libcgraph5, libexpat1 (>= 1.95.8),
lpm (>= 2.0.36~rc1~dfsg) | libgd2-xpm (>= 2.0.36~rc1~dfsg), libgraph4,
libgvpr1, libx11-6, libxaw7, libxmu6, libxt6

I'm a little puzzled as to exactly why they have this - the utilities are purely command-line. Probably some little utility you don't need has a GUI or something. Ah, the joys of binary distributions.

Right -- or maybe there's some kind of image-handling code that used something from libX11 just because, hey, it's always going to be there...

Of course, right now the execution servers are all running Debian -- seems to be roughly the same version (with the same X11 dependencies) as Ubuntu, though.

Surely you people with the power of access to gcc could bake your own from source?

If nothing else, you would find out the exact dependencies real quick.

Here is another one that claims to have a good url entry point. I say "claim" because the "how to" on this page is a bit fragmented and possibly incomplete. Is the one you were trying to remember earlier?

http://code.google.com/p/django-graphviz/

After you get graphviz installed, I will try and get that to work through an url & view.

I think you should grab what I did and make a howto or FAQ entry. The only wrinkle to that plan is the contacts app is a bit "young" - I think most users are going to need to customize it. And that is kinda hard when it installs as an egg file, no?

So I'm planning to uninstall the egg file and copy over the "src" tree instead.

I seem to remember that Graphviz has an unreasonably large number of built-time dependencies, but it's possible I'm remembering it as more painful than it really was. I do know that I tried to build the latest version for some minor feature or other and ended up just using the Ubuntu one in the end because it was too much of a pain to get the source distribution building.

Of course, that was some time ago and these things do change.

Building from source would probably solve at least the X11 dependency - running ldd on the dot binary, for example, yields the following actual runtime dependencies:

linux-vdso.so.1 =>  (0x00007fff5b5fd000)
libgvc.so.5 => /usr/lib/libgvc.so.5 (0x00002b394b5f5000)
libgraph.so.4 => /usr/lib/libgraph.so.4 (0x00002b394b872000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00002b394ba80000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00002b394be3e000)
libcdt.so.4 => /usr/lib/libcdt.so.4 (0x00002b394c042000)
libpathplan.so.4 => /usr/lib/libpathplan.so.4 (0x00002b394c249000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00002b394c451000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00002b394c67b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00002b394c893000)
/lib64/ld-linux-x86-64.so.2 (0x00002b394b3d0000)

Still, Graphviz is going to be a peripheral package at best, so using an off-the-shelf version is going to be a lot easier from a maintenance perspective.

Agreed, we certainly could build it but installing the binary will probably be faster. And if ldd doesn't say it needs the X11 stuff, it's certainly possible that we can avoid putting those into the user sandboxes too.

OK, it's on the list, and if it's easy then we'll get it in at the next package build. I don't have a timeline for that, but it's normally every week or so.

Excellent!

I was also going to say I'm happy to use another package that does the same thing: Generate UML (or UML-ish) graphs of your model file(s). This was the first "recent" solution I found, there may well be others.

Graphviz is about the only tool I've found which does a half-decent job of automatic layout - probably not surprising that it's a product of academia (AT&T Research Labs, originally) as it's a surprisingly tricky problem. This probably explains why it's quite widely used as opposed to people designing their own solution.

Re: online API, I just found that Google had an API for this but that they've deprecated it. Seems to be a lot of that going round at Google right now - presumably next year they'll deprecate their little, underused search functionality, too.

Would we need all the graphviz binaries, do you think?

/usr/bin/lneato /usr/bin/ccomps /usr/bin/gvgen /usr/bin/nop /usr/bin/gc /usr/bin/dotty /usr/bin/vimdot /usr/bin/mm2gv /usr/bin/gxl2gv /usr/bin/sccmap /usr/bin/diffimg /usr/bin/acyclic /usr/bin/gvpr /usr/bin/gvpack /usr/bin/gvcolor /usr/bin/dijkstra /usr/bin/dot /usr/bin/tred /usr/bin/prune /usr/bin/gml2gv /usr/bin/bcomps /usr/bin/unflatten /usr/bin/lefty /usr/bin/circo /usr/bin/sfdp /usr/bin/osage /usr/bin/neato /usr/bin/gv2gxl /usr/bin/twopi /usr/bin/gxl2dot /usr/bin/fdp /usr/bin/dot2gxl

Are they useful in their own right?

Haven't quite got time to check them all right now, but at least some are useful. Each layout engine is its own binary - dot is used for hierarchical directed graphs, for example, whereas neato is for undirected graphs with no real ordering. You can find the main layout engines mentioned on the Graphviz front page.

As to whether the others are useful, I suspect it's a mixed bag. The vimdot tool appears to be a simple wrapper around editing a file and updating the graphical rendering whenever it's saved, for example, which would be of little use without an X server.

I suspect some of the others may be potentially useful (such as dot2gxl which sounds like it translates from the native Graphviz dot format to something else) but possibly only for esoteric cases.

I'll try and find time to chase them all up, but in the meantime I doubt any are actually harmful - just potentially not useful.

I've spent a little time chasing these up and I think the tools can be divided into a few categories. Personally I think there's little harm in including all of the tools except those that require a GUI, but the core tools are the most important - those are the ones that third party software is most likely to rely on. If you feel that more binaries equates to more maintenance effort, however, you may wish to trim the list.

As an aside, I never realised how much obscure stuff there was as part of Graphviz - some of the graph filters in particular look quite interesting. Useful? Mmmmm, maybe in esoteric circumstances... But interesting.

Anyway, the list of core tools, which actually produce image files from graph definitions, are as follows - these correspond to the different layout engines so different types of graph will use a different tool:

• circo - circular layouts for undirected graphs.
• fdp - undirected graphs using a spring model (a newer / less mature alternative for neato).
• dot - standard tool for drawing directed graphs, probably the most commonly used one.
• neato - undirected graphs using a spring model.
• osage - draws clustered graphs recursively.
• sfdp - alternative to fdp for larger graphs.
• twopi - undirected graphs using a radial layout.

There are also a bunch of command-line tools to filter graph files in various ways. Most of these are of primary use to academics, but I can imagine more mainstream software having a specific use for them under specific circumstances. For example, ccomps might be useful for network visualisation software for a large network, to identify hosts in a cluster which have become partitioned:

• acyclic - reverse edges in a directed graph to remove any cycles.
• bcomps - split a graph into biconnected components.
• ccomps - split a graph into unconnected components.
• dijkstra - calculate distance of every node from a specified origin node.
• gvcolor - colour graph nodes according to the flows through them.
• gvpack - merges disjoint graphs into a single graph definition.
• gvpr - manipulate graphs arbitrarily, a Graphviz equivalent to awk.
• prune - label subgraphs rooted at specified nodes to be removed by gvpr.
• sccmap - split a graph into strongly connected components.
• tred - calculate the transitive reduction of a graph.
• unflatten - improve layout of graphs with many leaf or unconnected nodes.

There are a set of command-line tools which convert to and from the native Graphviz "dot" format (sorry, it appears I'm supposed to call it "gv" format these days according to the man pages):

• dot2gxl - old name for gv2gxl.
• gml2gv - convert from GML format to dot.
• gv2gxl - convert from dot to GXL format.
• gxl2dot - old name for gxl2gv.
• gxl2gv - convert from GLX format to dot.
• mm2gv - convert from Matrix Market format to dot.

There are also a few tools to do other tasks - mostly these are Graphviz-related, but I can't figure out why on earth diffimg is included. I mean, I can see it could be potentially a useful utility in a number of situations, but no idea whatsoever what it's got to do with graphs specifically. Maybe they use it for debugging their layout engines.

• diffimg - create an image which is the diff of two other images.
• gc - show graph statistics, the Graphviz equivalent to wc.
• gvgen - generate a graph from a set of predefined types.
• nop - pretty-print a graph source file.

Finally, I can't see how the following tools would be useful without an X-server. Potentially I guess people might want to run them over SSH with X-forwarding, but that's getting pretty obscure for PA to offer. I can't see any explicit harm in including them if it's easier just to include everything, but short of X-forwarding they'll mostly just quit with the usual error about not finding a display:

• dotty
• lefty
• lneato
• vimdot

Hopefully that's some use. It was also an interesting blast from the past - I haven't messed around with graphs this much since my Cambridge Uni days. Running gvgen -S6 | neato -Tpng -o/tmp/test.png kept me amused for, oooh, at least 30 seconds. Good times.

I know PA staff said it's "on the list," but I thought I should speak up so that you know there's more support for this... I'm evaluating PA before purchasing the service, and I would love to see graphviz added. It's not strictly a requirement, but I was hoping to deploy, at some point, a tool that I developed locally that requires graphviz.

PS -- I love your service. I'm a first-time python user, using web2py on PA, and it's a blast. Learning this is more addictive than gaming.

Thanks, evanphelps -- I've added an upvote, and I'll take a look on Monday to see if we can push this through -- I know Harry was taking a look last week.

Just to record my vote for this, same as evanphelps, I am still evaluating. I noted this when I tryed to access to web2py database graph model feature and got the "pygraphviz library not found" error.

Thanks for the heads-up, I've added another upvote. It does look a little trickier than we'd hoped, but we'll see what we can do.

In case it's any use, I know that Ubuntu has a bunch of pre-built packages for some of their releases. Even if the template system isn't Ubuntu, you might be able to find one with sufficiently similar library versions and crib the binaries from it and build your own .deb from those.

Any updates on adding Graphviz and the Python extensions to PAW?

graphviz and pygraphviz are in testing now, hoping to deploy them in the next few weeks...

And they're now released! Let us know how you get on with django-extensions...

Just tried a basic test of dot generating a PNG file and that seems to work fine. Great stuff!

Thanks for checking it out :-)

is now graphviz availlable ?

I tried import graphviz and pip install graphviz and both made an error

I also think this is an extremly important feature for some applications

We have all the graphviz binaries available (try eg which dot), and the pygraphviz module is available by default. If you want the "graphviz" python package, you can install it with

pip install --user graphviz

Or use a virtualenv. More info here: https://www.pythonanywhere.com/wiki/InstallingNewModules