I think keeping things in house is a laudable aim - that's usually my natural inclination to the extent that I have to fight it even when good quality third party apps and libraries are available. You might have noticed I have something of a "throw ideas in and see what sticks" approach.
@a2j: Not quite sure about SO-style comments - that's very much a focussed QA site, and you can see that despite their limitation on comment length and automatic collapsing of comments, threads still sometimes diverge significantly. Forking a topic into a separate thread would seem to be the cleaner solution, and hopefully quite easy to implement - the new thread would presumably acquire the post at the "fork point" as well a "this thread was a fork of...." link.
However, the ability to simply reply to another post would be handy - it would just pre-populate the post box with the quoted version of the original post, and also be tagged somewhere as "in reply to post X". While on the subject of quoting, minor point - could the size of quoted text be cranked down slightly? It always seems a little odd that text you're quoting to reply to is larger than the reply itself. I'd make them the same size, but indent the quote and colour it lighter grey or blue.
Closing a topic to further discussion is useful, but I'm not sure who should have the privilege to do it - it would be frustrating if someone posted some blatant misinformation and then the OP closed the topic before it could be corrected. I suggest that only admins be allowed this, initially.
The generic flags approach seems potentially useful, but just being able to "star" a thread as a favourite with a separate index page listing favourite threads (in the same order of last update as the main index) would probably be sufficient. Perhaps we could implement optional email updates for posts in starred threads (or have a separate set of "watched" threads, but that seems like unnecessary complexity).
Also, potentially a bit fiddly but a per-user "modified since you last viewed this thread" icon on each thread might be useful. Bit debatable this one - if implemented, it needs a manual "mark all as read" option. I'm inclined to say ignore this and use the RSS feed if that's what you want to do, really, but I thought I'd mention it as it's a feature that some forums offer.
On the subject of flags, perhaps we could tag posts for searching purposes, similar to SO. Personally I'm unconvinced of the value unless you curate the set of tags pretty carefully as they tend to be used inconsistently by different people, but I thought I'd throw it in there - some people seem to love tags. Could use these as an alternative to straight categories as the latter may cause people to miss stuff (say a support question is asked in the "general chat" category, people who don't read that category would miss it).
I guess that "favourites" and "mentioned" threads could potentially just become magic tags, so perhaps a tagging system of pre-defined tags is the way to go, if only just under the hood for future expansion. This keeps things somewhat simpler by there only being two classes of index page - the full, unfiltered index; and the same thing but filtered by a single tag. The rest is just deciding on the sugar coating (e.g. whether viewing posts with the "favourite" tag deserves its own special link instead of being presented to the user as "just another tag".
I guess this is fairly similar to the way Gmail works, really. Maybe they were on to something... (^_^)
Oh, and chat sounds like a great idea, if only for new users with support questions or who are deciding whether to use the service. Would be nice if we could make it IRC or something as opposed to web only, because there are already a number of good IRC clients for every platform, but that's probably more effort than knocking up a quick web service, and potentially more open to abuse. I would suggest reserving a room on an established network rather than running your own server if you do decide to go the IRC route, just because it seems to be fiddlier than it first appears (mostly security issues to worry about).
Sorry to blather on, but I'm sure you're used to that from me by now!