Introduction

The new Forge is implemented as a collection of tool applications on top of a robust and open platform. Some of the services provided by the platform include:

• Indexing and search
• Authentication and Authorization
• Email integration (every tool application gets its own email address)
• Asynchronous processing via RabbitMQ
• Simple autolinking between different artifacts in the forge
• Attachment handling
• Tool administration

Tools, on the other hand, provide the actual user interface and logic to manipulate forge artifacts. Some of the tools currently impemented include:

admin

This tool is installed in all projects, and allows the administration of the project's tools, authentication, and authorization

search

This tool is installed in all projects, and provides the ability to search a project for various types of artifacts.

home

This tool is installed in all projects, and provides the ability to customize the project landing page with "widgets" shared by other tools.

Git, Hg, SVN

These tools allow you to host a version control system in the Forge. They also provides the ability to "fork" Git and Hg repos in order to provide your own extensions.

Wiki

This tool provides a basic wiki with support for comments, attachments, and notifications.

Tracker

This tool provides an extensible ticketing system for tracking feature requests, defects, or support requests.

Forum

This tool provides a forum interface with full email integration as well. The forum also handles attachments to posts either via the web interface or via email.

Ming Databases and the Context Object

There are a few databases you need to be aware of when developing for the new Forge. The Forge maintains a 'main' database which contains the project index and user list as well as a project-local database for each project (sub-projects share databases with thier parent databases, however). Most of the time, you should not need to worry about these databases as they are automatically managed. The automatic management is based on various properties of the Pylons "context" object c:

project

The current project, used to determine which database to use for other objects

app

The current tool application object.

user

The current user

The Forge platform provides the following functions to manage the context object:

Artifacts

We've mentioned artifacts a couple of times now without definition. An artifact, as used in the new Forge, is some object that a tool needs to store in the forge. The platform provides facilities for controlling access to individual artifacts if that's what the tool designer favors. For instance, the Forum tool allows a user to edit or delete their own posts, but not to edit or delete others (unless the user has the 'moderate' permission on the forum itself). Some examples of artifacts in the current tools:

• Forum: Forum, Thread, Post, Attachment
• Wiki: Page, Comment, Attachment
• Tracker: Ticket, Comment, Attachment
• SCM: Repository, Commit, Patch

In order to implement your own artifact, you should override at least a few of the methods of the pyforge.model.artifact.Artifact class:

from ming.orm.property import FieldProperty
from pyforge.model import Artifact

class NewArtifact(Artifact):
    class __mongometa__:
        name='my_new_artifact' # collection where this artifact is stored
    type_s = 'My Artifact' # 'type' of the artifact used in search results

    # Add your own properties here (beyond those provided by Artifact)
    shortname = FieldProperty(str)

    def url(self):
        'Each artifact should have its own URL '
        return self.app.url + self.shortname + '/'

    def index(self):
        'Return the fields you want indexed on this artifact'
        result = Artifact.index(self)
        result.update(type_s=self.type_s,
                      name_s=self.shortname,
                      text=self.shortname)
        return result

    def shorthand_id(self):
        'Used in the generation of short links like [my_artifact]'
        return self.shortname

Platform services provided for artifacts

Whenever you create, modify, or delete an artifact, the platform does a couple of things for you:

• The artifact is added to the index and will appear in searches
• A shortlink is generated for the artifact (e.g. [MyWikiPage]). This allows you to reference the artifact from other artifacts. For instance, you might want to reference [Ticket#151] from [Commit#abac332a]. Whenever the commit message is displayed in the SCM tool, any references to [Ticket#151] will be automatically linked to that Ticket's page.

Shortlinks work only within a project hierarchy (in order to link to some other project's page, you'll have to use the full URL). Sometimes, a shortlink may need to be differentiated based on its location in a subproject or in one of many tools of the same type within a project. In order to do this, shortlinks may be prefixed by either the tool mount point or a project ID and tool mount point.

For instance, suppose we have an ticket tracker mounted at projects/test/tracker with Ticket #42 in it. Further suppose that there is an SCM repository mounted at projects/test/subproject/repo. A user could push a commit to that repository with the commit message [projects/test:tracker:42] - Fix weird issue. If you then examined the commit in the SCM tool, the shortlink would be clickable and would take you to the ticket itself. The Tracker tool would also list the commit message as a "related object" in a sidebar to allow for quick cross-referencing.

Asynchronous Processing

Much of the actual functionality of the new Forge comes from code that runs outside the context of a web request, in the reactor server (invoked by running paster reactor development.ini. Asynchronous processing is performed by two types of functions, auditors and reactors, differentiated as follows:

Auditor

Auditors listen to queues on the audit exchange. Messages sent to an auditor queue are interpreted imperatively ("do this"). Auditor-type messages should specify a project ID project_id, an application mount point mount_point, and a user ID user_id, which will be used by the platform to set the context before calling the registered callback, and all of which reference the recipient of the message. An auditor callback function is called once for each message received on its queue.

Reactor

Reactors listen to queues on the react exchange. Messages sent to a reactor queue are interpreted in an advisory manner ("this was done"). Reactor-type messages should specify a project ID project_id and a user ID user_id, which will be used by the platform to set the context before calling the registered callback, and all of which reference the source of the message. If the reactor callback is an instance method, it will be called once for each instance of the tool that exists for the given project for each message received on its queue. If it is a class method, it will be called once for each message received on its queue. For instance, the Tracker tool may be configured to react to SCM commit messages in order to generate links between SCM commits and Tracker tickets. All tracker instances in a project will be notified of SCM commits in such a case.

In order to create a callback function for an auditor or a reactor, simply add a method to the tool application class that is decorated either with the @audit or the @react decorator. For instance, the forum tool defines a reactor on the Forum.new_post message:

@react('Forum.new_post')
def notify_subscribers(self, routing_key, data):
    ....

If there are a large number of reactors, you can define them in a separate module and use the mixin_reactors() method as in the SCM tool:

from .reactors import reactors
...
class ForgeGitApp(Application):
    ...
mixin_reactors(ForgeGitApp, reactors)

$ paster reactor_setup development.ini

In order to actually send a message to either the audit or react exchange, a helper method is provided in the pylons global object g:

Email Integration

The Forge platform provides easy-to-use email integration. Forge email addresses are of the form <topic>@<mount_point>[.<subproject>]*.<subproject>.projects.sourceforge.net. When a message is received on such an email address, the address is parsed and the sending user is identified (if possible). Based on the parsed address, the pylons context attributes c.project and c.app are set, and the application is queried to determine whether the identified user has authority to send an email to the given app/topic combination by calling c.app.has_access(user, topic). If the user has access, the message is decomposed into its component parts (if a multipart MIME-encoded message) and one audit message is generated for each part with the following fields:

headers

The actual headers parsed from the body of the message

message_id

The Message-ID header (which should be universally unique and is generated by the email client), used for determining which messages are replies to which other messages

in_reply_to

The In-Reply-To header, used for determining which messages are replies to which other messages

references

The References header, used for determining which messages refer to which other messages

filename

Optional, if the part is an attachment with a filename, this will be populated

content_type

The MIME content_type of the message part

payload

The actual content of the message part

user_id

The ID of the user who sent the message

Once the message is generated, it is sent to the audit exchange with the routing key <Tool Type>.<topic>. For instance, a message to comment on a Wiki page might have the routing key Wiki.MainPage.

The Forge platform also provides full support for sending email without worrying about the specifics of SMTP or sendmail handling. In order to send an email, a tool needs simply to send an audit message with the routing key forgemail.send_email and the following fields:

from

Return address on the message (usually the topic@tool_name that generated it)

subject

Subject of the message

message_id

Value to put in the Message-ID header (the _id field of a :class:`pyforge.model.artifact.Message` is suitable for this)

in_reply_to (optional)

Value to put in the In-Reply-To header (the parent_id field of a :class:`pyforge.model.artifact.Message` is suitable for this)

destinations

List of email addresses and/or :class:`pymongo.bson.ObjectId` s for :class:`pyforge.model.auth.User` objects

text

Markdown-formatted body of the message (If the user has requested html or combined text+html messages in their preferences, the Markdown will be so rendered. Otherwise a plain text message will be sent.)

Migrations

Although Ming provides the Forge platform with some lazy migration facilities, there are some cases (adding an index, dropping an index, etc.) where this is insufficient. In these cases, the Forge platform uses the Flyway migration system. Migrations are organized into 'modules' which are specified by named entry points under the 'flyway.migrations' section. For instance, to specify a migrations module for the ForgeForum, you might have the following entry point:

[flyway.migrations]
forum = forgediscussion.migrations

Inside the :mod:`forgediscussion.migrations` module, you would specify the various migration scripts to be run:

from flyway import Migration

class V0(Migration):
    version=0
    def up(self):
        # Do some stuff with self.session to upgrade
    def down(self):
        # Do some stuff with self.session to undo the 'up'

class V1(Migration):
    version=1
    def up(self):
        # Do some stuff with self.session to upgrade
    def down(self):
        # Do some stuff with self.session to undo the 'up'

You can optionally supply up_requires() and down_requires() methods for your migration if it requires something more complex than the previous migration in the same module:

class V3(Migration):
    version=3
    def up_requires(self):
        yield ('pyforge', 3)
        for r in super(V3, self).requires():
            yield r

To actually run the migration, you must call the paster command flyway:

# migrate all databases on localhost to latest versions of all modules
$ paster flyway

# migrate the 'pyforge' database on 'myserver' to the latest version
$ paster flyway -u mongo://myserver:27017/pyforge

# migrate all the databases on 'myserver' to the latest version
$ paster flyway -u mongo://myserver:27017/

# migrate the forgediscussion module to the latest version on localhost
$ paster flyway forgediscussion

# migrate the forgediscussion module to the version 5 (up or down) on localhost
$ paster flyway forgediscussion=5

It's often helpful to see exactly what migrations flyway is planning on running; to get this behavior, pass the option -d or --dry-run to the flyway command.