|
a/Allura/docs/platform.rst |
|
b/Allura/docs/platform.rst |
| 1 |
Platform Architecture overview
|
1 |
Platform Architecture overview
|
| 2 |
===================================
|
2 |
===================================
|
| 3 |
|
3 |
|
| 4 |
I'm told that the reason you build a platform is to "reduce the marginal cost
|
4 |
I'm told that the reason you build a platform is to "reduce the marginal cost
|
| 5 |
of developing applications." Sounds good. Well, actually it sounds a bit
|
5 |
of developing applications." Sounds good. Well, actually it sounds a bit
|
| 6 |
dry. But it's about right, we want to make creating new online development
|
6 |
dry. But it's about right, we want to make creating new online development
|
| 7 |
tools faster, easier, and more fun, which I guess is the "reduce the marginal
|
7 |
tools faster, easier, and more fun, which I guess is the "reduce the marginal
|
| 8 |
cost" thing.
|
8 |
cost" thing.
|
| 9 |
|
9 |
|
| 10 |
Platform building blocks
|
10 |
Platform building blocks
|
| 11 |
---------------------------------------------------------------------
|
11 |
---------------------------------------------------------------------
|
| 12 |
|
12 |
|
| 13 |
Before we get into the details of how to extend the Allura platform, perhaps
|
13 |
Before we get into the details of how to extend the Allura platform, perhaps
|
| 14 |
it would be smart to explain some of the big pieces and why there are there.
|
14 |
it would be smart to explain some of the big pieces and why they are there.
|
| 15 |
|
15 |
|
| 16 |
We wanted Allura tools to be fast, we needed them to scale, and we had some
|
16 |
We wanted Allura tools to be fast, we needed them to scale, and we had some
|
| 17 |
complex requirements for data storage and extensibility. So, we needed a
|
17 |
complex requirements for data storage and extensibility. So, we needed a
|
| 18 |
**fast,** flexible, and easy to use data persistence system.
|
18 |
**fast,** flexible, and easy to use data persistence system.
|
| 19 |
|
19 |
|
| 20 |
We were very impressed by the general message architecture of Roundup, but we
|
20 |
We were very impressed by the general message architecture of Roundup, but we
|
| 21 |
wanted to extend it from just email messages to include scm commits, and we
|
21 |
wanted to extend it from just email messages to include scm commits, and we
|
| 22 |
added a message bus (RabbitMQ which we'll talk about in a second), to make
|
22 |
added a message bus (RabbitMQ which we'll talk about in a second), to make
|
| 23 |
it fast.
|
23 |
it fast.
|
| 24 |
|
24 |
|
| 25 |
.. image:: _static/images/messages.png
|
25 |
.. image:: _static/images/messages.png
|
| 26 |
:alt: Message Architecture
|
26 |
:alt: Message Architecture
|
| 27 |
|
|
|
| 28 |
We were also impressed by the flexibility of Roundup's Hypertable system in
|
|
|
| 29 |
allowing for ad-hock ticket schema additions.
|
|
|
| 30 |
|
27 |
|
|
|
28 |
We were also impressed by the flexibility of Roundup's Hypertable system in
|
|
|
29 |
allowing for ad-hock ticket schema additions.
|
|
|
30 |
|
| 31 |
It definitely seemed like something we wanted in a next generation forge,
|
31 |
It definitely seemed like something we wanted in a next generation forge,
|
| 32 |
because we wanted app tools to be able to:
|
32 |
because we wanted app tools to be able to:
|
| 33 |
|
33 |
|
| 34 |
* create and version their own document types,
|
34 |
* create and version their own document types,
|
| 35 |
* extend existing document structures,
|
35 |
* extend existing document structures,
|
| 36 |
* and to mange document revisions, access control lists, and other
|
36 |
* and to mange document revisions, access control lists, and other
|
| 37 |
platform level data.
|
37 |
platform level data.
|
| 38 |
|
38 |
|
| 39 |
In spite of the power and flexibility of the Roundup HyperTable
|
39 |
In spite of the power and flexibility of the Roundup HyperTable
|
| 40 |
implementation, we had some concerns about performance and scalability.
|
40 |
implementation, we had some concerns about performance and scalability.
|
| 41 |
|
41 |
|
| 42 |
Fortunately several of the Allura authors (including me) used MongoDB
|
42 |
Fortunately several of the Allura authors used MongoDB
|
| 43 |
in rewriting the download flow of SourceForge.net, and knew that it could
|
43 |
in rewriting the download flow of SourceForge.net, and knew that it could
|
| 44 |
handle huge loads (we saturated a 2gb network connection on the server
|
44 |
handle huge loads (we saturated a 2gb network connection on the server
|
| 45 |
with 6% cpu utilization).
|
45 |
with 6% cpu utilization).
|
| 46 |
|
46 |
|
| 47 |
We also knew that MongoDB's flexible replication system would allow us
|
47 |
We also knew that MongoDB's flexible replication system would allow us
|
| 48 |
to build the forge in such a way that we could easily provide a
|
48 |
to build the forge in such a way that we could easily provide a
|
| 49 |
package of all project data to developers concerned about lock-in.
|
49 |
package of all project data to developers concerned about lock-in.
|
| 50 |
|
50 |
|
| 51 |
Not only that but Rick Copeland had built a couple of custom Object
|
51 |
Not only that but Rick Copeland had built a couple of custom Object
|
| 52 |
*Non*-Relational Mappers (ONRMs?) before, including one for MongoDB,
|
52 |
*Non*-Relational Mappers (ONRMs?) before, including one for MongoDB,
|
| 53 |
and he whipped up Ming, which backed on MongoDB and gave us exactly
|
53 |
and he whipped up Ming, which backed on MongoDB and gave us exactly
|
| 54 |
what we needed.
|
54 |
what we needed.
|
| 55 |
|
55 |
|
| 56 |
As I mentioned before we also needed a fast, flexible message bus and queuing
|
56 |
As I mentioned before we also needed a fast, flexible message bus and queuing
|
| 57 |
system. RabbitMQ was (lightning) fast, (shockingly) flexible, but not supper
|
57 |
system. RabbitMQ was (lightning) fast, (shockingly) flexible, but not super
|
| 58 |
easy to use. Fortunately we didn't have to roll our own wrapper here, as
|
58 |
easy to use. Fortunately we didn't have to roll our own wrapper here, as
|
| 59 |
the Python community already whipped up Carrot, and Celery, which made
|
59 |
the Python community already whipped up Carrot, and Celery, which made
|
| 60 |
working with the RabbitMQ based AMQP bus a LOT easer.
|
60 |
working with the RabbitMQ based AMQP bus a LOT easer.
|
| 61 |
|
|
|
| 62 |
|
|
|
| 63 |
Pluggable Event Listeners
|
|
|
| 64 |
|
|
|
| 65 |
|
|
|
| 66 |
Have we mentioned Roundup already? Because here's another idea we stole
|
|
|
| 67 |
from them:
|
|
|
| 68 |
|
|
|
| 69 |
**Auditors** and **reactors**
|
|
|
| 70 |
|
|
|
| 71 |
**Auditors** are hooks that get called when events/messages come in,
|
|
|
| 72 |
they can modify the message before it is persisted to the document
|
|
|
| 73 |
store (via MongoDB).
|
|
|
| 74 |
|
|
|
| 75 |
Once the message is saved to the document store, it is then queued up for another
|
|
|
| 76 |
set of hooks -- **reactors** -- that are not allowed to change the
|
|
|
| 77 |
message, but can do things like send e-mail or push a new kind of event
|
|
|
| 78 |
onto another queue.
|
|
|
| 79 |
|
|
|
| 80 |
Nearly everything in Roundup is implemented as either an auditor or a reactor,
|
|
|
| 81 |
and Allura definitely steals that idea and runs with it.
|
|
|
| 82 |
|
|
|
| 83 |
TODO: Finish reactor overview (after reactor code is written).
|
|
|
| 84 |
|
61 |
|
| 85 |
|
62 |
|
| 86 |
Application Tools
|
63 |
Application Tools
|
| 87 |
---------------------------------------------------------------------
|
64 |
---------------------------------------------------------------------
|
| 88 |
|
65 |
|
| 89 |
Writing a tool for the new forge is as simple as defining a few controllers
|
66 |
Writing a tool for Allura is as simple as defining a few controllers
|
| 90 |
to handle particular URL's, templates to render pages, and defining the schemas
|
67 |
to handle particular URL's, templates to render pages, and defining the schemas
|
| 91 |
of any new forge document types that your tool requires.
|
68 |
of any Allura document types that your tool requires.
|
| 92 |
|
69 |
|
| 93 |
.. image:: _static/images/tools.png
|
70 |
.. image:: _static/images/tools.png
|
| 94 |
:alt: App Tools
|
71 |
:alt: App Tools
|
| 95 |
:align: right
|
72 |
:align: right
|
| 96 |
|
73 |
|
|
... |
|
... |
| 117 |
* Event publisher (optional)
|
94 |
* Event publisher (optional)
|
| 118 |
|
95 |
|
| 119 |
Users/groups and Permissions
|
96 |
Users/groups and Permissions
|
| 120 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
97 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 121 |
|
98 |
|
| 122 |
In order to facilitate more open processes, where more users can contribute
|
99 |
In order to facilitate more open processes, where more users can contribute
|
| 123 |
-- while still protecting data -- documents can easily be "versioned", and
|
100 |
-- while still protecting data -- documents can easily be "versioned", and
|
| 124 |
the platform provides tools to manage versioned documents for you.
|
101 |
the platform provides tools to manage versioned documents for you.
|
| 125 |
|
|
|
| 126 |
|
|
|