Open Source Projects Europe

--- a/Allura/docs/install.rst
+++ b/README.markdown
@@ -1,233 +1,243 @@
-Getting Started Guide
-=====================
-
-Quickstart Script
-----------------------
-
-If you have or can acquire a copy of the quickstart script setup-local.bash,
-that's probably the way to go.  For those who are gluttons for punishment, here
-are the manual directions....
-
-Setup your python development stack on a Mac
---------------------------------------------
-If you use a mac, you'll want to follow the `Mac OS install`_ instructions now.
-
-.. _`Mac OS install`: mac_install.html
-
-Setting up a virtual environment
-------------------------------------------
-
-The first step to installing the Forge platform is installing a virtual
-environment ("virtualenv").  To do this, you need to first download a copy of
-`ez_setup.py <http://peak.telecommunity.com/dist/ez_setup.py>`_.  Once you have
-this, you need to install virtualenv.  You can do this by executing::
-
-    $ sudo python ez_setup.py virtualenv
-
-Once you have virtualenv installed, you need to create a virtual environment.
-We'll call our environment 'anvil'::
-
-    $ virtualenv anvil --no-site-packages
-
-This gives us a nice, clean environment into which we can install all the forge
-dependencies.  In order to use the virtual environment, you'll need to activate
-it::
+# Sandbox Creation
+
+We'll use [VirtualBox](http://www.virtualbox.org) and [Ubuntu 10.10](http://ubuntu.com) to create a disposable sandbox for Forge development/testing.
+
+* Download and install [VirtualBox](http://www.virtualbox.org/wiki/Downloads) for your platform.
+  
+* Download a minimal [Ubuntu 10.10](https://help.ubuntu.com/community/Installation/MinimalCD) ISO (~15MB).
+  
+* Create a new virtual machine in Virtual Box, selecting Ubuntu (64 bit) as the OS type.  The rest of the wizards' defaults are fine.
+  
+* When you launch the virtual machine for the first time, you will be prompted to attach your installation media.  Browse to the `mini.iso` that you downloaded earlier.
+  
+* Consult [available documentation](https://help.ubuntu.com/) for help installing Ubuntu.
+  
+
+# Forge Installation
+
+Before we begin, you'll need the following additional packages in order to checkout the Forge source code.
+
+    ~$ sudo apt-get install git-core mercurial
+
+You'll also need additional development packages in order to compile some of the modules.
+
+    ~$ sudo apt-get install default-jdk python-dev libssl-dev libldap2-dev libsasl2-dev
+
+Running the test suite requires additional packages.
+
+    ~$ sudo apt-get install libtidy-0.99-0 gitweb
+
+... and our messaging server, RabbitMQ.
+
+    ~$ sudo apt-get install rabbitmq-server
+
+And finally our document-oriented database, MongoDB.
+
+    ~$ sudo apt-get install mongodb
+
+
+## Setting up a virtual python environment
+
+The first step to installing the Forge platform is installing a virtual environment via `virtualenv`.  This helps keep our distribution python installation clean.
+
+    ~$ sudo apt-get install python-setuptools
+    ~$ sudo easy_install-2.6 -U virtualenv
+
+Once you have virtualenv installed, you need to create a virtual environment.  We'll call our Forge environment 'anvil'.
+
+    ~$ virtualenv --no-site-packages anvil
+
+This gives us a nice, clean environment into which we can install all the forge dependencies.  In order to use the virtual environment, you'll need to activate it.  You'll need to do this whenever you're working on the Forge codebase so you may want to consider adding it to your `~/.bashrc` file.
 
     ~$ . anvil/bin/activate
-    (anvil)~$ 
-
-Now that that's out of the way, we'll go ahead and install turbogears::
-
-    (anvil)~$ easy_install pylons==0.9.7 # TurboGears 2 doesn't work with Pylons 1.0
-    (anvil)~$ easy_install -i http://www.turbogears.org/2.1/downloads/current/index tg.devtools TurboGears2
-
-And the tip of easywidgets::
-
-    (anvil)~$ cd ~/src
-    (anvil)~/src$ hg clone http://bitbucket.org/rick446/easywidgets
-    (anvil)~/src$ cd easywidgets
-    (anvil)~/src/easywidgets$ python setup.py develop
-
-Installing the code
--------------------------
-
-.. sidebar:: System Required Packages
-
-    Before performing the steps below, you'll want to make sure you have the
-    following packages installed on your system.  (There may be others, these are
-    the ones I know about at the moment.)
-
-    - python-dev
-    - libldap2-dev
-    - libsasl2-dev
-    - rabbitmq-server
-
-    To run Solr you need Java compiler. For Ubuntu::
-
-       $ sudo apt-get install sun-java6-bin sun-java6-demo sun-java6-jdk sun-java6-jre
-       $ sudo update-alternatives --config java # select Sun Java
-       $ export JAVA_HOME=/usr/lib/jvm/java-6-sun
-
-OK, now we can get down to actually getting the forge code and dependencies
-downloaded and ready to go.  The first thing we'll need to set up is `Ming
-<http://merciless.sourceforge.net>`_.  To do this, we'll check out the source
-from git::
+
+Now that that's out of the way, we'll go ahead and install TurboGears.
+
+    (anvil)~$ easy_install pylons==0.9.7
+    (anvil)~$ easy_install -i http://www.turbogears.org/2.1/downloads/2.1b2/index/ tg.devtools==2.1b2 TurboGears2==2.1b2
+    (anvil)~$ easy_install http://httplib2.googlecode.com/files/httplib2-0.4.0.tar.gz
+
+
+## Installing the Forge code and dependencies
+
+Now we can get down to actually getting the Forge code and dependencies downloaded and ready to go.
 
     (anvil)~$ mkdir src
     (anvil)~$ cd src
-    (anvil)~/src$ git clone git://merciless.git.sourceforge.net/gitroot/merciless/merciless Ming
-
-Now, we'll need to set up Ming for development work::
-
-    (anvil)~/src$ cd Ming
-    (anvil)~/src/Ming$ python setup.py develop
-
-Once this is done, we'll check out & set up out forge codebase::
-
-    (anvil)~/src/Ming$ cd ..
-    (anvil)~/src$ git clone ssh://engr.geek.net/forge
-    (anvil)~/src$ cd forge/Allura
-    (anvil)~/src/forge/Allura$ python setup.py develop
-    (anvil)~/src/forge/Allura$ cd ../ForgeDiscussion
-    (anvil)~/src/forge/ForgeDiscussion$ python setup.py develop
-    (anvil)~/src/forge/ForgeDiscussion$ cd ../ForgeMail
-    (anvil)~/src/forge/ForgeMail$ python setup.py develop
-    (anvil)~/src/forge/ForgeMail$ cd ../ForgeGit
-    (anvil)~/src/forge/ForgeGit$ python setup.py develop
-    (anvil)~/src/forge/ForgeGit$ cd ../ForgeSVN
-    (anvil)~/src/forge/ForgeSVN$ python setup.py develop
-    (anvil)~/src/forge/ForgeSVN$ cd ../ForgeHg
-    (anvil)~/src/forge/ForgeHg$ python setup.py develop
-    (anvil)~/src/forge/ForgeHg$ cd ../ForgeTracker
-    (anvil)~/src/forge/ForgeTracker$ python setup.py develop
-    (anvil)~/src/forge/ForgeTracker$ cd ../ForgeWiki
-    (anvil)~/src/forge/ForgeWiki$ python setup.py develop
-    (anvil)~/src/forge/ForgeWiki$ cd ../HelloForge
-    (anvil)~/src/forge/HelloForge$ python setup.py develop
-    (anvil)~/src/forge/HelloForge$ cd ..
-
-Hopefully everything completed without errors.
-
-Initializing the environment
------------------------------------
-
-The forge consists of several components, all of which need to be running to have
-full functionality:
-
-mongod
-  MongoDB database server -- generally set up with its own directory (I like
-  ~/var/mongodata).  To run, execute the following::
-
-      (anvil)~/src$ mkdir -p ~/var/mongodata 
-      (anvil)~/src$ mongod --dbpath ~/var/mongodata 
-
-RabbitMQ
-  Message Queue -- to set this up for use, you'll need to run the following commands::
-
-      (anvil)~/src$ sudo rabbitmqctl add_user testuser testpw
-      (anvil)~/src$ sudo rabbitmqctl add_vhost testvhost
-      (anvil)~/src$ sudo rabbitmqctl set_permissions -p testvhost testuser ""  ".*" ".*"
-
-  If you get errors running these, it's likely because rabbit isn't running. It can be run as a daemon (instructions vary per architecture) or directly from a console window, e.g.::
-
-	    $ cd <rabbitmq_server_directory> # not needed on MACOS
-	    $ sudo rabbitmq-server
-
-SOLR server
-  This is our search and indexing server.  We have a custom config in
-  ~/src/forge/solr_config::
-
-      (anvil)~/<path_to_solr>/example$ java -Dsolr.solr.home=$(cd;pwd)/src/forge/solr_config -jar start.jar
-
-Forge "reactor" server
-  This is the server that will respond to RabbitMQ messages.  To set it up to
-  receive messages, you'll need to run the following commands::
-
-      (anvil)~/src$ cd forge/Allura
-      (anvil)~/src/forge/Allura$ paster reactor_setup development.ini
-      (anvil)~/src/forge/Allura$ paster reactor development.ini
-
-Forge SMTP server
-  This server routes messages from email addresses to tools in the forge::
-    
-      (anvil)~/src/forge/Allura$ paster smtp_server development.ini
-
-TurboGears application server
-  This is the main application that will respond to web requests.  We'll get into
-  details later.
-
-In order to initialize the forge database, you'll need to run the following::
+    (anvil)~/src$ git clone git://git.code.sf.net/p/allura/git.git forge
+
+Although the application setup.py files define a number of dependencies, the `requirements.txt` files are currently the authoritative source, so we'll use those with `pip` to make sure the correct versions are installed.
+
+    (anvil)~/src$ cd forge
+    (anvil)~/src/forge$ easy_install pip
+    (anvil)~/src/forge$ pip install -r requirements-dev.txt
+
+And now to setup each of the Forge applications for development.  Because there are quite a few (at last count 15), we'll use a simple shell loop to set them up.
+
+    for APP in Allura* Forge* NoWarnings pyforge
+    do
+        pushd $APP
+        python setup.py develop
+        popd
+    done
+
+Hopefully everything completed without errors.  We'll also need to create a place for Forge to store any SCM repositories that a project might create.
+
+    for SCM in git svn hg
+    do
+        mkdir -p ~/var/scm/$SCM
+        chmod -R 777 ~/var/scm
+        sudo ln -s ~/var/scm/$SCM /
+    done
+
+
+## Initializing the environment
+
+The forge consists of several components, all of which need to be running to have full functionality.
+
+
+### MongoDB database server
+
+Generally set up with its own directory, we'll use ~/var/mongodata to keep our installation localized.  We also need to disable the default distribution server.
+
+    (anvil)~$ sudo service mongodb stop
+    (anvil)~$ sudo update-rc.d mongodb remove
+
+    (anvil)~$ mkdir -p ~/var/mongodata ~/logs
+    (anvil)~$ nohup mongod --dbpath ~/var/mongodata > ~/logs/mongodb.log &
+
+
+### SOLR search and indexing server
+
+We have a custom config ready for use.
+
+    (anvil)~$ cd ~/src
+    (anvil)~/src$ wget http://apache.mirrors.tds.net/lucene/solr/1.4.0/apache-solr-1.4.0.tgz
+    (anvil)~/src$ tar xf apache-solr-1.4.0.tgz
+    (anvil)~/src$ cd apache-solr-1.4.0/example/
+    (anvil)~/src/apache-solr-1.4.0/example/$ nohup java -Dsolr.solr.home=$(cd;pwd)/src/forge/solr_config -jar start.jar > ~/logs/solr.log &
+
+
+### RabbitMQ message queue
+
+We'll need to setup some development users and privileges.
+
+    (anvil)~$ sudo rabbitmqctl add_user testuser testpw
+    (anvil)~$ sudo rabbitmqctl add_vhost testvhost
+    (anvil)~$ sudo rabbitmqctl set_permissions -p testvhost testuser ""  ".*" ".*"
+
+
+### Forge "reactor" processing
+
+Responds to RabbitMQ messages.  We'll need to perform some setup operations before we can start it.
+
+    (anvil)~$ cd ~/src/forge/Allura
+    (anvil)~/src/forge/Allura$ paster reactor_setup development.ini
+    (anvil)~/src/forge/Allura$ nohup paster reactor development.ini > ~/logs/reactor.log &
+
+
+### Forge SMTP for inbound mail
+
+Routes messages from email addresses to tools in the forge.
+
+    (anvil)~/src/forge/Allura$ nohup paster smtp_server development.ini > ~/logs/smtp.log &
+
+
+### TurboGears application server
+
+One of the modules we need for Subversion integration is not available in egg format, however the distribution copy will work fine.
+
+    (anvil)~$ sudo apt-get install python-svn
+    (anvil)~$ echo '/usr/lib/python2.6/dist-packages' >> ~/anvil/lib/python2.6/site-packages/setuptools.pth
+
+In order to initialize the forge database, you'll need to run the following:
 
     (anvil)~/src/forge/Allura$ paster setup-app development.ini
 
-This shouldn't take too long, but it will start the reactor server doing tons of
-stuff in the background.  It should complete in 5-6 minutes.  Once this is done,
-you can start the application server::
-
-      (anvil)~/src/forge/Allura$ paster serve --reload development.ini
-
-And now you should be able to visit the server running on your 
-`local machine <http://localhost:8080/>`_.
-
-Logging In, Getting Around
-----------------------------------------------
-
-Part of the base system includes the test-admin and test-user accounts.  The
-password for both accounts is `foo`.  The `test` project has several tools
-already configured; to configure more, you can visit the `Admin` tool
-(accessible in the top navigation bar when inside the `test` project).  
-
-Running the Tests
----------------------------------
-
-The test setup is a little bit different from the dev/production setup so as not
-to create conflicts between test data and development data.  This section will
-tell you how to set up your test environment.
-
-mongod
-  We'll need a test MongoDB server to keep from stomping on our development data::
-
-      (anvil)~/src$ mkdir -p ~/var/mongodata-test
-      (anvil)~/src$ mongod --port 27018 --dbpath ~/var/mongodata-test
-
-RabbitMQ
-  Here, we'll set up a second virtual host for testing.  We also need to set up
-  the RabbitMQ queues using reactor_setup::
-
-      (anvil)~/src$ sudo rabbitmqctl add_vhost vhost_testing
-      (anvil)~/src$ sudo rabbitmqctl set_permissions -p vhost_testing testuser ""  ".*" ".*"
-      (anvil)~/src$ cd forge/Allura
-      (anvil)~/src/forge/Allura$ paster reactor_setup test.ini#main_with_amqp
-
-SOLR server
-  We are using the multicore version of SOLR already, so all the changes to use
-  core1 (the testing core) rather than core0 (the dev core) are encapsulated in
-  test.ini.
-
-To actually run the tests, just go to the tool directory you wish to test (or
-to the Allura directory) and type::
-
-    (anvil)~/src/forge/Allura$ nosetests
-
-Some options you might find useful for nosetests:
-
---pdb
-  Drops into a PDB prompt on unexpected exceptions ("errors" in unittest
-  terminology)
-
---pdb-fail
-  Drops into a PDB prompt on AssertionError exceptions in tests  ("failures" in unittest
-  terminology)
-
--s
-  Do *not* capture stdout.  This is essential if you have embedded pdb
-  breakpoints in your test code.  (Otherwise, you will not see the prompt; your
-  test will just mysteriously hang forever.)
-
--v
-  Print the name of the test as it runs.  This is useful if the test suite takes a while
-  to run and you want to let it continue to run while you begin debugging the
-  first (few) failures.
-
+This shouldn't take too long, but it will start the reactor server doing tons of stuff in the background.  It should complete in 5-6 minutes.  Once this is done, you can start the application server.
+
+    (anvil)~/src/forge/Allura$ nohup paster serve --reload development.ini > ~/logs/tg.log &
+
+And now you should be able to visit the server running on your [local machine](http://localhost:8080/).
+
+
+## Next Steps
+
+
+### Generate the documentation
+
+Forge documentation currently lives in the `Allura/docs` directory and can be converted to HTML using `Sphinx`:
+
+    (anvil)~$ cd ~/src/forge/Allura/docs
+    (anvil)~/src/forge/Allura/docs$ easy_install sphinx
+    (anvil)~/src/forge/Allura/docs$ make html
+
+You will also want to give the test suite a run, to verify there were no problems with the installation.
+
+    (anvil)~$ cd ~/src/forge
+    (anvil)~/src/forge$ ./run_tests
 
 Happy hacking!
+
+
+# Current Local Tweaks
+
+[#1405] Remove sf.phpsession from forge/Allura/setup.py
+
+    diff --git a/Allura/setup.py b/Allura/setup.py
+    index f9fc0c4..ea1eb16 100644
+    --- a/Allura/setup.py
+    +++ b/Allura/setup.py
+    @@ -37,7 +37,6 @@ setup(
+             "EasyWidgets >= 0.1.1",
+             "PIL >= 1.1.7",
+             "iso8601",
+    -        "sf.phpsession==0.1",
+             "chardet==1.0.1",
+             "feedparser>=4.1,<=4.2",
+             "oauth2==1.2.0",
+
+
+[#1407] Remove all rtstats traces from Allura/development.ini
+
+     diff --git a/Allura/development.ini b/Allura/development.ini
+     index 71c3166..0496619 100644
+     --- a/Allura/development.ini
+     +++ b/Allura/development.ini
+     @@ -184,15 +184,15 @@ templating.mako.reloadfromdisk = true
+      keys = root, allura, sqlalchemy, paste, amqp
+
+      [handlers]
+     -keys = console, rtstats
+     +keys = console
+
+      [formatters]
+     -keys = generic, rtstats
+     +keys = generic
+
+      # If you create additional loggers, add them as a key to [loggers]
+      [logger_root]
+      level = DEBUG
+     -handlers = console, rtstats
+     +handlers = console
+
+      [logger_allura]
+      level = DEBUG
+     @@ -224,17 +224,7 @@ args = (sys.stderr,)
+      level = NOTSET
+      formatter = generic
+
+     -[handler_rtstats]
+     -class = sfx.lib.loggers.RTStatsHandler
+     -args = ('rtstats.log', 'allura', 1)
+     -level = NOTSET
+     -formatter = rtstats
+     -
+      # If you create additional formatters, add them as a key to [formatters]
+      [formatter_generic]
+      format = %(asctime)s,%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
+      datefmt = %H:%M:%S
+     -
+     -[formatter_rtstats]
+     -format = %(asctime)s %(created)d %(kwpairs)s
+     -datefmt = %d/%b/%Y:%H:%M:%S UTC