testing: add a Vagrant environment for Bugzilla
authorGregory Szorc <gps@mozilla.com>
Fri, 04 Jul 2014 01:10:34 -0700
changeset 358988 0b10d2213115f25c0c03ff1ad179122af7744325
parent 358987 e367a2102af216f71e7e6a1ded7b6b18dc90b9a0
child 358989 36c2991d681711f41c1099385e4441ad797e2b6b
push id16998
push userrwood@mozilla.com
push dateMon, 02 May 2016 19:42:03 +0000
testing: add a Vagrant environment for Bugzilla
.hgignore
testing/bmoserver/Puppetfile
testing/bmoserver/README.rst
testing/bmoserver/Vagrantfile
testing/bmoserver/files/apache24.patch
testing/bmoserver/files/apache_extra.conf
testing/bmoserver/files/apache_perlswitches.conf
testing/bmoserver/files/checksetup_answers.txt
testing/bmoserver/files/elasticsearch.patch
testing/bmoserver/files/fastimport.py
testing/bmoserver/files/fkpatch.patch
testing/bmoserver/files/nodefaultproduct.patch
testing/bmoserver/fileserver.conf
testing/bmoserver/manifests/bugzilla.pp
testing/bmoserver/vagrant-bootstrap.sh
--- a/.hgignore
+++ b/.hgignore
@@ -1,8 +1,10 @@
 coverage/
 pylib/rbbz/rbbz.egg-info/
 pylib/rbmozui/rbmozui.egg-info/
+testing/bmoserver/.vagrant
+testing/bmoserver/files/Mozilla-Bugzilla-Public*
 venv/
 \.pyc$
 \.pyo$
 \.swp$
 ~$
new file mode 100644
--- /dev/null
+++ b/testing/bmoserver/Puppetfile
@@ -0,0 +1,5 @@
+forge "https://forgeapi.puppetlabs.com"
+
+mod "puppetlabs/apache", "1.0.1"
+mod "puppetlabs/mysql", "2.2.3"
+mod "puppetlabs/vcsrepo", "1.0.2"
new file mode 100644
--- /dev/null
+++ b/testing/bmoserver/README.rst
@@ -0,0 +1,98 @@
+Bugzilla Server
+===============
+
+This directory contains code for running a Bugzilla server.
+
+We want to run a Bugzilla server that is as close as possible to the
+configuration of bugzilla.mozilla.org so that testing is accurate
+and there won't be any surprises when we push to production.
+
+Usage
+=====
+
+1. Install Vagrant
+2. Run ``vagrant up``
+3. Wait a while (there are lots of packages to download and install)
+
+Basic image creation should take 10-15 minutes on a modern machine with
+an SSD and a fast internet connection. A lot of that is installing Perl
+packages from CPAN.
+
+If you install the Bugzilla data dump (see below) expect image creation
+to take ~40 minutes longer (50+ minutes on a modern machine).
+
+Inside the VM, Bugzilla is available at http://localhost:80/. Outside of
+the VM, Bugzilla is available at http://localhost:12000/.
+
+The admin username and password is ``admin@example.com`` and
+``password``.
+
+Choice of Vagrant
+=================
+
+We use Vagrant + Puppet for creating and provisioning a virtual
+machine running Bugzilla.
+
+We would ideally use Docker. However, Docker is still quirky with
+regards to containers that run multiple processes, thus requiring
+an init process. After much toiling with Docker, it was decided that
+it would be easier to use Vagrant for the time being.
+
+The Puppet config is sufficiently standalone that it could be leveraged
+by Docker as well.
+
+MySQL Server Settings
+=====================
+
+Credentials are *root*/*root* and *bugs*/*bugs*. The Bugzilla database
+is *bugs*.
+
+The MySQL server is tuned for fast importing of the BMO data set.
+Settings are adjusted to favor writes over reads. However, read
+performance for basic analysis workloads should hopefully not be
+impacted too much.
+
+The server is also tuned for SSDs. If you are running on magnetic
+storage, BMO data set import will take a long time.
+
+bugzilla.mozilla.org Data
+=========================
+
+If you would like to import a dump of Mozilla's bugzilla.mozilla.org
+data, grab a file from https://people.mozilla.org/~mhoye/bugzilla/
+and save it into ``files/``. e.g.
+``files/Mozilla-Bugzilla-Public-04-May-2014.sql.gz``.
+During Puppet provision, the content from this file will be imported
+into the database automatically.
+
+Adding dump data later
+----------------------
+
+If you provision without a dump file and attempt to reprovision with
+a dump file, this will likely result in error. To correct this,
+drop the ``bugs`` database and reprovision::
+
+   $ vagrant ssh
+   $ mysql -uroot -proot -e 'DROP DATABASE bugs;'
+   $ exit
+   $ vagrant provision
+
+Multiple dump files
+-------------------
+
+If there are multiple dump files in the ``files/`` directory, behavior
+is undefined. Please only place 1 dump file in that directory!
+
+Performance of dump importing
+-----------------------------
+
+MySQL has been tuned to make dump importing fast. In addition, we have
+a wrapper script (``fastimport.py``) that adds some statements to the
+import transaction to avoid excessive I/O.
+
+If you have a modern machine, disk write I/O during import should be
+40+ MB/s with read I/O almost non-existent (except during imports of
+tables with FULLTEXT indexes).
+
+Even with all the optimizations, it still takes ~40 minutes on modern
+machines to import the dump. You have been warned.
new file mode 100644
--- /dev/null
+++ b/testing/bmoserver/Vagrantfile
@@ -0,0 +1,12 @@
+Vagrant.configure("2") do |config|
+  config.vm.box = "ubuntu-1404-amd64"
+
+  config.vm.network "forwarded_port", guest: 80, host: 12000
+
+  config.vm.provider "virtualbox" do |v|
+    v.memory = 4096
+    v.cpus = 4
+  end
+
+  config.vm.provision :shell, path: "vagrant-bootstrap.sh"
+end
new file mode 100644
--- /dev/null
+++ b/testing/bmoserver/files/apache24.patch
@@ -0,0 +1,12 @@
+diff --git a/.htaccess b/.htaccess
+index a901a44..715899e 100644
+--- a/.htaccess
++++ b/.htaccess
+@@ -1,6 +1,6 @@
+ # Don't allow people to retrieve non-cgi executable files or our private data
+ <FilesMatch (\.pm|\.pl|\.tmpl|\.swf|localconfig.*)$>
+-  deny from all
++  Require all denied
+ </FilesMatch>
+ <IfModule mod_expires.c>
+ <IfModule mod_headers.c>
new file mode 100644
--- /dev/null
+++ b/testing/bmoserver/files/apache_extra.conf
@@ -0,0 +1,2 @@
+PerlSwitches -w
+PerlConfigRequire /home/bugzilla/bugzilla/mod_perl.pl
new file mode 100644
--- /dev/null
+++ b/testing/bmoserver/files/apache_perlswitches.conf
@@ -0,0 +1,1 @@
+PerlSwitches -T
new file mode 100644
--- /dev/null
+++ b/testing/bmoserver/files/checksetup_answers.txt
@@ -0,0 +1,26 @@
+ $answer{'db_host'}   = 'localhost';
+ $answer{'db_driver'} = 'mysql';
+ $answer{'db_port'}   = 0;
+ $answer{'db_name'}   = 'bugs',
+ $answer{'db_user'}   = 'bugs';
+ $answer{'db_pass'}   = 'bugs';
+ $answer{'db_sock'}   = '';
+ $answer{'db_check'}  = 1;
+ $answer{'db_mysql_ssl_ca_file'}     = '';
+ $answer{'db_mysql_ssl_ca_path'}     = '';
+ $answer{'db_mysql_ssl_client_cert'} = '';
+ $answer{'db_mysql_ssl_client_key'}  = '';
+ $answer{'urlbase'} = 'http://localhost:12000/';
+ $answer{'create_htaccess'} = '';
+ $answer{'use_suexec'} = '';
+ $answer{'index_html'} = 0;
+ $answer{'cvsbin'} = '/usr/bin/cvs';
+ $answer{'interdiffbin'} = '/usr/bin/interdiff';
+ $answer{'diffpath'} = '/usr/bin';
+ $answer{'webservergroup'} = 'bugzilla';
+ $answer{'ADMIN_OK'} = 'Y';
+ $answer{'ADMIN_EMAIL'} = 'admin@example.com';
+ $answer{'ADMIN_PASSWORD'} = 'password';
+ $answer{'ADMIN_REALNAME'} = 'Admin';
+ $answer{'NO_PAUSE'} = 1;
+
new file mode 100644
--- /dev/null
+++ b/testing/bmoserver/files/elasticsearch.patch
@@ -0,0 +1,19 @@
+diff --git a/Bugzilla/Install/Requirements.pm b/Bugzilla/Install/Requirements.pm
+index 1c9c913..384df22 100644
+--- a/Bugzilla/Install/Requirements.pm
++++ b/Bugzilla/Install/Requirements.pm
+@@ -386,13 +386,6 @@ sub OPTIONAL_MODULES {
+         version => '0',
+         feature => ['memcached'],
+     },
+-
+-    # BMO - metrics
+-    {
+-        package => 'ElasticSearch',
+-        module  => 'ElasticSearch',
+-        version => '0',
+-    },
+     );
+
+     my $extra_modules = _get_extension_requirements('OPTIONAL_MODULES');
+bugzilla@vagrant-ubuntu-trusty-64:~/bugzilla$
new file mode 100755
--- /dev/null
+++ b/testing/bmoserver/files/fastimport.py
@@ -0,0 +1,23 @@
+#!/usr/bin/python
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+# Add some guards around a mysql dump file to make importing faster.
+# See https://dev.mysql.com/doc/refman/5.6/en/optimizing-innodb-bulk-data-loading.html
+
+from __future__ import print_function
+
+import sys
+
+# These drastically cut down on I/O during bulk import and make importing
+# significantly faster, even on solid state storage.
+print('SET unique_checks=0;')
+print('SET foreign_key_checks=0;')
+
+while True:
+    data = sys.stdin.read(32768)
+    if not data:
+        break
+
+    sys.stdout.write(data)
new file mode 100644
--- /dev/null
+++ b/testing/bmoserver/files/fkpatch.patch
@@ -0,0 +1,16 @@
+diff --git a/Bugzilla/DB.pm b/Bugzilla/DB.pm
+index cf828d7..1df5fb8 100644
+--- a/Bugzilla/DB.pm
++++ b/Bugzilla/DB.pm
+@@ -577,8 +577,10 @@ sub bz_add_column {
+     my $current_def = $self->bz_column_info($table, $name);
+ 
+     if (!$current_def) {
++        my $trimmed_def = dclone($new_def);
++        delete $trimmed_def->{REFERENCES};
+         my @statements = $self->_bz_real_schema->get_add_column_ddl(
+-            $table, $name, $new_def, 
++            $table, $name, $trimmed_def, 
+             defined $init_value ? $self->quote($init_value) : undef);
+         print get_text('install_column_add',
+                        { column => $name, table => $table }) . "\n"
new file mode 100644
--- /dev/null
+++ b/testing/bmoserver/files/nodefaultproduct.patch
@@ -0,0 +1,12 @@
+diff --git a/checksetup.pl b/checksetup.pl
+index 501138c..0091ab5 100755
+--- a/checksetup.pl
++++ b/checksetup.pl
+@@ -230,7 +230,7 @@ Bugzilla::Install::reset_password($switch{'reset-password'})
+ # Create default Product
+ ###########################################################################
+
+-Bugzilla::Install::create_default_product();
++#Bugzilla::Install::create_default_product();
+
+ Bugzilla::Hook::process('install_before_final_checks', { silent => $silent });
new file mode 100644
--- /dev/null
+++ b/testing/bmoserver/fileserver.conf
@@ -0,0 +1,4 @@
+[files]
+path /vagrant/files
+allow *
+
new file mode 100644
--- /dev/null
+++ b/testing/bmoserver/manifests/bugzilla.pp
@@ -0,0 +1,280 @@
+$packages = [
+  'build-essential',
+  'g++',
+  'graphviz',
+  'libdaemon-generic-perl',
+  'libgd-dev',
+  'libssl-dev',
+  # Installing PerlMagick from CPAN gives a compiler error.
+  'perlmagick',
+  'pkg-config',
+  'unzip',
+]
+
+package { $packages:
+  ensure       => installed,
+  responsefile => 'puppet:///files/debconf-responses',
+}
+
+user { 'bugzilla':
+  gid      => 'bugzilla',
+  groups   => ['bugzilla'],
+  uid      => 1002,
+  shell    => '/bin/bash',
+  password => 'bugzilla',
+}
+
+group { 'bugzilla':
+  gid => 500,
+}
+
+file { '/home/bugzilla':
+  ensure  => directory,
+  owner   => 'bugzilla',
+  group   => 'bugzilla',
+  require => User['bugzilla'],
+}
+
+vcsrepo { 'bugzilla':
+  path     => '/home/bugzilla/bugzilla',
+  source   => 'https://git.mozilla.org/webtools/bmo/bugzilla.git',
+  provider => git,
+  owner    => bugzilla,
+  group    => bugzilla,
+  ensure   => latest,
+  revision => 'production',
+  require  => File['/home/bugzilla'],
+}
+
+file { '/home/bugzilla/bugzilla':
+  mode    => 0755,
+  owner   => bugzilla,
+  group   => bugzilla,
+  require => Vcsrepo["bugzilla"],
+}
+
+# Changing this will cause the log file to be deleted. Due to
+# Exec['fix_innodb']. You have been warned.
+$innodb_log_file_megabytes = 512
+$innodb_log_file_bytes = $innodb_log_file_megabytes * 1024 * 1024
+
+class { '::mysql::server':
+  # BMO uses 5.6. Ubuntu 14.04 ships with 5.5 by default.
+  package_name   => 'mysql-server-5.6',
+  purge_conf_dir => true,
+  root_password  => 'root',
+  override_options         => {
+    'mysql'                => {
+      'max_allowed_packet' => '1G',
+    },
+    'mysqld'                    => {
+      'default_storage_engine'  => 'InnoDB',
+      'key_buffer_size'         => '32M',
+      'max_allowed_packet'      => '1G',
+      'innodb'                  => 'FORCE',
+      'character-set-server'    => 'utf8mb4',
+      'collation-server'        => 'utf8mb4_general_ci',
+
+      'tmp_table_size'         => '32M',
+      'max_heap_table_size'    => '32M',
+      'query_cache_type'       => '0',
+      'query_cache_size'       => '0',
+      'max_connections'        => '500',
+      'thread_cache_size'      => '50',
+      'table_definition_cache' => '1024',
+      'table_open_cache'       => '2048',
+
+      'innodb_flush_method'            => 'O_DIRECT',
+      # 0 is optimized for SSDs.
+      'innodb_flush_neighbors'         => '0',
+      'innodb_log_files_in_group'      => '2',
+      'innodb_log_file_size'           => "${innodb_log_file_megabytes}M",
+      # This is a bit large to optimize for bulk inserts.
+      'innodb_log_buffer_size'         => '64M',
+      'innodb_flush_log_at_trx_commit' => '2',
+      'innodb_file_per_table'          => '1',
+      # Decrease this if you don't run the VM with 4 GB of memory.
+      'innodb_buffer_pool_size'       => '2G',
+      'innodb_write_io_threads'       => '8',
+      'innodb_read_io_threads'        => '8',
+      'innodb_change_buffer_max_size' => '75',
+
+      'log_error' => '/var/lib/mysql/mysql-error.log',
+    },
+  },
+
+  databases => {
+    'bugs'  => {
+      ensure  => 'present',
+      charset => 'utf8',
+    },
+  },
+
+  users => {
+    'bugs@localhost' => {
+      ensure => 'present',
+      # bugs
+      password_hash => '*F6143BCA58806D14CD1C97998C6792405D8AE8AE',
+    },
+  },
+
+  grants => {
+    'bugs@localhost/*.*' => {
+      ensure     => 'present',
+      options    => ['GRANT'],
+      privileges => 'ALL',
+      table      => '*.*',
+      user       => 'bugs@localhost',
+    },
+  },
+}
+
+# We adjust the default innodb log file sizes. This causes MySQL to
+# throw a fit about mismatched size because the log files created during
+# package install and subsequent initial run are a different size from
+# the config. We forcefully remove the log file after log file size
+# change.
+exec { 'fix_innodb':
+  command   => "/sbin/stop mysql; /bin/rm /var/lib/mysql/ib_logfile0 /var/lib/mysql/ib_logfile1",
+  onlyif    => "/usr/bin/test -e /var/lib/mysql/ib_logfile0 -a \$(/usr/bin/du -b /var/lib/mysql/ib_logfile0 | /usr/bin/awk '{ print \$1 }') -ne ${innodb_log_file_bytes}}",
+}
+
+Class['mysql::server::config'] ~> Exec['fix_innodb'] ~> Class['mysql::server::service']
+
+# Bugzilla dump file to load. This is optional.
+# Dumps can be obtained from https://people.mozilla.org/~mhoye/bugzilla/.
+$bmo_dump_file = 'Mozilla-Bugzilla-Public-*.sql.gz'
+
+exec { 'load_bmo':
+  onlyif  => [
+    "/usr/bin/test -f /vagrant/files/${bmo_dump_file}",
+    "/usr/bin/test ! -f /home/bugzilla/bmo_loaded",
+  ],
+  user    => 'bugzilla',
+  command => "/bin/zcat /vagrant/files/${bmo_dump_file} | /vagrant/files/fastimport.py | /usr/bin/mysql -ubugs -pbugs bugs && /usr/bin/touch /home/bugzilla/bmo_loaded",
+  timeout => 14400,
+}
+
+Class['mysql::server'] ~> Exec['load_bmo']
+
+exec { 'reset_git':
+  require => Vcsrepo['bugzilla'],
+  command => '/usr/bin/git reset --hard HEAD && /usr/bin/git clean -f',
+  user    => 'bugzilla',
+  cwd     => '/home/bugzilla/bugzilla',
+}
+
+# ElasticSearch isn't available on CPAN. Hack around failure installing
+# it.
+exec { 'patch_es':
+  require => Exec['reset_git'],
+  before  => Exec['bzmodules'],
+  command => '/usr/bin/patch -p1 < /vagrant/files/elasticsearch.patch',
+  user    => 'bugzilla',
+  cwd     => '/home/bugzilla/bugzilla',
+}
+
+exec { 'bzmodules':
+  require => [Exec['patch_es'], Package[$packages], Class[::mysql::server]],
+  command => '/home/bugzilla/bugzilla/install-module.pl --all',
+  user    => 'bugzilla',
+  cwd     => '/home/bugzilla/bugzilla',
+  timeout => 1800,
+}
+
+# Linux::Pid is a dependency for Apache::SizeLimit. For whatever reason
+# it isn't detected by CPAN.
+# XMLRPC::Transport::HTTP is needed by xmlrpc.cgi. For whatever reason
+# the dependency isn't installed.
+exec { 'pm_extra':
+  require => Vcsrepo['bugzilla'],
+  command => '/home/bugzilla/bugzilla/install-module.pl Linux::Pid XMLRPC::Transport::HTTP',
+  user    => 'bugzilla',
+  cwd     => '/home/bugzilla/bugzilla',
+}
+
+# Bug 769829. Foreign key issue with component watch schema updating.
+exec { 'patch_db':
+  before  => Exec['bzchecksetup'],
+  require => Exec['reset_git'],
+  command => '/usr/bin/patch -p1 < /vagrant/files/fkpatch.patch',
+  user    => 'bugzilla',
+  cwd     => '/home/bugzilla/bugzilla',
+}
+
+# Bug 1034678. Error creating default product.
+# This is only relevant is we don't do a BMO load. But it's harmless of
+# we do.
+exec { 'patch_defaultproduct':
+  before   => Exec['bzchecksetup'],
+  require  => Exec['reset_git'],
+  command  => '/usr/bin/patch -p1 < /vagrant/files/nodefaultproduct.patch',
+  user     => 'bugzilla',
+  cwd      => '/home/bugzilla/bugzilla',
+}
+
+# The .htaccess isn't compatible with Apache 2.4. We fix that.
+exec { 'patch_htaccess':
+  require => Exec['reset_git'],
+  before  => Service['httpd'],
+  command => '/usr/bin/patch -p1 < /vagrant/files/apache24.patch',
+  user    => 'bugzilla',
+  cwd     => '/home/bugzilla/bugzilla',
+}
+
+# checksetup.pl appears to not always refresh data/params if the
+# answers have been updated. Force it by removing output.
+file { '/home/bugzilla/bugzilla/data':
+  ensure  => absent,
+  recurse => true,
+  force   => true,
+}
+
+exec { 'bzchecksetup':
+  require  => [Exec['bzmodules'], Exec['load_bmo'], File['/home/bugzilla/bugzilla/data']],
+  command => '/home/bugzilla/bugzilla/checksetup.pl /vagrant/files/checksetup_answers.txt',
+  user     => 'bugzilla',
+  cwd      => '/home/bugzilla/bugzilla',
+}
+
+# Restore pristine repo state (undo local patch hacks).
+exec { 'unpatch_bugzilla':
+  require => Exec['bzchecksetup'],
+  command => '/usr/bin/git checkout -- Bugzilla/DB.pm Bugzilla/Install/Requirements.pm checksetup.pl',
+  user    => 'bugzilla',
+  cwd     => '/home/bugzilla/bugzilla',
+}
+
+file { '/etc/apache2/conf.d/50perlswitches.conf':
+  source => '/vagrant/files/apache_perlswitches.conf',
+  owner  => 'root',
+  group  => 'root',
+  notify => Service['httpd'],
+}
+
+class { 'apache':
+  default_vhost => false,
+  group         => 'bugzilla',
+  manage_group  => false,
+  mpm_module    => 'prefork',
+}
+
+class { 'apache::mod::perl': }
+class { 'apache::mod::rewrite': }
+
+apache::vhost { 'bugzilla':
+  port        => '80',
+  docroot     => '/home/bugzilla/bugzilla',
+  directories => [
+    {
+      path           => '/home/bugzilla/bugzilla',
+      addhandlers    => [{ handler => 'cgi-script', extensions => ['.cgi'] }],
+      directoryindex => 'index.cgi',
+      allow_override => ['All'],
+      options        => ['Indexes', 'FollowSymLinks', 'ExecCGI'],
+    },
+  ],
+  additional_includes => '/vagrant/files/apache_extra.conf',
+  require             => Exec['pm_extra'],
+}
+
new file mode 100644
--- /dev/null
+++ b/testing/bmoserver/vagrant-bootstrap.sh
@@ -0,0 +1,38 @@
+#!/bin/sh
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+# This script provisions a Vagrant VM for development mode.
+
+set -e
+
+MODULES_DIR='/var/puppet-modules'
+
+if [ ! -f /root/provision.initial ]; then
+  apt-get update
+  # We don't need chef since we use puppet.
+  apt-get -y remove chef
+  apt-get -y autoremove
+  touch /root/provision.initial
+  apt-get -y dist-upgrade
+  apt-get -q -y install git ruby-dev
+fi
+
+if [ ! -d $MODULES_DIR ]; then
+  mkdir -p $MODULES_DIR
+fi
+
+cp /vagrant/Puppetfile $MODULES_DIR/Puppetfile
+
+if [ `gem query --local | grep librarian-puppet | wc -l` -eq 0 ]; then
+  gem install librarian-puppet
+  cd $MODULES_DIR && librarian-puppet install --clean
+else
+  cd $MODULES_DIR && librarian-puppet update
+fi
+
+puppet apply --verbose \
+  --modulepath=$MODULES_DIR/modules/ \
+  --fileserverconfig=/vagrant/puppet/fileserver.vagrant.conf \
+  /vagrant/manifests/bugzilla.pp