Nova Resource:Deployment-prep/SAL/Archive 1

May 2

15:14 labs-logs-bottie: petrb: making syslog on apaches be /data/project/apaches_log
11:52 labs-logs-bottie: petrb: changed sudo policies on web to test if puppet override it
11:52 labs-logs-bottie: j: add transcoding settings to CommonSettings.php again
11:46 labs-logs-bottie: petrb: purged stuff on transcoding and freed some 119468kb
11:08 labs-logs-bottie: hashar: install dsh package on deployment-dbdump
09:54 labs-logs-bottie: hashar: /usr/local/apache/conf is now an independant git repository

April 30

13:31 labs-logs-bottie: petrb: rebooting web5
13:27 labs-logs-bottie: petrb: web4 reboot for same reason
13:26 labs-logs-bottie: petrb: same for web
13:24 labs-logs-bottie: petrb: rebooting web3 broken /data/project/
11:57 labs-logs-bottie: petrb: fixed transcoding
11:28 j^: reboot deployment-transcoding(i-00000105)

April 25

17:01 labs-logs-bottie: j: Changing uid and group of apache user from 48 to 33 to match www-data on web3,web4,web5
16:44 Platonides: With the uid change to deployment-web, it is now writing into /data/project/errors.log
16:42 hashar: Apaches no more log anything. This is because rsyslog sends logs to a blackhole :-D
16:27 Platonides: Changing uid and group of apache user from 48 to 33 to match www-data
15:20 hashar: deleting testswarmmysqlconf , it is of no use :-/
14:43 hashar: Creating temporary instance to test a MySQL puppet snippet
13:01 labs-logs-bottie: hashar: I am a hero
13:00 labs-logs-bottie: petrb: test
12:24 hashar: added a basic puppet skeleton in manifests/labs/beta/ with https://gerrit.wikimedia.org/r/5790 (test branch)
11:39 hashar: manually purged debian package `ack`, installed `ack-grep`
11:22 hashar: puppet finished migration of web{3,4,5} to apache::service
11:19 hashar: changed squid visible name to squid001.beta.wmflabs.org
11:02 hashar: migrate web{3,4,5} from webserver::php5 to apaches::service
10:47 hashar: Cleaning out squid peers list
10:11 hashar: made /etc/ a git repository on deployment-squid and committed existing /etc/squid/
10:03 hashar: adding generic::packages::git-core on deployment-squid so we can track

April 24

23:21 labs-logs-bottie: petrb: disabled $wmgEnableCaptcha
18:54 hashar: wrap -dbdump motd to 80 chars
15:42 hashar: deployement-web host does work! :-]
15:41 hashar: made deployement-web host a wikimedia-task-appserver , add to create some apache2 configuration placeholder. Apache2 does launch but it is not working though (timeout)
15:40 petan|wk: I told hashar to log stuff, if he won't, slap him
13:18 hashar: added apache::service on deployment-web host
12:18 labs-logs-bottie: petrb: moved the log file storage to gluster
12:18 labs-logs-bottie: petrb: updated git and commited all changes

March 21

03:22 mutante: mysqld on deployment-sql is stopped - did not start it though after i heard petan is working on corrupted db's
03:06 mutante: added myself as a member just to see the instance names and check for the sql server...

March 20

16:16 labs-logs-bottie: petrb: it seems that corruption of db is worse than I expected, need to restore backup old few months
16:12 labs-logs-bottie: petrb: mysql is back up
15:41 labs-logs-bottie: petrb: getting sql server down I found a bunch of corrupted db's, rollback is necessary
15:41 labs-logs-bottie: j: install php-pear on deployment-web3/4/5 required by TMH

March 19

08:18 labs-logs-bottie: root: restoring sql tables from backup

March 15

20:17 labs-logs-bottie: root: restored ok
20:14 labs-logs-bottie: root: restoring database from backup
10:16 labs-logs-bottie: petrb: failed auth on db server reboot was required
09:53 labs-logs-bottie: petrb: scheduling auto replication of sql server

March 14

14:37 labs-logs-bottie: petrb: switching en.wikipedia to older previous v

March 11

22:14 Damianz: Increased nofile on deployment-squid and added max_filedesc option to squid config. Also installed squidclient.
04:36 Ryan_Lane: also deployment-web3
04:35 Ryan_Lane: also deployment-web
04:34 Ryan_Lane: make that deployment-web5
04:34 Ryan_Lane: rebooting deployment-web, it OOM'd

March 9

14:12 labs-logs-bottie: petrb: rebooting -nfs
13:52 labs-logs-bottie: root: updated apt on webs1
13:34 j^: add ppa:j/timedmediahandler and install ffmpeg on web3 and web5

March 6

22:21 labs-logs-bottie: petrb: some instances will need to reboot, however site seems to be ok now
22:09 labs-sexy-bottie: petrb: updating svn
22:07 labs-sexy-bottie: petrb: fixed squid a bit
15:23 labs-sexy-bottie: petrb: test
14:53 labs-sexy-bottie: root: disabling bot for a while
03:11 Andrew: facepalm: apparently all reboots are failing, so this will be down until Ryan brings it all back up tomorrow
02:57 Andrew: rebooting a few hosts, there is something seriously wrong with fetching resources at the moment

March 5

15:48 labs-sexy-bottie: petrb: temporary disabled ssl server
15:48 labs-sexy-bottie: petrb: reconfigured squid
15:36 labs-sexy-bottie: petrb: restarted servers
15:24 labs-sexy-bottie: petrb: temporary changed code of localsettings to debug site
15:20 labs-sexy-bottie: petrb: fixed broken memc :o
15:04 labs-sexy-bottie: petrb: inserted new wiki to sul
15:01 labs-sexy-bottie: petrb: please ignore some of the previous lines in log we were just testing bot
14:58 labs-sexy-bottie: petrb: updated live
14:58 labs-sexy-bottie: petrb: meh
14:36 labs-sexy-bottie: petrb: created a new log system, just type log message to log your change on prep
14:35 labs-sexy-bottie: petrb: this is test :o

March 4

12:00 Andrew: Finished deployment of het deploy, added a new ee.

March 1

16:35 Platonides: Installed dpkg-dev on deployment-dbdump
16:03 Platonides: Installed joe on deployment-dbdump
16:03 petan|wk: platonides needs to check the project name

February 27

08:20 petan: creating 2 more web servers to handle load
08:19 petan: rebooting both web servers, starting with web1

February 23

08:52 petan|wk: fixing the squid

February 22

01:03 Ryan_Lane: reconfiguring the web server instances to remove puppet classes that no longer exist

February 17

02:34 Andrew: Moving /usr/local/apache/common/live to /usr/local/apache/common/live-hom and symlinking live to live-hom
01:56 Andrew: running afl_rev_id patch on all wikis
01:52 Andrew: installing ack (source code search tool) on dbdump
00:59 petan: if anything is broken, it was me
00:51 petan: I broke it!
00:27 petan: switched to HEAD

February 16

10:11 j^: add video/webm to /etc/mime.types on web/webs1/web2

February 13

10:08 petan|wk: removing the puppetized memcached
08:54 petan|wk: removing some extensions from config which are missing in latest branch

January 30

18:46 petan: configuring some boxes for cluster to handle high load

January 29

01:13 hexmode: oom reboot -web

January 27

13:30 j^: install upstart script /etc/init/timedmediahandler.conf on deployment-transcoding and start service
13:05 j^: touch /etc/wikimedia-image-scaler on deployment-transcoding; transcoding needs more wgMaxShellMemory too
12:47 Platonides: updated /usr/local/apache/common/live/extensions/TimedMediaHandler to r110117 per j^request
10:11 j^: add-apt-repository ppa:j/timedmediahandler and update ffmpeg on deployment-web to support frame extraction from WebM videos
06:35 j^: update ffmpeg on deployment-transcoding (new security release from ppa)

January 26

00:00 petan: configured new firewall rule irc

January 25

23:52 petan: linked /usr/local/apache/common-local to /usr/local/apache/common
23:06 petan: updating svn
22:28 petan: reverted unlogged changes made to config which broke whole site
10:03 j^: installed ffmpeg on deployment-web (required by TMH to extract stills)

January 24

20:52 petan: created db user oren and new database for temporary wiki
13:53 petan|wk: reconfigured new instance and fixed some issues on puppet, no logs in sal regarding it
00:51 hexmode: svn up * updatedata

January 23

19:31 hexmode: restart memcache on nfs-memc
19:06 hexmode: aptitude update deployment-web

January 22

09:16 petan: configured nfs to listen for backup server
01:01 petan: configured firewall for backup instance
00:45 petan: creating a backup instance in -prepbackup project for online backup of mysql from deployment project + fs backup
00:30 petan: updating /live to head
00:21 petan: installed timedmediahandler (trunk) to commons

January 16

23:41 hexmode: to solve the trusted XFF problem, I installed tinycdb and created an 0 length file in the right place
20:35 Ryan_Lane: released unused IP address from project

January 15

15:45 petan: ran live/extensions/TrustedXFF/generate.php
11:20 petan: updated to latest head all wikis

January 14

18:49 petan: enabled global blocking
18:21 johnduhart: Removed myself from the project.
14:32 petan: separated common to own deployment file
14:30 hexmode: enabled webfonts for mywiki properly in IntialiseSettingsDeploy.php
14:25 johnduhart: Updated wmf-config/InitialiseSettings.php from production
14:25 johnduhart: Reverted change to wmf-config/InitialiseSettings.php
14:24 hexmode: enabled webfonts for mywiki

January 13

21:34 petan: assigning new dns
21:33 petan: moved deployment to beta.wmf...
13:07 petan|w: installed jdk on search

January 12

18:49 petan: installed all requested sw on search
18:46 petan: mounted conf files
18:38 petan: installed updates on new instances and rebooting it
06:48 Ryan_Lane: added nfs mounts to the fstab for deployment-web
06:47 Ryan_Lane: remounted /mnt/upload on deployment-web as nfs rather than nfs4
06:47 Ryan_Lane: modified export options on deployment-nfs-memc; removed nfs4 specific options, and removed other options not necessary for our environment.
00:38 johnduhart: Unmounted /mnt/export from /tmp on -web

January 11

23:02 hexmode: svn up live
03:19 johnduhart: Added production commons to ForeignFileRepos http://en.wikipedia.deployment.wmflabs.org/wiki/File:BradPittBAR08.jpg
02:28 johnduhart: Moved config to http://config.deployment.wmflabs.org/viewfile.php?file=CommonSettings.php
01:44 johnduhart: updated databases and interwiki
01:41 johnduhart: setup flagged revisions

January 10

23:06 petan: checks done
23:04 petan: disabled sql for fs checks
22:30 petan: created nfs:/mnt/export/backup use it for all files which aren't versioned
21:44 petan: deleting big db's expect db lags ^^
21:35 petan: maintenance on simple
21:17 petan: created squid box
19:05 petan: tweaked memcached and flushed cache
16:48 johnduhart: installing wikimedia-task-appserver on -web
16:29 johnduhart: live and extensions recheckedout into a new folder
16:04 johnduhart: Site broken, currently recreating live folder
12:27 johnduhart: Running updatedata (very slowly)
12:19 johnduhart: Fixed wmf-config permissions on -nfs-memc
05:32 johnduhart: thumbnails now working
05:30 johnduhart: Installed imagemagick on web
05:11 johnduhart: Adding apache config for upload.deployment.wmflabs.org
05:11 johnduhart: Made a quick stab at upload config
05:00 johnduhart: Mounted that export onto -web
05:00 johnduhart: Created nfs export /mnt/upload on deployment-nfs-memc
04:37 johnduhart: unmounted deployment-nfs-memc:/mnt/export on /mnt from deployment-web
03:32 johnduhart: Last update solves an issue where CentralAuth would make 70+ queries per page
03:31 johnduhart: Updated databases
03:26 johnduhart: svn up'd live
03:04 johnduhart: interwiki now works

January 9

21:29 johnduhart: Updated databases
21:29 johnduhart: Created test.wikimedia
21:25 petan: reconfigured global permissions
20:38 petan: created hi wiki + de wiki
20:00 petan: creating commons, de wiki, en_wiktionary etc. etc...
18:57 petan: configured ip back to -web and removed temporary ip
18:53 petan: disasociated 208.80.153.219
18:50 petan: turned off -test definitely and reconfiguring ip
18:47 petan: moved all stuff -web reconfiguring IP
18:18 petan: moving configuration of apache to web
16:40 petan|work: reconfigured apache on test
16:15 johnduhart: svn up /usr/local/apache/common/live
15:59 johnduhart: Ran update on metawiki
15:58 johnduhart: Recreated metawiki enwiki enwikibooks
15:57 johnduhart: Imported centralauth
15:55 johnduhart: Dropped all wikis except simplewiki
15:55 johnduhart:
15:53 petan|work: restarted memcached
15:24 johnduhart: Created enwiki and enwikibooks
15:20 johnduhart: Restarted memcached
15:17 johnduhart: Created simplewiki
15:04 petan|work: restarted memcached
15:02 johnduhart: Ran update.php on metawiki
14:59 johnduhart: Creating metawiki
14:55 petan|work: disabled current site
14:55 johnduhart: Creating centralauth db
14:52 johnduhart: DROPing new configuration tables, will recreate
14:38 johnduhart: Forget the metawiki dump
14:09 petan|work: created backup of broken db of meta and replaced it with auth db
13:52 petan|work: test is done, restored test SUL to previous state
13:43 petan|work: created backup of central auth and replace the testing SUL with current data, merged with current SUL so that we can use same logins on all sites
13:19 petan|work: updated svn
12:44 johnduhart: Ran update.php on metawiki
12:32 johnduhart: Created enwikibooks http://en.wikibooks.deployment.wmflabs.org/wiki/Main_Page
08:12 johnduhart: Starting import of metawiki
07:37 johnduhart: This only affects my new configuration though
07:36 johnduhart: WARNING: some how new users are showing up on some feed outside of labs, and was picked up on a monitoring robot. wtf.
07:15 johnduhart: Importing simplewikibooks
07:10 johnduhart: SiteMatrix is now working http://meta.wikimedia.deployment.wmflabs.org/wiki/Special:SiteMatrix
07:09 johnduhart: Created simple wikibooks http://simple.wikibooks.deployment.wmflabs.org/
07:09 johnduhart: Adding wikibooks configuration
06:31 johnduhart: Installing git on deployment-test
06:23 johnduhart: Created simplewiki http://simple.wikipedia.deployment.wmflabs.org/wiki/Main_Page
05:50 johnduhart: Downloaded metawiki dump ont dbdump, extracting now. poor labs.
05:36 johnduhart: Recreated hwiki, had a bad JohnTest account. Central auth now works fully
05:15 johnduhart: Running update.php on hiwiki
05:14 johnduhart: Importing prefstats table to hiwiki
04:41 johnduhart: Centralauth is working :)
04:28 johnduhart: Imported db schema to centralauth
04:25 johnduhart: Created centralauth database centralauth
04:18 johnduhart: Created metawiki http://meta.wikimedia.deployment.wmflabs.org/
04:18 johnduhart: http://meta.wikimedia.deployment.wmflabs.org/
04:09 johnduhart: Created meta docroot and added favicons to meta and wp
04:08 johnduhart: Adding remnant.conf to httpd.conf
04:07 johnduhart: Adding metawiki apache config to /usr/local/apache/conf/remnant.conf
04:03 johnduhart: Lot of configuration done, much tuning to come. Missing.php now works http://nope.wikipedia.deployment.wmflabs.org/wiki/Main_Page
01:24 johnduhart: Installed mysql-client on deployment-test

January 8

23:45 johnduhart: Installed php5-cli on deployment-test
23:21 johnduhart: Enabled rewrite rules on deployment-test
20:39 petan: created new group for /mnt/www
19:33 petan: clearing cache
18:47 hexmode: set up relative paths for config files
16:45 petan: moved memcached to deployment-nfs-memc and update global
14:04 petan: created temporary memcached instance for data
12:17 petan: restarted apache to fix some problem with config
00:11 hexmode: set config for timeline
00:11 hexmode: "apt-get install ploticus ttf-freefont" for timeline
00:04 jeremyb: booting memcached fixed it
00:02 jeremyb: booting memcache in case that fixes interwiki
00:02 jeremyb: [auth] added bugzilla to interwiki.sql and ran it, doesn't seem to be working

January 7

23:55 hexmode: Fix path for PoolCounter: PoolCounter.php -> PoolCounterClient.php
23:52 hexmode: Fix path for OAI: OAI.php -> OAIRepo.php
23:31 hexmode: set up git for /var/www/global
23:08 petan: disabled LQT on other wikis
23:07 petan: changed configuration of ajax
23:07 petan: fixed InitialiseSetting
22:42 petan: updated all wikis to latest head
22:36 petan: disabled LQT because it's broken
22:34 petan: ran update on auth db
22:10 petan: removed puppetized memcached, because its configuration suck

January 5

20:26 petan: importing MW ns full history to en_wikipedia
16:48 petan|w: created instance for dumps, current instance is overloaded
15:53 petan|w: import now running using mwimport

January 4

20:27 petan: opened port 80 for wide net
20:26 petan: registered deployment.wmflabs.org
20:24 petan: allocated ip208.80.153.215
20:22 mutante: raised floating IP quota to 1
16:10 petan|work: create instances for apache and mysql
16:08 petan|work: configured firewall for webserver
16:06 mutante: added members MarkAHershberger & Petrb - added them to sysadmin and netadmin roles
16:02 mutante: added new project deployment-prep for hexmode and petan
16:02 mutante: added new project deployment-prep for hexmode and petan