Nova Resource:Deployment-prep/SAL/Archive 1: Difference between revisions
Content deleted Content added
Labslogbot (talk | contribs) petrb: changed sudo policies on web to test if puppet override it (labs-logs-bottie) |
Labslogbot (talk | contribs) petrb: making syslog on apaches be /data/project/apaches_log (labs-logs-bottie) |
||
Line 1: | Line 1: | ||
=== May 2 === |
=== May 2 === |
||
* 15:14 labs-logs-bottie: petrb: making syslog on apaches be /data/project/apaches_log |
|||
* 11:52 labs-logs-bottie: petrb: changed sudo policies on web to test if puppet override it |
* 11:52 labs-logs-bottie: petrb: changed sudo policies on web to test if puppet override it |
||
* 11:52 labs-logs-bottie: j: add transcoding settings to CommonSettings.php again |
* 11:52 labs-logs-bottie: j: add transcoding settings to CommonSettings.php again |
Revision as of 15:14, 2 May 2012
May 2
- 15:14 labs-logs-bottie: petrb: making syslog on apaches be /data/project/apaches_log
- 11:52 labs-logs-bottie: petrb: changed sudo policies on web to test if puppet override it
- 11:52 labs-logs-bottie: j: add transcoding settings to CommonSettings.php again
- 11:46 labs-logs-bottie: petrb: purged stuff on transcoding and freed some 119468kb
- 11:08 labs-logs-bottie: hashar: install dsh package on deployment-dbdump
- 09:54 labs-logs-bottie: hashar: /usr/local/apache/conf is now an independant git repository
April 30
- 13:31 labs-logs-bottie: petrb: rebooting web5
- 13:27 labs-logs-bottie: petrb: web4 reboot for same reason
- 13:26 labs-logs-bottie: petrb: same for web
- 13:24 labs-logs-bottie: petrb: rebooting web3 broken /data/project/
- 11:57 labs-logs-bottie: petrb: fixed transcoding
- 11:28 j^: reboot deployment-transcoding(i-00000105)
April 25
- 17:01 labs-logs-bottie: j: Changing uid and group of apache user from 48 to 33 to match www-data on web3,web4,web5
- 16:44 Platonides: With the uid change to deployment-web, it is now writing into /data/project/errors.log
- 16:42 hashar: Apaches no more log anything. This is because rsyslog sends logs to a blackhole :-D
- 16:27 Platonides: Changing uid and group of apache user from 48 to 33 to match www-data
- 15:20 hashar: deleting testswarmmysqlconf , it is of no use :-/
- 14:43 hashar: Creating temporary instance to test a MySQL puppet snippet
- 13:01 labs-logs-bottie: hashar: I am a hero
- 13:00 labs-logs-bottie: petrb: test
- 12:24 hashar: added a basic puppet skeleton in manifests/labs/beta/ with https://gerrit.wikimedia.org/r/5790 (test branch)
- 11:39 hashar: manually purged debian package `ack`, installed `ack-grep`
- 11:22 hashar: puppet finished migration of web{3,4,5} to apache::service
- 11:19 hashar: changed squid visible name to squid001.beta.wmflabs.org
- 11:02 hashar: migrate web{3,4,5} from webserver::php5 to apaches::service
- 10:47 hashar: Cleaning out squid peers list
- 10:11 hashar: made /etc/ a git repository on deployment-squid and committed existing /etc/squid/
- 10:03 hashar: adding generic::packages::git-core on deployment-squid so we can track
April 24
- 23:21 labs-logs-bottie: petrb: disabled $wmgEnableCaptcha
- 18:54 hashar: wrap -dbdump motd to 80 chars
- 15:42 hashar: deployement-web host does work! :-]
- 15:41 hashar: made deployement-web host a wikimedia-task-appserver , add to create some apache2 configuration placeholder. Apache2 does launch but it is not working though (timeout)
- 15:40 petan|wk: I told hashar to log stuff, if he won't, slap him
- 13:18 hashar: added apache::service on deployment-web host
- 12:18 labs-logs-bottie: petrb: moved the log file storage to gluster
- 12:18 labs-logs-bottie: petrb: updated git and commited all changes
March 21
- 03:22 mutante: mysqld on deployment-sql is stopped - did not start it though after i heard petan is working on corrupted db's
- 03:06 mutante: added myself as a member just to see the instance names and check for the sql server...
March 20
- 16:16 labs-logs-bottie: petrb: it seems that corruption of db is worse than I expected, need to restore backup old few months
- 16:12 labs-logs-bottie: petrb: mysql is back up
- 15:41 labs-logs-bottie: petrb: getting sql server down I found a bunch of corrupted db's, rollback is necessary
- 15:41 labs-logs-bottie: j: install php-pear on deployment-web3/4/5 required by TMH
March 19
- 08:18 labs-logs-bottie: root: restoring sql tables from backup
March 15
- 20:17 labs-logs-bottie: root: restored ok
- 20:14 labs-logs-bottie: root: restoring database from backup
- 10:16 labs-logs-bottie: petrb: failed auth on db server reboot was required
- 09:53 labs-logs-bottie: petrb: scheduling auto replication of sql server
March 14
- 14:37 labs-logs-bottie: petrb: switching en.wikipedia to older previous v
March 11
- 22:14 Damianz: Increased nofile on deployment-squid and added max_filedesc option to squid config. Also installed squidclient.
- 04:36 Ryan_Lane: also deployment-web3
- 04:35 Ryan_Lane: also deployment-web
- 04:34 Ryan_Lane: make that deployment-web5
- 04:34 Ryan_Lane: rebooting deployment-web, it OOM'd
March 9
- 14:12 labs-logs-bottie: petrb: rebooting -nfs
- 13:52 labs-logs-bottie: root: updated apt on webs1
- 13:34 j^: add ppa:j/timedmediahandler and install ffmpeg on web3 and web5
March 6
- 22:21 labs-logs-bottie: petrb: some instances will need to reboot, however site seems to be ok now
- 22:09 labs-sexy-bottie: petrb: updating svn
- 22:07 labs-sexy-bottie: petrb: fixed squid a bit
- 15:23 labs-sexy-bottie: petrb: test
- 14:53 labs-sexy-bottie: root: disabling bot for a while
- 03:11 Andrew: facepalm: apparently all reboots are failing, so this will be down until Ryan brings it all back up tomorrow
- 02:57 Andrew: rebooting a few hosts, there is something seriously wrong with fetching resources at the moment
March 5
- 15:48 labs-sexy-bottie: petrb: temporary disabled ssl server
- 15:48 labs-sexy-bottie: petrb: reconfigured squid
- 15:36 labs-sexy-bottie: petrb: restarted servers
- 15:24 labs-sexy-bottie: petrb: temporary changed code of localsettings to debug site
- 15:20 labs-sexy-bottie: petrb: fixed broken memc :o
- 15:04 labs-sexy-bottie: petrb: inserted new wiki to sul
- 15:01 labs-sexy-bottie: petrb: please ignore some of the previous lines in log we were just testing bot
- 14:58 labs-sexy-bottie: petrb: updated live
- 14:58 labs-sexy-bottie: petrb: meh
- 14:36 labs-sexy-bottie: petrb: created a new log system, just type log message to log your change on prep
- 14:35 labs-sexy-bottie: petrb: this is test :o
March 4
- 12:00 Andrew: Finished deployment of het deploy, added a new ee.
March 1
- 16:35 Platonides: Installed dpkg-dev on deployment-dbdump
- 16:03 Platonides: Installed joe on deployment-dbdump
- 16:03 petan|wk: platonides needs to check the project name
February 27
- 08:20 petan: creating 2 more web servers to handle load
- 08:19 petan: rebooting both web servers, starting with web1
February 23
- 08:52 petan|wk: fixing the squid
February 22
- 01:03 Ryan_Lane: reconfiguring the web server instances to remove puppet classes that no longer exist
February 17
- 02:34 Andrew: Moving /usr/local/apache/common/live to /usr/local/apache/common/live-hom and symlinking live to live-hom
- 01:56 Andrew: running afl_rev_id patch on all wikis
- 01:52 Andrew: installing ack (source code search tool) on dbdump
- 00:59 petan: if anything is broken, it was me
- 00:51 petan: I broke it!
- 00:27 petan: switched to HEAD
February 16
- 10:11 j^: add video/webm to /etc/mime.types on web/webs1/web2
February 13
- 10:08 petan|wk: removing the puppetized memcached
- 08:54 petan|wk: removing some extensions from config which are missing in latest branch
January 30
- 18:46 petan: configuring some boxes for cluster to handle high load
January 29
- 01:13 hexmode: oom reboot -web
January 27
- 13:30 j^: install upstart script /etc/init/timedmediahandler.conf on deployment-transcoding and start service
- 13:05 j^: touch /etc/wikimedia-image-scaler on deployment-transcoding; transcoding needs more wgMaxShellMemory too
- 12:47 Platonides: updated /usr/local/apache/common/live/extensions/TimedMediaHandler to r110117 per j^request
- 10:11 j^: add-apt-repository ppa:j/timedmediahandler and update ffmpeg on deployment-web to support frame extraction from WebM videos
- 06:35 j^: update ffmpeg on deployment-transcoding (new security release from ppa)
January 26
- 00:00 petan: configured new firewall rule irc
January 25
- 23:52 petan: linked /usr/local/apache/common-local to /usr/local/apache/common
- 23:06 petan: updating svn
- 22:28 petan: reverted unlogged changes made to config which broke whole site
- 10:03 j^: installed ffmpeg on deployment-web (required by TMH to extract stills)
January 24
- 20:52 petan: created db user oren and new database for temporary wiki
- 13:53 petan|wk: reconfigured new instance and fixed some issues on puppet, no logs in sal regarding it
- 00:51 hexmode: svn up * updatedata
January 23
- 19:31 hexmode: restart memcache on nfs-memc
- 19:06 hexmode: aptitude update deployment-web
January 22
- 09:16 petan: configured nfs to listen for backup server
- 01:01 petan: configured firewall for backup instance
- 00:45 petan: creating a backup instance in -prepbackup project for online backup of mysql from deployment project + fs backup
- 00:30 petan: updating /live to head
- 00:21 petan: installed timedmediahandler (trunk) to commons
January 16
- 23:41 hexmode: to solve the trusted XFF problem, I installed tinycdb and created an 0 length file in the right place
- 20:35 Ryan_Lane: released unused IP address from project
January 15
- 15:45 petan: ran live/extensions/TrustedXFF/generate.php
- 11:20 petan: updated to latest head all wikis
January 14
- 18:49 petan: enabled global blocking
- 18:21 johnduhart: Removed myself from the project.
- 14:32 petan: separated common to own deployment file
- 14:30 hexmode: enabled webfonts for mywiki properly in IntialiseSettingsDeploy.php
- 14:25 johnduhart: Updated wmf-config/InitialiseSettings.php from production
- 14:25 johnduhart: Reverted change to wmf-config/InitialiseSettings.php
- 14:24 hexmode: enabled webfonts for mywiki
January 13
- 21:34 petan: assigning new dns
- 21:33 petan: moved deployment to beta.wmf...
- 13:07 petan|w: installed jdk on search
January 12
- 18:49 petan: installed all requested sw on search
- 18:46 petan: mounted conf files
- 18:38 petan: installed updates on new instances and rebooting it
- 06:48 Ryan_Lane: added nfs mounts to the fstab for deployment-web
- 06:47 Ryan_Lane: remounted /mnt/upload on deployment-web as nfs rather than nfs4
- 06:47 Ryan_Lane: modified export options on deployment-nfs-memc; removed nfs4 specific options, and removed other options not necessary for our environment.
- 00:38 johnduhart: Unmounted /mnt/export from /tmp on -web
January 11
- 23:02 hexmode: svn up live
- 03:19 johnduhart: Added production commons to ForeignFileRepos http://en.wikipedia.deployment.wmflabs.org/wiki/File:BradPittBAR08.jpg
- 02:28 johnduhart: Moved config to http://config.deployment.wmflabs.org/viewfile.php?file=CommonSettings.php
- 01:44 johnduhart: updated databases and interwiki
- 01:41 johnduhart: setup flagged revisions
January 10
- 23:06 petan: checks done
- 23:04 petan: disabled sql for fs checks
- 22:30 petan: created nfs:/mnt/export/backup use it for all files which aren't versioned
- 21:44 petan: deleting big db's expect db lags ^^
- 21:35 petan: maintenance on simple
- 21:17 petan: created squid box
- 19:05 petan: tweaked memcached and flushed cache
- 16:48 johnduhart: installing wikimedia-task-appserver on -web
- 16:29 johnduhart: live and extensions recheckedout into a new folder
- 16:04 johnduhart: Site broken, currently recreating live folder
- 12:27 johnduhart: Running updatedata (very slowly)
- 12:19 johnduhart: Fixed wmf-config permissions on -nfs-memc
- 05:32 johnduhart: thumbnails now working
- 05:30 johnduhart: Installed imagemagick on web
- 05:11 johnduhart: Adding apache config for upload.deployment.wmflabs.org
- 05:11 johnduhart: Made a quick stab at upload config
- 05:00 johnduhart: Mounted that export onto -web
- 05:00 johnduhart: Created nfs export /mnt/upload on deployment-nfs-memc
- 04:37 johnduhart: unmounted deployment-nfs-memc:/mnt/export on /mnt from deployment-web
- 03:32 johnduhart: Last update solves an issue where CentralAuth would make 70+ queries per page
- 03:31 johnduhart: Updated databases
- 03:26 johnduhart: svn up'd live
- 03:04 johnduhart: interwiki now works
January 9
- 21:29 johnduhart: Updated databases
- 21:29 johnduhart: Created test.wikimedia
- 21:25 petan: reconfigured global permissions
- 20:38 petan: created hi wiki + de wiki
- 20:00 petan: creating commons, de wiki, en_wiktionary etc. etc...
- 18:57 petan: configured ip back to -web and removed temporary ip
- 18:53 petan: disasociated 208.80.153.219
- 18:50 petan: turned off -test definitely and reconfiguring ip
- 18:47 petan: moved all stuff -web reconfiguring IP
- 18:18 petan: moving configuration of apache to web
- 16:40 petan|work: reconfigured apache on test
- 16:15 johnduhart: svn up /usr/local/apache/common/live
- 15:59 johnduhart: Ran update on metawiki
- 15:58 johnduhart: Recreated metawiki enwiki enwikibooks
- 15:57 johnduhart: Imported centralauth
- 15:55 johnduhart: Dropped all wikis except simplewiki
- 15:55 johnduhart:
- 15:53 petan|work: restarted memcached
- 15:24 johnduhart: Created enwiki and enwikibooks
- 15:20 johnduhart: Restarted memcached
- 15:17 johnduhart: Created simplewiki
- 15:04 petan|work: restarted memcached
- 15:02 johnduhart: Ran update.php on metawiki
- 14:59 johnduhart: Creating metawiki
- 14:55 petan|work: disabled current site
- 14:55 johnduhart: Creating centralauth db
- 14:52 johnduhart: DROPing new configuration tables, will recreate
- 14:38 johnduhart: Forget the metawiki dump
- 14:09 petan|work: created backup of broken db of meta and replaced it with auth db
- 13:52 petan|work: test is done, restored test SUL to previous state
- 13:43 petan|work: created backup of central auth and replace the testing SUL with current data, merged with current SUL so that we can use same logins on all sites
- 13:19 petan|work: updated svn
- 12:44 johnduhart: Ran update.php on metawiki
- 12:32 johnduhart: Created enwikibooks http://en.wikibooks.deployment.wmflabs.org/wiki/Main_Page
- 08:12 johnduhart: Starting import of metawiki
- 07:37 johnduhart: This only affects my new configuration though
- 07:36 johnduhart: WARNING: some how new users are showing up on some feed outside of labs, and was picked up on a monitoring robot. wtf.
- 07:15 johnduhart: Importing simplewikibooks
- 07:10 johnduhart: SiteMatrix is now working http://meta.wikimedia.deployment.wmflabs.org/wiki/Special:SiteMatrix
- 07:09 johnduhart: Created simple wikibooks http://simple.wikibooks.deployment.wmflabs.org/
- 07:09 johnduhart: Adding wikibooks configuration
- 06:31 johnduhart: Installing git on deployment-test
- 06:23 johnduhart: Created simplewiki http://simple.wikipedia.deployment.wmflabs.org/wiki/Main_Page
- 05:50 johnduhart: Downloaded metawiki dump ont dbdump, extracting now. poor labs.
- 05:36 johnduhart: Recreated hwiki, had a bad JohnTest account. Central auth now works fully
- 05:15 johnduhart: Running update.php on hiwiki
- 05:14 johnduhart: Importing prefstats table to hiwiki
- 04:41 johnduhart: Centralauth is working :)
- 04:28 johnduhart: Imported db schema to centralauth
- 04:25 johnduhart: Created centralauth database centralauth
- 04:18 johnduhart: Created metawiki http://meta.wikimedia.deployment.wmflabs.org/
- 04:18 johnduhart: http://meta.wikimedia.deployment.wmflabs.org/
- 04:09 johnduhart: Created meta docroot and added favicons to meta and wp
- 04:08 johnduhart: Adding remnant.conf to httpd.conf
- 04:07 johnduhart: Adding metawiki apache config to /usr/local/apache/conf/remnant.conf
- 04:03 johnduhart: Lot of configuration done, much tuning to come. Missing.php now works http://nope.wikipedia.deployment.wmflabs.org/wiki/Main_Page
- 01:24 johnduhart: Installed mysql-client on deployment-test
January 8
- 23:45 johnduhart: Installed php5-cli on deployment-test
- 23:21 johnduhart: Enabled rewrite rules on deployment-test
- 20:39 petan: created new group for /mnt/www
- 19:33 petan: clearing cache
- 18:47 hexmode: set up relative paths for config files
- 16:45 petan: moved memcached to deployment-nfs-memc and update global
- 14:04 petan: created temporary memcached instance for data
- 12:17 petan: restarted apache to fix some problem with config
- 00:11 hexmode: set config for timeline
- 00:11 hexmode: "apt-get install ploticus ttf-freefont" for timeline
- 00:04 jeremyb: booting memcached fixed it
- 00:02 jeremyb: booting memcache in case that fixes interwiki
- 00:02 jeremyb: [auth] added bugzilla to interwiki.sql and ran it, doesn't seem to be working
January 7
- 23:55 hexmode: Fix path for PoolCounter: PoolCounter.php -> PoolCounterClient.php
- 23:52 hexmode: Fix path for OAI: OAI.php -> OAIRepo.php
- 23:31 hexmode: set up git for /var/www/global
- 23:08 petan: disabled LQT on other wikis
- 23:07 petan: changed configuration of ajax
- 23:07 petan: fixed InitialiseSetting
- 22:42 petan: updated all wikis to latest head
- 22:36 petan: disabled LQT because it's broken
- 22:34 petan: ran update on auth db
- 22:10 petan: removed puppetized memcached, because its configuration suck
January 5
- 20:26 petan: importing MW ns full history to en_wikipedia
- 16:48 petan|w: created instance for dumps, current instance is overloaded
- 15:53 petan|w: import now running using mwimport
January 4
- 20:27 petan: opened port 80 for wide net
- 20:26 petan: registered deployment.wmflabs.org
- 20:24 petan: allocated ip208.80.153.215
- 20:22 mutante: raised floating IP quota to 1
- 16:10 petan|work: create instances for apache and mysql
- 16:08 petan|work: configured firewall for webserver
- 16:06 mutante: added members MarkAHershberger & Petrb - added them to sysadmin and netadmin roles
- 16:02 mutante: added new project deployment-prep for hexmode and petan
- 16:02 mutante: added new project deployment-prep for hexmode and petan