Maemo Wiki Mirror - User contributions [en]

Migrating to Community-driven Infrastructure

2013-05-03T20:46:26Z

93.200.111.117:

== Introduction ==
[up to date as of 2013-02-08]

Albeit Nokia's plans about discontinuation of maemo support been known since spring 2012, Nokia gave "Go" to Nemein (service provider on behalf of Nokia) for the real migration work not earlier than 2 weeks before christmas 2012.

As of January, 18th 2013 the *.maemo.org infrastructure has been consolidated from a 20+ physical servers (aka "irons") to current config and completely migrated to new locations independant of Nokia servers. This task been accomplished by Nemein. Talk.maemo.org forum been integrated with the other infra, many thanks to Nemein for donating the VM for that. Also many thanks to Nemein for this incredible piece of work done during times when others (as well as the guys there) usually are already away for winter holidays.

The current setup (see below) consists of around 10 Virtual Machines hosted by Nemein on their xen-grid. This is an interim solution. Nokia paid Nemein for this consolidation/migration and hosting until end of February.

Handing over control of servers still pending, right now (2013-01-30) it's still Nemein and affiliates to control that infra.

Transfer of control over (*.)maemo.org DNS entries ("the domain") is still being negotiated between Nokia and HiFo, all DNS changes done so far been done by Nokia dnsmaster on Nemein's request

The plans of council and HiFo board so far are: kindly ask Nemein to have *.maemo.org nicely bundled. We hope for this setup to be free of major known bugs (I.E. autobuilder working, repository working albeit maybe slow) when Nemein hands us the package.

[2013-02-08] Negotiations about direct migration to one of our 3 options (see below) are ongoing.

===further plans, state of migration===
Further plans are to migrate again to some hosted root servers, either on a sponsor like http://osuosl.org/about-osuosl or to our own stuff we may rent from e.g. Hetzner.

[2013-02-08] currently we're in negotiations about 3 possible ways into future hosting:
* osuosl (could provide VM or rootservers or CoLo [UPS server shipping: 48h:1200EUR, 7d:630EUR, +customs])
* IPHH, a ISP in Hamburg. Falk contacted them and they are willing to offer CoLo basically free of charge. Of course we will put their name on our maemo.org frontpage to give due credit. HW service will be done by Falk. (costs ~300EUR for setup and HW upgrade, plus 50..300EUR for shipping the iron to Hamburg)
* get own paid rootservers, like 2 of http://www.hetzner.de/en/hosting/produkte_rootserver/ex10 (costs ~300EUR/month, 400EUR setup)
Depending on option chosen, we might or might not keep the SuperMicro.

[2013-02-17] Hildon Foundation board has agreed on following IPHH if the contract is good and keeping OSUOSL open as an alternative. Sending iron to IPHH on 2013-02-18/19, and also negotiating with OSUOSL about how a possible migration to them would look like so we get a decent checklist in case we need it.

Falk's mail forwarded form IPHH to HiFo:
<pre>
Hi everyone,

these are the details, what IPHH is willing to offer us.

Best regards,

Falk

Begin forwarded message:

> From: Rene Sasse <support@iphh.xxx>
> Subject: [IPHH #442659] Re: maemo.org
> Date: 18. Februar 2013 11:08:33 MEZ
> To: falk@fourecks.xxx
> Cc: joerg@openmoko.xxx
> Reply-To: support@iphh.xxx
>
> Falk,
>
> IPHH offers the following services to Hildon Foundation for one year free of
> charge:
>
> * Colocation/electricity for the following devices:
> - 1 Server (2RU)
> - 1 Switch (1RU)
> * 1 100MBit/s Uplink Port
> * A /27 IPv4 Network
>
> This offer is valid for one year and has to be discussed for renewal after 11
> month.
>
> Legal Contact will be:
>
> Hildon Foundation
> 120 West 10th Street, Erie, PA, 16501, USA
>
> Technical Contact will be:
>
> Falk Stern (FS7182-RIPE)
> Rathmann-Cohrs-Straße 12, 21357 Bardowick, Germany
> Mobile: +49-160-71560xx
>
>
> best regards
> Rene
>
> --
> Rene Sasse E-Mail: support@iphh.xxx
> Technical Consultant Tel: +49 (0)40 374919-xx
> IPHH Internet Port Hamburg GmbH Fax: +49 (0)40 374919-xx
> Wendenstrasse 408 AG Hamburg, HRB 76071
> D-20537 Hamburg Geschaeftsfuehrung: Axel G. Kroeger--7E94C7404EC25FD69CC85C3653348297
>
</pre>

Iron to move form: ( http://nemein.com/fi/ )
Nemein Oy
tel. +358 20-198 6030
Vilhonvuorenkatu 11 D, 8 krs
00500 Helsinki, FINLAND
FIN-1647219-2
support AT nemein.com

to ( http://www.iphh.net/en/contact.html )
IPHH Internet Port Hamburg GmbH
#444615
Wendenstrasse 408
20537 Hamburg
Germany
T : +49 40 37 49 19-0
F : +49 40 37 49 19-29
E : info@iphh.net

Package details:
size
x: 100cm
y: 66cm
z: 28cm
weight: ~40kg

Shipping accomplished:
Shipment number 1139212793
Status from Wed, 20.02.2013 10:57 hours Delivered - signed for by Herr POLROK*
Recipient TPHH
Delivered on Herr POLROK*
via DHL account provided by Nokia/Pekka (many thanks!) on 2013-02-19.
Courtesy Aslan and Eero of Nemein.

Hosting migration timing plan:
https://docs.google.com/spreadsheet/ccc?key=0AuQnrvWRbTtzdFhERzlDbEp0eVNQQTdfNWpQbzdIT0E&usp=sharing

Alternatives - however obvious - for the above plans have been discussed with Nemein and HiFo and are not feasible. E.G. there was no way we could get the money instead of the server iron hardware. Sustaining the current xen-grid based VM hosting would be ~1500EUR per month plus a basically not evadable 2200EUR on top for maintenace. We want to switch away from that by all means, thus the 2nd migration.

This page is intended as a central place where status and other operational information can be gathered.

=== Plan for migration / Timeline [2013-03-15]===

* Friday, 22.2. (falk)
** Rack Hardware @ IPHH - Hardware is racked
** Install base system (CentOS 6.3 with patches from xes)

* Saturday, 23.2. (xes/falk)
** Start migrating repository.m.o
** Start migrating VMs with static data

* ... (hidden DNS master set up)
** sync databases, switch DNS entries
** DNS switched [Nokia] to new IPs on 2013-03-14 1700UTC. Final sync established 1900. since then machines up and running on *new*

VMs we need to migrate:

{|
! Name !! Disk Size !! Location of act. instance !! _migrated? !! _Comments on *new* instance
|+
| static || 30G || nemein || synced+up || works
|+
| wiki || 20G || nemein || synced+up || works
|+
| repository || 900G || nemein || synced+up || We need to check the disk size, this might be too big for current hw, maybe split tablets-dev off.
|+
| mail || 20G || nemein || synced+up || also has lists
|+
| scratchbox || 100G || iphh || setup! || will be setup new
|+
| vcs || 50G || nemein || synced+up || has NFS mounts from garage and repository (copying)
|+
| garage || 100G || nemein || synced+up || has NFS mounts from stage and vcs (copied, seems to work)
|+
| db || 100G || nemein || synced+up || works, needs tuning
|+
| builder || 50G || nemein || copied+up || still needs fixing several aspects
|+
| talk || 20G || nemein || synced+up || up since 2013-03-13, via HTTP-forward
|+
| dns || ?? || ipph || setup! || dns records/serial incomplete, bind inactive
|}

=== State of final migration ===

* talk.maemo.org is running on "new" hardware
* DNS switchover should happened at 14.3., around 17:00UTC
* all VM got synced and are up and running. *old* should be out of business when DNS change propagated

=== Setup with IPHH ===

==== Networks ====

We have 2 /28 Subnets (213.128.137.0/28 and 213.128.137.16/28)

Networks are configured as follows:

{|
! IPv4 !! IPv6 !! VLAN !! Xen Bridge !! default GW
|+
| 213.128.137.0/28 || not yet || 1 || xenbr0 || 213.128.137.14
|+
| 213.128.137.16/28 || not yet || 2 || xenbr1 || 213.128.137.17
|+
| 10.0.1.0/24 || not yet || 3 || xenbr2 || 10.0.1.1
|}

IP Plan for vlan 1

{|
! IPv4 !! IPv6 !! Hostname
|+
| 213.128.137.1 || n/a || firewall-carp
|+
| 213.128.137.2 || n/a || firewall-a
|+
| 213.128.137.3 || n/a || firewall-b
|+
| 213.128.137.4 || n/a || blade-a
|+
| 213.128.137.5 || n/a || blade-b
|+
| 213.128.137.6 || n/a || portforwarding for monitor
|+
| 213.128.137.7 || n/a ||
|+
| 213.128.137.8 || n/a ||
|+
| 213.128.137.9 || n/a ||
|+
| 213.128.137.10 || n/a ||
|+
| 213.128.137.11 || n/a ||
|+
| 213.128.137.12 || n/a || IPHH Router 1
|+
| 213.128.137.13 || n/a || IPHH Router 2
|+
| 213.128.137.14 || n/a || IPHH-VRRP
|}

IP Plan for vlan 2

{|
! IPv4 !! IPv6 !! Hostname !! Aliases
|+
| 213.128.137.17 || n/a || firewall-carp || -
|+
| 213.128.137.18 || n/a || firewall-a || -
|+
| 213.128.137.19 || n/a || firewall-b || -
|+
| 213.128.137.20 || n/a || www || static, maemo.org, planet, downloads
|+
| 213.128.137.21 || n/a || wiki || bugs
|+
| 213.128.137.22 || n/a || repository || stage
|+
| 213.128.137.23 || n/a || mail || lists
|+
| 213.128.137.24 || n/a || scratchbox || -
|+
| 213.128.137.25 || n/a || vcs || drop
|+
| 213.128.137.26 || n/a || garage || -
|+
| 213.128.137.27 || n/a || builder || -
|+
| 213.128.137.28 || n/a || talk || -
|+
| 213.128.137.29 || n/a || DNS || -
|+
| 213.128.137.30 || n/a || - || -
|}

IP Plan for vlan 3

{|
! IPv4 !! IPv6 !! Hostname
|+
| 10.0.1.1 || n/a || firewall-carp
|+
| 10.0.1.2 || n/a || firewall-a
|+
| 10.0.1.3 || n/a || firewall-b
|+
| 10.0.1.10 || n/a || db
|+
| 10.0.1.11 || n/a || monitor
|+
| 10.0.1.200 || n/a || blade-a/IPMI
|+
| 10.0.1.201 || n/a || blade-b/IPMI
|+
| 10.0.1.202 || n/a || maemo-switch
|}

==== Disk Layout of blade-[ab] ====

Both disks have the following partitioning:

RAID1 Volume for /boot (/dev/md0), consisting of /dev/sda1 and /dev/sdb1 (200M)

RAID1 Volume /dev/md1 consisting of /dev/sda2 and /dev/sdb2 (around 970G)
The RAID1 Volume contains a physical LVM volume.
We only have one VolumeGroup (vg_blade[ab]), which has LogVol00 with 20G as root volume, LogVol01 with 2 Gig as swap and vmstore with the rest as VM Storage mounted on /vmstore.

==== Tips & Tricks for migration ====

Copying:

Create an image on vmhost
<pre>fallocate -l 200g image.img</pre>
or, in case fallocate is unavailable
<pre>dd if=/dev/zero of=image.img bs=1 count=1 seek=200G</pre>

Attach as loop-device
<pre>losetup -f image.img</pre> (find the loop-device and create a filesystem on it)

Copy stuff
<pre>tar --create -p -j --one-file-system . | pv -br | ssh root@host 'cd /mountpoint ; tar xpj '</pre> or
<pre>cd / ; rsync -arvSxz . root@host:/mount/point</pre>

==== Stuff to do [2013-03-15] ====

* Implement a proper service monitoring for all machines and applications - nagios pending, http://monitor.maemo.org/ganglia/
* Setup a common policy for root/user accounts and sudo permissions
* Change root-passwords - done
* Make SSH root-login key-only - done?
* Find out, what to sync for final migration - done
* Configure internal DNS server in /etc/resolv.conf
* Coordinate DNS setup with Nokia - partially done
* Consolidate Databases - WIP
* Add disks to system - done, 4TB on blade-a
* Setup bugtracking system for infrastructure - done: roundup?
* fix NFS mounts - WIP
* update VMs to 3.2.0-38

==== Problems we walked into ====

===== Machines throwing their network away =====

Apparently, XEN has issues if a vm sends too many/too large network packets.

http://lists.xen.org/archives/html/xen-devel/2013-01/msg00198.html
has an interesting read about that problem.

Symptom:

xenbr1: port 8(vif51.0) entered forwarding state
vif vif-51-0 vif51.0: Too many frags
vif vif-51-0 vif51.0: fatal error; disabling device
xenbr1: port 8(vif51.0) entered disabled state

in dmesg

Temporary fix: Disable all offloading on eth0

for i in rx tx sg tso gso gro lro; do
ethtool -K eth0 $i off
done

Source of this problem:

https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/include/linux/skbuff.h?h=v3.3.1&id=9d4dde5215779f4099730194ad30624fdba3d8b2

We fixed that problem on our machines by ensuring dom0 and domU use same MAX_SKB_FRAGS

== Inventory (obsolete, please update) ==

As a first step we try to gather information about the present infrastructure at *.maemo.org. This "inventory" is intended to provide an overview about all components of the infrastructure as well as to provide information that will later on aid during the actual migration.

Currently the following topics are considered important for the migration:

* Legal Issues (Names, Trademarks, Domain Names, etc.)
* Infrastructure (Web Site, Forum, Wiki, Autobuilder, Mailinglists, Garage, etc.)

== Legal Issues ==

=== What is the state about the name "Maemo"? ===
"... Maemo is currently a registered trademark of Nokia and the domain name is owned by Nokia.

=== Who owns "maemo.org"? ===
Negotiations about domain ownership still ongoing between Hildon Foundation board and Nokia (2013-01-20), if community can't get control over the DNS, we might revert to maemocommunity.org.

Domain ID:D105692361-LROR 
Domain Name:MAEMO.ORG 
Created On:07-Feb-2005 16:26:32 UTC 
Last Updated On:07-Jan-2013 10:25:55 UTC 
Expiration Date:07-Feb-2014 16:26:32 UTC 
Sponsoring Registrar:MarkMonitor Inc. (R37-LROR) 
Registrant ID:mmr-31461 
Registrant Name:Nokia Corporation 
Registrant Organization:Nokia Corporation 
Registrant Street1:P.O.Box 226 
Registrant Street2:Nokia Group 
Registrant Postal Code:00045 
Registrant Country:FI 
Registrant Phone:+358.718008000 
Registrant FAX:+358.718034496 
Registrant Email:dnsauthority@nokia.com 

We're planning to ask Nokia to allow a hidden primary [http://fengnet.com/book/DNS.and.BIND.5th.Edition/dns5-CHP-16-sect-1.html] for maemo.org, that we will host on a persistent VM (dns) sponsored by Nemein (thanks Eero! :-D ). The purpose is to allow swift changes of IPs under maemo.org without bothering Nokia's DNSmaster, as long as the domain still belongs to Nokia. Once the domain will get transferred to HiFo, this will become less useful but also not exactly any problem. in 6 months or so we can consider tearing down the hidden primary and manage our domain directly.

== What is needed for the community to run maemo.org? ==
TMO forums donated to Hildon Foundation: http://maemo.org/community/board/tmo_forums_donated_to_hildon_foundation/

== What are the costs? ==
Nokia paid for hosting until end of February.
Current (2013-01-30) interim config (VM on Nemein's xen-grid) will cost 1300EUR/month for the VM, plus 2200EUR/month for the maintenance. For the colocation rackspace, traffic, energy etc of the iron(s) Nokia donates to community there will be another 500+EUR/month. All excl VAT.

At end of February we hope to drop the xen-grid VM since they shall run in a virtualization on our iron by then.

If you're willing to donate, please visit http://hildonfoundation.org/support/

== What about the personal information of the users? ==
Please refer to the privacy policy posted on the website. If you want info about what's the data stored about you inside *maemo.org, or want this data / your account getting permanently deleted, please contact council@maemo.org

== Operational Platform ==
[2013-03-20] All of maemo.org is running on our supermicro server colocated at IPHH

List of hardware Nokia will donate to HiFo, according to Nemein's plans. [2013-02-08]

{| class="sortable wikitable" border="2"
|-
! ID
! Hostname
! Mgmt IP Address
! OOB Mgmt IP Address
! Type (Virtual / Baremetal)
! System Admin
! HW Vendor
! HW Model
! Form Factor
! CPU
! Memory
! Disk
! Acquisition Date
! Warranty
! Services
! Comment
|-
| 01
| blade-a.maemo.org
|
|
| Baremetal
| Falk(warfare)
| Supermicro
| http://www.supermicro.nl/products/system/2u/2027/SYS-2027TR-HTRF.cfm?parts=SHOW
| 2U 19" Rackmount
| Intel® Xeon® processor E5-2620
| 32GB
| (raid1:2*)1TB, 2*2TB=4TB aux.
|
| 3 years
| Falk (for HH CoLo)
| only 2 of the 4 blades populated
|-
| 02
| blade-b.maemo.org
|
|
| Baremetal
|
|
|
|
| Intel® Xeon® processor E5-2620
| 32GB
| (raid1:2*)1TB
|
|
|
|}

=== OS and virtulization on community iron (planning, discussion) ===
Please don't forget to tag your contributions with your nick!
====Server OS====
===== alternative A =====
blabla-OS
===== alternative B =====
===== alternative C =====

====Virtualization====
===== alternative A =====
XEN (with OS blabla of above)
===== alternative B =====
VMware
===== alternative C =====

=== Services ===
The following table is intended to give a concise and easily perceivable overview of the *.maemo.org services. Please use the next sub-section for providing more detailed information.

{| class="sortable wikitable" border="2"
|-
! Resource
! URL (If Applicable)
! Migration Status (DONE/WIP/NST)
! Service Maintainer
! System Admin
! Software Name
! Software Version
! Software License
! Known Issues
! Last status update
|-
| Maemo Main Web Site
| http://www.maemo.org
| style="background-color:yellow;" |1 BUGS
| ?
| Nemein
|
|
|
| orphaned links/404s: http://maemo.org/community/council/system_operator_needed/; '''Login doesn't work'''
| 2013-01-25
|-
| Maemo Forums
| http://talk.maemo.org
| style="background-color:lime;" |1 DONE
| [[User:chemist|chemist]], Reggie
| Falk, [[User:chemist|chemist]]
| vBulletin
|
| Unlimited duration, no upgrades included, acquired on 2012-20-12
| Captcha image issues
| 2013-02-10
|-
| Maemo Wiki
| http://wiki.maemo.org
| style="background-color:yellow;" |1 BUGS
| ?
| Nemein
|
|
|
| '''(Watch) Email not working'''; random connection timeouts
| 2013-01-25
|-
| Repositories
| http://repository.maemo.org
| style="background-color:yellow;" |1 BUGS
| X-Fade, Merlin1981
| Nemein
|
|
|
| former akamai serverfarm, now points to stage.m.o VM master of farm. '''Hashsum errors legacy'''
| 2013-02-20
|-
| Blog aggregator
| http://planet.maemo.org
| style="background-color:lime;" |1 DONE
| ?
| Nemein
|
|
|
|login flawed?
| 2013-02-10
|-
| Maemo Garage
| https://garage.maemo.org/
| style="background-color:lime;" |1 DONE
| ?, Woody
| Nemein
|
|
|
|
| 2013-01-25
|-
| Maemo Autobuilder
|
| style="background-color:red;" |1 NST
| X-Fade
| Nemein
|
|
|
| '''OFFLINE''', x-fade working on it
| 2013-02-20
|-
| Maemo Nameservers
|
| style="background-color:yellow;" |1 WIP
| Merlin, Falk
| Nokia
|
|
|
| Still using Nokia Nameservers; following hidden primary plan til domain transfer to HiFo established
| 2013-01-25
|-
| Drop
| http://drop.maemo.org
| style="background-color:yellow;" |1 WIP
| X-Fade
| Nemein
|
|
|
|
| 2013-02-10
|-
| VCS
| http://vcs.maemo.org
| style="background-color:yellow;" |1 WIP
|
| Nemein
|
|
|
|
| 2013-02-10
|-
| Listserv
| https://lists.maemo.org
| style="background-color:yellow;" |1 BUGS
|
| Nemein
|
|
|
| '''occasional lockups resp interface down'''
| 2013-02-20
|-
| Static
| http://static.maemo.org
| style="background-color:yellow;" |1 WIP
|
| Nemein
|
|
|
| '''temporary fix via NAT port81 redir''', instable?
| 2013-02-20
|-
| Stage
| http://stage.maemo.org
| obsolete
| X-Fade
| Nemein
|
|
|
| VM got assigned to repository.m.o
| 2013-02-20
|-
| Bugs
| http://bugs.maemo.org
| style="background-color:lime;" |1 DONE
| Andre
| Nemein
|
|
|
| -
| 2013-01-25
|-
| Scratchbox
| http://scratchbox.org/
| style="background-color:yellow;" |1 WIP
| thedead1440
| Nemein, thedead1440
|
|
|
| 80.248.164.245, Logica Finland Oy, migration pending
| 2013-02-20
|-
| Voting Infrastructure
| ?
| style="background-color:yellow;" |1 WIP
| woody14619
|
|
|
| ?
| ?
| 2013-02-20
|}

=== More Detailed Information ===
In this sub section more detailed information about the entries in the table can be placed. The intent is to keep the table concise while still being able to have all relevant information at hand.

List of VMs and their associated IPs:
IP adresses
188.117.59.198 test.maemo.org
# www.maemo.org maemo.org
188.117.59.200 www.maemo.org
188.117.59.200 planet.maemo.org
188.117.59.200 static.maemo.org
188.117.59.199 drop.maemo.org
188.117.59.207 garage.maemo.org
188.117.59.204 lists.maemo.org
188.117.59.202 wiki.maemo.org
188.117.59.212 bugs.maemo.org
# 188.117.59.203 repository.maemo.org scrubbed
188.117.59.205 stage.maemo.org repository.maemo.org (reassigned)
188.117.59.206 vcs.maemo.org

List of internal IP/VM
127.0.0.1 MaemoTemplate
10.0.0.1 maemo static maintenance
10.0.0.2 wiki bugs
10.0.0.121 stage repository
10.0.0.4 mail smtp lists
10.0.0.5 scratchbox
10.0.0.6 dns
#10.0.0.7 repository
10.0.0.9 vcs drop
10.0.0.10 garage
10.0.0.11 db backup
10.0.0.12 builder
10.0.0.254 fw

Cpu Cores, RAM (in MB), storage (DISK, in GB), of the VMs
<pre>
Current VMs actually in use (some more were reserved originally since it
was not certain what services could be merged)

Name C RAM DISK
------------------------
MaemoFW 1 1024 10
Builder 1 4096 150
garage 2 8192 100
test 2 2048 30
wikib 2 2048 50
www 2 6144 70
vcs 2 8192 200
db 2 8192 260
mail 2 2048 30
stage 2 2048 870
talk 2 4096 15
========================
20 48128 1785

sb 2 2048 30
dns 2 2048 30
========================
25 52224 1845
</pre>

==== Forum (talk.maemo.org) ====
Unlike the other services, talk.maemo.org is not behind the endian firewall. Maintenence access is not via test jumpserver.

Software: vBulletin
licence: Unlimited duration, no upgrades included, acquired on 2012-20-12

=== Scratchbox ===
Scratchbox is also sponsored by Nokia. (Please verify?)
Scratchbox is required for running the Fremantle and Harmattan SDK.

Currently there's a VM on Nemein's xen-grid named "scratchbox", but state of the case is unclear.

===Tracker for Sysops and Maintainers===
This tracker is meant for maemo staff and affiliated only
web frontend: roundup.fourecks.de/maemo/
mail access (read docs!): maemo-issue AT fourecks.de

===Service Maintainers (please update/augment/fix)===
(please don't usually pester maintainers directly! First try to contact council@maemo.org, we'll forward)

These are the Service Maintainers (in spe), for services like forum (tmo), wiki, bugs, etc. They are (generally) not sysops of the machines their service is running on.

{|
! From !! Nick !! Full Name !! E-Mail !! Services Maintained !! Status !! Comments
|+
| Nemein || mashiara || Rambo Eero af Heurlin || eero.afheurlin at <to be disclosed by owner> || (sysop) || || [leaving?] ||
|+
| Nemein || x-fade || Niels Breet || Niels<at>maemo.org || (mail, IRC, builder, ???...) || || [leaving?] ||
|+
| Nemein || ferenc || Ferenc Szekely || ferenc<at>maemo.org || (mail, sysop, ???...) || || [leaving?] ||
|+
| maemo || warfare || Falk Stern || falk<at>fourecks.de || (maemo master sysop) || ||
|+
| maemo || chemist || Ruediger Schiller || webmaster<at>talk.m.o || Talk || || ||
|+
| maemo || merlin1991 || Christian Ratzenhofer || <at> || Repos || || [preliminary accepted] ||
|+
| ??? || andre_ || Andre Klapper || ???<at>??? || Bugs || || [???] ||
|+
| || || || || ??? (wiki) || ||
|+
| || || || || (planet???) || ||
|}

== Unsorted Hints ==
=== ssh access ===
All legacy accounts got ported to new infra.

Access to any VM is via plain direct ssh:
ssh <user>@<VM>.maemo.org

=== backup ===
we're doing backups to the 4TB auxiliary storage on blade-a, using backupPC:
ssh -L8088:localhost:80 blade-a
konqueror http://localhost:8088
backup-master is Falk

talk VM sysop (chem|st) has access to it and control over own backups, via ssh confic on blade-a:
command="sleep 1d",permitopen="127.0.0.1:80" <ssh-pubkey>

== Steering ==

council is in charge of any steering.

Joerg Reisenweber got appointed for "maemo.org infra administration coordinator" and thus is the single point of coordination for any detail questions.

If you got any questions, suggestions, critics, whatever, please contact Joerg (DocScrutinizer) or any other of council members via IRC. or send a mail to council AT maemo.org. We're just community's proxies acting in best intention to do what's probably community's best interest. If you don't agree with what we do or have suggestions how we could do better, please holler. Best place: Friday 1800UTC IRC:(freenode.net)#maemo-meeting

== More ==

* OBS @ TiZen or SuSe : https://bugs.tizen.org/jira/browse/TINF-48?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

== Autobuilder and friends ==

maemo autobuilder setup

autobuilder consists of multiple VMs

=== drop VM ===
this VM has /etc/passwd synchronised with garage and ~ folders mounted via NFS from garage

account synchronisation is handled by scripts running on garage VM and then sync is triggered using ssh and scripts in /usr/local/bin

packages are uploaded to /mnt/incoming-builder via SCP

=== garage VM ===
this is the VM where stuff happens

password/account sync to gforge/postgresql is done using
*/10 * * * * root /usr/local/bin/add_groups_users_git_ssh.sh > /tmp/add_groups_users_git_ssh.log dev/null 2>&1
this also updates ~/.ssh/authorized_keys

garage also handles web extras-uploader (/var/lib/extras-assistant/) - package is uploaded and then moved to the same folder as packages uploaded to drop and then chowned using

/var/lib/extras-assistant/bin/copy_package_files_to_autobuilder.sh

A lot of jobs on garage VM is done using local root crontab (/var/spool/cron/crontabs/root)

after package is uploaded it's processed by buildME

buildME runs as builder user and it's started from cron every minute
* * * * * builder /home/builder/buildme

buildme is configured using /etc/buildme.conf

buildme takes care of couple things
* verify that .tar.gz and other files are correct (checked using checksum from .dsc file)
* select free destination (buildme can handle parallel builds on multiple hosts/users)
* scp all required files to selected destination
* start sbdmock on the destination
* copy results back and resulting .deb to repository incoming folder (result_dir = /mnt/builder/%(product)s and repo_queue = /mnt/incoming/extras-devel/%(product)s/)
* send emails to list and user uploading package

=== builder VM ===

this VM has standard installation of scratchbox with no targets configured (it's not required for sbdmock)

when sbdmock is started it cleans up old build folder, creates new target and prepares build enviroment and then runs dpkg-buildpackage

sbdmock also generates logfiles that are parsed by buildme

=== repository/stage VM ===

this is where repository management happens
*/2 * * * * repository /home/repository/queue-manage-extras-devel.sh
*/5 * * * * repository /home/repository/queue-manage-extras.sh
*/5 * * * * repository /home/repository/queue-manage-community-testing.sh
*/5 * * * * repository /home/repository/queue-manage-community.sh

those scripts (and scripts inside /home/repository/queue-manager-extras) check for new packages in repository incoming folder and then move those to /var/repository/staging, regenerate Packages

(using sums that were previously cached) and sign it if required and then if any changes happened
#touch .changed file, so we know that we need to sync to live
touch /var/repository/staging/community/.$dist.changed

this file is then checked by
1003 10634 1 0 Mar18 ? 00:00:00 /bin/sh /usr/local/bin/packages/rqp.sh
started by /etc/init.d/repository-qp

this script starts rsync when required to sync to live repository

this script also starts repository-queue-proc.php that processes repository updates coming from midgard (old package cleanup and promotions)

Maemo.org team/How Elections Work

2013-05-03T20:45:19Z

93.200.111.117:

This page details the current [http://maemo.org/vote voting system] and how it works. It is presented as a means of offering transparency into how voting is handled and how the current system provides both anonymity and verifiability in community elections and referendums.

= Overview =

The voting system provides voters with a multiple-choice interface which can be used to create an ordered list of selected options. Voters can select some, none, or all of the choices, and order those choices as they prefer them.

For multi-seat elections, voters may choose any number of candidates, be that greater than, equal to, or less than the number of seats (see Single Transferable Vote below). For the special case of a two-choice referendum, voters should be aware that there is no (additional) sense in choosing both options, though the voting system may in fact allow them to do so. ''Only the first selection in a 2 option election will be counted.''

Voters that dislike the choices, or want a choice not presented, may show their desire to participate by choosing nothing and casting an empty ballot. If enough voters cast empty ballots it can affect an election by making fewer (or no) candidates reach the required "quorum" levels to be elected.

= Parts of an Election =

The current voting process is divided into several pieces. Some of these are not visible to the voter, or are taken as a given. To ease the process of explaining the process, we will follow the "life" of a single vote, from the creation of the election to the final results.

== Elections ==

Elections are created by an administrator (often Council) and stored in the database with several vital fields about that election. Included are the start and stop times, the title of the election, text about the election, candidates for the election, and the number of "winners". The contents of these tables generally show up in one form or another in the interface for voters.

== Electorate ==

The system also contains an electorate list; the list of ''users eligible to vote'' in an election. That list generally gets updated from the main maemo.org list before elections are started. The date this happens is the relevant date for calculating karma, account age, and so on. Only users that meet the conditions ''at that date'' are included to the electorate table. It is '''difficult''' (but not impossible) to update or adjust this list after elections have started. To avoid complexity it is asked that voters check their account settings, '''particularly their email address,''' and setup any linkages needed at least a few weeks before elections start.

== Voting Tokens ==

When an election is setup and ready to be used, and the electorate is verified, an administrator instructs the system to create and mail tokens for the electorate. A script in the system then generates a random voting token for every member of the electorate, and mails that token to each voter. The voting token, used as a password for the voting login screen, is valid for one vote in the election for a given voter. Administrators never see these tokens (aside from any token they may get as a voter).

The tokens are stored in a separate table, with a numeric pointer to the voter and election for which it is to be used. Storing them this way, in separate tables, makes it harder for an administrator to "accidentally" see a voter's e-mail and token together if they need to look at the database for any reason. The odds of anyone guessing a token is also quite slim, and both the token and e-mail address is required to log in to vote.

== Voting Process ==

The entire voting process is state driven. When a voter chooses an election and logs in, the system validates the token to allow them into the voting interface. This is a "courtesy validation" to allow the voter to know that the token is still valid. Nothing is altered by this step.

The list of choices is presented to the voter, and stored in a ''javascript applet'' in the voters browser that allows the voter to chose as many options as they like, in the order they like (note that you can undo choices by clicking on the list of chosen items). When the voter is happy with their choice(s) and proper order, they submit the choices for validation and confirmation. The vote selection is validated, displayed to the voter to confirm that the system got their choice(s) and order correct, and the voter is asked to submit the vote for final processing.

The next step is the most complex and critical. Once the vote has been submitted, several things happen all in one single action:
# A random vote validation string is created for this vote.
# The vote is prepared for entry into the list of votes with this random string.
# The system atomically (as one step):
## Validates that the vote token is still valid
## Adds the vote to the list of votes
## Deletes the valid vote token

At this stage, one of two things happens:
* If an error occurs, all sub-steps are undone, leaving a valid vote token, and an error is reported to the voter.
* If all works as expected, the random vote validation string is presented to the voter.

Lets look at those sub-steps a little closer:

The first sub-step generates a random validation string for your vote. When the vote is stored in the vote table, that is the *only* reference as to "who" cast this vote. Being that it's random, the only way one can know it is by seeing it on their screen, just after submitting their vote. This is important because '''it allows each voter to validate the public ballot later''' by looking up their unique entry to verify their vote was counted as they cast it.

The third step guarantees that a voter always has either a vote token, or has a vote cast with a random validation string. This prevents multiple votes per voter, and prevents technical problems from leaving a voter stranded without a vote. For this reason, '''at any time up to the point one hits the "submit vote" button on the validation page, and gets a validation token random string displayed, one may close their browser, or hit back, to change their choices, or start all over anew with the process.'''

== Post Election Ballot Validation ==

Once the vote has closed, the ballot of all votes cast is ''publicly'' viewable (and downloadable) by everyone. The list includes the votes cast, including the order of the choices for multi-choice votes, and the random validation strings for each vote. Since each voter knows his or her validation string, they can validate their vote is correct. They can also count the votes, and validate that the ''results'' of the election are correct. But since only one random validation string is known per voter, voters (as well as admins) can not determine who voted for which candidates in the election. Since there are no links to the original owner or starting token in the final voting table, there's also no way for an administrator (or anyone with database access) to determine who voted for which candidates either.

= Technical Details =

This area is for a more technical look at how things work, and may be less interesting for the general voter.

== Source Code ==

For those interested, the voting code is public domain and can be reviewed. At this point, the entire voting system consists of PHP code, a small piece of javascript that runs in the browser, and a perl script that handles e-mail delivery. Since the addition of the admin interface, even the administration functions are now in PHP. The OpenSTV portion is written in perl, and is also opensource, though there is a requested donation (which has been made on behalf of Maemo for our use of it).

== Database Setup ==

The database schema for the voting database is also checked in as an SQL script that can be used to create all tables needed for the scripts to function. Any updates to tables should be included in this schema.

=== Individual Tables ===

There are several tables in the election interface. Below is a brief description of each, as they exist at the time of writing this wiki page.

{| class="wikitable"
|-
! Table Name
! Description
|-
| admin_users
| This table contains the information needed to validate users of the administration interface.
|-
| elections
| This contains a list of all elections, past, present, and future.
|-
| election_anon_tokens
| This contains a list of all validation strings ever made, and indicates which election the token was made for, but not which voter.
|-
| election_choices
| This contains the choices for all elections, from Yes/No to candidates for elections.
|-
| election_results
| This contains html blobs that are shown as official election results.
|-
| election_tmp_tokens
| This contains the list of all tokens for all elections. Each token knows which voter and election it belongs to (via numeric index). Obsolete tokens (for elections that are closed) shall get deleted from this table.
|-
| election_votes
| This contains the actual votes for an election. The only references here are to election_anon_tokens, to indicate which votes go to which verification string.
|-
| electorate
| This contains the list of all eligible voters for all elections.
|-
| midgard_users
| This is a temporary table, imported from Midgard and/or the garage to allow scripts to update the electorate in an easier fashion. Creation date of this table determines the karma, account age etc used to decide eligibility of a user.
|-
| outbound_email
| This system handles queuing and batching mail to prevent us from looking like spammers. Temporary table. Depletes as mails get sent out.
|}

== Cloning Voting Tokens ==

One option available for administrators is to "clone" voting tokens from one election into another. This allows one "password" to be used in two or more elections. This process is often desirable when two elections are happening at the same time or in close succession, as it cuts down on the number of tokens mailed out, and eliminates confusion about which voting token string to use for a given election.

For example, the Coding Competitions often have 5 to 10 categories, each with a separate election. Having 5 to 10 tokens mailed to voters could cause spam triggers, and/or could confuse voters as to which token to use for which category. Instead, one token is created, and cloned to the other categories. The allows a voter to use the same e-mail and password to vote in all Coding Competition elections, but still only allows one vote per voter per category.

Cloning works because each token has a reference to the voter and election for which it exists. When a token is cloned a '''copy''' is made with a ''different'' election identifier. This means two tokens exist with the same random "string", but each points to the same user and a ''different'' election. When a voter casts a vote in one election, just the token for that election gets destroyed.

This is also the reason tokens can't be cloned after voting has opened on an election. Since voting tokens are destroyed as the vote is cast, cloning from an active election would mean those who already voted in the election would not have a token to copy, and thus would not be able to vote in the cloned election.

{| class="wikitable" border=1
|-
!
! align="center"| Voting Tokens
! align="center"| Votes & Verify Strings
|-
! Setup Referendum 1
|
'''(3UA9VTp,Alice,Ref1)'''
'''(oUB3VTq,Bob,Ref1)'''
'''(yUC9VTr,Chad,Ref1)'''
... [all eligible electorate]
|
|-
! Clone Referendum 1 to Election 2
|
(3UA9VTp,Alice,Ref1)
(oUB3VTq,Bob,Ref1)
(yUC9VTr,Chad,Ref1)
...
'''(3UA9VTp,Alice,Elect2)'''
'''(oUB3VTq,Bob,Elect2)'''
'''(yUC9VTr,Chad,Elect2)'''
...
|
|-
! Alice & Chad Vote In Referendum 1
|
<strike>(3UA9VTp,Alice,Ref1)</strike>
(oUB3VTq,Bob,Ref1)
<strike>(yUC9VTr,Chad,Ref1)</strike>
...
(3UA9VTp,Alice,Elect2)
(oUB3VTq,Bob,Elect2)
(yUC9VTr,Chad,Elect2)
...
|
'''"Yes" ⇒ OMGuVotedN0wg0prty (Ref1)'''
'''"No" ⇒ Yualw4ysSoN3gatlve (Ref1)'''
|-
! Alice & Bob Vote In Election 2
|
(oUB3VTq,Bob,Ref1)
...
(wUZ3VT5,Zarrah,Ref1)
<strike>(3UA9VTp,Alice,Elect2)</strike>
<strike>(oUB3VTq,Bob,Elect2)</strike>
(yUC9VTr,Chad,Elect2)
...
|
"Yes" ⇒ OMGuVotedN0wg0prty (Ref1)
"No" ⇒ Yualw4ysSoN3gatlve (Ref1)
'''"Kyle","Lucy" ⇒ M4nUVote4YurBudzEh (Elect2)'''
'''"Jim","Mandy" ⇒ OMGuVotedTo0g0prty (Elect2)'''
|}

Maemo.org team/How Elections Work

2013-05-03T20:44:46Z

93.200.111.117:

This page details the current [http://maemo.org/vote voting system] and how it works. It is presented as a means of offering transparency into how voting is handled and how the current system provides both anonymity and verifiability in community elections and referendums.

foobar (just testing)

= Overview =

The voting system provides voters with a multiple-choice interface which can be used to create an ordered list of selected options. Voters can select some, none, or all of the choices, and order those choices as they prefer them.

For multi-seat elections, voters may choose any number of candidates, be that greater than, equal to, or less than the number of seats (see Single Transferable Vote below). For the special case of a two-choice referendum, voters should be aware that there is no (additional) sense in choosing both options, though the voting system may in fact allow them to do so. ''Only the first selection in a 2 option election will be counted.''

Voters that dislike the choices, or want a choice not presented, may show their desire to participate by choosing nothing and casting an empty ballot. If enough voters cast empty ballots it can affect an election by making fewer (or no) candidates reach the required "quorum" levels to be elected.

= Parts of an Election =

The current voting process is divided into several pieces. Some of these are not visible to the voter, or are taken as a given. To ease the process of explaining the process, we will follow the "life" of a single vote, from the creation of the election to the final results.

== Elections ==

Elections are created by an administrator (often Council) and stored in the database with several vital fields about that election. Included are the start and stop times, the title of the election, text about the election, candidates for the election, and the number of "winners". The contents of these tables generally show up in one form or another in the interface for voters.

== Electorate ==

The system also contains an electorate list; the list of ''users eligible to vote'' in an election. That list generally gets updated from the main maemo.org list before elections are started. The date this happens is the relevant date for calculating karma, account age, and so on. Only users that meet the conditions ''at that date'' are included to the electorate table. It is '''difficult''' (but not impossible) to update or adjust this list after elections have started. To avoid complexity it is asked that voters check their account settings, '''particularly their email address,''' and setup any linkages needed at least a few weeks before elections start.

== Voting Tokens ==

When an election is setup and ready to be used, and the electorate is verified, an administrator instructs the system to create and mail tokens for the electorate. A script in the system then generates a random voting token for every member of the electorate, and mails that token to each voter. The voting token, used as a password for the voting login screen, is valid for one vote in the election for a given voter. Administrators never see these tokens (aside from any token they may get as a voter).

The tokens are stored in a separate table, with a numeric pointer to the voter and election for which it is to be used. Storing them this way, in separate tables, makes it harder for an administrator to "accidentally" see a voter's e-mail and token together if they need to look at the database for any reason. The odds of anyone guessing a token is also quite slim, and both the token and e-mail address is required to log in to vote.

== Voting Process ==

The entire voting process is state driven. When a voter chooses an election and logs in, the system validates the token to allow them into the voting interface. This is a "courtesy validation" to allow the voter to know that the token is still valid. Nothing is altered by this step.

The list of choices is presented to the voter, and stored in a ''javascript applet'' in the voters browser that allows the voter to chose as many options as they like, in the order they like (note that you can undo choices by clicking on the list of chosen items). When the voter is happy with their choice(s) and proper order, they submit the choices for validation and confirmation. The vote selection is validated, displayed to the voter to confirm that the system got their choice(s) and order correct, and the voter is asked to submit the vote for final processing.

The next step is the most complex and critical. Once the vote has been submitted, several things happen all in one single action:
# A random vote validation string is created for this vote.
# The vote is prepared for entry into the list of votes with this random string.
# The system atomically (as one step):
## Validates that the vote token is still valid
## Adds the vote to the list of votes
## Deletes the valid vote token

At this stage, one of two things happens:
* If an error occurs, all sub-steps are undone, leaving a valid vote token, and an error is reported to the voter.
* If all works as expected, the random vote validation string is presented to the voter.

Lets look at those sub-steps a little closer:

The first sub-step generates a random validation string for your vote. When the vote is stored in the vote table, that is the *only* reference as to "who" cast this vote. Being that it's random, the only way one can know it is by seeing it on their screen, just after submitting their vote. This is important because '''it allows each voter to validate the public ballot later''' by looking up their unique entry to verify their vote was counted as they cast it.

The third step guarantees that a voter always has either a vote token, or has a vote cast with a random validation string. This prevents multiple votes per voter, and prevents technical problems from leaving a voter stranded without a vote. For this reason, '''at any time up to the point one hits the "submit vote" button on the validation page, and gets a validation token random string displayed, one may close their browser, or hit back, to change their choices, or start all over anew with the process.'''

== Post Election Ballot Validation ==

Once the vote has closed, the ballot of all votes cast is ''publicly'' viewable (and downloadable) by everyone. The list includes the votes cast, including the order of the choices for multi-choice votes, and the random validation strings for each vote. Since each voter knows his or her validation string, they can validate their vote is correct. They can also count the votes, and validate that the ''results'' of the election are correct. But since only one random validation string is known per voter, voters (as well as admins) can not determine who voted for which candidates in the election. Since there are no links to the original owner or starting token in the final voting table, there's also no way for an administrator (or anyone with database access) to determine who voted for which candidates either.

= Technical Details =

This area is for a more technical look at how things work, and may be less interesting for the general voter.

== Source Code ==

For those interested, the voting code is public domain and can be reviewed. At this point, the entire voting system consists of PHP code, a small piece of javascript that runs in the browser, and a perl script that handles e-mail delivery. Since the addition of the admin interface, even the administration functions are now in PHP. The OpenSTV portion is written in perl, and is also opensource, though there is a requested donation (which has been made on behalf of Maemo for our use of it).

== Database Setup ==

The database schema for the voting database is also checked in as an SQL script that can be used to create all tables needed for the scripts to function. Any updates to tables should be included in this schema.

=== Individual Tables ===

There are several tables in the election interface. Below is a brief description of each, as they exist at the time of writing this wiki page.

{| class="wikitable"
|-
! Table Name
! Description
|-
| admin_users
| This table contains the information needed to validate users of the administration interface.
|-
| elections
| This contains a list of all elections, past, present, and future.
|-
| election_anon_tokens
| This contains a list of all validation strings ever made, and indicates which election the token was made for, but not which voter.
|-
| election_choices
| This contains the choices for all elections, from Yes/No to candidates for elections.
|-
| election_results
| This contains html blobs that are shown as official election results.
|-
| election_tmp_tokens
| This contains the list of all tokens for all elections. Each token knows which voter and election it belongs to (via numeric index). Obsolete tokens (for elections that are closed) shall get deleted from this table.
|-
| election_votes
| This contains the actual votes for an election. The only references here are to election_anon_tokens, to indicate which votes go to which verification string.
|-
| electorate
| This contains the list of all eligible voters for all elections.
|-
| midgard_users
| This is a temporary table, imported from Midgard and/or the garage to allow scripts to update the electorate in an easier fashion. Creation date of this table determines the karma, account age etc used to decide eligibility of a user.
|-
| outbound_email
| This system handles queuing and batching mail to prevent us from looking like spammers. Temporary table. Depletes as mails get sent out.
|}

== Cloning Voting Tokens ==

One option available for administrators is to "clone" voting tokens from one election into another. This allows one "password" to be used in two or more elections. This process is often desirable when two elections are happening at the same time or in close succession, as it cuts down on the number of tokens mailed out, and eliminates confusion about which voting token string to use for a given election.

For example, the Coding Competitions often have 5 to 10 categories, each with a separate election. Having 5 to 10 tokens mailed to voters could cause spam triggers, and/or could confuse voters as to which token to use for which category. Instead, one token is created, and cloned to the other categories. The allows a voter to use the same e-mail and password to vote in all Coding Competition elections, but still only allows one vote per voter per category.

Cloning works because each token has a reference to the voter and election for which it exists. When a token is cloned a '''copy''' is made with a ''different'' election identifier. This means two tokens exist with the same random "string", but each points to the same user and a ''different'' election. When a voter casts a vote in one election, just the token for that election gets destroyed.

This is also the reason tokens can't be cloned after voting has opened on an election. Since voting tokens are destroyed as the vote is cast, cloning from an active election would mean those who already voted in the election would not have a token to copy, and thus would not be able to vote in the cloned election.

{| class="wikitable" border=1
|-
!
! align="center"| Voting Tokens
! align="center"| Votes & Verify Strings
|-
! Setup Referendum 1
|
'''(3UA9VTp,Alice,Ref1)'''
'''(oUB3VTq,Bob,Ref1)'''
'''(yUC9VTr,Chad,Ref1)'''
... [all eligible electorate]
|
|-
! Clone Referendum 1 to Election 2
|
(3UA9VTp,Alice,Ref1)
(oUB3VTq,Bob,Ref1)
(yUC9VTr,Chad,Ref1)
...
'''(3UA9VTp,Alice,Elect2)'''
'''(oUB3VTq,Bob,Elect2)'''
'''(yUC9VTr,Chad,Elect2)'''
...
|
|-
! Alice & Chad Vote In Referendum 1
|
<strike>(3UA9VTp,Alice,Ref1)</strike>
(oUB3VTq,Bob,Ref1)
<strike>(yUC9VTr,Chad,Ref1)</strike>
...
(3UA9VTp,Alice,Elect2)
(oUB3VTq,Bob,Elect2)
(yUC9VTr,Chad,Elect2)
...
|
'''"Yes" ⇒ OMGuVotedN0wg0prty (Ref1)'''
'''"No" ⇒ Yualw4ysSoN3gatlve (Ref1)'''
|-
! Alice & Bob Vote In Election 2
|
(oUB3VTq,Bob,Ref1)
...
(wUZ3VT5,Zarrah,Ref1)
<strike>(3UA9VTp,Alice,Elect2)</strike>
<strike>(oUB3VTq,Bob,Elect2)</strike>
(yUC9VTr,Chad,Elect2)
...
|
"Yes" ⇒ OMGuVotedN0wg0prty (Ref1)
"No" ⇒ Yualw4ysSoN3gatlve (Ref1)
'''"Kyle","Lucy" ⇒ M4nUVote4YurBudzEh (Elect2)'''
'''"Jim","Mandy" ⇒ OMGuVotedTo0g0prty (Elect2)'''
|}