Opt Problem

2010-01-12T02:19:42Z

66.25.130.174: /* Some goals */

= The /opt Problem =

There is not enough space on root to fit all applications, therefore additional partition has to be used. But just how?

There is a section in the Maemo 5 Developer Guide explaining the basics:
http://wiki.maemo.org/Documentation/Maemo_5_Developer_Guide/Packaging%2C_Deploying_and_Distributing/Installing_under_opt_and_MyDocs

= Current situation =

* 256MB OneNAND chip: bootloader, kernel, root (ubifs - with compression)
* 32GB eMMC: /home ~2GB (ext3), /home/user/MyDocs ~29GB (vfat), swap 0.7GB
* symlink: /opt/ -> /home/opt/
* OneNAND is probably "much" faster than the eMMC (hard numbers? read? write? small blobs? big blobs?)
* OneNAND performance figures: Sustained read performance: 108MB/s. Sustained write performance: up to 17MB/s. See [http://www.samsung.com/global/business/semiconductor/products/fusionmemory/downloads/onenand_brochure_200609.pdf Samsung OneNAND specs].
* eMMC performance figures: read performance 37MB/s. Write performance 20MB/s. Write speed of 17MB/s has been achieved in this test at [http://symbianbriefs.blogspot.com/2010/01/n900-review-part-10-transferring-data.html symbianbriefs.blogspot.com]. My own read tests however show ~11MB/s (370MB file).
* Decisions were taken in a "what shall we do about /opt?" BOF at the [[Maemo Summit 2009]]. Minutes: http://lists.maemo.org/pipermail/maemo-developers/2009-October/021289.html

= Some goals =

* Firmware should be flashable.
* Should not break applications.
* One should be able to install more than 256mb worth of software.
* Standards compliance, less need to make uncompatible quirks.
* A fix should be doable after shipping (before shipping, probably not?)

== Current problems / developer and packaging issues ==

This should be a list of current problems developers are experiencing related to optification:

* [http://lists.maemo.org/pipermail/maemo-developers/2009-October/021419.html Optification breaks package on upgrade from non-optified older version]

* Symlinks take up space in the rootfs. This may become an issue if there are lots of them.

* [[/Non-Optified_packages|List of non-optified packages and applications]] - Please feel free to add every piece of software which is using too much space in rootfs

= Possible solutions, even the bad ones =

==== Applications are made relocatable and installed to /opt ====
* Relocatable applications can move everywhere without breaking.
* No symlinks get scattered over the rootfs (symlinks take up space, too).
* Not all applications can easily be made relocatable.
* Requires more work by the programmer.
* May not be feasible for ports from an upstream source.
* Putting a startup script into /usr/bin setting up environment variables such as LD_LIBRARY_PATH can help.
* autotools-based packages can use --prefix=/opt/<packagename>.
* Python apps can make use of the [http://docs.python.org/reference/datamodel.html#index-821 __file__] module-scope variable to determine where they are installed (e.g. installdir = os.path.dirname(__file__)).
* Could be taken into account as an alternative to maemo-optify wherever feasible.

==== Have a 2-5gb partition on the emmc for addon packages mounted to /usr/local or /opt ====
* Most apps by default usually go there
* for /opt store according to the FHS standard /opt/<package> and symlink to /opt/{bin,lib,share,...} actually the standard is /opt/<vendor>/whatever In this case using /opt/maemo directly would work.
* Having ld.so.conf and PATH set properly avoids all the symlink madness
* Don't do symlinks for /usr/local|/opt - make it a real partition
* Explain to people why they don't have full 32GB of storage - or use LVM on the eMMC to dynamicaly change the size as things get installed through some dpkg hooks or such - we do know how much it will take ahead of time so if there isn't enough storage do a resize(VFAT should be resizable with something like gparted maybe?) ext3/2 are both online resizeable (growth only!)
* Requires a bit more work when packaging - make some defaults pre-set in SDK
* All apps should use a normal usr layout underneath any path. Example: /opt/maemo/{bin,sbin,lib, etc...} same if it would be in /usr/local - works like GNU stow

==== store addon packages completely in /home/opt hierarchy ====
* like previous solution but on the standard /home partition
* very clean solution if speed advantage of OneNAND over eMMC turns out to be a myth
* /etc and /var could be copied there as well and mounted to root using bind. This way OneNAND would only be changed be reflashes and if /home is not mounted (e.g., by a keypress during boot) a fallback system could boot from unmodified OneNAND.
* if OneNAND is significantly faster, mark some often used shared libraries as candidates for a /usr/lib/cache directory in which they are copied during installation until a certain cache limit is reached.

==== During packaging move all files >500kb to /opt and symlink to root ====
* Optify is basically this.
* Install is not slowed down, work is done when packaging.
* Some packages break when replaced with symlinks. How to prevent - hack on hack?
* A buildbot could force this to debian/rules automatically.

==== Use aufs/unionfs to overlay OneNAND root with eMMC filesystem. ====
* This could be partial overlay, for example only /lib and /usr.
* Could keep essential libraries on OneNAND so they would work faster.
* Has anyone measured the speed penalty for aufs/unionfs usage?
* Is it faster or slower that moving root to eMMC?
* Might be difficult to have new programs to benefit from faster OneNAND.

==== Hack with --datadir, --libdir etc. through debian/rules ====
* Could be forced by a buildbot.
* Should move most of the big lumps on larger media if there are some.
* Works at least with packages with configure.
* Requires at least additional $LIBDIR setup?
* Programs which have hardcoded paths (ugly code) break.
* Could additionally symlink files from opt to root.

==== Require that applications have --prefix /opt ====
* Could be forced by a buildbot.
* Should not break anything.
* Programs would not benefit from faster OneNAND.
* Requires at least additional $PATH setup?
* Programs which have hardcoded paths (ugly code) break.
* Could additionally symlink files from opt to root.
* Just reminds me of 770's /var/lib/install..
* Against standard

==== Move root from OneNAND to eMMC and combine with /home to create a new root ====
* How much faster OneNAND is? Numbers?
* Can flasher flash on eMMC?
* Leave only the bootable and rescue stuff in the rootfs
* Maybe use it uncompressed for speed and add a /fast/usr folder to OneNAND to store some system files that would benefit from the speed (example: move /usr/something to /fast/usr and symlink it or just place /fast/usr in the path)

==== Use mount -o bind (or -o rbind) ====
* Method: Do not use NAND for any additional data. Boot into standard image. Any additional changes made are only available after -o bind or -o rbind which are defined in /etc/fstab . Setup provides potential fail-over; if -o bind or -o rbind fails user gets into 'rescue shell' providing 1) recovery method 2) quick install to vanilla firmware without 3rd party utilities, networking, or computer (think: no network available, expensive roaming, no computer with fwflasher or Nokia Software Downloader available, ...)

==== Turn the problem around, pivot the root with /opt ====
* After pivot /opt would be root and old root would be /opt. Symlink the old root contents to new root would make those available.
* Can the base system dependencies handle this?
* Can Nokia created software handle this?
* Could this work? If so, then all additional packages would install directly on eMMC.

==== After install move all files >500kb to /opt and symlink to root ====
* Could be done at background after install or with post-install hook.
* There might not be enough space on / even for install.
* Install is already slow, could make it even slower.
* Some packages break when replaced with symlinks. How to prevent - hack on hack?
* Uninstall might be messy, some post-remove magic needed.

==== Chroot install in /opt, symlink to where it belongs ====
* A Bit like stow.
* Maybe somebody can see something good in this?
* Difficult hack.. would need all the tools etc to be visible in the chroot.

[[Category:Development]]

Maemo Wiki Mirror - User contributions [en]

Opt Problem