Maemo Wiki Mirror - User contributions [en]

User:Tanner

2010-06-03T17:50:54Z

178.165.183.141: /* full optification */

= Packages =

info about some of the packages I've uploaded to extras-devel

== [http://maemo.org/packages/view/debhelper7/ debhelper7] ==

see also http://wiki.maemo.org/Packaging

A backport of Debhelper 7 for Fremantle is available in extras-devel.
It works transparently and can coexist with debhelper 5 in the SDK.
It also works on the autobuilder, if you specify
Build-Depends: debhelper7
in debian/control.
The following lines in debian/rules are necessary to use it:

PATH:=/usr/bin/dh7:/usr/bin:$(PATH)
export PATH
SBOX_REDIRECT_IGNORE=/usr/bin/perl
export SBOX_REDIRECT_IGNORE

We are working on enhancement to make optification is easier, i.e. direct installation of files in the /opt hierarchy.

If you also need a more recent cdbs, then use the package [http://maemo.org/packages/view/cdbs-dh7/ cdbs-dh7],
which conflicts with the standard cdbs and does not work on autobuilder yet.

To replace several tools in scratchbox you can also use the following line as a template
SBOX_REDIRECT_IGNORE:=$(shell echo /usr/bin/{perl,dh_*} | sed "s/ /:/g")

== CLI icon ==

For your convenience I have uuencoded Tim Samoff's [http://lists.maemo.org/pipermail/maemo-community/2010-January/003870.html CLI icon]
which is supposed to be added to the debian/control file of CLI apps. You can download it [http://maemory.com/N900/tools/cli.icon.txt here].

== [http://maemo.org/packages/view/kernel-flasher-maemo/ custom Linux kernel] ==

My custom kernel which contains probably everything one could need on the N900:

This custom kernel contains additional modules for IPv6, packet filtering,
NAT, tunneling, kernel configuration, Wifi mesh networking,
ext4, XFS, reiserfs, NTFS reading, NILFS2, ISO9660, UDF, CIFS, automounter,
UNIONFS, device mapper and dmloop, cryptography, cryptoloop,
EFI partitions, UTF8 codepages, mouse+joystick input,
USB/IP and generic USB device drivers.
It shows a boot logo.

Just install the package "kernel-flasher-maemo" from extras-devel/non-free (section System) and reboot.
Reverting to the original kernel is simple: "apt-get install --reinstall kernel kernel-flasher" or reflash only the original kernel image.
see [http://talk.maemo.org/showthread.php?t=43420 TMO page] for more details.

== GNU tools ==
the crippled busybox is installed on the N900.
command line users prefer the real GNU tools.
these are the optified packages for the device.
Most of them do not take use more RAM than busybox (a few KBs)!
[http://maemo.org/packages/view/coreutils-gnu/ coreutils-gnu]
[http://maemo.org/packages/view/diffutils-gnu/ diffutils-gnu]
[http://maemo.org/packages/view/findutils-gnu/ findutils-gnu]
[http://maemo.org/packages/view/grep-gnu/ grep-gnu]
[http://maemo.org/packages/view/sed-gnu/ sed-gnu]
[http://maemo.org/packages/view/tar-gnu/ tar-gnu]

== [http://maemo.org/packages/view/texlive-latex-base/ TeXLive 2009] ==
[http://talk.maemo.org/showthread.php?t=42087 talk page]

== [http://maemo.org/packages/view/r-base-core/ R] (statistics) ==

bug: installation fails with ucfr missing
solution: wait or edit r-base-core.postinst and comment line 44 out

== [http://maemo.org/packages/view/libblas3gf/ BLAS] [http://maemo.org/packages/view/liblapack3gf/ LAPACK] [http://maemo.org/packages/view/libsuitesparse-3.1.0/ suitesparse] ==

bug: linking against BLAS and LAPACK currently hardcodes the links found in the -dev package.

== [http://maemo.org/packages/view/python-numpy/ numpy], [http://maemo.org/packages/view/python-scipy/ scipy], [http://maemo.org/packages/view/python-sympy/ sympy] ==

[http://talk.maemo.org/showthread.php?t=33020 some discussion]

== [http://maemo.org/packages/view/bonnie++/ bonnie++] (filesystem benchmarks) ==
see [http://talk.maemo.org/showpost.php?p=504332&postcount=76 post and ff]

== [http://maemo.org/packages/view/tcl8.5/ tcl8.5] [http://maemo.org/packages/view/tk8.5/ tk8.5] [http://maemo.org/packages/view/python-tk/ python-tk] ==

== [http://maemo.org/packages/view/lzma/ lzma] and its successor [http://maemo.org/packages/view/xz-utils/ xz-utils] ==

an extremely efficient compressor. it is much faster than bzip2 for decompression and yields about 30% smaller files.

== [http://maemo.org/packages/view/xskat/ xskat] ==

bug: keyboard doesn't work.
bug: window close button is not visible

== experimental packages ==

some experimental ports which I have not yet uploaded to extras-devel can be found in my repository

deb http://www.maemory.com/N900/ fremantle free incompatible

currently: python2.6, perl5.10, GNU compiler collection 4.4.3, autoconf 2.64, automake1.10

----

= Ideas/plans: =

== [https://garage.maemo.org/projects/moebian/ Moebian]: Maemo/MeeGo on Debian ==

[http://wiki.maemo.org/Maemo_on_Debian Wiki] and [http://talk.maemo.org/showthread.php?t=44967 discussion]

== Brainstorm: [https://maemo.org/community/brainstorm/view/more_efficient_and_flexible_use_of_internal_flash Repartitioning] ==

[http://talk.maemo.org/showthread.php?t=37869 brainstorm discussion]
[http://talk.maemo.org/showthread.php?t=35122 repartitioning thread]

== Brainstorm: [http://maemo.org/community/brainstorm/view/stylus_and_finger_friendly_uis/ Stylus and finger friendly UIs] ==

[http://talk.maemo.org/showthread.php?p=525854 brainstorm discussion]

== alternative USB host solution ==

[http://talk.maemo.org/showthread.php?t=41870 USB/IP]

== full [http://wiki.maemo.org/Opt_Problem optification] ==

Ideally addon software should be installed only in /opt without any symlinks in / (NAND)
in accordance with [http://www.pathname.com/fhs/2.2/fhs-3.12.html FHS].
/usr/local should be moved and symlinked to /opt/local and is reserved for own programs not maintained by dpkg.

A package may either install its files directly (for small packages) in the /opt/maemo hierarchy (bin,etc,lib,man,share)
or in a separate directory /opt/<package> with its own hierarchy (bin,etc,lib,man,share).
For the latter solution packages should symlink all non-private files to /opt/maemo/{bin,etc,lib,man,share}
and the search paths of Maemo need to be adjusted in /etc/profile and /etc/osso-af-init/af-defines.sh:

* PATH=$PATH:/opt/maemo/usr/bin (and sbin for root)
* ld.so.conf: add /opt/maemo/usr/lib
* locales: [https://bugs.maemo.org/show_bug.cgi?id=8363 LOCPATH]=LOCPATH=/usr/share/locale:/opt/maemo/usr/share/locale
* MANPATH=/opt/maemo/usr/share/man
* pkgconfig: /usr/share/pkgconfig:/opt/maemo/usr/share/pkgconfig
* info in info?: /usr/share/info:/opt/maemo/usr/share/info
* apps in hildon: [https://bugs.maemo.org/show_bug.cgi?id=10531 Bug about Application Menu] /usr/share/applications:/opt/maemo/usr/share/applications
* icons in hildon ?: /usr/share/icons:/opt/maemo/usr/share/icons
this list may be incomplete - please help to complete it

As NAND is much faster than eMMC a cache manager could mantain a list
of very often used binaries and libraries in /opt and automatically
copy them and symlink them to a fixed size cache in NAND.

TMPDIR=/var/tmp should be moved to /opt/tmp
or /var/opt should symlink to /opt/var and contain a larger tmp directory.

== chroot instead of /opt ==
I have described my currently preferred [http://maemo.org/community/brainstorm/view/remove_256mb_limitation_of_the_rootfs_partition_in_the_n900/ solution as #5]
----

= Tips and tricks: =

== fresh builds ==

[http://maemo.org/packages/repository/latest/fremantle_extras-devel_free_armel/all/ latest packages]

[https://garage.maemo.org/builder/fremantle/?C=M;O=D autobuilder current builds]

== [[Repartitioning the flash]] ==

[http://talk.maemo.org/showpost.php?p=424811&postcount=66 how to swap FAT and ext3 parition]
[http://talk.maemo.org/showpost.php?p=425359&postcount=68 how to implement USB mass storage with a loopback file]

== Setting up a repository mirror on your server ==

[http://www.maemory.com/mirror/repro-maemo.tgz reprepro configuration]

== Safely free some space on rootfs ==

[http://maemory.com/N900/tools/moveroot.sh This script] shows you how much space is freed.
It can also detect whether some directories have already been moved.
WARNING: Although this script works for most users, you should always perform a backup before running it.
Do not use this script if you have already used some other methods or scripts from [[Free_up_rootfs_space]]!
In this case reflashing is recommended.

# install the rootsh package from extras
# download the [http://maemory.com/N900/tools/moveroot.sh script] to your MyDocs (or upload it via SSH)
# in Terminal run
sudo gainroot
sh /home/user/MyDocs/moveroot.sh

It should have freed about 95MB on your rootfs.
Upgrading the firmware OTA after using the script is supposed to work.
If you reflash the firmware the script will automatically remove the files from the old firmware
when you run it again.

[http://maemory.com/N900/tools/unmoveroot.sh This untested script] reverts the modifications and moves as many as possible files back to rootfs.
Make sure that you have enough space on rootfs for that. Otherwise it will abort the operation. Instructions are the same as for moveroot.sh,
except that you call unmoveroot.sh.

== On-device flashing of a custom kernel image ==

This how to flash a kernel on-device (without USB/flasher 3.5):

# you have a kernel image /path/zImage (with it's modules installed)
# install the package "sdk-fiasco-gen" from SDK
# become root: "sudo gainroot"
# make a "fiasco" :)
fiasco-gen -o /path/zImage.fiasco -g -k /path/zImage
# flash it
/sbin/fiasco-image-update /path/zImage.fiasco

== Reflashing & Restore ==

this is a slightly modified version of the script I use to restore my N900 after a reflash.
My reflashing procedure is:

# store all settings with Backup on my SD card
# reflash with standard http://wiki.maemo.org/Updating_the_firmware, i.e.
sudo flasher-3.5 -F ''<current firmware image>''.bin -f -R
# restore old backup except programs
# install rootsh from System
# in Terminal:
sudo sh /home/backup/restore.sh
with [http://maemory.com/N900/tools/restore.sh this script (for PR<1.2] or [http://maemory.com/N900/tools/restore1.2.sh this script (PR1.2)] in /home/backup (feedback and patches are welcome).
It expects that you made a backup of /root and your selected modified files (e.g. /etc/ssh) in /home/backup.
The file "pkgs" should contain a list (line by line) of the package names you want to install.

Opt Problem

2010-06-03T14:38:26Z

178.165.183.141: fix typo

= The /opt Problem =

There is not enough space on the [[rootfs]] to fit all applications, therefore an additional partition has to be used. But how?

There is a section in the Maemo 5 Developer Guide explaining the basics:
[[Documentation/Maemo_5_Developer_Guide/Packaging,_Deploying_and_Distributing/Installing_under_opt_and_MyDocs|Installing under /opt and /MyDocs]]

= Current situation =

* 256MB OneNAND chip: bootloader, kernel, root (ubifs - with compression)
* 32GB eMMC: /home ~2GB (ext3), /home/user/MyDocs ~29GB (vfat), swap 0.7GB
* symlink: /opt/ -> /home/opt/
* OneNAND is probably "much" faster than the eMMC (hard numbers? read? write? small blobs? big blobs?)
* file system [http://talk.maemo.org/showpost.php?p=504332&postcount=76 benchmark results] for different media
* OneNAND performance figures: Sustained read performance: 108MB/s. Sustained write performance: up to 17MB/s. See [http://www.samsung.com/global/business/semiconductor/products/fusionmemory/downloads/onenand_brochure_200609.pdf Samsung OneNAND specs].
* eMMC performance figures: read performance 37MB/s. Write performance 20MB/s. Write speed of 17MB/s has been achieved in this test at [http://symbianbriefs.blogspot.com/2010/01/n900-review-part-10-transferring-data.html symbianbriefs.blogspot.com]. My own read tests however show ~11MB/s (370MB file).
* Decisions were taken in a "what shall we do about /opt?" BOF at the [[Maemo Summit 2009]]. Minutes: http://lists.maemo.org/pipermail/maemo-developers/2009-October/021289.html

= Some goals =

* Firmware should be flashable.
* Should not break applications.
* One should be able to install more than 256mb worth of software.
* Standards compliance, less need to make incompatible quirks.
* A fix should be doable after shipping (before shipping, probably not?)

= Current problems / developer and packaging issues =

This should be a list of current problems developers are experiencing related to optification:

* [http://lists.maemo.org/pipermail/maemo-developers/2009-October/021419.html Optification breaks package on upgrade from non-optified older version]

* Symlinks take up space in the rootfs. This may become an issue if there are lots of them.

* [[/Non-Optified_packages|List of non-optified packages and applications]] - Please feel free to add every piece of software which is using too much space in rootfs

= Tool for detection of non-optified packages =
[[Image:Storageusage.png|thumb|300px|<br>[http://code.google.com/p/storageusage/ Storage Usage] provides a great visual way to detect non-optified packages.
]]

= Possible solutions, even the bad ones =

==== Applications are made relocatable and installed to /opt ====
* Relocatable applications can move everywhere without breaking.
* No symlinks get scattered over the rootfs (symlinks take up space, too).
* Not all applications can easily be made relocatable.
* Requires more work by the programmer.
* May not be feasible for ports from an upstream source.
* Putting a startup script into /usr/bin setting up environment variables such as LD_LIBRARY_PATH can help.
* autotools-based packages can use --prefix=/opt/<packagename>.
* Python apps can make use of the [http://docs.python.org/reference/datamodel.html#index-821 __file__] module-scope variable to determine where they are installed (e.g. installdir = os.path.dirname(__file__)).
* Could be taken into account as an alternative to maemo-optify wherever feasible.

==== Have a 2-5gb partition on the emmc for addon packages mounted to /usr/local or /opt ====
* Most apps by default usually go there
* for /opt store according to the FHS standard /opt/<package> and symlink to /opt/maemo/{bin,lib,share,...} (actually the standard is /opt/<vendor>/whatever In this case using /opt/maemo directly would work).
* Having ld.so.conf and PATH set properly avoids all the symlink madness
* Don't do symlinks for /usr/local|/opt - make it a real partition
* Explain to people why they don't have full 32GB of storage - or use LVM on the eMMC to dynamicaly change the size as things get installed through some dpkg hooks or such - we do know how much it will take ahead of time so if there isn't enough storage do a resize(VFAT should be resizable with something like gparted maybe?) ext3/2 are both online resizeable (growth only!)
* Requires a bit more work when packaging - make some defaults pre-set in SDK
* All apps should use a normal usr layout underneath any path. Example: /opt/maemo/{bin,sbin,lib, etc...} same if it would be in /usr/local - works like GNU stow

==== store addon packages completely in /home/opt hierarchy ====
* [http://wiki.maemo.org/User:Tanner#full_optification see this page]
* like previous solution but on the standard /home partition
* very clean solution as no files (except for /var/lib/dpkg which may be moved as well) are created or modified on the NAND.
* /etc and /var could be copied there as well and mounted to root using bind. This way OneNAND would only be changed be reflashes and if /home is not mounted (e.g., by a keypress during boot) a fallback system could boot from unmodified OneNAND.
* if OneNAND is significantly faster, mark some often used shared libraries as candidates for a /usr/lib/cache directory in which they are copied during installation until a certain cache limit is reached.

==== During packaging move all files >500kb to /opt and symlink to root ====
* Optify is basically this.
* Install is not slowed down, work is done when packaging.
* Some packages break when replaced with symlinks. How to prevent - hack on hack?
* A buildbot could force this to debian/rules automatically.

==== Use aufs/unionfs to overlay OneNAND root with eMMC filesystem. ====
* This could be partial overlay, for example only /lib and /usr.
* Could keep essential libraries on OneNAND so they would work faster.
* Has anyone measured the speed penalty for aufs/unionfs usage?
* Is it faster or slower that moving root to eMMC?
* Might be difficult to have new programs to benefit from faster OneNAND.

==== Hack with --datadir, --libdir etc. through debian/rules ====
* Could be forced by a buildbot.
* Should move most of the big lumps on larger media if there are some.
* Works at least with packages with configure.
* Requires at least additional $LIBDIR setup?
* Programs which have hardcoded paths (ugly code) break.
* Could additionally symlink files from opt to root.

==== Require that applications have --prefix /opt ====
* Could be forced by a buildbot.
* Should not break anything.
* Programs would not benefit from faster OneNAND.
* Requires at least additional $PATH setup?
* Programs which have hardcoded paths (ugly code) break.
* Could additionally symlink files from opt to root.
* Just reminds me of 770's /var/lib/install..
* Against standard

==== Move root from OneNAND to eMMC and combine with /home to create a new root ====
* How much faster OneNAND is? Numbers?
* Can flasher flash on eMMC?
* Leave only the bootable and rescue stuff in the rootfs
* Maybe use it uncompressed for speed and add a /fast/usr folder to OneNAND to store some system files that would benefit from the speed (example: move /usr/something to /fast/usr and symlink it or just place /fast/usr in the path)

==== Use mount -o bind (or -o rbind) ====
* Method: Do not use NAND for any additional data. Boot into standard image. Any additional changes made are only available after -o bind or -o rbind which are defined in /etc/fstab . Setup provides potential fail-over; if -o bind or -o rbind fails user gets into 'rescue shell' providing 1) recovery method 2) quick install to vanilla firmware without 3rd party utilities, networking, or computer (think: no network available, expensive roaming, no computer with fwflasher or Nokia Software Downloader available, ...)

==== Turn the problem around, pivot the root with /opt ====
* After pivot /opt would be root and old root would be /opt. Symlink the old root contents to new root would make those available.
* Can the base system dependencies handle this?
* Can Nokia created software handle this?
* Could this work? If so, then all additional packages would install directly on eMMC.

==== After install move all files >500kb to /opt and symlink to root ====
* Could be done at background after install or with post-install hook.
* There might not be enough space on / even for install.
* Install is already slow, could make it even slower.
* Some packages break when replaced with symlinks. How to prevent - hack on hack?
* Uninstall might be messy, some post-remove magic needed.

==== Chroot install in /opt, symlink to where it belongs ====
* A Bit like stow.
* Maybe somebody can see something good in this?
* Difficult hack.. would need all the tools etc to be visible in the chroot.

==== Make /usr another partition ====
You know, historically /usr exists entirely for the purpose of separating the application partition from the underlying system, which is exactly our problem.
* All the core nokia software and libraries should be relocated to / or maybe a /maemo.
* /usr and /opt should become symlinks to /home/usr and /home/opt.
* Maemo targeted GUI software would be encouraged to use /opt/[application].
* Linux libraries and CLI software could safely use /usr, minimizing the required modifications.
* Any frequently used accessed files in /home but not /home/user/MyDocs could be cached in /cache, which then gets overlaid with /home using unionfs or something more specialized.
* Another location like /alt could be used by packages that know they need the NAND's speed, but this is discouraged.

[[Category:Development]]