find /sys/devices/ -name driver -exec file “{}” \; | sed “s|^.*\.\.||” | sort -u | wc

Building mobi’s from XML files:

I was trying to convert a docbook based document from xml to mobi. http://files.kroah.com/lkn/lkn_xml.tar.gz  to mobi.

The steps are:run dbtoepub after getting the xml / docbook stuff, and any dtd files (say the safary_future.dtd files needed) Then run the kindlegen program http://www.amazon.com/gp/feature.html?ie=UTF8&docId=1000234621 on the ebub file.

FWIW I ended up getting the dtd file by cloning a repo on git hub git://github.com/sebastianbergmann/phpunit-documentation.git and copying its dblite directory to my /opt/tools/docbook/ directory.

Then I still had problems with a few missing xml files so I deleted their reference (metadata.xml, cyprt.xml)  from the book.xml The resulting mobi file looks to work fine on my tablet running the kindle reader apk.

Linux kernel spelunking of the /sys directory

The sysf directory has a lot of interesting data in it.  I’m looking to identify all the drivers and modules and devices my system thinks it has in it.  To this end I’ve been using grep and find and now setting up to implement a python program to extract the “drivers” the running kernel thinks it has access too as well as the “device”s  it knows about, and the set of “modules” it has (even if they are not loaded).  I should be able to map the drivers in use to devices and kernel config settings.  I should also be able to identify devices that are missing drivers.  I should be able to identify drivers that really are unneeded.

At least thats what I “feel” I should be able to do.  The reality is there are things happening to complicate my simple assumptions.  As I get into this exploration I’ll fill in the details.  But, by running a few find commands I can identify all the drivers the system thinks it has access too at runtime (by searching for bind in the /sysfs tree using find).  I can also know all the modules the system knows about by listing the /sys/module/ directory.  I can identify all the drivers associated with devices by searching for “driver” using find.  I can find all the devices and drivers that have issues a udev envent by finding all instances of  ”modalias” using find.

The problems I have is that the numbers don’t add up yet:

Linux mgross-MOBL 3.9.0 #8 SMP Sun May 5 12:30:05 PDT 2013 x86_64 x86_64 x86_64 GNU/Linux
mgross@mgross-MOBL:~$ find /sys/ -name device -exec file "{}" \; | sed "s|^.*\.\.||" | sort -u | wc
 124 172 2795
mgross@mgross-MOBL:~$ find /sys/ -name driver -exec file "{}" \; | sed "s|^.*\.\.||" | sort -u | wc
 42 44 1121
mgross@mgross-MOBL:~$ find /sys/ -name unbind | wc
 60 63 2226

mgross@mgross-MOBL:~$ find /sys/ -name bind | wc
 62 65 2192

mgross@mgross-MOBL:~$ find /sys/ -name refcnt -exec cat "{}" \; | wc

 101 101 225

mgross@mgross-MOBL:~$ find /sys/ -name refcnt -exec cat "{}" \; | grep -v 0 | wc
 67 67 138
mgross@mgross-MOBL:~$ lsmod | wc
 102 361 4543
mgross@mgross-MOBL:~$ find /sys/ -name modalias | wc
 135 135 8093

mgross@mgross-MOBL:~$ find /sys/ -name module | wc
 33 33 1198

mgross@mgross-MOBL:~$ ls /sys/module/ | wc
 144 144 1332

So I have 101 modules loaded and 43 drivers (modules tend to be drivers and I feel the 43 “drivers” should include drivers that are loaded a modules. Yet I can bind/unbind only 62(or 60) drivers from devices and the system thinks there are 144 modules going on.

Something isn’t adding up for me just yet.  Clearly my definition of device, driver, and module I am assuming does not match up with reality here.

I spent a bunch of time today getting apache2 with mod_wsgi to share up the Flask sample application: “flaskr”  this is posting to capture the issues and solutions so when I look at this again I won’t be starting from scratch.

Setting up the virtual env within the *.wsgi file failed. I tried the activate_this.py thing but it never worked.

I kept getting a syntax error form mod_wsgi in the /var/FLASK/flaskr/flaskr.wsgi file on the line “ import sys”!!!! I eventually worked around it by setting the WSGIPythonPath in the apache2.conf file. (no clue on why it failed or why changing the apache2.conf fixed it)

I forgot to turn my flask app into a “module” the from flaskr import app line of the flaskr.wsgi file was failing because my app was a program an not a module that could be imported. Then I needed to fix up the import path to work with the sys.path I set up.

Finally the search path to the sqlite3 data base was path relative in my flask.py. It needs to be absolute for mod_wsgi to work ok.

yup it was a pain in the butt.

Ref URLs

Configurations:

wsgi user account:

useradd -M flask
usermod -s /bin/false flask
usermod -L flask
adduser flask www-dat

/etc/apache2/apache2.config:

add the following to the end of the file:

WSGIPythonHome /var/FLASK/venv

/etc/apache2/sites-available/default:

put the following in the <VirtualHost *:80> section

 WSGIDaemonProcess flaskr user=flask group=www-data threads=5 home=/var/FLASK
 WSGIScriptAlias /flaskr /var/FLASK/flaskr/flaskr.wsgi
 <Directory "/var/FLASK/flaskr">
   WSGIProcessGroup flaskr
   WSGIApplicationGroup %{GLOBAL}
   WSGIScriptReloading On
   Order deny,allow
   Allow from all
 </Directory>

Set up working directory and virtualenv to use (do as root):

mkdir /var/FLASK/
 cd /var/FLASK
 virtualenv --no-site-packages venv
 . venv/bin/activate
 pip install Flask
 cp -r /home/<username>/dev/flaskr .
 chown -R flask:flask /var/FLASK

/var/FLASK/flaskr/flaskr.wsgi:

import sys
sys.path.insert(0, '/var/FLASK')
from flaskr.flaskr import app as application

I wanted to figure out how to deploy Flask applications on my work computer without screwing it up too much with experimentation and hacking around in crazy ass ways.  So I decided to deploy it on a Virtual box ubuntu guest OS.  But to make it work I need the VBox guest to open up port 80 that my host could somehow send http requets too.

This post is basically a SOCKS enhanced version of the bridging information I posted about in 2009 http://thegnar.org/sync/?p=1

The steps are:

• HOST:

  • sudo apt-get install bridge-utils uml-utilities ssh

  • add to /etc/ssh/sshd_config: AllowTcpForwarding yes

  • create a vboxbridge.sh scrip:

# create a tap
sudo tunctl -t tap1 -u <username>
sudo ip link set up dev tap1

# create the bridge
sudo brctl addbr br0
sudo brctl addif br0 tap1

# set the IP address and routing
sudo ip link set up dev br0
sudo ip addr add 10.1.1.1/24 dev br0
sudo ip route add 10.1.1.0/24 dev br0

• Install VirtualBox
• build a ubuntu guest VM from an ISO file.
• After settup and installations are finished you will Configure it to use bridged networking on the tap1 interface
• For initial setup you will use NAT to because of proxy issues with bridging to tap1.
• Guest
  • apt-get install tmux apache2 vim  ssh pyflakes pep8 python-setuptools python-virtualenv proxychains libapache2-mod-wsgi
  • /etc/proxychains.conf: [ProxyList] socks5 127.0.0.1 9999
  • Firefox: about:config (in addressbar):  network.prox.sock_remote_dns true
  • Firefox manual proxy setting:  localhost Port 9999, SOCKSv5
  • ssh -C <host_username>@10.1.1.1 -D 9999 -N

with this configuration / set up the Vbox guest is now able to host a web server to 10.1.1.2:80 to the host OS.  Also, the guest web browser works ok and I can use “proxychains apt-get” and proxify other non proxy aware applications using proxychains.

tmux handles paned windows and resizing so much better and its currently under active development where screen has been stagnant since 2005 (I think).

Getting going with tmux I only really need a handful of commands and for me it seems less cryptic than screen.  BTW I’m sticking with the ctrl-b as the command prefix key because I use minicom and with tmux I can have a minicom session in a screen pane.  yay!

TMUX commands I use:

• ^b %  — vertical split
• ^b “  — horizontal split
• ^b <arrow key> — move between panes.
• ^b <alt>+<arrow>  — adjust size of current pane
• ^b <pg-up>  — scroll up the history buffer
  • when in copy mode ^s for search down (will wrap in buffer)
  • when in search mode n will go to next and N will search the opposite direction
  • q exist page up / copy mode
• ^b ,   — rename current window tab
• ^b n — cycle to next window tab
• ^b p — cycle to previous window tab
• ^b d  — detach from session
• tmux att -t <session name>  — attach to session by name
• tmux ls  — list sessions
• ^b :break-pane  — move current pane to a tab’ed window
• ^b :join-pane -t dest-wind  — move current window to dest-wind as a pane.
• ^b ? brings up a help page of key bindings

There are copy and past things but I will use my host to do those outside of tmux so I have not bothered figuring that out.

.tmux.conf file:

set -g history-limit 50000                                                                                                                             │
set -g default-terminal screen-256color                                                                                                                │
setw -g aggressive-resize on

• http://youtu.be/XloTXFgNL7s
• installed laptop-mode-tools
• changed my sysctl.conf to set “vm.swappieness=2″
• set my fstab to use ramdisks for temp directories
  • tmpfs /tmp       tmpfs noatime,mode=1777,size=1350m 0 0
    tmpfs /var/tmp   tmpfs noatime,mode=1777,size=1350m 0 0
    tmpfs /var/spool tmpfs noatime,mode=1777,size=1350m 0 0
    tmpfs /var/log   tmpfs noatime,mode=0755,size=1350m 0 0
• mounted my root using “noatime,nodiratime,discard,errors=remount-ro”
• turned off the discrete graphics in my rc.local !  This dropped my burn rate from 45W to 13W!!
  • echo ‘OFF’ > /sys/kernel/debug/vgaswitcheroo/switch
•  lastly I put the following into my rc.local to avoid some latencies in the file system.  It may be overkill:

• for dev in sda sdb sdc
  do
  QSZ=1024
    DEV=/sys/block/$dev/queue/nr_requests
    if [ -r $DEV ]; then
      echo $QSZ > $DEV
    fi
  
  QSZ=1024
    DEV=/sys/block/$dev/queue/read_ahead_kb
    if [ -r $DEV ]; then
      echo $QSZ > $DEV
    fi
  done
  
  

To make a local copy of the package selection states:

dpkg --get-selections >myselections

You might transfer this file to another computer, and install it there with:

dpkg --clear-selections

dpkg --set-selections <myselections

Note that this will not actually install or remove anything, but just set the selection state on the requested packages. You will need some other application to actually
download and install the requested packages. For example, run

apt-get dselect-upgrade.

The power HAL came to AOSP in the Jelly Bean release.  Its reason for being was to end of life the early suspend notification call chains in the kernel that Android power management was dependent on.  It has since then expanded to become the user mode agent for boosting the platform performance when user interactivity is important.  This blog posts documents its current implementation in the AOSP master branch as of Feb 2, 2013.

Important files:

• Interface spec for power HAL plug in power.$(TARGET_BOARD_PLATFORM).so:
  • hardware/libhardware/include/hardware/power.h
  • The comments are very useful and pretty much explain the powerHAL design in use.
• Sample implementations of power HAL power.$(TARGET_BOARD_PLATFORM).so file can be found (hint: look at the Android.mk’s for these as well as the c code):
  • hardware/qcom/power
  • device/asus/grouper/power
  • device/samsung/manta/power
  • device/samsung/tuna/power
  • device/generic/goldfish/power
• Callers of setInteractive powerHAL entry point:
  • frameworks/base/services/jni/com_android_server_power_PowerManagerService.cpp
• Callers of powerHint powerHAL entry point:
  • frameworks/base/services/jni/com_android_server_power_PowerManagerService.cpp
  • frameworks/native/services/surfaceflinger/DisplayHardware/PowerHAL.cpp

Pretty simple.  setInteractive is basically called when the screen is turned on and off with different parrameters with the expectation that the powerHAL .so file will poke the kernel interfaces to notify whatever is needed to be notified that the screen is off or on.  powerHint is used for busting the performance on user interactivity or graphics activity going on (UI / UX animations)

There isn’t much more to say other than you should look a the example implementations of powerHAL’s available to get an idea on how people have implemented it for specific hardware and mimic them for yours.  When ready you simply add “power.<your target board platform name>” to the PRODUCT_PACKAGES list in your device’s AndroidProducts.mk (or a mk file that AndroidProducts includes) and you have a Power HAL.

lunch full_x86;

Dual Socket workstation 2 Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz cpu’s (each cpu has 8 cores with HT) 32GB 1333MHz ram 7200RPM HD

no ccache time nohup make -j 32

mgross@mtgdev:~/work/master$ time nohup make -j 32
nohup: ignoring input and appending output to `nohup.out’

real    17m12.243s
user    392m0.962s
sys    14m10.229s

with a primed ccache on SSD:

time nohup make -j 32
nohup: ignoring input and appending output to `nohup.out’

real    8m27.191s
user    66m11.256s
sys    4m43.330s

mgross@mtgdev:~/work/master$ ccache -s
cache directory                     /mnt/ssd/home/mgross/work/master/ccache
cache hit (direct)                     0
cache hit (preprocessed)           15346
cache miss                         15400
called for link                      944
unsupported source language          680
unsupported compiler option          180
files in cache                     30800
cache size                           4.1 Gbytes
max files                       10000000
max cache size                      16.0 Gbytes

HP Envy 17: i7 quad Intel(R) Core(TM) i7-2760QM CPU @ 2.40GHz 16GB 1600Mhz DDR3 7200RPM HD

no ccache: time nohup make -j 8

nohup: ignoring input and appending output to `nohup.out’

real 63m0.631s
user 440m56.765s
sys 13m22.894s

After priming the ccache (is on SSD, build on 7200RPM HD) and then rebuilding (after a reboot):

time nohup make -j 8
nohup: ignoring input and appending output to `nohup.out’

real 17m44.246s
user 76m37.079s
sys 4m51.046s
mgross@mgross-MOBL:~/work/master$ ccache -s
cache directory /mnt/ssd/home/mgross/work/master/ccache
cache hit (direct) 0
cache hit (preprocessed) 15345
cache miss 15401
called for link 944
can’t use precompiled header 961
unsupported source language 680
compiler check failed 329
unsupported compiler option 180
files in cache 30802
cache size 4.1 Gbytes
max files 500000
max cache size 8.0 Gbytes
mgross@mgross-MOBL:~/work/master$

Lenovo ThinkPad X220 : i5 dual core  Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz 8GB 1600Mhz DDR3 SSD

no ccache: time nohup make -j 4nohup:

ignoring input and appending output to `nohup.out’

real    111m1.760s
user    412m5.845s
sys    13m2.241s

Building with a primed ccache (build + ccache on same SSD) time nohup make -j 4
nohup: ignoring input and appending output to `nohup.out’

real    24m13.602s
user    77m27.790s
sys    5m4.503s

mgross@MGROSS-X220VM:~/work/master$ ccache -s
cache directory                     /mnt/ssd/home/mgross/work/master/ccache
cache hit (direct)                     0
cache hit (preprocessed)           15345
cache miss                         15401
called for link                      944
unsupported source language          680
unsupported compiler option          180
files in cache                     30802
cache size                           4.1 Gbytes
max files                       10000000
max cache size                      16.0 Gbytes

HP G62 : i3 dual core Intel(R) Core(TM) i3 CPU       M 330  @ 2.13GHz 8GB 1066Mhz DDR3 7200RPM HD

no ccache: time nohup make -j 4

nohup: ignoring input and appending output to `nohup.out’

real    176m56.178s
user    638m14.293s
sys    19m45.450s

 using primed ccache : time nohup make -j 4
nohup: ignoring input and appending output to `nohup.out’
real    45m38.374s
user    122m15.186s
sys    7m46.397s

mgross@G62-mgross:~/Work/master$ ccache -s
cache directory                     /home/mgross/Work/master/ccache
cache hit (direct)                     0
cache hit (preprocessed)           15344
cache miss                         15402
called for link                      944
unsupported source language          680
unsupported compiler option          180
files in cache                     30804
cache size                           4.1 Gbytes
max files                        1000000
max cache size                      10.0 Gbytes
mgross@G62-mgross:~/Work/master$

HP Mini 210:  Intel(R) Atom(TM) CPU N455   @ 1.66GHz 2GB ram SSD

time nohup make -j 1
nohup: ignoring input and appending output to `nohup.out’

real    1311m52.342s
user    1337m23.875s
sys    55m11.231s

mostly worked but I was missing a package at build time libsdl-ttf2.0-dev

apt-get install build-essential libgtk2.0-dev libgnome2-dev libsdl1.2-dev libsdl-ttf2.0-dev

this works to build version mame0146s : http://mamedev.org/downloader.php?file=releases/mame0146s.zip on 64 bit ubuntu 11.04

http://blog.decat.org/aosp-step-by-step-instructions-for-setting-up-a-chrooted-ubuntu-10-04-64bit-build-environment-on-ubuntu-11-10-and-12-04-hosts

Starting from the 12.04 command line do the following things (mostly in order)

• sudo apt-get update
• sudo apt-get upgrade
• sudo apt-get install dchroot debootstrap

sudo tee -a /etc/schroot/schroot.conf <<EOF
 [lucid]
 description=Ubuntu 10.04 LTS (Lucid Lynx)
 location=/var/chroot/lucid
 priority=3
 users=<<your user name here>>
 groups=<<your user name here>>
 root-groups=root
 EOF

sudo debootstrap --variant=buildd --arch amd64 lucid /var/chroot/lucid http://archive.ubuntu.com/ubuntu/

sudo tee -a /var/chroot/lucid/etc/debian_chroot <<EOF
 $HOSTNAME-chroot-lucid)
EOF

• sudo cp /etc/resolv.conf /var/chroot/lucid/etc/
• sudo cp /etc/passwd /var/chroot/lucid/etc/
• sudo cp /etc/group /var/chroot/lucid/etc/
• FSTAB changes: (to host 12.04 /etc/fstab)

#bind mount for lucid
/proc /var/chroot/lucid/proc bind defaults,bind 0 0
/home /var/chroot/lucid/home bind defaults,bind 0 0

• sudo chroot /var/chroot/lucid/
• sources.list changes:  (the chroot /etc/apt/sources.list)

 #make sure the following 2 lines are in the sources.list
 deb http://archive.ubuntu.com/ubuntu lucid main
 deb http://archive.ubuntu.com/ubuntu lucid universe

• now install the android build dependencies;
• apt-get update

  sudo apt-get install git-core gnupg flex bison gperf build-essential zip curl \
               zlib1g-dev gcc-multilib g++-multilib libc6-dev-i386 lib32ncurses5-dev \
               ia32-libs x11proto-core-dev libx11-dev lib32readline5-dev lib32z-dev \
               tofrodos gawk mkisofs

Feel free to add whatever else you may need (vim, ctags, cscope ccache, etc...)

• Now you may want to have a script “lucid.sh” to log in into the chroot and as the correct user:
• note: if you installed Java the way you are *supposed to* in $HOME/bin/ and set up the paths in your .profile then the java support “just works” otherwise you deserve what you get.

#!/bin/sh
sudo chroot /var/chroot/lucid/ su - `whoami`

Seems to work.