Tuesday, November 11, 2014

Well, no it wasn't

As simple as zypper dup?


But not that painful.

The first effort left us with a corrupted yast2 (although yast worked) and no network connectivity.

Using a Windoze machine I downloaded the installation ISO and burned it to a USB flash drive. I used this to upgrade over the previous zypper dup effort. All the repositories had been converted, so it was relatively easy, although it wiped the startup splash screen and still no network. And AutoKey was hosed for a dependency conflict issue.

But it fixed the video and yast2 issues.

Searching for answers I stumbled across


It seems they've changed the network infrastructure from inetd to something called "wicked".

That doesn't work on wifi laptops. 


They tell you how to nuke wicked, but it would not work because we didn't have NetworkManager installed.

So we used the ISO once again to install NetworkManager, and followed the rules in the link above to nuke wicked.

That fixed networking, so I now updated everything, which then fixed AutoKey.

So all is copacetic.

Sort of:

The default video is still to big and the fonts too big, and I miss the splash screen hiding all the bootup trash.

But everything else seems to work.

Tomorrow is another day.

Monday, November 10, 2014

Reinstalling the operating system

BLUF: It's easy when you know how.

There are some times that try men's souls, and some are when you have to change your computer architecture or operation system (OS). 

This is referred to as migration.

Now, imagine the trials associated with doing both on two integrated computers, all at the same time.


This is how it went:

There is really only one computer (Hewlett Packard tm2t Touchscreen) in this case, with 8 Gigabytes (GB) of Random Access Memory (RAM) and a 500GB Hard Drive (HD) that has been divided ("partitioned") into two parts:

100GB "root" partition (called "/")
400GB "data" partition (called "/data")

A normal computer system comprises two basic physical elements: the physical Hard Drive (HD) and a set of memory chips for Random Access Memory (RAM).

Typically the HD has hundreds of "zerabytes" while the RAM only has 5-10 zerabytes.

(Here, zera is a placeholder for Kilo, Mega, Giga, Peta.... I can't find that it actually is assigned any value, but your mileage may vary. At present we are talking zera = GIGA: One Gigabyte (GB) =109 bytes.)

The HD may be either an actual physical spinning platter of some sort or a virtual drive comprising zillions of  chips on a super chip.

The RAM is most decidedly an array of chips on a board.

So in today's terms, a system might comprise 500 GB of HD space and 8 GB of RAM. The HD is used for storing information while the RAM is used for executing applications that use the stored information.

But then it gets more complicated, as the HD can be logically separated into several “partitions”, each with its own file structure and contents. Deciding on which directories to put in how many partitions is very much an art, not a science.

The basic idea is to allow disposable or easily reproducible information to reside in one partition and the more precious irreplaceable information to be barricaded somewhere else, so that if the first fails or is corrupted then the latter is protected.

In this representative system:

We have one partition (the "/" or "root" partition) containing all the elements associated with the operating system (OS): basic OS, the application files, hardware driver files, and so forth. Especially, files containing the user preferences, or “Settings”.

Then we have a second ("/data") partition that contains all the personal data, documents, and other information we want to keep safe. This drive has been encrypted with yet another password.

Then there is a bit of hybrid data, such as Settings, that need to live in the / partition to be read by the executing applications but are something of a pain to set up and a pain to lose if you have to reinstall. In this case we:

  • Build them in the / partition
  • Move them to the /data partion and then
  • Provide a pointer (called a symbolic link) in the / partition that points to their location in the /data partition.

The idea is to install the operating system and all the stuff you might want to change or upgrade in the / partition while keeping personal data safe from exploitation or corruption in the encrypted /data partition.

That way, when you upgrade the system you can reformat the / partition and install the new versions there without losing all your personal data:

You'll have to reconstruct the symbolic links, but that is a relatively trivial (easy when you know how) task if you've kept records of your links. See PPPPPP below...

So, having taken the time and discipline to record the applications, links, and repositories (more on those later) that you are using, it is an easy matter to install a new or reinstall the former OS. You simply reformat the first partition, replacing all its contents, reinstall the applications from your saved sources or updated repositories, and then reconstruct the symbolic links to settings.

Seldom more than perhaps an hour or so of work.

OK, so the first step was to migrate the basic machine from a 32-bit architecture to a 64-bit architecture.

What does this mean?

Architecture refers to the underlying physical structure of the computer. This includes the dynamic Random Access Memory (RAM) chips, where the moment-to-moment execution of program steps takes place, the more static Hard Drive (HD) storage where programs and data are stored (but not executed), and the overall structure of all of these.

Perhaps the most fundamental issue is “How much stuff can you usefully save?”

This depends not only on how big the box might be (RAM and HD size) but also whether you can find it once it is in the box.

Finding stuff on the HD is easy, you simply specify a path for all the directories and subdirectories to the file you seek.

But RAM has specific locations ("addresses"): different sets of transistors in the array of transistors where information starts and stops for a particular record. So you have to know how many addresses you can keep track of and read.

This, in turn, depends on how big your addresses are.

These addresses are defined by a certain number of bits: single states of a group of two-state (binary: on-off) transistors. So dig out your statistics book and calculator:

If you have x bits (transistors) per address you can save 2x different combinations of x bits, assuming you have enough RAM.
If you have 32-bit addresses then you can address 4,294,967,296 different (or about 4.3 GB) locations of memory.

But these days, RAM comes in tens of GB, so we need something else. By moving to a 64-bit architecture we can now address ~1.8 x 1019 (18 followed by eighteen digits, actually 18,446,744,073,709,551,616) different addresses, which indeed is a very big number.

Which might be a good thing to do. 

So we proceed with this:

  • Download the new operating system as an image of a DVD and burn it to a new DVD-R disk
  • Write and save a record in /data that summarizes:
    • All the programs we needed to reinstall after installing the new OS 
    • All the links we needed to reconstruct
    • Network settings and passwords
    • And, importantly, all the repositories that were being used
               In short, exercised the axiom: Prior planning prevents {...} poor performance (PPPPPP)...

  • Put the disk in the DVD drive and reboot the system
  • Make sure it is running well, and, well...
  • Go to the gym.
When we came back the new 64-bit OS was running, just fine.

Next, as expected, we reinstalled the programs and symbolic links from our record in /data.

Reinstalling the programs requires downloading them from the distribution's repositories.

OK, so what is all this blather about repositories?

Repositories are simply that: a place where piles of stuff are being saved. In computer terms they are internet locations where developers upload and "serve" the latest versions of the various applications and other files. And there are zillions of them. 

The point is that with repositories you have some semblance of assurance that the programs offered ("served") there have had some sort of oversight and review so that they are neither corrupted nor malware-infected. Not a guarantee, but a stronger assurance. Furthermore, the programs served there have more likely been checked for the other files that are needed for them to run (called "dependencies") so your hassle may be minimized in loading the applications from the repository.

Look: there are some people who like a top-down Bottom-Line-Up-Front approach (guess who) and some who like to work at perfecting the finest details of an object. Thanks be to God for both. The repositories are the domain of the latter. In Free and Open Source Software (FOSS) there are literally thousands, if not millions, of people around the world working through an elaborate cooperative scheme to develop, track, debug, and serve thousands if not millions of different applications, at least for Linux-based systems. These people fix bugs, find corruptions, and generally make the world a better and safer place.

For free. Because they want to and can.

So repositories are your friend. Download software from "official" repositories  whenever possible.

Configuring repositories is yet another art, rather than a science. They are a complex, albeit straightforward, functional decomposition of applications by categories. So if you are interested in a particular type of software, e.g., Education, then there is probably a repository that contains only applications associated with Education.

Topic for another time, but (remember the six Ps) if you record the repositories in the old installation you can then (at least for openSUSE) just change the version:


simply becomes


And so forth for the remaining (23 in my case) different repositories...


So, after all that we were done.

Well, Sort of. It's a bit more complicated.

You see, we're running yet another machine in software inside of this machine. This is called Virtualization:

Virtualization is a technique that allows you to embed one computer inside of another.

A virtualization application, such as Oracle VirtualBox, VMWare, or several others, creates an entirely new computer in software; hence the term “virtual machine” or VM. 

The virtualization application (e.g., VirtualBox) resides in the first (/ or root) partition and is executed in RAM. It reads and operates on the virtual machine (VM), which is a rather huge single file that resides in the /data partition of the parent (host) machine and is referred to as the guest machine.

The guest in turn contains one or more (virtual) HD partitions of its own, each with its own set of an operating system, applications, and user data which can be and usually are completely different from those of the host system. And it has its own (virtual) RAM, necessarily smaller than that of the host, since it has to use the host RAM resources.

This process of virtualizing a machine within a machine within yet another machine... can continue as long as time (and actual hard disk and RAM space) exist..

(I'm reminded of the broom in the Sorcerer's Apprentice... smart man, Walt Disney...)

So, upgrading all this is not a trivial matter: 

You can choose to upgrade the host system without affecting the embedded (guest) system.
You can choose to upgrade the guest without affecting the parent host system.
But there are physical limitations to HD and RAM space, so if you want to upgrade both, then there is a definite cascade of steps that you must take in order to succeed:

First upgrade the top level host system:

The new main operating system (in this case openSUSE 13.1 ) is a 64-bit system, while the older system was 32-bit. So we could not use the normal zypper distribution upgrade process,   which simply requires renaming the repositories and issuing a single command:
     zypper dup

Instead, it required a "clean" install: reformatting the / partition, reinstalling all the applications, and reconstituting the symbolic links. Not a huge effort but, as noted, still more effort than
     zypper dup.

Next upgrade the first level guest system
The Windows migration in the virtual machine was from 32-bit Windows XP to 64-bit Windows 7. This required a bit more effort:

  • First we had to increase RAM in the VM from 1GB to 2GB. Merely a click of a button once we knew how, but finding out how took a few hours. First we had to take the machine apart to confirm that there was enough physical RAM present, since free only showed 2GB. It turns out that we were using a 32-bit system without PAE that, for the reasons discussed above, cannot see more than 4 GB.  The VM assumed the host only had 4 GB and would not allow increasing VM RAM beyond 1 GB.

This was solved by installing a PAE kernel and trying again (push the car back up the hill... :-(

  • Next we had to increase the virtual HD from 20 GB to 40 GB, according to Microsoft's system requirements page. But hello, that required a partitioning tool. Progressive and straight-forward, once you know how, but more time gone, downloading, mounting, and farkling about. You have to first increase the size that VirtualBox allows in the host, then actually go into the guest operating system and increase the partition there as well.
  • Then we cloned the system to ensure we still had a system if all else failed (see PPPPP)
  • This caused a conflict since the new drive had the same UUID as the old drive (duh, look up clone - how do you spell "genetically identical") so there was some more messing about to figure out how to change the UUID.

But eventually we were ready and followed the excellent instructions at


to succeed.

And of course there are always some dirty little secrets that you have to figure out...


Well that was then (a year ago) and this is now. Now I want to move from 64-bit openSUSE 13.1 to 64-bit openSUSE 13.2 without changing the contained VM.

According to this account that should simply be a matter of renaming the repositories in yast2 sw-single from 13.1 to 13.2, and then issue the command zypper dup from the root command line.

Can it really be that simple? Well, not quite that simple, but simple enough.

Watch this space:

     zypper repos --uri
     cp -Rv /etc/zypp/repos.d /etc/zypp/repos.d.Old
     sed -i 's/13\.1/13.2/g' /etc/zypp/repos.d/*
     zypper ref
          The following failed, so we chose ignore and then removed them from the repo list:
               file systems
          All the rest worked fine.

So now holding breath: There are dire warnings of consequences, so I'll close everything down, log out to a new root session, and issue the fatal command:
     zypper dup --download "in-advance"

Friday, November 7, 2014

Très cool: Convert .ics to .csv

One more step towards freedom.

This gives me an ICS ==> CSV translation that renders

in chronological order.


I know about logging programs, billable time gigs, and the lot. There are lots of them out there, you can Google for them.

But I wanted something that would just simply log what I do. And, if I decide to take a nap, record idle time, and in the morning tell me what time I really went to sleep.

Simple stuff.


The only thing I 've found like that is a wonderful app for Windows, but I'm not on Windows. I've given him a salute there, but this is here: Linux.

Eventually I found KTimeTracker. It is indeed very cool, collects what you do by the window title, when you started, when you stopped. And has a bunch of reports on how much time you spent on this or that.

But it doesn't give you the times...

It does have a File > Edit History feature that shows exactly what I want, but there is no way to show it or export it.

KTimeTracker does export an .ics file. I had written a BASH script to restart it with a new .ics file. The script is called time_new, which I run every day when I start work. This starts a new .ics file and calls a routine called time_awk.

Now the rubber is starting to meet the road.

I had written this some while ago to parse the .ics file into single line log and event rows, but it was still a mess.

So, then doing what I knew how to do, I delved into writing some Windows Visual Basic macros to further refine it into that which I wanted, that of KTimeTracker's History screen, that could then be saved as an Excel spreadsheet.

But this is tedious. First, I almost never use Windows except for this, then there is the bloatware of starting up Windows 7 (under VirtualBox in my case), and then the actual parsing takes forever under VirtualBox. After the macro runs you get the hourglass forever unless you switch out of the guest to the host Linux platform and back. I haven't taken the time to try to figure out that one.

What a PITA.

So I finally bit the bullet. It took me a couple of days, but I have now constructed an AWK file that does exactly what I want: 
  • Translate an ICS file exported by KTimeTracker into a CVS file that can be opened in LibreOffice to show:

in chronological order.

In the process, I've learned a lot more about AWK, SED, and the interactions with the various shells: SH, BASH, CSH, etc.

And yes, Virginia, there is a difference. The most maddening is in the ability (or not) to assign shell variables values from within AWK.

I won't take the time now to explore all that, just give you the answer. It is well commented, but as always, if you don't know what you're doing you might want to wait until you do.

In the meantime, all you gotta do is point this at your .ics file and the result will be a CSV-formatted .xls file with the same name, as in:

     time_awk 141105_andy.ics

that results in


Easy when you know how.



Here it is, under the standard GPL:

#! /bin/bash
# License: LGPL v3+ (see the file LICENSE)
# (c)2001-2014 C. Andrews Lavarre
# email : alavarre@gmail.com
# This program is free software; you can redistribute it and/or modify #
# it under the terms of the GNU General Public License as published by #
# the Free Software Foundation; either version 3 of the License, or    #
# (at your option) any later version.                                  #
#                                                                      #
# This program is distributed in the hope that it will be useful,      #
# but WITHOUT ANY WARRANTY; without even the implied warranty of       #
# GNU General Public License for more details.                         #
#                                                                      #
# The GNU General Public License is posted at                          #
#    http://www.gnu.org/licenses/gpl.txt                               #
# You may also write to the Free Software Foundation, Inc.,            #
# 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA             #

# This routine was modified 141106 by C. A. Lavarre (Andy).
# It invokes AWK to parse an ICS file
# Arguments:
# $1 the path (optional) and filename of the ics
# e.g., /data/info/korganizer/archive/andy/130112_andy.ics
# Usage
# time_awk $1
# Save the input
# Check the input
if [ "$target" == "" ]; then
#Complain and offer options
echo "Usage:"
echo "time_awk workingdirectoryandicsfilename"
echo "e.g., /data/info/korganizer/archive/andy/130112_andy.ics"
echo "Quitting"
exit 0
# Finish
# Get the word and its length
# Get the file
mydir=$(dirname $source)
# If blank then
if [ "$mydir" == "." ]; then
# Set the working directory and basename
# Finish
# Show resultsmyfile
# Name the new .xls file
    cd $basedir
    echo "Saving "$target" awk results to "$output
    echo "Continue? (y|n):"
    read user
    if [ "$user" == "n" ]; then
# Quit
echo "cd to the correct directory, follow the usage."
echo "Quitting"
exit 0
# Finish
# Build the file
# Usage: awk [POSIX or GNU style options] [--] 'program' file ...
# awk -F 'all the instructions' $target
# the -F : option declares the colon as a field separator
# Between DTSTART and DTEND {do THIS} (^ means look at start of line)
# Then after SUMMARY {do THAT} 
# and after COMMENT {do THE OTHER}
# and after END:VEVENT {FINISH}
# THIS is 
# {printf "%s," $2} which instructs to
# print the second field $2 as a string
# THAT is
# {printf "\"%s\",", $2} which instructs to
# print the second field $2 as a string in double quotes followed by an @ sign as the delimiter
# {c=$2} which instructs to
# set the variable c to (copy) the value of the second field
# {print c; c=""} which instructs to
# Print the COMMENT and clear the copy variable
# then pipe the whole lot to the tr command:
# Usage: tr [OPTION]... SET1 [SET2]
#Translate, squeeze, and/or delete characters from standard input,
# tr -d '\015' deletes character with octal value 015 (CR)
# Pipe it to awk to strip off the first line
# Pipe it to sed
# sed 's/,//g' removes all commas 
# Pipe it to awk to remove lines with only two fields
# Pipe it to Sort it by columns #2 and #3 
# then repeat this for all lines 
# and send the lot to $output
awk -F :\
'/^DTSTART/||/^DTEND/ {printf "%s@", $2}\
/^SUMMARY/ {printf "\"%s\"@", $2}\
/END:VEVENT/ {print c; c=""}' $target\
| tr -d '\015'\
| awk 'NR>1'\
| sed 's/,//g'\
| awk -F @ '$3!=""'\
| sort -t "@" -k2\
> $output
chmod +x $output
# Create alternate work files
  # Pipe it to awk to 
# Move field $3 to $4
# and send the lot to $output1
awk -F @\
'{$(NF)=$3;}1' OFS="," $output > $output1
# Split field #2 on the T delimiter into $2 and $3:
# Split field $4 on the T delimiter into $4 and $5
# and send the lot to the file $output2
awk -F ,\
{split($2, a, "T"); $3 = a[2]; $2 = a[1];}\
{split($4, a, "T"); $5 = a[2]; $4 = a[1];}\
{ print $0;}
' OFS="," $output1 > $output2
# Convert columns $2 and $4  in YYYYMMDD to MM/DD/YYYY
# Convert columns $3 and $5 to HH:MM:SS
# and send the lot to $output
awk -F ,\
{$2 = substr($2,5,2)"/"substr($2,7,2)"/"substr($2,1,4)}\
{$3 = substr($3,1,2)":"substr($3,3,2)":"substr($3,5,2)}\
{$4 = substr($4,5,2)"/"substr($4,7,2)"/"substr($4,1,4)}\
{$5 = substr($5,1,2)":"substr($5,3,2)":"substr($5,5,2)}\
{print $0;}
' OFS="," $output2 > $output
# Delete the temporary files
rm $output1
rm $output2
# Report

echo "Parsing complete."