Monday, November 10, 2014

Reinstalling the operating system

BLUF: It's easy when you know how.

There are some times that try men's souls, and some are when you have to change your computer architecture or operation system (OS). 

This is referred to as migration.

Now, imagine the trials associated with doing both on two integrated computers, all at the same time.


This is how it went:

There is really only one computer (Hewlett Packard tm2t Touchscreen) in this case, with 8 Gigabytes (GB) of Random Access Memory (RAM) and a 500GB Hard Drive (HD) that has been divided ("partitioned") into two parts:

100GB "root" partition (called "/")
400GB "data" partition (called "/data")

A normal computer system comprises two basic physical elements: the physical Hard Drive (HD) and a set of memory chips for Random Access Memory (RAM).

Typically the HD has hundreds of "zerabytes" while the RAM only has 5-10 zerabytes.

(Here, zera is a placeholder for Kilo, Mega, Giga, Peta.... I can't find that it actually is assigned any value, but your mileage may vary. At present we are talking zera = GIGA: One Gigabyte (GB) =109 bytes.)

The HD may be either an actual physical spinning platter of some sort or a virtual drive comprising zillions of  chips on a super chip.

The RAM is most decidedly an array of chips on a board.

So in today's terms, a system might comprise 500 GB of HD space and 8 GB of RAM. The HD is used for storing information while the RAM is used for executing applications that use the stored information.

But then it gets more complicated, as the HD can be logically separated into several “partitions”, each with its own file structure and contents. Deciding on which directories to put in how many partitions is very much an art, not a science.

The basic idea is to allow disposable or easily reproducible information to reside in one partition and the more precious irreplaceable information to be barricaded somewhere else, so that if the first fails or is corrupted then the latter is protected.

In this representative system:

We have one partition (the "/" or "root" partition) containing all the elements associated with the operating system (OS): basic OS, the application files, hardware driver files, and so forth. Especially, files containing the user preferences, or “Settings”.

Then we have a second ("/data") partition that contains all the personal data, documents, and other information we want to keep safe. This drive has been encrypted with yet another password.

Then there is a bit of hybrid data, such as Settings, that need to live in the / partition to be read by the executing applications but are something of a pain to set up and a pain to lose if you have to reinstall. In this case we:

  • Build them in the / partition
  • Move them to the /data partion and then
  • Provide a pointer (called a symbolic link) in the / partition that points to their location in the /data partition.

The idea is to install the operating system and all the stuff you might want to change or upgrade in the / partition while keeping personal data safe from exploitation or corruption in the encrypted /data partition.

That way, when you upgrade the system you can reformat the / partition and install the new versions there without losing all your personal data:

You'll have to reconstruct the symbolic links, but that is a relatively trivial (easy when you know how) task if you've kept records of your links. See PPPPPP below...

So, having taken the time and discipline to record the applications, links, and repositories (more on those later) that you are using, it is an easy matter to install a new or reinstall the former OS. You simply reformat the first partition, replacing all its contents, reinstall the applications from your saved sources or updated repositories, and then reconstruct the symbolic links to settings.

Seldom more than perhaps an hour or so of work.

OK, so the first step was to migrate the basic machine from a 32-bit architecture to a 64-bit architecture.

What does this mean?

Architecture refers to the underlying physical structure of the computer. This includes the dynamic Random Access Memory (RAM) chips, where the moment-to-moment execution of program steps takes place, the more static Hard Drive (HD) storage where programs and data are stored (but not executed), and the overall structure of all of these.

Perhaps the most fundamental issue is “How much stuff can you usefully save?”

This depends not only on how big the box might be (RAM and HD size) but also whether you can find it once it is in the box.

Finding stuff on the HD is easy, you simply specify a path for all the directories and subdirectories to the file you seek.

But RAM has specific locations ("addresses"): different sets of transistors in the array of transistors where information starts and stops for a particular record. So you have to know how many addresses you can keep track of and read.

This, in turn, depends on how big your addresses are.

These addresses are defined by a certain number of bits: single states of a group of two-state (binary: on-off) transistors. So dig out your statistics book and calculator:

If you have x bits (transistors) per address you can save 2x different combinations of x bits, assuming you have enough RAM.
If you have 32-bit addresses then you can address 4,294,967,296 different (or about 4.3 GB) locations of memory.

But these days, RAM comes in tens of GB, so we need something else. By moving to a 64-bit architecture we can now address ~1.8 x 1019 (18 followed by eighteen digits, actually 18,446,744,073,709,551,616) different addresses, which indeed is a very big number.

Which might be a good thing to do. 

So we proceed with this:

  • Download the new operating system as an image of a DVD and burn it to a new DVD-R disk
  • Write and save a record in /data that summarizes:
    • All the programs we needed to reinstall after installing the new OS 
    • All the links we needed to reconstruct
    • Network settings and passwords
    • And, importantly, all the repositories that were being used
               In short, exercised the axiom: Prior planning prevents {...} poor performance (PPPPPP)...

  • Put the disk in the DVD drive and reboot the system
  • Make sure it is running well, and, well...
  • Go to the gym.
When we came back the new 64-bit OS was running, just fine.

Next, as expected, we reinstalled the programs and symbolic links from our record in /data.

Reinstalling the programs requires downloading them from the distribution's repositories.

OK, so what is all this blather about repositories?

Repositories are simply that: a place where piles of stuff are being saved. In computer terms they are internet locations where developers upload and "serve" the latest versions of the various applications and other files. And there are zillions of them. 

The point is that with repositories you have some semblance of assurance that the programs offered ("served") there have had some sort of oversight and review so that they are neither corrupted nor malware-infected. Not a guarantee, but a stronger assurance. Furthermore, the programs served there have more likely been checked for the other files that are needed for them to run (called "dependencies") so your hassle may be minimized in loading the applications from the repository.

Look: there are some people who like a top-down Bottom-Line-Up-Front approach (guess who) and some who like to work at perfecting the finest details of an object. Thanks be to God for both. The repositories are the domain of the latter. In Free and Open Source Software (FOSS) there are literally thousands, if not millions, of people around the world working through an elaborate cooperative scheme to develop, track, debug, and serve thousands if not millions of different applications, at least for Linux-based systems. These people fix bugs, find corruptions, and generally make the world a better and safer place.

For free. Because they want to and can.

So repositories are your friend. Download software from "official" repositories  whenever possible.

Configuring repositories is yet another art, rather than a science. They are a complex, albeit straightforward, functional decomposition of applications by categories. So if you are interested in a particular type of software, e.g., Education, then there is probably a repository that contains only applications associated with Education.

Topic for another time, but (remember the six Ps) if you record the repositories in the old installation you can then (at least for openSUSE) just change the version:

simply becomes

And so forth for the remaining (23 in my case) different repositories...


So, after all that we were done.

Well, Sort of. It's a bit more complicated.

You see, we're running yet another machine in software inside of this machine. This is called Virtualization:

Virtualization is a technique that allows you to embed one computer inside of another.

A virtualization application, such as Oracle VirtualBox, VMWare, or several others, creates an entirely new computer in software; hence the term “virtual machine” or VM. 

The virtualization application (e.g., VirtualBox) resides in the first (/ or root) partition and is executed in RAM. It reads and operates on the virtual machine (VM), which is a rather huge single file that resides in the /data partition of the parent (host) machine and is referred to as the guest machine.

The guest in turn contains one or more (virtual) HD partitions of its own, each with its own set of an operating system, applications, and user data which can be and usually are completely different from those of the host system. And it has its own (virtual) RAM, necessarily smaller than that of the host, since it has to use the host RAM resources.

This process of virtualizing a machine within a machine within yet another machine... can continue as long as time (and actual hard disk and RAM space) exist..

(I'm reminded of the broom in the Sorcerer's Apprentice... smart man, Walt Disney...)

So, upgrading all this is not a trivial matter: 

You can choose to upgrade the host system without affecting the embedded (guest) system.
You can choose to upgrade the guest without affecting the parent host system.
But there are physical limitations to HD and RAM space, so if you want to upgrade both, then there is a definite cascade of steps that you must take in order to succeed:

First upgrade the top level host system:

The new main operating system (in this case openSUSE 13.1 ) is a 64-bit system, while the older system was 32-bit. So we could not use the normal zypper distribution upgrade process,   which simply requires renaming the repositories and issuing a single command:
     zypper dup

Instead, it required a "clean" install: reformatting the / partition, reinstalling all the applications, and reconstituting the symbolic links. Not a huge effort but, as noted, still more effort than
     zypper dup.

Next upgrade the first level guest system
The Windows migration in the virtual machine was from 32-bit Windows XP to 64-bit Windows 7. This required a bit more effort:

  • First we had to increase RAM in the VM from 1GB to 2GB. Merely a click of a button once we knew how, but finding out how took a few hours. First we had to take the machine apart to confirm that there was enough physical RAM present, since free only showed 2GB. It turns out that we were using a 32-bit system without PAE that, for the reasons discussed above, cannot see more than 4 GB.  The VM assumed the host only had 4 GB and would not allow increasing VM RAM beyond 1 GB.

This was solved by installing a PAE kernel and trying again (push the car back up the hill... :-(

  • Next we had to increase the virtual HD from 20 GB to 40 GB, according to Microsoft's system requirements page. But hello, that required a partitioning tool. Progressive and straight-forward, once you know how, but more time gone, downloading, mounting, and farkling about. You have to first increase the size that VirtualBox allows in the host, then actually go into the guest operating system and increase the partition there as well.
  • Then we cloned the system to ensure we still had a system if all else failed (see PPPPP)
  • This caused a conflict since the new drive had the same UUID as the old drive (duh, look up clone - how do you spell "genetically identical") so there was some more messing about to figure out how to change the UUID.

But eventually we were ready and followed the excellent instructions at

to succeed.

And of course there are always some dirty little secrets that you have to figure out...


Well that was then (a year ago) and this is now. Now I want to move from 64-bit openSUSE 13.1 to 64-bit openSUSE 13.2 without changing the contained VM.

According to this account that should simply be a matter of renaming the repositories in yast2 sw-single from 13.1 to 13.2, and then issue the command zypper dup from the root command line.

Can it really be that simple? Well, not quite that simple, but simple enough.

Watch this space:

     zypper repos --uri
     cp -Rv /etc/zypp/repos.d /etc/zypp/repos.d.Old
     sed -i 's/13\.1/13.2/g' /etc/zypp/repos.d/*
     zypper ref
          The following failed, so we chose ignore and then removed them from the repo list:
               file systems
          All the rest worked fine.

So now holding breath: There are dire warnings of consequences, so I'll close everything down, log out to a new root session, and issue the fatal command:
     zypper dup --download "in-advance"

No comments: