Saturday, May 6, 2017

Transition Evolution to a new installation

I'm moving technology posts to 
     https://privustech.com/

We recently converted the operating system from openSUSE Leap 42.2 to Linux Mint (8) KDE. Doing so required transfering our Evolution data to the new OS.
Details at

Friday, February 24, 2017

Chromium video playback [Solved]

http://privustech.com/index.php/2017/02/24/chromium-video-playback-solved/


We were unable to view video in Chromium
     Version 54.0.2840.100 (64-bit)
     Linux openSUSE 42.1.
The solution is here:
     Open
          chrome://flags/
     in Chromium.

Disable:
     Override software rendering list Mac, Windows, Linux, Chrome OS,
     AndroidOverrides the built-in software rendering list and 
     enables GPU-acceleration on unsupported system configurations.
     #ignore-gpu-blacklist
Restart Chromium.
Problem solved.

Monday, February 13, 2017

Missing Cat

This happens all the time. It just happened again in the neighborhood. 

After having provided this information several times in the past to others, it suddenly occurred to me: why not just blog it? Then I can simply provide the URL and we all win.


======================


So sorry, I feel your pain. My little buddy went missing for a month.

Finally found him: curiosity almost killed the cat:

There was a family moving out a few houses up the street, so the basement covers were open, and curious Hank went in. As they had just finished loading the truck they slammed the covers shut and left, trapping Hank inside. 


The out-of-town owner returned a month later to prep for the next set of renters and heard Hank yowling (Siamese). He had seen my posters, so knew who to call.


Hank had lost five of ten pounds. The only thing that kept him alive was a leaking pipe, so he wasn't dehydrated. 

All's well that ends well and he lived another seven years. 

So in the meantime, some tips:

1. Report this to the local authorities (
e.g., Police, Potter League, Newport Animal Hospital), and other veterinarians and animal shelters. People often turn in lost animals, and like it or not (reality): there are some nasty people out there (some of whom I met in my ordeal) who will simply snatch a pet out of jealousy or just plain meanness.

2. Register with www.tabbytracker.com

3. Offer a reward. (The finder refused. Thank you.)

4. Make posters and put them up around the neighborhood. Many owners are out of town and don't read the local social media. For example:

5. Go around knocking door to door.


6. Read up on "Pet Detectives". Google is your friend:

     http://tinyurl.com/hpnt7zx

I went with Carl:

     http://www.petdetectiveusa.com/

At the time he offered a DIY paper with all the considerations for $30 (as I recall), I can't find that now, sorry.

There is a whole culture out there about this problem: they hide in storm drains, get disoriented and can't find their way home, especially if hurt or scared, etc. etc. etc.

7. When you find him, erect an invisible fence so it won't happen again:
     Moriarty in Portsmouth or Warwick:

          http://tinyurl.com/jlfnkz3
They are designed for dogs, but work equally as well for cats. I'd put the collar on him and let him out in the yard, so he could do his cat thing of lying in the sun and chasing birds and other critters, but if he tried even approaching the fence within a foot or so he got a nasty shock. So no way to jump the fence. Tough love.

Again, so sorry, but there is much you can do beyond social media. 


I hope you find your buddy.

Friday, February 10, 2017

Easy when you know how: Let's Encrypt open source "automated" TLS/SSL certificates and keys

We recently renewed our StartCom certs for our vhost domains.

The sites immediately failed SSL:
     Chromium reported
          NET::ERR_CERT_AUTHORITY_INVALID

     Firefox reported
          SEC_ERROR_REVOKED_CERTIFICATE
     Qualys
          https://www.ssllabs.com/ssltest/analyze.html

          reported 
          Trusted No NOT TRUSTED

It turns out that Apple, Mozilla, Google no longer trust StartCom:
https://linustechtips.com/main/topic/688200-apple-google-and-mozilla-disavow-wosign-and-startcom-certificates/
https://serverfault.com/questions/829298/my-certificate-issued-by-startssl-is-not-accepted-by-my-clients.
so the CA (Certifying Authority) cert(ificate) was not being accepted.

So, what to do? We decided to go with Let's Encrypt:
     https://letsencrypt.org/

Easy when you know how, except we didn't. But we do now. After working through this tutorial you shall also. 
This post provides a top level of the gotchas we found along the path.

==========

The system is a bit short on concise documentation (hence this post) but there is a helpful wiki:
     https://community.letsencrypt.org/latest
and, as always, Google is your friend. 


First, Let's Encrypt is a system put in place by the EFF (Electronic Frontier  Foundation) to provide free TLS/SSL certificates:
     https://letsencrypt.org/
     https://letsencrypt.org/how-it-works/

The objective of Let’s Encrypt and the ACME protocol is to make it possible to set up an HTTPS server and have it automatically obtain a browser-trusted certificate, without any human intervention. This is accomplished by running a certificate management agent on the web server.

The agent operates under the ACME (Automatic Certificate Management Environment) protocol. Any number of clients for the ACME server exist: the protocol is open source, so anyone can write a client for it and many have.

letsencrypt.org recommends we use their “official” client CertBot:
https://certbot.eff.org/about/
Certbot is an easy-to-use automatic client that fetches and deploys SSL/TLS certificates for your webserver. Certbot was developed by EFF.
Certbot is part of EFF’s larger effort to encrypt the entire Internet. Websites need to use HTTPS to secure the web.


Except that we were unable to get it to work under openSUSE. It is not in the normal repositories. We found several ymp packages but all failed either to install or installed but failed to run:
     https://software.opensuse.org/package/certbot fails to run after installation.

     https://rootco.de/2016-05-16-letsencrypt-on-leap/ clones a git installation. It runs, but only checks its installation:
     certbot-auto certonly --webroot -w /srv/www/htdocs/gvhl \ -d genietvanhetleven.org
Bootstrapping dependencies for openSUSE-based OSes...
Loading repository data...
Reading installed packages..

xxx is already installed...
Nothing to do.


     https://letsencrypt.org/docs/client-options/ lists a huge number of alternative. But which?

     https://community.letsencrypt.org/t/cerbot-failing-when-installing-python-packages/22722 suggests GetSSL:
     GetSSL is designed so you can also run it from a different server, providing you have SSH / SFTP to the server you want to place the certs on if needed, for servers / devices where you couldn't run most other clients.

Ultimately we went with Dehydrated:
     https://github.com/lukas2511/dehydrated

We found 
     https://software.opensuse.org/download.html?project=security&package=dehydrated
and it just runs. But it needs configuration.



Configuring Dehydrated

To begin please note that, unlike StartCom, CertBot, and GetSSL, you need to install and run Dehydrated on the target server machine, not a proxy.

These are the steps:
1. Install Dehydrated
2. Set the Staging
3. Create domains.txt
4. Create well-known
5. Create /etc/dehydrated/config

6. Configure Apache2
7. Start it:
     # dehydrated -c -f /etc/dehydrated/config
Now: Success!
8. Reverse the Staging and reexecute
9. Get the CA certs
10. Link the /etc/dehydrated/certs files to /etc/apache2/ssl.crt and ...key
11. Amend the apache .conf files to address these.
12. Now check the certs.
13. Go to production.

1. Install

openSUSE installation requires that you add the repository with yast
on the server:

http://download.opensuse.org/repositories/security/openSUSE_Leap_42.1/

Then use yast sw_single to install:
     dehydrated
     dehydrated-apache

2. Staging

https://github.com/lukas2511/dehydrated/blob/master/docs/staging.md
Let’s Encrypt has stringent rate limits in place.
If you start testing using the production endpoint (which is the default), you will quickly hit these limits and find yourself locked out.
To avoid this, please set the CA property to the Let’s Encrypt staging server URL in your
config file:
CA="https://acme-staging.api.letsencrypt.org/directory"
CA_TERMS=
"https://acme-staging.api.letsencrypt.org/terms"


https://community.letsencrypt.org/t/cn-fake-le-intermediate-x1/13437
That means you issued those certificates against the staging server (possibly with --test-cert or --dry-run). You should reissue against the production server by removing those flags from your client invocation, if present.

Once you've gotten things working mostly, comment out these to get actual (not fake) certs:
https://calomel.org/lets_encrypt_client.html
Once your testing is done against the test certificate authority server (happy hacker fake CA) you need to edit the script to point to the production certificate authority server to generate a valid certificate. Edit the script and look for the variable at the top called "CA=" and uncomment out the "official server" and comment the "testing server". For example:
# The Lets Encrypt certificate authority URL
#CA="https://acme-staging.api.letsencrypt.org" # testing server, high rate limits. "happy hacker fake CA"
CA="https://acme-v01.api.letsencrypt.org" # official server, rate limited to 5 certs per 7 days


DO NOT delete the staging certs: it will break apache! Just let letsencrypt replace them when you go to production.


After all the testing is done you will need to run it with the -x option, since you will have created test certificates and it will refuse to regenerate actual certificates unless forced to do so:
https://community.letsencrypt.org/t/this-is-not-going-well/27366/4
     dehydrated -c -x -f /etc/dehydrated/config


3. domains.txt

https://github.com/lukas2511/dehydrated/blob/master/docs/domains_txt.md
Create /etc/dehydrated/domains.txt on the server:
Dehydrated uses the file domains.txt as a configuration file for which certificates should be requested.
     /etc/dehydrated/domains.txt:
          example1.com www.example1.com
          example2.net www.example2.net

4. well-known

https://github.com/lukas2511/dehydrated/blob/master/docs/wellknown.md
     O
n the server machine:

     # cd /var     
     # mkdir www     
     # mkdir www/dehydrated
     # mkdir www/dehydrated/.well-known
     # mkdir www/dehydrated/.well-known/acme-challenge
     # touch www/dehydrated/.well-known/acme-challenge/m4g1C-t0k3n

Change the owner and group
     # cd /var/www/dehydrated
     # chown -R wwwrun:www .well-known

Change the mod from 0755 to 0775

     # chmod -R 0775 .well-known

Copy it to each :

     # cp -R .well-known /srv/www/htdocs/
     # cp -R .well-known /srv/www/htdocs/gvhl

     # cp -R .well-known /srv/www/htdocs/nptbeyond
     # cp -R .well-known /srv/www/htdocs/privustech
     # cp -R .well-known /srv/www/htdocs/truthcourage
/etc/dehydrated/config:

     WELLKNOWN=/var/www/dehydrated

5. config file

https://www.aaflalo.me/2016/09/dehydrated-bash-client-lets-encrypt/
     /etc/dehydrated/config
is mostly commented out, showing the default values.

Two values must be uncommented and corrected:
     # E-mail to use during the registration (default: )
     
     CONTACT_EMAIL=alavarre@gmail.com
      HOOK=/etc/dehydrated/hook.sh

6. Configure Apache2

We must add a dehydrated.conf file in /etc/apache2/sysconfig.d:
Edit /etc/apache2/httpd.conf to add
     Include /etc/apache2/sysconfig.d/dehydrated.conf

Create /etc/apache2/sysconfig.d/dehydrated.conf to read:
     Alias /.well-known/acme-challenge /var/www/dehydrated     
     Options None

     AllowOverride None
     # Apache 2.x     
     Order allow,deny

     Allow from all
     # Apache 2.4     
     Require all granted

The #Apache 2.x section is deprecated so we delete it.

Restart the server.

7. Start it

The first time you run it it will be in the Staging mode.
     # dehydrated -c -f /etc/dehydrated/config
Now: Success!


8. Reverse the Staging and reexecute


9. Get the CA certs

LetsEncrypt keeps their root offline, so we need to use the active Intermediate as the CA cert. But which? They have a number of different intermediate certs:
     https://letsencrypt.org/certificates/
     Our intermediate “Let’s Encrypt Authority X3” represents a single public/private key pair. The private key of that pair generates the signature for all end-entity certificates (also known as leaf certificates), i.e. the certificates we issue for use on your server.

The one that works for us is the cross-signed cert:
lets-encrypt-x3-cross-signed.pem
https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem

So we would expect that cert to verify the leaf cert:
lavarre:/etc/apache2/ssl.crt # openssl verify -verbose -CAfile lets-encrypt-x3-cross-signed.pem genietvanhetleven.org.pem
genietvanhetleven.org.pem: OK
 

And indeed, it does. Woohoo!

10. Links

Link the /etc/dehydrated/certs files to /etc/apache2/ssl.crt and ...key

Link to the CA cert:
# cd /etc/apache2/ssl.crt
# ln -s lets-encrypt-x3-cross-signed.pem letsencrypt_CA.pem


Link to the Domain Certificates and Keys Link to each cert in /etc/apache2/ssl.crt.
Link to each key in /etc/apache2/ssl.key.
We wrote a BASH script tldr_link that automates this:
     for DOMAIN_TLDR in {example1.com,example2.net}
     do
     
     cd /etc/apache2/ssl.crt
      ln -s /etc/dehydrated/certs/$DOMAIN_TLDR/cet.pem $DOMAIN_TLDR.pem
     ln -s /etc/dehydrated/certs/$DOMAIN_TLDR/chain.pem $DOMAIN_TLDR_chain.pem
     ln -s /etc/dehydrated/certs/$DOMAIN_TLDR/fullchain.pem $DOMAIN_TLDR_fullchain.pem
     cd /etc/apache2/ssl.key
     ln -s /etc/dehydrated/certs/$DOMAIN_TLDR/privkey.pem $DOMAIN_TLDR_key.pem
     done



11. Amend the apache.conf files

Amend the apache .conf files to address these. entities.

We check configuration files. Different files have different server names CA cert files. So we make them all point to
     ServerName privustech.com:993
     ServerAdmin alavarre@privustech.com
     SSLCACertificatePath /etc/ssl/certs
     SSLCACertificateFile /etc/ssl/certs/letsencrypt_CA.pem


Update all of these
     /etc/apache2/default-server.conf
     /etc/apache2/default-vhost-ssl.conf
     /etc/apache2/default-vhost.conf
     /etc/apache2/vhost.conf



12.Check the certs

https://www.aaflalo.me/2016/09/dehydrated-bash-client-lets-encrypt/
Now you should find a folder /etc/dehydrated/certs/ with a folder for each of your domain set in your domains.txt file.
In each of those folders, you’ll find two important symbolic links that you need to use in all your application that rely on that certificate-key pair.
fullchain.pem :  /etc/dehydrated/certs/example.com/fullchain.pem
privkey.pem :  /etc/dehydrated/certs/example.com/privkey.pem
The first one is your certificate will the different root certificates prepended to it, in other words, the one you need to set for your service. The second one is the private key of the certificate.

1. Verify the key-cert pairs (If more than one hash is displayed, they don't match):
# (openssl x509 -noout -modulus -in /etc/apache2/ssl.crt/example1.com.pem | openssl md5;openssl rsa -noout -modulus -in /etc/apache2/ssl.key/example1.com_key.pem | openssl md5) | uniq
(stdin)= 6b93be29e02041403ad7b0903bd8acc4


2. Verify the chain
# openssl verify -verbose -CAfile example1.com_chain.pem example1.com.pem
example1.com.pem: OK


3. Check the dates
# openssl x509 -startdate -enddate -noout -in example1.com.pem
notBefore=Jan 23 20:17:30
2017 GMT
notAfter=Jan 23 20:17:30
2020 GMT


13. Go to production

Reverse the Staging in /etc/dehydrated/config and run
dehydrated -c -x -f /etc/dehydrated/config

Restart the server:
     service apache2 restart
and check your site(s) in a browser.
https:// is working!

So much for "automatic". But easy when you know how. We all now do. :-)

Thursday, January 26, 2017

Statistical Analysis Overview

Statistical Analysis Overview1
Major Entities
There are two major entities in Applied Statistics:
• Data
• Meta Data
Data
Data comprise the raw sample information we collect as well as the results of analyzing the samples.
Canonical notation presents these as multivariate variables2 in an array, e.g.:
A sequence or other collection of random variables is independent and identically distributed ("i.i.d.") if each random variable has the same probability distribution as the others and all are mutually independent.3
The results of a function of a variable are also data:
R = F(X)
These are our test data for the R Project for Statistical Computing4 program that we are using:
We will use these for demonstration in the remainder of this paper.
Metadata
Metadata present the overlying processes and methods that interrelate the data.
Processes, functions, and analytic parameters are metadata.
The processes of statistical analysis comprise models and tests on both the model and its results.5
Statistics
Descriptive Statistics
Descriptive Statistics comprise calculations that describe the sample set:
Models
We build a mathematical model (a “regression”) to describe the relationships between input variables and the observed results. Most frequently the model is a linear regression of the form:
[R] = [A]x[X]+[B]
in standard matrix algebra notation.
Correlation analysis6
The first step is to validate the model. We do this with correlation analysis to determine how closely the model matches the observed samples. The sample data are used to compute r, the correlation coefficient for the sample. The symbol for the population correlation coefficient is ρ, the Greek letter "rho":
• ρ = population correlation coefficient (unknown)
• r = sample correlation coefficient (known; calculated from sample data)
If the test concludes that the correlation coefficient is significantly different from 0, we say that the correlation coefficient is "significant".
Significance is indicated by the value:
α = 1-ρ
α = 5 indicates a 95% correlation and is considered “significant”.
Factor Analysis
The analyst must seek the causes if the model does not adequately match the samples. Factor Analysis is one tool for this purpose.
Statistical Control
The process of statistical quality control37 is one of determining whether a process and its results are “under control” or “out of control”.
A process that is operating with only chance causes of variation present is said to be in statistical control.
A process that is operating in the presence of assignable causes is said to be an out-of-control process. A process is considered to be out of control when its results exceed the Upper Specification Limit (USL) or Lower Specification Limit (LSL):
Control Charts
The USL/LSL correspond to the Upper Control Limit (UCL) or Lower Control Limit (LCL) in a control chart. These limits typically are taken to be three standard deviations (3σ) above and below the process mean:

The latter chart is an example of the R program output.
Hypothesis Testing8
A hypothesis test examines two opposing hypotheses about a population: the null hypothesis and the alternative hypothesis. The null hypothesis is the statement being tested. Usually the null hypothesis is a statement of "no effect" or "no difference". The alternative hypothesis is the statement you want to be able to conclude is true.
Based on the sample data, the test determines whether to reject the null hypothesis (to decide that the second hypothesis is correct). You use a “p-value”, to make the determination. If the p-value is less than or equal to the level of significance α then you can reject the null hypothesis.
p-value
The p-value is defined as the probability of obtaining a result equal to or "more extreme" than what was actually observed, when the null hypothesis is true. In layman's terms, it is the probability of being wrong by rejecting the null hypothesis. So we reject the null hypothesis when the p-value is sufficiently small, that is, less than the significance level α, which is a cut-off point that you define.
We cannot know the exact p-value, but there are a number of different tests for estimating the p-value depending on the known characteristics of the sample sets at hand:9

t-test10

This delivers a random variable t that approximates the p-value. A t-test is used for testing the mean of one population against a standard or comparing the means of two populations if you do not know the populations’ standard deviation and when you have a limited sample (n < 30).
R returns the following paired t-test result:
data: y and V2
t = -4.2636, df = 5, p-value = 0.007987
indicating a close correlation between y and V2

z-test

A z-test is used for testing the mean of a population versus a standard, or comparing the means of two populations, with large (n ≥ 30) samples whether you know the population standard deviation or not.
Other tests
We may wish to compare other statistics (characteristics) of different sample sets.

F-test

An F-test is used to decide if 2 populations’ variances are the same, assuming both populations are normally distributed. The samples can be any size.

Levene's test

Levene's test is used to decide if 2 populations’ variances are the same, assuming both populations are continuous but NOT normally distributed.

Anderson–Darling test11

The Anderson–Darling test is a statistical test of whether a given sample of data is drawn from a given probability distribution. 

Confidence interval

A confidence interval is a range of likely values for a population parameter (such as the mean μ) that is based on sample data.
Use a confidence interval to make inferences about one or more populations from sample data, or to quantify the precision of your estimate of a population parameter, such as μ.

Test and CI for Two Variances

This calculates the ratio of the variances (Σ) of two sample sets.
This summarizes the various tests:12

1©Privus Technologies LLC, P.O. Box 149, Newport, RI 02840, 2017
21 Richard A. Johnson and Dean W. Wichern, Applied Multivariate Statistical Analysis, 6th Ed., ISBN 0-13-187715-1
5Douglas C Montgomery, Introduction to Statistical Quality Control, 6th Edition, ISBN 978-0-470-16992-6
6Barbara Illowsky. “Testing the Significance of the Correlation Coefficient.” Collaborative Statistics Boundless, 26 May. 2016. Retrieved from https://www.boundless.com/users/235422/textbooks/collaborative-statistics/linear-regression-and-correlation-13/testing-the-significance-of-the-correlation-coefficient-181/testing-the-significance-of-the-correlation-coefficient-424-15972/
7Montgomery, Section 5.2

Thursday, January 12, 2017

Wireshark question: How to get Wireshark to see usbmon0?

I seem to have a habit of embarking on projects and getting to a point at which neither I nor anyone else seems able to find an answer.
Typically, I then go find a forum to post a question. This usually works. But this time it has not.
So, it occurred to me, since all my hours of Googling have failed me, perhaps I should try having Google do the work by posting the question to a blog. That way, anyone halfway interested in the components of my question will be directed here.It may be gratifying to them to know that someone else has the same problem. Perhaps then we can all contribute and we all win.
So here goes. I'll let you know how it works out... If you have thoughts please leave comments by clicking Comments at the bottom of the post. Thanks in advance.
=======================================================
How to get Wireshark to see usbmon0?
 ls -l /dev/usbmon shows 
crw-r--r-- 1 root root 248, 0 Jan 10 14:50 /dev/usbmon0 
crw-r--r-- 1 root root 248, 1 Jan 10 14:50 /dev/usbmon1 
crw-r--r-- 1 root root 248, 2 Jan 10 14:50 /dev/usbmon2
but Wireshark only sees the latter two.
1. We have a piece of boat gear (RayMarine C120W) that bridges NMEA 0183 (ASCII) and EtherNet ("SeaTalk-HS") data for transmission to Windows software (RayTech Navigation System—RNS). The bridged data are wired to a DB-9F chassis connector near the laptop. We did have a Serial to Ethernet cable that connected to an older laptop running the software that had an Ethernet Socket. It worked fine.
2. We have not touched the boat wiring, but have lost the cable and necessarily moved the software to a new laptop (openSUSE Leap 42.1 Linux) that does not have an Ethernet socket, only USB.
3. We have a Gigaware 2603487 USB-A to Serial Cable. It is recognized by the laptop and connected to ttyUSB0. We can read that port at the command line interface—CLI—with cat /dev/ttyUSB0 and see the NMEA 0183 ASCII sentences but not the Ethernet stream. 
3.1 I understand that the EtherNet traffic is higher frequency and multiplexed, yada yada, so will address that aspect ("EtherNet over USB") in due course, but first we need Wireshark to see the basic USB data that we can see on the CLI (presumably on usbmon0) to ensure that Wireshark is reading the USB connection.
4. We have laboriously followed https://wiki.wireshark.org/CaptureSetup/USB and many of its adherents, particularly http://stackoverflow.com/questions/31054437/how-to-install-wireshak-on-linux-and-capture-usb-traffic — yes, they misspelled Wireshark. As a result we have:
4.1 Sorted out usbmon. It needs to be restarted after each reboot (modprobe usbmon), a PITA we'll address later.
4.2 Added the requisite capabilities to dumpcap
4.3 Changed permissions as directed (644) on /dev/usbmon*, added the wireshark group and added the user to the group.
4.4. Configured Wireshark for non-root use, but that shows the same results as running it as root (yes, I know, a no-no).
5. https://wiki.wireshark.org/CaptureSetup/USB says the special "usbmon0" interface receives events from all USB buses.
5.1 After a new modprobe usbmon after a reboot ls -l /dev/usbmon* returns
crw-r--r-- 1 root root 248, 0 Jan 10 14:50 /dev/usbmon0 
crw-r--r-- 1 root root 248, 1 Jan 10 14:50 /dev/usbmon1 
crw-r--r-- 1 root root 248, 2 Jan 10 14:50 /dev/usbmon2
so others (user, wireshark group) should be able to read.
5.2 So indeed usbmon0 exists but it does not appear in Wireshark. Wireshark only shows usbmon1 and usbmon2.  Neither has any interesting traffic, certainly not the ASCII stream that we can see on the CLI.
6. We have attempted using a USB connected EtherNet to USB adapter with a Serial to Ethernet cable. It is recognized by the OS and Wireshark sees it as eth0 but there is zero traffic on it.
We can proceed further with EtherNet over USB once we have determined that Wireshark can read usbmon0 (ttyUSB0).
How to get Wireshark to see usbmon0?