Exploring log files from Linux terminal cheatsheet

less

less and zless are available in all versions of Linux and almost all of Unix. We can use them to read the log files.

Browse the file

CTRL + F – move forward one window
CTRL + B – go back one window
CTRL + D – advance half window
CTRL + U – go back half window
j – advance one line, 10j advance 10 lines
k – go back one line, 20k go back 20 lines
G – end of file
10G – go to line 10 from the end
g – start of file
10g – go to line 10
CTRL + G – show file name, line, byte and percentage of progress
q or ZZ – exit
h – help 🙂

Search and filter

Forward
• / – search for a pattern
• n – navigate to the next occurrence
• N – navigate to the previous occurrence
Backward
• ? – look for a pattern
• n – navigate back to next occurrence
• N – navigate to next occurrence forward

Show only lines that meet a pattern:

&Pattern

To redisplay all lines:

& <Enter>

Bookmarks

When the file is large and we want to mark a position to which we want to return we can use marks.

The marks are identified with a letter. We create them with the m command and navigate to them with the single quote:

ma
'a

Simulate tail -f

When we read running log files they will continue to increase. We can indicate unless it shows us the last lines with F. If we want to stop monitoring the new lines, press Ctrl + C.

Show line numbers

We can open a file showing the line numbers by adding the -N option. Once the file is open we can add the line numbers by typing -N and pressing enter.

grep

The grep command allows us to filter text files in this way we can filter log files to keep the lines that may be relevant to us.

Filter lines that do not meet a pattern:

grep -v [\ STDOUT \] logfile.log> logfile_filtered.log

In this case we have created a file with the lines that do not have the text [STDOUT].

head and tail

They allow us to keep the first or last rows of a file.
For example we can keep them with the last lines of the file from a row number. This is valid if we are not interested in the first lines of the file:

tail -n +587040 logfile.log> log_from1500.log

First lines of a file. In the following file we have identified in which line the 16 hours start and we have been left with the one hour logs.

head -n 284523 log_from1500.log> log_from1500to1600.log

References

Unix Less Command: 10 Tips for Effective Navigation

Create a Timelapse with a Rapsberry Pi

Enable the camera

sudo raspi-config

Go to the camera option and select enable, finish and reboot.

Image capture

The software to capture images and videos from the Rapsberry camera is already installed in the Raspbian distribution. To capture still images we will use  raspistill command.

It takes too long to take a single photo. In my case it takes around 5 seconds. Thus do not use crontab if you want images in a shorter interval as it will fail for too many frames.

Raspistill provides two convenient options to make timelapses: “t” and “tl” options .

Still if the interval is too short it will skip frames. A message like this will be shown when skipping frames:

mmal: Skipping frame 584 to restart at frame 585

This is a problem because the tools to make the video from the images expect consecutive numbering and they will stop in the first skip.

In the following link we can find a script that can be used to avoid number jumps:
http://brruchstuecke.blogspot.com.es/2014/01/raspberry-pi-camera.html

raspistill -md 5 -w 1280 -h 720 -vf -hf -o test%06d.jpg -tl 1000 -t 7200000

If we set the camera upside down we can flip the image in both directions with “vf” and “hf” options.

We limit the size of the picture to the really needed image size as it does not look sensible to make the poor raspberry to work with insanely large images.

Video encoding

Some articles suggest to use mencoder, ffmpeg or avconv to encode the video. Some of these tools may not be available in the last releases of Raspbian Jessie or may be incomplete and you will spend quite a time trying to make them work to realize that they are too slow to work in a Raspberry.

Raspberry Pi GPU has support for hardware accelerated h264 encoding using OpenMAX. The only available software that seems to be able to use it is gstream.

Make sure that the GPU has enough memory asigned, otherwise it will fail with a cryptic error. (TODO try to find the error message)

Latest version of Raspbian Jessie seems to already provide the needed debian packages. Just install the “gstreamer1.0-tools” package.

For older versions of Debian there are repositories that provide the package.

The simplest pipeline that I have found that creates a timelapse video is the following one:

gst-launch-1.0 multifilesrc location=test%06d.jpg index=1 caps="image/jpeg,framerate=24/1" ! jpegdec ! omxh264enc ! avimux ! filesink location=timelapse.avi

Troubleshooting

To debug any problem with gstreamer you can launch it with the “v” option and run the following command before gstreamer.

export GST_DEBUG="*:3"

 

 

References:

http://www.raspberryconnect.com/hardware-add-ons/item/137-raspberry-pi-camera-module

Create and configure Let’s Encrypt certificate in Apache 2.2

The Let’s Encrypt documentation states that it’s client supports the autoconfiguration of apache/2.x (working on Debian 8+ and Ubuntu 12.04+). This is currently not entirely true as support for Apache 2.2.x is being worked as part of an ongoing issue. If you do not want to wait for a version of the client that supports Apache 2.2 you can still use it to validate the domain and create the certificate and then manually configure Apache.

To validate and create our certificate while our apache server is running we will use the webroot option of the client:

./letsencrypt-auto certonly -a webroot --webroot-path /var/www/example \
  -d example.com -d www.example.com

In the previous command we need to replace the domains we want the certificates for and the folder where the pages for the domain are served from.

The previous command will create some files in the /etc/letsencrypt/live/<domain> folder.

Now we need to tell to Apache to use these files for our domain. We will do it in the VirtualHost configuration file for our site. In Debian like Linux distributions it will be in the file /etc/apache2/sites-available/<domain>.

There, in the VirtualHost for port 443 we need to add the following statements:

SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/<domain>/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/<domain>/privkey.pem 
SSLCertificateChainFile /etc/letsencrypt/live/<domain>/fullchain.pem

You can get a full example of the file for Apache an for NGINX here.

Now you should be able to reload the configuration of apache and your server will use the new certificate. In a Debian like distribution you can use:

sudo service apache2 reload

Letsencrypt options for dummies

Recently Let’s Encrypt started to offer it’s services in public beta. The project is great because it offers free and easy to set up encryption for websites. Getting free or very cheap certificates was already possible but still it was needed to follow a cumbersome process to obtain and install them. Let’s Encrypt promises to change this and provides a tool to  perform the process automatically.

The projects Quick Start Guide offers a good introduction to the options but the terminology used in it did not make complete sense to me in the start.

I have rephrased some of the options in a way that makes more sense to me. I hope it also helps other muggles like me.

./letsencrypt-auto –apache

This is the option you should use if it is supported for your web server and OS combination. The documentation states that Debian like OSs with Apache 2.X are supported. Still, currently, the support is only for version 2.4 onwards.

This option validates that the server really serves the requested domain, creates the certificate and install in the Apache server.

./letsencrypt-auto certonly -a standalone

This option only validates that the server really serves the requested domain and creates the certificate but does not configure the web server. To provide the validation it starts a web server in the 80 and 443 ports. This means that you need to stop your current server which makes it inconvenient if you are already serving a site.

The generated files will be available in the /etc/letsencrypt/live/<domain> folder and it is up to you to configure the server to use them.

./letsencrypt-auto certonly -a webroot –webroot-path /var/www/example

This option is quite similar to the previous one but it uses your current running web server to perform the domain validation. We need to provide the web server root. That means the folder where our served documents or pages are in the server. I suppose that it creates some kind of file there to validate that our server is really serving the requested domain. We can use this option if we have an already running web server and the first option does not work for us. I have used it to configure an Apache 2.2 server.

As in previous option it is up to you to configure the web server to use the certificate.

./letsencrypt-auto certonly -a manual

This guides you through all the needed steps but it does not perform any of them. I have not personally used this option but this interesting guide on how use Let’s Encrypt in a shared Dreamhost account uses it.

 

Java Enums vs Constants

Very often I meet projects with large constant utility classes. In fact I use this class and function utility class as a measure of developers misunderstanding of object oriented programming. And the results of this measure is almost always discouraging.

Most of the time these constants can be best coded as Java Enums.

As I need to explain the usage of enums and it’s advantages too frequently I will try to summarize them in this entry.

If there is a copy of Effective Java at hand Item 30 to 34 are a good read.

Enums are Type Safe and enforce a limited set of values

If we use constants the following code is possible:

public static final String MALE = "M";
public static final String FEMALE = "F";

private String g = "G Force?"; // What is g?
private String gender = "Male";
// How should I write the gender,
//I'm in hurry, do I need to read the Javadoc? Is it even written?

When using an IDE it will show the developer that a String is needed, but it won’t be any hint on the accepted values.

We can write the same using Enums:

public enum Gender {
    MALE, FEMALE
}

private Gender g = "G Force?"; // Compile error!!
private Gender gender = "Male"; // Compile error!!

private Gender maleInstance = Gender.MALE;
private Gender femaleInstance = Gender.FEMALE;

Now we do not open the door to erroneous values and when using an IDE it offers us a list of the acceptable values making the developer more productive and the code less error prone.

You can also use enums in switch statements

switch (gender) {
    case MALE:
        // do sthg
        break;
    case FEMALE:
        // do other thg
        break;
}

Enums can list all possible options

Often we need to offer a list of all the possible options, by example, in a user interface. Enum includes the values() method.

With constants:

public static final String MALE = "M";
public static final String FEMALE = "F";

// We need to remember to change this method when we add another option
public String[] getOptions() {
    return new String[]{MALE , FEMALE};
}

This code seems final and we do not expect to change it.

Really?

http://www.theguardian.com/world/2014/apr/02/third-gender-must-be-recognised-by-nsw-after-norrie-wins-legal-battle

Ouch! Now it is not enough to add a new constant, We also need to change the getOptions method.

With enums:

public enum Gender {
    MALE, FEMALE
}
// We DO NOT need to remember to change this method when we add another option
public Gender[] getOptions() {
    return Gender.values();
}

Just adding a third option does it.

Compare values

Comparing constant options forces to use the equals method:

public static final String MALE = "M";
public static final String FEMALE = "F";

String gender = myComplexServiceBackedByMegaComplexObjecRelationalMapper.getGender();

if (gender == FEMALE) {
    // Probably my service is not referencing the constant object
}

if (FEMALE.equals(gender)) {
    // I am forced to use equals
}

As Java ensures that there is only one instance of the enum the equality operator works right:

public enum Gender {
    MALE, FEMALE
}

Gender gender = myComplexServiceBackedByMegaComplexObjecRelationalMapper.getGender();

if (gender == Gender.FEMALE) {
 // Single instance means we are comparing the references
}

if (Gender.FEMALE.equal(gender)) {
 // Of course this also works
}

But I get the codes from an external source.

It is common to load the data from an external source (file, database, …) that already uses a code for the options. We can code the enum this way:

public enum Gender {

    MALE("M"), FEMALE("F");
    private final String code;

    private Gender(String code) {
        this.code = code;
    }

    public String code() {
        return code;
    }

    public static Gender fromCode(String code) {
        if (code != null) {
            for (Gender g : Gender.values()) {
                if (code.equalsIgnoreCase(g.code)) {
                    return g;
                }
            }
        }
        return null;
    }
}

The create our variable this way:

Gender gender = Gender.fromCode("M");

But constants are more efficient

An int is a primitive and it seems that using a primitive is faster than using an object. Still Java assures that there is only ever one instance of the enum, making its usage very fast.

References:

• http://stackoverflow.com/questions/11575376/why-use-enums-instead-of-constants
• http://stackoverflow.com/questions/1858829/static-string-constants-vs-enum-in-java-5
• http://stackoverflow.com/questions/9969690/whats-the-advantage-of-a-java-enum-versus-a-class-with-public-static-final-fiel
• http://stackoverflow.com/questions/9254637/java-enum-vs-int

Reading More Like This results from Solr with Solrj

It is possible to configure Solr to offer documents related to a document based on the information stored in the documents.

I wanted to perform a search and offer related documents for each of the result documents. Unfortunately there is no method in QueryResponse to retrieve that information. I searched for solutions on Internet and I was proposed to use a SearchHandler to retrieve the MLT information which forced me to make a request for each retrieved document.

Anyway the information I needed was there in the XML returned from Solr so I thought that it was possible to retrieve it. I managed to get it using the Map returned by getResponse().

        SolrServer server = new HttpSolrServer("/server/url");
        SolrQuery query = new SolrQuery();
        // add params to the query
        try {
            QueryResponse rsp = server.query(query);
            SolrDocumentList results = rsp.getResults();
            SimpleOrderedMap mltResp = (SimpleOrderedMap) rsp.getResponse().get("moreLikeThis");
            for (SolrDocument doc : results) {
                SolrDocumentList mltDocumentList = (SolrDocumentList) mltResp.get((String) doc.get("id"));
                //do sth with mltDocumentList
            }
        } catch (Exception ex) {
            // log
        }

Installing TOra with Oracle Support on Ubuntu 12.10 (Quantal Quetzal)

Get instant client packages from Oracle download page.

• oracle-instantclient11.2-basic-11.2.0.3.0-1.x86_64.rpm
• oracle-instantclient11.2-devel-11.2.0.3.0-1.x86_64.rpm
• oracle-instantclient11.2-sqlplus-11.2.0.3.0-1.x86_64.rpm

If you do not have alien installed install it:

sudo apt-get install alien

Install the Oracle packages using alien:

sudo alien -i oracle-instantclient11.2-basic-11.2.0.3.0-1.x86_64.rpm
sudo alien -i oracle-instantclient11.2-sqlplus-11.2.0.3.0-1.x86_64.rpm
sudo alien -i oracle-instantclient11.2-devel-11.2.0.3.0-1.x86_64.rpm

Download the sources and the dependencies to compile the ubuntu package:

apt-get source tora
sudo apt-get build-dep tora

Prepare the environment:

export ORACLE_HOME="/usr/lib/oracle/11.2/client64"
export LD_LIBRARY_PATH="${ORACLE_HOME}/lib"
export TNS_ADMIN="${ORACLE_HOME}"
export ORACLE_PATH_INCLUDES=/usr/include/oracle/11.2/client64/

Modify the FindOracle.cmake file:

vi tora-2.1.3/cmake/modules/FindOracle.cmake

to add the following line:

SET(ORACLE_PATH_INCLUDES $ENV{ORACLE_PATH_INCLUDES})

Now you can build and install the package:

cd tora-2.1.3/
fakeroot debian/rules binary
sudo dpkg -i ../tora_2.1.3-2_amd64.deb

References:
Installing TOra with Oracle Support on Ubuntu 10.04 (Lucid Lynx)
HowToBuildToraWithOracle

Remember to clear the cache to reload the feeds plugins

I firstly installed the feeds module and oh! I also needed to install the data module. Ok both installed, lets configure feed importers. An error appears:

Missing Feeds plugin FeedsDataProcessor. See feed_fast. Check whether all required libraries and modules are installed properly.

I wasn’t able to find any reference on Google about the error, nor on data or feeds modules issues. The plug-in was there, on feed module and the file feeds.plugins.inc seemed to to be there initialize them all. A condition to check the presence of the data module was in place and it should evaluate to true now. And the my bulb lighted: a Drupal rule of thumb. I installed the data module after the feeds module, so the evaluation on feeds.plugin.inc should have been cached, bingo! Whenever you see strange behaviours on Drupal clear the cache!!