Change Hive metastore from derby to MySQL

Change Hive metastore from derby to MySQL

 

Machine : UBUNTU-14.04 | Hive : HIve 1.2.1

To change Hive Metastore from Derby to MySQL we need to follow these 8 simple steps,

 

Step-1 :

First we need to install Mysql and its dependencies on system.

Command-1 : sudo apt-get install mysql-server

Note: Click Next > Next and set the password for MySQL.

Command-2 : sudo apt-get install libmysql-java

 

Step-2 :

Create soft-link for connector in Hive lib directory.

Command-1 : ln -s /usr/share/java/mysql-connector-java.jar $HIVE_HOME/lib/mysql-connector-java.jar

 

Step-3 :

Access your MySQL and create one new database metastore for hive,

Command : $ mysql -u root -p

Enter password:

mysql> CREATE DATABASE metastore;

 

Step-4 :

Then we need one MySQL account for Hive to access the metastore. It is very important to prevent this user account from any type of change in schema.

Command :

mysql> CREATE USER ‘hive’@’%’ IDENTIFIED BY ‘password’;

mysql> GRANT all on *.* to ‘hive’@localhost identified by ‘password’;

mysql> flush privileges;

 

Step-5 :

Now we need to configure Hive to access MySQL metastore, for this we need to update hive-site.xml file(If file does not exist then create a new one)

<configuration>

<property>

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:mysql://192.168.8.99/metastore?createDatabaseIfNotExist=true</value>

<description>metadata is stored in a MySQL server</description>

</property>

<property>

<name>javax.jdo.option.ConnectionDriverName</name>

<value>com.mysql.jdbc.Driver</value>

<description>MySQL JDBC driver class</description>

</property>

<property>

<name>javax.jdo.option.ConnectionUserName</name>

<value>hive</value>

<description>user name for connecting to mysql server</description>

</property>

<property>

<name>javax.jdo.option.ConnectionPassword</name>

<value>password</value>

<description>password for connecting to mysql server</description>

</property>

</configuration>

 

Note: While updating please take all properties-tag only.

 

Step-6 :

Now we need to run the Hive schematool to initialization MySQL metastore.

For this we need to go to $HIVE_HOME>bin> folder

Command-1 : schematool -initSchema -dbType mysql

Note : When you have found that your metastore is corrupted, then we need to update metastore.

  • Before you run hive for the first time, run

Command : schematool -initSchema -dbType mysql

  • If you already ran hive and then tried to initSchema and if it’s failing:

Command : mv metastore_db metastore_db.tmp

You find your metasore_db file at $HIVE_HOME location.

  • Re run

 

Step-7 :

Start your Hive and access your tables.

 

Step-8 :

To validate it ,

Connect and open your hive

hive>

Then create a table in it and insert one record.

hive> create table saurzcode(id int, name string);

hive> insert into saurzcode(1, “Helical”);

Later access your MySQL and open metastore database

mysql -u root -p

Enter password:

mysql> use metastore;

And see your table as a record in TBLS table of metastore database.

mysql> show tables ;

mysql> select * from TBLS;

 

Hurray, Completed !!!!

Thanks for visiting , Have a great day.

PUSHPRAJ KUMAR

WATCHDOG

WATCHDOG

INTRODUCTION

For those embedded systems that can’t be constantly watched by a human, watchdog timers may be the solution.

The watchdog timer is an important device in the embedded system , which is used to develop reliable products. Most of the embedded systems need to be self-reliant in order to restart and restore the system if any software bug disturbs the system. It is not always possible for human operators to wait for rebooting the system for every software problem. The watchdog timer is a piece of hardware that provides ultimate solutions for the real-time industries, which used to detect system abnormalities automatically and to reset the processor.

attiny10-blinker_yamaguchi_flash

 

 

Watchdog Timers(WDT)

A watchdog timer is a piece of hardware that can be used to automatically detect software anomalies and reset the processor if any occur. Generally speaking, a watchdog timer is based on a counter that counts down from some initial value to zero. The embedded software selects the counter’s initial value and periodically restarts it. If the counter ever reaches zero before the software restarts it, the software is presumed to be malfunctioning and the processor’s reset signal is asserted. The processor (and the embedded software it’s running) will be restarted as if a human operator had cycled the power.

Figure 1 shows a typical arrangement. As shown, the watchdog timer is a chip external to the processor. However, it could also be included within the same chip as the CPU. This is done in many micro-controllers. In either case, the output from the watchdog timer is tied directly to the processor’s reset signal.

image1

kicking the dog

The process of restarting the watchdog timer’s counter is sometimes called “kicking the dog.” The appropriate visual metaphor is that of a man being attacked by a vicious dog. If he keeps kicking the dog, it can’t ever bite him. But he must keep kicking the dog at regular intervals to avoid a bite. Similarly, the software must restart the watchdog timer at a regular rate, or risk being restarted.

Advantage of Watchdog Timers:

  • Resets automatically without human intervention.

  • Detects the errors in the program and reboot the system

  • Cost sensitive

  • Saves the time and money

  • No need to place the employers to monitor the software debugs.

  • Increases the system performance.

SUMMARY

A number of considerations must go into any design that uses a watchdog as a monitor. Once the timeout period is determined, the system software must be analyzed to determine where to locate the watchdog restart instructions. For an effective design, the number of watchdog restarts should be kept to a minimum, and some consideration should be given to the likelihood of incorrectly executing a restart. As mentioned previously, some system software is too convoluted or data-dependent to ensure that all software flow paths are covered by a watchdog restart. This may dictate that a self-diagnostic software approach might be required. If there is an expected failure mechanism such as a periodic EMI burst or power supply glitch, the watchdog timeout should consider this period.

 

 

Pushpraj Kumar 🙂

INNODB Backup and Recovery using mysqldump

InnoDB is a storage engine for MySQL. MySQL 5.5 and later use it by default. It provides the standard ACID-compliant transaction features, along with foreign key support (Declarative Referential Integrity). It is included as standard in most binaries distributed by “MySQL AB”, the exception being some “OEM” versions.

The key to safe database management is making regular backups. Depending on your data volume, number of MySQL servers, and database workload, you can use these techniques, alone or in combination: hot backup with MySQL Enterprise Backup; cold backup by copying files while the MySQL server is shut down; physical backup for fast operation (especially for restore); logical backup with mysqldump for smaller data volumes or to record the structure of schema objects

Issue : —

C:\Calpont\bin>mysqldump.exe  -u root -p  –database  TestDB > C:\Pushpraj\infinidbPushpraj.sql 

Enter password: *******

mysqldump: Got error: 138: IDB-2029: Lock table command is currently not supported in InfiniDB. when using LOCK TABLES

Solution  :-  use –skip-lock-tables

LOGICAL BACKUP

Windows :::

To take the dump from your mysql for infinidb —-

C:\Calpont\bin>mysqldump.exe –skip-lock-tables -u root -p  –database  TestDB > C:\Pushpraj\infinidbPushpraj.sql 

Enter password: *******

It will create your dump file .

To restore the dump in your mysql —-

First create an empty DB in your Mysql desitination DB.

C:\Calpont\bin>mysql.exe -u root -p  TestDB < C:\Pushpraj\infinidbPushpraj.sql 

Enter password: *******

 

🙂   Have fun   🙂

 

PUSHPRAJ KUMAR

Reason for “glibc – detected” and the possible cause.

Most common reason for glibc – detected:-

  • When we are trying to free some memory  which is logically not available for “free ()” .
  • When we are trying to free some memory which is already free.
  • When we are trying to use multiple “free()”  for the same variable.
  • When we are trying to free pointers.
  • When we are trying to free the variable from outside the scope of variable.
  • Using an uninitialized pointer.
  • Writing outside your array bounds.

To avoid or solve glibc detection follows some basic and common rule:-

  1.    Set pointer to NULL after free
  2.    Check for NULL before freeing.
  3.     Initialize pointer to NULL in the start.
  4.    Always allocate storage blocks with calloc or some other means of initializing every block to known all-zeros.
  5.    Write test-scripts to independently exercise all major pieces of the program in isolation to the others. Run these tests all the time to catch regressions.
  6.    Use debugging-tools that add “eyecatchers” around storage-blocks to detect “off-by-one” overwrites and other similar problems. Consider leaving them in, in the production(!) code.

Example:-

int main()

{

char* s = (char*) malloc(sizeof(char)*10);

s=”hello”;

free(s);

}

Here we get  “** glibc detected *** free(): invalid pointer: 0x0000000000400b2c”

Solution-        s=”hello”;

Here we are assigning another address to s, to a statically allocated memory. Freeing it is not correct. Also, since we are doing this, we are basically leaking the memory you have allocated here:

.                     char* s =(char*) malloc(sizeof(char)*10);

#include <stdio.h>

#include <string.h>

#include<stdlib.h>

int main(){

static const size_t kBufferSize = 10;

char* s = (char*)malloc(sizeof(char)*kBufferSize);

strncpy(s,”hello”, kBufferSize); // better than strcpy, you are protecting

// yourself from a buffer overflow

free(s);

}

 

IMPORTANT TIMESTAMP COMMAND

Timestamp

timestamp is a sequence of characters or encoded information identifying when a certain event occurred, usually giving date and time of day, sometimes accurate to a small fraction of a second.

A timestamp is the current time of an event that is recorded by a computer. Through mechanisms such as the Network Time Protocol ( NTP ), a computer maintains accurate current time, calibrated to minute fractions of a second. Such precision makes it possible for networked computers and applications to communicate effectively.

How to Get timestamp in UNIX

Command         : –       date +%sResult         : –        1405407447

How to convert timestamp into Date – time in UNIX

Command         : –    date -d @1405407447 +’%Y-%m-%d %H:%M:%S’

Result         : –                2014-07-15 01:57:27

How to set system Date Time in UNIX

Command       : –     date -s “11/20/2003 12:48:00”

Set the system date and time to November 20, 2003, 12:48 PM

How to get  GMT time Information  in UNIX

GMT time according to selected country..

Command  : –    TZ=’America/Los_Angeles’ date

Result         : –        Tue Jul 15 00:28:10 PDT 2014

Command  : –    TZ=’India’ date

Result         : –      Tue Jul 15 07:32:40 India 2014

How to convert Date into Unix timestamp

To convert Date time in to specific Timestamp

Method: 1

Command       : –    date -d “2010-10-02 17:56:56” “+%s”

Result               : –        1286060216

Method: 2

Command       : –   date –date=’06/12/2012 07:21:22′ +”%s”

Result               : –        1339503682

How to add and subtract Date

To add some days in specific Date

Command       : –    date -d ‘Jan 1 1982 + 11 days’

Result               : –     Tue Jan 12 00:00:00 EST 1982   

To add some Days and get the output in timestamp

Command       : –    date -d ‘Jan 1 1982 + 11 days’ +%s

Result              : –          379659600

To subtract some days in specific Date

Command       : –    date -d ‘Jan 1 1982 – 1 days’  

Result              : –    Thu Dec 31 00:00:00 EST 1981