I had to connect to a MySQL instance to do some very quick reporting. But after installing the latest MySQL drivers, Tableau threw an error with this:
connection using old (pre-4.1.1) authentication protocol refused
The error mentioned 3.51.0 as some version number. Lo and behold that’s a driver version when you go to download MySQL drivers. I downloaded and installed 3.51.30 drivers and the connection was happy.
Drivers are here (Tableau Drivers page links here):
How do you enable slow query logging in MySQL without having to restart?
set @@global.slow_query_log = 1;
I prefer to log these things to the mysql.slow_log table rather than to a file. This way you can view the slow queries with this query: "select * from mysql.slow_log":
set @@global.log_output = "TABLE";
Set the threshold above which a query is considered a “slow query” with this (in seconds):
set @@global.long_query_time = .33;
You can view your current settings for all these configs with these queries:
select @@global.slow_query_log;
select @@global.log_output:
select @@global.long_query_time;
You can make these setting permanent so they are set when mysql starts by updating the my.cnf file with these:
--slow_query_log = 1
--log-output = "TABLE"
--long_query_time = .3
Clear out the slow query data logged to that mysql.slow_log table:truncate table mysql.slow_log (or any sql delete statement I think)
A note on select @@long_query_time vs. @@global.long_query_time:
@@long_query_time shows your current session settings, while @@global.long_query_time shows global settings for all sessions.
Just use @@global.long_query_time, @@global.slow_query_log, @@global.log_output.
set @@global.log_slow_queries is what was used in older versions of mysql. Don’t use it now though, if only for “staying with the times”.
In this case I’ll do it with the Hadoop filecrush project here: http://ift.tt/1gzGOis
There are no JAR files to download.
What do you do?
You build it 
How?
Look for pom.xml. That means it can all be built by maven. Here’s how to do that:
1. install Java Development Kit (Open-JDK in this case)
– yum install java-1.7.0-openjdk-devel (the “-devel” includes the actual JDK… the version without “-devel” doesn’t have the JDK… despite the name)
– I found that by first going “yum list *jdk* and looking through what it displays
– set JAVA_HOME environment variable
– export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64/jre
2. install wget
– yum install wget
2. install maven
– Maven is not in the yum repository, so we’re going to just follow the directions from the Maven site: http://ift.tt/1j5lN1M
– wget http://ift.tt/12xh1qc
– tar xzf apache-maven-3.2.3-bin.tar.gz -C /usr/local/
– cd /usr/local
– export M2_HOME=/usr/local/apache-maven-3.2.3/
– export PATH=${M2_HOME}/bin:${PATH}
– “mvn -version” to make sure it’s installed
– you should see no error messages
3. Download the filecrush project:
– wget http://ift.tt/1wKXqR5
4. Install unzip
– yum install unzip
5. decompress project
– unzip master.zip
6. Run Maven to build the project
– cd filecrush-master
– mvn package
– it’ll download a bunch of stuff now
– “mvn package” automatically looks for a “pom.xml” file, since that file contains all the build instructions.
– “mvn package” produces a JAR file, whereas “mvn install” would install the files somewhere on the system.
– hmm… seems to throw exceptions… looking for error about “Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project filecrush: There are test failures.”
7. Found something: http://ift.tt/1uM35ml
– this is someone reporting what appears to be this very error
– The filecrush author says it looks to be a non-deterministic problem that we can safely ignore with the following build command:
– mvn -Dmaven.test.skip=true package
– there we go. Seems to build fine
8. Get built JAR
– the file at target/filecrush-2.2.2-SNAPSHOT.jar
Say you’ve done some fairly complex query on a 6 billion row table. It took 5 hours but it completed. Yay! However, now you have a 200GB tempdb and your drives are all full. Since this was a one-time deal, you want to shrink that tempdb back down to something much smaller.
If you try doing the typical database shrinking stuff but it doesn’t actually finish – in SSMS > System Databases > right click on tempdb > shrink > files > then find the big file and try to shrink it. Check the specified initial size in the database files properties – right-click tempdb > properties > Files > look for initial size column. One file may have a really big initial size specified. Just make that number smaller.
LZO makes Impala faster because files compressed with LZO can be split up and assigned to different nodes. If you use GZIP compression and write a single large file to HDFS, it gets split up across many HDFS blocks. Since Hadoop and Impala are all about parallel processing, this makes it slower because only a single node can process any given compressed file… unless it’s been compressed with LZO.
Anyways, the docs were *kinda* clear on what to do, but they were still confusing about whether or not you have to muck around on the command line. Well, you don’t. You can just do it all via Cloudera Manager and the Parcels systems (assuming you chose to install using Parcels when you initially set up the cluster).
In short, the LZO compression features are contained in the GPL Extras parcel. Here’s the page on how to install it:
It’s not too clear, especially since they don’t list anything for the 5.2 release, which is what I have. I just used http://ift.tt/1vT806E without appending any version number to the end. So, later on when there’s 5.3 or something, you may need to put in http://ift.tt/1zXhxJ6 or something like that.
Specifically, I did the following to set it up and get the parcel installed:
1. Cloudera Manager > at the top-middle-rightish, click on the parcel icon (looks like a gift box) > Edit Settings at the top right > In “Remote Parcel Repository URLs” add a new entry and paste in “http://ift.tt/1vT806K;
2. Save changes
3. Restart Cluster
4. I had to redeploy client configurations after the cluster restarted (there were icons mentioning as much in the “Home” page of Cloudera manager).
Now, you can go to each of your nodes and run “yum install lzop -y”. Once that’s done, LZO should be magically available for things like filecrush.
Figured I’d mention a few things I discovered that for some reason didn’t stand out to me in the docs.
If you have a Hive table that you’ve been dumping new data into, and you want to have your analysis tools pointed at an Impala/Parquet version of the data in that table, you can’t simply do a “create table mytable_parquet_partitioned like mytable partitioned by (year int) stored as parquet”.
You have to do it this way:
if mytable is structured like this:
mytable
(
year int,
day int,
transactiontime timestamp,
product string,
user int,
ip string
)
and you want to partition on year for your mytable_parquet_partitioned, then here is what your “create table” statement and your insert needs to look like:
create table mytable_parquet_partitioned
(
day int,
transactiontime timestamp,
product string,
user int,
ip string
)
partitioned by (year int)
stored by parquet;
Notice how you don't include the "year" column in the column list of the "create table" statement. Rather, you have to put it *only* in the "partitioned by" clause. Where does that "year" column end up being in the table structure?
insert into mytable_parquet_partitioned
partition (year)
select
day int,
transactiontime timestamp,
product string,
user int,
ip string
from mytable
Notice how you do not select the year column from the source table. Rather, you specify it in the partition clause.
The resulting table will be structured like this:
mytable_parquet_partitioned
(
day int,
transactiontime timestamp,
product string,
user int,
ip string,
year int
)
Notice how the partition column is at the end of the list of columns?
You can do it, e.g. configure a DataNode to store data on multiple directories on the same drive.
Something that happens is that your total cluster storage estimate double-counts the available space on that volume. Each DFS directory you have configured assumes its the only one on the volume, so the available space is summed.
You really should run Splunk as a non-root user. Here’s how to do it:
1. create a new user (if it doesn’t already exist)
– useradd splunk
– passwd splunk
2. Stop splunk
– /opt/splunk/bin/splunk stop
3. give ownership of all splunk files to the “splunk” user
– chown -R splunk:splunk /opt/splunk/
4. set splunk to start up under the “splunk” user at system boot
– /opt/splunk/bin/splunk enable boot-start -user splunk
5. reboot and make sure splunk starts up as expected
– top
If it doesn’t start up, the most likely thing is that for some reason the “splunk” user does not have permissions on some file somewhere.
I saw this sort of error come up when using Impala:
When I execute some aggregation query, a red bar comes up that indicates some failures but doesn’t really tell me much at all. I hit “execute” again on the same query, usually the below sort of error comes up in the Query Log and the query fails.
“impala Backend 3:Memory Limit Exceeded Process: memory limit exceeded. Limit=”
In my case the query memory usage was somewhere just above 8GB and the limit was right around 8GB. Well, this means the configured memory limit for Impala queries was reached on at least one node. Yeah, if just one node dies, then the entire thing dies. In my case, there was just one Impala group that was set to around an 8GB memory limit for Impala.
Here’s how to check and change:
Cloudera Manager > Home > Impala in desired Cluster > Configuration > ”
Impala Daemon Memory Limit” > make those as big as you can > restart cluster.
I think this is really cool. Prior to this upcoming release if you wanted to use parquet files you had to do a separate create and insert statement then drop the “incoming” table.
Hive .14 will support Avro as a full-class storage format. You’ll be able to do “create table mytable () stored as avro”. Different type of format than parquet, yeah, but these are examples of cool stuff coming soon
At the bottom of every action config window there is a field that says “Job XML”. These sorts of things have always scared me. Well, here’s what it means in Hadoop-land.
If you don’t do anything with this field and you set up a Sqoop task, that task can run along just fine, happy as can be… until it needs to do something involving Hive. At that point, it has no idea what or where to do stuff in Hive…. because it doesn’t know anything about where Hive is. And that’s why the hive-site.xml file has to be specified there. You click the dot-dot-dot and upload a file – the hive-site.xml file you get from here:
Cloudera Manager > Cluster > Hive > Actions drop-down on the top right > “Download Client Configuration” > in that zip file will be hive-site.xml. That’s the file you upload. That’s the file that defines where anything Hive-related will be.
You could get fancy and store your hive-site (and all the other *-site.xml files in some HDFS folder you just point to), but that’s fancy and I’m not ready for that 
At Database setup during the cluster setup, you see this error when you test the connections to the DBs:
JDBC driver cannot be found. Unable to find the JDBC database jar on host :
You need to make sure you get the latest mysql connector jar file and put it here (named this way too):
/usr/share/java/mysql-connector-java.jar (the below error messages say it’s explicitly looking for a file by that name)
In /var/log/cloudera-scm-server/cloudera-scm-server.log I saw this:
+ exec /usr/java/jdk1.7.0_67-cloudera/bin/java -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:///dev/urandom -cp ‘/var/run/cloudera-scm-agent/process/6-HIVE-test-db-connection:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*’ com.cloudera.enterprise.dbutil.DbCommandExecutor db.properties
I want to store all log and event data in HBase. I want to generate a Hive schema for each event type. I want to then query with Impala, ideally inserting data into parquet-backed tables. CDH 5.2 is my Cloudera Hadoop install.
No dice. There must be a bug; that or we’re all missing some crucial bit of info.
0% Complete (0 out of 2) Backend 1:DoNotRetryIOException: Failed after retry of OutOfOrderScannerNextException: was there a rpc timeout? CAUSED BY: OutOfOrderScannerNextException:
Here are a few mentions from people wanting to do the same thing and coming up with nada:
This Apache bug *seems* to be the same. The solution seems to be increase the RPC timeout in Impala:
There is a config in the Impala service for RPC timeout to HBase (but it didn’t seem to do anything for me; it just kind of sits there forever at 0% complete):
In Cloudera Manager > “Home” > Impala in Cluster > Configuration > type “rpc” in search > HBase RPC Timeout > change to something larger
Besides this Impala-HBase issue, I saw some seriously lackluster performance with Hive doing the same query. It would get up into the mid-to-high 90% range in the Map stage, but then just sort of stall out and get never really finish (not by the time I called it after some “reasonable amount of time”).
This is discouraging. It looks like I won’t be able to use HBase as the core datastore for events and log data for all products, from which I then dynamically maintain Hive table schemas and and materialize things into parquet-backed tables which I then query with Impala. It’s just not ready. Time to step back and just have Sqoop lay down Avro files and hopefully figure out how to get it to create external Hive table schemas, so nobody accidentally drops one of the tables.
I noticed that my Cloudera cluster install seemed to put all the DFS directories on the root volume of each node at /dfs/dn. There were 2 volumes on each machine and the larger volume was mounted at /home. I can’t recall why or if I had something to do with that. However, needless to say, most of the space on this cluster was not even being used by the cluster in any way. Here’s how I got my Hadoop install to use the extra space:
Note you will have to restart your cluster; or if you have Enterprise, do a rolling restart.
On each node, create a directory at /home/dfs/dn. It’s in /home *only* because I didn’t want to redo all the mount points and change up partitions. I plan on nuking each data node one at a time and installing more disk space anyways, so this will do for now. Anyways…
1. create the directory you want HDFS to use
– mkdir /home/dfs
– mkdir /home/dfs/dn
– chown -R hdfs:hdfs /home dfs
2. Go to your Cloudera Manager web interface and click on “Home” at the top > HDFS service in the relevant Cluster > Instances > “DataNode” (you’ll do this for each node) > Configuration > now click the “+” sign in the DataNode Data Directory config section and type in “/home/dfs/dn”.
3. Go back to “Home” and you’ll see an icon next to the HDFS service that indicates restarts are necessary. Do that.
Once the cluster comes back up, new data writes should start going to the new directories. You should also see the bar in the HDFS Summary area indicate the additional available space.
JSON – consider it an alternative to xml. It’s smaller, faster, and easier to read. Just use it.
Avro – a data storage system that stores JSON along with the schema for the JSON. Think of it as a file that contains loads of objects stored in JSON, and then the schema is stored along with it. In addition,
“When Avro is used in RPC, the client and server exchange schemas in the connection handshake”.
Trevni – a columnar storage format. Instead of writing out all the columns for a row then moving on to the next row, Trevni writes out all the rows for a given column and then moves on to the next row. This means all the column values are stored sequentially, which allows for much faster BI-like reads.
Parquet – Cloudera and Twitter took Trevni and improved it. So, at least in the Cloudera distribution, you’ll see Parquet instead of Trevni. I suspect most BI-type systems will be using Parquet from now on.
Really, JSON and Avro are not directly related to Trevni and Parquet. However, Serializers/Deserilizers (SerDe) come by default with Hive, so it’s good to know.
If you use Avro, it means you can do strong typing while moving around lots of NOSQL data. Sqoop, for instance, now can export directly to Avro files – it generates the JSON schema and everything for you based on the columns in the source tables.
Avro:
Trevni:
Parquet:
Solving the Small Files problem in Hadoop. The Filecrush project on Github by Edward Capriolo seems to be a viable solution. Amazon has release S3DistCp, which would be another solution. For this, I’m covering filecrush.
You need the filecrush JAR, which is not something included in the Github project. There are links to www.jointhegrid.com, but that site has been down for me. Not sure why. Other searches for a JAR of filecrush only yields sketchy results. Sooooo… let’s build it.
Hint, when you see a file named “pom.xml” on a Github or Bitbucket project, it means you can build the thing with Maven… pretty seamlessly.
For Maven you need to install a Java SDK (if openJDK, make sure to get the one with “-devel” at the end of the package name, e.g. yum install java-1.7.*-devel.
For Maven, you need to download the Tar, untar it, move it to the expected location, and export some environment variables.
Download the Github project (the whole zip file). Install unzip and unzip it.
From inside the unzipped filecrush folder (same level as pom.xml) you need to run Maven.
In the case of filecrush, currently (2.2.2) you need to tell Maven to skip the tests, since they seem to be breaking due to some nontrivial reason caused by a recent Hadoop release. It appears it’s not an issue though: http://ift.tt/1uM35ml (from the auther of filecrush)
mvn -Dmaven.test.skip=true package
It will download dependencies and then eventually spit out a jar file in the targets directory. Copy that up to your Hadoop cluster – I put mine in /user/hive/aux_libs/. Then, “Refresh Cluster” from the Home of Cloudera Manager on the relevant cluster.
Run filecrush like this:
From a SSH session on one of the nodes:
hadoop jar filecrush-2.2.2-SNAPSHOT.jar com.m6d.filecrush.crush.Crush –input-format text –output-format text /user/root/ingfolderwithloadsoffiles/ /user/root/outputfolder/ 20101121121212
(I did the –input-format and –output-format because the files were gzipped text files)
See the docs for more usage options: http://ift.tt/1gzGOis
If you add a SerDe via the “Add Jar” command in a Hive query and then create a table that uses that SerDe, note that you will not be able to later drop that table from a different Hive session without first adding the same SerDe via “Add Jar” command.
Moral of the story – make sure you hang on to every custom SerDe any of your users ever uses to create a Hive table.
I’m sure there are other ways to drop a table without the SerDe present, but still… just something to be aware of.
You need 2 SerDe jar files, and you need to configure the Hive Auxiliary Jars path.
1. Pick the directory where you will always put all your globally-accessible additional SerDe jars:
– these will be usable by everyone who uses Hive, so consider that I guess
– I’m going with /var/lib/hive/aux_jars
– mkdir /var/lib/hive/aux_jars
– do this on each node that is running HiveServer2 or HiveServer.
2. From the following 2 projects get the SerDe jars and somehow copy them into the /var/lib/hive/aux_jars folder on all your nodes running HiveServer2 and/or HiveServer:
– http://ift.tt/1xm1hEC
– http://ift.tt/1eVPKm1
– make sure to do a chown -R hive:hive /var/lib/hive/aux_jars
3. In Cloudera Manager, click on the Hive service and go to the configuration tab:
– type “aux” to filter the configs to show the
Hive Auxiliary JARs Directory config. enter /var/lib/hive/aux_jars
– this is my own path, not something official or some magic number
– it’s just telling Hadoop-land which directory on the HiveServer2 nodes to look for additional SerDe jars.
– redeploy and restart – just do whatever Cloudera Manager tells you do to in order to deploy the config changes
4. Now you can use those SerDes as they are documented in step 3 above. If not, double-triple check your path spelling. I’ve not had it *not* work for me for any other reason.
You need to install the 32 and 64 bit ODBC drivers/Connectors for one of or both Hive and Impala from here:
You will need to run the ODBC Administrator program that’s built into Windows to configure the “System DSN” that gets created for each one when you install the drivers/Connectors. Just hit the Windows key and type “odbc admin”. There’s a 32-bit one and a 64-bit one. The 32-bit one can only edit the 32-bit DSNs. The 64-bit one can only edit the 64-bit DSNs. Find them on the “System DSN” tab. And, yes, you need to edit both to point at your Impala and/or Hive server. If you’re not sure which node(s) that is, you can find them in Cloudera Manager:
1. Cluster > Hive > Instances > HiveServer2
– Whichever machine is running that role instance is where you need to point all your DSNs
2. The ports to point at are found here:
– Cluster > Impala > Configuration > Ports > “Impala Daemon HiveServer2 Port” (default is 21050)
– Cluster > Hive > Configuration > Ports > “HiveServer2 Port” (default is 10000)
Some errors when you click on Details:
MainThread agent ERROR Could not determine hostname or ip address; proceeding.
agent.py: error: argument –hostname is required
Make sure your hostnames match:
/etc/sysconfig/network
/etc/hosts
notably, check the hosts file on the machine on which you’re running Cloudera Manager, e.g. if its hosts file has a different hostname for the machine at 192.168.1.222 than what that machine has in either its /etc/hosts or /etc/sysconfig/network files.
If you want to import data into HDFS via Sqoop and have it auto-create the Hive table schema, currently (1.4.5) Sqoop jobs will fail if the the Hive table already exists. Running the same “sqoop import” command directly will successfully complete.
The reason is that when a Sqoop job is created the “PROPNAME” of hive.fail.table.exists is set to true. If you update that via SQLTool in the hive metastore DB so it’s set to false, the jobs will run fine.
I can’t find anything in the docs that indicates how you can specify this behavior. I’ve had to manually run “update” statements directly on the HSQLDB instance…. that or just before creating your sqoop job run a “sqoop import” command with –create-hive-table and specifying “where 1=0″ in the query somehow so the table exists…. then when you create your sqoop job removing the –create-hive-table bit from the import config.
I think we just need to add another configurable parameter that lets you specify if you want the import to fail if the hive table already exists.
Here’s the SQL statement that will make it so it won’t fail if the table already exists:
update sqoop_sessions set propval = false where propname = ‘hive.fail.table.exists';
I submitted my very first Apache Jira bug for this
I hate citing just one URL and asking questions, as doing so seems too mypic. However, I think it will serve to address a question I’ve had about fluoride that I’ve not really found a great answer to.
Question: Would it be cheaper and less risky if we skipped fluoridating water and instead just did twice-annual fluoride treatments?
This review seems to indicate there would be no difference in outcomes: http://ift.tt/1xP65yH
“Initial studies of community water fluoridation demonstrated that reductions in childhood dental caries attributable to fluoridation were approximately 50%–60% (94–97). More recent estimates are lower — 18%–40% (98,99). This decrease in attributable benefit is likely caused by the increasing use of fluoride from other sources, with the widespread use of fluoride toothpaste probably the most important.”
“Clinical trials conducted during 1940–1970 demonstrated that professionally applied fluorides effectively reduce caries experience in children (233). In more recent studies, semiannual treatments reportedly caused an average decrease of 26% in caries experience in the permanent teeth of children residing in nonfluoridated areas”
80% of caries in children happen in 25% of the population. It seems widespread fluoridation is kind of like carpet bombing everything just to make sure we get it all. In a world of web content personalization, it seems we ought to be able to target with at least some level of precision.
Sqoop seems to forget I told it to store the password in the database. It’ll run fine without prompting for the password one or two times. Then out of the blue it seems to start prompting again. This means Oozie-scheduled tasks fail and I have to recreate the job (after copy-pasting the –last-value I get by doing a sqoop job –show job-name).
This is only a problem when you specify a –password. If you can pack the username/password in the connection string to the DB, it can’t “lose” that. For SQL Server, I can do that. For MySQL, it’s not worked yet…. mrfgh…
When running a Sqoop job via an Oozie task, I got the following errors:
WARN org.apache.sqoop.tool.SqoopTool – $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
ERROR org.apache.sqoop.Sqoop – Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
These were because the mysql connector jdbc driver was not in the Oozie sharelibs folder in HDFS. hdfs://user/share/lib/lib_20141030223533. Note you need to restart the Oozie service in the cluster in order for the driver to get picked up. The Cloudera docs seem to say you don’t have to restart Oozie, but I’ve tested it out and I have to restart Oozie from Cloudera Manager.
If you delete one of the jars from the sharelib and run an Oozie job that somehow depends on that jar, you get insta-failed
You submit an oozie job and it fails. If you see the following error followed by a stack trace in the Logs:
Launcher exception: begin > end in range (begin, end): (1415382327807, 1415382308114)
-AND/OR this-
java.lang.IllegalArgumentException: begin > end in range (begin, end): (1415382327807, 1415382308114)
Check the system time on all your hadoop nodes. Those times are not coming from your query – they’re coming from your hadoop nodes comparing their system times.
$> date
If they’re skewed even a little bit, that error will come up. So, make sure you’ve got ntp running on all the servers and make sure they’re all pointed at the same ntp server.
Here’s an existing Jira task in the Apache project for Oozie that I finally came across while searching (the submitter mentions just an 8 second time skew causing this):
/etc/hosts
Say it looks like this:
127.0.0.1 localhost.localdomain localhost.localdomain localhost4 localhost4.localdomain4 localhost mycpu001
::1 localhost.localdomain localhost.localdomain localhost6 localhost6.localdomain6 localhost mycpu001
192.168.1.51 mycpu001
192.168.1.52 mycpu002
192.168.1.53 mycpu003
192.168.1.54 mycpu004
Sometimes if you do nslookup mycpu001, it will resolve to “localhost” or some variant in that line. Not sure that would be entirely expected.
I’m going to remove the “mycpu001″ from the first 2 lines (and the equivalent on all other machines)
in /var/log/mysqld.log you see this line toward the end:
“Fatal error: Can’t open and lock privilege tables: Table ‘mysql.host’ doesn’t exist”
Do yourself a favor and just uninstall mysql, disable selinux, reboot, then reinstall mysql:
1. yum remove mysql-server
– rm -rf /var/lib/mysql
2. nano /etc/selinux/config
– “enforcing” -> “disabled”
3. reboot
4. yum install mysql-server
5. continue where you left off
– your my.cnf file will remain
I’m sure someone really really disagrees with the “disable selinux” bit
14/11/06 17:25:04 INFO mapreduce.Job: Running job: job_1415319646271_0006
14/11/06 17:25:14 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=FAILED. Redirecting to job history server
14/11/06 17:25:14 INFO mapreduce.Job: Job job_1415319646271_0006 running in uber mode : false
14/11/06 17:25:14 INFO mapreduce.Job: map 0% reduce NaN%
14/11/06 17:25:14 INFO mapreduce.Job: Job job_1415319646271_0006 failed with state FAILED due to:
14/11/06 17:25:14 INFO mapreduce.ImportJobBase: The MapReduce job has already been retired. Performance
14/11/06 17:25:14 INFO mapreduce.ImportJobBase: counters are unavailable. To get this information,
14/11/06 17:25:14 INFO mapreduce.ImportJobBase: you will need to enable the completed job store on
14/11/06 17:25:14 INFO mapreduce.ImportJobBase: the jobtracker with:
14/11/06 17:25:14 INFO mapreduce.ImportJobBase: mapreduce.jobtracker.persist.jobstatus.active = true
14/11/06 17:25:14 INFO mapreduce.ImportJobBase: mapreduce.jobtracker.persist.jobstatus.hours = 1
14/11/06 17:25:14 INFO mapreduce.ImportJobBase: A jobtracker restart is required for these settings
14/11/06 17:25:14 INFO mapreduce.ImportJobBase: to take effect.
14/11/06 17:25:14 ERROR tool.ImportTool: Error during import: Import job failed!
I have not idea why this is happening. It may be something to do with a copy-paste of the command from notepad. If I retype the last parts of the command in the shell, it doesn’t throw that error.
The lame thing is that Google searches for “mapreduce.ImportJobBase: mapreduce.jobtracker.persist.jobstatus.active = true” get nearly nothing besides source code dumps. Whatever the case, if you get that sort of result after running a sqoop command, just retype the whole mess, especially if you copied and pasted it in to the console window.
This was the single remaining config error I was seeing in Cloudera Manager, and I was not finding too much about it in my searches:
Thrift Server role must be configured in HBase service to use the Hue HBase Browser application.
I’d click on the details for that config error, and the only option was “None”. In Hue, the error was that it couldn’t connect to localhost:9090. Well, the problem was that there were no HBase Thrift instances install in my cluster. Heh heh. After installing one it showed up as an option other than “None” in the config file.
My thoughts after trying them all (on local VMs)
1. You need more than 18GB of RAM on your machine in order to effectively test. Just do it.
2. Cloudera is the easiest to install. AND it sets up Hue for you. Hortonworks and MapR require a LOT of manual edits to arcane config files (they seem arcane when you’re new). Hortonworks is the next-easiest. MapR was the hardest.
3. MapR, to me, has the most promise, given it’s closer to the metal. The promise of random writes directly to the HDSF cluster just seems really really good.
4. MapR is the hardest to install. It just takes more command-line work.
5. Adding nodes to a cluster is strightforward with Hortonworks and Cloudera. With Mapr, you have to do more command-line prep than a noob will prefer.
6. MapR seemed to have some of the best documentation on how Hadoop works. Hortonworks was up there too. Cloudera seemed a little less than screamingly clear… but that could have been due to the fact that theirs was the first docs I had started reading.
7. Hortonworks installs a MySQL instance for the Hive metastore. Cloudera and MapR use some embedded Postgre DB, which they repeatedly say not to use for much beyond a proof of concept cluster.
8. Cloudera has some proactive notifications on config best practices. However, I’m not sure why something like Java heap size configs would differ – I suppose the installer may set things to some percentage of available RAM.
I intend on having 2 separate namenode servers, and on each one I will be running a MySQL instance. Hive metastore data, Impala metadata, namenode info, Cloudera Manager, and other roles will all use these DBs. Not sure how it’ll all go yet, but my first attempt last week didn’t go so well.
Read through this, then at each step, read ahead a few steps, as the info is a bit spread out: http://ift.tt/1wPj4BF
Do yourself a favor and disable selinux before installing. Otherwise you’ll likely see this error when mysql fails to start:
Fatal error: Can’t open and lock privilege tables: Table ‘mysql.host’ doesn’t exist
(it’s due to permissions on the /var/lib/mysql folder
– nano /etc/selinux/config
– “enforcing” change to “disabled”
– reboot machine
– yum remove mysql-server
– rm -rf /var/lib/mysql
– yum install mysql-server
1. yum install mysql mysql-server
2. in /etc/my.cnf (maybe make a copy of your original, e.g. cp my.cnf my.cnf.orig)
– paste in the sample my.cnf file contents from here (the paths should match the default install paths, but check just in case): http://ift.tt/1wPj4BF
– uncomment the binlog lines and set it to “mixed” (otherwise it’ll error out when your cluster starts up)
3. service mysqld start
4. chkconfig mysqld on
5. /usr/bin/mysql_secure_installation
6. Install Extra Packages for Enterprise Linux (EPEL repo)
– wget http://ift.tt/1htTSIk
– sudo rpm -Uvh epel-release-6*.rpm
7. Copy in the mysql connector (do this on both machines)
– mkdir /usr/share/java
– cp mysql-connector-java-5.1.33/mysql-connector-java-5.1.33-bin.jar /usr/share/java/mysql-connector-java.jar
– note the filename changes when you copy it (if it’s not exactly that you will get errors when testing DB connections)
8. At this point I installed Cloudera Manager on machine1
– from here: http://ift.tt/1wPj4BJ
– wget http://ift.tt/1eEtSMH
– chmod u+x cloudera-manager-installer.bin
– ./cloudera-manager-installer.bin
– the installer downloads everything you need
9. I’m having it install Java
– later I’ll install the mysql connector
– I had one node fail during installation – just “retry failed hosts” a few times. Likely it was some connectivity thing
10. Now for the Cluster Role Assignments step
– All services that require a DB will be on the hosts that have mysql installed and running
– NameNode – first server
– Secondary NameNode – second server
– Cloudera Management Server Activity Monitor – second server
11. Database setup
– “Use Custom Database”
– Database Type: MySQL
– create databases and users for each service listed at the bottom of this page: http://ift.tt/1wPj4BF
– this page provides another description of this process: http://ift.tt/1qb7xXE
– from inside a mysql client on the host mentioned has storing the activity monitor:
– create database amon DEFAULT CHARACTER SET utf8;
– grant all on amon.* TO ‘amon’@’%’ IDENTIFIED BY ‘amoncrazypassword';
– from inside a mysql client on the host mentioned as storing the hive metadata:
– create database hive DEFAULT CHARACTER SET utf8;
– grant all on hive.* TO ‘hive’@’%’ IDENTIFIED BY ‘hivecrazypassword';
– In the “Database Host Name:” field, mine looked like this:
– “:3306″ since that’s the port mysql was running on
– I’d recommend not explicitly specifying a hostname because there are a lot of configs the installer auto-configures.
12. You should get some nice green check marks indicating successful connections. If not:
– service iptables status
– check the port mysql is configured to listen on (default is 3306)
– service mysqld status
– make sure the right database and user are created on the appropriate host
13. If the install finishes smoothly, make sure to set up backups.
– I’ve seen it fail on the “Creating Hive Metastore Databse Tables” step with an error about there needing to be a hostname configured. Just open up a new tab pointed at your Cloudera manager and then look for the red configuration alert next to the Hive service in your cluster. It should take you directly to the offending config. In my case, I clicked on the button to revert it to default. Then go back to the Cluster Setup tab and click Retry. Things were happy again.
14. I’d suggest the first thing you do is restart your cluster
– there are usually some errors that a restart will clear up
15. Hue creates its own embedded database. If you want to have it run on your mysql instances:
– for Hue: http://ift.tt/1qb7vPy
– from a mysql client on the host running hue:
– create database hue DEFAULT CHARACTER SET utf8;
– grant all on hue.* TO ‘hue’@’%’ IDENTIFIED BY ‘huecrazypassword';
– in Cloudera Manager click on the Hue Service > Configuration tab at the top > look for “Database” configs on the left
– update appropriately
16. Honestly, expect the startup of a new cluster to feel a bit like whack-a-mole with the errors on services. Just click on them, look at the logs, and then restart them a few times. The HDFS and HBase services, if not working right, can cause other services to error out, especially the health checks. All that means is focus on HBase and HDFS first.
MapR has a fantastic feature where it exposes the Hadoop file system directly via NFS. So, you can read and write to the Hadoop cluster directly from client machines, rather than having to use other tools you have to learn. Now, to be fair, Hortonworks and Cloudera have NFS share features too. The difference is that they buffer the incoming data before actually writing it to the cluster. With MapR, the data is written as it’s received. MapR should have better performance with NFS shares too.
The use cases for DWH are pretty obvious: if you can export csv files and have them simply write directly to the Hadoop cluster, then you’ve cut out an entire step in the data workflow. No need to use special ODBC drivers in SSIS to write data to Hadoop, no need to move very large files around over your network. You can even mount the NFS share on your web servers and have them log directly to that mount point. As the data is laid down, it is queryable too
It’s easy enough to mount an NFS share in Linux and OSX, but for Windows you have to either install some third party NFS client, or you can do this:
Enable the NFS client features in Windows – search for Add Features
regedit.exe
HKEY_LOCAL_MACHINE > SOFTWARE > Microsoft > ClientForNFS > Default > add 2 DWORDS:
AnonymousGid
AnonymousUid
The values need to be 0, which will be set by default.
from a command line:"nfsadmin client restart" (or reboot)
Now when you browse to your mapR machine’s IP via Explorer, you will see the shared folders. You can copy stuff in or out, delete, etc. Note, I’ve only ever seen 30MB/s at the fastest.
mapR docs on the subject:
Notes on security:
I haven’t looked yet, but it’d probably be a good idea to restrict which hosts can hit which folders on the Hadoop cluster. I haven’t yet messed with how mapR exports the NFS shares, but I suppose IP restrictions should be alright.
Goal:
Let data get firehosed into HBase. Then auto-generate and maintain Hive external table schema based on the actual key-value pairs in the HBase column families. It seems nobody’s really doing this too much. However, it seems this would be a general solution for data warehousing (as long as you can get all data into JSON format in an HBase table).
How to generate dynamic Hive tables based on JSON:
The good JSON serde:
Generate a Hive schema based on a “curated” representative JSON doc:
In a comment on his own OP, how to create an external table pointed at an HBase table that returns everything as JSON. It may be useful as the source of the “curated” JSON doc:
I’ll update on progress…
I found this video by the CTO of MapR to be very explanatory about now Mapr does things.
In short, there are no NameNodes to die in your Hadoop cluster, their MaprDB seems even better than HBase, their NFS allows for random read-write, and everything on the cluster is HA (not sure still why exactly there are a limited number of control nodes though – more to learn).
https://www.mapr.com/resources/videos/architectural-overview-mapr
Figured I’d consolidate what I did to get data into a Hadoop cluster with SSIS/SSDT. I used SSDT 2013 – the BI version install. In this case, it was a Hortonworks Hadoop cluster.
1. Install both the 32 and 64 bit versions of the Hortonworks Hive ODBC driver: http://ift.tt/1wsXYrp
– edit the DSNs that should already be in there from the installer to point at whichever node is running a HiveServer2 service (find it in Ambari)
– you have to edit the 32 bit one via the 32 bit ODBC Data Source Administrator and the 64 bit DSN via the 64 bit ODBC Data Source Administrator
– In “Authentication”, I could only get it to work with “username” or “username and password”. “No Authentication” resulted in a test just hanging for a long time. I used the “hive” account creds as defined in Ambari
2. At this point you must have a Hive table schema already existing. Just map columns from the data flow to columns in the Hive table schema and you should be good to go.
– Now, to be honest, I keep getting a permissions issue with writing data to the table. I haven’t solved it yet, but it may be problematic. I’ll try to update later with what works.
How do you create a table in Hive? You can use Hue or HCat or whatever web-based tool may be installed on your Hadoop cluster. But here is how you can do it via the command-line:
1. SSH into one of the nodes on your cluster that’s running Hive
2. change use to a user that has permissions in Hive:
– su hive
3. start hive
– hive
4. Once you’re in hive, it’s just like when you’re logged in to a mysql client – generally the same commands are available)
– show databases;
– use database default;
– create table (Year int, Month);
Installing their manager was simple enough:
Install steps here: http://ift.tt/1Dm56HJ
You can use any VM – not just the Vagrant steps described.
1. install the Hortonworks repo:
– wget http://ift.tt/1fESf8t
– cp ambari.repo /etc/yum.repos.d
2. make sure you set up your hosts file at /etc/hosts
2. install the ambari-server
– yum install ambari-server
– ambari-server setup
– just go with defaults
3. start ambari-server
– ambari-server start
– it’ll fail if your hosts file doesn’t include the hostname of the machine you’re running it on
4. in order to add machines to a cluster you need to have the ambari agent installed on each machine
– http://ift.tt/1Dm56HL
– the Ambari wizard will do it automatically ONLY if you have ssh keys installed
– if you don’t have ssh keys installed you will have to manually install the ambari agent on each machine
– wget http://ift.tt/1fESf8t
– cp ambari.repo /etc/yum.repos.d
– yum install ambari-agent
– point the agent at the Ambari server
– nano /etc/ambari-agent/conf/ambari-agent.ini
Notes:
1. The step where you register and check the servers: http://ift.tt/1tA1cuT
– Hosts will fail if you don’t have the /etc/hosts files synced up to reflect all the hosts
2. during install, you can tail -f /var/log/yum.log to see everything Ambari is installing
3. During the “Install, Start, and Test” step, I got a failure on all 3 nodes with this message: “Puppet has been killed due to timeout”. I see yum still installing stuff on each node, so I presume it’s just a case where my internet connection was too slow and it really did timeout. I’m going to let yum continue and then simply click the “retry” button in Ambari…. yeah, after waiting clear through when it installed the mysql-connector packages, I clicked the “Retry” button in Ambari – it’s progressing now.
4. This is kinda cool – it installs mysql as the backend DB for the manager and Hive.
5. It will install Nagios server and agents – again, kinda cool
6. It seems to use Puppet for deployment
7. Note the mysql install could be more secure. Run this on the mysql host at some point: mysql_secure_installation (use all defaults except the mysql root password)
Problems I encountered:
1. Not all services started right up after the install. I had to manually start a few
2. In the initial setup wizard I changed the oozie configs to use a mysql database (the one that was configured to be used for hive). However, the oozie username does not get created and neither does the database. I had to create mysql users oozie@% and oozie@horton2 (the name of the machine oozie was running on, not it was also the same host). The oozie@% wasn’t strictly necessary, but I added it for future machines I might run oozie on.
3.
I’m going to give a step-by-step as I’m doing it. I’m running on a fresh CentOS 6.5 minimal install:
1. Download installer from here: http://ift.tt/1yw9kLV
– I had to do yum install wget
– mv mapr-setup mapr-setup.sh
– chmod u+x mapr-setup.sh
– ./mapr-setup.sh
– I see Run “/opt/mapr-installer/bin/install” as super user, to begin install process
– /opt/mapr-installer/bin/install
2. “Unable to install package sshpass”
– looking at logs in /opt/mapr-installer/var/mapr-installer.log I see it tried doing yum install sshpass
– well, I guess it wasn’t found in the default repos I have
– I’m doing this: http://ift.tt/12u8GUO (installing maprTech and EPEL repos)
3. Some questions about the install come up
– I typed in the hostname when it asked for the hostname for the control nodes (I guess a consistent /etc/hosts file would be a good idea)
– I go with default answers (I hit enter a bunch)
4. “Disks not set for node: c1″ (c1 is the hostname of my control node, which is the machine I’m running this installer on)
– I select to modify the options
– “d”
– “Enter the full path of disks for hosts separated by spaces or commas []:”
– I open a new SSH window and do lsblk to see possible drives. I’m going to add a new disk to this machine really quick
– echo “- – -” > /sys/class/scsi_host/host0/scan
– lsblk (don’t see any new disks
– echo “- – -” > /sys/class/scsi_host/host1/scan
– lsblk (still no new disks)
– echo “- – -” > /sys/class/scsi_host/host2/scan
– lsblk (ahh… there’s a new disk at sdb
5. /dev/sdb is the full path to the disks I want to use
– does this mean all hosts need to have the same device paths?
– not sure, but I’m going to find out how things go with adding data nodes later on
6. I put in the SSH creds
7. This is a decent wait
– while poking around in /opt/mapr-installer I noticed an ansible directory – cool way to install stuff – just use an ansible playbook
– Cloudera requires the Oracle JDK. MapR uses OpenJDK, which seemed to download and install faster than the Cloudera install.
– I noticed MapR has some good instructions on how to set up a local repo: http://ift.tt/12u8GUO
8. Well, that install took about 20 minutes (on the safe side)
9. I can log in. Now I’ll see how things look, create a few data node VMs, create a new cluster (or is the cluster technically created and I just need to add more nodes?)
Clone a working CentOS 6.5 machine and start it up:
update the hostname here:
nano /etc/sysconfig/network
Reset the NICs:
rm -rf /etc/udev/rules.d/70-persistent-net.rules
reset the UUID and MAC address:
nano /etc/sysconfig/network-scripts/ifcfg-eth0
– remove the stuff after “HWADDR=” just on that line though
– remove the stuff after “UUID=” just on that line)
– make sure you see ONBOOT=yes
reboot
After this, you may not need to mess with the MAC address, as mentioned below, but I figured I’d include how I’ve found to do it.
After reboot, do one of the below to set the MAC address:
1. Go to that above 70-persistent-net.rules file (it gets recreated), get the MAC address and then paste it into “HWADDR=” line
-OR-
2. run system-config-network-tui and confirm the network settings are as desired, then save it. This will put in the MAC address
Situation:
Running Tableau behind a NAT
Tableau Server is behind a proxy server
No DNS from public internet
Wanting to extract Google Analytics data automatically via Tableau Server
Note:
The OAuth setup will not work without the proxy settings configured for the Run As account. Here’s what the error looked like after clicking “Accept” on the IE popup confirming I wanted to allow Tableau Server to connect to my Google Account via OAuth (after a 20 second pause):
Tableau Server encountered an internal error.
Request ID: VElOGwoxNiEAADC0qPEAAAHu
Tableau Server version: Version 8.2.3
(8200.14.0925.1717) 64-bit
Here are the steps I took to set up Tableau Server OAuth. They clarify a few things not super-explicitly mentioned here: http://ift.tt/1tmnlLZ:
1. Set up Google OAuth
– create new project here: http://ift.tt/1fljiJ4
– set up a new Client ID under “Credentials
– REDIRECT URI needs to be set to the hostname/domain name you use in your browser to connect to Tableau Server when you set up your Google Analytics key. Note, this cannot be an IP address. “localhost” works if you want to set up OAuth while logged in to the Tableau Server. Really, this whole thing is complicated by the fact that there is no direct, consistent public IP/domain name for accessing Tableau.
– JAVASCRIPT ORIGINS – just blank it out
2. Enable the “Analytics API” in the APIs page of your project
2. log in to Tableau Server (Web UI) with the hostname/domain name you put in the above REDIRECT URI field.
3. The Tableau Server admin needs to go to Admin > Maintenance and check the boxes in the Settings section to allow users to save their own OAuth tokens
3. On the top-right click on your name > User Preferences > click on “Add” next to Google Analytics
Notes on REDIRECT URI:
1. IP addresses result in Google saying you need to supply a device_id and something else – doesn’t work
3. You could make up a domain name and add it to your hosts file (and all your end users’ hosts files…. yikes), and telling IE to not use a proxy server for addresses beginning with that made-up domain name
The Google Analytics connector will throw odd 500 errors if you don’t configure the Run As account to use the proxy server. If you happened to have Tableau Desktop installed on the Tableau Server machine, everything would have worked just fine…. because you had likely set your own proxy server settings. You also need to set the proxy server configs for your Tableau Server’s Run As account… there, I said it twice.
You have to login/RDP to the Tableau Server as the Run As account to do this. Running IE as the Run As account and configuring the proxy server didn’t “stick” (after setting it, I immediately opened up the configs again and they were reset to default). You have to RDP in. If you don’t, here’s an error you see when publishing a workbook to the server:
An error occurred when publishing the data source.
GetOAuthUsername caught exception processing specs Response code: 500
The Cluster Setup step (step 5) where it creates the Hive metastore tables kept failing with the below error (seen in the logs when you click on Details > stderr). I was installing the embedded Postgres DB.
Error connecting to db with user ‘\”hive’\” and jdbcUrl ‘\”jdbc:postgresql://localhost:7432/hive’\”’
It seems I got it to get past this by manually clicking to start the Hive metastore service:
Coudera Manager > in the cluster section on the left click on Hive > click on Instances tab > select the Hive metastore server and from Actions select “start”. For whatever reason, after doing that and then going back and clicking “Retry” on the Cluster Setup page, it didn’t throw the error and the cluster seemed to start up alright.
The “Cluster Installation” phase of installing the Hadoop cluster didn’t have CDH 5.2 – it only showed 4.7. It had also auto-selected some of the other parcel options (Accumulo, etc.), when every other time I’ve done this installation it had by default selected “None”. The following step where it downloads and distributes the parcels seemed to hang after downloading; so when I clicked “Back”, the cluster installation restarted and I had to reselect the nodes I wanted to include in the cluster. After that, the usual default selections of CDH 5.2 and none of the other parcels showed up; and this time it seems the download, distribution and activation worked fine.
My URL looked like this, in case you’ve encountered the same thing:
/cmf/express-wizard/wizard#step=parcelInstallStep
I don’t think this something is particularly broken, but I figured I’d post this as an edge case help.
At this point I think you may be able to start a new cluster installation from a fresh Cloudera Manager install: http://192.168.1.201:7180/cmf/express-wizard/. I’m not sure about that, but I’ll update later if it is indeed the case.
BTW, the “Abort Installation” button on the Cluster Installation (step 3) applies to hosts whose installation has NOT completed successfully. It won’t roll back all your hosts, in the case where just one host is unhappy for some reason. I’ve seen a fairly regular occurrence of just one host having some intermittent network problem for example, that results in the express-wizard getting hung up waiting on that machine, but that machine doesn’t know to tell the wizard anything. So you could be sitting there forever. In that case, just click that “Abort Installation” button. You should then see buttons replace it about rolling back failed hosts and then attempt reinstall on failed hosts.
Cloudera Manager will set all the Hadoop stuff up for you. However, by default it will do it with an embedded PostGres DB, which they say will not scale well as you grow your cluster. Also, it may be harder to do backups and such. It seems that’s the only significant blocker from simply saying “just run the Cloudera Manager auto-installer”.
I figured I’d install MySQL, which is one of their supported DB platforms. But we don’t really have to do that until the Cloudera Manager cluster setup is to the point where you’re setting up the Hive metastore (step 3 of “Cluster Setup”, well into the whole shebang). The installation instructions seem to make a big deal about having to have the databases and such figured out before you ever run the Cloudera Manager installer…. you don’t – you can just install them when the Cloudera Manager gets to the database setup step.
The URL I’m at has “step=showDbTestConnStep” in it. I see “Database Setup”, with options for “Use Custom Database” and “Use Embedded Database”. At this point I went to the node that has the hive role and installed mysql this way:
yum install mysql-server -y
chkconfig mysqld on
/usr/bin/mysql_secure_installation
I did a CentOS minimal install, so I got some errors Cloudera not being able to find the JDBC driver to connect to the mysql database, so I did this to install it, and things were happy.
sudo yum install mysql-connector-java
After all this I got an error saying Cloudera couldn’t find the specified database, so I did this to simply create it:
mysql
create database hive;
After installing things on a few nodes I saw this message when Cloudera Manager was “checking things for correctness”:
Cloudera recommends setting /proc/sys/vm/swappiness to 0. Current setting is 60. Use the sysctl command to change this setting at runtime and edit /etc/sysctl.conf for this setting to be saved after a reboot. You may continue with installation, but you may run into issues with Cloudera Manager reporting that your hosts are unhealthy because they are swapping. The following hosts are affected:
Run this on each node: sudo sysctl -w vm.swappiness=0
I did a minimal install, so here were the necessary things:
1. Disable SELINUX
– nano /etc/selinux/config
– set it to “disabled”
2. install python
– yum install python
3. install wget (to get cloudera-manager installer)
– yum install wget
4. install nano (because I like it)
– yum install nano
5. allow root to log in remotely (Cloudera Manager requires this)
6. disable iptables
– only because I’m running all machines within a locked-down subnet
– yeah, scary with allowing root to log in remotely
Likely convenient things:
1. set a common hosts file for all machines:
– list out all the IPs and hostnames for each machine in this file
– this is where you can specify the fully qualified domain name as well as host names
– Cloudera Manager will work more seemlessly in various ways
– You can have Cloudera Manager search for nodes to add by these host names
2. set up SSH keys so Cloudera Manager can use those instead of SSH passwords (
– though it can use SSH passwords as well
Note that when Cloudera Manager sets up a host, it will download 500+ MBs of install packages on each machine. In my case, the YUM install logs showed this:
Transaction Summary
================================================================================
Install 117 Package(s)
Total download size: 513 M
Installed size: 894 M
Downloading Packages:
If you want to use “bridged” connections, where the VMs appear as any other machine directly attached to your router (and dhcp server), you may need to go Edit > Virtual Network Editor > select the “Bridged” Type row, then below in the “Bridged to:” drop-down, select your NIC that is your main internet connection. I’ve tried loads of different virtualization and VPN solutions, so I have about 12 different network connections.
Once I did the above, my VMs set to “bridged” NICs got an IP from my dhcp server just fine.
Critical points that may or may not be obvious:
1. If you’re using the Sqoop stuff in the web-based interface, you’re actually using Sqoop2
2. You have to download and install the JDBC driver for SQL Server yourself
– curl -L ‘http://ift.tt/1CfDcMU; | tar xz
– sudo cp sqljdbc_4.0/enu/sqljdbc4.jar /var/lib/sqoop2/
– while you’re at it, you may as well put it in the sqoop directory too: sudo cp sqljdbc_4.0/enu/sqljdbc4.jar /var/lib/sqoop/
3. Sqoop2 home directory is /var/lib/sqoop2/ (maybe not…….)
4. restart Sqoop2 service after copying in the JDBC driver file:
– sudo service sqoop2-server restart
5. connection string in “Manage Connections” is like this: jdbc:sqlserver://192.168.1.102
6. for an action, leave schema and table name fields blank and just paste in your TSQL query, then append this to the end of it: +and+${CONDITIONS}. Don’t mess with the boundary query stuff until some other time (Sqoop2 will automatically query for the min/max of the Partition column name you provide).
7. if you mess with the connection you create in Hue/Sqoop2, note you have to type in the password
8. if you get errors, don’t fight it – you have to log in via SSH and look at /var/log/sqoop2/sqoop2.log
You can only run one or the other; not both at the same time. When I tried running a VM on VirtualBox on my Windows 8.1 machine, I got this error:
“vt-x is not available”
This post in the VirtualBox forums was my first indication that you can’t have 2 hypervisors running at the same time: http://ift.tt/1vRwgFG
This post is a solution that has worked for me:
In short, you create a new boot profile that disables Hyper-V in Windows 8.1 (covered in the Hanselman blog post). It’s actually kinda slick. When you click “restart”, hold down the Shift key and the menu will come up where you can select a different boot profile.
Test with the following (paste into a shell prompt):
env x='() { :;}; echo vulnerable’ bash -c “echo this is a test”
If you’re vulnerable, you’ll see this:
$ env x='() { :;}; echo vulnerable’ bash -c “echo this is a test”
vulnerable
this is a test
To fix it:
In CentOS:
yum update bash
In Ubuntu:
apt-get install bash
When you run the text code again, you should see this:
$ env x='() { :;}; echo vulnerable’ bash -c “echo this is a test”
bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `x’
this is a test
If you’re using Salt Stack, you could run this to update all your minions’ bash installs:
salt ‘*’ pkg.install bash
There are some more sophisticated ways to do it, especially for larger environments, but I figured I’d mention it.
There was an “E” in the freezer temperature display and an “F” in the refrigerator temperature display inside the refrigerator side.
The problem was that the freezer side fan could not spin due to ice build up. When I pulled out the freezer shelves, I saw water had been flowing down the back wall and was frozen. When I pulled off the panels, sure enough, the fan did not spin and there was a ton of ice all around the fan housing. I removed the ice and plugged the unit back into the wall; everything is happy now.
We have had some really hot weather lately, and there was a power outage for a while. Combine that with kids standing with the freezer door open forever looking for popsicles…. That may have been enough to melt some ice that was sitting in the ice dispenser storage tray.
I had seen a number of people posting about this issue who replaced the mainboard (for nearly $150 it seems). Others replaced the freezer fan. I suspect I’m not the only one that’s had an ice buildup problem. So, in some of those cases, I bet they didn’t actually have to replace anything – they just needed to unplug the freezer and let the ice melt a bit.
The “E F” code means either the freezer fan or the refrigerator fan has some sort of problem.
Here are some pics for your perusal fun:
The answer in this thread pointed me to the fan in the freezer:
The video on this page will give you an idea of what it takes to get at the fan in the freezer:
Here are a number of links to threads of varying usefulness:
This:
Exactly like that person, I cannot figure out how to get it working. All other search results address a slightly different situation (I think).
Not any of the following:
In short, the RDC client that installs as part of MS Office (version 2.1.1 in my case) doesn’t work. Instead, use use this free app MS put in the iTunes app store:
I didn’t know there was an app for OSX in the Apple AppStore. I had always used what came with MS Office. Well, the MS Office one didn’t want to work on a Windows Server 2012 machine.
It’s one of those somewhat long-standing problems with the MS Office RDC client, but I had always [sort of] gotten it to work. Here’s a thread that encapsulates what I’ve experienced (some of the proposed solutions have worked for me at one point or another, but nothing worked just now).
We were seeing some very very slow forum page loads about 2 weeks after upgrading to a new version of IPB – approaching 10-15 seconds in some cases. A server restart seemed to help initially. However, the slowdown showed up again shortly after.
I noticed the Processes were consistently showing a number of queries waiting for a table level lock. After refreshing for a while I saw cases where the queries were waiting for 7 seconds. They were waiting for “delete” queries on the sessions table. The sessions table had nearly 100,000 rows, as a note.
Looking at the delete query in a bit more detail, it was of this sort: “delete from sessions where (ip_address=x.x.x.x or running_time<142255332)
I noticed there was no index on ip_address, so we added one. Since adding that index on sessions.ip_address, there have been no queries waiting on a table level lock, and forum page load times have gone down even more.
Moral of the story? If you’re seeing slow page loads in your IPB forums and you have a lot of traffic to your forum, add an index to sessions.ip_address. There is no scenario where that would be a bad thing, and there are many scenarios where it is a good thing.
The following applies this situation: SSAS is installed on a machine not connected to a Domain. Tableau Server installed on a machine not connected to a Domain. Tableau Server users are set up in the local Tableau user database (not Domain user accounts). You want all users to be able to view data pulled from the SSAS instance without having to enter additional credentials beyond their Tableau Server creds.
If you want to use Tableau Server with SSAS cubes, you need to set up Tableau Server to be running under a machine account that has permissions to access the SSAS cube (on non-domain machines this means creating matching usernames/passwords on each machine). IN ADDITION, when you publish a Tableau workbook or data source to the Tableau Server you need to be running Tableau Desktop under a machine account that has permissions to access the SSAS cube. This is the only way you will see the option to have Tableau Server use the RunAs account when it connects to the SSAS cube.
If, from Tableau Desktop, you connected to the SSAS cube by typing in credentials, you will not see the RunAs authentication option when you publish the workbook or data source to Tableau Server (meaning your users cannot use the workbook or data source).
The following help articles, while useful, didn’t seem to address the above use case (not in its entirety anyways):
If you point Tableau at SSAS multidimensional cubes, you will want to be able to use relative date filtering. Here’s the easiest way:
You will filter on dimension attributes whose values are of this form: YYYY-MM-DD, YYYY-MM, and YYYY. The properties of the YYYY-MM look like this:
If you put those attributes in a hierarchy, do not hide them, as SSAS suggests (with a blue squiggly that has a tooltip that says “Avoid visible attribute hierarchies for attributes used as levels in user-defined hierarchies”). When you’re in Tableau, you need to tell Tableau to treat these 3 attributes as dates, and you cannot do that to attributes that are *only* in a hierarchy.
Once your cube has processed, go in to Tableau and connect to it. You will see this by default:
You want to change it to this:
After that, relative date filtering seems to work fine.
This blog post mentioned something about having to change how unique values are determined by Tableau: http://ift.tt/1pyOuoo
I didn’t have to though. This is what seems to be working, and it’s the default that Tableau decided on:
You *MUST* be able to increase disk space without having to reboot. Imagine your application has blown up and your production database servers are filling up disk space with binary logs. It’d be unacceptable to have to reboot your DB servers just to add more space.
Here’s how to do it without having to reboot:
1. Add another disk – physical disk virtual disk… just add another one (don’t extend an existing virtual disk – you’d have to reboot to see that extra space).
2. fdisk -l
Tell CentOS to rescan for disks:
echo “- – -” > /sys/class/scsi_host/host0/scan
-then check to see if it found a new disk -
fdisk -l
(you should see a new device in the output when compared to the first time you ran fdisk -l… if not, rescan on different “host” numbers, e.g.
echo “- – -” > /sys/class/scsi_host/host1/scan
echo “- – -” > /sys/class/scsi_host/host2/scan
echo “- – -” > /sys/class/scsi_host/host3/scan)
3. fdisk /dev/sdb
n
p
1
“enter” “enter” “enter” to use all space
t
1 (this may be auto-selected… it’ll say)
8e (this is default CentOS 6.5)
w
4. pvcreate /dev/sdb1 (the “1″ is really the number you selected in the previous step)
5. vgextend /dev/vg_centos6 /dev/sdb1 (the vg_centos6 is the default virtual group with CentOS 6.5 install. Use “vgdisplay” to show all your virtual groups if your setup is different)
6. lvextend -l +100%FREE /dev/vg_centos6/lv_root (“lv_root” is from default CentOS install. use “lvdisplay” to see all your logical volumes in case you have a different setup)
7. Now resize the filesystem to fill the space on the newly-expanded logical volume:
resize2fs /dev/vg_centos6/lv_root
I’ve copied and pasted in these steps on several servers now (including production MySQL-Percona DB servers, and it just works). I’ve read this method of adding an additional disk and then adding that to an existing virtual group is the recommended way to expand disk space on Oracle installs, so I’m assuming performance is perfectly fine. You could take a snapshot of your VM just before doing this, just in case… though I’m not sure if there might be issues with the new disk… try it on some test VM before doing it to your prod machines).
I had a case where I wanted to see the directory and files being served up. In this case, doing so would provide an approximation of what players were doing inside the web/mobile game. I had already set up Logstash to parse nginx access logs, but this was some additional parsing. Well, here’s the best way to do it: Just grok the same field again. Below is my actual Logstash filter file. The second “grok {…… }” section is what I added.
filter {
if [program] == “nginx-access” {
grok {
match => [ "message" , "%{IPORHOST:remote_addr} - %{USERNAME:remote_user} \[%{HTTPDATE:time_local}\] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer$
}
grok {
match => [ "message" , ".*GET /%{WORD:ActionType}/%{WORD:Action}?.*" ]
}
geoip {
source => “remote_addr”
database => “/opt/logstash/vendor/geoip/GeoLiteCity.dat”
}
}
}
I used this to write the additional grok parse: http://ift.tt/1sPTniB
I do not understand the intricacies of Elasticsearch, but this is one thing that may help contribute to anyone else having problems with unassigned shards of data. If you have either the “head” or “kopf” plugins installed (graphical web interface for administering an elasticsearch cluster), I had some shards showing up at the bottom of the list of nodes in a row called “unassigned”.
Quick searches didn’t yield much, but I tried to “close” the offending index. And they got integrated into the nodes for that index and the cluster health went from “yellow” to “green” and things seem happy again.
Here are a few search results I had reviewed (but they seemed more involved than I was hoping to get into):
When I clicked on the unassigned shards, this is some of what showed up in kopf and head plugins:
state: UNASSIGNED
primary: false
node: null
relocating_node:
null shard: 0
In my case, it seems to have been a bug as described here: http://ift.tt/1zLJNjK
I could see errors in /var/log/salt/master that looked like this:
Received function _ext_nodes which is unavailable on the master, returning False
I did have a few minions that were on 2014.1.4 while my master was on 2014.1.5.
The fix described in the above post is to effectively just paste in a dummy function by that name:
find where the file master.py is and then inside that (there may be 2, so search for the one applicable here), insert the following just above “def _master_opts”:
def _ext_nodes(self, load):
”’
Stub out until 2014.1.5 minion installed
”’
return {}
There’s no minion config file at /etc/salt/minion after installing via Apt. I had to copy one over from a CentOS machine. Seems to work fine then, fwiw.
I did this:
apt-get install salt-minion
When it’s done. No config file. And, yeah, you see lots of unhappy log lines about not being able to find the host “salt” (if you have to specify an IP to the salt master).
Saltstack’s cmd.run is great, but what if you want to run multiple commands and you don’t want to mess with a script?
The only way I could get it to work was to have separate cmd.run blocks written in the order you want things executed. An example would be if you wanted to kill and restart a process every highstate:
Kill the process:
cmd.run:
– name: killall myproc.sh
Wait a few seconds:
cmd.run:
– name: sleep 3s
Start the process:
cmd.run:
– name: /var/lib/myproc/myproc.sh
You can have multiple commands run from one block, but these commands seem to get executed in random order:
Kill then wait then restart:
cmd.run:
– names:
– killall myproc.sh
– sleep 3s
– /var/lib/myproc/myproc.sh
While restoring a forums DB, we hit this error:
File ‘D:\MSSQL\Data\my_forum_db_file.mdf’ is claimed by ‘ftrow_file_new’(3) and ‘my_forum_db_file’(1). The WITH MOVE clause can be used to relocate one or more files.
In this case, the problem was because by default the restore database wizard doesn’t automatically change the destination file name of the full text index data file. In the Options section of the restore database window, just change the destination file name to something different than the main data file of the database.
I know – this may not be too clear, but you should see what I’m talking about when you right-click on the DB instance > Restore DB > point at the backup file > Options on the left top.
It was not obvious to me in the docs exactly where to put scheduling “stuff”. I was just looking at the scheduling docs page, so I’m sure some other doc mentioned this. Here’s how to get you started in case you don’t want to read more
nano /srv/pillar/top.sls
base:
‘*’:
– schedule
mkdir /srv/pillar/schedule
nano /srv/pillar/schedule/init.sls
schedule:
highstate:
function: state.highstate
minutes: 60
maxrunning: 1
All that stuff will then result in all your minions running a highstate every 60 minutes. You can obviously filter by changing the ‘*’ to whatever partial minion name you want, just like with salt states.
You can see these schedules in each minion’s pillar data:
salt ‘lbtest*’ pillar.data. For me, it showed up at the bottom of the returned data.
I figure this could or should replace cron jobs, say, for database backups, though I’m not sure if there are any *serious* dangers of the salt minion dying or otherwise failing.
Here are a few event IDs to look for:
4625 – classic failed logon attempt
5156 – means your computer permitted the connection (look at your firewall to see if you’re allowing inbound connections from IP ranges you really don’t need to allow – more on that below)
A good way to know what your Windows machine’s firewall is allowing inbound is to do this:
Server Manager > Configuration > Windows Firewall with Advanced Security > Inbound Rules > then on the right go Filter by State > Filter by Enabled > then sort everything by the “Remote Address” column
Then, just go down the list and look for Remote Address ranges you really don’t need to allow in to your machine. Generally speaking, make everything either “Local subnet” or specific IP addresses/ranges you *know* you need to allow in to this machine. If you see “Any”, then that local port should be something like 80, 8080, 443, or something you actively want every computer on the entire intarwebs to be able to access.
I just have to believe Salt Stack is a great low-overhead and simple way to monitor servers.
One way you can start with this is by using the PsUtil interface module that comes with Salt now.
I had come across this slide deck. It seemed to involved for my liking, but something about the Salt Scheduler and this mention of commands that start with “ps.” caught my eye.
Well, Salt has a built-in module that will return various PsUtil outputs. You have to have PsUtil installed on each minion though… specifically the Python version. Here’s how I did it on CentOS 6.5:
yum install python-psutil.x86_64
you could do this concurrently on all Salt minions by typing this on the salt master:
salt ‘*’ cmd.run ‘python-psutil.x86_64 -y’
Or, you could make a global Salt State that applies to all minions that looks something like this (I put this in a folder called globalpackageinstalls/init.sls):
python-psutil.x86_64:
pkg.installed:
– pkg_verify: True
and then in your top.sls you would put something like this:
base:
‘*’:
– globalpackageinstalls
Whatever the case, once the python version of psutils is installed, you can run any of the commands for all minions at once. Making use of the “Returners” feature should allow you to get the results into something like ElasticSearch.
salt ‘*’ ps.cpu_percent
salt ‘*’ ps.disk_partition_usage
This is great and all, but alerting is sort of what you really really want in order to be proactive. I’m not sure how to do that *easily*. But maybe alerts are never really all that easy to set up and manage… bleh
Yeah, it’s odd. I don’t see any minion config file at /etc/salt/minion
Must be an install bug… that or I’m totally missing something. But that’s where the config file is supposed to be.
I just type “apt-get install salt-minion”. No file at /etc/salt/minion
I’m excited about the push_dir feature that I think could be used to do filesystem backups
It pushes files to a cache directory on the salt master. Then it should just be a case of rsyncing those directories to a long-term storage server. I’m going to see if I can change the target directory used with push_dir.
An article on *how* epigenetics can work and how these changes can be passed on to subsequent offspring. It’s a rather profound thought that your persistent decisions today can genetically affect your grand children.
These are always interesting reads, if a bit slow-going when you’re not already familiar with the specific terminology used.
Cygwin users have to first have a Windows account.
1. Create a new user. The permissions you assign to this user are the “real” permissions on things you access through Cygwin.
2. Once the Windows user is created, you need to tell Cygwin about it with this command:
mkpasswd -l -u the_new_windows_user_you_created >> /etc/passwd
This creates the appropriate user in the /etc/passwd file, which, like regular Linux installs, shows the available users on the install. Notice the Windows user UID is one of the things that shows up in the line in this file – Cygwin tells Windows “this UID wants to do thus-and-such”.
If you naively clone a machine with the salt minion on it, things won’t work when the clone comes up (assuming you want to keep the original running too). You need to have the salt minion on the newly cloned machine redo the pki “stuff”.
The easiest way is to just uninstall, delete /etc/salt/ and then reinstall, eg. “yum install salt-minion”
Make sure to update the IP of your salt master in /etc/salt/minion
It’s a bit more involved than simply cloning and being done with it, but it’s not too bad. The problem really only happen if you change the host name of the cloned server. This is specifically with Percona XtraDB 5.6.15-rel63.0.
- clone the VM
- nano /etc/my.cnf
– change the serverid to something different, just in case
- nano /var/lib/mysql/auto.cnf
– change the server-uuid somehow (just change any ol’ “a” to “b”… this is a guid, it’s arbitrary anyways). If you don’t, you’ll get this message spamming the logs: “Slave: received end packet from server, apparent master shutdown:”
- nano /var/lib/mysql/relay-log.inf
– change the name of the relay-bin file name (in my case it was still pointed at the previous host name.
Restart the mysql service and then check the error log here: /var/log/mysqld.log (or wherever your install has logs)
I think that’s everything… Hope it helps.
This pointed me in the right direction with the “received end packet….” thing: http://ift.tt/1wTCG6t
