Quickest Steps to Install any Splunk version really really easily

I’m using CentOS 6.6, but any Linux variant should work fine.

You want the .tgz file from here (use wget to downoad the file to the /tmp directory on your server):

http://ift.tt/18rHcT7. Mine looked like this:



wget -O splunk-6.2.2-255606-Linux-x86_64.tgz 'http://ift.tt/1FAMGVO'

--2015-02-26 01:30:13-- http://ift.tt/1FAMGVO


Extract splunk and put it in the /opt directory:



tar xvzf splunk-*.tgz -C /opt


Run Splunk for the first time:



/opt/splunk/bin/splunk start --accept-license


Set Splunk to start at boot:



/opt/splunk/bin/splunk enable boot-start


Deploying VM Clones of Centos 6.6 in VirtualBox

Make a nice, clean CentOS VM, complete with all the utilities you like.

Make a script and put it in whatever default folder you go to when logging in:



nano reset-networking.sh


Put the following in:

# this script resets the networking, as in the case of a fresh cloned VM

# after the reboot, DHCP should pick up a new IP and you're off to the races

# assumes you have nano installed (yum install nano)

# assumes you have removed the "MACADDR=" and "HWADDR=" lines from

# /etc/sysconfig/network-scripts/ifcfg-eth0

# suggested: in /etc/sysconfig/network file, add DHCP_HOSTNAME=

# .... the hostname will then show up in your dhcp table

# remove the existing network interfaces

rm /etc/udev/rules.d/70-persistent-net.rules

# bring up an editor to change the hostname

nano /etc/sysconfig/network

# reboot to complete things

reboot

now make sure the script is executable:



chmod u+x reset-networking.sh


Open /etc/sysconfig/network-scripts/ifcfg-eth0:

Remove the MACADDR= line and the HWADDR= line

Shut down the machine

shutdown -P now

Now you can clone the VM.

When a new clone comes up, log in and execute the script – it’ll reboot the machine and then it should be all good:

./reset-networking.sh

Helpful links:

http://ift.tt/1A8hoRv

http://ift.tt/1EteBc3

http://ift.tt/1A8hoRw

nlog config changes are reflected in real time – and they won’t take effect if the change invalidates the config file

Here’s kind of a nice way to test…. on PROD!

If you need to change your logging configs, you can just do it on your production server. If your changes break anything, they won’t take effect; the already-running configs will stay in place. It won’t crash your application either.

Note, if you stop your application, make a change to the config that breaks the config, the application will not start up. So, hack way… but only while the application process is running!

Fun dangerous things are a blast.

Redshift sql alchemy dialect installation

It was not obvious to me how to install this thing – I wanted to use csvkit to auto-generate table schema and execute it on a DB server.

Install csvkit:

pip install –upgrade csvkit

install client-side postgresql packages (needed for sqlalchemy dialect install):

apt-get install libpq-dev

install python dev (thanks to this: http://ift.tt/1Eci5Q2):

apt-get install python-dev

install the redshift sqlalchemy dialect (found you could do this from here: http://ift.tt/1GaLSYx):

pip install http://ift.tt/1GaLSYz

Piping Linux command line output to multiple commands or multiple files

Here’s what I found that answered this:

http://ift.tt/1hoJanU

command | tee >(next command) >outputfile

The key is to pipe the output to “tee” and then effectively you just list all the programs or files you want the output to go to.

Backup mysql with mysqldump – except one file per table

Add a new disk to a Linux machine (EC2 and Ubuntu in this case)

You need more space. Create a new volume in EC2 and then attach it to the running machine. Then, log in to the machine via Putty or whatever you prefer.



fdisk -l should show the new volume, but of course it's not partitioned. This gives you the /dev/xvdf path (or whatever)

fdisk /dev/xvdf

n

1 (likely will be default)

enter (default)

enter (default)

w (writes the partition to the disk)

mkfs -t ext3 /dev/xvdf1 (that "1" at the end refers to the partition number you typed in (or selected because it was default) just above)

tune2fs -m 1 /dev/xvdf1 (reserve only 1% of space for root user - it's 5% by default, so it prevents non-root users from using this reserved space)


Now you need to mount this new disk:



mkdir /backup (I'm mounting this disk to use as a backup volume)

nano /etc/fstab (edit the fstab file, which defines which disks are mounted where on bootup)

create a new line that looks like this (then save and exit):

/dev/xvdf1 /backup ext3 defaults 0 0

mount -a (mounts all the stuff in /etc/fstab)


Now you should see your new volume showing up in /backup

Run a linux process in the background so you can log out of the console or not have to keep it up

I had a loooong mysql dump to run, and I didn’t want to risk it getting cut off part way through just because my local machine’s wireless network connection timed out or something.

This did it:

1. Run the process (mysqldump cmd with options)

2. Ctrl+z

3. bg

4. disown -h

5. Exit the terminal

Amazon RDS MySQL instance – when it’s really really big

750GB big. If you want to restore a mysql dump to an RDS instance, expect to encounter some issues on the first go-round. You’ll likely have to create a custom parameter group. You’ll likely have to increase your max_allowed_packet parameter for the instance.

So, do yourself a favor and create a parameter group up front and search for my.cnf tweaks that optimize the sort of workload your database will see.

By the way, the following error may be that your max_allowed_packet size is too small for your restore:

ERROR 2006 (HY000) at line 1466: MySQL server has gone away mysqldump: Got errno 32 on write

Recovering a Windows Administrator Password in Amazon EC2

Say you’ve lost the admin creds to your Windows EC2 instance. And then you lose the private key. Bummer. You can still recover the password.

http://ift.tt/1ApWQcd

Just do it step-by-step.

When it says in step 8. (Optional) If your temporary instance is based on the same AMI that the original instance is based on, and the operating system is later than Windows Server 2003, you must complete the following steps or you won’t be able to boot the original instance after you restore its root volume because of a disk signature collision.

What does that look like? Well, when you reattach the dirve and start up the instance, it will continually stay in the “initializing” state. You will see it sort of flash to something else then return to “Initializing”. It’ll do this forever, so if it’s been 5 minutes and it’s still in the “initializing” state, then you have to do step 8.

When, in step 8 it says to “In the Registry Editor, load the following registry hive into a folder named BCD: d:\boot\bcd.” Just open regedit.exe, select the root node, then search for “Windows Boot Manager”. It’ll find something, but look at the registry tree for the one named 11000001.

If you mount the disk to the original instance, by default it puts it at xvdf… you have to mount it at /dev/sda1, otherwise you will get an error message about there not being any root volume mounted.

Regular Expressions, REX, Eval and Splunk – some tips to make it easier on yourself

Splunk uses “PCRE” Regular expressions, so when you use this tool (and you really should) select that from the dropdown: http://ift.tt/1fMXgx6

Just paste in a sample event and it’ll match it in real time, right before your eyes! Granted, Splunk does have this sort of feature, but honestly, I find it helpful to sort of “back up and out” of what I’m doing in Splunk to solve a problem that’s not strictly a Splunk thing.

Reset Splunk Universal Forwarder so it will resend logs

Rreplacing backslashes in Splunk

Say you are extracting data that has nested JSON. Splunk may auto-escape double quotes. You can’t then directly run spath on that field and get anything out of it. You have to remove the backslashes. You need to use the “eval” function and for some reason stuff in 4 backslashes. Like this:

| eval MyDataField=replace(MyDataField,”\\\\”,””)

Splunk answer about this:

http://ift.tt/1DOsu1z

Splunk props.conf – yeah, you have to restart splunk entirely when you make changes

On this page, it seems to indicate you could search with ” | extract reload=T” and it would reload your props.conf file. I didn’t see it work, at least not with extract-time transformations (it may work with search-time stuff).

http://ift.tt/1zpDC5H

Multiple events logged to a single log line – how to work with it in Splunk

I have log lines that are really multiple lines of events (JSON in this case, and many events are batch-logged at once). I need Splunk to split them into individual events.

props.conf is where you have to muck around. And, yes, you have to restart splunk any time you make changes there.

Ultimately, this worked for me:

[name_of_my_sourcetype]

LINE_BREAKER = (\n)

SHOULD_LINEMERGE = false

TRUNCATE = 0

#TIME_PREFIX = “Timestamp”:”

#TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N

#TZ = UTC

KV_MODE = JSON

I’ve commented out the TIME…. things, but they do set the timestamp of each event to be whatever immediately follows that TIME_PREFIX regex.