Archive

Archive for the ‘virtualbox’ Category

Minimal Debian VM upgraded to wheezy / Jenkins with OpenJDK 7

December 20, 2013 Leave a comment

I have upgraded my minimal Debian VM to current Debian stable (“wheezy”). It comes in OVA format which is deployable to Virtualbox or VMware.

The Debian system JDK, i.e. the location of java, javac, etc. commands in the PATH, is still OpenJDK 6 because that is the default-jdk on wheezy and the Jenkins deb packages from jenkins-ci.org depend on it. In particular this means that the Tomcat / Jenkins process itself is executed by OpenJDK 6.

But my Jenkins installer now also installs OpenJDK 7 and pre-configures it as the default JDK for Jenkins jobs. That means you can now use this Jenkins instance to build your Java 7 projects, as well as older Java projects.

Please follow the step-by-step installation instructions if you want to use the VM. It consists completely of Free / Open Source software. I provide it for download “as is” without any warranty of any kind.

Categories: debian, java, jenkins, virtualbox Tags: ,

Sqoop daily Oracle data into Hive table partition

August 9, 2013 Leave a comment

The following bash script can be used to import Oracle records into a Hive table, partitioned by date. It uses Sqoop. Both Hive and Sqoop are part of typical Hadoop distributions, like the Hortonworks Sandbox, for example.

#!/bin/sh

function upper() {
  echo "$1" | tr [a-z] [A-Z]
}

if [ $# -ge 5 ]; then
  schema=$(upper $1)
  table=$(upper $2)
  column_to_split_by=$(upper $3)
  date_column=$(upper $4)
  date_value="$5"
else 
  echo
  echo "Usage: $(basename $0) schema table column-to-split-by date-column YYYY-MM-DD"
  echo
  echo "Imports all records where value of date-column is \$date_value from"
  echo "Oracle table \$schema.\$table as a Hive table partition."
  echo "Hadoop will split the import job based on the column-to-split-by."
  echo "* The table must have the columns specified as column-to-split-by and date-column."
  echo "* The column-to-split-by must be finer granularity than date-column, ideally unique."
  echo "* The date_value must be in YYYY-MM-DD format."
  echo "* If date_value is unspecified, the current date will be used."
  exit 1
fi

echo "schema = $schema"
echo "table = $table"
echo "column_to_split_by = $column_to_split_by"
echo "date_column = $date_column"
echo "date_value = $date_value"

# we have to drop the partition, because --hive-overwrite does not seem to do it
hive -e "use $schema; alter table $table drop if exists partition($date_column='$date_value');"

columns=$( \
sqoop eval \
--options-file /usr/local/etc/sqoop-options.txt \
--query "select column_name from all_tab_columns where table_name = '$table'" \
| tr -d " |" \
| grep -Ev "\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-|COLUMN_NAME|$date_column" \
| tr '\n' ',' \
| sed -e 's/\,$//'
)

query="select $columns from $schema.$table \
       where $date_column = to_date('$date_value', 'YYYY-MM-DD') \
       and \$CONDITIONS"

echo "query = $query"

sqoop import \
--options-file "/usr/local/etc/sqoop-options.txt" \
--query "$query" \
--split-by "$column_to_split_by" \
--target-dir "$schema.$table" \
--hive-import \
--hive-overwrite \
--hive-table "$schema.$table" \
--hive-partition-key "$date_column" \
--hive-partition-value "$date_value" \
--outdir $HOME/java

JDBC connection details

Put them into /usr/local/etc/sqoop-options.txt, in a format like this:

--connect
jdbc:oracle:thin:@hostname:port:hostname
--username
oracle_username
--password
oracle_password

Categories: bash, linux, mac os, virtualbox Tags:

Jenkins on minimalistic Debian Virtualbox VM (64bit)

November 23, 2011 Leave a comment

A Jenkins build server (LTS release) can now be easily installed on the minimalistic Debian VM:

  1. Download and install Virtualbox
  2. Download debian-stable-amd64-minimal.ova and import it into Virtualbox
  3. Start the “debian-stable-amd64-minimal” VM in Virtualbox
  4. If you are outside Nova Scotia, please review debian-stable-amd64-minimal.txt and adjust locale, timezone and Debian mirror based on your location
  5. Start an ssh session to localhost, port 1111 (using PuTTY, for example)
  6. Log in as user (default password is “user”)
  7. Issue “sudo install.sh jenkins” (default root password is “root”)
  8. Press enter for any questions during installation
  9. Open http://localhost:8888/ in a browser on the host OS for Jenkins web ui

You can go to “Manage Jenkins” – “Configure System” and see that JDK, Ant and Maven entries are already configured for you.

Both OpenJDK 6 and OpenJDK 7 are installed automatically for you in the VM. Please note that openjdk-6 is the Debian stable system default, while openjdk-7 is configured as the default for Jenkins build jobs.

Important: Make sure to change root and user passwords to something secure, as mentioned in debian-stable-amd64-minimal.txt.

Categories: debian, java, jenkins, virtualbox

Convert Virtualbox VDI for VMware Player

November 15, 2011 1 comment

Regarding the VM that I mentioned in my previous post – this way I also got it to work in VMware Player:

  • Install and start Virtualbox (https://www.virtualbox.org/wiki/Downloads)
  • Go to File – Import Appliance (and import the ova file mentioned in my last post)
  • Close Virtualbox
  • Use VBoxManage on the command line to clone the VDI to VMDK, roughly as described here (but do not try to uninstall guest additions):  http://scottlinux.com/2011/06/24/convert-vdi-to-vmdk-virtualbox-to-vmware/
  • Create a new empty Linux/Debian VM in VMware Player
  • Close VMware Player
  • Edit the new vmx file so that it points to the cloned vmdk
  • Then open the VM in VMware player again. Now it should boot into Debian.
Categories: debian, virtualbox

Minimalistic Debian VM (64bit)

November 10, 2011 Leave a comment

I uploaded a VM image to dev.doepner.net: It is a minimalistic Debian VM that can be used as a base for lean servers. It is in OVA format, exported from Virtualbox 4.1.6. Details are in the corresponding txt file.

You can add the VM to your Virtualbox using “File” – “Import Appliance …”. VMware should also work but might require some compatibility conversion of the ova file.

Recipes for specific Nexus, Jenkins, Wiki, Bugzilla setups will follow … I will publish those as scripts/instructions relative to the minimal base image, rather than maintaining several VMs.

Categories: debian, os, virtualbox

Virtualbox clipboard workaround

January 11, 2008 Leave a comment

With Virtualbox OSE 1.5.2, host Kubuntu 7.10 and guest Win XP I have intermittent clipboard problems: Garbled text, often even from one Linux app to another.

I tried upgrading to VirtualBox 1.5.4 non-free but had network problems with it.

Finally I came up with the following workaround. The idea is to write/read the clipboard using a temporary shared file. Prerequisites:

  • Install xclip in Kubuntu.
  • Put the cat, pclip and gclip commands in the PATH of the Windows system.
  • Set up a Samba share that both guest and host can read and write to.

In my setup the share is /share in the guest and mounted as network drive S: in the guest. The temporary file will be in /share/clipboard/.

Create an application shortcut in Kubuntu with the Application command

xclip -o > /share/clipboard/clip.txt

and drag it on the taskbar so that you can execute it with one click. To avoid annoying startup notifications disable them in the “Advanced Options” in the “Application” tab of the shortcut. Now you should be able to copy the content of the X clipboard to the temporary file with one click.

In the guest system (Windows) you do the same thing in reverse but with a keyboard shortcut: Put a file clip-in.bat into a safe location (where you won’t accidentally delete it) with the following one line of code:

cat S:\clipboard\clip.txt | gclip

Create a desktop shortcut pointing to clip-in.bat and define a keybard shortcut for it (under right-click, Properties..), e.g. ALT-CTRL-C.

Now you should be able to mark and paste from the host to the guest system in the following way:

  • Mark some text in the host (Kubuntu) system
  • Click the shortcut in the KDE taskbar
  • In the host system press CTR-ALT-C and then CTRL-V (to paste)
Categories: virtualbox

Windows in a box

January 8, 2008 Leave a comment

Until quite recently I did not use any operating system virtualization software. On my main PC I ran only Kubuntu and on my laptop I had a dual boot setup with the Windows XP installation that came pre-installed and Kubuntu.

VMware was not very appealing to me since it is proprietary software, not Open Source. A few weeks ago I gave VirtualBox a try. It is available as an Open Source Edition (OSE). There are packages included in the Ubuntu distribution so it was quite easy to install.

I must say that I am very impressed. The UI is intuitive, well documented and straightforward to use. I installed Windows XP Professional as a guest system and it worked without problems.

I had to realize, though, that “Shared Folders” are only supported in the Commercial Edition. That is not a problem since Folder Sharing can be done with Samba.

“Seamless mode” makes the experience even better: The windows from the Win XP desktop and the Kubuntu desktop appear as parts of one united desktop, with the taskbar and desktop background changing according to what window is currently active.

VirtualBox is developed by the German company InnoTek. Keep up the good work, guys!

Categories: debian, os, virtualbox