The SUN side of life

Evil ZVOL/ISCSI Tuning?

2010-05-04T23:21:00.003+02:00

After a few days of VDI in service it was about time to start tuning the whole beast. The W7 guests show a rather poor performance when it came to disk IO. The virtual disks live on ISCSI targets provided by a X4540, so one would hope for decent performance. But the W7 guest hardly ever got more than 5MB/s from the virtual disks. So something needed some serious checking.
As a comparison I got close to 100MB/s when using NFS from the storage to the boxes running Virtualbox (VBox is running on X4450s with 64GB RAM, all NICs aggregated). So there was hope. Next I tried Solaris' ISCSI initiator and got transfer rates around 60MB/s. Both way better than the VBox' own ISCSI initiator in combination with whatever W7 does.
Digging around a little I found this blog post. So it seems like Windows tries to flush the cache pretty often. First thing I tried was to enable fast ack for incoming wirtes to the ISCSI target(s) with
iscsitadm modifiy admin -f enable
Luckily the server is backed by UPS power so I chose to ignore the warning that this might be dangerous. (Anyway there a no data on the client images, so worst case I can just clone the guest again if it goes wrong ;-))
Next thing I set the MaxRecvDataSegmentLength to 128MB, that should give the clients a bit more memory to dump their stuff faster:
iscsitadm modifiy target -m 128M
That made things a little better. I got nearly twice the throughput. Still way too slow.
So as a last resort I remember this post by Robert Milkowski about enabling write cache for zvols. As the X4540 is running Solaris 10 the Comstar option to ignore cache flushes doesn't exist yet (or more precise no comstar in Sol10). So mileks program to enable the write cache on zvols does help a lot.
I now get about 30MB/s for the W7 guest, which I consider decent enough for a virtual Windooze box ;-)
For all above always keep in mind, if you loose power, you may loose data.

Gone live

2010-04-16T12:40:00.002+02:00

Heureka! The first bunch of SunRays went live yesterday. The are served by SunVDI, using VirtualBox for the VMs and Windows7 Enterprise as guest OS.
You may have guessed. Samba4 is running as DC.
The most annoying bit of the last day was the central Infoblox DHCP server. That thing is pretty stupid in terms of in what order it hands out DHCP options.
We found it to give the options in this order:
- based on host object
- entire network
- based on subnet
So since there already was another SunRay installation and we couldn't provide the required options based on the subnet the guys from central network management are now busy setting up the DHCP option for every DTU I have to install. Not nice, but works for now. I guess this will get funny when we move to another building in two years and the IP addresses of the servers change...

Samba4 AD Domain Controller to serve Solaris and Windows

2010-04-10T23:26:00.008+02:00

The installation of Samba4 as DC is almost finished. Despite some minor issues and a bit of work involved it was a rather pleasant and nice thing to do. I try to document the steps I had to do here as a little reference. Implementing Linux clients is still open, so for now this is very Solaris centric. But should be easy to reproduce on Linux.

Start off with the Samba4 how-to here. It gives all information needed to build and setup your DC. Once this is completed you are ready to serve Windows clients.
But I want to do a little bit more and use the data in Samba4's LDAP to serve the Unix boxes as well.

For wide parts I followed Scott Lowe's post on integrating Solaris 10 and Active Directory.
Since a few things differ when using Samba4 as AD I'll list the steps with short comments here again:

I skipped the part about installing the "Server for NIS" on the AD server. When I started off setting things up I didn't the schema use by AD and SFU at hand. But you only need that if you plan to manage you users with the MS tools (MMC and the likes). Since I tend to do that stuff from my Solaris box with a few scripts I found the presence of the posixAccount and shadowAccount objectclasses sufficient. As it turns out the need schema for the SFU stuff can be found here.

I created a test account with

net newuser test password

and set the required attributes with ldapmodify:

ldapmodify -h ldapserver -D adminsitrator@YOURDOMAIN
dn: cn=test,cn=users,dc=YOURDOMAIN
changetype: modify
add: objectclass
objectclass: posixaccount
-
add: objectclass
objectclass: shadowaccount
-
add: uidnumber
uidnumber: 1000
-
add: gidnumber
gidnumber: 100
-
add:unixhomedirectory
unixhomedirectory: /home/test
-
add: loginshell
loginshell: /bin/tcsh

This will be done with a litte script later, including creating a ZFS dataset for the user, finding the next free uidnumber and so on. For now this is just fine.

Well almost perfect. Unfortunately the MS AD schema used by Samba4 doesn't know anything about automount maps. So I had to add the necessary schema.
The schema definition is here. This creates an auto_master map with entries for /net and /home and the auto_home map with an entry for
* fileserver:/export/home/&
autmount.ldif can be added using ldapadd. The creation of the actual maps with ldapadd fails though. It gives a naming violation and I haven't found out why yet. Luckily Samba4 brings ldbmodify that can process LDIFs and write them into the database.
Point it to the ldb file for your domain. (Those files are in $SAMBAHOME/private/sam.ldb.d/ ):
ldbmodify -H $SAMBAHOME/private/sam.ldb.d/DC%3DYOURDOMAIN.ldb -U administrator autmount_maps.ldif
You may want to make a backup before ;-)

Next thing is to setup Kerberos. For that to work we need to create service principals in the Kerberos database for each host that we want to join.
Scott's post suggests to create an user account for each Solaris machine and to use ktpass.exe to generate a keytab file. Since I couldn't find a version of ktpass that would work on the Windows7 (virtual machines) I have here, I reverted to an alternative way. The easiest was to just use Samba3 and join the machines as domain members using a minimal smb.conf with
workgroup = DOMAIN
realm = REALM
security = ADS

When you have the smb.conf in place do the basic Kerberos setup on your Solaris client. Either use kclient to configure everything or copy the krb5.conf from your S4 DC. That file is created during the provision and can be found in $SAMBAHOME/private/krb5.conf. krb5.conf goes into /etc/krb5 on Solaris.
Test your Kerberos setup with
kinit administrator@REALM (mind the uppercase for the realm). That should exit without failure. You can check your ticket with klist.
If that worked you can now do a
net ads join -U administrator
to join the domain. Upon a successful join we can use
net keytab create
to write the principal key into a local file. The Samba3 that comes with Solaris10 writes the krb5.keytab in /etc. The Solaris Kerberos client expects that file to be in /etc/krb5 though. So just move it there.
Now we have Kerberos configured and the needed keytab in place.
Tell PAM to use Kerberos for authentication by modifying /etc/pam.conf to look like this:

other auth requisite pam_authtok_get.so.1
other auth required pam_dhkeys.so.1
other auth sufficient pam_krb5.so.1
other auth required pam_unix_cred.so.1
other auth required pam_unix_auth.so.1

The last missing piece is the LDAP configuration. For now I trimmed down the ldapclient line from Scott's post to;
ldapclient manual \
-a defaultSearchBase=dc=example,dc=com \
-a domainName=example.com \
-a defaultServerList=172.16.1.10 \
-a attributeMap=passwd:gecos=cn \
-a attributeMap=passwd:homedirectory=unixHomeDirectory \
-a serviceSearchDescriptor=passwd:cn=users,dc=example,dc=com \
-a serviceSearchDescriptor=group:cn=users,dc=example,dc=com

Problem is that this expects the users to have objectclass posixAccount and shadowAccount (no problem for me, since I set those in my scripts). The windows tools only set objectclass user (see the objectClassMap in Scott's post).
I tried with that mapping in place, but found that getent passwd fails to list LDAP users, getent passwd username works, finger fails, logins work. I assume there is something wrong with getent in Solaris but I need to investigate a litte more.
Without the objectclassmap in place everything works just fine.
One advise - ldapclient copies /etc/nsswitch.ldap to /etc/nsswitch.conf. So go and check that file, especially look at the hosts line. You sure want DNS in there... ;-)

Now users can login with their domain accounts and have a common password for both Windows and Solaris.
I need to play around with the mappings a little more and get a few more maps into LDAP (auth_attr, prof_attr and project at least). Also at the moment I cannot change the password from a Solaris client.
Let me know If you need more details on the bits of the setup, I'll try to help.

New gear

2010-04-10T22:50:00.002+02:00

last week the new gear arrived. A little over a ton (1080kg including packaging) of various equipment was delivered. A few servers, racks, UPSs, SunRays, monitors... After everything was distributed to the final destination we fitted the servers into the cabinets and by 4p.m. my physical exercise for the next month was done ;-)
Over the next two days I installed Solaris 10 on the main servers, added the needed software (SamFS on the fileserver, Sun VDI and SunRay Server Software on the VDI servers).
The xVM servers will be running OpenSolaris and provide different flavours of Linux as compute farm to the users.

Directory services for all platforms will be provided with a Samba AD DC. That serves nicely for the Windows clients and also gives LDAP and Kerberos for the Unix clients. The main goal is to have one central account management and a common password across all platforms without the hassle of synchronising between the worlds.

Trying Samba4

2010-03-22T13:53:00.002+01:00

It's been a while since my last post. I'm still alive and running (Open)Solaris whenever I have the option to choose.
A lot of exciting things happened in the past months and now a new challenge arises.
We start a new research branch and I got to plan (and soon to install) most of the infrastructure. It involves quite a lot of things that are pretty exciting by themselves (SamFs, VDI, xVM/Xen).
The one thing that kept me interested the past few days was the integration of the planned Windows7 guest for the virtual desktop infrastructure. As I try to avoid too much contact with the M$ world I am looking for a way to do it utilizing Unix power. As Microsoft removed support for NT4 style domains and group policies that became quite a challenge. Right now I'm trying out Samba4 as Active Directory replacement. So far it looks rather promising. A few questions arose already but with the help of the folks on #samba-technical those were cleared quite easily.
As I want to do some fancy stuff (like the shadow volume/previous version stuff in combination with zfs snapshots) it will become a hybrid setup with Samba4 as AD DC and Samba3 doing the fileserver part.
I tried to join the OpenSolaris CIFS service to the Samba4 domain but failed so far (that's being investigated by the samba folks). But since the storage server is going to be Solaris10 anyway that isn't a top priority at the moment.

I'll post more when I do the interesting parts of the planned installation.

Crossbow, Xen & Linux fun

2009-05-07T23:51:00.002+02:00

Recently I had to re-install an OpenSuse on a box a co-worker had installed about two years ago. After his native Linux installation had managed to shred a little data (roughly 1TB loss) on an external raid, I decided to put something decent on the box ;-)
Ok to be fair it was kind of a bad combination that the raid box had a failure on one of the ports and Linux kept writing happily to the drives despite the errors it must have encountered (at least there where log entries about write errors). Seems OpenSuse's default settings for Ext3 are a really *BAD* idea. It is set to "on error continue". Looks like a safe way to lose data.
Luckily not a real loss in data, raw data are in a safe place, but some guys have to re-run their compute jobs (and hopefully keep in mind to make a copy to our HSM once they're finished).

Well after all it is a Sun Box (x4600), so it is now running SXCE (b112 at the moment). ZFS handling the attached disks (now the raid box is down to simply a JBOD) just makes me sleep better ;-)
The Suse installation is now running as a xVM pvm domU. Solaris doing the disk I/O and sharing the data to the domU, Suse running Matlab.
domU installation was a bit trickier than the Fedora 8 install I did a while ago for a few other domUs. It took me a while to find the right combination to make the Suse domU boot off the virtual disk.
After that I decided to try out the crossbow stuff. Especially the etherstubs looked interesting. So two dladm commands later I had an etherstub (kind of a virtual network switch within Solaris) and a virtual nic setup. Plumb the vnic, set an IP address, ping the "switch" works just fine. So next thing I attached another nic to my domU and brigded it over the etherstub. The new vnic gets created by xVM and shows up in the domU.
Assign an IP address, send a ping and wait... and wait... and wait... I did expect an answer. Well, seems like something went wrong.
A little snooping on the Solaris instance and I see packets coming and going. They just never make it into the Linux world.
The only thing that made me wonder was the different MTU on Solaris and Linux. Solaris quite happily set the MTU to 9000, while Linux had it set to 1500. After ditching the vnics, setting the MTU on the etherstub to 1500 and recreating the vnic with MTU 1500 ping finally got an answer.
Not sure if this is a Solaris problem, a feature or something completely different. Though I tend to blame Linux for not supporting jumbo frames ;-)
Well maybe it does. But at least not for the xen-provided virtual network card.
So if you come to a point where you want to make a Linux domU talk to a Solaris box over an etherstub and lose the packets you may want to check the MTU setting.

Thanks Igor (or how to get VT-x on a Sony notebook)

2009-03-16T18:51:00.002+01:00

I already ranted about Sony disabling VT-x in all notebooks (except for the BZ line). They also don't want to release a BIOS were one could actually change settings other than boot device order. There have been tutorials about patching the NVRAM on models with a phoenix bios for quite some time. Unfortunately mine uses an UEFI based AMI Aptio bios so until now I was out of luck.
Checking for news on this issue again I came across a thread in an notebook related forum. That guy - Igor (an Intel Black Belt btw) - did an awesome job of reverse engineering on the bios structure and came up with a program that utilizes AMI's afudos to first dump the current bios, parse the resulting file for the pattern which disable VT-x, patch that to something more desirable ;-) and use afudos again to write back the patch bios image.
The challenge was to get a DOS bootdisk onto the usb pendrive, the rest worked just like a charm.
You can also go to his website directly and visit the download section.
Just phantastic work. For me it's now off to play xVM with hvms on the notebook!
Thanks a lot Igor!

LCD panel backlight control vs Sony Vaio AW11Z part 2

2009-01-06T14:24:00.002+01:00

Heureka!
Finished. The port of NVClock/SmartDimmer to OpenSolaris is complete. It works nicely on my notebook (and I only crashed it once on the way *g*)
A little reading, browsing source code and learning how to access PCI stuff was... ummm... interesting :-) Thankfully there exists a nice thing called libpciaccess which makes life a lot easier.
NVClock now has a Solaris backend that gives me the ability to control the backlight on my notebook.
For now the source code can be found here
Have a go and try it if you have a notebook with an NVidia graphics card. Please report if you have any issues with the Solaris specific part. Any comments are welcome. But please don't sue me if it blows your box into pieces ;-)
Mind you - smartdimmer and nvclock need to be run as priviledged user to set the registers on the graphics card.

I hope the author of NVClock accepts my patch so OpenSolaris gets supported out of the box.

The really amazing thing is how much battery time the backlight actually sucks. With the 'big' battery runtime went up by 1hour just by dimming from 100% down to 15%. PowerTop reports 40Watts vs 26.5Watts. That is just impressive!

LCD panel backlight control vs Sony Vaio AW11Z part 1

2009-01-05T21:02:00.002+01:00

As mentioned in an earlier post, the backlight control on the Vaio doesn't work out of the box with OpenSolaris. A lot of googling and reading revealed that the backlight isn't controlled via ACPI but via the NVidia graphics card. A lot of more googling and reading lead me to a nifty little tool called nvclock / smartdimmer. It features a number of options to tweak some settings. One of the options is the brightness level of the panel backlight. Now there was just one problem. OpenSolaris isn't (yet) supported.
A little more reading and digging around the code...
Heureka!
I now have a first version which can dim the backlight of the panel. Still a lot of things hardcoded (only one graphics card, fixed base mem address). Still... it's the first step.

Battery time on the default battery went up by 35mins by dimming from 100% down to 15%.

Off to part two... and make the beast auto-detect the graphics card, fetch the base mem address etc.

As soon as I'm done with that I'll post it here and on the relevant list.

Multimedia keys and Songbird on Opensolaris

2008-12-23T18:17:00.002+01:00

As I posted earlier the Multimedia keys on the Vaio work quite nicely with Gnome apps like Rhythmbox. It's just that I prefer songbird. And while moving a server to a new pair of disks a little music is more than welcome. So let's rock and roll.
Only thing the multimedia keys don't work. Bugger. Songbird seems to ignore the gnome keybindings. The MMKeys add-on only seems to work on Linux. I didn't have the nerve to look into it yet. But there seems no easy way to bind keys to actions in Songbird.
A little annoying.

Moving box to S10U6 with ZFS root

2008-12-23T17:57:00.007+01:00

So what do you do just before christmas? Well I started updating a T1000 from Solaris 10U4 to U6 with ZFS root. As this box serves a patient administration system in a hospital there is not much room for longer down times. Well except around official holidays.
The T1000 is a nice box to run databases on boot it has one problem - not a real problem more like a hurdle in terms of updating. It came with just a single hard disk when we got it. It has a StorEdge 3510 attached though.
Well sounds quite easy. Setup LiveUpgrade and off you go. In theory yes. Creation of the new boot environment worked nicely. Upgrade went fine too. The damn thing just wouldn't boot off the external disk stack. Bugger.
As there are now double disks in a single bracket available (well for some time now) I ordered a pair of disks to move the installation from one disk to the new pair.
Now the dirty part begins ;-)
As I am too lazy to replicate the installation from scratch (I could do it but it would require more time than I have at hand) I chose the adventurous route ;-)

First of all dump the current installation to the external disk stack.
Swap the disks.
Boot off the net (the T1000 doesn't have a DVD drive and I didn't want to try an USB drive)
Slice one of the new disks as desired
Restore the dump from the external disks
Reboot
Be done.

Well not quite.

Booting fails as I forgot to run installboot on the new disk, so boot net again
Install the boot block on the new drive with installboot
Reboot
Done

Works nicely. Including all copying (roughly 30GB local data), net booting etc the box was running again in less than 2 hours.

What's next?
Create a new boot environment on the second disk. As we are still running S10U4 it has to be UFS again. Right now it is still populating the new BE with data from the first disk. (Yes, we are running live at this point.) Luckily the new disks (SATA vs SAS) are much faster, so this is running faster than expected.
Next thing will be the update to S10U6 followed be reboot into the new BE running S10U6. That gives us ZFS root. Once this is done and working I'll delete the first BE and create a new boot environment on a ZFS root pool. And one last reboot will bring us to where we want to be. Mirror root pool with much easier upgrade procedures next time (and just a single reboot needed).

Update
The upgrade to S10U6 took a bit over 2 hours. No problems here. Try to boot the new BE.
With the shutdown running I knew there was something missing.
Exactly.
The luactivate. Things like this happen when you drive home (90min commute) to finish this from home. Oh well!
Anyway, activated and booted the new BE.
Zpool created (remember it has to be a slice in order to be bootable! - Yes I did read the man page on that point ;-))
So now we are copying the S10U6 UFS BE to the final destination on a zpool.

Update 2
Everything made it to the new BE on the zpool. It boots. This was work for today then.

Merry Christmas everyone!

More support fun

2008-12-17T15:52:00.002+01:00

Update on the Vaio Support story. 2 days after logging two further calls (both regarding LCD panel brightness) an answer two both of them.
1. How to dim the panel with no OS loaded?
-> We do not support that.

2. Why isn't the panel brightness persistent across reboots (even with vista)?
-> Have you tried the latest update for your graphics card?

When trying to answer via the support web site it hit me twice now that I was typing for too long and got logged out. Of course back in the browser didn't work and I had to type again...

Sony function keys and OpenSolaris

2008-12-17T15:38:00.004+01:00

The sound vol up/down/mute buttons on the Vaio nearly worked out of the box, all I had to do was to map them in the Gnome Keyboard Shortcuts preferences.
Running
# dtrace -n '::KeyboardConvertScan:entry {printf("%x", arg1)}'
I found the keycodes for some other keys (Play/Pause, Stop, Next, Prev).
Funny that the Keyboard Shortcuts preferences dialog didn't see the keycodes when I tried to map them directly. So I had a look at acerkb from frkit which I had been using for ages on my Acer Ferrari. Seems like I was lucky that acerkb mapped the keycodes for the sound control already and it therefore just worked. So I replaced the keycodes from the Ferrari with the ones from the Vaio. And voila I was able to map them to the according functions in Gnome.

I wonder how I can find out which keycode sequences the keys generate that I couldn't catch with above dtrace command. There are 4 more Sony keys (S1, S2, AV mode, Eject) that I couldn't grab. Also the keys for LCD brightness, suspend and internal/external display didn't generate a key code. So there is more research needed.

How much support to expect

2008-12-17T14:14:00.003+01:00

So after playing around with the Vaio and logging a service call about the numlock issue I found that no matter what I did the notebook would only charge the battery to 80%. Since I had ordered a high capacity battery as well I put that one into the notebook to find the exact same behaviour. Charged to 80% and off. Both Vista an Opensolaris reported the notebook was running on mains, battery connected, charged to 80% neither charging or discharging.
Time for another service call.
By now it was day 4 (in words four) after I logged the call about numlock.
And finally some kind soul from Sony's Vaio support answered to the numlock question. Indeed there is no such option in the BIOS (that much I knew already). There also is no way to turn off numlock at bootup using any other tool (Vaio Control Center may be). No BIOS update available. So here I am stuck with a turned on numlock on every boot and it takes Sony's support 4 days to find that out. Great work!

"Only" two days after logging the call on the battery question Vaio Support reacted and asked me a lot of questions. Yes this is the original power supply and batteries that Sony shipped. Yes my power outlet works just fine (why would the notebook run on mains otherwise?). Yes the power supply gets warm after being plugged in for a while. I tried both batteries as well. The guy then suggested a nice list of things to try (I was waiting for "dance around naked" until the end...) No real solution suggested though as I already had all cables removed and reinserted etc.
Funny I had a look around in the "Vaio Control Center" that comes installed with Vista. There is an option to extend battery life by only charging the battery to 80%. This was of course enabled. Switched off I have fully charged batteries.
And of course tech support didn't come up with the thing to check. But it has been only two days on this call.
Let's see how my other calls go. 2 more open for 2 days now.

Welcome blog, welcome new toy

2008-12-12T10:41:00.001+01:00

Welcome to my blog. I'll mainly post some stuff about (Open)Solaris und a little bit of this and that now and then.

So short before christmas, right in time a new toy arrived. A Sony Vaio AW11Z/B. Quite a nice bit of gear, though a little bit too big. But the AR series models I was going to get wasn't available any more. Since I didn't want anything with less than 1650x1080p display resolution and choise of brand is limited I "had" to go for the AW model.
Unfortunately it comes with Windows Vista preinstalled. That we're about to solve pretty soon :-)
First of all wait a bit until the preloaded Vista is ready to welcome you. Took like 15minutes or so before I was logged in. The notebook comes with a nice collection of software (Adobe Acrobat Distiller, Photoshop Elements etc.). We also might need Vista to watch BluRay discs. So I'm going for a dual boot.
Before we start lets create the recommended recovery dvds (yes there's 3! recovery dvds). Took well over an hour to create these. By the time I was this far I had lunch and my boss was there to take a look at the new toy.
The AW11Z/B comes with two 320GB harddisks. Those will make a nice ZFS mirrored pool. I used Vista's own hard disk and partition manager to shrink the original C: partition down to about 50GB on the first drive. I didn't touch the recovery partition (you never know when you might need it ;-)) That gave me roughly 230GB for the Solaris installation.
So now it's time to boot the Solaris installation dvd (I used SXCE build 103). Choose the text mode install and went for a ZFS root installation. I allocated 30GB to the root pool, the rest will go into homes and stuff.
About 40mins later it's time for the first "real" boot into Solaris.
Everything works just as expected. The first nice surprise is that Wifi works out of the box. (It has a Intel WiFi Link 5100).
For the Marvell Yukon 88E8055 we grab Masayuki Murayama's myk driver and are ready to go on the wired NIC too.
Sound works out of the box as well. Nivida graphics just works too.
Even the built-in webcam just works. All in all a very pleasant and easy thing to do.
Next thing is to create a solaris partition on the second harddisk, create two slices, attach on to the root pool and create a mirrored pool for all the data.
There is also a driver for the SD-Card reader which seems to work (it attached fine). I'll have to test that when I have an SD-Card.

Now everything is ready to put some data on it and install all needed software.

Though a few things remain open at the moment.
Suspend to ram seems to suspend the notebook ok but didn't wake up properly again. I'll have to check that.
The screensaver should switch off the backlight of the display after a while, but didn't.
The key for display brightness don't work so battery runtime is not where I want it to be yet.
The bios is really crippled. All one can change is the time and boot device order. I couldn't even switch off the bootup numlock. That really annoys me and I logged a service request with Sony. So far their support hasn't reacted at all within a day.
When I logged into Opensolaris (I use GDM as display manager) it didn't set the permissions on the sound and webcam device, so I had to fix that manually to have those working as non-root user.

Despite those issues great work from everyone who contributed to (Open)Solaris!