Tuesday, May 4, 2010

Evil ZVOL/ISCSI Tuning?

After a few days of VDI in service it was about time to start tuning the whole beast. The W7 guests show a rather poor performance when it came to disk IO. The virtual disks live on ISCSI targets provided by a X4540, so one would hope for decent performance. But the W7 guest hardly ever got more than 5MB/s from the virtual disks. So something needed some serious checking.
As a comparison I got close to 100MB/s when using NFS from the storage to the boxes running Virtualbox (VBox is running on X4450s with 64GB RAM, all NICs aggregated). So there was hope. Next I tried Solaris' ISCSI initiator and got transfer rates around 60MB/s. Both way better than the VBox' own ISCSI initiator in combination with whatever W7 does.
Digging around a little I found this blog post. So it seems like Windows tries to flush the cache pretty often. First thing I tried was to enable fast ack for incoming wirtes to the ISCSI target(s) with
iscsitadm modifiy admin -f enable
Luckily the server is backed by UPS power so I chose to ignore the warning that this might be dangerous. (Anyway there a no data on the client images, so worst case I can just clone the guest again if it goes wrong ;-))
Next thing I set the MaxRecvDataSegmentLength to 128MB, that should give the clients a bit more memory to dump their stuff faster:
iscsitadm modifiy target -m 128M
That made things a little better. I got nearly twice the throughput. Still way too slow.
So as a last resort I remember this post by Robert Milkowski about enabling write cache for zvols. As the X4540 is running Solaris 10 the Comstar option to ignore cache flushes doesn't exist yet (or more precise no comstar in Sol10). So mileks program to enable the write cache on zvols does help a lot.
I now get about 30MB/s for the W7 guest, which I consider decent enough for a virtual Windooze box ;-)
For all above always keep in mind, if you loose power, you may loose data.

Friday, April 16, 2010

Gone live

Heureka! The first bunch of SunRays went live yesterday. The are served by SunVDI, using VirtualBox for the VMs and Windows7 Enterprise as guest OS.
You may have guessed. Samba4 is running as DC.
The most annoying bit of the last day was the central Infoblox DHCP server. That thing is pretty stupid in terms of in what order it hands out DHCP options.
We found it to give the options in this order:
- based on host object
- entire network
- based on subnet
So since there already was another SunRay installation and we couldn't provide the required options based on the subnet the guys from central network management are now busy setting up the DHCP option for every DTU I have to install. Not nice, but works for now. I guess this will get funny when we move to another building in two years and the IP addresses of the servers change...

Saturday, April 10, 2010

Samba4 AD Domain Controller to serve Solaris and Windows

The installation of Samba4 as DC is almost finished. Despite some minor issues and a bit of work involved it was a rather pleasant and nice thing to do. I try to document the steps I had to do here as a little reference. Implementing Linux clients is still open, so for now this is very Solaris centric. But should be easy to reproduce on Linux.

Start off with the Samba4 how-to here. It gives all information needed to build and setup your DC. Once this is completed you are ready to serve Windows clients.
But I want to do a little bit more and use the data in Samba4's LDAP to serve the Unix boxes as well.

For wide parts I followed Scott Lowe's post on integrating Solaris 10 and Active Directory.
Since a few things differ when using Samba4 as AD I'll list the steps with short comments here again:

I skipped the part about installing the "Server for NIS" on the AD server. When I started off setting things up I didn't the schema use by AD and SFU at hand. But you only need that if you plan to manage you users with the MS tools (MMC and the likes). Since I tend to do that stuff from my Solaris box with a few scripts I found the presence of the posixAccount and shadowAccount objectclasses sufficient. As it turns out the need schema for the SFU stuff can be found here.

I created a test account with

net newuser test password

and set the required attributes with ldapmodify:

ldapmodify -h ldapserver -D adminsitrator@YOURDOMAIN
dn: cn=test,cn=users,dc=YOURDOMAIN
changetype: modify
add: objectclass
objectclass: posixaccount
-
add: objectclass
objectclass: shadowaccount
-
add: uidnumber
uidnumber: 1000
-
add: gidnumber
gidnumber: 100
-
add:unixhomedirectory
unixhomedirectory: /home/test
-
add: loginshell
loginshell: /bin/tcsh

This will be done with a litte script later, including creating a ZFS dataset for the user, finding the next free uidnumber and so on. For now this is just fine.

Well almost perfect. Unfortunately the MS AD schema used by Samba4 doesn't know anything about automount maps. So I had to add the necessary schema.
The schema definition is here. This creates an auto_master map with entries for /net and /home and the auto_home map with an entry for
* fileserver:/export/home/&
autmount.ldif can be added using ldapadd. The creation of the actual maps with ldapadd fails though. It gives a naming violation and I haven't found out why yet. Luckily Samba4 brings ldbmodify that can process LDIFs and write them into the database.
Point it to the ldb file for your domain. (Those files are in $SAMBAHOME/private/sam.ldb.d/ ):
ldbmodify -H $SAMBAHOME/private/sam.ldb.d/DC%3DYOURDOMAIN.ldb -U administrator autmount_maps.ldif
You may want to make a backup before ;-)

Next thing is to setup Kerberos. For that to work we need to create service principals in the Kerberos database for each host that we want to join.
Scott's post suggests to create an user account for each Solaris machine and to use ktpass.exe to generate a keytab file. Since I couldn't find a version of ktpass that would work on the Windows7 (virtual machines) I have here, I reverted to an alternative way. The easiest was to just use Samba3 and join the machines as domain members using a minimal smb.conf with
workgroup = DOMAIN
realm = REALM
security = ADS

When you have the smb.conf in place do the basic Kerberos setup on your Solaris client. Either use kclient to configure everything or copy the krb5.conf from your S4 DC. That file is created during the provision and can be found in $SAMBAHOME/private/krb5.conf. krb5.conf goes into /etc/krb5 on Solaris.
Test your Kerberos setup with
kinit administrator@REALM (mind the uppercase for the realm). That should exit without failure. You can check your ticket with klist.
If that worked you can now do a
net ads join -U administrator
to join the domain. Upon a successful join we can use
net keytab create
to write the principal key into a local file. The Samba3 that comes with Solaris10 writes the krb5.keytab in /etc. The Solaris Kerberos client expects that file to be in /etc/krb5 though. So just move it there.
Now we have Kerberos configured and the needed keytab in place.
Tell PAM to use Kerberos for authentication by modifying /etc/pam.conf to look like this:

other auth requisite pam_authtok_get.so.1
other auth required pam_dhkeys.so.1
other auth sufficient pam_krb5.so.1
other auth required pam_unix_cred.so.1
other auth required pam_unix_auth.so.1


The last missing piece is the LDAP configuration. For now I trimmed down the ldapclient line from Scott's post to;
ldapclient manual \
-a defaultSearchBase=dc=example,dc=com \
-a domainName=example.com \
-a defaultServerList=172.16.1.10 \
-a attributeMap=passwd:gecos=cn \
-a attributeMap=passwd:homedirectory=unixHomeDirectory \
-a serviceSearchDescriptor=passwd:cn=users,dc=example,dc=com \
-a serviceSearchDescriptor=group:cn=users,dc=example,dc=com

Problem is that this expects the users to have objectclass posixAccount and shadowAccount (no problem for me, since I set those in my scripts). The windows tools only set objectclass user (see the objectClassMap in Scott's post).
I tried with that mapping in place, but found that getent passwd fails to list LDAP users, getent passwd username works, finger fails, logins work. I assume there is something wrong with getent in Solaris but I need to investigate a litte more.
Without the objectclassmap in place everything works just fine.
One advise - ldapclient copies /etc/nsswitch.ldap to /etc/nsswitch.conf. So go and check that file, especially look at the hosts line. You sure want DNS in there... ;-)

Now users can login with their domain accounts and have a common password for both Windows and Solaris.
I need to play around with the mappings a little more and get a few more maps into LDAP (auth_attr, prof_attr and project at least). Also at the moment I cannot change the password from a Solaris client.
Let me know If you need more details on the bits of the setup, I'll try to help.

New gear

last week the new gear arrived. A little over a ton (1080kg including packaging) of various equipment was delivered. A few servers, racks, UPSs, SunRays, monitors... After everything was distributed to the final destination we fitted the servers into the cabinets and by 4p.m. my physical exercise for the next month was done ;-)
Over the next two days I installed Solaris 10 on the main servers, added the needed software (SamFS on the fileserver, Sun VDI and SunRay Server Software on the VDI servers).
The xVM servers will be running OpenSolaris and provide different flavours of Linux as compute farm to the users.

Directory services for all platforms will be provided with a Samba AD DC. That serves nicely for the Windows clients and also gives LDAP and Kerberos for the Unix clients. The main goal is to have one central account management and a common password across all platforms without the hassle of synchronising between the worlds.

Monday, March 22, 2010

Trying Samba4

It's been a while since my last post. I'm still alive and running (Open)Solaris whenever I have the option to choose.
A lot of exciting things happened in the past months and now a new challenge arises.
We start a new research branch and I got to plan (and soon to install) most of the infrastructure. It involves quite a lot of things that are pretty exciting by themselves (SamFs, VDI, xVM/Xen).
The one thing that kept me interested the past few days was the integration of the planned Windows7 guest for the virtual desktop infrastructure. As I try to avoid too much contact with the M$ world I am looking for a way to do it utilizing Unix power. As Microsoft removed support for NT4 style domains and group policies that became quite a challenge. Right now I'm trying out Samba4 as Active Directory replacement. So far it looks rather promising. A few questions arose already but with the help of the folks on #samba-technical those were cleared quite easily.
As I want to do some fancy stuff (like the shadow volume/previous version stuff in combination with zfs snapshots) it will become a hybrid setup with Samba4 as AD DC and Samba3 doing the fileserver part.
I tried to join the OpenSolaris CIFS service to the Samba4 domain but failed so far (that's being investigated by the samba folks). But since the storage server is going to be Solaris10 anyway that isn't a top priority at the moment.

I'll post more when I do the interesting parts of the planned installation.

Thursday, May 7, 2009

Crossbow, Xen & Linux fun

Recently I had to re-install an OpenSuse on a box a co-worker had installed about two years ago. After his native Linux installation had managed to shred a little data (roughly 1TB loss) on an external raid, I decided to put something decent on the box ;-)
Ok to be fair it was kind of a bad combination that the raid box had a failure on one of the ports and Linux kept writing happily to the drives despite the errors it must have encountered (at least there where log entries about write errors). Seems OpenSuse's default settings for Ext3 are a really *BAD* idea. It is set to "on error continue". Looks like a safe way to lose data.
Luckily not a real loss in data, raw data are in a safe place, but some guys have to re-run their compute jobs (and hopefully keep in mind to make a copy to our HSM once they're finished).

Well after all it is a Sun Box (x4600), so it is now running SXCE (b112 at the moment). ZFS handling the attached disks (now the raid box is down to simply a JBOD) just makes me sleep better ;-)
The Suse installation is now running as a xVM pvm domU. Solaris doing the disk I/O and sharing the data to the domU, Suse running Matlab.
domU installation was a bit trickier than the Fedora 8 install I did a while ago for a few other domUs. It took me a while to find the right combination to make the Suse domU boot off the virtual disk.
After that I decided to try out the crossbow stuff. Especially the etherstubs looked interesting. So two dladm commands later I had an etherstub (kind of a virtual network switch within Solaris) and a virtual nic setup. Plumb the vnic, set an IP address, ping the "switch" works just fine. So next thing I attached another nic to my domU and brigded it over the etherstub. The new vnic gets created by xVM and shows up in the domU.
Assign an IP address, send a ping and wait... and wait... and wait... I did expect an answer. Well, seems like something went wrong.
A little snooping on the Solaris instance and I see packets coming and going. They just never make it into the Linux world.
The only thing that made me wonder was the different MTU on Solaris and Linux. Solaris quite happily set the MTU to 9000, while Linux had it set to 1500. After ditching the vnics, setting the MTU on the etherstub to 1500 and recreating the vnic with MTU 1500 ping finally got an answer.
Not sure if this is a Solaris problem, a feature or something completely different. Though I tend to blame Linux for not supporting jumbo frames ;-)
Well maybe it does. But at least not for the xen-provided virtual network card.
So if you come to a point where you want to make a Linux domU talk to a Solaris box over an etherstub and lose the packets you may want to check the MTU setting.

Monday, March 16, 2009

Thanks Igor (or how to get VT-x on a Sony notebook)

I already ranted about Sony disabling VT-x in all notebooks (except for the BZ line). They also don't want to release a BIOS were one could actually change settings other than boot device order. There have been tutorials about patching the NVRAM on models with a phoenix bios for quite some time. Unfortunately mine uses an UEFI based AMI Aptio bios so until now I was out of luck.
Checking for news on this issue again I came across a thread in an notebook related forum. That guy - Igor (an Intel Black Belt btw) - did an awesome job of reverse engineering on the bios structure and came up with a program that utilizes AMI's afudos to first dump the current bios, parse the resulting file for the pattern which disable VT-x, patch that to something more desirable ;-) and use afudos again to write back the patch bios image.
The challenge was to get a DOS bootdisk onto the usb pendrive, the rest worked just like a charm.
You can also go to his website directly and visit the download section.
Just phantastic work. For me it's now off to play xVM with hvms on the notebook!
Thanks a lot Igor!