Tuesday, May 4, 2010

Evil ZVOL/ISCSI Tuning?

After a few days of VDI in service it was about time to start tuning the whole beast. The W7 guests show a rather poor performance when it came to disk IO. The virtual disks live on ISCSI targets provided by a X4540, so one would hope for decent performance. But the W7 guest hardly ever got more than 5MB/s from the virtual disks. So something needed some serious checking.
As a comparison I got close to 100MB/s when using NFS from the storage to the boxes running Virtualbox (VBox is running on X4450s with 64GB RAM, all NICs aggregated). So there was hope. Next I tried Solaris' ISCSI initiator and got transfer rates around 60MB/s. Both way better than the VBox' own ISCSI initiator in combination with whatever W7 does.
Digging around a little I found this blog post. So it seems like Windows tries to flush the cache pretty often. First thing I tried was to enable fast ack for incoming wirtes to the ISCSI target(s) with
iscsitadm modifiy admin -f enable
Luckily the server is backed by UPS power so I chose to ignore the warning that this might be dangerous. (Anyway there a no data on the client images, so worst case I can just clone the guest again if it goes wrong ;-))
Next thing I set the MaxRecvDataSegmentLength to 128MB, that should give the clients a bit more memory to dump their stuff faster:
iscsitadm modifiy target -m 128M
That made things a little better. I got nearly twice the throughput. Still way too slow.
So as a last resort I remember this post by Robert Milkowski about enabling write cache for zvols. As the X4540 is running Solaris 10 the Comstar option to ignore cache flushes doesn't exist yet (or more precise no comstar in Sol10). So mileks program to enable the write cache on zvols does help a lot.
I now get about 30MB/s for the W7 guest, which I consider decent enough for a virtual Windooze box ;-)
For all above always keep in mind, if you loose power, you may loose data.

Friday, April 16, 2010

Gone live

Heureka! The first bunch of SunRays went live yesterday. The are served by SunVDI, using VirtualBox for the VMs and Windows7 Enterprise as guest OS.
You may have guessed. Samba4 is running as DC.
The most annoying bit of the last day was the central Infoblox DHCP server. That thing is pretty stupid in terms of in what order it hands out DHCP options.
We found it to give the options in this order:
- based on host object
- entire network
- based on subnet
So since there already was another SunRay installation and we couldn't provide the required options based on the subnet the guys from central network management are now busy setting up the DHCP option for every DTU I have to install. Not nice, but works for now. I guess this will get funny when we move to another building in two years and the IP addresses of the servers change...

Saturday, April 10, 2010

Samba4 AD Domain Controller to serve Solaris and Windows

The installation of Samba4 as DC is almost finished. Despite some minor issues and a bit of work involved it was a rather pleasant and nice thing to do. I try to document the steps I had to do here as a little reference. Implementing Linux clients is still open, so for now this is very Solaris centric. But should be easy to reproduce on Linux.

Start off with the Samba4 how-to here. It gives all information needed to build and setup your DC. Once this is completed you are ready to serve Windows clients.
But I want to do a little bit more and use the data in Samba4's LDAP to serve the Unix boxes as well.

For wide parts I followed Scott Lowe's post on integrating Solaris 10 and Active Directory.
Since a few things differ when using Samba4 as AD I'll list the steps with short comments here again:

I skipped the part about installing the "Server for NIS" on the AD server. When I started off setting things up I didn't the schema use by AD and SFU at hand. But you only need that if you plan to manage you users with the MS tools (MMC and the likes). Since I tend to do that stuff from my Solaris box with a few scripts I found the presence of the posixAccount and shadowAccount objectclasses sufficient. As it turns out the need schema for the SFU stuff can be found here.

I created a test account with

net newuser test password

and set the required attributes with ldapmodify:

ldapmodify -h ldapserver -D adminsitrator@YOURDOMAIN
dn: cn=test,cn=users,dc=YOURDOMAIN
changetype: modify
add: objectclass
objectclass: posixaccount
-
add: objectclass
objectclass: shadowaccount
-
add: uidnumber
uidnumber: 1000
-
add: gidnumber
gidnumber: 100
-
add:unixhomedirectory
unixhomedirectory: /home/test
-
add: loginshell
loginshell: /bin/tcsh

This will be done with a litte script later, including creating a ZFS dataset for the user, finding the next free uidnumber and so on. For now this is just fine.

Well almost perfect. Unfortunately the MS AD schema used by Samba4 doesn't know anything about automount maps. So I had to add the necessary schema.
The schema definition is here. This creates an auto_master map with entries for /net and /home and the auto_home map with an entry for
* fileserver:/export/home/&
autmount.ldif can be added using ldapadd. The creation of the actual maps with ldapadd fails though. It gives a naming violation and I haven't found out why yet. Luckily Samba4 brings ldbmodify that can process LDIFs and write them into the database.
Point it to the ldb file for your domain. (Those files are in $SAMBAHOME/private/sam.ldb.d/ ):
ldbmodify -H $SAMBAHOME/private/sam.ldb.d/DC%3DYOURDOMAIN.ldb -U administrator autmount_maps.ldif
You may want to make a backup before ;-)

Next thing is to setup Kerberos. For that to work we need to create service principals in the Kerberos database for each host that we want to join.
Scott's post suggests to create an user account for each Solaris machine and to use ktpass.exe to generate a keytab file. Since I couldn't find a version of ktpass that would work on the Windows7 (virtual machines) I have here, I reverted to an alternative way. The easiest was to just use Samba3 and join the machines as domain members using a minimal smb.conf with
workgroup = DOMAIN
realm = REALM
security = ADS

When you have the smb.conf in place do the basic Kerberos setup on your Solaris client. Either use kclient to configure everything or copy the krb5.conf from your S4 DC. That file is created during the provision and can be found in $SAMBAHOME/private/krb5.conf. krb5.conf goes into /etc/krb5 on Solaris.
Test your Kerberos setup with
kinit administrator@REALM (mind the uppercase for the realm). That should exit without failure. You can check your ticket with klist.
If that worked you can now do a
net ads join -U administrator
to join the domain. Upon a successful join we can use
net keytab create
to write the principal key into a local file. The Samba3 that comes with Solaris10 writes the krb5.keytab in /etc. The Solaris Kerberos client expects that file to be in /etc/krb5 though. So just move it there.
Now we have Kerberos configured and the needed keytab in place.
Tell PAM to use Kerberos for authentication by modifying /etc/pam.conf to look like this:

other auth requisite pam_authtok_get.so.1
other auth required pam_dhkeys.so.1
other auth sufficient pam_krb5.so.1
other auth required pam_unix_cred.so.1
other auth required pam_unix_auth.so.1


The last missing piece is the LDAP configuration. For now I trimmed down the ldapclient line from Scott's post to;
ldapclient manual \
-a defaultSearchBase=dc=example,dc=com \
-a domainName=example.com \
-a defaultServerList=172.16.1.10 \
-a attributeMap=passwd:gecos=cn \
-a attributeMap=passwd:homedirectory=unixHomeDirectory \
-a serviceSearchDescriptor=passwd:cn=users,dc=example,dc=com \
-a serviceSearchDescriptor=group:cn=users,dc=example,dc=com

Problem is that this expects the users to have objectclass posixAccount and shadowAccount (no problem for me, since I set those in my scripts). The windows tools only set objectclass user (see the objectClassMap in Scott's post).
I tried with that mapping in place, but found that getent passwd fails to list LDAP users, getent passwd username works, finger fails, logins work. I assume there is something wrong with getent in Solaris but I need to investigate a litte more.
Without the objectclassmap in place everything works just fine.
One advise - ldapclient copies /etc/nsswitch.ldap to /etc/nsswitch.conf. So go and check that file, especially look at the hosts line. You sure want DNS in there... ;-)

Now users can login with their domain accounts and have a common password for both Windows and Solaris.
I need to play around with the mappings a little more and get a few more maps into LDAP (auth_attr, prof_attr and project at least). Also at the moment I cannot change the password from a Solaris client.
Let me know If you need more details on the bits of the setup, I'll try to help.

New gear

last week the new gear arrived. A little over a ton (1080kg including packaging) of various equipment was delivered. A few servers, racks, UPSs, SunRays, monitors... After everything was distributed to the final destination we fitted the servers into the cabinets and by 4p.m. my physical exercise for the next month was done ;-)
Over the next two days I installed Solaris 10 on the main servers, added the needed software (SamFS on the fileserver, Sun VDI and SunRay Server Software on the VDI servers).
The xVM servers will be running OpenSolaris and provide different flavours of Linux as compute farm to the users.

Directory services for all platforms will be provided with a Samba AD DC. That serves nicely for the Windows clients and also gives LDAP and Kerberos for the Unix clients. The main goal is to have one central account management and a common password across all platforms without the hassle of synchronising between the worlds.

Monday, March 22, 2010

Trying Samba4

It's been a while since my last post. I'm still alive and running (Open)Solaris whenever I have the option to choose.
A lot of exciting things happened in the past months and now a new challenge arises.
We start a new research branch and I got to plan (and soon to install) most of the infrastructure. It involves quite a lot of things that are pretty exciting by themselves (SamFs, VDI, xVM/Xen).
The one thing that kept me interested the past few days was the integration of the planned Windows7 guest for the virtual desktop infrastructure. As I try to avoid too much contact with the M$ world I am looking for a way to do it utilizing Unix power. As Microsoft removed support for NT4 style domains and group policies that became quite a challenge. Right now I'm trying out Samba4 as Active Directory replacement. So far it looks rather promising. A few questions arose already but with the help of the folks on #samba-technical those were cleared quite easily.
As I want to do some fancy stuff (like the shadow volume/previous version stuff in combination with zfs snapshots) it will become a hybrid setup with Samba4 as AD DC and Samba3 doing the fileserver part.
I tried to join the OpenSolaris CIFS service to the Samba4 domain but failed so far (that's being investigated by the samba folks). But since the storage server is going to be Solaris10 anyway that isn't a top priority at the moment.

I'll post more when I do the interesting parts of the planned installation.