Diary of a reverse-engineer and the SPARC T3-1

How does one get into the business of reverse engineering? For me it started at an early age, I loved to take things apart. Of course it's also desirable that, if whatever it is that's being taken apart was working, it should be again, once it's put back together. It took a little while before I was able to put them back together successfully. Sometimes the problem was that there would be extra parts left over during the re-assembly process. But the thing I always took the most pride in was being able to fix something that was broken.

In the late sixties it was old watches, transistor radios and telephones that were some of my first "projects". The old crank type phones were some of my favorites. I still have an old 5-bar magneto from one of those phones. As the years went by the technology changed, but my interest and curiosity remained, progressing into computers and programming.

So, this year I find myself doing some reverse engineering again, bringing the new Oracle/Sun SPARC T3-1 systems into an existing Solaris jumpstart environment. The first issue is that this environment is customized and the person who wrote the scripts is no longer around, thus that reverse engineering that I've been talking about.

First problem, the SPARC T3-1 wouldn't jumpstart in the existing environment because the custom scripts which try to specify the boot drives in one of two ways:

1. try to look up the drives in a disk map file based on the machine type and a wild card for logical disk names.
2. or use physical paths (assigned to a custom devalias) to look-up the logical path (/dev/dsk/c#t#d#) once the jumpstart is initiated.

In the product notes (821-2059-15) for the SPARC T3-1, it's stated that this system now uses a SAS 2.0 WWID (16 digit hexadecimal number) in place of the "t#". Of course this WWID is unique to each disk drive, so that rules out a wild card in the disk mapping file.

Option two was also a bust, because once the SPARC T3-1 boots, the physical path (whether using the format listing, or an "ls -l" command of the logical disks) doesn't provide the same physical path used at the eeprom boot level. This issue also comes into play during post jumpstart processing as well, when we try to write more devalias values for the root drive and its mirror. But I'm getting ahead of myself.

Example: ls -l and partial format listing (both show the scsi_vhci as the path)

/dev/dsk/c05000CCA00AD91644d0s2 -> ../../devices/scsi_vhci/disk@g5000cca00ad91644:c

AVAILABLE DISK SELECTIONS:

0. c1t5000CCA00AD91644d0
/scsi_vhci/disk@g5000cca00ad91644

To solve the first problem, I did two things, I modified the script that looks up the boot drives to recognize the devalias when they were given as a logical device, vs physical. This allowed the jumpstart to proceed normally and only involved a few commands from the "ok" prompt to determine the WWID of the disks that I wanted to specify. In this case I wanted to use the first drive of each of the two internal controllers.

{0} ok probe-scsi-all

/pci@400/pci@2/pci@0/pci@4/scsi@0

#--------^^^^^ 2^nd controller

FCode Version 1.00.54, MPT Version 2.00, Firmware Version 5.00.17.00

Target 9

Unit 0 Disk HITACHI H103030SCSUN300G A2A8 585937500 Blocks, 300 GB

SASDeviceName 5000cca0150611a0 SASAddress 5000cca0150611a1 PhyNum 0

#-------------^^^^^^^^^^^^^^^^ this will be the root mirror

/pci@400/pci@1/pci@0/pci@4/scsi@0

---------^^^^^ 1^st controller

FCode Version 1.00.54, MPT Version 2.00, Firmware Version 5.00.17.00

Target 9

Unit 0 Disk HITACHI H103030SCSUN300G A2A8 585937500 Blocks, 300 GB

SASDeviceName 5000cca01502f6e4 SASAddress 5000cca01502f6e5 PhyNum 0

#-------------^^^^^^^^^^^^^^^^ this will be the primary root disk

These disks are in HDD slot 0 (for pci@1) and HDD slot 4 (for pci@2). Note, switch any alpha characters to uppercase, as that's how they are listed in the /dev/{r}dsk/ directory.

{0} ok nvalias boot1 /dev/dsk/c0t5000CCA01502F6E4d0

{0} ok nvalias boot2 /dev/dsk/c0t5000CCA0150611A0d0
{0} ok nvstore

Now the jumpstart could proceed as normal, but as I mentioned above, the additional post processing (more of the custom stuff) tries to create devalias entries for the root disk and its mirror. The problem here was the same as mentioned earlier, i.e., the "ls -l" doesn't provide the info needed to re-construct a physical path that would be valid at the eeprom level.

One other things to note, during jumpstart the normal post processing does write the bootpath to the eeprom "boot-device" entry. However, it includes the unique WWID. This can be problematic if that disk should need to be replaced (i.e., how do we know the WWID of the new drive?). The solution I used in this case was to write the values for the root drive and its mirror using the default paths of the devalias listing for disks0 thru disk 7 (this system has an 8 disk backplane vs 16).

Example of the devalis. Note: The disk#’s match the HHD#’s

disk7 /pci@400/pci@2/pci@0/pci@4/scsi@0/disk@p3

disk6 /pci@400/pci@2/pci@0/pci@4/scsi@0/disk@p2

disk5 /pci@400/pci@2/pci@0/pci@4/scsi@0/disk@p1

disk4 /pci@400/pci@2/pci@0/pci@4/scsi@0/disk@p0

disk3 /pci@400/pci@1/pci@0/pci@4/scsi@0/disk@p3

disk2 /pci@400/pci@1/pci@0/pci@4/scsi@0/disk@p2

disk1 /pci@400/pci@1/pci@0/pci@4/scsi@0/disk@p1

disk0 /pci@400/pci@1/pci@0/pci@4/scsi@0/disk@p0

So, at this point I decided to scrap the old method and create the bootdisk and bootmirror devalias values based on these default disk devalias values, and they are based on the disk "port#"/HDD# slot position and not the WWID numbers. Note the addition of the ",0:a" to provide a slice/partition selection (needed to support a possible live upgrade to an alternate partition).

{0} ok nvalias bootdisk /pci@400/pci@1/pci@0/pci@4/scsi@0/disk@p0,0:a

{0} ok nvalias bootmirror /pci@400/pci@2/pci@0/pci@4/scsi@0/disk@p0,0:a

{0} ok nvstore

{0} ok setenv boot-device bootdisk bootmirror

{0} ok boot

This solution should be short lived as we are currently looking at implementing a different method of provisioning Solaris systems. If anyone has other suggestions for a solution, I welcome them.

Sun Alumni/Community site migrating to Planets

I sent an email off the other day regarding a problem that I was seeing on Community.Sun.Com (CSC) and I got the following in an email response.

The aggregator that runs community.sun.com is being replaced by planets.sun.com. The community.sun.com (CSC) team will then redirect CSC to http://planet.sun.com/alumni/group/blogs/.
But in the meantime, could you please point directly to the planet.sun.com (PSC) URL?

Since, the PSC application meets the need, we won't be debugging the CSC application.

Please let us know if you have any further questions. Thank you.

Regards,
Sun Blogs Support

So my next question was, will this mean that the Community site will look more like the BSC site, which just had some updates like "Most Recent Tags" and "New Bloggers", but alas the PSC site doesn't have those feature. But there may be some hope, I got this as a reply to the "new look" question.

There is a redesign in progress for the sun.com community venues, so community.sun.com will have a new look -- I'm hoping the roll out will happen by the next quarter.

Dark Matter/Dark Energy

I ran across an interesting article on SLAC's website* called The Search for Dark Energy . It sounds like Science Fiction, but researchers will be conducting over 2 dozen experiments to find that missing 70% of the stuff that makes up our universe.

* It's interesting what you find while looking up something completely unrelated. I went to SLAC's website because of a Sun blogger talking about SLAC getting one of the first Sun Blackboxes.
(Full Article)

Solaris 10: laptop config alternative

While I was at Sun, I helped out with several Installfests. For those who aren't familiar with the process, they were a gathering of people with the common goal of getting Solaris loaded on their laptops. Sometimes it was a mixture of luck and magic, to get the target laptop to retain the original OS (if desired) and then partition the disk(s) to load Solaris and/or linux. A fairly typical scenario was a double or even triple boot system;

1 - Windows
2 - Solaris (looked like a linux swap partition)
3 - Linux (Ext partition with root and swap)

The fourth DOS partition would usually be a shared data volume, configured as fat32, so that any OS you chose to boot could mount and access the shared volume.

However, I must be getting lazy, because I didn't want to have to deal with the disk re-partitioning and the risk to my company provided laptop this time around, so here's the alternative, which had some side benefits that I'll get to as I go along.

The first thing I did was download and install vmware Server 1.0.1, which is a freely available at the vmware website. Once the vmware server software is installed* and setup for Solaris, you can load a Solaris 10 x86 CD/DVD media and start installing Solaris as you would on any other hardware. So the first benefit was the time saved by not having to resize my Windows partition and re-partition the disk. You'll just need enough free disk space to hold the guest OS, plus swap space and any other Solaris partitions.
*If you don't already have it, it requires the MS .Net framework 1.1.

During the VM setup I chose to "hide" the VM behind the vmware "nat" and allow vmware to provide the IP address via DHCP. Here's the next benefit, I didn't have to load any special network drivers, the VM connects to the default network that the "master" OS is using at the time, whether that's wired or wireless. Also, sound support worked without any extra packages being loaded.

At this point I could use a little more memory, but I'm getting by pretty well, for my purposes with 1GB of memory. I haven't been able to get USB working, so that I could plug in a memory stick/thumb drive, but that's about the only thing I'm missing at this point. I I can boot the OS to have access to Solaris commands, manpages, do dtrace demos and even setup a Solaris container.

To busy to blog?

That's not necessarily a bad thing, it's kind of like being to busy to catch a cold, it's the good kind of busy, which is way better than the alternative. I've had lots of customer meetings the last few weeks, some are interested in storage solutions, everything from some additional storage (Sun StorageTek 3510 FC) to a solution to replace an optical WORM archiving system (SUN StorageTek 5320 NAS). I've also been busy doing some Total Cost of Ownership studies, these are time consuming, but they're interesting to perform and the benefits can be huge when comparing the costs of older hardware, power consumption and performance and factoring in virtualization, whether it's Solaris Containers or vmware.

On the biking front, my schedule and the shorter days have kept me from riding as much as I would like, but I have been using my trainer (Cyclops fluid2) and some video (that I shot with my miniDV camera mounted on the bike this past summer) as as motivation.

Solaris Virtualization and consolidation

Here's an interesting website that goes into some of the specifics of moving a Solaris 10 zone/container from one hardware platform to another. One area that it didn't mention, but which some customers are already using, is to locate the '/zones' filesystem on a SAN. By doing this, the container becomes highly portable to any platform within the companies infrastructure, that is attached to the SAN. By having the zones on the SAN you don't need to deal with the archiving steps, just detach the container from one platform and attach it to another (oversimplified, but that's the basics).

I'm currently in the middle a design for a Highly Utilized Data Center. I'm not quite sure what to call it, but that name describes the idea behind it. Sometimes when I'm putting a presentation together I feel like I'm scripting a episode of Star Trek with all the technobabble (e.g., business speak) But, to get back on topic, with various forms of virtualization (e.g., vmware and Solaris containers) the trend is to do more with fewer (notice I didn't say less). The idea is of course to have a platform (for the hardware piece I'll use the term platform) that can support many systems and/or services. For the X64 based platform the possibilities range from; vmware to support multiple OSs on a single platform, to Solaris 10 x86 by using zones/containers.

Weekend Rides:
This past weekend marked the start of some near seasonal temperatures (Indian Summer), so I took advantage of it and got out on my bike on both Sat. and Sun., managing about 62 miles total.

CEC 2006

This will be the first CEC that I've missed, I had attended all of them (while I was at Sun) since 1994. I either attended them as a participant, presenter or assisted at the last few Installfests. Oh, wait there was that year that they didn't have it, but even then I contributed a presentation / audio cast that was used as part of a Virtual CEC. I would always bring back something of value, to help customers; troubleshooting tips, new product info and contacts. People from all over the world attend the CEC, and it might be the only chance you would get to meet someone, who helped you with a problem, in person.

I just celebrated my 1 year anniversary at Laurus Technologies. It's taken a little getting used to, moving from the service side of things to sales, but so far, so good. Plus we have 7 Ex-Sun people at Laurus now (counting two that were added in just the past week or so), so it seems like old home week sometimes.

Thursday Ride: I opt'ed out tonight, it looked like rain and I took the family out for pizza. The weekend club rides should be fun, if the weather clears up.

I was even "dugg" a few weeks ago with my posting about the concept for a 6th Gen iPod.

Where Oh Where is my T2000?

Back on March 31st, 2006 I won a Sun Fire T2000 as part of a promotion during a SUN Partner Training webinar. It was even written up in a SUN Partner newsletter, congratulating me on the win.
But, (yes that's a big but) I still haven't received the T2000 that was promised. I think I've been pretty patient about it, all very pleasent emails between the SUN contact and myself. It's just disappointing that it is taking so long. I wanted a chance to play with the system a little and then set it up for customer demos. Hopefully I'll get the RoHS version, now that it's been this long.

Thursday ride: We were rained out tonight, oh well, it's a good time to get some blogging done.

Blog was misfed

No, it wasn't eating too much junk food, it looks like the feed url should have been ...rss.xml vs the default ...atom.xml. This seemed to be causing either an error message or a prompt for username and password when trying to read my blog entries from the SUN Community site.

Tuesday Ride for 8/8/06: Weather should be great. The route will probably be out and around Ottawa Lake, starting and ending at the North High School parking lot in Waukesha (round trip about 36 miles).