text:
tag:blogger.com,1999:blog-97268332019-02-12T03:33:01.129+00:00The Trouble with Tribbles...Peter Tribblehttp://email@example.com:blogger.com,1999:blog-9726833.post-56420930442061295112019-02-10T14:10:00.000+00:002019-02-10T14:10:48.486+00:00Thoughts on SPARC support in illumosOne interesting property of illumos is that its legacy stretches back decades - there is truly ancient code rubbing shoulders with the very modern.

An area where we have really old code is on SPARC, where illumos has support in the codebase for a large variety of Sun desktops and servers.

There's a reasonable chance that quite a bit of this code is currently broken. Not because it's fundamentally poor code (although it's probably fair to say that the code quality is of its time, and a lot of it is really old), but it lives within an evolving codebase and hasn't been touched in the lifetime of illumos, and likely much longer. Not only that, but it's probably making more assumptions about being built with the old Studio toolchain rather than with gcc.

What of this code is useful and worth keeping and fixing, and what should be dropped?

A first step in this was that I have recently removed support for starfire - the venerable Sun E10K. It seems extremely unlikely that anyone is running illumos on such a machine. Or indeed that anyone has them running at all - they're museum pieces at this point.

A similar, if rather newer, class of system is the starcat, the Sun F15K and variants. Again, it's big, expensive, requires dedicated controller hardware, and is unlikely to be the kind of thing anyone's going to have lying about. (And, if you're a business, there's no real point in trying to make such a system work - you would be much better off, both operationally and financially, in getting a current SPARC system.)

And if nobody has such a system, then not only is the code useless, it's also untestable.

The domained systems, like starfire and starcat, are also good candidates for removal because of the relative complexity and uniqueness of their code. And it's not as if the design specs for this hardware are out there to study.

What else might we consider removing (with starfire done and starcat a given)?

The serengeti, Sun-Fire E2900-E6800. Another big blob of complex code.

The lw8 (lightweight 8), aka the V-1280. This is basically some serengeti boards in a volume server chassis.

Anything using Sbus. That would be the Ultra-2, and the E3000-E6000 (sunfire). There's also the socal, sf, and bpp drivers. One snag with removing the Ultra-2 is that it's used as the base platfrom for the newer US-II desktops, which link back to it.

The olympus platform. That's anything from Fujitsu. One slight snag here is that the M3000 was quite a useful box and is readily available on eBay, and quite affordable too.

Netra systems. (Specifically NetraCT - there's a US-IIi NetraCT, and two US-IIe systems, the NetraCT-40 and the NetraCT-60. Code names montecarlo and makaha (something about Tonga too). Also CP2300 aka snowbird.

Server blade. I'm talking the old B100s blade here.

Binary compatibility with SunOS 4 - this is kernel support for a.out, and libbc.

I'm not saying at this point that all of this code and platform support will go, just that it lists the potential candidates. For example, I regard support for M3000 as useful, and definitely worth thinking about keeping.

What does that leave as supported? Most of the US-II and US-III desktops, most of the V-series servers, and pretty much all the early sun4v (T1 through T3 chips) systems. In other words, the sort of thing that you can pick up second hand fairly easily at this point.

Getting rid of code that we can never use has a number of benefits:

We end up with a smaller body of code, that is thus easier to manage.

We end up with less code that needs to be updated, for example to make it gcc7 clean, or to fix problems found by smatch, or to enable illumos to adopt newer toolchains.

We can concentrate on the code that we have left, and improve its quality.

If we put those together into a single strategy, the overall aim is to take illumos for SPARC from a large body of unknown, untested, and unsupportable code to a smaller body of properly maintained, testable, and supportable code. Reduce quantity to improve quality, if you like.

As part of this project, I've looked through much of the SPARC codebase. And it's not particularly pretty. One reason for attacking starfire was that I was able to convince myself relatively quickly that I could come up with a removal plan that was well-bounded - it was possible to take all of it out without accidentally affecting anything else. Some of the other platforms need bit more analysis to tease out all the dependencies and complexity - bits of code are shared between platforms in a variety of non-obvious ways.

The above represents my thoughts on what would be a reasonable strategy for supporting SPARC in illumos. I would naturally be interested in the views of others, and specifically if anyone is actually using illumos on any of the platforms potentially on the chopping block.Peter Tribblehttp://firstname.lastname@example.com:blogger.com,1999:blog-9726833.post-86104166300833783472019-02-08T22:21:00.000+00:002019-02-08T22:21:23.002+00:00SPARC and tod modules on illumosFollowing up from removing starfire support from illumos, I've been browsing through the codebase to identify more legacy code that shouldn't be there any more.

Along the way, I discovered a little tidbit about how the tod (time of day) modules - the interface to the hardware clock - work on SPARC.

If you look, there are a whole bunch of tod modules, and it's not at all obvious how they fit together - they all appear to be candidates for loading, and it's not obvious how the correct one for a platform is chosen.

The mechanism is actually pretty simple, if a little odd.

There's a global variable in the kernel named:

tod_module_name

This can be set in several ways - for some platforms, it's hard-coded in that platform's platmod. Or it could be extracted from the firmware (OBP). That tells the system which tod module should be used.

And the way this works is that each tod module has _init code that looks like

if (tod_module_name is myself) {
 initialize the driver
} else {
 do nothing
}

so at boot all the tod modules get loaded, but only the one that matches the name set by the platform actually initializes itself.

Later in boot, there's an attempt to unload all modules. Similarly the _fini for each driver essentially does

if (tod_module_name is myself) {
 I'm busy and can't be unloaded
} else {
 yeah, unload me
}

So, when the system finishes booting, you end up with only one tod module loaded and functional, and it's the right one.

Returning to the original question, can any of the tod modules be safely removed because no platform uses them? To be honest, I don't know. Some have names that match the platform they're for. It's pretty obvious, for example, that todstarfire goes with the starfire platform, so it was safe to take that out. But I don't know the module names returned by every possible piece of SPARC hardware, so it isn't really safe to remove some of the others. (And, as a further problem, I know that at least one is only referenced in closed source, binary only, platform modules.)Peter Tribblehttp://firstname.lastname@example.com:blogger.com,1999:blog-9726833.post-37043975962146366582018-11-04T19:53:00.001+00:002018-11-04T19:53:52.916+00:00Tribblix and the python transitionIt's been over a decade since python 3 came out, and a lot of the world is still using python 2.

But we're at the point now where the world has said enough is enough, and it's time to finally get the 2-to-3 transition over with.

And while Tribblix is all about retro styling, it's also all about keeping up. So I put together a plan for migrating Tribblix from python 2 to python 3.

Ship all the modules for python 3 as well as python 2, ready to switch

Move the python consumers (eg mercurial) across to python 3

Make python 3 the default

Deprecate and remove python 2

This is made a little easier by the fact that there's nothing in Tribblix itself that uses python directly - I haven't made the mistake of having my packaging system or anything like that written in python, for example.

There was a little wrinkle in all this. I had got this planned out, and then python 3.7 was just around the corner. So I ended up waiting a little, and put a python 3.6 to 3.7 transition at the beginning of the list.

So where am I right now? I've now got all the modules built and packaged for python 3.7, and python 3.6 has been removed from Tribblix. This was made somewhat easier by the fact that no packages in Tribblix yet depended on python 3 - the transition hadn't been started properly, so I could simply throw away all the python 3.6 stuff.

As an aside, this had the odd side effect that all the python 3.7 modules were packaged straight away for SPARC, whereas python 3.6 was never finished there - the 3.6 to 3.7 switch was all scripted, rather than manual, so was very little actual work. There were a couple of modules that needed to be updated anyway to work with 3.7 (pyyaml for example), and I took the opportunity to do a bunch of routine module updates at the same time.

So just having all the modules turned out to be nearly trivial. Now there's going to be a longer slog migrating all the python consumers across and making python 3 the default. (It might be easiest to make python 3 the default first, so that when building the consumers they automatically pick up the python I want.)

I was originally thinking of a fairly slow and structured approach where each step would be a point release of 3.7. But I'm well ahead of that already, and the remaining steps are likely to occur fairly promptly. (Or, as promptly as I have time to do the work.)

So it won't be long before we bid farewell to python 2 in Tribblix.Peter Tribblehttp://email@example.com:blogger.com,1999:blog-9726833.post-27782907504901683402018-07-17T08:10:00.000+01:002018-07-17T08:10:32.741+01:00Tribblix - minimal plus pkgsrcThe latest update (0m20.5) of Tribblix is now available for download.

This one comes relatively quickly after its predecessor, but includes the eager fpu and ldt fixes.

The visible change this time is that there's a minimal ISO available. At just over 200M, it's about a fifth of the size of the regular ISO.

The only difference between the regular and minimal ISO is in what extra packages are dropped on the ISO. If you aren't going to add overlays at install, then the minimal ISO is much better.

The use cases for the minimal ISO are the obvious minimal server installs (including cloud-based), but also if you decide to use pkgsrc for your software rather than Tribblix native packages.

Tribblix is all about choice and flexibility, so if you want to use pkgsrc, go right ahead. You'll get a broader choice of software, certainly. In some areas you'll get newer versions than I provide, in some areas Tribblix will update more aggressively.

One thing I have noticed, though, is that if you want to use pkgsrc, do so exclusively. Mixing and matching native packages and pkgsrc really doesn't work.

With that in mind, the minimal ISO makes a great base for pkgsrc. To make it even easier, the pkgsrc bootstrap is available on the ISO itself. So if you want to install Tribblix with pkgsrc, just invoke the installer like so:

./live_install.sh -B c1t0d0 pkgsrc

Then, as root

export PATH=/opt/local/bin:$PATH

and you're all set to install packages from pkgsrc.

To get the X server and basics:

pkgin install modular-xorg

To install a graphical desktop, such as Xfce - there are many others, of course:

pkgin install xfce4

And then, as a regular user

export PATH=/opt/local/bin:$PATH
startxfce4

And, lo and behold, you should have a functional basic Xfce desktop running.

Peter Tribblehttp://firstname.lastname@example.com:blogger.com,1999:blog-9726833.post-36686996030902625842018-06-01T22:18:00.000+01:002018-06-01T22:18:03.306+01:00Tribblix - creating zones from imagesOne of the things I've done with Tribblix is try and hide some of the complexity around managing zones - rather than having to mess around with zonecfg and zoneadm and all that, just have one simple command that creates a zone correctly.

Tribblix also has the capability to drop an alternative illumos distribution into a zone - so called alien zones.

The OmniTribblix variant has LX zones, so you can run Linux in a zone.

Up to now, you've had to manually download the appropriate image, save it somewhere, and then install the zone from that image.

Wouldn't it be much easier to have Tribblix do that for you? Well, as of the m20.4 release, it can!

So, for example, OmniOS have images as zfs send streams available for download - the .zfs.bz2 files. So you can now build an OmniOS zone on Tribblix like so:

zap create-zone -z omnios -t alien -i 10.0.2.26 -I omnios:r151026

That's it.

And, if you're running OmniTribblix you could create an Ubuntu zone like so:

zap create-zone -z ubuntu -t lx -x 10.0.2.27 -I ubuntu

or, if you want Alpine (and this is really quick)

zap create-zone -z alpine -t lx -x 10.0.2.28 -I proxmox:alpine

The images are downloaded and cached, so creating subsequent zones will be much quicker.

It's a proof of concept at this point, and needs fleshing out a bit more to make it even more friendly, but it shows that it's possible and useful.

Peter Tribblehttp://email@example.com:blogger.com,1999:blog-9726833.post-32098825332920302632017-11-06T19:47:00.001+00:002017-11-06T19:47:35.230+00:00Selecting relay smarthosts and using SMTP AUTH on illumosA problem I looked at recently involved configuring a system to send (relay) email via a customer's own SMTP servers. There are 2 parts to this:

Select the relay host depending on some condition

Authenticate with the remote relay using SMTP AUTH

Search for SMTP AUTH with sendmail on illumos or Solaris, and you invariably end up with advice on how to build Cyrus SASL and sendmail from scratch.

For example, Andrew has some good instructions.

However, if you look at the sendmail we ship on illumos you'll find that it's already been built with SASLv2 support:

# /usr/lib/sendmail -bt -d0.1 < /dev/null
Version 8.14.4+Sun
 Compiled with: DNSMAP LDAPMAP LOG MAP_REGEX MATCHGECOS MILTER MIME7TO8
 MIME8TO7 NAMED_BIND NDBM NETINET NETINET6 NETUNIX NEWDB NIS
 PIPELINING SASLv2 SCANF STARTTLS TCPWRAPPERS USERDB
 USE_LDAP_INIT XDEBUG

And, if you telnet to port 25 and look at the EHLO response it includes:

250-AUTH GSSAPI DIGEST-MD5 CRAM-MD5

However, that's not actually the part we want here (but I'll come back to that later). I don't want to authenticate against my own server, I need my system to authenticate against a remote server.

Back to the problem at hand.

The first part - selecting the right smarthost - can be achieved using smarttable. All you need is the smarttable.m4 file, and then build a configuration using it by enabling the smarttable feature.

The second part, SMTP AUTH, should also be very simple. Again, it's all documented, and just involves enabling the authinfo feature. But wait - on illumos, there is no authinfo.m4 file, so that won't work.

In fact, it does. So what you need to do is to download the sendmail source, unpack it, and there in the cf/feature directory you'll find the authinfo.m4 file.

OK, so copy both files - smarttable.m4 and authinfo.m4 - into the /etc/mail/cf/feature directory on a server. Copy and edit the sendmail.mc file (i'm going to copy it to /tmp and edit it there) to add the 2 feature lines, like this fragment of the file here:

...
define(`confFALLBACK_SMARTHOST', `mailhost$?m.$m$.')dnl
FEATURE(`authinfo')dnl<br />FEATURE(`smarttable')dnl
MAILER(`local')dnl
...

Basically, just add the features above the MAILER line. Then compile that:

cd /etc/mail/cf/cf
m4 ../m4/cf.m4 /tmp/sendmail.mc > /tmp/sendmail.cf

That's your new sendmail.cf ready. It uses 2 databases in /etc/mail, to create these (initially empty):

cd /etc/mail
touch smarttable
touch authinfo
makemap hash smarttable < smarttable
makemap hash authinfo < authinfo

then copy your new sendmail.cf into /etc/mail and restart sendmail

cp /tmp/sendmail.cf /etc/mail
svcadm restart sendmail

So far so good, but what should those files look like?

First the smarttable file, which is just a map of sender to relay host. For example, it might just have:

email@example.com smtp.gmail.com

Which means that if I want my home system to send out mail with my address on it, it should route it through gmail's servers rather than trying to deliver it direct (and likely getting marked as spam).

Then the authinfo file, which looks like

Authinfo:smtp.gmail.com "U:root" "I:firstname.lastname@example.com" "P:mypassword" "M:LOGIN PLAIN"
Authinfo:smtp.gmail.com:587 "U:root" "I:email@example.com" "P:mypassword" "M:LOGIN PLAIN"
(There are just 2 lines there, starting with Authinfo:, even if the blog shows it wrapped.)

Basically, for gmail, you need to supply your email address as the identifier and your password as, well, the password. (Note: if you've got two-factor authentication set up, you'll need to set up an app key.)

Of course, the authinfo files ought to to readable only by root, otherwise anyone one your system can read your password in the clear.

There are a couple of non-standard tweaks you'll need for gmail to work. First, you need to go to your gmail account settings and allow less secure apps. Second, you will need the "M:LOGIN PLAIN" entry in the authinfo file, else you'll get an "available mechanisms do not fulfill requirements" error back.

Redo the two makemap commands above and you're good to go.

That's SMTP AUTH the one way. At which point you're probably thinking, can we authenticate against an illumos sendmail using SMTP AUTH?

The answer, sadly, is no. At least as far as I can tell. While our sendmail is built correctly against SASLv2, illumos doesn't seem to ship enough supporting bits of the SASL infrastructure to make this work. You should be able to create the file /etc/sasl/Sendmail.conf to configure it. Unfortunately the only pwcheck_method available is auxprop (using shadow, which would allow you to authenticate against local system accounts, isn't available; neither is saslauthd, and there's no saslauthd anyway). Worse, illumos has no auxprop plugins, so the whole thing is rather useless. Note that rebuilding sendmail alone won't fix this, as the problem is in the underlying sasl implementation.

The above notes were developed on Tribblix, but ought to apply to any illumos distribution using the vanilla illumos sendmail+sasl combination.Peter Tribblehttp://firstname.lastname@example.com:blogger.com,1999:blog-9726833.post-58001846849205085402017-11-01T20:59:00.000+00:002017-11-01T20:59:52.853+00:00Building illumos-gate on AWSHaving talked about running Tribblix on AWS, one of the things that would be quite neat would be to be able to build illumos-gate.

This is interesting because it's a relatively involved process, and might require proper resources - it's not really possible to build illumos inside VirtualBox, for instance, and many laptops don't run illumos terribly well. So it's hard for the average user to put together a decent - most likely dedicated - rig capable of building or developing illumos, which is clearly a barrier to contribution.

Here's how anyone can build illumos, using Tribblix.

Build yourself an EC2 instance as documented here, with 2 changes:

The instance type should be m4.large or bigger - m4.xlarge or c4.xlarge would be better. The bigger the instance, the quicker the build, but m4.large is pretty much the minimum size.

Attach an EBS volume to the instance, at least 8G in size. If you want to do multiple builds, or do lint or debug builds, then it has to be larger. I attach the volume as /dev/sdf, which is assumed below. (You could keep the volume around to persist the data, of course.)

Once booted, log in as root. You then need to set up the zfs pool (the disk showing up as c2t5d0 below matches the /dev/sdf attachment point) and create a couple of file systems that can be used to host the build zone and store the build.

zpool create storage c2t5d0
zfs set compression=lz4 storage
zfs destroy rpool/export/home
zfs create -o mountpoint=/export/home storage/home
zfs create -o mountpoint=/export/zones storage/zones

You should then do an update to ensure packages are up to date, and install the develop overlay to get you some useful tools.

zap refresh
zap update-overlay -a
zap install-overlay develop

Then create a user, which you're going to use to do the build. For me, that is:

groupadd -g 10000 it
useradd -g it -u 11730 -c "Peter Tribble" -s /bin/tcsh \
 -d /export/home/ptribble ptribble
mkdir -p /export/home/ptribble
chown -hR ptribble:it /export/home/ptribble
passwd ptribble

Then create a build zone. It has an IP address, just pick any unused private address (I simply use the address above that of the global zone, which you can get with ifconfig or from the AWS console - note that it's the private address, not the public IP that you ssh to).

zap create-zone -z illumos-build -t whole \
 -i 172.xxx.xxx.xxx -o develop \
 -O java -O illumos-build -U ptribble

What does this do? It creates a new zone, called illumos-build. It's a whole root zone, with its own exclusive set of file systems. The IP address is 172.xxx.xxx.xxxx. The develop overlay is installed (in this case, copied from the global zone); the java and illumos-build overlays are added to this new zone (note the upper-case -O here). Finally, the user account ptribble is shared with the zone.

Give that a few seconds to boot and log in to it, then a couple of tweaks that are necessary for illumos to build without errors.

zlogin illumos-build
rm /usr/bin/cpp
cd /usr/bin ; ln -s ../gnu/bin/xgettext gxgettext

Now log out and log back in to the instance as your new user. We're going to create somewhere to store the files, and check out the source code.

mkdir Illumos
cd Illumos
git clone git://github.com/illumos/illumos-gate.git
wget -c \
 https://download.joyent.com/pub/build/illumos/on-closed-bins.i386.tar.bz2 \
 https://download.joyent.com/pub/build/illumos/on-closed-bins-nd.i386.tar.bz2

Now we set up the build.

cd illumos-gate
bzcat ../on-closed-bins.i386.tar.bz2 | tar xf -
bzcat ../on-closed-bins-nd.i386.tar.bz2 | tar xf -
cp usr/src/tools/scripts/nightly.sh .
chmod +x nightly.sh

There are two more files we need. Go to the tribblix-build repo and look in the illumos directory there. Grab one of the illumos.sh files from there and put it into your illumos-gate directory with the name illumos.sh. If you need to change how the build is done, this is the file to edit (but start from one of those files so you get one appropriate for Tribblix as the host). Also, grab Makefile.auditrecord and use it to replace usr/src/cmd/auditrecord/Makefile.

Now log in to the zone and start the build.

pfexec zlogin -l ptribble illumos-build
cd Illumos/illumos-gate
time ./nightly.sh illumos.sh

On an m4.xlarge instance, this took me just under 75 minutes. Look in the log directory and check that the mail_msg looks clean without errors, and you'll have the built files in the proto directory and an IPS repo under packages.

For more behind the scenes details on the illumos build process itself, look at the how to build illumos page.

Peter Tribblehttp://email@example.com:blogger.com,1999:blog-9726833.post-4955863760954395132017-10-31T19:12:00.002+00:002017-10-31T19:12:20.071+00:00Public Tribblix AMI now availableThere's now a public Tribblix AMI available to run on AWS.

This was built according to the notes I gave earlier. And is part of making Tribblix the illumos for everyone.

This is to be considered slightly experimental, and there are a couple of constraints:

First, the AMI is only available in the London region for now (I'm in the UK, so that's where I'm running things). I could make it available elsewhere, but there are costs associated with doing so and, as everything related to Tribblix comes out of my own pocket, I'm not going to incur costs unless there's a demonstrable need. If you want to run in a different region, then you can always copy the AMI.

Second, the size of the image is quite small. Again, there's a constraint on cost. But the idea here is that you wouldn't store any non-trivial data in the image itself - you would create an appropriately sized EBS volume, attach that and create a zfs pool for your data. The Tribblix repo server does just that - the package repo lives on the second pool.

So, how to use this? I'm going to assume some level of AWS familiarity, that you have an account and know basically how to use AWS, and that your account is set up with things like an ssh key pair.

Go to the AWS console, and navigate to the EC2 dashboard. Unless you've copied the AMI to the region of your choice, make sure you're working in London - the dropdown is in the top right:

Then hit the launch instance button:

Now you get to choose an Amazon Machine Image (AMI). Click on "Community AMIs" and enter "Tribblix" or "illumos" into the "Search community AMIs" search box. At the time of writing, you'll only get one result, but more may appear in future:

OK, go and select that one. Then you can Choose an Instance Type. A great thing about Tribblix is that it's pretty lightweight, so the t2.micro - available on the free tier - is a good choice.

Click on "Review and Launch". On the next screen you can edit the storage to add an additional volume, but the one thing you must do is edit the security group.

If you leave it like that, you'll have no way to access it. So Edit it, and the simplest thing to do at this point is to create a new security group that allows ssh only, with the source being your own IP address, which you can get by selecting "My IP" from the source dropdown.

(I've got a saved security group that does just that, to let me straight in from home.)

Click on "Review and Launch" to go back to the main screen, and then "Launch". The is when you get to choose which key pair you can use to log in to your instance:

It will take a little while to start (although it's usually ready before the status checks say so), and you should then be able to ssh in to it (as root, with the key pair you set up).

ssh -i peter1-london.pem \
email@example.com

And you're good to go. What you do then is up to you; I'll cover some scenarios in upcoming posts. Be aware that the base AMI has a pretty minimalist set of packages installed, so you probably want to add some more packages or overlays to do anything useful.

Peter Tribblehttp://firstname.lastname@example.com:blogger.com,1999:blog-9726833.post-12089595276330769832017-09-20T19:28:00.000+01:002017-09-20T19:28:51.799+01:00The commoditization of IT?IT, so the story goes, is now a boring commodity. But is this true?

Let's first define what a commodity is. There are a range of definitions we could use, but I'm going to think of a commodity as something that is functionally undifferentiated and available from multiple sources. The key aspect here is that of interchangeability (aka fungibility).

As an example, most computer components fall into the commodity category. Memory DIMMS, disk drives, network interfaces - you can (in principle) use any vendor's disk drives or memory and your computer will still work. You can use a mouse, keyboard, or monitor from any vendor and things will work just fine. Vendors have to differentiate in other ways - performance, cost, reliability, service.

What about smartphones? I would say that the phone piece is a commodity. Whether for a mobile or a land line, you can switch your telephone for another make or model, and you can switch from one telephony provider to another.

But the smart part of smartphones isn't properly interchangeable. You can't simply swap an Apple handset for an Android and carry on as you were; you have to switch everything to a different domain. And the suppliers here are keen to enforce differentiation and prevent interchangeability. We live in a world of proprietary walled gardens.

In most non-trivial cases, databases aren't commodities. Big database companies rely on the fact that you couldn't migrate to another database vendor even if you wanted to.

Operating systems are clearly differentiated. You can't swap Solaris for Windows, or either for Linux or BSD. You can't even treat different distributions as commodities if you restrict yourself to the Linux domain.

Although the operating system landscape is changing a little, in that Docker and containerization offer the prospect of interchangeability - you could, in theory, run a Docker image anywhere and on anything.

Cloud computing definitely isn't a commodity. (Thinking of it as a utility might be slightly more accurate.) Heck, there are sufficient differences over what's available that migrating between different AWS regions isn't smooth, let alone migrating between cloud providers.

Vendor lock-in is the big thing, and it's diametrically opposed to being a commodity - what vendor wants to make it easy for its customers to leave? (Despite that being one of the key attractions of any vendor in practice.)

One of the requirements for interchangeability is standardization, and there's a tension here between standardizing things (thereby making things the same) and innovation, which necessarily implies change. I could (and probably will at some point) go on at length about innovation, but I see precious little innovation in practice, more constant reinvention of the square wheel. Meanwhile the standards we have are either efforts like POSIX, which is largely codifying accidental implementations from the 1970s, or ad-hoc emergence of initial implementations that were cobbled together with little or no thought for actual suitability.

Rather than commoditization being a standard base, with a rising tide lifting all boats, any commoditization chips away the good stuff to leave the lowest common denominator, while everyone deliberately introduces incompatibilities in the name of differentiation.

So it seems to me that, far from being commoditized, IT has been monopolized and mediocritized.

Peter Tribblehttp://email@example.com:blogger.com,1999:blog-9726833.post-29325020621749223702017-09-10T21:55:00.000+01:002017-09-10T21:55:07.215+01:00Tribblix - illumos for everyone?When I was doing a bit of a branding exercise for Tribblix, part of which generated the rather amateurish logo I now have - something I needed to make some business cards and stickers to take to FOSDEM this year - one of the things I wondered about was a good tagline.

In the end, I ended up with "the retro illumos distribution". Of course, it was pointed out that illumos was retro enough on its own, so the idea of a retro variant was a bit unnecessary.

The other tagline I came up with was "illumos for everyone". I rejected it in the end because it was a bit preposterous - I'm not really building something for everyone.

Yet the underlying idea here was simple - that I would actively seek to build a distribution that was inclusive, not exclusive. That's why:

I have a SPARC version as well as x86

On x86, I support 32-bit as well as 64-bit systems

Tribblix is suitable for both desktop and server use

Tribblix is a flexible system, not an appliance or hypervisor

I work on ensuring Tribblix will work successfully on systems with more minimal resources than other distributions

A variety of installation methods are supported - media, network, iPXE

I've worked on installation in the cloud, both KVM-based and AWS, in addition to bare metal or other hypervisors

I've tried to make key features such as zones easier to configure and use

Generally, the idea is to reduce the barriers and limitations for installing and using Tribblix.

This summer, I came across a much better way of putting it. Rather than "illumos for everyone", a variation of the UK Labour Party's slogan expresses the idea much more elegantly. Tribblix would be "illumos for the many, not the few". It's a shame that the slogan is already taken, as it expresses the philosophical aim rather neatly.Peter Tribblehttp://email@example.com:blogger.com,1999:blog-9726833.post-60947681253791610582017-08-03T18:45:00.002+01:002017-08-03T18:45:56.469+01:00Creating a decent Tribblix AMIPreviously, I've described how I created my first Tribblix AMI, then how to do it properly in hvm mode so you can run on modern instances in all regions.

That creates something that will work, but is it actually in a state that's useful?

The first thing is to add an EC2 credential service. That's the thing that will query for metadata and install the keys on the system so you can log after the instance is created. I tried the ec2-credential service from OmniOS, but for some reason it didn't work right on Tribblix. I've tweaked mine a little, forcing it to run after the network comes up, adding retries in case there's a problem, and also disabling it in non-global zones.

Of course, there's more instance metadata that I could query and use, but I haven't yet had a need for anything other than the initial key.

The other thing I've been wondering about is the configuration of Tribblix itself - specifically what the storage should look like and what the default software installation should look like.

My image is built on an 8G "disk" or EBS volume. That might seem a little small, but remember that Tribblix is pretty lean and mean. For a typical server configuration you'll probably be looking at about 1G or so, and that's without any special work. The most annoying thing here is that by default you lose 2G to each of dump and swap, so that's effectively half the disk gone. There's opportunity to modify those, especially as I'm typically using t2.micro instances on the free tier that only have 1G of memory. You might not even want dump at all. As for swap, you do want some (so that anonymous reservations don't eat into actual RAM) but you could cut that down a bit.

As I'm writing this I do wonder whether I could pull some of the instance metadata and shrink the dump and swap volumes appropriately.

The assumption I'm making here, though, is that if you're storing any reasonable amount of data that you're going to attach a separate EBS volume, and you can then size that appropriately to the need at hand. (And you can then move that data around independent of your running instance.) So I think that keeping the root volume fairly small is reasonable. It also keeps my AWS bill down, an important consideration as any charges here come out of my own pocket.

Then, what should the baseline software install look like? Tribblix uses overlays, and there's an assumption that you always start from the base overlay. I'm currently using a dedicated overlay that pulls in cli-tools - essentially you get basic shells, compression tools, basic utilities, but not much else. Many of the normal server utilities don't apply to running in the cloud, as they're aimed at monitoring or managing hardware.

The base set of packages is that installed on the ISO. That includes most storage and network drivers, which are irrelevant - on EC2 you know exactly which drivers you need, so almost all the drivers that are installed are unnecessary. What I need here is a better way of handling installation variants, so it knows the drivers aren't supposed to be there - at the moment I could remove them, but updates and upgrades would simply put them back. In the same vein, I could only ship a 64-bit kernel, as we know there are no 32-bit instance types available.

At the moment I have an LX variant, which is a bit of a hack in terms of the way I've packaged it together, but as the number of interesting variants grows I'm going to have to come up with a better way of handling it, especially as you might want multiple variants together - for instance a 64-bit LX-enabled cloud-optimised image.Peter Tribblehttp://email@example.com:blogger.com,1999:blog-9726833.post-63730308222337175922017-07-31T13:54:00.002+01:002017-07-31T13:54:46.429+01:00Building a Tribblix AMI - hvm modeAfter having created a Tribblix AMI to prove that Tribblix basically works on EC2, I then moved on to the next issue - how to create an AMI that will run in hvm mode?

As a reminder, pv mode AMIs are deprecated, aren't supported by all instance types, and don't work in all regions. So you really need something that runs in hvm mode.

The first thought might be to convert the existing pv image to a hvm image. I've tried that and, while you can do the conversion, the image doesn't actually work. The problem here is that ZFS has the physical paths of the devices it's installed on embedded in the pool metadata. Changing from pv to hvm mode changes the emulated hardware, in particular the disk paths, so the ZFS pool isn't where it thought it was and the system panics. If you have a mismatch between the disk layout where the pool was created and where you're running you'll get a panic something like this:

If you had console access and could boot from media you could fix this, but AWS doesn't provide that. (And if you could boot from media you could just do a regular install without all the shenanigans involved in producing an AMI.)

So, you have to create the image on a system that looks like EC2. Which means using xen.

Fortunately, this road has been travelled before. These instructions are exactly what you need. They're for OpenIndiana, but will apply to any illumos distribution. And they're the process used by the OpenZFS project to do their testing. (I'll also mention that the OpenZFS folks have put a number of fixes back into illumos that improve the EC2 experience for us.)

I'm not going to repeat those instruction, that would be boring, so I'll talk about what I had to do or change to make those instructions work for me.

I got one of my spare desktop PCs out and installed Ubuntu 16.04 on it. (I must be spoilt by Tribblix, the Ubuntu install was horrendously slow and very high maintenance.) And then installed xen, rebooted as dom0, and set up the bridge networking.

That was my first pothole. There's this thing called systemd that's come along, and it changes the way network configuration is done. Much cussing and googling, but I got it right first time.

Then I discover that there's a new toolstack here. It's all xl not xm, but otherwise seems the same.

I then tried to start a VM, only to be given a completely meaningless and unhelpful error message. Why tell the user what's wrong when you can just vomit a stack trace?

After a bit of head-scratching I worked out that the system didn't actually support hvm mode. If you run xl info and look for virt_caps, it should mention hvm. That's a bit odd, the sticker on the front of the box looks right.

Manufacturers ship hardware with VT-x disabled in the BIOS, it appears. Into the BIOS we go, to find that the relevant settings are greyed out and you need a BIOS password to get into them. Open the box and start looking for jumpers. Fortunately I found a helpful article - the key here was the bit about the jumper being blue, little details like that make all the difference.

OK, so having wiped the BIOS password, gone into the BIOS and enabled VT-x, I go back to xen. Looking at virt_caps now shows hvm, as it should, and my domain starts.

The idea here is that you connect to the console with VNC. Easy enough, but by the time I had got my ssh tunnel set up and started up my VNC client, my VM had gone. I started it again, it starts booting just fine but then issues a few warnings and then a kernel panic. It's all over pretty quick.

In order to catch what it said, I then used vnc2flv. Someone asked me about screen recorders a while back, and I suggested they did what they wanted to do in a vnc session and use vnc2flv to record it. But it's the same here. Once I had the session recorded I can watch the movie and pause it to see what errors it's spitting out.

This, I think, is related to illumos bug 7186. It looks like we can't handle the network presented by newer versions of xen.

To get round this I simply disabled the network interface in the VM definition. Then the VM boots just fine and can be installed. You're a little bit limited in that you can't do updates but, as long as nwam is enabled then it will get itself on the network when you do run it on something that does have a compatible network.

For OmniOS, this means you have to manually enable nwam, as they have networking switched off by default. And remember that you must have networking enabled if you're running on EC2 as there's no other way to access your system.

What you'll also need to ensure at this point is that you have a functional user account you can get in to via ssh. With Tribblix and OpenIndiana you have jack, other distros might need to create a user. You wouldn't want that on a production AMI, of course, but you need to be able to log in to the system the first time in order to complete any configuration and add the various bits of AWS integration that you'll need.

Having got my image installed I followed the instructions through and got an AMI that works just fine.

The configuration file I used is:

builder='hvm'
name='ami-template'
vcpus=1
memory=1024
disk=[ 'file:/var/tmp/tribblix-0m20.1.iso,hdb:cdrom,r',
 'file:/root/ami-template.img,xvda,w' ]
boot='d'
vnc=1
vnclisten='0.0.0.0'
vncconsole=1
on_crash='preserve'
xen_platform_pci=1
serial='pty'
on_reboot='destroy'

The one crucial thing here, apart from not having a vif line to create a network, is that you must use xvda for the disk. That's what EC2 will present to you, if you use something else you'll get the same panic on boot that I saw when attempting to convert a pv image.

We're almost done. Next time I'll talk about how to go from something that minimally boots up to something that's done well.Peter Tribblehttp://email@example.com:blogger.com,1999:blog-9726833.post-22693721392225335792017-07-31T09:28:00.000+01:002017-07-31T09:28:18.813+01:00Running illumos on AWS - the first Tribblix AMII've run Tribblix on all sorts of hardware - desktops, servers, even the occasional laptop. I've had success running it on some of the smaller cloud providers that allow you to install from a custom ISO, or iPXE, such as my adventures with Vultr.

However, running on AWS has eluded me. You might wonder why you would want to, but the reality is that AWS is a huge player, with many people turning to it as their default (and often only) option. So giving everyone who uses AWS access to Tribblix would be a good thing, and would also offer an easy route for people who might want to play with Tribblix to do so.

The first thing to realize is that AWS is not so much a single cloud as a set of independent clouds. Each region is independent, and has a different set of capabilities. For example, EFS is only available in a few regions. These differences can affect us.

On AWS, there are 2 different types of guest. We have pv (the older, paravirtualized) and hvm (the newer, hardware assisted). Any given AMI (Amazon Machine Image) will only run as either a pv or a hvm guest. And some EC2 instance types are pv, others hvm. Newer regions (such as London) are exclusively hvm, so pv isn't an option.

Building an AMI from scratch looked a little daunting, so I looked to see what other illumos distributions might have made AMIs available. If you go to the community AMI page when launching an instances, the only one you'll find is OmniOS. They even have a page explaining how it was done. The snag is that all their images are pv. For my first set of experiments then, I was operating in the Dublin region.

The OmniOS AMI boots up just fine and works pretty much as you would expect. No problems there. How to get Tribblix running though?

The answer lies in the beauty of ZFS and Boot Environments. The basic approach here is to take a running OmniOS image, create a new Boot Environment, install Tribblix into that Boot Environment, and make the Tribblix Boot Environment the one to boot from next time. Once I've successfully booted the Tribblix image, I can clean up and delete the original OmniOS files.

One of the advantages of Tribblix is that I have my own installer. It's quite a bit simpler than some of the other distros, and thus much easier to mangle to do things in new environments. I decided to use the iPXE image as used in my Vultr experiment, because it was easy and I had it to hand. I then wrote a modified installer script (source here) called img_install that was based on my over_install script used to drop Tribblix into an existing ZFS pool. The difference is that the old over_install was run in the context of a Tribblix Live CD; the new img_install is run in the context of an alternative distro. The other thing in that script is that I don't do any boot loader fiddling - the pv instances have a special pv-grub, which I'm careful not to touch.

(By the way, the same trick will work for other illumos distributions. You just need a source archive of some sort and a script to unpack it. For example, I have a script to unpack some of the ISO images in the tribblix-zones repo, which I use to create alien-root zones. It's the same idea of installing an image in a alternate path.)

So all that was involved was to:

Start up an OmniOS instance (a micro instance on the free tier works fine)

Run the img_install script to create the alternate BE

Reboot, so you boot into Tribblix

Delete the old OmniOS BE

Finish off the install and apply updates

Then you can do the normal create an image trick on the AWS console, and you have a nice shiny Tribblix AMI.

That all worked out just beautifully. Tribblix runs on EC2 just fine.

In the next article, I'll describe how to create a hvm AMI.Peter Tribblehttp://email@example.com:blogger.com,1999:blog-9726833.post-55967227223020059982017-07-22T12:41:00.000+01:002017-07-22T12:41:17.986+01:00Mucking around with IPv6 and illumos zones
The world is running out of IPv4 addresses, and it's time to move to IPv6.

I remember that story being told over and over at conferences in the mid 1990s. Yet, here we are in 2017 and while there has been progress, we're definitely not there yet.

With zones, illumos (and Solaris) give you virtualized application environments (containers is the trendy term - we tend not to use that in the Solarish context because it got polluted by Sun marketing). Those environments (usually) need to be networked, so why not with IPv6.

So here goes with a few notes on the subject.

Shared-IP zones

With zones, the original networking model was a shared-ip stack, where the zone is given a fully configured network that is just a virtual IP configured on an existing interface. All the setup is done in the global zone, which makes it very easy.

(By the way, this was the cause of the limit of 8192 zones per system, because you can only have 8192 virtual addresses on a single physical interface.)

And configuring an IPv4 address is just a case of adding a net section to the zone configuration:

add net
set physical=aggr1
set address=172.18.1.172/24
end

It's exactly the same for IPv6, the only interesting issue is what the IPv6 address would be. Let's start with the link-local address - the one that starts with fe80:: - as you will generally need that even if you don't have a routed IPv6 network. For a physical interface, the IPv6 address is usually derived from the MAC address. We can't use that one, because we're sharing the interface and the global zone has already grabbed it. So the convention here is to construct something from the IPv4 address. It's then guaranteed to be unique in a broadcast network, which is all that matters for a link-local address. So all we have to do is convert the IPv4 address to hex, for example with printf:

printf "%x%x:%x%x\n" 172 18 1 172

which gives ac12:1ac, so the link-local address would be configured as:

add net
set physical=aggr1
set address=fe80::ac12:1ac/10
end

You're pretty much done here, if you do that your global and non-global zones will be able to communicate using IPv6 on the local subnet.

If you had a routable prefix, then the same scheme can be applied. Just put the fragment onto the end of your prefix.

add net
set physical=aggr1
set address=XXXX:XXXX:XXXX:XXXX::ac12:1ac/64
end

Of course, if you're assigned specific IPv6 addresses then you can use those directly. The above scheme is pretty trivial to script, though (and it actually makes it fairly easy to keep your DNS zone files up to date too).

Exclusive-IP zones

For an exclusive-ip zone, you just hand over a network interface to a zone and let it go figure. So it can assign whatever addresses it likes.

In particular, the zone can use the normal MAC address scheme to generate its IPv6 link-local address.

Originally in older Solaris, you needed to use a genuine physical interface. Which limited you a little bit as there are only so many network cards you can jam into a server. OpenSolaris introduced full network virtualization in the form of Crossbow, so any illumos distribution or Solaris 11 can create fully virtualized network stacks and present those to zones in the same way.

In Tribblix, I use zap to manage zones, and it takes care of creating the appropriate vnics and, if appropriate, etherstubs, and wiring things together. I also poke functional /etc/hostname.* and /etc/defaultrouter files into the zone so the networking at least gets configured when the zone boots. Adding IPv6 to the zone is simply a case of creating matching empty /etc/hostname6.* files (one for each vnic) and the IPv6 addresses will get autoconfigured.

There's one wrinkle with exclusive-ip that deserves a whole section, that of restricting the zone to only using addresses that you've set.

Restricting with allowed-address

Remember that an exclusive-ip zone can manage the network interface. So it could allocate the wrong address and generally cause havoc on the network. To prevent this, set the allowed-address property on the interface. For example:

add net
set physical=vnic1
set allowed-address=172.18.1.172/24
end

The zone manages the interface, but an attempt to configure an invalid address will be thwarted.

You can see what's happening under the hood by running dladm show-linkprop. You'll see that the protection and allowed-ips properties are set.

(As an aside, it would be fantastic if illumos got the configure-allowed-address feature that Solaris 11 has, which would bypass my trickery in having to poke the network setup files in the zone.)

The same thing works for IPv6. The first problem I discovered is that (unlike Solaris 11) illumos won't accept multiple addresses in a list. Initially I thought this was something about IPv6, but it turns out you need to specify each address you want to add in a separate block.

The next question is going to be - what is the IPv6 address going to be? It's derived from the MAC address, so isn't fixed in advance.

The first step is to get the properties of the vnic. Running dladm show-vnic will give you the properties you need, including the MAC address. If you just want the one field, that's fairly easy too.

# dladm show-vnic -p -o MACADDRESS vnic1
2:8:20:c6:71:d

The IPv6 address is made up from that as a EUI-64 address, which is basically the fe80:: prefix, the first 3 octets, then ff:fe, then the last 3 octets. Oh, and the 7th bit gets flipped. And conventionally the leading zero gets suppressed. An ugly way of scripting this in ksh looks like:

/usr/sbin/dladm show-vnic -p -o MACADDRESS $VNIC | \
 /usr/bin/sed 's=:= =g' | read o1 o2 o3 o4 o5 o6
integer -i2 vi=16#$o1
integer -i2 nvi
nvi=$(($vi ^ 2#00000010))
integer -i16 xv=$nvi
no1=${xv/16#/}
if [ "$no1" = "0" ]; then
 no1=""
fi
if [ "$no3" = "0" ]; then
 no3=""
fi
if [ "$no5" = "0" ]; then
 no5=""
fi
printf "fe80::%s%s:%sff:fe%s:%s%s/10" "$no1" "$o2" "$o3" "$o4" "$o5" "$o6"

So, what you want to do is add something like:

add net
set physical=vnic1
set allowed-address=fe80::8:20ff:fec6:71d/10
end

to your zone configuration. And if you have a routable IPv6 address, you'll need to duplicate the block again for that.

In practice, this didn't quite work for me. If you don't set the allowed-address properties then the addresses get configured correctly, but with the protection set the address doesn't come up properly. If you try it then you get:

# ifconfig vnic1 inet6 up
ifconfig: setifflags: SIOCSLIFFLAGS: vnic1: Invalid argument

However, if you explicitly set the address, running ifconfig by hand:

# ifconfig vnic1 inet6 fe80::8:20ff:fec6:71d/10 up

then it works perfectly.

Peter Tribblehttp://email@example.com:blogger.com,1999:blog-9726833.post-2051489658943820382017-07-13T13:27:00.000+01:002017-07-13T13:27:52.553+01:00What gets into Tribblix?The software available for Tribblix is a bit of an eclectic mix. How do I choose what software to package?

There are actually a number of different reasons why you get a particular package.

The basics

Some packages are just basic,and you expect to find them. Much of the GNU stack comes in this way. And often things like Perl and Python are a foundational requirement for a lot of other tools.

What I want personally

There are a number of areas where I have specific interests - I'm a bit of a magpie when it comes to X11 window managers, for example. And I need to open office documents, so I had to get LibreOffice working. There a few games or emulators that I like. This also explains why some things might not be present too - I have no real interest in video or multimedia, for example, so that's an area with relatively little coverage.

Oh, that looks cool

I'm often interested in new stuff. (Even if it's actually old stuff that's just new to me.) So if I come across a piece of software and think "that looks cool" I'll often try and build and package it. If it works, fine, it ends up in the repository. Even if I might not end up doing anything with it, I've gone to the effort of making a package so it may as well stay there and somebody else might make use of it.

I have a $DAYJOB

Yes, I have a day job (a very good one, thank you very much), and it involves running applications on illumos. If I'm going to evaluate software I'll do it on Tribblix first. Building stuff on Tribblix is much easier than on, say, OmniOS - I have total control of the environment, and many more tools and prerequisite packages to give me a head start. So I can easily screen out any software that simply isn't going to work, and identify any patches or modifications necessary, before heading into the rather more constrained work environment.

Can you make X or Y or Z available

I get requests from users. I'll pretty much always at least try to add the software asked for - the fact that someone's bothered to ask indicates it might be useful, and I might find it interesting as well. This doesn't always work, of course, and I'll have to punt.

I got bored one day

Sometimes I get a bit of free time (no, this doesn't happen very often), and start looking for packages that might be worth adding. Sometimes this involves looking at other systems to see what they ship.

It's a prerequisite for something else

Dependency hell is a fact of life, so a lot of the time is actually spent building prerequisites. This is one reason for speculatively trying things out - it identifies prerequisites, and they're often going to be needed by other packages too. Although what you'll find is a number of packages with no obvious consumers, because the software I wanted didn't actually work. As I mentioned before, though, I'll keep those packages I built, and they might come in useful later. Peter Tribblehttp://firstname.lastname@example.com:blogger.com,1999:blog-9726833.post-14862450433514851332017-07-06T21:38:00.002+01:002017-07-06T21:38:59.834+01:00Running LX zones with TribblixI mentioned a few months ago a little project I had been working on - nicknamed omnitribblix, it's regular Tribblix with the illumos components coming from illumos-omnios (now via OmniOS Community Edition) rather than vanilla illumos-gate.

One of the changes I made in the recent Milestone 20 update was to split out the release packages to give more flexibility.

Thiis allowed me to release a micro update to Milestone 20 (imaginatively called m20.1 or update 1), which updates the illumos bits but shares the same main package repository as the main Milestone 20 release.

And the other thing I can now do is build variant releases. So Tribblix has an LX variant!

You can download the omnitribblix ISO image from the Tribblix download page. It installs, operates, and is packaged just like regular Tribblix. If you don't use LX zones, you probably wouldn't notice the difference.

(It's versioned as m20lx.1 - the update 1 there means that it's a parallel release to the regular Tribblix Milestone 20 update 1.)

You can also update to the LX variant from either the regular Milestone 20 or Milestone 20 update 1 releases, in the normal way. It's a micro update, or sidegrade perhaps, but uses the same upgrade process as regular upgrades.

And, because of the magic of boot environments, if there's a problem you can roll back.

Anyway, once you have omnitribblix installed, how do you create an LX zone? Very easily, in the same way you create and destroy other zones on Tribblix, using the zap utility.

Before you can do that, though, you need a Linux image of some sort to install.

I've been using the same images I use under Docker. So, for example, if I want Alpine then I would go:

docker run alpine uname -a

and then get the name of the container

docker ps -a

and then export that with

docker export romantic_galileo > alpine.tar

Then copy the alpine.tar file to your omnitribblix system. If you want something a bit richer, then Ubuntu will work. But generally exporting a Docker container like this will work, and the image characteristics will be a good fit for a zone.

And then all you do to create the zone is use zap, specifying that it's an lx brand and telling it where the tarball is:

zap create-zone -z alpine -t lx \
-x 10.0.2.99 -I /tmp/alpine.tar

and just zlogin to it as normal.

There are constraints around networking - you have to be exclusive-ip (the -x flag) and zap will create (and destroy) the vnic for you automatically. But the networking in the zone won't actually be configured. (While you specify the IP address in the command, that just tells zap how to configure the network plumbing and the vnic.) You'll have to log in to the zone and use the native tools to identify and configure the network, like so:

/native/sbin/ifconfig -a
/native/sbin/ifconfig znic0 inet 10.0.2.99 up
/native/usr/sbin/route add net default 10.0.2.2

And off you go. Sitting on an illumos box with all its goodness, with access to the wide variety of the Linux ecosystem at your fingertips.

Peter Tribblehttp://firstname.lastname@example.com:blogger.com,1999:blog-9726833.post-31612654587549133102017-06-18T17:52:00.000+01:002017-06-18T17:52:19.553+01:00Tweaking binaries with elfeditOn Solaris and illumos, you can inspect shared objects (binaries and libraries) with elfdump. In the most common case, you're simply looking for what shared libraries you're linked against, in which case it's elfdump -d (or, for those of us who were doing this years before elfdump came into existence, dump -Lv). For example:

% elfdump -d /bin/true

Dynamic Section: .dynamic
 index tag value
 [0] NEEDED 0x1d6 libc.so.1
 [1] INIT 0x8050d20

and it goes on a bit. But basically you're looking at the NEEDED lines to see which shared libraries you need. (The other field that's generally of interest for a shared library is the SONAME field.)

However, you can go beyond this, and use elfedit to manipulate what's present here. You can essentially replicate the above with:

elfedit -r -e dyn:dump /bin/true

Here the -r flag says read-only (we're just looking), and -e says execute the command that follows, which is dyn:dump - or just show the dynamic section.

If you look around, you'll see that the classic example is to set the runpath (which you might see as RPATH or RUNPATH in the dump output). This was used to fix up binaries that had been built incorrectly, or where you've moved the libraries somewhere other than where the binary normally looks for them. Which might look like:

elfedit -e 'dyn:runpath /my/local/lib' prog

This is the first example in the man page, and the standard example wherever you look. (Note the quotes - that's a single command input to elfedit.)

However, another common case I come across is where libtool has completely mangled the link so the full pathname of the library (at build time, no less) has been embedded in the binary (either in absolute or relative form). In other words, rather than the NEEDED section being

libfoo.so.1

it ends up being

/home/ptribble/build/bar/.libs/libfoo.so.1

With this sort of error, no amount of tinkering with RPATH is going to help the binary find the library. Fortunately, elfedit can help us here too.

First you need to work out which element you want to modify. Back to elfedit again to dump out the structure

% elfedit -r -e dyn:dump /bin/baz
 index tag value
 [0] POSFLAG_1 0x1 [ LAZY ]
 [1] NEEDED 0x8e2 /home/.../libfoo.so.1

It might be further down, of course. But the entry we want to edit is index number 1. We can narrow down the output just to this element by using the -dynndx flag to the dyn:dump command, for example

elfedit -r -e 'dyn:dump -dynndx 1' /bin/baz

or, equivalently, using dyn:value

elfedit -r -e 'dyn:value -dynndx 1' /bin/baz

And we can actually set the value as well. This requires the -s flag to set a string, but you end up with:

elfedit -e 'dyn:value -dynndx -s 1 libfoo.so.1' /bin/baz

and then if you use elfdump or elfedit or ldd to look at the binary, it should pick up the library correctly.

This is really very simple (the hardest part is having to work out what the index of the right entry is). I didn't find anything when searching that actually describes how simple it is, so I thought it worth documenting for the next time I need it.

Peter Tribblehttp://email@example.com:blogger.com,1999:blog-9726833.post-26637995571636001842017-06-09T19:46:00.001+01:002017-06-09T19:46:28.078+01:00On Tribblix Milestone 20Having released a new update for Tribblix, I thought I would add a little commentary on the progress that's being made and the direction things are going in.

This goes beyond the rather dry release notes and list of what's changed.

The big structural change is that the ISO has been built as a single root archive, rather than the old way with a split-off /usr that's lofi-mounted from a compressed image.

The original reason for doing this (and I experimented with it a while ago) was to allow installation on systems without drivers for the device that you're booting from. This might be a system with only USB3 ports, or I've had problems with laptops where illumos doesn't recognize the CD drive. The boot loader (and BIOS) load the initial boot archive, so if you don't need to ever talk to the media device again you're in much better shape.

While we now have USB3 support, this simplified boot is a good thing in any case, and it allows some neat tricks like iPXE boot.

Another logical change is in the release mechanism itself. I've discussed the Tribblix package repositories before. The snag with the traditional repository layout was that the packages that defined a release were in the main Tribblix repository. So, every time I make a new release I end up having to create a whole new Tribblix repository. Every time I update the illumos packages, I needed a new Tribblix repository. Creating a new one isn't too bad; ongoing support for multiple repositories is a lot of unnecessary work.

The way to fix this is to split out the packages (there are 3 of them) that define the properties of a release into their own separate repo. This allows at least 2 new possibilities:

I can release updated illumos packages without spinning a whole new Tribblix release. It would still use the same upgrade mechanism, but the main Tribblix repo is shared and it's a much lighter release process.

I could create variants or spins. For example, I could create a variant that has LX (see omnitribblix). This would just have a different set of illumos packages but shares everything else. Or I could build a 32-bit or 64-bit only distro.

I haven't yet done either of those things, but it's going to happen.

Behind the scenes I've been gradually working to get more packages - especially those that deliver libraries - built as both 32-bit and 64-bit.

Tribblix is fairly clear that it will continue to support 32-bit and 64-bit hardware, at least for a while. (Whereas both OmniOS and OpenIndiana have effectively dropped 32-bit compatibility, mostly by neglect rather than design.) Of course, there is a reasonable amount of software now that's only 64-bit (anything built with go, for example, or OpenJDK 8), but there's a reasonable chance the people using 32-bit hardware aren't necessarily going to want the latest and greatest applications. (This isn't 100% true, by the way - sometime you have to interoperate with other facilities in the environment.) But eventually we're going to have to make a full 64-bit transition, and it would be good to be ready.

That gives a rough idea of the work that's currently underway. Looking ahead, there are a whole long list of packages that need adding or updating (such is a maintainer's life). The one significant place I have been falling behind is that I haven't updated gcc, so that needs work. And, of course, I'm trying to get SPARC into some sort of reasonable shape. But, overall, Tribblix is now pretty solid and a bit more polish and attention to detail would benefit it greatly.

Peter Tribblehttp://firstname.lastname@example.com:blogger.com,1999:blog-9726833.post-74866295181990874962017-06-07T16:13:00.002+01:002017-06-07T16:13:23.518+01:00Installing Tribblix on Vultr using iPXEOne of the new features in Tribblix 0m20 is that booting and installing using iPXE now works.

Here's an example of using this functionality to install a server running Tribblix in the Vultr cloud. A similar mechanism ought to work for any other provider that allows iPXE boot.

I'm assuming you have signed up and logged in, then go to deploy a server.

First choose where you want to deploy the server. I'm in the UK, so London is a good choice.

Then the critical bit, selecting the Server Type. The bit you want here is in a slightly confusing location, under the "Upload ISO" tab. But then select the "iPXE" radio button and put in the value http://pkgs.tribblix.org/m20/ipxe.txt

The other key option is Server Size. As with many providers, there's a simple scale. For testing, an instance with 1G of memory is more than adequate.

The deploy it. After a few seconds of installing you can then click the link to manage the server, and then view the console, which uses VNC.

If you're reasonably quick you get to see the initial iPXE screen, and can see it downloading the images:

What you can see here is that it's downloaded the original ipxe script we specified. This looks like:

#!ipxe
dhcp
kernel /m20/platform/i86pc/kernel/amd64/unix
initrd /m20/platform/i86pc/boot_archive
boot

Which just says to set up the network using dhcp (this might have already been done, but if you're booting off an ipxe iso it may not have been, so we do it anyway), then download the kernel and the boot archive, then boot from what you've just downloaded.

The kernel and the boot archive are on the iso, I've just unpacked them on the server (so the URL given above for the ipxe script will be reasonably permanent for anybody to use). The only slight tweak I've had to make is that the original boot archive is actually gzip compressed and iPXE can't handle that, so it's been uncompressed. The boot archive also now contains the /usr file system as well, rather than it being split off as before. While I'm sure you could mangle the system to download it and sort things out, it's so much easier to put it inside the boot archive.

Then you get into the normal installer, so log in as jack, su to root, and see what disk(s) are available using the new diskinfo tool. Then you can install Tribblix to that disk:

Don't bother adding additional overlays at this point. It won't work - and you'll get an error about not being able to install overlays (you'll get the error anyway because the installer always tries to add some packages that aren't needed in the live environment). This will be fixed in a future update, but it's relatively harmless.

The other thing you should do before the installation is to change the passwords for root and jack. If you change them before running the installer than the change will propagate to the installed system (because all it's doing is a copy). You really don't want the system to boot up wide open to the internet with the default (and well known) passwords.

Once the (pretty quick) install finishes, it'll look like this:

That's just like a normal install, other than the missing overlays. Then just reboot and you'll soon see the new loader, followed by the system booting.

Due to the missing overlays, you'll get an error about the intrd service failing. You'll have to log in (ssh will work at this point) and then add at least the base overlay:

zap install-overlay base

Plus whatever other overlays you might want. Then you can clear the intrd service and you're good to go.

Peter Tribblehttp://firstname.lastname@example.com:blogger.com,1999:blog-9726833.post-66753450789983772232017-06-02T20:18:00.002+01:002017-06-02T20:18:46.038+01:00Tribblix memory requirementsCompared to the other illumos distributions, Tribblix has lower memory requirements.

I'm not talking about crazy stunts like running in 48M; here I'm talking about running a fully fledged system.

I've been doing a bit of testing of the upcoming release, which includes running the install under a range of configurations. The test here is to boot the ISO image in VirtualBox with a range of memory sizes and then install the kitchen sink.

The live image won't boot at all on a 256M system

The live image will boot on a 512M system, but installing to zfs will fail

However, installing to ufs works on a 512M system

With 768M, installation to zfs is rather slow

With 1G or more, you're fine

The upcoming release is going to be built slightly differently, in that it's no longer a split-off /usr configuration. (I discussed how that worked and those strange zlib files some time ago.) The latest OmniOS is a single image; SmartOS likewise. It's just so much easier to construct, and far more reliable.

That change explains the 256M failure - the ramdisk is about 300M, so it simply won't fit. It's likely to have an impact on the 512M case too - in the old scenario you only paged in the bits of the /usr filesystem as and if you needed them, now it's locked into memory.

On a limited memory system there's a way to make things a bit easier. Simply install the base (no additional overlays) from the installer, then add the rest of the overlays and packages later. The point here is that running from disk doesn't lock up anywhere near as much memory as the full OS being resident in RAM does. And some of the packages in the kitchen sink are rather large, which causes problems.

Once you've got Tribblix installed, how well does it cope? Surprisingly well, to be honest. The Xfce desktop runs quite well in either 512M or 768M of memory. I can run firefox on the 768M system without too many problems (given the way it consumes memory, probably not for a long intensive browsing session), while firefox on a 512M system does run, but it's clearly starting to grind. Java applications work, some smaller ones at least. You need to be realistic in your expectations, but the point is that smaller systems do work.

The most limited systems would tend to be older, possibly 32-bit hardware. I could build a 32-bit only image which would be quite a bit smaller - maybe only two-thirds the size. (And if you really wanted to you could get it even smaller - but then you're in the realms of building custom images using mvi or the like.)

However, the aim of keeping Tribblix viable on smallish systems isn't just to allow the use of old hardware, beneficial though that is. If you're running a service on a cloud or hosting provider then being able to use a 1G server instead of a 2G server will halve your costs, and that's a very good thing to be able to do.Peter Tribblehttp://firstname.lastname@example.com:blogger.com,1999:blog-9726833.post-6456092389456727802017-05-29T14:59:00.000+01:002017-05-29T14:59:25.973+01:00Tribblix SPARC progressTribblix is one of the relatively few illumos distributions that runs on both SPARC and x86 hardware.

There are valid reasons for the lack of SPARC support in other distributions. For those backed by commercial entities, it makes no sense to support SPARC as they don't have paying customers to foot the bill. Which leaves SPARC support firmly in the hobbyist realm.

Even in Tribblix, SPARC support has lagged the x86 version somewhat. Again, for entirely predictable reasons. While I do have SPARC hardware, it's relatively slow, noisy, power hungry, and heat-producing compared to my regular x86 boxes. And my day to day use is my x86 workstation, so that drives a lot of the desktop work.

But SPARC development of Tribblix hasn't stopped. Far from it, it's just naturally slower.

The current download ISO image at this time is still Milestone 16. Just to clarify the versioning here - that means it was built from exactly the same illumos commit as the corresponding x86 release. Because it took a little longer to get ready, the userland packages (such as they were) tended to be a bit newer.

There have been 3 more Tribblix release on x86 since then. Over the winter (when it was cold and the heat output from the T5140 I use as a build server was a good thing) I tried building updated illumos versions. The T5140 I'm using to do the builds is running a cobbled-together frankendistro of bits of Tribblix, bits of OpenSXCE, some random bits from other people working on SPARC, and a whole lot of elbow grease. I managed to build illumos at the m17 and m18 release points, but m19 was a step too far (some of the native stuff assumes that the host OS isn't terribly antiquated). What this means is that I need to replace that by a current system, and get a properly self-hosting illumos build.

That modernizes the underlying illumos components a bit. What about the rest of the system? The primary effort there was to replace the old core components that had been been borrowed from OpenSXCE while bootstrapping the distribution in the first place with native packages (and that are then up to date and match the x86 build). Some of the components here are pretty crucial - zlib and libxml2, for instance. At one point I messed up libxml2 slightly - not enough to kill SMF (which would be a big worry) but enough to stop zones working (which, apart from indicating that I had broken it, also left me without an imortant test mechanism). Rebuild everything enough times and the problem eventually cleared.

I also had a go at getting my SunBlade 1500 workstation working. It's not terribly quick, but it's quiet enough and sufficiently low power that I can have it running without negatively impacting the home office. That was a bit of a struggle, the bge network driver currently in illumos doesn't work - I assume I'm seeing bug 7746 here, but the solution - to use an older version of the driver - works well enough. With that box available I not only have more testing available but also a lightweight machine that I can use to keep the package backlog under control.

Graphics on SPARC is an interesting problem. OK, so I don't expect this to be a priority, but it would be nice to have something that worked. The first problem I found (a while ago) was that some of the binary graphics drivers wouldn't work at all. For example, the m64 driver (which is what might drive the graphics in my SunBlade 2000) uses hat_getkpfnum which was removed from illumos courtesy of bug 536. Graphics drivers that load often simply don't work, and getting an X server to start is a bit of a nightmare. After far too much manual fiddling I did manage to get a twm desktop running on the aforementioned SunBlade 1500, but don't expect native graphics support to improve any time soon.

Applications is another matter, there's no reason you couldn't run at least some applications on a SPARC system and display them back on your desktop machine. After all, X11 is a network display protocol (despite all the effort to eradicate that and turn it into a local-only display protocol). Or run a VNC server and access that remotely. So I've started (but not finished) building up the components for useful applications.

I haven't yet got an ISO image. That's likely to be a while, but if you have an existing SPARC system running Tribblix m16 then the upgrade to m18 ought to work. Although I would recommend a couple of changes to the procedure if you're going to try this:

Refresh and update everything: 'zap refresh ; zap update-overlay -a'

Download the current upgrade script from github and run that script in place of 'zap upgrade'

After booting into the newly updated BE, refresh and update everything again, just to make sure you're up to date

Peter Tribblehttp://firstname.lastname@example.com:blogger.com,1999:blog-9726833.post-48135350406728896372017-04-29T11:47:00.000+01:002017-04-29T11:47:52.963+01:00OmniTribblixIn Tribblix, it's a basic principle that I ship upstream software unmodified. I don't impose my own views on installation layout, nor do I customize it. Generally, I apply patches only to make stuff compile.

This means that what you see in Tribblix is exactly what the upstream author intended, and not some distro-specific bastardization of it.

It also makes my life easier, I don't have to maintain patches, and updating software is much easier if it's unmodified.

In particular, I use an absolutely vanilla illumos-gate. (For a long time it differed only in that I had the fix for 5188 applied, relevant because Tribblix actually uses SVR4 packaging, but now that's integrated I don't even need to do that.)

Again, this makes my life easier. (When you're maintaining a distro on your own in your spare time, making decisions that simplify your job is essential.)

But it also has another benefit: because I have no "special" features that I've added, I'm not tied to one particular version or variant or commit of illumos. Any version of illumos-gate will do just fine. When it comes time to make a release, I just clone the gate, build, and go.

What I could do, then, is build an instance of Tribblix atop some other fork of the gate. For example, illumos-omnios.

I did just that, built the gate (it needed a couple of changes to Makefiles because of the way that perl and snmp are slightly different in OmniOS than it is in Tribblix), created packages, built an ISO, booted and installed it in VirtualBox.

As expected, it just works.

But just demonstrating that it works isn't really the reason I wanted to do this. What I'm really after is the LX brand, which has been integrated into current OmniOS.

Installing an LX zone requires a Linux image. The original (Joyent) work was for their own deployment mechanism, using ZFS images. As soon as it was available in OmniOS the first thing I did was use tarballs, which OmniOS now supports. The easiest way to create a Linux image is to create a Docker container the way you like it, and then export it to a tarball. I did that for Alpine and installed a zone based on that.

Then you can do very simple things like:

# zlogin lx1 /bin/uname -a
Linux lx1 4.4 BrandZ virtual linux x86_64 Linux

It's an attractive idea to simply use this as the base for the next Tribblix release. However, that requires illumos-omnios to be supported in the long term, which is currently at risk.

Peter Tribblehttp://firstname.lastname@example.com:blogger.com,1999:blog-9726833.post-28707205170504608592017-04-12T20:10:00.000+01:002017-04-12T20:10:37.371+01:00Noisy TribblixI've had a couple of Tribblix users ask me why audio doesn't work.

This was something I had noticed myself, and the reason was not that audio was in some way broken, but that the permissions on the audio devices were wrong - owned and only writeable by root.

Now I only wanted to actually get any audio out on fairly rare occasions, so a quick chown wasn't that much of an imposition. But it obviously needed fixing properly.

My assumption here is that most desktop users will be logging in through the SLiM login manager. So all I need to do is fix the permissions just before it calls setuid() to the logged in user. And then reset them back once the user is done.

Now, I could have made up a bunch of chowns myself, or written a helper. There's actually code in SLiM to call ConsoleKit - but I don't have ConsoleKit, and don't really see the need to maintain a port of it just for this.

But illumos already has the capability to do this, and the normal login mechanisms use it. There's code in libdevinfo that sets the permissions according to the rules laid out in the /etc/logindevperm file. So the code is really just a call to di_devperm_login() and di_devperm_logout(), and all is well.

This also fixed another irritating bug - I can now eject memory sticks as myself, without needing to be root.

The next thing that happens, of course, is that it doesn't take very long to realise that Twitter has a lot of videos that play automatically. So I'm sitting there and I can hear either the internal loudspeaker or my headphones warbling away.

So the next thing I need is a way to shut the thing up. Historically, I used the old CDE sdtaudiocontrol, which was pretty good. (In general, I detested CDE as a desktop, the mailer and calendar were decent enough for their time, and the audio control was the only other thing I used much.) I use Xfce as my desktop, it used to have xfce4-mixer but that's now unmaintained and deprecated (and I removed that as part of the migration from gstreamer-0.10 to gstreamer1). Which pretty much leaves the command line audio utilities in illumos, specifically audioctl. I've added the package so users who update will automatically get that as well.

The command

audioctl set-control volume 0

silences things, while

audioctl set-control volume 75

puts the volume back to normal. I've created aliases mute and unmute for those. A more sophisticated approach would be to save the volume and restore it afterwards, but this is enough for now.Peter Tribblehttp://email@example.com:blogger.com,1999:blog-9726833.post-25799025695630221782017-03-12T14:17:00.001+00:002017-03-12T14:17:55.765+00:00How old are illumos man pages?I've recently been looking at improving the state of the illumos man pages.

One thing you'll notice is that the date on some of the manpages is old - really old, some of them are dated 1990. That presumably means that they haven't been modified in any meaningful sense for a quarter of a century.

(By date here I mean the date displayed by the man command. Which isn't necessarily the date somebody last touched it, but should correspond to the last meaningful change.)

The distribution of dates looks like this:

As you can see, the dates go all the way back to 1990. There's not just the odd one or two either, there are a decent number of man pages that comes from the 1990s.

There are some obvious features in the chart above.

There's a noticeable spike in 1996, which of course significantly predates illumos or OpenSolaris. It's not entirely obvious why there should be a spike, but 100 of those man pages are related to libcurses.

I suspect the dip in 2005 is a result of the launch of Solaris 10, when everyone had a bit of a breather before development kicked into full gear again.

Then there's a drop in 2010. In fact, there's just a single page for 2010. That's when the OpenSolaris project closed, so there was little work done at that point. Also, our man pages were only integrated into illumos-gate in early 2011 (prior to which they were kept separate), and it's taken a while for man page updates to pick up again.

Of course, one reason the man pages are so old is that the software they're documenting is old. That's not necessarily a bad thing, if it ain't broke don't fix it as they say, but there is a certain amount of total rubbish that we ought to clear out.Peter Tribblehttp://firstname.lastname@example.com:blogger.com,1999:blog-9726833.post-43567256448077781872017-02-02T21:11:00.000+00:002017-02-02T21:11:43.914+00:00Creating a Tribblix package repositoryI've previously described how Tribblix packages are built.

The output of that process should be a zap file, which is an SVR4 package in filesystem format, zipped up. The file naming convention is

PKG_NAME.VERSION.zap

Where PKG_NAME is the SVR4 name of the package (you can define aliases to be more user-friendly), VERSION is obvious - but has to match the installed version as shown, for example, by 'pkginfo -x'.

You can install those packages directly. There's a little helper /usr/lib/zap/instzap that will automate the process of unpacking the zap file and running pkgadd on it, or you can do it by hand.

You don't have to use my tooling. If you've got a scheme for building SVR4 packages already, then you could simply convert those.

However, what you would really want to do is stick the packages in a repository somewhere, so they can be accessed using the normal zap commands.

In Tribblix, a repository is just a web-accessible location that contains the zap files and minimal metadata. The metadata is the catalog and a list of aliases.

The aliases file contains lines with two fields separated by a vertical pipe "|". The first field is the friendly alias, the second the real name of the package. If you want multiple aliases, just add multiple lines to the file.

The catalog file contains lines with 5 fields separated by a vertical pipe "|".The first field is the package name, the second the current version, the third a space-separated list of packages this package depends on, the fourth the size of the file in bytes, the fifth is an MD5 checksum of the file. There's a trailing "|" in this case to terminate the line. The size and checksum are used to verify the download was successful and didn't corrupt the file. If you want to be sure that it's actually the package it claims to be then packages can also be signed.

Here's an example line in the catalog:

TRIBabook|0.5.6.0|TRIBreadline|66923|d98f77cfb3e92ee6495e3902cc46486f|
So the TRIBabook package has a current version of 0.5.6.0, depends on the TRIBreadline package, and has the given size and checksum. It's in the file called TRIBabook.0.5.6.0.zap.

The build repo has scripts that create the catalog and aliases files, which I use for convenience. They do make some assumptions about my package build pipeline, so it might be easier to manage the files by hand.

So, having got a nice place on the web that has your packages and a catalog and some aliases, how does Tribblix know to use it?

Assuming your repository is called myrepo, you need to add a file /etc/zap/repositories/myrepo.repo, containing something like the following (which is the configuration for the main tribblix repo in the milestone 18 release)

NAME=tribblix
DESC=Tribblix packages
URL=http://pkgs.tribblix.org/tribblix-m18/
SIGNED=tribblix

If you aren't signing packages (and how to add the keys to the client is an exercise for the reader) then omit the SIGNED line. The NAME in this case would be myrepo, the DESC is whatever you make it, and the URL points at the directory containing the files.

That's almost it. The other thing you need to do is add the repository to the list of repos that zap will search, which is held in the /etc/zap/repo.list file. By default, this contains

100 tribblix
200 illumos
300 oi

That's a simple list, and the number is a priority, lower numbers have higher priority - they're searched first.

(What the priority scheme means is that if you have a package with the same name in multiple repos, the one in the highest priority repo is the one that will be used. For example, the ssh packages used to come from illumos. As we've moved to openssh, all I had to do was put the new openssh packages into the tribblix repo with the same package names they had before and they were installed instead. Of course, you have to be careful that the replacement package delivers the correct functionality, especially if it's delivering shared libraries.)

If your package names are unique to you (for instance, if you name them MYPKGfoo rather than TRIBfoo) then the priority doesn't matter. If you're deliberately trying to override some of the built-in packages, then your repo has to be the highest priority one so it gets searched first.

If you then utter 'zap refresh' it should go and retrieve the catalog and aliases files and then be set up to use them.

And yes, it would be nice if zap had the facility to manage repos for you - that's planned, I just need to implement it.

Peter Tribblehttp://email@example.com


text:
I had the last week off work, and the weather was atrociously hot, so I wound up watching a bunch of movies. Here's the run-down.
Shutter Island – The surprise wasn't all that surprising, but it was still worth the journey. With this film, Martin Scorsese delivered a masterful psychological thriller, and DiCaprio did a pretty good job of convincing me that he can act. (8 out of 10)
Grown Ups – This was exactly what I expected from the ads. This movie pulled together a great cast of some of the best comedy actors out there: Adam Sandler, Kevin James, Chris Rock, David Spade, and Rob Schneider. But somehow when you put this all together, it was less than the sum of its parts. This cast can't help but put together a funny movie, but with that kind of comedic firepower, I should have been doubled over laughing, not just chuckling here and there. (3 out of 10)
Eclipse – Yes, I'll admit to having read the "Twilight" books, and to having watched the first two movies. This wasn't a great movie by any stretch of the imagination, but it was much better than the first two movies. The only downside was Bryce Dallas Howard, who seemed in over her head as Victoria; though she didn't get much screen time, I much preferred Rachelle Lefevre in this role. (5 out of 10)
Knight and Day – It's been almost 10 years since I enjoyed a movie that starred Tom Cruise (Minority Report in 2002). This one won't revive his career, but it was a satisfactory escape on a hot day. It was a totally predictable action-comedy with a smidgen of romance thrown in. (6 out of 10)
Toy Story 3 – Although their reputation is built on outstanding computer animation, the truth is that Pixar has always been focused on telling compelling stories. Computer animation is their chosen medium, but they consistently put the message first. The short film, "Day & Night" that preceded Toy Story was charming, but the feature film drew me in so quickly and completely that I had soon forgotten it. A few critics have called this the best of the three Toy Story movies. I haven't seen the other two in many years, so I'll refrain from that judgment, but it was certainly the best movie I've seen in a couple years. (10 out of 10)
Blown Away – After Toy Story 3, it was inevitable that the next movie would come up short, and boy did this one! Tommy Lee Jones's Irish accent was terrible, and Jeff Bridges performance was truly forgettable. It was like watching Speed with worse acting (yes, worse than Keanu Reeves) and a few more explosions. (3 out of 10)
Iron Man – Another disappointment, the more so because I had heard good things about this one. This was good for about the first 30 minutes, but then went quickly downhill. The middle of the movie spent far too long showing him trying to perfect the Iron Man suit, and the end was too cartoonish after the rest of the movie taking a realistic approach. (4 out of 10)
The Italian Job – Low expectations, high satisfaction. I'd not really heard anything positive about this movie, but it turned out to be a reasonably entertaining crime caper. My only complaint is that I'd really like to see Seth Green cast as someone other than the tech guy. (6 out of 10)
Also, mixed in there, I saw a few previews for movies that might be promising. The most promising of the bunch looks like Inception, directed by Christopher Nolan and starring Leo DiCaprio. Other intriguing previews were for The Green Hornet, The Other Guys, and Salt.

subset: commoncrawl


text:
Channel Four in the UK has a fairly impressive in-house creative team, as evidenced by its magnificent Paralympics campaign from a few years back. And now, to launch its new ondemand platform All4, it's peered 100 years into the future and predicted a same sex royal wedding, robotic horse racing and a series of mutant hosts.
Not surprisingly, the first regal same sex kiss is getting a fair amount of attention online and, interestingly, according to Pink News, Channel 4 recently brought into effect a "tough new diversity quota system, which calls for six percent of employees to be LGBT by the year 2020 – more than double the 2.4% of Channel 4 staff that currently identify as LGBT".
And with research showing there are plenty of benefits if employers allow true authenticity in the workplace, something explained in ex BP chief executive John Browne's book The Glass Closet, it's good to see businesses making some changes.


text:
Premier Tasting Tour for 2
- Süre: 5 Saat (Tahmini)
- Ürün kodu: PVF0QN
Our Premier Tasting Tour for couples is a private tour and tasting experience which is perfect for discerning wine tasters - Aficionados of fine food & wine! who like to taste, select and purchase premium wines to take home whilst hoping to avoid the crowds. It will be an opportunity to learn first hand from a winemaker or a viticulturist about the art of producing quality wine. Travel in a a modern twin cab 4x4 with opportunity to ride through the vines depending on vineyard operations. (This tour is not suitable for first time wine tasters or children)
Tour includes
- expert tour guide
- visit and tasting at 4 award winning cellar doors
- max group size 3 on tour.
- a sumptuous ploughmans platter, picnic box or cellar door platter.
- a guided tour through a vineyard and winery (available depending on vineyard operations).
- complimentary bottle of wine per tour booking to take home.
- pickup and drop off from your local accommodation
- gift vouchers can be purchased online with a 12 month validation from date of purchase


text:
Posting ID : A1068546930
Date Posted : 2015-08-19
Category : Customer Service
Do you love pets and enjoy walking and being on the go? We have the perfect gig for you. Joyful Pawz, a locally owned and operated pet care services company is looking for a pet lover to work as a pet sitter / dog walker.
Please respond to this posting with resume, cover letter and three professional references. All three of these items are needed in order to be considered.
Again COVER LETTER, RESUME AND REFERENCES. Please email them.
-great atmosphere -- outside!
-great pay
-have fun and earn money
-Have a flexible schedule. 🔴 Must be free from 8am-5pm during the day!
-Must have open availability. Walks will be on weekends (day and night), however you will be working daytime hours during the week - hours worked will vary and depend on client load
-Must be availability to work holidays.
-Must have flexibility and availability. This is an on-call position. Sometimes we need to fulfill walks/visits on very short notice.
-be physically able to walk dogs of all sizes (sometimes up to 3 at a time) for 30-45 minutes straight
-have experience loving animals
-highly dependable, reliable and trustworthy.
-excellent communication and interpersonal skills
-compassion toward people and animals
-servant's heart, willingness to go above and beyond for people and pets
-reliable transportation
-smart phone with texting, camera and email capabilities- THIS IS A MUST!🔴


text:
Mixtape: Gucci Mane – Trap God (Download)
Gucci Mane releases his latest mixtape on Q's new site WorldstarTapes. The mixtape features Waka Flocka, Rick Ross, Meek Mill, Birdman and more.
Get the Gucci Mane "Trap God" Tracklist & Download after the jump.
Gucci Mane – Trap God Tracklist
1. Intro produced by Lex Luger
2. "Head Shots" feat Rick Ross produced by Tarentino, 808 Mafia
3. "Money Habits" feat Young Scooter produced by 808 Mafia
4. "Crazy" feat Waka Flocka Flame produced by Drumma Boy
5. "Get Money Nigga" feat Meek Mill produced by 808 Mafia
6. "Rolly Up" feat Young Scooter, Waka Flocka Flame produced by Mike WiLL Made It
7. "Fuck The World" feat Future produced by Mike WiLL Made It
8. "Act Up" feat T-Pain produced by T-Pain
9. "Never See" feat Verse Simmonds produced by Shawty Redd
10. "That's That" feat Kevin McCall produced by Kevin McCall
11. "Servin" produced by 808 Mafia
12. "Shooter" feat Young Scooter, Yung Fresh produced by Zaytoven
13. "Dead Man" feat Young Scooter, Trae Tha Truth produced by Metro Beats
14. "Baby Wipes" feat Waka Flocka Flame produced by Zaytoven
15. "Don't Trust" feat Young Scooter
16. "Suckaz" produced by Shawty Redd
17. "Gas and Mud" produced by C4
18. "Fuck Something" feat Kirko Bangz, Waka Flocka Flame, Young Scooter produced by 808 Mafia
19. "I Fuck With That" produced by Mike WiLL Made It, Southside, C-Note
20. "Get Lost" feat Birdman produced by Detail
Download: Gucci Mane Trap God
Leave a Reply


text:
WASHINGTON — H.B. Atkinson paid $6,000 for his satellite dish TV system, set it up in his front yard, and shielded it from view with trees to pacify the neighbors. But as soon as he and six of those neighbors gathered around his television set to watch a boxing match, Atkinson became the victim of a technical knockout.
''I turned on the set, and pretty soon a voice came through and said this fight is scrambled, there's no use to keep on trying to get it, because you're not,'' said Atkinson, of Vienna, Va.