One Man's Journey Into Linux

F#, Erlang and GPUs

2009-12-16T11:20:00.018Z

Christmas time in olde Rochester towne, and as is traditional there is a fair in the grounds of the castle. I like carousels, and so do my kids, but here’s the thing: you can’t just go up to the carousel and go for a ride. The first thing you have got to do is wait for the carousel to stop and wait for the other people to get off. Once you get on you still don’t go anywhere; the guy running the carousel doesn’t want to run it half empty, so you have to wait for it to fill up. Only once it’s filled up, or if no one else turns up, do you get to ride. Finally, everyone gets the same ride: the music plays, the cranks turn, the horses go up and down, the same for everyone.

At ForensiT we’re right at the start of planning a major new project. For the first time, we’re thinking seriously about F#. F# is now a fully featured .NET language and ships as part of Visual Studio 2010. But why even consider F# instead of, say, Erlang?

When I was kid, a long time ago before the computer was personal, there used to be a lot of moaning a about the Japanese. All they did was take the products that we had invented and make them cheaper... oh, and... er... better. From a purely financial point of view, it’s not such a bad business model. Microsoft have a reputation for doing much the same thing...

Here’s some F# code that Luca Bolognese demonstrates in his excellent PDC 2008 presentation:

Here’s some Erlang code that does the same thing

Is it just me, or are there some similarities here? Except this is where F# begins, it is not where it finishes. It is always vital to keep in mind that Microsoft has, to all intents and purposes, unlimited resources. If Microsoft are going to do a functional programming language then, eventually at least, they are going to come up with the most powerful functional programming language there is.

F# is a .NET language, that means that a F# programmer has complete access to all the .NET code that Microsoft and others have developed over the last decade. More importantly perhaps, a F# programmer will have access to all the future innovations in the .NET framework. What I’m thinking of here in particular is future support for multi-core processors and concurrency. The speed at which Microsoft can assimilate and support new hardware and software technologies is just way, way beyond what even well supported projects like Erlang can do. If you choose Erlang over F# you have to be aware that you are giving all this up.

One technology that is already pressing is the GPU and programming languages like CUDA. There are already third party .NET libraries for CUDA and OpenCL. Without doubt, native support will follow. For Erlang to support GPU processing would require the Erlang runtime to be rewritten. Can we really expect the Erlang developers to spend precious time and effort on a technology which may not stand the test of time?

Notwithstanding the general point about future innovations in the .NET framework, how desirable would GPU support in Erlang actually be? The beauty of Erlang is in its message passing: multiple processes (“agents” in F#) signal each other when they want something done, or when they have a response. Just as a thought experiment, let’s imagine a moderately sized Erlang application with many processes. Let’s also assume that that the vast majority of these processes carry out the same task in response to the same message; in other words, they are different instances of the same code. The application runs and messages start flying around; realistically, the different processes receive messages at different times.

What happens when a process receives a message with some data that it has to process? Here’s the first point where GPU code might be useful. If the task was sufficiently complex, and sufficiently parallelizable, being able to call GPU-code would be a big help. (I don’t really have a clear idea about what sort of task this would be, but the calculation of a Mandelbrot set is the kind of thing I have in mind.) You can of course call C functions from Erlang, so some CUDA interop is already possible; leaving aside the question of how efficient it would be.

But what if the task each process carries out is not complex? Could GPU code make the basic infrastructure of Erlang, the framework of messaging passing, more efficient? Let’s go back to our thought experiment. A message comes in with data that needs to be processed. GPUs work with batches of data, they are fundamentally SIMD (Single Instruction, Multiple Data) devices. To take advantage of the GPU, the process would need to hand the data off to some kind of scheduler and wait. The scheduler wouldn’t want to run the GPU half empty, so it would wait for a certain time until either it had enough data to use the available GPU cores, or the waiting time was up. Once the ride was over, the GPU would then have send a result back to each of the waiting processes.

The question then, becomes one of whether it is faster to wait for the batching and unbatching of the GPU code, or faster just to wait for a time-slice on a CPU core. I certainly don’t have the answer. What we can say is that our thought experiment is very GPU-friendly. If you only have a small number of processes, or the processes are doing different things, a GPU is not going to give you any advantages.

The truth is that functional, message-passing languages and GPUs occupy two different parts of the parallel computing world. It is not as obvious, as I first thought it was, that they actually need each other.

Heads in the Clouds

2009-03-23T10:41:00.012Z

I've been trying to understand Cloud Computing. No, let me rephrase that, I've been trying to understand what's different about Cloud Computing. Microsoft are busy prepping their Azure Services Platform in readiness for taking on Amazon Web Services, Sun, Google and the rest. Even in the worst economic climate since the 1930s, these companies are spending tens of millions of dollars on building massive data centres that will host "The Cloud." Why?

Microsoft have some Case Studies to help those us having problems with the whole Cloud thing get it. Here's an extract

Ok. But is there really anything new here? What happens if we replace the word "Cloud" with something more prosaic and old-fashioned, something like the word "Internet":

You would be forgiven for thinking that these two paragraphs mean exactly the same thing. But they can't do. "Cloud Computing" must mean something, right?

Well, I'm not going to labour the point. Yes it does mean something different: it means having your code run on someone else's hardware. It also means having your data stored on someone else's hardware. In the example above, Infosys don't have a SQL database running on an Internet server that they own, they have a SQL database running on an Internet server that Microsoft owns. Why would they do something like that? Because of the benefits that Microsoft and Amazon and everyone else claims for Cloud Computing: start-up costs are minimal - you don't have to buy a server infrastructure up front, just rent what you need and scale up (and up) as you need to; redundancy is massive - if the data center hosting your application falls over, your application is instantly shifted to another data centre in another time zone, or in another country, or on another continent. As Microsoft says, it is about "having global-class credibility."

So the benefits are real. But that really is it. There's no new code, no new Internet. As Oracle's Larry Ellison said last year "The interesting thing about cloud computing is that we've redefined cloud computing to include everything that we already do." He went on to say "The computer industry is the only industry that is more fashion-driven than women's fashion."

What Cloud Computing emphatically is not is "where thousands of computers cooperate through the Internet to compute a result. Google’s proprietary MapReduce* framework is the standard bearer for this..." This definition comes from a recent Intel primer on Parallel programming. The author is either being idealistic or naive. Let's be generous: this is what we would like Cloud Computing to be - unlimited access to a hyper-computer where we only pay for the time we use. But it is not what is on offer. Yes, you can set up multiple virtual servers on AWS to form a cluster, but they might all be running on the same physical box! Never forgetting, of course, that the interconnect is the cluster.

Neither is Cloud Computing "where IT power is delivered over the Internet as you need it, rather than drawn from a desktop computer." At least, not yet. This is a more generalized and subtle, but just as idealistic definition. Notice the word "desktop" - we're not talking about replacing web servers here, which is what Amazon and Microsoft are punting. In this definition the "standard bearer" is Google's Gmail. What is replaced is the mail program on your PC. But there is nothing remotely new just in web applications like Gmail that requires the term "Cloud Computing." To have any force, this web applications paradigm would have to advance to the point where people were accessing not just their email, but the vast majority of their applications from multiple consumer devices (like mobile phones, TVs and digital cameras) which didn't just supplement the PC, but actually subsumed it. This promise is implicit in Cloud Computing, but again it is not what's on offer - and just building data centers won't in itself make it happen.

This definition comes from The Guardian's article on GNU founder Richard Stallman's now famous attack on Cloud Computing. Stallman argued that Cloud Computing was "worse than stupidity" and "simply a trap aimed at forcing more people to buy into locked, proprietary systems that would cost them more and more over time." Crucially he argued that "One reason you should not use web applications to do your computing is that you lose control..." (My emphasis.) This issue of control has been heavily debated. (There is one example here.) I just want to comment about one aspect of control.

In their book Fire in the Valley about the birth of the PC industry, Paul Freiberger and Michael Swaine talk about the feeling programmers, technicians and enginneers had in the 1960s and 70s of being "locked out of the machine room." The Personal Computer changed all that; it was genuinely subversive, putting technology and computing power in the hands of anyone who wanted to use it. Those of us who build Beowulf clusters work in the same tradition. The danger of Cloud Computing is that once again the machine room door will be slammed in our faces.

Two more nodes

2009-03-21T16:37:00.011Z

I've added two additional nodes to my cluster. The new nodes have Asus rather than Abit motherboards and, unfortunately, CentOS 5.2 doesn't support the Nvidia nForce 630a chipset. So I was left with a choice of either trying to build the necessary drivers, or finding an alternative. In the end I decided to give CAOS Linux another try. There seems to be no link on the website anymore to download a VNFS image, but if you Google you can find the FTP site easily enough. I used version 1.0 rather than the RC1 version I had tried previously. It is still a "bare bones" distribution - it doesn't even come with vi - but it worked. I even managed to get Python installed from source using "make DESTDIR", which didn't happen before. There are a few quirks: I don't seem to be able to specify which eth device gets which MAC address, which means I can't use it on all the nodes, but for the new nodes it's fine.

I wasn't going to go on about performance results anymore, they are just a bit of fun after all. However, having added the two new nodes they had to be tested :-) The new nodes have Athlon 64 X2 5200+ (2700MHz) processors instead of 4200 (2200MHz) processors, and 4Gb RAM instead of 2Gb RAM. So a big increase in performance right? Er... no. There is an increase in performance - from 13.3 to 14.25 Gflops on a two node test, but this modest increase in output is offset by a big drop in computational efficiency: down from 75.6% to 66%. What's going on?

On the original nodes, with 2Gb RAM, pretty much all I had to do to find the maximum performance was keep increasing the problem size (N) until xhpl ran out of memory and crashed. The bigger the problem size, the better the measured performance. With the nodes with 4Gb RAM, the measured performance peaks before (way before) we run out of memory. What we see is an initial steep increase in performance as the problem size increases, leading to a peak, and then a slow decline.

It's as if the processors can't make effective use of all that extra memory. Or perhaps we've hit the limit of what the bonded Gigabit Ethernet interconnect can deliver.

Supersize me

2009-02-19T12:01:00.012Z

One simple way to improve network performance is to enable Jumbo packets. Jumbo packets - or frames - are, as their name suggests, bigger than normal network packets. This means that more data can be transferred across the network at once: up to 9000 bytes instead of 1500 bytes. Because the computer CPU is not being interupted as often for the same volume of data, CPU utilization is increased along with throughput. What's more, on Linux enabling Jumbo frames is just a matter of setting the MTU (Maximum Transmission Unit) size on the network device:

#ifconfig eth1 mtu 9200

To make this permenant, you just need to add MTU=9200 in the /etc/sysconfig/network-scripts/ifcfg-eth1 file. On Perceus this is done in the /etc/perceus/modules/ipaddr file. The only other thing you need is a switch that has Jumbo frame support enabled. That done, you can test everything is working by pinging with a Jumbo packet size:

ping -s 9000 -M do 192.168.4.16

Easy! So why haven't I done this before? I originally bought D-Link DGE-528T Gigabit cards for the cluster. These were cheap, but seemed to offer everything I needed - including Jumbo frames support. The maximum MTU was only 7200 bytes, not the more usual 9200 bytes, but I could ping the cards with packets of 7000 bytes. What I found, however, was that I could not run the HPL benchmark xhpl with Jumbo packets of any size enabled. I could start up the mpd ring, and run mpdtrace, but when I actually tried to run xhpl it crashed:

Fatal error in MPI_Send: Other MPI error, error stack:
MPI_Send(173).............................: MPI_Send(buf=0x555d1c0, count=1, dtype=USER, dest=2, tag=2001, comm=0x84000001) failed

Setting MTU back to 1500 again fixed the problem straight away. So is this a driver problem or a xhpl problem? I'm guessing it is a driver problem, but life's too short...

I made the decision to junk the D-Link DGE-528T cards and go with Intel PRO/1000 MT cards - another eBay triumph! This time there were no problems with xhpl crashing with Jumbo frames enabled. Interestingly, you enable Jumbo frames on the bonded device, bond0, not on the slave devices, and the bonding driver does the rest. Very cool.

Output on a four node test rose from 24.08 to 24.86 Gflops (70.6% compEff.) Running over all six nodes of the cluster produced 36.13 Gflops (68.4% compEff.)

I'm going to leave it there, and not spend any more time tuning. It is time to put the cluster to work running distributed Erlang. What performace critera are most relevent to this task is another question altogether...

I think we're bonding...

2009-02-19T11:51:00.001Z

Having doubled the measured efficiency of my cluster by linking HPL to ACML, I looked around for something else that could take me closer to the 75% efficiency achieved by "real" clusters. One obvious area to look at is the "interconnect" - the network that connects the nodes together to actually make the cluster. Top clusters use Infiniband for the interconnect, but - even though prices continue to fall - Infiniband is still out of reach for just about all Beowulf clusters like mine. Whether Infiniband will ever be widely used for these kind of clusters is open to debate: perhaps we'll all be using 10GE cards in a few years. For now though, we have to get the most from our 1 Gigabit network cards.

The highest rated Gigabit cluster on the Top 500 list comes in at number 78. It's computational efficiency is only 53.26%. The average computational efficiency of all Gigabit clusters in November 2008's Top500 list is even lower at just 50.52%. The most efficient is 63.04% efficient, the lowest at 40.34% - and that is with 5096 cores! That is a lot of wasted clock cycles. Next to these figures, the 60.3% efficiency I achieved on my four node cluster doesn't look too bad. However, the efficency was nearer 70% with two nodes, so as I add nodes it looks like overall efficency will fall. Reason enough to try and maximize performance.

One of the things I wanted to investigate from the very start was network card bonding, or - perhaps more strictly - network card aggregration. This is where you "bond" two (or more) network cards together to create a single logical network interface. In other words, two network cards share the work of one, thereby giving you an increase in network performance.

Linux support for network card bonding is definitive. Not only that, it is very easy to impliment. (There is a good article here.) From a perceus cluster point of view, you just need to add the lines

alias bond0 bonding
options bond0 mode=802.3ad miimon=100

to the /etc/modprobe.conf on the vnfs image, and then edit the /etc/perceus/modules/ipaddr file. (This is used by the perceus 50-ipaddr.pl perl script to automatically generate the the ifcfg* files in /etc/sysconfig/network-scripts/):

n00004 bond0(USERCTL=no&BOOTPROTO=none&ONBOOT=yes):192.168.4.14/255.255.255.0/192.168.4.1 eth0(HWADDR=00:18:37:07:FB:3A):[default]/[default] eth1(MASTER=bond0&SLAVE=yes): eth2(MASTER=bond0&SLAVE=yes):

This creates a bond0 interface from eth1 and eth2, the "slave" interfaces.

Linux, then, supports bonding extremely well - but Linux is only one half of the connection. At the other end of the wire is the switch. I was using a Dell PowerConnect 2724 switch which advertises itself as supporting "Link Aggregation." Unfortunately "link aggregation" can mean different things to different people. To me it means "using multiple network cables/ports in parallel to increase the link speed beyond the limits of any one single cable or port..." But it can also refer to using bonded network devices for load balancing and fail-over protection. It is in this latter sense that the PowerConnect 2724 switch supports "aggregation." It doesn't support cabling devices in parallel to increase performance. For that, you need LACP (the Link Aggregation Control Protocol) which the Dell 27xx switches do not support; switches that do are twice the price. It was a case on getting on eBay, where I found a "pre-enjoyed" PowerConnect 5324 which does support LACP.

So was it worth it? On a two node test, output went from 12.22 Gflops to 12.93 Gflops; that doesn't sound a lot, but it means an increase in computational efficiency from 69.3% to 73.5%. On a four node test, output went from 21.24 Gflops to 24.08 Gflops; this is an increase in computational efficiency from 60.3% to 68.4%. So the benefit of aggregation is much greater on the four node cluster. This makes sense: by using aggregation we are flattening the performance curve; we are alleviating the bottleneck effect of the additional interconnects.

Network cards are cheap - at least relative to the cost of memory or processors. If you have a switch that supports LACP, link bonding definitely seems worthwhile.

VLANs to the rescue

2009-02-13T10:00:00.003Z

I've overcome the two issues that caused me so much frustration. The problem of node network devices picking up different names (eth0, eth1) between boots was fixed by specifying the MAC addresses in the /etc/perceus/modules/ipaddr file:

eth0(HWADDR=00:18:37:07:05:A8):[default]/[default]

My thanks go to Greg Kurtzer for helping me get this fixed.

The general network problems I was having with provisiond, amongst other things, were resolved by splitting my "management" network (with 192.168.3.x addresses) and "application" network (with 192.168.4.x addesses) over two VLANs. This was easy to do on the switch and really cleaned things up. Now the node will always boot from the network device I want it to boot from, and management traffic doesn't wander over the application network.

DNS Disaster!

2009-01-25T15:35:00.001Z

I needed to make a change to the configuration of one of the network cards on the "host" machine. (I tend to use the word "host" rather than "head node" because I don't really think of it as a node - it's not where any applications will run. I suppose I could also use the more Perceus-like "master".) Because I come from a Windows background I still tend to look for a GUI rather than use the command line, so I ran system-config-network. Big, big mistake. Suddenly I can no longer ping any of the cluster nodes by name.

When I set up Perceus I'm sure I did no more than follow the instructions in the User Guide and add

nameserver 127.0.0.1

to the /etc/resolv.conf file on the host. (I also had to put the host address in the /etc/resolv.conf file on the nodes, of course: if your nodes boot slowly, you probably forgot.) system-config-network wiped my resolv.conf file, so I added the nameserver line back in and then ran /etc/init.d/preceus reload. No good. Looking in /var/log/messages (via gnome-system-log, ofcourse!) showed this when I did the reload:

perceus-dnsmasq[5286]: ignoring nameserver 127.0.0.1 - local interface

But is this a problem? I read somewhere that it isn't, however, subsequently I've not seen this message repeated when things have worked.

It struck me that I was assigning static IP addresses to my nodes in /etc/perceus/modules/ipaddr. Does perceus-dnsmasq pick these up, or do I need to add them to the /etc/hosts file? I didn't need to before. It also struck me that the messages log was filling up with entries like

perceus[5295]: ERROR: Unknown node just tried to contact us! (NodeID=00:1C:F0:6E:C8:53)

This was despite the fact that for a few minutes after booting there were no errors and perceus node status showed the node as "ready" and regularly responding, only to fail to respond later. Despite the fact too, that other nodes which showed up as "init" were not generating "Unknown node" errors.

At this point I was just a bit confused...

Time to get back to some certainties. The first thing to do was to get DNS working. I use two "networks" for the cluster. The "management" traffic is sent over a network card assigned a static 192.168.3.x address. The application traffic is send over a second network card with a static 192.168.4.x address. How do you make sure the right card gets the right IP address? You run /sbin/ifconfig on the node to get the order of the network devices and edit /etc/perceus/modules/ipaddr accordingly.

Easy, right? Well no. I've found that which node network device gets which name, eth0, eth1, etc. can change between boots, or if the host has been rebooted, or if the vnfs image has been updated.

I edited the /etc/perceus/modules/ipaddr file so that the management card would get a "Default" address. Perceus first looks in the /etc/hosts file for the node name, if it doesn't find one it assigns a DHCP address. However, the DHCP address that is assigned is not the same as the address assigned on boot up! As a result, if I tied to ping the node, the node name was resolved to the boot up address again. This cannot be right. There must be something wrong with perceus-dnsmasq - and if there isn't there should be.

So I added the node name to the /etc/hosts file with its static 192.168.3.x address. Finally, things started to work. At least, I was able to ping the nodes by name. However, perceus node status was still not being updated and I was still getting "unknown node" errors. I will leave that investigation to another post.

With the node names being resolved to their 192.168.3.x addresses, I needed to change the way I launched mpi applications. Essentially this is just a case of saying which interface hostname to use. So in the mpd.hosts name file I added entries like this:

n00005 ifhn=192.168.4.25
n00004 ifhn=192.168.4.24

Then bring up the ring of mpds specifying the local interface hostname:

#mpdboot --ifhn=192.168.4.1 -n 3 -v -f mpd.hosts

I could then add the application network addresses I wanted to use to the machine file, just:

192.168.4.25
192.168.4.24

and run the application:

# mpirun -machinefile machines -n 4 ./xhpl

I hadn't needed to do any of this previously. I'm left with the uneasy feeling that I haven't got to the bottom of why the problem arose, or whether what I've done is really the solution or just a workaround. If I had 1000 nodes, or 10,000 nodes, would Perceus expect me add all those addresses to the /etc/hosts file? perceus-dnsmasq should handle that, shouldn't it?

Installing Erlang

2009-01-18T08:31:00.006Z

One of my key goals for my experimental cluster is to run (and program) distributed Erlang. The first task is simply to get Erlang installed on the nodes. Unfortunately, there is no CentOS rpm package for Erlang. This is a bit surprising: it means that there is no rpm package for RHEL either. It's not a big problem, we just need to install from the source code.

The first thing to do is download and unpack the source file. Do not do what I did and use File Roller, the GNOME archive manager. If you do you get a make error. Instead just follow the instructions in the readme file:

gunzip -c otp_src_R12B-5.tar.gz tar xf -
zcat otp_src_R12B-5.tar.gz tar xf -

To build Erlang I needed to install the ncurses and OpenSSL development libraries:

yum install ncurses-devel
yum install openssl-devel

That done, Erlang built without any problems. But that's only half the job. The nice thing about running the same OS on both the "host" machine (where the nodes are managed from) and on the nodes themselves, is that you can build software on the host and then just copy it to the nodes. The Perceus user guide states that you should be able to do something like this:

make DESTDIR=/mnt/centos-5.2-1.stateless.x86_64 install

I have to say, however, that I've never got this to work. Happily, Erlang's file structure is quite simple. The progam files are in /usr/local/lib/erlang (by default) and there are a bunch of links in /usr/local/bin. Once Erlang is installed on the host you can copy the files to the mounted vnfs:

cp -r /usr/local/bin/* /mnt/centos-5.2-1.stateless.x86_64/usr/local/bin
cp -r /usr/local/lib/erlang /mnt/centos-5.2-1.stateless.x86_64/usr/local/lib

I rebooted a node, connected over ssh, and typed erl. Everything looks fine.

HPL Tuning

2009-01-05T12:50:00.003Z

You might think that the way to test the relative performance of different computer systems would be to give them all the same problem to solve, and then see which system solved the problem fastest. That is not how HPL works. Instead, the linear equations that get solved by the Linpack Benchmark are a means to an end, not an end in themselves: they are really just there to give the cluster's CPUs something to chew on. The aim is simply to run the CPUs as fast and as efficiently as possible. To this end, there are a number of parameters you can tweak in the HPL.dat configuration file to optimize or tune the way HPL runs.

There are 31 lines in the HPL.dat file, but some are definitely more important than others. In my, admittedly limited, experience the three most crucial parameters are the "problem size" (N), the "block size" (NB), and the "process rows" and "process columns" (P and Q). Essentially, the "problem size" needs to fill up as much RAM as possible - without any of it being swapped out to disk, and bearing in mind that the Operating System needs RAM too. 80% of total memory seems a good starting point; this is recommended in the HPL FAQs and seems to be borne out in practice. There's a nice graph on the Microwulf website showing how performance increases with problem size.

The "block size" is best determined by testing, but I've found values between 160 and 256 to produce the best results, depending on the system. I have a feeling that a good block size is related to the size of the processor cache, but I've no evidence whatsoever...

The P and Q parameters specify the dimensions of the "process grid." Again, I've found the best values by trial and error - I've no theory.

So I fired up my cluster full of hope and expectation, only to find that the results were truly terrible! Running four nodes, the best I could do was just 1.94 Gflops (with N=20000, NB=120, PxQ=2x4.) To put this in some sort of perspective, Microwulf - a four node cluster with 8Gb of RAM - is capable of delivering 26.25 Gflops. Earlier in the year, I ran HPL on Fedora 8 on my Dell D630 laptop, and achieved what I thought was an amazing 9.54 Gflops*. My laptop does have 4Gb of RAM, and the 2.20GHz Intel Mobile Core 2 Duo processor has a 4096 KB L2 cache, but that's nearly 10 Gflops on a laptop! I couldn't get 2 Gflops on the cluster... Clearly there was something badly wrong. But what?

To cut a very long story short, by experimenting with different values for NB and P and Q, and by carefully tuning N to the available RAM, I was able to increase performance from 4 nodes running 8 processes to 10.43 Gflops. (WR00L2L4: N=27648, NB=192, P=1, Q=8.) This was a big improvement -and shows how important tuning HPL is - but is still pretty poor.

You can measure the performance of a cluster by its Computational Efficiency. Computational Efficency is simply the the measured performance of the cluster (R_Max) divided by its theoretical maximum peak performance (R_Peak):

compEff = R_Max/R_Peak

R_Peak can be estimated as the number of nodes in the cluster, multiplied by the number of cores per node, multiplied by the number of floating point units per core, multipled by the clock speed. In the case of my test server R_Peak= 4 x 2 x 2 x 2.2 = 35.2. With the cluster only delivering 10.43 Gflops, compEff = 10.42/35.2 = 29.6%. This is not a good number :-( As a comparison, "real" clusters like those in the TOP500 list are around 75% efficent. Microwulf runs at an execeptional 82%. My D630 comes in at 108% :-o But then it isn't a cluster...

I needed a "game changer." My cluster is built using AMD Athlon™ 64 X2 processors. I replaced the GotoBLAS library with AMD Core Math Library (ACML) which is optimized for AMD processors. There is a really excellent article on using ACML with HPL here. The results were outstanding. Performance doubled to 21.24 Gflops. (WR00L2L2: N=27392, NB=192, P=2, Q=4.) Computational Efficiency doubled to 60.3%. All I had done was link HPL to a different library!

The figures are still not brilliant, but at least they are approaching acceptability. Interestingly, Computational Efficiency is around 70% when running on any two nodes. There are still some things to try.

*WR00L2L2: N=18944, NB=256, P=1, Q=4. Time taken 475.22.

Show us yer FLOPS!

2008-12-23T09:48:00.001Z

What's the first thing you want to know about your cluster? Or anyone else's cluster, come to that? You want to know how fast it is, right? The way of measuring cluster performance is to count the FLOPS - the FLoating point Operations Per Second - the cluster is capable of executing. Counting FLOPS is the biggest pissing competition in the world of computing. Right now the cluster that can piss higher up the wall than any other is RoadRunner, capable of 1.7 petaflops (that's a 1 with 15 noughts - a million, billion FLOPS, a million gigaflops.)

The way to measure FLOPS is to run the High Performance Computing Linpack Benchmark - HPL. You can download HPL from Netlib. To build HPL you need two more things - mpi and a BLAS library. BLAS stands for Basic Linear Algebra Subroutines. GotoBLAS is recognized as being a fast implimentation of BLAS, so I downloaded that. I ran the GotoBLAS quickbuild.64bit script and everything seemed OK, so I left it there.

It is probably worth mentioning that HPL can use Fortran 77, so I installed the g77 "compat-gcc-34-g77-3.4.6-4" package.

Building HPL is a case of creating a make file for your architecture. Fortunately, you can just edit one of the default files in the hpl "setup" sub directory. I used the Make.Linux_PII_FBLAS file and set the MPI directory as follows:

MPdir = /opt/mpich2-install

and the Linear Algebra library like this:

LAdir = /home/David/Sources/GotoBLAS

Then its just a case of calling make specifying the right architecture:

make arch=Linux_PII

(Obviously the make file needs to match.)

The file you need to start benchmarking is xhpl in the (in my case) hpl/bin/Linux_PII_FBLAS/ sub directory. We need to run this with mpirun. However, there's an issue. In order to run an application across multiple nodes, that application's binary file (and any supporting libraries) needs to be on each node. Rebuilding a node image every time we want to run a new application is obviously out of the question, so what do we do? Fortunately Perceus can come to our aid.

Perceus supports what is known as "Hybridization". Hybridization is essentially file sharing. The idea is that files or folders in the VNFS image are replaced by symbolic links pointing to network-based files or folders. Unfortunately, it is at this point that Perceus' careful abstraction of the node file system falls down. To specify which files or folders get redirected you have to get into the guts of how Perceus organizes things.

I want to create a shared directory where I can put the binaries I want to run with mpi. This directory is going to be /opt/mpirun. Importantly, this directory has to exist on the head node, as well as all the cluster nodes. The first step is to add /opt/mpirun to the hybridize configuration file located at /etc/perceus/vnfs/vnfs_capsule_name. However, the /opt/mpirun directory specified in the hybridize file is not the /opt/mpirun directory on the host machine (the head node.) Oh no, this is the /opt/mpirun directory located on the physical representation of the VNFS file system that actually underlies the "virtual" VNFS file system of the nodes. In reality this is located at /var/lib/perceus/vnfs/vnfs_capsule_name/rootfs/opt/mpirun. It is this directory that actually gets shared. So because I want /opt/mpirun to exist on the head node as well, /opt/mpirun on the head node has to be a symbolic link back to /var/lib/perceus/vnfs/vnfs_capsule_name/rootfs/opt/mpirun. Not pretty. Finally, you need to mount the VNFS image and edit /etc/fstab so that the node connects to the share.

That done, we are ready to start testing the performance of the cluster by running xhpl. (Well, almost, I had to copy the libg2c.so.0 library to /usr/lib64/ on the nodes first.) Copy xhlp and HPL.dat to /opt/mpirun and go...

Installing MPI

2008-12-22T14:24:00.005Z

MPI stands for Message Passing Interface. When different instances of a process -usually running on different nodes - need to talk to one another, they do so using MPI. It has become a sort of de facto standard.

The are various implimentations of MPI. I'm using MPICH2. Building and installing MPICH2 on the head node is no problem - just follow the instructions in the "From A Standing Start to Running an MPI Program" section of the Installer's Guide. The only thing you need to keep in mind is that you will need to replicate the install on all the nodes. For this reason I installed to /opt/mpich2-install via configure -prefix. I could then copy the mpich2-install directory to the mounted VNFS image:

perceus vnfs mount centos-5.1-1.stateless.x86_64
cp -r /opt/mpich2-install /mnt/centos-5.1-1.stateless.x86_64/opt/
perceus vnfs umount centos-5.1-1.stateless.x86_64

The key to getting mpi working, however, is to be able to ssh onto a node from the head node, and onto the head node from a node, without needing to enter a password. You must be able to ssh both ways. Happily, Perceus sets up the connection from the head node to the node, so you don't have to do anything. But to ssh onto the head node from a node - without needing a password - you have to do some work. On the node:

[root@node ~]#ssh-keygen -t rsa
[root@node ~]#cat .ssh/id_rsa.pub ssh root@head_node 'cat >> .ssh/authorized_keys'

This generates a private/public key pair for root on the node and copies the public key to the head node so that root will no longer need to enter a password when using ssh to connect from the node. That's fine, except the cluster nodes are stateless - they don't have their own harddrives - so the next time the node reboots the configuration will be lost. The solution is to copy the keys and ssh settings back to the node VNFS image. If the VNFS image is mounted, you can do this:

[root@node ~]#scp -r ./.ssh head_node:/mnt/centos-5.1-1.stateless.x86_64/root

Keep in mind that a .ssh directory probably already exists, so you might want to get it out of the way first. But that's it - job done!

There a few things worth noting here. You only have to generate the public/private keys once. The same keys work for all nodes - they are not related to the host name of the node (which I thought they might be.) This also means that the keys work regardless of what nic your mpi traffic is going over. (More of which later.) On mpi itself, all you have to do is make sure that the files are in the same place on each system, and it does the rest. Very smart. After that it is just a case of starting mpi on each node:

mpdboot -n 5 -f mpd.hosts

Running mpdtrace should now show you the names of your nodes.

Let's go clustering...

2008-12-22T11:10:00.008Z

The reason I need CentOS is because I want to experiment with clustering. I first looked at clustering earlier this year, building a small 4 node beowulf cluster by following the "Configuration Notes" for Joel Adams' inspirational Microwulf. However, the Microwulf model is not easily scalable: for example, you need to manually create and configure a file partition on the host for each of the nodes. I want to look at something more industrial. After reading this article on the Linux Magazine website, I thought Perceus sounded just what I needed.

Perceus is an "enterprise and cluster provisioning toolkit" and supersedes the older Warewulf provisioning tools.

Perceus turned out to be pretty easy to build and install. I ended up needing to download and build all of the dependencies from the Perceus website, but nothing too onerous. I then downloaded, and "imported" into Perceus, the Caos NSA 1 VNFS "capsule". Let me just unpack that :-) Caos is a high performance, lightweight distribution of Linux. (NSA stands for "Node, Server, Appliance".) VNFS stands for Virtual Node File System. The idea is that you package up an Operating System - like Caos - into a VNFS capsule which you can then easily distribute, run and manage on your cluster nodes with Perceus. In fact the VNFS system works really well.

My "head node", running Perceus on top of CentOS 5.2, has three network cards. One nic talks to the outside world, while the other two talk to the cluster. To allow this, I fixed the firewall to completely open up the two internal network connections by adding the following lines to /etc/sysconfig/iptables:

-A RH-Firewall-1-INPUT -i eth0 -j ACCEPT
-A RH-Firewall-1-INPUT -i eth1 -j ACCEPT

That done, I started a node with a monitor attached and watched it boot into Caos. Very cool. Except that's when my problems began...

I ran perceus node status on the head node to see what state Perceus thought my node was in. Unfortunately it showed "init" and not "ready". Then I remembered that I hadn't installed provisiond on the node image. provisiond is a client-side daemon that runs on each node and talks to perceus (running on the head node) to let it know what is going on with the node.

Following the instructions in the Perceus "User Guide" I spent the best part of two DAYS trying to install provisiond. It should be as easy as this:

rpm -ivh --root /mnt/caos-nsa-node-1.0-1.stateless.x86_64 \
/usr/src/rpm/RPMS/x86_64/perceus-provisiond-1.4.0-1898.x86_64.rpm

The problem is that Caos is so high performance and so lightweight that it doesn't seem to have a working version of rpm - or any other package manager - installed. Most of those two days were spent trying to install rpm, or trying to work out what I'd missed in the User Guide. I hadn't missed anything: the User Guide is simply wrong.

Towards the end of the second day I thought I had better just check that provisiond wasn't already installed on the Caos image. It was. So the User Guide is doubly wrong: the wrong instructions for something that didn't need to be done in the first place. My node status problem had nothing to do with provisiond.

Nevertheless, the lack of a package manager is a big problem for me. One of the things most people will want to do is run mpi based applications on their cluster. That means you have to install mpi on the nodes. mpi is dependent on Python. Python isn't installed on Caos, so how are you going to install it? Not with rpm or yum or apt-get, that's for sure. Want to build it from source? How are you going to install a compiler? Perhaps that isn't an issue; perhaps you can compile it on the head node and use ./configure --prefix or something to install it to the mounted VNFS image. Are you sure all the libraries are going to be there?

May be this problem doesn't arise of you are also running Caos on the head node - I don't know. I gave up on Caos and used the centos-5.1-genchroot.sh script in /usr/share/perceus/vnfs-scripts (not "vnfs-tools" as it says in the User Guide) to create a CentOS VNFS image. The script worked perfectly and provisiond installed instantly first time.

My node status problem was down to a combination of two things. Firstly a DNS problem fixed by setting the correct nameserver entry to the head node in the /etc/resolv.conf file on the nodes; and secondly by getting the eth0 and eth1 device IP entries in the right order in the /etc/perceus/modules/ipaddr configuration file. Easy when you know how...

All CentOS is theft... Or is it?

2008-12-03T15:24:00.006Z

Up to now I have been running Ubuntu as my Linux Server OS of choice: for me, the Fedora stack is updated far to frequently for it to be a viable server option. However, for reasons that will become apparant in future posts, I need to run a Red Hat type server. The problem is I don't want to pay for it... Wouldn't it be good if I could get hold of the Red Hat Enterprise code, without having to pay for support? Enter The Community Enterprise Operating System:

CentOS is an Enterprise-class Linux Distribution derived from sources freely provided to the public by a prominent North American Enterprise Linux vendor. CentOS conforms fully with the upstream vendors redistribution policy and aims to be 100% binary compatible. (CentOS mainly changes packages to remove upstream vendor branding and artwork.)

That "prominent North American Enterprise Linux vendor" is, of course, Red Hat. The CentOS FAQs make the following points:

CentOS-x is NOT Red Hat® Linux, it is NOT Fedora™ Core. It is NOT Red Hat® Enterprise Linux. It is NOT RHEL.
CentOS-x does NOT contain Red Hat® Linux, Fedora™ Core, or Red Hat® Enterprise Linux.
CentOS is built from publicly available open source SRPMS.

And so, dear reader, we enter the bizarre world of Open Source licensing. I am not saying that the folks at CentOS are a bunch of liars and thieves - clearly what they are doing is perfectly acceptable in the Open Source world. But where else would it be acceptable? What if I go and get a can of Diet Coke, scrub off the printing, and then put on my own label? In what sense would that NOT be Diet Coke? Would the The Coca-Cola Company not sue my sorry ass if I tried to pass this stuff off as my own - even if I was giving it away for free?

Now any Open Sourcers out there reading this may well be saying you just don't get it. And you know what? I don't. I write software for a living. If someone took my code, changed the logos, and then passed it off unchanged as their own, I would have some issues with that. But Open Source software isn't like that, right? It is a community effort, right? RHEL a community effort? The creation of those SRPMS files a community effort? I don't think so. (I'd better stop there, I'm starting sound like Lewis Black from the Daily Show...)

So I'm not going to use CentOS? Wrong! I'm definitely going to use CentOS. He who lives by the sword dies by the sword. If Red Hat want to play in a world where software has no value, that's up to them.

Going /home

2008-12-03T12:37:00.012Z

With the release of Fedora 10 I thought I would take another look at running Linux on my laptop. I haven't changed my mind, however: I stick by my assertion that Linux is a great workbench, but a lousy desktop. I love working on Linux, but I'll be installing Fedora 10 on a separate hard drive.

One of things that might have caused my previous installation of Fedora to break so badly was that I upgraded it: from version 7 to version 8. It turns out that just because you can upgrade a system doesn't mean that you should: section 2.1.4.3 of the Release Notes states "In general, fresh installations are recommended over upgrades." But if the OS is changing every 6 months, what do you do? Backing up your data (and all your settings) every time and then restoring it all to your new system is a serious pain. One answer to this is to create a separate /home partition. By doing this you can keep your stuff out of harms way when the system and the applications get nuked by the installer.

Creating a separate /home partition is beautifully easy on Fedora 10. When the installer gets to the bit about "Select which drive(s) to use for this installation", you just need to check the "Review and modify partitioning layout" box. On the next page, click the "New" button and then add a new partition with /home as the mount point; the only other thing you just need to do is set the size. I also checked "Encrypt" - meaning my personal data would be encrypted :-)

The Fedora installer does everything else for you - including resizing the other partitions so that your new /home partition fits.

Cool. Except before I could do this, I still needed to backup my data and settings from my old Fedora installation that didn't have a separate /home partition. To get this done, I created an archive of my old home directory:

#cd /home
#tar cfv David.tar David/

I then burnt David.tar to a DVD. Once Fedora 10 was installed I created my new "David" user account and let Fedora create a home directory. I then copied my David.tar file from the DVD to /home:

#cd /media
#cd "Personal Data, Dec 01, 2008"
#cp David.tar /home

I then deleted the "David" home directory that Fedora had created for me, and recreated it from the .tar file:

#cd /home
#rm -rf /home/David
#tar xvf ./David.tar

Finally, I needed to make my user account the owner of the directory.

chown -hR David /home/David

I logged on, and it worked! My desktop appeared just as I remembered it. Maybe I didn't need to create a /home partition after all...

Footnote. Sadly, though, not everything worked. I had hoped that the version of Evolution on Fedora 10 would just pick up all my email. No chance. It starts some conversion process... and then crashes everytime. What a piece of crap that application is. The lack of an enterprise class mail client is one of the biggest failures of the Linux desktop.

The End of the Affair

2008-07-21T14:00:00.002+01:00

How do you fall out of love? It’s hard to say. The little annoyances that you have always put up with slowly become festering grievances. But do you fall out of love because they become grievances, or do they become grievances because you fall out of love?

There is usually some event which, when you look back, you realise was the moment when things began to go wrong. For Fedora and I, it was a few months ago when after yet another kernel update the sound on my laptop - which had been working perfectly – stopped working. The volume control suddenly had a red X next to it. Bizarrely system-config-soundcard showed that the sound card was configured correctly and could play a test sound. Clicking on the volume control however, just resulted in “No volume control GStreamer plugins and/or devices found.” Why?

Yes, I found a post on LinuxQuestions.org that discussed the problem. But by now I realized I was beginning to wonder why I should bother - so perhaps it was me. Was there a problem with PulseAudio? Was it a security issue? After spending several hours completely removing PulseAudio and after reinstalling all the ALSA packages, sound was working in MPlayer and Adobe Flash Player in Firefox, but I still had no volume control.

There have been other grievances of course; there is never just one thing. Suspend was working fine: now it doesn’t. If I suspend my laptop I have to hard boot it to get it to come back to life. Could I fix this? Probably. But it is one thing to have to get something working that has never worked: it is quite another to have to fix something that has been needlessly broken. I’m fed up spending my weekends scouring the Linux support forums for fixes to problems I shouldn’t have. Life is too short.

I want to get my laptop working with Internet Connection Sharing on my phone. Yes, I could get it working… But I’ve got a wife and children I want to spent time with, and code that I want to write.

Late in the first decade of the twenty-first century what do we use our Personal Computers for? You will have your own ideas, but I would suggest browsing the Internet, reading email, storing and playing music, watching video. Linux is not very good at any of these things. Firefox is hampered by font issues, Evolution is grim, MPlayer is great – when it works. Why do I have to reinstall the MPlayer plugins whenever Firefox is upgraded? Because they are two separate applications and nothing on Linux is joined up. I would pay for a distro that took away all the hassle - that managed my installation to keep it updated but kept it working. There is no such distro.

So after eighteen months of using Fedora as my everyday laptop OS I’m going back to Vista. It’s not that I won’t be using Linux anymore - far from it. Linux now has a permanent place in our server and development environments. Linux is a great workbench but it is a lousy desktop.

Am I giving up completely on a UNIX compliant desktop which works, is stable, and supports the twenty first century? Well there is always this :-)

Samba and LDAP: a Wind-up

2008-06-10T11:26:00.008+01:00

I had to revisit this because I thought I must have missed something: must have got something wrong. But I really don't think I have. The developerWorks document I quoted in an earlier post was written before the release of Samba 3. It says that "There are two things a Samba/LDAP installation cannot do 'out of the box' ". The first is "Retrieve user account information from an Windows 2000 Active Directory server"; the second is "Alleviate the need for /etc/passwd." Both these issues, the document confidently expects "will be resolved with the release of Samba 3.0." But it didn't happen. Instead, the Samba documentation states that "The second item [removing the need for /etc/passwd] can be accomplished by using LDAP NSS and PAM modules." Except that these modules are already installed on Fedora. Checking the /etc/nsswitch.conf file for the necessary entries:

passwd: files ldap
group: files ldap

shows that these entries are in place.

Everything I've read implies that I should be able to achieve what I want to do with Samba and LDAP: create groups and user accounts in LDAP and have them access file and print resources on the Samba server. And yet I've not been able to achieve that in the time I've had available. I'm disappointed. My feeling is that unless you really need to use the features of LDAP you are probably better off using Samba's tdbsam backend instead.

CLDAP

2008-06-10T09:32:00.006+01:00

I couldn't of course just leave things as they were, so I did some digging. Everything I've read indicates that openldap, and by extension FDS, does not support UDP. As a result, an openldap server cannot respond to CLDAP queries such as those made by a Windows client. That doesn't always seem to have been the case. Earlier versions of openldap seem to have had a complile time option --enable-cldap. This option appears to have been dropped. I found a reference here to the to option no longer being available in version 2.1.22. Certainly in the configure script of the current 2.4.10 source code there is no mention of --enable-cldap.

I couldn't find any formal announcement, but perhaps the reason CLDAP support was dropped from openldap was because the protocol has been buried as an Internet standard. It's epitaph is recorded in RFC 3352.

From a practical point of view, all this means is that you have to treat a Samba server - even one with an LDAP backend - as a NT 4.0 server and connect to it via NetBIOS. If you specify a DNS name, Windows thinks you are connecting to Active Directory. This is made clear by the subtly different error message you get on Vista:

Don't think this is the end of CLDAP, however. It is obviously still being used by Active Directory, and if it is still being used by Active Directory, it will have to be supported by Samba 4.0.

Joining a Samba Domain

2008-06-09T13:48:00.012+01:00

Joining a Windows workstation to a Samba domain is quite instructive: it tells you an awful lot about how a Windows workstation joins a Windows domain...

Here's what I got the first time I tried to join a Windows XP machine to my RIVERSIDE Samba domain:

The error message goes on: "The query was for the SRV record for _ldap._tcp.dc._msdcs.RIVERSIDE"

What's happened here is that Windows has first tried to look up the NetBIOS name "RIVERSIDE" assuming - correctly - that it is not a DNS name. When the NetBIOS name look up failed, Windows then queried DNS. Most of this error message relates to the DNS look up, and as such is quite misleading.

The standard solution you will find on all the Samba discussion forums is to add the Samba server to workstation's list of available WINS servers. One simple way of doing this is to get your DHCP server to pass on the address of the Samba server. If your DHCP server is Linux you just need to add a line like the following to your /etc/dhcp3/dhcpd.conf:

option netbios-name-servers 192.168.2.8;

When the workstation next gets an IP address from the DHCP server it will know about your Samba server (running WINS) and will be able to resolve the domain NetBIOS name. This works. And maybe, if you're sensible, you should stop there. However, aren't you a bit curious about the " SRV record " stuff?

If, instead of supplying a NetBIOS domain name, we supply a DNS domain name, like riverside.forensit.com, Windows will attempt to look up - not the DNS domain name itself - but a service location (SRV) resource record for a Domain Controller for the domain. In other words, it queries DNS for something like:

SRV _ldap._tcp.dc._msdcs.RIVERSIDE.FORENSIT.COM

In order for your DNS server to answer this query you need to add some lines to your DNS config files:

_ldap._tcp IN 1H SRV 0 100 389 medway
_ldap._tcp.riverside.forensit.com-site._sites IN 1H SRV 0 100 389 medway_ldap._tcp.riverside.forensit.com_site._sites.dc_msdcs IN 1H SRV 0 100 389 medway
_ldap._tcp.gc._msdcs IN 1H SRV 0 100 389 medway_ldap._tcp.dc._msdcs IN 1H SRV 0 100 389 medway
_ldap._tcp.dc._msdcs.riverside.forensit.com IN 1H SRV 0 100 389 medway_ldap._tcp.gc._msdcs.riverside.forensit.com IN 1H SRV 0 100 389 medway
_gc._tcp IN 1H SRV 0 100 3268 medway_gc._tcp.riverside.forensit.com-site._sites IN 1H SRV 0 100 3268 medway

On Fedora, these get added to the /var/named/*.db file. You can add these entries using system-config-bind, although (rightly) it does complain.

After restarting named on the DNS server, the Windows workstation should be about to get a response to its SRV query. On my riverside.forensit.com domain it can... But - and throughout this whole long exercise there has always been a but - that doesn't get me much further. Windows returns the error "A domain controller for the domain RIVERSIDE.FORENSIT.COM could not be contacted." The details are:

DNS was successfully queried for the service location (SRV) resource record used to locate a domain controller for domain RIVERSIDE.FORENSIT.COM:

The query was for the SRV record for _ldap._tcp.dc._msdcs.RIVERSIDE.FORENSIT.COM

The following domain controllers were identified by the query:

medway.riverside.forensit.com

Which is all absolutely correct. I can ping medway.riverside.forensit.com (the DC.) I can look it up using nslookup. So what's going on now?

I resorted to Wireshark. Having successfully queried for the DC name, the XP workstation sent a search request for the "ROOT" base object of the LDAP directory over CLDAP (Connectionless LDAP.) It was this request that failed: the server returning "Destination unreachable (Port unreachable)" over ICMP. However, the LDAP error log on my server just reports " - slapd started. Listening on All Interfaces port 389 for LDAP requests." It's almost like a firewall problem, except Firestarter reports no blocked connections.

So that's as far as I've got. I hate leaving unanswered questions, but I've already spent far too long on Samba: the questions will have to wait. It maybe that what I'm attempting here can't be done. I've come across some posts by glauco.b on the Ubuntu Forums saying that openldap does not support UDP - which is what the CLDAP query goes over.

Roaming Profile Error Logging On To Samba

2008-06-06T14:11:00.005+01:00

When I attempted to log on to my Samba domain from an XP workstation I got this:

Followed by this:

Except I'm not using roaming profiles.

The solution was to explicitly set the "logon path" and "logon home" parameters to nothing in the /etc/samba/smb.conf file:

logon path =
logon home =

I seem to remember a customer complaining about this in the past. Now I know.

Samba A101

2008-06-06T10:26:00.011+01:00

RULE ONE: Before you can add a user to the smbpasswd password file, that user must have an UNIX/Linux account (typically stored in /etc/passwd) on the system hosting the Samba service.
-ibm.com/developerWorks

And that's what I was missing. You can't add a Samba user unless the posix user account exists first. This applies to Samba implementations with a LDAP "back end" just as much as it applies to traditional Samba implementations. It never occurred to me that this would be the case. On a Windows Active Directory server, you don't have to create a local user account as well as creating a user account in AD, the LDAP database. On NT 4.0 Servers - which Samba was originally developed to imitate - things are different, of course. But in February 2000 Windows moved on: Samba has still to catch up eight years later.

Suddenly the scales have been lifted from my eyes. Fedora Directory Services, and indeed other Linux LDAP implementations, are at best "bolt ons" for Samba. LDAP does not fundamentally change the way Samba operates, and emphatically Samba + LDAP does not equal AD.

Crucially, Samba cannot integrate LDAP with the Linux file system. On a Windows server, you can create a group in AD and then give that group access to a folder. Samba 4 will have its own LDAP directory that will allow it to be an LDAP server for AD clients. However, there will still be no integration with the file system. As a recent article in Linux Magazine puts it:

On a more somber note, Samba4 lacks great integration with the POSIX system on which it sits. It cannot map NT ACLs into POSIX ACLs, it requires users and groups to be added to Linux as well as to its internal ldb database.

So why bother implementing Samba (3) with LDAP at all? For the moment I can't think of a single good reason. Apparently a LDAP back end provides better scalability and performance. But any organization needing scalability and performance isn't going to be looking at hacking around with Samba and LDAP anyway. They are going to look to Active Directory, and the product that Active Directory set out to destroy - Novell Netware. I'm suddenly very interested in how OES integrates with the Linux file system - if indeed it does.

My illusions shattered, I got Samba and LDAP working quite quickly. It was just a case of creating the accounts I needed using useradd or system-config-users. This included creating the "Administrator" account (which then allowed me to use pdbedit to set the SID) and my Windows workstation accounts:

/usr/sbin/useradd -n -c "Workstation FDSPC" -M -d /nohome -s /bin/false FDSPC$

I then called "smbpasswd -D 10 -a Administrator" as I did before. This time, however, I got this:

smbldap_search_ext: base => [dc=riverside,dc=forensit,dc=com], filter => [(&(uid=Administrator)(objectclass=sambaSamAccount))], scope => [2]
ldapsam_getsampwnam: Unable to locate user [Administrator] count=0
pdb_set_username: setting username Administrator, was
pdb_set_full_name: setting full name Administrator, was
pdb_set_domain: setting domain RIVERSIDE, was
...etc...

which makes sense: Samba checks LDAP to see if the accounts exists and creates it if it doesn't. I used smbpasswd again to create the workstation accounts:

/usr/bin/smbpasswd -a -m FDSPC$

This done, I was finally able to add my Windows workstation to my Samba LDAP domain.

Getting More Information From smbpasswd

2008-06-05T16:05:00.003+01:00

smbpasswd has 10 debug levels specified by the -D switch. Level 10 is not recommended: the man page says "Levels above 3 are designed for use only by developers and generate HUGE amounts of log data, most of which is extremely cryptic." Really?

smbpasswd -D 10 -a testuser

gives us this:

smbldap_search_domain_info: Searching for:[(&(objectClass=sambaDomain)(sambaDomainName=RIVERSIDE))]
smbldap_search_ext: base => [dc=riverside,dc=forensit,dc=com], filter => [(&(objectClass=sambaDomain)(sambaDomainName=RIVERSIDE))], scope => [2]
The connection to the LDAP server was closed
smb_ldap_setup_connection: ldap://riverside.forensit.com
smbldap_open_connection: connection opened
ldap_connect_system: Binding to ldap server ldap://riverside.forensit.com as "cn=Directory Manager"
ldap_connect_system: successful connection to the LDAP server
ldap_connect_system: LDAP server does not support paged results
The LDAP server is successfully connected
pdb backend ldapsam:ldap://riverside.forensit.com has a valid init
smbldap_search_ext: base => [dc=riverside,dc=forensit,dc=com], filter => [(&(uid=testuser)(objectclass=sambaSamAccount))], scope => [2]
ldapsam_getsampwnam: Unable to locate user [testuser] count=0

testuser can't be found. Of course it can't be found - we're trying to create it!

Connecting to Samba... or not...

2008-06-03T13:35:00.004+01:00

I have managed to get over most of my connection issues. These were down to a combination of DNS configuration problems, and firewall issues.

The DNS problems stemmed from the fact that there were "allow-query" and "listen-on" entries in the /etc/named.conf file that effectively restricted queries to the localhost. So, for example, dig @localhost riverside.forensit.com would work, but using the IP address dig @192.168.2.8 riverside.forensit.com would fail. My guess is that these entries were written to the file when I was messing around with system-config-bind, although I don't recall changing anything.

After fixing the DNS problems, I installed Firestarter. I might have been able to get away with Fedora's own system-config-firewall, but I'm familiar with Firestarter - and it logs blocked connections so you can easily see what is going on. Once I'd opened up the Samba, DNS, LDAP and HTTP ports - not forgetting the Fedora Management Console port, I was starting to get somewhere. I was still unable to see my "Riverside" domain on the network, however. I couldn't see it from a Windows machine, and I couldn't even see it from "Network" on the server itself.

Being able to browse for the domain on the network is dependent on the Samba nmbd daemon (that's service for the Windows-minded.) nmbd is controlled by the line

wins support = yes

in the /etc/samba/smb.conf file. You would think then, that when you started Samba, nmbd would automatically be started too. Well I did. It wasn't. I manually started "nmb" via system-config-services and "Riverside" appeared under "Microsoft Windows Network" on my XP machine.

Almost there? Unfortunately there is still something missing. Although I can "see" the domain, I cannot get a Windows machine to join the domain. There's still more work to do.

Installing Samba with Fedora Directory Server

2008-05-28T11:15:00.006+01:00

The instructions are here:

http://directory.fedoraproject.org/wiki/Howto:Samba

All I can say is follow the instructions to the letter. Do not deviate from the instructions in any way. I had all sorts of problems, but they were all related to mistakes I'd made following the instructions. Eventually however, I did get the Samba service to start without any errors.

So I'm all set, right? No... Although everything appears to be running correctly locally, I cannot connect to the machine - as a domain controller - across the network, and I cannot connect via Fedora Management Console from another computer.

It really shouldn't be this difficult.

Installing VMware Tools on Fedora

2008-05-20T10:36:00.004+01:00

I always forget the steps involved, so here they are for reference.

1. Install the required development packages:

yum install kernel-devel gcc gcc-c++

2. Untar the vmware-tools-distrib from the "virtual" Install CD. (Don't bother with the .rpm file.)

3. cd to vmware-tools-distrib. Run ./vmware-install.pl

That's it! Except I have had to add the following lines to /etc/X11/xorg.conf to get the display to work:

Section "Monitor"
Identifier "vmware"
EndSection

There are much better instructions here.

Fedora Management Console Trouble

2008-05-14T14:19:00.012+01:00

When I tried to log from Fedora Management Console the first time, it didn't work. All I got was a HTTP 404 "Not found" error.

Checking the /var/log/apache2/error.log file showed that apache was trying to serve /var/www/admin-serv which didn't exist. When the machine booted, there was also an error when apache2 started: "VirtualHost _default_:8443 --mixing * ports and non-* ports with a NameVirtualHost address is not supported..."

I am not proud to say that it took me days to overcome this. It took me installing and setting up a Fedora 8 server (because I thought the problem was related to running FDS on Ubuntu) only to find that I got the exact same HTTP 404 "Not found" error logging on to the Fedora server from the Management console.

It may well be that there was more than one problem with the Ubuntu installation - when I have time I will go back and check. To connect to the Fedora server I eventually found that all I had to do was specify the port number used when I ran setup-ds-admin.pl.

Doh!

Installing Fedora Directory Server

2008-05-14T12:41:00.004+01:00

It turns out that our customer is using Fedora Directory Server. Installing an Ubuntu server was probably not the best option then...

No matter. It is certainly possible to install FDS on Ubuntu: the full instructions are here. Mercifully, however, someone who should be given a medal has created the install packages. These can be downloaded here. It is just a case of adding

deb http://ubuntu.opencodes.org gutsy main

to the /etc/apt/soures.list file, running apt-get update, and then running

apt-get install fedora-ds-admin

I also ran apt-get install fedora-idm-console. I don't know if this is really necessary. Ubuntu server doesn't have a desktop, but I do want to use the Windows Fedora IDM Console remotely. That done, I ran

setup-ds-admin.pl

It failed. The error was a result of a "Netscape Portable Runtime error - 5977: libicui18n.so.36: cannot open shared object file: No such file or directory" The solution to this was to install libicu36:

apt-get install libicu36

why this library wasn't already installed, I don't know - unless it is installed with the desktop. I also installed termcap (mistakenly) thinking this had something to do with it. The Ubuntu instructions say that Termcap should be installed. However, installing Termcap on 64-bit Ubuntu turns out to be a pain. It requires downloading the 64-bit rpm from Fedora, running alien to convert it to a .deb file, and then installing the file using dpkg. There are good instructions here.

Once libicu36 was installed, I ran setup-ds-admin.pl again and it worked perfectly! I just accepted the defaults - happily noting all the correct domain name settings :-) - and the job was done.

Configuring DNS on Fedora

2008-05-13T21:45:00.023+01:00

Out of interest I decided to take a look at Fedora's DNS configuration tool, system-config-bind. system-config-bind turns out to be pretty horrible. Coming from a Windows background, I suppose I've come to expect GUI tools to provide a level of abstraction, to take the complexity out of system configuration. That's what Windows Server Wizards do, and it's what we at ForensiT hope our own Wizards do. system-config-bind doesn't do that. If you don't have the knowledge to write your own bind configuration files, don't expect system-config-bind to help you out. The only level of abstraction it provides is from the actual configuration files themselves.

Start up system-config-bind and you see the following:

If you click the "New" button, or right click on "DNS Server", you get the chance to add a new item. (I'm not sure this is the right word to use, but it will do for now.) We want to create a new zone, so that's what I'll do. I then get this:

This gets my nomination for the worst GUI of the year award. There are three OK buttons! THREE! How are you supposed to know what to do first? It is an abomination to the art of user interface design.

What you have to do is click on each of the top two OK buttons. Starting with the top left, select the class from the drop-down list; in this case it is "IN". Click the top left OK button.

Great a dialog box that looks almost exactly the same as the first one! However, we're down to two OK button so we must be making progress. We're creating a Forward zone so we just click the top OK button.

Finally a dialog box that can be understood. We just need to type in our domain name - not forgetting the the dot at the end. system-config-bind does actually remind you about this. That done, when you click on OK you can enter the details for the Zone:

It is not the most friendly dialog box I've ever seen, but it is relatively straight forward. When you've filled in your settings and clicked OK, you have created your zone.

Next you need to create the A records for you domain: highlight the zone, right-click or click "New" and choose "A IPv4 Address"

There is some benefit to using system-config-bind then. It creates the reverse DNS settings for you, so you don't have to mess around creating and editing .arpa files. To be fair, there are other benefits to. By selecting "DNS Server" and clicking the "Properties" button you get to edit a whole range of, well, DNS server properties. It is just a pity the User Interface was - don't think we can use the word designed - created by someone who hasn't got a clue.

When you're done, you can right-click "DNS Server" and choose "Start Server"

Setting the FQDN

2008-05-13T14:07:00.004+01:00

Another quick tip for those like me who are consolely challenged... A linux machine's Fully Qualified Domain Name is set in the /etc/hosts file. The entry just needs to be something like this:

127.0.0.1 medway.riverside.forensit.com medway localhost

You can check the FQDN by running hostname with the -f switch.

Configuring DNS

2008-05-12T12:28:00.034+01:00

Having fixed my VMware woes, installation of Ubantu Server was easy. I choose the DNS, OpenSSH and Samba server options. OpenSSH is extremely useful: it allows you to open a secure console onto the server from another machine - including a Windows machine using a utility like PuTTY.

When you set up a new Windows domain you need to setup a DNS server for that domain. I'm assuming the same thing goes for Samba. Setting up a DNS server on Ubuntu isn't difficult, but it is long-winded and it does require that you edit a whole bunch of text files. This isn't so bad if you are using a GUI, but Ubuntu server doesn't install a desktop by default so you're stuck with console based text editors. You can install a desktop, like GNOME, but so much stuff you don't need gets installed along with it, stuff like Evolution and GIMP, that it's probably better to get along without it.

So which text editor? Unix hard men will now roll up their sleeves to reveal vi tattooed on their sallow, bloated arms. If you can get used to vi good luck to you. (There is a good tutorial here.) You can also use nano.

On Ubuntu there is a hierarchy of DNS configuration files. (It is different on Fedora, so what follows is probably not applicable. You are probably going to be using something like system-config-bind anyway.) Top of the pile is /etc/bind/named.conf. named.conf has entries to "include" two other files: /etc/bind/named.conf.options and /etc/bind/named.conf.local. You will probably not need to edit named.conf itself.

named.conf.options allows you to set a "forwarders" entry to a nameserver that can resolve all the domain names your DNS server doesn't know about. You just need to uncomment the block and enter the IP address. At ForensiT we already have a DNS server, so that server's IP address goes in here.

named.conf.local is where your domain really begins. Given that our new domain is going to be "riverside.forensit.com" and the server name is "medway", we need to add something like this:

zone "riverside.forensit.com" {
type master;
file "/etc/bind/zones/riverside.forensit.com.db";
};

We will also need to add a zone definition for reverse DNS:

zone "2.168.192.in-addr.arpa" {
type master;
file "/etc/bind/zones/rev.2.168.192.in-addr.arpa";
};

Now we need to create the two files referenced in the entries we just created. The zone .db file needs an entry like this:

A couple of things are worth pointing out in passing. "admin.riverside.forensit.com." is not a server: this information is interpreted as an email address(!) and is required. The line immediately below is the version of the file; it is based on the date with a number appended. As well as adding an "A" record for the server, I've added an "A" record for the domain as well - this follows what Windows does.

Similarly, we create the .arpa file:

We're getting there. However, a DNS server needs a static IP address, so before doing anything else we need to edit the /etc/network/interfaces file:

auto eth0
iface eth0 inet static
address 192.168.2.8
netmask 255.255.255.0
gateway 192.168.2.1

(I rebooted at this point.)

The use of all these config files is a recipe for trouble. Fortunately, before starting bind we can check that there aren't any problems with the files. We just need to run:

named-checkconf -z /etc/bind/named.conf

If all is well, we are almost ready to fire up our DNS server. There is one more file to edit, however. We need to change the entries in /etc/resolv.conf to reflect the new configuration:

search riverside.forensit.com
nameserver 192.138.2.8

(The previous entries were set by DHCP.) Finally, we can start bind:

sudo /etc/init.d/bind9 start

If you want to check for any errors on start up, you can look in the /var/log/daemon.log file. (Handy to know if, like me, you're only used to checking log files from the Desktop.) We can now use dig to make sure that our DNS server is doing what it should be:

dig riverside.forensit.com

Our existing DNS server needs to know about the new domain, so we add a forwarders entry for the new domain in the named.conf.local file of the existing DNS server:

zone "riverside.forensit.com"{
type forward;
forwarders{192.168.2.8;};
};

Don't forget to restart bind!

At ForensiT we find that customers frequently forget to do this when setting up a new domain, which leads to all kinds of problems. If your DNS server is a Windows server, you can find instructions in the User Profile Wizard User's Guide on creating a forwarders entry for the new domain.

The next step is to configure Samba. Before we do that, however, we need to set up LDAP.

If you're after some proper instructions on setting up DNS on Ubuntu try these links:

https://help.ubuntu.com/community/BIND9ServerHowto
http://ubuntuforums.org/showthread.php?t=236093
http://www.ubuntugeek.com/dns-server-setup-using-bind-in-ubuntu.html

VMware Problem Running 64-bit Ubuntu

2008-05-12T10:39:00.009+01:00

I'm not off to a good start :-( As soon as I try to boot the VMware virtual machine I get this:

I'm trying to install the 64-bit version Ubuntu server. The host machine is a Dell PowerEdge 2950 with dual quad-core 64-bit Xeons. I've got a choice: I can down the server - and all the virtual servers running on it, of course - and check the BIOS settings or just download the 32-bit version.

Installing the 32-bit version of Ubuntu Server wouldn't get to the bottom of the problem, so I went for a reboot and checked the BIOS. Sure enough, vitualization support was disabled. (Why?!) More importantly, enabling virtualization fixed the problem.

Installing a Samba Domain

2008-05-12T09:21:00.003+01:00

One of our potential customers is having problems joining a XP workstation to their new Samba 3.0.28 domain using User Profile Wizard. Although we've tested using Samba before, we really need to build an up to date Samba domain in our lab for troubleshooting purposes... So that's what I'm going to do.

I'm using Ubuntu because, much as I love Fedora, I don't want to have to upgrade the kernel every week. (That's OK if you're using the latest laptop, not so great for your server.) And I'm using Ubuntu 7.10 (Gutsy Gibbon) and not the current version 8.04 (Hardy Heron) just because I've got it to hand. The new server will be installed as a VMware virtual machine on VMware Server, which itself runs on Ubuntu Server 7.10. What I'm going to do here is record my steps, primarily for my own reference, but also because - if you're reading this - it might be of some use to you.

First step: create the virtual machine.

Automatically Connecting to Wireless Networks

2007-10-25T12:42:00.000+01:00

The Linux world still has the capacity to leave me shaking my head in bewilderment...

One of the things that Vista does really well is managing wireless network connections. Vista will automatically connect to an available wireless network without you having to do anything. If you look at a standard install of Linux, even an install of a "cutting edge" distro like Fedora 7, you would never guess that Linux offers the same functionality.

I have literally spent months looking for an easy way to connect to wireless networks. I looked at utilities like wlassistant and netgo; neither of which turned out to be much good. I even wrote my own bash script using iwlist to scan for available networks and iwconfig to connect. (This worked, but was let down by DHCP problems with dhclient when switching networks.)

The solution was installed on Fedora all the time. The only problem was that it was not enabled, and - worse - not visible. The solution is NetworkManager.

NetworkManager will automatically connect to an available wireless network. But that's not all it does. NetworkManager actively monitors your network connections so that if you plug in a network cable it will disconnect the wireless connection and use the faster wired connection instead; remove the cable and NetworkManager reconnects to the wireless network on the fly. Very, very, cool.

So how do you get it to work? To start NetworkManager manually you can just type the following on the command line:

/usr/sbin/NetworkManager

However, once you are happy NetworkManager is working for you, you will want to have it start automatically:

/sbin/chkconfig --level 345 NetworkManager on
/sbin/chkconfig --level 345 NetworkManagerDispatcher on
/sbin/service NetworkManager start
/sbin/service NetworkManagerDispatcher

Why was it so hard to find? Why isn't the Linux world trumpeting its existence? It is very strange. Perhaps the answer goes back to the issue of what Linux is for. Being a commercial rival to Windows doesn't seem to be what Linux is for- which is fine. So what are Redhat and Novell and Ubuntu doing?

There is more information on NetworkManager here and here.

VMWare - Bridging over Wireless

2007-10-22T13:49:00.001+01:00

I couldn't get my VMWare Virtual Machines to connect to a network via my Fedora wireless connection. The solution is to use a patched version of the vmnet.tar file and then run vmware-config.pl again. vmnet.tar is located in /usr/lib/vmware/modules/source on my machine.

The patch is discussed here. It is available via the VMWare Communities page here.

Enabling Bluetooth

2007-10-22T10:48:00.000+01:00

It's been a while since my last post. Mainly this is because Fedora 7 is working fine and I can just get on with my real job. I have to say it gives me a warm feeling to be using Linux: I don't have that nagging suspicion that I've paid to enter a pact where I'm allowed to use someone else's software in exchange for marketing my life. As a business desktop, Linux can best be described as practical. I've come to think of Linux as a screwdriver rather than an iPod.

Fedora has advanced in a few months to the point that it supports the D630's hardware natively; there is no longer a need to build the video and sound drivers, etc. One thing that I could just not get working however, was the Dell 360 bluetooth module. Finally, I found a discussion on an Ubuntu forum that explained that the problem only occurs on Dell machines with Vista pre-installed. (You may recall that Dell would not sell me a D630 without Vista installed.) The Vista bluetooth driver actually updates the firmware on the Dell 360 bluetooth module so that it can no longer be used by other Operating Systems, including Linux, but also XP. To fix the problem you need to install XP and downgrade the firmware by installing the XP bluetooth drivers.

So how to install XP? You can't install the XP drivers from a VMWare virtual machine, because the bluetooth module is obviously not visible to the host Linux OS, and so is not visible to the Guest OS. I don't have any unused space on my hard drive to create a new partition. How about installing XP on an external USB drive? Well it starts to install, but then won't boot because XP does not support booting from a USB drive. There is a way around this, however. You can craft a XP install CD to support USB drives: there are good instructions here if you need to do this. I didn't. I bought a second hard drive from Dell for £59 (about $120.)

Problem solved? Of course not. The D630 has a SATA hard drive and XP does not support SATA. What's more the Neanderthal XP installer allows you to install additional disk drivers only from the a: drive - and the D630 does not come with a floppy disk. To get around this one you need to craft another XP install CD to support SATA drives. This is not as onerous as it sounds. All you need to do is download the SATA drivers from the Dell website and then integrate them into the XP install image using the brilliant - and free - nLite Windows installation customizer. There are instructions here.

Once I had XP installed, I installed the XP bluetooth driver and that was it! When I restarted Fedora the Gnome Bluetooth Manager was showing the in notification area and running hcitool dev showed device hci0. All I need to do now is work out how to dial out...

So... Can Linux be a business friendly desktop?

2007-08-09T10:37:00.000+01:00

This can only ever be a preliminary opinion, of course... The answer is it depends on your business.

Unless you have the very latest hardware, the chances are that Linux will install perfectly on your kit. Linux is just as stable as Windows and vice versa, so much so that no one talks about Operating Systems crashing anymore. In terms of basic business applications, OpenOffice provides all the word processing and spreadsheet functionality that most users will ever need - although it doesn't look at good as Microsoft Office. So why wouldn't you install Linux?

If I look around my own business I can't see anyone who just needs basic OS and Office functionality. OK, we are Windows software developers, so our developers need to be running Visual Studio - which only runs on Windows. But then, the people in accounts use an accounts package - which only runs on Windows... and so it goes on. Unless you are starting your business from scratch, you will have already made a big investment in Windows software. If you are going to move your desktops to Linux, you are going to have to move your applications to Linux as well.

What if you start slowly? Just move one group of users at a time? If you do that, you have interoperability problems. Using Exchange? Forget it. Evolution just isn't up to the job as an Exchange client. That means making a strategic decision in advance to move key functionality to cross-platform software. In the case of mail, that would be something like Lotus Notes.

On top of application issues you have people issues. You will need to retrain your IT staff and your end users.

So why bother? Why would you want to take on that much pain? You wouldn't; of course you wouldn't. But perhaps your business is already running Lotus Notes, perhaps your business is using browser based applications, or Java applications; perhaps you only have a handful of Windows only programs. In that case, why not?

I can't help thinking however, that to use Linux as a business tool you have to lock it down. The real power of Linux lies in the fact that - unlike Windows - it isn't locked down. That's not a reason not to use it as your business desktop, of course. It's just that there may be better uses for Linux.

Desktop Linux

2007-08-08T11:11:00.000+01:00

The normally excellent The Register website has posted an ill-judged report from LinuxWorld by Ashlee Vance which begins "The Linux desktop reminds us of a dog humping a table leg. It's both fun and disturbing to watch, but ultimately there's very little payoff from the exercise." Whose we? It is obvious from the article that Vance has never actually run Linux on a desktop. The Register would have been better off sending someone who knew what they were talking about, rather than someone reliant on second hand opinions and their own prejudices.

It's a shame because there is a debate to be had about "Desktop Linux" - or rather, Linux on the Personal Computer, as Linux itself comes with multiple "desktops."

Vance writes that Linux "suffers from limited driver, multimedia and power management support." That is simply untrue. In this blog, I've documented my experience in installing Linux on Dell's very latest laptop hardware. Yes, I have had to wait for drivers to get my DVD and sound working, but it only took six weeks for Linux to catch up. With hardware manufactures focused on the 90% of the PC market that is Windows, Linux driver support is remarkably good. Vance goes on "The Linux community ... may well make something Windows comparable by 2020" - which is breathtakingly ignorant.

So what is the "payoff"? The issue I've been trying to resolve for myself is what is Linux on the desktop for? For example, is it aiming to be a replacement for Windows? If it is, which Windows? Windows at home? Windows in business? Or both? However, I think I've been asking the wrong question. Linux is Linux. What Linux is for is to "manage the hardware and software resources of a computer." What people then do with Linux is an entirely separate question. Of course, the people at Redhat and Novell want us to run Linux on our business desktops: they want to sell us support stuff. What we have to ask is, is that a practical propostion: can Linux be a business friendly desktop?

The do question is important. Not least because the open nature of Linux means that you can do more with Linux than you can do with Windows. If that's something you want to do...

Installing Python 2.4

2007-08-06T13:05:00.000+01:00

One of the things that spurred me on to look at Linux again was that I want to look at Numenta's HTM (Hierarchical Temporal Memory) neural network framework. (There is no Windows version.) The current version of the software is dependent on Python 2.4: it will not run with Python 2.5 which is installed on Fedora 7. So I needed to get Python 2.4 installed.

Building and installing Python 2.4 from the source files was no problem. However, when I ran python2.4 (or just python as 2.4 was now the default installation) I got “python: error while loading shared libraries: libpython2.4.so.1.0: cannot open shared object file: No such file or directory”. libpython2.4.so.1.0 was sitting happily /usr/local/lib/ of course.

The solution was to add the LD_LIBRARY_PATH environment variable:

LD_LIBRARY_PATH=/usr/local/lib/
export LD_LIBRARY_PATH

Adding these lines to my .bashrc file fixed the problem.

D630 Sound Update

2007-08-06T12:49:00.000+01:00

If I enable software sound mixing (ESD) so that I can have System Sounds, things start going wrong. Sound on a video playback, for example, is hopelessly distorted. If, during playback, I mute the system sound and then enable it again everything is fine again - until the next system sound. It's not a major problem to not use ESD, it's just an annoyance.

Java in Firefox

2007-08-06T12:25:00.000+01:00

For some reason when I installed the Java runtime, the Java plugin for Firefox did not get installed. To install it manually, you just need to go the Firefox plugins directory and create a link:

su
/usr/lib/mozilla/plugins
ln -s /usr/java/latest/plugin/i386/ns7/libjavaplugin_oji.so

To test if the Java plugin is working you can go here.

Video Enlightenment through MPlayer

2007-08-06T11:21:00.000+01:00

I'm sitting in ForensiT's new office waiting for BT to turn up and connect us to the outside world. As the office is still an empty box, I've got some time to update this blog.

Last Christmas my wife and I got ourselves a HDD Sony camcorder; no more tapes; no more messing around transferring video off tape and onto a computer. When I plug the camera into my laptop, Fedora does that cool thing that Fedora does, and asks me if I want to import my photos. When I tried to play a video clip however, Totem just complained that I did not have a decoder installed to handle the mpeg2 file format.

Totem uses either the xine-lib or GStreamer libraries. On Fedora it uses the GStreamer libraries by default, so I set about making and installing the gstreamer-mpeg2 plugin. To cut a long story short, although I could make and install the mpeg2 plugin I got no where trying to play my videos. I've never been impressed by Totem and I'm even less impressed now... Then I found MPlayer.

Basically, MPlayer is awesome! You do have to make and install the software yourself, but it isn't difficult, and it is well worth the effort. There are excellent instructions here. MPlayer played my video clips immediately with no additional configuration.

The only problems I have found with MPlayer are to do with Beryl. With Beryl running, you get a “X11 error: BadAlloc (insufficient resources for operation)” error when you try to play a video. The solution is simply to use the X11(XImage/Shm) video driver: you can set this in the MPlayer “Preferences” on the “video” tab if you are using the GUI; you should also set vo=x11 in the .mplayer/config file in case MPlayer gets called from elsewhere. Using the x11 driver fixes just about everything. The only exception I found is playing wide screen videos: the x11 driver forces the video to play at 4:3. You need to switch to the Metacity Window Manager and change the MPlayer video driver back to “xv” if you want to watch wide screen videos in all their glory. The only other problem with Beryl is that the MPlayer skin stays in a rectangular window rather than showing up on its own.

My feeling is that this is a small price to pay. However, this is Linux and what I'm learning is that one size does not fit all – so another solution might suit you better. There is one other advantage with MPlayer, though. MPlayer can handle RealPlayer files. What's more, you can install a plugin so that MPlayer will play multimedia content embedded in Firefox. There are different versions available; I installed mplayerplug-in. (If you do this, SELinux jumps in when you try to watch a video. However, the SELinux Troubleshooter tells you what to do – you chcon a couple of files - and all is well.) After removing the links to the RealPlayer plugins in the /usr/lib/mozilla/plugins directory, I can now watch RealPlayer content in Firefox without having to launch an external viewer – something RealPlayer itself could never do.

VMware

2007-08-01T14:59:00.000+01:00

What did I say about VMware being my "get out of jail free card"? Out of the box VMware 6.0 can't be configured to run on kernel 2.6.22.1. See this topic on the VMware discussion forum. If you do, check out the great post by Grogan on Jul 20, 2007 ;-) Fortunately, there is a great fix by "cf" called vmware-any-any-update113 available here.

Kernel Update Woes

2007-08-01T10:04:00.000+01:00

After upgrading to the 2.6.22.1-33.fc7 kernel, my D630 reverted to the Metacity Gnome Window Manager; trying to run Beryl caused the machine to lockup. So I spent a boring hour or so rebuilding the Intel graphics drivers as I did before. When I finished rebuilding the drivers glxinfo still reported "direct rendering: No" This was a bit strange: /dev/dri/card0 was in place and the Xorg.0.log file reported no errors. More to the point, Beryl worked fine - so I didn't worry about it.

This morning I boot up my machine and the 2.6.22.1-41.fc7 kernel update is available. So, somewhat reluctantly given that I thought I would have to rebuild the drivers all over again, I ran the update. However, this time there was no problem with Beryl at all. Good news! I had no sound, of course...

D630 Sound fixed!

2007-07-30T12:53:00.000+01:00

My thanks go to Mike for his comment here. The updated GATech driver fixed my D630 sound problem. I now have a "STAC92xx Analog" sound device.

I should go on holiday more often! I don't suppose anyone knows anything about Bluetooth? :-)

D630 DVD fixed

2007-07-30T12:19:00.000+01:00

The latest Fedora kernel update 2.6.22.1-33.fc7 fixes the D630 DVD issue. All good things come to those who wait.

Of Linux, Holidays and Bluetooth

2007-07-30T12:17:00.000+01:00

I've been on holiday. But, like a lot of people I guess, I'm not able to cut myself off from work completely and I still needed to check my emails. However, where I went holiday there is not even a phone, let alone a network. So what to do? What I've done in the past is to connect to the Internet via GPRS on my phone, and connect my laptop to my phone via bluetooth. This is fairly straight forward on XP, trivial on Vista, and on Linux?

The blue bluetooth symbol glows brightly on my D630. The lights are on, but there is no one home. The Dell 360 bluetooth card is not recognized. This is something of a surprise: the 360 card has been around for a while and seems to work with various Linux distros. I can only assume that it is not being discovered correctly by the kernel... So I'm faced yet again with a long hunt for drivers, patches, clues in discussion forums, whatever. Except I'm going on holiday and I really haven't got time for this. Fortunately, I've got an old USB Bluetooth adapter manufactured by a company called Bluetake. When I plug this in Fedora finds it instantly. But despite finding some excellent instructions here I just don't have the time to mess around getting it to work. This is Linux at its most hobbyist; this is Linux the Operating System for people who tinker in sheds. Why can't it just work?

I confess I gave up. I took my old Inspiron 1150 with Vista installed. The bluetooth connection to my phone worked flawlessly.

This is one of the great problems with Linux as a business desktop. Even if you are able to develop a standard loadset image that works with your hardware, there is always going to be some senior manger somewhere who will want to do something slightly out of the ordinary and connect to their PDA or phone or something. When the support call comes in someone is going to have to struggle with Linux to get things to work. Because hardware manufacturers target Windows, working with Windows will always be easier.

So why bother? Why spend your valuable time trying to get stuff to work with Linux when you can just plug it into Windows? Personally, I believe there are a number of reasons why the struggle is worthwhile. Here's just one.

Just before I went on holiday, Microsoft filed a patent for an "advertising framework" that would capture "context data" from your Windows PC so that advertisers could show you targeted adverts in any applications that you were using. According to The Register website: "Microsoft's envisions a "context manager" that would gather data from "various data sources" with a "profile manager" and "profile database" storing data "over a period of time" for use in "refining context data for advertisement selection." (See here.)

Now there are a number of reasons why companies take out patents. Sometimes it is a defensive measure: maybe Microsoft want to get a patent on this before someone else does; someone like Google. However, I can't help thinking of a presentation I attended a few years ago where a Microsoft employee talked about their vision of Office as a service. It could be that there will be more than one way to pay for that service - or maybe you won't get the choice at all. Either way, I don't want a "context manager" monitoring what I do on my own machine. Being able to read, and even compile, the code running on my machine has to be the best way of protecting myself.

D630 DVD Update

2007-07-18T10:25:00.000+01:00

My thanks go to Julien for his comment here. Following his suggestion I downloaded the updated kernel-2.6.22.1-20.fc7.i686.rpm from the Fedora updates/testing/7/i386/ directory. The update does fix the DVD problem. Unfortunately, the kernel update also knocked out the D630s wireless configuration. The wlan0 device no longer exists :-(

all-generic-ide

2007-07-05T16:15:00.000+01:00

The ide-generic kernel module, ide-generic.ko, is not present on Fedora 7. Possibly that's one reason why the all-generic-ide option won't work.

I guess its possible to build it from the kernel source...

D630 on Ubuntu

2007-07-05T14:09:00.000+01:00

There is an interesting post from on the Ubantu forum here. "Lek130" writes:

The issue still is that the d630 has some new components ... Device drivers updates for linux are not top prio for HW Vendors, I am afraid. That is an issue in itself.

I tried a few distros OpenSuse10.1, FedoraCore6, Fedora7, etc.
OpenSuse did not at even get close.
FC6 worked well from CD - no issue. But upgrading to F7 killed the system
F7 CD did not recognize the storage controller.

Feisty Fawn U7.04 was the one I could get quickly working.

Ubuntu 7.04 has the 2.6.20-16 kernel. F7 has the later 2.6.21-1 kernel. Given that FC6 worked, (I found myself that I could install from the FC6 CD,) it looks like something got broken in the later kernel.

D630 Update

2007-06-28T15:28:00.000+01:00

I'm getting no where quickly trying to get the DVD and sound to work on the D630.

There are a lot of posts on different Linux forums describing the DVD problem on similar - but not identical - chipsets. The best I found is here. The advice is to add the parameter all-generic-ide to the "kernel" line in the GRUB bootloader config file /etc/grub.conf:

kernel /vmlinuz-2.6.18-1.2798.fc6 ro root=/dev/VolGroup00/LogVol00 all-generic-ide rhgb quiet

Unfortunately, it doesn't work on the D630.

I have also tried different BIOS options. In the BIOS, under "Onboard Devices","SATA Operation" is set to ACHI (Advanced Host Controller Interface .) The BIOS says the factory default is ATA. Obviously that's not the Dell factory default. If you try to change it you are warned that "SATA Operation must be AHCI when the Flash Cache Module is enabled." Presumably, Dell have enabled the Flash Cache Module to allow the machine to support Vista ReadyDrive. I disabled the Flash Cache and set "SATA Operation" to ATA: Fedora would no longer boot. I did try booting from the Fedora 7 setup DVD (thinking that a reinstall might not be too high a price to pay to get the DVD working) but it was still not recognized - not even with all-generic-ide.

I've had no better luck with the sound card. I have rebuilt the ALSA (Advanced Linux Sound Architecture) driver, library and utilites but it has made no difference.

It is not just a case then, of getting the latest drivers or changing some settings. The brutal fact seems to be that the current builds of the Linux kernel and drivers simply do not fully support the GM965 chipset. I can probably live without the DVD and without any sound for a while until updates are released, which of course they will be. However, this does highlight one of the difficulties of choosing Linux as your OS - especially in a business environment. Businesses tend to have regular hardware refresh cycles; they need to be comfortable that their loadset images are going to work with the hardware they purchase. The dominance of Windows means that hardware vendors support for Linux will at best be an after-thought, or more likely just be left to the Linux community.

Dell D630 Wireless

2007-06-25T11:25:00.000+01:00

The Dell D630 wireless card is supported by Fedora out of the box. For once, I don't need to build any software. All that is required is that you configure it for your network as below.

The two outstanding items are the DVD drive and the sound card. The DVD drive is probably the most important, so I'm going to look at that first.

Dell D630 Graphics Fixed!

2007-06-25T09:58:00.000+01:00

I built the Intel 3D driver following the excellent instructions on the Intel website here. I've had no trouble at all building the drivers on Fedora 7 - so if you are thinking about it, don't be put off. Having built and installed the drivers however, I still had no 3D functionality. beryl no longer whitescreened: it didn't do anything. The Intel instructions tell you to run glxinfo to check that direct rendering has been enabled. You should see something like this:

$ glxinfo grep direct
direct rendering: Yes

(grep is one of the wonders of the unix world. I can't believe Microsoft didn't clone it in the early days.) What I saw was "direct rendering: no"

One of my problems is that I'm having to learn Linux as I go along, and it takes me a while to find out about the basic stuff that I really should know about. Every time the X Window System starts up, it creates a a log file at /var/log/Xorg.0.log. This log file is packed with information, and tells you just about everything you need to know about what's happening with your graphics setup. For example:

(II) Loading /usr/lib/xorg/modules/drivers//intel_drv.so
(II) Module intel: vendor="X.Org Foundation"
compiled for 1.3.0, module version = 2.0.0

I wish I'd checked this before upgrading the 2D driver. I would love to know if version 2.0.0 is installed when you install Fedora 7. If you know, please leave a comment.

Looking through Xorg.0.log I saw the following entry:

drmOpenDevice: node name is /dev/dri/card0
drmOpenDevice: open result is -1, (No such device)

There are lots of questions on the Linux forums along the lines The directory /dev/dri exists, but there is no card0 device. How can I create it? The answer is that you don't: the X server creates it as needed. What allows the X server to create the card0 device is drm. My mistake was that I hadn't rebuilt drm along with the graphics drivers. I rebuilt drm (again following the Intel instructions) and then rebuilt the 3D graphics drivers just in case. After a reboot /dev/dri/card0 was present and glxinfo reported "direct rendering: Yes"

I started beryl. Still nothing. By starting beryl in a terminal window with the -d switch, I could see errors like this:

$ beryl-manager -d

[...]

Xlib: extension "XVideo" missing on display ":0.0".
xvinfo: No X-Video Extension on :0.0

[...]

Checking for XSync extension : failed

No sync extension
beryl: No sync extension
Xlib: extension "SHAPE" missing on display ":0.0".

The X library "extensions" are loaded via libextmod.so, which was present on the system, so what was going on? The solution was simple (once I knew what it was.) All I had to do was to explicitly tell the X Windows System to load the module by adding Load "extmod" to the "Module" section of the /etc/X11/xorg.conf file:

Section "Module"
Load "glx"
Load "dri"
Load "extmod"
EndSection

That was it - job done! After a restart everything worked and beryl was back. I'm still struggling with the D630 screen. It is not as clear as when the same machine was running Vista, but that is only part of the problem. This a 1440x900 resolution screen that is physically only 30cm x 19cm in size; 14.1" wide aspect. My 19" desktop monitor runs at 1280x1024. Even on Vista most people would want to use larger scale 120dpi fonts. It requires a lot of fine tuning.

Building the Dell D630 Graphics Driver (Cont.)

2007-06-22T10:57:00.000+01:00

Thanks to this post I realised that I was missing the xorg-server.pc file. This is installed by the SDK for developing X server driver modules. Doh! Well you live and learn. You can install the SDK from the Fedora xorg-x11-server-sdk package. That done, the driver built beautifully.

./autogen.sh --prefix=/usr
make
su
make install

If you don's specify the prefix, the drivers get installed to /usr/local/lib/xorg/modules/drivers/ rather than /usr/lib/xorg/modules/drivers/ which is the drivers directory used by Fedora.

And the driver has made absolutely no difference at all :-(

The system says it is still using the "Intel - Experimental modesetting driver for Intel integrated graphics chipsets" video card. More importantly, the display looks just like it did before. Was Fedora already using this driver? I haven't been able to find out. However, the job is only half done. I've built the 2D driver; I now have to build the 3D driver.

Building the Dell D630 Graphics Driver

2007-06-22T09:17:00.000+01:00

Fedora 7 on the D630 is not a practical option as things stand, because of the lack of support in the OS for key components of the hardware. I could of course wait until that support arrives, but if I want to run Linux now I really only have one option: that is to enable support for the hardware myself. On the plus side, Intel have released open source drivers for the graphics card and (I believe) the sound card. I don't know about the DVD drive. On the downside, I need to build the drivers myself.

So, ignoring Intel's ominous warning that "Compiling and/or upgrading graphics drivers in Linux is a complex and error-prone task", I've downloaded the source code from Intel here. To download the driver software you need git, which is not installed on Fedora by default. However, you can easily install the git package using yum or pirut (Add/Remove Software.)

git-clone git://anongit.freedesktop.org/git/xorg/driver/xf86-video-intel

Once you have downloaded the driver software, cd to the xf86-video-intel directory that git creates, and see what happens when you start to compile the drivers:

./autogen.sh

This first thing that happened when I ran autogen.sh was that I got an error checking for intel-gen4asm. This was easily fixed by downloading and making the software:

git-clone git://anongit.freedesktop.org/git/xorg/app/intel-gen4asm
cd intel-gen4asm
./autogen.sh
make
su
make install

However, I'm now getting this error:

./configure: line 20243: syntax error near unexpected token `XINERAMA,'
./configure: line 20243: `XORG_DRIVER_CHECK_EXT(XINERAMA, xineramaproto)'

For once, there are plenty of hits on Google for this error, which is said to be related to the xineramaproto.pc file. However, I have the xineramaproto.pc in my /usr/lib/pkgconfig/ directory, which is on my PKG_CONFIG_PATH. I'm guessing it is an environment variable problem, but at the moment this is holding me up.

Installing Fedora 7 on a Dell D630 Laptop (Cont.)

2007-06-19T20:15:00.000+01:00

The machine did boot. However, all is not well. The DVD drive, which caused the installation problems, does not show up. I've no sound. The sound card detected is the "Intel Corporation 82801H (ICH8 Family) HD Audio Controller." Which - presumably - is wrong.

The system is using an "Intel - Experimental modesetting driver for Intel integrated graphics chipsets" driver for the D630's Intel GMA (Graphics Media Accelerator) X3100 graphics card. This works up to a point. It is successfully driving the display at 1440x900 resolution, but the screen has an oddly milky appearance. Support for the GM965 Chipset as a whole is obviously an issue. I have a vision of looking in Windows Device Manager and seeing all those yellow exclamation marks in the device tree.

And tragically trying to run Beryl is a disaster. All I get is a screen whiteout. It's probably the experimental graphics card driver. All that graphics capability is going to waste... I need to get better hardware support from somewhere, or things are looking bad.

Installing Fedora 7 on a Dell D630 Laptop

2007-06-19T13:54:00.000+01:00

I start by booting the Laptop from the Fedora 7 install DVD. On the "Installation Method" screen you are asked "Which type of media contains the packages to be installed?" I choose "Local CDROM" Immediately there is a problem. I get an error message saying "No driver found":

Unable to find any devices of the type needed for this installation type. Would you like to manually select your driver or use a driver disk

Selecting any of the listed drivers just brings me back to the same screen.

I try booting from a Fedora Core 6 CD instead of the Fedora 7 DVD. This time everything is fine. Obviously, the Fedora 7 installer doesn't contain any drivers for my DVD drive, or more likely the ICH8M Intel Ultra ATA controller it is hanging off of. Not good. Still, I should be able to get Fedora 7 loaded from CD instead of DVD. Accept... There aren't any CD images. The Fedora Project website states "Beginning with Fedora 7, it is recommended to use Live images to install on a system with a CD-ROM only." So I try that. The problem is the Fedora 7 Live CD doesn't work either! All I get is an error:

---------------------------------------------------
WARNING: Cannot find root file system!
---------------------------------------------------
Create symlink /dev/root and then exit this shell to continue the boot sequence.

For a few minutes it looks like I'm hosed and that I'm going have to go back to Vista. But then I have a better idea. I boot from the DVD again. This time when I'm asked what type of media contains the pacakges to be installed I choose "HTTP." I'm prompted to configure TCP/IP (and just accept the defaults) and the machine gets an IP address. I'm then prompted for a Web site name and the Fedora directory. It works! The installer is now pulling down the packages from a mirror site.

Trying to install Linux on the latest hardware is clearly a problem. I just hope the system boots when the installation is complete.

Decision Time

2007-06-19T12:32:00.000+01:00

My new Dell 630 laptop has been delivered. It is time to put my money where my mouth is. Will Fedora be my main OS on my new laptop?

If you have read any of the posts here, you will know what the answer is already. Despite the problems and compromises, it's a no-brainer: I'm going with Linux.

Of course, because of VMWare I do have a get-out-of-jail-free card: I can run multiple OSs on the same system - and being a Windows developer I will definitely need to do that. But because I really want to explore Linux, Linux has to be the main OS. If I was configuring this machine for a colleague, I probably wouldn't make the same choice. Probably... But I'm going to keep that discussion for another time.

Interestingly, despite Dell saying "The fully configurable Latitude D630 can come pre-loaded with new Windows Vista, Windows XP or FreeDOS to suit your business needs..." Dell wouldn't sell me a D630 without Windows pre-installed. However, after some negotiation with our account manager a mutually acceptable deal was struck :-)

My first impression of the D630 was how much smaller it seemed than the older latitudes I've used in the past. This is the result, I think, of the wide aspect screen which makes the machine narrower - and lighter. It is not a wholly positive impression - the screen does seem small. Still, I guess I'll get used to it.

I love Beryl (True)

2007-06-18T11:23:00.000+01:00

What is it about Linux? Perhaps because of the collaborative nature of the Linux community, because of the strict rules on how things are done necessitated by its collaborative nature, nobody wants to shout about anything, even when they have something to shout about. Fedora 7 is taken to be a "bleeding edge" distro. So why isn't beryl installed by default, or at least why isn't there an option to install it?

What's beryl? Beryl is an OpenGL accelerated desktop that gives you all the features you've heard about from Windows Vista: "glass" toolbars; 3-D Windows transformations; Windows "thumbnails." What's more, because it's based on OpenGL, you do not need 256Mb of dedicated Video RAM to get it to work. Beryl works beautifully on my old Toshiba laptop.

Now I have read articles where is beryl is discussed in dismissive tones as "eye-candy." Anyone who takes that position is betraying a profound ignorance of the computer market. As I have discussed elsewhere, the "wow" in Microsoft's multi-million dollar marketing campaign for Windows Vista is precisely this "eye-candy." Beryl blows it away.

So install it! On Fedora 7 it's right there. Just go to "Add/Remove Software" and install beryl-gnome - the "meta-package" to install all beryl components. Once you are happy that everything is running, you will want to get beryl to start automatically. Just type gnome-session-properties in a terminal window and add the Beryl Manager to Startup Programs. The default install location is /usr/bin/beryl-manager.

This being Linux, you can now spend many happy hours running the Beryl Setting Manager to configure all the different options with all their different parameters. Absolutely brilliant!

RealPlayer

2007-06-15T20:07:00.000+01:00

One area where it might be thought that Linux lags behind Windows and OS X is in delivering multimedia content, and in particular video. Possibly this is because the idea of a free Operating System does not sit comfortably with the restrictions imposed by Digital Rights Management (DRM) and enforced - not to say embraced - by Windows Media Player, QuickTime and iTunes.

So what if I want to watch some free content, say from the BBC news website? Well, things aren't good. If you try to watch a news story you just get a error saying "Not yet supported."

The software that does not yet support the video format is the Totem Movie Player for the GNOME desktop. At this point you only have one option, and that is to install RealPlayer.

Now sometimes small things trip you up. How difficult could it be to get RealPlayer working with Firefox? I've been trying (on and off) for a week.

There are two ways to download RealPlayer: as a binary file - RealPlayer10GOLD.bin - and as a .rpm. If you go for the binary you have to make it executable first:

su
chmod +x RealPlayer10GOLD.bin
./RealPlayer10GOLD.bin

You then get prompted for the install directory:

Directory: [/home/David/RealPlayer]: /usr/share/RealPlayer

Note that the default install location is completely unhelpful...

Unfortunately, there is a good chance that the install won't run at all. Instead you will get an error along the lines of "error while loading shared libraries: libstdc++.so.5: cannot open shared object file: No such file or directory." RealPlayer, it turns out, tries to link to an old C++ library. We fix this by calling yum which installs compat-libstdc++-33:

yum install libstdc++.so.5

At least if you run the binary you see this problem early. If you use the .rpm everything appears to be OK until you try to run RealPlayer. As is so often the case with the Linux GUI nothing happens. You have to run RealPlayer from a terminal to see that the library is missing.

Run RealPlayer from the "Applications/Sound & Video" menu first. RealPlayer will then configure the (Mozilla) Firefox plugin for you. So we're good to go, right? Er... no. If I go back and try and watch the news story again all I see is this:

Now what I can do is launch the video in a standalone RealPlayer, although that is hardly ideal. Sadly, that is as far as I've got. I've spent a long time trying different configurations and on searching for a solution, but without success. The BBC website itself notes:

You may find that you are only able to listen to live radio using the Listen using stand-alone Real Player link.

Disappointing.

Liberation Fonts

2007-06-14T11:46:00.000+01:00

The font issue that I described here is a serious one. Red Hat have made a new set of fonts available, called Liberation, which are free to use and free to distribute. You can read more here: http://www.press.redhat.com/2007/05/09/liberation-fonts/

Firefox still looks crap, but Red Hat should be given a great deal of credit for beginning to address the issue.

Fedora 7

2007-06-14T09:52:00.000+01:00

I've been running Fedora 7 for almost two weeks. To be honest, in terms of the user experience the changes over Fedora 6 are pretty minimal. The most notable features are the new Fedora default theme with its blue folders, and the advent of Fast User Switching à la Windows XP, Vista and OS X.

Like OS X, a "User Switcher" option appears in the Notification Area on the desktop next to the clock. Or, at least it should. I upgraded my previous installation of FC6 to Fedora 7 so that I could keep my home folder and settings (and all the work I'd done with Evolution.) The "User Switcher" does not appear in my upgraded desktop the Notification Area, although it does appear on the desktops of any new users I create. This isn't really a problem for me, and I haven't spent any time investigating.

Upgrading was easy. The biggest problem I had was that GRUB (the Linux boot loader) insisted on trying to boot FC6 - which no longer existed. I could hit a key to choose Fedora 7 from the menu, but I usually forgot. The answer was to edit the /boot/grub/menu.lst file and set the "default" value to zero so that GRUB would boot the first option listed (Fedora 7) rather than FC6. I suppose I could just have deleted the FC6 option altogether, but I was worried about screwing things up and not being able to boot at all...

Other than that, I had to rebuild my MadWifi wireless driver so that it would work with the new kernel, and - strangely - I had to reinstall the Java runtime. But that was it.

I continue to be very impressed with the Linux OS. My feeling about Linux "office" applications is less unequivocal. Fedora 7 comes with OpenOffice 2.2. Although OpenOffice applications are perfectly practical, because they are imitations of Microsoft Office applications you cannot help but compare them to the original: and if you do that, they are always going to fall short; they are always going to look and feel less slick and more clunky. I did download StarOffice, the paid-for version of OpenOffice from Sun. My advice would be don't bother. These is no difference in terms of functionality as far I could tell. StarOffice is supported, of course, but other than that there seems no reason to purchase.

Despite my misgivings about OpenOffice, and despite my frustrations with Evolution, I'm using Fedora as my laptop OS. Since installing Linux on an old Toshiba laptop about three weeks ago, I've only gone back to my Vista laptop once.

A final word on building Evolution 2.10

2007-06-04T09:49:00.000+01:00

Happily I have now resolved the two errors that were causing me problems. The first error -"Could not connect to Evolution Exchange backend process" - was caused by the wrong version of evolution-data-server running in the background. The problem seems to have been related to the evolution-data-server entry in the bonobo-activation-config.xml file - which I swear I had already updated. The second error - "Could not create composer window" - was also fixed by adding an entry to the bonobo-activation-config.xml file: this time, for gtkhtml. The key here was to rebuild gtkhtml with --prefix so the new version got installed into a different directory. I then put the new path in the bonobo-activation-config.xml file so that it was the new version that got called by Evolution not the version originally installed. This is an important lesson learnt. There is so much integration between gnome applications, like Evolution, and the gnome desktop that you really need to keep any updated builds separate.

So what is Bonobo? Bonono "is the GNOME architecture for creating reusable software components and compound documents." (See here.) It is a set of CORBA interfaces

which will ultimately provide the ability to [do] a lot of things that users of GUI-systems such as MacOS and Windows have gotten used to, such as embedding spreadsheets in word-processing documents, reusable user interface controls, scriptable components etc. (Here.)

It's COM for Linux. I won't go any deeper into this now. Suffice to say, I think, that unlike COM, CORBA has never really been wholly embraced by developers. Rightly or wrongly, it has a reputation for being overly complex - as if COM is not complex enough. However, it is clear that by editing the bonobo-activation-config.xml file we have some control over which version of a component gets called.

So Evolution 2.10.2 is working! What's more, Fedora 7 ships with version 2.10.1, so maybe my efforts have been of some practical value: I will still be running software I build myself on Fedora 7. I did finally find some instructions for building Evolution - although much too late for me. The website is http://www.go-evolution.org/ Thanks to Novell for providing a link on the main Evolution site... not. I have to say I have to disagree with the author of the instructions. They start off saying "building it ain't really hard at all" which is true; but then in the very next sentence go on to say:

Normal users (who aren't planning to start developing Evolution) shouldn't be reading this. Really. I'm serious about this! You will probably not understand a single thing about this documentation if you don't have a strong Linux-development background!

This is developer bull. You do not need to know anything at all about development to make and install software on Linux.

But that is enough about Evolution. Fedora 7 is out and its time for something new.

Running Evolution 2.10

2007-06-01T09:45:00.000+01:00

I wish I could say I was running Evolution 2.10. The truth however, is that despite having successfully built the software it isn't working.

The first error I get is "Could not connect to Evolution Exchange backend process" when trying to connect to Exchange:

My feeling is that this is related to having two versions of Evolution and evolution-data-server installed on the machine. If I force a shutdown with evolution-2.10 --force-shutdown I get the message "Shutting down evolution-data-server-1.8" which is the wrong version.

That's bad enough, but I can't reply to any mails at all without a "Could not create composer window" error - and a crash.

So why not just remove the old version of Evolution? The problem with that is the way that the Fedora Package Manager works - either though yum or pirut ("Add/Remove software".) The Package Manger is excellent at installing any dependencies a software package might need to run when it is installed, but when it comes to uninstalling the software things go a bit wrong. Evolution is dependent on Gnome libraries in order to run. So what do you think happens if you uninstall Evoluton? Gnome gets uninstalled as well. That's right, you lose your desktop. Genius!

Fortunately, I was running VMWare when I discovered this little quirk, so I could just rollback. However, this is a serious failing. You should surely be able to remove your email client without removing your desktop.

So perhaps I just need to find a way of getting Evolution-2.10 to run with the right build of evolution-data-server... But then, Fedora 7 has been released.

Building Evolution-Exchange

2007-05-31T09:21:00.000+01:00

Nailed it! Thanks to this post I discovered that evolution-data-server doesn't build libexchange-storage by default because the libexchange libraries are dependent on openldap - and by default evolution-data-server doesn't link with openldap. Thanks a lot. To fix the problem I needed to build evolution-data-server again, this time specifying openldap:

sh autogen.sh --with-openldap=yes --prefix=/opt/evolution-data-server

Once I had installed the new build of evolution-data-server I could finally build evolution-exchange. Having run make install, the Exchange option was available in Evolution 2.10:

Finally!

UPDATE: Unfortunately, although the Exchange option was available, there was no where to enter the OWA URL! I had to rebuild evolution with openldap enabled: sh autogen.sh --with-openldap=yes --prefix=/opt/evolution

The documentation on building Evolution is non-existent; I have been entirely dependent on forums for scraps of information. This is not so much open source as obscure source software: you can have the source code but don't expect us to help you. But why did I expect anything else? I wouldn't expect much from a small "community" based development project, but there is no getting away from the fact that Evolution is Novell's baby.

Now let's see if it works.

Wireless Revisited

2007-05-31T08:45:00.000+01:00

When I booted my machine this morning I got a message saying that updates were available. Dutifully I downloaded the updates and rebooted. When Fedora came back up I had lost my wireless connection. Not only had I lost the connection, I had lost my wireless device!

One of the automatic updates had updated the kernel, and this wiped out the wireless driver. To fix it I had to run the MadWifi make install again, and then run modprobe ath_pci. In other words, I had to reinstall the driver.

Now this is not too bad for me with my one machine, but what if I was a support engineer responsible for 100 or 1000 machines? Certainly most large enterprises don't just allow automatic updates to be run on Windows workstations without testing first, but if I was looking to move my organization to Linux I would want to look carefully at how updates are managed.

Building Evolution 2.10

2007-05-30T10:59:00.000+01:00

To give a detailed account of the problems I've had building Evolution 2.10 would take me days. Instead I'm going to post an annotated list of the steps I've taken. I have now managed to successfully build and run Evolution 2.10, but it has been something of a vanity project: I did it so that I could learn to do it. With Fedora 7 due out in a couple of days - with hopefully an updated version of Evolution that will work with Exchange - there was probably no actual necessity for me to do this. Still, I did learn a lot.

To build Evolution, we first need to build the modules Evolution depends on. The Evolution website tells us these are libsoup, gtkhtml and evolution-exchange. The recommended way to download the source files is via svn - the Subversion version control system. So in a terminal window we start out here:

svn checkout http://svn.gnome.org/svn/libsoup/tags/LIBSOUP_2_2_98 libsoup
cd libsoup
sh autogen.sh
no gnome-autogen.sh in
You need to install gnome-common from the GNOME CVS

We're in trouble straight away. So install gnome-common:
su
yum install gnome-common
exit

Try again:
sh autogen.sh
make
su
make install
exit

Download and build gtkhtml:
cd ..
svn checkout http://svn.gnome.org/svn/gtkhtml/branches/gnome-2-18 gtkhtml
cd gtkhtml
sh autogen.sh
***Error***: You must have intltool >= 0.25 installed

Trouble again. Install intltool:
su
yum install intltool
exit

Try to build gtkhtml again:
sh autogen.sh
checking for intltool >= 0.35.5... 0.35.0 found
configure: error: Your intltool is too old. You need intltool 0.35.5 or later.

Oops! yum has installed an old version. So we need to remove it...
su
yum remove intltool
exit

Download and build the latest version of intltool:
cd ..
svn checkout http://svn.gnome.org/svn/intltool/trunk intltool
cd intltool
sh autogen.sh
make
su
make install
exit

Back to gtkhtml:
cd ../gtkhtml
sh autogen.sh

***Error***: some autoconf macros required to build gtkhtml
were not found in your aclocal path, or some forbidden
macros were found. Perhaps you need to adjust your
ACLOCAL_FLAGS?

Groan... To fix this I added the ACLOCAL_FLAGS environment variable to my .bashrc file and set it to the aclocal directory which was installed by gnome-common. .bashrc is hidden in your home directory and sets variables for your bash terminal.

export ACLOCAL_FLAGS="-I /usr/local/share/aclocal"

Restart the terminal window and try to build gtkhtml again:
cd ../gtkhtml
sh autogen.sh
make
su
make install
exit

Now for Evolution itself. We don't want the new version of Evolution to clash with version 2.8 which is installed by Fedora. To do this we specify a new installation directory by using the --prefix configure parameter. We can pass this to configure via autogen.sh:

svn checkout http://svn.gnome.org/svn/evolution/branches/gnome-2-18 evolution
cd evolution
sh autogen.sh --prefix=/opt/evolution

But we get errors:
./configure: line 1354: GNOME_DOC_INIT: command not found
./configure: line 1359: syntax error near unexpected token `evolution,'
./configure: line 1359: `AM_INIT_AUTOMAKE(evolution, 2.10.2)'

To fix this, we update the PKG_CONFIG_PATH environment varaible. We can do this in .bashrc again. gnome-common also created the /usr/local/lib/pkgconfig directory.

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

Go again:
cd evolution
sh autogen.sh --prefix=/opt/evolution

checking for CAMEL_GROUPWISE... configure: error: Package requirements (camel-provider-1.2 libedataserver-1.2 >= 1.9.4 libegroupwise-1.2 >= 1.9.4) were not met:

Requested 'libedataserver-1.2 >= 1.9.4' but version of libedataserver is 1.8.3
Requested 'libegroupwise-1.2 >= 1.9.4' but version of libegroupwise is 1.8.3

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables CAMEL_GROUPWISE_CFLAGS
and CAMEL_GROUPWISE_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.

This is seriously bad news. These libraries are installed by evolution-data-server. The Evolution webpage didn't mention having to build evolution-data-server but it looks like we have no choice:

cd ..
svn checkout http://svn.gnome.org/svn/evolution-data-server/branches/gnome-2-18 evolution-data-server
cd evolution-data-server
sh autogen.sh --prefix=/opt/evolution-data-server
make
su
make install
exit

Amazingly it built! Update the PKG_CONFIG_PATH with the new libraries:

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/opt/evolution/lib/pkgconfig

Try Evolution again:
cd evolution
sh autogen.sh --prefix=/opt/evolution

checking for CAMEL_EXCHANGE... configure: error: Package requirements (libbonoboui-2.0 >= 2.4.2 libglade-2.0 libgnomeprint-2.2 libgnomeprintui-2.2 gthread-2.0 gconf-2.0 camel-provider-1.2 libebook-1.2 >= 1.9.4 libedataserverui-1.2 libexchange-storage-1.2 >= 1.9.4 libecal-1.2) were not met:

Requested 'libexchange-storage-1.2 >= 1.9.4' but version of libexchange is 1.8.3

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

WHAT??! After a lot of messing around I noticed that the development packages for the installed versions of Evolution and evolution-data-server were also installed on the system. I removed these using the Package Manager (Add/Remove Software) and tried again.

cd evolution
sh autogen.sh --prefix=/opt/evolution
make
su
make install
exit

Success! All we need to do is add our path /opt/evolution/lib/bonobo/servers to the /etc/bonobo-activation/bonobo-activation-config.xml configuration file (see here) and we're ready to run Evolution 2.10.

/opt/evolution/bin/evolution-2.10

Evolution even keeps your old settings.

Looking through this post, everything seems logical and straight-forward. In truth, it has taken me over a week to get to this point. It has been incredably frustrating. I have spent hours experimenting and googling. I've not found any clear instructions.

You may have noticed that despite building Evolution, I have not build evolution-exchange to allow me to connect Evolution to Exchange. This was the reason for building the updated version of Exchange in the first place. The reason I haven't built Evolution-Exchange is because I still don't know how to fix this error:

Download and attempt to build:
svn checkout http://svn.gnome.org/svn/evolution-exchange/branches/gnome-2-18 evolution-exchange
sh autogen.sh

checking for LIBEXCHANGE... configure: error: Package requirements (libsoup-2.2 evolution-shell-2.10 libedataserverui-1.2 libexchange-storage-1.2 libbonobo-2.0 libxml-2.0 gconf-2.0) were not met:

No package 'libexchange-storage-1.2' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables LIBEXCHANGE_CFLAGS
and LIBEXCHANGE_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.

But, as far as I can tell, libexchange-storage-1.2 is created by evolution-data-server...

Going Wireless

2007-05-30T09:13:00.000+01:00

The problem with having Fedora installed on VMWare is that I can only use it when I'm sat at my desk. With the holiday weekend coming up, I wanted to give myself some options. I remembered that we have an old Toshiba laptop in a cupboard, so I decided to install Fedora on that.

Everything went very smoothly - I had no installation problems at all. However, having the laptop wasn't going to be much good without Internet access, so I needed to install my NetGear wireless card: just inserting it in the card slot didn't work ;-)

The Netgear card uses an Atheros chipset. Fortunately, there is great support for Atheros based cards on Linux from MadWifi.

The first step is to install the driver. You need to download the driver source files which come as a bzip2 tar file. The easiest thing to do is just double-click the file to get the archive manager to open it, and drag the compressed source folder somewhere convenient - like your home directory. MadWifi have some really good instructions for building and installing the driver here.

One of the biggest differences I've discovered running Linux after running Windows is that building the software should be seen as the first stage of the installation process, not the last stage of the development process. This is a result of both the software being open source, and needing to run on multiple versions of multiple distributions of Linux. The good news is that you do not need to know anything at all about developing software to build the source code.

There are standard methods of building software. The first thing you need to do is run a terminal window and cd to the source directory (which you have probably just unzipped.) Next you may need to run autoconf. This will generate a configure file, although often the configure file has already been generated for you. You now run configure. configure will set up the files needed to build the software on your system.

To make life even easier, many source packages come with a shell script called autogen.sh which does all this for you. If there's a autogen.sh file you should run that instead of autoconf and configure. Just type sh autogen.sh.

Whether you have run autoconf and configure, or just autogen.sh you should now be prompted to run make. If the software builds correctly (and this can take some time depending on the software) you can su and then run make install which copies the compiled software into all the right directories.

With MadWifi all you have to do is run make and then make install. Because MadWifi is a driver you need to add it to the kernel using modprobe. (See the MadWifi instructions.)

Getting the wireless card to function isn't hard, but there are some curiosities. Firstly, by default, two wireless "devices" get created by MadWifi. This is because MadWiFi supports virtual access points. The first device is wifi0, which I think of as the wireless card. The second device is ath0 which is the virtual device which is actually used by the OS. It is ath0 that you configure in order to connect you your wireless network. The only problem here is that it is wifi0 that is brought up when the system boots, and the system will attempt to get an IP address. This leads to a long pause on bootup until the "connection" times out. To stop this you can use the "Network Configuration" utility (from the System/Administration menu) to configure wifi0 to use a static IP address which can just be left blank.

The MadWifi instructions tell you how to configure the wireless connection manually, but what if you want to start the wireless connection automatically everytime your machines boots? Isn't that what most people want to do?

The answer is to edit your /etc/rc.d/rc.local file. This shell file gets run on bootup after the system has finished loading. You just need to add the wireless configuration commands to the file:

#Bring up the virtual card
/sbin/ifconfig ath0 up

#Set the authentication mode
/sbin/iwpriv ath0 authmode 1

#Specify the Wireless Network
/sbin/iwconfig ath0 essid network_name key password

#Get an IP address
/sbin/dhclient ath0

Not too difficult at all.

Evolution Exchange Bugs

2007-05-22T10:44:00.000+01:00

Evolution appeared to have successfully connected to Exchange and synchronized my mail. However, when I clicked on one of my mail folders I immediately got an "Error while Refreshing folder" error saying "Lost connection to Evolution Exchange backend process"

To make matters worse, CPU usage hit 100% and stayed there until I closed down Evolution.

To cut a long and frustrating story short, this is a known bug. The "solution" is to delete the mail data that Evolution has cached by opening a terminal window and typing the following:

evolution -–force-shutdown
cd ~/.evolution
cd mail/exchange
ls (to get the name of the directory)
rm -rfv ./(name of directory)
Restart Evolution

I'm very grateful to the person who posted the fix here. However, this isn't really a fix. It doesn't stop the problem from occurring, it just removes the problem when it does. So why does it occur? My hunch is that the problem happens if you have your mail account open somewhere else. I'm still looking into Linux being my main laptop OS - I have (or had) my mail account open on another machine. Time will tell if just having my mail account on Fedora will prevent the problem from occurring.

But then there's another problem. Even when Evolution has correctly synchronized with Exchange and is happily connected, new mail does not show up in my Inbox. If I get new mail, the Inbox icon shows that I have an unread message, but the message itself does not appear in the Inbox. What's more, no amount of Send/Receiving or attempting to refresh the Inbox will help. The only way to see the new mail message is to shutdown Evolution and restart. This just about makes Evolution unusable as my default email client.

There is some hope, however. Fedora comes with Evolution 2.8, and version 2.8 of the Exchange Connector. New versions of both these pieces of software are available - although not via yum, or even via a rpm file. Looks like I might have to do a real Linux installation.

Connecting to Exchange

2007-05-21T15:02:00.000+01:00

I have to admit that this was the job I was dreading. I had visions of having to set up an IMAP connection to Exchange, with all the horrors of having to configure both Exchange and Fedora. In fact, it didn't turn out that way. Connecting to Exchange was relatively easy: the problems came after...

The Linux email client is Evolution. Despite being an open source application, Evolution seems to have been primarily developed by Novell who "sponsor" openSUSE and sell SUSE Linux Enterprise. The recommended way to connect Evolution to Exchange is via Outlook Web Access (OWA.) OWA is the Exchange webmail service and has been available since Exchange Server 5.0. OWA is enabled by default.

In order to connect Evolution to your Exchange server you need to install the Evolution Exchange connector. This is not installed by default on Fedora 6. Fortunately, with yum it is simple to install it:

yum install evolution-connector

Once the connector is installed, you can start Evolution.

If you are starting Evolution for the first time, the Evolution Account Assistant "Mail Configuration" Wizard will fire up to guide you through setting up your mail account. You click "Forward" to begin, and then enter your name and email address on the "Identity" page. Now the interesting bit. on the "Receiving Email" page you need to select "Microsoft Exchange" as the Server Type. If you haven't installed the Exchange connector, "Microsoft Exchange" will not be listed. You need to enter your user name and the OWA URL. This is not as scary as it sounds; the OWA URL is generally just something like:

http://exchange_server.mydomain.com/Exchange

Click the "Authenticate" button to enter your password and make sure that you can connect to your Exchange server. You can't click "Forward" until you do.

On the "Receiving Options" page, you need to add your GC (Global Catalog) server name. This is probably your Domain Controller, but depending on your domain architecture might not be. There are then some options to select. Perhaps the most crucial is "Automatically synchronize account locally." In other words, keep a copy of your email account on the machine so that you can read your email off line.

Why wouldn't you select this? After all, if you didn't want to access your email off line you could just type your OWA URL into Firefox and view your email that way - that's what it's for after all. The answer is that Evolution allows you per folder synchronization. In other words you can just keep email in specific mail folders locally on the machine. I would like my whole email account available offline, so I ticked the option. To finish setting up your account, you just need to click "Forward", give your account a name, click "Forward" again, and click "Apply" to finish. Evolution creates a new "Exchange" folder and then tries to connect to Exchange.

Unfortunately, it didn't work. What I got was "Error while scanning folders... Could not authenticate to server":

I quick check with Wireshark showed that once again the problem related to authenticating via NTLMSSP. There is obviously a problem with the way I have things set up on Fedora that I will have to investigate. To get things to work I edited my account settings (via "Preferences" on the "Edit" menu) and changed the "Authentication Type" on the "Receiving Email" tab to "Plaintext Password" instead of "Secure Password." I restarted Evolution; Evolution connected to Exchange, and began to synchronize my mail.

And that was when my problems began...

Why does Firefox look so crap?

2007-05-16T11:21:00.000+01:00

With the best will in the world, Linux has not got the best looking desktop you are ever going to see. Vista looks fantastic: the text is crisp and clear and easy on the eye. Linux is... functional. Well, perhaps I can live with that. What I find harder to put up with is browsing the Internet. Web browsing on Linux is a disgrace. If there is one application that defines someone's use of their computer today it is their web browser, so it had better be good. Web pages on Firefox look bad. Even Firefox webpages on Firefox look bad. The problem though, is a Linux problem not just a Firefox problem.

Now there are somethings that you can do to make things less bad. The key thing you can do is install some of Microsoft's fonts! Some years ago Microsoft released a set of TrueType fonts that people could download for free to make their browsing experience better. Microsoft have since removed the download from their website, but the set of fonts was picked up by the Linux community who have clung onto them ever since - presumably because they are the only decent fonts they've got.

The problem is fonts ain't free - and the idea that something someone has created through their own skill, imagination and endeavour might have a value is of course an anathema in the Linux world.

There are good instructions on installing the Microsoft fonts here. I won't repeat them, but I did need to reboot before they showed up in the list of fonts available. (See "System" menu, then "Preferences" and "Fonts".) The msttcorefonts-2.0-1.spec file that the instructions tell you to download is worth looking through if you are interested in learning about what is going on behind the scenes. The Microsoft fonts are installed in a "msttcorefonts" directory under /usr/share/fonts and the directory is added to the list of available fonts by calling /usr/sbin/chkfontpath --add %{fontdir}. The /usr/bin/fc-cache utility is then called. This showed some "invalid cache file" errors on my installation. However, the cache files in question were not to do with the Microsoft fonts. They were all *-x86 cache files. I'm running 64-bit Fedora, and I assume that may be the problem. Where they came from I don't know; perhaps they were installed by Fedora.

Once the Microsoft fonts are installed you can set about tweaking Firefox so that it looks more like Internet Explorer, i.e., less ugly. In Firefox you need to go to the "Edit" menu, and then "Preferences." You can then choose "Content" and change the fonts. I've chosen to mostly use Verdana. To be fair, the actual text on most web pages is now ok, although not on this one :-( Where things still look bad is on pages like BBC news that use a lot of bold text. The way Firefox (or Linux) renders bold text is extremely poor. As yet I haven't found a solution.

Yum

2007-05-15T10:09:00.000+01:00

When I was investigating the errors I got browsing the network I needed to install Wireshark. To install the software I used yum. yum is a is a software package manager and represents a massive leap forward over how things used to be on Linux not that long ago. No more decompressing tarballs and typing in long lists of commands from an install readme file: yum automatically works out dependencies, downloads the software and installs it for you. All I needed to do to install Wireshark was type:

yum install wireshark-gnome

The guys with the neck-beards (as Don Box calls them) must be shaking their heads!

You don't even need to go near a terminal window. You can browse and install software packages just by clicking on "Add/Remove Software" from the "Applications" menu. yum is a major step towards making Linux a business friendly operating system for the desktop. However, it is only a step.

Printing

2007-05-15T09:39:00.000+01:00

Connecting to shared Windows printer is very easy. It is simply a case of clicking on "Printing" from the "System/Administration" menu, clicking on the "New Printer" button and following the prompts. You choose "Windows Printer via SAMBA" on the "Select Connection" page and type in the path. (You need to specify a username and password even if you are authenticating via AD.)

The most difficult task is selecting which driver to use for your printer. I resorted to Google for advice. With the correct driver installed, printing is not a problem.

NetBios Name Service

2007-05-14T20:01:00.000+01:00

When I browse "Windows Network" it is the NetBios Name Service requests that Fedora issues on port 32770 that get blocked by the firewall. In fact it is not just port 32770: there are a range of ports. To date the port range that needs to be unblocked on the firewall is 32770-32796.

Connecting to a Windows Share

2007-05-14T11:55:00.000+01:00

If you are reading this my guess is that you are probably in the same situation as me: you're looking at moving from Windows to Linux. Obviously, I want to make sure any information I give out here is as accurate as possible. Connecting to a Windows file share is a very common thing to want to do, especially if you are coming from a Windows background. However, for me, it wasn't without its complications.

In a previous post I described joining my Linux machine to my Windows domain. Once I did this I had no problems at all mounting a Windows share using "Connect to Server..." from the "Places" menu on the Fedora desktop. (I'm using the Gnome desktop, which is the Fedora default. If you are using a KDE desktop, the menus are probably different. This is something I hope to come back to.) However, I got "The folder contents could not be displayed" error when I clicked on the "Windows Network" icon.

Like most companies, my company runs a network firewall. Having tried everything I could think of on Fedora, I enabled logging on the firewall just to check whether anything was showing up. Sure enough, the firewall was denying connections from my Linux machine on ports 32770 and 32771. Once I enabled the ports I was able to browse the Windows Network with no problems at all. Very cool...

Ok, but what if the machine is not joined to the domain? This is where things got complicated. The bottom line is that I have not been able to mount a Windows share using "Connect to Server..." from the desktop. I've spent a long time looking at this, but I've got no where. What I can do is mount the share from the command line:

mount -t cifs -o user=username,password=password
//Server/Share /mnt/Name

So what's going on? One possibility I considered was the issue of SMB packet signing. (There is a good discussion here.) However, if that was the case I wouldn't be able to mount the share from the command line. I downloaded Wireshark (formally Ethereal) and saw that I was getting a STATUS_LOGON_FAILURE error:

Doesn't help much... I ran a Wireshark again, this time to capture what happened when I ran mount. The difference was that mapping a drive from the desktop involved Fedora attempting to connect using SMB via NTLMSSP (NT Lan Manager Security Support Provider) whereas using mount did not.

I'm sure there is a solution to this somewhere; either by changing the configuration of the firewall, or the configuration of Fedora. However, I'm going to have to leave this for another time.

Camera

2007-05-10T20:03:00.000+01:00

I needed to download some pictures from my camera. I usually need XP to do this because Canon have not released any Vista drivers. I thought I would plug the camera into Fedora and see what happens. I immediately got a "Camera Import" message saying "A camera has been detected." I clicked on the "Import Photos" button and it just worked brilliantly. Why can't Vista do this?

Connecting to Active Directory

2007-05-10T13:58:00.000+01:00

As you may recall, my To-Do list looks like this:

Connect to my company's Exchange Server and get my mail.
Connect to my company's Windows 2003 Server and access files.
Print stuff.
Optionally authenticate via Active Directory.

As it turns out, the last and "optional" requirement is the first one that I've tackled. This is because it showed up on the "Create User" screen when I was installing. There is good news and then there is... well let me explain.

First the good news. Getting Fedora to authenticate via Active Directory turned out to be surprising easy. From the "System" menu you choose "Administration" and then "Authentication" - this brings up the "Authentication Configuration" dialog box. On the "Authentication" tab, you tick "Enable Kerberos Support" and click the "Configure Kerberos" button. The Kerberos "Realm" is just the DNS name of your domain; the KDCs are your Domain Controllers (port 88 by default); and the "Admin Servers" are your Domain Controllers again, this time using port 749.

Next tick "Enable SMB Support" and click the "Configure SMB..." button. The "Workgroup" is your domain; the "Domain Controllers" are your domain controllers. Simple.

Finally, tick to "Enable Winbind Support" and click the "Configure Winbind..." button. The "Security Model" needs to be "ads" and you just have to fill in the boxes with your domain details. Having done that, you click the "Join Domain" button and... not for the first time on Fedora nothing happens. I ran the "Active Directory Users and Computers" MMC Snap-in on a Windows machine, and sure enough my Linux workstation appeared in the default "Computers" container in AD. Would it have been so difficult to have a confirmation messagebox? I sometimes think that for Linux developers the GUI is an afterthought, and ease of use is not thought about much at all.

I logged off, and logged on again with my domain username and password - and it worked! Perhaps I shouldn't be surprised that this worked, but I was expecting the worst.

I then logged off again, and logged on with another Active Directory account. This time the logon failed with "Incorrect username or password." What was going on? It turns out that the reason the logon failed was that the user account did not already exist on the machine. There is a suble and important distinction to be made here between Linux and Windows. Linux only uses AD to authenticate the user's logon: the user's account exists on the machine. Windows also uses AD to authenticate a user's logon, but the user's account exists in AD.

So you may be wondering, with some justification, what's the point of authenticating via AD? There are some advantages, however. You can mount Windows network shares on the domain (via "Connect to Server..." on the "Places" menu) without needing to specify a username and password. Additionally, I hoped that I would be be able to browse my domain in "Windows Network." Unfortunately I'm getting an error:

There are obviously still some issues to be resolved.

Updates...

2007-05-10T10:34:00.000+01:00

The 259 updates took 12 hours to install! During that time the machine was essentially unusable. This is a one-off, of course, but I'm not impressed...

Here Beginneth the Lesson

2007-05-09T10:32:00.000+01:00

I'm running Fedora on VMWare. To get an Operating System to run smoothly on VMWare you need to install VMTools. Now if you are reading this, and you are not going to be running Linux on VMWare, I would still urge you to bear with me, because what I'm about to describe has got to be one of the most profound differences between Windows and Linux there is.

When you install VMWare tools - or just about anything else - on Windows you run a setup file; something like setup.exe. The setup file launches a Wizard, like InstallShield or Advanced Installer, you click "Next" a few times and your software just gets installed. Sometimes you may need to reboot, but that's about it. When you try to install VMWare Tools on Linux, things are kind of different...

Firstly, there is no setup.exe file. What you are presented with are two install "packages." One is a tar.gz file - basically a big zip file; the other is a .rpm file. One of the reasons that I chose Fedora is that it supports .rpm files. RPM files were developed by Redhat (RPM originally stood for Red Hat Package Manager) to make installing software simplier on Linux. The Package Manager unpacks the application files to the right places and can run the configuration scripts so that the application will run on your machine. Coming from Windows, that doesn't seem much to ask.

So I run the VMWare Tools .rpm file and eventually I get a messagebox saying that the software installed successfully- excellent. I click "Ok" and... nothing's changed. There's no dialog box, no readme file, nothing on the "Applications" menu, nothing at all to indicate what I should do next. So its off to Google... What I need to do is to is run a run a script (written in perl) called vmware-config-tools.pl. The script is installed in the /usr/bin directory. (I will be coming back to file locations in future posts.) I open a terminal, "su" to run as root, and run the script. I then get a truly magnificent message:

None of the pre-built vmmemctl modules for VMWare Tools is suitable for your running kernel. Do you want this program to try to build the vmmemctl module for your system (you need to have a C complier installed on your system)?
[Yes]

Just stop and think about that for a moment! The program is offering to compile a module -that is, actually build the binary code - on the fly. That is the difference between Linux and Windows right there. The program can only do that because the source code is open; the souce code is available.

So I hit return and keep hitting return to accept all the default answers. The program calls the gcc compiler and builds the VMWare Tools modules in front of my eyes.

As far as I can gather, there is nothing unusual about what has just happened; other applications can do the same thing. This is why you should install the Software Development software even if you are not doing any software development. I ran the .rpm file, but in this case I might just as well of run the tar.gz file to unpack the files. With some applications there will only be a tar.gz file, and you may have to do the compiling yourself. I hope though, that you are as excited by the possibilities as I am.

Installing Fedora

2007-05-08T13:40:00.000+01:00

Installing Fedora on VMWare is a breeze. For anyone who doesn't know, VMWare Workstation is desktop virtualization software that allows you to run multiple operating systems simultaneously on a single PC. It must be one of the most innovative software products to emerge in recent years.

To install Fedora I downloaded the 64-bit Fedora Core 6 CD .iso files from one of the Fedora mirror sites. Whether choosing the 64-bit version was a mistake, time will tell. There are 6 .iso files, but you only need the first three if you do a default installation. Then I created a new VMWare Virtual Machine, pointed the CD-ROM to the .iso file, and clicked "Start."

The Fedora install program is very straight-forward and will seem quite familiar to anyone who has installed Windows. You are prompted about disk partitioning, but if you are installing to a clean disk you just need to accept the defaults.

One thing that is worth pointing out is the Network Devices screen. This asks you how you want to set the machine's Hostname. The default is to set the hostname "automatically via DHCP." Coming from a Windows background, this was alien to me. As far as I know, Windows DHCP servers do not set workstation hostnames (although I may have missed something.) On Windows the workstation hostname is set on the workstation. Because I'm running Linux in a Windows envirionment, I decided to set the hostname manually.

The final screen asks you about about what additional software you want installed. By default, "Office and Productivity" software is ticked: in other words, you get OpenOffice. Optionally you can check "Software Development" and "Web Server." Even if you are not planning to do any software development, I would say that you should tick the option. I'll come on to the reason later. Click "Next" and the installation is ready to begin.

It takes well over an hour, which is about twice as long as XP or Vista. Having said that, the installation is completely automatic, so you are not tied to the machine. Once the machine reboots, you need to answer a few more questions. You need to accept the license agreement. You can then set the Firewall settings: in addition to trusting SSH, I ticked Samba because I want to run in a Windows Environment. I accepted the default setting for SELinux - which is enabled. You then need to create your user account. This is just like creating an account when you install Windows, accept that the username is case sensitive as well as the password.

The "Create User" screen has an interesting option. There is a button which says "Use Network Logon..." If you click the button - and you know what you're doing - you can configure the machine to authenticate users via Active Directory using Winbind. However, you can also configure Winbind once Fedora is installed, so for now I've postponed this.

Click "Forward" and you are just about finished. You can now logon to Fedora for the first time. Fedora will try and connect over the Internet to see if there are any security updates available. There were 259 updates available after I installed Fedora: maybe Linux isn't so different to Windows after all...

Which Linux?

2007-05-08T11:58:00.000+01:00

If you are planning to run Linux the first thing that you need to decide is which Linux you want to run. From the outside looking in, the dark continent of Linux looks depressingly tribal. LinuxCD.org boasts a 1000+ versions of Linux and BSD. What? And they think that's a good thing? I guess it depends on your point of view.

If you are coming to Linux from Windows you naturally want to know what's the best "distro" to install. Do you install openSUSE which is sponsored by Novell? Or Gentoo? Or Fedora which is sponsored by Redhat? Or Ubuntu which you are going to be able buy pre-installed on Dell machines? Or something else?

I'm going to stick my neck out here. You know what? It doesn't much matter. Linux is Linux: whichever "distro" you install you are going to get basically the same Operating System with the same applications.

It is time for me to fess up. I've been playing around with Linux for a long time. But "playing" is right: I've never really got to grips with Linux and I've never used Linux as my main OS. I first installed Linux from a Yggdrasil distro. I still have the CD which is dated "Fall 1993." The kernel version is 0.99.13. But then came Windows 95... Over the years I have regularly installed Linux to see what was happening. In the main it has been Redhat Linux that I have used. So, because I'm familiar with Redhat, but also because Redhat have a strong commercial rather than home focus, I'm going to install Fedora.

Here's the plan. My trusty Dell Inspiron 1150 laptop is slowly grinding to a halt. In a few months I'm going to buy a new one. Should I install Linux on my new laptop? Because I'm not brave/stupid (delete where applicable) enough to just dump Windows, I'm going to test Linux out on VMWare before making a decision. I must be able to:

Connect to my company's Exchange Server and get my mail.
Connect to my company's Windows 2003 Server and access files.
Print stuff.
Optionally authenticate via Active Directory.

That is a pretty modest set of objectives, I think. They are, of course, business focused objectives and the focus for this blog will be in using Linux for business.

It is time to download Fedora.

Why Linux?

2007-05-08T10:56:00.000+01:00

I'm a Windows man. I spend my days sitting in front of Windows Vista writing software for Windows computers. My company is a Microsoft Certified Partner. I am a Microsoft Certified Professional. So what am I doing getting involved with Linux?

I don't have a simple answer to the question. Partly, I'm interested in Linux for the same reason I got interested in computers in the first place: I want to know how it works. Linux is important. It is important in my industry both commercially and - to some extent - technically. Linux also has an importance culturally.

So who is this blog for? Why do people blog anyway? Who do they expect is going to read their stuff? My hope is that if someone is considering moving from Windows to Linux they might come across this blog and find something I have to say useful. If that's you, please post a comment! Otherwise how am I ever going to know? The only other audience I have in mind for this blog is myself. If I can keep track of what I've done, hopefully I won't repeat my mistakes.

My plan is to keep my posts here short but frequent. I will also try to keep my posts as contemporaneous as possible with what I'm actually doing with Linux.

It's time to start.