Setting up my new ASUS UX301LA

I spoilt myself in 2013 for my Birthday and Christmas and bought the beautiful ASUS Zenbook UX301LA. The model I ordered comes with a touch screen WQHD (2560x1440) display, an Intel i7 4558U CPU, 8GiB of RAM and 2 internal SSDs. Needless to say it's very cool!

The laptop comes with OEM Windows 8, which despite all the bad geek press online I actually like it. Without a touch screen though it would be useless but with a touch screen it works quite well. The 2560x1440 screen is almost too small - I find myself squinting at certain things, so I run the display at 1080p instead.

Regardless of all this my new toy was designed to encourage me to get back into developing on my own time, so a Linux distro is needed. The stock controller setting joins the disks into one RAID 0 volume, but I want independent disks for Windows and Linux, so first step was to change the controller mode from RAID to AHCI.

My Linux distro family of choice is Red Hat. Trying a Fedora 20 Live CD is very unstable - it constantly freezes, the Fedora installer gets python crashes, haven't been able to get through the installer once after several attempts. I generally get about 5-6 minutes in the disk partitioning area before it freezes again. Even with the ATA controller in AHCI mode (broken RAID) Fedora must be reading a few sectors into the disk because it still thinks the disks are part of a RAID disk. Trashing the partition table on /dev/sda, Fedora now allows me to try partition that disk, but it's still too unstable to install.

Faced with no operating system on my lovely Gorilla Glass encased toy, I torrent down a Win 8 ISO, preying there's no keyloggers buried in the image. Microsoft in their infinite wisdom don't make it easy for OEM users to download install media, and no activation key I could find on the Internet would either. I actually own a valid copy of Windows and still have to download an ISO. Go figure.

The Windows 8 ISO could only install onto sdb, said sda was an "unbootable disk". Strange, but that's what I want anyway - sda will be the Linux install - so away we go and Windows 8 re-installs perfectly fine. I need to put all the OEM software and drivers back on. ASUS are always good with this kind of stuff and the downloads are easy to find. Once they are all installed I've pretty much got a usable laptop back, and not just one very expensive mirror.

The Fedora Live CD is still freezing the laptop - I hoped it was a blue moon thing but I guess not. I tried Ubuntu 13.10 seems a bit better, but still get the odd freeze. This could be due to the virtual CDROM drive I'm using, but I got a working Ubuntu 13.10 install eventually. The touch screen and video support on Ubuntu is good, it supports the native 2560x1440 resolution out of the box.

I'm still such a Red Hat fan and feel lost on Debian systems, so went to try Fedora one last time. It got through an install. It's no where near the partition layout I'd like but I'm just glad to have an RPM based system at my fingertips again. First thing I discover is I can't lower the screen resolution on Fedora 20. Looks like the EDID for the laptop screen is not being picked up. I need to dive into X11 to force some different resolutions. Wow, haven't done this in 7 years... The ArchLinux wiki is always helpful but trying to force 1920x1080 crashes Gnome Shell.

I've also noticed that Fedora 20 and Ubuntu 13.10 activates some crazy power saving when the laptop is running off the battery which pushes the fan speed up to the highest setting. It's not completely unusable, but definitely noisy and annoying. The ArchLinux wiki has more information on the Intel Graphics drivers, but this post says not for Fedora 20 yet, in a few weeks perhaps.

The Fedora GRUB installation picked up the Windows install but the boot parameters are not enough to boot Windows 8 on the second disk. This Stack Exchange post contains exactly what is needed:

menuentry 'Windows 8.1 Pro' {
insmod part_gpt
insmod fat
set root='hd1,gpt2'
insmod search_fs_uuid
insmod chain
search --fs-uuid --no-floppy --set=root --hint-bios=hd1,gpt2 --hint-efi=hd1,gpt2 --hint-baremetal=ahci1,gpt2
chainloader /EFI/Microsoft/Boot/bootmgfw.efi
}

The touch screen is not as good on Fedora as well, so combined with the screen resolution I decide to swallow my Debian bigotry and go back to Ubuntu. For now.

There are two remaining problems. The first is on Linux and Windows: when the laptop is left untouched for some amount of time, sometimes it gets into some strange hibernation state and nothing I do can wake it up - it has to be power cycled.

The second is just in Linux - every few hours it freezes. Some times running videos or sound seems to trigger it a lot more frequently. Not sure how I'm going to track down what is most likely a driver problem. Suggestions are welcome.

Diagnosing a Drupal 7 cache generation problem

My company launched their new website recently. When we launched before Christmas we encountered a reoccurring problem that was more difficult than most to diagnose. The problem itself is very specific to our site so I doubt the exact details will help many people, but maybe the troubleshooting steps involved will prove interesting to someone. I'm not particularly proud of the time it took to track down nor our exact thought process (hardly blowing my own horn with this post) but here we go anyway.

The website platform was built for us by a third party, the technology is mostly Drupal 7 with some custom modules written for functionality we required. We wrote our own "Drupal deployment interface" that mirrors the contents of one Drupal site (our Dev server) onto our UAT or Live platform. The Live platform is a simple Apache / Drupal / Varnish stack with a load balancer in front of several web servers, the back-end is several MySQL servers.

When we deployed our final site to launch, we ran into a problem where a specific image on our front page was not displaying. Looking at the HTML source when the image is broken and we see that the image source rather bizarrely contains the hostname 127.0.0.1:

<img alt="" class="media-image" typeof="foaf:Image"
  src="http://127.0.0.1/sites/default/files/LMAX-intro-video.jpg" />

Not being Drupal 7 experts we can't code dive into it's PHP with much confidence so on comes the black box testing and some facts we discovered:

1. We confirm this section of HTML is dynamically generated - it's not a hard coded link to 127.0.0.1 someone's typed into the Drupal interface.
2. No other images are broken, just this one.
3. Looking at other images, the source of the image should be starting with "http://www.lmax.com/...".
4. If we request the correct image link directly it loads fine, so the image file is not missing nor does there appear to be a problem with Apache serving the file.
5. We restart Varnish on all web servers to see if this is problem between Varnish and Drupal but it does not fix the problem.
6. We dump all databases and grep for the offending string and pin the problem to one table.

The bad HTML is being stored in Drupal 7's cache_filter table. We delete the entry from the cache_filter table, refresh our site in a browser and the problem is solved, but unfortunately not for good.

The next day the problem re-occurred - our web developer says that he deployed a new copy of the site onto production and the image is missing again. We investigate again and find the same bad HTML on the database servers. We delete the entry from the cache_filter table again then check our website - the image is still broken. Looking back in the cache_filter table and we see that the same bad HTML has been regenerated, despite us deleting that row. Just to be sure, we truncate the entire cache_filter table on both databases and refresh - still contains 127.0.0.1. What we thought resolved our problem yesterday has not worked a second time and we now have no quick fix way of solving it.

We convey to the business that we can't fix this in five minutes and settle down for some more serious investigation. We now know:

1. The tail end of the problem is the Drupal generated HTML stored in the cache_filter table in our database(s).
2. The problem appears to occur after a deployment of new content from our Dev server to our Live servers.
3. We specifically don't restore any cache table content when doing a deployment to avoid any "stale" cache from Dev reaching Live - so after deployment, the cache tables are empty.
4. Something is continually repopulating the cache with bad HTML.

We have a UAT environment to test deployment specific problems, built for exactly these kinds of problems. We only managed to reproduce the problem once in several test deployments of the same content to our UAT environment - the issue is very intermittent on UAT and practically constant on Live. UAT does not have any load balancing normally, we add some but still cannot reproduce.

We search through the core Drupal 7 PHP code, our custom modules and contributed modules for mention of the host 127.0.0.1. It appears a few times but leads no where relevant. We also spend time playing with Varnish on UAT: we know that each Varnish server's Apache backend is configured over the loopback interface and it's written in the Varnish configuration file with '127.0.0.1'. Our work proves unhelpful there as well.

Trying a different approach, we turn on full query logging on the UAT database, deploy to UAT and browse around our website, looking for "insert into cache_filter" lines. Our thinking is to trace back through the queries for an idea of what occurs before the cache_filter insert and thus hint at what's populating this table. The UAT query log does not help much: we find the insert query but the problem has not occurred after the deployment so the cache_filter contents is correct. Other than witnessing a lot of queries against the domain table, the UAT query log is not very helpful.

We decide to turn on the full query log for one of the production MySQL servers, as we were not happy that our efforts in UAT had exhausted this avenue. We finally have our eureka moment: within seconds of turning on the log we see a queries against the domain table, but these are ever so slightly different:

SELECT domain_id, subdomain, sitename FROM domain WHERE subdomain = '127.0.0.1'

We had a general idea of the domain module and that it works by what hostname someone puts in the browser, so these queries said to us that someone or something was hitting localhost with URL requests and they are getting far enough into our web stack for Drupal to query for it. We immediately revisit Varnish but can't prove it is the cause on UAT yet again. We compare the Apache logs with the Varnish logs, we think on how UAT (unfortunately) differs to production and finally the sack of pennies drops.

The answer was in front of us the entire time - The load balancers use HTTP health checks of "GET /" against the web servers. The load balancer health checks run continuously almost every second and so when a deployment occurs against Live, the load balancers will almost always be the first request to the front page of the website. Since we effectively truncate the cache tables when we deploy, the load balancer health check triggers Drupal to repopulate it's cache. Something about the load balancer's request is causing Drupal to search for a '127.0.0.1' domain, perhaps incomplete HTTP headers, or maybe a REMOTE_HOST header of 127.0.0.1. Since we don't have a domain of 127.0.0.1 the request falls back to our default domain (a feature of the domain module) but somehow content for the front page is being generated incorrectly with details from the original request and cached.

To confirm what was only a theory at this point we changed the load balancer health check to just test the TCP connection rather than a HTTP test, waited for a request to come through and checked the site - the generated content from our request was correct. Rather than keep the TCP health check we found an example Drupal PHP script that does a minimal Drupal bootstrap to check the database health and return HTTP status codes appropriately.

The clarity of hindsight:

• When we had the issue the very first time, after deleting the bad cache_filter entry I must have refreshed the website faster than the load balancers check, hiding the problem until the next day.
• When we added the load balancer to UAT, we mustn't have set up a health check (or if so, only a TCP connection check), as we were unable to reproduce the problem in UAT. Lesson: if trying to mirror production, mirror production.

The vast majority of the problem is now worked around, but it is not solved - the issue still re-occurs every once every couple of weeks, in Live and UAT now as well. There are still several questions that I would like to answer:

1. What is the exact part of the load balancer request that caused Drupal to generate it's cache incorrectly? Is it the REMOTE_HOST header?
2. Is it just the load balancer or was Varnish also a catalyst? If we take Varnish out of the mix and just have the load balancer point to Apache directly, do we still have a problem?
3. What's causing the very infrequent re-occurrences of the problem now? Could it be the Varnish cache expiring and requesting a new copy of the object?

Like all Systems Administration problems though, it will get attention when it annoys someone enough to justify spending the time to fix it permanently. If only computers didn't exist, our lives would be so much simpler...

Sound through HDMI and an nVidia GeForce 210

Continuing on from the previous post about my new HP Microserver, I was mainly interested in seeing how the CPUs did playing a HD movie. Watching the CPU load while playing back a H.264 preview and it wasn't as bad as I expected: both cores sat around 30-60% usage, which includes background system tasks. For the usage I expect of my media server this is sufficient, it's rare that it's playing any video at the same time as anything else. It's also worth mentioning that considering the vast range of formats of most of my media it would be highly unlikely if even the majority would benefit from much offloading to the GPU, however it would be nice to have, so I was still keen on a graphics card with the necessary features.

Considering the space inside the Microserver and the positioning of the PCIe slots, I've got a rather restrictive list of graphics card requirements:

• heat sink cannot be “thicker” than the width of a PCI slot
• low profile
• nVidia (Linux drivers), preferably with a GPU with good PureVideo support
• fanless/passive
• HDMI out, but would settle for DVI

After a bit of searching and lots of squinting at images of cards trying to determine how high the heat sinks were, we found a range of Zotac GeForce 210 cards that fit the bill. The GT210 is one of the first chips to have nVidia's latest PureVideo technology (although now superseded by generation 5 and the GT520). It has a low form factor face plate replacement in the box and the heat sink is the same thickness as the slot. The card also has a HDMI port so can be my sound card as well. The exact model I ended up ordering was the Zotac ZT-20309-10L GT 210 512MB DDR3 LP Silent for £34. I would have preferred a 1GB RAM model but due to availability in the region I settled on the 512MB.

Installing the card was trickier than I thought. There's some annoying bits of metal at the back of the Micro case that the top of the graphics card's face plate was hitting. Had to bend the metal out of the way with a thin screw driver to get the card to slide back properly. Aside from that, the heat sink fits snugly along the side of the case.

HD video play back performance appears unchanged. This could be for several reasons: first off I've only attempted some very basic tuning of Ubuntu and VLC, there's surely many things I haven't tried. Secondly the format of my test MPEG-4 video may not be one that nVidia's PureVideo can offload. More importantly for me at this time is sound: Ubuntu is not playing out of the HDMI audio controller.

Some research and a lot of time later (this project was shelved for a few weeks), I've got Ubuntu playing out the nVidia audio controller. The XBMC wiki and Arch Linux forums contained the necessary tips, namely the installation of the ALSA sound modules, setting the correct options for the snd-hda-intel kernel module and loading the ALSA module into PulseAudio. The probe mask has ALSA using the nVidia codec of the card:

luke@nexus-micro:~$ grep snd-hda-intel /etc/modprobe.d/alsa-base.conf options snd-hda-intel enable_msi=0 probe_mask=0xfff2

Reboot or rmmod / modprobe, then see what device ALSA detects:

luke@nexus-micro:~$ aplay -l **** List of PLAYBACK Hardware Devices **** card 0: NVidia [HDA NVidia], device 3: HDMI 0 [HDMI 0]
Subdevices: 1/1
Subdevice #0: subdevice #0

ALSA says Card 0 Device 3 so that goes into PulseAudio :

luke@nexus-micro:~$ grep alsa /etc/pulse/default.pa load-module module-alsa-sink device=hw:0,3

And done. Next I'll be looking more into video playback performance and maybe doing a disk swap from old media server to new media server.