Moving emails from old server to new server using bash, python and offlineimap

I had a task of moving emails from an old server to a new server. Considering that stream of emails would be coming in on the live server, rsync didn’t look like a good option to sync the Maildirs.

So, I searched around the Internet for better solutions. The logically better solutions contained using the imap protocol to fetch the emails.

The first tool I came across was imapsync which uses imap to imap sync. But, I had to discard that option as it would involve me knowing passwords for all the users at both ends or resetting passwords on both the servers. There are other tools that would do two way sync. But that was again an overkill for my purpose.

Then I stumbled across offlineimap which is itself written in python. It fetches email from remote imap server and stores it in a local Maildir

First step was to reset the passwords for all the users on the remote server, this is a safe operation as the remote server is not in use anymore and will terminated soon. So, I reset the password of all the users

#!/usr/bin/env bash

PASS="mypass"
for USER in `awk -F: '{print $1}' /etc/passwd`
do
  echo $USER:$PASS >> user_list
done

cat user_list | chpasswd

view raw change-pass This Gist brought to you by GitHub.

Be careful to use the above script as it is. It will reset password for all the users. Modify it to filter as per your requirements

So, now we are all set to setup sync on the local server. First of all, download offlineimap from your distro’s repository. Now we need to set it up to sync the emails. of all the users that exist locally. To do that, we need to add all the users information to the ~/.offlineimaprc file.

Here’s a simple python script to populate the .offlinelimaprc file.

#!/usr/bin/python

userfile = open('/etc/passwd', 'r')
rcfile = open('~/.offlineimaprc','w')

entries = userfile.read().split("\n")
userfile.close()
userlist = []
content_repos = ""

for entry in entries:
  user = entry.split(":")[0]
  userlist.append(user)
  content_repos = content_repos + """
  [Account {0}]
  localrepository = {0}Local
  remoterepository = {0}Remote
  
  [Repository {0}Local]
  type = Maildir
  localfolders = /var/spool/imap/w/user/{0}
    
  [Repository {0}Remote]
  type = IMAP
  remoteserver = my.host.name
  remoteuser = {0}
  remotepass = mypass
  """.format(user)


content_accounts = """
[general]
  accounts = {0}
""".format(",".join(userlist))

content = content_accounts + content_repos
rcfile.write(content)
rcfile.close()

That’s it. Simple enough, we are done. Now run the offlineimap command to get syncing.

 

Getting Lava 720G to work in Ubuntu 11.10/12.04 Beta

I have a Lava 720g 3G USB modem. The modem is supported by the usb_modeswitch distributed with Ubuntu 11.10 and 12.04 Beta.
But, despite that it doesn’t connect out of the box. Here’s a very simple way in which I got it working.

Open a terminal and type following commands in it

sudo modprobe usbserial vendor=0x1c9e product=0×9605
sudo usb_modeswitch -v 0x1c9e -p 0×9605 -V 0x1c9e -P 0×9605 -M “55534243123456788000000080000606f50402527000000000000000000000″

Web.py on lighttpd

So, I wanted to host a small project made in web.py on my current server. I am running lighttpd as my HTTP server.

The process to set it up is pretty straight forward as explained in the webpy documentation.

I am writing this how-to to cover a couple of things which I didn’t find well documented and had to search around a lot to find

  • To get web.py running with lighttpd and fastcgi you need to install the package flup. In debian it is known as python-flup.
  • On this server, I run php based sites as well so I had to setup fastcgi for both. Here’s what the code looks like:
fastcgi.server = ( ".php" => (( 
"bin-path" => "/usr/bin/php5-cgi", 
"socket" => "/tmp/php.socket",
"max-procs" => 1,
"bin-environment" => (
 "PHP_FCGI_CHILDREN" => "4",
 "PHP_FCGI_MAX_REQUESTS" => "1000"
)
)),
"/code.py" => ((
"socket" => "/tmp/fastcgi.socket",
"bin-path" => "/path-to/webpy-app/code.py",
"max-procs" => 1,
"bin-environment" => (
 "REAL_SCRIPT_NAME" => ""
),
"check-local" => "disable"
))
)
  • To use web sockets instead of unix sockets change
"socket" => "/tmp/fastcgi.socket",
to
"host" => "127.0.0.1",
"port" => "8081",

 

 

Mediawiki on dotcloud

Unfortunately, there’s no guide available for installing mediawiki on dotcloud and googling for it doesn’t turn up anything much useful. But, I had to move out LUG wiki to dotcloud. I proceeded with it while refering to drupal documentation. At the moment, I don’t have a guide on installing mediawiki but I can highlight the problem I faced.
The big problem lies in the fact that the setup of MediaWiki was done on Apache and dotcloud runs nginx. So, the problem occured mainly with getting the path right.
The savior comes in form of the MediaWiki documentation for nginx.

Some of the changes I had to make were

LocalSettings.php

$wgUsePathInfo = true;

nginx.conf

try_files $uri $uri/ /index.php?title=$1&$args;

fastcgi.conf

fastcgi_param  PATH_INFO          $fastcgi_path_info;

 

This isn’t a full fledged guide or a how-to but hopefully a pointer for people who are stuck while getting started with MediaWiki on dotcloud or nginx.

Connecting GSM modem via CLI using NetworkManager

After getting my new HDD couple of days back, I decided to install Arch Linux on it. The idea with Arch is to sufficiently control what goes into my system but without too much management overhead unlike in Gentoo. Thus, I installed stated with Core install of Arch and added fluxbox over it as my default WM(I am not running any DM as of now, but might go with qingy). Thus, I have tried to avoid any GNOME or KDE libraries and do as much via minimal system as possible.

Since I disconnected my broadband a few months back, I have been living on Mobile Internet connectivity. So far I had been tethering my phone to the wifi router and then using LAN cable to connect to the PC. Horrible way to go, I agree! To simplify matters I got a USB GSM modem. It was as easy as pie to set up the connection on Ubuntu using Network Manager but with Arch, I decided not to go for nm-applet but manage it via CLI.

First I setup NetworkManager as instructed on the Arch Wiki and I got Wired Connection working in no time. The most important tool at this juncture was nmcli. I checked

nmcli dev

and it showed that gsm modem is being detected. But, on checking

nmcli con

It only showed the wired ethernet connection. Going through the man pages and googling for hints didn’t help at all. I finally turned to #nm IRC channel on Freenode.

Thanks to some excellent support, I was connected using the GSM Modem in no time.

Here’s the quick rundown

  1. Create a new file in /etc/NetworkManager/system-connections and as per the sample file. PS: You can generate uuid using uuidgen or input any random UUID as long it isn’t in use elsewhere in your system.
  2. Now, chmod the file to 0600.
  3. Restart NetworkManager
  4. Voila! What have we here. I could now see the new connection in the list.

Now to get connected to the new connection. First we disconnect the Wired connection

nmcli dev disconnect iface <iface name>

And then connect to the new interface

nmcli con up id <connection id>

For those using MTNL Mumbai, you can use my settings

[connection]
id=MTNL
uuid=15d742f1-2b5a-421e-9f27-fcb1fc26d72c
type=gsm
autoconnect=true

[ipv4]
method=auto

[gsm]
number=*99#
username=mtnl
password=mtnl123
apn=gprsppsmum

[serial]
baud=115200

 

 

 

WordPress Performance Tips

Do you find performance of your WordPress blog to be lousy? And have no clue on where to start with it?

There are lots of articles on the Internet which will tell you what to do. So, instead of repeating the same, I’ll cover each aspect and links to articles which provide useful how-to’s on the same which I have come across while working on OnlyGizmos and iPhoneHelp

I’ll divide it into sections on basis of the type of hosting as it is usually the limiting factor on how much tuning you can do.

Shared Hosting

Having your blog on shared hosting means there’s not much you can change but this is what will suffice for huge number of bloggers out there.

  • Caching Plugin: W3 total cache is the best and highly recommended plugin. With tons of features besides caching like Minification, CDN support, mobile support, Cloudflare integration it is by far the best cache plugin I have come across. For installation and configuration see this wonderful how-to from wpbeginner
  • Theme: Use themes which have been designed with SEO in mind. These themes follow the best practices to speed up your page loading time e.g. putting stylesheets at the top and minify them. The theme framework I would recommend is Thesis. It is a paid theme but having worked with it, I can say that it’s worth the price if you are serious about your website.
  • Scripts: Put javascript files at bottom as recommended in YSlow. This can be handled by W3 Total Cache.

Single Server

In addition to the above, if you have control over your server e.g. VPS or Dedicated server you can take following extra steps -

  • HTTP Server: Move over from Apache to Nginx. Here’s a good guide on moving over your wordpress blog to nginx.
  • MySQL Tuning: Run the mysqltuner script and follow the recommendations given by it to improve MySQL performance.
  • PHP Accelerators: Install PHP Accelerator like APC or XCache. List of accelerators is available on wikipedia. I recommend APC since it’s stable and is likely to be merged into PHP 6.
  • CDN: If you serve lot of static content and media do consider going with a CDN. And if you think you cannot afford CDN’s as they are too costly, think it over. MaxCDN offers excellent CDN at an affordable price of $39.95 for the 1st TB. Do check them out.

Multiple Servers

  • Memcached: One of the components I am yet to personally handle but have heard a lot of good stuff about it. It can provide superior object caching. It has a good support in W3 Total Cache thus making your work of integrating with wordpress easier.
  • Varnish: Place varnish in front of your HTTP server and see your websites fly. Here’s a good article on setting up varnish with nginx, w3tc and APC. And if it isn’t enough official website provides excellent documentation.

These are the things that I have learnt in the past one year. If you have something which I have not covered please share it with us in the comments.

Bye Bye Twitter, Bye Bye Facebook

Many people have been asking me as to why I quit twitter? And haven't got a proper reply from me. I do owe them a reply, and here it is.
I have been a bit prone to mood swings, and at the beginning of December when I started feeling low, I thought it was just one of those days. But, it went on happening for a couple of weeks with no solution in sight. And some of you on twitter may also have noticed me cribbing about some nonsense stuff. Twitter and Facebook were in a way contributing and making me feel worse. So, finally after pondering over the decision for 3-4 days I decided I will quit both Twitter and Facebook.
At the same time, I did not want to miss some of the amazing people I met on these sites. Thus, I decided I will back up my profile and keep it safe so I have it available if I ever decide to return. I know, I will be missing some of these people, I already do. But, I had to take that decision and stick with it.
I still don't know if I will ever return back to those websites or not. Thankfully, facebook allows me to deactivate my profile and preserves it. I have backed up my twitter 'Friends' too.

And as to my mood, I have had the most amazing week of my life, coincidently after quitting Twitter and Facebook – 3 days cycling trip to Pune and Panchgani, a close friends wedding, nice tweetup where I met lots of people. The time spent outdoors and in real life is just what I needed. It has given me a much needed high. And with some of my very good friends likely to meet me this month, I hope it will be just what I needed.

Public Transport v/s Private Transport – My Take

The Problem

With each passing day, the traffic situation in Mumbai is worsening. Big traffic jams, accicents, parking woes, etc. Various solutions have been proposed to combat this problem.

Here's a slightly outdated statistical information on the traffic growth in Mumbai from 2005 to 2008. An annualised growth of 12%! That was 1.5 years ago, it has worsened further now.

""

This problem has been widely debated and discussed topic everywhere. I had the opportunity to be a part of it on two occasions, once on twitter and once during Raxit's session at BCM6. This prompted me to express my views on the topic on this blog.

Many solutions have been proposed to solve our traffic problems – promote public transport, improved road conditions, car pooling, levying of taxes in certain zones, allowing different vehicles on odd and even numbered days.

Preferred Solution

Public transport is the most viable solution, according to me. I am basing this on certain assumptions

  • Only road transport is covered
  • The scenario described is that of Mumbai, it may or may not apply to other places

Reasons

  1. Less Congestion: Mass means of transport like buses and trains can carry higher number of passengers in lesser space compared to vehicles like cars and bikes.
  2. Reusability: Once a person reaches their destination, the same space can be taken up by somebody else. Thus, there's resusability of the same space. In case of private vehicles the space lies vacant till used again.
  3. Space Saving: Since public transport is reusable, we need a lower number of public transport vehicles. Also, that mass means of transport can fit in more people per area. This leads to saving of space for parking and while plying the vehicles on the road.
  4. Flexibility of Timing: As compared to other options like car pooling, public transport is much more flexible with respect to time. Trains are much faster over longer distances, over shorter distances cabs and autorickshaws can take you to your destination as quickly.
  5. Lower Total Cost: Mass transport options like buses and trains are cheaper to travel. Though, cabs and rickshaws may sound expensive, when considered with option of sharing it becomes cheaper. Add to that, the cost of maintenance is to be borne by the vehicle owner unlike public transport.

My Story

But, there are reasons why I use public transport lesser:

  • Is much less fun unless when travelling in a big group over longer distances.
  • Going out and exploring some nice places is better done in your cycle, bike, car as compared to public transport
  • Going out biking with a group of friends once in a while is a must.
  • Trains and buses are too crowded, which makes me choose my 2 wheeler over those.
  • Still, I tend to balance between using public transport and my bicycle (now that my bike is stolen), though I tend much more heavily towards my bicycle.

BarCampMumbai6

The 6th edition of our favourite unconference – BarCamp Mumbai will be held on Sunday, October 11th, 2009 at S. P. Jain Institute of Management and Research.

""
BarCamp Mumbai is an ad-hoc gathering of people born from the desire to share and learn in an open environment. It is an event with discussions, demos and interaction from participants.

BarCamp Mumbai is in essence a conference without a pre-set agenda. We prefer the term ‘unconference’ actually. A bunch of smart people meet up over the weekend, put up a schedule on a wall and spend the rest of their time taking up sessions and discussions with each other. There is no audience. Only participants.

So host a session, help out with planning, ask questions, spread the word- Everybody is invited. There really are no walls – or rules for that matter!

You can register your attendance for the event at http://bcm6.eventbrite.com. If you wish to speak, you can register your topic at http://barcampmumbai.org/register.

Looking forward to meeting you there!

Address:

Sunday, October 11th, 2009.
S. P. Jain Institute of Management and Research,
Bhavans College Campus,
Munshi Nagar, Dadabhai Nagar Road,
Andheri West,
Mumbai

You can find more details on how to reach there on the website.

Curious about the last BarCamp in Mumbai? Check out photographs, and the newspaper coverage from previous barcamps: Photos and News.

Night Cycling

Mumbai is popularly known as city that never sleeps. I have been a witness and contributor to it since college days. Those days we used to spend time with friends sitting around Churchgate station having chai and bun maska or go out with my bike and travel and length and breadth of Mumbai. Mumbai is far from quiet at that hour. It's abuzz with activities from night time food hawkers to merry party goers to cars owners racing on the empty roads to various kinds of shady people lining the streets to make money.
Amongst such night creatures is us new breed of cyclists, who enjoy going out for rides in the night. Joined by a group of 10-15 cycling enthusiasts from all across the city, we gather at a pre-determined starting point and cycle around the city. We are greeted to the sights of curious passer-by's who inquisitively ask us about cycling, teenagers driving around rashly, deserted by-lanes of the city and so much more.

Some of my memorable moments from night ride are:-

  • Wide empty roads of Reclamation.
  • Climb up Mt. Mary road and fast paced descent down to Bandstand.
  • Silent lanes around Carter Road.
  • Curious people popping around to cheer us cyclists.
  • Sitting in group and late night grub at Dadar and CST.
  • Busted party at Worli Seaface.
  • Enjoying the breeze and waves at Worli Seaface and Marine Drive.
  • The tiring steep climb opposite PDP.

Switch to our mobile site