home.dataforce.org.uk Dataforce's Blog

Ident Server

Posted by Dataforce

I recently encountered a problem on a server that I manage where by the oidentd server didn't seem to be working.

Manual tests worked, but connecting to IRC Servers didn't.

I tried switching oidentd with ident2 and the same problem.

After switching back, and a bit of debugging later it appeared that the problem was that the IRC Servers were expecting spaces in the ident reply, whereas oidentd wasn't giving them.

I then quickly threw together an xinet.d-powered ident server with support for spoofing.

First the xinet.d config:

service ident
{
	disable = no
	socket_type = stream
	protocol = tcp
	wait = no
	user = root
	server = /root/identServer.php
	nice = 10
}

Unfortunately yes, this does need to run as root otherwise it is unable to see what process is listening on a socket. In future I plan to change it to allow it to run without needing to be root (by using sudo for the netstat part)

Now for the code itself:

#!/usr/bin/php
<?php
	/**
	 * Simple PHP-Based inetd ident server, version 0.1.
	 * Copyright (c) 2010 - Shane "Dataforce" Mc Cormack
	 * This code is licensed under the MIT License, of which a copy can be found
	 * at http://www.opensource.org/licenses/mit-license.php
	 *
	 * The latest version of the code can be found at
	 * http://home.dataforce.org.uk/index.php?p=news&id=135
	 *
	 * This should be run from inetd, it will take input on stdin and write to stdout.
	 *
	 * By default users can spoof ident by having a .ident file in /home/<username>/.ident
	 * If this is present, it will be read.
	 * It should be a file with a format like so:
	 *
	 * <pid> <ident>
	 * <local host>:<local port>:<target host>:<target port> <ident>
	 *
	 * The first line that matches is used, any bit can be a * and it will always match,
	 * so "* user" is valid. In future more sophisticated matches will be permitted
	 * (eg 127.*) but for now its either all or nothing.
	 *
	 * Its worth noting that <target host> is the host that requests the ident, so if this
	 * is likely to be different than the host that was connected to, then "STRICT_HOST" will
	 * need to be set to false.
	 *
	 * At the moment <local host> is ignored, in future versions this might be changed, so
	 * it is still required.
	 *
	 * Lines with a ':' in them are assumed to be of the second format, and must contain
	 * all 4 sections or they will be ignored.
	 *
	 * Lines starting with a # are ignored.
	 *
	 * There are some special values that can be used as idents:
	 *    ! = Send an error instead.
	 *    * = Send the default ident.
	 *    ? = Send a random ident (In future a 3rd parameter will specify the format,
	 *        # for a number, @ for a letter, ? for either, but this is not implemented yet)
	 *
	 * In future there will also be support for /home/user/.ident.d/ directories, where
	 * every file will be read for the ident response untill one matches.
	 * This will allow multiple processes to create files rather than needing to
	 * lock and edit .ident
	 */

	// Allow spoofing idents.
	define('ALLOW_SPOOF', true);

	// Requesting host must be the same as the host that was connected to.
	define('STRICT_HOST', true);

	// Error to send when '!' is used as an ident.
	define('HIDE_ERROR', 'UNKNOWN-ERROR');

	openlog('simpleIdent', LOG_PID | LOG_ODELAY, LOG_DAEMON);

	$result = 'ERROR : UNKNOWN-ERROR' . "\n";

	$host = $_SERVER['REMOTE_HOST'];

	syslog(LOG_INFO, 'Connection from: '.$host);

	// Red in the line from the socket.
	$fh = @fopen('php://stdin', 'r');
	if ($fh) {
		$input = @fgets($fh);
		$line = trim($input);
		if ($input !== FALSE && !empty($line)) {
			$result = trim($input) . ' : ' . $result;
			// Get the data from it.
			$bits = explode(',', $line);
			$source = trim($bits[0]);
			$dest = isset($bits[1]) ? trim($bits[1]) : '';

			// Check if it is valid
			if (preg_match('/^[0-9]+$/', $source) && preg_match('/^[0-9]+$/', $dest)) {
				// Now actually look for this!
				$match = STRICT_HOST ? ":$source .*$host:$dest " : ":$source.*:$dest";

				$output = `netstat -napW 2>&1 | grep '$match' | awk '{print \$7}'`;

				$bits = explode('/', $output);
				$pid = $bits[0];

				if (preg_match('/^[0-9]+$/', $pid)) {
					$user = `ps -o ruser=SOME-REALLY-WIDE-USERNAMES-ARE-PERMITTED-HERE $pid | tail -n 1`;

					$senduser = trim($user);

					// Look for special ident file: /home/user/.ident this is an ini-format file.
					$file = '/home/'.trim($user).'/.ident';

					if (file_exists($file)) {
						$config = file($file, FILE_SKIP_EMPTY_LINES | FILE_IGNORE_NEW_LINES | FILE_TEXT);
						foreach ($config as $line) {
							// Ignore comments.
							$line = trim($line);
							if (substr($line, 1) == '#') { continue; }

							// Make sure line is valid.
							$bits = explode(' ', $line);
							if (count($bits) == 1) { continue; }

							// Check type of line
							if (strpos($bits[0], ':') !== FALSE) {
								// LocalHost:LocalPort:RemoteHost:RemotePort
								$match = explode(':', $bits[0]);
								if (count($match) != 4) { continue; }

								if (($match[1] == '*' || $match[1] == $source) &&
								    ($match[2] == '*' || $match[2] == $host) &&
								    ($match[3] == '*' || $match[3] == $dest)) {
									syslog(LOG_INFO, 'Spoof for '.$senduser.': '.$line);
									$senduser = $bits[1];
									break;
								}
							} else if ($bits[0] == '*' || $bits[0] == $pid) {
								syslog(LOG_INFO, 'Spoof for '.$senduser.': '.$line);
								$senduser = $bits[1];
							}
						}

						if ($senduser == "*") {
							$senduser = trim(user);
						} else if ($senduser == "?") {
							$senduser = 'user'.rand(1000,9999);
						}
					}

					if ($senduser != "!") {
						$result = $source . ', ' . $dest . ' : USERID : UNIX : ' . trim($senduser);
					} else {
						$result = $source . ', ' . $dest . ' : ERROR : ' . HIDE_ERROR;
					}
				}
			}
		}
	}

	echo $result;
	syslog(LOG_INFO, 'Result: '.$result);
	closelog();
	exit(0);
?>

I welcome any comments about this, or any improvements and hope that it will be useful for someone else.

GMail – apply labels to email from group members

Posted by Dataforce

As Noted by Chris recently on IRC, Google Mail lacks a feature in its ability to automatically label/filter messages - you can't do it based on emails from people in a contact group, short of adding a filter with all their email address on it.

At the time it was mentioned this didn't affect me, however later when I got round to adding loads of labels/filters in gmail (yay for, nicely coloured inbox!) to nicely separate things for me I also ran into this problem, so came up with the following python script that does it for me.

It checks messages, sees if the sender is in the contacts, then checks each group to see if there is a label with that group name that is not already set, then checks to see if the contact is in the group, and finally sets the label if everything matches up.

I ran it initially to tag my entire inbox (set "checkAllIndex" to "True" change "ga.getMessagesByFolder(folderName)" to "ga.getMessagesByFolder(folderName, True)") and now have it running on a 15 minute cron (not using loopMode) to tag new messages for me.

Hopefully this will be useful to someone else, I'm not sure how well it works in general, it worked fine for me with ~700 messages at first, however after a few runs (due to regrouping some contacts) I was greeted by an "Account Lockdown: Unusual Activity Detected" message when trying to do anything - This went away after about 20 minutes, but don't say you wern't warned if it happens to you.

#!/usr/bin/env python
"""
 This script will login to gmail, and add labels to messages for contact groups.

 By default the script will only check items from the past 2 days where email
 was recieved.

 Loop mode can be enabled to save logging in repeatedly from cron.
 Loop mode may fail after some time if google kills the session, or gmail
 becomes unavailable or so. (Untested in these situations). On the other hand
 it may also just keep running indefinetly as if no problem occured, loop mode
 is relatively untested and was added as an after thought.

 When running in loop mode, it is best to have a crontab entry also that checks
 and restarts the script if it dies.

 Copyright 2008 Shane 'Dataforce' Mc Cormack

 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
 in the Software without restriction, including without limitation the rights
 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 copies of the Software, and to permit persons to whom the Software is
 furnished to do so, subject to the following conditions:

 The above copyright notice and this permission notice shall be included in
 all copies or substantial portions of the Software.

 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
"""

# Uncomment the lines below if python can't find libgmail on its own, and edit
# the sys,path.insert to point to where libgmail.py is.

# import sys
# sys.path.insert(0, 'libgmail')
import libgmail
import time

###############################################################################
# Configuration
###############################################################################

# Email Address
email = "YOUR EMAIL HERE"
# Password
password = "YOUR PASS HERE"
# Check all on index, rather than just the first 2 dates found
checkAllIndex = False
# Use Loop (if true the script will keep looping, and sleep between checking
# for new mail to modify)
useLoop = False
# Time in seconds to sleep when looping (300 = 5 mins)
loopTime = 300
# Label Prefix - if group-based labeles are prefixed, set the prefix here.
# (eg "Groups/")
labelPrefix = ""
# What folder to check? ('inbox' or 'all' are probbaly the most common settings)
folderName = 'inbox'

###############################################################################
# Helper classes/methods
###############################################################################

class ContactGroup:
	def __init__(self, id, name, contacts):
		self.id = id
		self.name = name
		self.contacts = contacts

	def containsContact(self, contact):
		for knownContact in self.contacts:
			if knownContact[0] == contact.id:
				return True
		return False

		def __str__(self):
			return self.name

# Get Contacts and Groups
# Modified from libgmail 0.1.10 to include groups aswell
def getContacts(account):
	"""
	Returns a GmailContactList object
	that has all the contacts in it as
	GmailContacts
	"""
	contactList = []
	groupList = []
	# pnl = a is necessary to get *all* contacts
	myUrl = libgmail._buildURL(view='cl',search='contacts', pnl='a')
	myData = account._parsePage(myUrl)
	# This comes back with a dictionary
	# with entry 'cl'
	addresses = myData['cl']

	# Now loop through the addresses and get the contacts
	for entry in addresses:
		if len(entry) >= 6 and entry[0]=='ce':
			newGmailContact = libgmail.GmailContact(entry[1], entry[2], entry[4], entry[5])
			contactList.append(newGmailContact)

	contacts = libgmail.GmailContactList(contactList)

	# And now, the groups
	for entry in addresses:
		if entry[0]=='cle':
			newGroup = ContactGroup(entry[1], entry[2], entry[5])
			groupList.append(newGroup)

	return contacts, groupList

###############################################################################
# Setup
###############################################################################

print "Running.."
print "Use Loop:", useLoop
if useLoop:
	print "  Loop Time:", loopTime
print "Check all on index:", checkAllIndex
print "Label Prefix:", labelPrefix
print "Checking Folder:", folderName
print "libgmail Version:", libgmail.Version
print ""

# Login to gmail
print "Logging in as", email
ga = libgmail.GmailAccount(email, password)
ga.login()

# Loop at least once.
loop = True;

while loop:
	loop = useLoop

	print "Getting label names.."
	# Get Labels
	labels = ga.getLabelNames(refresh=True)
	# Get Messages
	print "Getting messages.."
	inbox = ga.getMessagesByFolder(folderName)
	# Get Contacts
	print "Getting contacts and groups"
	contacts, groups = getContacts(ga)

	# Check each thread in the inbox
	lastDate = '';
	secondDate = False;
	for thread in inbox:
		# Only check dates we are supposed to.
		if not checkAllIndex:
			# Get the date
			threadDate = thread.__getattr__('date');
			# Make sure a date is set
			if lastDate == '':
				lastDate = threadDate

			# If this date is different to the last one do something.
			if lastDate != threadDate:
				# If we are already on the second date, then we stop now
				if secondDate:
					break;
				# Otherwise, if the new data is a non-time date, we can change to the
				# second date.
				elif "am" not in threadDate and "pm" not in threadDate:
					lastDate = threadDate
					secondDate = True

		print "Thread:", thread.id, len(thread), thread.subject, thread.getLabels(), thread.__getattr__('date'), thread._authors, thread.__getattr__('unread')
		try:
			# Current Labels
			threadCurrentLabels = thread.getLabels();
			# We will add labels here first to prevent dupes
			threadLabels = set([])
			# Check each message in the thread.
			for msg in thread:
				print "  Message:", msg.id, msg.sender
				# Check if sender is a known  contact
				contact = contacts.getContactByEmail(msg.sender)
				if contact != False:
					# Check each group for this contact
					for group in groups:
						# If we have a label with this group name
						labelName = labelPrefix+group.name
						if (labelName in labels) and (labelName not in threadCurrentLabels):
							# And the group contains the contact we want
							if group.containsContact(contact):
								# Add it to the list
								print "    Sender Label:", labelName
								threadLabels.add(labelName)
		except Exception, detail:
			print "  Error parsing messages:", type(detail), detail

		# Now add the labels
		for label in threadLabels:
			print "  Adding Label:", label
			thread.addLabel(label)
		# If thread was unread, make it unread again.
		if thread.__getattr__('unread'):
			print "  Remarking as unread"
			ga._doThreadAction("ur", thread)

	if loop:
		print ""
		print "Sleeping"
		time.sleep(loopTime)
	else:
		print "Done"

On a related note, I've also recently started to use the "Better Gmail 2" addon for firefox (Official page seems down atm, but more info here) mostly for the grouping of labels feature.

Edit: Script will now preserve unread status of threads.

Email Woes

Posted by Dataforce

On a daily basis, I get around 800 emails to my email accounts, of which most of it is spam.

Now as good as thunderbird is at detecting spam, even it fails at a lot of the spam I recieve, leaving me with arround 100-200 spam per day that gets into my inbox.

I've dealt with and accepted this for over a year now, before a discussion on IRC made me decide to do something about it. (When I say discussion, I mean Chris pasted one line showing how good the UTD-Hosting mail server was at preventing junk getting to him)

So, I recently (Today and yesterday) started prodding my postfix config to help with the problem.

Firstly I added some RBL checking, this was easy enough, 3 lines to my config in the smtpd_recipient_restrictions bit:

    reject_rbl_client list.dsbl.org
    reject_rbl_client zen.spamhaus.org
    reject_rbl_client dnsbl.sorbs.net

I also added:

    reject_non_fqdn_recipient
    reject_unknown_recipient_domain

I also added the following lines:

smtpd_helo_required = yes

smtpd_delay_reject = yes

smtpd_helo_restrictions =
    permit_mynetworks
    check_helo_access hash:/etc/postfix/helo_access
    reject_non_fqdn_hostname
    reject_invalid_hostname
    permit

smtpd_sender_restrictions =
    permit_mynetworks
    reject_non_fqdn_sender
    reject_unknown_sender_domain
    permit

/etc/postfix/helo_access looks like this (Its surprising how many mails this catches, 114/7500 - altho they would probably be caught later on):

soren.co.uk            REJECT You are not me.
207.150.170.50         REJECT You are not me.

Next step was SPF checking, this involved adding to smtpd_recipient_restrictions:

    check_policy_service unix:private/policy

and to master.cf

policy  unix  -       n       n       -       -       spawn
        user=nobody argv=/usr/bin/perl /usr/lib/postfix/policyd-spf-perl

(One can apt-get install postfix-policyd-spf-perl or download it from http://www.openspf.org/Software)

Currently I use catch-all on all my domains (yes this is stupid I know) and as a result, I get alot of spam to 1) Addresses that don't exist and never have 2) Addresses that used to exist for others but now don't.
To combat this, I added this line to smtpd_recipient_restrictions:

    check_recipient_access hash:/etc/postfix/recipient_access

/etc/postfix/recipient_access looks something like this:

foo@example.com REJECT This account is no longer valid.
bar@example.com REJECT This account is no longer valid.
baz@example.net REJECT This account is no longer valid.

The result of all this can be seen by running the mailstats script Chris was kind enough to share with me:

root@soren:/etc/postfix# ./mailstats.php

Incoming --(7500)--> Valid HELO --(6707)--> Valid Sender --(6705)--> Passed by dsbl --(6136)--> Passed by spamhaus --(811)--> Passed by sorbs --(568)--> Passed by relay check --(565)--> Passed by SPF --(542)--> Forwarded to shinobu --(390)--> To a valid domain --(339)--> To a valid user --(306)--> Dropped Spam --(306)--> Delivered.
Total Rejections: 7194 (Unknown Reason: 0 | Pretended to be me: 114)

The "Forwarded to shinobu" entry is a server for which I am the backup MX for, this accounts for 152 mails (about 2%)
The delivered count of 306 mails is about 4%, meaning that 94% of all the junk mail I recieve is now dropped by postfix and not delivered to my mailbox!

These simple additions have made a huge difference! I have a 10day holiday coming up, and now rather than coming home to 8000 mails, I'll only come home to 320!

As a further line of defense, prior to being sent to my mailbox, those 4% of mails get filtered thorugh spamassassin (which I have configured to only run for certain domains, with different scores for different domains/users as needed) which does a good job of catching the spam that thunderbird misses, configuring a mail filter on thunderbird to filter these mails (Which get subject tagged with {Spam?}) into my junk folder (aswell as configuring thunderbird to trust what the spamassassin headers say) means very little, if any, spam now reaches my inbox!

Brilliant!