Planet FreeBSD Summer of Code 2008

Today I’m going on a vacation to Greece for one week. I’ll be back next Thursday (10th). I’ve polished my project pkg_improved so I think that people could test it out to see if it works. There’s not a whole lot of change, just some speed gains here and there, you can check the CHANGES file posted for more details. You could also make a diff with /usr/src/usr.sbin/pkg_install/ but you will get a whole lot of syntax changes =(

If you want to test it, then just compile and install (NB! this will replace your normal pkg_* tools). You could always get back your original pkg_* tools by compiling and installing from /usr/src/usr.sbin/pkg_install . If you don’t want to replace your pkg_* tools you should just compile it and rename the executables (NB! You will not get proper behaviour from ports unless you change some PKG_* variables in /usr/ports/Mk/bsd.port.mk (see PKG_CMD). The first thing you should do after install is to run pkg_convert to cache the existing flat database, after that you don’t have to run it again unless you use the original pkg_* programs.

There’s some ideas that I’ve been looking into like adding date’s to installed packages (perhaps a @comment in +CONTENTS), recovering +REQUIRED_BY in pkg_add (after e.g. pkg_delete -f zip; pkg_add zip). These should probably be posted to a mailing list, but I’ll wait until I get home from Greece to post them.

pkg_improved: http://home.no.net/andenore/pkg_improved.tar.gz

(please e-mail me if you find bug’s and the like)

Real Life really doesnt like me working on this project. Thanks to exam time at university I was have to cover a few shifts for other students this past week, and ended up doing 6 days in a row. Thankfully I have some holiday time coming up starting this Thursday, and while I’ll be away for both weekends I’m on holiday I’ll have a full week to work on MPLS from the moment I get up to the moment I go to sleep, without any distractions.

Accomplished Last Week:

- Worked on my FreeBSD wiki pages.

- Went over OpenBSD/Ayame code again to try and get a bit more understanding on how it works.

Plans for Next Week:

- Port over the protocol description file mpls_proto.c, then begin working on porting over code for simple sending and receiving of MPLS packets.

- Once sending and receiving is working, look at implementing static label switching, then working towards dynamic label switching off a routing table.

There used to be a flaw in using ipfw on bridge interface. It’s impossible to distinguish incoming packets on member interface from incoming packets on bridge itself. For example consider two rules:
add 1 allow ip from any to any in recv bridge
add 2 allow ip from any to any in recv member

First rule will never match. The logic is ok here (if you are aware of ipfw’s handling of interface options). But what do you expect if you disable filtering on member interfaces and perform filtering on bridge only. You expect rule 1 to match all incoming packets on bridge. It gets extremely annoying when using stateful filtering.

First time I came across this issue several years ago. But didn’t figure out how to fix it. At that time I’ve decided to switch to pf.

Actually ipfw is the only firewall that allow rules like
allow ip from any to any out recv if1 xmit if2
Such tricks are possible because ipfw gets input interface from mbuf of a packet. pf for example relies on pfil to provide interface.

I’ve added a hack into if_bridge to work around it. It contradicts traditional ipfw behaviour a little but seems to be much more useful. I think patches are useful enough and can be commited into FreeBSD:

perforce.freebsd.org/changeView.cgi?CH=143921
perforce.freebsd.org/changeView.cgi?CH=144238

This has been somewhat of a amputated week, last weekend I was away for a bicycle race and this weekend I’m going to a wedding, so it hasn’t been a lot of time for programming. Anyways I’ve added human-readable output for pkg_info’s -s option (the old is also available with the -b option). The environment variable PKG_DBCACHE_FILE is used to specify where the cache-file is located. And I’ve added features for the pkg_convert program that checks if key’s exists within the database and the printing of the data indexed in the database. The pkg_create -O option has been modified to cache information as well and installing ports from /usr/ports/ will then work if you specify the PKG_CMD in /usr/ports/Mk/bsd.port.mk to point to this pkg_create.

I’m hoping to release a test-version next week so that I can receive comments and bugs which I’m sure there is. On Thursday next week I’m going on a one week vacation to Greece so that you will have a good time to test the tools and fill my mailbox

Man these weeks are flying by. I managed to get a bit of work done on MPLS, but not much, before a suprise trip to visit friends up in Auckland dragged me away for the weekend.

Accomplished Last Week:

- Began looking over the OpenBSD implementation, looking through what has been submitted on their CVS, and just generally trying to get how the code does what it needs to do, and where it does it.

- Finally got a call back with an answer from IBM about my application. They have decided not to invite me to their interview day this time, but apparently if they hold another one then they will invite me to that. I’m not holding my breath. It appears as though it is very hard to get a job in New Zealand working with operating systems or networking stuff, unless you have experience. Unfortunately for me, my grades in my first 2 years of uni where anything but flash, and I seem to be suffering for it now. It’s almost making me wish I had some PHP or web programming under my belt, which is what all the NZ jobs seem to be looking for. Oh well, onwards and upwards with the job hunt hopefully.

Plans for Next Week:

- Once I have sorted out the licencing stuff with using OpenBSDs code (I havent had anything to do with Open Source licences befire, and I dont want to step on anyones toes by doing the wrong thing) then I will be porting over the code that I need to start working on.

- Figure out where exactly the code from OpenBSD has gotten to in terms of my plans, and then start working on improving it or moving on to new features I need.

Thats all for now. If anyone knows of any jobs in NZ for someone studing networking and operating systems, let me know

Yep. I’ve made some changes that break backward compatibility
But I’ve tried not to break anything intentionally but to do a cleanup work.

First of all most of sysctl’s responsible for layer2 filtering were replaced by per interface flags.

net.link.ether.ipfw and net.link.bridge.ipfw are replaced by l2filter interface flag.
So sysctl net.link.ether.ipfw=1 became ifconfig if1 l2tag.

net.link.bridge.ipfw_arp was renamed to net.link.bridge.pfil_layer2_arp

Introduced l2tag interface flag. It’s purpose is to add mbuf tag containing source and destination layer2 addresses to every packet passing through interface. Note that l2tag filtering against layer2 addresses is performed in layer3.

When invoked from layer2 ipfw no longer touches layer2 headers. So they following rule won’t work anymore:
ifpw allow ip from 10.1.1.1 to any src-ether 00:11:11:11:11:11 layer2

ipfw mac option was replaced by to two options: src-ether and dst-ether. ipfw still accepts mac option but translates it into src-ether and dst-ether.

Lookup tables support layer2 addresses now:
ipfw table 1 add 10.1.1.1 ether 00:11:11:11:11:11
ipfw allow ip from table(1) to any

ipfw mac-type was renamed to ether-type. Support for mac-type preserved.

Stateful filtering remains somewhat special. The problem here is that l2tag is added to a packet only in input path (when invoked from ether_demux). Such decision was intentional, mainly because it’s impossible to get tag added in output path without serious layer violations or entire pfil framework and packet handling redesign. That’s why a packet that has no l2tag attached, will pass against layer2 dynamic rule.

Dynamic rules (state created by the rule) do not check both source and destination layer2 address, but just the addresses specified by the rule created it. For example
ifpw allow ip from 10.1.1.1 to any src-ether 00:11:11:11:11:11 keep-state
will create dynamic rule that checks only source ethernet address of a packet, but not destination.

I’m working on improving layer2 packet filtering in FreeBSD as a part of Google Summer of Code 2008.
See wiki for details.

Progress reports are to follow.

I’ve been hacking the NetBSD lii(4) driver so that it works under FreeBSD. This driver is most notably found on the Asus line of sub-notebooks, Eee PC. So far, so good. I did not finish the porting yet, but the mechanical changes are mostly done.
The reason for this is that I bought an Eee PC 701, hence I need this driver, :-), although I haven’t touched my Eee PC yet (it’s at my parents house). But I will do the first testing this weekend.

If you have this hardware and would like to help with the effort, please email me.

The effort is being revision controlled at //depot/user/rpaulo/lii/.

So, I found some time to continue my SoC work. tcpad is now capable of handling the most important TCP FSM transitions, like CLOSE_WAIT, FIN_WAIT_1, SYN_SENT, etc. I also implemented a basic timer facility that cleans up old connections in TIME_WAIT state. This still doesn’t honor the 2MSL required by the RFC, but it’s a start.
I also cleaned the code a little and improved the debugging macro.

Next is SEQ/ACK analysis.

Changes 

These are the main things I’ve done so far some abstract and some more specific, if you want to check out the code you can get it from the perforce depot: //depot/projects/soc2008/andenore_pkginstall/

pkg_convert:

•  Added a convert program (pkg_convert) that indexes information to PKG_DBCACHE_FILE (defaults to $PKG_DBDIR/pkgcache.db)

pkg_info:

• Improved performance for -W option using the dbcache
• A quick profiling of pkg_info revealed that the default behavior for pkg_info was to call read_plist for each package even though not needed so I’ve added a check for this and it runs significantly faster.

pkg_add:

• Indexes information to dbcache according to the add

pkg_delete:

• “Deindexes” information according to the delete

lib:

• Changed methods in match.c to use cache if available
• Added global variable ‘database’ of type DB* (the database object)
• Added database helper functions

I’ve done some quick profiling of pkg_version and it spends much of it’s time in read_plist too (just to fetch the origin). So I plan to index the origin of each package and speed this up in pkg_version.