FreeBSD Quarterly Status Report
2004 continues on with wonderful progress. Work continues on locking
down the network stack, ACPI made more great strides, an ARM port
appeared in the tree, and the FreeBSD 4.10 release cycle wrapped up.
Once 4.10 is released, the next big focus will be FreeBSD 5.3. We
expect this is be the start of the 5-STABLE branch, meaning that not
only will it be stable for production use, it will also be largely
feature complete and stable from an internal API standpoint. We expect
to release 5.3 in mid-summer, and we encourage everyone to download the
latest snapshots from
- ATA project Status Report
- Automatic sizing of TCP send buffers
- Binary security updates for FreeBSD
- Book: The Design and Implementation of the FreeBSD Operating System
- CAM lockdown and threading
- Convert ipfw2 to use PFIL_HOOKS mechanism
- Cronyx Tau-ISA driver
- FreeBSD Dutch Documentation Project
- FreeBSD threading support
- GEOM Gate
- Improved Multibyte/Wide Character Support
- Move ARP out of routing table
- Network interface naming changes
- Network Stack Locking
- OpenOffice.org porting status
- PCI Powerstates and Resource
- Porting OpenBSD's packet filter
- SMPng Status Report
- Status Report
- Sync protocols (Netgraph and SPPP)
- The FreeBSD Simplified Chinese Project
- TrustedBSD Audit
- TrustedBSD Mandatory Access Control (MAC)
- TrustedBSD Security-Enhanced BSD (SEBSD) port
- Verify source reachability option for ipfw2
Contact: Nate Lawson <njl@FreeBSD.org>
Much of the ACPI project is waiting for architectural changes to be completed. For instance, the cpufreq driver requires newbus attachments for CPUs. Support code for this should be committed at the time of publication. Other architectural changes needed include rman support for memory/port resources and a generic hotkey and extras driver. Important work in other areas of the kernel including PCI powerstate support and APIC support have been invaluable in improving ACPI on modern platforms. Thanks go to Warner Losh and John Baldwin for this work.
Code which is mostly completed and will go in once the groundwork is finished includes the cpufreq framework, an ACPI floppy controller driver, and full support for dynamic Cx states.
ACPI-CA was updated to 20040402 in early April. This has some GPE issues that persist in 20040427 that will hopefully be resolved by the date of publication.
I'd like to welcome Mark Santcroos (marks@) to the FreeBSD team. He has helped in the past with debugging ACPI issues. If any developers are interested in assisting with ACPI, please see the ACPI TODO and send us an email.
Contact: Søren Schmidt <sos@FreeBSD.org>There is finally support (except for RAID5) for the Promise SX4/SX4000 line of controllers. The support is rudimentary still, and doesn't really make any good use of the cache/sequencer HW yet. The Silicon Image 3114 support has been completed. Lots of bug fixes and cleanups. Future work now concentrates on new controller chips (Marvell SATA chips probably the most prominent) and getting the SATA support finished so that hotswap etc works with SATA HW as well. Also ATA RAID is about to get rewritten to take advantage of the features that the ATA subsystem now offers, including support for the HW on Promise/Marvell and the like controllers. A number of new RAID metadata
formats (Intel, AMI) is also in the works.
Contact: Andre Oppermann <andre@FreeBSD.org>
The current TCP send and receive buffers are static and set to a conservative value to preserve kernel memory. This is sub-optimal for connections with a high bandwidth*delay product because the size of the TCP send buffer determines how big the send window can get. For high bandwidth trans-continental links this seriously limits the maximum transfer speed per TCP connection. A moredetailed description from the last status report can be found with the link above.
Work on this project has been stalled due to some other network stack projects with higher precedence (ipfw2 to pfil_hooks and ip_input/ip_output cleanups).
Contact: Colin Percival <firstname.lastname@example.org>
Having recently passed its first birthday, FreeBSD Update is now being used on about 170 machines every day; on a typical day, around 60 machines will download updates (the others being already up to date). To date, over 157000 files have been updated on over 4200 machines.
The new Book "The Design and Implementation of the FreeBSD Operating System" is the successor of the legendary "The Design and Implementation of 4.4BSD" book which has become the de-facto standard for teaching of Operating System internals in universities world-wide.
This new and completely reworked edition is based on FreeBSD 5.2 and the upcoming FreeBSD 5.3 releases and contains in-details looks into all areas (from virtual memory management to interprocess communication and network stack) of the operating system on 700 pages.
It is now in final production by Addison-Wesley and will be available in early August 2004. The ISBN is 0-201-70245-2.
Contact: Scott Long <scottl@FreeBSD.org>
Work has begun on locking down the CAM subsystem. The project is divided into several steps:
- Separation of the SCSI probe peripheral from cam_xpt.c to scsi_probe.c
- Threading of the device probe sequence.
- Locking and reference counting the peripheral drivers.
- Locking the XPT and device queues.
- Locking one or more SIMs and devising a way for non-locked drivers to function.
While the immediate goal of this work is to lock CAM, it also points us in the direction of separating out the SCSI-specific knowledgefrom the core. This will allow other transports to be written, such as SAS, iSCSI, and ATA.
Progress is being tracked in the FreeBSD Perforce server in the camlock branch. I will make public patches available once it has progressed far enough for reasonable testing. So far, the first two items are being worked on.
Contact: Andre Oppermann <andre@FreeBSD.org>
ipfw2 is built directly into ip_input() and ip_output() and it makes these functions more complicated. For some time now we have the generic packet filter mechanism PFIL_HOOKS which are used by IPFILTER and the new OpenBSD PF firewall packages to hook themselves into the IP input and output path.
This patch makes ipfw2 fully self contained and callable through the PFIL_HOOKS. This is still work in progress and DUMMYNET and IPDIVERT plus Layer2 firewall are not yet fully functional again but normal firewalling with it works just fine.
The patch contains some more cleanups of ip_input() and ip_output() that is work in progress too.
Contact: Roman Kurakin <rik@FreeBSD.org>
ctau(4) driver for Cronyx Tau-ISA was added. Cronyx Tau-ISA is family of synchronous WAN adapters with various set of interfaces such as V.35, RS-232, RS-530(449), E1 (both framed and unframed). This is a second family of Cronyx adapters that is supported by FreeBSD now. The first one was Cronyx Sigma-ISA, cx(4).
Cronyx Tau-PCI family will become a third one. The peculiarity of this driver that it contains private code. This code is distributed as obfuscated source code with usual open source license agreement.Since code is protected by obfuscation it is satisfy needs of commerce. On the other hand it still stays a source code and thus it becomes closer to open source projects. I hope this form of private code distribution will become a real alternative to object form.
Contact: Remko Lodder <email@example.com>
The FreeBSD Dutch Documentation Project is a ongoing project in translating the handbook and other documentation to the Dutch language. Currently we have a small team of individuals who translate, check other's work, and publish them on the internet. You can view the current status on the webpage (listed above). Still we can use more people helping out, since we have a long way to go. Every hand that wants to help, contact me, and i will provide you details on how we work etc. Currently the project has translated the handbook pages of: The X Windows System, and Configuration and Tuning, they only need to be checked before publishing.
Contact: Doug Rabson <dfr@FreeBSD.org>
Contact: Julian Elischer <julian@FreeBSD.org>
Contact: Marcel Moolinar <marcel@FreeBSD.org>
Contact: Dan Eischen <deischen@FreeBSD.org>
Threading developers have been active behind the scenes though not much has been visible. Real Life(TM) has been hard on us as a group however.
Marcel and Davidxu have both (individually) been looking at the support for debugging threaded programs. David has a set of patches that allow gdb to correctly handle KSE programs and patches are being considered for libthr based processes. Marcel added a Thread ID to allow debugging code to unambiguously specify a thread to debug. He has also been looking at corefile support. Both sets of patches are preliminary.
Dan Eischen continues to support people migrating to libpthreads and it seems to be going well.
Doug Rabson has done his usual miracle work and produced a set of preliminary patches to implement TLS (Thread Local Storage) for the i386 platform.
Julian Elischer is investigating some refactoring of the kernel support code.
i386, amd64, ia64 libpthread works.
alpha, sparc64 not implemented.
Contact: Olivier Houchard <cognet@FreeBSD.org>
FreeBSD/arm is now in the FreeBSD CVS tree. Dynamic libraries now work, and NO_CXX=true NO_RESCUE=true buildworld works too (with patches for toolchain that will live outside the tree for now). Now the focus should be on xscale support.
Contact: Pawel Jakub Dawidek <pjd@FreeBSD.org>
GEOM Gate class is now committed as well as ggatec(8), ggated(8) and ggatel(8) utilities. It makes distribution of disk devices through the network possible, but on the disk level (don't confuse it with NFS, which provides exporting data on the file system level).
Contact: Tim Robbins <tjr@FreeBSD.org>
New locales: Unicode UTF-8 locales have been added to the base system. All of the locales previously supported by FreeBSD now have a corresponding UTF-8 version, along with one or two new ones -- 53 in all.
Library changes: The restartable conversion functions (mbrtowc(), wcrtomb(), etc.) in the C library have been updated to handle partial characters in the way prescribed by the C99 standard. The <wctype.h> functions have been optimized for handling large, fragmented character sets like Unicode and GB18030. Documentation has been improved.
Utilities: The ls utility has been modified to work with wide characters internally when determining whether a character in a filename is printable, and how many column positions it takes on the screen. Character handling in the wc utility has been made more robust. Other text-processing utilities (expand, fold, unexpand, uniq) have been modified, but these changes have not been committed until the performance impact can be evaluated. Work on a POSIX-style localedef utility has started, with the aim to have it replace the current mklocale and colldef utilities in FreeBSD 6. (It is currently on the back-burner awaiting a response to a POSIX defect report.)
Future directions: wide character handling functions need to be optimized so that they are more competitive with the single-byte functions when dealing with 8-bit character sets. Utilities need to be modified to handle multibyte characters, but with a careful eye on performance. Localedef needs to be finished.
Contact: Tim Kientzle <kientzle@FreeBSD.org>
Both bsdtar and libarchive are now part of -CURRENT. A few minor problems have been reported and addressed, including performance issues with many hard-links, and options required by certain packages. For now, the "tar" command is still an alias for "gtar." Those who would like to use bsdtar as the default system tar can define WITH_BSDTAR to make "tar" be an alias for "bsdtar."
My current plan is to make bsdtar be the default in -CURRENT in about another month, probably after the 5-STABLE split, and remove gtar from -CURRENT sometime later. It's still open if and when this switch will occur in 5-STABLE. On the one hand, I see potential problems if 5-STABLE and 6-CURRENT have different tar commands; on the other hand, switching could be disruptive for some users.
The ARP IP address to MAC address mapping does not belong into the routing table (FIB) as it is currently done. This will move it to its own hash based structure which will be instantiated per each 802.1 broadcast domain. With this change it is possible to have more than one interface in the same IP subnet and layer 2 broadcast domain. The ARP handling and the routing table will be quite a bit simplified afterwards. As an additional benefit full MAC address based accounting will be provided.
Luigi has become the driver of this project and posted a first implementation for comments on 25. April 2004 (see link).
Contact: Brooks Davis <brooks@FreeBSD.org>
An enhanced network interface cloning API has been created. It allows interfaces to support more complex names than the current name# style. This functionality has been used to enable interesting cloners like auto-configuring vlan interfaces. Other features include locking of cloner structures and the ability of drivers to reject destroy requests. A patch has been posted to the freebsd-net mailing list for review and will be committed in early May. This work is taking place in the perforce repository under: //depot/user/brooks/xname/...
Contact: Robert Watson <rwatson@FreeBSD.org>
This project is aimed at converting the FreeBSD network stack from running under the single Giant kernel lock to permitting it to run in a fully parallel manner on multiple CPUs (i.e., a fully threaded network stack). This will improve performance/latency through reentrancy and preemption on single-processor machines, and also on multi-processor machines by permitting real parallelism in the processing of network traffic. As of FreeBSD 5.2, it was possible to run low level network functions, as well as the IP filtering and forwarding plane, without the Giant lock, as well as "process to completion" in the interrupt handler.
Work continues to improve the maturity and completeness of the locking (and performance) of the network stack for 5.3. The network stack development branch has been updated to the latest CVS HEAD, as well as the following and more:
- Review of socket flag and socket buffer flag locking; so_state broken out into multiple fields covered by different locks to avoid lock orders in frobbing the so_state field. Work in progress.
- WITNESS now includes hard ordering for many network locks to improve lock order debugging process.
- MAC Framework modified to use pcbs instead of sockets in a great many situations to avoid socket locking in network layer, especially when generating new mbufs.
- New annotations relating to socket and interface locking.
- Began NetGraph review and corrected NetGraph socket locking problems.
- sendfile() locking appears now to be fixed, albeit holding Giant more than strictly necessary.
- if_ppp global variable locking performed and merged.
- A variety of race conditions and bugs in soreceive() locking fixed, including existing race conditions triggered only rarely in -HEAD and -STABLE that triggered easily with SMP and Giant-free operation.
- Locking of socket buffer and socket fields from fifofs. Proposed patch to correct lock order problem between vnode interlock and socket buffer lock order problems. fifofs interactions with UNIX domain sockets cleaned up.
- Research into KQueue issues. Feedback to KQueue locking patch authors.
- netatalk AARP locked down, MPSAFE, and merged to CVS.
- Lock order issues between socket, socket buffer, and UNIX domain socket locks corrected. Race conditions and potential deadlocks removed.
- if_gif recursion cleanups, if_gif is much more MPSAFE.
- First pass MPSAFE locking of NFS server uses an NFS server subsystem lock to allow so_upcall() from socket layer without Giant. This closes race conditions in the NFS server when operating Giant free. Second pass for data based locking is also in testing.
- if_sl.c (SLIP) fine-grained locking completed and merged to CVS.
- if_tun.c (tunnel) fine-grained locking completed and merged to CVS.
- Merge of conditional Giant locking on debug.mpsafenet to CVS; semantics now changed so that Giant isn't just twiddled over the forwarding path, but the entire stack. Must be used with caution unless running with our patches. Callouts also convered to conditional safety.
- if_gif, if_gre global variables locked and merged to CVS.
- netatalk DDP cleanup (break out PCB from protocol code), largely locked down at the PCB level. Some work remains to be done before patches can be distributed for testing, but close to MPSAFE.
- Began review of netipx, netinet6 code for locking requirements, some bugs corrected.
- Race conditions in handling of socket so_comp, so_incomp debugged and hopefully closed through new locking of these fields.
- Many new locking annotations, field documentation, lock order documentation.
Netperf patches are proving to be quite stable in a broad variety of environment, as long as non-MPSAFE chunks are avoided. Kqueue, IPv6, and ifnet locking remain the most critical areas where additional functionality is required. Focus is shifting from new development to in depth testing, performance measurement, and interactions with other subsystems.
This work would not be possible without contributions from the following people (and no doubt many others): John Baldwin, Bob Bishop, Brooks Davis, Pawel Jakub Dawidek, Matthew Dodd, Julian Elischer, Ruslan Ermilov, John-Mark Gurney, Jeffrey Hsu, Kris Kennaway, Roman Kurakin, Max Laier, Sam Leffler, Scott Long, Rick Maklem, Bosko Milekic, George Neville-Neil, Andre Oppermann, Luigi Rizzo, Jeff Roberson, Tim Robbins, Mike Silberback, Bruce Simpson, Seigo Tanimura, Hajimu UMEMOTO, Jennifer Yang, Peter Wemm. We hope to present these patches on arch@ within a few days, although some elements required continued refinement (especially socket locking).
Contact: NAKATA Maho <maho@FreeBSD.org>
After almost three years efforts for porting OpenOffice.org 1.0.x and 1.1.0 for FreeBSD by Martin Blapp (mbr@FreeBSD.org) and other contributors, There are four version of OpenOffice.org (OOo) in ports tree. 1.1.1: stable version, 1.1.2: next stable, 2.0: developer and 1.0.3: legacy.
Stable version 1.1.1 in /usr/ports/editors/openoffice-1.1/ builds/installs/works fine for 5.2.1-RELEASE. Packages for 5.2.1-RELEASE, 26 localized versions and 4.10-PRELEASE only English version, are available at http://oootranslation.services.openoffice.org/pub/OpenOffice.org/ooomisc/ (note: source of OOo 1.1.1.RC3 is identical OOo 1.1.1)
Patches needed to build are currently 18 for 1.1.1, and 161 for 1.0.3 the number of patches are greatly reduced.
OOo 1.1.2, the next stable version in /usr/ports/editors/openoffice-1.1-devel is also builds/installs/works fine for 5.2.1-RELEASE. We are planning to upgrade this port as soon as 1.1.2 will be released.
Next major release, 2.0 (planned to be released at January 2005 according to http://development.openoffice.org/releases/OpenOffice_org_trunk.html), /usr/ports/editors/openoffice-2.0-devel, now compiles for 5.2.1-RELEASE but have big problem that prohibits to remove BROKEN.
Legacy version, OOo 1.0.3: /usr/ports/editors/openoffice-1.0/ I'm not interested in this port. We hope someone else will maintain this.
For builds, my main environment is 5.2.1-RELEASE, and I have no access to 4-series, so several build problems had been reported for 5-current and 4-stable, however, they now seems to be fixed. Please make sure your Java and/or kernel are up-to-date.
For version 1.1.1, yet we have serious reproducible core dumps, this means OOo cannot pass the Quality Assurance protocol of OpenOffice.org (http://qa.openoffice.org), so we cannot release OOo as quality assured package. It seems to be FreeBSD's userland bug, since some reports show that there are no problem for 4-stable but we still searchingthe reason.
Note that developers should sign JCA (Joint Copyright Assignment) before submitting patches via PR or e-mail, otherwise patches won't be integrated to OOo's source tree. We seriously need more developers, testers and builders.
Contact: Warner Losh <imp@FreeBSD.org>
Lazy allocation of pci resources has been merged into the main tree. These changes allow FreeBSD to run on computers where PnP OS is set to true. In addition, the saving and restoring of the resources across suspend/resume has helped some devices come back from suspend.
Future work will focus on bus numbering.
The two months after the import was done were actually rather quiet. We imported a couple of minor fixes from the OpenBSD stable branch. The import of tcpdump 3.8.3 and libpcap 0.8.3 done by Bruce M.Simpson in late March finally put us into the position to build a working pflogd(8) and provide rc.d linkage for it. Tcpdump now understandsthe pflog(4) pseudo-NIC packet format and can be used to read the log-files.
There has also been work behind the scenes to prepare an import of the OpenBSD 3.5 sources. The patches are quite stable already andwill be posted shortly. Altq is in the making as well and going alongquite well based on the great work from rofug.ro, but as it needs modifications to every network driver which have to be tested thoroughly it needs more time.
Several folks continue to work on the locking the network stack as noted elsewhere in this report. Outside of the network stack, the following items were worked on during the March and April time frame. Giant was pushed down in the fork, exit, and wait system calls as far as possible. Alan Cox (alc@) continues to lock the VM subsystem and push down Giant where appropriate. A few system calls and callouts were marked MP safe as well.
A few changes were made to the interrupt thread infrastructure. Interrupt thread preemption was finally enabled on the Alpha architecture with the help of the recently added support to the scheduler for pinning threads to a specific CPU. An optimization to reduce context switches during heavy interrupt load was added as well as rudimentary interrupt storm protection.
Contact: Roland van Laar <firstname.lastname@example.org>
This patch if for if_wi current. It enables you to disable the ssid broadcasting and it also allows you to disable clients connecting with a blank ssid.
Contact: Roman Kurakin <rik@FreeBSD.org>
As part of my work on synchronous protocol stack a ng_sppp driver was added to the system. This driver allows to use sppp as a Netgraph node. Now I plan to update sppp driver as much as possible to make it in sync with Cronyxs one (PPP part). Also I work on FRF.12 support in FreeBSD (now I have FRF.12 support for Netgraph and SPPP (and for Cronyx linux fr driver) but only End-to-End). I plan to test it by my self within a week and after that I plan to make full support of FRF.12.
If you want to get current version and test it, please feel free to contact me.
Contact: Xin LI <email@example.com>
We have finished about 75% of the Handbook translation work. In the last two months we primarily worked on bringing the handbook chapters more up to date. To make the translation more high quality we are also doing some revision on it.
We are still looking for manpower on SGML'ifying the FAQ translation which has been done last year by several volunteers.
The TrustedBSD Project is producing an implementation of CAPP compliant Audit support for use with FreeBSD based on the Apple Darwin implementation.
Experimentally integrated the XNU audit implementation from Apple's Darwin 7.2 into Perforce.
Adapted audit framework to compile into FreeBSD -- required modifying memory allocation and synchronization to use FreeBSD SMPng primitives instead of Mach primitives. Pushed down the Giant lock out of most of the audit code, various other FreeBSD adaptations such as suser() API changes, using BSD threads, td->td_ucred, etc.
Adapted per-thread audit data to map to FreeBSD threads
Cleaned up userspace/kernel API interactions, including udev_t/ dev_t inconsistencies between Darwin and FreeBSD.
Use vn_fullpath() instead of vn_getpath(), which is a less complete solution we'll need to address in the future.
Basic kernel framework now operates on FreeBSD; praudit tool written that can parse FreeBSD BSM and Solaris BSM.
The TrustedBSD Mandatory Access Control (MAC) Framework permits the FreeBSD kernel and userspace access control policies to be adapted at compile-time, boot-time, or run-time. The MAC Framework provides common infrastructure components, such as policy-agnostic labeling, making it possible to easily development and distribute new access control policy modules. Sample modules include Biba, MLS, and Type Enforcement, as well as a variety of system hardening policies.
The TrustedBSD MAC development branch in Perforce was integrated to the most recent 5-CURRENT.
mdmfs(8) -l to create multi-label mdmfs file systems (merged).
Diskless boot updated to support MAC.
Re-arrangement of MAC Framework code to break out mac_net.c into mac_net.c, mac_inet.c, mac_socket.c (merged).
libugidfw(3) grows bsde_add_rule(3) to automatically allocate rule numbers (merged). ugidfw(8) grows 'add' to use this (merged).
pseudofs(4) no longer requires MAC localizations.
BPF fine-grained locking now used to protect BPD descriptor labels instead of Giant (merged).
Prefer inpcb's as the source of labels over sockets when creating new mbufs throughout the network stack, reducing socket locking issues for labels.
TrustedBSD "Security-Enhanced BSD" (SEBSD) is a port of NSA's SELinux FLASK security architecture, Type Enforcement (TE) policy engine and language, and sample policy to FreeBSD using the TrustedBSD MAC Framework. SEBSD is available as a loadable policy module for the MAC Framework, along with a set of userspace extensions support security-extended labeling calls. In most cases, existing MAC Framework functions provide the necessary abstractions for SEBSD to plug in without SEBSD-specific changes, but some extensions to the MAC Framework have been required; these changes are developed in the SEBSD development branch, then merged to the MAC branch as they mature, and then to the FreeBSD development tree.
Unlike other MAC Framework policy modules, the SEBSD module falls under the GPL, as it is derived from NSA's implementation. However, the eventual goal is to support plugging SEBSD into a base FreeBSD install without any modifications to FreeBSD itself.
Integrated to latest FreeBSD CVS and MAC branch.
New FreeBSD code drop updated for capabilities in preference to superuser checks.
Installation instructions now available!
Contact: Andre Oppermann <andre@FreeBSD.org>
The verify source reachability option for ipfw2 has been committed on 23. April 2004 to FreeBSD-CURRENT. For more information see the links above.
News Home | Status Home