CL-HTTPCL-HTTP Development Projects

Overview
Project Ideas
  1. Server Enhancements
  2. Server Documentation
  3. Client Enhancements
  4. Web Walker Enhancements
  5. Mail HyperArchive
  6. Database Projects
  7. LambdaVista
  8. Log Analysis
  9. Platform-Specific Projects
    1. Lisp Machine
    2. Macintosh Platform
    3. UNIX Platform
    4. PC Platform
  10. Extensions, Add-ons & Applications

Overview

This page describes a set of development projects that you could contribute to the CL-HTTP community. You will work with an international team of CL-HTTP developers, and perhaps learn some interesting things about advanced Common Lisp programming as you win recognition and build your reputation in the Lisp community.

Please be a good citizen and contribute something, no matter how small, to improve these tools for yourself and other users. Not only did somebody else write this server, client, web walker, proxy, full-text search engine,
mail hyperarchive, and numerous extensions, they also empowered you with the source code. Why not invest a little effort in return? In the end, you will help yourself by showing that the Lisp culture can produce shared resources and motivating the major developer to continue providing high-quality advanced software to you without charge.

Before contributing code, you may wish to consult the guidelines and efficiency considerations.

If you are working on some code for CL-HTTP, please send a paragraph (or a URL) to www-cl@ai.mit.edu (or bug-cl-http@ai.mit.edu if you would prefer less notice) explaining what you are doing, your time frame, and how to contact you. That way, people will know what you are doing and can contact you if they have ideas, suggestions, or want to help out. It can also help avoid duplicated effort. Have a look at the WWW-CL archives to see what other people are doing.
 

Project Ideas

  1. Server Enhancements

    1. Protocol Conformance
      • HTTP 1.1 Proxying:  Complete remaining details of HTTP proxying. Maximize efficiency and flexibility. Support on all platforms.
      • HTTP 1.1 Digest Authentication: Update the current digest implementation to conform with the latest draft of the digest specification (proposed standard digests).
      • File Upload: Complete and polish implementation of file upload standard, making sure that it works on all platforms. Obtain code from John Mallery.
      • New Cookies: Implement the new cookie extension specification for HTTP.
      • HTTP Features: Implement the HTTP feature extension that allows clients and servers to advertise the set of features that they support in a standardized way.
      • Content Negotiation: Implement the HTTP 1.1 content negotiation specification. (see: RFC 2068)
      • Multiple Languages & Character Sets
      • HTTP 1.1 Details: Make sure all advanced and obscure features of HTTP 1.1 are implemented.
    2. HTML 4.0 Generation: Implement HTML 4.0 as an extension of current generation tools.
    3. XML Generation & Parsing:  Provide efficient tools for generating and parsing XML. This should include RDF and some RDF schemas as a high priority.
    4. Cascading Style Sheets: Implement style sheet generation tools (CSS2) for integration with HTML 3.2 generation facilities.
    5. CLIM Server Window: Overhaul the existing CLIM window so that:
      • It runs on multiple platforms
      • It offers a full set of relevant commands
      • It edits user access control information
      • It offers a variety of useful displays, including:
        • Realtime log display
        • Performance statistics
        • Metering of URL access times
    6. Efficiency Enhancements
      • Header Buffers: Parse HTTP headers into resourced header buffers. Resource header objects. Raw header strings become offsets into the header buffer.
      • Resource URLs: Uninterned URL are created with increasing frequency by search URLs, proxying, and other applications, these should be resourced to minimize consing.
      • Data Caching Facility:
        • Develop caching policies for the pathname datacaching facility.
        • Develop directory-oriented caching policy that is invoked via a keyword on directory export.
        • Where applicable,  provide areas to hold cache data structures and GC policies for optimal performance.
        • Integrate the caching proxy with the datacache facility.
      • Server Metering: Use Henrik Frystyk's (frystyk@w3.org) TCP metering tools and test suite to improve server latency, throughput and efficiency.
      • Double-Buffered Asynchronous Transfer: Develop platform-specific facilities, where applicable, for multi-buffered asynchronous TCP transfers. This improves performance by queuing new buffers TCP to ship, and thus, freeing Lisp to continue computing rather than blocking waiting for TCP to complete delivery of the current buffer.
    7. Implementation Enhancements
      • Declarative Header Parsing/Generation
      • Declarative URL Parsing/Generation
      • Escape Characters In Exported Pathnames
      • Fill CLOS Caches On Start-Up Or Save of Binary Images
      • Factor Into Component Modules
        • Client
        • HTML Synthesis
    8. Compressed Page Transfer Encoding
      • Provide GZIP compression of static files in a manner analogous to CRLF encoding.
      • Implement Deflate Compression and Uncompression
      • Integrate with Server & Client
      • Support
        • Compressed Static Pages
        • On-the-fly Compression
        • On-the-fly Uncompression
    9. Portable Dynamic Form Processing: Use the W3P presentation system to replace CLIM in the Dynamic Forms module. Make Dynamic Forms run portably. The Dynamic Form processing facility is a high-level object-oriented form processing facility that predates HTML form and allows very high level coding of forms with maximal code reuse. It has been used in COMLINK for 5 years and currently drives all the forms used over SMTP and HTTP on the White House Publications system.
    10. Secure HTTP: Implement HTTP tunneling using after-download cryptography.  The first step involves implementing a digest authenticated tunnel that allows the client to know  with complete certainty that the server received the HTTP message in tact and processed the request successfully.
    11. Proxy Service: Add intelligent caching to proxy implementation (see the CERN server). This is pretty important for many companies that operate through firewalls. It is also very handy for downloading URLs to a powerbook for disconnected use
    12. Custom Logging: Develop a variety of custom log formats that yield useful information about HTTP operations. Useful ideas include logging:
      • Form submissions and responses
      • Metering information for URLs
      • Proxy operation
      • Precomputed server analyses
    13. Portable Remote CLOS Interface: Develop an object interface using Inter-Language Unification (ILU) from Xerox's Palo Alto Research Center. Develop interfaces to readily available persistent object stores and CORBA.
    14. Performance Tools
      • Logfile Replayer (aka Web Monkey): Given an HTTP logfile, re-execute the requests against a selected server. This is an indispensable tool for serious performance tuning of CL-HTTP ports.
      • Metering: Develop higher level metering facilities atop the current facilities that capture computation expended on a per URL basis.
      • HTML Display of Performance Statistics: Write HTML displays to facilitate use of metering information to enhance URL performance.
      • Tracing Tools
        • Show Execution Process
    15. Secure Sockets Layer (SSL): Integrate SSL support into the Web server and client using foreign function calls into the freely available reference implementation in C. The C code and specification distributed by Netscape allows non-commercial use for free and commerical use for a license fee. A more ambitious effort would rewrite the C code in Lisp tuned for speed. See also Apache-SSL
    16. Digital Signatures:
      • Develop a framework for digitally signing static and dynamic resources.
      • Ensure portability DSS signature code (available from John Mallery).
      • Devise key management  system.
      • Document secure operation of digital signature facility.
    17. Site Management Tools: Develop a change management facility and incorporate it into tools for managing the content of Web site. Contact Rainer Joswig for existing code and further suggestions.
    18. Portable Lisp Substrate: Develop an integrate substrate from the various port sources that allows CL-HTTP to run across platforms with minimal platform-specific code. Special attention needs to be paid to creating common frameworks for:
      • Networking
      • Processes
      • Locking
      • Streams
    19. Cruise the source code and look for bugs, incomplete features, better numerical declarations, inefficiencies and write patches to fix them.

  2. Server Documentation Projects

    1. Develop a CL-HTTP Tutorial Using Interbook. Contact Peter Brusilovsky plb@cs.cmu.edu for further information.
    2. Extend the online manual or tutorial by writing one or more sections that explains how to do something. The manual should be written in HTML and should explain things with live working examples. We need to target these audiences:
      • Naive users who just want to serve web pages with minimal fuss.
      • Novices who want to Learn Lisp
      • Lisp-capable people who want to write their own response functions and hook up their own systems but do not want to learn more about CL-HTTP than it takes to solve their problem.
      • Lisp Programmers who want to Learn the Server
        • Extend and annotate the example file exports.lisp. Improve the comments to make this example more of a live tutorial. Add additional examples illustrating any previously undocumented features.
        • Improve documentation strings and comments in source code: Give priorities to those areas people are most likely to use or find confusing.
      • Wizards who want to understand the details of CL-HTTP and the Web so they can experiment with new extensions to CL-HTTP and the Web. There may be a category of wizards who are not lisp hackers and we should have a way to reach them.
    3. Overview Server & Architecture
    4. Interfacing Existing Applications
      • CLIM To Web
      • GGIs To LISP
    5. Create icons and gifs for use spicing up CL-HTTP generated pages and advertising the technology.
    6. Dynamic FAQ: Implement a facility for capturing, storing, searching, and displaying FAQs. The facility should provide ultra-targeted answers to user queries based on an inventory of questions and answers,
    7. Graphical Class Hierarchy Browser: Implement a Web-based interface for browsing CLOS class hierarchy and viewing associated documentation strings. Use this to document CL-HTTP class structure. The interface should utilize Lisp's introspection capabilities, using basic (widely supported) Meta-Object Protocol calls to generate and update the tree structure. Pick a few standard starting points, such as the url class. Tree layout algorithms could be combined with the GIF generation code or Java to provide a nice graphical front-end.

  3. Client Projects

    1. Client HTTP 1.1 Conformance: Identify all the client MUSTs in the http 1.1 specification. Prioritize them. Implement them in the order of priority.
    2. Make the client RPC facility faster and more robust

  4. Web Walker Projects

    1. Multithreaded: Use the task queue facility to make the Web walker operate asynchronously with multiple threads.
    2. Persistence: Develop a persistent storage model for use with the Web walker.
    3. CLIM Interface: Develop a CLIM interface to operate the Web Walker. It should include facilities for:
      • Defining Activities
      • Defining Constraints
      • Monitoring Operation
      • Debugging Errors
    4. Web Site Maintenance: Develop an Web walker application for finding all the dead urls on a web site.
    5. Generate VRML views of Sites
    6. Develop Facilities for Periodic Site Walking: This involves developing persistent actions that awake a certain times to execute a Web Walking operation. The operations and any predicates need to be store persistent as well. See: server realms (file or OODB storage) and COMLINK persistent actions.
    7. URL Reminders: Improve the URL reminder application from Rainer Joswig.
    8. XML Parsing: Integrate an XML parser.
    9. Index Web Structures: Use LambdaVista to index and retrieve Web structures.

  5. HyperArchive

    1. Archive Migration: Develop management policies and associated implementations that migrate messages to archives at specified intervals or archive sizes.
    2. MIME Support: Add MIME support so that MIME multipart messages can be viewed using appropriate helper applications.

  6. Database Projects

    1. Flatfile Database: Develop a database building from the specialized code used by the Mail HyperArchive.
    2. Generic API: Provide a generic database API so that platforms can specialize into various native database backends.

  7. LambdaVista

    1. Documentation: Write documentation strings for major functions and datastructures.
    2. Document Universes: Abstract the document-universe data structure into CLOS mixins.
    3. Web Walker Support
    4. Documentation Search: Full text search in CL documentation.
    5. Substring Search via Thesaurus
    6. Easy Document Indexing With Fragment Support
    7. WAlS Interface
    8. WordNet Interface: Implement an interface to WordNet designed to enable higher level applications atop a tookit of WordNet operations.
      • Utilize to improve indexing & retrieval performance of LambdaVista

  8. Log Analysis

    1. Efficient Storage Format: Develop a storage format that requires no parsing or analysis when loading log files into memory. The format should be modular so that only logfiles in a time interval need to be examined.
    2. KDTREE Implementation: Revise the Common Lisp KDTree implementation for high efficiency needed in logging applications.

  9. Platform Specific Projects

    1. Lisp Machine Platform

      1. Mail HyperArchive: Recognize epsilon encodings and render them in corresponding HTML.
      2. Concordia Interface: Develop a Web interface that provides access to the Symbolics document examiner.

    2. Macintosh Platform

      1. OpenTransport Polish: Tune the MCL OpenTransport interface. correctly. See: ``Technote 1145 Living in a Dynamic TCP/IP Environment'' which describes how to write OpenTransport code that correctly handles multiple IP addresses, dial-up links, sleep and wakeup on PowerBooks, modem disconnection, and user reconfiguration.
      2. Binary Application with User Interface: Develop a stand-alone binary app so people without MCL can run the server. This involves developing a GUI user interface from which the user can operate and maintain the server. Among other things, it means an interface to specifying the configuration file and ensuring that it is loaded on start up. Have a look at the remote administration interface for MacHTTP. Alternatively, a forms-based approach would be portable across multiple platforms and would be safe once secure http is incorporated.
        • Special Application
        • Top-Level Loop
        • Catch Errors
        • Handle Apple Events
      3. QuickDraw 3D Integration
      4. Wood DB Interface
      5. ODBC Interface
      6. MCL-Based WYSIWYG Web Client In FRED
      7. Improve Performace of MD5 & SHA: Integrate some 16bit MD5 code from Digitool and apply the sae ideas to SHA.

    3. UNIX Platforms

      1. Send-Mail Interface
      2. Revise CGI Interfaces to Conform with CGI Work in MCL

    4. PC Platform

      1. ACL 5.0: Bring port up to HTTP 1.1 conformance
      2. Port to C Lisp
      3. ODBC DB Interface

  10. Extensions, Add-Ons & Applications

    1. Higher-Level HTML Synthesis
      • Cliche & Abstractions
        • Lisp To HTML Translator
        • Graphics
          • Bar Charts
          • Graphs
          • Tree-Drawing
        • Command Processor
      • Automatic Generation of Navigational Links
      • Graphics For Space Filling (e.g., Backgrounds)
    2. Java Synthesis
      • Java Presentations For Client
      • Java Primitives Composable From LISP
    3. Indexing & Search For Web Regions
      • Gatherers On The Server
      • Brokers For Delivery To Clients
      • Specification For Collecting & Distributing Site Indices. See work by Harvest group.
    4. Server Collaboration Site
      • Register Servers
      • Register People & Interests & Abilities
      • Share Extensions
      • Advertise Extensions
      • Peer Review Extensions
      • Automatic Patch Distribution
    5. Code Sharing and Distribution
      • Build Web Collaboration Systems For Sharing Code: Provide a forms interface that allows people to submit code or extensions to CL-HTTP and retrieve submissions of others. Incorporate a peer review system that allows people to rate software along various dimensions.
      • Java Library: Use the facility to:
        • Capture Java applets from users.
        • Document applet APIs.
        • Locate applets based on search criteria.
        • Invoke applets dynamically.
    6. Mobile Lisp Code
      • Secure Lisp Listener
      • Safe Common Lisp Package
      • No Side Effects
      • Bounded Computation Time & Space
        • Bounded Stack Depth
      • Fasl Streams To Clients
    7. Modern Common Lisp Defsystem: This will provide superior system management that is not wedged in the past, i.e., based on pathnames. It should accept URNs and provide a patching facility and version control system at least as good as the Lisp Machine.
      • Start from CLOS-based portable defsystem
      • URN fragments provide relinearization capability
      • Tacking of code dependency enables configurable loading
      • Tree shaking provides small application images.
    8. Synchronized Multimedia: Implement the current standards for synchronized multimedia Web display.
      • Emit Streamed Audio
        • MacInTalk Source
        • Streamed
      • Emit Streamed Video: Implement facilities for emitting streamed video using standard formats.
    9. Integrate CLIM Graphics: Port the SRI CLIM generation tools to CL-HTTP and portable lisp
    10. MacroMedia Shockwave Generation: Develop a facility for generating Shockwave on the fly.
    11. VRML 2.0 Synthesis: Implement a VRML 2.0 generation package analogous to the distribution VRML 1.0 facility. 4.0 browsers from Netscape and Microsoft implement the standard.
    12. Active-X Synthesis
      • Active-X Presentations For Client
      • Active-X Primitives Composable From LISP
    13. Use SMTP And NNTP To Distribute Hyperlinks
      • Hyperlink-Based Mail Reader On The Web
    14. SMTP Mailer: Extend the current portable SMTP transmission code into a full-fledged mailer. Abstract the storage componet so that a database can serve as the backend. Use the mailer as an SMTP gateway for the server.
    15. NNTP Gateway: Extend the NNTP code (available from John Mallery) to gateway between NTTP and the server. Ensure portability of implementation.
    16. Server Trouble Shooter: Diagnose of server problems based on a case library acquired by experience.
John C. Mallery (jcma@ai.mit.edu) Rainer Joswig (joswig@lavielle.com)

M.I.T. Artificial Intelligence Laboratory