The mailing list is not quite ready yet, but will be soon. Subscribe by sending mail to ippm-request@psc.edu. The paper meeting roster will NOT be used to preload the mailing list. It will be preloaded from the NANOG mailing list maintained at MERIT.
[Most of the slides were text only, so they are imbeded in the minutes.]
What makes a good metric? How do we design a metric that works for all the users and providers?
It has been suggested that the broader list is really a reflection of user expectations and requirements. Different user expectations exists and need to be addressed in some way. Matt thinks that some of this discussion reflects the differences between the roles of an IP wholesaler and retailer. It makes sense to start working on the core metrics and work towards a broader set. Therefore, The IPPM charter should include the broader metrics, but since they are "fuzzy" and subject to non consensus associated with subjective measures, they should be "tabled" until after the core metrics are assured convergence.
We listed existing tools to give us confidence that we had a a complete metrics wish list.
It is an example of a bulk transfer metric. It provides a lot of useful information. It provides a reasonable mimicry of TCP, but does make use if ICMP just like regular traceroute. Christian Huitema said that ICMP was not intended for performance measurement, so as long as this is the case, ICMP does not provide a good mechanism for doing measurement. There is considerable variety among routing vendors in their ICMP processing. Since some vendors do respond to ICMP as fast as they forward packets and some don't, this may cause vendors' equipment to be at a disadvantage because of choices the ICMP responses architecture. A routing requirements working group does exists where this can be discussed. Some folks suggested that the RA machines might be used as landmark machines. Bill Manning said that should not be done. Other landmark machines could be identified.
What is the future of treno? Does it really emulate idealized TCP? If not, vendors may optimize their software to make it work well with the tool and not really help make the network work better. Sally Floyd volunteered to work with Matt to see if treno does emulate an appropriate TCP. How do we use it for testing? There are issues in creating a mechanism to do testing (a standard procedure). John Boone volunteered to work on the development of a standard testing procedure to measure bulk transfer using treno.]
With the new Internet Architecture, different sites will use different paths to get to the same places. The RA mission is to insure a consistent set of routes for all users. By using a RIPE-181-based database and through an analysis, a consistent routing configuration can be presented.
There are a number of things that might be measured:
What about reporting? Since this information can only be really be collected at exchange points, what would be done with the data collected? Defining a scope is really critical to doing this. Concerns should be sent to the RA at ra-team@merit.edu or ra-team@isi.edu.
The RA will be all NAPs and the two MAEs and other interconnects. The NAP operators are not willing to do these type measurements since it is not related to the level 2 service they are provisioning.
General information about the RA and current North American routing architecture is availible at http://www.isi.edu/div7/ra under "papers"
A number of other issues were also discussed during this meeting.
Once this work is complete, specific ownership issues should be considered. Some standards groups assert ownership of the metrics and the tools that produce them. This may be appropriate here. There was also a concern that the meaning of the results should be as immutable over time (some folks called this tool drift). This may be too hard to do and is definitely hard to consider when the metrics and the tools are not yet articulated.
Is it possible to create a metric that could be used by an independent party that would produce a meaningful result? Can the metrics be defined in an unambiguous way? Can specific issues such as the interactive performance of a telnet situation from the user perspective be quantified?
There was also considerable concern about providers testing themselves and other providers versus consumers doing the testing and issues involving the need for disclosure of components involved (Black-box versus Clear-box testing). One attendee asserted that Information in the current NSFNET has been available publicly for years and folks have not exploited that to unreasonably bash on anyone. Matt would like there to be some way for this to be done using RA-type infrastructure. This data should be sanitized in such a way that any bias is effectively removed. If this issue is not resolved by IETF and the providers, it is believed that the consumer will assert their right-to-know.
There was also an open issue about how any of this work might related to IPv6.
There was strong consensus that IPPM work is important and should be done within IETF.
There was consensus that this work will initially focus on the delivery and routing datagrams. (i.e. not related services, such as NOC charteristics)
However IPPM appears to be within the charter of an existing working group (BMWG, the Bench Marking WG). Their charter clearly meant to include measurements on real operational networks. It may be easier to take the IPPM effort and move it. On the other hand, the effective orientation of BMWG has clearly been towards the laboratory environment. There was some concern that the BMWG is a little too open ended, and that a focused charter for IPPM would be better. Christian Huitema was particularly encouraging on this issue.
Matt called for volunteers to chair or co-chair the WG. Nobody volunteered publicly [but there have since been several private responses - MM].
The group decided that a subcommittee would convene to draft a charter. The following people volunteered: Paul Love, Teunis J Ott, John Boone and Matt.
Matt hopes that the charter and the relationship to the other working groups will be finalized by the May NANOG (May 21, 1995) meeting.
Final review of the minutes by Matt Mathis.