> So who's here?

Hi -- I'm an Architect at IBM's Pacific Development Center in Vancouver.
I'm working on a couple of projects in the banking & financial sector that
have some pretty interesting distribution issues to deal with. I've been on
the dist-obj mailing list
(http://www.distributedcoalition.org/mailing_lists/dist-obj/dist-obj.html)
for a bunch of years and heard about this list on xml-dist-app.

I never seem to get the time to contribute much to the lists I'm on... too
much travelling & such... Hopefully I'll muster up a few useful posts on
this list...

My interests include building reliable distributed systems, software
engineering (metrics, methods, etc.), architecting systems so they use
mobile code, etc... and of course widely distributed, decentralized
systems.

cheers,

- steve


Clay Shirky <clay@...> on 07/17/2000 05:28:04 PM

Please respond to decentralization@egroups.com

To:   decentralization@egroups.com
cc:
Subject:  [decentralization] who's here?




So who's here?

I just had a piece in the Times about Napster, and while it doesn't
address decentralization per se, it does address one of its effects in
the music biz, so I offer it here as a possible ice breaker:

http://www.nytimes.com/00/07/15/oped/15shir.html

--
Clay Shirky                 |    shirky.com - Essays on the Internet:
http://www.shirky.com/      |        Culture, Economics, Globalization



------------------------------------------------------------------------
Get a NextCard Visa, in 30 seconds!
1. Fill in the brief application
2. Receive approval decision within 30 seconds
3. Get rates as low as 2.9% Intro or 9.9% Fixed APR
http://click.egroups.com/1/6628/3/_/_/_/963880337/
------------------------------------------------------------------------

To unsubscribe from this group, send an email to:
decentralization-unsubscribe@egroups.com

Ben -

Can you talk about what consensus there is (if any) for future
direction of Gnutella-NG?

- Lucas

(I am definitely interested to hear your thoughts on trust
implementations, but maybe that is better off-list.  Anyone else
interested maybe send an email and we'll form a smaller group to
pick through details?)

----- Original Message -----

From: Burd, Greg

To: 'decentralization@egroups.com'

Sent: July 19, 2000 9:31 AM

Subject: RE: [decentralization] the view from GnutellaNG (Re: who's here?)

Ben,

      I too am glad to see you hanging out on this m-board.  If we do this
right, the possibilities are endless.

      In regards to #1, I agree.  We need this random, transient network
of cooperating nodes to self organize into some efficient structure.  My
feeling is that a particular node's peers (the nodes directly connected to
it) should participate in a process that determines the role (as you
outlined in #2) and service quality of the node in question.  In other
words, a nodes position in the net-cloud is determined by its peers and can
change at any time.  A node should understand and advertise its
capabilities, but not be able to place itself in the network structure where
ever it likes.  That would open up a large hole for hackers to subvert the
network.

      A second point I would like to get out there to the Gnutella clan.
Keep it up, but don't be so narrow in scope.  Separate the idea of
peer-to-peer (or servant based networks) from query/response systems for the
discovery of files.  That may be one good example of where p2p networks are
effective, but it isn't the only one.  There are many other uses of p2p
network ideas, some may even be legal.  ;-)

      So that leads me to my third point which is more of a question than
a statement.  I don't think the p2p/decentralized/servant/whatever
communities have any consensus as to the model of communication between p2p
participants.  What is the ultimate goal of the p2p network infrastructure?
If you look at Gnutella, WorldOS, FreeNet, etc they all have different goals
and are taking different approaches to reach those specific goals.  Maybe
what we have here is a need for a "OSI Seven Layer Model" approach to this
problem.  Can we break down the parts of this idea into distinct pieces and
then find a way to fit that puzzle together such that you can setup any
number of different p2p network interactions which satisfy all the various
goals people have?

* connection to a net-cloud, or discovery (ala Jini)
* msg content negotiation, is it BXXP or HTTP or what? maybe a PPP like
negotiation?
* security, is this cloud secure?  how can I tell?  is my identity safe?
how?
* negotiation, what does a net level conversation look like?
* participation, who can participate in this cloud?
* routing
* broadcast, unicast, anycast, what?
* anonymous tunneling (ala FreeNet)
* ???

thoughts?

-greg

-----Original Message-----
From: Ben Houston [mailto:ben@...]
Sent: Tuesday, July 18, 2000 6:44 PM
To: decentralization@egroups.com
Subject: [decentralization] the view from GnutellaNG (Re: who's here?)


Hi,


I'm a student who is on the working group of the gnutella-next
generation protocol (http://gnutella-ng.wego.com).  We are
unfortunately mostly young students thus disorganized and overloaded
with unpolished "ideas". :-/


The summary of the best ideas (IMHO) are as follows:

1. Ego-centric self organization is needed to optimize the p-2-p
network

Nodes join the p2p network randomly.  A random network is useless for
efficient p2p communication thus some organization has to take
place.  Thus the focus becomes on "emergent" like algorithms that
work on an ego-centric bases to organize the network.  The network
should become a something like a tree structure in terms of the
message passing but it should remain a random network in terms of the
maintained connections.  Thus if a client on the main communication
tree spontaneously combusts there is an enough other connections that
the network can reorganize into another tree (or "near-tree").

2. Clients should know their capability and take roles in the network
that maximizes their capably.

A client should be able to determine its capabilities (bandwidth,
cpu, memory, diskspace).  The capabilities are limited first by the
actually capabilities of the computer, and second by what the owner
of the computer is willing to allow.  Computers with large bandwidth
would organized themselves into the backbone of the network while
computers with low bandwidth would relegate themselves to the edges. 
Computers with low bandwidth may maintain a lot of redundant
connections in order to maintain network coherence when a central
client drops.


I hope I'm writing that's somewhat useful to you as well as on
topic. :-)
-ben houston
http://www.exocortex.org/~ben



--- In decentralization@egroups.com, Clay Shirky <clay@s...> wrote:
> So who's here?
>
> I just had a piece in the Times about Napster, and while it doesn't
> address decentralization per se, it does address one of its effects
in
> the music biz, so I offer it here as a possible ice breaker:
>
> http://www.nytimes.com/00/07/15/oped/15shir.html
>
> --
> Clay Shirky                 |    shirky.com - Essays on the
Internet:
> http://www.shirky.com/      |        Culture, Economics,
Globalization



------------------------------------------------------------------------
Old school buds here:
http://click.egroups.com/1/7081/3/_/_/_/963960313/
------------------------------------------------------------------------

To unsubscribe from this group, send an email to:
decentralization-unsubscribe@egroups.com


To unsubscribe from this group, send an email to:
decentralization-unsubscribe@egroups.com

>  So that leads me to my third point which is more of a question
> than a statement.  I don't think the p2p/decentralized/servant/
> whatever communities have any consensus as to the model of
> communication between p2p participants.  What is the ultimate goal
> of the p2p network infrastructure?

Surely for a list called 'decentralization', you don't *want* an
ultimate goal, nu? Following Lucas evolutionary biology idea, surely
the goals of all the p2p systems are short-term, namely to acquire
users, so as to have a chance of surviving to another rev.

Think of it as the Selfish Source Code model -- all a piece of code
has to do to survive is to stay in use.

> If you look at Gnutella, WorldOS, FreeNet, etc they all have different goals
> and are taking different approaches to reach those specific goals.

Thats true if you look at birds, lizards, and beetles as well, or
doctors, lawyers, and indian chiefs.

> Maybe what we have here is a need for a "OSI Seven Layer Model"
> approach to this problem.

The hilarious part of the "OSI model" is that despite its popularity,
it is rarely used in practice (or maybe its so popular because its so
rarely used, so it seems more ideal than the internet (which, of
course, it is)).

My favorite explanation of why OSI sucks so bad is at
http://www.bath.ac.uk/~ma7jkh/notes/78.html

   * Comparison between OSI and Internet models.

   The OSI model was developed before the implementation.
   The Internet model was developed after the implementation.

   OSI distinguishes between the model and the implementation.
   The Internet is fuzzier in this respect.

   The OSI model is general.
   The Internet model is specific, it has only one implementation (the
   Internet).

   OSI had sublayers- very complicated.
   The Internet model is simple

   Why isn't the OSI model used much in practice?

        Bad Timing: OSI took too long to formulate. TCP/IP was already
        popular by the time the standards were published.
        Bad Technology: The model layers in OSI are not good. Why choose
        seven layers?
        Bad Implementations: They were big and slow, and got a reputation
        as being bad. Later implementations were better, but too
        late. TCP/IP was free and fast.
        Bad Politics: OSI was designed by committee, and while they were
        talking, the Internet was being implemented for real.

   The Internet model is not perfect.

        Separation of implementation and model is poor.
        Model is not general.
        Only applies to TCP/IP.
        Does not separate physical and data link layers.

   The OSI model is good, and widespread.
   The OSI model implementation is rarely used.

   The Internet model is poor, but the implementation is widespread.

That seems to me to be true of Napster as well -- the model is poor,
but the implementation is widespread.

-clay

--
Clay Shirky                 |    shirky.com - Essays on the Internet:
http://www.shirky.com/      |        Culture, Economics, Globalization

Ben,

	 I too am glad to see you hanging out on this m-board.  If we do this
right, the possibilities are endless.

	 In regards to #1, I agree.  We need this random, transient network
of cooperating nodes to self organize into some efficient structure.  My
feeling is that a particular node's peers (the nodes directly connected to
it) should participate in a process that determines the role (as you
outlined in #2) and service quality of the node in question.  In other
words, a nodes position in the net-cloud is determined by its peers and can
change at any time.  A node should understand and advertise its
capabilities, but not be able to place itself in the network structure where
ever it likes.  That would open up a large hole for hackers to subvert the
network.

	 A second point I would like to get out there to the Gnutella clan.
Keep it up, but don't be so narrow in scope.  Separate the idea of
peer-to-peer (or servant based networks) from query/response systems for the
discovery of files.  That may be one good example of where p2p networks are
effective, but it isn't the only one.  There are many other uses of p2p
network ideas, some may even be legal.  ;-)

	 So that leads me to my third point which is more of a question than
a statement.  I don't think the p2p/decentralized/servant/whatever
communities have any consensus as to the model of communication between p2p
participants.  What is the ultimate goal of the p2p network infrastructure?
If you look at Gnutella, WorldOS, FreeNet, etc they all have different goals
and are taking different approaches to reach those specific goals.  Maybe
what we have here is a need for a "OSI Seven Layer Model" approach to this
problem.  Can we break down the parts of this idea into distinct pieces and
then find a way to fit that puzzle together such that you can setup any
number of different p2p network interactions which satisfy all the various
goals people have?

* connection to a net-cloud, or discovery (ala Jini)
* msg content negotiation, is it BXXP or HTTP or what? maybe a PPP like
negotiation?
* security, is this cloud secure?  how can I tell?  is my identity safe?
how?
* negotiation, what does a net level conversation look like?
* participation, who can participate in this cloud?
* routing
* broadcast, unicast, anycast, what?
* anonymous tunneling (ala FreeNet)
* ???

thoughts?

-greg

-----Original Message-----
From: Ben Houston [mailto:ben@...]
Sent: Tuesday, July 18, 2000 6:44 PM
To: decentralization@egroups.com
Subject: [decentralization] the view from GnutellaNG (Re: who's here?)


Hi,


I'm a student who is on the working group of the gnutella-next
generation protocol (http://gnutella-ng.wego.com).  We are
unfortunately mostly young students thus disorganized and overloaded
with unpolished "ideas". :-/


The summary of the best ideas (IMHO) are as follows:

1. Ego-centric self organization is needed to optimize the p-2-p
network

Nodes join the p2p network randomly.  A random network is useless for
efficient p2p communication thus some organization has to take
place.  Thus the focus becomes on "emergent" like algorithms that
work on an ego-centric bases to organize the network.  The network
should become a something like a tree structure in terms of the
message passing but it should remain a random network in terms of the
maintained connections.  Thus if a client on the main communication
tree spontaneously combusts there is an enough other connections that
the network can reorganize into another tree (or "near-tree").

2. Clients should know their capability and take roles in the network
that maximizes their capably.

A client should be able to determine its capabilities (bandwidth,
cpu, memory, diskspace).  The capabilities are limited first by the
actually capabilities of the computer, and second by what the owner
of the computer is willing to allow.  Computers with large bandwidth
would organized themselves into the backbone of the network while
computers with low bandwidth would relegate themselves to the edges.
Computers with low bandwidth may maintain a lot of redundant
connections in order to maintain network coherence when a central
client drops.


I hope I'm writing that's somewhat useful to you as well as on
topic. :-)
-ben houston
http://www.exocortex.org/~ben



--- In decentralization@egroups.com, Clay Shirky <clay@s...> wrote:
> So who's here?
>
> I just had a piece in the Times about Napster, and while it doesn't
> address decentralization per se, it does address one of its effects
in
> the music biz, so I offer it here as a possible ice breaker:
>
> http://www.nytimes.com/00/07/15/oped/15shir.html
>
> --
> Clay Shirky                 |    shirky.com - Essays on the
Internet:
> http://www.shirky.com/      |        Culture, Economics,
Globalization



------------------------------------------------------------------------
Old school buds here:
http://click.egroups.com/1/7081/3/_/_/_/963960313/
------------------------------------------------------------------------

To unsubscribe from this group, send an email to:
decentralization-unsubscribe@egroups.com

> It has taken the internet infrastructure way too long to catch up to the
> shift from e-mail to HTTP traffic

From my point of view, it has taken exactly the right amount of time,
then til now. Engineers who pine for more order will always lament how
slow and stupid and chaotic everything is, but I think we _want_ the
net to be behind the traffic. Thats what makes it adaptive.

> ...and with these viral P2P systems, the change in network traffic
> characteristics will be over night....

Already happening. Now we will react.

The issue is not how to design p2p systems -- they're already out
there, and you know how the Protocol Fairy is about not granting
do overs. The issue is how to improve the p2p systems we've got.

--
Clay Shirky                 |    shirky.com - Essays on the Internet:
http://www.shirky.com/      |        Culture, Economics, Globalization

Ben -

Let me see if I understand your points.  Please correct me if I
am wrong.

> 1. Ego-centric self organization is needed to optimize the p-2-p
> network

The goal is to provide an an optimized tree, where the number of
hops needed to find a resource is reduced in advance.  Because
nodes join and leave the network on their own schedule, there is
no way to do forced a general recalc, where an entire subnet is
mapped and optimized.  So the desirable behavior will have to be
emergent, and this means dynamic remapping.

If the nodes you are connected to have optimized connections,
then your connections are semi-optimized.  So you want to improve
your own connections just a little, and whenever you do improve
your connections you also improve the connections around you.

The point is to lower your average hop count.  So let's say you
are handling MP3 requests, and are seeing high hop counts in the
returned results.  You may be able to optimize your own subnet by
requesting direct connections to any node IPs that seem to have
travelled a large number of hops.  ...I believe that this is a
good beginning, but pretty tricky to get right.

> 2. Clients should know their capability and take roles in the network
> that maximizes their capably.

I have seen this idea on the wego boards before - maybe in your
proposal? - and was not comfortable with it.  I believe that
clients should take on whatever they see as being to their own
advantage.  The distinction is between
1) self interest that serves the community as a secondary effect
2) acting for the good of the community and serving your self
interest as a secondary effect

It's kinda an evolutionary logic.  Broader goals have to be
served via narrower goals.  #1 manipulates the small scale to get
desirable emergent behaviors.  #2 requires person-to-person
coordinatin.

BTW, I'm happy to see you hanging out here.  The split between
the various decentralization factions is causing plenty of work
to be duplicated.

- Lucas


Ben Houston wrote:
>
> Hi,
>
> I'm a student who is on the working group of the gnutella-next
> generation protocol (http://gnutella-ng.wego.com).  We are
> unfortunately mostly young students thus disorganized and overloaded
> with unpolished "ideas". :-/
>
> The summary of the best ideas (IMHO) are as follows:
>
> 1. Ego-centric self organization is needed to optimize the p-2-p
> network
>
> Nodes join the p2p network randomly.  A random network is useless for
> efficient p2p communication thus some organization has to take
> place.  Thus the focus becomes on "emergent" like algorithms that
> work on an ego-centric bases to organize the network.  The network
> should become a something like a tree structure in terms of the
> message passing but it should remain a random network in terms of the
> maintained connections.  Thus if a client on the main communication
> tree spontaneously combusts there is an enough other connections that
> the network can reorganize into another tree (or "near-tree").
>
> 2. Clients should know their capability and take roles in the network
> that maximizes their capably.
>
> A client should be able to determine its capabilities (bandwidth,
> cpu, memory, diskspace).  The capabilities are limited first by the
> actually capabilities of the computer, and second by what the owner
> of the computer is willing to allow.  Computers with large bandwidth
> would organized themselves into the backbone of the network while
> computers with low bandwidth would relegate themselves to the edges.
> Computers with low bandwidth may maintain a lot of redundant
> connections in order to maintain network coherence when a central
> client drops.
>
> I hope I'm writing that's somewhat useful to you as well as on
> topic. :-)
> -ben houston
> http://www.exocortex.org/~ben
>
> --- In decentralization@egroups.com, Clay Shirky <clay@s...> wrote:
> > So who's here?
> >
> > I just had a piece in the Times about Napster, and while it doesn't
> > address decentralization per se, it does address one of its effects
> in
> > the music biz, so I offer it here as a possible ice breaker:
> >
> > http://www.nytimes.com/00/07/15/oped/15shir.html
> >
> > --
> > Clay Shirky                 |    shirky.com - Essays on the
> Internet:
> > http://www.shirky.com/      |        Culture, Economics,
> Globalization
>
> ------------------------------------------------------------------------
> Old school buds here:
> http://click.egroups.com/1/7081/3/_/_/_/963960313/
> ------------------------------------------------------------------------
>
> To unsubscribe from this group, send an email to:
> decentralization-unsubscribe@egroups.com

Hi,


I'm a student who is on the working group of the gnutella-next
generation protocol (http://gnutella-ng.wego.com).  We are
unfortunately mostly young students thus disorganized and overloaded
with unpolished "ideas". :-/


The summary of the best ideas (IMHO) are as follows:

1. Ego-centric self organization is needed to optimize the p-2-p
network

Nodes join the p2p network randomly.  A random network is useless for
efficient p2p communication thus some organization has to take
place.  Thus the focus becomes on "emergent" like algorithms that
work on an ego-centric bases to organize the network.  The network
should become a something like a tree structure in terms of the
message passing but it should remain a random network in terms of the
maintained connections.  Thus if a client on the main communication
tree spontaneously combusts there is an enough other connections that
the network can reorganize into another tree (or "near-tree").

2. Clients should know their capability and take roles in the network
that maximizes their capably.

A client should be able to determine its capabilities (bandwidth,
cpu, memory, diskspace).  The capabilities are limited first by the
actually capabilities of the computer, and second by what the owner
of the computer is willing to allow.  Computers with large bandwidth
would organized themselves into the backbone of the network while
computers with low bandwidth would relegate themselves to the edges.
Computers with low bandwidth may maintain a lot of redundant
connections in order to maintain network coherence when a central
client drops.


I hope I'm writing that's somewhat useful to you as well as on
topic. :-)
-ben houston
http://www.exocortex.org/~ben



--- In decentralization@egroups.com, Clay Shirky <clay@s...> wrote:
> So who's here?
>
> I just had a piece in the Times about Napster, and while it doesn't
> address decentralization per se, it does address one of its effects
in
> the music biz, so I offer it here as a possible ice breaker:
>
> http://www.nytimes.com/00/07/15/oped/15shir.html
>
> --
> Clay Shirky                 |    shirky.com - Essays on the
Internet:
> http://www.shirky.com/      |        Culture, Economics,
Globalization

> congestion control in these systems is going to be too complex for
most systems to intelligently deal with, so optimization in terms of
network traffic is going to be important so that we don't bring the
internet to its knees.


this also leads to the question of whether the incumbent client-
server protocols like HTTP can support the requirements of p2p
communications .

take the viral nature of p2p : the utility lies in both the rapid
distribution and lifespan of a request . if the lifespan isn't
managed in a proper way , the traffic will be crippling . one
solution is to provide all management at the application level ,
assuming that developers will adhere to limiting the lifespan
effectively . however , taking a cue from the chaotic nature of
things [ plus a nod to said Murphy ] , it is likely that some
developers will not control the lifespan of their traffic .

hence , enabling a protocol to bear some of the lifespan management
might be a good idea . whether or not that protocol could ride
piggyback on HTTP is another issue up for debate .

brian .

>
> On the other hand, in any fully P2P situation, why bother optimizing?

One thing to consider is that optimization in this case is not just a
performance issue, its a friendliness issue.  With the viral distribution
nature of P2P systems one has to keep in mind the effect that these
systems are going to have on internet backbone traffic.  Internet-scale
congestion control in these systems is going to be too complex for most
systems to intelligently deal with, so optimization in terms of network
traffic is going to be important so that we don't bring the internet to
its knees.

It has taken the internet infrastructure way too long to catch up to the
shift from e-mail to HTTP traffic and with these viral P2P systems, the
change in network traffic characteristics will be over night....

-Justin

We used to stuff like this all the time on the work we did at my last
employer.  Basically you need to standardize on some sort of mobile code
system like Java.  For negotiation you simply have an XML document that
describes the interface/contract you are communicating with, and contains
a pointer to where the code is that implements that interface.  Bill
Venner's with the Jini ServiceUI project did something like this for
pushing down UI components for Jini services which is quite elegant.

But then again the problem with these solutions is that you have to
standardize on a mobile code system/framework.

>
> Is there anything in the real world that does this now? It sounds like
> a good idea in theory, but given that even the negotiating protocol
> would be non-standard, mightn't it be bette rto just eat the overhead
> for run-of-the-mill stuff, and tweak when pure performance is needed
> (akin to deciding between SQL and native db calls?)
>

>
> My belief is that having an LCD like HTTP is a valuable asset.
> There should always be a fallback lingua franca, particularly
> because decentralized systems are so varied.  Also, many
> applications don't need to pass a lot of messages.  So let's say
> there does need to be a least common denominator, a legacy system
> like http.
>

Definately agreed, and what I think you'll find is will see alot more
active networking sort of stuff in this space, where nodes are pushing
protocol implementations to other nodes....using HTTP as a bootstrap (or
fallback).  I think there will be some standardization on common themes in
these systems, but for the most part I believe the protocols will end up
being application specific.

For Dave I could see some sort of HTTP/XML based transport pushing
scripts/agents to end nodes to perform work would probably be very
interesting.....I think it would be entirely possible to implement an
interesting peer-2-peer system using the web standards support within
mozilla.  Adam Beberg is pushing his portable C stuff with cosm, and I
would probably tend to advocate Java or JPython code being pushed to the
nodes.  Peer-2-peer systems may be the first place that we actually start
seeing some useful mobile agents.

-Justin

> BTW, are you the Clay Shirky that wrote the op-ed piece in the NY
> Times, if so, a hat-tip and thanks for the inspiration. Great piece,
> right to the point.

The same. Glad you liked it.

Thats really my interst in p2p -- the freedom angle. My new motto is

"Its Prohibition, and Napster is a bathub full of gin."

-clay

> Is there anything in the real world that does this now?

TLS for one (resets a non-encrypted socket to an encrypted one on
the fly).  WorldOS for another.

The app server needs to allow cgi scripts (or whatever does that
kind of work) to reset the transport handler.  The transport
handler is any executable object that can read a stream and
convert it to a common object type.  So you negotiate the upgrade
by calling reset_transport_handler.cgi, and the cgi calls into
the server to set the transport handler.

Clay Shirky wrote:
>
> > Pairs of nodes should also be able to negotiate a protocol
> > upgrade, EG from the slow LCD to a faster but less common
> > protocol.
>
> Is there anything in the real world that does this now? It sounds like
> a good idea in theory, but given that even the negotiating protocol
> would be non-standard, mightn't it be bette rto just eat the overhead
> for run-of-the-mill stuff, and tweak when pure performance is needed
> (akin to deciding between SQL and native db calls?)
>
> -clay
>
> ------------------------------------------------------------------------
> Old school buds here:
> http://click.egroups.com/1/7081/3/_/_/_/963945435/
> ------------------------------------------------------------------------
>
> To unsubscribe from this group, send an email to:
> decentralization-unsubscribe@egroups.com

>>Is there anything in the real world that does this now? It sounds like
a good idea in theory, but given that even the negotiating protocol
would be non-standard, mightn't it be bette rto just eat the overhead
for run-of-the-mill stuff, and tweak when pure performance is needed
(akin to deciding between SQL and native db calls?)

I absolutely agree.

It's all about apps, what are you doing.

For example, if I were running Napster's servers, and I was using XML (they
aren't) to wire the clients to the database, I would very carefully study
everything about the connection, and cut fat where ever it improved perf,
and to satisfy Murphy I'd probably do it in places where it wasn't backed up
by numbers or logic.

On the other hand, in any fully P2P situation, why bother optimizing? It's
best to know why. Keeping everything super-high-level offers the quickest
deployment. People often overlook how widely available XML-RPC is. You save
time by using already-burned-in code, and when it comes time to hire another
programmer to work on the server, you have more choices.

BTW, are you the Clay Shirky that wrote the op-ed piece in the NY Times, if
so, a hat-tip and thanks for the inspiration. Great piece, right to the
point.

Dave

> Pairs of nodes should also be able to negotiate a protocol
> upgrade, EG from the slow LCD to a faster but less common
> protocol.

Is there anything in the real world that does this now? It sounds like
a good idea in theory, but given that even the negotiating protocol
would be non-standard, mightn't it be bette rto just eat the overhead
for run-of-the-mill stuff, and tweak when pure performance is needed
(akin to deciding between SQL and native db calls?)

-clay

The open question is whether these results are specific to RMI
and HTTP, or to what degree they apply to all
low-level/high-level protocol comparisons.

My belief is that having an LCD like HTTP is a valuable asset.
There should always be a fallback lingua franca, particularly
because decentralized systems are so varied.  Also, many
applications don't need to pass a lot of messages.  So let's say
there does need to be a least common denominator, a legacy system
like http.

Pairs of nodes should also be able to negotiate a protocol
upgrade, EG from the slow LCD to a faster but less common
protocol.

Can't say that I love RMI in this context - java centricity seems
like a mistake.  In a distributed system, many nodes are working
together under a single controller.  That single controller has
the option of bridging language and protocol types; so it can
decree that everyone speak a common dialect like Java/RMI.
Decentralized systems are different from distributed systems in
that authority is delegated to outside parties.  Those outside
parties don't have to take instruction from the consumer - it is
up the consumer to make their query understandable to the
provider.


Justin Chapweske wrote:
> Here is an interesting tidbit that I posted on HTP a while back.  It is
> basically a technique for setting up a direct point to point connection
> between two boxes behind seperated NAT'd networks without going through a
> mirror type server:

lovely!



David Orchard wrote:
>
> I did similar tests in Oct '99.
>
> Two tests are of interest: transport times and serialization times.
>
> I found that connection-oriented RMI (5 ms) is about 250% faster than
> connection-oriented HTTP 1.1 (12 ms).
> Connection-less RMI is about 80% (12 ms) faster than connection-less HTTP
> (20 ms).
> I did not have a chance to check out HTTP 1.1 pipelining or compression.
>
> I found that DOM serialization took about 50% longer than Java
> serialization.
>
> I suggest that most usage patterns of distributed objects are
> connection-less, where HTTP performs better as a percentage.
>
> Cheers,
> Dave Orchard
> XML Architect
> Jamcracker, Inc.
> 935 Stewart Dr.
> Sunnyvale, CA 94086
> p: 408.830.1886
> f: 408.328.0936
>
> Named to Red Herring's list of 100 Most Important Companies:
> www.redherring.com/mag/issue79/herring100/jamcracker.html
>
> Named to Fortune's list of Cool Companies 2000:
> http://www.fortune.com/fortune/cool/coo.html
>
> -----Original Message-----
> From: Dave Reynolds [mailto:der@...]
> Sent: Tuesday, July 18, 2000 7:07 AM
> To: decentralization@egroups.com
> Subject: [decentralization] Re: Fractional Horsepower HTTP Server
>
> I have done some small scale experiments testing performance
> curves of Java implementations of SOAP and SOAP-like protocols
> against Java RMI for a range of payload sizes (10b-100kb) in
> a Lan environment using PII-400 class PC hardware. The performance
> curves against payload size are complex but for middle-of-the-range
> stuff (a few kb) RMI ran around one order of magnitude faster (5ms
> versus 40ms round trip) with substantially reduced latency. Those
> were on a WinNT platform both ends. On Linux (Redhat 6.1) the
> different was much greater (5ms versus 120ms) for reasons I've not
> managed to track down.
>
> This is not science (or even engineering). Stuff like this depends
> on the details of hardware and OS setups, network traffic, maturity
> of the protocol implementations etc. However, at least it is
> marginally better than just handwaving.
>
> Whether these differences in actual round-trip and latency matter in
> a given application will vary a lot. The difference between 10ms and
> 100ms may be huge or irrelevant depending on context.
>
> Dave Reynolds, HP Laboratories, Bristol
>
> --- In decentralization@egroups.com, Lucas Gonze <lucas@g...> wrote:
> > > So what is YATP?
> >
> > Yet Another Transport Protocol.
> >
> > > About the overhead of XML, I've yet to see anyone provide
> benchmarks on
> > > modern hardware that proves that encoding and decoding are big
> problems in
> > > everyday applications, but I've heard lots of people say it's too
> expensive.
> > > I tune out people who are willing to have such opinions without
> any data.
> >
> > I'd like to see the data myself.  Has anyone ever seen data like
> > this?
>
> ------------------------------------------------------------------------
> Get great brand name shoes with just the click of a mouse. Check out
> the huge selection at Zappos.com, the Web's Most Popular Store!
> http://click.egroups.com/1/6994/3/_/_/_/963929221/
> ------------------------------------------------------------------------
>
> To unsubscribe from this group, send an email to:
> decentralization-unsubscribe@egroups.com
>
> ------------------------------------------------------------------------
> Old school buds here:
> http://click.egroups.com/1/7081/3/_/_/_/963942067/
> ------------------------------------------------------------------------
>
> To unsubscribe from this group, send an email to:
> decentralization-unsubscribe@egroups.com

Here is an interesting tidbit that I posted on HTP a while back.  It is
basically a technique for setting up a direct point to point connection
between two boxes behind seperated NAT'd networks without going through a
mirror type server:

From http://www.alumni.caltech.edu/~dank/peer-nat.html

"Peer-to-peer protocols that wish to be NAT-friendly must be aware that
any addresses they embed in their data packets may be
invalid once the packets pass through the NAT, and compensate
accordingly. One way to do it is as follows:

All traffic between the peers is done via a single UDP port. There is an
address server which is not behind any NAT. Users connect to
the address server first, and send it the IP address they think they
have; the server notes both that address and the address it sees in
the UDP header. The server then sends both addresses to the other
peers. At this point, everyone knows everyone else's address(es).

To open up peer-to-peer connections, all old peers send a UDP packet to
the new peer, and the new peer sends a UDP packet to
each of the old peers. Since nobody knows at first whether they are behind
the same NAT, the first packet is always sent to both the
public and the private address.

This causes everyone's NAT to open up a bidirectional hole for the UDP
traffic to go through. Once the first reply comes back from
each peer, the sender knows which return address to use, and can stop
sending to both addresses. "

Thanks for the numbers! Excellent. We need an engineering culture in
XML-land. This is going the right direction. Dave


----- Original Message -----
From: "David Orchard" <orchard@...>
To: <decentralization@egroups.com>
Sent: Tuesday, July 18, 2000 10:41 AM
Subject: RE: [decentralization] Re: Fractional Horsepower HTTP Server


> I did similar tests in Oct '99.
>
> Two tests are of interest: transport times and serialization times.
>
> I found that connection-oriented RMI (5 ms) is about 250% faster than
> connection-oriented HTTP 1.1 (12 ms).
> Connection-less RMI is about 80% (12 ms) faster than connection-less HTTP
> (20 ms).
> I did not have a chance to check out HTTP 1.1 pipelining or compression.
>
> I found that DOM serialization took about 50% longer than Java
> serialization.
>
> I suggest that most usage patterns of distributed objects are
> connection-less, where HTTP performs better as a percentage.
>
> Cheers,
> Dave Orchard
> XML Architect
> Jamcracker, Inc.
> 935 Stewart Dr.
> Sunnyvale, CA 94086
> p: 408.830.1886
> f: 408.328.0936
>
> Named to Red Herring's list of 100 Most Important Companies:
> www.redherring.com/mag/issue79/herring100/jamcracker.html
>
> Named to Fortune's list of Cool Companies 2000:
> http://www.fortune.com/fortune/cool/coo.html
>
> -----Original Message-----
> From: Dave Reynolds [mailto:der@...]
> Sent: Tuesday, July 18, 2000 7:07 AM
> To: decentralization@egroups.com
> Subject: [decentralization] Re: Fractional Horsepower HTTP Server
>
>
> I have done some small scale experiments testing performance
> curves of Java implementations of SOAP and SOAP-like protocols
> against Java RMI for a range of payload sizes (10b-100kb) in
> a Lan environment using PII-400 class PC hardware. The performance
> curves against payload size are complex but for middle-of-the-range
> stuff (a few kb) RMI ran around one order of magnitude faster (5ms
> versus 40ms round trip) with substantially reduced latency. Those
> were on a WinNT platform both ends. On Linux (Redhat 6.1) the
> different was much greater (5ms versus 120ms) for reasons I've not
> managed to track down.
>
> This is not science (or even engineering). Stuff like this depends
> on the details of hardware and OS setups, network traffic, maturity
> of the protocol implementations etc. However, at least it is
> marginally better than just handwaving.
>
> Whether these differences in actual round-trip and latency matter in
> a given application will vary a lot. The difference between 10ms and
> 100ms may be huge or irrelevant depending on context.
>
> Dave Reynolds, HP Laboratories, Bristol
>
> --- In decentralization@egroups.com, Lucas Gonze <lucas@g...> wrote:
> > > So what is YATP?
> >
> > Yet Another Transport Protocol.
> >
> > > About the overhead of XML, I've yet to see anyone provide
> benchmarks on
> > > modern hardware that proves that encoding and decoding are big
> problems in
> > > everyday applications, but I've heard lots of people say it's too
> expensive.
> > > I tune out people who are willing to have such opinions without
> any data.
> >
> > I'd like to see the data myself.  Has anyone ever seen data like
> > this?
>
>
> ------------------------------------------------------------------------
> Get great brand name shoes with just the click of a mouse. Check out
> the huge selection at Zappos.com, the Web's Most Popular Store!
> http://click.egroups.com/1/6994/3/_/_/_/963929221/
> ------------------------------------------------------------------------
>
> To unsubscribe from this group, send an email to:
> decentralization-unsubscribe@egroups.com
>
>
>
>
>
> ------------------------------------------------------------------------
> Old school buds here:
> http://click.egroups.com/1/7081/3/_/_/_/963942067/
> ------------------------------------------------------------------------
>
> To unsubscribe from this group, send an email to:
> decentralization-unsubscribe@egroups.com
>
>
>

I did similar tests in Oct '99.

Two tests are of interest: transport times and serialization times.

I found that connection-oriented RMI (5 ms) is about 250% faster than
connection-oriented HTTP 1.1 (12 ms).
Connection-less RMI is about 80% (12 ms) faster than connection-less HTTP
(20 ms).
I did not have a chance to check out HTTP 1.1 pipelining or compression.

I found that DOM serialization took about 50% longer than Java
serialization.

I suggest that most usage patterns of distributed objects are
connection-less, where HTTP performs better as a percentage.

Cheers,
Dave Orchard
XML Architect
Jamcracker, Inc.
935 Stewart Dr.
Sunnyvale, CA 94086
p: 408.830.1886
f: 408.328.0936

Named to Red Herring's list of 100 Most Important Companies:
www.redherring.com/mag/issue79/herring100/jamcracker.html

Named to Fortune's list of Cool Companies 2000:
http://www.fortune.com/fortune/cool/coo.html

-----Original Message-----
From: Dave Reynolds [mailto:der@...]
Sent: Tuesday, July 18, 2000 7:07 AM
To: decentralization@egroups.com
Subject: [decentralization] Re: Fractional Horsepower HTTP Server


I have done some small scale experiments testing performance
curves of Java implementations of SOAP and SOAP-like protocols
against Java RMI for a range of payload sizes (10b-100kb) in
a Lan environment using PII-400 class PC hardware. The performance
curves against payload size are complex but for middle-of-the-range
stuff (a few kb) RMI ran around one order of magnitude faster (5ms
versus 40ms round trip) with substantially reduced latency. Those
were on a WinNT platform both ends. On Linux (Redhat 6.1) the
different was much greater (5ms versus 120ms) for reasons I've not
managed to track down.

This is not science (or even engineering). Stuff like this depends
on the details of hardware and OS setups, network traffic, maturity
of the protocol implementations etc. However, at least it is
marginally better than just handwaving.

Whether these differences in actual round-trip and latency matter in
a given application will vary a lot. The difference between 10ms and
100ms may be huge or irrelevant depending on context.

Dave Reynolds, HP Laboratories, Bristol

--- In decentralization@egroups.com, Lucas Gonze <lucas@g...> wrote:
> > So what is YATP?
>
> Yet Another Transport Protocol.
>
> > About the overhead of XML, I've yet to see anyone provide
benchmarks on
> > modern hardware that proves that encoding and decoding are big
problems in
> > everyday applications, but I've heard lots of people say it's too
expensive.
> > I tune out people who are willing to have such opinions without
any data.
>
> I'd like to see the data myself.  Has anyone ever seen data like
> this?


------------------------------------------------------------------------
Get great brand name shoes with just the click of a mouse. Check out
the huge selection at Zappos.com, the Web's Most Popular Store!
http://click.egroups.com/1/6994/3/_/_/_/963929221/
------------------------------------------------------------------------

To unsubscribe from this group, send an email to:
decentralization-unsubscribe@egroups.com

I have done some small scale experiments testing performance
curves of Java implementations of SOAP and SOAP-like protocols
against Java RMI for a range of payload sizes (10b-100kb) in
a Lan environment using PII-400 class PC hardware. The performance
curves against payload size are complex but for middle-of-the-range
stuff (a few kb) RMI ran around one order of magnitude faster (5ms
versus 40ms round trip) with substantially reduced latency. Those
were on a WinNT platform both ends. On Linux (Redhat 6.1) the
different was much greater (5ms versus 120ms) for reasons I've not
managed to track down.

This is not science (or even engineering). Stuff like this depends
on the details of hardware and OS setups, network traffic, maturity
of the protocol implementations etc. However, at least it is
marginally better than just handwaving.

Whether these differences in actual round-trip and latency matter in
a given application will vary a lot. The difference between 10ms and
100ms may be huge or irrelevant depending on context.

Dave Reynolds, HP Laboratories, Bristol

--- In decentralization@egroups.com, Lucas Gonze <lucas@g...> wrote:
> > So what is YATP?
>
> Yet Another Transport Protocol.
>
> > About the overhead of XML, I've yet to see anyone provide
benchmarks on
> > modern hardware that proves that encoding and decoding are big
problems in
> > everyday applications, but I've heard lots of people say it's too
expensive.
> > I tune out people who are willing to have such opinions without
any data.
>
> I'd like to see the data myself.  Has anyone ever seen data like
> this?

Dave Winer wrote:

> And HTTP is totally the way to go because it's in every scripting and
> programming environment. It's the LCD protocol, and much cleaner than SMTP
> or FTP. It's just simple, I send you something and you send me something
> back. That's basically what computers do.
>

I could not agree more with this statement.  It's also great because you can
take an inexperienced programmer (or an experienced one who knows little about
networking), hand them the http spec, and have them come out with enough
knowledge to understand and build distributed systems around the
request-response paradigm.  Plus, it's trivially debuggable since it's
ascii-text based.  That's power.

> About the overhead of XML, I've yet to see anyone provide benchmarks on
> modern hardware that proves that encoding and decoding are big problems in
> everyday applications, but I've heard lots of people say it's too expensive.
> I tune out people who are willing to have such opinions without any data.

My only problem with XML is people who hail it as a panacea for all
communications ills.  It's great as an encoding format -- we can pre-agree on a
spec for our communications and then XML lets us be sure we're both speaking the
same dialect to one another.  But some folks claim that it will somehow enable
software to semanically understand the meaning behind the tags (my former PhD
advisor now has DARPA funding for this), which ain't gonna happen until we solve
the AI problem :)

-s
--
Steve Dossick
Founder and Chief Architect
iPal
310-578-8331 (voice)
310-578-8336 (fax)

> So what is YATP?

Yet Another Transport Protocol.

> About the overhead of XML, I've yet to see anyone provide benchmarks on
> modern hardware that proves that encoding and decoding are big problems in
> everyday applications, but I've heard lots of people say it's too expensive.
> I tune out people who are willing to have such opinions without any data.

I'd like to see the data myself.  Has anyone ever seen data like
this?

So what is YATP?

And HTTP is totally the way to go because it's in every scripting and
programming environment. It's the LCD protocol, and much cleaner than SMTP
or FTP. It's just simple, I send you something and you send me something
back. That's basically what computers do.

About the overhead of XML, I've yet to see anyone provide benchmarks on
modern hardware that proves that encoding and decoding are big problems in
everyday applications, but I've heard lots of people say it's too expensive.
I tune out people who are willing to have such opinions without any data.

I'm getting a 900 Mhz desktop machine next week btw. It was pretty cheap. I
belong to the church of letting hardware work for me. People used to use the
same argument against scripting, you don't hear that so much these days.

Dave


----- Original Message -----
From: "Lucas Gonze" <lucas@...>
To: <decentralization@egroups.com>
Sent: Monday, July 17, 2000 8:13 PM
Subject: Re: [decentralization] Re: Fractional Horsepower HTTP Server


> Got it - a web server with every client.  You may be interested
> in Magi (http://magi.endeavors.org/), who are implementing it.
>
> I also spent some time working on the fractional horsepower http
> server concept.  I moved away from it for pretty much the reasons
> in Wesley's post:
>
> > My complaint with HTTP is not its textualness but the tight
> > request/response coupling and order (requests can only go in one
> > direction, responses only in the other, and each request must have
exactly
> > one response).
>
> Also, I find HTTP's clunky handling of proxying to be supremely
> annoying, because proxying does so much work in a peer network.
>
> _But_ we all know that HTTP is not going away any time soon.
> Developers seem to be more or less evenly split between the
> fractional horsepower faction and the YATP faction.  I am placing
> bets on both sides, with an http layer in worldOS that uses http
> for wOS messages and also, of course, the wOS YATP.
>
> Dave Winer wrote:
> >
> > Lucas, here's a pointer to a piece I wrote about this:
> >
> > http://davenet.userland.com/1997/09/14/FractionalHorsepowerHTTPSe
> >
> > BTW, I subscribed to this list and pointed to it from my site.
>
> Thanks for the pointer.
>
> ------------------------------------------------------------------------
> Get a NextCard Visa, in 30 seconds!
> 1. Fill in the brief application
> 2. Receive approval decision within 30 seconds
> 3. Get rates as low as 2.9% Intro or 9.9% Fixed APR
> http://click.egroups.com/1/6630/3/_/_/_/963890619/
> ------------------------------------------------------------------------
>
> To unsubscribe from this group, send an email to:
> decentralization-unsubscribe@egroups.com
>
>
>

Got it - a web server with every client.  You may be interested
in Magi (http://magi.endeavors.org/), who are implementing it.

I also spent some time working on the fractional horsepower http
server concept.  I moved away from it for pretty much the reasons
in Wesley's post:

> My complaint with HTTP is not its textualness but the tight
> request/response coupling and order (requests can only go in one
> direction, responses only in the other, and each request must have exactly
> one response).

Also, I find HTTP's clunky handling of proxying to be supremely
annoying, because proxying does so much work in a peer network.

_But_ we all know that HTTP is not going away any time soon.
Developers seem to be more or less evenly split between the
fractional horsepower faction and the YATP faction.  I am placing
bets on both sides, with an http layer in worldOS that uses http
for wOS messages and also, of course, the wOS YATP.

Dave Winer wrote:
>
> Lucas, here's a pointer to a piece I wrote about this:
>
> http://davenet.userland.com/1997/09/14/FractionalHorsepowerHTTPSe
>
> BTW, I subscribed to this list and pointed to it from my site.

Thanks for the pointer.

Lucas, here's a pointer to a piece I wrote about this:

http://davenet.userland.com/1997/09/14/FractionalHorsepowerHTTPSe

BTW, I subscribed to this list and pointed to it from my site.

Dave

On Mon, 17 Jul 2000, Lucas Gonze wrote:

> My feeling is that there will always be a split between
> slow but extensible text protocols and fast but high maintenance
> binary protocols.  I wonder if there is a way to have a single
> protocol with two facades?

It's not clear to me that text protocols are that much slower than binary
protocols, but it depends on the size of messages and the path MTU size
(due to TCP slow start).

My complaint with HTTP is not its textualness but the tight
request/response coupling and order (requests can only go in one
direction, responses only in the other, and each request must have exactly
one response).

Wesley Felter - wesf@... - http://www.cs.utexas.edu/users/wesf/

I'm interested to hear more about the Fractional Horsepower HTTP
Server.  It is a good idea to talk about using HTTP as a
substrate for decentralized apps.  There are good reasons for and
against.  My feeling is that there will always be a split between
slow but extensible text protocols and fast but high maintenance
binary protocols.  I wonder if there is a way to have a single
protocol with two facades?

> These apparently
> separate worlds seem not to know much about the others

Strange but true.  Maybe things will pull together over the next
year or so.

I have cc'd this to the list at decentralization@egroups.com

Dave Winer wrote:
>
> UserLand is very active in this area, and has a lot of ideas, and software.
> Our involvement in SOAP is centered on the Fractional Horsepower HTTP Server
> idea, which is the same thing as SOAP, P2P and Napster. These apparently
> separate worlds seem not to know much about the others, but it's the same
> story. Count us in. Dave
>
> ----- Original Message -----
> From: "Lucas Gonze" <lucas@...>
> To: <xml-dist-app@...>
> Sent: Monday, July 17, 2000 12:16 PM
> Subject: RFC for a group to discuss decentralization
>
> > I wonder if there is a consensus to form a group or mailing list
> > specifically for discussing decentralization (a.k.a P2P).  I see
> > this as a followup to the issues raised at Twist 2000.
> >
> > For example:
> > * Is decentralization ever a good idea?  If so, when?  Is there
> > non-anecdotal evidence of costs and benefits?
> > * What protocol issues are there?  Can we begin assembling a good
> > protocol for decentralized messaging?  To what degree do the
> > protocols for Freenet, Gnutella or WorldOS meet the need (and how
> > do they fail)?  Do we need an application protocol or something
> > lower level?  Can HTTP do the job?  Can we implement peer routing
> > as an add-on to existing protocols?  Is there a call to develop
> > an IETF working group?
> > * Given that authoring and versioning are critical but hard in a
> > decentralized environment, how can we approach the job?  Is it
> > possible to integrate WebDAV with peer networking?
> > * What are the business issues?  Who are the players?  Who else
> > stands to win or lose, and why?
> >
> > At present many people and groups are working on the issues in
> > isolation, some for competitive reasons and others for lack of an
> > alternative.  My belief is that a communal approach will be more
> > productive.
> >
> > In hope that there will be interest, I have created a list
> > accessible at http://www.egroups.com/group/decentralization.
> >
> > - Lucas Gonze
> >

I'm here. Hacker-pundit that I am, I wrote a bit about technical P2P
issues today:

http://wmf.editthispage.com/2000/07/17

Wesley Felter - wesf@... - http://www.cs.utexas.edu/users/wesf/

So who's here?

I just had a piece in the Times about Napster, and while it doesn't
address decentralization per se, it does address one of its effects in
the music biz, so I offer it here as a possible ice breaker:

http://www.nytimes.com/00/07/15/oped/15shir.html

--
Clay Shirky                 |    shirky.com - Essays on the Internet:
http://www.shirky.com/      |        Culture, Economics, Globalization