Resourceful Computing

Thoughts about using Resource Oriented Computing to build simpler software

Kotlin and NetKernel

with 2 comments

Kotlin is a new programming language from JetBrains, the publishers of IntelliJ and other development tools. Their goal is to make Kotlin an industrial strength replacement for Java.

I am intrigued because Kotlin has type safety, modern language features, very good IDE support, and can interoperate easily with existing Java programs and libraries.

To facilitate experimentation, I developed a NetKernel module urn:org:netkernelroc:lang:kotlin and made it available on the NetKernelROC community Apposite repository. The current release (version 0.1.0 on June 14, 2013) includes Kotlin M5.2 (the 0.5.748 build). This is a Kotlin milestone release. While I have found the NetKernel code I’ve written with it to run just fine, you should probably consider this experimental. I will update the NetKernel module as Kotlin releases are published. Once you connect your NetKernel instance with the NetKernelROC Apposite repository you should be automatically notified of the updates.

The NetKernel Kotlin module is hosted on GitHub at https://github.com/netkernelroc/urn.org.netkernelroc.lang.kotlin Currently the module contains Kotlin JAR files with language runtime support. My goal is to also include Kotlin code (in the form of classes and extension functions) to extend internal NetKernel code such as the NKF API and HDS to make programming easier than when using Java. If you are interested in working at this level, then you should fork and clone a copy of this module from GitHub and propose updates. You will find that I have written two Kotlin based HDS builders. The first is a Kotlin replacement for HDSBuilder (facilitated by the IntelliJ “convert Java code to Kotlin” service). The second simply wraps the NetKernel HDSBuilder and delegates to it. I wrote both because I’m not certain which will be the better approach.

My first experimentation with a NetKernel module that uses Kotlin can be found in the NetKernel module urn:org:netkernelroc:util:file, also hosted on GitHub, at https://github.com/netkernelroc/urn.org.netkernelroc.util.file . This module, also in a very early release, contains the service active:fileList which returns a list of files in a specified directory. It uses the Kotlin replacement for HDSBuilder. If you look at the module urn:org:netkernelroc:util:file:test you will find an experimental Kotlin DSL to facilitate the creation of XUnit testlist.xml resources.

My hope is that Kotlin will turn out to be a good language to standardize on for NetKernel system development. I experimented with Groovy, but found it frustrating (that’s another discussion). I wrote a Scala language module for NetKernel a while ago, but Scala programs seem too opaque.

I released this module on GitHub and through the NetKernelROC community to make it easy for anyone in the community to join me in exploring Kotlin.

Written by Randolph Kahle

June 13, 2013 at 2:41 pm

Posted in API, NetKernel

NetKernel request execution and threads

leave a comment »

One aspect of NetKernel that confuses developers is the relationship between a request and a thread.

A request is a demand placed on a software system. A request could be triggered by a user interacting with the system, a Cron timer reaching a particular time, or another system in need of information. At the completion of request processing either the state of the system has changed, an information representation is returned, a request is made of another system, or some combination of these occurs.

A thread is an abstract execution context which is implemented by a real instruction counter and registers in a physical CPU core. The Java thread abstraction is simple and relatively straight forward to use. A program (or container) can set up a pool of threads which can be executed on one, two, or more physical CPU cores. Because a thread is an abstraction, the physical level execution can employ highly optimized techniques which remain invisible to the developer.

The thread abstraction has worked really well. As we have progressed from single core computers to two and four cores, developers have used a model of “processing a request on a thread” to build reasonably performant systems. Complexities arise, however, when writing code running on multiple threads that mutates state. Doug Lea’s early work and the later incorporation of this thinking into the Java concurrency libraries has made multi-threaded development and state mutation tenable.

For most developers, this is the “state of the art”. Some have used more advanced patterns to achieve very high concurrency and performance. There are also libraries, frameworks, etc. that support these more sophisticated patterns, such as Node.js. Also of note is Rich Hickey’s brilliant work with the data structures in Clojure.

NetKernel provides a much simpler abstraction.

In NetKernel the concept of a request and thread are completely separate. There is no notion of “processing a request on a thread”. Instead, a request (implemented as a Java object, which, with other associated objects, contains the complete processing context) is issued and the client waits for the requested information to be returned. While a thread actually does the processing of this request, that abstraction is hidden, just as the optimized physical execution of a thread is hidden from a Java developer. The following code show how a request is created:

INKFRequest request = context.createRequest("res:/helloworld.txt");

and this code tells the kernel that it should process the request:

context.issueRequest(request);

Notice the method name – issueRequest(). This is similar to typing in a URL into a browser and requesting information. In both cases there is no notion of a thread – this is just a request for information (in our case for the resource “res:/helloworld.txt”). Of course these two methods calls must execute on a thread – it just doesn’t matter which thread. Requests processing and threads are completely orthogonal.

Compare this with traditional Java multi-thread coding. To create a new thread of execution, several things must happen: an instance of a Java object that implements the Runnable interface is created, the runnable is provided to a new Thread, and the thread is instructed to start:

Runnable runnable = new MyRunnableClass();
Thread runner = new Thread(runnable);
runner.start();

In this model a thread is a visible abstraction – the exact opposite with NetKernel. In fact, threads and thread synchronization are primary and very necessary concerns for a developer.

What does this mean?

The implications of using the NetKernel Resource Oriented Computing abstraction are interesting to consider, especially as it relates to the emergence of 8-core, 16-core, 64-core, etc. computers and the importance of distributed computing. If a developer builds a system that uses ROC requests then the number and location of the threads that execute the work are not even visible. They are decoupled from the request processing code. This means that the size of worker thread pools, the number of cores, and whether they are local or remote, are not coding concerns – they are architectural and deployment / configuration decisions. And all of these decisions can be changed for an already built application independent of the code.

Written by Randolph Kahle

September 18, 2012 at 5:44 am

What language should we use to talk about ROC?

with one comment

Humans thrive on communication and we have invented many different forms. For example – music, dance, written communication which includes novels, poetry, short stories, essays, blogs, etc., as well as visual forms such as paintings. For particular domains, such as mathematics, we have developed symbolic languages that allows us to succinctly express our ideas. In the field of information systems we have different ways of expressing ourselves from the highest altitude view to executable code.

Resource Oriented Computing is an approach to conceive and implement information systems. It is natural to start with a familiar language to discuss the abstraction and then extend the language when required. In the eight years I have been working with Resource Oriented Computing I have wrestled with this language. It is frustrating because the concepts behind ROC are very simple and yet, we do not have an agreed upon set of words or notational system. I now think that the ROC community needs a new language.

I realize that the language we use helps us articulate out thoughts and shapes our thinking by facilitating and constraining. A language facilitates thought by making available words or symbols that have an agreed upon meaning and which allow us to easily note what we want to communicate. If that language lacks words or symbols for a particular new thought, then it constrains us and it forces us to either extend the language or take more time to build a line of reasoning.

Which languages have we tried? I have used pictures with address spaces (that look like clouds), endpoints (that look like rounded rectangles), and requests (that look like arcs). These symbols were first designed by Peter Rodgers for the NetKernel documentation. In conversations with Peter he has shown me a more mathematical and symbolic way to describe ROC. Tom Hicks has shared his ideas about a Lisp-like notation. I have experimented with a DSL to extend the Groovy programming language. The NetKernel explorer tool uses a graphical rendering. The NetKernel nCode composition tool presents a graphical view of endpoints and relationships but ignores address spaces. And finally, to create a running NetKernel system, one uses a text editor to create an XML document.

My final complaint about our current state is that no language supports a discussion about information flow through a NetKernel system. All of the languages are about the composition of spaces, endpoints, and request / response processing. What actually happens in a NetKernel system is that information flows through the ROC architecture. Isn’t that the whole point of an information system? If so, why is that not the primary focus of a language?

I leave this as a challenge to the NetKernel / ROC community. Let us create a new language for ROC.

Written by Randolph Kahle

September 1, 2012 at 7:29 am

Posted in ROC, Thoughts

1060 Research Weekly Newsletter

leave a comment »

Peter Rodgers, CEO of 1060 Research, publishes a weekly newsletter that includes interesting comments about Resource Oriented Computing. This page contains an index to these newsletters, a brief summary and, if I have thoughts about the newsletter, a link to a page with my comments.

Date Summary Volume Issue
March 11, 2011 Sometimes Peter Rodgers hits a home run with his newsletter. This week he covered several topics that I found really interesting. First, he announced a new SSH Client Library module which I first worked on for a client in a consulting engagement. Next he talks about the new TransientFileRepresentation which leads to a very simple implementation of support for the SSH transfer of arbitrarily large files. Finally, he provides an excellent discussion about how to think about the ROC architecture of a system and divide the work into teams. 2 20
March 4, 2011 Transactional overlays – introducing the new SQL Transactional overlay and a general discussion about implementing your own. 2 19
February 25, 2011 More about Shakespeare and infinite monkeys. 2 18
February 18, 2011 Infinite monkeys. 2 17
February 11, 2011 Negotiated HTTP Cache Consistency. 2 16
February 4, 2011 Several interesting topics – Asynchronous transports, dealing with corporate firewalls and Apposite, IP6 support and the stability of ROC solutions. 2 15
January 28, 2011 Advanced HTTP Handlers, Asynchronous HTTP Handler, SOAP client enhancements, HTTP Client review, XML Serializer and ROC resource and solution stability. 2 14
January 21, 2011 Brian Sletten’s article about why he uses NetKernel and an update on NetKernel performance testing. 2 13
January 14, 2011 A discussion about the NetKernel Portal Framework. 2 12
January 7, 2011 A discussion about Transport scope confirming that the space in which a transport is declared is the top of the super-stack. It was motivated by a problem ticket describing a transport, declared within a mapper-wrapped space, unable to request its configuration resource. 2 11
December 24, 2010 A year-end summary issue that looks back over the last series of articles and looks forward to the future of ROC. Peter uses a metaphor of bridge building technology to explain the difference between traditional framework and code-centric thinking and an ROC way of thinking for building information systems. 2 10
December 17, 2010 The fifth article about ROC and languages looks at functions and set processing. 2 9
December 10, 2010 The fourth article about ROC and languages examines what code and functions are all about. 2 8
December  3, 2010 The third article about ROC and languages focuses on arguments – Pass-by-Value, Pass-by-Reference and Pass-by-Request. 2 7
November 26, 2010 HTML 5 introduces Web Sockets. While they are not RESTful, they do work well with the Resource Oriented Abstraction supported by NetKernel, as demonstrated by downloadable demos. 2 6
November 19, 2010 The second article on ROC and languages looks at imports and extrinsic functions. 2 5
November 12, 2010 The first article on ROC and languages. 2 4
November 5, 2010 This week there is no commentary on ROC. 2 3
October 29, 2010 An exploration of state balancing and ROC level request routing. 2 2
October 22, 2010 On State and consistency. Golden Thread pattern 2 1

Written by Randolph Kahle

December 20, 2010 at 5:00 pm

Posted in NetKernel, ROC

Monitoring configuration changes

leave a comment »

In NetKernel transport endpoints can monitor their configuration information enabling a user to dynamically change their settings while an application is running.

I will show an implementation pattern you can use within a transport that will provide this dynamic behavior. The key is understanding that a transport can request its configuration information as a resource request, retain a reference to the response and later check to see if that response has expired. The response will report that it has expired when the configuration information represented by the resource has been changed.

Since I am writing a transport for the new Twitter Stream API, I will use this as the example to illustrate the pattern. The Twitter Stream API is relatively new and allows you to set up a filtered stream of status messages (formatted in JSON or XML) that are sent directly from the Twitter servers. An example configuration change is the addition or removal of a Twitter ID from those being filtered or monitored. By implementing this pattern my transport will detect when the file containing the configuration changes and will cause the transport to reconnect to Twitter with the new filter information.

For the Twitter stream transport, the configuration is available from the resource identified as res:/etc/system/TwitterStreamConfig.xml. The transport can request the configuration information represented as an HDS node with the following code:

String configURI = "res:/etc/system/TwitterStreamConfig.xml";
IHDSNode configuration = context.source(configURI, IHDSNode.class);

However, we want to maintain a reference to the response returned by the request and therefore use this code:

IHDSNode configuration;
INKFResponseReadOnly configurationResponse;
String configURI = "res:/etc/system/TwitterStreamConfig.xml";
configurationResponse = context.sourceForResponse(configURI, IHDSNode.class);
configuration = (IHDSNode)configurationResponse.getRepresentation();

Later, we can check to see if the response associated with the endpoint configuration is still valid with a call to isExpired():

if (configurationResponse.isExpired())
  {
  // Re-request the configuration information
  }

The easiest way to set up periodic monitoring of the configuration response is to create a monitor thread, which can be done in the postCommission(...) method. The corresponding
preDecommission(...) method would contain the code to interrupt and hence terminate the monitor thread.

In the following code I show the portion portion of the TwitterStreamTransport class which provides the dynamic reconfigurability of the endpoint. (The part of the transport that deals with Twitter has not been included as that would obscure the illustration of this pattern.)

package net.databliss.netkernel.tpt.twitter;

import org.netkernel.layer0.nkf.INKFRequestContext;
import org.netkernel.layer0.nkf.INKFResponseReadOnly;
import org.netkernel.layer0.representation.IHDSNode;
import org.netkernel.module.standard.endpoint.StandardTransportImpl;

public class TwitterStreamTransport extends StandardTransportImpl
  {
  private static final int POLL_INTERVAL = 10000; // ms

  // Thread monitoring changes in configuration resources
  private Thread monitorThread;

  // The configurationResponse is the link to the cached configuration
  // When the response expires then we must re-request the configuration
  // using its identifier to get the updated information.
  private INKFResponseReadOnly configurationResponse;

  private volatile boolean pollConfiguration = true;

  //==================== Override Superclass Methods ==================

  @Override
  protected void postCommission(INKFRequestContext inkfRequestContext) throws Exception
    {
    // Start the configuration monitoring thread
    monitorThread = new ConfigurationPoll();
    monitorThread.start();
    }

  @Override
  protected void preDecommission(INKFRequestContext inkfRequestContext) throws Exception
    {
    // Stop the configuration monitor thread by interrupting it
    pollConfiguration = false;
    monitorThread.interrupt();
    }

  //==================== Private Implementation Methods ==================

  private void checkConfig() throws Exception
    {
    if (configurationResponse == null || configurationResponse.isExpired())
      {
      // If the configuration has never been requested or if it has expired
      // then we need to request it using the identifier is supplied by the
      // config parameter.
      INKFRequestContext context = this.getTransportContext();
      configurationResponse = context.sourceForResponse("param:config", IHDSNode.class);
      configure((IHDSNode) configurationResponse.getRepresentation());
      }
    }

  private void configure(IHDSNode configuration)
    {
    // Code to configure the transport is included here.
    }

  private class ConfigurationPoll extends Thread
    {
    public void run()
      {
      while (pollConfiguration)
        {
        try
          {
          checkConfig();
          }
        catch (Exception e)
          {
          e.printStackTrace();
          }
        try
          {
          sleep(POLL_INTERVAL);
          }
        catch (InterruptedException e)
          {  //Do nothing
          }
        }
      }
    }
  }

If the response indicates that the configuration has expired, it does not mean that the configuration has new information. For example, a user could edit the file and add some space characters, which would invalidate the response but would not change the configuration. The private method configure(...) should include a test to see if the configuration information is new before making changes to the state of the transport.

Written by Randolph Kahle

October 20, 2010 at 12:07 pm

ROC – A Tale of Two Phases

with 3 comments

In the resource oriented computing (ROC) abstraction, resource requests are processed in two phases – the Resolution and Evaluation phase. In the resolution phase a search is conducted to find an endpoint that will resolve the request. During the evaluation phase, the endpoint found in the resolution phase is asked to perform whatever computation it requires to create and return a representation of the requested information.

Each request triggers these two phases, which means that a subsequent request for a particular information identifier could resolve to a different endpoint. This is a critical feature of ROC that we will argue later provides architectural flexibility.

The following diagram illustrates the resolution phase. The endpoint acting as a requestor issues a request into its containing space and a  search is conducted to find the first endpoint that resolves the request. In the diagram, the search reaches the Import endpoint which forwards the search into a related space. Within that space we see an endpoint finally resolving the request, which ends the search.

ROC Request Resolution (logical)

The next diagram illustrates the start of the evaluation phase. From the perspective of the endpoint acting as the requestor, a temporary relationship is established with the resolved endpoint and the request is sent for evaluation.

ROC Request Evaluation (logical)

When the endpoint creates a representation, it returns it to the requestor and the relationship between the requestor and the endpoint is forgotten.

ROC Request Evaluation (logical)

The cycle of searching for an endpoint followed by request evalution and then amnesia repeats for each request. This is fundamentally different from what most of us are used to. Most software systems work hard to optimize run time efficiency by remembering relationships instead of forgetting about them. Traditional compiled languages such as C, C++, Pascal, etc. generate compiled object libraries that are linked together by a linker. In Java, a Classloader brings class definitions into memory and created objects are linked through method address resolution tables when methods are first called. If we want to introduce dynamic relationships in these systems, we have to work at it and introduce points of indirection. Some languages make this easier, such as Groovy and its Meta Object Protocol (MOP); however, what is different with ROC is that the separation of a requestor and a service endpoint is fundamental.

The implication is that with ROC, we can create software that is reconfigurable at run time. At any level of an application architecture. In fact, all facets and any level of a system can be reconfigured dynamically. In the next post we will look at an implementation of ROC and explore why its performance is not all that bad while future posts will examine how we can harness this flexibility to our advantage.

Written by Randolph Kahle

April 1, 2010 at 4:03 am

Posted in Thoughts

Tagged with

Resource Oriented Computing

leave a comment »

This blog explores the use of Resource Oriented Computing to build simpler software. So, what is ROC? This post takes a preliminary, high-level look.

The fundamental abstraction is illustrated in the following diagram.

Resource Oriented Computing Abstraction

In ROC, requests are made for information by a Requestor by issuing a request into a Space. Information is modeled as Resources and each resource has one or more Identifiers. The diagram shows a resource request for a member of a Resource Set being resolved to an Endpoint. When the endpoint evaluates the request it returns an immutable copy of the information in the form of a Representation.

This diagram illustrates how the World Wide Web works! A browser (requestor) issues a request for information and identifies that information with a URI (the URL of the resource). The domain name service (DNS) resolves the identifier (within a global space) to a web server (endpoint) and then the browser issues a request directly to the endpoint for evaluation. The web server (endpoint) evaluates the request and returns a copy of the current information in the form of a web page (representation).

The ROC abstraction becomes particularly interesting when implemented as a very small software platform to support software applications development. The resulting software takes on characteristics of the Web, such as scaling, flexibility and malleability.

In the next several posts we will explore many of the facets of ROC and see how they impact the quality and simplicity of software.

Written by Randolph Kahle

March 29, 2010 at 12:01 pm

Posted in Thoughts

Tagged with