One of the exciting changes in the RETS community has been the formation of the “Research and Development Workgroup” (R&D WG), which solicits and review submitted business cases from the real estate community and identifies how RESO can contribute to the benefit of that business process. The submitted cases are reviewed with representatives from the impacted stakeholder groups to document specific goals and objectives. The resultant business case opportunity is then submitted to the RESO board of directors for approval and assignment. The benefit of this workgroup approach is that new technologies are not explored just for the sake of exploration, but to solve real business problems.
So, when Mike Seguin from the Contra Costa Association spoke at a session at NAR and suggested a new RETS Transport – bypassing the business evaluation process and jumping right to “cool new tech”, I was a bit annoyed that he had bypassed the process and the community. But I have huge respect for Mike, so I figured I’d best give him – and his ideas – a chance, get some more detail, and try to pull these new ideas into the RETS standards process.
The core idea here is that instead of RETS servers being queried by client applications to figure out what has changed, the server would use a well-established standard called AMQP (amqp.org) in order to push changes out to the clients who are subscribed to that ‘channel’. An implementation of the AMQP protocol is called a ‘message broker’. It is an intelligent, highly flexible tool to get data where it needs to go.
The business problem such an implementation solves is that, at the current time, a RETS server is queried by (in Mike’s market) about 500 client applications at multiple frequencies from 5 minutes to several hours each – even if nothing has changed in the MLS. Two immediate issues jump out, the unnecessary queries, and the unnecessay delay.
The unnecessary queries cause spikes in usage that is wasteful. Ultimately it means the vendor has to provide a RETS server with more horsepower than it might if things were more efficient. And, because some RETS clients do not query efficiently, this can potentially have an impact on performance for other RETS application (everyone else). What also happens is that sometimes RETS clients (for whatever reason) miss something esoteric that has changed and then they get out of synch with the server. So, it seems like a “push” solution would solve these problems.
Simply put, the MLS implements ‘messaging broker’ software that ‘listens’ for changes in the MLS servers data (new listing, price change, etc),and takes action when it happens. This is certainly nothing new, there are many technical solutions to finding when data has changed, and acting on that information. What is new is the flexibility and elegance of the logic based messaging ‘channels’ built into the messaging broker software. The same technology, without a ‘stack’ of other tools, is able to do many things, from detecting the update, to passing it to the message brokers exchange where logic gets applied and routing assigned, to the message queue, which securely and efficiently provides reliability for the the delivery of updates.
Not only is the negotiation of information delivery secure – since the server sends data to pre-approved (whitelisted) clients subscribers only, and SSL or TLS could be mandated for authentication and transmission – but the delivery can be highly reliable if properly implemented. The AMQP transport has solutions built into it to deal with cases where a RETS client is unavailable at the time of the push from the server. The server can track when items that have been queued have not been received.
So, AMQP also has potential to reduce the bandwidth needs of RETS and reduce the load on the RETS server. It does this by separating out the RETS function from the queuing function. So, when something changes on the RETS server, the RETS server just needs to create the update information once and sends it to the queuing (AMQP) server – and then the responsibility for dealing with all of the clients is on the queuing server’s plate – rather than the RETS server’s. The way it saves bandwidth is by providing a more reliable update, compared with typical RETS implementations, where typically more than 5% of queries fail in some way and then require re-querying. In Mike’s implementation, he was able to reduce bandwidth (he says he will provide me hard numbers in the next week or so) over the traditional transport while providing more timely updates (practically real time).
Which brings us to the other immediate problem AMQP solves; timeliness of informtion updates. With more and more by consumers for internet based listing information, it stands to reason that timely updates are increasingly important. Certainly an hour is not long to wait. But what if that hourly process fails, and doesn’t repeat for an hour? Or what if the consumer of the information isn’t ultimately a consumer, but itself an application generating real time price or invenory information on a large scale? Faster seems better. By reducing the payload (a single price change can be multicast immediately to all ‘subscribers’ to that channel) to very small packets of data, we can realize nearly real time updates from entry to display on the end users side. By analogy, this sort of multicast real time push is like arriving in hawaii after a long flight, and wanting to let everyone you know that you are there safely. You could make 500 individual phone calls, or you could tweet it to all of them just once.
Finally, since AMQP was built as a wire level protocol to create genuine interoperability for any platform, it is a great candidate for adoption as the transport basis for future RETS implementations. Implementations of Java based message brokers will be just as efficient and robust as an open source implementation of RabbitMQ using erlang (CCAR’s model), or other language implementations. Just like http works for any browser, AMQP works for any server.
Certainly there are pros and cons with everything. After having gone through the implementation experience and adapting all of his various systems incrementally, Mike tells me that the learning curve is the one con that jumps out at him. Building the message broker environment was actually quite rapid, over a two week span, but there was a good amount of prep work before that. Like any new technology, it starts slow, and then at some point you pass a critical point and realize the return on the technology itself. Mike tells me that he is currently maintaining many times more data and services, but at only a very modest increase in expense. And that ratio continues to improve. Modeling new message channels gets easier and less expensive as you go, and the results in turn more robust.
So, this transport should work great – improving reliability, decreasing bandwidth, and providing more timely updates – to implementations where a parallel database is being created. This does NOT mean that the current “pull” transport is obsolete – there are plenty of cases where it is not – for instance, an application hooked directly to a RETS server where every query is different – like an agent querying from a mobile phone or WyldFyre. Technologies like this should be adopted opportunistically, when there is already a major revisioning in process. Since there is a lot happening in our industry right now, from national parcel aggregation to mergers and statewide MLSs, Mike thought this was a good time to bring this forward. What he fnds fascinating is the potential to have this one process adapting to changes in data in his CRM, Compliance tool, MLS Mirror, Membership system, Member website, and then all of the end user messaging that processes as a result of that. The potential to provide business intelligence is evident.
But it all starts with MLS data acquired via the RETS standard, and Mike believes that AMQP offers a potential strong option to handle the transport layer.
So, let’s consider adding this new method to RETS.
How can this be done? I’ll work with Mike to fill out a business case worksheet and work to push it through the RETS R&D workgroup – hopefully people will be excited by the business benefit of adding a transport to RETS. Since it will not replace, but only be in addition to the current transport, there should be no impact on current systems. Vendors for server platforms, and then for client platforms, would then be able to add this to their system if they wished to reap the benefits.
Kudos, Mike – I look forward to moving this into the RETS standards process. It deserves attention and discussion within the community.