[brycemason]

Debated posting this, but so many ramifications I felt it best to do so. Original post in my forum. Reported here for broad applicability. The question has been why my systems (and I imagine those of many) have been less capable of doing business since the latest LC change. I detail the change and consequences below.

The issue is that P2P-Picks relies on LC providing its loan application data via the CSV or API, and that these are no longer the most timely way to get loan application data. It is affecting me personally and all of my clients, too, and there isn't much I can do about it except complain to LC, which I have done.

It used to be that when you clicked the link to get the CSV, LC would instruct its systems to generate a fresh CSV based on the loan applications that were available at that very moment. This, I imagine, was expensive for them when thousands of people were clicking at the magic hours. Hence why we saw rate limiters in place last November. In the last few days, they moved to a model of linking to a static CSV available for download that is updated for freshness just once every minute. While this relieves the stress off their servers, it has had some unintended (or perhaps intended) consequences.

Here is the rub. LC releases the loans onto their platform around 30-40 second after the hour. The loans are available to website users a full 20-30 seconds before they ever even appear in the CSV. The API has always been out of date a number of seconds, just like the CSV is now. Now, the absolute fastest way to get loan application data is to use the website interactively via their filter processes (or to screen scrape for automation). This puts all of us who want to do active loan selection in a position of either sitting at our terminals every release twitching like some kid playing a video game or breaking the rules (even more than we already have to do) and resorting to screen scraping loan data off the website in order to automate things. Of course, the web pages don't even have a majority of the credit variables anyway, so this may be of limited value.

I have witnessed this occur on both the fractional and whole loan side. Reliably, at 30 after the hour, new loans will appear. Within a few seconds, the "good stuff" (to the extent you wish to believe in such a concept) is either purchased outright (in the whole loan market) or is mostly already full (on the fractional side) by the time the other two loan data retrieval methods are updated. Here is an example from 6pm a couple nights ago to illustrate:


5:55:00 - Initial count: 31 whole loans on platform
6:00:05 - Unused whole loans expire (12 hour rule): 19 whole loans on the platform
6:00:30 - 170 new whole loans added to platform (as viewed from the webpage) - total inventory = 189
6:00:45 - 24 whole loans disappear off the platform - total inventory = 165
6:01:00 - browseNotes.csv and LoansBrowseLoans update with just 165 whole loans in them

Why would LC do this? I don't want to go into pet theories, even though I have some, before allowing LC a chance to respond. And to be clear LC is absolutely still providing an even playing field. Same rules for everyone. It's just that this change has benefitted a certain type of user (the twitching kid or the scraper).

My recommendation is to synch the timing of all forms of loan application data, website, CSV, and API. If the Royal Bank of Canada wrote Thor to synchronize orders across multiple stock exchanges, I'm pretty sure this is a feasible request.

[Ran]

What happened to LC's own autoinvest? Does it get loans as fast as screen scrapping?

[TravelingPennies]

That is a very important question. All third party automated investment systems designed for retail investors that I know of rely on the now-outdated CSV and API for loan input data. Either there are a lot of twitch buyers sitting on the site, or LC's own Prime service is getting first crack at automated investing. Those are the only two options for explaining why so many fractional loans are largely filled by the time of the CSV/API update. If it's the latter, it could be argued that their internal systems have an unfair advantage, as they wouldn't have to resort to screen scraping to organize the data (which takes time to poll individual web pages).

[sociallender]

This is troublesome. I don't understand the need for an API if the information is stale.  I can understand limiting the CSV file and metering to deter screen scraping.  However, an API should provide the same level of service as their front end.  This may have the opposite affect of now requiring auto invest users to use their front end (screen scrape) to place orders.

PLS does have the feature of using the web site front end to place orders based on previously saved LC filters.  However, there is no way to apply sophisticated logic such as AI, ratios,etc..  In other words, we can still place orders faster than those who are doing it the manual way.  However, we won't have the ability to analyze the loan data until the API or CSV is updated.  IMHO, the reason for the API is the ability to automate the loan selection process.  I don't understand why they would give the web site an unfair advantage?  I see this as a migration from the API to the web site which may be an unanticipated consequence. 

Bryce, in your analysis, is the API updated at the same time as the CSV?  Are there any advantages with one over the other? 

 

[TravelingPennies]

The API appears to update simultaneously with the CSV, as least when it matters at 1 minute after the hour. Historically the API was a little stale in my experience. This is all just my limited experience. I'm not going about this in a super scientific way, but have observed a number of times.

[lascott]

Timing seems to matter!  I was quite happy with P2P-Picks and BV the way it was working!  I suspect LendingRobot is affected as well.  We have another account using it but it is pretty selective so not sure if the note investing have slowed down yet. They seem to be.

Title: Mad Rush at Lending Club Loan Release Time: Part I
Posted by Anil Gupta | 08 Jun 2014 22:11:48 | Category: Lending Club
https://www.peercube.com/blog/post/21

[TravelingPennies]

Quote

[TravelingPennies]

I encourage others to verify my own observations.

[TravelingPennies]

Another issue is when to download the CSV or API data?  Prior, the note count could be used to determine when to download the CSV/API loan data.  As soon as the note count increased by a significant amount, you could download the loan data in one shot.  Now, I can't think of a way to determine when to download the API/CSV.  There is no note count in the API (which is absolutely ridiculous).  The only thing I can think of is to continue to monitor the note count then continuously poll the CSV/API for an update?  If that is the case, then we are back to the main problem of the expense/load to handle the poll traffic.  That is, we have to resort to a continuous download of the CSV/API until it catches up with the note count.  Any ideas on how to make this a smoother process? 

[GS]

I can't even begin to imagine why the website, API, and CSV can't be updated simultaneously ...

LC has set up system that encourages screen scrapping and exploiting the shopping cart lock out while the crafty users wait for the CSV to update.  I mean, if I wanted the best loans, and was willing to break LC's rules and possibly the law (is willfully delaying an open market order, and preventing others from placing an order, while one waits for updated information to be posted, illegal market manipulation? -- well, we know people are doing it already), I would design my automation to screen scrape my prospects from the website based on the limited info posted there, put them in the shopping cart, then wait for the expanded CSV to be posted before I select which ones to keep....

This can't be the vision that LC has in mind for the future of Automation ....

[Lovinglifestyle]

I use the shopping cart for recent listings of filters-applied EFG loans because I have more uninvested cash than the new listings will consume.  If LC had a way to sort KEEP from DISCARD into two piles I could get through the list faster.  I don't waste time "waiting" for a .csv to update.  I  go straight to work on what's there and check the 3rd sites when I think they've had time to publish.  Waiting instead of loading the cart would mean losing all the lower amount, 36 month loans without even being able to see them in order to make a decision! HOW ON EARTH IS THIS CHEATING??  It seems like common sense to me.

If LC's filters had the same filter options the 3rd party sites do there would be a chance I could use their automation.

A point of interest to me in all this recent discussion about the timing of available loans is that Interest Radar is very, very fast to have them all.  Rev must have a secret sauce.  Unfortunately for me, I'm addicted to an additional data point only available on a different service, but it too is ready while I'm still paring down my list.

[TravelingPennies]

I don't know if that last post was directed at me, or not (I never called anything "hoarding" or "cheating", nor did anyone else), but I completely agree that as long as LC is going to tilt the field to encourage screen scrapping and use of the shopping cart to retain notes while you think about whether or not you really want to buy it, well, the crafty investors will do just that.  In fact, I posted on the PLS section (http://www.lendacademy.com/forum/index.php?topic=1907.0) several months ago that I'd love to see PLS modified to do a quick LC filter grab of notes, then exploit the shopping cart lock out while it runs it's models.

As you pointed out, it works best if you have lots of cash to invest, and grab all or nearly all the new notes, which is why I think this has probably become the choice tactic of the institutional investors. 

SO, I'm not being judgmental, I'm right there with you, that it is the best way to get the best notes.  When I used to "hand invest", I'd do the same thing.  Refresh like a mo-fo until the new loans appeared, quick filter, sort, click-click-click the top notes with the highest yields, add to cart .... then start looking at the quality of what I selected.  However, that strategy stopped working about 6-7 months ago when I found I could not do it by hand fast enough to get the top notes in my filter, which led me to believe the game had changed and the computers were now doing what I was doing, but doing it much faster. 


But, while I do admit to doing this myself, I've always wondered if the tactic was legal, in that I was preventing others from participating in a fast paced market while I took my sweet time to decide what I wanted to keep, and what I wanted to throw back.  Can you imagine if the SEC discovered that some hedgefund had discovered a way to prevent other's from trading a stock while they ran models to see if they wanted to buy it or not .... I'd imagine they would not be amused.  That's a much closer analogy to consider than taking two pairs of pants into the dressing room. 

[mchu168]

I see people locking up loans from high income borrowers right at feeding time, but then release them after further review.  There's definitely a lot of gaming the system going on to get some perceived "edge."  Someone needs to write a version of "Flash Boys" as it pertains to the Lending Club world.

[TravelingPennies]

It's not cheating to use the website to buy a bunch of loans right at release. Nobody said it was. To the contrary my original post said the playing field is indeed even. Everyone gets the same info at the same time. We are just complaining that the supported official ways to get loan applications are unequal. LendingClub is my main hobby and I have spent hundreds of hours building models, culturing relationships, and writing code to automate loan selection that is based on the premise that use of the API or CSV will allow me to be competitive in the marketplace. For two years that was true. Last week it got turned on its head. We simply can't do the type of analysis we want to do now.

[Fred]

It is not too difficult for LC to implement a "shopping cart" like that of Amazon:

http://www.amazon.com/gp/help/customer/display.html?nodeId=468468
Quote